Skip to content

fix(process): do not start a shell for a session that is already gone - #372

Open
kshivang wants to merge 1 commit into
masterfrom
fix/pty-spawn-after-teardown
Open

fix(process): do not start a shell for a session that is already gone#372
kshivang wants to merge 1 commit into
masterfrom
fix/pty-spawn-after-teardown

Conversation

@kshivang

Copy link
Copy Markdown
Owner

Seen in a live install

Updating terminal-tab reloaded the plugin, and during the teardown a pty spawn arrived at the already-closed classloader:

ClassNotFoundException: Plugin classloader for '...terminaltab' is UNLOADED;
refusing to resolve 'com.pty4j.util.PtyUtil' against the host classloader.
Something still referenced the plugin after it was unloaded - that reference is the bug.
  at com.pty4j.unix.PtyHelpers.<clinit>(PtyHelpers.java:206)
  at com.pty4j.PtyProcessBuilder.start(PtyProcessBuilder.java:175)
  at DesktopProcessService.spawnProcess(PlatformServices.desktop.kt:102)
  at TabController$initializeTerminalSession$1.invokeSuspend(TabController.kt:1546)

The host's guard was right to refuse. The first pty spawn in that classloader happened during its own teardown, so PtyHelpers ran its static initialiser against a loader that was already closed.

Three faults, not one

1. The spawn was attempted for a cancelled session.

TerminalTab.dispose does call coroutineScope.cancel(). But cancellation is cooperative and PtyProcessBuilder.start() is a blocking JNI chain with no suspension point, so a coroutine already inside it never notices. Session init does real work before that call - environment assembly, shell-integration injection - and that whole span is when a tab can be disposed.

currentCoroutineContext().ensureActive() before the call narrows the window from all of session init to an instruction gap. It does not close it, and nothing in the code or the tests claims otherwise.

2. catch (e: Exception) did not catch the failure.

A closed classloader raises NoClassDefFoundError / ExceptionInInitializerError. Those are Errors, not Exceptions, so the documented "returns null on failure" contract broke in exactly the case that produced the stack above: rather than a connection error on the tab, the throwable escaped to the host. Now catch (t: Throwable).

3. Cancellation would have been swallowed by that same clause and reported as "Failed to spawn process" on a tab already on its way out. The CancellationException clause sits above it and rethrows.

Why the pty start is now injectable

One reason: a LinkageError cannot be provoked through the real PtyProcessBuilder, so without a seam the Throwable-vs-Exception distinction is untestable - and an untested catch clause is how it came to be Exception-only in the first place. The default argument is the original builder chain, unchanged.

This was not hypothetical caution. My first version of the cancellation test used withContext(cancelledJob), which throws at the withContext boundary before spawnProcess runs, so it passed against the Exception-only catch and proved nothing. The seam is what made both properties real.

Tests

1089 compose-ui tests, all green. Verified by reverting each change separately - each fails exactly one test and no others:

Reverted Fails
catch (t: Throwable)catch (t: Exception) a linkage error resolves to null rather than escaping
the CancellationException clause cancellation from inside the spawn propagates
ensureActive() a cancelled caller does not get a process

I also deleted a test I had written with assertFalse(false, "placeholder…") in it. It asserted nothing; the limitation it was gesturing at is stated in the KDoc instead.

Deliberately not done

Eagerly warming the pty4j classes at startup would remove this stack entirely, since <clinit> could never run late. It costs class loading plus a JNA library load on every launch for users who never open a terminal, which is the wrong trade for a startup path. The two guards make a late spawn resolve cleanly instead of crashing.

Seen in a live install. Updating terminal-tab reloaded the plugin, and during
the teardown a pty spawn arrived at the closed classloader:

    ClassNotFoundException: Plugin classloader for '...terminaltab' is
    UNLOADED; refusing to resolve 'com.pty4j.util.PtyUtil' against the host
    classloader. Something still referenced the plugin after it was unloaded.
      at com.pty4j.unix.PtyHelpers.<clinit>
      at DesktopProcessService.spawnProcess
      at TabController$initializeTerminalSession$1.invokeSuspend

The first pty spawn in that classloader happening during its own teardown, so
PtyHelpers ran its static initialiser against a loader that was already closed.

Three faults, each fixed and each pinned by a test that fails without it.

The spawn was attempted for a cancelled session. `TerminalTab.dispose` does
cancel `coroutineScope`, but cancellation is cooperative and
`PtyProcessBuilder.start()` is a blocking JNI chain with no suspension point,
so a coroutine already inside it never notices. Session init does real work
first - environment assembly, shell-integration injection - and that whole span
is when a tab can be disposed. `ensureActive()` before the call narrows it to
an instruction gap. It does not close it, and nothing here claims otherwise.

`catch (e: Exception)` did not catch the failure. A closed classloader raises
NoClassDefFoundError / ExceptionInInitializerError, which are Errors, so the
documented "returns null on failure" contract broke in precisely the case that
produced the stack above: instead of a connection error on the tab, the
throwable escaped to the host. Now `catch (t: Throwable)`.

Cancellation would have been swallowed by that same clause and reported as
"Failed to spawn process" on a tab already on its way out, so the
CancellationException clause sits above it and rethrows.

The pty start is injectable now, for one reason: a LinkageError cannot be
provoked through the real PtyProcessBuilder, so without a seam the
Throwable-vs-Exception distinction is untestable - and an untested catch clause
is how it came to be Exception-only. Verified by reverting each of the three
changes separately; each fails exactly one test and no others.

Deliberately not done: eagerly warming the pty4j classes at startup would
remove this stack entirely, but it costs class loading and a JNA library load on
every launch for users who never open a terminal. The two guards above make a
late spawn resolve cleanly instead.
@claude

claude Bot commented Aug 21, 2026

Copy link
Copy Markdown

Review

The diagnosis is right and the write-up is unusually good — the three faults are genuinely distinct, the clause ordering (CancellationException above Throwable) is correct, and the revert matrix is the right way to prove each test pins one change. The seam justification holds up: a LinkageError really can't be provoked through the real PtyProcessBuilder.

Two things I'd want fixed before merge, plus some smaller notes.


1. an active caller still gets a process will fail on the Windows CI leg

.github/workflows/test.yml runs :compose-ui:desktopTest across [macos-latest, ubuntu-latest, windows-latest]. /bin/sh doesn't exist on Windows, so spawnProcess returns null there and assertNotNull(handle, "a normal spawn was refused") trips — red CI on a change that is correct.

The repo already has the pattern for this: daemon/TerminalSessionCoreTest.kt guards every real-pty test with

if (ShellCustomizationUtils.isWindows()) return  // POSIX sh marker; pty differs on Windows

Worth the same guard on a command that cannot start resolves to null rather than throwing too — whether a bogus path fails at PtyProcessBuilder.start() or later is a ConPTY-vs-forkpty detail, not something the contract pins. The other three tests use the seam or never reach the spawn, so they're platform-clean as written.

2. The daemon path converts the new cancellation into a spurious error

ensureActive() now throws CancellationException out of spawnProcess on a path that previously just proceeded. The three UI call sites are fine — TabController.kt:1500, TabController.kt:1729 and EmbeddableTerminal.kt:1206 each have a catch (e: CancellationException) clause that rethrows, which is presumably why this looked clean.

daemon/TerminalSessionCore.kt does not. start()'s init coroutine has only catch (e: Exception) at line 269, and close() calls scope.cancel() at line 358. So a close arriving during buildEnvironment() now lands as:

_connectionState.value = State.Error("Terminal initialization failed: ${e.message}")
connected.complete(false)
log.error("session {} init failed: {}", id, e.message)

That is exactly fault 3 from your description — a session on its way out reported as a failure — just relocated to the daemon. It's also a swallowed CancellationException, so the coroutine completes normally inside a cancelled scope. The pre-existing clause was merely unreachable for this cause; the guard makes it reachable. A catch (e: CancellationException) { connected.complete(false); close(); throw e } above line 269 would match what the other three sites already do.

(The spawn avoidance is a straight improvement for the daemon — TerminalSessionCore.kt:196 currently spawns and then kills via the if (closed) branch. It's only the reporting that regresses.)

3. catch (t: Throwable) is wider than the fault it targets

The comment argues the LinkageError case convincingly, but the clause as written also turns OutOfMemoryError and StackOverflowError into "return null, render a connection error" with a printStackTrace() — the JVM carries on in an undefined state and the only trace goes to stderr. Two clauses would encode the intent more precisely than the comment does:

} catch (e: LinkageError) {   // closed plugin classloader: NoClassDefFoundError / ExceptionInInitializerError
} catch (e: Exception) {

LinkageError covers both throwables named in the stack, and a VirtualMachineError keeps propagating.

4. com.pty4j.PtyProcess in a public constructor signature

pty4j is implementation at compose-ui/build.gradle.kts:106, so the type isn't on any consumer's compile classpath. Every caller is inside compose-ui today (PlatformServices.desktop.kt:20 and the test), so nothing breaks — but the seam does put an implementation-scoped dependency into the module's public API. Making the parameter (or a fun interface wrapping it) internal, or having the lambda return ProcessService.ProcessHandle instead, keeps the testability without exporting pty4j.

5. Minor

  • assertFalse is imported in ProcessSpawnTeardownTest.kt but unused — leftover from the placeholder test you removed. Also a stray blank line before the closing brace.
  • t.printStackTrace() — the daemon side of this codebase uses slf4j. Since you're already rewriting the clause, a logger call would make the plugin-teardown case visible to a host that captures logs rather than only to stderr.
  • The startPty seam takes both command and config when command is derived from config; folding the arrayOf(...) into the lambda would make the parameter list self-evident. Cosmetic.

Nothing above touches the core change, which I think is correct. Item 1 is the merge blocker; item 2 is the one that would show up in a live daemon install the same way the original stack did.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant