Skip to content

Keep a session visible in Picture in Picture while another app is in front #39

Open

Description

@m96-chan

The reason to run a terminal on a phone is usually something that takes a while — a build, a deploy, a tail -f. Right now watching one means leaving iOSSH in front and doing nothing else. The session already survives backgrounding; what is missing is being able to see it.

README.md already states the retention half: "Screen lock and backgrounding retain the current SSH session and terminal contents." Picture in Picture would make retained state visible rather than only recoverable.

What this would be

A read-only window. AVPictureInPictureController gives no text input and no scrolling — the standard PiP controls are all the user gets. The value is watching output, not driving it. Tapping the window returns to the app, which is the right affordance for "something finished, go look".

Available from iOS 15 through AVPictureInPictureController.ContentSource(sampleBufferDisplayLayer:playbackDelegate:), so the iOS 17 deployment target in project.yml is not a constraint.

What stands in the way

Four things, in rough order of how much they decide whether this is worth doing.

1. It needs the audio background mode, and this is not an audio app. AVPictureInPictureController requires UIBackgroundModes to include audio. App/Info.plist declares no background modes at all today. A terminal claiming the audio background mode is the kind of thing review asks about, and docs/APP_STORE.md already notes that the encryption and privacy answers were deliberately not guessed. This is the open question that should be settled before any code is written — everything below is wasted if the entitlement is refused.

2. The renderer deliberately stops when the app stops being active.

TerminalView.swift:250  addObserver(... UIApplication.willResignActiveNotification ...)
TerminalView.swift:863  @objc private func resignedActive() { suspend() }
TerminalView.swift:554  renderer?.isActive = false

README.md §5 states the rule this implements: "Stop rendering when the app enters the background (GPU work is not allowed in the background)." PiP is the sanctioned exception to that rule, but only while PiP is actually running — the condition has to become "not active and not in PiP", not "not active".

3. The model stops producing snapshots too, so even a renderer that kept drawing would redraw the same frame:

ConnectionModel.swift:687  guard isVisible, !isInBackground, snapshotTask == nil else { return }
ConnectionModel.swift:349  isInBackground = true; ...; snapshotTask?.cancel()

Only one session should keep publishing — the one in the PiP window. The other retained iPad tabs should stay suspended, or backgrounding the app turns into four live parser pipelines.

4. The frames have to become sample buffers. PiP consumes CMSampleBuffers through an AVSampleBufferDisplayLayer; the terminal draws through Metal into an MTKView. The path is an offscreen render target backed by a CVPixelBuffer through a CVMetalTextureCache, wrapped and enqueued. The renderer is already damage-driven and hands the view immutable snapshots, so a second destination is not a rewrite — but it is a second destination at a different size, which the glyph atlas and cell metrics are not currently asked to produce.

Worth deciding early

  • Which session, on an iPad with up to four tabs. The visible one at the time PiP starts is the obvious answer, but it should be a decision rather than a default that falls out of the code.
  • What it costs in battery. This keeps a Metal renderer and an SSH read loop alive behind another app. A tail -f on a chatty log is the worst case, and the existing credit gate is the only thing bounding it.
  • What happens on disconnect while in PiP, where there is nowhere to show a banner or a Reconnect button.

Priority

Low, as filed. It is a feature rather than a defect and 0.1.0 does not need it. The label is worth raising if the audio background mode turns out to be acceptable to review, because that is the part that decides whether this is a weekend or a dead end.

Activity

  1. m96-chan commented on Sep 15, 2026

    @m96-chan
    OwnerAuthor

    Parked on branch pip-terminal rather than deleted.

    It builds and installs, and the offscreen render path is the part that took the work: MetalRenderer.render(into:background:completion:) reuses encodeFrame, so the terminal is drawn a second time into a CVPixelBuffer without duplicating any drawing code.

    The window has never appeared on a device. Two defects were found and fixed along the way, and neither of them is known to have been the reason:

    • A deadlock. Frames were only produced once PiP was active, but AVPictureInPictureController will not report isPictureInPicturePossible until its layer has content, and the system only starts a window automatically when it is possible. No frames, never possible, never starts.
    • The source layer covered the terminal. A sublayer composites above its parent's own contents, and this one's parent is the Metal layer the terminal draws into — so sizing it to the view put a one-frame-per-second feed over the live terminal. It is transparent now.

    The gap that the iPad question exposed is also fixed: WorkspaceSessionStore.enterBackground() stops every session publishing, so the window would have frozen on its last frame. Only the session the window is showing keeps producing frames, which is the frontmost tab because the workspace hands every tab the same renderer.

    Why it is parked

    The comparison that prompted it: on Android, enterPictureInPictureMode() hands the system your actual UI and it scales it down — no video pipeline, no entitlement, no audio session, and RemoteAction buttons if you want them. On iOS, PiP only accepts AVPlayerLayer or AVSampleBufferDisplayLayer, so what this does is render the terminal a second time and present it as a live video stream. It is a facsimile, and what it buys is a small window nobody can type into.

    Against that: UIBackgroundModes: audio declared by an app that plays nothing, a second render path, and an audio session to keep active. The review question this issue opens with is still unanswered and still decides everything.

    #47 takes the part of #46 that needed none of this — a background task assertion, about thirty seconds, no entitlement. That is where the common case lives.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpriority: lowNice to have; no release impact

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions