Add WebRTC ClientWindow to t2v app

docs/source/developer_guides/index.rst

@@ -41,6 +41,13 @@ Developer Guides
       How public runner names are registered, parsed, matched to manifests,
       and dispatched to integration-owned demo launch modes.
 
+   .. grid-item-card:: V2 WebRTC lifecycle
+      :link: v2_webrtc_lifecycle
+      :link-type: doc
+
+      Ownership and data flow across the CLI, application runner, persistent
+      browser window, resident model, and sequential sessions.
+
    .. grid-item-card:: Add a new method
       :link: new_integration
       :link-type: doc
@@ -70,6 +77,7 @@ generated clip, see :doc:`/quickstart/index`.
    inference_pipeline_overview
    config_system
    runner_slugs
+   v2_webrtc_lifecycle
    new_integration
    local_benchmarks
 

docs/source/developer_guides/v2_webrtc_lifecycle.md

@@ -0,0 +1,156 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# V2 WebRTC application and session lifecycle
+
+The v2 WebRTC path keeps an application and its model alive while it creates
+one session at a time from browser requests. A session owns one rollout; it
+does not own the server or reload the model.
+
+The Self-Forcing entry command remains:
+
+```bash
+uv run --project integrations_v2/t2v_self_forcing flashdreams-run-v2 \
+  t2v-self-forcing --mode webrtc --port 8080
+```
+
+Open the printed URL, enter a prompt, and select **New session**. Closing or
+refreshing the page ends the active rollout, but the process and loaded model
+remain available for the next request. Ctrl-C ends the application.
+
+## Ownership
+
+| Component | Owns | Does not own |
+| --- | --- | --- |
+| `cli.py` | Argument parsing, application and window construction, final runner cleanup | Model state, session state, browser events |
+| `ApplicationRunner` | The initialized application and the serial application-level session loop | Browser event types or per-step scheduling |
+| `run_session()` | Exactly one session, its generation loop, its I/O thread, and session cleanup | The application or a persistent window after a successful handoff |
+| `WebRTCClientWindow` | The WebRTC server and the thread-safe browser-event queue | Application or session state |
+| `WebRTCServer` | HTTP, one active peer connection, its data channel, and its media track | Runtime lifecycle decisions |
+| `T2VApplication` | The loaded model pipeline shared by every session | A rollout cache or browser connection |
+| `T2VSession` | One prompt and one rollout cache | Model loading or server lifetime |
+
+The CLI constructs the window, then transfers its lifetime cleanup
+responsibility to the runner. While the server is idle, the runner's calling
+thread opens and polls the window. During a session, `run_session()` lends the
+window to one dedicated I/O thread, which performs the meaningful close when
+the window should end. The runner does not touch it again until that thread has
+stopped and the old session has closed. Its final, idempotent close is a fallback
+for setup failures and interrupted handoffs. This is sequential access, not
+concurrent access.
+
+## Data flow
+
+1. `cli.py` selects the `t2v-self-forcing` application and the `webrtc` window
+   mode. Arguments after `--` belong to the application.
+2. `ApplicationRunner.init()` calls `T2VApplication.init()`. The pipeline is
+   set up, moved to its device, evaluated, and retained by the application.
+3. The WebRTC mode creates `WebRTCClientWindow`, which starts its server thread
+   and exposes the browser URL.
+4. `ApplicationRunner.run()` resolves the CLI's partial `SessionDescRequest`
+   against the initialized application's default `SessionDesc`.
+5. A window whose `keeps_open_between_sessions` capability is true opens with
+   that resolved stream format before a session exists. This lets the browser
+   negotiate WebRTC and send its first request without loading a session cache
+   first. Other windows start the resolved session immediately.
+6. The server validates every data-channel message and invokes the callback
+   registered by `WebRTCClientWindow`. The callback only appends the event to a
+   thread-safe queue.
+7. While idle, `wait_for_new_session()` drains that queue. It translates the
+   latest close/new-session transition into a complete `SessionDesc` and
+   returns only that description. `ApplicationRunner` therefore never parses a
+   `UserInputEvent`.
+8. The application validates the description and creates a `T2VSession` over
+   its resident pipeline. `T2VSession.init()` creates only the per-rollout
+   prompt/cache state.
+9. `run_session()` starts one I/O thread. That thread opens the same window for
+   the actual session boundary, drains input each UI tick, calls `step_ui()`,
+   and writes completed results. The calling thread runs model steps.
+10. A new-session event stops that rollout and returns the requested next
+    `SessionDesc`. A close or natural completion returns no replacement. In
+    a persistent window the runner keeps it open, closes the old session, and
+    either starts the replacement or waits for another browser request.
+11. Ctrl-C closes the peer/server and then the application. Releasing the
+    application drops the one resident pipeline after every session cache has
+    already been released.
+
+## Why the WebRTC window is opened twice
+
+The two calls mark different boundaries:
+
+- The first call prepares the fixed stream format so a browser can connect
+  while no session exists.
+- The per-session call discards source frames queued by the previous rollout.
+  Event timestamps retain one window-lifetime monotonic origin, so opening a
+  replacement cannot reorder buffered events from the old and new browser.
+
+They do not create two servers or two peer connections. A connected peer is
+reused. The application must either accept the already-resolved `SessionDesc`
+or reject it; it cannot silently change width, height, layout, or playback rate
+after the browser stream was prepared.
+
+## Lifecycle event ordering
+
+Close and new-session events can arrive in one drained batch during a page
+refresh. Their order is meaningful, so the latest transition wins:
+
+- close, then new session: start the new browser's request;
+- new session, then close: cancel the request because that browser went away.
+
+Any close or replacement also advances the session generation. A model step
+that was already running may finish, but its result is tagged with the old
+generation and is not written. The WebRTC media track similarly clears frames
+that it has not yet handed to the encoder. At most one already-encoded frame
+can still be in flight; a transport cannot recall a frame it has sent.
+
+## WebRTC connection lifecycle
+
+Offer negotiation is serialized, and only one browser is admitted. Data
+channel and peer callbacks capture the peer that registered them, so callbacks
+from an old page cannot close or mutate its replacement. Closing the active
+data channel releases the peer immediately, allowing a refreshed page to
+negotiate without an indefinite series of HTTP 409 responses.
+
+The new-session button starts enabled and reads **Opening...** while signaling
+is in progress. A click during that interval keeps the latest valid prompt in
+the page and sends it as soon as the data channel opens. The UI therefore does
+not make model loading or a brief reconnect look like an unavailable action.
+
+The media track uses `frames_per_second_for_step`, the generated video's
+playback rate. `frames_per_second_for_ui` controls only how often the runtime
+polls input and presents completed work.
+
+The media source queue is currently unbounded. T2V rollouts are finite, and a
+session replacement clears frames that have not reached the encoder, so its
+size remains bounded by a rollout in the current application. A future
+long-running producer will need an explicit block-or-drop policy chosen for
+that application's latency requirements.
+
+## Why these boundaries are useful
+
+- The core generation pipeline and checkpoint load once and stay resident
+  across browser refreshes and sequential prompts. The existing Wan pipeline
+  may release and reload its one-shot text encoder after prompt encoding to fit
+  within GPU memory; it does not reload the diffusion model between sessions.
+- Session cleanup is complete before the next session is created, so rollout
+  caches cannot overlap accidentally.
+- WebRTC code transports validated events but never makes application
+  lifecycle decisions.
+- `ApplicationRunner.run()` visibly owns the possibly multi-session run, while
+  `run_session()` visibly owns exactly one rollout.
+- Failure paths keep the first useful exception while still releasing the
+  session I/O thread, WebRTC server thread, window, and partially initialized
+  model.
+
+`SessionDescRequest` remains separate from `SessionDesc` on purpose: the former
+means “only the fields the CLI explicitly supplied,” while the latter is the
+fully resolved contract shared by the application, session, and window. No
+additional host, session-runner class, or WebRTC-specific lifecycle wrapper is
+needed.
+
+The bundled browser page is currently prompt-oriented because text-to-video is
+the application that needs client-created sessions today. A generic UI schema
+should be introduced only when another application has a concrete, different
+request format.

flashdreams/flashdreams/api_v2/application.py

@@ -27,17 +27,19 @@ def init(self, commandline_args: Sequence[str]) -> None:
         """Parse application arguments and validate startup state."""
         ...
 
-    def session_desc(self) -> SessionDesc | None:
-        """Return the description of a session this application would generate.
+    def default_session_desc(self) -> SessionDesc | None:
+        """Return this initialized application's default session description.
 
-        A caller has to describe a session before there is one to describe, and
-        only the application knows what its model was trained for. Asked before
-        :meth:`init`, so describing a session costs nothing.
+        The application owns its model's output requirements and defaults, such
+        as its layout, preferred dimensions, and frame rate. The runtime asks
+        after :meth:`init`, then applies the caller's explicit requests to this
+        default before calling :meth:`create_session`. That method accepts the
+        resolved description or rejects it; it does not silently change the
+        stream the runtime already asked the client window to prepare for.
 
         Returns:
-            The session to create when nobody asks for another, or ``None``,
-            the default, from an application that generates whatever it is
-            asked for. Its caller describes the session instead.
+            The default session description, or ``None`` when the application
+            has no output requirements and accepts the runtime's defaults.
         """
         return None
 
@@ -49,8 +51,7 @@ def create_session(self, session_desc: SessionDesc) -> ISession:
             session_desc: Session the runtime is asking for.
 
         Returns:
-            A session for ``session_desc``, resolved to what this application can
-            actually produce.
+            A session that produces ``session_desc``.
 
         Raises:
             ValueError: The application cannot honour ``session_desc``.

flashdreams/flashdreams/api_v2/client_window.py

@@ -3,7 +3,7 @@
 
 """Client window abstract interface."""
 
-from abc import ABC
+from abc import ABC, abstractmethod
 
 from .input_source import InputSource
 from .output_sink import OutputSink
@@ -13,14 +13,37 @@ class IClientWindow(InputSource, OutputSink, ABC):
     """Handle application input and output for one client window.
 
     The runtime opens the window with the session's description, then reads input
-    and writes results until the run ends, and closes it then. A window stays open
-    across a session reset.
+    and writes results until the run ends. A window stays open across a session
+    reset. When the client asks for a replacement session, the runtime closes the
+    old session and opens the same window with the replacement's description. A
+    persistent window can also remain open between sessions while the runtime
+    waits for another client request.
 
     A window does not describe the output shape. The session does, and the window
     is given that description in :meth:`OutputSink.open`.
 
-    One thread makes every call on a window, so an implementation needs no
-    locking except when its backend delivers input from another thread.
+    One runtime thread at a time makes every call on a window, so an
+    implementation needs no locking except when its backend delivers input from
+    another thread.
 
     Created by the runtime, never by an application.
     """
+
+    keeps_open_between_sessions: bool = False
+    """Whether sessions start on demand and the window persists between them.
+
+    When false, the runtime starts the resolved initial session immediately and
+    returns after it ends unless the client requested a replacement. When true,
+    the runtime opens the window before creating a session, waits for a client
+    request, and returns to waiting after completion or disconnection.
+    """
+
+    @abstractmethod
+    def close(self) -> None:
+        """Release this window's resources.
+
+        This must be safe before :meth:`OutputSink.open` and after an earlier
+        call. The session loop performs the meaningful close on its I/O thread;
+        the application runner calls it again as a lifetime-cleanup fallback.
+        """
+        ...

flashdreams/flashdreams/api_v2/session.py

@@ -26,9 +26,11 @@ class ISession(ABC):
 
     @abstractmethod
     def init(self) -> None:
-        """Load the model and anything else this run needs.
+        """Prepare the state owned by this run.
 
-        Must not do client I/O, since this can run before a client connects.
+        Shared model loading belongs to the application. This method prepares
+        per-session state such as an encoded prompt or KV cache. It must not do
+        client I/O, since this can run before a client connects.
         """
         ...