Forward-merge release/0.6 into main#476
Merged
Merged
Conversation
> [!WARNING] > **BREAKING CHANGE — Rust and Go stream APIs:** `LlmJsonStream` is no longer a `Pin<Box<dyn Stream<...>>>` alias; Rust callers that construct it directly or rely on the aliased boxed-stream type must migrate to `LlmJsonStream::new` or `LlmJsonStream::from_closeable`. Go `LlmStream.Close` now returns an `error`, so code that depends on its previous `Close()` method type must update. Early explicit close now finalizes the partial response and emits an interrupted lifecycle end event. #### Overview Make explicit close behavior for managed LLM streams consistent across Rust, Python, Node, Go, and the C FFI. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Replace the opaque core stream alias with a close-aware managed stream wrapper that finalizes once, emits an interrupted end event, and rejects reads after close. - Preserve Python `await stream.aclose()` while making it wait for async-generator cancellation and `aclose()` cleanup. - Add Node `await stream.close()`, Go `stream.Close() error`, and C `nemo_relay_stream_close`. - Make early close drain queued chunks and surface producer-cleanup errors consistently in each binding. - Document explicit-close behavior and early-stop examples for the primary Rust, Python, and Node bindings. - Add core, Python, Node, Go, and FFI regression coverage for idempotence, partial finalization, cleanup, and post-close exhaustion. #### Where should the reviewer start? Start with `crates/core/src/api/runtime/callbacks.rs` and `crates/core/src/stream.rs` for the shared close contract, then review the Python, Node, FFI, and Go bridge implementations. See `docs/instrument-applications/code-examples.mdx` for consumer usage. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: RELAY-496 ## Summary by CodeRabbit * **New Features** * Added explicit async stream cleanup/close APIs across Python (`aclose`), Node.js (`close`), Go (`Close() error`), and C (`nemo_relay_stream_close`), all with idempotent behavior and observable completion. * Streaming now supports cooperative cancellation and explicit completion signaling. * **Bug Fixes** * Stopping early now reliably halts producers, finalizes partial responses, and emits the expected “interrupted” end behavior. * **Documentation** * Updated migration notes, examples, and glossary to require explicit close/cleanup when exiting streams early and to clarify finalizer behavior. * **Tests** * Updated/expanded coverage to validate the new close/cancellation lifecycle. Authors: - Will Killian (https://github.com/willkill07) Approvers: - Eric Evans II (https://github.com/ericevans-nv) - Maryam Najafian (https://github.com/mnajafian-nv) - Zhongxuan (Daniel) Wang (https://github.com/zhongxuanwang-nv) URL: #465
Author
|
SUCCESS - forward-merge complete. |
github-actions
Bot
added
size:XL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to release/0.6 that creates a PR to keep main up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.