Skip to content

stream: fix Utf8Stream flush handling - #66473

Open
jasnell wants to merge 1 commit into
nodejs:mainfrom
jasnell:jasnell/utf8stream-flush-fix
Open

jasnell wants to merge 1 commit into
nodejs:mainfrom
jasnell:jasnell/utf8stream-flush-fix

Conversation

@jasnell

@jasnell jasnell commented Oct 3, 2026

Copy link
Copy Markdown
Member

Fix several issues with Utf8Stream flushing:

  • flush() now writes buffered data regardless of minLength and invokes the callback only after pending writes complete, including when minLength is zero and a write is in flight.
  • Multiple concurrent flush() calls are tracked correctly and end() waits for pending flushes before closing.
  • flushSync() throws ERR_INVALID_STATE if called while an asynchronous write is in progress instead of corrupting output.
  • Periodic flushes no longer stack up while a flush is pending.
  • fsync is skipped for stdout/stderr file descriptors.
  • reopen(), end() and destroy() behave correctly when the stream is destroyed while still opening.

Separated out from #65840

@jasnell
jasnell requested review from mcollina and ronag October 3, 2026 01:48
@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/streams

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. stream Issues and PRs related to Node.js streams. labels Oct 3, 2026
@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.40741% with 34 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.42%. Comparing base (9746ebc) to head (6a11fbc).

Files with missing lines Patch % Lines
lib/internal/streams/fast-utf8-stream.js 87.40% 34 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #66473      +/-   ##
==========================================
+ Coverage   90.40%   90.42%   +0.02%     
==========================================
  Files         791      791              
  Lines      276120   276315     +195     
  Branches    53022    53085      +63     
==========================================
+ Hits       249618   249853     +235     
+ Misses      16897    16871      -26     
+ Partials     9605     9591      -14     
Files with missing lines Coverage Δ
lib/internal/streams/fast-utf8-stream.js 88.62% <87.40%> (+5.55%) ⬆️

... and 31 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mcollina

This comment was marked as resolved.

@mcollina

This comment was marked as outdated.

@jasnell
jasnell force-pushed the jasnell/utf8stream-flush-fix branch from 8034879 to d52d366 Compare October 4, 2026 11:13
@jasnell

jasnell commented Oct 4, 2026

Copy link
Copy Markdown
Member Author

... why? If it’s a file, this is useful.

You're right. changing.

Updated for both issues

mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 4, 2026
- flush() with minLength 0 now waits for in-flight writes before
  calling the callback.
- end() waits for pending flush callbacks before closing.
- Concurrent flush() calls are tracked with counters.
- flush() no longer fsyncs stdout/stderr and ignores EBADF.
- Periodic flushes do not stack while a flush is pending.
- end()/reopen()/destroy() while opening no longer double-close or throw;
  pending flushes fail with an error when the stream is destroyed.

The flushSync() change from the upstream PR (throwing while a write
is in progress) is intentionally left out as it is breaking.
mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 4, 2026
- flushSync() preserves ordering where it can: it cancels a pending
  EAGAIN/EBUSY retry and writes everything in order, writes the
  remainder of a partial write first when called from a 'write'
  listener, and writes to the previous file while reopening. With an
  async write in flight, only the queued data is written.
- 'write' is emitted after the written bytes are released.
- flushBufferSync() keeps _lens in sync when re-queuing _writingBuf.
- flush() fsyncs stdout/stderr again (useful when redirected to a
  file) and ignores EBADF/EINVAL/ENOTSUP/EOPNOTSUPP/EROFS from fsync.

@mcollina mcollina left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 5, 2026
Align with nodejs/node#66473: when destroy() is called while a write
is in flight, a pending flush() now succeeds if all the data was
written, and fails only if data is left. A pending EAGAIN/EBUSY retry
is cancelled on destroy instead of writing to the closed fd.
@mcollina

mcollina commented Oct 5, 2026

Copy link
Copy Markdown
Member

This needs a rebase now

mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 5, 2026
* fix: flush handling (port of nodejs/node#66473)

- flush() with minLength 0 now waits for in-flight writes before
  calling the callback.
- end() waits for pending flush callbacks before closing.
- Concurrent flush() calls are tracked with counters.
- flush() no longer fsyncs stdout/stderr and ignores EBADF.
- Periodic flushes do not stack while a flush is pending.
- end()/reopen()/destroy() while opening no longer double-close or throw;
  pending flushes fail with an error when the stream is destroyed.

The flushSync() change from the upstream PR (throwing while a write
is in progress) is intentionally left out as it is breaking.

* fix: port updated flushSync and fsync handling from nodejs/node#66473

- flushSync() preserves ordering where it can: it cancels a pending
  EAGAIN/EBUSY retry and writes everything in order, writes the
  remainder of a partial write first when called from a 'write'
  listener, and writes to the previous file while reopening. With an
  async write in flight, only the queued data is written.
- 'write' is emitted after the written bytes are released.
- flushBufferSync() keeps _lens in sync when re-queuing _writingBuf.
- flush() fsyncs stdout/stderr again (useful when redirected to a
  file) and ignores EBADF/EINVAL/ENOTSUP/EOPNOTSUPP/EROFS from fsync.

* fix: settle pending flushes after an in-flight write on destroy

Align with nodejs/node#66473: when destroy() is called while a write
is in flight, a pending flush() now succeeds if all the data was
written, and fails only if data is left. A pending EAGAIN/EBUSY retry
is cancelled on destroy instead of writing to the closed fd.
mcollina added a commit to pinojs/sonic-boom that referenced this pull request Oct 5, 2026
Align with nodejs/node#66473: destroy() while the file is opening now
sets `destroyed` right away, so write()/end()/flush() behave as on any
destroyed stream instead of silently accepting data that is dropped.
A pending flush() is failed with an 'error' event.
Fix several issues with `Utf8Stream` flushing:

* `flush()` now writes buffered data regardless of `minLength`
  and invokes the callback only after pending writes complete,
  including when `minLength` is zero and a write is in flight.
* Multiple concurrent `flush()` calls are tracked correctly and
  `end()` waits for pending flushes before closing.
* `flushSync()` throws `ERR_INVALID_STATE` if called while an
  asynchronous write is in progress instead of corrupting output.
* Periodic flushes no longer stack up while a flush is pending.
* `fsync` is skipped for stdout/stderr file descriptors.
* `reopen()`, `end()` and `destroy()` behave correctly when the
  stream is destroyed while still opening.

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode
@jasnell
jasnell force-pushed the jasnell/utf8stream-flush-fix branch from d52d366 to 6a11fbc Compare October 5, 2026 21:59
@jasnell

jasnell commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

@mcollina rebased... PTAL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ci PRs that need a full CI run. stream Issues and PRs related to Node.js streams.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants