Skip to content

Reduce Allocations in StorageService NNTP Ingestion - #245

Merged
enteriaza merged 3 commits into
masterfrom
vatp-removal
Oct 23, 2026
Merged

enteriaza merged 3 commits into
masterfrom
vatp-removal

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Description

Optimises the StorageService NNTP ingestion hot path to reduce per-article allocations while preserving protocol correctness, buffer ownership, and queue handoff semantics.

Changes

  • Reuse a session-local read scratch buffer.
  • Reduce the initial article assembly buffer rental.
  • Avoid unnecessary command-line and message-ID allocations.
  • Skip debug hex formatting when debug logging is disabled.
  • Add an in-memory ingestion allocation benchmark.
  • Isolate transport counter tests from parallel execution and restore exact assertions.
  • Preserve pipelined TAKETHIS handling and buffer ownership guarantees.

Validation

  • 418 StorageService tests passed.
  • Focused NNTP tests passed, including pipelined TAKETHIS and transport timeout coverage.
  • Release x64 solution build succeeded.
  • dotnet format --verify-no-changes passed.

Performance Notes

The in-memory benchmark measured a reduction from approximately 5,659 to 3,259–3,260 allocated bytes per article, around 42% less allocation under the same harness.

This is a relative measurement; the harness also includes client-side status-line processing and allocations, so the absolute figure is not server-only.

TLS throughput measurements remain inconclusive. Five optimised-build runs varied from 10,624 to 16,613 accepted articles over three seconds, so no definitive throughput improvement is claimed.

- Replace the VATP listener and session with NNTP STREAMING ingestion
- Route TAKETHIS articles through the existing accepted-article queue
- Remove obsolete VATP transport components, tests, and configuration
- Fix pipelined command buffer corruption and oversized article handling
- Add NNTP ingestion and large-residue regression tests
- Update StorageService lifecycle tests and protocol documentation
- Preserve the existing fleet advertisement wire contract
- Reuse session-local read scratch buffer
- Reduce initial article assembly buffer rental
- Avoid unnecessary command and message-ID allocations
- Skip debug hex formatting when debug logging is disabled
- Add in-memory ingestion allocation measurements
- Isolate transport counter tests and restore exact assertions
- Preserve NNTP framing and buffer ownership semantics
@enteriaza enteriaza self-assigned this Oct 23, 2026
@enteriaza enteriaza added the enhancement New feature or request label Oct 23, 2026
- Remove forced garbage collection from allocation measurements
- Remove an unused destuffed article assignment
- Document expected cancellation during benchmark shutdown
- Preserve benchmark measurement and ingestion behaviour
@enteriaza
enteriaza merged commit 6ca0958 into master Oct 23, 2026
7 checks passed
@enteriaza
enteriaza deleted the vatp-removal branch October 23, 2026 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants