Skip to content

feat(storage): implement per-spool capacity management, LFU eviction, and ARTICLE retrieval - #247

Merged
enteriaza merged 2 commits into
masterfrom
storage-services
Oct 24, 2026
Merged

enteriaza merged 2 commits into
masterfrom
storage-services

Conversation

@enteriaza

Copy link
Copy Markdown
Collaborator

Description

This PR extends VectorNNTP.StorageService with active per-spool capacity management, LFU-based eviction, NNTP article retrieval, and improved ingress resource control. It also consolidates CanonicalV1 delivery framing into Common and adds opt-in diagnostic benchmarks for investigating filesystem performance and resilience.

Storage Capacity and Eviction

  • Enforce per-spool capacity policies for placement and admission.
  • Introduce independent per-spool eviction monitoring and High-to-Low LFU reclamation.
  • Track article access and storage metadata in SQLite schema v3.
  • Protect articles being served through read leases.
  • Keep queued and in-flight article payloads accounted for while waiting for storage capacity.
  • Coordinate capacity recovery with blocked persistence workers.

NNTP Protocol and Ingress

  • Add ARTICLE <message-id> retrieval from published CanonicalV1 articles.
  • Preserve shared article-delivery framing and dot-stuffing semantics.
  • Add bounded article assembly and receive deadlines.
  • Improve handling of capacity backpressure and resource-constrained ingress.

Queue Management and Lifecycle

  • Strengthen graceful shutdown and queue-drain behaviour.
  • Report failed or abandoned work rather than silently treating shutdown as successful.
  • Integrate the new capacity and eviction components into StorageService dependency injection and lifecycle management.

Shared Delivery Framing

  • Move CanonicalV1 NNTP delivery framing into Common.
  • Retain BackFiller compatibility through a thin framing facade.
  • Add focused unit tests for the shared implementation.

Diagnostics and Testing

  • Add opt-in filesystem, publication-strategy, ingress-resilience, and memory-envelope benchmarks.
  • Extend regression coverage for capacity management, LFU eviction, ARTICLE retrieval, queue accounting, and shutdown.
  • Keep diagnostic benchmarks isolated from production behaviour.

Validation Notes

The current working-tree review identified previous successful test and build results, but the complete set of current changes has not yet been revalidated together. Full-tree validation should be completed before merging.

The filesystem diagnostics are investigative tools; this PR does not claim to resolve the observed filesystem tail-latency issue.

feat(storage): enforce per-spool capacity, LFU eviction, and ARTICLE retrieval
@enteriaza enteriaza self-assigned this Oct 24, 2026
@enteriaza enteriaza added the enhancement New feature or request label Oct 24, 2026

@github-advanced-security github-advanced-security AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

@enteriaza
enteriaza merged commit 5f8109d into master Oct 24, 2026
7 checks passed
@enteriaza
enteriaza deleted the storage-services branch October 24, 2026 11:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants