feat(storage): implement bounded in-memory DLR storage

[lykakis]

Parent epic: #337
Delivery step 7; after file-backed DLR storage #340 and before PostgreSQL outbound durability #333

Status

Delivery step 7 in #337. Begin this issue after #340 is merged and the shared DLR backend-selection, readiness, and operation-instrumentation structure is stable.

This issue is not a blocker for filesystem SMS pending storage in #339, file-backed router/routed state in #345, or the database-free filesystem profile completed by #340.

Problem

Sendium supports durable DLR correlation and downstream delivery through PostgreSQL and, after #340, file-backed storage. Some deployments do not require DLR state to survive restart and prefer bounded process-local handling without opening either durable DLR store.

Disabling DLR handling entirely is not equivalent: disabled mode does not correlate provider message IDs or deliver HTTP and downstream SMPP receipts while the process is running.

Goal

Implement a configurable, bounded in-memory backend for the existing DlrStorage behavior.

Memory mode must provide normal DLR correlation and downstream delivery while Sendium remains active, with explicit capacity and retention limits. It is intentionally non-durable, single-process, and unsuitable for multiple replicas.

This issue does not change the default DLR backend. PostgreSQL and file-backed DLR storage remain the durable choices.

Durability boundary

Outbound-message durability and DLR durability remain separate selections:

• Outbound storage determines whether an accepted SMS survives until provider processing reaches a terminal outcome.
• DLR storage determines whether provider correlations and pending downstream HTTP/SMPP deliveries survive afterward.

With filesystem outbound SMS storage from #339 and #345 and memory DLR storage, an accepted SMS remains recoverable until provider processing and the required in-memory DLR handoff complete. After the pending SMS file is removed, a restart may lose its provider correlation or pending downstream receipt.

With memory SMS processing and memory DLR storage, both boundaries are process-local. After #333, PostgreSQL outbound storage may also be combined with memory DLR storage where that profile is explicitly supported.

Backend selection

The DLR backend selector must support:

• postgresql: existing durable PostgreSQL handling.
• file: durable single-process local handling from feat(storage): implement file-backed DLR storage #340.
• memory: bounded process-local handling from this issue.
• disabled: no Sendium-owned DLR correlation or downstream delivery.

Backend selection is startup-only. Selecting memory must not initialize a PostgreSQL DLR datasource or open the file-backed DLR store. Unsupported SMS/DLR profile combinations fail startup rather than silently weakening durability.

Functional behavior

While the process remains active, memory mode must:

• Correlate provider message IDs with gateway message IDs.
• Support multiple provider IDs for one multipart submission.
• Apply first-terminal-DLR-wins semantics.
• Preserve intermediate provider states without prematurely consuming correlations.
• Create provider-rejection results for supported downstream channels.
• Deliver and retry HTTP callbacks.
• Deliver downstream SMPP receipts.
• Retain pending SMPP receipts while the client is disconnected.
• Persist delivery-attempt fencing in memory so stale callbacks cannot complete or retry a newer attempt.
• Remove expired correlations and pending deliveries through periodic cleanup.
• Never fall back silently to disabled or another backend after initialization failure.

All correlations, reservations, callbacks, retries, attempt state, and pending SMPP receipts are lost when the process stops.

Capacity and admission

Capacity counts logical gateway messages, not provider-correlation entries. A multipart message with multiple provider message IDs consumes one tracked-message slot.

Reservation must be atomic under concurrent submissions and must never exceed the configured maximum.

For a new HTTP or SMPP submission requesting a DLR:

1. Reserve one memory DLR slot.
2. Admit the SMS through the selected outbound backend.
3. Send protocol success only after both operations succeed.
4. Release the reservation if outbound admission fails.

If no slot is available, reject or backpressure the DLR-requesting submission with the configured protocol response. Non-DLR submissions continue subject to normal outbound limits.

After restart, the memory backend starts empty. A pending SMS recovered from filesystem or PostgreSQL outbound storage must acquire a new DLR reservation before provider submission. If capacity is unavailable, keep the outbound record pending and retry reservation later rather than submitting an SMS whose requested DLR cannot be tracked.

Do not silently evict active entries to make room. Release a reservation when:

• Processing terminates without requiring further DLR state.
• The provider correlation expires before terminal resolution.
• The downstream DLR is delivered successfully.
• A pending downstream delivery expires.
• Another terminal path explicitly proves that no correlation or downstream delivery remains.

Configuration

Configuration is equivalent to:

sendium.dlr.storage.backend=memory
sendium.dlr.memory.max-tracked-messages=100000
sendium.dlr.provider-correlation-ttl=3D
sendium.dlr.pending-delivery-ttl=7D
sendium.dlr.memory.cleanup-interval=1M

Suggested environment-variable mappings:

SENDIUM_DLR_STORAGE_BACKEND
SENDIUM_DLR_MEMORY_MAX_TRACKED_MESSAGES
SENDIUM_DLR_PROVIDER_CORRELATION_TTL
SENDIUM_DLR_PENDING_DELIVERY_TTL
SENDIUM_DLR_MEMORY_CLEANUP_INTERVAL

The initial three-day provider-correlation and seven-day pending-delivery values preserve current PostgreSQL retention behavior. Final defaults must be checked against the heap cost of representative DLR records.

Observability

Logs

• Log the selected DLR backend, configured capacity, retention periods, and cleanup interval at startup.
• Emit a prominent warning that memory-backed DLR state is non-durable and single-process.
• Log capacity rejections and periodic cleanup summaries.
• Do not log message payloads, provider message IDs, or other sensitive message data.

Readiness

Expose the selected backend through the existing sendium-dlr-storage readiness check.

Memory mode remains UP while the backend is initialized and internally operational. Reaching configured capacity does not make the entire service unready because non-DLR submissions may still be accepted.

Metrics

Expose only:

• The selected backend, tagged with backend=memory.
• Storage operation latency and success/error outcome.
• Current reserved logical-message capacity and configured maximum capacity.

Do not introduce separate expiration, cleanup, rejection, correlation, or delivery-state metrics in this issue.

Verification

• Focused DlrStorage behavior tests run against memory mode.
• Concurrency tests prove reservation count never exceeds capacity.
• Admission tests prove capacity is reserved before HTTP or SMPP success and released when outbound admission fails.
• Restart tests prove memory DLR state is cleared while pending durable SMS records remain recoverable.
• Recovery tests prove a durable pending SMS reacquires capacity before provider submission.
• Tests cover multipart correlations, first-terminal-wins, intermediate receipts, callback retry, disconnected SMPP delivery, cleanup, attempt fencing, and reservation release.
• Configuration tests prove memory mode starts without PostgreSQL DLR or file-backed DLR initialization.

Acceptance criteria

• memory is a selectable DLR backend and does not require a PostgreSQL datasource or file-backed DLR store.
• HTTP submissions requesting DLRs receive correlated HTTP callbacks while the process remains active.
• SMPP submissions requesting DLRs receive correlated downstream receipts while the process remains active.
• Provider rejection results work through both supported downstream channels.
• Multipart provider IDs resolve to the correct logical gateway message while consuming one capacity slot.
• First-terminal-wins and stale-attempt fencing match the shared DlrStorage behavior.
• Restart clears every memory-backed correlation, reservation, retry, and pending delivery.
• Recovered durable SMS messages reacquire DLR capacity before provider submission.
• Expired correlations and pending deliveries release capacity.
• Active entries are never silently evicted.
• New DLR-requesting submissions are rejected or backpressured when capacity is unavailable.
• Non-DLR submissions are not rejected solely because memory DLR storage is full.
• Memory mode is reported through startup logs, readiness data, selected-backend metrics, and capacity metrics.
• Documentation explains the non-durable boundary and every supported combination with filesystem, memory, and future PostgreSQL outbound storage.

Open decisions

• Whether to replace or deprecate sendium.dlr.persistence.enabled.
• Final property names and environment-variable mappings.
• Default maximum number of tracked logical messages.
• Whether retention configuration applies uniformly to PostgreSQL, file-backed, and memory backends.
• Exact HTTP and SMPP responses when DLR capacity is unavailable.
• Whether explicit multi-replica configuration causes startup failure or a prominent warning in memory mode.

Non-goals

• Surviving application or container restart in memory mode.
• Sharing memory-backed DLR state between processes or replicas.
• Exactly-once callback or SMPP receipt delivery.
• Replacing PostgreSQL or file-backed durable DLR storage.
• Changing the default DLR backend.
• Persisting SMS bodies as part of memory DLR state.
• Dynamically switching backend without restart.

[lykakis]

Parent epic: #337.

This issue provides the bounded in-memory DLR implementation for the epic's memory profile. #337 owns the shared storage architecture, supported backend combinations, and eventual defaults. PostgreSQL remains the current durable DLR backend until the file-backed profile defined by the epic is implemented; memory mode remains explicitly non-durable.