feat(delivery): openframe-machine-delivery — engine, dispatcher and the tool-installation spec (PR 1 of the plan)

[semen-flamingo]

This PR adds a module and one spec and wires nothing into existing flows: no service changes behaviour after merge, with or without the flag. The wiring (ack routing, success hook, call site) is the stacked follow-up #2212.

Types roll out one at a time. This PR carries the pilot, TOOL_INSTALLATION — the highest-traffic flow and the one that can be exercised on demand through /force/tool-agent/reinstall. Client uninstall, scripts and the two update flows each come as their own spec PR once the pilot has been through dev and stage.

Context

Four server→agent flows (tool install, tool update, client update, client uninstall) go through JetStream with a durable consumer per machine; the server keeps no state and has no ack, watchdog or metric for them. Scheduled scripts already use an application-level model (ScriptDeliveryRetryService, ScriptExecutionAcknowledgeListener, watchdog, openframe-rmm.yaml alerts). This is the shared engine that the four flows (and later the scripts) move onto; each delivery type is a spec, the same pattern as NotificationTypeSpec + NotificationTypeRegistry + NotificationEmitter.

Module layout

openframe-machine-delivery   com.openframe.delivery — the engine, no NATS dependency
  └─ openframe-data-mongo-sync        (MachineDelivery + repository, MachineRepository)
openframe-data-nats          NATS specs, today: ToolInstallationDeliverySpec
  └─ openframe-machine-delivery
api-lib, api-service-core, management-service-core, client-core, tool-agent-nats-installation → both (#2212+)

Publishers of deliveries live in five modules of three services (api, management, client); only client will run the sweep. Hence a module every publisher can include, with the engine's only contact with the machine document isolated in MachineOnlineStatus. "Machine" in the name is deliberate: the engine is transport-agnostic (specs own publish) but assumes a recipient with online/offline presence — it is not a generic outbox for push or Slack.

The pattern, mirrored from notifications

Notifications	Delivery
NotificationType	DeliveryType
NotificationSeed (per type, nested Spec.Seed)	DeliverySeed (per type, nested Spec.Seed)
NotificationTypeSpec<S>	DeliverySpec<S, P> — getType, getSeedClass, getPayloadClass, request(seed), publish(machineId, payload), onFailed(row, failure)
NotificationTypeRegistry	DeliverySpecRegistry
NotificationEmitter.notify(request)	DeliveryDispatcher.dispatch(seed)

A caller hands a seed to the dispatcher and knows nothing else:

deliveryDispatcher.dispatch(new ToolInstallationDeliverySpec.Seed(machineId, toolAgent, tool, reinstall));

The dispatcher resolves the spec by seed.type(), the spec builds the DeliveryRequest (payload, targetId, machineId), DeliveryRecorder writes the PENDING row, the spec publishes. A spec is self-contained: ToolInstallationDeliverySpec maps the payload itself (the two data-nats mappers) and publishes on machine.{id}.tool-installation through NatsMessagePublisher. It does not wrap ToolInstallationNatsPublisher; that class is untouched here and deleted in #2212 once its last caller moves to the dispatcher. The targetId a spec publishes with is the one its completion hook will receive from the agent: toolAgent.getKey() ↔ agentType in installed-agent. The pair (targetId, machineId) always reads "which agent on which machine".

How SUCCESS is detected and when re-delivery stops (wired in #2212)

A row in machine_delivery is PENDING → ACKED → DONE | FAILED. The sweep only ever selects PENDING, so re-delivery stops the moment the row leaves PENDING. No new subject carries success — the signals the agent already sends will close the row:

Type	Agent signal (exists today)	Server hook	Transition
any	machine.{id}.execution.acknowledge + type, targetId	ScriptExecutionAcknowledgeListener → DeliveryTracker.acknowledge	PENDING → ACKED (re-sends stop)
TOOL_INSTALLATION	machine.{id}.installed-agent {agentType, version} after install()	InstalledAgentService → complete(TOOL_INSTALLATION, agentType, machineId)	→ DONE
CLIENT_UNINSTALL (later)	HTTP POST /api/agents/uninstall	AgentUninstallService → complete(CLIENT_UNINSTALL, "openframe-client", machineId)	→ DONE
SCRIPT_SCHEDULE (later)	script-execution.result	RmmResultService → complete(SCRIPT_SCHEDULE, executionId, machineId)	→ DONE
*_UPDATE (later)	same installed-agent, matched on version		→ DONE only when version == target

What the agent does not send today is a failure: after ack, a failed install is visible only as FAILED/TIMEOUT from the watchdog; an explicit failure result is a Rust-side follow-up.

Engine

DeliveryDispatcher.dispatch(seed)         spec = registry.require(seed.type()); request = spec.request(seed)
                                          recorder.record(request)  → row PENDING (MongoDeliveryRecorder / NoopDeliveryRecorder by flag)
                                          spec.publish(machineId, payload)
DeliveryTracker.acknowledge / complete    PENDING → ACKED → DONE   (MongoDeliveryTracker / NoopDeliveryTracker by flag)
DeliverySweepScheduler.tick()  every openframe.delivery.sweep.interval under ShedLock, only where sweep.enabled=true:
   DeliverySweepService.retryPending()    PENDING older than ack-threshold →
       MachineOnlineStatus says OFFLINE → wait (RETRY_ON_RECONNECT) until reconnect-window, or FAILED now (SKIP)
       attempts < max                    → spec.publish(machineId, payload from payloadJson), attempts+1
       else                              → FAILED exhausted
   DeliveryWatchdogService.reapAcked()    ACKED older than result-timeout → FAILED timeout
DeliveryFailureRecorder.fail(row, reason) FAILED + expiresAt (TTL) + metric + spec.onFailed(row, reason)

Rows use a composite _id TYPE:targetId:machineId (DeliveryId): the agent's signals carry only that natural key, so one open row per command per machine is a requirement, not a storage choice. Re-dispatching the same command overwrites its row; there is no per-attempt history by design.

Override chain

Numbers never live in a spec. Resolution order, all in DeliveryProperties: openframe.delivery.defaults.* (required, validated at startup) → openframe.delivery.types.<TYPE>.* → row-level offlineBehavior / reconnectWindowSeconds (carried from ScheduleScript). A spec overrides behaviour only: request, publish, onFailed.

openframe:
  delivery:
    enabled: true
    sweep:
      enabled: true          # client only
      interval: 30000
      lock-at-most-for: 2m
      lock-at-least-for: 10s
    defaults:
      ack-threshold-seconds: 30
      max-attempts: 3
      offline-behavior: RETRY_ON_RECONNECT
      reconnect-window-seconds: 86400
      result-timeout-seconds: 600
      ttl-seconds: 604800
    types:
      SCRIPT_SCHEDULE:
        offline-behavior: SKIP

Rollout mechanics

No new subject and no dual publish. A JetStream stream stores every message published on its subjects, whether it came from js.publish or plain publish. So when #2212 switches the spec from publishPersistent to core publish on the same machine.{id}.tool-installation, old agents keep receiving it through the TOOL_INSTALLATION stream and their durable consumers, and new agents receive it through a core subscription. The stream is deleted only at cleanup, once no old agent is left. Rows (and therefore retries and alerts) are created only for machines whose agent version acks — old agents never ack and would otherwise be re-sent to.

What is in this PR

• openframe-machine-delivery (new): DeliverySeed, DeliverySpec, DeliverySpecRegistry, DeliveryRequest, DeliveryDispatcher, DeliveryRecorder (+ mongo / noop), DeliveryTracker (+ mongo / noop), DeliveryId, DeliveryProperties, DeliverySweepService, DeliveryWatchdogService, DeliveryFailureRecorder, DeliveryMetrics (openframe.delivery.retried{type}, openframe.delivery.failed{type,reason}), DeliverySweepScheduler, MachineOnlineStatus. 27 unit tests.
• data-mongo-common: DeliveryType, DeliveryStatus, DeliveryFailure, MachineDelivery in document.delivery; data-mongo-sync: MachineDeliveryRepository.
• data-nats: ToolInstallationDeliverySpec with nested Seed, self-contained (payload mapping + subject + publish). 3 tests. Existing publishers untouched.
• Root pom: module + dependencyManagement entry; shedlock-spring pinned like client-core (not managed by the parent).

Every flag-dependent bean has a no-op counterpart selected by matchIfMissing, so services that include the module without the YAML boot as before. With the flag off only DeliverySpecRegistry, DeliveryDispatcher and the two no-op beans exist.

Plan by PRs

1. this PR — module, engine, dispatcher, tool-installation spec. Prod: no change.
2. feat(delivery): wire ack, success and the tool-installation call site into the delivery engine (PR 2 of the plan) #2212, stacked on this branch — usage for the pilot: type/targetId on ScriptExecutionAcknowledgeMessage + listener routing; complete() in InstalledAgentService; ToolInstallationService through DeliveryDispatcher; ToolInstallationNatsPublisher deleted; spec publishes over core NATS; row only for machines whose agent version acks. Flag still off everywhere.
3. saas-tenant — openframe.delivery YAML block with explicit enabled: false in base, sweep.enabled for client only, pin oss.libs.version; then enabled: true dev → stage → prod.
4. saas-shared — Grafana rules on openframe_delivery_failed_total{type,reason} in openframe-rmm.yaml.
5. Rust (agent team) — tool_installation_message_listener.rs → core subscribe + ack with type/targetId (pattern: execution_listener.rs); explicit failure result. First real rows, metrics and alerts appear with this agent.
6. Next types, one PR each with the same shape (spec + usage + Rust listener): client uninstall (targetId = "openframe-client", onFailed restores PENDING_DELETION), scheduled scripts (ScheduleFireDispatcher → dispatcher, delete ScriptDeliveryRetry* / ScheduleDeliveryRepublisher; ad-hoc scripts from api-lib gain retry), client/tool updates (machine.all.* → per-machine fan-out with version on the row, delete PublishState + AgentVersionUpdatePublishFallbackScheduler; CLIENT_UPDATE last, it is the agent rollout channel).
7. saas-tenant cleanup per type once its fleet is migrated — drop $JS.API.* from the NATS configmap, delete the stream explicitly (its consumers go with it), Mongock drop script_delivery_retry, remove openframe.rmm.execution.retry.*.

In parallel, outside the plan: NATS service/admin passwords into ExternalSecret per tenant.

Open decisions

1. Ack format: extend ScriptExecutionAcknowledgeMessage with type/targetId (feat(delivery): wire ack, success and the tool-installation call site into the delivery engine (PR 2 of the plan) #2212) vs. a new subject.
2. Update broadcasts: fan-out rows per machine (proposed) vs. pull model.
3. PENDING_DELETION on exhausted uninstall: revert + alert (proposed) vs. leave.
4. Agent-side durable consumers only (proposed) vs. server-side too.

Verification

Full reactor mvn test on this branch: 12 modules with tests, 464 tests, 0 failures (JDK 21). client-core and api-service-core sources are byte-identical to main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BnYUpYgKvuMw6hB5VZSimY