Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277

Activity

1. iamwavecut

   added

   agent:createdCreated by incident automationCreated by incident automation

   agent:queuedQueued for an automated incident investigationQueued for an automated incident investigation

   needs-triageIncident cause still requires investigationIncident cause still requires investigation

   needs-humanAutomated investigation requires the owner decisionAutomated investigation requires the owner decision

   on Oct 23, 2026
2. iamwavecut commented on Oct 25, 2026

   

   iamwavecut

   on Oct 25, 2026

   Last edited by iamwavecut

   OwnerAuthor

   More actions

   Problem

   A dialog task on the dialog worker queue reached terminal state failed and emitted the routing event all_attempts_exhausted with an empty route object and a protocol-class reason; this matches open issue 277, now at 51 occurrences. The newest occurrence ran the deployed head, which equals the pinned revision 7970c21. The scoped attempt capture for this occurrence retains a single attempt_failed row on the primary self-hosted lane inside the task window, with no failure reason retained. Verified in the pinned tree: two dialog session-engine exits (crates/openplotva-app/src/dialog_turn/session.rs:843-902 sanitizer-empty/raw-empty/forced-image-tool-missing, and 1044-1074 outbound-validation rejection) hard-code the protocol class although the provider answered, bypassing the model-output/protocol split that PR 151 introduced in the shared classifier (openplotva-llm/src/retry.rs:248-269) and most adapters; the job-retry path then requeues the whole turn to exhaustion and emits the terminal event with ids unset, matching the incident snapshot exactly. A genuine endpoint fault on the engaged lane cannot be excluded because no per-attempt reason is retained, so both an our-code misclassification funnel and an external fault remain possible; the discriminating evidence is the per-attempt error strings, which the deep stage must pull before any patch is declared causal.

   External cause: possible. Code defect: possible.

   Observations

   • Terminal routing event all_attempts_exhausted for the dialog workflow with an empty route object and a protocol-class reason; 51 occurrences since 2026-09-17, status diagnosing; the newest occurrence ran minutes before this triage on the deployed build.
   • Deployed runtime revision, build revision and repository HEAD all equal the pinned base revision 7970c21; restart_count 0, no OOM kill, so the occurrence ran the current build including every dialog change in the provided history; the working tree is clean with no prior patch to continue.
   • Scoped evidence for this occurrence: a dialog task on the dialog worker queue, state failed, created 1793478044, started 1793478050, completed 1793478174; the scoped attempt capture retains only one attempt_failed row, on the primary self-hosted lane at 1793478087, inside the task window; no attempt row retains a failure reason or error string.
   • Host, runtime and database evidence is healthy (about 9.4 GB memory available, ample disk, moderate load, zero waiting locks, normal pool and connections), ruling out local resource starvation, crash loops and lock or pool exhaustion.
   • Queue counters show 34525 completed versus 146 failed dialog tasks (under 0.5 percent), so the routing path works for the vast majority of dialog tasks and the trigger is conditional.
   • Verified in the pinned tree at crates/openplotva-app/src/dialog_turn/session.rs:843-902: when the answer is empty after sanitization (or raw-empty, or the explicitly requested image tool was not executed), the session engine passes FailureReason::ProviderProtocolError to the job-retry handler even though the provider answered and our own sanitizer emptied or withheld the answer.
   • Verified in the pinned tree at crates/openplotva-app/src/dialog_turn/session.rs:1044-1074: when the outbound deliverability validator rejects the sanitized answer, the session engine again passes FailureReason::ProviderProtocolError to the job-retry handler.
   • Verified in the pinned tree at crates/openplotva-llm/src/retry.rs:248-269: the shared message classifier maps validator rejections including 'empty final text', 'pathological final text' and 'no tool calls' to ModelOutputRejected (same-model re-sampling, no breaker charge) while decode failures such as 'decode chat completion payload' and 'response body is empty' map to ProviderProtocolError; the two session-engine exits bypass this split by hard-coding the class.
   • Verified in the pinned tree at crates/openplotva-llm/src/gemini.rs:2638-2641 and 2940-2961: one adapter family still types the identical 'empty final text' condition as ProviderProtocolError, disagreeing with the shared rules (crates/openplotva-llm/src/aifarm.rs:721-724 maps the same phrase to the 'empty' rejection bucket).
   • The job-retry path repeats the failure reason in the terminal routing event detail as last_retryable_reason with provider and model ids unset (crates/openplotva-app/src/dialog_jobs/retry.rs:231-235, routed_attempts.rs:732-733), matching the incident snapshot's empty route object and protocol-class reason exactly.
   • The signature's first_seen day coincides with the merge day of the model-output/protocol reclassification (PR 151), and the recurrence spanned every later dialog merge including the deployed head.

   Hypotheses

   • Misclassification funnel (our code): the answer for this prompt was emptied by the sanitizer or rejected by outbound validation on the engaged lane, the session engine typed it as a protocol error, breakers were charged and the chain walked to weaker fallbacks, the whole turn requeued and replayed, and the terminal event repeated the protocol-class reason; a prompt-conditioned rejection reproduces on every fallback model.
   • Genuine endpoint fault (external): the engaged lane really failed in the window (connect, timeout or decode failure), the classifier typed a protocol error correctly, and the chain and job budgets exhausted with the identical terminal signature.
   • Both funnels coexist and alternate across the recurring signature's 51 occurrences; a single-attempt capture spanning seconds fits both a fast validator rejection and a brief endpoint fault.

   Supporting

   • The exact terminal signature (event type, workflow, empty route object, protocol-class reason with ids unset) is reproducible in the deployed tree from the two session-engine validator exits through the job-retry exhaustion path, requiring no provider fault at all.
   • The two exits contradict the documented policy of the model-output/protocol split introduced by PR 151: when the provider answered and our validators rejected the output, the shared classifier's intended class is ModelOutputRejected with same-model re-sampling, not a protocol class that charges breakers and walks the chain.
   • The signature's first_seen coincides with the split merge day, and the count grew while the classifier and both unmigrated exits are unchanged in the deployed head.
   • Merged PR 247 documents live empty-answer replay-to-exhaustion runs on the primary dialog hop, confirming the funnel is exercised in production, not merely theoretical.
   • Healthy host, runtime and database evidence plus the overwhelming dialog success rate rule out local resource exhaustion, crash loops and lock or pool starvation.

   Contradicting

   • No captured attempt record carries a failure reason, so a genuine endpoint fault correctly typed as a protocol error cannot be excluded for this occurrence.
   • The current scoped capture shows a single attempt on the primary lane rather than the four-route engagement seen at the retained earlier analysis, which weakens the all-route repeat-failure pattern for this occurrence specifically; the capture may be truncated.
   • A concurrent open issue (279) records availability bursts on an external provider in the same period, so external availability faults remain live.
   • A pure provider-outage story does not by itself explain the empty route object in the terminal event, which the verified job-retry path produces by repeating the reason with ids unset; the adapter/classifier disagreement on the empty-final-text phrase means even a retained reason string would not fully disambiguate the funnel.

   Related changes

   • PR 151 (merged 2026-09-17, in the deployed head) split model-output rejections from protocol errors and added walker-level same-model re-sampling; it migrated adapter sites but not the two session-engine validator exits, leaving the misclassification branch this signature exercises.
   • PR 247 (merged 2026-10-12) documents and caps the live empty-answer replay storm on the primary self-hosted dialog hop but leaves the failure class and requeue-to-exhaustion behavior unchanged.
   • PR 262 (merged) changed dialog route sampling parameters; newest dialog sampling change in the provided history with no causal evidence either way.
   • PR 98 introduced the multi-hop dialog cascade and wall-clock budget whose tail reachability and per-round truncation these terminal events question.
   • PR 30 audited the response-decoding surface that defines genuine protocol-class failures, sharpening the split the two validator exits bypass.
   • PR 24 added the external [configured resource] routing pool that forms the tail route engaged in earlier occurrences.
   • The deployed head 7970c21 is itself a dialog tool-call prompt-contract merge (PR 285) observed via git log; it postdates all of the above with no causal evidence either way.

   Missing

   • Per-attempt failure reasons or retry-stage job events for this occurrence: the error strings would discriminate the session-engine validator exits (sanitization-empty, outbound-validation rejection) from endpoint decode/connect failures and attribute the observed protocol-class reason to one of the two funnels.
   • Raw provider response bodies behind the attempts; retention exists per the backend merge (PR 8) but no bodies are in scope.
   • Live database values of the dialog workflow's hop budget, wall clock, job-attempt cap and route role/ordering/weight fields at failure time, needed to judge how the observed attempts plus the whole-turn requeue fit the configured rounds.
   • Confirmation of whether the current single-attempt scoped capture is complete or truncated relative to the four-attempt capture retained at the earlier analysis.
   • A design decision for the deep stage: whether the raw-empty branch belongs to the model-output class (as the shared rules map empty final text) or the protocol class (as one adapter family still types it), and whether the adapter/classifier disagreement should be unified.
   • Whether the operator-facing admin report should distinguish validator-exhaustion terminal events from endpoint protocol faults, so healthy providers stop appearing faulty.
   • Timestamps of all 51 exhaustion events to establish whether they cluster around specific dialog content patterns or provider incidents.

   Acceptance

   • A regression test where a step provider answers with text that sanitizes to empty must end with a model-output-class (or validator-specific) last_retryable_reason in the job-retry terminal routing event, not the protocol class.
   • A regression test where the outbound deliverability validator rejects the answer must produce the same reclassification.
   • A regression test where the endpoint returns an empty or non-decodable body must still classify as the protocol class, preserving genuine protocol-fault semantics.
   • Deep-stage verification must pull this occurrence's retry-stage job events or per-attempt error strings and attribute the observed reason to one of the two funnels before any patch is declared causal for the observed occurrences.

   Impact

   51 captured terminal events.

   Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.
3. iamwavecut commented on Oct 28, 2026

   

   iamwavecut

   on Oct 28, 2026

   Last edited by iamwavecut

   OwnerAuthor

   More actions

   Problem

   A dialog task on the dialog queue failed terminally with event all_attempts_exhausted, reason provider_protocol_error and an empty route object; the incident record shows 128 occurrences of the signature. Re-verified in the repository at the deployed head (identical to base revision): two session-engine exits hard-code the protocol-error class for outcomes where the provider answered and our own validation rejected or emptied the answer (sanitizer collapse to empty at the raw-empty branch, outbound deliverability rejection), the job-retry handler requeues the whole turn per attempt and on exhaustion emits the observed terminal event with provider and model unset while repeating the protocol-class reason, and the shared message-rule classifier maps the same validator conditions to the model-output class, so the two exits bypass that split. A genuine endpoint protocol fault is not excluded: per-attempt error strings and routing attempt rows for the newest occurrence are not retained in scope, and the classifier's decode-failure path produces the identical terminal signature. The deep stage must attribute the observed reason to one funnel before any patch is declared causal.

   External cause: possible. Code defect: possible.

   Observations

   • Incident snapshot: terminal event all_attempts_exhausted with reason provider_protocol_error, workflow dialog, empty route object; 128 recorded occurrences of the signature per the incident record, and the dialog queue's failed counter is 127 against roughly 34.2k completed (under 0.5 percent).
   • Scoped task evidence for the newest occurrence: dialog task on the dialog queue in state failed, about 19 seconds from creation to start and about 2 seconds from start to completion, a fast terminal window consistent with either an immediate validator rejection after a fast answer or a fast endpoint failure.
   • Scoped route-attempt evidence for the newest occurrence returns no_matching_evidence, and no per-attempt error strings or raw provider bodies are retained in scope, so the providers and models actually tried and the failing stage remain unknown.
   • Host and runtime evidence is healthy: deployed revision equals repository head and base revision, restart_count 0, no OOM kill, ample memory and disk, moderate load; the database reports zero waiting locks with an active pool, ruling out local resource starvation or a crash loop.
   • Re-verified at the deployed head (session engine, dialog_turn/session.rs lines 843-903): when the final answer is empty, becomes empty after response sanitization, or an explicitly requested image tool was skipped, the turn reports FailureReason::ProviderProtocolError to the job-retry handler; the outbound deliverability validator rejection (lines 1044-1075) does the same. In all these cases the provider answered and our own validation rejected what it wrote.
   • Re-verified at the deployed head (dialog_jobs/retry.rs lines 216-235): every retryable reason requeues the whole turn on the same queue up to the attempt cap, and on exhaustion the terminal routing event all_attempts_exhausted is emitted with provider_id and model_id None and detail.last_retryable_reason repeating the failure reason, matching the incident snapshot's empty route object and protocol-class reason exactly.
   • Re-verified at the deployed head (openplotva-llm retry.rs message rules lines 248-269): validator rejections such as 'empty final text', 'only protocol artifacts' and 'pathological final text' map to the model-output class, while decode failures such as 'decode chat completion payload' and 'response body is empty' map to the protocol class; the two session-engine exits bypass this split by hard-coding the protocol class.
   • Re-verified at the deployed head: one adapter family types its own 'empty final text' rejection as a protocol-class error while the shared message rules map the identical phrase to the model-output class, so the layers disagree on the same condition and a reason string alone does not identify the funnel.
   • Existing tests assert the terminal ledger reason codes for these funnels (sanitized_empty_exhausted, undeliverable_answer_exhausted) and one routing-event test asserts last_retryable_reason only for a provider-unavailable funnel (dialog_jobs/tests.rs line 2236); no test asserts the terminal routing event's last_retryable_reason for the sanitizer-empty or undeliverable funnels, so the incident-facing classification is unguarded.

   Hypotheses

   • Misclassification funnel (our code): the model's answer was emptied by our sanitizer or rejected by our outbound validation, the session engine typed it as a protocol error, the turn requeued to the attempt cap, and the terminal event repeated the protocol-class reason; the code path to the observed signature is verified end-to-end in the deployed tree without any provider fault.
   • Genuine endpoint protocol fault (external): the endpoint returned empty or non-decodable bodies for these requests, the classifier correctly typed protocol errors, attempts hopped and charged breakers, and the chain or job budget exhausted with the identical terminal signature.
   • Both funnels coexist and alternate across the occurrences; the few-second final processing window fits both an instant validator rejection after a fast answer and a fast endpoint failure.

   Supporting

   • The exact terminal signature (event type, workflow, empty route, protocol-class reason) is reproducible in the deployed tree from the two session-engine exits through the job-retry handler, so a complete code path to the observed event exists without any provider misbehavior.
   • The two exits contradict the split's own policy established by the reclassification merge: when the provider answered and our validators rejected the output, the intended class is the model-output one with same-model re-sampling, not a protocol-class provider failure that charges the breaker and walks to a weaker fallback.
   • The recurrence continued under a deployed head in which the classifier and these two exits are unchanged since the reclassification merge, and the newest occurrence provably ran the current build (deployed revision equals head).
   • Healthy host, runtime and database evidence plus the overwhelming dialog success rate rule out local resource exhaustion, crash loops and lock starvation.

   Contradicting

   • A pure our-code story is not established: without the newest occurrence's per-attempt error strings or retry-stage job events, a genuine endpoint protocol fault that the classifier correctly reports cannot be excluded.
   • A pure external-endpoint story is not established: the verified validator-rejection funnel produces the identical signature without any provider fault, and nothing in the retained scope evidences an endpoint fault specifically.
   • The classifier inconsistency on the 'empty final text' condition between one adapter family and the shared rules means even a correct-looking reason string does not by itself identify the funnel.

   Related changes

   • Merge Re-sample a rejected model before falling back to one without tools #151 (merged 2026-09-17, in the deployed head) split model-output rejections from protocol errors and added walker-level same-model re-sampling for validator rejections; it migrated the adapter sites but not the two session-engine validator exits, which still emit the protocol class.
   • Merge Sample Gemma dialog turns as the model card recommends #262 (dialog route sampling parameters) is the newest dialog-behavior change before the continued recurrence; no causal evidence either way.
   • Merge Route memory workflows through independent cascades #98 set dialog to weighted-primary multi-hop routing with a workflow budget; these protocol-class terminal events question whether the fallback tail was reachable under that budget.
   • Merge Audit LLM tool parser parity #30 followed up a post-deploy dialog exhaustion audit by tightening the tool-call parsing surface, adjacent to the exhaustion surface but not to the failure-class split.
   • Merge Coordinate incident feedback, quota recovery, and actionable notifications #141 owns the incident-feedback and notification surface these terminal events flow into.
   • The prior analysis of this signature (issue Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277) references merges extending the model-output repair verdict carry-back and capping a live empty-answer replay storm; those merge records are not in the provided history and could not be re-verified in this pass.

   Missing

   • The per-attempt error strings or retry-stage job events for the newest occurrence: the event's error field distinguishes the session-engine sites ('dialog answer became empty after sanitization', 'dialog answer rejected by outbound validation: ...', 'dialog provider returned no answer, response, or queued tool material') from a decode-failure string, which would attribute the observed reason to one funnel.
   • Retained routing attempt rows for the newest occurrence (the scoped attempt query returns no matching evidence), so the set of providers and models actually tried is unknown.
   • Raw provider response bodies behind the occurrences; retention exists but no bodies are in scope.
   • A design decision for the deep stage: whether the raw-empty branch belongs to the model-output class (the shared rules map 'empty final text' there) or the protocol class (one adapter family types it there), and whether the two validator exits should reuse the model-output reason or a validator-specific reason; this also decides whether the inconsistent adapter typing must be aligned.
   • Live database values of the dialog workflow's hop budget, wall clock, job-attempt cap and route role/ordering/weight fields at failure time, needed to judge whether the fallback tail was reachable.
   • Whether the operator-facing admin report should distinguish validator-exhaustion terminal events from endpoint protocol faults, so operators stop seeing healthy providers as faulty.
   • Content of the merges referenced by issue Dialog turns terminate as provider_protocol_error after our own validators reject the answer: two session-engine exits bypass the model-output split and requeue to exhaustion #277's body but absent from the provided history (repair-verdict carry-back, empty-answer replay cap), needed to confirm they leave the failure class unchanged.

   Acceptance

   • Deep-stage verification must pull the newest occurrence's retry-stage job events or error strings and attribute the observed reason to one of the two funnels before any patch is declared causal for the observed occurrences.
   • A regression test where a step provider answers with text that sanitizes to empty must end with a model-output-class (or validator-specific) last_retryable_reason in the job-retry terminal routing event, not provider_protocol_error.
   • A regression test where the outbound deliverability validator rejects the answer must produce the same reclassification.
   • A regression test where the endpoint returns an empty or non-decodable body must still classify provider_protocol_error, preserving genuine protocol-fault semantics.

   Impact

   128 captured terminal events.

   Exact infrastructure identities and operational evidence remain in the private incident context available to the repair worker.