feat(processing): classify artifact processing failures with safe diagnostics (#2126)

docs/en/development/meta.json

@@ -9,6 +9,7 @@
     "plugin-contract",
     "pydantic-ai-inference",
     "observability",
+    "processing-failures",
     "remote-access-implementation",
     "dashboard",
     "desktop-delivery-plan"

docs/en/development/observability.md

@@ -74,6 +74,9 @@ JSON formatting deliberately ignores unknown extras. When a new field is require
 4. Update the operational field contract in both locales and add a focused observable-output regression test.
    Review compatibility before renaming or removing published fields or event names.
 
+See [Processing failure diagnostics](processing-failures.md) for the shared failure vocabulary and the
+per-Family coverage matrix.
+
 Service identity comes from `src/powercontext/server/telemetry.py`; logs and traces use the same resource identity.
 Error classification comes from `powercontext._logging.error_category`, with domain-owned categories such as `blocked`
 set at their owning boundary. Do not create independent vocabularies in each adapter.

docs/en/development/processing-failures.md

@@ -0,0 +1,87 @@
+# Processing failure diagnostics
+
+This page is the shared contract for classifying artifact-processing failures and for the safe
+diagnostics attached to them. It extends the existing operational logging fields; it does not
+introduce a second telemetry system.
+
+## Shared fields
+
+Every processing failure keeps the existing identity and outcome fields (`family`, `operation`,
+`worker_id`, `request_generation`, `outcome`, `error_code`, `exception_type`) and may add:
+
+| Field | Meaning | Values |
+|---|---|---|
+| `processing_stage` | The stage that owned the failure | `generation`, `embedding`, `output_parsing`, `business_validation`, `persistence`, `usage_accounting` (reserved for usage reporting or persistence) |
+| `failure_reason` | A bounded reason, only when evidence supports it | `context_overflow`, `output_truncated`, `rate_limit`, `timeout`, `schema_invalid`, `provider_unavailable`, `configuration`, `invalid_output`, `storage_failure`, `usage_limit` |
+| `input_item_count` | Number of input items in the failing call | Non-negative integer |
+| `estimated_input_tokens` | Estimated input tokens | Non-negative integer |
+| `token_estimator_id` | Identity of the estimator that produced the estimate | Bounded string |
+| `effective_budget` | Budget applied to the call | Non-negative integer |
+| `provider_request_id` | Provider request identity, when reported | Bounded string |
+| `finish_reason` | Provider finish reason, when reported | Bounded string |
+| `failure_location` | Stable internal location or operation identifier | Bounded string |
+| `cause_exception_type` | Original provider or library exception type before mapping | Bounded string |
+
+`processing_stage` is distinct from the existing `stage` field. `stage` names the emitting boundary or
+domain location: Worker and supervisor values (`worker_process`, `work_admission`, `supervisor`,
+`lease_renewal`, `scope_discovery`, `pending_observation`, `term_revocation`, `worker_termination`),
+domain block names such as `topic_memory`, and the child Worker's `worker` fallback when an exception
+carries no stage. A child failure without a domain block may omit `stage` entirely. `processing_stage`
+is the bounded domain stage. Counts and identifiers are bounded before they reach a log record, and
+unknown or unsupported values are omitted rather than guessed.
+
+## Privacy boundary
+
+Diagnostics never carry Source or Artifact bodies, prompts, model responses, credentials,
+authorization headers, SQL parameters, or arbitrary exception text. Provider identifiers and reasons
+are length-limited printable text. A field being allowlisted is not a redaction engine: values must
+still be safe at the call site.
+
+## Coverage matrix
+
+`Covered` means the stage has a stable classification and a test. `Partial` means some failures are
+classified through the inferred operation or exception type but not every raise site is tagged.
+`Missing` means the stage is not classified yet. `N/A` means the Family does not perform that stage
+in its processing path.
+
+| Family | Generation | Embedding | Output parsing | Business validation | Persistence | Usage accounting |
+|---|---|---|---|---|---|---|
+| Memory / Atomic Memory | Partial | Covered | Partial | Covered | Partial | Missing |
+| Topic Memory | Partial | Partial | Partial | Missing | Missing | Missing |
+| Experience | Partial | N/A | Partial | Missing | Missing | Missing |
+| Profile | Partial | N/A | Partial | Missing | Missing | Missing |
+| Skill | Partial | N/A | Partial | Missing | Missing | Missing |
+| Handoff | Partial | N/A | Partial | Missing | Missing | Missing |
+| Prompt | Partial | N/A | Partial | Missing | Missing | Missing |
+
+Memory / Atomic Memory is tracked at operation granularity while it is the active slice:
+
+| Operation | Stage | Status | Notes |
+|---|---|---|---|
+| `atomic-memory.extract` | Generation | Partial | Provider timeout, unavailability, rate limit, and configuration are classified; a provider finish reason is only visible where the adapter exposes it. |
+| `atomic-memory.extract` | Output parsing | Partial | Schema and Pydantic output failures are classified; some provider truncation paths surface as schema failures. |
+| `atomic-memory.extract` | Business validation | Covered | Candidate content and cited-evidence rules carry `business_validation` / `invalid_output`. |
+| `atomic-memory.reconcile` | Generation | Partial | Same provider mapping as extraction. |
+| `atomic-memory.reconcile` | Output parsing | Partial | Same schema mapping as extraction. |
+| `atomic-memory.reconcile` | Business validation | Covered | Reconciliation output and content rules carry `business_validation` / `invalid_output`. |
+| Embedding (extract, reconcile, query) | Embedding | Covered | Provider status, wrong vector count/order/type, and embedding profile/input failures are classified. |
+| Publish / index | Persistence | Partial | `AtomicMemoryIndexError` codes are mapped; other storage failures keep unknown. |
+| Usage accounting | Usage accounting | Missing | Usage-reporting and persistence failures are not classified; inference usage limits are reported in their originating stage (generation, or embedding when an adapter reports the limit). |
+
+The first rollout slice covers Memory / Atomic Memory: provider failures, budget exhaustion, and
+extraction/reconciliation validation are classified, and safe accounting fields travel from the
+child Worker to the supervisor log. Other Families keep their current behavior until their own slice
+lands; the matrix is updated in the same change as each slice.
+
+## Rollout
+
+1. Shared diagnostics contract, allowlisted fields, Worker propagation, and the Memory / Atomic
+   Memory slice (this change).
+2. Topic Memory: tag its staged generation, index/embedding, and persistence failures.
+3. Experience, Profile, and Skill: tag generation output validation and persistence entry points.
+4. Handoff and Prompt: tag generation and publication failures, then add usage-accounting
+   classification and complete the matrix.
+
+For new instrumentation, cover the observable behavior: successful classification, propagation
+across a real spawned Worker, sensitive-value exclusion, and that a diagnostics failure does not
+change the business outcome.

docs/en/docs/operate/observability.md

@@ -34,6 +34,7 @@ including code stages. Code-specific diagnostic fields are included only for `co
 | `request_id`, `trace_id`, `span_id` | Bound ingress request and current trace context, when available |
 | `worker_id`, `family`, `binding`, `request_generation`, `supervisor_group`, `supervisor_generation` | One worker attempt and its accepted work/fence; correlate child output with supervisor diagnostics |
 | `duration_ms`, `error_code`, `error_category` | Duration in milliseconds, diagnostic code, and bounded operational error classification |
+| `stage`, `processing_stage`, `failure_reason`, `input_item_count`, `estimated_input_tokens`, `token_estimator_id`, `effective_budget`, `provider_request_id`, `finish_reason`, `failure_location`, `cause_exception_type` | Processing failure stage/reason and safe accounting fields; `stage` names the emitting boundary or domain location (Worker/supervisor stage, domain block name, or `worker` fallback), `processing_stage` the bounded domain stage. See [Processing failure diagnostics](../../development/processing-failures.md) |
 
 Server roles are `all` (API and supervisor), `api`, and `background`; child processes use `worker`. Explicit logging
 configuration without a deployment role uses `server`. JSON identity fields correspond to trace resource attributes

docs/zh/development/meta.json

@@ -9,6 +9,7 @@
     "plugin-contract",
     "pydantic-ai-inference",
     "observability",
+    "processing-failures",
     "remote-access-implementation",
     "dashboard",
     "desktop-delivery-plan"

docs/zh/development/observability.md

@@ -75,6 +75,8 @@ JSON formatter 会忽略未登记的 extras。确需新增字段时：
 4. 更新中英文 operational field contract，并补充针对可观察输出的 regression test。
    重命名或删除已发布字段、event name 前，必须检查 compatibility。
 
+共享失败词汇与各 Family 覆盖矩阵见 [处理失败诊断](processing-failures.md)。
+
 Service identity 来自 `src/powercontext/server/telemetry.py`，logs 和 traces 共用同一 resource identity。
 Error classification 来自 `powercontext._logging.error_category`；`blocked` 等 domain-owned category
 由所属 boundary 明确设置。各 adapter 不应自行定义不同词汇。

docs/zh/development/processing-failures.md

@@ -0,0 +1,77 @@
+# 处理失败诊断
+
+本文是 Artifact 处理失败分类与安全诊断的共享契约。它扩展现有的可观测性日志字段，不引入第二套
+telemetry 系统。
+
+## 共享字段
+
+每次处理失败保留现有的身份与结果字段（`family`、`operation`、`worker_id`、`request_generation`、
+`outcome`、`error_code`、`exception_type`），并可按需增加：
+
+| 字段 | 含义 | 取值 |
+|---|---|---|
+| `processing_stage` | 失败所属的处理阶段 | `generation`、`embedding`、`output_parsing`、`business_validation`、`persistence`、`usage_accounting`（预留给账务上报或持久化） |
+| `failure_reason` | 有证据支持时才给出的有界原因 | `context_overflow`、`output_truncated`、`rate_limit`、`timeout`、`schema_invalid`、`provider_unavailable`、`configuration`、`invalid_output`、`storage_failure`、`usage_limit` |
+| `input_item_count` | 失败调用涉及的输入条目数 | 非负整数 |
+| `estimated_input_tokens` | 估算的输入 token 数 | 非负整数 |
+| `token_estimator_id` | 产生估算值的估算器标识 | 有界字符串 |
+| `effective_budget` | 本次调用生效的预算 | 非负整数 |
+| `provider_request_id` | provider 返回的请求标识 | 有界字符串 |
+| `finish_reason` | provider 返回的结束原因 | 有界字符串 |
+| `failure_location` | 稳定的内部位置或操作标识 | 有界字符串 |
+| `cause_exception_type` | 映射前的原始 provider/库异常类型 | 有界字符串 |
+
+`processing_stage` 与既有的 `stage` 字段不同：`stage` 表示发出失败的位置或领域位置，包括
+Worker/supervisor 值（`worker_process`、`work_admission`、`supervisor`、`lease_renewal`、`scope_discovery`、
+`pending_observation`、`term_revocation`、`worker_termination`）、领域 block 名称（如 `topic_memory`），
+以及异常未携带 stage 时 child Worker 的 `worker` 兜底；没有领域 block 的 child 失败可能完全省略 `stage`。
+`processing_stage` 表示有界的领域阶段。计数与标识在写入日志前会被限界；未知或不受支持的值直接省略，
+不做猜测。
+
+## 隐私边界
+
+诊断不得包含 Source/Artifact 正文、prompt、模型响应、凭据、授权头、SQL 参数或任意异常文本。provider
+标识与原因只保留有长度上限的可打印文本。字段进入白名单不等于自动脱敏，调用点仍必须保证值本身安全。
+
+## 覆盖矩阵
+
+`Covered` 表示该阶段已有稳定分类与测试；`Partial` 表示部分失败可经 operation 或异常类型推断，但尚未覆盖
+所有抛出点；`Missing` 表示尚未分类；`N/A` 表示该 Family 的处理路径不执行该阶段。
+
+| Family | Generation | Embedding | Output parsing | Business validation | Persistence | Usage accounting |
+|---|---|---|---|---|---|---|
+| Memory / Atomic Memory | Partial | Covered | Partial | Covered | Partial | Missing |
+| Topic Memory | Partial | Partial | Partial | Missing | Missing | Missing |
+| Experience | Partial | N/A | Partial | Missing | Missing | Missing |
+| Profile | Partial | N/A | Partial | Missing | Missing | Missing |
+| Skill | Partial | N/A | Partial | Missing | Missing | Missing |
+| Handoff | Partial | N/A | Partial | Missing | Missing | Missing |
+| Prompt | Partial | N/A | Partial | Missing | Missing | Missing |
+
+Memory / Atomic Memory 作为当前切片，按 operation 粒度跟踪：
+
+| Operation | Stage | 状态 | 说明 |
+|---|---|---|---|
+| `atomic-memory.extract` | Generation | Partial | provider timeout、不可用、限流、配置错误已分类；仅 adapter 暴露 finish reason 时可见。 |
+| `atomic-memory.extract` | Output parsing | Partial | schema 与 Pydantic 输出失败已分类；部分 provider 截断仍表现为 schema 失败。 |
+| `atomic-memory.extract` | Business validation | Covered | 候选内容与引用证据规则输出 `business_validation` / `invalid_output`。 |
+| `atomic-memory.reconcile` | Generation | Partial | 与提取相同的 provider 映射。 |
+| `atomic-memory.reconcile` | Output parsing | Partial | 与提取相同的 schema 映射。 |
+| `atomic-memory.reconcile` | Business validation | Covered | 对账输出与内容规则输出 `business_validation` / `invalid_output`。 |
+| Embedding（提取、对账、查询） | Embedding | Covered | provider 状态、向量数量/顺序/类型错误、profile/input 失败均已分类。 |
+| Publish / index | Persistence | Partial | `AtomicMemoryIndexError` 的 code 已映射；其他存储失败保持 unknown。 |
+| Usage accounting | Usage accounting | Missing | 尚未分类账务上报与持久化失败；推理用量限制归入其来源阶段（generation，或 embedding adapter 上报时归入 embedding）。 |
+
+第一个落地切片覆盖 Memory / Atomic Memory：provider 失败、预算耗尽、提取/对账校验均有分类，安全计数字段
+可从子 Worker 传播到 supervisor 日志。其他 Family 在各自切片落地前保持现有行为；每次切片在同一变更中更新
+本矩阵。
+
+## 落地顺序
+
+1. 共享诊断契约、日志白名单字段、Worker 传播，以及 Memory / Atomic Memory 切片（本次变更）。
+2. Topic Memory：标注分阶段生成、索引/嵌入与持久化失败。
+3. Experience、Profile、Skill：标注生成输出校验与持久化入口。
+4. Handoff、Prompt：标注生成与发布失败，再补 usage accounting 分类并补全矩阵。
+
+新增 instrumentation 时，需要覆盖可观察行为：分类正确、跨真实 spawn Worker 传播、敏感值不出现，以及诊断
+失败不改变业务结果。

docs/zh/docs/operate/observability.md

@@ -32,6 +32,7 @@ Code 专用诊断字段仅在 `code.*` events 中输出。
 | `request_id`、`trace_id`、`span_id` | 已绑定的 ingress request 和当前 trace context；没有上下文时省略 |
 | `worker_id`、`family`、`binding`、`request_generation`、`supervisor_group`、`supervisor_generation` | 一次 worker attempt 及其已接受的工作和 fence，用于关联 child 与 supervisor logs |
 | `duration_ms`、`error_code`、`error_category` | 毫秒耗时、诊断 code 和有限取值的 operational error category |
+| `stage`、`processing_stage`、`failure_reason`、`input_item_count`、`estimated_input_tokens`、`token_estimator_id`、`effective_budget`、`provider_request_id`、`finish_reason`、`failure_location`、`cause_exception_type` | 处理失败 stage/reason 与安全计数字段；`stage` 表示发出失败的位置或领域位置（Worker/supervisor stage、领域 block 名称或 `worker` 兜底），`processing_stage` 表示有界的领域阶段。详见 [处理失败诊断](../../development/processing-failures.md) |
 
 Server role 为 `all`（API 与 supervisor）、`api` 或 `background`，child process 使用 `worker`。
 显式配置 logging 但没有指定 deployment role 时使用 `server`。JSON identity 字段对应 trace resource attributes：

src/powercontext/builtin/artifacts/atomic_memory/extraction.py

@@ -19,14 +19,20 @@
 from dataclasses import dataclass
 from typing import TYPE_CHECKING, Any, Literal
 
-from pydantic import BaseModel, ConfigDict, Field, JsonValue, TypeAdapter
+from pydantic import BaseModel, ConfigDict, Field, JsonValue, TypeAdapter, ValidationError
 
 from powercontext.artifacts import ArtifactRef
 from powercontext.builtin.artifacts.atomic_memory.models import AtomicMemoryContent
 from powercontext.builtin.artifacts.memory.prompts import MemoryExtractionProfile
 from powercontext.builtin.inference import InvalidInferenceOutputError, StructuredGenerator, TokenEstimator
 from powercontext.builtin.persistence.records import _canonical_source_text
 from powercontext.builtin.persistence.sources import SourceRepository, StoredSource
+from powercontext.builtin.processing_diagnostics import (
+    ProcessingDiagnostics,
+    ProcessingFailureReason,
+    ProcessingStage,
+    attach_diagnostics,
+)
 from powercontext.sources import TEXT_EVIDENCE_PROJECTION_KEY, Source, SourceObservation, SourceRef, TextEvidence
 
 _JSON = TypeAdapter(JsonValue)
@@ -120,13 +126,38 @@ class AtomicMemoryGenerationPipeline:
 
     async def extract(self, value: AtomicMemoryExtractionInput) -> tuple[AtomicMemoryCandidate, ...]:
         result = await self.extractor.generate(value)
-        output = AtomicMemoryExtractionOutput.model_validate(result.output)
+        try:
+            output = AtomicMemoryExtractionOutput.model_validate(result.output)
+        except ValidationError as error:
+            raise attach_diagnostics(
+                InvalidInferenceOutputError("atomic-memory-extract", "output does not match the extraction schema"),
+                ProcessingDiagnostics(
+                    stage=ProcessingStage.OUTPUT_PARSING,
+                    reason=ProcessingFailureReason.SCHEMA_INVALID,
+                    failure_location="atomic-memory-extract",
+                ),
+            ) from error
         supplied = {item.evidence_id for item in value.evidence}
         for candidate in output.candidates:
-            AtomicMemoryContent(kind=candidate.kind, text=candidate.text)
+            try:
+                AtomicMemoryContent(kind=candidate.kind, text=candidate.text)
+            except ValidationError as error:
+                raise attach_diagnostics(
+                    InvalidInferenceOutputError("atomic-memory-extract", "candidate content is invalid"),
+                    ProcessingDiagnostics(
+                        stage=ProcessingStage.BUSINESS_VALIDATION,
+                        reason=ProcessingFailureReason.INVALID_OUTPUT,
+                        failure_location="atomic-memory-extract",
+                    ),
+                ) from error
             if not set(candidate.evidence_ids) <= supplied:
-                raise InvalidInferenceOutputError(
-                    "atomic-memory-extract", "candidate cites unavailable Source evidence"
+                raise attach_diagnostics(
+                    InvalidInferenceOutputError("atomic-memory-extract", "candidate cites unavailable Source evidence"),
+                    ProcessingDiagnostics(
+                        stage=ProcessingStage.BUSINESS_VALIDATION,
+                        reason=ProcessingFailureReason.INVALID_OUTPUT,
+                        failure_location="atomic-memory-extract",
+                    ),
                 )
         return output.candidates