RFC: publish @codeburn/core as a pure parsing and detection library

[laulpogan]

Summary

Publish a standalone @codeburn/core package from this repository. The package would expose:

• pure provider parsers;
• normalized, content-minimized observations;
• pure detector functions whose findings carry structured evidence, confidence, and an algorithm version;
• a documented semantic-versioning policy for every serialized export schema; and
• no file discovery, database opening, subprocess launch, credential lookup, or network initialization.

The end state is one engine and one implementation. @codeburn/core owns every deterministic decode, observation transform, and detector. The existing codeburn CLI becomes Core's first consumer: it keeps discovery, SQLite access, caches, pricing, aggregation, configuration, UI, and apply behavior, then delegates every parse and detect call to Core. This is an extraction of the engine, not a parallel SDK that can drift from the product.

Motivation

CodeBurn already has the hard assets: broad provider coverage, provider-specific normalization, a common ParsedProviderCall boundary, and a substantial detector suite. Those capabilities currently live behind a CLI-oriented package boundary:

• src/providers/types.ts combines pure parsing contracts with source discovery.
• Provider modules commonly read files, databases, environment variables, or network services in the same module that decodes provider records.
• ParsedProviderCall still contains userMessage, so it is not a safe default interchange format.
• WasteFinding has a stable ID and human explanation, but no structured evidence, detector confidence, or algorithm version.
• The published package bundles the CLI and does not expose a supported library entry point or TypeScript declarations.

A small, stable core would let contributors test parser changes without booting the application and let other local tools build reports, dashboards, and integrations over CodeBurn-compatible telemetry. Making the CLI consume Core keeps community work on the live product path.

Goals

1. Make provider decoding deterministic and testable from supplied input.
2. Define one content-minimized observation schema suitable for persistence or transport.
3. Make detector results auditable and comparable across releases.
4. Guarantee that importing or calling Core cannot initialize network access.
5. Publish runtime schemas and TypeScript types under an explicit compatibility policy.
6. Keep the current CLI behavior and provider fidelity during migration.
7. End with every CodeBurn provider parser and detector implemented once, in Core, with the CLI as a real caller.

Non-goals

• Moving file discovery, cache management, pricing refreshes, configuration, or UI code into Core.
• Turning Core into a daemon, service, database, or telemetry collector.
• Preserving raw prompts, responses, tool arguments, shell commands, file contents, or provider payloads in normalized output.
• Migrating all providers before the first preview release. Preview coverage may grow incrementally, but partial extraction is not the final architecture.
• Freezing internal CodeBurn types as public API. Core should expose a smaller purpose-built contract.

Proposed boundary

host adapters
  files · SQLite · protobuf · APIs · credentials · caches
                         |
                         v
                  @codeburn/core
  pure decoders -> minimized observations -> pure detectors
                         |
                         v
host policy
  pricing · aggregation · storage · CLI · web · desktop

codeburn would own every side effect. @codeburn/core would own deterministic transformation.

For local providers, a host adapter discovers and reads the source, then passes strings, bytes, or decoded rows to Core. For SQLite providers, Core receives selected rows rather than opening the database. For network-backed providers, Core may decode an already-received response, but request construction, authentication, retrying, and fetching remain outside the package.

End-state invariant

This RFC proposes a new ownership boundary, not a second implementation beside the CLI.

@codeburn/core owns:

• every provider decoder and parser state machine;
• conversion into normalized, content-minimized observations;
• parser diagnostics and deterministic deduplication rules;
• every detector, evidence builder, confidence rule, and algorithm version; and
• public schemas, types, conformance fixtures, and compatibility migrations.

The codeburn CLI owns:

• file and directory discovery;
• SQLite, protobuf, and network I/O;
• credentials, configuration, caches, and scheduling;
• price resolution, aggregation, storage, and presentation; and
• interactive or mutating features such as UI, guard, optimize apply, and undo.

CLI adapters may select and frame transport records or rows, pass explicit state into Core, and translate Core output into legacy CLI shapes during migration. They may not decode provider semantics or contain parsing rules, observation semantics, detector thresholds, or duplicate detection logic.

The extraction is complete only when every provider registered by the CLI delegates parsing to Core, every detector exposed by the CLI delegates to Core, and the superseded CLI implementations have been deleted.

Package shape

I suggest a workspace package in this repository:

packages/core/
  src/
    schema.ts
    observations.ts
    diagnostics.ts
    providers/
    detectors/
    index.ts
  schemas/
    observation-1.0.0.json
    finding-1.0.0.json
  package.json
  README.md

Illustrative public entry points:

@codeburn/core
@codeburn/core/schema
@codeburn/core/providers/claude
@codeburn/core/providers/codex
@codeburn/core/detectors

The root entry point should stay small. Provider subpath exports let consumers load only the formats they use. The package should publish declarations, declare sideEffects: false, and expose ESM through an explicit exports map.

Parser contract

Each parser should accept caller-supplied records plus explicit context and state. It should never discover its own input.

const result = parseClaude({
  records,
  context: {
    sourceId,
    projectId,
    privacyKey,
  },
  state: previousState,
})

The exact API can vary by source format, but every parser should obey these rules:

• Same input, context, and state produce byte-equivalent observations and diagnostics.
• Incremental state is explicit, serializable, and returned to the caller. No module-global deduplication or cursor state.
• Record failure is isolated. One malformed record produces a content-free diagnostic without discarding valid sibling records.
• Stable observation IDs derive from provider identity and record identity. They must not embed prompts, paths, commands, or other sensitive content.
• Parsers preserve separate input, output, cache-read, cache-write, and reasoning token classes when the provider supplies them.
• Measured, derived, and estimated values remain distinguishable.
• Provider-reported cost may be preserved as a measurement. Model pricing and counterfactual cost calculation remain host policy for the initial release.

The package should publish supportedProviders() metadata so preview coverage is explicit during migration. A provider is supported only when the CLI wrapper calls its Core parser and fixture tests cover that path. This manifest describes transitional coverage; it does not weaken the end-state requirement that every CLI provider moves to Core.

Normalized observation schema

The initial schema should be intentionally smaller than ParsedProviderCall. A possible envelope:

type ObservationEnvelope = {
  schema: 'codeburn.observation'
  schemaVersion: '1.0.0'
  id: string
  kind: 'usage' | 'activity'
  eventAt: string
  provider: string
  sourceFormat: string
  sourceFormatVersion?: string
  sessionId: string
  turnId?: string
  callId?: string
  projectId?: string
  quality: {
    basis: 'measured' | 'derived' | 'estimated'
    confidence: number // 0..1
    caveats?: QualityCaveat[]
  }
}

A usage observation adds model identity, speed, token classes, request counts, and optional provider-reported cost. An activity observation adds a normalized operation such as read, edit, execute, search, tool, agent, or skill.

Content minimization should be a schema invariant, not a caller convention:

• No userMessage, prompt, response, thinking text, tool arguments, raw command line, file contents, attachments, or raw provider record.
• No absolute home directory or raw working directory.
• Observation, source, project, session, turn, and call IDs are opaque stable identifiers or keyed fingerprints, never raw provider identifiers by default.
• Repeated-resource detection uses a keyed resourceFingerprint supplied through explicit parser context.
• Path-sensitive detectors use a coarse resourceClass, such as source, dependency, build, vcs, or unknown, rather than the original path.
• Command analysis uses a safe commandFamily, such as git, test, or build, rather than arguments.
• Free-text source fields are normalized into bounded identifiers or controlled enums before export. Unknown or oversized values produce diagnostics rather than becoming observation text.
• Diagnostics may include record index, byte offset, controlled error code, and a keyed fingerprint, but never rejected content or raw exception messages.

Runtime validation should use the same source definition as the TypeScript types. Published JSON Schemas make the contract usable outside TypeScript.

Detector contract

Detectors should be ordinary pure functions over normalized observations or summaries derived from them:

const findings = detectDuplicateReads(observations, detectorOptions)

Every finding should include:

type Finding = {
  schema: 'codeburn.finding'
  schemaVersion: '1.0.0'
  id: string
  detector: {
    id: string
    algorithmVersion: string
  }
  subject: {
    projectId?: string
    sessionId?: string
  }
  evidence: Evidence[]
  confidence: {
    score: number // 0..1
    basis: ConfidenceBasis[]
  }
  impact: 'low' | 'medium' | 'high'
  estimate?: {
    value: number
    unit: string
    method: string
  }
  title: string
  explanation: string
}

Evidence must be machine-readable. For example, a repeated-read finding could cite observation IDs plus observed: 7, threshold: 3, and unit: 'reads'. A prose explanation may summarize that evidence but cannot replace it.

Confidence measures how strongly the recorded evidence supports the detector's classification on its documented scale. It does not express causal certainty, business value, or savings accuracy. Estimated savings therefore remain a separate, labeled field with their own method.

QualityCaveat and ConfidenceBasis should be documented controlled vocabularies, not free-text escape hatches. Human titles and explanations must be generated only from normalized identifiers and aggregate metrics.

Every detector stamps an algorithm version. Threshold or classification changes must bump that version so consumers can compare like with like and invalidate cached findings safely.

Side-effect and network guarantee

Importing any @codeburn/core entry point must not:

• inspect the home directory or environment;
• open files or databases;
• spawn a process;
• read credentials;
• bind a socket; or
• call fetch, http, https, net, tls, or a provider SDK.

Calling a parser or detector must obey the same rule. Network-backed provider support means pure response decoding only.

CI should enforce this with an import smoke test that replaces network primitives with throwing stubs before importing every public entry point. A second test should call representative parsers and detectors under the same guard.

Versioning policy

Package, serialized schema, and detector algorithm versions solve different problems and should advance independently.

Package version

@codeburn/core follows semantic versioning for its exported TypeScript and JavaScript API.

• MAJOR: remove or rename an export, make an optional argument required, or change an exported value's meaning.
• MINOR: add an optional argument, export, provider parser, or detector without breaking existing calls.
• PATCH: compatible bug fixes, performance improvements, and documentation changes.

Observation and finding schema versions

Every serialized object carries schema and schemaVersion.

• MAJOR: remove or rename a field; make an optional field required; change a type, unit, range, nullability, or meaning; or add a closed-union variant that can break exhaustive consumers.
• MINOR: add an optional field whose absence preserves current meaning.
• PATCH: clarify documentation or validation without changing the accepted shape or field semantics.

New providers do not require a schema bump when they emit an already-supported shape.

Core should retain parsers and validators for the current schema major plus at least the previous major. A migration function should upgrade the previous major to the current one when the conversion is lossless and unambiguous.

Detector algorithm versions

Each detector uses semantic versions for its decision procedure.

• MAJOR: change what the finding claims or make results incomparable with prior output.
• MINOR: add evidence or improve precision while preserving the finding's meaning.
• PATCH: correct an implementation bug within the documented procedure.

Golden tests should pin schema versions and algorithm versions independently of the package version.

Migration plan

1. Contract first. Land schemas, parser/detector interfaces, privacy invariants, and conformance helpers.
2. Prove three source shapes. Extract Claude JSONL, Codex JSONL, and one SQLite-backed provider. This exercises line-oriented, stateful, and row-oriented parsing.
3. Wire the CLI immediately. Existing provider wrappers retain discovery and I/O but delegate decoding to Core. Compatibility adapters preserve current ParsedProviderCall behavior during migration.
4. Extract representative detectors. Start with duplicate reads, low read/edit ratio, and context bloat. They exercise activity fingerprints, thresholds, token metrics, evidence, and confidence.
5. Publish a bounded preview. Publish only after the CLI consumes every parser and detector exported by that preview and the side-effect/network gates pass. The supported-provider manifest documents its exact transitional coverage.
6. Complete the extraction. Move every remaining CLI provider parser and detector into Core, wire the CLI to each Core implementation, and delete the superseded implementations.
7. Enforce the boundary. Add an architecture test or lint rule that rejects new provider parsing or detector decision logic outside Core.

No parser or detector should exist in both packages after its migration. One implementation serves Core, the CLI, and tests. A preview release is a migration checkpoint, not the target architecture.

Conformance and acceptance criteria

• Importing every public entry point performs no file, database, subprocess, credential, environment, or network initialization.
• Each parser passes deterministic golden fixtures and malformed-record isolation tests.
• CLI fixture output remains equivalent for every migrated provider.
• Every provider registered by the CLI delegates parsing to Core at migration completion.
• Every detector exposed by the CLI delegates detection, evidence, confidence, and algorithm stamping to Core at migration completion.
• CLI compatibility adapters perform I/O and type translation only; they contain no provider parsing rules or detector decisions.
• An architecture test or lint rule prevents new parser or detector implementations from landing outside Core.
• The CLI is a real caller of every parser and detector exported by each Core release.
• Runtime schemas, generated JSON Schemas, and TypeScript declarations agree.
• A content-smuggling fixture proves prompts, responses, commands, paths, tool arguments, and unknown provider fields cannot enter observations, findings, or diagnostics.
• Every finding has non-empty structured evidence, a confidence score and basis, and an algorithm version.
• Package, schema, and detector versioning policies appear in the published README.
• Provider coverage is machine-readable and tested.
• The package can be installed without the CLI, desktop assets, pricing bundle, or native provider dependencies.

Open questions

1. Should stable fingerprints require a caller-provided privacy key, or should Core also offer an explicitly weaker deterministic default for simple local use?
2. Which provider subset should define the first preview: Claude and Codex only, or Claude, Codex, and one SQLite provider?
3. Should the first release include a pure price-snapshot rater, or keep provider parsing and pricing fully separate until the observation contract settles?

If this direction is acceptable, the first implementation step would be a small contract PR: package scaffold, schemas, conformance tests, and one migrated parser wired back into the CLI. That PR starts the extraction; completion still requires every CLI parser and detector to move to Core under the end-state invariant above.

[laulpogan]

Thanks—this is the intended end state. I edited the RFC to make it an explicit invariant rather than an implication.

The revision now states that:

• @codeburn/core becomes the only implementation of provider decoding, content-minimized observation transforms, and pure detection;
• the CLI keeps all side effects and host policy, then delegates every parse and detect call to Core;
• compatibility adapters may perform I/O framing and type translation only;
• preview provider coverage is transitional, not the target architecture;
• migration completes only after every registered CLI provider and exposed detector delegates to Core and the superseded implementations are deleted; and
• an architecture gate prevents new parser or detector decision logic from landing outside Core.

That gives us the long-term single-engine architecture you described while preserving an incremental migration path.