Skip to content

Sources: drafts, and every problem on its pipeline stage (#104) - #108

Open
phornstein wants to merge 3 commits into
mainfrom
feat/104-source-drafts
Open

phornstein wants to merge 3 commits into
mainfrom
feat/104-source-drafts

Conversation

@phornstein

Copy link
Copy Markdown
Collaborator

Phase 2 of #104: the backend for drafts and structured errors.

Behaviour (as decided in #104):

  • Drafts. A source that isn't complete yet saves as a draft instead of being refused. Only a spec that doesn't parse, an invalid id or name, or a disallowed file path stops a save.
  • Drafts can't be enabled. This is enforced in the API and in the store. Enable checks the source again and stores what it finds, so a draft whose problem was resolved elsewhere (for example a schema published since) can be enabled without being saved again.
  • An enabled source is saved only when complete. Otherwise the save is refused and the running source is kept. This is enforced inside the store's write, so a concurrent enable can't get around it.
  • Every problem at once. A refusal is a 422 with {error, problems: [{kind: invalid|incomplete, stage, field?, message}]}. stage is source, decode, map, static_join, registry, affiliation, filter, tracker, throttle or output, and null for the source as a whole. error is still the first problem's message, so existing callers keep working.
  • Rows carry their status. Each row has draft and problems, kept in a new column (migration 0020, SQLite schema 20). Sources saved before this migrate as complete.
  • Empty transport settings are named: host, port, URL, bind address, topic, method or path. Every mapping rule's problems are reported, not just the first.
  • Config backups carry drafts. An enabled source in an imported file must be complete, and every problem it has is listed.

Code

  • SourceSpec::problems() (ot-source) does all the checking. validate() returns its first result, as before; validate_against is removed.
  • TransportConfig::missing() and MappingSpec::problems() are new.

UI. Nothing visible changes. The client types gain draft, problems and SourceProblem, and ApiError.problems is kept for the editor in phase 4. The current Add Source wizard now updates a source it has already saved as a draft, instead of trying to create it again (which returned 409).

Review. /code-review high found 10 issues. All 10 are fixed in the second commit.

Gate (dev server): cargo fmt --check and cargo clippy --workspace --all-targets -D warnings pass. cargo test --workspace passes except nodes_share_one_picture_under_one_number, the date bug fixed by #106. UI: tsc -b passes, oxlint has 0 errors, vitest 58/58.

🤖 Generated with Claude Code

phornstein and others added 2 commits October 21, 2026 15:32
A source that is not complete yet saves as a draft that cannot be
enabled, instead of being refused:
- SourceSpec::problems lists every problem, each with its stage
  (source, decode, map, ..., output) and setting. Only an invalid id or
  name stops a save. validate() still returns the first, as before.
- Empty transport settings are reported by name.
- MappingSpec::problems reports every rule's problems.
- Sources store their problems (migration 0020); a draft cannot be
  enabled, in the store as well as the API.
- PUT on an enabled source is refused unless the spec is complete;
  enable re-checks.
- A refusal is a 422 whose body lists the problems beside the error.
- Configuration backups carry drafts; an enabled source in a file must
  be complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Enable stores what checking again finds, so a draft whose problem was
  resolved elsewhere (a schema published since) can be enabled without
  re-saving. A refused enable keeps the problems without logging a
  decision.
- The store refuses to save problems onto an enabled source inside its
  write, so a concurrent enable cannot leave it running a draft.
- One draft test (a non-empty JSON array) everywhere; an import's
  non-array problems read as none.
- Transport checks run even with an empty setting; a gRPC source's
  protobuf compiles once.
- Config import lists every problem of a source.
- validate_against is gone (problems covers it); ApiError has one
  constructor.
- The client keeps a refusal's problems on ApiError. The Add Source
  wizard updates a source it already saved (a draft) instead of creating
  it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@phornstein phornstein added the sources Sources, transports, codecs, pipelines label Oct 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sources Sources, transports, codecs, pipelines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant