Repository navigation
Sources: a workspace of steps — flows with any number of steps, Output and NATS topics (#104) - #110
Merged
Merged
Conversation
FlowCanvas with palette groups for the Sources flow editor. Every release since 0.2.1 is additive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since stareSDK 0.2.10 CodeEditor imports @codemirror/autocomplete and @codemirror/lang-html, so the production build fails without them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cargo deny reported 28 advisories against main: - rustls 0.23.46: RUSTSEC-2026-0346. - rustls-webpki 0.103.16, webpki-roots and webpki-root-certs 1.0.10: distrusted emSign intermediates (RUSTSEC-2026-0381, -0382, -0383). - wasmtime and wasmtime-wasi 49.0.4: RUSTSEC-2026-0321 to -0327, -0352 to -0364, -0378, -0379. - zip 9 (RUSTSEC-2026-0367 to -0369), through calamine 0.37 and rust_xlsxwriter 0.101. - yoke-derive 0.8.4 replaces a yanked release. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source that is not complete yet saves as a draft that cannot be enabled, instead of being refused: - SourceSpec::problems lists every problem, each with its stage (source, decode, map, ..., output) and setting. Only an invalid id or name stops a save. validate() still returns the first, as before. - Empty transport settings are reported by name. - MappingSpec::problems reports every rule's problems. - Sources store their problems (migration 0020); a draft cannot be enabled, in the store as well as the API. - PUT on an enabled source is refused unless the spec is complete; enable re-checks. - A refusal is a 422 whose body lists the problems beside the error. - Configuration backups carry drafts; an enabled source in a file must be complete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Enable stores what checking again finds, so a draft whose problem was resolved elsewhere (a schema published since) can be enabled without re-saving. A refused enable keeps the problems without logging a decision. - The store refuses to save problems onto an enabled source inside its write, so a concurrent enable cannot leave it running a draft. - One draft test (a non-empty JSON array) everywhere; an import's non-array problems read as none. - Transport checks run even with an empty setting; a gRPC source's protobuf compiles once. - Config import lists every problem of a source. - validate_against is gone (problems covers it); ApiError has one constructor. - The client keeps a refusal's problems on ApiError. The Add Source wizard updates a source it already saved (a draft) instead of creating it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Phases 3 to 5 of #104: - sourceFlow.ts: a source as steps in the pipeline's fixed order, which connect themselves; the palette's transports, codecs (with codec plugins), stages and outputs; adding a stage puts it in its place, removing one closes the gap; Source, Decode, Map and Output stay. - SourceEditor at #sources/<id>: the palette and the steps on the left; the selected step's settings and samples, or the source's own settings, status and history, on the right; problems on their steps, live from the dry run; Enabled, Save, Delete and a whole-spec JSON view. - Capture samples on the Source step; the detected decode settings and the suggested mapping are offered, never applied unasked. - Add source asks for a name, id and transport, and saves a draft with its four steps. - The transport form shows only the Source or the Decode step's part, and gains the file transport. - The dry run (/sources/validate) runs a draft as far as its decode, map and tracker stages allow, and lists the problems. - The wizard, the source drawer, the pipeline designer and their views are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The JSON view takes only text with a source's shape, so a half-typed pipeline cannot crash the editor. - A step taken out another way (JSON, emptying the Reject rules) is no longer selected, so its settings are not shown without a value. - Switching codec takes away the framing the old one set (CoT's end tag, a plugin's own); framing set by hand stays. - The dry run says why a pipeline cannot run when the checks did not catch it, on its stage, instead of 'invalid source'. - The sources list stops polling while a source is open. - The editor's state badge is sourceState's; the dead Transport-tab codec helpers are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ery problem (#104) - Static join is shown while the Map has a static rule, as the pipeline runs it; it is not added or removed on its own. - A dry run that cannot run lists what stops it first, then the rest, so every step shows what it needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The settings list them all; long messages no longer spill onto the next step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pipeline is a flow: one input (transport and codec), then Reject, Map, Filter, Entity links, Affiliation, Tracker and Throttle steps the user connects, any number of each, branching to the source's one Output (system tracks) and any NATS topics (mapped data as JSON). The older fixed form reads as the same stages in a line, so every existing spec, example and test runs unchanged. Problems name their step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The worker runs a source's flow: NATS topic steps publish what reaches them (with the source's security label) on their subjects; a Tracker step anywhere is run on its timer; static caches are kept per Map. - Problems name their step (node) instead of a fixed stage; the dry run says on which step a flow cannot run. - Raw output and its consent are gone: a source saved with a raw-output subject gets a NATS topic step beside its Output, once, on start, and a configuration file's raw output is read the same way. The raw-output endpoints are removed. A NATS topic's subject must lie outside the system tracks' subjects. - The engine reads a source's untracked detections and entity links from its Output's path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The editor is a full workspace: the step list on the left (Records: Reject, Map; Observations: Filter, Entity links, Affiliation, Tracker, Throttle; Outputs: Output, NATS topic), steps dragged anywhere and connected by hand, one input each, any number of each step, branching to the source's one Output and any NATS topics. Clicking a step opens its settings and samples over the workspace; the input's are the transport and codec. The feed's own settings, status and history open from the top row. Connections that would feed a step the wrong data, a second input or a loop are refused as they are drawn. The topology on the Sources list draws each source's flow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workspace narrows rather than hides steps under the panel; the status no longer lists a raw output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The admin guide, reference and NATS output docs describe the source workspace, flows, NATS topic steps and their migration from raw output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
phornstein
changed the title
The workspace keeps its full width; the settings (or the feed's) open over its right side in stareSDK's SideNav, resizable, scrolling, and closed with its close control or Escape. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
phornstein
marked this pull request as ready for review
October 22, 2026 23:28
# Conflicts: # CHANGELOG.md # crates/ot-server/src/api.rs # crates/ot-server/src/config_backup.rs # crates/ot-source/src/source.rs # ui/src/api/client.ts # ui/src/pages/sources/AddSourceWizard.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phases 3–5 of #104, reworked from the fixed-order editor to the free workspace you asked for. Stacked on #105, #107 and #108; still a draft until you've tested it on 8091.
What you see
Workspace. A full-width workspace with the step list on the left:
Drag steps anywhere and connect them by hand. Every step except Output can be used any number of times, and a step can feed several, so a flow can branch.
Connection rules, enforced as you draw. One input per feed and one input per step, no loops, and each step only takes the data it works on: records before a Map, observations after.
Outputs. Output sends system tracks to correlation (one per feed). Each NATS topic publishes the mapped data as JSON on its own subject, without correlation.
Step settings. Clicking a step opens its settings beside the workspace, with its samples. The input's settings are the transport and the codec. Problems show on their step, live.
Feed (top row) opens the feed's own settings: name, description, tracks or detections, publish-alone, confirm-after, priority, security label and emitter motion. Its status and history are there too.
JSON shows the whole source and fills the page height.
The Sources list topology draws each source's flow.
Underneath
ot-source::flowstores the pipeline as{codec, nodes, edges}, and the pipeline runs it as a graph. A spec in the old fixed form reads as the same stages in a line, so every existing spec, example and pipeline test (including the STANAG 4607 replays) runs unchanged. 112 ot-source tests pass.node), and the dry run says which step a flow can't run on.op:migration)./raw-outputendpoints are removed.tracks.>.Checked on the dev server
cargo fmt, clippy-D warnings,cargo test --workspace.tsc -b, oxlint 0 errors, vitest 67/67,vite build.Docs and CHANGELOG are updated.
🤖 Generated with Claude Code