Repository navigation
refactor(cti)!: one CTI store, and the whole bundle in context - #42
Merged
Merged
Conversation
cti_technique_grounding anchors each technique against the source text and stores the surrounding sentence as x_cti_evidence, plus x_cti_confidence for how strong the anchor was. make_attack_pattern looked for those under the bare names `evidence` and `confidence`, never found them, and the guards that write them onto the SDO never fired. The effect was that a bundle carried only MITRE encyclopedia prose, identical for every report citing the same technique, and nothing of what the report actually said. Two reports describing completely different intrusions produced the same grounding text. Both spellings are accepted so a hand-written IR keeps working.
create_windows_ability built {"name": "windows", "platform": "psh"} and
create_linux_ability built {"name": "linux", "platform": "sh"}. Both are the
wrong way round.
Caldera matches an ability to an agent with
ability.find_executors(self.executors, self.platform) in
app/objects/c_agent.py, whose signature is find_executors(names, platform):
the name is the runner the agent supports, the platform is its OS. Transposed,
no agent matches, so every ability the author workflow created was
undispatchable no matter how good the command was.
Verified by reading core, not by running an operation.
A bundle this pipeline produces holds tens of objects, not thousands. The blackcat fixture yields 35 across 137 STIX objects, about 13k tokens, which fits a modern context window several times over. Embedding that to retrieve three chunks threw away the other thirty-two, and the query it retrieved against was the operator's raw prompt, which rarely contains technique language, so the three it kept were close to arbitrary. Published long-context evaluations put the point where retrieval starts to pay at the reader's effective context window, far above a single report. The rag capability becomes the cti capability: read every attack-pattern and threat-actor the attached bundles carry, render them in document order, and hand the lot over. Retrieval, the embedding model, its endpoint, its credentials and topk are gone, and so is the sentinel string the empty-corpus path returned, which was truthy and reached the model as intelligence. A selection larger than cti.context_budget_tokens is trimmed from the end, keeping objects whole, and the prompt states how many were dropped so the agent can call a CTI tool for the rest. Two behaviours change beyond the swap: - The ATT&CK id now reaches the prompt. The old chunk was "name | description" and left external_id in external_references, so the agent could not cite or pass an id it had just been shown. 34 of 35 objects in the fixture carry one. - A capability the operator enabled no longer fails open. Every enrich exception was logged at WARNING while the run finished green and ungrounded, which looked identical to a grounded run. It now stops the run. author also gains cti_pipeline as an optional server. It declared optional_servers=[], so the CTI tools were filtered out of its sidebar and it was structurally cut off from the pipeline whose output it is meant to build from.
Author's picker was backed by GET /plugin/mcp/rag/list, which globs plugins/mcp/data/*.json. The pipeline writes to data/outputs_stix/, so the picker listed a directory the pipeline never touches and Author's file list was empty on every install. The only file that would ever appear there is data/sessions.json, the plugin's own chat transcript. GET /plugin/mcp/stix/list already served the right directory with better metadata, so the duplicate is deleted rather than repointed, along with /plugin/mcp/rag/upload, which wrote bundles into the same unreachable place. Nothing globs data/*.json now, so sessions.json stops being offered as intel. The sidebar had grown two incompatible halves of one feature: a Capabilities checkbox plus a RAG panel for Author, a CTI section plus an embedding-config block for Plan and Execute, each mounted for one workflow and hidden from the other. Both are replaced by a single Intel section gated on the workflow declaring the cti capability, which both do. Gone with it: the embedding model, endpoint, key, TLS toggle and TopK inputs, in both copies. They configured a retrieval step that no longer exists. The Temperature, Tool calls and Max tokens inputs went too; they reached the payload and were read by nobody. Attaching is now the whole configuration, so the capability checkbox is gone as well. It never gated the panel, ticking it alone did nothing, and selecting files enabled the capability whether it was ticked or not, so it taught a control relationship that did not exist. The nine cti_rag_* keys the sidebar emitted are replaced by one attached_intel list. The old emit was refreshed by a watch that observed two of its nine sources, and the panel holding the other seven only rendered after a bundle was selected, so those inputs were stale by construction. The listing now reports how many attack-pattern and threat-actor objects each bundle carries, and the picker shows the total for what is attached. A grounded run and an ungrounded one were previously indistinguishable.
The Author signature says "Do NOT run operations or deploy infrastructure", but caldera_core is a required server and exposes core_create_operation and core_add_link_to_operation, and cti_pipeline is now optional there and exposes cti_pipeline_run_operation. Server scoping is all-or-nothing, so Author needed most of caldera_core while having no way to refuse three of its tools. That made the docstring the entire boundary between an authoring run and a live operation on a real agent. denied_tools on Workflow closes it. The tool is dropped while the run assembles its list, so the agent is never offered it and cannot call what it cannot see. Filtering happens before the name-collision check so a denied name does not occupy the namespace either. Author denies the three operation starters and keeps everything it needs, including cti_pipeline_build_adversary: authoring an adversary is its job, running one is not. Plan and Execute denies nothing, which is the point of it. Read via getattr in mcp_svc, since a workflow contributed by another plugin may predate the field. A denylist fails open, so the test asserts coverage rather than a fixed list: it reads both servers for tools matching the operation-starting naming convention and fails if Author can reach one it does not deny. Verified it fails by removing cti_pipeline_run_operation from the declaration, which named exactly that tool.
Three things the refactor left behind, found reviewing its own diff. context_budget_tokens was readable but not writable. profile_defaults does not apply the allowlist so the capability read it fine, but set_config gates the cti section on WORKLOAD_OVERRIDABLE and the key was not in it. A deployment on a short-window model could not lower the budget, which the comment in default.yml told them to do. A bundle deleted on the CTI page stayed attached. ChatRagPanel used to drop selections missing from the listing; nothing did after it was deleted, so the name survived in localStorage and the next run failed in _enrich with "attached CTI bundle not found". The listing now prunes what is gone. The sidebar emitted attached_intel into workflowContext while ChatWorkflow derived the same value from the session ref and overwrote it. That is the two-sources-for-one-value shape this branch removed, reintroduced in miniature, so the emit and the whole workflowContext channel go. Also removed with them: embed_model, rag_topk and rag_embed_model from the config allowlists, the embed_model fallback in llm_client, and the block in mcp.vue that seeded capabilitySettings.rag from /defaults keys that no longer exist. All configured a capability that is gone.
CTI is the vocabulary operators already use, so the section that holds attached bundles carries it on both workflow sidebars. The controls inside still act on intel, which is the object: you attach intel, in the CTI section. The heading is what a rebuild of magma changes, so this needs one.
author declared its own cti_context description while plan_execute used the shared CTI_CONTEXT_DESC, so the two workflows were told different things about the same field. author's copy said only "relevant CTI information including attack patterns, techniques, and threat actor behaviors" and dropped the guardrails: that the field is technique-level only and never carries hosts, accounts, domains, software or infrastructure, and that it is grounding rather than permission to invent what is missing. Author is the workflow that writes commands into abilities, so it is the one that most needs to be told not to read targeting facts out of a report.
The two grounded signatures were still DSPyCalderaFactoryClientWithRAG and DSPyCalderaPlannerClientWithRAG, and the stage tag still read "executing DSPy ReAct with RAG", for a code path that no longer retrieves anything. The MLflow stage is operator-visible, so it was telling them the run did something it did not. data/stix_cti also survived in three _resolve_pipeline_file candidate tuples. No code writes it: upload was redirected to outputs_stix with a comment saying writing there made every bundle invisible to list, get, download and delete. Keeping it as a lookup candidate meant a stale copy in a directory nothing maintains could win over pipeline output. The 'inputs' candidate goes with it. No such directory exists.
technique_id is optional on both ability builders and nothing derives it from technique_name, so the model can create an ability with no ATT&CK id at all. cti_pipeline_build_adversary matches stockpile abilities on technique_id, so an untagged ability is invisible to it: the operator authors an ability to cover T1059.001, re-runs coverage, and still sees T1059.001 unmatched. Stated in the field description rather than enforced. The id belongs to the technique, and the model has it whenever the id came from cti_context, so this gives it the reason without hardcoding a requirement that would break the cases where no technique is known.
format_plan_execute_context still read workflow_context["selected_stix_files"] and "cti_uses_rag", but the GUI stopped sending either when the nine emitted cti_rag_* keys collapsed to one attached_intel list. Neither key arrives now, so operation_context was always empty and the planner was never told what the operator attached. It could not name a source or pass a real filename to cti_pipeline_build_adversary, so it would have had to guess one. Dropped cti_uses_rag rather than renaming it. It announced that bundles were "also enabled as RAG context", describing a retrieval step that no longer exists and a mode the operator could not turn off. Note that the suite did not catch this: nothing asserts the GUI's workflow_context keys against what the prompt formatter reads, and both sides were internally consistent.
Replacing three sidebar sections with one removed a contiguous block that also held eight definitions the surviving template still references. Vue resolves a missing binding to undefined and renders nothing, silently, so neither pyflakes nor the suite could see it and the section looked correct until an operator attached a bundle. Broken by it: - timesIcon, so the detach button on an attached bundle and the picker's close button rendered as empty squares. This is what the operator hit. - folderOpenIcon on the Attach button and collapseIcon on the rail toggle. - modelToggleIcon, endpointProfiles, selectedEndpointProfileName, endpointProfileDraftName and modelConfigSummary, which between them are the whole LLM Endpoint section that Plan and Execute renders: the profile dropdown, the draft-name input and the summary label. All eight are restored verbatim from 02ad21a^, since none of them had anything to do with retrieval. Found by diffing template bindings against script declarations rather than by reading, because reading is what missed them the first time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Four separate breaks between the CTI pipeline and the workflows that consume
it, all of which failed silently.
Author could not see pipeline output. Its picker globbed
data/*.json;the pipeline writes
data/outputs_stix/. The list was empty on every install,and the only file that would ever appear there was
data/sessions.json, theplugin's own chat transcript.
The report's own words never reached the bundle. Grounding stores each
technique's source sentence as
x_cti_evidence; the STIX builder read it asevidenceand dropped it. Two unrelated intrusions citing the same techniqueproduced identical grounding text.
Retrieval discarded most of the rest. A bundle holds tens of objects, not
thousands, so embedding it to pick three threw the rest away for no gain, and
the query was the operator's raw prompt, which rarely contains technique
language. Published long-context work puts the point where retrieval starts to
pay at the reader's context window, far above one report, so the crossover is
now a token budget rather than a
k.Measured on the same APT29 bundle:
The sidebar had two incompatible halves of one feature, each mounted for
one workflow and hidden from the other. Now one CTI section, gated on the
workflow declaring the capability, which both do. Attaching is the whole
configuration, so the capability checkbox is gone: it never gated the panel and
selecting files enabled the capability regardless.
Author also declared
optional_servers=[], so the CTI Pipeline server wasfiltered out of its sidebar entirely. Optional now, which makes the coverage
loop work:
build_adversaryreports what the stockpile cannot cover, and thatis what Author should write abilities for.
Two fixes alongside, because grounding is pointless if the output cannot run:
executor
name/platformwere transposed in both ability builders so nothingAuthor created could match an agent, and
technique_idnow states that anuntagged ability is invisible to
build_adversary.Finally
denied_toolsonWorkflow. Author's signature says not to runoperations while
caldera_corehands itcore_create_operation, so adocstring was the entire boundary. Tools are now dropped as the run assembles
its list.
Type of change
Breaking: capability id
ragbecomescti(stored selections prune once);/plugin/mcp/rag/uploadand/rag/listdeleted,/stix/*already served theright directory; payload shape is now
{cti: {cti_files: [...]}};data/stix_ctiis no longer a lookup candidate.How Has This Been Tested?
pytest plugins/mcp/tests: 470 passed, 42 of them live-endpoint testsagainst a running Caldera. Suite confirmed inert:
data/count andconf/local.ymlabsence identical before and after.pyflakesclean.npm run buildin magma, bundle grepped for the newstrings,
package-lock.jsonrestored.tests/test_workflow_tool_scope.py, proven to fail by removingcti_pipeline_run_operationfrom the declaration.18 ATT&CK ids, 5 with report evidence, ~7,105 tokens inside the 24k budget.
Budget 800 trims and announces. Empty corpus returns
''. A bundle with nobehavioural objects raises.
every chat component.
The executor fix was verified against
app/objects/c_agent.pyin core ratherthan by dispatching an ability.
Checklist: