Repository navigation
fix(cti): make the plugin and CTI pipeline work on a clean clone - #36
Merged
Merged
Conversation
Adds api_base support so the MCP plugin can target an OpenAI-compatible
gateway (e.g. AIP) instead of api.openai.com. Includes ${VAR} expansion
in default.yml so credentials live in the environment, not the file.
When yaml configures api_base + model, the yaml model wins over any GUI
submission — the gateway has a constrained model list, and stale
localStorage defaults like "gpt-4o" would otherwise produce a 401.
Backend changes flow api_base end-to-end: factory + planner clients,
the subprocess DSPY_API_BASE env, and the global dspy.configure path
in mcp_svc.
Adds an Enabled MCP Servers checkbox list to the Global Model Config panel. On mount, fetches /plugin/mcp/servers, defaults checked state to each server's default_enabled, persists picks to localStorage, and ships enabled_servers in the factory and planner POST payloads so the backend actually spawns the chosen subset. Without this, every run silently fell back to caldera_core only, so plugin-provided servers like RANGE were never reachable from the UI. Also bumps the form's default model to openai/nemotron-3-super to match the new yaml default; backend has a yaml-authority rescue for existing localStorage caches stuck on gpt-4o. Requires a magma rebuild (npm run build) for the bundle to pick this up.
Three layers, runnable independently with pytest:
1. discovery — server_registry picks up range/mcp_server.py and
parses MCP_METADATA via ast.literal_eval
2. spawn — range/mcp_server.py boots over stdio and exposes
range_-prefixed tools (no name collisions)
3. multi-merge — mcp_factory_client.run() spawns both servers and
merges tools without collision; LLM call is stubbed
so no API key is needed
Catches regressions where discovery breaks, range tools lose their
namespace prefix, or the multi-session merge stops working.
Two related observability fixes that together unblock the Ability Factory trajectory in MLflow. mcp.dspy.autolog() was commented out in the factory client, so every factory run completed without writing thought_/tool_name_/observation_ tags. The /plugin/mcp/status route reads exactly those tags, so the GUI saw an empty trajectory and polled forever waiting for it to fill in. Re-enabling it brings factory parity with the planner client. mcp_svc._run_execution() also called dspy.configure(lm=...) in a worker async task, which raised "can only be called from the same async task that called it first" the moment a second request landed. The call was caught in try/except, so it never crashed — but it was redundant: both clients already wrap react.acall in dspy.context(lm=...) which is task-local. Removed the configure block (and now-dead _yaml_llm import) so the warning stops printing.
Adds /plugin/mcp/defaults so the UI can pull yaml-resolved config (model, api_base, temperature, max_tokens, max_tool_calls, rag_embed_model, rag_topk) from a single source instead of duplicating hard-coded defaults in mcp.vue. api_key is always returned blank — the user supplies their own. Fields the yaml doesn't (yet) name come from a small _CONFIG_FALLBACKS dict so adding a new yaml key automatically flows through. Wires /plugin/mcp/history/runs and /plugin/mcp/history/run — the mcp_history.vue tab calls these endpoints and the handlers (list_runs, get_run_detail) already existed in mcp_api, but no route was registered so the History UI silently failed. Removes the unused setup_routes() function in mcp_api.py — hook.py registers all routes inline; setup_routes was never called.
Stops hard-coding model, temperature, max_tool_calls, max_tokens, ragEmbedModel, and ragTopK in mcp.vue. On mount, fetches /plugin/mcp/defaults and fills any field the user hasn't already set via localStorage. This kills the drift class that produced the "GUI=gpt-4o vs yaml=openai/nemotron-3-super" 401 — there is now one source of truth (conf/default.yml plus _CONFIG_FALLBACKS in mcp_api), and the UI reflects it. User overrides via localStorage still win; the fetch only fills blanks.
Reorganize the flat app/ directory into the layered structure documented in REFACTOR_PLAN.md. Pure mechanical moves with no logic changes; all imports updated, syntax checks pass, integration tests still pass. app/mcp_factory_client.py -> app/workflows/author.py app/mcp_planner_client.py -> app/workflows/plan_execute.py app/server_registry.py -> app/discovery/servers.py app/rag.py -> app/capabilities/rag.py app/factory.py -> app/dspy_env.py The caldera_core extension half (app/mcp_server.py and app/mcp_tools/) stays at the top of app/ deliberately. Putting it under app/mcp/ would create the path doubling plugins.mcp.app.mcp.tools, which only affects the plugin literally named mcp. Updated imports in hook.py, mcp_svc.py, mcp_api.py, mcp_server.py, and the integration test file. Updated the in-test fallback path in workflows/author.py and workflows/plan_execute.py so mcp_server.py is located one directory up from the new workflows/ folder.
Defines the two extension contracts the discovery layer will use to register plugin contributions. No registry consumers yet; this is pure type definition so future commits can land incrementally. Workflow ties an agent persona (DSPy signature) to its server scope, the capability ids it accepts, an optional plan validator dotted path, and a UI component path. The plan_validator field reserves space for the plan-then-execute pattern that pairs LLM inference with a deterministic spec-to-call translator on the tool-owning plugin side. Capability defines an orthogonal modifier with an enrich() function that runs before the workflow's agent loop and contributes named input fields to the workflow's signature.
Adds WORKFLOWS lists at the bottom of both built-in workflow modules so the discovery layer can pick them up once it lands. The orchestrator still routes by the legacy ExecuteStyle string for now, so behavior is unchanged for users. Author requires caldera_core, accepts the rag capability, and forbids range tooling because its signature already says do not deploy infrastructure. Plan and Execute requires caldera_core, accepts range optionally so the user can opt into infrastructure tools when needed, and accepts rag so CTI reports can ground both operation planning and infrastructure provisioning.
Adds a CAPABILITIES list at the bottom of rag.py exposing the existing RAGService through the Capability contract. The enrich function loads selected bundles from plugins/mcp/data/, builds a RAGService, retrieves the most relevant chunks for the user's prompt, and returns a cti_context string. Workflows that accept the rag capability and have a cti_context input field will receive this content automatically once the orchestrator switches to the capability registry. Inlines a copy of the rag context formatter inside the capability so it no longer depends on workflow code; the duplicate inline formatters in workflows/author.py and workflows/plan_execute.py will be removed when the orchestrator switch lands. This declaration is dormant; the legacy _build_rag_service_from_files path in mcp_svc still wires RAG into runs directly so behavior is unchanged for users.
Adds discover_workflows() and discover_capabilities() that mirror the
existing discover_mcp_servers() pattern. They scan two sources each:
1. plugins/<other_plugin>/mcp/workflows.py and capabilities.py
(the new external-plugin contract)
2. plugins/mcp/app/workflows/*.py and capabilities/*.py
(built-in registrations)
Built-ins are discovered after externals so they win on id collision.
Unlike server discovery which uses ast.literal_eval to avoid running
plugin code, workflow and capability discovery have to import the
modules because the registrations reference live Python classes
(signatures) and async functions. Import failures are logged and
skipped rather than blocking the rest of the registry.
hook.enable() now runs all three discovery passes and hands the
registries to MCPService. The service stores them but does not consume
workflow_registry or capability_registry yet; the orchestrator switch
is the next commit. /plugin/mcp/servers continues to work unchanged.
Local verification with the current plugin tree returns:
servers: ['caldera_core', 'range']
workflows: ['author', 'plan_execute']
capabilities: ['rag']
The mcp_svc orchestrator now dispatches by workflow id from the
workflow_registry built at boot, instead of branching on the
ExecuteStyle enum. The new flow is:
1. Look up the workflow by id.
2. Filter enabled_servers against the workflow's required and
optional set; required servers are pinned on regardless.
3. Filter enabled_capabilities against the workflow's accepted set.
4. Run each capability's enrich() in order; merge their outputs
into a context dict.
5. Invoke workflow.run(prompt, lm_obj, **context) so capability
fields like cti_context land in the signature's input fields
by name.
Both workflow run() functions now accept cti_context as a kwarg and
absorb unknown capability outputs via **_extra_capability_context, so
adding a new capability does not require touching workflow code.
The legacy /execute payload (type plus model.config.rag_files) still
works. mcp_svc.execute() accepts both shapes and translates the
legacy one into the new one via _LEGACY_TYPE_MAP. The translation
keeps RAG settings working, including the implicit RAG behavior the
old UI relied on where attaching rag_files enabled RAG even when type
was plain factory or planner.
ExecuteStyle is removed. The legacy shim and its callers can be
deleted once the UI ships the new payload exclusively.
The dspy.context() per-run pattern in each workflow's run() is left
in place; we deliberately do not call dspy.configure() from the
orchestrator (it raises across async tasks).
Adds two new GET routes that the UI uses to render itself dynamically
from discovery instead of hardcoding tabs:
GET /plugin/mcp/workflows
Returns each discovered workflow with its display name,
description, required and optional servers, accepted
capabilities, ui_component path, example prompts, and the
plan_validator dotted path (when set). Workflows whose required
servers are not all present in the discovered server registry
are filtered out, so for example a workflow that requires the
RANGE plugin disappears from the list when RANGE is not
installed.
GET /plugin/mcp/capabilities
Returns each discovered capability with its display name,
description, ui_settings_component path, and the input field
names it contributes.
Both endpoints read directly off MCPService.workflow_registry and
MCPService.capability_registry, which are populated once at boot by
the discovery layer.
mcp.vue is now data-driven: it fetches /plugin/mcp/workflows and
renders one card per discovered workflow instead of two hard-coded
"Factory" and "Planner" buttons. The card shows the workflow's
display name, description, and a one-line summary of which servers
and capabilities it can use. Adding a new workflow from any plugin
now results in a new card with no edits to mcp.vue.
The Global Model Config panel keeps only LM credentials and limits.
The "Enabled MCP Servers" checklist moves out of Global Model Config
and into each workflow's session page, scoped to that workflow's
required and optional servers. Required servers render disabled so
the user cannot turn them off. The "RAG TopK" and "RAG Embed Model"
fields move into the RAG capability's settings panel inside the
workflow page, since they only make sense when RAG is in play.
Workflow session pages now accept the workflow registration as a
prop. The two previously-distinct factory and planner pages collapse
into the same workflow shell that reads everything (title, examples,
allowed servers, accepted capabilities) from props.workflow. The
files keep their old names (local_mcp_ability_factory.vue and
public_mcp_ability_factory.vue) for now to keep the magma bundle
diff small; they can be renamed in a follow-on commit.
The /execute payload sent by the workflow shell switches to the new
shape: workflow_id, lm_config, enabled_servers, enabled_capabilities,
capability_settings. The legacy {type, config} shape is still
accepted by the backend, so a stale browser tab keeps working until
the user reloads.
The in-page Extension Guide is updated to teach the new contract:
mcp_server.py at plugin root, mcp/workflows.py and mcp/capabilities.py
inside an mcp/ folder, optional translator.py for plan-then-execute
workflows, and an optional Vue session page referenced by the
workflow's ui_component field.
localStorage shape gains three new keys (serversByWorkflow,
capabilitiesByWorkflow, capabilitySettings) so a user's per-workflow
choices and per-capability settings persist across reloads. RAG and
similar capability settings are stored globally rather than per
workflow, so picking a TopK once applies everywhere RAG is enabled.
…esign Captures the agreed shape for the MCP plugin redesign: three composable blocks (servers, workflows, capabilities) discovered at boot, the two-layer pattern for CTI to infrastructure (inference scaffolding plus spec-to-call translation), the new directory structure, plugin contracts, migration steps in order, and the open design questions. Documents the deliberate asymmetry where external plugins group MCP contributions under an mcp/ folder but the MCP plugin itself does not (to avoid the plugins.mcp.app.mcp.tools path collision).
The signatures' process_result OutputField descriptions previously read "Message that summarizes the result", which the model dutifully interpreted as a recap of which tools it called rather than the data those tools returned. Real symptom from a plan_execute run: process_result: "I first listed the available profiles to see what environments are configured. Then I retrieved the list of templates to know what deployment blueprints exist. After that, I fetched the VM images for each profile..." That tells the user nothing. The user wanted counts, names, and statuses, not a play-by-play of the agent's tool selection. The new descriptions explicitly forbid methodology narration and require the actual content (real names, ids, counts, statuses) be returned in a short list, with an exception for naming missing data when a tool failed. The WithRAG variants keep the secondary CTI note but subordinate it to the substantive results. Author and Plan and Execute share the same shape so they get the same fix in parallel.
Adds .env and .env.local to ignore so credentials never get tracked. Adds the plugin's planning and handoff markdown files to ignore so proposal documents stay local while implementation lands in code.
The plugin previously called load_dotenv() inside the caldera_core MCP subprocess, which meant the parent process never saw the file's contents and only worked because the subprocess re-loaded the same file. Move the load to hook.py so the parent reads .env at plugin enable; spawned subprocesses inherit the environment automatically. Adds python-dotenv to requirements.txt to back the load.
Adds app/config.py as the single place that merges yaml defaults, .env
credentials, and per-request UI overrides. Callers used to do this merge
inline in three different places with subtly different rules; the new
resolver makes the precedence explicit:
1. yaml defaults (model, api_base, ...)
2. env-resolved api_key (read from the env var named in api_key_env)
3. UI overrides (only non-empty fields, only fields not declared
fields_locked: true in yaml)
Renames the credential env vars so each one says what it is for:
CALDERA_API_KEY -> CORE_CALDERA_API_KEY (Caldera REST admin key)
yaml's ${API_KEY} -> MCP_LLM_API_KEY (LLM provider key, named
via api_key_env in yaml)
This is a breaking change for anyone with a stale .env. Update local
.env files to use the new names.
Restructures conf/default.yml to drop the dead factory: block and the
non-functional ${API_KEY} substitution, add a caldera: block declaring
which env vars to consult, and add a fields_locked map so deployment
constraints (gateway-fixed model lists) are explicit instead of silently
overriding UI input.
Updates .env.example to match. Updates the test that forwards env to a
range subprocess to use the new var name.
mcp_svc.execute() previously skipped DSPy client construction when the UI did not send an api_key, leaving the workflow to read yaml itself on the way down. The resolver now runs unconditionally for every request so the merge happens in one place: yaml defaults -> env credential -> UI overrides The rag capability still defaults its api_key to the resolved LLM key when the caller doesn't supply embed_api_key explicitly. mcp_api.defaults() switches to llm_defaults() from the shared config module, drops the dead workflows.author import, and now emits fields_locked so the UI can disable inputs that the deployment fixes (matching the resolver's behavior of dropping locked-field overrides).
Both workflow modules previously redefined get_llm_config and
_expand_env, then re-merged yaml with the caller's lm_obj inside
run(). When yaml had api_base set the merge silently overwrote the
UI's model field, and the override was invisible to the user.
mcp_svc now resolves the full LM config once via resolve_llm_config
(yaml + env + UI overrides, with fields_locked enforced). The
workflow's job is just to consume the resolved dict and build a
dspy.LM from it. Removes:
- _expand_env (the ${API_KEY} substitution it backed never worked)
- get_llm_config (replaced by config.llm_defaults for the test path)
- configure_llm (dead, never called)
- the build_lm_from_dict / yaml-overrides-UI block in plan_execute.py
- the lm_obj_safe / yaml_cfg merge block in author.py
The dspy.LM construction helper stays in plan_execute.py as
_build_lm_from_settings, callable from a single resolved dict.
The RAG capability previously hard-coupled embedding credentials to the chat LLM key. That breaks when the chat model lives behind a gateway with a constrained model list and the embedding model is hosted somewhere else (e.g. chat on MITRE gateway, embeddings on OpenAI). RAGService now accepts an api_base alongside api_key and forwards both to dspy.Embedder. The capability's enrich() honors two new optional settings: embed_api_key override for the embedding key embed_api_base override for the embedding provider's base URL When neither is supplied, the resolved chat-LLM key continues to flow through unchanged so existing single-provider deployments work without configuration.
McpGUI used to be vestigial: the class was instantiated in hook.py but no route was registered, splash() was unrouted, and the jinja2 template was the scaffold's Alpine.js click-the-button demo. Give the file a real job. splash() now assembles a readiness snapshot via the unified config resolver and the discovered registries: - Whether MCP_LLM_API_KEY is set (server-side default for LLM calls) - The model and api_base that requests will use - The Caldera REST URL the caldera_core subprocess will hit - Discovered MCP servers, workflows, and capabilities by name The template renders that snapshot as a status card and surfaces a warning when the LLM key is missing so the user is told up front that runs will fail until the .env or per-session UI key is supplied. Wires GET /plugin/mcp/gui to splash so the page actually renders when Caldera opens the plugin. The Vue SPA workspace stays the primary UI; this splash is the plugin's diagnostic / entry surface.
…ess halves
The single config.py file paired with the side-effect-import bootstrap
in dspy_env.py expressed two halves of one contract awkwardly. Promote
config to a package so the two halves are siblings under one
namespace:
app/config/__init__.py re-exports the parent-side public API so
existing imports keep working unchanged
(resolve_llm_config, llm_defaults,
caldera_connection)
app/config/parent.py the resolver code (was config.py). The
BaseWorld import is deferred into
_load_defaults so this module imports
cleanly even in subprocess contexts that
do not have Caldera's framework on
sys.path
app/config/subprocess.py the subprocess-side LM bootstrap (was the
top of dspy_env.py). Renamed from
configure_dspy_from_env to
ensure_lm_configured. Lazy and idempotent;
raises a clear RuntimeError when
DSPY_API_KEY is missing instead of
silently no-op'ing
dspy_env.py loses its module-level configure call. CreateCommand.__init__
calls ensure_lm_configured() so importing dspy_env is now inert; the
global LM state changes only when CreateCommand actually needs it. The
file's remaining job (signatures plus the CreateCommand module) is
unchanged.
The DSPY_* env-var names that bridge parent and subprocess are now
constants in config/subprocess.py, documented as the contract the
parent's get_env helper writes to.
The execute() and _run_execution() methods were broken up by ASCII section dividers (---- 1. ... ----, ---- 2. ... ----) that added visual noise without making the flow clearer. Remove the dividers and the comments that just described the next block; keep only the ones that explain a non-obvious decision (legacy payload mapping, deliberate dspy.configure absence, RAG default-key injection). No behavior change. Class init, signatures, and side effects are unchanged.
The config package introduced a __init__.py that re-exported the parent API, plus a deferred BaseWorld import inside _load_defaults to keep parent.py importable from subprocess context. Both were structural concessions to a layout that did not earn them: the package only ever held two siblings, and the subprocess never imported parent anyway. Flatten back to a single app/config.py. Move the BaseWorld import to the top of the file (it is the parent-side resolver; the subprocess does not import this module). Drop the __init__.py and the package directory entirely. Existing imports (from plugins.mcp.app.config import resolve_llm_config) keep working unchanged. The subprocess-side bootstrap moves into dspy_env.py in the next commit; this one only renames and rewrites the parent half. config_subprocess.py is staged for deletion in the next commit.
…with workflows
dspy_env.py is now the single owner of the DSPy library contract for
the caldera_core MCP subprocess:
- The DSPY_* env-var names that bridge parent and subprocess
- The lazy LM bootstrap (ensure_lm_configured) that runs on first
CreateCommand() instantiation
- The signatures and CreateCommand module mcp_server.py uses
The cross-module reach into config.subprocess is gone. dspy_env.py
imports only os and dspy at the top of the file. Importing it has
no side effects; DSPy is configured only when CreateCommand actually
needs it. Missing DSPY_API_KEY raises a clear RuntimeError naming
the parent-side env-forwarding contract.
The parent's get_env helper in workflows/author.py and
workflows/plan_execute.py now imports the ENV_* constants from
dspy_env, so the env-var contract has a single source of truth in
the file that authoritatively reads them.
Deletes app/config_subprocess.py (the in-between file). With this
commit the layout is two flat modules:
app/config.py parent-side stack and resolver
app/dspy_env.py subprocess-side DSPy library bootstrap
Both workflow views polled status into pollFinalResult but never rendered it. Reasoning and Thoughts displayed; the substantive answer the agent produced was dropped on the floor. Add a Result panel above Thoughts that shows pollFinalResult when the run is finished. Uses a violet accent matching the workflow theme to distinguish the answer from intermediate reasoning.
The workflows already return the agent's process_result, reasoning,
and trajectory back up the call chain to mcp_svc. mcp_svc was
discarding everything except a 250-char preview tag, and the API
was reaching back into MLflow to recover the same data the agent
had just returned in memory.
Add an OrderedDict run cache on MCPService keyed by run_id with an
LRU bound. _run_execution writes RUNNING on entry, FINISHED with
the agent's full payload on success, and FAILED with the error
message on exception. get_run(run_id) returns the snapshot for the
upcoming /status read switch.
Both workflows now return {process_result, reasoning, trajectory}
instead of just process_result so the cache has everything the UI
needs without scraping MLflow tags.
Every existing mlflow.set_tag and mlflow.log_param call stays in
place. MLflow keeps its observability role; this commit only adds
a parallel in-memory state path. The /status endpoint is switched
to read from the cache in the next commit.
The /status endpoint now calls mcp_svc.get_run(run_id) and returns
the cached snapshot directly. The agent's process_result, reasoning,
and trajectory travel agent -> workflow -> mcp_svc cache -> API
without round-tripping through MLflow's tag and param store.
Side effects:
- Field-name mismatches between writer and reader can no longer
cause the UI to silently render empty panels. There is one
source of truth shared by writer and reader.
- Polls hit an in-memory dict instead of an MlflowClient RPC.
- Returns 404 when the run_id is older than the LRU bound or the
server restarted mid-run; the History endpoints still surface
those runs from MLflow.
Drops the log_param('process_result', ...) band-aid that was added
in plan_execute and author so MLflow could ferry the answer back
to the API. The corresponding set_tag('process_result', ...) stays
so the answer is still visible in the MLflow UI for that run.
History endpoints (list_runs, get_run_detail) still read from
MLflow; that is the right boundary.
The Global panel kept model, endpoint and limits in localStorage while the
server kept its own in conf/local.yml, so the two panels named two different
models: Global showed the browser's copy and CTI Extraction showed the file.
Neither was wrong about its own source, which is what made it hard to see.
The connection is server state now. The panel reads it from /defaults and
those values overwrite rather than fill, saveConfig no longer persists them,
and a browser upgrading from v2 hands its copy to the server once before
defaults are read back so an existing endpoint is not replaced by whatever
default.yml ships.
get_config now returns a resolved view alongside the raw file. The CTI panel
layered the profiles itself, in JavaScript, over unresolved yaml, so it
printed 'not set' for an endpoint that came from MCP_LLM_API_BASE while
extraction dialled it correctly. That copy of the allowlist is gone.
An absent workload profile now inherits the connection instead of resolving
to {}, which raised "No LLM profile 'cti'" for a deployment that had simply
deleted the block.
The promotion existed to keep a conf/local.yml that carried the connection under a workload profile working. That shape was written by a skipped test rather than chosen, and supporting it indefinitely meant the loader had two ideas of where the connection lives. There is one shape now. A connection left under cti is ignored and the warning names the keys to move. The browser shim that pushed a v2 localStorage connection to the server before reading defaults back is gone for the same reason; the connection is read from the server on mount and nowhere else. The reload test moves to tests/test_client_reload.py, which is what it was always about.
Each ChainOfThought step returns a Prediction, and the next signature declares list[str]. Passing the object straight through made DSPy stringify it, reasoning included, into the following prompt: the ranker received a paragraph of prose where it declared a list, and every call logged a type mismatch. The command still came out, so this read as noise rather than a defect, but each step was prompted with the previous step's reasoning instead of its answer.
The txt and md branch of the cleaner kept the original extension while the html and pdf branches normalise to .txt, and Stage 1 globs clean/*.txt. So a .md report, which three UI surfaces accept, was cleaned and then skipped. On a deployment with any prior IR on disk the run still reported complete and the row still went green, because nothing empties outputs_ir/complete. Also in the extraction path: The name-only technique lookup could never match. name_index is keyed '<kind>:<lowername>' and the lookup used the bare name, so every technique identified by name fell through to the non-enriched fallback, which carries no external_references and is therefore invisible to adversary authoring. Both enriched attack-pattern paths omitted created and modified while the fallback set them, so bundles were internally inconsistent and stix2.parse fabricated the timestamps from the wall clock on every read. The MITRE object's own timestamps are used, which also stops those objects changing between runs. The dropped-technique counter was module state under a thread pool, so it reported every drop since process start rather than the file's own.
…shot The sync compared each field to what /defaults reported at mount and never updated that snapshot, so editing a value away and back again sent nothing on the way back: the server kept the intermediate value while the panel displayed the original. It also refused to send an empty string, so blanking api_base to fall back to MCP_LLM_API_BASE was impossible. It now diffs against what the server is known to hold, updated after each successful save and deliberately not updated after a failed one, so the next edit retries the whole diff. Leaving the panel also flushes a queued save. The CTI panel reads the connection from the server when it mounts, so navigating inside the debounce window showed it the pre-edit endpoint until the next visit.
Roughly forty lines of comment for twenty of config, with some of it stale. Dropped the claim that provider is read with no default, which resolve_env_indirection has supplied since it was written, and the em dash. Parsed config is unchanged.
An MCP stdio session fails inside nested anyio task groups, so the exception reaching a handler stringifies as 'unhandled errors in a TaskGroup (1 sub-exception)'. The workflow handler logged that with str(e) and the service handler logged the real summary afterwards, but MLflow params are immutable, so the first write won and the second was dropped. The history endpoint returns every param, so that useless string is what the History view showed as the reason a run failed. The summariser moves out of mcp_svc to where both callers can reach it, and the workflow handlers use it, so the message written first is already the provider error.
These POST to a running server, so they wrote conf/local.yml for real. An earlier version of this file is how a deployment ended up pinned to a gateway nobody had chosen, with the connection under cti where the resolver no longer reads it. An autouse fixture now restores the file and clears the server's cache, and the connection payloads move to the llm profile the writer accepts. TestRestoreConfig is deleted. It re-pinned that same gateway on every run, which is the behaviour the fixture exists to prevent.
Several suites POST to a running server, so their writes land in the real config. The file this session cleaned came back from a full run carrying offline: true, temperature: 2.0 and max_tokens: 8192, with every comment gone, because the server rewrites it with yaml.safe_dump. The guard is autouse in conftest and unconditional, so the next suite to grow a live POST does not have to remember. The per-file fixture it replaces is removed.
Both panels displayed these and both stored their own copy, so they drifted: CTI Extraction read 0 and 8192 while Global Model Config read 0.5 and 24000, with nothing to say which a run would use. They move to the llm profile, so there is a single value rather than two kept in step. Editing either panel edits the same key, and a cti override is dropped and logged like any other. default.yml sets them explicitly, which also settles the disagreement between app/config.py's numeric fallbacks and the llm_client wire defaults: both resolvers now read the same number. Returning from a sub-view re-reads /defaults, because this view is hidden rather than unmounted and the CTI panel can have changed either value while it was hidden. temperature ships 0.0, not the 0.5 the chat fallback used, because Stage 1 parses the model's own output as JSON and sampling makes that less reliable. top_p, timeout, stream, offline and embed_model stay per workload.
Every branch of the cleaner wrote <stem>.txt, so report.md and report.txt both became report.txt. The cleaner gathers files concurrently, so whichever finished second replaced the other, and an interleaved pair produced text belonging to neither, which is then what the extractor parsed. Everything downstream is keyed on that stem, so the two reports also collapsed onto one IR and one bundle. Normalising md to .txt introduced the txt/md pair; html and pdf could already collide with each other the same way. clean_stem folds the source extension in, so report.md becomes report_md.txt. Stage 1's selection filter and the clean-to-raw reverse lookup use the same helper, the latter having matched on a bare stem and so moved the wrong raw file to processed. Clean, IR and STIX artifacts are renamed by this. Existing ones are not migrated; they are re-derived on the next run.
finalize_run took the selection and never read it, so every upload was retired to raw/processed whether or not this run touched it. list_cti_raw then stamped a literal status on that directory, so a report that was never selected, never cleaned and never extracted rendered green. file_status, which computes the status from the IR and its mtime, was already written and never called. It is wired in for both directories and its IR path now goes through clean_stem. The unused dedup set is used.
The misplaced-key guard was scoped to workload profiles, so the llm section took any key at all and any new top-level section was created on demand. An Authorization header, an x-api-key and an extra_headers block all reached conf/local.yml in plaintext, beside tracked config, under a file header promising credentials never go there. mlflow.host reached it too, and hook.py passes that to 'mlflow server --host'. One save could rebind an unauthenticated server holding every logged prompt and response to a non-loopback interface. get_config grew a 'resolved' sibling, which broke the exact-key-set test in unwrap_config_envelope, so a GET-edit-POST round trip was no longer recognised as an envelope and slipped past the per-section rules entirely. Sections and keys are now allowlisted, the envelope tolerates its sibling, and the secret pattern covers header and token spellings. A credential-named key is still stripped rather than refused, so an older cached bundle that posts api_key can save the rest of its payload.
is_valid_actor called name.isalpha() on the whole string, which is False as soon as there is a space, so every two-word actor fell through: Berserk Bear, Wizard Spider, Fancy Bear and Cozy Bear included, two of which the function's own docstring says to keep. Digits are allowed now too, since FIN7 and TA505 are names, but a bare number is not. mcp_server had a bare 'from dspy_env import' above the sys.path insert meant to make it resolvable, so the module imported only when sys.path[0] happened to be app/. Four tests asserted things that are no longer true or never were: test_preserves_dots wanted normalize_phrase to keep dots in an IP, but the function is documented as canonical alphanumeric form and its only callers use it as a dedup key, where stripping punctuation is the point. The entity extraction that cared about IPs was removed earlier on this branch. test_slash_actor_split expected ALPHV/BlackCat to split into two actors. No slash handling exists anywhere in the module, and those are two names for one group, so splitting would invent a duplicate. test_server_class_imports and test_get_server imported MCPServer and _get_server. git log -S finds no commit that ever added either. They are replaced by guards on what the module does expose, including the import that was broken. test_cti_has_parameter_fields asserted the cti section declares temperature and max_tokens. Those live on llm now; it asserts the resolved profile.
Emitted a DeprecationWarning on every STIX bundle, observed-data object and uploaded file. Timezone-aware now, and the isoformat suffix stays Z rather than +00:00 so bundle timestamps keep the shape STIX expects.
convert_ir_to_stix reads attack_patterns, threat_actors and hashes. It has never had a branch for malware, tools or infrastructure, so every one of those was extracted, canonicalised, enriched with command lines, written to the IR and then dropped at conversion. The LLM prompt asked for all three, so this was also paid for in tokens on every report. Removed: the three fields from the prompt and the schema, the well-known tool scan, the spaCy NER tool pass, the regex infrastructure pass and the tool and malware halves of the MITRE entity index, plus the command extraction that existed only to attach commands to them. The infrastructure pass was the IP and domain regex that survived the earlier removal of entity extraction, which this branch dropped on the grounds that CTI says what to run and the environment says where. Verified by running the pipeline end to end offline before and after: the emitted bundle is identical once the wall-clock timestamps are normalised.
The blanket utcnow replacement in 91520a7 wrote datetime.now(datetime.UTC) into a module that does 'from datetime import datetime', where that attribute does not exist. hashes_to_stix_observed_data raised AttributeError on every report carrying a hash, and cti_pipeline_stage1 turns that into 'Stage 1 failed for all N files', so one hash took the whole batch down. Nothing crossed that seam: the function is reached only from convert_ir_to_stix when the IR carries hashes, and no test produced one. It has tests now. The object was also missing spec_version, first_observed and last_observed, so stix2.parse rejected it outright and no consumer could load a bundle that contained one. It round-trips through the reference parser now.
Both workflows built the subprocess environment with
str(lm_settings.get('temperature') or 0.5). 0.0 is falsy, and 0.0 is the
shipped value: conf/default.yml sets it because Stage 1 parses the model's own
output as JSON. So the one setting that exists to keep extraction
deterministic was the one silently replaced. max_tokens had the same shape,
and the ssl_verify and timeout lines beside them were already correct.
dspy_env._DEFAULTS held temperature 0.5 and max_tokens 10000 against
default.yml's 0.0 and 24000, a third resolver disagreeing with the two that
5fbc99d had just reconciled. A test now asserts they stay equal.
Also replaces the two utcfromtimestamp calls 91520a7 missed, in a file it
already touched.
Every symbol below was grepped across app/, gui/, tests/, conf/, docs/, mcp_server.py, hook.py and the markdown before removal. The only dynamic dispatch in the plugin is app/discovery reading module-level WORKFLOWS and CAPABILITIES attributes; MCP tools are explicit decorators and routes are explicit add_route calls, so nothing here was reachable by name. cti_linguistics.py goes from 401 lines to 66. It kept HASH_PATTERNS and extract_hashes, which Stage 1 uses, and lost behaviour normalisation, fuzzy canonicalisation, entity typing and the dynamic technique extraction, none of which had a caller. Its normalize_behavior_text was a second, diverging copy of the one cti_nlp_enhancements defines and actually uses. Also removed: the STIX open-vocabulary loader and its 75-line data file, the infrastructure type triggers, init_mlflow, build_dspy_lm, prepare_raw_uploads, a duplicate new_stix_id, _is_valid_cti_text, _caldera_root, two copies of an uncalled create_tool_function, five section helpers in the ability library loader, and the rag module-level shims, which tested a global that is defined nowhere and so could only ever return their placeholder string. Then the imports and locals those orphaned. pyflakes is clean across the plugin. Verified by running the pipeline end to end offline on a report carrying a file hash: both bundles parse with stix2.
test_set_config_with_different_models POSTed four model names in a loop to localhost:8888, so whichever ran last was left in the operator's conf/local.yml and in the server's config cache. That is how a deployment configured for Qwen ended up asking Featherless for meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 and taking a 404 on every Author run. The conftest guard restores the file but not the running server: set_config caches what it wrote, so the value kept being served from /defaults, the panel read it back, and the browser wrote it to disk again. Restoring a file cannot undo a write to a live process. Retargeting these from cti to llm in ecaf472 is what made it reach the connection at all; on cti the allowlist had been rejecting them. set_config has 37 offline tests that drive the real handler against tmp_path, so the four live classes plus two more in the other suites were duplicating that coverage while rewriting a running deployment. Removed. No live POST to set_config remains.
The README config example showed temperature and max_tokens under cti, which 5fbc99d moved to llm; layered_profile drops them and set_config returns 400. Its key list also disagreed with default.yml. Both now match WORKLOAD_OVERRIDABLE, asserted against the frozenset. The CTI page still advertised entity extraction, removed in d918d15.
It globbed data/outputs_stix, which is gitignored, so on a clean checkout the glob was empty and it printed and returned without asserting. Where it did find files it checked five objects and only spec_version on threat-actor. It now builds a bundle from our own builders and runs stix2.parse over it, plus spec_version, id, created and modified on every object. Verified by removing spec_version from a builder: the test fails.
Module scope killed whatever held the MLflow port, ran subprocess.Popen, then blocked up to 10s polling. So any import did that, including pytest collecting a module that imports hook. Moved to _ensure_mlflow_server(), called from enable(). Import now takes 3s and spawns nothing. kill_existing_mlflow is deleted. It ran only inside 'if not is_port_open', so nothing was ever listening to find, and it never checked the victim was MLflow before killing it.
A wrong model surfaced three different ways, none of which named the setting:
litellm said the model does not exist, Featherless said it is not available
for inference, and for the same model on another day said it was temporarily
at capacity. The CTI path dumped the provider's raw JSON.
LLMHTTPError carries the status and explains itself, and is_transient
separates a wrong setting from a blip. cti_parsing degrades to offline
extraction only when the failure is transient; a bad model or a rejected key
now fails loudly instead of silently producing a worse bundle.
summarize_exception gains the same rewrites for the DSPy path.
before: litellm.NotFoundError: Custom_openaiException - The model
`meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8` does not exist.
after: Model not available: the provider has no
meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8. Set a model it
serves in Global Model Config, or in llm.model in conf/local.yml.
Four tests POSTed to cti/run against a running server with step 'all'. An empty file list means everything, so they ran Stage 1 and Stage 2 over the operator's real corpus: a paid LLM call per report, and every bundle in outputs_stix re-minted with a new id. Each asserted only that the status was one of two or three values, so none could fail. Removed. Three uploads, pytest_upload.txt, pytest_report.html and pytest_listing_test.txt, were never deleted and accumulated in the real data/raw/uploads. An autouse fixture now sweeps them, alongside the existing conf/local.yml guard. Verified: conf/local.yml md5 and the 38 files under data/ are identical before and after a full run.
Popen's handle was discarded and Caldera has no plugin teardown hook, so the tracking server was reparented to init on shutdown and held the port across every restart. Observed on this machine: mlflow pid 8451, parent 1. atexit terminates it, then kills it if it ignores that. The handle is set only when this process actually spawned one, so a server the operator started themselves is left alone. Nothing can be done about SIGKILL.
HackedRico
force-pushed
the
fix/cti-pipeline
branch
from
August 31, 2026 14:17
73b4621 to
6345a44
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The plugin did not run on a clean clone. It imported a RANGE plugin, a
detections plugin and a D3FEND ontology that are not in this repository, and
the CTI pipeline referenced modules that had been removed with them.
This branch makes it work standalone, and removes what it was reaching for.
Config. The connection (provider, model, api_base, credentials, ssl_verify)
now lives on the
llmprofile alone. A workload profile such asctimay setonly
top_p,timeout,stream,offlineandembed_model; anything else isdropped and
set_configreturns 400.temperatureandmax_tokensmoved tollmtoo, because two panels display them and two stored copies drifted. TheGlobal Model Config panel is now server-authoritative and writes
conf/local.yml; the API key stays in the browser.Security. Every route now requires authorization. Both delete endpoints are
contained to their own directory. Credentials cannot reach
conf/local.yml, andmlflow.hostcannot be rebound off loopback through the API.Pipeline. Fixes a markdown upload being accepted and never parsed, two
reports cleaning to one file, a file hash aborting Stage 2, attack-patterns
missing
created/modified, and a name-only technique lookup that could nevermatch. Extraction is scoped to the selected reports. Malware, tools and
infrastructure are no longer extracted, since
convert_ir_to_stixnever readthem.
Removed. RANGE, D3FEND, the detections tool, the CALDERA fact bridge, the
mock response toggle, and roughly 780 lines that nothing referenced.
A fresh clone runs offline with no API key:
conf/default.ymlshipsoffline: true, and extraction falls to spaCy plus the MITRE taxonomy.Type of change
14 commits are marked
!. The behavioural ones: the connection allowlist, theclean-file naming (existing clean, IR and STIX artifacts are re-derived on the
next run, not migrated), and the removal of the features listed above.
How Has This Been Tested?
pyflakesclean across the plugin.emitted bundles parse with
stix2.pre-existing
conf/local.yml.conf/local.ymlanddata/byte-identical,after finding that live tests were writing both.
switch models.
Checklist:
Known and deferred
cti.vuefetches do not checkres.ok, and auth returns 200 with anHTML login page, so an expired session renders as empty tables.