fix(cti): make the plugin and CTI pipeline work on a clean clone

README.md

@@ -17,7 +17,7 @@ The plugin is designed to keep the LLM grounded in MCP tools and server-side con
 ## Features
 
 - **Author workflow**: Create CALDERA abilities and adversaries from an operator prompt while using available MCP server tools.
-- **Plan and Execute workflow**: Select or upload CTI/STIX, build an adversary from the observed techniques, run the CALDERA operation against available agents, and summarize detection coverage.
+- **Plan and Execute workflow**: Select or upload CTI/STIX, build an adversary from the observed techniques, run the CALDERA operation against available agents, and report which techniques it covered.
 - **CTI ingest pipeline**: Upload raw CTI in HTML, PDF, plaintext, or Markdown and produce STIX 2.1 bundles for retrieval and planning.
 - **STIX selection for planning**: Pick generated STIX bundles from the Plan and Execute workspace. Selecting STIX automatically enables CTI retrieval for that run.
 - **Model and CTI/RAG profiles**: Configure the chat model, API base, API key, temperature, token budget, and tool-call budget. CTI/RAG can use the chat model by default or a saved profile when it needs a different endpoint.
@@ -68,9 +68,45 @@ The CTI ingest workflow stages raw CTI files, runs extraction, and displays gene
 
 <br>
 
-### CTI Pipeline Fidelity
+### What the pipeline produces
 
-The CTI pipeline extracts structure from STIX and related observables, including hosts, identities, user accounts, domains, software, ATT&CK platform hints, and CVE references. These become CALDERA facts so an operation runs against values the report actually named, not placeholders.
+A threat report tells you **what** to run. Your environment tells you **where**.
+
+The pipeline extracts ATT&CK techniques, the named threat actor, and file
+hash observables, and nothing else. It does not extract hosts, accounts, domains, services or
+software, and does not turn them into CALDERA facts. Those describe the
+report's victim, not your estate; CALDERA discovers facts about your hosts
+at runtime, which is why the abilities that use them carry `has_agent_copy`
+and `no_backwards_movement` requirements.
+
+The one insertion point into core CALDERA is the adversary profile:
+
+```
+upload report(s) -> clean -> IR -> ATT&CK techniques -> STIX bundle
+                                                            |
+  "build an adversary from this and run it"                 v
+      build_adversary   attack-pattern -> ability.technique_id -> Adversary
+                        + unmatched_techniques + platform_excluded
+      run_operation     Adversary + your agents + runtime-discovered facts
+```
+
+### Known limits
+
+- **Extraction quality.** Scored against the committed measuring stick with
+  `tests/test_pipeline_score.py`: precision 0.83, F1 0.89. That fixture names
+  its technique ids explicitly, so its recall is an upper bound, not a field
+  estimate. Recall on prose-only reports is materially lower, and fusing
+  several reports on the same actor is the cheapest way to raise it.
+- **No detection scoring.** The `detections` plugin does not ship in this
+  repository, so nothing here scores an operation against SIEM rules.
+- **Platform matching is coarse.** `build_adversary` knows which platforms
+  your agents run, not whether a Windows agent is domain-joined. It reports
+  the gap in `platform_excluded` rather than closing it.
+- **Fusion does not check identity.** Fusing bundles for two different
+  actors merges them into one profile without complaint.
+- **Operations are not agent-scoped.** `run_operation` runs against the whole
+  `red` group; the `agent_paws` argument is not honoured by core's operation
+  schema.
 
 ## Installation
 
@@ -79,7 +115,7 @@ Install MCP like a standard CALDERA plugin.
 1. Clone or copy this repository into the CALDERA plugin directory:
 
 ```bash
-git clone https://gitlab.mitre.org/caldera/caldera-mcp.git plugins/mcp
+git clone https://github.com/mitre/mcp.git plugins/mcp
 ```
 
 2. Add `mcp` to your CALDERA plugin list in `conf/local.yml`. Keep local configuration in `conf/local.yml`; do not commit that file.
@@ -92,25 +128,39 @@ plugins:
   - mcp
 ```
 
-3. Optional integrations can be enabled by adding their plugins as well:
+3. Install plugin requirements from the CALDERA virtual environment:
 
-```yaml
-plugins:
-  - magma
-  - sandcat
-  - stockpile
-  - detections
-  - mcp
+```bash
+source venv/bin/activate
+pip install -r plugins/mcp/requirements.txt
 ```
 
-4. Install plugin requirements from the CALDERA virtual environment:
+4. Install the spaCy model. pip cannot do this from `requirements.txt`:
 
 ```bash
-source venv/bin/activate
-pip install -r plugins/mcp/requirements.txt
+python -m spacy download en_core_web_lg
+```
+
+   Install poppler as well, which supplies `pdftotext`. Without it the
+   pipeline still ingests TXT, MD and HTML, and PDF uploads report the
+   missing dependency rather than failing silently:
+
+```bash
+brew install poppler          # macOS
+apt install poppler-utils     # Debian/Ubuntu
 ```
 
-5. Copy `.env.example` to `.env` and set the LLM endpoint and credential. Both are required; nothing ships in `conf/default.yml`.
+   To run the test suite, also install the dev requirements:
+
+```bash
+pip install -r plugins/mcp/requirements-dev.txt
+```
+
+5. Copy `.env.example` to `.env` and set the LLM endpoint and credential.
+   Both are optional: with neither set, IR extraction falls back to a
+   deterministic offline extractor. That extractor trades roughly 15 to 20
+   percent of recall for sub-second processing, so configure a model when
+   extraction quality matters.
 
 ```bash
 cp plugins/mcp/.env.example plugins/mcp/.env
@@ -164,12 +214,14 @@ Use Upload CTI to stage raw reports, run CTI extraction, and produce STIX 2.1 bu
 Use Plan and Execute for CTI-driven operations. The workflow can:
 
 - Read selected STIX bundles and CTI/RAG context.
-- Extract hosts, operating systems, domains, users, and software.
-- Turn those into a CALDERA fact source so operations run on values the report named.
-- Build and run the CALDERA operation.
-- Summarize operator-review gaps and detection coverage.
+- Fuse several bundles describing the same actor into one.
+- Build an adversary from the observed techniques, previewing it first.
+- Run the CALDERA operation against agents that have checked in.
+- Report which techniques had no matching ability, and which had one that
+  no live agent can run.
 
-The workflow should report missing evidence rather than inventing hosts, users, domains, or services.
+The CTI names techniques, not your estate. The workflow reports a gap
+rather than answering a question about your environment from a report.
 
 ## Configuration
 
@@ -184,28 +236,32 @@ MCP supports saved endpoint profiles in the UI. Profiles can carry:
 - Maximum tool calls
 - Maximum tokens
 
-CTI/RAG model settings can use the chat endpoint by default or a separate saved profile when CTI extraction benefits from a different model.
+CTI and RAG share the chat endpoint. Extraction differs from chat only in its generation settings, which the CTI Extraction Model panel edits.
 
 ### Local Settings
 
 Prefer environment variables for local values, and keep `conf/default.yml` limited to safe defaults.
 
-To pin values on disk instead, set them in `conf/local.yml`. That file is overlaid onto `conf/default.yml` key by key, so it only needs the keys it changes.
-
-The `llm` block sits at the top level of the file:
+To pin values on disk instead, create `conf/local.yml`. That file is a **sparse overlay**, not a full config: it is deep-merged onto `conf/default.yml` key by key, so it needs only the keys it changes. A file containing one key is valid and complete.
 
 ```yaml
 llm:
   model: openai/gpt-oss-120b
   api_base: https://api.example.com/v1
-  api_base_env: MCP_LLM_API_BASE
-  api_key_env: MCP_LLM_API_KEY
-  temperature: 0.5
-  max_tool_calls: 5
-  max_tokens: 24000
+  offline: false
+
+cti:
+  timeout: 120
+  offline: false
 ```
 
-A value set here outranks the environment variable named beside it, so leave `api_base` empty to keep resolving it from `MCP_LLM_API_BASE`. The `cti` block takes the same shape and may name a different variable if CTI extraction should reach a separate endpoint.
+Three rules that are easy to trip over:
+
+- **`cti` layers over `llm`, but only for its own generation settings.** A workload profile may set `top_p`, `timeout`, `stream`, `offline` and `embed_model`. Everything else belongs to `llm`: the connection (`provider`, `model`, `api_base`, credentials, `ssl_verify`), and `temperature` and `max_tokens`, which two panels display and so are stored once. Anything else under `cti` is dropped and logged, and `set_config` rejects it outright.
+- **Precedence differs by field.** `api_key` resolves environment-first and is never read from this file. `api_base` resolves yaml-first, so a value here beats `MCP_LLM_API_BASE`; leave it empty to keep using the variable.
+- **A key with no value is `null`, and null overrides.** A bare `llm:` heading wipes the shipped block rather than falling through to it. Omit the block instead.
+
+The GUI's **Global Model Config** panel writes the connection to the `llm` profile in `conf/local.yml`, so it configures every workload including CTI extraction. Only fields you change are written, so a value still coming from `MCP_LLM_API_BASE` is left alone rather than pinned to disk. The API key is the exception: it stays in `localStorage` and rides each request, and `set_config` strips credentials before writing.
 
 ## API Surface
 
@@ -220,6 +276,10 @@ The plugin exposes MCP workflow APIs through CALDERA's aiohttp server. The UI us
 - Run CTI pipeline steps.
 - Read run history and transcripts.
 
+The CTI pipeline MCP server exposes five tools: `cti_pipeline_ingest_cti`,
+`cti_pipeline_fuse`, `cti_pipeline_build_adversary`,
+`cti_pipeline_run_operation` and `cti_pipeline_wait_for_agents`.
+
 When adding new UI functionality, prefer extending the existing MCP API routes instead of creating separate side channels.
 
 ## Architecture

app/capabilities/rag.py

@@ -34,7 +34,7 @@ def __init__(
         if stix_bundle_path:
             self.load_stix_bundle(stix_bundle_path)
 
-    def load_stix_bundle(self, stix_bundle_path: str, embed_model: str = 'openai/text-embedding-3-small'):
+    def load_stix_bundle(self, stix_bundle_path: str, embed_model: str | None = None):
         """Load STIX bundle from file path and build embeddings."""
         try:
             with open(stix_bundle_path, 'r') as f:
@@ -45,7 +45,7 @@ def load_stix_bundle(self, stix_bundle_path: str, embed_model: str = 'openai/tex
         except json.JSONDecodeError:
             raise ValueError(f"Invalid JSON in STIX bundle: {stix_bundle_path}")
 
-    def initialize_from_bundles(self, stix_bundles: List[dict], embed_model: str = 'openai/text-embedding-3-small'):
+    def initialize_from_bundles(self, stix_bundles: List[dict], embed_model: str | None = None):
         """Initialize the RAG service with multiple STIX bundles and create retriever."""
         all_corpus = []
         all_adv_step = {}
@@ -56,7 +56,19 @@ def initialize_from_bundles(self, stix_bundles: List[dict], embed_model: str = '
 
         self.corpus = all_corpus
         self.adv_step = all_adv_step
-
+
+        if not self.corpus:
+            # Embeddings() raises numpy AxisError on an empty corpus, and the
+            # caller swallows it and silently runs without CTI grounding.
+            # Bundles with no attack-pattern and no threat-actor are the
+            # normal way to get here. search_cti_title handles search=None.
+            self.search = None
+            self.log.warning(
+                "[RAG] No attack-pattern or threat-actor objects in the "
+                "selected STIX; CTI retrieval is unavailable for this run"
+            )
+            return
+
         self.log.info("Initializing embeddings and retriever for STIX corpus")
         embedder_kwargs = {"api_key": self.api_key}
         if self.api_base:
@@ -77,10 +89,7 @@ def extract_text_chunks(self, stix_bundle: dict) -> tuple[List[str], Dict[str, s
         adv_step = {}
 
         for obj in stix_bundle.get("objects", []):
-            if obj.get("type") in [
-                "attack-pattern", "malware", "tool", "threat-actor", 
-                "intrusion-set", "identity", "indicator", "report"
-            ]:
+            if obj.get("type") in ["attack-pattern", "threat-actor"]:
                 name = obj.get("name", "")
                 description = obj.get("description", "")
 
@@ -160,18 +169,7 @@ def get_context_for_task(self, task: str) -> Dict[str, any]:
         }
 
 
-# Legacy functions for backward compatibility
-def search_cti_title(query: str) -> list[str]:
-    """Legacy function - use RAGService instead."""
-    if 'global_rag_service' in globals():
-        return global_rag_service.search_cti_title(query)
-    return ["RAG service not initialized"]
 
-def search_cti_data_by_title(name: str) -> str:
-    """Legacy function - use RAGService instead."""
-    if 'global_rag_service' in globals():
-        return global_rag_service.search_cti_data_by_title(name)
-    return "RAG service not initialized"
 
 
 # ---------------------------------------------------------------------------
@@ -184,7 +182,6 @@ def search_cti_data_by_title(name: str) -> str:
 # capability registry, this declaration is dormant; the legacy code path in
 # mcp_svc still wires RAG into runs directly.
 
-import os
 import asyncio
 from pathlib import Path
 
@@ -253,7 +250,7 @@ async def _enrich(prompt: str, settings: dict) -> dict:
     settings keys (all optional):
       rag_files       list of filenames under plugins/mcp/data/
       topk            int, defaults to 5
-      embed_model     str, defaults to openai/text-embedding-3-small
+      embed_model     str, falls back to the chat model when unset
       api_key         str, embedding key fallback (mcp_svc fills this with
                       the resolved chat-LLM key when the caller does not
                       override it via embed_api_key)
@@ -271,7 +268,15 @@ async def _enrich(prompt: str, settings: dict) -> dict:
     embed_api_key = settings.get("embed_api_key") or settings.get("api_key") or ""
     embed_api_base = settings.get("embed_api_base") or settings.get("api_base") or None
     ssl_verify = settings.get("ssl_verify")
-    embed_model = settings.get("embed_model") or "openai/text-embedding-3-small"
+    # An undeclared embedding model falls back to the chat model. A deployment
+    # pointing at its own gateway has no reason to reach for an OpenAI model
+    # name it never chose, and that name would be sent to that gateway.
+    embed_model = settings.get("embed_model") or settings.get("model")
+    if not embed_model:
+        raise ValueError(
+            "No embedding model: set embed_model, or a model on the global "
+            "LLM profile for it to inherit."
+        )
     topk = int(settings.get("topk") or 5)
 
     base_dir = Path(__file__).resolve().parent.parent.parent / "data"

app/cti_ingest_svc.py

@@ -3,28 +3,21 @@
 Logging is preserved verbatim for debug capture via:
 python3 cti_ingest_svc.py --base-dir data --step all 2>&1 | tee -a debug_mitre.log
 
-| Step          | Description                       |
-| ------------- | -------------------------------------- |
-| Raw → Clean   | Normalize source material              |
-| Clean → IR    | LLM extraction of entities & behaviors |
-| NLP Layers    | Improve IR quality                     |
-| MITRE         | Map behaviors to ATT&CK                |
-| Relationships | Build entity graph                     |
-| Validation    | Ensure graph correctness               |
-| Outputs       | Persist IR + summary                   |
-| Scenario      | (Optional) Narrative generation        |
+| Step         | Description                                    |
+| ------------ | ---------------------------------------------- |
+| raw-to-clean | Normalize source material                      |
+| debug-ir     | Stop after IR extraction                       |
+| stage2       | IR -> STIX bundle only                         |
+| all          | Everything, in order                           |
 
 '''
 
 import argparse
-import asyncio
-import json, re
 import shutil
 from enum import Enum
 from pathlib import Path
 
 from plugins.mcp.app.cti_pipeline_stage1 import (
-    ensure_dirs,
     step_raw_to_clean,
     step_parse_to_ir
 )
@@ -70,20 +63,14 @@ def run_stage(self, base_dir: Path, step: str):
                     step_raw_to_clean(base_dir)
 
                 case "debug-ir":
-                    step_parse_to_ir(base_dir, stop_after="ir")
-
-                case "debug-mitre":
-                    step_parse_to_ir(base_dir, stop_after="mitre")
-
-                case "debug-relationships":
-                    step_parse_to_ir(base_dir, stop_after="relationships")
+                    step_parse_to_ir(base_dir, stop_after="ir", only=self.selected)
 
                 case "stage2":
                     self.run_stage2(base_dir)
 
                 case "all":
                     step_raw_to_clean(base_dir)
-                    step_parse_to_ir(base_dir)
+                    step_parse_to_ir(base_dir, only=self.selected)
 
                     # Only run Stage 2 if IR exists
                     ir_dir = base_dir / "outputs_ir" / "complete"
@@ -121,12 +108,25 @@ def status(self):
         }
 
     def finalize_run(self, base_dir: Path, selected: list[str]):
+        """Retire the reports this run actually processed.
+
+        This moved everything in uploads/ and never read its own argument, so
+        a report the operator did not select was retired having never been
+        cleaned or extracted. list_cti_raw stamps a literal "processed" on
+        that directory, so it then rendered green.
+
+        An empty selection still means everything, matching the convention
+        step_parse_to_ir uses for the same reason.
+        """
         uploads = base_dir / "raw" / "uploads"
         processed = base_dir / "raw" / "processed"
 
         processed.mkdir(parents=True, exist_ok=True)
 
+        names = set(selected or [])
         for f in uploads.iterdir():
+            if names and f.name not in names:
+                continue
             shutil.move(str(f), processed / f.name)
 
     @staticmethod