FabDefectTriage 🔬

A multimodal agentic crew for industrial defect/anomaly triage: a vision model classifies a defect image (wafer map / surface scan), a retrieval step matches it against historical defect/root-cause records, an LLM hypothesizes the root cause, and a reporter agent drafts a structured disposition — with confidence-gated hand-off to a human when the vision and LLM stages disagree or are uncertain. Mirrors classic semiconductor SEM/wafer defect and root-cause-analysis work, reframed as a vision + LLM agent crew with full pipeline lineage tracked in MLflow.

⚠️ LLM-generated root-cause hypotheses and disposition recommendations can make mistakes. FabDefectTriage's reasoning and reporter agents are LLM calls — always review the evidence (vision confidence, retrieved precedent cases) and confirm before acting on a disposition, especially for any case not already flagged for human review.

Architecture

 image (wafer map / surface scan)
   │
   ▼
 graph.py           LangGraph StateGraph
   │
   ├─ VisionAgent      — ONNX-served ResNet-18 classifier (vision/infer.py)
   │                     -> defect class + confidence + full class distribution
   │
   ├─ RetrievalAgent    — semantic search over a Chroma vector store of
   │                      historical defect/RCA cases (memory/case_store.py)
   │
   ├─ ReasoningAgent    — LLM hypothesizes root cause from the vision result
   │                      + retrieved precedent, self-reports confidence and
   │                      whether it agrees with the vision class
   │
   ├─ [conditional edge] route_after_reasoning:
   │     low vision confidence, low reasoning confidence, or a vision/LLM
   │     disagreement -> escalate; otherwise -> auto
   │
   └─ ReporterAgent     — LLM drafts a structured disposition (accept /
                          rework / scrap / escalate) + rationale + next
                          actions. The escalate path additionally writes the
                          report into data/artifacts/human_review_queue/ —
                          the hand-off point a human triage engineer picks
                          up from.

 Every run's full lineage (vision confidence, retrieved case ids, reasoning
 confidence, final disposition) is logged to MLflow (mlops/mlflow_tracking.py).

• Vision model (vision/anomaly_model.py, vision/infer.py): a timm ResNet-18 classifier fine-tuned on wafer-map images, exported to ONNX and served in-process via onnxruntime.InferenceSession — this is the "ONNX for serving" step, distinct from and unrelated to the ONNX Runtime GenAI LLM server below. Classifies into the WM-811K defect-pattern taxonomy: none, center, donut, edge-loc, edge-ring, loc, random, scratch, near-full.
• Case store (memory/case_store.py): an embedded, on-disk Chroma collection of historical defect records with resolved root causes and dispositions, seeded from memory/seed_cases.json — 14 realistic semiconductor-manufacturing RCA records spanning the defect taxonomy.
• Agents (agents/): each a thin run(...) function — VisionAgent has no LLM call at all (pure model inference); RetrievalAgent is a pure case-store lookup; ReasoningAgent and ReporterAgent both use llm/structured.py's schema-validated generation so every LLM call returns a typed Pydantic object, not free-form text to parse.
• Serving: api.py (FastAPI: POST /triage, POST /triage/synthetic, GET /health, GET /classes, plus the case-store curation endpoints below) + app.py (Streamlit UI, calls the API over FABDEFECTTRIAGE_API_BASE_URL). Each triage report in the UI can be downloaded as PDF, Markdown, or JSON, and confirmed into the case store for future retrieval — see "Case store curation" below.
• MLOps: mlops/mlflow_tracking.py logs both vision-model training runs and every triage pipeline run's full lineage to MLflow (local file-store by default, under data/mlruns/).
• Cloud: fabdefecttriage/sagemaker/{train,deploy}.py launch vision-model training jobs and real-time inference endpoints on AWS SageMaker; infra/aws/ provisions the S3 artifact bucket, SageMaker execution role, and (optionally) a Step Functions state machine reference for running the pipeline as managed cloud steps instead of an in-process LangGraph run — see infra/aws/README.md.

Case store curation

The case store (memory/case_store.py) starts seeded from the bundled memory/seed_cases.json (14 records), but the UI lets you grow and manage it as real triage decisions come in:

• Confirm and add to case store — each triage report has a button that adds it as a new historical case (POST /cases/confirm), so future retrieval can surface it as precedent. Idempotent per report, so re-confirming the same report is a no-op.
• Automatic backups — the store snapshots itself to a timestamped seed_cases_backup_*.json file once it has grown by case_store_backup_every_n_cases (default 10) cases since the last backup, keeping the most recent case_store_backup_keep (default 5) and pruning older ones.
• Make a backup now — a sidebar button that snapshots the store on-demand, independent of the automatic count threshold. Manual backups are never pruned and don't count toward case_store_backup_keep — they stay until you remove them.
• Reset the case store — hard-resets to either the bundled seed_cases.json or any backup (automatic or manual), picked from a dropdown in the sidebar (gated behind a confirmation checkbox since it's destructive). The sidebar's "Backend" box always shows the current case count and which source the store was last reset to.
• Remove a backup — delete any backup you no longer need from the sidebar. You can't remove the backup currently in use as the reset source, or the bundled default.

LLM backends

Backend	Where	Notes
vllm (default)	Local	Point at your own vLLM server — requires a GPU
onnx	Local	Point at your own ONNX Runtime GenAI server — no GPU required
ollama	Local	Point at your own Ollama server
llamacpp	Local	Point at your own llama.cpp (llama-server) instance
claude	Cloud	Anthropic API, requires FABDEFECTTRIAGE_ANTHROPIC_API_KEY

All five expose the same LLMClient.generate(system_prompt, user_prompt) interface (llm/client.py). The reasoning and reporter agents need schema-validated JSON out of the model, so llm/structured.py prefers grammar-constrained decoding (response_format={"type": "json_schema", ...}) on backends that support it (vLLM, Ollama, llama.cpp) and otherwise falls back to embedding the schema as prompt text plus a validate-and-retry loop (ONNX, Claude).

Each backend's base URL and model name are configured via .env (see .env.example) — point them at whichever local LLM server you already run, or leave the defaults if it's on localhost at the standard port for that tool.

Datasets

• Synthetic wafer-map generator (data/datasets.py:generate_synthetic_wafer_dataset) — a deterministic, seeded generator that draws circular wafer maps with each of the 9 WM-811K defect-pattern shapes (center clusters, donut/edge rings, scratches, etc.) using OpenCV-style geometric masks. This is what the demo model is trained on and what the UI's "generate a synthetic sample" tab uses — no external download required.
• Public datasets (loader stubs in data/datasets.py, each raising NotImplementedError with exact download instructions): WM-811K wafer-map dataset (Kaggle), MVTec AD anomaly dataset (mvtec.com), Severstal steel-defect dataset (Kaggle competition), NEU surface-defect database (Northeastern University). Each requires either a Kaggle account or an institution-hosted archive this repo can't fetch unattended — wire real images into the same (N, H, W, 3) uint8 / (N,) int label array shape the synthetic generator produces and everything downstream (training, ONNX export, inference) is unchanged.

---

Quick Start — Local

1. Install FabDefectTriage

cd FabDefectTriage
python3.12 -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # defaults work for local vLLM mode (requires a GPU)

2. Train the demo vision model

python scripts/train_demo_model.py
# Trains a ResNet-18 on the synthetic wafer-map dataset (a few minutes on
# GPU, longer on CPU) and exports data/artifacts/anomaly_model/anomaly_model.onnx.
# Logs the run to MLflow. Swap in a real dataset (see "Datasets" above) by
# passing --dataset-npz path/to/real_dataset.npz once one is prepared.

3. Start the LLM backend (if it isn't already running)

start_fabdefecttriage.sh expects the LLM server to already be up — it only checks reachability and warns if it isn't, it doesn't start/stop your LLM server itself. Start the default backend, vLLM (requires a GPU), or point .env at whichever local server you already have running:

# e.g. serving a model like Qwen2.5-7B-Instruct on localhost:8000/v1 via vLLM.
# No GPU? Use a local ONNX Runtime GenAI server instead (CPU int4 model,
# slower per request) and pass --backend onnx below.

4. Start everything else

./scripts/start_fabdefecttriage.sh

Open http://localhost:8501. Use the "Generate a synthetic sample" tab to try the full pipeline with no real dataset on hand, or upload your own wafer-map/surface image under "Upload an image". Each report can be confirmed into the case store or downloaded (PDF/Markdown/JSON) — see "Case store curation" above. Stop the API/UI with ./scripts/stop_fabdefecttriage.sh (this never touches your LLM server — stop it separately if you're done with it).

Other local backends

start_fabdefecttriage.sh --backend <name> only points the API at a different already-running LLM server — start that server yourself first, then:

./scripts/start_fabdefecttriage.sh --backend ollama
./scripts/start_fabdefecttriage.sh --backend llamacpp
./scripts/start_fabdefecttriage.sh --backend onnx

---

Tutorial

A hands-on, step-by-step walkthrough lives under tutorial/ as a set of Jupyter notebooks — each one imports the real fabdefecttriage package and runs it for real (training the vision model, querying the case store, calling a live LLM backend, running the full agent pipeline), while writing its own output into tutorial/artifacts/ so running it never touches this repo's own data/. See tutorial/README.md to get started.

---

Cloud Deployment — ECS/Fargate + S3 + SageMaker

cd infra/aws
terraform init
terraform apply \
  -var="container_image=<account>.dkr.ecr.<region>.amazonaws.com/fabdefecttriage:latest" \
  -var="s3_bucket_name=<globally-unique-bucket-name>"

This provisions an ECR repository, an ECS/Fargate service running the API+UI container behind an ALB, an S3 bucket for datasets/models/report artifacts, and a SageMaker execution role for fabdefecttriage/sagemaker/train.py / deploy.py. Set FABDEFECTTRIAGE_RUN_MODE=cloud and FABDEFECTTRIAGE_LLM_BACKEND=claude on the deployed container (the Terraform already wires these). See infra/aws/README.md for the optional Step Functions pipeline definition.

Build and push the image first:

docker build -t <account>.dkr.ecr.<region>.amazonaws.com/fabdefecttriage:latest .
docker push <account>.dkr.ecr.<region>.amazonaws.com/fabdefecttriage:latest

infra/aws/ is written to be deploy-ready and passes terraform validate, but requires your own AWS credentials to apply.

---

Testing

source venv/bin/activate
pytest tests/ -v

The suite runs offline against a fake LLM client, a fake vision model (or a freshly-exported untrained ONNX model for the inference-pipeline tests), and a temp-directory Chroma case store — the one exception being the case store's one-time sentence-transformers embedding-model download on first run.

License

Apache License 2.0 — see LICENSE.