DataPilot V1.1

DataPilot is a local, runnable data analysis agent web application. Users upload a CSV/XLSX dataset, ask natural-language questions, and a LangGraph agent autonomously calls data tools to produce tables, ECharts visualizations, and a final report.

V1.1 (this branch) hardens the agent for reliability and adds an eval suite:

• Reviewer never fails open: reviewer/structured-output failures produce an explicit REVIEW_ERROR outcome, never PASS.
• Anti-infinite-loop limits: a hard cap on tool executions per run plus a LangGraph recursion limit.
• Full tool traces: every tool record carries call_id, sequence, duration_ms, and error (visible in the UI timeline and SSE events).
• Checkpoint / resume: the graph runs on a LangGraph InMemorySaver checkpointer keyed by thread_id; interrupted runs resume from their last checkpoint via REST endpoints.
• Minimal eval suite covering tool selection, SQL/result correctness, review/revision, and unsupported-conclusion scenarios.

Architecture

Frontend (Next.js + TypeScript + ECharts)
    │  REST /api  (+ SSE /api/analyze/stream)
    ▼
Backend (FastAPI)
    │
    ▼
LangGraph Agent  (compiled with InMemorySaver checkpointer, thread_id-scoped)
 profile → planner → analyst ↔ tools → reviewer → final
    │
    ├── Tools (Pydantic validated)
    │     inspect_dataset, query_dataset, calculate_statistics,
    │     get_dimension_values, create_chart
    │
    └── Data layer (Pandas + DuckDB read-only)

LangGraph workflow

profile → planner → analyst
                      │  ▲
          tool calls? │  │ tool results
                      ▼  │
                     tools
                      │
                      ▼
                   reviewer ──REVISE (< max_revisions)──→ analyst
                      │
                      │ PASS / REVIEW_ERROR
                      ▼
                    final

The analyst is an LLM agent that selects tools dynamically. No fixed tool-calling sequence is hardcoded. The reviewer returns a structured PASS/REVISE verdict and reason; a conditional edge sends REVISE back to the analyst while revision_count < max_revisions.

Review outcomes (V1.1). The reviewer can only terminate with one of:

Outcome	Meaning
PASS	Analysis verified by the reviewer.
REVISE	Feedback sent back to the analyst (bounded by max_revisions).
REVIEW_ERROR	Reviewer LLM unavailable, structured output failed, or verdict malformed. The workflow still finishes, but the report is not marked as reviewed — the agent never fails open into PASS.

PASS is never a fallback anywhere: AnalyzeResponse.review has no default, and every serialization boundary (API response, SSE review event, final report) resolves a missing or invalid outcome to REVIEW_ERROR with an explanatory reason. Only an explicit reviewer PASS serializes as PASS.

Implementation note: the profile stage is registered in LangGraph as the node profile_dataset because LangGraph (>= 0.2) forbids node names that collide with state keys (the state key is profile).

Reliability (V1.1)

• Anti-infinite-loop guards. DATAPILOT_MAX_TOOL_CALLS (default 20) caps tool executions per run: once the cap is reached, further tool calls are aborted with an explicit error record and the analyst is routed to review. DATAPILOT_MAX_REVISIONS (default 2) caps the revision loop, and DATAPILOT_MAX_ITERATIONS (default 50) sets the LangGraph recursion limit.
• Tool trace metadata. Each entry in tool_calls includes call_id (the LLM's tool call id), sequence (1-based order), duration_ms (execution time), and error (message when the call failed). The same fields are streamed in SSE tool_call events and rendered in the frontend timeline.
• Mock LLM state isolation. The deterministic demo LLM is a process-wide singleton but holds no global script counters: the analyst script step is derived from the conversation history and reviewer counters are scoped per run_id. run_id is generated fresh for every fresh run (it never equals a reused thread_id), so consecutive or interleaved runs — including consecutive fresh runs on the same thread_id — never pollute each other.

Checkpoint / resume (V1.1)

Every run is checkpointed after each super-step by an in-process InMemorySaver (the version locked in requirements.txt). Checkpoints are keyed by thread_id; run_id is a separate, always-fresh per-run id that scopes per-run state (e.g. mock reviewer counters) and is preserved untouched across a resume:

• POST /api/analyze / POST /api/analyze/stream accept an optional thread_id in the request body. Without one, a fresh thread id is generated and returned in the response.
• If the thread already holds an unfinished checkpoint, the request resumes the run from that checkpoint instead of starting over.
• A resume is only accepted when the request's question and dataset_id match the checkpoint's canonical values. A mismatch on an unfinished thread is rejected with 409 THREAD_INPUT_MISMATCH (HTTP: JSON detail with code: "THREAD_INPUT_MISMATCH"; SSE: an error event with the same code) — the old run is never resumed while echoing the new request's metadata.
• A completed thread receiving a new request starts a fresh run on the same thread_id (new run_id, no inherited state) in both the REST and stream paths.
• GET /api/threads/{thread_id} returns the checkpoint snapshot: {thread_id, completed, next_nodes, state} (404 when the thread never ran).
• POST /api/threads/{thread_id}/resume resumes an interrupted thread from its last checkpoint and returns the final AnalyzeResponse (404 when the thread never ran).

Example:

# Run 1 (simulate an interruption) — remember the thread id
curl -X POST http://localhost:8000/api/analyze \
  -H 'Content-Type: application/json' \
  -d '{"question":"What are sales by region?","dataset_id":"abc123","thread_id":"run-1"}'

# Inspect the checkpoint
curl http://localhost:8000/api/threads/run-1

# Resume the interrupted run
curl -X POST http://localhost:8000/api/threads/run-1/resume

Note: the checkpointer is in-memory, so checkpoints live only as long as the backend process (see Known limitations).

Tool list

Tool	Purpose
inspect_dataset	Show dataset schema, missing values, and preview rows
query_dataset	Read-only DuckDB SQL over the uploaded table (dataset)
calculate_statistics	Descriptive stats for a column
get_dimension_values	Top unique values / counts for a categorical column
create_chart	Build an ECharts-compatible option (bar/line/pie/scatter)

All tools use Pydantic args_schema for parameter validation. query_dataset rejects modification keywords and multiple statements.

Eval suite (V1.1)

backend/eval is a minimal, deterministic eval suite. It drives the real agent graph with scripted LLM behavior and runs the real tools (including DuckDB SQL) against each case dataset, then asserts on the final state:

Scenario	Covers
tool_selection_and_sql_correctness	Tool selection order + query rows match ground truth
review_revision_loop	REVISE once, then PASS; chart added after feedback
unsupported_conclusion_rejected	Reviewer rejects a conclusion that contradicts the data; the fixed report drops it
unsupported_conclusion_terminates	Analyst never fixes: the revision cap terminates the run
reviewer_failure_never_passes	Failing reviewer yields REVIEW_ERROR, never PASS

Run it from the repository root:

python -m backend.eval.run_eval
# DataPilot V1.1 eval report
# ------------------------------------------------------------
# [PASS] tool_selection_and_sql_correctness (123ms)
# ...
# 5/5 scenarios passed

The suite is also executed by pytest via backend/tests/test_eval.py. Eval runs are reproducible: counter-based reviewer behavior is instantiated fresh per scenario run, so calling run_eval() twice in the same process yields identical, all-green results.

Repository layout

backend/
  main.py              FastAPI app (analyze, stream, threads/checkpoint API)
  config.py            Environment config
  schemas.py           API schemas (tool trace, thread snapshot)
  data/store.py        In-memory dataset store
  tools/               Five data tools + registry
  agent/               LangGraph state, nodes, prompts, graph, mock LLM
  eval/                V1.1 eval suite (cases, runner, CLI)
  tests/               Pytest tests (incl. reliability / review-error / eval)
frontend/
  app/                 Next.js App Router pages
  components/          Chat, preview, timeline, table, chart, report
  lib/                 API client and TypeScript types
.github/workflows/     CI (backend pytest+eval, frontend npm ci/lint/typecheck/build)
sample.csv
requirements.txt
.env.example

Getting started

1. Backend

cd DataPilot
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS/Linux
source .venv/bin/activate

pip install -r requirements.txt
cp .env.example .env
# Edit .env: either option works — DeepSeek or any OpenAI-compatible endpoint.
uvicorn backend.main:app --reload --port 8000

LLM providers

The agent talks to any OpenAI-compatible API through LangChain ChatOpenAI. No OpenAI key required — DeepSeek works out of the box:

# DeepSeek (only the key is required; endpoint and model auto-fill)
DEEPSEEK_API_KEY=sk-xxxx
# DEEPSEEK_MODEL=deepseek-chat            # default; supports tool calling
# DEEPSEEK_BASE_URL=https://api.deepseek.com  # auto-filled

# or OpenAI:
# OPENAI_API_KEY=sk-xxxx
# OPENAI_MODEL=gpt-4o-mini

DEEPSEEK_* takes precedence when both are set. For the analyst's autonomous tool selection, use a model with function calling support (e.g. deepseek-chat); reasoning-only models such as deepseek-reasoner may not support tool calls.

No API key at all? Set DATAPILOT_MOCK_LLM=true in .env (or the environment) to run the full workflow — tools, charts, revision loop — with a deterministic scripted LLM so you can demo the UI without any key. The real analyst path is used as soon as a key is set.

2. Frontend

cd frontend
npm install
npm run dev

Open http://localhost:3000. Upload sample.csv, then ask e.g. “What are total sales by region and category?”

On Windows you can also use start_backend.bat and start_frontend.bat in two terminals.

3. Tests / checks

# Backend (unit + reliability + eval suite)
pytest backend/tests -q

# Eval suite standalone
python -m backend.eval.run_eval

# Frontend
cd frontend
npm run lint
npm run typecheck
npm run build

The same checks run in CI (.github/workflows/ci.yml) on every push/PR.

Configuration

All secrets are read from environment variables (see .env.example). No API keys are committed. Either DEEPSEEK_API_KEY or OPENAI_API_KEY is required for the autonomous LLM analyst/reviewer; without one the workflow still runs but the reviewer reports REVIEW_ERROR (it never fabricates a PASS), and DATAPILOT_MOCK_LLM=true provides a keyless demo.

Variable	Default	Purpose
DATAPILOT_MAX_REVISIONS	2	Max reviewer revision loops
DATAPILOT_MAX_TOOL_CALLS	20	Max tool executions per run (loop guard)
DATAPILOT_MAX_ITERATIONS	50	LangGraph recursion limit per invocation
DATAPILOT_MOCK_LLM	off	Use the deterministic scripted LLM
DATAPILOT_CORS_ORIGINS	http://localhost:3000	Allowed CORS origins

Known limitations

• In-memory dataset storage: data is lost when the backend restarts.
• In-memory checkpointer: thread checkpoints are lost when the backend restarts; thread_id is only meaningful within one process lifetime.
• No login, permissions, multi-tenancy, RAG, vector DB, MCP, Redis, or complex DB.
• Designed for local demos and moderate-size files.
• The quality of analysis depends on the configured LLM; the reviewer is a safeguard, not a guarantee.