LLM Agent Security Evaluation and Control Lab

A reproducible evaluation framework for tool-using LLM agents. The lab measures how agents fail under adversarial inputs, then compares defenses by security gain and capability loss.

This is an empirical security research system, not an offensive toolkit. Agents run against sandboxed tools with controlled consequences. Every experiment writes configs, trajectories, and metrics that can be inspected later.

Research question

Can we systematically discover and measure failure modes in tool-using LLM agents, then test whether defenses reduce those failures without destroying useful capability?

What this project does

1. Runs an agent loop with constrained tools (filesystem and HTTP fixtures by default).
2. Applies attack scenarios such as direct prompt injection and indirect injection through retrieved content.
3. Optional defenses wrap prompts, gate tool calls, or combine both.
4. Scores attack success, task success, detection, HTTP exfiltration, and secret leakage.
5. Runs a paired benign utility suite so capability loss is not confounded with attack outcomes.
6. Writes run artifacts under runs/ for audit and comparison.

How it was built

The system is a Python package (agentsec) with Hydra YAML configs composing model, environment, attack, defense, and evaluation settings.

Architecture

• Harness: multi-turn agent loop that records assistant messages, tool calls, and tool results in OpenAI-compatible chat order.
• Backends: a deterministic mock model for offline tests, plus a local OpenAI-compatible HTTP client aimed at Ollama (also usable with vLLM, llama.cpp, or LocalAI).
• Tools: sandboxed filesystem reads and writes, plus fixture-backed HTTP requests. Live network fetch is off by default.
• Attacks: scenario builders that plant synthetic secrets and injection text inside controlled fixtures.
• Defenses: none, prompt_boundary (marks tool data as untrusted), tool_policy (blocks denylisted hosts and sensitive paths), and combined.
• Evaluation: keyword and channel-aware scoring across the full transcript, with refusal-aware checks to reduce false positives.
• Compare: sweeps defenses on one attack family and a paired utility suite, then reports security gain and capability loss.

Design choices

• Baseline system prompts stay helpful rather than pre-hardened, so defenses can be measured against a normal starting point.
• Strong injection keeps secrets in .env only, so text leak and HTTP exfil can be scored as separate channels.
• Capability is measured on benign tasks (repo summary, web summary) so utility is not mixed with attack compliance.
• The default real-model path is free and local. No paid cloud API key is required.

Stack

Area	Choice
Language	Python 3.11+
Packaging	pyproject.toml, editable install
Config	Hydra + OmegaConf
CLI	Typer entrypoints and python -m agentsec.*
Quality	pytest, ruff
Local models	Ollama via OpenAI-compatible /v1

Requirements

• Python 3.11 or newer
• pip and a virtual environment
• Optional for real-model runs: Ollama and a pulled instruct model such as qwen2.5:3b

Setup

git clone <this-repo>
cd llm-agent-security-evaluation

python -m venv .venv

Activate the environment:

# Windows (PowerShell)
.\.venv\Scripts\activate

# Linux / macOS
source .venv/bin/activate

Install the package:

pip install -e ".[dev]"

Running the lab

Offline smoke (mock model)

No Ollama required. Useful for verifying the install and reading artifacts quickly.

python -m agentsec.run
python -m agentsec.compare evaluation.repetitions=3
pytest -q

Local open-weight model (Ollama)

ollama pull qwen2.5:3b
python -m agentsec.doctor
python -m agentsec.run model=ollama evaluation.repetitions=1
python -m agentsec.compare model_group=ollama evaluation.repetitions=3

doctor checks that Ollama is reachable and that the configured model is installed.

Useful overrides

# Defenses
python -m agentsec.run defense=prompt_boundary
python -m agentsec.run defense=tool_policy
python -m agentsec.run defense=combined

# Attacks
python -m agentsec.run attack=direct_injection
python -m agentsec.run attack=direct_injection_strong
python -m agentsec.run attack=indirect_injection

# Benign utility suites
python -m agentsec.run attack=utility_repo
python -m agentsec.run attack=utility_web

# Compare with more trials or another attack family
python -m agentsec.compare model_group=ollama attack_name=direct_injection_strong evaluation.repetitions=5
python -m agentsec.compare model_group=ollama attack_name=indirect_injection evaluation.repetitions=3

Hydra overrides can change model, attack, defense, and evaluation settings without editing source.

What a run produces

Each experiment creates a folder under runs/:

runs/<run_id>/
  config.yaml
  trajectories.jsonl
  metrics.json
  model_metadata.json
  stdout.log

compare also writes a summary table plus JSON and CSV under runs/compare/. Reported fields include attack success rate, utility success rate, detection rate, HTTP exfil rate, secret leak rate, security gain, and capability loss.

Repository layout

conf/                 Hydra configs (model, attack, defense, environment, evaluation)
src/agentsec/         Package source
  backends/           Mock and OpenAI-compatible LLM clients
  harness/            Agent loop and scoring helpers
  tools/              Sandboxed filesystem and HTTP tools
  attacks/            Attack and utility scenario builders
  defenses/           Defense plugins
  eval/               Metric aggregation
  store/              Run artifact writers
experiments/          Named experiment configs (optional)
tests/                Unit tests
runs/                 Experiment outputs (gitignored)

Example findings from local runs

On qwen2.5:3b with the strong direct-injection suite, undefended runs often show high attack success while benign utility stays high. Prompt boundary alone may not reduce attack success. Tool policy and the combined defense can reduce attack success by blocking sensitive reads and denylisted hosts, with little utility loss on the paired benign suite.

Results vary by model and prompt.

Tests

pytest -q

Scope and non-goals

In scope: controlled evaluation of agent failures and defenses, sandboxed tools, local open-weight models, reproducible artifacts.

Out of scope for this codebase: attacking real third-party systems, unrestricted host shell or network access, or claiming safety certification of any model.