Final-year BTech project (CSE, AI/ML) at Keshav Memorial Institute of Technology, Hyderabad.
AI agents read untrusted text - web pages, bills, tool output - and then call tools, including money transfers. This project asks a simple question: what actually stops a hijacked agent from sending money to an attacker? The answer came from a measurement, not a guess.
1. We built a detector. Stage 1: canon() -> 3-BERT ensemble (RoBERTa + SecureBERT + DistilBERT majority vote). Parser + LIME were not in the 144 loop - the gate missed 0/125.
2. We tested it in a live agent loop. The detector ran as a gate on AgentDojo banking: 144 attack episodes plus 16 benign tasks, agent gpt-oss-120b.
3. It failed, and we published the failure. The detector missed every injected tool output (0 out of 125). Attack success with the detector was 70.14%, worse than 65.97% with no defense at all. Recognizing bad text does not stop a bad tool call.
4. So we built a lock instead. Stage 2 stopped reading the attacker's text and started checking the agent's arguments. Before send_money runs, the gate reads the destination and amount from the user's own task text only, and allows the call only on an exact match. A payee that lives only on a bill goes to a human. A bill is never the contract.
5. Same 144 episodes, zero theft. Attack success fell to 0.00% across 293 money calls. The price is honest: benign tasks whose payee exists only in a file now need human confirmation.
pdp/- the Stage 2 policy gate: contract extraction, allow/deny/confirm decisions, runtime taint tracking, human-in-the-loop flow.demo/- LEDGER, an interactive console (FastAPI + static UI, no API keys, runs offline). It has a live Stage 1 detector tab (both checkpoints, OLD and D1), a live Stage 2 gate with four scenarios, the frozen results table, and an honest limits page.src/- Stage 1 research code: ensemble, parser, LIME, calibration.data/p1_1_runs/- the frozen 144-episode scores everything here claims.tests/- 90 pytest tests, also run on GitHub Actions (Python 3.11).
python -m venv .venv
.venv\Scripts\activate # Linux/mac: source .venv/bin/activate
pip install -r requirements.txt
pytest tests/ -q
python demo/server.pyrequirements.txt covers the demo and tests. Model training weights are optional and stay out of git.
Same suite, same agent, four arms. Source: data/p1_1_runs/buckets.json.
| Stage | Arm | What it is | Attack success | Benign tasks |
|---|---|---|---|---|
| - | No defense | agent, no gate | 65.97% (95/144) | 13/16 |
| 1 | Detector only | text classifier on tool output | 70.14% (101/144) | 11/16 |
| 2 | Contract gate | tool/intent lock (decide) |
0.00% | 10/16 |
| 2 | Dest/amount lock | user-text lock + taint (decide_args) |
0.00% | 7/16 |
Evaluation counts "please confirm" as blocked, so attack payments are zero on the last two rows. Benign utility drops because some honest tasks keep the account number only in a file; in the demo a person can still press Allow.
A follow-up that rewrites tool output also failed to get an automatic payment (0/80). If the attacker can rewrite the user's own prompt, the lock allows 16/16 - that is the design, not a bug: the user's text is the license.
On each money-tool call:
- Mismatch: the destination or amount is not what the user wrote - deny.
- Unbound: the destination never appeared in the user text - ask a human.
- Bound match: both fields match the user text - allow.
Tool output and file text cannot authorize a destination. The Stage 1 detector still ships in the dashboard for analysis, but it cannot approve a payment.
Tested on AgentDojo banking, one agent model, a static 144-episode suite. Lab prototype, not a banking product. The lock trades convenience for safety on bill-payment tasks, and it only understands exact text matches today.
Python, FastAPI, Hugging Face Transformers (BERT fine-tuning and inference), LIME explainability, adversarial evaluation design, human-in-the-loop systems, honest failure analysis.
MIT. Copyright 2026 Mohd Ibadullah.