Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
1efda96
fix: make the gate fail closed and the README true
Vibherpunk Sep 19, 2026
bc0a518
merge: resolve conflicts with main (preserve truthful fixes)
Vibherpunk Sep 19, 2026
1952421
Merge pull request #1 from Vibherpunk/fix/truthful-gate
Vibherpunk Sep 19, 2026
69b1197
feat: HardTruth v2.0 - failure-biased execution ledger & inverted gate
Vibherpunk Sep 20, 2026
d8ca2d1
feat(v2.1): implement fail-closed hard gate, polyglot anti-stubbing, …
Vibherpunk Sep 20, 2026
1bcff76
feat(v2.1): harden tier2 execution, universal manifest integrity, mul…
Vibherpunk Sep 20, 2026
e5b5bc1
test(v2.1): provide explicit exitCode in test harness mock steps
Vibherpunk Sep 20, 2026
5c5414f
feat(v2.1): address Opus adversarial review findings (A1-A7, B1-B5, C…
Vibherpunk Sep 20, 2026
b04c74b
feat(v2.1): address Opus round 2 findings (A1-A12, B, C2-C3)
Vibherpunk Sep 20, 2026
d4e242b
fix(install): use nested hook command schema for Claude Code in setti…
Vibherpunk Sep 20, 2026
fba17f2
feat(v2.1): resolve Opus round 3 blocking findings (P1-P6, S1-S3, L6)
Vibherpunk Sep 20, 2026
f8fb868
feat(v2.1): add empirical A/B dogfood benchmark and harden baseline w…
Vibherpunk Sep 20, 2026
50e2256
feat(benchmarks): add and execute 100 legitimate physical verificatio…
Vibherpunk Sep 20, 2026
11373a2
docs(readme): add dual-audience overview, 100-test benchmark results,…
Vibherpunk Sep 20, 2026
e216cfa
fix(gate): isolate session source tracking from dirty rooms, add unit…
Vibherpunk Sep 21, 2026
e1c1187
fix(reconciliation): support SYSTEM_MESSAGE and async task descriptio…
Vibherpunk Sep 21, 2026
4db04df
fix(workspace): support dict and URI formats in get_workspace_dir
Vibherpunk Sep 21, 2026
f0a48f4
fix(tests): allow subprocess execution in test_tier2_runner_detection…
Vibherpunk Sep 21, 2026
22a5e8f
fix(reconciliation): support unittest discover and env prefix in suit…
Vibherpunk Sep 21, 2026
7bc2a8b
fix(weakening): ignore comments and docstrings in test weakening check
Vibherpunk Sep 21, 2026
8c1dcfd
fix(weakening): anchor test weakening patterns to statement start
Vibherpunk Sep 21, 2026
e9cd30c
fix(reconciliation): sync premise_str with reconciled failures
Vibherpunk Sep 21, 2026
aa2ec20
fix(rule4a): support basename matching in claimed file refs
Vibherpunk Sep 21, 2026
0d77a34
fix(claims): filter negative assertions from action trigger evaluation
Vibherpunk Sep 21, 2026
94559cc
feat(stop-gate): implement B1-B6 hardening, 45s deadline belt, sub-se…
Vibherpunk Sep 21, 2026
0509465
fix(reconcile): update last_test_step from all verified test commands…
Vibherpunk Sep 21, 2026
bc9f499
feat(hardening): battle-tested adversarial defenses, concurrency fuzz…
Vibherpunk Sep 21, 2026
45a382c
fix(daemon): implement Plan B CrossProcessLedgerLock, atomic POSIX ap…
Vibherpunk Sep 23, 2026
a60806a
fix(ledger): preserve strict path encompassment and support python3 -…
Vibherpunk Sep 23, 2026
f3aa5ec
feat(nli): upgrade neural verification engine to ModernBERT 2.0.0
Vibherpunk Sep 24, 2026
b0114c2
feat: compile hardtruth-gate to rust for <5ms cold starts
Vibherpunk Sep 25, 2026
bac046a
fix: adjust protojson compatibility for post_tool hook
Vibherpunk Sep 25, 2026
79a512d
feat: gracefully fallback to manual verification ledger if daemon unr…
Vibherpunk Sep 25, 2026
ec37706
fix(hardtruth-gate): enforce empty protojson contract on post_tool al…
Sep 25, 2026
30f950c
WIP: snapshot of deployed HardTruth (~/.hardtruth/lib) as of 2026-09-26
Sep 27, 2026
f2f269d
fix: Claude Code support, taint supersession, session-scoped Tier 2, …
Sep 27, 2026
19ff36d
fix: fail closed on unverifiable claims when daemon is down; make sui…
Sep 27, 2026
d19a7e3
feat: SessionStart baseline so pre-existing dirty files aren't this s…
Sep 28, 2026
33fb24a
preserve: working-tree state 2026-10-11 (Mac cleanup Phase 2)
Oct 11, 2026
f4409e9
client: bring hook in line with installed copy (2026-10-11 blocked-ed…
Oct 12, 2026
bf755c1
tier2: Go projects and DB-backed tests verify in the sealed container…
Oct 12, 2026
c59b744
tier2: detect TEST_DATABASE_URL in shared test-helper packages, not o…
Oct 12, 2026
98bba1b
merge: reconcile main (Rust v0.2.0) into the live Python v2 line
Oct 12, 2026
8f83c92
install: register the Claude Code SessionStart hook
Oct 12, 2026
14df40d
license: MIT for the Rust sources too
Oct 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: CI

on:
push:
branches: [ main, fix/truthful-gate ]
pull_request:
branches: [ main ]

jobs:
test:
runs-on: ubuntu-latest

steps:
- name: Check out repository
uses: actions/checkout@v4

- name: Set up Python 3.11
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements-dev.txt

- name: Run test suite with daemon UNREACHABLE (fail-closed verification)
run: |
SYSTEM_ONE_URL=http://127.0.0.1:9 pytest tests/

- name: Start HardTruth Daemon in background
run: |
HARDTRUTH_API_TOKEN=dummy-ci-token python3 daemon/app.py &
for i in {1..30}; do
if curl -s http://127.0.0.1:8000/health | grep -q "healthy"; then
echo "Daemon is healthy"
break
fi
sleep 1
done

- name: Run test suite with daemon UP (live NLI verification)
run: |
SYSTEM_ONE_URL=http://127.0.0.1:8000 HARDTRUTH_API_TOKEN=dummy-ci-token pytest tests/

- name: Stop Host Daemon
run: |
curl -X POST http://127.0.0.1:8000/shutdown || true
pkill -f "daemon/app.py" || true
sleep 2

- name: Run test suite with Live Containerized Daemon
run: |
docker build -t hardtruth-daemon -f docker/Dockerfile .
docker volume create hardtruth_state
docker run -d -p 127.0.0.1:8000:8000 \
-v hardtruth_state:/root/.hardtruth \
-e HARDTRUTH_API_TOKEN=dummy-ci-token \
-e HARDTRUTH_WORKSPACE_MAP="{\"$GITHUB_WORKSPACE\": \"/workspace\"}" \
-v "$GITHUB_WORKSPACE:/workspace:ro" \
--name hardtruth-daemon-container hardtruth-daemon
for i in {1..30}; do
if curl -s http://127.0.0.1:8000/health | grep -q "healthy"; then
echo "Container Daemon is healthy"
break
fi
sleep 1
done
SYSTEM_ONE_URL=http://127.0.0.1:8000 HARDTRUTH_API_TOKEN=dummy-ci-token pytest tests/
11 changes: 11 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,15 @@ __pycache__/
.venv/
env/
.pytest_cache/

# Handoff and audit battery documents
HANDOFF_PROMPT.md
*audit_findings*.md
round*_audit_findings.md

scratch/
benchmarks/
goose-plugin-hardtruth/

# Rust build output (hardtruthd / hardtruth-gate v0.2.0, Cargo.toml at root)
target/
1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ version = "0.2.0"
edition = "2021"
authors = ["Adam Matar <adam@skidnir.io>"]
description = "High-performance resident anti-hallucination and integrity daemon and thin-client CLI gate"
license = "MIT"

[lib]
name = "hardtruth"
Expand Down
152 changes: 152 additions & 0 deletions HARDTRUTH_SPECIFICATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
# HARDTRUTH SPECIFICATION: COMPREHENSIVE SOURCE OF TRUTH (v2.1)
## Autonomous Anti-Hallucination & Anti-Drift Execution Gate for Coding Agents

---

### 1. MISSION & SCOPE (NON-NEGOTIABLE INVARIANTS)

* **Primary Purpose:** Prevent autonomous coding models (`agy`, Claude Code, Goose, local LLMs) from hallucinating progress, fabricating code/test assertions, and drifting over long multi-turn execution loops.
* **Core Invariant:** "Evidence Before Assertions." An agent is physically blocked from completing its session until every action it claims to have taken (files created/modified, commands run, test suites passed) is corroborated by immutable OS- and git-level evidence.
* **Explicit Anti-Scope:** HardTruth explicitly rejects the "Stoin Universal Evidence Contract" and generalized multi-domain enterprise attestation frameworks (accounting, PMS, etc.). HardTruth is strictly a deterministic, language-aware execution and integrity gate for software development.
* **Security Invariant:** HardTruth must not increase the security risk of the host computer. Untrusted agent-written code must never execute within the verifier daemon's security domain where signing keys and physical ledgers reside.

---

### 2. THREAT MODEL & TRUST BOUNDARIES

* **Tier 0 (Confabulation & Drift - Primary Focus):** The model is not actively attacking the host OS, but due to context truncation, needle-in-haystack degradation, or sycophancy, it hallucinates that it wrote code, executed commands, or fixed bugs when it did not.
* **Tier 1 (Goal-Directed Evasion):** The agent operates with shell execution privileges and will attempt shortcuts to satisfy stop conditions (e.g. `git stash`, modifying test assertions to `assert True`, `pytest || true`, renaming test files, editing config switches). HardTruth must resist Tier 1 evasion deterministically.
* **Tier 2 (Root OS Compromise):** An attacker with full root access or hardware control is out of scope. HardTruth assumes standard user-space isolation.
* **Trust Boundary Invariant:** The verifier daemon is authoritative; the agent hook is an untrusted probe. All enforcement policies, timeouts, and kill switches are configured daemon-side. The agent cannot disable Tier 2 or clear halts via local environment variables.

---

### 3. HARNESS ADAPTER CONTRACT

HardTruth integrates across different agent runtime harnesses. Each harness requires an exact adapter mapping:

| Harness | Stop Event | Tool Event | Blocking Verb / Schema | Exit Code Handling | Timeout Budget |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Antigravity (`agy`)** | `Stop` | `PostToolUse` | `{"decision": "continue", "reason": "..."}` | Stdout string regex + ExitCode | Hook: 90s, Tier 2: 60s |
| **Claude Code** | `Stop` | `PostToolUse` | `{"decision": "block", "reason": "..."}` | ExitCode / stderr parsing | Hook: 90s, Tier 2: 60s |
| **Goose** | `on_stop` | `post_tool_execution` | `{"action": "halt", "message": "..."}` | Process exit code | Hook: 90s, Tier 2: 60s |

* **Timeout Invariant:** The installer must ensure `hook_timeout >= tier2_timeout + 25s`. If Tier 2 is allocated 60 seconds, the client harness hook timeout must be set to at least 85–90 seconds.

---

### 4. EVASION REGISTER & HARDENED DEFENSES

* **E1: `git stash` / Dirty Tree Cleanse:**
* *Vector:* Agent stashes broken code before stop so `git status` reads clean and Rule 1 vacuously passes.
* *Defense:* Halt if `git stash list` is non-empty at stop, or if files recorded as modified in the session ledger do not appear in the cumulative baseline diff.
* **E2: Test Runner Substitution / Suppression:**
* *Vector:* Agent adds `pytest.ini` with `addopts = -k none` or creates dummy test directories to fool detection.
* *Defense:* The canonical test runner is detected and pinned at **Session Bootstrap (Step 0)** and locked into the ledger; it is never re-detected at stop.
* **E3: Assertion Weakening & Test Dropping (Rule 5):**
* *Vector:* Agent changes `assert func() == True` to `assert True` or adds `@pytest.mark.skip` / `test.skip`.
* *Defense:* Compare test AST / assertion counts against baseline git blobs (`git show baseline:path`). Halt if test identifiers disappear, assertion counts decrease, or skip decorators are added without human waiver.
* **E4: Branch & History Rewriting:**
* *Vector:* Agent switches branches or checks out detached HEAD to hide diffs.
* *Defense:* Enforce `git merge-base --is-ancestor baseline HEAD`.
* **E5: Shell Masking & Taint Categories:**
* *Vector:* Agent uses `pytest || true` or `; exit 0` to fake success.
* *Defense:* Categorize commands into:
1. *Masking* (`|| true`, `; exit 0`, `|| echo pass`) $\rightarrow$ **TAINTED FAILURE** (permanent until resolved).
2. *Exit-Preserving* (`cd dir && pytest`, env prefixes `CI=1 pytest`) $\rightarrow$ **VALID VERIFICATION COMMAND**; extract effective directory.
3. *Exit-Lossy* (unpiped `|` without pipefail, uncaptured subshells) $\rightarrow$ `UNCORROBORATED_EXECUTION` (ignored: neither credit nor failure).
* **E6: Linter Substituted for Tests:**
* *Vector:* Agent runs `ruff` or `eslint` to satisfy "verification command" requirement.
* *Defense:* Split patterns into `TEST_EXECUTION` vs. `STATIC_CHECK`. Rule 1 strictly requires at least one passing `TEST_EXECUTION`.
* **E7: Synthetic Exit 0 Fallback Elimination:**
* *Vector:* `handle_post_tool_use()` defaulting to `observed_exit_code = 0` when transcript path is missing.
* *Defense:* Never synthesize `0`. If not corroborated by transcript or process runner metadata, mark as `UNCORROBORATED`.
* **E8: Transcript Spoofing:**
* *Vector:* Agent writes fake completion strings directly to transcript files.
* *Defense:* HardTruth cross-validates transcript entries against OS process execution receipts and daemon ledger records.

---

### 5. EXECUTION PIPELINE SPECIFICATION

When `handle_stop()` triggers, execution follows this strict sequence:

```
[Step 0: Session Bootstrap (at Init)]
(Capture baseline_sha, pin test runner, snapshot test ASTs, lock ledger)
↓
[Step 1: Check Halt Circuit Breaker]
(Differentiate agent-remediable halts vs. environment-undetermined)
↓
[Step 2: Universal Git & Worktree Integrity]
(Verify merge-base ancestry, empty stash, and cumulative diffs)
↓
[Step 3: Polyglot Anti-Stubbing Linter (Rule 3)]
(Scan changed hunks in TS, JS, PY, RS, GO for empty stubs)
↓
[Step 4: Source vs. Verification Check (Rule 1)]
(If source changed, at least 1 TEST_EXECUTION command must have run)
↓
[Step 5: Unresolved Failure Check (Rule 2)]
(All recorded command failures must have matching clean resolutions)
↓
[Step 6: Test-Surface Monotonicity Audit (Rule 5)]
(Verify no deleted assertions, dropped test cases, or new skip markers)
↓
[Step 7: Deterministic Claim-to-Action Grounding (Rule 4A)]
(Extract file paths & test commands from unstripped prose; verify in ledger)
↓
[Step 8: Semantic Contradiction Fallback (Rule 4B)]
(DeBERTa-v3 fallback for high-level narrative assertions)
↓
[Step 9: Tier 2 Independent Runner Handoff]
(Clean containerized execution; regression-based comparison against baseline)
↓
[DECISION: ALLOW]
```

#### Step 0: Session Bootstrap
* Upon session start (`v1/session/start` or first tool hook), HardTruth registers:
1. `baseline_commit_sha` via `git rev-parse HEAD`.
2. Pinned canonical test runner (e.g. `bun test`, `pytest`, `cargo test`).
3. Baseline test surface metrics (set of test IDs, assertion counts per test file).
4. Regression baseline: run Tier 2 once on baseline commit. Record any pre-existing failures (`baseline_failures`).

#### Step 3: Polyglot Anti-Stubbing (Hunk-Scoped)
* Only inspects lines modified by the agent (hunks in `git diff baseline..HEAD`).
* Detects:
* Python: empty `pass`, `...`, `raise NotImplementedError` in non-exception, non-abstract methods.
* TypeScript/JavaScript: `throw new Error("Not implemented")`, `// TODO: implement`, empty function blocks.
* Rust/Go: `todo!()`, `unimplemented!()`, `panic("not implemented")`.

#### Step 7: Rule 4A - Deterministic Claim-to-Action Grounding
* Operates on **unstripped agent prose** (including markdown backticks and tables).
* Extracts referenced file paths. If the prose affirms creating or modifying path `P`:
* Path `P` must exist in `git diff --name-only baseline..HEAD` with non-trivial hunks.
* If claimed and missing: **HARD REJECT (Claimed File Modification Not Corroborated)**.
* Extracts test execution assertions (e.g., *"I ran `cargo test` and all checks passed"*):
* Ledger must contain a matching `TEST_EXECUTION` command with confirmed `exit 0` run *after* the latest file edit.
* If claimed and missing: **HARD REJECT (Claimed Test Execution Not Corroborated)**.

#### Step 9: Regression-Based Tier 2 Hard Gate
* **Isolation:** Disposable container with `--security-opt no-new-privileges`, `--cap-drop ALL`, memory/cpu limits, `--network none`.
* **Dependencies:** Mount host dependency cache (`node_modules`, `.venv`) as strictly read-only (`:ro`).
* **Evaluation:** Tier 2 passes if:
$$\text{stop\_failures} \subseteq \text{baseline\_failures}$$
Any *new* failure not present at baseline halts the gate. Pre-existing failures do not block the agent.

---

### 6. CONCURRENCY & LEDGER INTEGRITY

* **Serialized Ledger Appends:** Ledger writes must acquire an exclusive `flock()` and in-process mutex. `prev_hash` is re-read under the lock before computing the HMAC digest and appending the record. Parallel tool calls can never create broken hash chains.
* **Fail-Closed Principle:** Every pipeline step returns `PASS`, `HALT`, or `UNDETERMINED`. Exceptions or unreachable dependencies return `UNDETERMINED` (halts execution; does not burn the agent's 3-strike escalation budget).

---

### 7. VERIFICATION & ADVERSARIAL EVALUATION HARNESS

* HardTruth v2.1 must pass a dedicated adversarial evaluation suite before deployment:
1. **Red Corpus (E1–E10 Scenarios):** Automated test runs attempting each evasion vector (stashing, assertion weakening, stubbing, prose lying, shell masking, synthetic exit codes). Required outcome: 100% caught (`HALT`).
2. **Green Corpus:** $\ge 30$ legitimate completion scenarios (pure refactors, new tests added, `cd dir && test`, pre-existing red tests, doc-only edits). Required outcome: 100% clean `ALLOW` (False Positive Rate $\le 2\%$).
3. **Host Safety:** Subprocess environment scrub and container permission drops verified against credential exfiltration attempts.
Loading
Loading