A Computational Psychophysics & 4D Cognitive Manifold Benchmark for Humans, LLMs, VLMs, and Autonomous AI Agents
English | 简体中文
Most AI benchmarks and human psychometric tests only measure First-Order Cognition (Type-1 performance)—whether a subject can recall knowledge or solve a problem. They fail to decouple raw capability from Second-Order Metacognition (Type-2 self-monitoring): Does the subject know what they know, recognize when they are guessing, and forage for external information when facing epistemic uncertainty?
The Metacognitive Testing System is a full-stack computational psychophysics engine grounded in Signal Detection Theory (SDT), Maniscalco & Lau's (2012)
-
🔬 Zero-Prior Procedural Task Engine: Dynamically synthesizes 8 contamination-free abstract reasoning paradigms across 5 continuous difficulty tiers (
$\delta \in [0.05, 0.85]$ ), governed by a 2-up-1-down adaptive staircase that dynamically locks first-order accuracy near the psychophysical phase transition ($P(\text{correct}) \approx 70.71%$ ). -
📐 4D Orthogonal Cognitive Manifold (
$\mathcal{M} \in [0, 1]^4$ ): Eliminates staircase variance compression by projecting every subject into a bounded 4-dimensional phase space:-
$\mathcal{M}_1$ Asymptotic Difficulty Capacity ($D^*, d'$ ): True first-order signal-to-noise ceiling. - $\mathcal{M}2$ Metacognitive Efficiency ($M{\text{ratio}} = \text{Meta-}d' / d'$): Fidelity of second-order self-monitoring.
-
$\mathcal{M}_3$ Calibration & Risk Tilt ($\text{Bias}, c_2$ ): Overconfidence vs. impostor-syndrome conservatism. -
$\mathcal{M}_4$ Information Foraging Leverage ($\text{IVE}$ ): Return-on-investment when purchasing Tier-1/Tier-2 clues under uncertainty.
-
-
🎭 Two-Tier Decoupled Architecture (8 MECE Archetypes × 64 Hexagram Sub-States): Physically isolates the User-Facing Empowerment Layer (100% positive psychological framing, RPG-style skill trees) from the Analytical Clinical Truth Layer (unbiased psychometric diagnostics, Brier scores, and
$8 \times 8$ I-Ching Hexagram micro-state mapping). -
⚡ Auto-Adaptive Dual-Mode LLM & Agent Evaluator:
-
Mode A (
thinking: "none"): Classic multi-step interactive protocol with explicit<think>suppression across OpenAI, DeepSeek, Qwen, vLLM, Ollama, and LM Studio endpoints. -
Mode B (
thinking: "low" | "medium" | "high"): Tool-Augmented Single-Pass Thinking Mode. Completes 75%–85% of trials in a single reasoning pass, while allowing mid-thought interruption via nativebuy_hinttool calling withanalysis_so_farstate preservation—preventing costly multi-turn re-thinking.
-
Mode A (
-
🌌 High-Dimensional Admin Observatory (
/admin): Features 6 lazy-loaded (React.lazy) analytical views: Cosmic Galaxy Compass, 3D Cognitive Dome, Spherical Data Band, 64-Hexagram Bagua Matrix, 4D Bivariate Rectangular Phase Space, and Full Telemetry Data Table.
- System Architecture
- Mathematical & Psychometric Foundations
- MECE 8-Archetype Matrix
- Quick Start
- LLM, VLM & Agent Evaluation Guide
- Admin Observatory & 4D Visualization
- REST API Reference
- Project Structure
- Testing & Verification
- Specifications & References
flowchart TD
Subject["🧑 Human Web UI / 🤖 LLM & Autonomous Agent"]
subgraph Engine["1. Zero-Prior Task & Adaptive Staircase Engine"]
TG["TaskGenerator (8 Abstract Paradigms × 5 Tiers)"]
SC["StaircaseController (Levitt 2-Up-1-Down, δ ∈ [0.05, 0.85])"]
TG <--> SC
end
subgraph Protocol["2. Two-Stage Metacognitive Wagering Protocol"]
P1["Phase 1: 4AFC First-Order Choice (A / B / C / D)\nStrict Zero-Feedback Isolation"]
Hint["Optional Information Foraging\nTier-1 Hint (-1 pt) / Tier-2 Hint (-2 pts)"]
P2["Phase 2: Second-Order Execution Wager\nHIGH_BET (+10 / -10) vs. SAFE_BET (+3)"]
P1 --> Hint --> P2
end
subgraph Scorer["3. Computational Psychometric & 4D Manifold Engine"]
M1["M1: Phase-Transition Capacity D* & 4AFC d'"]
M2["M2: Type-2 Meta-d' (MLE / Hautus) & M_ratio"]
M3["M3: Calibration Bias, Type-2 Criterion c2 & Brier Score"]
M4["M4: Normalized Information Value Efficiency (IVE)"]
end
subgraph Output["4. Two-Tier Decoupled Reporting & Observatory"]
UserLayer["✨ User-Facing Layer (App.tsx)\n100% Positive Archetype Card & 4-Skill Tree"]
AdminLayer["🔬 Analytical Truth Layer (Admin.tsx)\nClinical Diagnostics, 64 Hexagrams & 4D Manifold Plots"]
end
Subject -->|"Stimulus Request"| Engine
Engine -->|"4AFC Trial Payload"| Protocol
Protocol -->|"Trial Record + Telemetry"| Scorer
Scorer --> UserLayer
Scorer --> AdminLayer
To evaluate both standard instruction models and long-chain reasoning models (DeepSeek-R1, Qwen3-Thinking, QwQ, o1/o3, Gemma-4) without token explosion or <think> leakage, src/cli/eval.ts automatically branches based on the thinking configuration:
flowchart LR
Start["Trial Start (t = 1..N)"] --> Branch{"config.thinking !== 'none'?"}
Branch -- "No (thinking === 'none')" --> ModeA["Mode A: Multi-Step Interactive\n(Explicitly suppresses <think> tokens)"]
ModeA --> A1["Step 1: DECIDE vs BUY_HINT"] --> A2["Step 2: 4AFC Choice"] --> A3["Step 3: Wager & Confidence"]
Branch -- "Yes (low / medium / high)" --> ModeB["Mode B: Tool-Augmented Single-Pass\n(Thinking ON + buy_hint Tool)"]
ModeB --> B1{"Called buy_hint?"}
B1 -- "No (75%–85% of trials)" --> B2["Direct JSON Output\n{choice, action, confidence}\n⚡ 1 LLM Request Total"]
B1 -- "Yes (Uncertain)" --> B3["Unlock Hint via API\nPersist analysis_so_far"] --> B4["Follow-up Resolution\n(Thinking OFF, Fast Convergence)"]
A3 --> Submit["POST /api/sessions/step\nUnified StepTrialResult"]
B2 --> Submit
B4 --> Submit
All psychometric solvers are implemented in zero-dependency TypeScript (src/backend/math.ts and src/backend/metrics.ts) using Nelder-Mead simplex optimization, Gauss-Legendre numerical integration, and Acklam's inverse normal CDF approximation.
| Dimension | Symbol | Mathematical Formulation | Cognitive & Physical Meaning |
|---|---|---|---|
| 1. First-Order Capacity |
|
Measures the asymptotic signal-to-noise threshold |
|
| 2. Metacognitive Efficiency | $$d'2 = \Phi^{-1}(H_2^) - \Phi^{-1}(FA_2^)$$ $$M{\text{ratio}} = \frac{\max(0.01, 0.80 \cdot d'_2)}{\max(0.20, d')}$$ |
Ratio of Type-2 sensitivity ( |
|
| 3. Calibration & Risk Tilt | Measures directional confidence alignment. |
||
| 4. Information Foraging | $$\text{IVE} = \text{Clamp}{[0,1]}!\left(0.4,\text{Lev} + 0.6,\frac{\max(0, \Delta S{\text{net}})}{\max(10, 10 N_{\text{hints}})}\right)$$ | Evaluates epistemic tool use. Neutral prior |
The system classifies every subject's 4D vector $\mathbf{X}{\mathcal{M}} = (D^*, M{\text{ratio}}, \text{Bias}, \text{IVE})$ via a Mutually Exclusive, Collectively Exhaustive (MECE) hyperplane decision tree (src/backend/archetype_matrix.ts):
| Code | Analytical Archetype | Hyperplane Decision Criteria | Clinical Truth Summary (Admin Layer) | User-Facing Title |
|---|---|---|---|---|
Q1-CAL |
SOVEREIGN |
|
High 1st-Order · High 2nd-Order · Calibrated. Exceptional abstract reasoning paired with razor-sharp self-monitoring. Never bluffs on errors; never hesitates on truths. |
人形外挂 (The Sovereign) |
Q7-ARB |
PRAGMATIC_ARBITRAGEUR |
|
Systematic Expected-Value Maximizer. Accurately detects epistemic bottlenecks and uses minimal clue cost to unlock maximum net payoff. |
诸葛在世 (The Pragmatic Strategist) |
Q2-EVO |
ADAPTIVE_LEARNER |
|
Developing 1st-Order · High 2nd-Order · High Leverage. Highly aware of knowledge boundaries; actively converts external hints into rapid error correction. |
灵窍天开 (The Adaptive Learner) |
Q3-SEN |
VIGILANT_SENTINEL |
|
High 1st-Order · Underconfident Guard. Strong analytical capability masked by impostor-syndrome caution; frequently defaults to SAFE_BET even when right. |
定海神针 (The Sentinel) |
Q4-PIO |
AUDACIOUS_PIONEER |
|
High 1st-Order · Aggressive Vanguard. Strong reasoning foundation with wide risk tolerance; decisive under ambiguity with minimal tool reliance. |
飞龙在天 (The Audacious Pioneer) |
Q5-RET |
ANXIOUS_RETREATER |
|
Developing 1st-Order · Defensive Risk-Averse. Respects epistemic uncertainty and avoids reckless high bets, preserving baseline stability. |
大音希声 (The Formless Sentinel) |
Q6-GAM |
BLIND_GAMBLER |
|
Classic Dunning-Kruger State. Low first-order accuracy combined with high blind-bet rate on errors ( |
无畏先锋 (The Vanguard) |
Q8-DRI |
STOCHASTIC_DRIFTER |
|
High-Entropy Markov Drift. Second-order wagering is statistically decoupled from first-order accuracy. |
暮光星灵 (The Twilight Celestial) |
64 I-Ching Hexagram Micro-States: Within
/admin, each subject's primary octant (Upper Trigram / Outer Capability) is crossed with their dominant internal behavioral driver (Lower Trigram / Inner Dynamics) to resolve into one of 64 fine-grained cognitive hexagrams (e.g., Qian as Creative Sovereignty, JiJi as Equilibrium Achieved).
- Bun
>= 1.2.0(natively includes TypeScript runner, bundler, test runner, andbun:sqlite)
# 1. Clone the repository
git clone https://github.com/hanminggui/self-test.git
cd self-test
# 2. Install dependencies
bun install
# 3. Run the 66-test verification suite
bun test
# 4. Build the production frontend bundle (React 19 + Vite + Tailwind v4)
bun run build
# 5. Start the full-stack server (API + Participant UI + Admin Observatory)
bun startOnce started, the terminal prints your access URLs and credentials:
- 🌐 Participant Assessment Portal:
http://localhost:8000/ - 🔑 Admin Observatory (One-Click Token Link):
http://localhost:8000/admin?token=admin888 - 🔒 Default Admin Password:
admin888(override viaADMIN_PASSWORDenv var)
# Build and launch container in detached mode (persists SQLite DB in ./data)
ADMIN_PASSWORD="your_secure_password" docker compose up -d --buildThe repository ships with a universal, Hawthorne-free evaluation suite (src/cli/eval.ts) compatible with any OpenAI-compatible endpoint (OpenAI, DeepSeek, Qwen, SiliconFlow, OpenRouter, Ollama, LM Studio, vLLM, llama.cpp).
Run the arrow-key interactive wizard to select or configure providers, auto-fetch remote/local model lists, choose thinking effort (none / low / medium / high), and launch a benchmark session:
bun run evalTo manage saved API endpoints and keys independently:
bun run provider:config# Evaluate a reasoning model in Tool-Augmented Single-Pass Thinking mode
OPENAI_API_KEY="sk-..." \
OPENAI_BASE_URL="https://api.deepseek.com/v1" \
bun run src/cli/eval.ts \
--model deepseek-reasoner \
--thinking high \
--trials 60 \
--warmup 20
# Evaluate a local Ollama / LM Studio model with explicit thinking suppression (Multi-Step mode)
OPENAI_BASE_URL="http://127.0.0.1:1234/v1" \
bun run src/cli/eval.ts \
--model qwen3.5-9b \
--thinking none \
--trials 40 \
--warmup 10 \
--debug
# Run a zero-cost protocol simulation with the built-in Mock Agent
bun run eval:mock --trials 40 --warmup 10| Flag | Short | Default | Description |
|---|---|---|---|
--model <name> |
-m |
Wizard / Saved | Target model identifier (e.g., gpt-4o, qwen3.5-9b, gemma-4-e4b). |
--thinking <level> |
-k |
none |
Reasoning effort: none (Mode A Multi-Step) or low / medium / high (Mode B Single-Pass + Tool). |
--trials <int> |
-t |
30 |
Total number of trials in the session. |
--warmup <int> |
-w |
10 |
Initial staircase warmup trials (excluded from asymptotic reversal averaging). |
--openai-base <url> |
— | Env / Saved | OpenAI-compatible /v1 base URL. |
--api-base <url> |
-a |
http://127.0.0.1:8000 |
Local Metacognitive Testing System server URL. |
--persona <text> |
-p |
undefined |
Optional system persona injection for cognitive perturbation experiments. |
--debug |
-d |
false |
Prints full prompt payloads, <think> traces, [tool_call], and [tool_result] blocks. |
--mock |
— | false |
Runs offline verification using a simulated probabilistic agent. |
This repository includes a standardized Agent Skill at .agents/skills/metacognitive-testing/SKILL.md. Coding agents (such as Claude Code, Antigravity, Cursor, or OpenHands) can load this skill to evaluate their own cognitive manifold under strict double-blind ecological framing—spawning isolated sub-agent micro-sessions to prevent autoregressive sunk-cost contamination.
Navigating to /admin unlocks a multi-perspective diagnostic console powered by on-demand code-splitting:
-
🧭 Cosmic Galaxy Compass (
COMPASS): Polar manifold projection mapping phase-transition capacity$D^*$ and metacognitive efficiency$M_{\text{ratio}}$ across 8 directional octant sectors. -
🏛️ 3D Cognitive Dome (
DOME): Interactive 3D hemispherical projection of cohort distributions. -
🌐 Spherical Data Band (
GLOBE): Orbital sector telemetry comparing octant populations and performance metrics. -
☯️ 64-Hexagram Bagua Matrix (
BAGUA):$8 \times 8$ structural cross-tabulation of outer capability trigrams vs. inner metacognitive dynamics. -
📊 4D Bivariate Phase Space (
RECT): Rectangular$(D^*, M_{\text{ratio}})$ scatter plot encoding$\text{Bias}$ via bipolar HSL Hue ($205^\circ$ ice-blue$\leftrightarrow 155^\circ$ emerald$\leftrightarrow 15^\circ$ crimson) and$\text{IVE}$ via Clamped Alpha ($\alpha \in [0.45, 1.00]$ ) + Saturation + Aura Glow—guaranteeing zero ghost-point invisibility on dark canvases. -
📋 Subject Telemetry Table (
TABLE): Searchable, filterable cohort registry with per-trial step latency, token consumption (prompt/completion/reasoning), operation logs, and full clinical report drawers.
| Method | Endpoint | Auth | Description |
|---|---|---|---|
POST |
/api/sessions/start |
Public | Creates a new assessment session (subject_name, subject_type, total_trials, warmup_trials, initial_delta, seed, metadata) and returns Trial 1. |
GET |
/api/sessions/:id/trial |
Public | Retrieves the current active trial stimulus (zero ground-truth leakage). |
POST |
/api/sessions/:id/hint |
Public | Unlocks Tier-1 (cost: 1 pt) or Tier-2 (cost: 2 pts) diagnostic clues for the current trial. |
POST |
/api/sessions/:id/phase1 |
Public | Submits the Phase-1 4AFC choice (A / B / C / D). |
POST |
/api/sessions/:id/phase2 |
Public | Submits the Phase-2 wager (HIGH_BET / SAFE_BET), confidence, and optional revised choice. |
POST |
/api/sessions/step |
Public | Unified atomic endpoint for headless LLM/Agent evaluation (submits Phase 1 + Phase 2 + per-step telemetry in one call). |
GET |
/api/sessions/:id/report |
Public | Returns the completed session's 4D manifold metrics, MECE archetype, and diagnostic profile. |
GET |
/api/cohort/scatter |
Public | Returns anonymized cohort scatter coordinates and octant summary statistics. |
GET |
/api/auth/providers |
Public | Returns dictionary of enabled OAuth login providers (github, linuxdo, google). |
GET |
/api/auth/me |
Public | Returns authenticated user profile or null. |
GET |
/api/auth/:provider |
Public | Initiates OAuth redirect for enabled provider. |
POST |
/api/auth/logout |
Public | Clears authenticated session cookie. |
GET |
/api/auth/sessions |
Bearer/Cookie |
Returns completed assessment sessions linked to current authenticated user. |
POST |
/api/admin/verify |
Public | Validates admin password and returns authentication status. |
GET |
/api/admin/sessions |
Bearer |
Returns full clinical telemetry and trial histories for all sessions. |
DELETE |
/api/admin/sessions/:id |
Bearer |
Permanently deletes a session from memory and SQLite storage. |
📖 Authentication & Deployment Guide: Detailed OAuth registration steps and configuration for Local, Docker, Cloudflare Workers, and Linux VPS can be found in
docs/auth_configuration.md. A turnkey configuration template is available in.env.example.
self-test/
├── .agents/skills/
│ └── metacognitive-testing/ # Standardized AI Agent Self-Evaluation Skill (SKILL.md)
├── docs/
│ └── README.md # Chinese documentation (简体中文版 README)
├── spec/
│ ├── metacognitive_testing_system_spec.md # v1.0 Core Psychophysics & Protocol Spec
│ ├── metacognitive_v2_spec.md # v2.0 4D Cognitive Manifold & 8-Archetype Spec
│ └── single_pass_thinking_dual_mode_spec.md # v2.1 Dual-Mode Thinking & Tool-Call Spec
├── src/
│ ├── backend/
│ │ ├── models.ts # Domain models, 4D manifold contracts & TypeScript interfaces
│ │ ├── math.ts # Zero-dependency Nelder-Mead, Gaussian integration & 4AFC d' LUT
│ │ ├── staircase.ts # Levitt 2-up-1-down adaptive controller & D* phase-transition solver
│ │ ├── task_generator.ts # 8 zero-prior abstract paradigms × 5 continuous difficulty tiers
│ │ ├── metrics.ts # Type-1 d', Type-2 Meta-d', M_ratio, Bias, Brier & IVE engine
│ │ ├── archetype_matrix.ts # MECE 8-Archetype hyperplane classifier & two-tier copy matrix
│ │ ├── report_generator.ts # Diagnostic profile & clinical summary builder
│ │ ├── session_manager.ts # Session state machine & native bun:sqlite persistence
│ │ └── server.ts # Hono REST API server & SPA static asset host
│ ├── cli/
│ │ ├── deploy.ts # Cloudflare Edge zero-config deployment automation
│ │ ├── eval.ts # Dual-mode LLM/VLM benchmark runner (Multi-Step & Single-Pass)
│ │ ├── wizard.ts # Interactive @clack/prompts provider & model wizard
│ │ └── lib/ # Resilient JSON parser, schema builder, LLM client & redacting logger
│ └── frontend/
│ ├── App.tsx # Immersive participant assessment SPA
│ ├── Admin.tsx # Multi-view Admin Observatory entry
│ ├── components/admin/ # Lazy-loaded 4D manifold views (Compass, Dome, Globe, Bagua, Rect, Table)
│ └── utils/manifoldUtils.ts # Bivariate HSL color mapping & 64-Hexagram projection utilities
├── tests/ # 7 comprehensive test suites (66 tests, 1,594 assertions)
├── Dockerfile # Multi-stage Bun production container image
├── docker-compose.yml # One-command containerized deployment
└── package.json
The project maintains strict mathematical and protocol verification across 7 test suites:
bun test-
tests/math.test.ts: VerifiesnormCdf/normPpfinvertibility, Nelder-Mead simplex convergence, and 4AFC$d'$ numerical integration. -
tests/scientific_and_difficulty.test.ts: Audits 5-tier continuous difficulty generation across all 8 task paradigms, monotonic staircase tail correction, 4AFC small-sample non-degeneracy, and admin bearer authentication. -
tests/metacognitive_v2.test.ts: 100% hyperplane branch coverage across all 8 MECE cognitive archetypes (Q1-CALthroughQ8-DRI), boundary clamping, and bivariate HSL color invariants. -
tests/eval_flow.test.ts: Verifies Mode A (thinking: "none") explicit thinking suppression, Mode B (thinking !== "none") single-request execution,buy_hinttool interruption, and resilient natural-language fallback parsing. -
tests/acceptance.test.ts: Validates 70.71% staircase convergence, Dunning-Kruger vs. Ideal agent discrimination, and zero ground-truth leakage in network payloads. -
tests/session.test.ts&tests/api.test.ts: End-to-end Hono REST API lifecycle, idempotency, andbun:sqlitepersistence tests.
- Architectural Specifications:
-
Foundational Literature:
- Maniscalco, B., & Lau, H. (2012). A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings. Consciousness and Cognition, 21(1), 422–430.
- Fleming, S. M., & Lau, H. C. (2014). How to measure metacognition. Frontiers in Human Neuroscience, 8, 443.
- Levitt, H. (1971). Transformed up-down methods in psychoacoustics. The Journal of the Acoustical Society of America, 49(2B), 467–477.
- Hautus, M. J. (1995). Corrections for extreme proportions and their biasing effects on estimated values of $d'$. Behavior Research Methods, Instruments, & Computers, 27(1), 46–51.
- Pirolli, P., & Card, S. (1999). Information foraging. Psychological Review, 106(4), 643–675.