Skip to content

Commit b4612a0

Browse files
author
Vitor Castro
committed
Refactor the agent harness into phase toolsets, oracle gates, and orchestrated card runs.
Add benchmark fixtures/tasks, split .NET skills, and cover the new modules with unit tests.
1 parent 73c81ad commit b4612a0

161 files changed

Lines changed: 9311 additions & 1542 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.cursor/rules/english-texts.mdc‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
description: All project texts (prompts, guardrails, skills, rules, markers, READMEs, fixtures) are written in English
3+
alwaysApply: true
4+
---
5+
6+
# English-only texts
7+
8+
Everything the model or a developer reads in this repo is English:
9+
10+
- system prompts, nudges, guardrail/oracle notes, tool results and markers (`…[clipped: …]`, `(already in context …)`)
11+
- skills and rules (`extensions/codeforge/resources/**`, `governance/builtins.ts`)
12+
- status/activity labels, reports (`BUGS.md`, plan report), READMEs, `HEURISTICS.md`
13+
- benchmark prompts (`task.json`) and fixture docs/cards (`Depends on:`, `Acceptance criteria`)
14+
15+
Portuguese is allowed only in **matchers of user input** (regexes, skill `triggers`, stopwords), because users write in Portuguese:
16+
17+
```ts
18+
// ✅ matcher — keeps recognizing Portuguese requests
19+
/\b(review|revis\w*|encontr\w* (os )?bugs?)\b/i
20+
// ❌ text sent to the model
21+
`…[cortado: mostradas linhas 1-${n}]`
22+
// ✅
23+
`…[clipped: showing lines 1-${n} of ${total}]`
24+
```
25+
26+
Chat replies to the user follow the user's language; this rule is about files in the repo.

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,3 +83,5 @@ vscode/
8383

8484
# Local planning docs (keep offline)
8585
BLUEPRINT-02-Open-Source-AI-IDE-Revised-Embedded-AI-Platform.md
86+
87+
benchmarks/results/

‎CodeForge.sln‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
Microsoft Visual Studio Solution File, Format Version 12.00
2+
# Visual Studio Version 17
3+
VisualStudioVersion = 17.5.2.0
4+
MinimumVisualStudioVersion = 10.0.40219.1
5+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "benchmarks", "benchmarks", "{66320409-64EC-F7C5-3DEF-65E7510DAAD1}"
6+
EndProject
7+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "fixtures", "fixtures", "{AFA72641-1A30-5FD9-46FF-8D3CF9AF32B9}"
8+
EndProject
9+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "billing", "billing", "{D9734131-DA50-2FA4-E292-8A411DE028FF}"
10+
EndProject
11+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{87F2DFEF-6771-6C5F-308A-534D1B05A8F9}"
12+
EndProject
13+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Acme.Billing", "benchmarks\fixtures\billing\src\Acme.Billing\Acme.Billing.csproj", "{F2A9ADC8-9C72-F8A6-E9F4-2AE26A42D867}"
14+
EndProject
15+
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{75E704B8-F95A-BA8B-0982-342EDA781354}"
16+
EndProject
17+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Acme.Billing.Tests", "benchmarks\fixtures\billing\tests\Acme.Billing.Tests\Acme.Billing.Tests.csproj", "{2CBCF198-0448-A543-E33E-B0A55F3610E8}"
18+
EndProject
19+
Global
20+
GlobalSection(SolutionConfigurationPlatforms) = preSolution
21+
Debug|Any CPU = Debug|Any CPU
22+
Release|Any CPU = Release|Any CPU
23+
EndGlobalSection
24+
GlobalSection(ProjectConfigurationPlatforms) = postSolution
25+
{F2A9ADC8-9C72-F8A6-E9F4-2AE26A42D867}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
26+
{F2A9ADC8-9C72-F8A6-E9F4-2AE26A42D867}.Debug|Any CPU.Build.0 = Debug|Any CPU
27+
{F2A9ADC8-9C72-F8A6-E9F4-2AE26A42D867}.Release|Any CPU.ActiveCfg = Release|Any CPU
28+
{F2A9ADC8-9C72-F8A6-E9F4-2AE26A42D867}.Release|Any CPU.Build.0 = Release|Any CPU
29+
{2CBCF198-0448-A543-E33E-B0A55F3610E8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
30+
{2CBCF198-0448-A543-E33E-B0A55F3610E8}.Debug|Any CPU.Build.0 = Debug|Any CPU
31+
{2CBCF198-0448-A543-E33E-B0A55F3610E8}.Release|Any CPU.ActiveCfg = Release|Any CPU
32+
{2CBCF198-0448-A543-E33E-B0A55F3610E8}.Release|Any CPU.Build.0 = Release|Any CPU
33+
EndGlobalSection
34+
GlobalSection(SolutionProperties) = preSolution
35+
HideSolutionNode = FALSE
36+
EndGlobalSection
37+
GlobalSection(NestedProjects) = preSolution
38+
{AFA72641-1A30-5FD9-46FF-8D3CF9AF32B9} = {66320409-64EC-F7C5-3DEF-65E7510DAAD1}
39+
{D9734131-DA50-2FA4-E292-8A411DE028FF} = {AFA72641-1A30-5FD9-46FF-8D3CF9AF32B9}
40+
{87F2DFEF-6771-6C5F-308A-534D1B05A8F9} = {D9734131-DA50-2FA4-E292-8A411DE028FF}
41+
{F2A9ADC8-9C72-F8A6-E9F4-2AE26A42D867} = {87F2DFEF-6771-6C5F-308A-534D1B05A8F9}
42+
{75E704B8-F95A-BA8B-0982-342EDA781354} = {D9734131-DA50-2FA4-E292-8A411DE028FF}
43+
{2CBCF198-0448-A543-E33E-B0A55F3610E8} = {75E704B8-F95A-BA8B-0982-342EDA781354}
44+
EndGlobalSection
45+
GlobalSection(ExtensibilityGlobals) = postSolution
46+
SolutionGuid = {DC61E0B0-E006-49BD-AAAE-6F2B47245DB7}
47+
EndGlobalSection
48+
EndGlobal

‎benchmarks/README.md‎

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,38 @@
11
# CodeForge Benchmarks
22

3-
This directory contains benchmark suites for evaluating CodeForge's AI capabilities.
3+
## Harness benchmark (available now)
4+
5+
Semi-automatic suite: the agent runs inside CodeForge; `check` scores with hidden tests + session metrics.
6+
7+
### JavaScript (`fixtures/shop`) — T1–T4
8+
Quick harness smoke (node). Good for catching retrieve/read loops; **not** your real C# workload.
9+
10+
### C# / .NET 10 (`fixtures/billing` + `scaffold`) — CS1–CS4
11+
Closer to ContextRouter: `dotnet test`, classlibs, docs-driven features, VIP navigation, and a **scaffold** task (create solution from a C01 card).
12+
13+
```bash
14+
npm run bench -- list
15+
npm run bench -- setup CS1 --label bonsai --open # or CS2 / CS3 / CS4
16+
# New Chat in that folder → bonsai → paste the printed prompt
17+
npm run bench -- check latest
18+
npm run bench -- report
19+
npm run bench -- selftest CS # validates fixtures + solutions
20+
```
21+
22+
Score (0–100): hidden 60 · visible 15 · protected untouched 10 · checks 10 · no junk files 5.
23+
24+
### Long and review (`fixtures/router-cards`, `fixtures/billing-bugs`) — CS5–CS6
25+
26+
- **CS5 (review)**: 6 bugs planted in `billing-bugs` (visible tests still pass). The agent writes
27+
`BUGS.md` with `path:line` references; the manifest lives in `tasks/CS5-review-billing-bugs/bugs.manifest.json`
28+
(outside the fixture). Score: F1 × 80 · protected 10 · `BUGS.md` exists 10 (±3 line tolerance).
29+
- **CS6 (long)**: cards C01–C05 with "Depends on:" lines. Hidden tests run **per card** (only that
30+
card's tests are compiled in); the hidden ratio is the average over cards.
31+
32+
Every result also reports `falseDone` (the agent claimed green while tests were red) and
33+
`harness` (chars of harness text injected into the conversation).
34+
35+
## Planned suites
436

537
## Categories
638

0 commit comments

Comments
 (0)