From 438010335820229b207a8f19f870ee27ea026d65 Mon Sep 17 00:00:00 2001 From: Omkar P <45419097+omkar-foss@users.noreply.github.com> Date: Thu, 13 Aug 2026 18:40:11 +0530 Subject: [PATCH 1/4] Add scoring doc, update readme Signed-off-by: Omkar P <45419097+omkar-foss@users.noreply.github.com> --- README.md | 42 ++++++++++++++++++++++++++---------------- SCORING.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 16 deletions(-) create mode 100644 SCORING.md diff --git a/README.md b/README.md index f19dff1..2038747 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# CHAOSS AI Detection Tool +# CHAOSS AI disclosure detection -A standalone CLI tool and GitHub Action that detects AI-generated contributions in git repositories. It works entirely from git-level data (commit emails, messages, trailers) using [go-git](https://github.com/go-git/go-git), with no platform API dependencies in the core. A separate text-scanning mode lets wrappers pipe in PR descriptions, issue comments, or any other text. +A standalone CLI tool and GitHub Action that detects disclosed AI-generated contributions in git repositories. It works entirely from git-level data (commit emails, messages, trailers) using [go-git](https://github.com/go-git/go-git), with no platform API dependencies in the core. A separate text-scanning mode lets wrappers pipe in PR descriptions, issue comments, or any other text. The goal is to help open source maintainers understand when AI tools are involved in contributions, and to give community health projects like [CollectOSS](https://github.com/chaoss/collectoss/) and [GrimoireLab](https://github.com/chaoss/grimoirelab/) a way to track AI usage across repositories. @@ -8,27 +8,27 @@ The goal is to help open source maintainers understand when AI tools are involve The built-in detectors run against each commit, each producing findings at a confidence level: -**High confidence** -- strong signals that an AI tool authored or co-authored the commit: +- `Co-Authored-By` and `Assisted-By` trailers with known AI tool emails (Claude Code, Cursor, Aider). +- Known commit trailers in formats unique to specific tools (such as Aider, EntireIO, Replit Agent/Assistant etc.) or footers (like `Generated with Claude Code`) that can contain values indicative of AI use. - Known AI bot committer emails (Claude, Copilot, Cursor, Codex, Gemini Code Assist, Amazon Q, Devin, Cline, Continue.dev, Cody, JetBrains AI, CodeRabbit). Also matches on the numeric prefix of GitHub noreply emails, so bot username renames don't break detection. -- `Co-Authored-By` trailers with known AI tool emails (Claude Code, Cursor, Aider). - `git-ai` authorship logs stored in git notes under `refs/notes/ai`, including the attributed tool and model when available. -- AI session ID trailers (such as Replit-Commit-Session-Id) combined with other known commit trailers, indicating that the commit was generated as part of an AI conversation or workflow. - -**Medium confidence** -- patterns in the commit message itself: -- `aider:` prefix (Aider's default commit format). -- `Generated with Claude Code` footer. -- Known commit trailers in formats unique to specific tools (such as EntireIO, Replit Agent/Assistant) that can contain values indicative of AI use. - Branch names following conventions used by AI coding CLIs/agents (e.g. `codex/`, `claude/`, `cursor/`, `copilot/`, `devin/`, `cline/`, `aider/`, `gemini/`). - - -**Low confidence** -- mentions of AI tool names in text: -- Word-boundary matches for tool names like Claude, Copilot, Cursor, Aider, ChatGPT, Windsurf, Devin, etc. This detector also runs against commit messages, and is the primary detector for the text-scanning mode (PR bodies, comments). +- AI session ID trailers (such as Replit-Commit-Session-Id) combined with other known commit trailers, indicating that the commit was generated as part of an AI conversation or workflow. +- Mentions of tool names like Claude, Copilot, Cursor, Aider, ChatGPT, Windsurf, Devin, etc. This detector also runs against commit messages, and is the primary detector for the text-scanning mode (PR bodies, comments). +- Disclosure of AI use from checkboxes in pull request description or comments, in text-scanning mode. Checkbox labels configurable by user. ## CLI usage ``` -disclosure scan [--range=BASE..HEAD] [--format=json|text] [--min-confidence=low|medium|high] [repo-path] -disclosure text [--format=json|text] [--input=FILE|-] +disclosure scan \ + [--range=BASE..HEAD] [--format=json|text] \ + [--min-confidence=low|medium|high] \ + [--confidence-levels="low=30,medium=70,high=100"] + [repo-path] +disclosure text \ + [--format=json|text] [--input=FILE|-] \ + [--checkbox-label-ai-used="AI was used"] \ + [--checkbox-label-ai-not-used="AI was not used"] disclosure version ``` @@ -57,6 +57,11 @@ echo "I used Claude to write this PR" | disclosure text --format=json disclosure text --input=pr-body.txt ``` +### Numeric scoring + +Please see [SCORING.md](SCORING.md) for more information on disclosure's +scoring methodology. + ### Use as a CI gate The exit code makes it usable in shell pipelines and CI scripts: @@ -181,6 +186,11 @@ cmd/ CLI subcommands action/ GitHub Action (composite action + labeling) ``` +## Other AI disclosure/attribution tools + +- [AItrributor](https://github.com/block/aittributor) - Prepare-commit-msg hook that adds AI agent attribution to git commits. +- [Usagescale](https://usagescale.org/) - An open standard for declaring how a work was made, whose knowledge it carries, and who stands behind it. + ## Contributing See [CONTRIBUTING.md](CONTRIBUTING.md). diff --git a/SCORING.md b/SCORING.md new file mode 100644 index 0000000..03bd1aa --- /dev/null +++ b/SCORING.md @@ -0,0 +1,45 @@ +# Disclosure numeric scoring (20th May 2026) + +Related issue: https://github.com/chaoss/ai-detection-action/issues/12 + +## Simple additive scoring + +We use simple additive scoring per detector to compute the final score. Steps are as follows: +1. Every detector produces one or more findings per commit. +2. For every commit, the scoring is then grouped per-detector e.g. for commit C if there are two +findings for detector `trailer`, one with score 35 and another with score 45, then `max()` is used +to aggregate per detector findings at each commit. So in this case, commit C will have score 45 for +detector type `trailer`. +3. The per detector scores are then adds for each commit to get the score for a particular commit. +e.g. if commit C gets per detector scores of 75.0 and 85.0 from detectors `trailer` and `toolmention` +detectors respectively, then the total score for commit C will be 75 + 85 = 160.0 +4. Confidence is calculated at commit as well as finding level. It's based on the default confidence +levels unless user-specified: +- low confidence for score 0 to 30 +- medium confidence for score 31 to 70 +- high confidece for score 71 to 100 + +### Example (branch feature/sample-commit) +```git +Author: Jon Snow +Date: Sat May 17 11:42:08 2026 +0530 + +feat(auth): add JWT refresh token rotation and session invalidation + +Implemented refresh token rotation for improved session security. +Users now receive a new refresh token on every refresh request, +and reused/expired tokens invalidate the session automatically. + +Co-authored-by: Claude +``` + +In above commit, scoring will be as follows: +1. Trailer - Yes, one Co-Author finding, matches known trailer Claude Code (40) with known email (35) = 75.0 +2. Committer - No, committer email address doesn't match known AI bot email addresses = 0.0 +3. Branch - No, branch does not have known tools = 0.0 +4. Gitnotes - No gitnotes found = 0.0 +5. toolmention - Yes, one finding, tool Claude matched = 20.0 + +**Total score: 75 + 0 + 0 + 0 + 20 = 95 pts** + +95 pts lies in 71 to 100 range, so it falls in confidence level here is **high**. From 29ac294644f07632cd6f7dc416bacc9d6994e8b1 Mon Sep 17 00:00:00 2001 From: Omkar P <45419097+omkar-foss@users.noreply.github.com> Date: Fri, 21 Aug 2026 14:30:14 +0530 Subject: [PATCH 2/4] Update scoring examples Signed-off-by: Omkar P <45419097+omkar-foss@users.noreply.github.com> --- SCORING.md | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 68 insertions(+), 3 deletions(-) diff --git a/SCORING.md b/SCORING.md index 03bd1aa..3e587b4 100644 --- a/SCORING.md +++ b/SCORING.md @@ -1,6 +1,6 @@ # Disclosure numeric scoring (20th May 2026) -Related issue: https://github.com/chaoss/ai-detection-action/issues/12 +Related issue: https://github.com/chaoss/disclosure/issues/12 ## Simple additive scoring @@ -15,7 +15,8 @@ e.g. if commit C gets per detector scores of 75.0 and 85.0 from detectors `trail detectors respectively, then the total score for commit C will be 75 + 85 = 160.0 4. Confidence is calculated at commit as well as finding level. It's based on the default confidence levels unless user-specified: -- low confidence for score 0 to 30 +- no confidence has score 0 +- low confidence for score 1 to 30 - medium confidence for score 31 to 70 - high confidece for score 71 to 100 @@ -31,10 +32,11 @@ Users now receive a new refresh token on every refresh request, and reused/expired tokens invalidate the session automatically. Co-authored-by: Claude +Assisted-by: Github Copilot ``` In above commit, scoring will be as follows: -1. Trailer - Yes, one Co-Author finding, matches known trailer Claude Code (40) with known email (35) = 75.0 +1. Trailer - Yes, one Co-Author finding matches known trailer with tool Claude Code (40) and known email (35). Another known Assisted-by finding matches known trailer using tool Github Copilot (75.0) = 75.0 2. Committer - No, committer email address doesn't match known AI bot email addresses = 0.0 3. Branch - No, branch does not have known tools = 0.0 4. Gitnotes - No gitnotes found = 0.0 @@ -43,3 +45,66 @@ In above commit, scoring will be as follows: **Total score: 75 + 0 + 0 + 0 + 20 = 95 pts** 95 pts lies in 71 to 100 range, so it falls in confidence level here is **high**. + +### CLI Output Example + +#### Example 1: disclosure scan +```sh +$ disclosure scan +Scanned 2 commits, 1 with AI signals + +Tools detected: + Claude Code: 1 + +Commit abc123def456 (score: 100.0, confidence: high) + [score: 100.0, confidence: high] Claude Code [Opus 4] (trailer): Co-Authored-By trailer with email noreply@anthropic.com +``` + +#### Example 2: disclosure text + +For a file `pr-body.txt` with checkboxes: +``` +Generative AI disclosure + +Please select one option: + +[x] This PR uses AI/LLMs +[ ] This PR does not use AI/LLMs + +If AI tools were used, please provide details below: +- What tools were used? Claude +- How were these tools used? For code review +- Did you review these outputs before submitting this PR? Yes +``` + +When run through `disclosure text`, it produces findings that confirm AI use: +```sh +$ disclosure text --input=pr-body.txt --confidence-levels=low=30,medium=70,high=100 --format=json --enable-checkbox-detection +{ + "commits": { + "commits": [ + ... + ], + "summary": { + "total_commits": 9, + "ai_commits": 0, + "tool_counts": {}, + "by_confidence": {}, + "per_detector_scores": null + } + }, + "text": { + "findings": [ + { + "detector": "toolmention", + "tool": "Claude", + "confidence": "high", + "score": 95, + "detail": "checkbox confirms AI was used and text mentions Claude" + } + ], + "score": 95, + "confidence": "high" + } +} +``` From bf0736314933f88b4341afc9b560a83530b3b5c0 Mon Sep 17 00:00:00 2001 From: Omkar P <45419097+omkar-foss@users.noreply.github.com> Date: Fri, 21 Aug 2026 16:15:22 +0530 Subject: [PATCH 3/4] Update scoring examples, add separate one for checkbox detection Signed-off-by: Omkar P <45419097+omkar-foss@users.noreply.github.com> --- SCORING.md | 75 ++++++++++++++++++++++++++++-------------------------- 1 file changed, 39 insertions(+), 36 deletions(-) diff --git a/SCORING.md b/SCORING.md index 3e587b4..cc8ffad 100644 --- a/SCORING.md +++ b/SCORING.md @@ -5,22 +5,25 @@ Related issue: https://github.com/chaoss/disclosure/issues/12 ## Simple additive scoring We use simple additive scoring per detector to compute the final score. Steps are as follows: + 1. Every detector produces one or more findings per commit. 2. For every commit, the scoring is then grouped per-detector e.g. for commit C if there are two -findings for detector `trailer`, one with score 35 and another with score 45, then `max()` is used -to aggregate per detector findings at each commit. So in this case, commit C will have score 45 for -detector type `trailer`. + findings for detector `trailer`, one with score 35 and another with score 45, then `max()` is used + to aggregate per detector findings at each commit. So in this case, commit C will have score 45 for + detector type `trailer`. 3. The per detector scores are then adds for each commit to get the score for a particular commit. -e.g. if commit C gets per detector scores of 75.0 and 85.0 from detectors `trailer` and `toolmention` -detectors respectively, then the total score for commit C will be 75 + 85 = 160.0 + e.g. if commit C gets per detector scores of 75.0 and 85.0 from detectors `trailer` and `toolmention` + detectors respectively, then the total score for commit C will be 75 + 85 = 160.0 4. Confidence is calculated at commit as well as finding level. It's based on the default confidence -levels unless user-specified: + levels unless user-specified: + - no confidence has score 0 - low confidence for score 1 to 30 - medium confidence for score 31 to 70 - high confidece for score 71 to 100 ### Example (branch feature/sample-commit) + ```git Author: Jon Snow Date: Sat May 17 11:42:08 2026 +0530 @@ -36,6 +39,7 @@ Assisted-by: Github Copilot ``` In above commit, scoring will be as follows: + 1. Trailer - Yes, one Co-Author finding matches known trailer with tool Claude Code (40) and known email (35). Another known Assisted-by finding matches known trailer using tool Github Copilot (75.0) = 75.0 2. Committer - No, committer email address doesn't match known AI bot email addresses = 0.0 3. Branch - No, branch does not have known tools = 0.0 @@ -49,8 +53,9 @@ In above commit, scoring will be as follows: ### CLI Output Example #### Example 1: disclosure scan + ```sh -$ disclosure scan +$ disclosure scan --confidence-levels=low=30,medium=70,high=100 Scanned 2 commits, 1 with AI signals Tools detected: @@ -60,9 +65,30 @@ Commit abc123def456 (score: 100.0, confidence: high) [score: 100.0, confidence: high] Claude Code [Opus 4] (trailer): Co-Authored-By trailer with email noreply@anthropic.com ``` -#### Example 2: disclosure text +#### Example 2: disclosure text command + +For a file `file.txt` with checkboxes: + +``` +Claude and Chatgpt were used for the code, while Copilot was used for reviews, +and some other AI tools may have been used for documentation. +``` + +When run through `disclosure text`, it produces findings that confirm AI use: + +```sh +$ disclosure text --input=pr-body.txt --format=text --enable-checkbox-detection +Found 3 AI signal(s): +Score: 20.0, Confidence: low + [score: 20.0, confidence: low] Claude (toolmention): text mentions Claude + [score: 20.0, confidence: low] Chatgpt (toolmention): text mentions Chatgpt + [score: 20.0, confidence: low] Copilot (toolmention): text mentions Copilot +``` + +#### Example 3: disclosure text command with checkbox detection enabled For a file `pr-body.txt` with checkboxes: + ``` Generative AI disclosure @@ -78,33 +104,10 @@ If AI tools were used, please provide details below: ``` When run through `disclosure text`, it produces findings that confirm AI use: + ```sh -$ disclosure text --input=pr-body.txt --confidence-levels=low=30,medium=70,high=100 --format=json --enable-checkbox-detection -{ - "commits": { - "commits": [ - ... - ], - "summary": { - "total_commits": 9, - "ai_commits": 0, - "tool_counts": {}, - "by_confidence": {}, - "per_detector_scores": null - } - }, - "text": { - "findings": [ - { - "detector": "toolmention", - "tool": "Claude", - "confidence": "high", - "score": 95, - "detail": "checkbox confirms AI was used and text mentions Claude" - } - ], - "score": 95, - "confidence": "high" - } -} +$ disclosure text --input=pr-body.txt --format=text --enable-checkbox-detection +Found 1 AI signal(s): +Score: 95.0, Confidence: high + [score: 95.0, confidence: high] Claude (toolmention): checkbox confirms AI was used and text mentions Claude ``` From 81a83bace49a1dbfc06d2e479f8e7fd18ec78de6 Mon Sep 17 00:00:00 2001 From: Omkar P <45419097+omkar-foss@users.noreply.github.com> Date: Fri, 21 Aug 2026 16:46:38 +0530 Subject: [PATCH 4/4] Update scoring examples, reformat markdown Signed-off-by: Omkar P <45419097+omkar-foss@users.noreply.github.com> --- SCORING.md | 76 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 64 insertions(+), 12 deletions(-) diff --git a/SCORING.md b/SCORING.md index cc8ffad..713814e 100644 --- a/SCORING.md +++ b/SCORING.md @@ -1,6 +1,6 @@ # Disclosure numeric scoring (20th May 2026) -Related issue: https://github.com/chaoss/disclosure/issues/12 +Related issue: ## Simple additive scoring @@ -46,7 +46,7 @@ In above commit, scoring will be as follows: 4. Gitnotes - No gitnotes found = 0.0 5. toolmention - Yes, one finding, tool Claude matched = 20.0 -**Total score: 75 + 0 + 0 + 0 + 20 = 95 pts** +**Total score**: 75 + 0 + 0 + 0 + 20 = **95 pts** 95 pts lies in 71 to 100 range, so it falls in confidence level here is **high**. @@ -54,22 +54,65 @@ In above commit, scoring will be as follows: #### Example 1: disclosure scan +When `disclosure scan` is run on its [own git repository](https://github.com/chaoss/disclosure), +at the time of this writing: + ```sh $ disclosure scan --confidence-levels=low=30,medium=70,high=100 -Scanned 2 commits, 1 with AI signals +Scanned 103 commits, 9 with AI signals Tools detected: + Aider: 3 + Claude: 2 Claude Code: 1 - -Commit abc123def456 (score: 100.0, confidence: high) - [score: 100.0, confidence: high] Claude Code [Opus 4] (trailer): Co-Authored-By trailer with email noreply@anthropic.com + Cline: 1 + Codex: 1 + Copilot: 1 + Cursor: 2 + Devin: 1 + GLM-4: 1 + Kimi: 1 + Replit: 3 + t3.chat: 2 + +Commit 64cddd958faa (score: 20.0, confidence: low) + [score: 20.0, confidence: low] Codex (toolmention): text mentions Codex + [score: 20.0, confidence: low] Claude (toolmention): text mentions Claude + [score: 20.0, confidence: low] Cursor (toolmention): text mentions Cursor + [score: 20.0, confidence: low] Copilot (toolmention): text mentions Copilot + [score: 20.0, confidence: low] Devin (toolmention): text mentions Devin + [score: 20.0, confidence: low] Cline (toolmention): text mentions Cline + [score: 20.0, confidence: low] Aider (toolmention): text mentions Aider +Commit 496ebded2c69 (score: 20.0, confidence: low) + [score: 20.0, confidence: low] Kimi (toolmention): text mentions Kimi + [score: 20.0, confidence: low] GLM-4 (toolmention): text mentions GLM-4 +Commit f4f9781121c8 (score: 20.0, confidence: low) + [score: 20.0, confidence: low] Replit (toolmention): text mentions Replit +Commit 1d7b738fedb3 (score: 20.0, confidence: low) + [score: 20.0, confidence: low] t3.chat (toolmention): text mentions t3.chat +Commit 497c2fb26185 (score: 20.0, confidence: low) + [score: 20.0, confidence: low] Replit (toolmention): text mentions Replit +Commit 2082e559760d (score: 20.0, confidence: low) + [score: 20.0, confidence: low] Cursor (toolmention): text mentions Cursor + [score: 20.0, confidence: low] Aider (toolmention): text mentions Aider + [score: 20.0, confidence: low] Claude Code (toolmention): text mentions Claude Code +Commit 7c1dd7d8eed9 (score: 20.0, confidence: low) + [score: 20.0, confidence: low] Aider (toolmention): text mentions Aider + [score: 20.0, confidence: low] Claude (toolmention): text mentions Claude +Commit b90a1f0530a4 (score: 20.0, confidence: low) + [score: 20.0, confidence: low] Replit (toolmention): text mentions Replit +Commit 938740b59216 (score: 20.0, confidence: low) + [score: 20.0, confidence: low] t3.chat (toolmention): text mentions t3.chat ``` +In above output, the `low` confidence findings indicate mentions of AI tools in +commit messages during development. + #### Example 2: disclosure text command -For a file `file.txt` with checkboxes: +For a file `file.txt`: -``` +```text Claude and Chatgpt were used for the code, while Copilot was used for reviews, and some other AI tools may have been used for documentation. ``` @@ -77,19 +120,23 @@ and some other AI tools may have been used for documentation. When run through `disclosure text`, it produces findings that confirm AI use: ```sh -$ disclosure text --input=pr-body.txt --format=text --enable-checkbox-detection +$ disclosure text --input=file.txt --format=text Found 3 AI signal(s): Score: 20.0, Confidence: low [score: 20.0, confidence: low] Claude (toolmention): text mentions Claude - [score: 20.0, confidence: low] Chatgpt (toolmention): text mentions Chatgpt + [score: 20.0, confidence: low] ChatGPT (toolmention): text mentions ChatGPT [score: 20.0, confidence: low] Copilot (toolmention): text mentions Copilot ``` +In the above output, `low` confidence indicates that the specified tools are mentioned in the text +body. Given that the tools are simply mentioned, further research is warranted to understand the +extent to which the those tools were in the given context of the specified text. + #### Example 3: disclosure text command with checkbox detection enabled For a file `pr-body.txt` with checkboxes: -``` +```text Generative AI disclosure Please select one option: @@ -106,8 +153,13 @@ If AI tools were used, please provide details below: When run through `disclosure text`, it produces findings that confirm AI use: ```sh -$ disclosure text --input=pr-body.txt --format=text --enable-checkbox-detection +$ disclosure text --input=pr-body.txt \ + --format=text --enable-checkbox-detection \ + --cb-disclosed-ai="This PR uses AI/LLMs" --cb-disclosed-noai="This PR does not AI/LLMs" Found 1 AI signal(s): Score: 95.0, Confidence: high [score: 95.0, confidence: high] Claude (toolmention): checkbox confirms AI was used and text mentions Claude ``` + +In above output, a `high` confidence indicates that the user disclosed AI use by ticking the +`This PR uses AI/LLMs` checkbox, with Claude as the mentioned tool.