Add ai use checkbox detection with supporting tests

[omkar-foss]

Description
Add ai use checkbox detection with supporting tests.

This PR fixes #55.

Notes for Reviewers

Because this has some scoring related functionality, this PR has been branched out of #65. So this should be reviewed and merged only after #65 is merged.

Signed commits

• Yes, I signed my commits.

Generative AI disclosure

Please select one option:

• This contribution was NOT assisted or created by Generative AI tools.
• This contribution was assisted or created by Generative AI tools.

If AI tools were used, please provide details below:
- What tools were used?
- How were these tools used?
- Did you review these outputs before submitting this PR?

[Copilot]

Pull request overview

Adds checkbox-based AI-use detection (aligned with Issue #55) and extends the scanning pipeline to produce numeric scores and confidence levels per finding/commit/report, with CLI configuration and expanded test coverage.

Changes:

• Add PR-body checkbox detection to the tool-mention detector, including configurable checkbox labels.
• Introduce numeric scoring + confidence mapping across detectors, exposing per-detector scores and aggregate scores in scan results and output formats.
• Extend CLI flags (confidence mapping overrides, checkbox-label flags) and update/expand tests accordingly.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
scan/scan.go	Enrich commit/report results with score, confidence, and per-detector scores.
scan/scan_test.go	Updates scan tests for new scoring/confidence behavior and additional detectors.
output/output.go	Include score/confidence in text and JSON outputs; handle short hashes safely.
output/output_test.go	Adds/updates output format tests for new score/confidence fields and edge cases.
detection/trailer/trailer.go	Convert trailer detection to score-based findings and derived confidence.
detection/trailer/trailer_test.go	Update trailer tests to assert scores/confidence.
detection/toolmention/toolmention.go	Add checkbox-based AI-use confirmation, custom labels, scoring, and confidence.
detection/toolmention/toolmention_test.go	Add checkbox-focused test cases and scoring/confidence expectations.
detection/gitnotes/gitnotes.go	Add score + derived confidence to git-notes findings.
detection/gitnotes/gitnotes_test.go	Add helper assertions for score/confidence/detector metadata.
detection/detection.go	Add confidence “none”, JSON (un)marshal, score aggregation, and confidence mapping utilities.
detection/detection_test.go	Add unit tests for score consolidation, confidence mapping, and config parsing.
detection/constants.go	Add default checkbox labels and numeric scoring constants.
detection/committer/committer.go	Convert committer detection to score-based findings and derived confidence.
detection/committer/committer_test.go	Update committer tests for scoring/confidence metadata.
detection/branchname/branchname.go	Add score + derived confidence for branch-name detection.
detection/branchname/branchname_test.go	Update branch-name tests for scoring/confidence metadata.
cmd/cmd.go	Add confidence-level override parsing, propagate detector config, and add checkbox label flags to text.
cmd/cmd_test.go	Add/expand CLI tests for new flags, formats, and JSON outputs.

Suppressed comments (1)

detection/toolmention/toolmention_test.go:469

• This table-driven test defines wantScore but never asserts f.Score for each finding, so score regressions in toolmention detection won’t be caught.

			gotTools := make([]string, len(findings))
			for i, f := range findings {
				gotTools[i] = f.Tool

				if f.Confidence != tt.wantConfidence {
					t.Errorf(
						"confidence = %d, want %d", f.Confidence, tt.wantConfidence,
					)
				}
				if f.Detector != "toolmention" {
					t.Errorf("detector = %q, want %q", f.Detector, "toolmention")
				}

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[omkar-foss]

@andrew it'd be great if we finish up with #65 before getting into this PR, as this one depends on #65. I've mentioned this in pr description above for your reference.

[andrew]

• Match the GitHub task-list syntax used by the repository template (- [x] ...) and add a regression test using the actual template line.
• Expose checkbox labels as GitHub Action inputs and pass them to disclosure text, as requested in #55.
• Make the CLI help examples use the registered --check-label-ai-used and --check-label-ai-not-used flag names.

[andrew]

• detection/toolmention/toolmention.go: Handle CRLF input when matching checkbox lines. The current [ \t]*$ suffix leaves \r unmatched on non-final lines. GitHub returns PR bodies with CRLF endings, so a checked checkbox followed by more content produces no checkbox finding. Normalize line endings or accept an optional \r, and add a regression test.

• action/action.yml: Expose the checkbox labels as action inputs and forward them to disclosure text. The action currently cannot configure checkbox text for repositories with custom PR templates, so the earlier review request and issue requirement remain unresolved.

[omkar-foss]

• detection/toolmention/toolmention.go: Handle CRLF input when matching checkbox lines. The current [ \t]*$ suffix leaves \r unmatched on non-final lines. GitHub returns PR bodies with CRLF endings, so a checked checkbox followed by more content produces no checkbox finding. Normalize line endings or accept an optional \r, and add a regression test.

Done

* `action/action.yml`: Expose the checkbox labels as action inputs and forward them to `disclosure text`. The action currently cannot configure checkbox text for repositories with custom PR templates, so the earlier review request and issue requirement remain unresolved.

It'd be nice if we do this in a separate PR as there might be some iterations on this.

[andrew]

• detection/toolmention/toolmention.go: Strip HTML comments before tool matching or emit the checkbox as a separate finding. With the repository PR template unchanged apart from checking the AI-used box, the hidden instructional examples produce high-confidence ChatGPT and Claude findings. Add a regression test using the full template.

• action/action.yml: Expose the checkbox labels as action inputs and forward them to disclosure text. Custom PR templates still cannot configure this feature through the action. If this work moves to another PR, remove Fixes #55 and track the remaining requirement separately.

• cmd/cmd_test.go: Fix TestScanCommandConfidenceLevelsValid. It expects ExitError for valid mappings and passes because the package working directory is not a Git repository, so it does not test successful parsing. Pass a test repository and expect ExitNoAI or ExitAI, or remove this unrelated test change.

[omkar-foss]

• detection/toolmention/toolmention.go: Strip HTML comments before tool matching or emit the checkbox as a separate finding. With the repository PR template unchanged apart from checking the AI-used box, the hidden instructional examples produce high-confidence ChatGPT and Claude findings. Add a regression test using the full template.

I think we'll keep my initial plan of adding annotation (html) comments separate from this issue, in here let's go forward with the implementation outlined here. Also see my comment here for marking annotation comments for later (separate issue may be).

* `action/action.yml`: Expose the checkbox labels as action inputs and forward them to `disclosure text`. Custom PR templates still cannot configure this feature through the action. If this work moves to another PR, remove `Fixes #55` and track the remaining requirement separately.

Done, I've updated the action to add the checkbox labels.

* `cmd/cmd_test.go`: Fix `TestScanCommandConfidenceLevelsValid`. It expects `ExitError` for valid mappings and passes because the package working directory is not a Git repository, so it does not test successful parsing. Pass a test repository and expect `ExitNoAI` or `ExitAI`, or remove this unrelated test change.

I've simplified those tests now, this is done.

[omkar-foss]

I'm testing out the checkbox labels in this PR, ticking the checkbox above to AI used, let's see if it gives us AI_DETECTED=true in the workflow output.

[omkar-foss]

I'm testing out the checkbox labels in this PR, ticking the checkbox above to AI used, let's see if it gives us AI_DETECTED=true in the workflow output.

Yep, it detects both chatgpt and claude from the annotation comments after ticking the checkbox above, see: https://github.com/chaoss/disclosure/actions/runs/31892725137/job/95034821686?pr=81#step:3:9643

I'll push the annotation comment stripping, apart from that everything works well.

[omkar-foss]

I'll push the annotation comment stripping, apart from that everything works well.

I checked this thoroughly and the problem is that if we strip the annotation (html) comments, it also changes the behaviour of the toolmention detector e.g. if there's an annotation comment in free text like a pr template file on which disclosure is run on, it will strip the comment and result in a false negative.

So best way to handle this seems to retain the toolmention findings for the annotation comment and let the user handle it by adjusting the confidence level to medium, this is more robust as this will work well for other things too (like code comments in pr description etc). I've added a comment in the action file for user reference which should help configure it.

[MoralCode]

Strip HTML comments before tool matching or emit the checkbox as a separate finding. With the repository PR template unchanged apart from checking the AI-used box, the hidden instructional examples produce high-confidence ChatGPT and Claude findings. Add a regression test using the full template.

The algorithm i proposed should not have this problem.

IMO this checkbox detection should work by checking for the provided string (via the input flag), searching for an exact (or maybe case insensitive exact) match in the provided PR description, grabbing just the one line, looking for the presence of the github checkmark syntax (checked or unchecked) and then deciding whether to report a result based on whether it is checked. if any of that fails partway through (i.e. no checkbox detected), then ignore that line.

this feedback from andrew feels like we are very much mixing the text and checkbox detections here in a way thats creating extra complexity