docs: add real-world bash/ops-tooling repo example to Performance#747
Open
saddestmartian wants to merge 1 commit into
Open
docs: add real-world bash/ops-tooling repo example to Performance#747saddestmartian wants to merge 1 commit into
saddestmartian wants to merge 1 commit into
Conversation
The existing Performance section benchmarks large polyglot application codebases (Linux kernel, Django). Bash already scores in the Excellent parsing tier under Language Support, but there's no worked example of the agentic token-efficiency payoff on a repo shape that's mostly shell scripts, YAML/JSON config, and Markdown docs rather than a typical multi-language application. Adds one real session's measurement: a single search_code call vs a grep-based sub-agent fan-out on an internal dev-tooling repo, clearly labeled as one real-world data point rather than a controlled benchmark. Signed-off-by: saddestmartian Signed-off-by: saddestmartian <saddestmartian@gmail.com>
Owner
|
Hey, looks good. Can you remove the last line? Basically embedd the insights naturally :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a small "Real-World Example" subsection to the Performance section of the README.
The existing Performance benchmarks (Linux kernel, Django) are measured on large polyglot application codebases. Bash already scores in the "Excellent" parsing tier under Language Support, but there's no worked example of what that translates to for a different repo shape: infra/ops-tooling repos that are mostly shell scripts, YAML/JSON config, and Markdown docs rather than a typical multi-language application.
This adds one real session's measurement on an internal dev-tooling repo (bash + JSON + Markdown, ~14.9k indexed nodes / ~20.5k edges): a single
search_codecall reproducing a config-file consumer inventory in ~375ms vs. a grep-based sub-agent fan-out taking ~131s across 12 tool calls for the same result. It's explicitly labeled as one real-world data point, not a controlled multi-trial benchmark, to keep it honest alongside the hardware-benchmarked numbers above it.Docs-only change — no source touched.
Checklist
git commit -s) — required, CI rejectsunsigned commits (DCO, see CONTRIBUTING.md)
make -f Makefile.cbm test) — N/A, docs-only change, no C source touchedmake -f Makefile.cbm lint-ci) — N/A, docs-only change, no C source touched