perf: index the Clover file lookup instead of rescanning per source path - #618
Conversation
`get_src_path_line_nodes_clover` walked every `.//file` in the report on every call, and it is called once per source path, so N source paths against a report of M files did N*M element visits and N*M `relative_path` calls. Build the lookup once per report: a map from repository-relative path, plus a map from the last path segment for the suffix match that absolute-path reports rely on. Both store document position, so matches are still returned in document order. Follow-up to the review on Bachmann1234#617.
|
This is looking good, but yeah, if you can cleanly separate the clover helpers that would be nice |
Adding the index in this branch took violations_reporter.py to 1019 lines, past pylint's too-many-lines limit of 1000. The lookup is self-contained, so it moves to violationsreporters/clover.py as CloverFileIndex, and the module is back to 958 lines. XmlCoverageReporter keeps the per-report cache and builds the index on first use, so the dispatch in _cache_file and the signature of get_src_path_line_nodes_clover are unchanged. The moved code resolves GitPathTool in its own module, so the Clover tests patch it there. Skipping a <file> element that carries neither path nor name had no test of its own; it has one now. Requested in review on Bachmann1234#618.
|
Done in 90823f0. The lookup now lives in
The moved code resolves |
|
Thanks! Ill release this shortly |
Follow-up to your review on #617 — the second half of what I said I'd send there.
The problem
get_src_path_line_nodes_cloverwalksxml_document.findall(".//file")on everycall, and it is called once per source path. So a run over N source paths against a
report containing M files does NM element visits and, more expensively, NM
GitPathTool.relative_pathcalls, each of which normalises a path.The format detection you flagged was the other half of the same problem and went in
with #617; this is the file lookup itself.
The fix
Build the tables once per report, on first use:
by_relative_path— repository-relative path to<file>elements;by_last_segment— final path segment to the candidates for the suffix test.The match here isn't equality: reports carrying absolute paths resolve through
normalized_file_path.endswith(f"/{src_path}"), which is what makes theno-
path-attribute PHPUnit reports resolve at all. A suffix can only match when thelast path segment matches, so grouping candidates by that segment keeps the fallback
O(few) rather than O(all files), and the common case O(1).
Both tables store each element's document position, so matches come back in document
order exactly as the linear walk produced them.
Checks
test_file_lookup_is_built_oncecountsGitPathTool.relative_pathcalls: 3 filesagainst 3 source paths made 9 calls before and 3 after. Reverting only the
source file fails it with
9 == 3, so the test pins this change rather than passingincidentally.
test_absolute_paths_still_match_by_suffixcovers two files that share a lastsegment under different directories, plus a path that must not match. It passes both
before and after — that is the point, the matching semantics are unchanged.
black --check .,isort --check .anddoc8 README.rst --ignore D001are clean.diff-quality --violations pylintand--violations flake8over this diff:100%, 0 violations across 95 changed lines.
Two things worth flagging
get_src_path_line_nodes_cloverbecomes an instance method taking the report index,matching
get_src_path_line_nodes_cobertura. It reads like a public name, so if you'drather keep a static shim for compatibility, say so and I'll add one.
This also takes the module from 979 to 1019 lines, so pylint's
too-many-lines(limit1000) now fires at module level. It's advisory in CI and doesn't land on a changed
line, but if you'd prefer, the Clover helpers are a clean candidate for their own
module and I'm happy to do that instead.