Skip to content

perf: index the Clover file lookup instead of rescanning per source path - #618

Merged
Bachmann1234 merged 2 commits into
Bachmann1234:mainfrom
Eljees:perf/clover-file-index
Aug 16, 2026
Merged

perf: index the Clover file lookup instead of rescanning per source path#618
Bachmann1234 merged 2 commits into
Bachmann1234:mainfrom
Eljees:perf/clover-file-index

Conversation

@Eljees

@Eljees Eljees commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Follow-up to your review on #617 — the second half of what I said I'd send there.

The problem

get_src_path_line_nodes_clover walks xml_document.findall(".//file") on every
call, and it is called once per source path. So a run over N source paths against a
report containing M files does NM element visits and, more expensively, NM
GitPathTool.relative_path calls, each of which normalises a path.

The format detection you flagged was the other half of the same problem and went in
with #617; this is the file lookup itself.

The fix

Build the tables once per report, on first use:

  • by_relative_path — repository-relative path to <file> elements;
  • by_last_segment — final path segment to the candidates for the suffix test.

The match here isn't equality: reports carrying absolute paths resolve through
normalized_file_path.endswith(f"/{src_path}"), which is what makes the
no-path-attribute PHPUnit reports resolve at all. A suffix can only match when the
last path segment matches, so grouping candidates by that segment keeps the fallback
O(few) rather than O(all files), and the common case O(1).

Both tables store each element's document position, so matches come back in document
order exactly as the linear walk produced them.

Checks

  • test_file_lookup_is_built_once counts GitPathTool.relative_path calls: 3 files
    against 3 source paths made 9 calls before and 3 after. Reverting only the
    source file fails it with 9 == 3, so the test pins this change rather than passing
    incidentally.
  • test_absolute_paths_still_match_by_suffix covers two files that share a last
    segment under different directories, plus a path that must not match. It passes both
    before and after — that is the point, the matching semantics are unchanged.
  • Full suite: 382 passed before, 384 after, no failures.
  • black --check ., isort --check . and doc8 README.rst --ignore D001 are clean.
  • diff-quality --violations pylint and --violations flake8 over this diff:
    100%, 0 violations across 95 changed lines.

Two things worth flagging

get_src_path_line_nodes_clover becomes an instance method taking the report index,
matching get_src_path_line_nodes_cobertura. It reads like a public name, so if you'd
rather keep a static shim for compatibility, say so and I'll add one.

This also takes the module from 979 to 1019 lines, so pylint's too-many-lines (limit
1000) now fires at module level. It's advisory in CI and doesn't land on a changed
line, but if you'd prefer, the Clover helpers are a clean candidate for their own
module and I'm happy to do that instead.

`get_src_path_line_nodes_clover` walked every `.//file` in the report on
every call, and it is called once per source path, so N source paths against
a report of M files did N*M element visits and N*M `relative_path` calls.

Build the lookup once per report: a map from repository-relative path, plus
a map from the last path segment for the suffix match that absolute-path
reports rely on. Both store document position, so matches are still returned
in document order.

Follow-up to the review on Bachmann1234#617.
@Bachmann1234

Copy link
Copy Markdown
Owner

This is looking good, but yeah, if you can cleanly separate the clover helpers that would be nice

Adding the index in this branch took violations_reporter.py to 1019 lines,
past pylint's too-many-lines limit of 1000. The lookup is self-contained, so
it moves to violationsreporters/clover.py as CloverFileIndex, and the module
is back to 958 lines.

XmlCoverageReporter keeps the per-report cache and builds the index on first
use, so the dispatch in _cache_file and the signature of
get_src_path_line_nodes_clover are unchanged.

The moved code resolves GitPathTool in its own module, so the Clover tests
patch it there. Skipping a <file> element that carries neither path nor name
had no test of its own; it has one now.

Requested in review on Bachmann1234#618.
@Eljees

Eljees commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

Done in 90823f0. The lookup now lives in diff_cover/violationsreporters/clover.py as CloverFileIndex, and violations_reporter.py is back to 958 lines, under the 1000 that this branch had pushed it past.

XmlCoverageReporter keeps the per-report cache and builds the index on first use, so _cache_file and the signature of get_src_path_line_nodes_clover are unchanged.

The moved code resolves GitPathTool in its own module, so the Clover tests patch it there. One branch came across without a test of its own — a <file> element carrying neither path nor name — so I added test_file_without_a_path_is_skipped; it fails with a TypeError if the guard is dropped. 385 tests pass, black --check, isort --check and doc8 are clean, and diff-cover over the diff reports 100%.

@Bachmann1234

Copy link
Copy Markdown
Owner

Thanks! Ill release this shortly

@Bachmann1234
Bachmann1234 merged commit 46e8b53 into Bachmann1234:main Aug 16, 2026
14 checks passed
@Bachmann1234

Copy link
Copy Markdown
Owner

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants