Skip to content

Add pks list-references to list the references between packs - #71

Merged
dduugg merged 2 commits into
mainfrom
list-references
Oct 12, 2026
Merged

dduugg merged 2 commits into
mainfrom
list-references

Conversation

@dduugg

@dduugg dduugg commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

pks list-references prints, as JSON, every reference from one pack to a constant that another pack defines, with the violations pks check finds in it. Each violation gives its type, whether it's strict, and whether a package_todo.yml records it. A reference that the packs' rules allow, such as one covered by a declared dependency or to a public constant, is listed with an empty violations list.

check reports only the references that break a rule, so today nothing shows how packs use each other where the rules allow it. Tools that read package_todo.yml files, such as query_packwerk, see only the recorded violations. This is the pks half of rubyatscale/query_packwerk#8. A References collection in query_packwerk would be built from this output.

$ pks list-references --defining-pack packs/bar
{"references":[{"file":"packs/foo/app/services/foo.rb","line":3,"column":4,"constant_name":"::Bar","referencing_pack_name":"packs/foo","defining_pack_name":"packs/bar","violations":[{"violation_type":"dependency","strict":false,"recorded":false},{"violation_type":"privacy","strict":false,"recorded":false}]}]}
  • --defining-pack and --referencing-pack limit the list to references to or from the packs they name. Each can be repeated. An unknown pack is an error, and a trailing slash is accepted, as in rm.
  • It exits 0 whether or not references have violations.
  • schema/list-references-output.json describes the output, as schema/check-output.json does for check -o json.

This PR doesn't bump the version. The entry goes under ## Unreleased.

Design notes

  • References within a pack, and to constants that no pack defines (a gem's, say), are left out.
  • Several definitions. The experimental parser can resolve a constant to more than one file in a pack, and then the reference has a copy for each file. The copies are listed as one reference with every violation any of them makes, which is what check reports. Taking one copy's violations would drop a privacy violation whenever that copy's file was public, and which copy comes first varies between runs.
  • ERB. pks keeps no source positions in ERB and reports line 0 there, so an ERB reference has a null line and column, and an ERB file's references to one constant are listed once. check -o json reports these as "line": 0, which its own schema forbids (minimum: 1). That's left for a follow-up, along with keeping positions through the ERB-to-Ruby conversion.
  • Output goes through a BufWriter, since a large app produces tens of MB of JSON.
  • The sort and the checker loop run in parallel, as check's checkers do. par_chunk_by needs rayon 1.10, so Cargo.toml's requirement goes up from 1.7; Cargo.lock isn't tracked, so nothing else guarantees it.
  • get_checkers becomes pub(crate) so the new module can run the checkers itself. check_all and pks rm's reference search are unchanged.

Who this affects

  • pks list-references is new. Nothing else changes.

Test plan

  • tests/list_references_test.rs checks every output against the schema. It covers: the full output for simple_app; strict and recorded flags in every combination (uses_strict_mode_partially_recorded, contains_package_todo); every violation type; both filters, combined, and --defining-pack repeated and with a trailing slash; an unknown pack for each filter; leaving out a reference within a pack; several uses of a constant in one file, and one at line 1, column 0; the order of files; ERB, with two constants and two files; and, under the experimental parser, a constant with a public and a private definition, and one that two packs define.
  • Each of these makes a test fail on every run: keeping references within a pack, not merging copies, reporting ERB's line 0, leaving out the sort, leaving any field out of the key that merges copies, or taking only one copy's violations.
  • On a large Rails monolith, without a cache, it takes about as long and as much memory as pks check. Its unrecorded violations match check -o json's violations exactly, line and column included. Every reference matches the schema, and none repeats.
  • cargo test, cargo fmt --check, and cargo clippy --all-targets --all-features with -Dwarnings. Locally, test_gitignore_negation_patterns fails on main too when the global gitignore has *.log, and it passes without one.

`pks check` reports only the references that break a rule. Nothing
shows how packs use each other's constants where the rules allow it,
as through a declared dependency or a public constant, and tools that
read package_todo.yml files, such as query_packwerk, see only recorded
violations (rubyatscale/query_packwerk#8).

`pks list-references` prints, as JSON, every reference from one pack to
a constant that another pack defines, with the violations `check` finds
in it. Each violation gives its type, whether it is strict, and whether
a package_todo.yml records it, so an allowed reference lists none. It
exits 0 either way. `--defining-pack` and `--referencing-pack` limit the
list to references to or from the packs they name, and can be repeated.
An unknown pack is an error, as in `rm`.

References within a pack, and to constants that no pack defines, are
left out. A constant defined in several files of a pack, which only the
experimental parser resolves to more than one definition, yields a copy
of the reference for each file. The copies are listed as one reference
with every violation any of them makes, as `check` reports them. Taking
one copy's violations would drop a privacy violation whenever that
copy's file was public, and which copy comes first varies between runs.

pks keeps no source positions in ERB and reports line 0 there, so an
ERB reference has a null line and column, and an ERB file's references
to one constant are listed once. schema/list-references-output.json
describes the output, and the tests check each output against it.
The schema now says what a consumer needs to rely on the output:

- References are sorted by file, line, column, constant and defining
  pack, and a constant that two packs define gives a reference to each
  at the same place.
- `violations` lists recorded violations as well, which `check` leaves
  out, in checker order. An empty list means the reference breaks no
  enforced rule, which includes a rule that isn't enforced.
- `strict` is the strictness of the pack that enforces the rule: the
  referencing pack for dependency and layer, the defining pack for the
  rest.
- `column` counts bytes, and it is null exactly when `line` is. ERB's
  null is `check -o json`'s 0.

The sort and the checker loop run in parallel, as `check`'s checkers do.
On a synthetic app with 1M references, the command took 5.4 s instead of
7.2 s, and its output was byte for byte the same. Copies of a reference
give the same output in any order, so an unstable sort is enough.
`par_chunk_by` needs rayon 1.10, and since Cargo.lock isn't tracked,
Cargo.toml's requirement is what guarantees it.

Dropping `line`, `column`, `constant_name` or `defining_pack_name` from
`identity()` left every test passing, since no fixture had two
references that differ only in one of them. New tests cover each, along
with line 1 and column 0, the order of files, a violation recorded but
not strict, and the visibility, layer and folder_privacy types.

The several-definitions test caught a regression to one copy's
violations only when that copy came first. Now `packs/foo` ignores
dependencies on the private copy's folder, so each copy makes a
violation the other doesn't, and taking either copy alone fails every
run.

The tests drop `RUST_LOG`, which makes pks log to stderr, since they
expect stderr to be empty.
@dduugg
dduugg merged commit d13f6db into main Oct 12, 2026
16 checks passed
@dduugg
dduugg deleted the list-references branch October 12, 2026 19:35
@github-project-automation github-project-automation Bot moved this from Triage to Done in Modularity Oct 12, 2026
dduugg added a commit that referenced this pull request Oct 12, 2026
#71 needed `rayon`'s requirement raised from 1.7.0 to 1.10.0 by hand,
for `par_chunk_by`. Nothing else raises a requirement, so a new API is
found missing only when the build breaks. This takes the bundler and
github-actions setup from rubyatscale/packs-specification: monthly, all
updates in one group per ecosystem, with a 7-day cooldown.

Cargo.lock isn't tracked, and Dependabot's default strategy leaves a
requirement alone when it already allows the new release, as `1.7.0`
allows 1.12. `versioning-strategy: increase` raises the requirement to
the newest release instead, so Cargo.toml keeps up with what CI builds.

dist generates release.yml and pins its actions in dist-workspace.toml,
so the actions update leaves that file out.
dduugg added a commit that referenced this pull request Oct 12, 2026
…7.0 (#72)

* Print `--print-files` progress to stderr

`-p` printed "Started processing" and "Finished processing" lines to
stdout, so the JSON from `check -o json` and `list-references` came
after them and didn't parse. They are progress, not output, so they now
go to stderr, where pks's warnings already go.

* Read `<%==` tags as Ruby in ERB

Rails' `<%== %>` outputs without escaping, but the regex that pulls Ruby
out of ERB took only one `=` after `<%`, so `<%== foo(Bar) %>` became
`= foo(Bar)`. That doesn't parse. lib-ruby-parser recovers only part of
the file, and the tag's references and every reference after it were
missed: with `<%= SomeConcern %>`, `<%== Bar.bar %>` and `<%= Baz %>`,
pks found only `::SomeConcern`.

In three large Rails apps, these were the only files that failed to
parse. Packwerk reads ERB with a real parser and handles the tag.

* Warn about files that don't parse

When lib-ruby-parser couldn't parse a file, pks used whatever partial
tree it recovered, or none, and said nothing, so `check` could pass
while missing the file's references. Packwerk reports a syntax error as
an offense. pks now prints a warning to stderr with the file and the
first error, and its line in Ruby. ERB gets no line, since its lines are
those of the Ruby pks makes from it.

The error rides on `ProcessedFile` as `syntax_error`, which isn't
serialized, so the cache format that pks shares with packwerk doesn't
change. A file pks warns about isn't written to the cache, so the
warning comes back on every run until the file is fixed, instead of
only when the file is first parsed.

Only the app's own files, its included files, warn.
`expose-monkey-patches` also parses Ruby's and the gems' sources, which
can use syntax pks can't parse and aren't the app's to fix; they stay
quiet and cached.

An empty file has no AST but no error either, so it doesn't warn. Three
large Rails apps give no warnings under either parser, now that `<%==`
is read correctly. The parser can't read some newer syntax, such as
Ruby 3.2's anonymous argument forwarding, and those files warn too.

* Read only the referencing packs' files in `list-references`

`--referencing-pack` filtered references after pks had parsed every
included file, so asking about one pack cost as much as asking about
all of them. Only a pack's own files can make references from it, so
pks now passes just those to `get_all_references`, as `check <paths>`
passes its paths. The experimental parser still reads every file for
definitions inside `get_all_references`, so it gains less.

For two packs of a large Rails app, the output is the same as the full
list limited to those packs, under both parsers, and the run took 2.6 s
instead of 8.0 s with the default parser. A test checks that a file in
another pack isn't read: one that doesn't parse would make pks warn.

* Share test helpers, and keep `RUST_LOG` out of tests that check stderr

pks's logger reads `RUST_LOG` even without `--debug`, so a developer
with `RUST_LOG=debug` set saw every test that expects an empty stderr
fail: 7 in `rm_test.rs`, all of `list_references_test.rs` and 5 in
`check_test.rs`. CI doesn't set it, so only local runs were affected.

`rm_test.rs` and `list_references_test.rs` each had the same `pks()`
helper, and `list_references_test.rs` and `check_test.rs` each had a
schema validator. Both move to `tests/common`, where `pks()` removes
`RUST_LOG`; the new syntax error tests use it too. The `check_test.rs`
tests run against the fixtures in place rather than through `Fixture`,
so those five remove it themselves. With `RUST_LOG=debug`, the suite
now fails only on `test_gitignore_negation_patterns`, which depends on
the global gitignore rather than on logging.

* Share list-references' reference selection with `rm`

`rm`'s search for references into the pack it removes filtered, sorted
and deduplicated the references the way `list-references` does, with
its own copy of each step. `between_packs` now holds the filter and the
sort, and `rm` asks it for the references to one pack from the others,
then keeps one copy of each, as `list-references` merges them. It and
`identity` move to `reference_extractor`, which both commands already
use, rather than one command reaching into the other's module. The
order `rm` prints references in is unchanged, since `identity` only
adds the defining pack, which is the same for all of them.

Nothing tested that `rm` lists a reference once when several of the
pack's files define its constant, so a new test does, under the
experimental parser. It passes before and after this change and fails
without the deduplication.

* Have Dependabot update Rust dependencies and actions monthly

#71 needed `rayon`'s requirement raised from 1.7.0 to 1.10.0 by hand,
for `par_chunk_by`. Nothing else raises a requirement, so a new API is
found missing only when the build breaks. This takes the bundler and
github-actions setup from rubyatscale/packs-specification: monthly, all
updates in one group per ecosystem, with a 7-day cooldown.

Cargo.lock isn't tracked, and Dependabot's default strategy leaves a
requirement alone when it already allows the new release, as `1.7.0`
allows 1.12. `versioning-strategy: increase` raises the requirement to
the newest release instead, so Cargo.toml keeps up with what CI builds.

dist generates release.yml and pins its actions in dist-workspace.toml,
so the actions update leaves that file out.

* Bump version to 0.7.0

`pks list-references` is a new command, and pre-1.0 a new feature wants
a minor bump. Once this merges, auto-release.yml tags and releases
v0.7.0, since Cargo.toml's version has no tag yet.

Retitle `## Unreleased` to `## 0.7.0`, the heading dist takes the
release notes from.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant