Skip to content

Assert every manifest case id is unique and every fixture is reachable #68

Description

@kikashy

Goal

Add a repository test that holds conformance/manifest.json to two structural invariants: case ids are unique, and every fixture on disk is named by exactly one case.

Why

The manifest is the authoritative index of the document-conformance corpus, and the count statements around the repository are now derived from it. Two ways it can drift quietly:

  • a duplicated id — a copy-paste when adding a case. The suite would run one of them, or both under one name, and the count would still look right.
  • an orphaned fixture — a file under conformance/ that no case names. It is dead weight that looks like coverage, and a contributor adding a sibling case may reasonably assume it is exercised.

Neither is currently checked. The second is the more interesting one, because an unreferenced fixture is indistinguishable from a covered case when you are reading the directory.

Scope

  • Add a test to tests/test_repository.py.
  • Assert that case ids are unique, naming any duplicate.
  • Assert that every case's path exists.
  • Assert that every fixture file under the corpus directories is named by exactly one case, naming any that is not.
  • Use subTest so a failure identifies the id or the path rather than only the count.
  • Do not add, remove, or rename any fixture or case in this issue. If the test finds an orphan, report it in the pull request and let a maintainer decide.

Acceptance criteria

  • A duplicated id fails, and the message names it.
  • A case pointing at a missing file fails, and the message names it.
  • A fixture no case names fails, and the message names it.
  • Try each of the three locally before opening the pull request, so the test is known to bite rather than assumed to.
  • python -m unittest discover -s tests -v passes on the unmodified tree.
  • Every commit is signed off (git commit -s).

Contributor learning

Treating an index as an artifact with its own invariants, and the difference between a file existing and a file being exercised.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions