Skip to content

fix(conformance): carry the schema the suite judges against - #12

Closed
glatinone wants to merge 1 commit into
feat/paginationfrom
fix/conformance-portable-spec
Closed

glatinone wants to merge 1 commit into
feat/paginationfrom
fix/conformance-portable-spec

Conversation

@glatinone

Copy link
Copy Markdown
Owner

fix(conformance): carry the schema the suite judges against

The default spec path was Path("spec/v0.1.0/memory-cell.schema.json"), relative
to the working directory. Run the suite the way it is meant to be run - against
somebody else's server, from wherever they happen to be standing:

amp-conformance --base-url https://their-server.example.com
FAIL schema: schema-spec-file -- spec not found at spec\v0.1.0\...; pass --spec

That is the suite failing to find its own fixture, reported in the same red as a
server failing a vector. A third party reading it sees a failure about a file they
have never heard of, naming neither their server nor their mistake. The goal of
this suite is to be runnable by implementations that are not this one, and a
directory-dependent fixture works against exactly that.

  • The normative schema now ships inside the wheel (a hatch force-include from
    spec/v0.1.0/, so the source of truth stays in one place and the copy is
    generated, not duplicated in the tree).
  • The default is resolved by searching the working directory, then each of its
    parents, then the packaged copy. Running from a checkout subdirectory now works
    too - that case was silently failing as well.
  • When no schema can be found, the schema vectors report skip with the flag to
    pass, and the summary counts them separately. --only schema exits non-zero,
    because asking for those vectors by name and getting a skip that exits 0 is the
    exact failure mode this suite exists to catch in other people's servers.
  • --spec is unchanged and still wins when given.

Two existing tests pinned the old behaviour; both are rewritten to assert the new
one rather than deleted - one of them existed precisely to check that a missing
spec was visible, and that requirement survives as "a skip is counted and named".

Verified: 19 conformance tests pass, the suite run from conformance/ now reports
14 schema vectors passing where it previously reported a failure, and the built
wheel contains amp_conformance/spec/memory-cell.schema.json.

The default spec path was `Path("spec/v0.1.0/memory-cell.schema.json")`, relative
to the working directory. Run the suite the way it is meant to be run - against
somebody else's server, from wherever they happen to be standing:

    amp-conformance --base-url https://their-server.example.com
    FAIL schema: schema-spec-file -- spec not found at spec\v0.1.0\...; pass --spec

That is the suite failing to find its own fixture, reported in the same red as a
server failing a vector. A third party reading it sees a failure about a file they
have never heard of, naming neither their server nor their mistake. The goal of
this suite is to be runnable by implementations that are not this one, and a
directory-dependent fixture works against exactly that.

- The normative schema now ships inside the wheel (a hatch force-include from
  `spec/v0.1.0/`, so the source of truth stays in one place and the copy is
  generated, not duplicated in the tree).
- The default is resolved by searching the working directory, then each of its
  parents, then the packaged copy. Running from a checkout subdirectory now works
  too - that case was silently failing as well.
- When no schema can be found, the schema vectors report `skip` with the flag to
  pass, and the summary counts them separately. `--only schema` exits non-zero,
  because asking for those vectors by name and getting a skip that exits 0 is the
  exact failure mode this suite exists to catch in other people's servers.
- `--spec` is unchanged and still wins when given.

Two existing tests pinned the old behaviour; both are rewritten to assert the new
one rather than deleted - one of them existed precisely to check that a missing
spec was visible, and that requirement survives as "a skip is counted and named".

Verified: 19 conformance tests pass, the suite run from `conformance/` now reports
14 schema vectors passing where it previously reported a failure, and the built
wheel contains `amp_conformance/spec/memory-cell.schema.json`.
@glatinone

Copy link
Copy Markdown
Owner Author

Landed on master in the v0.1.0 chain: the branch was fast-forward merged as part of b940905..92b88ee and released as v0.1.0. Closing so the open list matches reality - the commits are in master, and the tag points at them.

@glatinone glatinone closed this Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant