|
1 | 1 | ## Tests |
2 | 2 |
|
3 | | -To support the language-neutral testing of `purl` implementations, a test |
4 | | -suite is provided as JSON document named `test-suite-data.json`. This JSON |
5 | | -document contains an array of objects. Each object represents a test with |
6 | | -these key/value pairs some of which may not be normalized: |
7 | | - |
8 | | -- **purl**: a `purl` string. |
9 | | -- **canonical**: the same `purl` string in canonical, normalized form |
10 | | -- **type**: the `type` corresponding to this `purl`. |
11 | | -- **namespace**: the `namespace` corresponding to this `purl`. |
12 | | -- **name**: the `name` corresponding to this `purl`. |
13 | | -- **version**: the `version` corresponding to this `purl`. |
14 | | -- **qualifiers**: the `qualifiers` corresponding to this `purl` as an object |
15 | | - of {key: value} qualifier pairs. |
16 | | -- **subpath**: the `subpath` corresponding to this `purl`. |
17 | | -- **is_invalid**: a boolean flag set to true if the test should report an |
18 | | - error |
19 | | - |
20 | | -To test `purl` parsing and building, a tool can use this test suite and for |
21 | | -every listed test object, run these tests: |
22 | | - |
23 | | -- parsing the test canonical `purl` then re-building a `purl` from these |
24 | | - parsed components should return the test canonical `purl` |
25 | | - |
26 | | -- parsing the test `purl` should return the components parsed from the test |
27 | | - canonical `purl` |
28 | | - |
29 | | -- parsing the test `purl` then re-building a `purl` from these parsed |
30 | | - components should return the test canonical `purl` |
31 | | - |
32 | | -- building a `purl` from the test components should return the test canonical |
33 | | - `purl` |
| 3 | +The Package-URL (PURL) specification provides a JSON Schema and test |
| 4 | +files to support language-neutral testing of PURL implementations. The |
| 5 | +objectives for the PURL schema and test files are to enable tools to conform |
| 6 | +to the PURL specification for tool functions such as: |
| 7 | +- build a canonical PURL string from a set of PURL component data |
| 8 | +- parse a canonical PURL string into a set of PURL components |
| 9 | +- parse a PURL string input and rebuild it as a canonical PURL string |
| 10 | + |
| 11 | +The current JSON schema is available at: [`purl-spec/schemas/purl-test.schema-0.1.json`](https://github.com/package-url/purl-spec/blob/main/schemas/purl-type-definition.schema-1.0.json). |
| 12 | + |
| 13 | +The test files are available at: |
| 14 | +- [`purl-spec/tests/spec/`](https://github.com/package-url/purl-spec/tree/main/tests/spec): |
| 15 | +This folder contains JSON test files that are not for a specific PURL type. |
| 16 | + - [`specification-test.json`](https://github.com/package-url/purl-spec/blob/main/tests/spec/specification-test.json) - This file contains an array of test objects |
| 17 | + that primarily cover testing the validity of individual PURL components, |
| 18 | + separators between PURL components, and complete PURL strings. |
| 19 | + - There is a proposal to add separate test files in this folder for each |
| 20 | + PURL component. |
| 21 | +- [`purl-spec/tests/types/`](https://github.com/package-url/purl-spec/tree/main/tests/types): This folder contains one JSON test file for each registered PURL type. These |
| 22 | +tests should be focused on test cases that are specific to a PURL type, such |
| 23 | +as those for namespace or qualifiers. |
| 24 | + |
| 25 | +Two key properties in the PURL test JSON schema are: |
| 26 | +- Test groups |
| 27 | +- Test types |
| 28 | + |
| 29 | +### Test groups |
| 30 | + |
| 31 | +There are two PURL test groups: |
| 32 | +- **base**: Test group for base conformance tests. Base tests are pass/fail. |
| 33 | +- **advanced**: Test group for advanced tests. Advanced tests are more |
| 34 | +permissive than base tests. They may correct some errors. |
| 35 | + |
| 36 | +### Test types |
| 37 | + |
| 38 | +There are three PURL test types: |
| 39 | +- **build**: A test to build a canonical PURL output string from an input of |
| 40 | +decoded PURL components. See also [`/docs/how-build.md`](https://github.com/package-url/purl-spec/blob/main/docs/how-to-build.md). |
| 41 | +- **parse**: A test to parse decoded components from a canonical PURL |
| 42 | +input string. See also [`/docs/how-parse.md`](https://github.com/package-url/purl-spec/blob/main/docs/how-to-parse.md). |
| 43 | +- **roundtrip**: A test to parse an input PURL string and then rebuild it as a |
| 44 | + canonical PURL output string. |
| 45 | + |
| 46 | +### Test case basics |
| 47 | + |
| 48 | +#### Test case fields |
| 49 | +The fields for each test case are: |
| 50 | +- description: description of the test case purpose |
| 51 | +- test_group: base or advanced |
| 52 | +- test_type: build, parse, or roundtrip |
| 53 | +- input: PURL string or set of PURL components |
| 54 | +- expected_output: canonical PURL string, set of PURL components or null for |
| 55 | +an expected test failure |
| 56 | +- expected_failure: true or false |
| 57 | +- expected_failure_reason: description of the reason for the test case failure |
| 58 | + |
| 59 | +#### Error handling |
| 60 | +The standard error-handling behaviour for all test cases is based on two |
| 61 | +properties from the PURL test schema: |
| 62 | +``` |
| 63 | +"expected_failure": { |
| 64 | + "title": "Expected failure", |
| 65 | + "description": "true if this test input is expected to fail to be processed.", |
| 66 | + "type": "boolean", |
| 67 | + "default": false |
| 68 | + }, |
| 69 | + "expected_failure_reason": { |
| 70 | + "title": "Expected failure reason", |
| 71 | + "description": "The reason why this test is expected to fail if expected_failure is true.", |
| 72 | + "default": null, |
| 73 | + "type": [ |
| 74 | + "string", |
| 75 | + "null" |
| 76 | + ] |
| 77 | + } |
| 78 | +``` |
| 79 | +In the case of a test case failure the `expected_output` is null. |
| 80 | + |
| 81 | + |
| 82 | + |
| 83 | + |
| 84 | + |
0 commit comments