Skip to content

About

HTTP service that validates QTI 3 items, tests and packages against the 1EdTech schemas and rules

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

QTI 3 Validator

CI Release Docker pulls License: MIT

An HTTP service that checks QTI 3.0 and 3.0.1 items, tests and complete content packages against the official 1EdTech schemas and rules. Send it a document or a ZIP package; it tells you whether it is valid and, if not, exactly what is wrong and where.

  • Complete checks: well-formed XML, the QTI 3 XML Schemas (XSD), the 7,400+ Schematron rules 1EdTech publishes alongside them, and additional checks on how the parts of an item fit together.
  • Precise errors: line, column and element path for every problem.
  • Small and fast: a ~21 MB image that uses about 30 MB of memory when idle, is ready in just over a second and validates a typical item in well under a millisecond.
  • Self-contained: the schemas are built in. The service never contacts the internet and never stores or logs the documents you send.

Quick start

Start the service:

docker run --rm -p 8080:8080 ghcr.io/feyst/qti3-validator

Validate a package:

curl -X POST http://localhost:8080/api/validate \
  -H 'Content-Type: application/zip' \
  --data-binary @package.zip

Or a single item:

curl -X POST http://localhost:8080/api/validate \
  -H 'Content-Type: application/xml' \
  --data-binary @item.xml

The answer is a validation report. summary.outcome is the verdict; see Outcomes:

{
  "summary": {
    "outcome": "VALID",
    "fatals": 0,
    "errors": 0,
    "warnings": 0,
    "exceptions": 0,
    "notRun": 0,
    "totalRun": 3,
    "valid": 1
  },
  "errors": [],
  …
}

To see what findings look like, validate the example package with a problem of each kind, examples/with-errors.zip.

What is checked

Every document goes through four checks:

  1. Well-formedness: the document must be well-formed XML in UTF-8.
  2. XML Schema: the document must conform to the QTI XSDs of its version, including what QTI imports, such as MathML 3, SSML 1.1 and IEEE LOM.
  3. Schematron rules: the rules 1EdTech embeds in those schemas, for what an XSD cannot express, such as allowed attributes, max-choices against min-choices and ARIA roles.
  4. Additional checks: whether the parts of an item, test or package fit together, for example that every interaction is bound to a declared response variable, that the files a package refers to are in it, and that values and expressions have the right types.

A package is checked file by file. It needs a QTI 3 imsmanifest.xml in its root; every XML file is validated, media files are not opened, and XML files that are not a QTI document are reported as not run, unless the manifest lists them as a QTI resource. A QTI fragment (XML that another document includes with XInclude, whether or not the manifest lists it as a resource of type imsqti_fragment_xmlv3p0) is checked as a fragment; see the report.

Accepted documents are QTI items, tests, sections, stimuli, response processing templates, outcome declarations, package manifests, LOM and QTI metadata. Others, including QTI 2.x, are rejected with unsupported_document, unless a custom validator adds them.

QTI versions

QTI 3.0 (3.0.0) and 3.0.1 are supported, each with its own official schemas and rules. Both use the same XML namespace, so the service picks the version like this:

Situation Version used
?version=3.0.0 or ?version=3.0.1 in the request That version, for every document
A package, or a manifest on its own The manifest's <schemaversion>
Any other document The latest version, 3.0.1

The report says which version was used, in specification.version. To check 3.0 content against 3.0.1, pass ?version=3.0.1: the manifest's <schemaversion>3.0.0</schemaversion> then gives a version_overridden warning instead of an error.

API

Method Path Purpose
POST /api/validate Validate a package or a single XML document
GET /api/validators List the validatorId values
GET /health Liveness and readiness check
GET /version Service and QTI versions

POST /api/validate

Send a ZIP package or a single XML document as the request body. The service recognises which it is; a Content-Type is optional:

Content-Type Validated as
application/zip A package
application/xml, text/xml A single document
none, or any other A package if the body is a ZIP, otherwise a single document

Query parameters, all optional:

Parameter Meaning
version 3.0.0 or 3.0.1: the QTI version every document is validated against
name The name of the input in the report; default package.zip or document.xml
validatorId Qti30Inspector; see GET /api/validators

For example, to validate a 3.0 package against QTI 3.0.1:

curl -X POST 'http://localhost:8080/api/validate?version=3.0.1&name=toets.zip' \
  -H 'Content-Type: application/zip' \
  --data-binary @toets.zip

Outcomes

Every request that could be validated gets HTTP 200 and a report, also when the content is invalid; a validation that had to stop before the end gets HTTP 422 with the report so far. Each finding is listed under its outcome (errors, warnings, fatals, …), and summary.outcome is the most severe one:

Outcome When Valid?
VALID A document passed every check yes
WARNING Something to look at that does not break the item, such as an outcome that is never used yes
ERROR A document breaks the XML Schema or a rule, or a package misses its manifest no
FATAL Something could not be read at all: XML that is not well-formed, not UTF-8 or has a DOCTYPE, a document that is not QTI 3, or a body that is not a ZIP no
EXCEPTION The validation did not finish: it stopped because a document is too complex or took longer than REQUEST_TIMEOUT (code too_complex, HTTP 422), or the service itself failed (code internal, HTTP 500) no
NOT_RUN A check was skipped: an XML file that is not a QTI document, or the checks of a document that was not validated, such as after a FATAL does not count

In a package, every file gets its own findings, so one package can contain all of them. examples/with-errors.zip shows each kind. The validation report describes every field and code.

Status codes

A request that could not be validated at all gets an error status and a short JSON body:

{
  "code": "unsupported_version",
  "message": "version must be one of 3.0.0, 3.0.1"
}
HTTP Code When
400 unsupported_version version is not a supported version
400 unknown_validator validatorId is not listed by GET /api/validators
400 invalid_request The body could not be read
408 request_timeout The body did not arrive within REQUEST_TIMEOUT
413 too_large, too_many_files The request, package, or the XML in a package exceeds a limit, the request is larger than all of MAX_SPOOL_SIZE, or a document has more nodes than can be checked in MAX_VALIDATION_MEMORY; the message gives the numbers
422 too_complex Validation stopped before the end: a document's rules need more work than one document may take (a budget of XPath steps that grows with the size of the document, up to a fixed maximum), or validation took longer than REQUEST_TIMEOUT; the body is a report with outcome EXCEPTION and the findings so far
500 internal The service failed; the body is a report with outcome EXCEPTION
503 busy No room for the request now: no validation slot or validation memory came free within REQUEST_TIMEOUT, MAX_QUEUE requests are already waiting, or the uploads in progress fill MAX_SPOOL_SIZE and leave no room for the next bytes of this one; retry after the seconds in Retry-After

Configuration

All settings are environment variables. Sizes are in bytes.

Variable Default Description
ADDR :8080 Address and port to listen on
MAX_REQUEST_SIZE 10485760 (10 MiB) Largest single XML document
MAX_PACKAGE_SIZE 104857600 (100 MiB) Largest request body for a package
MAX_FILE_SIZE 10485760 (10 MiB) Largest uncompressed XML file inside a package
MAX_FILES 1000 Most entries a package may contain
MAX_UNCOMPRESSED_SIZE 268435456 (256 MiB) Largest total of uncompressed XML in one package
MAX_ERRORS 100 Most errors per document from the XML Schema and the rules; the rules' warnings have a limit of the same size. See what is limited
MAX_DEPTH 256 Deepest element nesting accepted
MAX_CONCURRENT number of CPUs Validations that run at the same time; further requests wait. The CPUs are those of the container's CPU limit, or GOMAXPROCS if you set it; the service itself runs with one thread more. One document under 256 KiB more runs beside them, on an express slot of its own, so large documents do not keep typical items waiting. A package never takes that slot, however small it is
MAX_QUEUE 4 × MAX_CONCURRENT Requests that may wait for a validation slot or for validation memory, as many small ones (under 256 KiB, taking up to 4 MiB) as large ones; further requests get 503 at once
MAX_SPOOL_SIZE 4 × MAX_PACKAGE_SIZE Request bodies held at once, counted by the bytes received: in /tmp, or in memory for a body under 256 KiB; an upload that does not fit gets 503 at once, one larger than all of it 413. Lowered automatically when /tmp is a tmpfs; see Temporary files
MAX_VALIDATION_MEMORY the memory limit less 72 MiB Memory the validations in progress may reserve together, in bytes. A validation reserves what its document takes, which its number of nodes decides rather than its size, and waits for room like for a slot, holding nothing while it waits; so the documents being validated fit in the container whatever MAX_CONCURRENT and MAX_QUEUE are. A share of it, 4 MiB per slot, is kept for typical items. Only a document that could never fit gets 413, with the numbers. Derived at startup from the memory limit of the container, or of a group above it; 1 GiB when there is none. See Memory and CPU
REQUEST_TIMEOUT 60s Time limit per request: receiving the body, waiting for a free slot and for validation memory, and validating. Running out while the body arrives gives 408; while waiting for a slot or memory, 503; a validation still running then stops with too_complex, 422 (the XML Schema check of the current document finishes first)
VALIDATORS_DIR /validators Directory with extra validators; see Custom validators

For example:

docker run --rm -p 8080:8080 \
  -e MAX_PACKAGE_SIZE=52428800 \
  -e MAX_CONCURRENT=4 \
  ghcr.io/feyst/qti3-validator

Running in production covers memory, uploads, public deployments, a read-only container, Docker Compose, Kubernetes and logging.

Limitations

  • Scoring: whether response processing gives the scores you intend is not checked; the additional checks only check that it can run, and that its values and expressions have the right types.
  • Encoding: documents must be UTF-8 (ASCII is fine). A DOCTYPE is not accepted.
  • SSML: two SSML restrictions are not enforced: version and xml:lang on <ssml:speak>, and name on <ssml:mark>.

Documentation

License

MIT. The patched copy of github.com/jacoelho/xsd in third_party/ and the fixtures from php-qti3 in testdata/ keep their own MIT licenses.

About

HTTP service that validates QTI 3 items, tests and packages against the 1EdTech schemas and rules

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages