eQ Questionnaire Runner

Run with Podman

Install Podman from Self Service as the container runtime.

Make sure the Podman machine started every time you want to use container images:

podman machine start

The Makefile detects the container runtime automatically (Podman on arm64, Docker on amd64).

For convenience when typing container commands, add an alies to your shell profile:

alias docker='podman'

This will only affect interactive shells. Make targets use the Makefile's runtime detection instead.

To get eq-questionnaire-runner running the following command will build and run the containers

RUNNER_ENV_FILE=.development.env podman compose up -d

To launch a survey, navigate to http://localhost:8000/

When the containers are running you are able to access the application as normal, and code changes will be reflected in the running application. However, any new dependencies that are added would require a re-build.

To rebuild the eq-questionnaire-runner container, the following command can be used.

RUNNER_ENV_FILE=.development.env podman compose build

If you need to rebuild the container from scratch to re-load any dependencies then you can run the following

RUNNER_ENV_FILE=.development.env podman compose build --no-cache

Run locally

Clone the repository

git clone git@github.com:ONSdigital/eq-questionnaire-runner.git

Pre-Requisites

The following must be installed and working before you start:

• Miniconda: Python, node and system package management (install from source, see bellow)
• Podman: Container runtime for supporting services (machine created and running)
• wkhtmltopdf: PDF generation (installed separately, see below)
• gcloud: Pulling images from Google Artifact Registry

Example conda installation:

curl -L -o /tmp/miniconda.sh \
https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh

bash /tmp/miniconda.sh -b -p $HOME/miniconda3

$HOME/miniconda3/bin/conda init zsh

Verify

conda --version

If conda reports command not found, the installer did not write the conda block into ~/.zshrc Confirm the install is present and wire it in:

ls -d $HOME/miniconda3
$HOME/miniconda3/bin/conda init zsh

Open a new terminal tab and re-check conda --version

Python, Node.js, Poetry, snappy and jq are all provided by the conda environment created in environment.yml

Verify if the remaining prerequisites are available:

podman --version
gcloud --version
make --version

Make sure that the Podman machine is running:

podman machine list
podman machine start

wkhtmltopdf

Runner uses pdfkit, which calls wkhtmltopdf, to generate the download PDF.

In Docker

The Runner image runs on Debian 13, which no longer packages wkhtmltopdf. The Dockerfile installs it from Debian 12's apt in a separate build stage and copies the binary and the libraries it needs into the runtime image. This is a temporary workaround until wkhtmltopdf is replaced.

Locally on macOS

wkhtmltopdf is not reliably available on conda-forge for macOS ARM, so it is installed outside the conda environment. Download the macOS .pkg from the wkhtmltopdf downloads page and run the installer, or from the command line:

curl -fsSLO https://github.com/wkhtmltopdf/packaging/releases/download/0.12.6-2/wkhtmltox-0.12.6-2.macos-cocoa.pkg
sudo installer -pkg wkhtmltox-0.12.6-2.macos-cocoa.pkg -target /
wkhtmltopdf --version

Note that wkhtmltopdf is an archived project and no longer receives updates.

The macOS package is upstream's patched-Qt build, which uses an older WebKit engine that doesn't support CSS variables. The Design System relies on them, so PDFs generated locally on macOS won't be styled correctly. To check PDF styling, run Runner from its Docker image instead.

Setup

Application version

Create .application-version for local development

This file is automatically created and populated with the git revision id during CI for anything other than development, but the file is absent when the repo is first cloned and is required for running the app locally. Setting the contents to local removes the implication that any particular revision is used when run locally.

echo "local" > .application-version

Python version

It is preferable to use the version of Python locally that matches that used on deployment. This project has a .python-version file for this purpose. Both are read manually and declared in the conda environment file below, conda does not read them automatically.

cat .python-version
cat .nvmrc

Conda environment

Python and Node.js versions are pinned in the committed environment.yml, matching .python-version and .nvmrc as closely as conda-forge availability allows:

Note: conda-forge does not publish every Node patch release. Where the exact .nvmrc version is unavailable, pin the closest available patch below it and note the substitution in environment.yml.

If .python-version or .nvmrc change, or if you want to use another version of Python or Node, update environment.yml to match.

Create and activate the environment:

conda env create -f environment.yml
conda activate eq-runner

Version can be changed by editing environment.yml and running conda env update -f environment.yml --prune

If conda env create fails with NoWritablePkgsDirError or a permission error on the notices cache, run Repair ownership of user conda directory in Self Service, then retry.

Build flags for python-snappy

python-snappy compiles against libsnappy, which is provided by the conda environment. Add an activation hook so the compiler can find it:

mkdir -p "$CONDA_PREFIX/etc/conda/activate.d"
cat > "$CONDA_PREFIX/etc/conda/activate.d/build-flags.sh" <<'EOF'
export CPPFLAGS="-I${CONDA_PREFIX}/include ${CPPFLAGS}"
export LDFLAGS="-L${CONDA_PREFIX}/lib ${LDFLAGS}"
EOF
conda deactivate && conda activate eq-runner

These flags now apply automatically whenever the environment is active.

Dependencies

With the environment active, install the Python dependencies:

poetry install

Install the JavaScript dependencies:

npm ci

When updating dependencies, use poetry update, only where appropriate:

• Patching a transitive dependency that Dependabot cannot bump directly
• A periodic refresh of the whole tree, followed by a full test run
• Regenerating the lock after changing a version constraint in pyproject.toml

Prefer poetry update <package> to update specific package and its dependencies

Design system templates

To update the design system templates run:

make load-design-system-templates

Schemas

To download the latest schemas from the Questionnaire Registry:

make load-schemas

Run server

Run the server inside the virtual env created by Poetry with:

make run

Supporting services

Runner requires five supporting services - a questionnaire launcher, a storage backend, a cache, the supplementary data service and the collection instrument registry.

Run supporting services with Docker

First, authenticate to make sure Docker can pull from GAR

gcloud auth login

To run the app locally, but the supporting services in containers, make sure you have Podman installed and machine is running from this step, then run:

make dev-compose-up

Note that on Linux you will need to use:

make dev-compose-up-linux

Run supporting services locally

Questionnaire launcher

Note: the eq-questionnaire-launcher image published to Artifact Registry is amd64-only and segfaults under emulation on Apple Silicon. Use the compose setup above, which runs it successfully.

podman run -e SURVEY_RUNNER_SCHEMA_URL=http://host.docker.internal:5000 -e SDS_API_BASE_URL=http://host.docker.internal:5003 -e CIR_API_BASE_URL=http://host.docker.internal:5004 -it -p 8000:8000 europe-west2-docker.pkg.dev/ons-eq-ci/docker-images/eq-questionnaire-launcher:latest

Mock Supplementary data service

podman run -it -p 5003:5003 europe-west2-docker.pkg.dev/ons-eq-ci/docker-images/sds:latest

Mock Collection Instrument Registry

podman run -it -p 5004:5004 europe-west2-docker.pkg.dev/ons-eq-ci/docker-images/cir:latest

Storage backends

Note: the DynamoDB and Datastore emulator images referenced below are hosted in third-party registries and may no longer be accessible. The compose setup above uses maintained images and is the recommended path.

DynamoDB

podman run -it -p 6060:8000 onsdigital/eq-docker-dynamodb:latest

or

Google Datastore

podman run -it -p 8432:8432 knarz/datastore-emulator:latest

Cache

podman run -it -p 6379:6379 redis:4

Using Google Cloud Platform for supporting services

To use EQ_STORAGE_BACKEND as datastore or EQ_SUBMISSION_BACKEND as gcs directly on GCP and not a docker image, you need to set the GCP project using the following command:

gcloud config set project <gcp_project_id>

Or set the GOOGLE_CLOUD_PROJECT environment variable to your gcp project id.

---

Integration Tests

There is a dev-convenience script that auto generates the lines of code for a user journey. See README for more information and how to run the script.

Frontend Tests

The frontend tests are written using Playwright, which requires NodeJS to run. To handle different versions of NodeJS it is recommended to install Node Version Manager (nvm). It is similar to pyenv but for Node versions. To install nvm use the command below (make sure to replace "v0.39.5" with the current latest version in releases:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash

You will need to have the correct node version installed to run the tests. To do this, use the following commands:

nvm install
nvm use

Install npm dependencies and playwright browsers:

npm install
npx playwright install --with-deps

Runner needs to be run with the functional test environment variables:

RUNNER_ENV_FILE=.functional-tests.env make run

The functional tests use page models generated for each of the test schemas, generate them with:

make generate-pages

Then you can run either run the tests with:

make test-functional

or headless with:

make test-functional-headless

Both commands delete the tests/functional/generated_pages directory and regenerates all page models from the schemas.

Run a specific spec with (you only need the spec filename, not the path):

make test-functional-spec SPEC=<spec filename>

To run a single test, add .only into the name of any test or describe function:

Only this single test: test.only('Skip conditions', async ({ page }) => {...})

Only tests in this describe block: test.describe.only('Question description', () => {...})

In playwright.config.ts on line 29 we forbid .only in CI, so if you have .only in your tests, and you run the tests in CI, the tests will fail.

Run against a remote environment with:

EQ_FUNCTIONAL_TEST_ENV=https://staging-new-surveys.dev.eq.ons.digital/ make test-functional

More detailed information on running and debugging can be found in functional-tests.md

---

Deploying

For deploying with Concourse see the CI README.

Deployment with gcloud

To deploy this application with gcloud, you must be logged in using gcloud auth login and gcloud auth application-default login.

When deploying with gcloud the environment variables specified in Deploying the app must be set.

Then call the following command with environment variables set:

./ci/deploy_app.sh

Deploying credentials

Before deploying the app to GCP you need to create the application credentials. Run the following command to provision the credentials:

PROJECT_ID=PROJECT_ID EQ_KEYS_FILE=PATH_TO_KEYS_FILE EQ_SECRETS_FILE=PATH_TO_SECRETS_FILE ./ci/deploy_credentials.sh

For example:

PROJECT_ID=eq-test EQ_KEYS_FILE=dev-keys.yml EQ_SECRETS_FILE=dev-secrets.yml ./ci/deploy_credentials.sh

Deploying the app

The following environment variables must be set when deploying the app.

Variable Name	Description
PROJECT_ID	The ID of the GCP target project
DOCKER_REGISTRY	The FQDN of the target Docker registry
IMAGE_TAG

The following environment variables are optional:

Variable Name	Default	Description
REGION	europe-west2	The region that will be used for your Cloud Run service
CONCURRENCY	80	The maximum number of requests that can be processed simultaneously by a given container instance
MIN_INSTANCES	1	The minimum number of container instances that can be used for your Cloud Run service
MAX_INSTANCES	1	The maximum number of container instances that can be used for your Cloud Run service
CPU	4	The number of CPUs to allocate for each Cloud Run container instance
MEMORY	4G	The amount of memory to allocate for each Cloud Run container instance
GOOGLE_TAG_ID		The Google Tag ID - Specifies the GTM account
WEB_SERVER_TYPE	gunicorn-threads	Web server type used to run the application. This also determines the worker class which can be async/threaded
WEB_SERVER_WORKERS	7	The number of worker processes
WEB_SERVER_THREADS	10	The number of worker threads per worker
WEB_SERVER_UWSGI_ASYNC_CORES	10	The number of cores to initialise when using "uwsgi-async" web server worker type
DATASTORE_USE_GRPC	False	Determines whether to use gRPC for Datastore. gRPC is currently only supported for threaded web servers

To deploy the app, run the following command:

./ci/deploy_app.sh

---

Internationalisation

We use flask-babel to do internationalisation. To extract messages from source and create the messages.pot file, in the project root run the following command.

make translation-templates

make translation-templates is a command that uses pybabel to extract static messages.

This will extract messages and place them in the .pot files ready for translation.

These .pot files will then need to be translated. The translation process is documented in Confluence here

Once we have the translated .po files they can be added to the source code and used by the application

Environment Variables

The following env variables can be used

Variable Name	Default	Description
EQ_SESSION_TIMEOUT_SECONDS	2700 (45 mins)	The duration of the flask session
EQ_PROFILING	False	Enables or disables profiling (True/False) Default False/Disabled
EQ_GOOGLE_TAG_ID		The Google Tag Manger ID - Specifies the GTM account
EQ_ENABLE_HTML_MINIFY	True	Enable minification of html
EQ_ENABLE_SECURE_SESSION_COOKIE	True	Set secure session cookies
EQ_MAX_HTTP_POST_CONTENT_LENGTH	122_880	The maximum http post content length that the system wil accept
EQ_MINIMIZE_ASSETS	True	Should JS and CSS be minimized
MAX_CONTENT_LENGTH	122_880	max request payload size in bytes
EQ_APPLICATION_VERSION_PATH	.application-version	the location of a file containing the application version number
EQ_ENABLE_LIVE_RELOAD	False	Enable livereload of browser when scripts, styles or templates are updated
EQ_SECRETS_FILE	secrets.yml	The location of the secrets file
EQ_KEYS_FILE	keys.yml	The location of the keys file
EQ_SUBMISSION_BACKEND		Which submission backend to use ( gcs, rabbitmq, log )
EQ_GCS_SUBMISSION_BUCKET_ID		The bucket name in GCP to store the submissions in
EQ_GCS_FEEDBACK_BUCKET_ID		The bucket name in GCP to store the feedback in
EQ_RABBITMQ_HOST
EQ_RABBITMQ_HOST_SECONDARY
EQ_RABBITMQ_PORT	5672
EQ_RABBITMQ_QUEUE_NAME	submit_q	The name of the submission queue
EQ_SERVER_SIDE_STORAGE_USER_ID_ITERATIONS	10000
EQ_STORAGE_BACKEND	datastore
EQ_DYNAMODB_ENDPOINT
EQ_REDIS_HOST		Hostname of Redis instance used for ephemeral storage
EQ_REDIS_PORT		Port number of Redis instance used for ephemeral storage
EQ_DYNAMODB_MAX_RETRIES	5
EQ_DYNAMODB_MAX_POOL_CONNECTIONS	30
EQ_QUESTIONNAIRE_STATE_TABLE_NAME
EQ_SESSION_TABLE_NAME
EQ_USED_JTI_CLAIM_TABLE_NAME
WEB_SERVER_TYPE		Web server type used to run the application. This also determines the worker class which can be async/threaded
WEB_SERVER_WORKERS		The number of worker processes
WEB_SERVER_THREADS		The number of worker threads per worker
WEB_SERVER_UWSGI_ASYNC_CORES		The number of cores to initialise when using "uwsgi-async" web server worker type
DATASTORE_USE_GRPC	False	Determines whether to use gRPC for Datastore. gRPC is currently only supported for threaded web servers
ACCOUNT_SERVICE_BASE_URL	https://surveys.ons.gov.uk	The base URL of the account service used to launch the survey
ONS_URL	https://www.ons.gov.uk	The URL of the ONS website where static content is sourced, e.g. accessibility info
SDS_API_BASE_URL		The base URL of the SDS API used for fetching supplementary data
CIR_API_BASE_URL		The base URL of the CIR API used for fetching collection instruments
OIDC_TOKEN_BACKEND	gcp	The backend to use when fetching the Open ID Connect token
OIDC_TOKEN_LEEWAY_IN_SECONDS	300	The leeway to use when validating OIDC tokens
SDS_OAUTH2_CLIENT_ID		The OAuth2 Client ID used when setting up IAP on the SDS
CIR_OAUTH2_CLIENT_ID		The OAuth2 Client ID used when setting up IAP on the CIR

The following env variables can be used when running tests

EQ_FUNCTIONAL_TEST_ENV - the pre-configured environment [local, docker, preprod] or the url of the environment that should be targeted

---

JWT Integration

Integration with the survey runner requires the use of a signed JWT using public and private key pair (see https://jwt.io, https://tools.ietf.org/html/rfc7519, https://tools.ietf.org/html/rfc7515).

Once signed the JWT must be encrypted using JWE (see https://tools.ietf.org/html/rfc7516).

The JWT payload must contain the following claims:

• exp - expiration time
• iat - issued at time

The header of the JWT must include the following:

• alg - the signing algorithm (must be RS256)
• type - the token type (must be JWT)
• kid - key identification (must be EDCRRM)

The JOSE header of the final JWE must include:

• alg - the key encryption algorithm (must be RSA-OAEP)
• enc - the key encryption encoding (must be A256GCM)

To access the application you must provide a valid JWT. To do this browse to the /session url and append a token parameter. This parameter must be set to a valid JWE encrypted JWT token. Only encrypted tokens are allowed.

There is a python script for generating tokens for use in development, to run:

python token_generator.py

---

Profiling

Refer to our profiling document.

---

Updating / Installing dependencies

Python

To add a new dependency, use:

poetry add [package-name]

This will add the required packages to your pyproject.toml and install them

To update a dependency, use:

poetry update [package-name]

This will resolve the required dependencies of the project and write the exact versions into poetry.lock

Using the poetry up plugin we can update dependencies and bump their versions in the pyproject.toml file

To update dependencies to the latest compatible version with respect to their version constraints specified in the pyproject.toml file:

poetry up

To update dependencies to their latest compatible version:

poetry up --latest

NB: both the pyproject.toml and poetry.lock files are required in source control to accurately pin dependencies.

JavaScript

To add a new dependency, use npm install [dev dependency] --save-dev or npm install [dependency] then use npm install to install all the packages locally.

---

Testing Design System changes (locally) without pushing to actual CDN

On Design System Repo

Checkout branch with new changes on

You will need to install the Design System dependencies. If you haven't installed Yarn, install it with npm i -g yarn. To install the dependencies run yarn in the terminal. If you haven't you will also need to install gulp.

Then in the terminal run:

yarn cdn-bundle
cd build
browser-sync start --cwd -s --http --port 5678

You should now see output indicating that files are being served from localhost:5678. So main.css for example will now be served on http://localhost:5678//css/main.css

Now switch to the eQ Questionnaire Runner Repo

On eQ Questionnaire Runner Repo

In a separate terminal window/tab: Checkout the runner branch you want to test on

Edit your .development.env with following:

CDN_URL=http://localhost:5678
CDN_ASSETS_PATH=

Edit the Makefile to remove load-design-system-templates from the build command. Should now look like this:

build: load-schemas translate

Run make load-design-system-templates in the terminal to make sure you have the Design System templates loaded

Then edit the first line in the templates/layout/_template.njk file to remove the version number. Should now look like this:

{% set release_version = '' %}

Then spin up launcher and runner with make dev-compose-up and make run

Now when navigating to localhost:8000 and launching a schema, this will now be using the local cdn with the changes from the Design System branch