Skip to content

Adding a scenario to the databases config, and improving the init-see… #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alneberg merged 1 commit into from
Oct 21, 2026
Merged

Adding a scenario to the databases config, and improving the init-see… #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 33 additions & 20 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ Holds views and test data to fire up a dev instance of StatusDB (CouchDB). This
The [databases](/DATABASES.md) documentation attempts to give an overview of each database and explain its purpose.

# Views
The first part of this repository is the code for the CouchDB views used for the NGI Stockholm StatusDB. The views are organised in the directory StatusDB_views and listed in the views_config.yaml.
The first part of this repository is the code for the CouchDB views used for the NGI Stockholm StatusDB. The views are organised in the `StatusDB_views` directory and listed in `views_config.yaml`.

## History
StatusDB_views was kept private since it could contain some project details or other details that was not supposed to be public. However this decision was changed in 2026 since there is no real reason to keep sensitive information in this repository. The content of the repository was moved to StatusDB_NGI since the git history might still hide details which are not supposed to be made public. Here are the contributors of the old repository:
StatusDB_views was kept private since it could contain some project details or other details that were not supposed to be public. This decision was changed in 2026, since there is no real reason to keep sensitive information in this repository. The content was moved to StatusDB_NGI without its git history, which might still contain details that were not supposed to be public. Here are the contributors of the old repository:

## Contributors of historic StatusDB_Views repository
- @aanil
Expand Down Expand Up @@ -35,21 +35,21 @@ They need to follow the naming pattern `<view_name>.map.js` and `<view_name>.red

Each database, design document and view is listed in the config file `views_config.yaml`.

## Checking and updating the views_config.yaml
To try to keep the views_config.yaml file up to date with the content of the repository, the script script/validate_views_config.py can be used:
## Checking and updating views_config.yaml
To check that `views_config.yaml` is in sync with the repository content, use `scripts/validate_views_config.py`:

```
> python scripts/validate_views_config.py --config views_config.yaml --views-dir StatusDB_views --check
> python3 scripts/validate_views_config.py --config views_config.yaml --views-dir StatusDB_views --check
✅ Config is up to date with directory structure
```

And to update the config (default is that a new view will not be included in dev):
To update the config from the directory structure (new views default to `scenarios: [stage]`):
```
> python scripts/validate_views_config.py --config views_config.yaml --views-dir StatusDB_views --update
> python3 scripts/validate_views_config.py --config views_config.yaml --views-dir StatusDB_views --update
```

# Seed Data
The second part of this repository is the seed data, which is mock data that can be used for testing and demoing a range of internal services that use StatusDB as its backend. The data is located in the seed_data directory and is used to populate the database when building the docker image.
The second part of this repository is the seed data, which is mock data for testing and demoing the internal services that use StatusDB as their backend. The data lives in the `seed_data` directory, is copied into the Docker image, and is loaded into CouchDB on first startup.

## Seed Data Structure

Expand All @@ -64,39 +64,42 @@ seed_data/
└── doc2.json
```

JSON files placed directly in `seed_data/` (legacy layout) are loaded into a `statusdb` database.

### Document Format

Each JSON file should contain a single CouchDB document. If the document has an `_id` field, it will be used as the document ID. Otherwise, CouchDB will auto-generate an ID.

Example document (`seed_data/example_project.json`):
Example document (abridged from `seed_data/projects/P12345.json`):

```json
{
"_id": "project_001",
"type": "project",
"name": "Example Genomics Project",
"_id": "P12345",
"project_id": "P12345",
"project_name": "Test RNA-seq Project",
"application": "RNA-seq",
"status": "ongoing",
...
}
```

# Docker Image

The docker image is most commonly used in the docker-compose setup for genomics status, see [Using with Genomics Status](#using-with-genomics-status) In that case, the docker is built automatically as part of the setup. However, in case the docker image should be used standalone, it needs to be built using:
The Docker image is most commonly used in the docker-compose setup for genomics-status (see [Using with Genomics Status](#using-with-genomics-status)), where it is built automatically as part of the setup. To use the image standalone, build it with:

```
cd Statusdb_NGI
docker build .
cd StatusDB_NGI
docker build -t statusdb_ngi .
```

And starting a container using
and start a container with

```bash
docker run -d \
-p 5984:5984 \
-e COUCHDB_USER=admin \
-e COUCHDB_PASSWORD=admin \
ghcr.io/scilifelab/StatusDB_NGI:latest
statusdb_ngi
```

After this, CouchDB will be available at:
Expand All @@ -105,7 +108,17 @@ After this, CouchDB will be available at:
- Fauxton UI: <http://localhost:5984/_utils>
- Credentials: `admin` / `admin`

The seed data is automatically loaded on first startup along with all views in the Statusdb_Views directory. It's prepared to use different scenarios where only a subset of views and data could be loaded, but currently there is only one `stage` scenario where all views and data is loaded.
On first startup, the seed data and the design documents from `StatusDB_views` are loaded into CouchDB. What is loaded is controlled by *scenarios*, selected with the `SCENARIO` environment variable (default: `stage`).

A scenario decides which databases and views are created:

- A database can list the scenarios it belongs to with a `scenarios` key in `views_config.yaml`. A database without a `scenarios` key is included in all scenarios.
- Inside an included database, a view is only deployed if the scenario is also in the view's own `scenarios` list.
- Seed data follows the database: if a database is not included in the scenario, the database is not created and its seed documents are not loaded.

Initialization is tracked per scenario, so it only runs on the first startup for a given scenario. Changing `SCENARIO` triggers a new initialization on the next startup; loading is idempotent, so documents that already exist are overwritten.

Currently the only scenario in use is `stage`, where all databases, views and seed data are loaded.

### Persisting Data

Expand All @@ -129,8 +142,8 @@ docker run -p 5984:5984 -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=admin StatusDB
```

## Container registry
There is a github workflow file written to facilitate automatic build and publishing of the seed data container, but it is not yet in use.
A GitHub workflow (`.github/workflows/docker-publish.yml`) for building and publishing the image to ghcr.io is in place, but the publish job is currently disabled.

## Using with Genomics Status

The [genomics-status](https://github.com/SciLifeLab/genomics-status) repository is configured to use this image in its docker compose setup. When starting docker compose, it will automatically build the image and populate it with the seed data, using the stage scenario.
The [genomics-status](https://github.com/SciLifeLab/genomics-status) repository is configured to use this image in its docker compose setup. When starting docker compose, the image is built automatically and the seed data is loaded on startup, using the `stage` scenario.
Loading