Expand documentation

.github/workflows/ci.yml

@@ -0,0 +1,60 @@
+name: CI
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  checks:
+    name: Lint and OpenAPI schema
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          submodules: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: "pip"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Lint
+        run: ruff check app utils tests run.py
+
+      - name: Check docs/openapi.json is up to date
+        run: python utils/export_openapi.py --check
+
+  tests:
+    name: Tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          submodules: true
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: "pip"
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      - name: Run tests
+        run: python -m pytest -q

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "moalmanac-db"]
-	path = moalmanac-db
+	path = moalmanac_db
 	url = https://github.com/vanallenlab/moalmanac-db

.vscode/settings.json

@@ -9,7 +9,6 @@
     "python.defaultInterpreterPath": "/opt/miniconda3/envs/moalmanac-api/bin/python",
     "python-envs.defaultEnvManager": "ms-python.python:conda",
     "python-envs.defaultPackageManager": "ms-python.python:conda",
-    "python-envs.pythonProjects": [],
     "[python]": {
         "editor.defaultFormatter": "charliermarsh.ruff"
     },

README.md

@@ -7,7 +7,7 @@ API for the [moalmanac-db](https://github.com/vanallenlab/moalmanac-db) and [moa
 This repository can be downloaded through GitHub by either using the website or terminal. To download on the website, navigate to the top of this page, click the green `Clone or download` button, and select `Download ZIP` to download this repository in a compressed format. To install using GitHub on terminal, type:
 
 ```bash
-git clone https://github.com/vanallenlab/moalmanac-api.git --recursive-submodules
+git clone https://github.com/vanallenlab/moalmanac-api.git --recurse-submodules
 cd moalmanac-api
 ```
 
@@ -20,15 +20,17 @@ git submodule update --init --recursive
 To update moalmanac-db to the latest commit, use [update_submodule_to_commit.sh](update_submodule_to_commit.sh):
 
 ```bash
-bash update_submodule_to_commit
+bash update_submodule_to_commit.sh
 ```
 
 Alternatively, to update moalmanac-db to a specific commit:
 
 ```bash
-bash update_submodule_to_commit <commit hash>
+bash update_submodule_to_commit.sh <commit hash>
 ```
 
+The script also regenerates [docs/openapi.json](docs/openapi.json) (see [OpenAPI schema](#openapi-schema)) and commits it together with the submodule update, so run it with this repository's Python environment activated, or point `PYTHON` at it (e.g. `PYTHON=/path/to/python bash update_submodule_to_commit.sh`). It refuses to run while `app/` has uncommitted changes, so the committed schema matches the committed code, and it commits nothing if regenerating the schema fails.
+
 ### Python dependencies
 
 This repository uses Python 3.12. We recommend using a [virtual environment](https://docs.python.org/3/tutorial/venv.html) and running Python with either [Anaconda](https://www.anaconda.com/download/) or [Miniconda](https://conda.io/miniconda.html).
@@ -61,7 +63,7 @@ ipython kernel install --user --name=moalmanac-api
 To update the database content from [moalmanac-db](https://github.com/vanallenlab/moalmanac-db):
 
  ```bash
-rm data/moalmanac.sqlite3; python -m app.populate_database -i moalmanac-db/referenced/ -c config.ini
+rm data/moalmanac.sqlite3; python -m app.populate_database -i moalmanac_db/referenced/ -c config.ini
 ```
 
 ### Environment configuration
@@ -95,6 +97,36 @@ Gunicorn is launched using the provided `ExecStart` command:
 
 Systemd and Gunicorn manage launching the application for production using the [service/moalmanac-api.service](service/moalmanac-api.service) file, so there is no need to run `python run.py` for production use.
 
+## Development
+
+### OpenAPI schema
+
+The API's interactive documentation is served at [https://api.moalmanac.org](https://api.moalmanac.org). A copy of its OpenAPI schema is committed at [docs/openapi.json](docs/openapi.json), which the [documentation site](https://docs.moalmanac.org) renders. Response schemas and examples in it are built from moalmanac-db's JSON Schemas and records (see [app/schemas.py](app/schemas.py)).
+
+Regenerate it whenever [app/routers/main/routes.py](./app/routers/main/routes.py) changes (routes, their parameters, or docstrings) or moalmanac-db is updated, and commit it with the change:
+
+```bash
+python utils/export_openapi.py
+```
+
+[update_submodule_to_commit.sh](update_submodule_to_commit.sh) does this automatically when updating moalmanac-db.
+
+### Continuous integration
+
+[GitHub Actions](.github/workflows/ci.yml) runs on every pull request and every push to `main`:
+
+- `ruff check app utils tests run.py`
+- `python utils/export_openapi.py --check`, which fails if [docs/openapi.json](docs/openapi.json) is out of date
+- `python -m pytest`
+
+To run the same checks locally before pushing:
+
+```bash
+ruff check app utils tests run.py
+python utils/export_openapi.py --check
+python -m pytest
+```
+
 ## Citation
 
 If you find this tool or any code herein useful, please cite:

app/dereferenced.py

@@ -1,39 +1,26 @@
 """
 Builds the in-memory cache of dereferenced records served by the API.
 
-The API's SQLite tables mirror moalmanac-db/schemas/referenced (see app.referenced).
-Endpoints instead serve records shaped like moalmanac-db/schemas/dereferenced, with
+The API's SQLite tables mirror moalmanac_db/schemas/referenced (see app.referenced).
+Endpoints instead serve records shaped like moalmanac_db/schemas/dereferenced, with
 foreign keys resolved to embedded objects. Rather than reimplementing that
 resolution, this module rebuilds the referenced tables from SQLite
 (`app.referenced.load_referenced`) and runs them through the moalmanac-db
-submodule's own `utils.dereference`, so the submodule stays the single source of
-truth for the output shape.
+submodule's own `moalmanac_db.utils.dereference`, so the submodule stays the single
+source of truth for the output shape.
 
 `build_cache` is called once, at startup (see app.main.create_app), and its result
 is stored on `app.state.dereferenced`. It is a plain read-only dict and is never
 written to disk or mutated afterward.
 """
 
 import copy
-import pathlib
-import sys
 import typing
 
 import sqlalchemy.orm
 
 from app import referenced
-
-_MOALMANAC_DB_ROOT = (
-    pathlib
-    .Path(__file__)
-    .resolve()
-    .parents[1]
-    / "moalmanac-db"
-)
-if str(_MOALMANAC_DB_ROOT) not in sys.path:
-    sys.path.insert(0, str(_MOALMANAC_DB_ROOT))
-
-from utils import dereference
+from moalmanac_db.utils import dereference
 
 # Every table `dereference.Database` needs to resolve foreign keys, keyed by the
 # same names as app.referenced.TABLES. `urls` has no entry in _CONCEPT_DIRS (it is
@@ -71,16 +58,16 @@ def build_cache(session: sqlalchemy.orm.Session) -> Cache:
 
     Rebuilds the referenced tables from SQLite, feeds them into a fresh
     `dereference.Database`, and dereferences each entity in the same order
-    `utils.dereference.write_all_concepts` uses. This is the entire dereferencing
-    logic; the API does not duplicate any of it.
+    `moalmanac_db.utils.dereference.write_all_concepts` uses. This is the entire
+    dereferencing logic; the API does not duplicate any of it.
 
     Args:
         session (sqlalchemy.orm.Session): The database session to read from.
 
     Returns:
         Cache: `{entity: {id: record}}` for every entity in
-        `utils.dereference._CONCEPT_DIRS`, each record equal to its file in
-        moalmanac-db/dereferenced/<entity>/. Dict order (both the outer entities and
+        `moalmanac_db.utils.dereference._CONCEPT_DIRS`, each record equal to its file in
+        moalmanac_db/dereferenced/<entity>/. Dict order (both the outer entities and
         each inner id) matches the referenced source, which callers rely on for
         deterministic, file-order output.
     """

app/main.py

@@ -3,7 +3,7 @@
 
 import fastapi
 
-from app import database, dereferenced
+from app import database, dereferenced, schemas
 from app.routers.main import router as main_router
 
 
@@ -21,6 +21,36 @@ def render(self, content: object) -> bytes:
         return json.dumps(content, indent=2, ensure_ascii=False).encode("utf-8")
 
 
+def strip_nullable_query_parameters(schema: dict) -> dict:
+    """
+    Removes the null branch from each query parameter's schema, in place.
+
+    Optional query parameters are typed `X | None` so their None default type
+    checks, which Pydantic renders as `anyOf: [X, {"type": "null"}]`. A query
+    parameter is omitted rather than sent as null, so the null branch only clutters
+    the documentation. Response schemas are left alone, where null is a real value.
+
+    Args:
+        schema (dict): The OpenAPI schema generated by FastAPI.
+
+    Returns:
+        dict: The same schema, with `anyOf: [X, null]` replaced by `X` for every
+            query parameter.
+    """
+    for path_item in schema.get("paths", {}).values():
+        for operation in path_item.values():
+            for parameter in operation.get("parameters", []):
+                if parameter.get("in") != "query":
+                    continue
+                parameter_schema = parameter.get("schema", {})
+                branches = parameter_schema.get("anyOf", [])
+                non_null = [branch for branch in branches if branch != {"type": "null"}]
+                if len(branches) == 2 and len(non_null) == 1:
+                    del parameter_schema["anyOf"]
+                    parameter_schema.update(non_null[0])
+    return schema
+
+
 def create_app(config_path: str = "config.ini") -> fastapi.FastAPI:
     """
     Creates and configures the FastAPI application instance.
@@ -51,7 +81,15 @@ def create_app(config_path: str = "config.ini") -> fastapi.FastAPI:
         description=(
             "The Molecular Oncology Almanac (MOAlmanac) is a paired knowledgebase and clinical interpretation "
             "algorithm for precision cancer medicine. Visit [our website](https://moalmanac.org) for more "
-            "information."
+            "information.\n\n"
+            "**Filtering**\n\n"
+            "- Different filters combine with AND: `?gene=BRAF&disease=Melanoma` returns records "
+            "matching all of them.\n"
+            "- Repeating a filter combines its values with OR: `?gene=BRAF&gene=KRAS` returns records "
+            "matching any of them.\n"
+            "- Exception: on `/search`, the `document`, `indication`, and `agent_id` filters narrow "
+            "each proposition's statement aggregates rather than which propositions are returned. "
+            "See the `/search` endpoint for details."
         ),
         docs_url="/",
         license_info={
@@ -82,7 +120,7 @@ def create_app(config_path: str = "config.ini") -> fastapi.FastAPI:
     if not os.path.exists(database_path):
         raise FileNotFoundError(
             f"{database_path} does not exist. Run "
-            "`python -m app.populate_database -i moalmanac-db/referenced -c "
+            "`python -m app.populate_database -i moalmanac_db/referenced -c "
             f"{config_path}` to create it.",
         )
 
@@ -94,6 +132,24 @@ def create_app(config_path: str = "config.ini") -> fastapi.FastAPI:
     app.state.dereferenced = cache
 
     app.include_router(main_router)
+
+    def openapi() -> dict:
+        """
+        Generates the OpenAPI schema once, without null branches on query parameters
+        and with moalmanac-db's record schemas added as components.
+
+        Returns:
+            dict: The cached OpenAPI schema.
+        """
+        if app.openapi_schema is None:
+            schema = fastapi.FastAPI.openapi(app)
+            strip_nullable_query_parameters(schema=schema)
+            schema.setdefault("components", {}).setdefault("schemas", {}).update(
+                schemas.build_components(),
+            )
+        return app.openapi_schema
+
+    app.openapi = openapi
     return app
 
 

app/models.py

@@ -1,5 +1,5 @@
 """
-SQLAlchemy models mirroring moalmanac-db/schemas/referenced.
+SQLAlchemy models mirroring moalmanac_db/schemas/referenced.
 
 Each entity table has one column per scalar field of its referenced schema, and the
 column name is the JSON key. Where the JSON key cannot be a Python attribute
@@ -543,17 +543,16 @@ class Documents(Base):
             nullable=True,
         )
     )
-    # Integer or string (or null) per the schema, so stored as JSON
-    identification_number = sqlalchemy.Column(
-        sqlalchemy.JSON,
-        nullable=True,
-    )
     publication_date: sqlalchemy.orm.Mapped[datetime.date | None] = (
         sqlalchemy.orm.mapped_column(
             sqlalchemy.Date,
             nullable=True,
         )
     )
+    regulatory_identifier = sqlalchemy.Column(
+        sqlalchemy.String,
+        nullable=True,
+    )
     status = sqlalchemy.Column(
         sqlalchemy.String,
         nullable=False,

app/populate_database.py

@@ -481,7 +481,7 @@ def main(referenced_directory: str, config_path: str = "config.ini") -> None:
     arg_parser.add_argument(
         "--input",
         "-i",
-        default="moalmanac-db/referenced",
+        default="moalmanac_db/referenced",
         help="Directory for referenced moalmanac db json files",
     )
     arg_parser.add_argument(

app/referenced.py

@@ -320,8 +320,8 @@ def _mappings(association: type[models.Base], owner: str) -> ListField:
             "drug_name_brand",
             "drug_name_generic",
             "first_publication_date",
-            "identification_number",
             "publication_date",
+            "regulatory_identifier",
             "status",
         ),
         lists={

app/routers/main/__init__.py

@@ -1 +1,3 @@
 from .routes import router
+
+__all__ = ["router"]

app/routers/main/handlers.py

@@ -7,8 +7,9 @@
 another entity (for example `?gene=BRAF` on `/propositions`). The route then calls
 `execute_query` for the matching ids and `serialize` to look them up in
 `app.state.dereferenced[entity]`. Because that cache already holds fully resolved,
-correctly ordered records (built by moalmanac-db's own `utils.dereference`), no
-handler builds or reorders a response dictionary itself.
+correctly ordered records (built by moalmanac-db's own
+`moalmanac_db.utils.dereference`), no handler builds or reorders a response
+dictionary itself.
 """
 
 import typing
@@ -509,7 +510,8 @@ def perform_joins(
     ) -> tuple[sqlalchemy.Select, set[type[models.Base]]]:
         """
         Filters by `document` (id), and `agent`/`agent_id` (delegated to
-        Agents.perform_joins once Documents is joined).
+        Agents.perform_joins once Documents is joined). `agent_type` is not a
+        filter on Documents, Indications, or Statements, so it is ignored.
 
         When `base_table` is Statements, joins through
         AssociationDocumentsAndStatements (`reportedIn`); when Indications, through
@@ -566,7 +568,11 @@ def perform_joins(
 
         statement, joined_tables = Agents.perform_joins(
             statement=statement,
-            parameters=parameters,
+            parameters={
+                key: values
+                for key, values in parameters.items()
+                if key != "agent_type"
+            },
             base_table=models.Documents,
             joined_tables=joined_tables,
         )