Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -284,9 +284,9 @@ definitions cannot drift from the local run. Six restate their commands inline:
definition is hand-kept in step with its `ci-local.sh` counterpart and will diverge if only
one side is edited.

**These workflows have not been exercised.** The repository was developed without a git
remote, so no run has ever happened and no tag has ever been cut. Treat an early red as a
pipeline defect until proven otherwise, and report it rather than working around it.
**These workflows are young.** They first ran at publication, and `release.yml` has run
once, for `v1.0.0-rc.1`. Plenty of legs have still run only once or twice, so treat an
early red as a possible pipeline defect and report it rather than working around it.

`all` covers every non-Docker check, including `sbom`, `conformance`, `crypt4gh`, the
`fuzz-smoke` compile-check of the excluded fuzz harnesses, and the real-data `corpus`.
Expand Down Expand Up @@ -952,7 +952,12 @@ Then:
`## [X.Y.Z]` section, so an un-rolled changelog blocks the release rather than being
silently dropped.
4. **Commit and tag.** Commit the above, then tag that same commit `vX.Y.Z` and push the
tag. `v0.x` tags publish as pre-releases and do not move `:latest`.
tag. A `v0.x` tag and any suffixed tag publish as pre-releases and do not move
`:latest`; only a stable tag does.

Update the version strings in the prose too. `README.md`, `docs/deployment.md`,
`docs/gdi-dataset-tool.md`, `docs/operating.md` and `docs/testing.md` name the current
release, and nothing checks them: `git grep 'v1\.0\.0-rc\.1'` finds the lot.
5. **What the tag enforces.** `release.yml` gates `publish` and `image` behind `gate`
(`ci-local.sh rust supply-chain`), `version-guard`, and the changelog assembly, and
`ci.yml` also runs on the tag. If anything is red, nothing publishes: fix and re-tag.
Expand Down
98 changes: 67 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,11 @@ queries in a common [GA4GH](https://www.ga4gh.org) language: does this variant e
and at what frequency? This node answers that from aggregate counts and publishes the
metadata a harvester needs. It does nothing else, which is why it stays small: one process,
no database, no message bus. It runs on a laptop, and
[Quickstart 1](#quickstart-1--see-it-work) has it answering a query minutes after the build.
[Quickstart 1](#quickstart-1--see-it-work) has it answering a query in minutes.

**Pre-release.** The tree is at `1.0.0-rc.1` and nothing is tagged, so there is no GitHub
Release, image or prebuilt binary, and everything below builds from source.
**Pre-release.** `v1.0.0-rc.1` is out. Binaries, a container image and checksums are on the
[Releases page](https://github.com/GenomicDataInfrastructure/gdi-node-standalone/releases).
It is a candidate, so the image has no `:latest` tag.
[CHANGELOG.md](CHANGELOG.md) records what `1.0.0` will commit to.

## What you get
Expand Down Expand Up @@ -67,7 +68,7 @@ starter kit if you need access control or a maintained stack.

| You are | Start at |
| --- | --- |
| evaluating it | [Quickstart 1](#quickstart-1--see-it-work): a node serving sample data, minutes after the build |
| evaluating it | [Quickstart 1](#quickstart-1--see-it-work): a node serving sample data, in minutes |
| operating a node | [Quickstart 2](#quickstart-2--a-real-node), then [Before you go live](#before-you-go-live) |
| a data provider | the provider half of Quickstart 2, then [Providers: your own data](#providers-your-own-data) |
| integrating against the API | [Integrating](#integrating) |
Expand Down Expand Up @@ -136,29 +137,48 @@ keyless, since nothing crosses a trust boundary. For the design, the trust bound
disclosure control, see [architecture.md](docs/architecture.md) and
[threat-model.md](docs/threat-model.md).

## Build
## Get it

The service runs on Linux `x86_64` and `aarch64`, the tool also on macOS and Windows,
with MSRV **1.96** ([deployment.md § Binaries and platforms](docs/deployment.md#binaries-and-platforms)).
You need `rustup` (the pinned toolchain installs itself on first use), a C toolchain
(`build-essential`, `gcc`, or the Xcode Command Line Tools), `git` and `curl`, plus `jq` for
the query in Quickstart 1; `scripts/dev-setup.sh --check` verifies the toolchain.
**Download.** Two binaries, plus the two sample files Quickstart 1 uses. The service
binary is Linux only and is the full build; the tool also ships for macOS and Windows
([platforms](docs/deployment.md#binaries-and-platforms)).

```bash
V=v1.0.0-rc.1
U=https://github.com/GenomicDataInfrastructure/gdi-node-standalone/releases/download/$V
R=https://raw.githubusercontent.com/GenomicDataInfrastructure/gdi-node-standalone/$V/crates/gdi-dataset-tool/tests/fixtures
mkdir -p ~/gdi-demo && cd ~/gdi-demo
curl -fsSLo gdi-node-standalone "$U/gdi-node-standalone-$V-x86_64-unknown-linux-gnu"
curl -fsSLo gdi-dataset-tool "$U/gdi-dataset-tool-$V-x86_64-unknown-linux-gnu"
curl -fsSLO "$R/covid-package.yaml" -O "$R/COVID.monogneic.aggregate.AFs.GRCh38.vcf"
chmod +x gdi-node-standalone gdi-dataset-tool
export PATH="$PWD:$PATH"
```

Both have `musl` builds for Alpine. The service also builds for `aarch64`; the tool does
not, so compile that one there. Checksums and a provenance attestation sit beside the
binaries, and
[operating.md §20](docs/operating.md#20-verifying-release-artifacts--the-container-image)
has the verify commands. There is an image too, `linux/amd64` only:
`docker pull ghcr.io/genomicdatainfrastructure/gdi-node-standalone:v1.0.0-rc.1`.

**Build** on macOS or Windows, where the service has no binary, or to work on the code. It
needs MSRV **1.96**, `rustup` and a C toolchain (`build-essential`, `gcc`, or the Xcode
Command Line Tools); `scripts/dev-setup.sh --check` checks them. The first build takes tens
of minutes, because arrow, parquet and noodles compile from source.

```bash
git clone https://github.com/GenomicDataInfrastructure/gdi-node-standalone.git
cd gdi-node-standalone
cargo build --release -p gdi-node-standalone -p gdi-dataset-tool # lite node + the tool (Quickstart 1)
cargo build --release -p gdi-node-standalone --features full # node with S3, Vault and PME (Quickstart 2)
cargo build --release -p gdi-node-standalone -p gdi-dataset-tool # lite node + the tool
cargo build --release -p gdi-node-standalone --features full # + S3, Vault and PME
export PATH="$PWD/target/release:$PATH"
```

The first build compiles the arrow, parquet and noodles crates from source and takes tens
of minutes; later builds reuse the cache. The default **lite** node has no S3, Vault or
at-rest encryption and makes no outbound connections, which is enough for an inbox node. A
config with `[[s3.buckets]]`, `[vault]` or `[vault].transit_key` needs the **full** build;
a lite binary refuses such a config and says why. The container image is always full:
`docker build -t gdi-node-standalone:local .`, plus the two build args that stamp a git SHA
into `/version` ([deployment.md § Container image](docs/deployment.md#container-image)).
A build defaults to **lite**: no S3, Vault or at-rest encryption, and no outbound
connections, which is enough for an inbox node. A config with `[[s3.buckets]]`, `[vault]`
or `[vault].transit_key` needs **full**; a lite binary refuses it and says why. Downloads
are full already.

## Quickstart 1 — see it work

Expand Down Expand Up @@ -197,24 +217,27 @@ Now be the provider: build the sample into a staging directory, drop it into the
make it visible, query it.

```bash
gdi-dataset-tool build crates/gdi-dataset-tool/tests/fixtures/covid-package.yaml --cc EE -o build
gdi-dataset-tool build covid-package.yaml --cc EE -o build # in a checkout: crates/gdi-dataset-tool/tests/fixtures/
ID=$(ls build) # build/ was empty; every build mints a new id
gdi-dataset-tool deploy build/$ID --inbox ~/gdi-demo/inbox --wait --management-url http://127.0.0.1:9090
gdi-dataset-tool publish $ID --inbox ~/gdi-demo/inbox # writes {id}.state.json; the node applies it on its next scan
curl -s http://127.0.0.1:9090/datasets/$ID/state # -> {"state":"visible",…} within rescan_interval_seconds

curl -s -X POST http://localhost:8080/aggregated/beacon/v2/g_variants \
-H 'content-type: application/json' \
-d '{"query":{"requestParameters":{"referenceName":"3","start":[45823239],"referenceBases":"T","alternateBases":"C","assemblyId":"GRCh38","requestedGranularity":"RECORD"}}}' \
| jq '.responseSummary, .response.resultSets[0].results[0].frequencyInPopulations[0].frequencies[-1]'
# -> {"exists": true, "numTotalResults": 1} and the Total population: alleleCount 618 / alleleNumber 8000
-d '{"query":{"requestParameters":{"referenceName":"3","start":[45823239],"referenceBases":"T","alternateBases":"C","assemblyId":"GRCh38","requestedGranularity":"RECORD"}}}'
# One line of JSON. Look for "exists":true and "numTotalResults":1, and in the Total
# population "alleleCount":618 with "alleleNumber":8000. Pipe to `jq` if you have it.
```

For a Beacon that answers like a real export (1 637 sites, twelve populations, chrX/Y/M),
build the realistic sample instead and repeat the `deploy` and `publish` lines with its id:

```bash
gdi-dataset-tool build crates/test-util/tests/fixtures/sample/gdi-sample.package.yaml --cc EE -o build-sample
V=v1.0.0-rc.1
S=https://raw.githubusercontent.com/GenomicDataInfrastructure/gdi-node-standalone/$V/crates/test-util/tests/fixtures/sample
curl -fsSLO "$S/gdi-sample.package.yaml" -O "$S/gdi-sample.GRCh38.vcf.gz"
gdi-dataset-tool build gdi-sample.package.yaml --cc EE -o build-sample # checkout: crates/test-util/tests/fixtures/sample/
ID=$(ls build-sample)
```

Expand All @@ -228,7 +251,8 @@ Point needs a `[fairdp]` block; Quickstart 2 has one.
S3 ingest with the node's crypt4gh key on disk, no Vault and no PME. That is the usual
production shape for public aggregated data ([`node.quickstart.toml`](node.quickstart.toml),
[operating.md §0](docs/operating.md#0-quickstart-first-production-bring-up)). It needs the
full build and an S3-compatible bucket: Garage, Ceph RGW, MinIO or AWS. The `[fairdp]`
full node (the download is one; from source, `--features full`) and an S3-compatible
bucket: Garage, Ceph RGW, MinIO or AWS. The `[fairdp]`
block is what makes the node a FAIR Data Point. Fill in every field, including the
publisher, the Health Data Access Body and both contact points, or delete the block for a
Beacon-only node.
Expand Down Expand Up @@ -381,7 +405,7 @@ do about it.
your CNI enforces. Read its "Before you apply" list first.
- **Vault and at-rest encryption (PME).** With a `[vault]` block the node reads its
identity and S3 credentials from Vault or OpenBao, and `[vault].transit_key` encrypts the
Parquet at rest (Parquet Modular Encryption). Both need the full build. Public aggregated
Parquet at rest (Parquet Modular Encryption). Both need the full node. Public aggregated
data does not need PME; an encrypted volume is the baseline. Try it on the dev stack
(root token, no TLS):
```bash
Expand All @@ -398,11 +422,23 @@ do about it.
## Providers: your own data

You have a VCF with per-population allele counts or frequencies and you want it
discoverable through a node. Build the tool ([Build](#build)) and let the wizard walk you
through it, as in the provider half of [Quickstart 2](#quickstart-2--a-real-node). If the
node is on the same machine, `deploy --inbox` and `publish --inbox`
([Quickstart 1](#quickstart-1--see-it-work)) need no keys and no profile. Worth knowing
before your first real build:
discoverable through a node. You need the tool, not the node, and it is one file from the
[Releases page](https://github.com/GenomicDataInfrastructure/gdi-node-standalone/releases):

- **Linux, x86-64**: `gdi-dataset-tool-<version>-x86_64-unknown-linux-gnu`, or the `-musl`
one on Alpine.
- **macOS, Apple silicon**: `gdi-dataset-tool-<version>-aarch64-apple-darwin`.
- **Windows**: `gdi-dataset-tool-<version>-x86_64-pc-windows-msvc.exe`.

On Linux and macOS, `chmod +x` it and put it on your `PATH`. A browser download is
quarantined on macOS, which `xattr -d com.apple.quarantine gdi-dataset-tool` clears. Intel
Macs and aarch64 Linux have no published build, so compile from source there
([Get it](#get-it)).

Then let the wizard walk you through it, as in the provider half of
[Quickstart 2](#quickstart-2--a-real-node). If the node is on the same machine,
`deploy --inbox` and `publish --inbox` ([Quickstart 1](#quickstart-1--see-it-work)) need no
keys and no profile. Worth knowing before your first real build:

- **Your VCF** can be on GRCh37 or GRCh38. Populations come from the INFO fields (`AF`,
`AC` and `AN`, one set per population), and only those aggregate counts leave your
Expand Down
22 changes: 9 additions & 13 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ The service is a single binary, built in two flavours:

Prefer `gnu`. Reach for `musl` only when the gnu binary will not start.

No release has been cut, so there is no `v*` tag: the Releases page is empty, the GHCR
image does not exist, and the download commands below do not resolve. Build from source
until a tag exists — it is the only working install path:
There are prebuilt binaries for `v1.0.0-rc.1`, listed further down. To build from source:

```bash
cargo build --release --features full -p gdi-node-standalone
Expand All @@ -56,7 +54,7 @@ source in its own temporary target directory, sharing nothing with the others. A
`--target-dir target`, or set `CARGO_TARGET_DIR` for the session, to share the work. Cargo
prints no progress when its output is piped or redirected, so a silent terminal is normal.

Once a release exists, download a prebuilt, signed, checksummed binary from the
Or skip the build: prebuilt, attested, checksummed binaries are on the
[Releases page](https://github.com/GenomicDataInfrastructure/gdi-node-standalone/releases).
The per-target service assets are named
`gdi-node-standalone-<tag>-{x86_64,aarch64}-unknown-linux-{gnu,musl}`; fetch one plus the
Expand Down Expand Up @@ -149,14 +147,12 @@ single-writer invariant.

## Container image

> **No image has been published yet.** No `v*` tag exists, so nothing has ever been pushed
> to GHCR and the `docker pull` below fails with not-found. Build your own from the shipped
> `Dockerfile` — see [Building your own image](#building-your-own-image) — until the first
> tag ships. This section is the procedure that becomes live with that release.
> **There is no `:latest`.** The workflow moves it only on a stable release, and
> `v1.0.0-rc.1` is a candidate. Pull the exact tag.

Tagged releases will publish a container image to GHCR, packaged from the same released
`gnu` (glibc) binary onto a minimal `distroless/cc` base. There is no recompile: the binary
is byte-identical to the bare-metal download.
Tagged releases publish a container image to GHCR, packaged from the same released `gnu`
(glibc) binary onto a minimal `distroless/cc` base. There is no recompile: the binary is
byte-identical to the bare-metal download.

```bash
docker pull ghcr.io/genomicdatainfrastructure/gdi-node-standalone:<tag>
Expand Down Expand Up @@ -701,8 +697,8 @@ an individual-level network only if the node really serves individuals.

## Compatibility

The declared support set. No prebuilt artifacts are published yet, so build the row you
need from source. `scripts/ci-local.sh` (see [`CONTRIBUTING.md`](../CONTRIBUTING.md))
The declared support set. Most rows ship as prebuilt binaries; build the rest from
source. `scripts/ci-local.sh` (see [`CONTRIBUTING.md`](../CONTRIBUTING.md))
build-verifies the Linux rows with its `cross` and `cross-arm` legs; both live in the
`release` target rather than `all`, so treat a row as verified once you have run it
yourself.
Expand Down
8 changes: 4 additions & 4 deletions docs/gdi-dataset-tool.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,10 @@ version is 1.96, and every crate builds a C-dependent build script, so a linker
Nothing else: no S3 endpoint, no Vault, no at-rest encryption. The binary does always link
the S3 and network stack, as the note below explains.

> **No prebuilt binaries yet.** No release has been tagged, so the Releases page is empty
> and there is nothing to download. Building from source is currently the only way to get
> the tool. Once a release exists, per-platform binaries for Linux, macOS and Windows will
> be published there with checksums and provenance attestation.
> **There are prebuilt binaries now.** `v1.0.0-rc.1` ships the tool for Linux (`gnu` and
> `musl`), macOS on Apple silicon and Windows, with checksums and a provenance attestation,
> on the [Releases page](https://github.com/GenomicDataInfrastructure/gdi-node-standalone/releases).
> Building from source, below, still works.

During development, run it through Cargo:

Expand Down
19 changes: 7 additions & 12 deletions docs/operating.md
Original file line number Diff line number Diff line change
Expand Up @@ -2742,18 +2742,13 @@ operational invariant covered above under [Where things live](#where-things-live

## 20. Verifying release artifacts + the container image

> **Status: no release has been cut yet.** The repository carries no tags at all, and
> `release.yml` triggers on a `v*` tag alone, so
> [`release.yml`](../.github/workflows/release.yml) has never run and none of the artifacts
> below have been published: no GitHub Release, no `SHA256SUMS`, no attestation, no SBOM,
> no `ghcr.io/…` image. The commands here fail with not-found until the first tag ships.
> Deployments today are built from source and carry no provenance or checksum you can
> verify. This section is the procedure that becomes live with the first release.

Once tagging begins, each release will publish, alongside the per-platform binaries: one
`SHA256SUMS`, a keyless SLSA build-provenance attestation via GitHub OIDC, needing no
signing-key secrets, and a CycloneDX SBOM per shipped binary (`gdi-node-standalone`,
`gdi-dataset-tool`) rather than one for the whole release. Verify before deploying.
> **`v1.0.0-rc.1` is the current release.** Being a candidate, it has no `:latest` image
> tag, so use the exact tag in the commands below.

Each release publishes, alongside the per-platform binaries: one `SHA256SUMS`, a keyless
SLSA build-provenance attestation via GitHub OIDC, needing no signing-key secrets, and a
CycloneDX SBOM per shipped binary (`gdi-node-standalone`, `gdi-dataset-tool`) rather than
one for the whole release. Verify before deploying.

**Checksums.** Download `SHA256SUMS` next to the artifact(s) and check:

Expand Down
7 changes: 3 additions & 4 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -297,10 +297,9 @@ comment: the upstream check enumerates on that comment, so an uncommented pin wo
flagged, merely unchecked.

`cargo-semver-checks` (public library API) belongs in this list but is not enforced: it is
in neither meta-leg, because its default baseline is HEAD and there is no `v*` tag to
compare against. Until the first release a public-API break is caught by nothing; after
one, run `SEMVER_BASELINE=v1.0.0 ./scripts/ci-local.sh semver-checks` against the previous
tag.
in neither meta-leg, because its default baseline is HEAD. Now that `v1.0.0-rc.1` exists,
run it against that tag by hand:
`SEMVER_BASELINE=v1.0.0-rc.1 ./scripts/ci-local.sh semver-checks`.

### Out-of-process

Expand Down