Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions .agents/docs/JOURNAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -7546,3 +7546,89 @@ the `[0.9.0]` OTLP entry states. The `imbhd` binary size was not re-measured, so
in TODO.md remains the August one and is labelled as such.

Not committed, not tagged, not published.

## v0.9.0 shipped without its archives, and what got them back (2026-09-15)

The `v0.9.0` GitHub Release page has no assets, and it never will. Release run
[34788914182](https://github.com/moriyoshi/imbh/actions/runs/34788914182) built and smoke-tested all
five targets, then `publish` failed in five seconds at its preflight. The archives were recovered
and published under the companion tag `dist-v0.9.0`.

### What actually happened

The Release was created **and published** by hand, ahead of the tag push, to stop CD creating one by
accident. Under GitHub's immutable releases that is the one irreversible move available: publishing
freezes the asset set at the moment of publication. By the time the five build legs finished there
was a published, frozen, empty Release sitting on the tag, and `release.yml`'s `published)` branch
refused rather than attempt an upload.

The refusal was correct on every count, and worth recording precisely because each half of it is
counter-intuitive:

* `gh api repos/moriyoshi/imbh/releases/tags/v0.9.0` reports `"immutable": true`. An upload returns
HTTP 422 `Cannot upload assets to an immutable release`. GitHub's wording is "Once you publish a
release as immutable, its assets can't be added, modified, or deleted" — *added* included, so this
is not a case of replacing a bad asset, it is a closed set.
* Turning the repository setting off does **not** unfreeze a release that was published while it was
on. The flag is stamped on the release, not read from the setting.
* Deleting the release to retry burns the name: "you cannot reuse the same tag name", and the
reservation survives repository deletion. Both the workflow error text and `README.md` already said
so in capitals; they were right.

### The gap the draft-first order does not cover

`release.yml` has published into a draft since the 0.3.0 incident, and that order is what protects
the pipeline from itself. It cannot protect against a Release that is already frozen *before the run
starts*. 0.3.0 and 0.9.0 are the same failure reached from opposite directions — CD publishing too
early, and a human publishing too early — and only the first one is a pipeline bug.

**The rule that generalises: never create a *published* Release for a version tag ahead of CD. If
one has to exist early, leave it a draft.** `release.yml:119` already handles that case explicitly —
it reuses an existing draft, fills it, and publishes it. A draft created early costs nothing; a
published one spends the version number.

### Why recovery was possible at all, and why 0.3.0 could not be

Only the *Release* was lost, not the binaries. Workflow artifacts have a 90-day retention, and the
run's `dist-*` artifacts were still live (expiring 2026-12-12). 0.3.0 was never recoverable because
by the time anyone looked its artifacts were long gone. That makes retention the real clock on this
class of incident: the window to rescue a frozen release is 90 days from the run, not from the
discovery.

The archives were pulled with `gh run download`, `SHA256SUMS` was rebuilt with the same
`cat ./*.sha256 | sort -k2` the `publish` job uses so the asset set is identical in shape to every
prior release, and the whole set went into a **draft** that was verified before publishing — the new
release is immutable too, so it also got exactly one attempt.

### Choosing the tag name

`dist-v0.9.0`, not `v0.9.0-dist`, and the prefix is load-bearing three times over:

* `release.yml` triggers on `push: tags: ['v*']`. A `v`-prefixed companion tag would have fired a
second full release run at publish time — five fat-LTO build legs, plus `image` and `plugin`
jobs pushing `0.9.0-dist` tags to GHCR as pollution.
* The `Version tags` ruleset scopes to `refs/tags/v*` and requires signatures and blocks deletion.
Outside it, the tag stayed deletable right up until the release was published, which is what made
the draft genuinely re-runnable.
* `v0.9.0-dist` is a semver *pre-release of 0.9.0* — it sorts as older than the release it carries
the binaries for.

### Verified

`sha256sum -c SHA256SUMS` passes 5/5; each archive carries `imbhd`, `imbh-tui`, `LICENSE` and
`THIRD-PARTY-NOTICES.txt`. The recovered `imbhd`, `imbh-tui` and notices are **byte-identical** to
the ones extracted from the published `ghcr.io/moriyoshi/imbh:0.9.0` (arm64) — so the archives are
provably the shipped artifacts, not a rebuild. The linux-gnu binaries pass the build job's own
assertions (`nothing to serve`, the `imbh-tui` usage line) and report a glibc floor of 2.34, under
the 2.36 limit. End to end from the public URL: `curl` + `sha256sum --ignore-missing -c` + `tar xzf`
+ run, clean. `v0.9.0` is still the repository's latest release, and the tag push triggered no
workflow.

### Left alone

`release.yml` is unchanged. It already does the right thing and already fails fast with an
explicit message; the failure was outside the pipeline. The run's separate
`THIRD-PARTY-NOTICES.txt differs from the checked-in copy` warning is untouched and still open — the
archives carry the run-generated copy, so it does not affect what was published here.

Docs not committed.
38 changes: 17 additions & 21 deletions .agents/docs/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,27 +76,23 @@ git history); this file tracks only what is still open.
the catalog's fold or bound them to the eval window. `exemplars()` likewise takes only a metric
name and is filtered by window client-side.

- [ ] **v0.9.0 is prepared but not cut.** The workspace is bumped to 0.9.0 and the prep is merged to
`main` (PR #52), but the release section has been **re-cut and re-dated 2026-09-14**: five more
PRs landed on top of the August prep (#53–#58), so the `[Unreleased]` entries they wrote were
folded into `[0.9.0]` rather than left for a 0.10.0 — see JOURNAL "Preparing v0.9.0" and
"Re-cutting v0.9.0". `README.md` / `docs/DOCKER_LOG_DRIVER.md` version strings and the notices
are still the August ones and still correct, because the version itself never moved. **Not
tagged and not published**: `v0.9.0` exists nowhere on the remote and crates.io still tops out
at 0.8.0. That is the user's call.

**A minor bump the signatures require.** Four breaking changes, all in the `[0.9.0]`
`### Changed` entries: `imbh-head`'s `dto::Series` gained a public `query_index` field and
became `#[non_exhaustive]`, `imbh-lgtm`'s `execute_traceql` gained an `S: Sync` bound,
`POST /v1/{logs,traces,metrics}` now answers the response the OTLP spec prescribes instead of
this server's own `{"accepted": …}` JSON, and `imbh_server::Response` gained a `headers` field.

**No new crates and no dependency change**, so the publish order is exactly v0.8.0's and the
footprint is unmoved — re-measured 2026-09-14 on the re-cut tree: 275 crates for the facade
(the gated axis, unchanged), `imbh-server` 298 default / 304 with `grpc`. The `imbhd` 33.5 MiB
binary figure is the August measurement and was not re-taken.

*(v0.8.0 — the entry this replaces — was tagged and released 2026-08-09.)*
- [x] **v0.9.0 is cut and published — but its Release carries no archives.** *(Closed 2026-09-15.)*
Tagged `v0.9.0` at `f483785` and published: all 12 crates are on crates.io at 0.9.0, and
`ghcr.io/moriyoshi/imbh:0.9.0` / `0.9` / `latest` plus the log-driver plugin tags are on GHCR.

**The GitHub Release was created and published by hand shortly before the tag was pushed**, so
when release run 34788914182 finished building all five targets the `publish` job had nowhere
to upload: an immutable release accepts no assets after publication, and the job's preflight
refused rather than let anyone burn the tag name by deleting it. The archives were recovered
from the run's workflow artifacts and published under the companion tag
[`dist-v0.9.0`](https://github.com/moriyoshi/imbh/releases/tag/dist-v0.9.0) on the same commit;
the binaries in them are byte-identical to those in the published image. `README.md`,
`CHANGELOG.md` and the v0.9.0 Release notes all point there. See the JOURNAL entry "v0.9.0
shipped without its archives" for the mechanism and the rule it implies.

*(The minor-bump rationale and the 2026-09-14 footprint re-measurement this item used to carry
are in the `[0.9.0]` CHANGELOG section and the JOURNAL "Re-cutting v0.9.0" entry. The `imbhd`
33.5 MiB binary figure quoted there is still the August measurement.)*

- [x] **Measure the promoted-column cost before setting any auto-promotion budget.** *(Closed
2026-08-08 — `examples/bench --bin promote-cost`. The gate passes, and it moved the budget from
Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,16 @@ release aborts if it is missing or duplicated.

## [0.9.0] - 2026-09-14

> **The archives are at [`dist-v0.9.0`](https://github.com/moriyoshi/imbh/releases/tag/dist-v0.9.0),
> not on the `v0.9.0` Release.** The Release was created and published by hand shortly before the tag
> was pushed, so when CD finished building it had nowhere to upload: an immutable release accepts no
> assets once published, and deleting it to retry would have reserved the `v0.9.0` name permanently.
> The `publish` job refused for exactly that reason. Everything else shipped normally — all 12 crates
> are on crates.io at 0.9.0, and the container image and log-driver plugin are published. The
> archives were recovered from the release run's workflow artifacts and published under a companion
> tag on the same commit; the binaries in them are byte-identical to those in the 0.9.0 image. See
> the note under the download recipe in `README.md`.

### Added

- **`Db::traces().get_many(ids, not_before)`** fetches every span of a list of trace ids in one
Expand Down
21 changes: 20 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ the reference server `imbhd` and the terminal explorer `imbh-tui`.

Every `vX.Y.Z` [GitHub release](https://github.com/moriyoshi/imbh/releases) carries a prebuilt
archive per platform, containing both binaries plus `LICENSE` and `THIRD-PARTY-NOTICES.txt` (with
one exception — see the note under the download recipe below about 0.3.0):
two exceptions — see the notes under the download recipe below about 0.9.0 and 0.3.0):

| Platform | Archive | `docker` log driver |
|---|---|---|
Expand All @@ -209,6 +209,16 @@ tar xzf imbh-${VERSION}-${TARGET}.tar.gz
./imbh-${VERSION}-${TARGET}/imbhd ./imbh-data # then point an OTLP exporter at :4318
```

**0.9.0's archives are under a different tag.** Its GitHub Release was published by hand before the
CD run could attach the assets, and an [immutable
release](https://docs.github.com/en/code-security/concepts/supply-chain-security/immutable-releases)
accepts none after it is published — so they were recovered from the release run's artifacts and
published at [`dist-v0.9.0`](https://github.com/moriyoshi/imbh/releases/tag/dist-v0.9.0), a
companion tag on the same commit. The release itself is otherwise complete. To use the recipe above
for 0.9.0, set `BASE=https://github.com/moriyoshi/imbh/releases/download/dist-v0.9.0`: the archive
names and `SHA256SUMS` are unchanged, and the binaries are byte-identical to the ones in the 0.9.0
[container image](#container-image).

**0.3.0 has no archives.** Its release run created the GitHub Release before uploading the assets;
under [immutable releases](https://docs.github.com/en/code-security/concepts/supply-chain-security/immutable-releases)
that publish froze it empty and reserved the `v0.3.0` tag name permanently, so the archives can
Expand Down Expand Up @@ -540,6 +550,15 @@ build when the tag's Release is already published. To recover from a failed CD r
jobs instead — the draft is reused and `--clobber` replaces a partial asset set. v0.3.0 lost its
Release to exactly this mistake, before the draft-first order was in place.

⚠️ **Never create the Release for a version tag ahead of CD either — or if you must, leave it a
draft.** The draft-first order protects the pipeline from itself; it cannot help when the Release
is already frozen before the run starts. v0.9.0 was published by hand shortly before its tag was
pushed, so every archive the run built had nowhere to go and `publish` failed at the preflight.
The workflow *fills and publishes an existing draft*, so a draft created early costs nothing. Those
archives were recovered from the run's artifacts and published under the companion tag
[`dist-v0.9.0`](https://github.com/moriyoshi/imbh/releases/tag/dist-v0.9.0), which only worked
because it happened inside the 90-day artifact retention window.

## FAQ

### Do I have to run a server?
Expand Down
Loading