From ba3d7564b394821fc92de70a37e21ef0734686ff Mon Sep 17 00:00:00 2001 From: Frank Loesche Date: Sat, 4 Jul 2026 16:38:06 -0400 Subject: [PATCH] docs(run-log): fix speed-from-fixed-framerate assumption, switch frame rows to NDJSON arrays MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Require FicTrac's own per-frame timestamp (col 22, `ts`) instead of assuming a constant camera framerate for speed/turning-rate calculations - Replace raw delta-rotation fields (cols 6-8) with integrated position (cols 15-16 `x`/`y`); keep heading (17) paired with movement direction (18) since they diverge during sideways slip - Define two logging levels: minimal (`ms, fc, index`) for correlating with FicTrac's own recording via frame counter, verbose (+ `ts, x, y, hd, dir`) for turning + forward locomotion inline - Drop FicTrac's separate sequence counter (col 23) — frame counter alone is sufficient for drop detection - Update camera rate assumption to ~100 Hz nominal and recompute all size estimates; drop the gzip option - Encode frame rows as positional JSON arrays instead of bare CSV so the whole log is valid NDJSON, dispatched by parsed type instead of a custom line-sniffing parser --- docs/development/run-log-encoding-options.md | 253 +++++++++++-------- 1 file changed, 146 insertions(+), 107 deletions(-) diff --git a/docs/development/run-log-encoding-options.md b/docs/development/run-log-encoding-options.md index a787ab7..2ec13f6 100644 --- a/docs/development/run-log-encoding-options.md +++ b/docs/development/run-log-encoding-options.md @@ -5,127 +5,166 @@ ## TL;DR The bench run-log (FicTrac-bridge JSONL, auto-committed per run to -`runlogs//` in the course repo) should carry **turning behaviour + -forward locomotion** without bloating. Recommendation: log **4 curated FicTrac -fields** (per-frame side / forward / yaw + integrated heading) as **compact CSV -frame rows** (relative-ms timestamps, 4–5 significant-figure floats) interleaved -with JSON event rows. That's **~1.8 MB for a 15-minute run** — ~90 % smaller than -dumping all 25 columns as JSON, and even smaller than today's (behaviour-less) -default — while staying human-readable and self-describing. +`runlogs//` in the course repo) exists to correlate **what the fly did** +with **what the arena displayed** — FicTrac itself already records everything +about the fly's behaviour independently, so this log's job is the *bridge* +between the two, not a replacement for FicTrac's own file. + +Two logging levels, both as **NDJSON**: every line is a valid JSON value — +frame rows are compact **JSON arrays** (positional, per a declared +`frame_schema`, relative-ms timestamps), interleaved with JSON object event +rows. A reader dispatches on the parsed type (`Array` → frame, `Object` → +event) — no custom line-format sniffing needed. + +- **Minimal** — `[ms, fc, index]`: bridge timestamp, FicTrac frame counter, + arena displayed frame. Enough to rejoin with FicTrac's own recording later + via `fc`. **~1.68 MB for a 15-minute run** at 100 Hz. +- **Verbose** — minimal + `ts, x, y, hd, dir`: FicTrac's own per-frame timestamp + plus integrated position/heading, so turning + forward locomotion can be read + straight out of the bridge log without a join. **~5.4 MB for a 15-minute run** + at 100 Hz. + +No gzip — committed logs stay plain text on GitHub. ## Context - **One committed log = one run.** The bridge rotates a fresh file at each run - start, so each file is scoped start-to-finish (bench-verified 2026-07-04). -- **GitHub ceiling ≈ 35 MiB/file** (Contents API; measured). Anything under a few - MB is comfortable. -- **Frame rate ≈ 50 Hz** on the current bench (measured: 2506 frames / 50.1 s). - A 15-min run ≈ **45,000 frames**. **Frame rows are ~99 % of the bytes**; - session/runner events are a handful. -- All sizes below are estimates at 50 Hz. **Double the camera rate → double the - file.** - -## The behavioural fields to keep - -Goal: turning + forward locomotion. FicTrac's **delta rotation vector in lab -coordinates (cols 6–8)** gives per-frame velocities on exactly these axes; -**integrated heading (col 17)** gives absolute orientation. - -| key | FicTrac col (1-based) | `fields[]` idx | meaning | unit | -| ------ | --------------------- | -------------- | ---------------------------- | --------- | -| `side` | 6 | 5 | sideways / slip velocity | rad/frame | -| `fwd` | 7 | 6 | **forward** velocity | rad/frame | -| `yaw` | 8 | 7 | **turning** velocity | rad/frame | -| `hd` | 17 | 16 | integrated heading (facing) | rad | - -**Scaling for analysis:** forward mm/s = `fwd × ball_radius × fps`; turning °/s = -`yaw × fps × 57.3`. The axis mapping (x = side, y = forward, z = yaw) is FicTrac's -standard lab-frame convention — worth a 10-second bench check (walk the ball -straight → `fwd` dominates; spin in place → `yaw` dominates) to confirm signs -before the course. - -### Every frame row also carries identity + timing - -The 4 fields above are the *behaviour*. Each frame row keeps them **in addition -to** the frame's identity + timing — **not instead of**: - -| key | source | meaning | -| ------- | ------------- | ------------------------------------------------------------------------------------------------------------------ | -| `ms` | bridge clock | timestamp, stored as **relative ms since run start** (the absolute epoch start is logged once in `run_metadata`) | -| `index` | arena | the frame index the controller **displayed** at that instant — needed to correlate behaviour with what the fly saw | -| `seq` | FicTrac | FicTrac's own frame counter — lets you detect **dropped frames**; the one field you may drop (~8 B) if you don't need that QC | - -So a full frame row is **`ms, seq, index, side, fwd, yaw, hd`** (7 values). The -size estimates below already include `index` + timestamp; `seq` adds ~8 B/frame -(~0.35 MB over a 15-min run) if kept. - -Optional add-ons: cols 15–16 (integrated x/y position) for the 2-D walking path; -the full 25 columns for archival/QC. - -## Encoding options — sizes for a 15-min run (45,000 frames @ 50 Hz) - -| # | Scheme | data kept | ~B/frame | 15-min file | vs #0 | readable | -| ----- | -------------------------------------------------- | ---------------------- | -------- | ----------- | ----- | ----------- | -| 0 | Full 25 cols — verbose JSON (`--log-frames` today) | everything | ~350 | **~16 MB** | — | yes | -| 1 | Full 25 cols — CSV rows + JSON events | everything | ~260 | ~12 MB | −26 % | yes | -| 2 | Curated 4 — verbose JSON | turn + fwd + slip + hd | ~125 | ~5.6 MB | −64 % | yes | -| 3 | Curated 4 — short-key JSON (drop `"type"`) | same | ~85 | ~3.8 MB | −76 % | ~ | -| 4 | Curated 4 — CSV rows + JSON events (abs ms) | same | ~53 | ~2.4 MB | −85 % | yes+schema | -| **5★**| **Curated 4 — CSV + relative-ms + rounded floats** | same | **~40** | **~1.8 MB** | −89 % | yes+schema | -| 6 | #5 + gzip (`.jsonl.gz`) | same | ~11 | ~0.5 MB | −97 % | no (opaque) | -| — | _today's default (no behaviour data)_ | seq / index / t | ~66 | ~3.0 MB | — | yes | - -**Two levers do most of the work** (independent of the row format): - -- **Relative time** — store ms-since-run-start (the epoch start is logged once in - `run_metadata`) instead of the 13-digit absolute epoch. Saves ~7–8 B/frame — - the single biggest field. -- **Float rounding** — FicTrac deltas are tiny; 4–5 significant figures is ample - behavioural resolution. - -**gzip** layers on top of _any_ row format (≈ 3–4× on CSV, ≈ 7× on verbose JSON) -as an escape hatch for an unusually long run — at the cost of GitHub inline -preview. - -## Recommended scheme (#5) - -Interleave: **JSON** for the few session/runner event rows, **bare CSV** for the -frame rows, with a one-time schema line. Parser rule: a line starting with `{` is -a JSON event; otherwise it's a CSV frame per the declared schema. + start, so each file is scoped start-to-finish. +- **GitHub ceiling ≈ 35 MiB/file** (Contents API; measured). Both levels above + are comfortably under it even at several times the assumed run length. +- **Camera runs at ~100 Hz nominal, but frame-to-frame spacing is not constant.** + You cannot recover per-frame speed by assuming a fixed framerate and dividing + by frame count — the actual inter-frame interval has to come from FicTrac's + own per-frame timestamp (column 22). This is why `ts` is a required verbose + field, not an optional QC add-on. +- **Frame rows are ~99% of the bytes**; session/runner events are a handful. +- All sizes below are estimates at 100 Hz over a 15-minute run (90,000 frames). + Double the camera rate → double the file. + +## Fields + +### Both levels: identity + arena correlation + +| key | source | FicTrac col | meaning | +| ------- | ------------- | ----------- | ------------------------------------------------------------------------------- | +| `ms` | bridge clock | — | timestamp, stored as **relative ms since run start** (epoch start logged once in `run_metadata`) | +| `fc` | FicTrac | 1 | FicTrac's frame counter — the join key back to FicTrac's own independent recording | +| `index` | arena | — | the frame index the controller **displayed** at that instant — needed to correlate behaviour with what the fly saw | + +`fc` alone is sufficient to detect dropped frames (it increments by 1 per +camera frame); FicTrac's separate sequence counter (column 23) is not logged +here. + +### Verbose adds: timing + behaviour + +| key | FicTrac col | meaning | unit | +| ----- | ----------- | ------------------------------------- | ---- | +| `ts` | 22 | FicTrac's own per-frame timestamp — the actual inter-frame interval, not an assumed one | ms | +| `x` | 15 | integrated x position, lab coords | rad (scale by ball radius) | +| `y` | 16 | integrated y position, lab coords | rad (scale by ball radius) | +| `hd` | 17 | integrated heading (facing direction) | rad | +| `dir` | 18 | instantaneous movement direction | rad | + +`x`/`y` (integrated position) replace the raw per-frame delta-rotation vector +(columns 6–8): the integrated path is more directly useful for reconstructing +the 2-D walking trace, and forward/turning velocities can be recovered from +successive `x`/`y`/`hd` samples divided by the actual `ts` delta — see below. +`hd` and `dir` are recorded together because they answer two different +questions: `hd` is which way the fly is **facing**, `dir` is which way it's +**actually moving** — the two diverge during sideways slip, which is exactly +the behaviour this log needs to be able to see. + +**Deriving velocities (verbose only):** for consecutive rows `i-1, i`: + +``` +dt_s = (ts[i] - ts[i-1]) / 1000 # actual inter-frame interval +dx, dy = x[i]-x[i-1], y[i]-y[i-1] # rad +speed_mm = hypot(dx, dy) * ball_radius_mm / dt_s +turn_dps = deltaAngle(hd[i], hd[i-1]) * 57.3 / dt_s # unwrap across ±π +``` + +Never substitute a nominal 100 Hz for `dt_s` — that's the assumption this +scheme exists to avoid. + +## Encoding options — sizes for a 15-min run (90,000 frames @ 100 Hz nominal) + +| Scheme | fields | ~B/frame | 15-min file | readable | +| ---------------------------------------------- | -------------------------------- | -------- | ----------- | ---------- | +| Today's default (verbose JSON, no behaviour) | `seq`/`index`/`t` per frame | ~66 | ~5.9 MB | yes | +| Full 25 cols — verbose JSON (archival/QC only) | everything | ~350 | ~31.5 MB | yes, but near the 35 MB ceiling | +| **Minimal — JSON array + relative ms** | `[ms, fc, index]` | **~19** | **~1.68 MB** | yes+schema | +| **Verbose — JSON array + relative ms** | `[ms, fc, index, ts, x, y, hd, dir]` | **~60** | **~5.4 MB** | yes+schema | + +**Three levers do most of the work** (independent of which level you pick): + +- **Relative time** — store `ms` as ms-since-run-start (the epoch start is + logged once in `run_metadata`) instead of the 13-digit absolute epoch. This + is the single biggest per-field saving. +- **Float rounding** — FicTrac's position/heading fields don't need more than + 4–5 significant figures for behavioural resolution. +- **Positional arrays over keyed objects** — a `[...]` row per the declared + `frame_schema` costs only 2 bytes (the brackets) over bare CSV, vs. repeating + every field's key name on every row (the "today's default" and "full 25 + cols" rows above). + +## Recommended scheme + +**Uniform NDJSON** — every line is one JSON value: session/runner events are +JSON **objects**, frame rows are JSON **arrays** (positional, per the +one-time `frame_schema` line declaring which level is in use and column +order). A reader does one `JSON.parse()` per line and dispatches on the +parsed type — `Array.isArray(v)` → frame row indexed against +`frame_schema.cols`, else → event object. No line-format sniffing, no +non-JSON lines anywhere in the file. + +Minimal: + +```jsonl +{"type":"logging_started","file":"arena-log-20260704-134533-129.jsonl","ms":1783187133129} +{"type":"run_metadata","run_id":"nkts77qj","experimenter":"michael","genotype":"Canton-S","rig_id":"bench01","protocol_filename":"g6_2x10_smoke.yaml","t0_ms":1783187133129,"ball_radius_mm":4.5,"fps":100} +{"type":"frame_schema","level":"minimal","cols":["ms","fc","index"]} +[0,1054712,64] +[10,1054713,64] +[20,1054714,65] +{"type":"runner","phase":"sequence-complete","rx_ms":1783187183251} +{"type":"logging_stopped","ms":1783187183252} +``` + +Verbose: ```jsonl {"type":"logging_started","file":"arena-log-20260704-134533-129.jsonl","ms":1783187133129} -{"type":"run_metadata","run_id":"nkts77qj","experimenter":"michael","genotype":"Canton-S","rig_id":"bench01","protocol_filename":"g6_2x10_smoke.yaml","t0_ms":1783187133129,"ball_radius_mm":4.5,"fps":50} -{"type":"frame_schema","cols":["ms","seq","index","side","fwd","yaw","hd"]} -0,1054712,64,-0.0123,0.0345,0.0082,1.2345 -20,1054713,64,-0.0110,0.0361,0.0075,1.2352 -40,1054714,65,-0.0098,0.0357,0.0069,1.2359 +{"type":"run_metadata","run_id":"nkts77qj","experimenter":"michael","genotype":"Canton-S","rig_id":"bench01","protocol_filename":"g6_2x10_smoke.yaml","t0_ms":1783187133129,"ball_radius_mm":4.5,"fps":100} +{"type":"frame_schema","level":"verbose","cols":["ms","fc","index","ts","x","y","hd","dir"]} +[0,1054712,64,1783187133129.4,12.3456,-4.5678,1.2345,1.1987] +[10,1054713,64,1783187133139.1,12.3501,-4.5701,1.2358,1.1990] +[20,1054714,65,1783187133149.6,12.3549,-4.5719,1.2371,1.1994] {"type":"runner","phase":"sequence-complete","rx_ms":1783187183251} {"type":"logging_stopped","ms":1783187183252} ``` -- **Self-describing** — `frame_schema` gives column order; `run_metadata` carries - the epoch start, ball radius, and fps needed to convert to real units. -- **Streamable** — the bridge writes each frame row as it arrives (no buffering). -- **~2.2 MB / 15-min run** with `seq` kept (~1.8 MB — table row #5 — if `seq` is - dropped), far under the 35 MB ceiling even at 2× the frame rate or 3× the - duration. +- **Self-describing** — `frame_schema` gives the level and column order; + `run_metadata` carries the epoch start, ball radius, and nominal fps needed + to convert to real units. +- **Streamable** — the bridge writes each frame row as it arrives (no + buffering). +- **Tool-compatible** — the whole file is valid NDJSON; any generic NDJSON/JSON-Lines + reader can consume it line-by-line without a bespoke parser. ## Open questions -1. **Fields** — are `side / fwd / yaw / hd` the right minimal set, or do you also - want raw delta-rotation-cam (cols 2–4) or absolute orientation (cols 12–14) - for QC? -2. **Frame rate** — confirm the course cameras run ~50 Hz (drives every size). - At 100 Hz, #5 is ~3.6 MB/15-min (still fine); full JSON (#0) would be ~32 MB, - right at the ceiling. -3. **Archival** — keep a `full` (all-25) option alongside `curated` in case - reprocessing later needs a dropped column? -4. **gzip** — acceptable to opt into `.jsonl.gz` for long runs, or must every - committed log stay plain text on GitHub? -5. **Precision** — OK to round floats to ~5 sig figs? (`seq`, `index`, and the - timestamp are all **kept** per frame alongside the 4 behavioural fields; `seq` - is the only droppable one, ~8 B/frame, if dropped-frame QC isn't needed.) +1. **`ts` clock domain** — FicTrac's column-22 comment says "video position or + epoch ms" depending on install/config. Need a bench check: if it's epoch ms + on the same clock as the bridge, `ts` can be relativized against the same + `t0_ms` as `ms` (shrinking it the same way); if it's a video-file position, + it must be stored as-is and only differences (`ts[i]-ts[i-1]`) are + meaningful. +2. **`ts`/`x`/`y` rounding** — FicTrac emits these as 6-decimal floats. OK to + round `ts` to ~0.1 ms and `x`/`y`/`hd`/`dir` to ~4–5 significant figures for + the committed log? +3. **Archival** — keep a `full` (all-25-column) option available on request + for reprocessing/QC, alongside `minimal`/`verbose`, given it sits close to + the GitHub file-size ceiling at 100 Hz? ## References