Run-log: curated FicTrac column subset (middle ground) + optional size reduction

[mbreiser]

Motivation

Bridge run logs (JSONL committed to the course repo) currently offer two extremes per FicTrac frame (fictrac-bridge/bridge.py write_frame):

• default — just {"type":"fictrac_frame","seq":…,"index":…,"t":…} (the arena index + timing; raw tracking discarded). ~65 B/line, ~196 KB per 50 s run.
• --log-frames — the full ~25-column FicTrac record appended as "fictrac":[…]. ~3–4× bigger (~8 MB per 10 min run).

We want a middle ground: keep a few scientifically-useful columns, not all 25, and not just seq/index/t. The index alone lets you recover heading (via gain) but permanently loses position/velocity/direction — unrecoverable after the fact.

Ask 1 — curated FicTrac column subset

Log a bounded, configurable subset. Candidate default set:

• integrated animal heading (col 17)
• integrated x / y position (cols 15–16)
• animal movement direction (col 18) + speed (col 19)

Implementation sketch:

• In Pipeline.handle_line the full fields list is already parsed (~line 254); write_frame decides what to store (~line 221).
• Add a --log-fields option (named preset like curated/full/min, or an explicit column list) so the bench chooses the payload. Store curated fields under short, stable keys (e.g. hd,x,y,dir,spd) so downstream parsing is simple.
• Keep --log-frames (= full) working.

Ask 2 — optional log size reduction (related)

Logs are plain JSONL with heavily repeated keys → very compressible. Options, best-first:

1. Gzip before commit (.jsonl.gz): ~5–10× lossless (browser CompressionStream('gzip') or bridge-side). Trades GitHub inline-preview for size; analysis tools read gzip natively. Turns the 60 MB worst case into ~6–8 MB.
2. Curated columns (Ask 1) — fewer floats/frame.
3. Terser schema — drop the constant "type":"fictrac_frame", short keys, or a columnar frame block with one header row (~40–60%, stays text).
4. Decimate — every Nth frame / on-change (loses temporal resolution).
5. Binary packing — smallest, opaque, needs a parser (last resort).

Recommendation

Ship the curated subset as the course default, keep small runs as human-readable JSONL, and add opt-in gzip for long runs. Revisit only if real course logs approach the ~35 MB GitHub Contents-API ceiling.

Context

• Shipped in Arena Studio v0.5: CSHL course data pipeline (Session 2) #139 (Arena Studio v0.5 course pipeline). Bench-verified: ~196 KB / 50 s default run.
• GitHub Contents-API commit ceiling ≈ 35 MiB/file (measured); bridge→browser WS export tested to 50 MB.
• Per-run rotation already scopes each committed log to one experiment.

[mbreiser]

Decision (2026-07-04): go with options 2 + 3 combined — curated columns in a terser schema, kept human-readable; gzip stays optional.

Rationale: this keeps the committed logs plain JSONL (previewable on GitHub, trivial to parse) while carrying the behavioral payload, at a modest size. ~50% smaller than the full-25 --log-frames option — and gzip (option 1) is deliberately not the default because we'd rather keep readability; it can still be layered on later for an unusually long run.

Concrete target for the implementer:

• Curated fields (default preset): hd heading, x/y integrated position, dir movement direction, spd speed — plus the existing seq/index/t.
• Terser encoding for the dominant frame rows: drop the constant "type":"fictrac_frame" (frames are the default row type — e.g. mark only the non-frame session/runner rows with a type, or use a 1-char discriminator), and use short keys (f seq, i index, t, hd, x, y, dir, spd). Round floats to a sane precision (e.g. 2–3 decimals) to trim further.
• Keep --log-frames (= full, all columns) and add a min preset (today's seq/index/t) so all three payload levels are selectable.
• Note vs. today: curated is slightly larger per line than the current bare default (it adds data), but far smaller than full; the terse keys keep the per-line overhead low.
• Document the frame schema (a header/legend line, or the README) so the short keys are self-describing for analysis.