Skip to content

docs(run-log): fix speed-from-fixed-framerate assumption, switch fram…#143

Merged
mbreiser merged 1 commit into
reiserlab:docs/run-log-encoding-optionsfrom
floesche:suggestion/docs/run-log-encoding-options
Jul 6, 2026
Merged

docs(run-log): fix speed-from-fixed-framerate assumption, switch fram…#143
mbreiser merged 1 commit into
reiserlab:docs/run-log-encoding-optionsfrom
floesche:suggestion/docs/run-log-encoding-options

Conversation

@floesche

@floesche floesche commented Jul 4, 2026

Copy link
Copy Markdown
Member
  • Corrects a wrong assumption in the run-log encoding proposal: per-frame speed can't be derived from an assumed fixed camera framerate, since frame-to-frame spacing varies. FicTrac's own per-frame timestamp (column 22) must be recorded instead and used for actual framerate calculation.
  • Replaces the previously-proposed raw delta-rotation fields (columns 6-8) with integrated position (columns 15-16) and pairs integrated heading (17) with movement direction (18), since the two diverge during sideways slip.
  • Restructures the proposal around two logging levels — minimal (ms, fc, index, rejoinable with FicTrac's own recording via the frame counter) and verbose (adds ts, x, y, hd, dir) — dropping FicTrac's separate sequence counter as redundant for drop detection.
  • Updates the camera-rate assumption from 50 Hz to ~100 Hz nominal and recomputes all file-size estimates; drops the gzip encoding option.
  • Switches frame-row encoding from bare CSV to positional JSON arrays, making the whole log valid NDJSON (uniform JSON.parse() + type dispatch, no custom line-format sniffing) for a ~2 bytes/frame cost.

…e rows to NDJSON arrays

- Require FicTrac's own per-frame timestamp (col 22, `ts`) instead of
  assuming a constant camera framerate for speed/turning-rate calculations
- Replace raw delta-rotation fields (cols 6-8) with integrated position
  (cols 15-16 `x`/`y`); keep heading (17) paired with movement direction
  (18) since they diverge during sideways slip
- Define two logging levels: minimal (`ms, fc, index`) for correlating
  with FicTrac's own recording via frame counter, verbose (+ `ts, x, y,
  hd, dir`) for turning + forward locomotion inline
- Drop FicTrac's separate sequence counter (col 23) — frame counter alone
  is sufficient for drop detection
- Update camera rate assumption to ~100 Hz nominal and recompute all size
  estimates; drop the gzip option
- Encode frame rows as positional JSON arrays instead of bare CSV so the
  whole log is valid NDJSON, dispatched by parsed type instead of a
  custom line-sniffing parser
@floesche

floesche commented Jul 4, 2026

Copy link
Copy Markdown
Member Author

This is my response to #141. I didn't want to overwrite your files, so feel free to modify what you want based on this.

@mbreiser mbreiser merged commit 69f7a70 into reiserlab:docs/run-log-encoding-options Jul 6, 2026
mbreiser added a commit that referenced this pull request Jul 6, 2026
Two-level NDJSON FicTrac run-log encoding for the course pipeline (minimal/verbose), a measured per-field byte audit, col-24 timing, fixed-decimal rounding, and resolved review questions. Incorporates Frank's revision (#143).

Co-authored-by: Frank Loesche <loeschef@janelia.hhmi.org>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@mbreiser

mbreiser commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Merged this — thanks, @floesche. The reframing is spot on and I kept all of it: bridge-not-replacement, the minimal level with fc as the join key, uniform NDJSON positional arrays, and hd+dir together for slip.

Two things I layered on in a follow-up (60acb6c), flagging since one revises your timing choice:

  1. Timing: switched ts (col 22) → dt (col 24, "ms since previous frame"). col 24 is the inter-frame interval directly, so speed = hypot(dx,dy)/dt needs no differencing and dodges the col-22 "video-position-or-epoch" ambiguity you flagged in Q1. Noted col 19 (FicTrac's own speed) and col 25 (capture time) as available too. Easy to revert to col 22 if you'd rather — say the word.
  2. Byte audit (Python json.dumps over a 90k-frame run): confirms your sizes (verbose ~58 B/frame ≈ 5.25 MB, minimal ~20 B ≈ 1.8 MB), and shows col 24 saves ~1 MB vs col 22 + that compact separators (separators=(",", ":")) are mandatory.

Still genuinely open (your Q1, now the only blocker for verbose): does our FicTrac build populate cols 24 & 25? Some configs leave them 0, and the sim can't tell us (it fakes a regular col 24). If they're empty on the real rig we fall back to your col-22-stored-as-is + differencing. Could you check a raw FicTrac line when you're next at a bench?

Full write-up now on main: docs/development/run-log-encoding-options.md. Decisions + implementation checklist tracked in #140.

@floesche

floesche commented Jul 6, 2026

Copy link
Copy Markdown
Member Author

Just quickly (and I can do the change later), but keeping dt (col 24) is wrong. If a frame is being skipped on the receiving end, you wouldn't be able to calculate the real time difference with 24. I suggest using 22 instead and calculating the difference in the bridge or during post processing.

This still also solve the follow up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants