Skip to content

Add a local-model provider alongside Gemini (--local) - #1

Merged
UditAkhourii merged 1 commit into
UditAkhourii:mainfrom
alisorcorp:local-provider
Aug 26, 2026
Merged

UditAkhourii merged 1 commit into
UditAkhourii:mainfrom
alisorcorp:local-provider

Conversation

@alisorcorp

Copy link
Copy Markdown
Contributor

Adds a second generation provider so cdaf generate can use a local multimodal model
instead of the Gemini Files API. Gemini stays the default — existing behaviour and
existing sidecars are unchanged.

cdaf generate ./clip.mp4 --local          # or --provider local
CDAF_PROVIDER=local cdaf generate ./clip.mp4

No API key, no per-clip cost, and the footage never leaves the machine. No new
dependencies
— stdlib urllib only. ffmpeg is required at runtime and checked for
explicitly.

Why it isn't just a swapped endpoint

The Files API path (upload the whole video, ask once) has no local equivalent, so the
provider samples frames itself. That forced three design choices which turned out to be
interesting on their own merits:

Boundaries come from the container, not the model. ffmpeg scene detection puts
segment boundaries on real cuts. On a synthetic 12s clip with cuts authored at exactly
3.0 / 6.0 / 9.0:

--scene-threshold detected
0.20 3.0
0.15 (default) 3.0, 6.0
0.06 3.0, 6.0, 9.0 — frame-exact

Precision is perfect; recall is threshold-dependent, which is why it's a flag rather
than a constant. A low-contrast transition can score below threshold and merge two shots.

Each shot is described without sight of the others. A describer that cannot see the
next shot cannot invent a causal chain across shots. This measurably suppresses a failure
mode I hit repeatedly with whole-video context: fluent, specific narration of an event the
footage deliberately withholds. More on that in the issue I'm filing separately.

Transcript words from the model, timing from measurement. The model transcribes
verbatim but times poorly — in my runs it snapped to round numbers, uniformly early
against measured onsets. So timing comes from a 300–3400 Hz RMS envelope instead.

One safety consequence worth calling out: when no speech is measured, the provider does
not ask the model. Handed a silent track, a model doesn't decline — it answers
conversationally, and "I am a large language model, trained by Google." lands in the
Transcript section looking like narration. Measured silence short-circuits ASR, and a
guard rejects replies that read as the assistant talking rather than transcribing.

Provenance in the header

Four x- keys (the prefix SPEC.md reserves for producers) so a consumer can tell
measured values from inferred ones:

x-shot-source: ffmpeg-scene-detect@0.06
x-shot-isolation: per-shot
x-continuity-pass: yes
x-transcript-timing: measured-rms

SKILL.md gains a section teaching agents to read those keys — and to distrust
completion claims and timestamps when they're absent.

Output

Same five-section body contract, validated by the existing CLI. From the synthetic clip
above:

## Segments
[00:00.0-00:03.0] The word 'ALPHA' is displayed in a white, stylized font against a solid blue background.
[00:03.0-00:06.0] Next, 'BRAVO' appears in a bold, white, rounded font against a solid red background.
[00:06.0-00:09.0] Then, 'CHARLIE' is shown in white, stylized capital letters centered on a solid green background.
[00:09.0-00:12.0] Finally, 'DELTA' is displayed in a white, stylized, slightly distorted font against a solid purple background.

## Transcript
(no speech)

Boundaries exact, all four labels read, silence correctly reported.

Tradeoffs, honestly

  • Slower per clip. Sequential per-shot calls plus local inference. Cost scales per
    shot rather than per second, so long footage is relatively cheaper here — but a
    short clip is faster via the API.
  • One clip at a time. No directory mode; the Gemini path keeps that.
  • A continuity pass runs after the isolated descriptions to restore pronouns and name
    objects that neighbouring shots make unambiguous — otherwise the subject is
    reintroduced from scratch every shot. It sees only text, never pixels, and is forbidden
    from adding actions or completions. Disable by passing continuity=False to
    describe_video_local if you'd rather ship the strictly literal output; happy to expose
    it as a CLI flag if you want it.
  • Reference target was Gemma 4 12B (its GGUF carries a combined vision+audio projector),
    but nothing is model-specific beyond the sampling defaults.

Testing

32 passed — the 10 existing tests plus 22 new ones covering shot construction,
truncated-JSON recovery, the non-transcript guard, header provenance, and provider
validation. All offline: no network, no ffmpeg, no model.

Also fixes cli/README.md, which still carried the pip install cdaf[generate] form
that 404s (the root README was fixed in 61dcde3; this one was missed).

Happy to split this, rename the flags, or drop the continuity pass — whatever fits how
you'd rather see providers organised.

cdaf generate --local (or --provider local, or CDAF_PROVIDER=local) generates a
sidecar via an OpenAI-compatible endpoint instead of the Gemini Files API. No API
key, no per-clip cost, and the footage never leaves the machine. Gemini remains the
default, so existing behaviour is unchanged.

The Files API path is not available locally, so the provider samples frames itself.
Three consequences worth noting, each measurable:

- Segment boundaries come from ffmpeg scene detection, so the boundaries it finds are
  frame-exact rather than model-inferred. Recall is threshold-dependent, exposed as
  --scene-threshold.
- Each shot is described without sight of the others, which prevents invented
  continuity between shots.
- Transcript words come from the model's audio encoder; timing is measured from a
  speech-band RMS envelope, because models transcribe accurately but time poorly.

Provenance is recorded in four x- header keys (reserved for producers by SPEC.md) so
a consumer can tell measured values from inferred ones.

No new dependencies: stdlib urllib only. ffmpeg is required at runtime.

Also fixes cli/README.md, which still had the pip form that 404s.
@UditAkhourii
UditAkhourii merged commit 2a51e11 into UditAkhourii:main Aug 26, 2026
7 checks passed
UditAkhourii added a commit that referenced this pull request Aug 27, 2026
The non-transcript guard from #1 substring-matched anywhere in the first
200 characters against a list including "sorry", "i cannot", "there is
no" and "trained by". Six plausible on-camera lines -- "Sorry, I'm
late", "I cannot believe how well this turned out", "There is no better
way to start the morning" -- were all discarded, silently: the transcript
vanishes and x-transcript-timing records none, so the sidecar looks
legitimately speechless.

Now two tiers. Assistant self-reference and explicit transcription
refusals still match anywhere. Generic refusal phrasings must both open
the reply and be paired with a reference to the recording, so ordinary
narration survives. All 7 real lines pass, all 6 chatbot replies blocked.

Adds test_local_body.py: drives describe_video_local end to end with
ffmpeg and the model mocked, asserting the assembled body parses as a
valid sidecar (including >1h timestamps and the brief/coarsen path). Its
first version passed for the wrong reason -- the mocked ffmpeg never
created the wav, so _transcribe bailed before the guard ran.

Merging #1 also left two overlapping trust sections in SKILL.md; they are
now one, keeping the PR's x-key table and phantom-brand-name warning.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants