Repository navigation
Add a local-model provider alongside Gemini (--local) - #1
Merged
Merged
Conversation
cdaf generate --local (or --provider local, or CDAF_PROVIDER=local) generates a sidecar via an OpenAI-compatible endpoint instead of the Gemini Files API. No API key, no per-clip cost, and the footage never leaves the machine. Gemini remains the default, so existing behaviour is unchanged. The Files API path is not available locally, so the provider samples frames itself. Three consequences worth noting, each measurable: - Segment boundaries come from ffmpeg scene detection, so the boundaries it finds are frame-exact rather than model-inferred. Recall is threshold-dependent, exposed as --scene-threshold. - Each shot is described without sight of the others, which prevents invented continuity between shots. - Transcript words come from the model's audio encoder; timing is measured from a speech-band RMS envelope, because models transcribe accurately but time poorly. Provenance is recorded in four x- header keys (reserved for producers by SPEC.md) so a consumer can tell measured values from inferred ones. No new dependencies: stdlib urllib only. ffmpeg is required at runtime. Also fixes cli/README.md, which still had the pip form that 404s.
UditAkhourii
added a commit
that referenced
this pull request
Aug 27, 2026
The non-transcript guard from #1 substring-matched anywhere in the first 200 characters against a list including "sorry", "i cannot", "there is no" and "trained by". Six plausible on-camera lines -- "Sorry, I'm late", "I cannot believe how well this turned out", "There is no better way to start the morning" -- were all discarded, silently: the transcript vanishes and x-transcript-timing records none, so the sidecar looks legitimately speechless. Now two tiers. Assistant self-reference and explicit transcription refusals still match anywhere. Generic refusal phrasings must both open the reply and be paired with a reference to the recording, so ordinary narration survives. All 7 real lines pass, all 6 chatbot replies blocked. Adds test_local_body.py: drives describe_video_local end to end with ffmpeg and the model mocked, asserting the assembled body parses as a valid sidecar (including >1h timestamps and the brief/coarsen path). Its first version passed for the wrong reason -- the mocked ffmpeg never created the wav, so _transcribe bailed before the guard ran. Merging #1 also left two overlapping trust sections in SKILL.md; they are now one, keeping the PR's x-key table and phantom-brand-name warning. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second generation provider so
cdaf generatecan use a local multimodal modelinstead of the Gemini Files API. Gemini stays the default — existing behaviour and
existing sidecars are unchanged.
cdaf generate ./clip.mp4 --local # or --provider local CDAF_PROVIDER=local cdaf generate ./clip.mp4No API key, no per-clip cost, and the footage never leaves the machine. No new
dependencies — stdlib
urllibonly.ffmpegis required at runtime and checked forexplicitly.
Why it isn't just a swapped endpoint
The Files API path (upload the whole video, ask once) has no local equivalent, so the
provider samples frames itself. That forced three design choices which turned out to be
interesting on their own merits:
Boundaries come from the container, not the model. ffmpeg scene detection puts
segment boundaries on real cuts. On a synthetic 12s clip with cuts authored at exactly
3.0 / 6.0 / 9.0:
--scene-thresholdPrecision is perfect; recall is threshold-dependent, which is why it's a flag rather
than a constant. A low-contrast transition can score below threshold and merge two shots.
Each shot is described without sight of the others. A describer that cannot see the
next shot cannot invent a causal chain across shots. This measurably suppresses a failure
mode I hit repeatedly with whole-video context: fluent, specific narration of an event the
footage deliberately withholds. More on that in the issue I'm filing separately.
Transcript words from the model, timing from measurement. The model transcribes
verbatim but times poorly — in my runs it snapped to round numbers, uniformly early
against measured onsets. So timing comes from a 300–3400 Hz RMS envelope instead.
One safety consequence worth calling out: when no speech is measured, the provider does
not ask the model. Handed a silent track, a model doesn't decline — it answers
conversationally, and
"I am a large language model, trained by Google."lands in theTranscript section looking like narration. Measured silence short-circuits ASR, and a
guard rejects replies that read as the assistant talking rather than transcribing.
Provenance in the header
Four
x-keys (the prefix SPEC.md reserves for producers) so a consumer can tellmeasured values from inferred ones:
SKILL.mdgains a section teaching agents to read those keys — and to distrustcompletion claims and timestamps when they're absent.
Output
Same five-section body contract, validated by the existing CLI. From the synthetic clip
above:
Boundaries exact, all four labels read, silence correctly reported.
Tradeoffs, honestly
shot rather than per second, so long footage is relatively cheaper here — but a
short clip is faster via the API.
objects that neighbouring shots make unambiguous — otherwise the subject is
reintroduced from scratch every shot. It sees only text, never pixels, and is forbidden
from adding actions or completions. Disable by passing
continuity=Falsetodescribe_video_localif you'd rather ship the strictly literal output; happy to exposeit as a CLI flag if you want it.
but nothing is model-specific beyond the sampling defaults.
Testing
32 passed— the 10 existing tests plus 22 new ones covering shot construction,truncated-JSON recovery, the non-transcript guard, header provenance, and provider
validation. All offline: no network, no ffmpeg, no model.
Also fixes
cli/README.md, which still carried thepip install cdaf[generate]formthat 404s (the root README was fixed in 61dcde3; this one was missed).
Happy to split this, rename the flags, or drop the continuity pass — whatever fits how
you'd rather see providers organised.