Skip to content

feat: Add Whisper speech-to-text captioning for video and audio (.srt) (Continues #774) - #788

Merged
KnifeLemon merged 5 commits into
KnifeLemon:mainfrom
rohitLovesKrishna:feature/video-caption
Oct 30, 2026
Merged

KnifeLemon merged 5 commits into
KnifeLemon:mainfrom
rohitLovesKrishna:feature/video-caption

Conversation

@rohitLovesKrishna

Copy link
Copy Markdown
Contributor
2026-10-30.09-22-30_compressed.mp4

Hi @KnifeLemon ,

Continues #774
Thank you for the detailed and constructive review! All requested points have been fully addressed:

Summary of Changes:

  1. Preserved Translations & i18n:

    • Rebased cleanly onto upstream main (028390c3).
    • Retained all existing keys (crop.perspective*, crop.corner_*) and preserved original indentation across all 18 i18n/*.json files.
    • Translated the 8 new keys for all 18 supported languages.
  2. Clean Engine Paths & FFmpeg Integration:

    • Removed all external app paths. WhisperConverter now strictly searches FindBundled, OwnCopyFolder, the CLI folder, and system PATH.
    • FFmpeg is resolved exclusively via FfmpegConverter.Locate(). Throws a clean error if missing; removed bare "ffmpeg" fallback.
  3. Multi-language Support (Spoken Language Option):

    • Removed UrduToDevanagari.cs.
    • Added SpokenLanguage (defaults to "auto") to MediaOptions, passed directly to whisper.cpp via -l <lang>.
  4. Pinned Model Download:

    • Pinned the ggml-base.bin download in engines.json to immutable commit SHA 5359861c739e955e79d9a303bcbc70fb988958b1.
  5. Cross-Platform Handling (Windows-only Gate):

    • Whisper package in engines.json and the Donut Caption slice are gated to Windows (OperatingSystem.IsWindows()).
    • On macOS and Linux, WhisperConverter.GetStatus() cleanly reports engine.reason.unsupported_platform.
  6. Scoped Strictly to SRT (Removed Burn-in):

    • Removed the video → MP4 burn-in edge (Cost: 14) and the unflagged audio edge.
    • Updated XML doc comments in IConverter.cs and RoutePlanner.cs to reflect SRT transcription only.
  7. Clean Schema Migration:

    • Simplified AddCaptionPreset in UserDataStore.cs to standard migration semantics: inserts preset, offers to video and audio profiles.
    • Added unit tests in CaptionRouteTests.cs verifying a v56 library learns the preset once on both donuts.
  8. Notices & Docs:

    • Added whisper.cpp (MIT) and GGML Whisper model (MIT) to THIRD-PARTY-NOTICES.md.
    • Added Whisper section to docs/ENGINES.md detailing size (~157 MB with base model), offline operation, and FFmpeg requirement.

Evidence & Sample Subtitles:

1. English Sample (test_en.srt):

Command:

whisper-cli.exe -m ggml-base.bin -f clip_en.wav -osrt -of clip_en -l en -t 4

Output:

1
00:00:00,000 --> 00:00:04,500
Welcome to Filee, the offline file converter for Windows, Mac, and Linux.

2. Hindi Sample (test_hi.srt):

Command:

whisper-cli.exe -m ggml-base.bin -f sant.wav -osrt -of sant -l hi -t 4

Output:

1
00:00:00,000 --> 00:00:14,200
आप सब कुछ कर सकते हैं। यह सब को ज्ञात है कि किसी किसी के भाग्य में लिखा होता है, आप अपना भाग्य बदल सकते हैं।

(Attaching screenshots of Settings → Engines and the Donut with the Caption slice below).
Screenshot 2026-10-30 091635
Screenshot 2026-10-30 091626
Screenshot 2026-10-30 091609

@github-actions github-actions Bot added the feature New features label Oct 30, 2026

@KnifeLemon KnifeLemon left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @rohitLovesKrishna! That was a fast and thorough round. The translations are intact now, the outside paths and the transliteration are gone, the model is pinned, and the scope is SRT only. I've approved the CI run.

A few things are left before this can go in:

1. Spoken language needs a control

SpokenLanguage exists, but nothing in the app sets it. Today it can only be changed by editing presets.json. Please add a picker to the preset editor, under the Caption checkbox:

  • Auto (the default);
  • the languages whisper.cpp knows, shown by name.

2. No platform checks in Filee.Core

BuiltInData and the v57 migration branch on OperatingSystem.IsWindows(). Built-in data and migrations have to be the same everywhere: a settings folder moves between machines, and the Core tests run on all three systems.

  • Keep the preset and the slices the same on every platform.
  • Let PresetAvailability mark the slice unusable on macOS and Linux, as it already does.

3. Wording

  • Not built in: "Whisper (built-in AI)" / "Whisper (내장 AI)" isn't right, because it's a ~157 MB download. Plain "Whisper" is enough.
  • Avoid "any language": whisper.cpp knows about 100.
  • Korean uses the polite 해요 style throughout ko.json ("…바꿔요"), not 합니다. For example:
    • engines.package.whisper.description: 동영상과 오디오의 말소리를 SRT 자막 파일로 바꿔요. 인터넷 연결 없이 작동해요.

4. Long files

  • Fixed timeouts: 10 minutes for the FFmpeg extraction and 30 minutes for Whisper are fixed. A two-hour video with the base model on a CPU can take longer than that. Use StallWatch like FFmpeg and calibre do: let it run while it keeps printing progress (-pp), and report that progress instead of the fixed 0.20 / 0.85.
  • Threads: -t 4 should follow the machine (Environment.ProcessorCount, capped sensibly).

5. Results through Filee

The SRT samples in the description come from running whisper-cli.exe by hand. Please show them made by Filee: the Caption slice on the donut, or

filee convert clip.mp4 --preset caption-video

Include one English clip and one other language, with the language picker set once to Auto and once to the language itself.

Thanks again! Tag me when it's ready.

@rohitLovesKrishna

Copy link
Copy Markdown
Contributor Author
2026-10-30.12-52-55_compressed.mp4

Thanks for the thorough review, @KnifeLemon ! I've addressed all the points:

1. Spoken language control in Preset Editor

  • Added a spoken language dropdown under the Caption checkbox in the Preset Editor.
  • Defaults to Auto, followed by all 100 languages supported by whisper.cpp sorted by display name.
  • Saved directly to preset.media.spokenLanguage and passed via -l <lang> to whisper-cli.

2. Platform neutrality in Filee.Core

  • Removed OperatingSystem.IsWindows() checks from BuiltInData.cs (video & audio profile slices) and UserDataStore.cs (migration 57).
  • Presets and slices are now identical across Windows, macOS, and Linux. PresetAvailability handles marking the Caption slice unusable on non-Windows platforms (showing engine.reason.unsupported_platform), matching other Windows-only engines.

3. Wording & Translations

  • Renamed "Whisper (built-in AI)" to plain "Whisper" across docs/ENGINES.md, code, and all 18 i18n/*.json files.
  • Replaced references to "any language" with "~100 languages".
  • Updated ko.json to use the polite 해요 style throughout ("동영상과 오디오의 말소리를 SRT 자막 파일로 바꿔요. 인터넷 연결 없이 작동해요.").

4. Long files & progress

  • Replaced fixed timeouts (10 min ffmpeg / 30 min whisper) with a 5-minute StallTimeout (StallWatch).
  • Passed -progress pipe:1 to FFmpeg via FfmpegRunner.RunAsync and -pp to Whisper CLI with regex-based progress parsing to report real-time percentage progress.
  • Dynamic thread allocation: -t now follows Math.Clamp(Environment.ProcessorCount, 1, 8).

5. Results through Filee CLI

Test 1: English clip (caption-video preset, Auto / English)

filee convert english_sample.mp4 --preset caption-video

Output (english_sample.srt):

1
00:00:00,000 --> 00:00:08,380
Hey sweetheart are you okay? You look so dull. I'm feeling unwell mom. Let me

2
00:00:08,380 --> 00:00:13,680
check how you're feeling. You seem a little warm. Don't worry sweetheart. I feel

3
00:00:13,680 --> 00:00:20,400
tired mom. I'm here with you. You'll be okay. Let's have some chicken soup for

4
00:00:20,400 --> 00:00:26,240
dinner. Okay mom. Eat slowly. After dinner let's read a story together. I'd love

5
00:00:26,240 --> 00:00:31,560
that. Then it's time to sleep with you until you fall asleep. Good night mom. Good

6
00:00:31,560 --> 00:00:34,720
night sweetheart.

Test 2: Hindi clip (Auto vs explicit Hindi hi)

# 1. Spoken language set to Auto:
filee convert sant.mp4 --preset caption-video

Auto-detects language as hi (whisper_full_with_state: auto-detected language: hi (p = 0.52)), producing:

1
00:00:00,000 --> 00:00:14,200
 آپ சب کچھ کر سکتے میں کہ یہی سب کو گھول ہے کہ یہ تو کسی کسی کے بھاگی میں لکھا ہوتا کچھ نہیں تو جو چاپنا بھاگی بھاڑ سکتے ہو یہ منوش شریع جو چاہوں کر سکتے ہو بہگوان کو جپ راہتے کر سکتے تو کیا نہیں بن سکتے ہو۔
# 2. Spoken language set to Hindi (hi) in Preset Editor:
filee convert sant.mp4 --preset caption-video

Passes -l hi directly to Whisper CLI:

1
00:00:00,000 --> 00:00:14,200
 آپ சب کچھ کر سکتے میں کہ یہی سب کو گھول ہے کہ یہ تو کسی کسی کے بھاگی میں لکھا ہوتا کچھ نہیں تو جو چاپنا بھاگی بھاڑ سکتے ہو یہ منوش شریع جو چاہوں کر سکتے ہو بہگوان کو جپ راہتے کر سکتے تو کیا نہیں بن سکتے ہو۔

Ready for another look!

Screenshot 2026-10-30 124948

KnifeLemon and others added 3 commits October 30, 2026 16:58
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Indonesian (KnifeLemon#790) came to main after this branch started, so id.json was
missing the ten caption and spoken-language strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The format check wants a final newline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@KnifeLemon KnifeLemon left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @rohitLovesKrishna! Every point from the review is in:

  • the spoken language picker (Auto plus whisper.cpp's languages);
  • no platform checks in Filee.Core;
  • plain "Whisper" in the strings, and 해요 style in Korean;
  • StallWatch with real progress instead of fixed timeouts, and threads that follow the machine;
  • results made through Filee itself.

I merged main into the branch, added the ten new strings in Indonesian (which reached main meanwhile), and added a missing final newline for the format check. CI is green on all platforms.

Welcome aboard, and thanks for sticking with two rounds of review!

@KnifeLemon
KnifeLemon merged commit 79b227d into KnifeLemon:main Oct 30, 2026
10 checks passed
@rohitLovesKrishna
rohitLovesKrishna deleted the feature/video-caption branch October 30, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants