[rohitLovesKrishna]

2026-10-29.21-49-12_compressed.mp4

Summary

This PR introduces speech-to-text AI captioning for both video and audio files using Whisper (whisper.cpp). It allows users to drop video or audio files onto the Caption preset slice on the donut to generate .srt subtitle files directly.

Key Changes

1. Whisper Engine Integration:

   • Added whisper to engines.json and engine download/environment infrastructure.
   • Users can install Whisper directly from the Engines tab in Settings.
   • Implemented WhisperConverter (src/Filee.Engines/Media/WhisperConverter.cs) using whisper-cli.exe with automatic 16kHz mono audio extraction via FFmpeg.

2. Direct SRT Subtitle Output:

   • Built-in caption-video preset configured with target format srt.
   • Added to both Video and Audio donut profiles.
   • Generates matching .srt subtitle files in the target directory (e.g., sample.mp4 -> sample.srt).

3. Hindi Devanagari Script Transliteration (UrduToDevanagari):

   • Addresses a common Whisper phonetic issue where Hindi speech is transcribed into Urdu (Perso-Arabic) script.
   • Automatically detects Perso-Arabic transcriptions and transliterates them into clean Devanagari Hindi while preserving SRT timestamps.

4. Settings & Schema Migration:

   • Bumped schema version to 57 with migrations ensuring existing user profiles receive the caption preset on their video and audio profiles without resetting customizations.

5. Localization & Testing:

   • Synchronized all 18 localization files (ar, de, en, es, fr, hi, it, ja, ko, ne, pl, pt-BR, pt-PT, ru, th, tr, zh-CN, zh-TW).
   • Added test coverage in CaptionRouteTests.cs and WhisperTests.cs. All unit test suites pass (730+ tests).

Verification

• Tested with filee-cli convert and within the desktop UI for both video (.mp4) and audio (.wav) files.
• Verified .srt output and Hindi Devanagari subtitle correctness.

[KnifeLemon]

Thanks a lot for this, @rohitLovesKrishna! Captions from speech would be a great addition to Filee, and the demo video looks good. I'd like to get this merged, but a few basics need fixing first. I've approved the CI run, so you'll see the results on this PR.

Must fix

1. Existing translations are deleted. In all 18 i18n/*.json files the diff removes keys that are on main:

   • crop.perspective, crop.perspective_tip, crop.perspective_pdf, crop.hint_keys_perspective
   • crop.corner_*

   It also re-indents the toolbar.chip_tip.* block. Please rebase on main and only add the new keys. Each language also needs the new strings translated: fr.json, ko.json and the others currently have the English text.

2. No paths from other apps. LocateCli and LocateModel search D:\ChiCaption\…, %LOCALAPPDATA%\ChiCaption\… and %LOCALAPPDATA%\Filee\models. Filee should only use its own engine folder, the user's own copy (Settings → Engines) and PATH. The same goes for FFmpeg: use FfmpegConverter.Locate(), and when FFmpeg is missing say so. Don't fall back to a bare "ffmpeg".

3. Remove the Urdu → Devanagari rewrite. UrduToDevanagari runs on every transcript that contains Perso-Arabic letters. That turns real Urdu (and possibly Arabic or Persian) subtitles into Devanagari, which is wrong for those users. If Hindi comes out in Urdu script, set the language instead: whisper.cpp's -l hi. A "spoken language" option on the preset (default auto) would cover every language the same way.

4. Pin the model download. huggingface.co/…/resolve/main/ggml-base.bin can change under the same URL, and then the SHA-256 check fails for everyone. Please use a commit URL (resolve/<commit>/ggml-base.bin).

5. macOS and Linux. whisper-bin-x64.zip is Windows only. On other platforms the package and the Caption slice should not be offered, and the engine should report engine.reason.unsupported_platform the way other Windows-only engines do. Please say in the PR what macOS and Linux show.

6. One scope per PR. The preset makes SRT, but the converter also has a video → MP4 burn-in edge (Cost: 14). It also has a second audio → SRT edge without the Caption flag, so any audio → SRT route goes through Whisper. Please keep this PR to SRT only and remove both. Burn-in can be its own PR later. The doc comments in IConverter.cs and RoutePlanner.cs say "burns styled captions into the video" and need the same change.

7. Schema migration. AddCaptionPreset still carries leftovers from an earlier version: the {name}_captioned check, the preset added in two places, and Offer called as well as the manual inserts. Please reduce it to what the other migrations do:

   • add the preset;
   • Offer it to the video and audio profiles;
   • add a test that a v56 library learns it once.

   One note: #705 also needs a schema step. Whichever lands second moves to the next number.

Also needed

• THIRD-PARTY-NOTICES.md: add whisper.cpp (MIT) and the ggml Whisper model (MIT).
• docs/ENGINES.md: a short Whisper section covering the download size (~150 MB with the model), that it runs offline, and the need for FFmpeg.

Results

When you push the changes, please add to the PR:

• a screenshot of Settings → Engines with the Whisper package;
• the donut with the Caption slice;
• the .srt from a short English clip and from one other language (Hindi is fine), with the CLI command or the steps you used.

Thanks again! Tag me when it's ready for another look.