Skip to content

Add OpenAI-compatible LLM translation provider with background jobs - #5

Merged
garanda21 merged 6 commits into
mainfrom
feat/llm-translation-provider
Oct 25, 2026
Merged

garanda21 merged 6 commits into
mainfrom
feat/llm-translation-provider

Conversation

@garanda21

@garanda21 garanda21 commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Closes #3

Summary

Adds an LLM translation engine as an alternative to DeepL, using any OpenAI-compatible chat API (OpenAI, OpenRouter, Anthropic, LiteLLM, Ollama), and moves translations to background jobs so long LLM runs no longer time out.

LLM provider

  • TRANSLATION_PROVIDER=llm with LLM_BASE_URL, LLM_API_KEY, LLM_MODEL. DeepL stays the default.
  • Only subtitle text is sent to the model, as JSON with ids; timestamps never leave the server and the SRT is rebuilt locally.
  • Bundled system prompt in server/prompts/llm-system-prompt.txt, overridable by mounting /config/llm-system-prompt.txt (or LLM_PROMPT_FILE).
  • Strict 1:1 id validation: malformed batches are retried and then split in half; rate limits and 5xx errors are retried with backoff.
  • LLM_FALLBACK_MODELS: models tried in order when the main one keeps failing; a failing model is skipped for 5 minutes.
  • Default batch size of 25, which was faster and produced fewer malformed responses than 50 with free models.

Background jobs

  • POST /translate_subtitle now returns 202 with a job; progress is exposed via GET /jobs and GET /jobs/:id. This avoids nginx's default 60s proxy timeout.
  • Jobs run one at a time; the same file and target language cannot be queued twice (409).
  • Plex is refreshed server-side when a job finishes, even if the tab was closed.
  • Failed jobs can be resumed (POST /jobs/:id/retry), skipping batches already translated. Works for both DeepL and LLM.
  • The UI shows a progress bar that survives page reloads, and a Resume translation button for failed jobs.

DeepL compatibility

  • With a stubbed DeepL API, translateSRT from main and from this branch produce byte-identical output with the same requests and batches.
  • Behaviour changes: translations are queued instead of running in parallel, a Plex refresh failure no longer marks the translation as failed, and a missing subtitle file returns 404.

Limitations

  • Jobs are kept in memory: they are lost on a server restart.

Testing

  • Lint and build pass; the :dev image was published and tested on a real server.
  • Real translation of a 1007-block episode with google/gemma-4-26b-a4b-it:free on OpenRouter: identical timestamps, good quality.
  • Mock provider tests for fallback, mid-run failure and resume, duplicate rejection, and progress surviving a page reload in the browser.

…jobs

- Add TRANSLATION_PROVIDER=llm using any OpenAI-compatible chat API
  (OpenAI, OpenRouter, Anthropic, LiteLLM, Ollama). Only subtitle text is
  sent to the model as JSON; timestamps never leave the server.
- Bundle a default system prompt, overridable by mounting
  /config/llm-system-prompt.txt (or LLM_PROMPT_FILE).
- Validate a strict 1:1 id mapping, retry malformed batches and split
  them in half; retry rate limits and server errors with backoff.
- Run translations as queued background jobs with batch progress, so
  long LLM runs no longer hit the 60s proxy timeout. Progress survives
  page reloads and Plex is refreshed server-side when a job finishes.
- Show LLM model and token usage in the header.

Closes #3
The server rejects a new job with 409 when the same subtitle file and
target language is already queued or running, and the translate button
is disabled in that case. Other files and languages can still be queued.
- LLM_FALLBACK_MODELS lists models tried in order when LLM_MODEL keeps
  failing. Models with a fallback fail over after 2 attempts and are
  skipped for 5 minutes; auth errors do not trigger a fallback.
- Jobs keep the translations of finished batches. A failed job can be
  resumed with POST /jobs/:id/retry, which skips completed batches for
  both DeepL and LLM providers.
- When one LLM batch fails, workers stop taking new batches and wait for
  in-flight ones, so a retry never overlaps a run still in progress.
- The UI shows where a job stopped and a Resume translation button.
@garanda21
garanda21 force-pushed the feat/llm-translation-provider branch from fe519d2 to acb3462 Compare October 25, 2026 11:26
@garanda21

Copy link
Copy Markdown
Owner Author

@claude review

@claude

claude Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Claude finished @garanda21's task in 32s —— View job


Review of PR #5: OpenAI-compatible LLM provider and background jobs

I read the server-side changes in full: llm.js, jobs.js, translator.js and routes/subtitle.js. I did not read the frontend files (src/), the README or the compose changes. I didn't run lint, build or any tests. git diff origin/main...HEAD failed because the checkout is shallow and has no merge base, so I read the files as they are on the branch.

The design is solid. The strict 1:1 id validation, the retry then split-in-half handling, the fallback cooldown, and the resumable partial array are all well thought out. Sending only text to the model and rebuilding the SRT locally is a good call. Below are the issues I found, most important first.

Should fix

  1. Infinite loop with a bad LLM_BATCH_SIZE (server/services/llm.js:11, :231).

    • LLM_BATCH_SIZE=0 makes start += BATCH_SIZE never advance, so the for loop pushes batches forever and the server runs out of memory.
    • A non-numeric value gives NaN. The loop then exits after one iteration with an empty entries.slice(0, NaN), so nothing gets translated.
    • LLM_CONCURRENCY=0 has a similar problem. Array.from({length: 0}) starts no workers, so the job "succeeds" and returns undefined for every text.
    • Suggested fix: clamp each setting to a positive integer, for example Math.max(1, parseInt(...) || default).
  2. Stale state when a failed job is retried (server/services/jobs.js:137-146).

    • retryJob clears error and finishedAt. It leaves startedAt alone, so it stays as the first attempt's time.
    • If an earlier attempt ever set plex or plexError, those are not reset either.
    • A job that reaches done after a retry will therefore show timings and Plex fields from earlier attempts. This is cosmetic, but it's a one-line delete for each field.
  3. Jobs hit by a transient error still burn model calls. In translateTextsLLM, a non-HTTP failure (malformed JSON) on a single entry throws after 2 attempts, and the whole job fails.

    • That is reasonable, and resume helps. Still, translateBatch uses error.status to tell HTTP errors from validation errors.
    • Any thrown error that happens to carry a .status skips the retry and split path. That includes the 503 for network failures. Consider documenting that contract with a comment, or using a dedicated error class.

Consider

  1. Fallback retry budget is asymmetric (llm.js:144). Non-last models get only 2 attempts, and the last model gets MAX_HTTP_RETRIES. With no fallbacks configured, the main model is the last one, so it gets the full 5. With fallbacks configured, a rate-limited main model gets only 2 tries before it is put on a 5-minute cooldown. That is probably intentional, and it should go in the README so users know a free-tier 429 will quickly push traffic to the fallbacks.

  2. Any non-auth error triggers the cooldown (llm.js:149). A 400 from a request the provider rejects (for example response_format unsupported by that model) benches the model for 5 minutes. That is fine for fallback purposes. It would help to say in the README that LLM_JSON_MODE=false is the fix for providers that reject response_format.

  3. Failed jobs count toward MAX_FINISHED_JOBS and hold their partial array (jobs.js:36-41). Once 20 or more jobs finish, the oldest failed job is pruned, and its resume progress goes with it. For a 1000-block file that is a lot of paid work to lose silently. You could prune done jobs first, or keep failed jobs until they are retried or dismissed.

  4. Path handling in /translate_subtitle (routes/subtitle.js:25-27). serie, season and srt_path come from the request body and go straight into join(...), so ../ segments can escape DATA_DIR. The same pattern already exists in /extract_subtitles, so this is not new. The new route reads the file and writes next to it, so a check that the resolved path stays under DATA_DIR or MOVIES_DIR would be worth adding. You could also restrict srt_path to .srt files.

  5. extract_subtitles still returns error.stack to the client (routes/subtitle.js:108-112). It's not part of this PR, but it's worth removing while you're in this file.

Nits

  • server/services/llm.js:61: the log line Using custom LLM system prompt prints on every translation. That's fine, but it does not say which job or language pair it belongs to.
  • jobs.js: processQueue() is called without .catch. runJob catches its own errors, so this is safe today. If pruneFinished ever throws, running stays true and the queue stops permanently. A try/finally around the loop would make it robust.
  • There are no automated tests for translateBatch validation, the split logic, or partial resume. Those are good candidates for unit tests with a mocked axios, since you described manual mock-provider testing.

Summary

Item 1 is a real hazard from a bad env var, and I would fix it before merging. Items 2 and 6 are small robustness fixes. The rest are optional.
· branch feat/llm-translation-provider

- Validate numeric LLM env vars and fall back to defaults; LLM_BATCH_SIZE=0
  previously looped forever and LLM_CONCURRENCY=0 produced empty output.
- Reject paths that resolve outside DATA_DIR/MOVIES_DIR, invalid language
  codes (they end up in the output file name), non-.srt files and unknown
  types in translate_subtitle and extract_subtitles.
- Stop returning error stacks from extract_subtitles.
- Clear the previous attempt's timings, path and Plex state on retry.
- Prune completed jobs before failed ones so resume progress is not lost.
- Keep the job queue running if an unexpected error escapes a job.
- Document the error status contract in translateBatch and the fallback
  retry budget and LLM_JSON_MODE in the README.
@garanda21

Copy link
Copy Markdown
Owner Author

@claude review

@claude

claude Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Claude finished @garanda21's task in 30s —— View job


Review of PR #5: OpenAI-compatible LLM provider and background jobs

I read server/services/llm.js, jobs.js, translator.js and routes/subtitle.js as they are on the branch head (bc1c37a). I did not read the frontend (src/), the README, the compose file or the prompt file. I did not run lint, build or any tests.

Most of the earlier review's points are already fixed on this head:

  • LLM_BATCH_SIZE, LLM_CONCURRENCY and the other numeric env vars are validated by numberFromEnv.
  • retryJob clears startedAt, plex and plexError.
  • Failed jobs are pruned last.
  • The /translate_subtitle path is checked with resolveInside, and the route accepts only .srt files.
  • processQueue has a try/catch/finally.
  • extract_subtitles no longer returns error.stack.

The status contract in translateBatch is also commented now. The points below are what I still see.

Should fix

  1. A response with no message content is misreported and retried as a network failure (server/services/llm.js:119-139).

    • The throw new Error('LLM response has no message content') is inside the try. The catch sees no error.response, so status is undefined and the error counts as retryable.
    • It is retried with exponential backoff up to maxRetries. It then surfaces as LLM API unreachable: LLM response has no message content with status 503.
    • This happens in practice with some free models and with content filters, which return content: null or an empty choices array. The user waits through the backoff and then sees a misleading message.
    • Fix: parse the content after the try/catch. Alternatively, throw an error that the catch recognises, so it gets the normal malformed-response path (retry once, then split the batch) instead of a 503.
  2. Retry-After has no upper bound (llm.js:141-142).

    • The header value is used as given, while the exponential fallback is capped at 30s. A provider that sends Retry-After: 3600 makes the job sleep for an hour, and it shows as "running" the whole time.
    • Suggestion: Math.min(retryAfter, 60) * 1000, or fail over to the next model when the wait is long.

Consider

  1. Resume safety check only compares lengths (translator.js:94). If the subtitle file is edited so that it has the same number of non-empty blocks, the saved partial translations are reused for different text. A cheap fix is to store a hash of texts next to partial and reset when it differs. It's an edge case, but a resumed job gives no sign that it happened.

  2. A model gets benched for any non-auth error (llm.js:163-167). A single 400 (for example response_format unsupported, or a context-length error on one oversized batch) puts that model on cooldown for 5 minutes for all batches. Only 429, 5xx and network errors really say "this model is unhealthy". Limiting the cooldown to those statuses would avoid benching a healthy model over one bad request. The README should also mention LLM_JSON_MODE=false for providers that reject response_format.

  3. Fewer retries when fallbacks are configured (llm.js:160). Non-last models get 2 attempts and the last gets LLM_MAX_RETRIES. This is probably intended, but it is not documented, and LLM_MAX_RETRIES has no effect on the main model once fallbacks exist.

  4. extract_subtitles still returns originalError: error.toString() (subtitle.js:142). This is not new in this PR. ffmpeg messages usually contain absolute server paths, so you may want to log it and not return it.

  5. Path check ignores symlinks (subtitle.js:17-21). resolveInside is purely lexical. A symlink inside the media directory pointing outside it passes. This is low risk if the media directories are trusted.

Nits

  • llm.js:77: Using custom LLM system prompt is logged on every call to loadSystemPrompt, so once per job. That's acceptable. Including the language pair would make the log easier to follow.
  • There are still no automated tests for translateBatch validation, the split path and partial resume. These are easy to cover with a mocked axios. Item 1 would have been caught by a test of an empty-content response.

Summary

Item 1 is a real bug. The slow failure and wrong error message are the visible effects. Item 2 is a one-line cap. The rest are optional.
· branch feat/llm-translation-provider

- Treat an empty or missing model response as a malformed response
  (retry, then split the batch) instead of a network failure that was
  retried with backoff and reported as "LLM API unreachable".
- Cap Retry-After waits at 60 seconds.
- Only put a model on cooldown for 429, 5xx and network errors; other
  HTTP errors fall back for that request only.
- Discard saved resume progress when the source texts change, using a
  hash instead of comparing only the number of blocks.
- Stop returning originalError from extract_subtitles and mask media
  directory paths in its error message.
@garanda21

Copy link
Copy Markdown
Owner Author

@claude review

@claude

claude Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Claude finished @garanda21's task in 16s —— View job


Review of PR #5 at head 1e873cd

I re-read server/services/llm.js in full. I read the resume section of translator.js (translateSRT), the top of routes/subtitle.js, and grepped jobs.js and the README. I did not read the frontend, the compose file or the prompt file, and I did not run lint, build or tests.

Everything from the last review is now addressed:

  • Empty message content: it is thrown with no .status, so it takes the retry-then-split path and no longer shows up as a 503.
  • Retry-After: it is capped at 60s.
  • Resume safety: translateSRT hashes the texts and discards partial when the file changed.
  • Cooldown: it now applies only to 429, 5xx and network errors. Other HTTP errors fall back for that request only.
  • README: it now documents the retry asymmetry with fallbacks and LLM_JSON_MODE=false.

I found no new blocking issues.

Consider (all optional)

  1. Cooldown is checked only before the first attempt on a model (llm.js:157). If every non-last model is on cooldown, requests go straight to the last model. That is the intended behavior. One side effect is that the last model never enters the cooldown map, so it is hit on every request even when it is the failing one. This is fine, since there is nothing left to fall back to.
  2. Network failures give up after 2 attempts on non-last models, with a short backoff (2s, then 4s). A brief connectivity blip with fallbacks configured will bench the main model for 5 minutes. This matches the documented behavior, so I'd leave it.
  3. resolveInside is purely lexical (routes/subtitle.js:17-21). Symlinks inside the media directory can still point outside it. This is low risk if the media directories are trusted, as noted before.
  4. Tests: there are still no automated tests for translateBatch validation, the split path or partial resume. These are good candidates for unit tests with a mocked axios, for example empty content, missing ids and a mid-run failure followed by resume. This is not a blocker.

Summary

The previous findings are fixed and the code reads well. I'd say this is ready to merge once you're satisfied with your own testing. The remaining points are optional.
· branch feat/llm-translation-provider

@garanda21
garanda21 merged commit b1c23a5 into main Oct 25, 2026
3 checks passed
@garanda21
garanda21 deleted the feat/llm-translation-provider branch October 25, 2026 12:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for LLMs using openai/compatible api

1 participant