Skip to content

feat: task-based video generation provider - #13

Closed
amosarc wants to merge 5 commits into
mofa-org:mainfrom
amosarc:feat/video-gen
Closed

feat: task-based video generation provider#13
amosarc wants to merge 5 commits into
mofa-org:mainfrom
amosarc:feat/video-gen

Conversation

@amosarc

@amosarc amosarc commented Aug 24, 2026

Copy link
Copy Markdown

What

  • ProviderKind::VideoTask (config kind "video_task") — for vendor video APIs that follow the async task pattern rather than a synchronous OpenAI-style call.
  • VideoTaskProvider: the full create → poll → download lifecycle used by DashScope video-synthesis (Seedance) and similar APIs:
    • POST {base}/video-synthesis with X-DashScope-Async: enable; prompt from the message, size/duration from request params
    • bounded polling of GET {base}/tasks/{id} — 2s interval, 10-minute deadline; SUCCEEDED proceeds, FAILED/CANCELED surface the vendor message
    • artifact downloaded into the output dir, reported through file/files like image generation
  • Engine's build_provider constructs it from provider config, so it joins routing/failover/cost tiers unchanged.

Testing

  • Full lifecycle against mock create/poll/download endpoints: PENDING→RUNNING→SUCCEEDED transitions, ≥3 polls observed, exact bytes land on disk
  • DashScope task-status payload parsing
  • Full suite green

Stacked on #12; will rebase on landing.

🤖 Generated with Claude Code

amosarc and others added 5 commits August 25, 2026 02:06
@amosarc @claude
feat: reasoning streaming and real token streaming for OpenAI-compati…
faa6588 
…ble providers

- StreamSink now carries a typed StreamDelta (Text | Thinking) instead of a
  plain String; the engine envelope emits a new StreamChunk::Thinking for
  reasoning deltas ahead of the answer text
- InferenceResponse gains `reasoning` (DeepSeek-R1 style reasoning_content)
  on both streaming and non-streaming paths
- OpenAiCompatProvider::stream is now a real SSE implementation for chat
  models: token-by-token deltas with cross-chunk line buffering, usage
  capture via stream_options.include_usage, reasoning_content mapped to
  Thinking deltas. Non-chat capabilities keep the single-delta compat path
- Provider params passthrough: temperature (was silently dropped),
  enable_thinking for hybrid reasoning models (Qwen3 style)
- with_http_client builder so callers (and tests) can inject a no-proxy
  client for loopback endpoints while the default keeps system proxy
  support for cloud providers
- tests: thinking-before-text ordering at the engine level, full SSE parse
  against a mock OpenAI-compatible endpoint (split frames, usage, param
  passthrough)

Co-Authored-By: Claude <noreply@anthropic.com>
@amosarc @claude
feat: image generation for OpenAI-compatible providers
3262ab3 
- `invoke_image_gen`: POST {base}/images/generations with prompt/n/size
  params from the request; b64_json artifacts decode to the output dir,
  url artifacts are downloaded first
- `InferenceResponse.files` lists every artifact produced by one request
  (n>1); `file` mirrors the first for single-image callers
- mock-endpoint test covers b64 round-trip (PNG magic on disk), files
  ordering, and request-body passthrough

Co-Authored-By: Claude <noreply@anthropic.com>
@amosarc @claude
feat: vision input — multimodal messages with images
1715ecb 
- kernel Message gains `images: Vec<String>` (data/http URLs, user turns);
  skipped in serialization when empty
- openai_compat: ChatMessage serializes with OpenAI vision semantics —
  plain `content` string without images (byte-compatible), multipart
  text/image_url parts with them; Vlm capability routes to the chat paths
  (invoke + stream)
- ollama: images map to the native bare-base64 message field (data URL
  prefixes stripped)
- tests: multipart wire format against a mock endpoint (text part first,
  image_url carries the data URL), plain-content compatibility, full
  suite 101 green

Co-Authored-By: Claude <noreply@anthropic.com>
@amosarc @claude
feat: runtime provider configuration API (BYOK setup)
44ad8f3 
- Engine.build_provider extracted from startup assembly and reused by
- Engine.add_provider_config: registers a provider at runtime behind an
  RwLock; find_provider/routing_providers/discover_all consult static +
  runtime registrations, so a runtime provider routes immediately
- SDK: POST /v1/config/providers (validates kind/name/key, appends a
  [[providers]] entry to the resolved config file for restart survival,
  registers + refreshes) and GET /v1/config/providers (masked listing
  derived from live capability cards — keys never leave the engine)
- EngineConfig::resolved_path: explicit env > ./config.toml >
  ~/.config/mofa-engine/config.toml
- test: register an OpenAI-compatible provider against a mock endpoint at
  runtime → capabilities discover it and invoke routes to it

Co-Authored-By: Claude <noreply@anthropic.com>
@amosarc @claude
feat: task-based video generation provider (TOOL-02)
44f0bb6 
- new ProviderKind::VideoTask + config kind "video_task"
- VideoTaskProvider: the create → poll → download lifecycle used by
  DashScope video-synthesis (Seedance) and similar vendor APIs —
  POST {base}/video-synthesis with X-DashScope-Async, bounded polling
  (2s interval / 10min deadline) over GET {base}/tasks/{id}, terminal
  statuses SUCCEEDED/FAILED/CANCELED, artifact downloaded to the output
  dir and reported via file/files
- engine build_provider constructs it from provider config (base_url,
  api_key, models, artifacts dir)
- tests: full lifecycle against mock create/poll/download endpoints
  (PENDING→RUNNING→SUCCEEDED transitions, ≥3 polls, bytes on disk),
  DashScope status-shape parsing

Co-Authored-By: Claude <noreply@anthropic.com>
amosarc added a commit to aios-pub/mofa-studio that referenced this pull request Aug 25, 2026
@amosarc @claude
feat(toolbox): async video generation with task cards (TOOL-02 studio…
4abdb3b 
… side)

Completes TOOL-02 on top of engine PR mofa-org/mofa-engine#13:

Gateway (server-core video_routes):
- POST /v1/videos/generations — validates the prompt, maps size/duration
  onto an engine video_gen invoke, registers a task, and drives the
  (minutes-long) engine call in the background; returns the task id
  immediately
- GET /v1/videos/generations/{id} — running/succeeded/failed with the
  finished clip as a data URL (artifact read + base64); failures carry
  the reason; a metadata span records every task outcome
- 2 integration tests: full submit→poll→succeeded lifecycle against a
  mock engine (request mapping asserted), prompt validation + unknown
  task 404

Frontend:
- video service (5 tests): submit/status, terminal detection,
  pollUntilTerminal with per-poll updates, PLAT-06 asset recording for
  succeeded clips (type=video, source=studio)
- VideoGenPage (/creation/video-gen, sidebar 视频生成): prompt/size
  (16:9/9:16/1:1)/duration(3/5/10s)/video-model picker with engine-route
  default and a no-model setup hint; task cards with live status tags,
  pulse progress bar, inline <video> playback + download on success,
  failure reasons surfaced; session-scoped task store keeps cards alive
  across navigation

32 server-core + 178 frontend tests green

Co-Authored-By: Claude <noreply@anthropic.com>
@amosarc

amosarc commented Aug 25, 2026

Copy link
Copy Markdown
Author

Closing: upstream main has since landed equivalent capability in parallel (thinking deltas in streaming, Message.images for vision, invoke_image_gen with b64+url handling, and the cloud_video_gen backend). This PR is now redundant against origin/main — our downstream gateway (mofa-studio) already reads the single file field with a files-preferred fallback, so nothing downstream depends on this branch. Thanks for the review consideration; happy to re-open specific deltas if the maintainers want any piece of it (e.g. InferenceResponse.files for n>1 artifacts).

@amosarc amosarc closed this Aug 25, 2026
@amosarc amosarc mentioned this pull request Aug 25, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@amosarc