Skip to content

feat(website): add Openjutsu, a character-swap Hub app, behind a staff-only flag - #20965

Merged
comfyrob merged 60 commits into
mainfrom
rob/app-openjutsu
Oct 19, 2026
Merged

comfyrob merged 60 commits into
mainfrom
rob/app-openjutsu

Conversation

@comfyrob

@comfyrob comfyrob commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds Openjutsu, a new Hub app at /hub/apps/openjutsu: swap the person in a 5 to 15 second part of a video for the character in a single image, keeping the scene, the camera and the clip's own sound. It shows only where the PostHog flag workshop-openjutsu-app-enabled is on, which today means Comfy staff only.

Resolves FE-3430 (ticket and design doc). Backend: BE-20545, Comfy-Org/cloud#13014 (merged) and Comfy-Org/cloud#14161 (this workflow's price and deployment).

What is in it:

  • The app (src/components/workshop/openjutsu, src/lib/workshop/openjutsu, useOpenjutsu): one state machine over the Re-shoot transport, so the app proxy, the local dev proxy and the in-page sample backend share one run path.
  • The workflow (openjutsu/2, see apps/website/workflows/openjutsu/README.md): Akatz Labs' H3 Character Swap workflow, using their node pack, SAM 3.1 masking, and 124-frame relay windows with 18 frames of history. It runs the v1.5 character LoRA and the FL2V Turbo 8-step LoRA on a dedicated deployment. The graph is generated from upstream's API file; the changes from upstream are listed in the README.
  • Prompt: every run sends the workflow's instruction with H3's reference tags. The page's text box is optional extra direction: character details, or a fix for the last take.
  • First-frame anchor (anchor.ts): before the job, the page takes the frame at the part's start and asks GPT Image 2.5 sunburst, at low quality through Comfy Router with the visitor's sign-in, to put the character into it, outfit included. The page uploads the result, and the graph crops it to the prepared size. If this fails, or nobody is signed in, the swap runs without an anchor. It is a separate Router charge, which the note under Generate mentions.
  • Price: per second by quality, 8 credits a second at 480p and 16 at 768p. It is read from H3CharacterSwapPrepareVideo, the same node and values the app proxy prices (Comfy-Org/cloud#14161); a test ties price.ts to the graph.
  • A reusable video trim popup (src/components/workshop/video-trim). It opens on every video a visitor adds; examples load straight in.
  • The full-screen editor kit (src/components/workshop/app-editor) and the redesign built on it, by @mariana953, merged in from feat(website): Openjutsu full screen on the shared editor #20636.
  • Registration: route, catalogue entry, flag entry, app proxy ids, and en and zh-CN copy. There is no zh-CN or ja page twin, matching the other Hub app pages.

Known limits, recorded in apps/website/workflows/openjutsu/HANDOFF.md:

  • The flag is visibility only, not access control. The proxy id is public in the bundle, so anyone signed in can call the app proxy directly; the catalog prices the graph from the node that sizes the run.
  • Order with the backend: until Comfy-Org/cloud#14161 merges, the app proxy prices Simplify TemplateHelper.mockIndex(): inline '{}' instead of JSON.stringify(customTemplates) #13014's old graph, so this page's runs are refused. It is staff-only either way, and main has no Openjutsu today, so either order is safe.
  • Clips that open with no person in frame (a fade from black, a title card) fail at the mask step before generating.
  • A run refused for credits is worded for an owner or a member through the shared studioGate, the same derivation Re-shoot has, which ADR-API-SERVER-FACTS-0042 wants moved to the backend.
  • This is large. The editor kit under app-editor is self-contained and can be split into its own PR if that helps review.

Verification

  • Tests: 188 Openjutsu unit tests pass. They cover the graph builder (with and without an anchor), the run path including the anchor fallback and a cancel, the price against the graph, the Router anchor request, and the panel and run components. pnpm typecheck:website, knip and the pre-commit lint and format pass. e2e/openjutsu.spec.ts and its fake app proxy are updated for the new graph, price and Router call, but have not been re-run locally since; CI is their first run.
  • Human QA, local proxy against the deployment (astro dev, unmetered on the app side): signed-in swaps at 480p with the Router anchor. The anchor is made, uploaded and used, the result keeps the character's full outfit from the first frame, and anchor-free runs succeed. Measured costs are in BENCHMARK.md.
  • Not yet verified: a swap through the staging app proxy with a signed-in account (quote, charge, refusal). That waits on Comfy-Org/cloud#14161.
  • Screenshots: the page is behind the flag; the preview deployment for this PR shows it to a staff account at /hub/apps/openjutsu/.

comfyrob and others added 30 commits October 7, 2026 16:40
Character swap for video on MiniMax H3 and the Akatz Labs Character Swap
LoRA. The page runs end to end against a stand-in transport in astro dev;
no deployment or app-proxy entry exists yet, so it reports itself
unavailable everywhere else.
Every video picked in Openjutsu now opens a trim dialog: a filmstrip
timeline with two handles, a looping preview and Confirm. The dialog and
its range rules are shared workshop pieces that take an app's own length
limits; a video too short for them cannot be confirmed.
Drops the frame-repeat step, the one node that needed a custom pack, and
names the LoRA by its published file name, as the Openjutsu build
installs it.
Generates the next H3 grid length at or above the chosen part and cuts
the result back to the part, so a 5 second pick gives 5 seconds.
…ength

A part that is the whole clip could miss the clip's own length by a
rounding and fail the strict trim, as a benchmark run did.
Quality is 768p by default or 480p. The canvas now follows the source's
own aspect ratio at the chosen pixel count, and the result is resized to
exactly the source's shape before saving.
…dev proxy

Under astro dev with PUBLIC_OPENJUTSU_PROXY set, the page needs no account:
the proxy holds the key and nothing is metered, as with Re-shoot.
…ler tested units

The stage is now a layout shell over separate player, status, part summary,
result/source toggle, actions and takes components. Run, gate and take rules
move out of the composable and the trim components into plain modules with
unit tests. No behaviour change.
…ite player, a cost line and a synced compare

The app now runs in the full-screen editor: settings in the floating panel (bottom sheet on phones), the source or take on the stage in the site video player, phases with elapsed time and Cancel over the stage while a take renders, and a result dock with Compare, Result, Source and Use these settings. Compare splits the result and its source under one set of player controls, the source following the result's play state and time. The quote shows under Swap character; a run the quote refuses says why on the button. The video player takes a default slot over the picture, and the run button's credit chip is optional.
…eld reset, tile keyboard, pinch, cancellable mock job
Video row matches the other apps' source row with trim as an inline link,
the character upload uses the kit tiles, the seed gains the dice, the panel
waits for a video like the other apps, and the run footer uses the kit
button when the account has nothing to act on.
…: prompt box with reference buttons, quality menu
…ed beyond the part

A part whose frame count already sits on the model's grid (6.6 s, 7.3 s,
14.4 s and up) generated no spare frames, so the final cut asked for more
than existed and the job failed at its last step.
…nel in place

A new video is used at once from its start, up to 15 s. A longer one gets a
quiet line under the video in the panel with a Change link to the trim. The
panel is always shown, the video can be taken out with an X and put back, and
the empty stage asks for a video without drag and drop.
@datadog-official

This comment has been minimized.

cloud#13014 now bills MiniMaxH3ReferenceToVideo.length tiered by the canvas
area (tier_match: at_most). The page prices the frames the job asks the model
to generate (gridFrames) at the rate of the smallest area tier the canvas fits,
refuses a canvas above every tier, and a test ties the price to the length the
workflow sets. 5 s is now 130 credits; 15 s stays 360.
coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 15, 2026
The twelve example clips and stills move from public/ to
media.comfy.org/website/workshop/openjutsu/examples, byte for byte, so
the repository carries no Openjutsu media.
coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 15, 2026
benjcooley
benjcooley previously approved these changes Oct 16, 2026
The card had no media. It now plays an 8 s split-wipe loop between the
source cowboy and the chrome-robot result, with the character reference
card, hosted on media.comfy.org beside the Re-shoot and Cinematic Studio
thumbnails (480x360 H.264 + 960x720 poster).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@comfyrob
comfyrob dismissed stale reviews from benjcooley and coderabbitai[bot] via 3e73121 October 16, 2026 22:39
coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 16, 2026
Replace the four starter swaps with singer → disco diva, boxer → Comfy
fan, painter → ronin and cowboy → chrome robot. The clips (5-6 s, with
their sound), posters and character stills are on media.comfy.org under
new names; the old files stay where they were.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 17, 2026
The examples no longer fill in who to replace, and the field is optional
everywhere: a blank one no longer holds Generate, the prompt names "the
person", and a take reads "Replaced the person". The label and
placeholder say it is optional and only needed when more than one person
is in the shot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
coderabbitai[bot]
coderabbitai Bot previously approved these changes Oct 17, 2026
Swap the graph to Akatz Labs' H3 Character Swap workflow (their node pack,
SAM 3.1 masking, relay windows) with the v1.5 character LoRA and the FL2V
Turbo 8-step LoRA, on its own deployment.

- Every run sends the workflow's instruction with H3's reference tags; the
  optional box adds character details or fixes a take.
- The page makes a first-frame anchor through Comfy Router (GPT Image 2.5
  sunburst, low) from the part's first frame and the character, uploads it,
  and the graph crops it to the prepared size. A failure runs the swap
  without one.
- Runs are priced per second by quality, from H3CharacterSwapPrepareVideo,
  as the app proxy charges them.
- The workflow contract, handoff and benchmark move to openjutsu/2.
@comfyrob
comfyrob enabled auto-merge October 19, 2026 02:32
@comfyrob
comfyrob added this pull request to the merge queue Oct 19, 2026
Merged via the queue into main with commit acbe33d Oct 19, 2026
98 checks passed
@comfyrob
comfyrob deleted the rob/app-openjutsu branch October 19, 2026 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high PR risk grade (advisory shadow check; grader-owned)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants