Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/known-limits.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,13 @@
Approaches already measured to fail.
Do not retry one without a new idea, and add a line here when you measure a new dead end.

- Geometric-mean scale convergence can move a previously passing per-frame median outside the
fixed band as visible depth support changes. After normal convergence, the automatic fitter
retains the earlier median-passing probe closest to unit geometric mean only if the final
medians fail, keeping that probe's scale and measured evidence together. Nonconvergence still
fails; explicit scales are unchanged. Median retention does not relax the independent depth
spread gate or establish acceptable geometry.

- The fresh city tracking baseline assigned person poses to street lamps and split the main
walker across tracks. Four of five retained tracks failed the source-appearance identity audit;
202 Mask R-CNN detections lacked a pose partner. These counts do not establish how many real
Expand Down
20 changes: 20 additions & 0 deletions docs/paid-recovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -513,6 +513,13 @@ Keep the original FPS and source clip. This option changes no current pipeline d
not authorize another paid clean, reset the shared inpainting allowance, or replace a failed
quality verdict. Inpainting remains subject to the same ledger, approvals and visual review.

Explicit-source clean masks are validated against the decoded frames selected by their recorded
source indices, with matching source SHA, PTS, timebase and packed shape. Their count need not
equal the FFmpeg FPS-filter count: a fractional-FPS tracking grid can retain a final source frame
that the FPS filter omits. Track-mask conversion still requires every declared track/sample mask
and complete sample coverage. Archives without explicit selection metadata must still match the
observed FPS-filter count; incomplete or inconsistent explicit metadata fails before paid cleaning.

## Experimental isolation of inpainting context

The direct runner and clean worker accept `--inpaint-mode isolated`; `legacy` remains the default.
Expand Down Expand Up @@ -650,3 +657,16 @@ Submission, cancellation and extraction are outside that wait. A timeout or inte
requests cancellation of the owned call and records whether the request outcome is known.
This receipt tracks the provider call. Tracking outputs still arrive through the returned
archive, and `result_received` precedes extraction and checks of the tracking report.

### Bounded avatar result waits

Frozen and tracked-avatar entrypoints record the owning app/profile and a local result deadline
before submission. The wait budget is the effective startup timeout plus execution timeout;
shortened tracked-animation execution deadlines shorten the corresponding wait. This does not
increase the provider deadline, retry allowance or spending budget.

On timeout or interruption the wrapper requests cancellation of only the call it submitted and
retains the recovery receipt. A cancellation request does not certify termination or a complete
checkpoint: inspect provider state and use the existing recovery path. Neither a wait failure
nor a missing checkpoint authorizes resubmission. The result wait still excludes image builds
and final output download; retain an overall operator deadline.
65 changes: 42 additions & 23 deletions scripts/run_clip.py
Original file line number Diff line number Diff line change
Expand Up @@ -1100,7 +1100,7 @@ def clean_flags(self):
return flags + (["--moved-mask"] if a.moved_mask else [])

def validate_clean_masks(self, path: Path):
"""Require packed masks for this source's observed FFmpeg sampling before any Modal call."""
"""Require masks matching decoded explicit selection or legacy FPS sampling before Modal."""
import numpy as np

if not path.is_file():
Expand All @@ -1126,24 +1126,6 @@ def validate_clean_masks(self, path: Path):
"--clean-masks masks must be a three-dimensional uint8 packed-bit array"
)

expected_shape = (
clean_sample_count(self.clip, self.a.fps),
expected_height,
expected_packed_width,
)
if tuple(packed.shape) != expected_shape:
raise ValueError(
"--clean-masks shape "
f"{tuple(packed.shape)} does not match observed source sampling {expected_shape}"
)
if declared is not None and (
declared.shape != (3,)
or tuple(declared.tolist()) != (expected_shape[0], expected_height, expected_width)
):
raise ValueError(
"--clean-masks shape metadata does not match the observed sampled frame count "
"and source dimensions"
)
if selected:
from worker.wander_worker.source_timing import select_source_frames

Expand All @@ -1159,7 +1141,7 @@ def validate_clean_masks(self, path: Path):
)
indices, pts = selection["source_indices"], selection["source_pts"]
if any(
value.shape != (expected_shape[0],) or not np.issubdtype(value.dtype, np.integer)
value.shape != (packed.shape[0],) or not np.issubdtype(value.dtype, np.integer)
for value in (indices, pts)
):
raise ValueError(
Expand All @@ -1175,12 +1157,29 @@ def validate_clean_masks(self, path: Path):
_, provenance = select_source_frames(
self.clip, self.a.fps, source_indices=indices.tolist()
)
expected_count = len(provenance["frames"])
if (
selection["source_sha256"].item() != provenance["sourceSha256"]
or selection["source_time_base"].item() != provenance["sourceTimeBase"]
or pts.tolist() != [frame["sourcePts"] for frame in provenance["frames"]]
):
raise ValueError("Explicit clean masks do not match the selected source pixels")
else:
expected_count = clean_sample_count(self.clip, self.a.fps)
expected_shape = (expected_count, expected_height, expected_packed_width)
if tuple(packed.shape) != expected_shape:
raise ValueError(
"--clean-masks shape "
f"{tuple(packed.shape)} does not match observed source sampling {expected_shape}"
)
if declared is not None and (
declared.shape != (3,)
or tuple(declared.tolist()) != (expected_count, expected_height, expected_width)
):
raise ValueError(
"--clean-masks shape metadata does not match the observed sampled frame count "
"and source dimensions"
)

def review(self):
"""The credit gate. 1600 credits are about to be spent on whatever these frames show."""
Expand Down Expand Up @@ -3075,13 +3074,22 @@ def scale_fit(self):
depth is ~1/scale0, so scale0 <- scale0 * geomean(ratio) converges in two or three probes.
"""
s, probes = self.a.scale0, []
selected_probe, selection_reason = "given", "explicit_scale"
if s is None:
best_median_probe = None
s = self.placement().get("floorMatchedScale") or 1.0
for i in range(self.a.scale_probes):
rows, fig, bands = self.diag(s, i)
ratios = [r for r, _, _ in rows]
gm = math.exp(sum(math.log(r) for r in ratios) / len(ratios))
probes.append((s, gm))
if all(SCALE_RATIO_BAND[0] <= r <= SCALE_RATIO_BAND[1] for r in ratios) and (
best_median_probe is None
or abs(math.log(gm)) < abs(math.log(best_median_probe[2]))
):
# Keep the measured evidence together. Diagnostic filenames round the
# scale, so a later probe can overwrite the same figure path.
best_median_probe = (i, s, gm, rows, bands, Path(fig).read_bytes())
say(
f" probe {i}: scale0 {s:.4f} -> ratios {[round(r, 3) for r in ratios]} "
f"(geomean {gm:.4f})"
Expand All @@ -3095,6 +3103,15 @@ def scale_fit(self):
f"{[(round(x, 4), round(g, 4)) for x, g in probes]}; pass "
f"--scale0 by hand or raise --scale-probes"
)
selected_probe, selection_reason = i, "converged_final"
if best_median_probe is not None and not all(
SCALE_RATIO_BAND[0] <= r <= SCALE_RATIO_BAND[1] for r in ratios
):
selected_probe, s, gm, rows, bands, figure_bytes = best_median_probe
ratios = [r for r, _, _ in rows]
fig = self.ctx / f"scale-fit-selected-probe-{selected_probe}.png"
fig.write_bytes(figure_bytes)
selection_reason = "earlier_median_pass_closest_log_geomean"
else:
rows, fig, bands = self.diag(s, "given")
ratios = [r for r, _, _ in rows]
Expand All @@ -3110,9 +3127,9 @@ def scale_fit(self):
if spread:
median_problem = (
f"depth ratios {[round(r, 3) for r in ratios]} are not all inside "
f"{SCALE_RATIO_BAND[0]}-{SCALE_RATIO_BAND[1]} at any single scale (geomean fit {s:.4f} "
f"gives {gm:.4f}). That is Marble geometry error or SfM drift, not a scale error -- it "
f"means the avatar reads ~{abs(1 - min(ratios)) * 100:.0f}% wrong at one end of the "
f"{SCALE_RATIO_BAND[0]}-{SCALE_RATIO_BAND[1]} at the evaluated scale {s:.4f} "
f"(geomean {gm:.4f}). This measurement alone does not rule out another scale. "
f"The depth mismatch is ~{abs(1 - min(ratios)) * 100:.0f}% at one end of the "
f"clip. Read {fig}, then either re-trim the clip to "
f"the window that fits or pass --allow-scale-spread to ship the geometric-mean fit "
f"with that caveat (gym, atrium and selfie did)."
Expand All @@ -3133,6 +3150,8 @@ def scale_fit(self):
p10=p10,
p90=p90,
probes=probes,
selectedProbe=selected_probe,
selectionReason=selection_reason,
gateOk=not spread and not wide,
medianOk=not spread,
spreadOk=not wide,
Expand Down
3 changes: 3 additions & 0 deletions scripts/stage_attempts.py
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,8 @@ def _read_authorization(path):
"worker/wander_worker/source_timing.py",
),
"worker/modal_lhm.py": (
"worker/stages/lhm_recovery.py",
"worker/stages/modal_profile.py",
"worker/stages/worker_resources.py",
"worker/stages/lhm_execution.py",
"worker/stages/lhm_person.py",
Expand All @@ -150,6 +152,7 @@ def _read_authorization(path):
"worker/stages/pose_instances.py",
),
"worker/modal_multiperson.py::animate": (
"worker/stages/modal_profile.py",
"worker/stages/worker_resources.py",
"worker/stages/lhm_execution.py",
"worker/stages/lhm_recovery.py",
Expand Down
Loading
Loading