A LangGraph agent that curates audio-video pairs: it identifies a sounding
object, segments it, erases it from the video, separates its sound from the audio, and
jointly refines the resulting pair with the LTX-2 denoising enhancer — discarding any
sample that fails a quality gate along the way.
caption→sounding objects→SAM3 segment→EffectErase inpaint→SAM-Audio remove→LTX-2 enhance→paired AV output
Successful curations
Four clips passed every gate — segmentation, visual removal, and audio removal —
and were jointly refined by the LTX-2 enhancer. Three are shown here; the fourth (an airplane
wing) is unpacked stage by stage further down.
Electronic keyboard cover ✓ all gates passed
sample --IOqc-XgYo_000000 · target: black electronic keyboard · mask area 0.168 · visual removal 1.00 · audio removal 0.90
“A young man with glasses sits at a black electronic keyboard, playing it
with both hands…” The keyboard is segmented, erased from the video, and its sound
separated out of the track. The keyboard was essentially the only sound source, so the residual
is near-silence — exactly what a clean separation should leave.
“A woman with dark hair, wearing a black top, sits and plays a large wooden
harp…” The woman is removed from the video and her harp playing separated from the
soundtrack.
“An orange Daewoo wheel loader is parked on a dirt and gravel
surface… The loader’s engine is idling, and a plume of dark smoke is emitted from
its exhaust pipe.” The machine is removed from the scene and its idling engine from the
soundtrack.
input
SAM3 mask
inpainted (silent)
final: enhanced AV pair
🔊 residual audio (engine removed)
How it works — one sample, stage by stage
The fifth success, the airplane-wing clip, unpacked through every node of the agent —
a textbook pass: mask area 0.588, visual removal 1.00.
0Input clip ✓ all gates passed
sample --5A5ZCa1dE · 8 s · 1280×720 @ 30 fps · with original audio
The raw audio-video pair entering the pipeline.
1AV caption generation
Qwen3-Omni (audio-visual captioner)
The agent first describes the clip — both what it sees and what it hears.
The video shows a view from an airplane window, focusing on the
aircraft’s white wing and its flaps. The plane is flying over a large body of water, with
several small, dark islands visible in the distance. The wing’s flaps are extended
downwards, indicating the plane is likely in the process of landing or taking off. The sky is
overcast with grey clouds. The sound of the airplane’s engines is audible throughout the
clip.
🛡 Quality assurance: the caption must explicitly name
what is heard, not just what is seen — the audio-visual captioner listens to the
soundtrack. A caption with no identifiable sound source causes the sample to be discarded at
the next stage.
2Sounding-object extraction
LLM extractor over the caption
From the caption, the agent extracts the objects that produce sound and picks the
removal target — here the wing, the visible part of the sound-producing aircraft.
🎯 target: airplane wing
🛡 Gate 1 — sounding object exists: the extractor
only accepts objects with a physical sound-production mechanism (engines, strings, voices
— not walls or roads). If it returns none, the route function sends the sample to
discard immediately.
3Target-object segmentation
SAM3 · text-prompted video segmentation
SAM3 tracks the target through every frame. The mask gates the sample: too small
an object isn’t worth curating.
mask overlaid on video
binary mask track
mask area ratio0.588
🛡 Gate 2 — the object must be big enough:
two-step check. First a cheap fail-fast: SAM3 segments only frame 0, and if the target
covers less than the first-frame threshold the sample is discarded before any full-video work
(this is what rejected the race car below in ~15 s). Then the full mask-area ratio must clear
MASK_AREA_THRESHOLD (0.15 production; 0.05 in these dev runs). Wing: 0.588 ✓.
4Video: object removal (inpainting)
EffectErase · Wan2.1-Fun-1.3B-InP + LoRA
The masked wing is erased and the sea, islands, and sky hallucinated in —
the video branch of the output pair (silent; audio is handled by its own branch below). The
visual check re-segments the result with SAM3 and finds nothing left of the wing.
visual removal score1.00
🛡 Gate 3 — closed-loop removal verification:
the checker doesn’t trust the inpainter. SAM3 re-segments the inpainted video with
the same text prompt and compares against the original mask → a removal score in [0,1]
(1 = nothing left of the object). It must clear VISUAL_SCORE_THRESHOLD (0.80
production; 0.40 in these dev runs). Wing: 1.00 ✓. This is the gate that rejected the
train derailment (0.435) and the bassoon (0.179) below.
Ten separation candidates are generated and scored with ImageBind; the winner
splits the soundtrack into the removed engine sound and the clean residual that pairs with the
inpainted video. Winner: visual prompt, seed 1453635084.
🔊 residual (engine removed) — kept
🚨 isolated target (engine) — discarded
ImageBind text↔target0.353
ImageBind text↔residual0.028
audio removal score0.90
🛡 Gate 4 — QA by tournament: instead of trusting
one separation, the agent generates 10 candidates (visual-mask and text prompts ×
5 fixed seeds) and scores each with ImageBind: text↔target similarity should be high
(the removed track really is the target sound) and text↔residual should be low (no
target sound left behind). Winner = max text↔target, tie-broken by min text↔residual.
A removal-score gate at 0.80 then decides whether the sample continues.
The inpainted video and residual audio are muxed into one clip (conformed to
185 frames), VAE-encoded, re-noised to strength 0.4219, and denoised back down — video and
audio refined together through cross-modal attention. Compare before / after:
before — muxed pair (inpaint + residual)
after — LTX-2 enhanced (final output)
denoise strength0.4219
frames185 @ 24 fps
audio outstereo 48 kHz
🛡 Quality assurance: three safeguards. (1) The input is
conformed to the VAE’s hard constraints before the 46 GB model loads (frames = 8k+1,
dimensions divisible by 32, AAC audio) and re-probed afterwards. (2) The denoise strength is the
lightest reachable on the distilled sigma grid — a single denoising step that
polishes texture without regenerating content. (3) The run fails loudly unless the output
carries an enhanced audio stream — a silent mux bug can’t slip through.
Where the agent says no
Four clips the agent’s gates reject: an ill-posed target, two inpaintings the visual
check flagged, and an object too small to bother with. Discarded samples stop at their gate
— that is the curation working as intended.
sample 9NcjRFu6C4I · target: train engine · mask area 0.095 · visual removal 0.435 < 0.80 production gate
“A blue and yellow CSX locomotive travels along a track… the train
derails and begins to flip over…” The derailing, flipping train is hard to erase:
SAM3 still finds over half of it after inpainting (removal 0.435). This run used a relaxed
development threshold of 0.40, so the pipeline completed all stages — but the production
gate (0.80) discards it, and rightly so: look closely at the inpainted frames.
input
SAM3 mask
inpainted — train still visible
enhanced (for comparison)
🔊 residual audio (train removed)
Eel on the ocean floor ⚠ visual gate: discard
sample --8Bq81udbw · target: flowing water · mask area 0.489 · visual removal −0.105
“An eel with a spotted body lies coiled on the sandy ocean floor…
The sound of flowing water is heard in the background.” The agent targeted the
flowing water sound, but the “water” region is half the frame —
removing the background is ill-posed, and the inpaint failed the visual check (score
−0.105). All stages were completed offline for illustration.
“On the right, a person’s hands are shown playing a bassoon…
The audio consists of the deep, resonant sound of the bassoon playing a melody.”
Segmentation found the bassoon, but the inpainter couldn’t cleanly erase it —
SAM3 still detects most of it afterwards (removal 0.179 < the 0.40 gate), so the agent
discarded the sample before the audio stages.
input
SAM3 mask
inpainted — rejected by the gate
Race car (sim) ⚠ mask gate: discard
sample ---lTs1dxhU · target: race car · first-frame mask < 0.05 → stopped at segmentation
“A black single-seater race car drives at high speed around a
racetrack… a loud, high-pitched whine…” The car covers only a tiny
fraction of the frame, so the first-frame mask check failed and the agent discarded the
sample immediately — no inpainting or audio work was spent on it.