Tangles
The above video was my first proof of concept for a project I’m calling “video tangles”. The idea is to film two (or more) videos where the subjects move through a similar sequence, and then interleave the frames to create a superposition of the two subjects where their identities blur together into one.
I have another, much more artfully choreographed artifact from this project that’s not fit for the public internet; you’ll need to track me down IRL to see it. Hoping to revisit this soon and make another cut that can be more accessible.
Here’s a sketch of what the pipeline looks like:
- Film videos with subjects performing the same movement sequence. It can be helpful to do multiple takes to generate more candidate frames to choose from later.
- Use
ffmpegto split the videos into individual.pngframes. - For each frame, create a transparency matte of the subject. I used a two stage process of first generating a trimap mask with BiRefNet, and then using VitMatte for the final matte. This step was pretty slow and probably overkill for my needs, but the resulting maps were very precise. Using the matted images allows them to be easily repositioned in the final composition.
- For each matted frame, generate keypoints and bounding boxes for the subject using a pose model. I used the Ultralytics YOLO26s pose model for this, which worked great, super fast and accurate. The pose model’s keypoints allow the definition of a distance function between frames: take the
(x, y)coordinates of each keypoint relative to the subject’s bounding box, and compute the L2 distance between those vectors. - Pick one video as the key and the other as the reference. Generate the output frames by iterating through the frames of the key video, alternating taking the
ithframe from the key video or the nearest neighbor from the reference video. - Stitch the output frames back together into a video with
ffmpeg.