Overlaying Styles in Video

My Mac’s currently on frame 59,160 of 66,150 of background removal for a 35 min video - it’ll finish in about 24 hours…

The Alpha Unscreen creates is binary - something is either in the foreground or in the background.

The Alpha DAE creates is more of a ‘how sure are we that this pixel is in the background / foreground?’ with values ranging from 0 to 255 (0 = 100% its background)

Taking either of these definitions you can simply invert each alpha frame which gives you a foreground removal result rather than a background removal one.

  1. I took my original test clip and extracted all the frames with ffmpeg
  2. Then I ran background removal on each individual frame - this is the only version with alpha I used
  3. I then ran every frame from (1) through all 121 styles resulting in a huge quantity of styled frames
  4. The complicated bit - for each frame of one of the styles (this will be new background) and another style (the new foreground)…
    a) Load background
    b) Load foreground
    c) Load alpha (from 2)
    d) Apply (c) to (b) - creating a version of (b) with alpha
    e) Overlay (d) on (a)
  5. Use ffmpeg again to build the new clip with frames version (e)

It sounds more complicated than it is :slight_smile:

To do this all you need is ffmeg, a way to automate background removal (DAE bug) and something to do (4) above

I’ll write some command line apps to do the missing bits and although DAE CLI BGR a directory should be fixed sometime you’ll defo need an app to do the overlay stuff.

It should be noted that you can retain the sound-track from a video as well, it’s just an extra ffmpeg step in the process - and in 6650-ish frames time I’ll have the source to do just that…

Tutorial to follow (sometime in Jan I hope)