I figured it out (again) after reading the previous post I made on the technique - there was enough information in there to refresh my memory.
The first thing to mention is that the background removal DAE uses actually identifies humans in a video and removes everything else. No way to do anything that ain’t a photo human (specialist background removal services offer more options). I’ve noted this situation to DAE.
Oh, you also need a lot of free disk space.
I just tested this command (I’ve renamed DeepArtEffectsCli)…
.\dae removebackground -input C:\art\video\frames\frame-0001.jpg -output C:\art\video\back\test.png
That takes the image C:\art\video\frames\frame-0001.jpg and creates C:\art\video\back\test.png which is the original with the background removed - specifically it’s now transparent, which is exactly what’s needed.
The process is to take a video then extract each frame as an image which is very easy to do using ffmpeg (this is actually how videos are processed by DAE but it does that bit as part of the video styling process)
You then apply a style to an image - which one depends on the result you want - I need to try this again.
You next combine two images to create a composite which has a style applied to the background or foreground.
Finally you use ffmpeg again to take all the composite images and convert them back into a video.
There are a few problems that need addressing though…
- A trivial issue is that you need to download ffmpeg and use it to extract the video into lots of images, one per frame. If you get this and want to test the command (it’s a command-line program) is simple - e.g.
If you’ve got a video called video.mp4 at the command line cd to that directory then…
mkdir frames
cd frames
ffmpeg -i ..\video.mp4 frame-%06d.jpg
You end up with a large number of jpegs in the frames directory called frame-000001.jpg … frame-00xxxx.jpg
-
DAE-cli won’t remove the background from the frames directory (reported to DAE) so you’d need to issue a command like the one near the top for every frame. I tested on a 34 second video I got from Pixabay, 1734 images. These images must be png (LOTS of disk space). Note that resolving this issue is actually fairly easy but DAE-cli needs to be run on every image - can be an overnight job.
-
Assuming you’ve worked around (2) then what you do next depends on whether you want to use DAE to style the background, foreground or both (in that order they each take longer than the previous and some involve an extra step)
-
At this point you need to combine the stupidly large quantity of images you’ve now got to create the final image - this can be done with imagemagick (can’t remember how ATM) but then again if I was doing it today I’d write a program to do it (this is fairly simple for what’s needed)
-
Finally you use ffmpeg to take all the finished frames and convert them back into a video - that ones - ffmpeg -i frame-%06d.jpg …\newvideo.mp4 (similar to extraction but with the arguments the other way around)
So, in essence, all that’s needed is a program that automates the annoying bits + the bits I’d do differently. Such a program is actually not that hard to write as all it’d actually do is call other programs, specifically dae and optionally ffmpeg. It’d also eat lots of disk while doing it - but then again a 1tb external USB is cheap these days so not an insurmountable problem