Dear Simon
I would like to have the capability of changing the foreground from the background for videos.
You have a posted on You tube on how to impose a video with a background image
https://youtu.be/uReF8-NAjHI. Very Dramatic.
What you presented requires some sophistication with programming etc. Is this function ever going to be in DAE?
The video I created was just a little experiment. As I did this a while ago I’ll have to refresh my memory and see if the new features in DAE help (actually, I’m positive that it’s easier when I did it).
Basically all that’s required is splitting the video into a load of png/jpg files then bulk-processing each frame (this can take a while).
The additional tools required are command-line utilities, ffmpeg + imagemagick
There will be the requirement to write out a big script file but that’s dead simple to automate.
As I say, I’ll have to do it again in order to properly instruct anyone on the method
Oh, I made some notes i posted, that’ll make things easier for me
Hi Peardox
I am super interested in this option of changing foreground and /or background in videos.
I look forward to you giving me specific instructions of what to do- when you figure it out.
Much thanks
Bob
I figured it out (again) after reading the previous post I made on the technique - there was enough information in there to refresh my memory.
The first thing to mention is that the background removal DAE uses actually identifies humans in a video and removes everything else. No way to do anything that ain’t a photo human (specialist background removal services offer more options). I’ve noted this situation to DAE.
Oh, you also need a lot of free disk space.
I just tested this command (I’ve renamed DeepArtEffectsCli)…
.\dae removebackground -input C:\art\video\frames\frame-0001.jpg -output C:\art\video\back\test.png
That takes the image C:\art\video\frames\frame-0001.jpg and creates C:\art\video\back\test.png which is the original with the background removed - specifically it’s now transparent, which is exactly what’s needed.
The process is to take a video then extract each frame as an image which is very easy to do using ffmpeg (this is actually how videos are processed by DAE but it does that bit as part of the video styling process)
You then apply a style to an image - which one depends on the result you want - I need to try this again.
You next combine two images to create a composite which has a style applied to the background or foreground.
Finally you use ffmpeg again to take all the composite images and convert them back into a video.
There are a few problems that need addressing though…
If you’ve got a video called video.mp4 at the command line cd to that directory then…
mkdir frames
cd frames
ffmpeg -i ..\video.mp4 frame-%06d.jpg
You end up with a large number of jpegs in the frames directory called frame-000001.jpg … frame-00xxxx.jpg
DAE-cli won’t remove the background from the frames directory (reported to DAE) so you’d need to issue a command like the one near the top for every frame. I tested on a 34 second video I got from Pixabay, 1734 images. These images must be png (LOTS of disk space). Note that resolving this issue is actually fairly easy but DAE-cli needs to be run on every image - can be an overnight job.
Assuming you’ve worked around (2) then what you do next depends on whether you want to use DAE to style the background, foreground or both (in that order they each take longer than the previous and some involve an extra step)
At this point you need to combine the stupidly large quantity of images you’ve now got to create the final image - this can be done with imagemagick (can’t remember how ATM) but then again if I was doing it today I’d write a program to do it (this is fairly simple for what’s needed)
Finally you use ffmpeg to take all the finished frames and convert them back into a video - that ones - ffmpeg -i frame-%06d.jpg …\newvideo.mp4 (similar to extraction but with the arguments the other way around)
So, in essence, all that’s needed is a program that automates the annoying bits + the bits I’d do differently. Such a program is actually not that hard to write as all it’d actually do is call other programs, specifically dae and optionally ffmpeg. It’d also eat lots of disk while doing it - but then again a 1tb external USB is cheap these days so not an insurmountable problem
Thanks for the detail information. Are you going to write such a program ? As I said earlier, I think it is a great feature.
It’s of extremely low priority to me ATM and DAE may add (some) of it anyway - the bulk removal of background looks possible now I’ve mentioned it.
I’ll get around to it when I want a break from other development (or when I actually need it)
I’m sure someone will make some really cool video from it, flog it for a large amount of cash (not a great motivational concept)
Hello Peardox
Thanks for your reply. When and if you get around to removing the background from videos and coming up with
different filters- let me know! I think you are right that it would make for cool videos! Regards
I’ve got around to experimenting and created a 10 second clip in 540p (I’ll be writing an article on it and 540p lets me show four videos in one HD frame)
Outstanding! Peardox… I look forward to the article that you will write. Where/when will the article appear?
This video is like a Christmas Gift!! Regards Bob
I write (very occasional) articles in my Blog at https://peardox.com (there are a load of DAE ones, all over a year old)
My experimentation is slow, my crap computing gear is a major issue (it’s all 5+ years old 'cos I’m skint). Also, I’m experimenting… so make a few mistakes - some big ones in this case were the size and framerate of the video I chose as a test.
The original video is 2560x1440 at 50 fps. Changing this to 960x540 made a huge difference in the most time-consuming part - rendering images with artfilter… Basically I’m getting about 3.33 fps - abut 2.5 hours to render the 500 frames. A similar (marginally less) amount of time is required for background removal but frame size is not a huge issue there.
Total time using DAE for the 10 second clip was therefore 4h 47m, then it hit me(just now actually) … drop the framerate to 25 fps for the next renders and it’ll halve the time (duhh - stupid me, experimentation is like this, learn from your mistakes)
Once I have all the individual images overlaying all 500 took 57 seconds and converting the images back into a video a similar time.
I’m also trying out removebg which does single frame removal differently and paid to get the video background removed (one reason it’s 10 seconds) - I was not impressed with the video background removal (it cost me £2.79 - which is a lot when you have a tenner in the bank - hell, I never have much more than that anyway…)
Ahh, the joy of experimentation. Just got CLI working on my Mac - over 6fps, twice the speed (aided by another trick)!
So, not posted on this subject for five days - the reason…
I had my aging Mac processing a (960x540) 10 second, 25fps video clip (250 frames per clip) frame by frame using all 121 available DAE Artfilters (that’s 30,250 images) - total time spent processing = 3 days 21 hours 31 mins (finished about an hour ago).
It’s worth noting that my Mac is a 2013 MacBook Pro - i.e. things would be a lot faster on the modern kit you probably have but I can’t afford (A decent CPU, Memory and a NVIDIA GPU running on Linux would probably be the ideal solution)
As a result I’ve now got 4.5Gb of jpgs (and 250 pngs for background removal)
The average size of a 10 second style directory is 35.5Mb - c. 145k per 960x540 image
The Background removed directory is 315Mb - 1.26Mb per image. This is the most important directory as it enables one image to be overlaid on another with or without a style applied to the foreground and / or background
There was one DAE crash (took me eight hours to notice)
That point reminds me why I’ve been so lax regarding DAE stuff for a long while… About a year ago I was processing a 36 minute video using the (then flaky) Mac GUI and it crashed after a week and a half - this can’t actually happen with the new setup, think I’ll give it another go…
Just located that video again - its a 4.2Gb MOV (c/o my also aging IPad Pro)