Overlaying Styles in Video

[Peardox]

This video is a split into five 10 second chunks.

1. The first 10 seconds are the original clip - the starting point
2. The second 10 seconds are the clip styled with Comic - you can do this in DAE anyway
3. The third 10 seconds are the clip styled with Sketch 1 - again you can do this in DAE anyway

Now to the fun stuff you can’t do with DAE…

4. The fourth 10 seconds are Comic (2) overlaid over Sketch 1 (3)
5. The last 10 seconds are Sketch 1 (3) overlaid over Comic (2)

You can, obviously, also do overlays with the unstyled clip (1) as well so there are another four variants I didn’t bother creating.

[bob]

Happy New Year… Congrats on overlaying styles in video. Nice Nice!!

Question 1… How do you combine the two videos so as to emphasize only background. What program or magic did you use for steps 4 and 5. Programs that I have do not allow me to emphasize one part of video vs other.

I attempted another approach than yours. You went for the complete background change -which is great - and I went for various aspects of video.

1. I converted video in DAE…man running
2. Took man out of video using Unscreen. (you are correct-it is glitchy)
3. Put man into different backgrounds.
   Problem with this approach is trying to take person or object out of video using Unscreen.
   Your approach offers other possibilities…

Finally… There is a glitch in DAE… (sorry DAE works well converting to different styles but when you go to save it sometimes saves video and all is well or it lets you put in name of file but then it starts to process the video again!! It comes and goes. I suggest you try converting videos for about an hour on and off and see if you can replicate the glitch or bug.

[Peardox]

My Mac’s currently on frame 59,160 of 66,150 of background removal for a 35 min video - it’ll finish in about 24 hours…

The Alpha Unscreen creates is binary - something is either in the foreground or in the background.

The Alpha DAE creates is more of a ‘how sure are we that this pixel is in the background / foreground?’ with values ranging from 0 to 255 (0 = 100% its background)

Taking either of these definitions you can simply invert each alpha frame which gives you a foreground removal result rather than a background removal one.

1. I took my original test clip and extracted all the frames with ffmpeg
2. Then I ran background removal on each individual frame - this is the only version with alpha I used
3. I then ran every frame from (1) through all 121 styles resulting in a huge quantity of styled frames
4. The complicated bit - for each frame of one of the styles (this will be new background) and another style (the new foreground)…
   a) Load background
   b) Load foreground
   c) Load alpha (from 2)
   d) Apply (c) to (b) - creating a version of (b) with alpha
   e) Overlay (d) on (a)
5. Use ffmpeg again to build the new clip with frames version (e)

It sounds more complicated than it is

To do this all you need is ffmeg, a way to automate background removal (DAE bug) and something to do (4) above

I’ll write some command line apps to do the missing bits and although DAE CLI BGR a directory should be fixed sometime you’ll defo need an app to do the overlay stuff.

It should be noted that you can retain the sound-track from a video as well, it’s just an extra ffmpeg step in the process - and in 6650-ish frames time I’ll have the source to do just that…

Tutorial to follow (sometime in Jan I hope)

[Peardox]

Bugger - my Pi4 crashed a week ago while rendering frame 421 of style Charles and I didn’t notice…

Dec 22 02:13 frame_000421.jpg

Same happened on the Mac when prepping for the overlay video but I spotted that one quickly enough (only 8 hours lost)…

Last log entry…

2020-12-22 02:14:03,033 DEBUG c.d.r.s.ArtFilter [main] downloading style 87c1ca55-1b84-11e7-afe2-06d95fe194ed (Charles)

My own fault for not checking things are still running, it had been happily running for 3.5 days when it died and had completed 10,021 frames in 16 styles and most of Charles

Reboot Pi4, edit scripts, continue

[10 mins later] Back in business - Charles 422 rendering

[kingofpain]

hey buddy, when I try to do higher res higher optimisation count artistic ai renders, the timer eventually goes negative and never finishes. even in lower chunk sizes. any ideas?

[Peardox]

Annoying, ain’t it? That specific problem is usually to do with Memory (not having enough).

Basically, if something like this happens and you’re using the GUI (as you obviously are), the first thing I’d try is rebooting your PC then as soon as you’re back in control don’t open anything (a web browser like Chrome eats memory for example). Just start DAE and try again and see if that fixes the problem (usually does).

Note that simply closing Chrome and trying DAE again is not the same thing as never starting Chrome in the first place…

If you looked in the log file you’d actually see it crashing (not always, but often) - the problem is that DAE’s GUI does nothing to recover from this situation - instead it just keeps on counting down, eventually going negative.

If the restart + run DAE trick don’t work then the next option is to use the command-line version - you even get to watch it crash if you’re unlucky enough / do it on purpose (huge image with huge chunks usually does the trick). Of course by using the CLI version you’re using even less memory.

DAE needs real memory available, the more the better - your swap file can help a little but only a little (eventually it won’t have anything it can swap out)

Personally I’ve got a lot of dev stuff on my main PC so usually hand off DAE stuff to another machine, the Mac for anything I want reasonably soon - which reminds me… Pi 4 is on Yira - almost finished it’s run

[kingofpain]

uses a lot of friggin memory! i have 32Gb ram, i7 3770K at 4.5 watercooled, 980Ti watercooled. (6Gb vram)
so riddle me this; it doesn’t use virtual memory at all? please advise. should I change system settings to manually or automatically allot virtual memory?

thanks

[Peardox]

Ahh, nobody ever mentions if they’re using a GPU… Assuming you’re using the GPU version of DAE things get more complicated.

When I had a working GPU (a year ago) I was able to test GPU versions. In that specific case I found that it appeared that DAE uploads the entire image to your GPU then runs everything there. I have a (currently disabled) 2Gb vram GPU and that broke as you describe when processing 1080p images.

The solution to this was to switch to the CPU version - it’s possible but messy to have both CPU + GPU versions of DAE installed at the same time (hey @Memnon make the GPU version co-habit with the CPU version pls) - I’ll have to write a Tutorial on this one…

It may sound counter-intuitive that your GPU (using the GPU version) can’t handle a 4K image while the same system will quite happily run 4K / 8K games. The fact is that any AI struggles with large images as it creates a huge number of different versions of the original image while processing the image (this is a vast over-simplification of what’s going on but is basically true). As image size doubles the memory required quadruples - 4k is, conceptually, four 1080p images in a square (important word mathematically - square)

So - still memory, but this time GPU Memory (and I can’t tell you what the limits are as I’ve got crap kit)

[Memnon]

As Peardox mention’ed it’s all about free memory which is the limiting factor.
If you’re using your GPU, it will render faster, but the memory of your GPU will be the limiting factor at some point.
If you’re using the CPU, it will render at normal speed, but the free memory of your system is the limiting factor.

You can check in your logs how much free memory your system has at time of startup of DAE.

• Open Deep Art Effects and select “ info ” --> “ about ” in the top menu bar.
• In the about dialog there is a button called “ show debug log ”
• A new window containing the log will appear

What you want to look for is this line: (example) RAM: 8695MB of 16333MB available
As you see I have a 16 GB rig, but right now there’s about 8GB free.
DAE requires 4 for most render processes, at higher settings or with longer processes such as artistic AI or bulk rendering, you’ll need more. If it runs out of free memory, the program will run into problems at the moment.

[Peardox]

For completeness sake it is possible to check your GPU’s memory usage as well using a command-line tool called nvidia-smi,

Although this tool is aimed at specialized use (e.g. server GPU farms) it does provide the information you need to check your GPU’s memory usage which is what we’re interested in.

For the command-line adverse I would suggest checking out GPU-Z as that may prove useful as well.

If I can get my GPU to stay on for more than a minute (my PC needs some TLC / replacing) I’ll test it out at the w/e so I can give you more detailed info.