I wanted to compare but yeah - they’re the same
It threw out some warnings (it’s an already reported GitHub issue) when converting which was why I was checking properly
So that’s what the model looks like- hmmm
1 Like
Onnx is just a more optimized model file for most parts… you can see the difference when using on extremely large renders (more artifacts) but on standard sizes unless using something like imagej for some forensics compare to the naked eye it’s almost impossible to see.
Edit: Indeed that’s what a model file looks like had to refresh my browser to see your attachment. It’s quite impressive as technology like mentioned previously having the financial resources I’d be re-compiling the vgg19 pack to get the most out of the models but right now with world events spending on toys unfortunately has to wait. Getting a new cli interface almost finished and retrieving all builds and packs…
I’ve just been reading the Python that does the training…
Coco 2014 has 82,783 images of varying sizes
Coco 2017 has 118,287 has Coco 2014 plus c35k more - again, all sizes
Training want images to be square so it therefore resizes each image to, by default, 256x256 retaining aspect then crops it to size.
Fairly obviously this step can be done once so all your training images are of the desired size for training. OK, you might want say 512x512 training images but that’s another story.
The resize and crop won’t take long but it’s completely obvious that if the images are already the desired size for training then some time will be saved.
Gonna resize all training images to 256 for now then again to 512 later and compare results in training time for one epoch
1 Like
Indeed if you optimize the source from starter you’ll save processing / wear time…
It took 18 mins on old PC (now running Linux) to convert all images to 256x256 so a good percentage of that should be saved not having Python doing it EVERY TIME…
More thought and I see that the 118k images in coco2017 could do with better organisation and this is something else I can automate when converting the images. Working it out if there images were stored three directories deep with 19 dirs in each level (00-18) then each image would belong in a 3rd level dir with a total of 18 other files - e.g. 01/12/08/somefile00-18.jpg. This would emmensely speed up disk access (having 118k files in one dir is a terrible idea) - some fairly simple maths works all this mapping out…
With luck I’ll get a nice speed improvement from this simple re-arangement and re-encoding of files which is really good for training.
There’s always HFS as an option as well.
I sorta guess reaserchers into AI either have no idea about analysis or they’re not bothered how long things take.
Oh, year - the space taken by the training images for coco2014 :@ 256z256 is 1.2G rather than 13G 
1 Like
Well that’s small enough to run it all on a ramdrive at that point…might give that a try… spending all day today sorting my bookmarks and tools and finishing off the more automated cli tool that I’ve been working on… A shame the daily are just repeats of the other 122 and not a new daily each day of the year.
Shawn
Done the resizing on Coco 2017 to 256 = 1.8G, currently running 512 (it’ll be somewhat lager obviously - just donnu how much bugger for c 30 mins)
Just found this as well which may be of interest - note the lite version is directly downloadable (25k images) GitHub - unsplash/datasets: 🎁 3,000,000+ Unsplash images made available for research and machine learning
1 Like
You reminded me of this
that links to https://www.deeparteffects.com/service/liststyles?e=1
But the main thing is that the main site programmableweb contains api’s to many image resource tables… could be a use for you as for unisplash I had seen it but wasn’t sure of the results I’d end up with.
I’ve finished conversion of images for transfer training using ~118k coco 2017
Original = 19G
256 = 1.8G
512 = 5.0G
Gonna try training sketch preview vs 512 to see what that’s like
I’ve dropped to 1 epoch from 2 (change really not visible) so training has been taking an hour now (it’ll go up again when I switch to 2017 tho)
The resizing of images made very little difference in training - it’s still got the Python code in there to resize (which shouldn’t do anything - I’ll have to try with the resize removed)
1 Like
Indeed if the python code is still there it’s still going to process the image… removing the routine should help as long as the conversion process of your final images are the same there shouldn’t be any difference in the final product.
The 512 could of went up to 7.2G however I suspect that the reduction of colors and the jpg compression helps a lot.
As for solving my 82h issue well with the ram drive seems I’m able to drop it by 50% so down to 40ish hours… not great but if I can drop it down to 1 a day model creation I’ll be happy with that the point is not having tons of models but good ones to work with.
3:30 pm here planning on doing a 48h on staying up to get lots of things done everything pointing in the right direction so no stopping the momentum now.
As for the epoch only makes a difference if you do 4+ 1-2 same for me no distinct difference.
Rats - 512 = 7.2Gb, that explains my OOM
I’ve only got 6G (Laptop) so with some fiddling around it appears my max is 384 (really close to 6Gb limit - Task Manager says I’m on 5.8 or 9)
Just wondering what Sketch will look like at that size (if it don’t OOM near the end - crossing things)
I’ve already planned to try a high epoc setting overnight - mind you I’ve got 42% more images using Coco2017 so that should help as well
1 Like
Sorry about being the bearer of bad new on that Gb size unfortunately a double xy is a 4times in size 384 should be about 5.5Gb indeed a push to your max limit.
We’ll keep those fingers crossed I’m also searching online for some decent pc’s to use as remotes for this process once I have a decent idea on how I want to proceed.
Indeed the bigger the sampler the better the chances I know I’ll probably check on archive.org for some iso compilations of packs and see what I can get from them or use firefox without a cache limit and use something like scrolller.com or anything with endless load for images and just build a library from there.
Well good luck with the results who knows theses tests may help improve DAE in the long run by suggesting things as normal users already have issues with a CLI I don’t see them doing what we do.
Just been looking at a Cloud GPU offering I might try out when I get cash
Can get a 3070 / 8GB with Ubuntu and 16GB RAM for about a buck an hour (can’t afford many hours really … dead poor) - might give it a go end of month (as they bill the following month)
That would allow for a quick test of 512 rendering
Making the image 384 has really hit my training speed - 45 mins and I’m at 34k/118k (~30%) so a full train is gonna be 2.5 hours at this rate (hope its worth it)
Oh - Coco2017 @ 384 = 3.8G
1 Like
Welcome to the club my last splurge was on DAE in attempting to make a business unfortunately at the same time the world went nuts and priorities have changed… from covid to monkeypox scam to countries giving their medical freedom to the WHO needless to say I’m on the rush to make sure the household ain’t missing nothing before the next fiasco… but we’ll get there and a buck an hour not bad considering the wear and tear + electrical costs of running a gpu… I’ll see what I can do to help with this en-devour of a project.
One hour remaining for a 6 epoch Coco2017 @ 256 (wonder what it’ll look like)
Also - AWS Activate for Startups, Founders, & Entrepreneurs
That’s enough for 1 month of A10G 24G GPU with the possibility of more
… And I just ttried starting one but it’s been sent for review 
You can get a very big bill if you’re not ulta careful so I guess this is sensible
+++++++++++++++++++++++++++++++++++++++++++++++++++
Discord Channel (for anyone who wants to chat actually) - Peardox