Differences in EOS web gallery generation time (new vs. updated gallery)

[FairfieldPhoto]

So I have my new PC and am doing performance benchmarks against my old system.

I have found something unusual.  Perhaps you can shed some light on it.

My most recent equestrian event had 3392 jpg images from a Nikon D2Hs.

When I run PM with EOS export, I can create a brand new gallery in 33 minutes.  31 of those minutes is generating the thumbnails and resized images.  2 of those minutes are generating all the HTML.

Now, I go back and run the export again so I can "re-use existing images".  This is to benchmark a gallery refresh where I may add a few new images.  The total time to generate the 3392 image gallery drops to 18.3 minutes when "re-using existing images" yet the part of the process that handles looking at the JPG images runs in 12.5 minute and the HTML file generation jumps to just over 6 minutes. 

I would have thought that the HTML generation part for EOS would have been the same in both generation methods (new vs. refresh).

Can you confirm this behavior on the development system?  It would be nice to get those 4 minutes back when you have customers standing in line to look at new photos.

For those still reading the post at this point, the new Core2Duo 2.16Ghz is 33% faster in EOS generation than the Pentium 4 2.6Ghz uni-processor I was using before.

-Mike

Background:
PM 4.5.1 Beta
Windows XP Pro
Core2Duo 2.16Ghz, 2Gb RAM, w/ 160Gb RAID 0 drives as primary while writing web gallery to separate 400Gb drive.  Seatbelt for the driver.

[Kirk Baker]

Mike,

When I run PM with EOS export, I can create a brand new gallery in 33 minutes.  31 of those minutes is generating the thumbnails and resized images.  2 of those minutes are generating all the HTML.

Now, I go back and run the export again so I can "re-use existing images".  This is to benchmark a gallery refresh where I may add a few new images.  The total time to generate the 3392 image gallery drops to 18.3 minutes when "re-using existing images" yet the part of the process that handles looking at the JPG images runs in 12.5 minute and the HTML file generation jumps to just over 6 minutes. 

I would have thought that the HTML generation part for EOS would have been the same in both generation methods (new vs. refresh).

We expect it would be the same too.  Did the settings change such that the template had more HTML to generate or its complexity was increased?

While looking into this we found ways to optimize the checking for reasonable (right sized) existing images and for me on the Mac, checking for 100 existing images now takes about a second instead of the former 25 seconds.  It should help quite a bit on 3392 images, though I am not certain my results on Mac OS X will be guaranteed on Windows, but it should be better nonetheless.

-Kirk

[FairfieldPhoto]

I look forward to seeing the performance bump!

I was using the exact same input folder and parameters.  The only difference was turning "on" the re-use existing images.  I tried to keep all other variables the same.

To replicate the condition, take 500 images (a big enough chuck of files to see the performance difference), generate a gallery from scratch (measuring the JPG time and the HTML time), then re-run exact test again with the same input folder, point to the same output gallery with only changing the "re-use images" setting and again, measure the jpg generation time and the HTML generation time. 

You should be able to see the difference (or not if the new fix addresses this).

-Mike

[FairfieldPhoto]

UPDATE with 4.5.2 Beta II code base.

WOW! The refresh speed is incredible.  Running on my Core2Duo with Windows XP Pro, 2GB RAM, and 2x 160Gb hard drives in RAID 0 (striped) mode results in a refresh speed of 6:30, down from 18:20 with 3392 images.  The image processing phase lasts just SEVEN SECONDS!!!  The rest of the time is HTML file generation.  WOW.  Tighten the seatbelts!

Now the bad news.  Where the original post pointed out that a full refresh (not reusing existing images) could be completed in 33:04 (again, the same directory of 3392 images), the original split was 31:00 in image generation and 2:04 in HTML file generation.  With Beta II, the image generation time dropped to 27:55, but the HTML generation added 6 minutes to bring total generation time to 33:54.  Overall speed is comparable between Beta I and Beta II, but whatever performance improvements were implemented in the JPG generation was eaten up by the change in HTML generation which now appears consistent between both full generation and reuse image modes at around 6.5 minutes.

From a severity level for this issue, I would lower this now since I don't do full refreshes all the time and have a workaround since I personally do incremental updates more often than full refreshes and the speed improvement in the incremental update (reuse images mode) is very welcome.  Anything you could do to find why Beta I had faster HTML generation and bring that to Beta II in a future enhancement release would be greatly appreciated.

-Mike

[Kirk Baker]

Mike,

UPDATE with 4.5.2 Beta II code base.

WOW! The refresh speed is incredible.  Running on my Core2Duo with Windows XP Pro, 2GB RAM, and 2x 160Gb hard drives in RAID 0 (striped) mode results in a refresh speed of 6:30, down from 18:20 with 3392 images.  The image processing phase lasts just SEVEN SECONDS!!!  The rest of the time is HTML file generation.  WOW.  Tighten the seatbelts!

Now the bad news.  Where the original post pointed out that a full refresh (not reusing existing images) could be completed in 33:04 (again, the same directory of 3392 images), the original split was 31:00 in image generation and 2:04 in HTML file generation.  With Beta II, the image generation time dropped to 27:55, but the HTML generation added 6 minutes to bring total generation time to 33:54.  Overall speed is comparable between Beta I and Beta II, but whatever performance improvements were implemented in the JPG generation was eaten up by the change in HTML generation which now appears consistent between both full generation and reuse image modes at around 6.5 minutes.

From a severity level for this issue, I would lower this now since I don't do full refreshes all the time and have a workaround since I personally do incremental updates more often than full refreshes and the speed improvement in the incremental update (reuse images mode) is very welcome.  Anything you could do to find why Beta I had faster HTML generation and bring that to Beta II in a future enhancement release would be greatly appreciated.

We'll see what we can do before 4.5.1 goes final.  I'll keep you posted.

-Kirk