Issue with contact sheets and large Tiff's

[maurits]

Hi, since a couple of months I am doing a lot of analog photography. I scan my negatives and the files are ordered in folder sets per film, or per shoot. So sometimes I have a folder containing 10 or more 100 MB + files. While my PM never had a problem with opening contact sheets containing lots of Nikon 12 MB NEF's, it now takes forever to load a page. Sometimes over a minute. When I preview a single file it can take up to 20 seconds or more for the single image to render. Zooming the same image to a 100% regularly takes over two minutes (per image).

More often than not, PM starts using so much CPU that other apps no longer operate. For example, my scanner simply stops scanning until I Force Quit PM. Force Quitting PM is about my only remedy to stop PM from taking 20 to 50% CPU. It rarely happens that PM settles and releases the CPU usage within a reasonable time frame, say a minute. So I simply Force Quit and restart PM. Working with scans from 4x5 negs (about 350 MB each) is near impossible.

Programs like Photoshop or Bridge render the same files and contact sheets in fractions of seconds.

I unchecked "high quality thumbnails" (which is a shame because now my contact sheets look plain ugly most of the time), but that does not help. My disk cache is 1024 MB, with 512 MB reserved on the disk cache volume. Cached files older than 15 days are purged. Memory cache is 700 MB. I am running Photo Mechanic version 4.6.2.1 and Mac OS X 10.6.2 on my Intel-based Mac (2.33 Core 2 Duo) with 3 GB RAM.

All in all, I desperately need some advise on how to properly setup PM to handle my sheets with large TIff's!

Cheers, Maurits

[Kirk Baker]

Maurits,

Hi, since a couple of months I am doing a lot of analog photography. I scan my negatives and the files are ordered in folder sets per film, or per shoot. So sometimes I have a folder containing 10 or more 100 MB + files. While my PM never had a problem with opening contact sheets containing lots of Nikon 12 MB NEF's, it now takes forever to load a page. Sometimes over a minute. When I preview a single file it can take up to 20 seconds or more for the single image to render. Zooming the same image to a 100% regularly takes over two minutes (per image).

More often than not, PM starts using so much CPU that other apps no longer operate. For example, my scanner simply stops scanning until I Force Quit PM. Force Quitting PM is about my only remedy to stop PM from taking 20 to 50% CPU. It rarely happens that PM settles and releases the CPU usage within a reasonable time frame, say a minute. So I simply Force Quit and restart PM. Working with scans from 4x5 negs (about 350 MB each) is near impossible.

Programs like Photoshop or Bridge render the same files and contact sheets in fractions of seconds.

I unchecked "high quality thumbnails" (which is a shame because now my contact sheets look plain ugly most of the time), but that does not help. My disk cache is 1024 MB, with 512 MB reserved on the disk cache volume. Cached files older than 15 days are purged. Memory cache is 700 MB. I am running Photo Mechanic version 4.6.2.1 and Mac OS X 10.6.2 on my Intel-based Mac (2.33 Core 2 Duo) with 3 GB RAM.

All in all, I desperately need some advise on how to properly setup PM to handle my sheets with large TIff's!

Photo Mechanic isn't designed to browse extremely large images efficiently.  It is designed to render images from cameras efficiently.  What resolution are the TIFF files?  Are they 8-bit or 16-bit?

-Kirk

[maurits]

Hello Kirk, most are around 6500 x 7000 px. Black and white are 16 bit, color 24 bit.

[Hayo Baan]

Hmm, as a test I just opened a contact sheet with 136 NEFs (15-20MB) and 6 extreme panorama's (1-1.5GB each!) and do not suffer any computer slowdown to speak of. It does take a while before all of the images get their proper thumbnail, but at least I can continue working on something else.

Note, during the processing of the contact sheet, one of my CPU cores is maxed-out (which in my case still leaves 7 idle), and also the HD is at (almost) full throughput (±90MB/s). My images are all on a second internal HD though so my "normal" work is not really hampered by the load on that HD.

For comparison I tried to open the exact same folder in Adobe Bridge to see how that behaves. Well, CPU usage was a bit higher while disk load was lower. The thumbnails for the extreme tiffs were not updated from the very small initial size, however; no wonder it loaded quicker. Maybe my  tiffs are actually too big (they're >30.000x4000) because the preview in Bridge didn't render them any bigger either. Quite useless 

Based on this, a couple of things that may be relevant in your case:
* how many cores does your system have? 1, 2, more?
* how much cpu does the scanning alone consume?
* do you have a fast HD, and do you have everything on one HD or spread-out over more?
* how many MBs are your TIFFs?

And as Kirk already mentioned, it is not to be expected that (large) TIFFs display anyway quickly. One of the reasons being that they do not contain a high res preview (as do most camera files) and that in turn means the tiff needs to be processed completely before it can be rendered...

Hope this explains things a bit and/or points you in a direction to get some more speed.

Cheers,
    Hayo

[maurits]

Hi Hayo, thanks for your elaborate answer!

My system has 1 core 2 duo processor, so 2 cores I guess. I have a reasonably fast drive, 500 Gb 7000 rpm. But just one drive. I guess I should start saving for that Mac Pro. My TIFFs are typically between 79 and 125 MB each. No folder contains more than 12 of these files.

Scanning consumes between 5 and 10% CPU. When I browse files during scanning, PM's processor load varies between 20 and 30%. Bridge uses the same percentage while browsing or opening a folder, sometimes even more (up to over 100% for a very short time). I was wrong thinking Bridge displayed the contents of a folder quicker. It was not quicker when I tested today. It does, however, seem to release the processor quicker after finishing building thumbs.

Previewing at fit to screen size and zooming to 100% in Bridge is definitely much quicker. After the thumbs were built, I could preview and zoom to 100% for each individual file immediately. CPU went up to 137% for a few seconds initially and then dropped to a steady 0.5%. In this folder there were 6 TIFFs, one 45 MB, three 100 MB and two 300 MB scans.

The same action in the same folder took PM about 5 to 25 seconds to render a fit to screen size preview for each individual file and over a minute to render a 100% view ( I did not use any other application at the same time). One 300 MB file was rendered only after more than two minutes and the last 300 MB file actually never rendered at 100% at all. CPU usage was between 166% and 29%. Sometimes it dropped down to around 4.5% for 20 seconds or more while rendering, without anything actually happening. Real memory usage jumped from 700 MB to 2 GB during the whole process. When closing the zoom, PM started building thumbs of the same files in the same folder all over again? This process took another three minutes using over 30% CPU constantly and 2 GB real memory. After that time, only after 6 minutes (doing nothing else in between), PM settled down to 4,5% CPU and 700 MB real memory again. I usually do not wait that long...

So my gripe basically is, why are there these enormous differences in rendering times and the long aftermath before everything has settled down?

Cheers, Maurits

 

[Kirk Baker]

Maurits,

Scanning consumes between 5 and 10% CPU. When I browse files during scanning, PM's processor load varies between 20 and 30%. Bridge uses the same percentage while browsing or opening a folder, sometimes even more (up to over 100% for a very short time). I was wrong thinking Bridge displayed the contents of a folder quicker. It was not quicker when I tested today. It does, however, seem to release the processor quicker after finishing building thumbs.

Previewing at fit to screen size and zooming to 100% in Bridge is definitely much quicker. After the thumbs were built, I could preview and zoom to 100% for each individual file immediately. CPU went up to 137% for a few seconds initially and then dropped to a steady 0.5%. In this folder there were 6 TIFFs, one 45 MB, three 100 MB and two 300 MB scans.

The same action in the same folder took PM about 5 to 25 seconds to render a fit to screen size preview for each individual file and over a minute to render a 100% view ( I did not use any other application at the same time). One 300 MB file was rendered only after more than two minutes and the last 300 MB file actually never rendered at 100% at all. CPU usage was between 166% and 29%. Sometimes it dropped down to around 4.5% for 20 seconds or more while rendering, without anything actually happening. Real memory usage jumped from 700 MB to 2 GB during the whole process. When closing the zoom, PM started building thumbs of the same files in the same folder all over again? This process took another three minutes using over 30% CPU constantly and 2 GB real memory. After that time, only after 6 minutes (doing nothing else in between), PM settled down to 4,5% CPU and 700 MB real memory again. I usually do not wait that long...

So my gripe basically is, why are there these enormous differences in rendering times and the long aftermath before everything has settled down?

Photo Mechanic is trying to build multiple previews in the direction you're browsing in the Preview window.  Because your images are so large, they use up all of the available memory and portions of the memory cache are flushed in order to try and fulfill your image Preview requests.  This results in cache 'thrashing'.  Reducing the size of your Memory Cache in PM by half may help somewhat.

TIFF files do not contain a preview at all (well sometimes they contain a low resolution thumbnail, but it isn't suitable) and so rendering them requires reading the entire file and scaling the full size image for each image request (even for a high quality thumbnail).  TIFF is an especially bad format for fast image previews.  Adobe's products can do something called 'image tiling' for which they have a patent on and Photo Mechanic cannot use this method (or we could get sued by Adobe for patent infringement) which is much more efficient once the tiles are created.

My short answer to you earlier was that PM is not optimized for viewing anything but images from digital cameras and that explanation while shorter than the above is still correct.  If you saved your images out as 20 megapixel JPEGs, your experience would be much better.  Basically, TIFF is just an especially bad format for PM for fast previewing.

-Kirk

[maurits]

Hi Kirk, thanks. I will have to live with this then...

Would it be possible to have PM handle TIFFs the same as RAW files (or only large size TIFFs), if one so chooses? I mean PM could show TIFFs and JPEGs combined (when they have the same name before the file extension) like it does for RAW+JPEG. The JPEGs could then be used for the thumbnails and for previewing and zooming and the TIFF sent to the image editing software when double clicked upon.

To be able to use this function it would, of course, be up the user to generate reasonable quality JPEGs from the TIFFs in the same folder.

Cheers, Maurits

[Kirk Baker]

Maurits,

Hi Kirk, thanks. I will have to live with this then...

Would it be possible to have PM handle TIFFs the same as RAW files (or only large size TIFFs), if one so chooses? I mean PM could show TIFFs and JPEGs combined (when they have the same name before the file extension) like it does for RAW+JPEG. The JPEGs could then be used for the thumbnails and for previewing and zooming and the TIFF sent to the image editing software when double clicked upon.

To be able to use this function it would, of course, be up the user to generate reasonable quality JPEGs from the TIFFs in the same folder.

Others have requested being able to pair up file types other than just RAW and JPEG.  We've considered doing it in the past but decided not to implement it because the number of changes to the application would be quite larger than we're willing to do at this time.

-Kirk

[Hayo Baan]

Others have requested being able to pair up file types other than just RAW and JPEG.  We've considered doing it in the past but decided not to implement it because the number of changes to the application would be quite larger than we're willing to do at this time.

Me being one of them 
Sad to hear you're not going to implement it any time soon though

This is a function that definitely (?) should be in the catalogue part though (one way or the other), wouldn't you agree? 

[Kirk Baker]

Hayo,

Others have requested being able to pair up file types other than just RAW and JPEG.  We've considered doing it in the past but decided not to implement it because the number of changes to the application would be quite larger than we're willing to do at this time.

Me being one of them 
Sad to hear you're not going to implement it any time soon though

This is a function that definitely (?) should be in the catalogue part though (one way or the other), wouldn't you agree? 

Please explain how cataloging has any impact on which images should be combined together and treated as a single image?

-Kirk

[Hayo Baan]

Well, part of the catalogue functionality, to me, would be to be able to see the images I want, with e.g. alternatives hidden until I specifically ask for them. I think Adobe calls this "stacks" and if I recall correctly from an earlier conversation, you were planning on implementing this. From stacks, it is a very small step to (automatically) combining (stacking) images of a different type. A preference order to the image-type you want to have shown for the combination (stack) would complete this.

I think this does make sense, and hope it does to you too

Thanks,
    Hayo

[Kevin M. Cox]

Reading this made me curious so I went and got my Canon EOS 1D "classic" out of the closet. It's RAW files are .tif instead of .cr2 like the newer bodies and Photo Mechanic combines these .tif RAW+JPEGs as a single file in the contact sheet.

I'm just curious, what is it about these ".tif" files that makes them combinable with JPEGs but TIFFs created in Photoshop not?

Maurits, this obviously isn't the optimal solution, but what about this as a workaround:

Create the JPEG copies for fast previews and then rename your TIFFs with the file extension .cr2   This will cause Photo Mechanic to combine the thumbnails in the contact sheet and speed up browsing I assume. When you open the .cr2 for editing Photoshop should recognize it as a TIFF file and open it correctly (at least it does in limited testing just now).

[maurits]

 
Kevin, excellent! I tried with two folders and my previews of full size JPEGs are much, much quicker. Even at 100%. Photoshop recognizes all renamed originals as TIFFs, with adjustments layers and all.

I did rename the extension to NEF though, being a Nikon guy... 

This improves my situation greatly. Many thanks!

Cheers, Maurits

[Kirk Baker]

Reading this made me curious so I went and got my Canon EOS 1D "classic" out of the closet. It's RAW files are .tif instead of .cr2 like the newer bodies and Photo Mechanic combines these .tif RAW+JPEGs as a single file in the contact sheet.

I'm just curious, what is it about these ".tif" files that makes them combinable with JPEGs but TIFFs created in Photoshop not?

TIFF files from the 1D are actually RAW files.  When PM sees a TIFF file it checks to see if it is really a RAW file from the 1D or 1Ds.  Canon later changed to the CRW and CR2 extension for its RAW file format.

-Kirk