Author Topic: TIFF Performance still very slow - anything I can do about this?  (Read 14926 times)

Offline Neil MacAvoy

  • Newcomer
  • *
  • Posts: 13
    • View Profile

I mentioned this problem a few months ago and am getting increasingly convinced that there's some performance issue with how PM handles large TIFF files.  I'm wondering if there's anything I'm doing wrong and/or if others are seeing a similar problem.

The description is pretty simple: When I try to browse through a folder of large (71 MB) TIFF files generated from Nikon Capture, PM takes forever to generate high quality views. I have literally seen it take > 30 seconds to switch from the low-res preview to a high-res preview. This happens a lot. If I try to browse forward, in a short time I hit an image that's waiting to refresh, and then it's time to get another cup of coffee. The CPU utilization is typically down around 5% during the wait period.  What initially attracted me to PM was the blinding fast speed on JPG and NEF previews, but now that my workflow includes TIFF it's suddenly a lot less interesting.

When I brought this up before, Kirk suggested PM could do nothing, since the bottleneck had to be the disk IO. Here's why I don't think that's the case: Both Adobe Bridge and PSCS2 can read these TIFF files about 10X faster, or so it appears.  If I point Bridge at a fresh folder of TIFFs (in other words, Bridge hasn't had a chance to build its cache), and I use "filmstrip" mode to get a decent size preview, if I click on one of the TIFF icons, after about 3 seconds, I get a nearly full-screen high-res preview. I don't think Bridge is using a preview embedded in the TIFF, but maybe I'm wrong about that. In any case, the preview Bridge gives me in 3 seconds is much better than the one that PM is showing me for 30 seconds. In Bridge, if I advance through the TIFFs, there's again about a 3 second delay while the file gets read, then I'm in business. So Bridge is currently a much faster browser for me for these files, which just doesn't make sense.  I get even faster TIFF loads in PSCS2; In under 2 seconds, I can load a large TIFF and start editing it. Likewise, if I use PM to select some TIFF files and move them to another hard disk, they don't take ~30 seconds per file to move. So....How is this a disk IO problem? Seems like my disk is serving up the bits to the application just fine.  I'm wondering if PM is trying to read ahead too much and just chokes, but really have no idea.

Some logistical details:

Windows XP SP2, all the patches accepted, etc.
Intel 840EE dual-core, running 3.6 GHz
2 GB ram
400 MB RAID disks. Defragmented regularly, including yesterday.
Running latest PM 4.4.3, but I don't think this is a new issue for me.
Tried various settings of disk and memory cache in PM. Currently
using 2048 MB disk cache and 512 MB ram cache.
Issue seen using Nikon D2X TIFFs, AdobeRGB colorspace, doesn't matter if I use NCapture or ACR3.X to generate the TIFFs.

- Has anyone else seen the same problem with large TIFFs?
- Am I the only person trying to browse a lot of large files? (I seriously doubt that..)

Kirk - if you send me your mailing address, I'd be glad to mail you a bunch of TIFF files so you can see if you can duplicate the PM vs. PSCS2 performance delta.  I love the interface of your program and am desparate to figure out why I'm seeing such slow performance. Thanks,

-Neil


Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: TIFF Performance still very slow - anything I can do about this?
« Reply #1 on: May 11, 2006, 09:46:38 AM »
I mentioned this problem a few months ago and am getting increasingly convinced that there's some performance issue with how PM handles large TIFF files.  I'm wondering if there's anything I'm doing wrong and/or if others are seeing a similar problem.

The description is pretty simple: When I try to browse through a folder of large (71 MB) TIFF files generated from Nikon Capture, PM takes forever to generate high quality views. I have literally seen it take > 30 seconds to switch from the low-res preview to a high-res preview. This happens a lot. If I try to browse forward, in a short time I hit an image that's waiting to refresh, and then it's time to get another cup of coffee. The CPU utilization is typically down around 5% during the wait period.  What initially attracted me to PM was the blinding fast speed on JPG and NEF previews, but now that my workflow includes TIFF it's suddenly a lot less interesting.

When I brought this up before, Kirk suggested PM could do nothing, since the bottleneck had to be the disk IO. Here's why I don't think that's the case: Both Adobe Bridge and PSCS2 can read these TIFF files about 10X faster, or so it appears.  If I point Bridge at a fresh folder of TIFFs (in other words, Bridge hasn't had a chance to build its cache), and I use "filmstrip" mode to get a decent size preview, if I click on one of the TIFF icons, after about 3 seconds, I get a nearly full-screen high-res preview. I don't think Bridge is using a preview embedded in the TIFF, but maybe I'm wrong about that. In any case, the preview Bridge gives me in 3 seconds is much better than the one that PM is showing me for 30 seconds. In Bridge, if I advance through the TIFFs, there's again about a 3 second delay while the file gets read, then I'm in business. So Bridge is currently a much faster browser for me for these files, which just doesn't make sense.  I get even faster TIFF loads in PSCS2; In under 2 seconds, I can load a large TIFF and start editing it. Likewise, if I use PM to select some TIFF files and move them to another hard disk, they don't take ~30 seconds per file to move. So....How is this a disk IO problem? Seems like my disk is serving up the bits to the application just fine.  I'm wondering if PM is trying to read ahead too much and just chokes, but really have no idea.

Some logistical details:

Windows XP SP2, all the patches accepted, etc.
Intel 840EE dual-core, running 3.6 GHz
2 GB ram
400 MB RAID disks. Defragmented regularly, including yesterday.
Running latest PM 4.4.3, but I don't think this is a new issue for me.
Tried various settings of disk and memory cache in PM. Currently
using 2048 MB disk cache and 512 MB ram cache.
Issue seen using Nikon D2X TIFFs, AdobeRGB colorspace, doesn't matter if I use NCapture or ACR3.X to generate the TIFFs.

Kirk - if you send me your mailing address, I'd be glad to mail you a bunch of TIFF files so you can see if you can duplicate the PM vs. PSCS2 performance delta.  I love the interface of your program and am desparate to figure out why I'm seeing such slow performance.

Email won't work.  There isn't an email server around that will accept even one 70+MB attachment.  Contact me privately and I will give you an FTP server login that you can upload your images to.  I'll look into it.

-Kirk

Offline sdick3

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: TIFF Performance still very slow - anything I can do about this?
« Reply #2 on: May 13, 2006, 06:47:46 PM »
I am not sure about you issue. But will relate my own experience to see if it helps. I am using ThumbsPlus and they supply a RAW file reader to read the IPTC data when you catalogue. But with this add-on set as the default for opening TIFF files PM takes forever to build thumbs.

I reset the default for TIFF and TIF back to Photoshop in the Windows File Associations and the speed increased significantly.

So what I am saying is check what the default app is for TIF as I think PM uses this to build the thumbs.

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: TIFF Performance still very slow - anything I can do about this?
« Reply #3 on: May 13, 2006, 07:02:02 PM »
I am not sure about you issue. But will relate my own experience to see if it helps. I am using ThumbsPlus and they supply a RAW file reader to read the IPTC data when you catalogue. But with this add-on set as the default for opening TIFF files PM takes forever to build thumbs.

I reset the default for TIFF and TIF back to Photoshop in the Windows File Associations and the speed increased significantly.

So what I am saying is check what the default app is for TIF as I think PM uses this to build the thumbs.

I can guarantee that Photo Mechanic does not rely on any external application to load TIFF files.  It uses libtiff which we compile into Photo Mechanic.  What can take longer to build the initial thumbnails is if the TIFF files themselves do not contain a thumbnail.  In that case PM has to read the entire image in order to create the thumbnails.

I have several of Neil's TIFF files now and tested them with Photo Mechanic on a machine that I built last night that is very similar to Neil's configuration and the speed was quite good, very impressive.  I am working directly with Neil to find out what we can do to solve his speed issue.

-Kirk

Offline pwp

  • Newcomer
  • *
  • Posts: 48
    • View Profile
Re: TIFF Performance still very slow - anything I can do about this?
« Reply #4 on: May 23, 2006, 04:57:03 AM »
I have exactly the same issues as Neil. My XP sp2 system is very similar. The slow loading of tiffs, single layer, 32 Mb Canon 1Ds files can only be described as glacial. Smaller 20D files are almost as slow, even with LZW compression. The RAW files load in an instant, but the slow tiff performance makes PM vitually unusable for me. The same folder of files would load 10-15 times faster in CS2 Bridge, known to be slow, and probably 50 times faster in BreezeBrowserPro. BB Classic is not far behind. The issue is the same on two different similarly specced workstations.

This is utterly frustrating, as the options & workflow with PM leaves the rest gasping for respectabilty. Kirk, we corresponded on this matter some time ago, I sent you files and even went as far as a tricky registry cleanup. Are there new solutions?

Cheers,
Paul

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: TIFF Performance still very slow - anything I can do about this?
« Reply #5 on: May 23, 2006, 07:11:17 AM »
I have exactly the same issues as Neil. My XP sp2 system is very similar. The slow loading of tiffs, single layer, 32 Mb Canon 1Ds files can only be described as glacial. Smaller 20D files are almost as slow, even with LZW compression. The RAW files load in an instant, but the slow tiff performance makes PM vitually unusable for me. The same folder of files would load 10-15 times faster in CS2 Bridge, known to be slow, and probably 50 times faster in BreezeBrowserPro. BB Classic is not far behind. The issue is the same on two different similarly specced workstations.

This is utterly frustrating, as the options & workflow with PM leaves the rest gasping for respectabilty. Kirk, we corresponded on this matter some time ago, I sent you files and even went as far as a tricky registry cleanup. Are there new solutions?

Yes.  Version 4.4.3.1 has much faster TIFF loading.  I went as far as purchasing components similar to Neil's and built a system very much like his.  Neil sent me about 15 usable 71 MB TIFF files.  They now load in about 2 seconds each and would load even faster if I had a RAID array like Neil has (my system is similar but not exactly the same.)  Neil's system is a Pentium-D system which shows as four CPUs when queried of the OS.  This meant that PM was loading up to four 71 MB TIFF files at once.  The disk I/O pattern then was fairly random instead of sequential.  We did a lot of testing with buffering, going as far as writing a simulator to determine how to most quickly load four 1 GB files.  It turns out that it is quicker on both Windows and Mac OS X to load each 1 GB file sequentially.  Otherwise each CPU needs a 64 MB (that's right, 64 million byte) buffer to approach the speed of sequential reads.  I put in some code to serialize our disk I/O and here are the results on my newly acquired Pentium-D system:

Version 4.4.3.1, loading 15 71MB TIFFs: 30.75 or about 2 seconds per photo loaded at high resolution
Version 4.4.3 and earlier, loading 15 71 MB TIFFs: 48.5 seconds or about 3.2 seconds per photo

This is a performance increase of 63%.  The improved speed indicates that this system is capable of sustained reads at about 35 MB/sec.

In Neil's case with a performance-oriented RAID array things were much worse, taking up to 20+ seconds to load a photo at high resolution.  With a test version of PM he can now load them in about a second flat.

On my 2.0 GHz G5 PowerMac, here are the timings:

Version 4.4.3.1, loading 15 71MB TIFFs: 22.75 or about 1.5 seconds per photo loaded at high resolution
Version 4.4.3 and earlier, loading 15 71 MB TIFFs: 35.15 seconds or about 2.34 seconds per photo

This is a performance increase of 64%.  The improved speed indicates that this system is capable of sustained reads at about 47 MB/sec.

In the case of BreezeBrowser and BreezeBrowser Pro, I expect that they load the TIFF thumbnail and stop there.  Photo Mechanic can be told to stop loading photos at the thumbnail stage if you want (Preferences->Contact Sheet tab->uncheck "Generate high quality thumbnails") but I really like to see all of my thumbnails at highest quality so I made sure that the new tunings makes for excellent performance when loading high-quality thumbnails.

Preview performance is also greatly enhanced.  I can zoom in on a 71 MB (12 Mega-pixel Nikon D2X 16-bit TIFF) in under two seconds, even faster if the photo's file data is in the system's disk cache.

Basically, loading of TIFF files is disk I/O bound.  The improvements to PM 4.4.3.1 mean that you will be able to load TIFFs as fast as you can copy them.  So once you have this version you will be getting the fastest performance available on *your* system.  If you want it to be faster then invest in a performance-tuned RAID array, as CPU performance is less of an issue.

We also made JPEG loading even faster on multi-CPU systems.  I don't have timings, but it is not quite as dramatic as the TIFF loading increase, though the seat-of-the-pants test feels faster.

HTH,

-Kirk