Slow ingest

[Graham Morgan]

Hi

Still finding that ingest is very slow 7-8 sec per image, 4-5 at best, 5D raw only. Backing up to usb ext HD 7200rpm

osx 10.5.4

macbook pro 2.2 intel dual

PM latest beta (6)


Thanks

Graham Morgan

[Kirk Baker]

Graham,

Still finding that ingest is very slow 7-8 sec per image, 4-5 at best, 5D raw only. Backing up to usb ext HD 7200rpm

osx 10.5.4

macbook pro 2.2 intel dual

PM latest beta (6)

How is the speed if you only ingest to your internal hard drive?

What is the format of the external hard drive?  You can use the Get Info command in the Finder on the disk icon to find out.

-Kirk

[Graham Morgan]

Hi Kirk

The drive format is 'extended' not journaled, when i first used PM, the files downloaded in about 2-3 seconds each, I can't understand how backing up to another drive would impact on this, is it simultaneous or not, where can the bottleneck be? What do you/others find when performing ingest.

Cheers

Graham Morgan

[Kirk Baker]

Graham,

The drive format is 'extended' not journaled, when i first used PM, the files downloaded in about 2-3 seconds each, I can't understand how backing up to another drive would impact on this, is it simultaneous or not, where can the bottleneck be? What do you/others find when performing ingest.

If you were previously only ingesting to the Primary destination and it took 2-3 seconds each and now you're ingesting to a Primary and a Secondary destination, it should take longer.  The file has to be read twice and written twice, and your external drive's interface is USB2 which will be much slower than the SATA interfaced internal drive on your MacBook.  The increased RPM speed of the drive won't help much when the interface feeding it is much slower.

-Kirk

[Graham Morgan]

Hi Kirk

Thanks, however Adobe Bridge downloader does this in less than half the time, this is unusual, something must have changed, perhaps at the os side. I would be happy to run the reporting version of PM.

Thanks

Graham Morgan

[Kirk Baker]

Graham,

Thanks, however Adobe Bridge downloader does this in less than half the time, this is unusual, something must have changed, perhaps at the os side. I would be happy to run the reporting version of PM.

Is the Bridge downloader downloading to two separate destinations at once?  Are we comparing against the same thing here?

-Kirk

[Graham Morgan]

Hi Kirk

Yes, gave it the same setup; in both cases PM takes more than twice the time.

thanks

Graham

[Kirk Baker]

Graham,

Yes, gave it the same setup; in both cases PM takes more than twice the time.

Contact me privately by clicking on my name to the left of this message and then click on the 'personal message' link.  Give me your contact information along with a best time to call.

Thanks,

-Kirk

[Kirk Baker]

Graham,

Thanks, however Adobe Bridge downloader does this in less than half the time, this is unusual, something must have changed, perhaps at the os side. I would be happy to run the reporting version of PM.

What kind of card reader are you using?  What interface does is use?  USB?  FireWire 400?  FireWire 800?

Can you post a screen shot of your Ingest dialog?  Also the Files tab and IPTC/XMP tab of the Photo Mechanic Preferences dialog, please.

Thanks,

-Kirk

[Graham Morgan]

Hi Kirk

Thanks for your prompt reply, the screen shots will be sent tomorrow. The card reader is USB 2, I'm not adding any info on ingest. How is it that copying from the card through finder can be so much faster? Getting files from the card wth the finder and then copying to the backup drive is about 1/3 of the PM ingest/backup option. Seems that the Bridge downloader and the Finder know something we don't. Waiting 40 min for 300 images seems much longer that normal. That's a long time with 4-5 full cards!
I'dont understand why USB 2 should be slower than Fire Wire, surley the read/write speed of the card will be an issue, way before hitting the max transfer speed of either. Do you think the this maybe Leopard indexing thing, the privacy option does'nt stick when i try to exclude the camera card.

Thanks

Graham Morgan

[Kirk Baker]

Graham,

Thanks for your prompt reply, the screen shots will be sent tomorrow. The card reader is USB 2, I'm not adding any info on ingest. How is it that copying from the card through finder can be so much faster? Getting files from the card wth the finder and then copying to the backup drive is about 1/3 of the PM ingest/backup option. Seems that the Bridge downloader and the Finder know something we don't. Waiting 40 min for 300 images seems much longer that normal. That's a long time with 4-5 full cards!
I'dont understand why USB 2 should be slower than Fire Wire, surley the read/write speed of the card will be an issue, way before hitting the max transfer speed of either. Do you think the this maybe Leopard indexing thing, the privacy option does'nt stick when i try to exclude the camera card.

Photo Mechanic should always copy files slower than the Finder.  The reason is that the Finder doesn't parse the files for metadata whereas Photo Mechanic does.  This takes time, though it shouldn't add more than a second to the overall time per file.  Then the file is copied and depending on your settings other things happen along the way, like folder creation, file renaming, insertion of IPTC data, etc.  All of these things take bits of time as well.  I would expect doing a single destination Ingest to be about 10-20% slower than a Finder copy.  Adding a secondary destination should slow things down a bit further, but certainly it shouldn't be three times slower.

We are currently benchmarking 4.5.3.2 and 4.5.3.1 on Mac OS X 10.4.11 and 10.5.4 to see how much time these operations take.  If they're slower than we expect, then we'll turn on the logging code and find out where the time is going.

One thing that will definitely slow things down (and is not fair when doing timing comparisons) is to have PM open the Contact Sheets as the files are being ingested.  In that case you're asking PM to do a lot more work than the Finder will ever have to do.  To do these time comparisons you need to set Ingest to open the Contact Sheets after the Ingest completes.

-Kirk

[Graham Morgan]

Hi Kirk

Thanks, adding to this, I have just viewed test results for various card/card reader combinations, the speed variation is surprising. My kingston cards and reader combination are among the poorer performers but it's not enough to explain the whole story. My ingest is just a simple download and copy, no renaming no IPTC info, 99% of the time a single folder is created, all the cards from one job downloading to this folder. Perhaps you can send me a shots of your recomended settings, though I suspect I'm doing things correctly. Yo can use my email for this.

Cheers

Graham Morgan

[Kirk Baker]

Graham,

Thanks, adding to this, I have just viewed test results for various card/card reader combinations, the speed variation is surprising. My kingston cards and reader combination are among the poorer performers but it's not enough to explain the whole story. My ingest is just a simple download and copy, no renaming no IPTC info, 99% of the time a single folder is created, all the cards from one job downloading to this folder. Perhaps you can send me a shots of your recomended settings, though I suspect I'm doing things correctly. Yo can use my email for this.

Here are the results of our tests with a Sandisk Extreme Ducati Edition UDMA CF card and a Lexar FireWire UDMA CF card reader.  The top throughput as measured with a Finder copy is 38 MB/sec.  Total data size on card was 2.64 GB.  Times are in seconds:

Mac OS X 10.5.4    Primary    Primary+Secondary
--------------------------------------------------------
PM 4.5.3.2             94           182
PM 4.5.3.1            142          182
Bridge CS3            94           174
Finder                   67           134

The Finder is always going to be faster than Photo Mechanic no matter what we try to optimize.  The Finder does not parse Exif data, Maker notes, etc.  I don't know what kind of parsing Bridge does while it reads the files but the times are similar to PM 4.5.3.2.  Notice that PM 4.5.3.1 is very much slower than PM 4.5.3.2 on Mac OS X 10.5.4.  We don't know what Apple changed about disk I/O in Mac OS X 10.5.3+ but it affected Photo Mechanic greatly.  We have spent several days finding faster I/O routines in the system and now our ingest speeds rival the speeds of Photo Mechanic 4.5.3.1 on trusty Mac OS X 10.4.11.

I would like to see your times.  Please note that you must unmount the card and remove it from the reader between each test in order for the test to be fair.  Otherwise the OS will have cached many of the files on the card and subsequent tests will go much faster which will skew the results.  Also, don't have PM open up the contact sheets during the Ingest.  It is not fair to make PM also render contact sheets in realtime along with the copying process.

One other thing that will affect times is that if you have PM set to unmount the card for you when the Ingest completes, then that will also skew the results.  Bridge CS3 and the Finder do not offer such a feature, and it can take several seconds in practice to unmount the card.  Photo Mechanic's Ingest will not signal ingest completion until the card completes the unmount procedure.  So if you really want to compare apples to apples, you need to turn that feature off.  We left it on in our tests and despite the additional time spent unmounting the card our speeds compare well against Bridge CS3 on Mac OS X 10.5.4.

Unless you can show otherwise with your own tests, using a stopwatch and unmounting the card between tests, we are considering this a non-issue and will not hold back the release of Photo Mechanic 4.5.3.2.

-Kirk

[Graham Morgan]

Hi Kirk

That's very helpful, I have some UDMA cards arriving on monday and will duplicate your setup.

Thanks

Graham Morgan