How does PM verify files when injesting?

[Mike-D]

Hi there

I just have a simple question: does PM verify files it has ingested? And if so, how?

It has been established that PM is modifying files as they are downloaded, so it can't be a CRC or MD5 check.  And I can't for the life of me think of a way that these files could be verified if they are not exact duplicates, which they won't be.  So how does this work?

...Mike

[Kirk Baker]

Mike,

I just have a simple question: does PM verify files it has ingested? And if so, how?

It has been established that PM is modifying files as they are downloaded, so it can't be a CRC or MD5 check.  And I can't for the life of me think of a way that these files could be verified if they are not exact duplicates, which they won't be.  So how does this work?

The short answer is no.  The long answer comes from the engineer who wrote Ingest:

It's the same as copying files off the card with the Finder or Windows Explorer.  The data read from the card is assumed to be correct, unless an I/O error occurs.  Verifying the data (ignoring for a moment that ingest can modify the file in transit) would essentially entail reading the file from the card twice.  Which is not as simple as it sounds, because the OS itself trusts the data it has already just read from the card, and caches it in RAM.  So if we asked the OS to read a file on the card twice, most if not all of the data on the second pass would come from the cache, which is really just a copy of the data read from the card on the first pass.

It might be possible to instruct the OS to clear its cache somehow, and force it to read from the card a second time.  However, that would definitely slow down the ingest.

-Kirk

[William R Wood]

To confirm what Kirk said, check out ImageIngesterPro, it verifies files during ingest but it is MUCH slower than PM.  To me speed is more important than verification since there is no evidence that I know of that straight copying is prone to errors.

Still ImageIngester does have some neat features: IIP lets you create up to 3 copies of your images during ingest, you can see what's on your cards before ingest and choose what you want ingested and it lets you convert RAWs to DNG during ingest (using Adobe's converter) all of which I will be lobbying Kirk to incorporate into PM. 

Regards

Bill Wood

[Mike-D]

I guess I'm in the minority then.  I would prefer data integrity over speed every time. Very often, especially when traveling, I dump cards to a laptop, but don't look at them right away, because of the painful speed of previewing when you use the actual RAW data and not the embedded JEPG file. So not having any way to verify that the copy was successful is a bit scary for me.

...Mike

[William R Wood]

I guess I'm in the minority then.  I would prefer data integrity over speed every time. Very often, especially when traveling, I dump cards to a laptop, but don't look at them right away, because of the painful speed of previewing when you use the actual RAW data and not the embedded JEPG file. So not having any way to verify that the copy was successful is a bit scary for me.

...Mike

I agree about the importance of data integrity and if I had to choose between speed and accurate files I would choose integrity over speed every time as well.  BUT, in my 25 years of running computers every day and copying untold thousands of files from one disc to another using DOS, windows file manager, backup software, etc including ingesting more thousands of files with PM and other photo software, I have never had a data integrity problem.  Not challenging you at all, but, given the extreme reliability of computers these days, I think it is reasonable to assume that data integrity is almost a non-issue and that's why I favor speed over verification - there is simply no evidence that there is any material risk of data corruption so why add the generally unnecessary step of verification which more than doubles ingest time.  Verification would be a great feature if you could turn it on and off at will and thus have it available for critical images and maybe that should be a feature request.

Don't understand your comment about painful preview speed using RAW files.  PM always uses the embedded JPG preview for RAW files (PM cannot render previews from RAW data) and PM is the speed king of image viewers.  That's why I bought it. 

Regards

Bill Wood

[jimb]

I guess I'm in the minority then.  I would prefer data integrity over speed every time. Very often, especially when traveling, I dump cards to a laptop, but don't look at them right away, because of the painful speed of previewing when you use the actual RAW data and not the embedded JEPG file. So not having any way to verify that the copy was successful is a bit scary for me.
...Mike

Mike,
What would be scary is if you dump them to a laptop, then reuse the cards without having some kind of backup of the images. In other words, the images you moved to the laptop should also be backup up on a cd/dvd, an external hard disk, or the cards themselves (providing you don't need to reuse them). Anything can happen while you are traveling --stolen laptop, hard disk crash, etc., etc., so having a backup of the trip's images is essential IMO.

[Mike-D]

Which is why I travel with two USB WD passport drives for secondary and tertiary backup.  But that is off topic.

I guess I'll adapt.

...Mike