Why RAW file verification is needed

[Mike-D]

You know how you have insurance on your house?  or your car?  And how you'll (if you're lucky) never have to use it?  Well some people do have to use it, and when it's needed it sure is handy.  Such is the case with RAW file verification during ingest.

I recently shot a wedding and downloaded 7 8GB cards simultaneously using Delkin Imagerouters and PM.  There was no indication of errors during the ingest. I scanned through the images in PM, and again saw no indication of errors.  However, when it came time to actually render the raw files, I had about 15% of the files corrupted beyond recognition.  The files on the cards themselves were absolutely fine, but something happened to them during ingest.

After I verified that the files on the cards were in fact OK, I began to troubleshoot and eventually discovered that I had some RAM go bad, above 7GB.  I am assuming that PM was using this address space during ingest, and the good data was read into bad ram, corrupting the file somehow...and then this corrupted file was written to the disk.

If some form of CRC or something had been done in the PM ingest, this would have been caught immediately, instead of a few days later.  I have the luxury of not needing my cards in back-to-back shoots, but if I had, I would have been screwed quite badly.

So the problem was bad RAM, but the bigger problem IMHO is that PM has no way to verify a file that it is ingesting.  Sure would be nice if this kind of insurance was there, though. I sure could have used it.

...Mike

[Kirk Baker]

Mike,

You know how you have insurance on your house?  or your car?  And how you'll (if you're lucky) never have to use it?  Well some people do have to use it, and when it's needed it sure is handy.  Such is the case with RAW file verification during ingest.

I recently shot a wedding and downloaded 7 8GB cards simultaneously using Delkin Imagerouters and PM.  There was no indication of errors during the ingest. I scanned through the images in PM, and again saw no indication of errors.  However, when it came time to actually render the raw files, I had about 15% of the files corrupted beyond recognition.  The files on the cards themselves were absolutely fine, but something happened to them during ingest.

After I verified that the files on the cards were in fact OK, I began to troubleshoot and eventually discovered that I had some RAM go bad, above 7GB.  I am assuming that PM was using this address space during ingest, and the good data was read into bad ram, corrupting the file somehow...and then this corrupted file was written to the disk.

If some form of CRC or something had been done in the PM ingest, this would have been caught immediately, instead of a few days later.  I have the luxury of not needing my cards in back-to-back shoots, but if I had, I would have been screwed quite badly.

So the problem was bad RAM, but the bigger problem IMHO is that PM has no way to verify a file that it is ingesting.  Sure would be nice if this kind of insurance was there, though. I sure could have used it.

If you have bad RAM, then it is entirely possible that a CRC-32 or MD5 could check out OK.  The RAM is bad after all and it can cause the data to be corrupted in the same way when verifying.

-Kirk

[Mike-D]

If you consider what would have to happen in order for your assertion to be true, you'd discover that it's doubtful the two CRCs would match.  Very doubtful actually. And ever more remote for an MD5.  But lets use CRCs for sake of discussion. 

In order for them to match, the CRC calculated from the good file on the card would have to be corrupted in RAM in such a way to make it identical to either
a) a valid CRC calculated on a corrupted file in RAM and stored in a valid section of RAM, or
b) a valid CRC calculated on a corrupted file in RAM which then itself becomes corrupted in RAM. 

I don't have any stats on this, but I find it extremely doubtful this would occur once, let alone for 200 files. And all it would take is Once to give an alert that something unexpected had happened and further investigation is required. To be honest I fail to understand the resistance to this feature.

...Mike

[Mike-D]

Here is a bit more info on how a CRC-32 is calculated, explaining how remote it would be for different files to have the same CRC-32 value.

From Dr Dobbs:
http://marknelson.us/1992/05/01/file-verification-using-crc-2/

...Mike

[Kirk Baker]

Mike,

Here is a bit more info on how a CRC-32 is calculated, explaining how remote it would be for different files to have the same CRC-32 value.

From Dr Dobbs:
http://marknelson.us/1992/05/01/file-verification-using-crc-2/

I am quite aware of how a checksum is calculated.  When you have bad RAM just about any transformation to the data can occur.  You may get false errors and errors may be missed.

I agree that it could be useful to have a verification step for those who are willing to wait the extra time.

-Kirk

[HPL]

This is the reason, I don’t use ingest.
I copy the files with the Windows explorer form card to disk and verify them later using Windiff.exe.

It’s longer but safer…

Peter

[Mike-D]

When you have bad RAM just about any transformation to the data can occur.  You may get false errors and errors may be missed.

While I agree that bad RAM can give a false error (which is still valuable to know), I think the chances of bad RAM causing an error to be missed are infinitesimal given the math behind what would have to happen in order to make that scenario occur.

I agree that it could be useful to have a verification step for those who are willing to wait the extra time.

It would be very useful, yes. Please!

...Mike

[vAfotoriporter]

I agree, it can be very useful if speed is less of an essence than security. It would be worth to add an option for it in the ingest dialog.

And once this CRC is calculated for all files individually, it could be stored with the files for later use. The only problem with it is that PM modifies the RAW files on all metadata edit causing the CRCs to mismatch.

But a possibility to verify is as good an idea as to add a CRC generation option for the burning features (one of my earlier requests). This way not only downloaded data could be verified but live image files on the drive and even archived (maybe cataloged with PM later?) data could be checked any time.

[William R Wood]

Verification would be a good option.

Meantime checkout ImageIngester and/or ImageVerifier:

http://basepath.com/ImageIngester/

[vAfotoriporter]

Verification would be a good option.

Meantime checkout ImageIngester and/or ImageVerifier:

http://basepath.com/ImageIngester/

My goal would be to use as few programmes as possible. Since PM does the best work with metadata it is obvious to use it for that stuff (like ingesting as the first step - I allways include original filenames and folder info in ingested images). And would rather see the verify option in PM than use yet another program for a feature mostly done by PM (with the exception of verification).

[Mike-D]

I'm bringing this one back from the dead to ask if, 6 years on, PM does any kind of image verification during ingest? Does anyone know?

[Kirk Baker]

Mike,

I'm bringing this one back from the dead to ask if, 6 years on, PM does any kind of image verification during ingest? Does anyone know?

I know for certain that PM does not perform any CRC/MD5/SHA1/etc. on the file and then verify that the file is the same at the destination.

-Kirk