Author Topic: Ingest files seem to copy OK, but the files are corrupt  (Read 6999 times)

Offline darinf

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Ingest files seem to copy OK, but the files are corrupt
« on: April 16, 2009, 09:28:26 AM »
I am using PM 4.5.3 on Windows XP sp3 with 4GB of RAM...

When I use ingest, all the files appear to copy fine, but when I try to access the RAW files in Capture One, Capture one cannot read about 1% of the files. (I like using Capture One since it generates it's own previews based on the RAW data, not the embedded thumbnails. this way I know for sure if the RAW data is intact.)

When I copy the files using Windows copy using the same card and same card reader, the files are 100% intact. When I copy using a third party copy utility called "Tera Copy" which does a verify after copying each file, the files are 100% OK.

When I use Downloader Pro with verify on or off, the files copy 100% fine.

Why doesn't PM have a verify option? Tera Copy copies the files almost as fast as PM, but does a checksum verification. Downloader Pro also does a verify and copies the files just as fast as PM.

Please advise? I love the ingest feature, but if it's not reliable, it's worthless.

Thanks,
-Darin

Offline Hayo Baan

  • Uber Member
  • ******
  • Posts: 2552
  • Professional Photographer & Software Developer
    • View Profile
    • Hayo Baan - Photography
Re: Ingest files seem to copy OK, but the files are corrupt
« Reply #1 on: April 17, 2009, 02:08:08 AM »
It could be a glitch of the card (reader) or perhaps even the OS (anti virus programs interfering perhaps?).

If I understand you correctly, you have tried two other methods of copying the same files of the same card, and these are fine. Have you tried to ingest them with PM a second time too? Were they correct then? If so, I guess there's something wrong with your card reader (and/or OS) that's causing data corruption. If not, there may be a problem with PM adding its specifics to the files. This is really unlikely though as thousands of people use PM without these symptoms…

So all I can suggest now is to (thoroughly) check your card reader and cards.


Regarding verification: I wonder what the other programs actually verify, and highly doubt this verification is effective anyway.
What PM (or any of the other programs) could do, without too big a performance impact, would be to read the just written file and check if the bits are all there. This is quite a useless step though as writing data to a HD is very very reliable and checking the data just written this way quite superfluous (let alone the fact that chances are the data will be read from the cache anyway).
A more time-consuming verify could also re-read the data from the card and then check if that was the same. This would double the ingest time and is still debatable if this would help.
Actually checking the data read would probably be the best option. To my knowledge, however, there are no checksums embedded in the file so the only thing left would be some structural integrity check, but I think what can be done is already done by PM implicitly (e.g, checking some key EXIF fields and adding its own specifics).

Cheers,
Hayo Baan - Photography
Web: www.hayobaan.nl

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Ingest files seem to copy OK, but the files are corrupt
« Reply #2 on: April 17, 2009, 10:06:38 AM »
Darin,

I am using PM 4.5.3 on Windows XP sp3 with 4GB of RAM...

When I use ingest, all the files appear to copy fine, but when I try to access the RAW files in Capture One, Capture one cannot read about 1% of the files. (I like using Capture One since it generates it's own previews based on the RAW data, not the embedded thumbnails. this way I know for sure if the RAW data is intact.)

When I copy the files using Windows copy using the same card and same card reader, the files are 100% intact. When I copy using a third party copy utility called "Tera Copy" which does a verify after copying each file, the files are 100% OK.

When I use Downloader Pro with verify on or off, the files copy 100% fine.

Why doesn't PM have a verify option? Tera Copy copies the files almost as fast as PM, but does a checksum verification. Downloader Pro also does a verify and copies the files just as fast as PM.

Please advise? I love the ingest feature, but if it's not reliable, it's worthless.

Verification is possible, but won't really have any effect unless PM causes the OS to flush all pending writes to the destination drive and then reads the destination files again.  This would slow down Ingest significantly and I can only surmise from your timing comparisons above that Tera Copy, and Downloader Pro do not flush pending writes when they do their verification.  So basically their verification is not entirely reliable anyway.

My guess would be that either your card is starting to have problems, your reader isn't entirely reliable, or you have some portion of your RAM that is bad.  What I would like to see is two files from you, both being the same file but one of them should be 100% correct and the other should have the errors that you suspect Ingest is causing.  I would want to compare them with a binary compare to see where the files differ.  If there are just a few random bytes that are incorrect, then I would suspect that your RAM is bad.  If a lot of data is bad then I would suspect the reader or the card as the culprit.

What happens if you repeat the Ingest again?  Does it still fail on 1% of the images and on the same images, or on different images?

If you have some samples that you can share with me, then please click on my name to the left of this message, then click on the 'personal message' link.  I will respond with upload instructions.

Thanks,

-Kirk

P.S. You should get PM 4.5.4 since it fixes bugs that were present in your version.

Offline darinf

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: Ingest files seem to copy OK, but the files are corrupt
« Reply #3 on: April 17, 2009, 10:24:47 AM »
As stated in the previous post, I used the EXACT SAME CARD READER and the EXACT same card with the EXACT same contents on that card for ALL transfers.

If it was the card reader or anti-virus software, then wouldn't I see errors using all software, not JUST PM?

In this case, I can copy the files using three other applications without any copy errors. That's 0% copying errors when copying thousands of files. But with PM I get about 1% copy errors or more. (I am not 100% sure whether one specific card is problematic or all cards exhibit the same problems. I have to do more testing in that respect. But again, ALL my cards copy fine using other software.)

The strange thing is that the PM "backup copy" on a different hard drive has the SAME errors. If I find a bad file on the main destination, the SAME file will be bad on the backup copy made by PM.

So, I can say with near certainty that this problem is specific to PM and not any hardware or other software problems.

If I do a second ingest with the same cards and card reader, I will get different files which are bad, so the errors do seem somewhat random.

I am not sure what the other programs are using for verification. I know Tera Copy lists source and target CRC values in their log after the copy. I will concede that any true verify would double the download time, so you definitely can't get "something for nothing" in terms of the verify.

In any case, there is definitely something going on with PM ingest as opposed to the other software.

I will see if I still have some "bad" files on my hard drive. In general I have been deleting the bad copies since they are dangerous to have on my drive.

I will try upgrading to 4.5.4.

Thanks,
-Darin

-Darin

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Ingest files seem to copy OK, but the files are corrupt
« Reply #4 on: April 17, 2009, 10:54:11 AM »
Darin,

As stated in the previous post, I used the EXACT SAME CARD READER and the EXACT same card with the EXACT same contents on that card for ALL transfers.

That's good.  Change as few variables as possible and that will help pinpoint the problem.

Quote from: darinf
If it was the card reader or anti-virus software, then wouldn't I see errors using all software, not JUST PM?

I didn't suggest that anti-virus software was a problem, though there have been some anti-virus programs that erroneously claimed that parts of PM were a trojan-horse.

Quote from: darinf
In this case, I can copy the files using three other applications without any copy errors. That's 0% copying errors when copying thousands of files. But with PM I get about 1% copy errors or more. (I am not 100% sure whether one specific card is problematic or all cards exhibit the same problems. I have to do more testing in that respect. But again, ALL my cards copy fine using other software.)

The strange thing is that the PM "backup copy" on a different hard drive has the SAME errors. If I find a bad file on the main destination, the SAME file will be bad on the backup copy made by PM.

So, I can say with near certainty that this problem is specific to PM and not any hardware or other software problems.

The issue could still be quite probably bad memory somewhere in your system's address space.  Other programs need not be affected if the memory buffer they use to transfer the data is not bad.  All it takes is a few bytes of bad memory in a specific location in memory to cause the issue that you're describing and not all programs would be affected.

Quote from: darinf
If I do a second ingest with the same cards and card reader, I will get different files which are bad, so the errors do seem somewhat random.

That also indicates that RAM could be bad.  PM does not reuse its memory buffers for copying, it allocates memory, copies data to/from it, and then frees that memory when the file copy is complete.

Quote from: darinf
I am not sure what the other programs are using for verification. I know Tera Copy lists source and target CRC values in their log after the copy. I will concede that any true verify would double the download time, so you definitely can't get "something for nothing" in terms of the verify.

In any case, there is definitely something going on with PM ingest as opposed to the other software.

I suggest that you get a utility program and do an exhaustive test of your RAM.

Quote from: darinf
I will see if I still have some "bad" files on my hard drive. In general I have been deleting the bad copies since they are dangerous to have on my drive.

No need.  If this problem happens as often as you have suggested, then a few new Ingests will cause the problem to occur again and then you'll have some samples to send me.

Please note that problems such as yours are extremely rare.  In each case that I have been involved in solving, the problem has either been bad RAM, a driver issue, a flaky card reader, or a card that is going bad.  Photo Mechanic itself has never been the root cause of the problem.

-Kirk

Offline darinf

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: Ingest files seem to copy OK, but the files are corrupt
« Reply #5 on: April 17, 2009, 11:00:54 AM »
Thanks for the quick response, Kirk.

I am uploading two sample files to my ftp server now. I will PM you with the download links.

I did notice that the copies I did with other software yields different file sizes than the same files copied with PM. I assume that's normal.

I did a file comparison of a good file and a bad file copied by PM. They have the exact same files size, but the files are different.

Do you have a recommendation for a good RAM testing utility (on Windows XP)?

Thanks,
-Darin

Offline darinf

  • Newcomer
  • *
  • Posts: 9
    • View Profile
Re: Ingest files seem to copy OK, but the files are corrupt
« Reply #6 on: April 17, 2009, 11:54:15 AM »
Well, I guess all bets are off, now.

I ran MemTest86 and found some bad RAM.

So hopefully replacing the RAM will fix any issues.

I will report back once I get the new RAM.

Thanks for your help.

-Darin

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Ingest files seem to copy OK, but the files are corrupt
« Reply #7 on: April 17, 2009, 11:56:38 AM »
Darin,

Thanks for the quick response, Kirk.

I am uploading two sample files to my ftp server now. I will PM you with the download links.

I did notice that the copies I did with other software yields different file sizes than the same files copied with PM. I assume that's normal.

I did a file comparison of a good file and a bad file copied by PM. They have the exact same files size, but the files are different.

The differences in file sizes is normal, especially since you have instructed PM to embed IPTC data into your TIFF-based RAW files (like CR2 or NEF).

I analyzed the two files.  They only differ in one bit out of 23,101,667 bytes.  In the good file at offset 0xBF699F (or 12,544,415 in decimal) the byte value is 0xA3 and in the bad file at the same offset the byte value is 0xE3.  If we look at those two values as their binary equivalent, we'll see that only one bit differs:

0xA3 = 1010 0011
0xE3 = 1110 0011

I hilighted the bit that differs.  Since PM does nothing to the RAW data as it is copied this indicates that your RAM is likely bad.

Quote from: darinf
Do you have a recommendation for a good RAM testing utility (on Windows XP)?

This one is free and is from Microsoft:

http://oca.microsoft.com/en/windiag.asp