Raw files get corrupted upon import on two different cameras and card readers

[esparrow]

I am using PM with the latest update on a Macbook Pro 2019 with the latest updates (10.15.7), though the problems occurred with the penultimate update of PM (I have not tried the latest version which I only installed today). I think the version was PhotoMechanic6R5216.

I am ingesting Raw files from a Nikon Z7. In the latest updates, I had corrupted Raw files in the ingest. They get copied but cannot be opened. They also do not get renamed. Upon copying, they get a new filename which is: „_Original filename“ (that is: the original filename with an additional underscore at the beginning).

Also, quite often, the connection to the XQD card gets interrupted during ingest; I have to take out the card, reenter it and continue the ingestion process.

First, I thought my camera or my XQD card might be damaged (both fairly new) - also because the photos then proved to be damaged also on the XQD card.

Then, however, the same thing occurred upon ingesting photos made with an Olympos EM-5 Mk II. Here, a different card reader was used, as, this time, and SD card was used.

The error does not occur, when I first copy the raw files directly to the HDD as a safety measure using the Mac Finder, then ingest with PM and then reformat the drive in the camera after having it ejected. When copying the files directly with the Mac Finder, the connection is also stable.

This is a very, very bad error. I lost quite many crucial pictures on an unrepeatable photo trip. I cannot say that PM is 100 per cent responsible, but it remains a very annoying experience.

Also, the ingest seems to run very slow. I am quite surprised how slow it works, given the fact that the XQD reader is USB 3.0 ready and connected via an adapter to an USB-C slot. The photos are copied to an external HDD, connected directly to an USB-C slot, using a cable USB-C/USB 3.0 Micro B plug.

Thank you very much for your help!

[Kirk Baker]

I am using PM with the latest update on a Macbook Pro 2019 with the latest updates (10.15.7), though the problems occurred with the penultimate update of PM (I have not tried the latest version which I only installed today). I think the version was PhotoMechanic6R5216.

I am ingesting Raw files from a Nikon Z7. In the latest updates, I had corrupted Raw files in the ingest. They get copied but cannot be opened. They also do not get renamed. Upon copying, they get a new filename which is: „_Original filename“ (that is: the original filename with an additional underscore at the beginning).

Also, quite often, the connection to the XQD card gets interrupted during ingest; I have to take out the card, reenter it and continue the ingestion process.

First, I thought my camera or my XQD card might be damaged (both fairly new) - also because the photos then proved to be damaged also on the XQD card.

Then, however, the same thing occurred upon ingesting photos made with an Olympos EM-5 Mk II. Here, a different card reader was used, as, this time, and SD card was used.

The error does not occur, when I first copy the raw files directly to the HDD as a safety measure using the Mac Finder, then ingest with PM and then reformat the drive in the camera after having it ejected. When copying the files directly with the Mac Finder, the connection is also stable.

This is a very, very bad error. I lost quite many crucial pictures on an unrepeatable photo trip. I cannot say that PM is 100 per cent responsible, but it remains a very annoying experience.

Also, the ingest seems to run very slow. I am quite surprised how slow it works, given the fact that the XQD reader is USB 3.0 ready and connected via an adapter to an USB-C slot. The photos are copied to an external HDD, connected directly to an USB-C slot, using a cable USB-C/USB 3.0 Micro B plug.

Thank you very much for your help!

Generally, a problem like this is caused by bad RAM in your system.  Photo Mechanic reads from disk into a memory block in RAM and then writes from that block in RAM to the destination file.  If the RAM doesn't hold its contents 100% correctly, then corruption will occur with the regularity of the failure of the RAM.  I'd like to see one of your images that you copied with the Finder and the same image copied with Photo Mechanic.  Please post them here or share with me a link to the images on a file sharing service of your choice.

Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your files.  You may have to make more than one post if the files are large.

Thanks,

-Kirk

[esparrow]

Dear Kirk,

thank you for your quick reply! Bad RAM on my mac would be very bad news. It is fairly new (a 2019 model). I did a hardware test holding "D" pressed on startup, with no error found. However, I do not know whether this is excluding bad RAM definitively (I fear not).

Some files got permanently destroyed on the XQD card in those occasions that I tried to copy them using finder AFTER having executed PM's digest. Since then, however, I often copy everything first using the finder as a backup and then ingest everything. So I do not have a pair for every corrupted file.

Sending the corrupted files via Wetransfer, as the uploading does not seem to work due to file size. I hope that is o.k.

Thank you very much for your help!

Best regards,

Michael

[Kirk Baker]

Michael,

thank you for your quick reply! Bad RAM on my mac would be very bad news. It is fairly new (a 2019 model). I did a hardware test holding "D" pressed on startup, with no error found. However, I do not know whether this is excluding bad RAM definitively (I fear not).

Some files got permanently destroyed on the XQD card in those occasions that I tried to copy them using finder AFTER having executed PM's digest. Since then, however, I often copy everything first using the finder as a backup and then ingest everything. So I do not have a pair for every corrupted file.

That should not be possible unless you browsed the card and made edits to the files on the card.  If you only did an Ingest then no files on the card should be modified.  Ingest is a read-only operation.  It does not modify the source files.

Sending the corrupted files via Wetransfer, as the uploading does not seem to work due to file size. I hope that is o.k.

Sure.  You'll get me a link soon?

-Kirk

[esparrow]

And here it is: 

https://we.tl/t-T0vwLexwdW

Thank you for your help!

Best regards,

Michael

[Kirk Baker]

Michael,

Thank you for the files.  I compared one pair (_A250016.ORF and 20201025_122545.ORF) and they're identical except for the first 64KB of data where the _A250016.ORF is completely random bytes, having no resemblance to the 20201025_122545.ORF or any other ORF RAW file.

Photo Mechanic is using one of the file copy routines provided by macOS to do the copying.  Here is a snippet of the actual source code:

Code: [Select]

BOOL success = [[NSFileManager defaultManager] copyItemAtPath:[NSString stringWithUTF8String:srcFilePath]
      toPath:[NSString stringWithUTF8String:destFilePath]
       error:&err];

It is curious that the corruption is exactly 64KB.  I still think it is likely that there is some small portion of your system's RAM that is bad.  I don't think that the built-in RAM test is exhaustive.  How long did it take to run?

I'd like to see your Photo Mechanic log.  In Photo Mechanic go to the Help menu and choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

[esparrow]

Dear Kirk,

thank you very much for your help! I will send you the support data in a PM and hope this is o.k.

Best regards,

Michael

[Kirk Baker]

Michael,

thank you very much for your help! I will send you the support data in a PM and hope this is o.k.

Yes, that would be fine.

Thanks,

-Kirk

[Kirk Baker]

Michael,

Thank you for the requested information.  I see no indication of a failure in copying the image.  This means that the images did not get a read error during the copy though their data was compromised.  It is likely that the corruption occurred on the card itself.  Unless you made edits on the card, the images should not be corrupted on the card.  They should be in the same state as they were written by the camera.  That said, if the card(s) are failing in some areas of the card, the data that was written by the camera may not be represented in the data that can be read from the card.  So if the media can't reliably store data then you're going to see corruption of the data.

I suggest switching to another set of cards.

-Kirk

[esparrow]

Dear Kirk,

thank you very much for your support and for checking that many information.

Do you allow me to ask some questions regarding your latest reply:

• Does the fact that the copying went well according to the log meant that my RAM is o.k.?
• How can the fact be explained that the error occured with two totally different cards (an XQD card/Nikon Z7 and an SD-card/Olympus EM5 MkII)? Is it possible that the cards became corrupted e.g. by not ejecting them correctly on a Mac? I do not say that this happend, but I am looking for an explanation why the answer should be that two different cards are defect and not the one computer, RAM or the program involved (i.e. PM6).

Thank you very much for your help!

Best regards,

Michael

[Odd Skjaeveland]

my RAM is o.k.?

A (free) utility like Memtest86 or Memtest86+ will probably provide an answer.
Let it run for some hours, over night for example.

[Kirk Baker]

Michael,

thank you very much for your support and for checking that many information.

Do you allow me to ask some questions regarding your latest reply:
• Does the fact that the copying went well according to the log meant that my RAM is o.k.?

No.  Your RAM could still have problems.

• How can the fact be explained that the error occured with two totally different cards (an XQD card/Nikon Z7 and an SD-card/Olympus EM5 MkII)? Is it possible that the cards became corrupted e.g. by not ejecting them correctly on a Mac? I do not say that this happend, but I am looking for an explanation why the answer should be that two different cards are defect and not the one computer, RAM or the program involved (i.e. PM6).

It's entirely possible that both cards are failing at the same time.

At this point I suggest trying another set of cards.  Also it would be worthwhile to run an exhaustive memory test like Mr. Skjaeveland suggested.  I have used Memtest86+ before on my systems and it has found bad RAM.

-Kirk

[esparrow]

Dear Kirk, dear Odd,

thank you very much for your help! I will check the RAM of my mac, although I haven't figured out yet how to do it with memtest86+ (my 2019 Mac does not recognise the created media on startup - neither in form of a CD nor in form of an USB-key - it seems to be all Windows).

One additional thought, though - could it also be a defective card reader - or even a defective cable adapter for that matter?

Thank you very much!

Best regards,

Michael

[Odd Skjaeveland]

...it seems to be all Windows

No, I don't think so. It may be all Intel, but your Mac matches that criteria, or no?

You boot the computer directly to Memtest+/Memtest, the memtest software runs with no operating system loaded at all. Thus Windows, Mac, Linux... is irrelevant. But your computer must be able to boot from the media where you install the memtest software.