PM changing the byte order ?

[vainlord]

Hi!

I have a problem with DXO optics prohttp://www.dxo.com/intl/photo/dxo_optics_pro

When i try to process files from my Sony A900 imported with PM DXO optics pro cant read/process the files.

It looks like this:



According to DXO support its due to the fact that PM done something to the files while copying them to the HD ?

Iv confirmed this by testing a file directly from the memory card, and it worked without any problems, but the files copied to my hd with PM gives me this result ?

Any ideas ?

Thanx

[Kirk Baker]

Hi!

I have a problem with DXO optics prohttp://www.dxo.com/intl/photo/dxo_optics_pro

When i try to process files from my Sony A900 imported with PM DXO optics pro cant read/process the files.

It looks like this:



According to DXO support its due to the fact that PM done something to the files while copying them to the HD ?

Iv confirmed this by testing a file directly from the memory card, and it worked without any problems, but the files copied to my hd with PM gives me this result ?

We do write some data to the files as they are ingested.  It sounds like the DXO folks have some fairly naîve parsers for the RAW file formats they process.  We certainly don't scramble the data.

-Kirk

[vainlord]

Hi Kirk!

This is a reply from DXO support when i told them that you dont scramble the data.

Oddly enough its just the A900 files that is transfered with PM that dont work with DXO.

Hello Paul,

All I can tell you is this. We have some very sophisticated programs and ways to examine the information in image files, which our experts did. If they note that the bits are scrambled, they are. Also, the only workflow we support is the use of Optics Pro being first in your workflow, unless you use the Lightroom plug-in.

Since all programs process images a little differently, and our software must have the original, unchanged information to work properly, when using Optics Pro, you cannot use the other programs first in your workflow.

If you have questions about any other aspects of using the program, please let us know. Optics Pro is working properly in this area.

[Kirk Baker]

Hi Kirk!

This is a reply from DXO support when i told them that you dont scramble the data.

Oddly enough its just the A900 files that is transfered with PM that dont work with DXO.

Hello Paul,

All I can tell you is this. We have some very sophisticated programs and ways to examine the information in image files, which our experts did. If they note that the bits are scrambled, they are. Also, the only workflow we support is the use of Optics Pro being first in your workflow, unless you use the Lightroom plug-in.

Since all programs process images a little differently, and our software must have the original, unchanged information to work properly, when using Optics Pro, you cannot use the other programs first in your workflow.

If you have questions about any other aspects of using the program, please let us know. Optics Pro is working properly in this area.

Please send us a sample file then.  We'll examine it.  Please click on my name to the left of this message, then click on the 'personal message' link.  I will respond with upload instructions.

-Kirk

[vainlord]

Thanx Kirk PM, Sent.

[Kirk Baker]

Paul,

This is a reply from DXO support when i told them that you dont scramble the data.

Oddly enough its just the A900 files that is transfered with PM that dont work with DXO.

Hello Paul,

All I can tell you is this. We have some very sophisticated programs and ways to examine the information in image files, which our experts did. If they note that the bits are scrambled, they are. Also, the only workflow we support is the use of Optics Pro being first in your workflow, unless you use the Lightroom plug-in.

Since all programs process images a little differently, and our software must have the original, unchanged information to work properly, when using Optics Pro, you cannot use the other programs first in your workflow.

If you have questions about any other aspects of using the program, please let us know. Optics Pro is working properly in this area.

Thanks for the sample files.  I can almost guarantee that DXO's experts did not even look into the issue.  I compared your two sample files and I know for a fact that the image data is not scrambled.  The image data is absolutely unchanged.  The only thing Photo Mechanic did was relocate the TIFF table (at offset 4 in the file is an offset to where the TIFF table starts and usually has the value of 8, meaning that the TIFF table starts at offset eight) and add some extra metadata.  It is likely that the engineers at DXO wrote their parser to just read the start of the file and expect that the TIFF table is there but follow the TIFF table offset and use it.  Unfortunately if they don't seek in the file to that offset and read that data into memory and instead just use it as an offset into uninitialized memory then they would get results like you see in their application.  I'm surprised their software doesn't crash when reading these files.

Hopefully they can make the small change that would be necessary to follow the offset to the TIFF table, or just ignore it and use the original unaltered TIFF table that starts at offset 8.  Either method would solve the problem.

-Kirk

[vinya]

I'm experiencing the same problem. DXO just updated their software but failed to make the necessary changes to enable their programme to read RAW which has been ingested by PM. If its so simple why did they ignore this issue?

Kirk, is there anyway to reset the  PM ingested files back to their original state so that I can process my PM altered files through DXO?

[Kirk Baker]

I'm experiencing the same problem. DXO just updated their software but failed to make the necessary changes to enable their programme to read RAW which has been ingested by PM. If its so simple why did they ignore this issue?

Kirk, is there anyway to reset the  PM ingested files back to their original state so that I can process my PM altered files through DXO?

Yes, use the "Revert TIFF-based RAW to original" command on the Tools menu.

-Kirk

[vinya]

Thanks Kirk. Well, I re-ingested the fotos to PM and have tried your suggestion several times but still no luck. Is there something else I can do?

[Hayo Baan]

Do you have the same problem with another camera model as well, or only with the Sony?

As kirk already wrote, PM does not alter the image data in any way, just the structure of the file, something many other image manipulation programs do by the way. I'm very surprised DXO doesn't cope well with this...

[vinya]

Only with A900 run through PM. My Nikon and Pentax files are fine with DXO. A900 ARW are fine too, at least the ones I didn't ingest into PM.  Its only the ones I ran through PM that DXO fails to read. I've tried converting to DNG and followed Kirks advice aamong other work-arounds but still DXO doesn't like them, I'm exasperated.

[Kirk Baker]

Only with A900 run through PM. My Nikon and Pentax files are fine with DXO. A900 ARW are fine too, at least the ones I didn't ingest into PM.  Its only the ones I ran through PM that DXO fails to read. I've tried converting to DNG and followed Kirks advice aamong other work-arounds but still DXO doesn't like them, I'm exasperated.

If you've ran the tool on your photos and DXO still doesn't like them, I suggest taking it up with them.  Send them a sample file and ask them what they think is wrong with it.  Ask them to take a look at this thread.  I explained how simple it is to parse a TIFF-based RAW file correctly and their engineers should be able to correct their parser.  As far as I know, DXO and Capture One are the only software packages that cannot function with a relocated TIFF table in a RAW file.

The tool should have put the TIFF table offset back to its original value so I don't know what else there is to do.  If you want, you can send us a sample image and we'll see if the changes have been undone or not.  Please click on my name to the left of this message, then click on the 'personal message' link.  I will respond with upload instructions.

-Kirk

[vinya]

I have emailed DXO and sent them a file. Here is their reply:

When the original file information we need is changed, or even shifted by a single byte, the process to extract the information Optics Pro needs from the image file can fail. We use precise file information provided to us by the camera manufacturers. Therefore, when the information in the file no longer follows their specifications, that can cause issues. And yes, our developers have investigated this issue in the past.

If other programs want to absolutely insure that no changes are made to image files that they use, one suggestion would be that they follow the Adobe model. Adobe allows their customers to write any changes made to an image file to a sidecar file. Therefore, the original file is left in the exact same state as it came from the camera. Adobe does this with their .xmp files, and it allows our customers to successfully use files that have been used in programs such as Adobe Bridge.

As I noted, thank you for writing back and for your input, and I appreciate your taking time to write to us.

If you have any other questions regarding the use of Optics Pro, please feel free to contact us.

regards,




Jeff
DxO Support Team   
DxO Labs

[Hayo Baan]

Wow, for such an advanced program they really have a backwards way of interpreting files and treating customers...

Anyway, seems your only option is to not embed the iptc info, but to have PM create a sidecar file instead.

[Kirk Baker]

When the original file information we need is changed, or even shifted by a single byte, the process to extract the information Optics Pro needs from the image file can fail. We use precise file information provided to us by the camera manufacturers. Therefore, when the information in the file no longer follows their specifications, that can cause issues. And yes, our developers have investigated this issue in the past.

That's really interesting.  I know of no way to get the camera manufacturers to give any form of specification on their RAW file formats.  If they did, our job would be much easier.  The only way to understand the file formats is to examine the files produced by each specific camera model, basically reverse-engineering them.  Even Thomas Knoll had to do this when he wrote the Adobe Camera RAW plugin.  I know this because I attended a conference where he described what he went through and how many hundreds of cameras Adobe had to purchase and continues to purchase so they have a known source of RAW file formats.

It sounds like to me that they have a parser that uses specific offsets instead of knowing the file format (which will tell you the offsets) like other applications (ACR, Bridge, Photoshop Lightroom, Bibble, Capture NX, Capture NX2, Canon Digital Photo Professional, Aperture, etc.) do.

I suggest you get the latest version of Photo Mechanic, and set it to not modify your RAW files, using an XMP sidecar workflow, and avoid using any of the GPS features since they do modify RAW files.  Or switch to a different conversion program as there are plenty of flexible alternatives available.

Do you still need us to look into your sample RAW file?

-Kirk