Tools to inspect RAW file structure?

[msbc]

Not directly related to PM, but I'm asking here because Camera Bits and/or it's users might have the info I need.

I have some Canon 10D RAW files (.CRW and .THM) that Mac OS X (10.4. is reporting as 'unsupported format'. OS X supports RAW formats directly in the O/S so this means that Finder, Preview, Aperture and iPhoto all see the images as unsupported. The strange thing is that all of my other image apps (PM, Canon DPP, COne, iView, LightRoom, PhotoShop) all have no problems with these files as they rely on their own RAW support.

So, it seems that somehow, in manipulating these particular files, something in their structure has changed to make them look corrupted to OS X.

I am therefore looking for information and/or software that will allow me to inspect the file structure so I can maybe compare it with other files and diagnose what the problem might be.

Any help would be greatly appreciated.

Mark

[Kirk Baker]

Mark,

Not directly related to PM, but I'm asking here because Camera Bits and/or it's users might have the info I need.

I have some Canon 10D RAW files (.CRW and .THM) that Mac OS X (10.4. is reporting as 'unsupported format'. OS X supports RAW formats directly in the O/S so this means that Finder, Preview, Aperture and iPhoto all see the images as unsupported. The strange thing is that all of my other image apps (PM, Canon DPP, COne, iView, LightRoom, PhotoShop) all have no problems with these files as they rely on their own RAW support.

So, it seems that somehow, in manipulating these particular files, something in their structure has changed to make them look corrupted to OS X.

I am therefore looking for information and/or software that will allow me to inspect the file structure so I can maybe compare it with other files and diagnose what the problem might be.

I usually use a hex editor.  There are many hex editing applications.  I use Hex Edit.

If you want to upload a sample (both the CRw and the THM file) we can take a look at it and see what's up.  Contact me privately for server info.

-Kirk

[dennis]

So, it seems that somehow, in manipulating these particular files, something in their structure has changed to make them look corrupted to OS X.

I am therefore looking for information and/or software that will allow me to inspect the file structure so I can maybe compare it with other files and diagnose what the problem might be.

Mark,

You might want to try ExifTool for examining file structure if you are up to it (there is quite a high level of nerdiness required).  However, I suspect that you won't see any problem with ExifTool since it is able to handle basic modifications to RAW files.

As we have mentioned in previous posts, Apple's OS X is not able to handle even simple modifications to certain RAW files (e.g. appending a single byte to an ORF).  It depends on the RAW format (e.g. Olympus ORF, Canon 1D TIF, and even Adobe DNG all seem to have various problems, and it may depend on the camera model).  We have notified Apple about these bugs with parsing certain RAW formats and hopefully they will respond with a new OS update to fix the parsing bugs (yes, an OS update is needed to fix these problems and consequently maybe all of your iApps as well).  But since there have been several OS updates already with no fixes for certain formats (e.g. ORF, 1D TIF), all we can say is to just keep your fingers crossed.  Naturally, since Aperture relies upon the OS for RAW formats, don't count on a fix for that app without OS help.

But as far as CRW files, I am surprised that you are having a problem since it is a different format from most other RAW formats that are TIFF based.  Certainly Photo Mechanic doesn't do anything to CRW files that should cause a problem with Mac OS X.  Are you using some other applications to edit these files?

--dennis

[msbc]

Hi Dennis,

Yes, I am familiar with ExifTool - I have examined a couple of the problem files and compared with good files and can't see any EXIF or IPTC differences that would indicate file changes.

The files in question haven't been touched in any way since Feb 05 so I can't recall what app or version I used back then - could have been PM, Canon DPP, iView or all 3!

I just started using Aperture after the 1.5 update so only noticed this problem recently.

I have accepted Kirk's kind offer and uploaded a sample to your server.

Thanks,
Mark

[dennis]

The files in question haven't been touched in any way since Feb 05 so I can't recall what app or version I used back then - could have been PM, Canon DPP, iView or all 3!

I looked at your file and it has a "CANON OPTIONAL DATA" segment appended to the end of the file (about 250K worth) which is written by DPP to store image adjustments.  The data isn't actually just appended like DPP does with CR2 files since CRW files are parsed recursively from the end-of-file.  But it looks like it is properly attached to the CIFF tree.

Apparently Apple didn't test CRW files with DPP

I'll file a bug report about this with Apple.  Hopefully they will fix all these parsing problems in a 10.4.9 update, otherwise we may have to wait till Leopard.

--dennis

[msbc]

Dennis,

Very interesting. Do you know of any way I could 'edit' the file to remove this extra info and return it to it's pre-DPP state?

Thanks,
Mark

[dennis]

Very interesting. Do you know of any way I could 'edit' the file to remove this extra info and return it to it's pre-DPP state?

Mark,

I tried removing the appended CANON OPTIONAL DATA but it still showed as "corrupted" in Preview 10.4.8 (even though it parsed just fine in PM and Photoshop).  So perhaps one would have to dig a little deeper.

I suspect that if you take a CRW file and make a copy of it, then edit the copy in DPP, then look at the difference in file sizes, you might be able to create a simple Unix command to truncate the difference from the end of file and retrieve the original version.  Of course you don't want to try this on your DPP-modified "originals", only copies of them.

The way CRW files are formatted is as a CIFF tree structure, and you can add more data to an existing CIFF tree by appending the data you want to add, along with the appropriate links.  However, I'm not sure if the header was changed at all.

I don't know why Apple would fail to parse this if they follow the CIFF parsing logic, but then again, there are other bugs such as appending a single byte to an Olympus E-300 ORF file that throws OS X off... 

--dennis

[msbc]

Dennis,

One of the guys from the Aperture team read my post about this on the Apple Disscussions board and contacted me. I've sent him a sample image and they are going to investigate.

Cheers,
Mark

[msbc]

Just thought I would let you know that the Digital Camera RAW Support Update fixes the problem with my .CRW files.