Hi Giles,
I've looked into this problem regarding Nikon's codec and Microsoft's Photo Info (MPI) add-on and here is the scoop from examining a "corrupted" RAW NEF photo. I first need to say that I haven't fully tested all options (e.g. XP with MPI, Vista with MPI, Vista with MPI + Nikon codec).
Readers should first read Microsoft's FAQ to understand what is officially going on here:
http://www.microsoft.com/windowsxp/using/digitalphotography/prophoto/photoinfofaq.mspxMPI uses a more "intrusive" method to add metadata than Photo Mechanic. It essentially rewrites the RAW file, whereas Photo Mechanic "patches" the RAW file by creating a new TIFF table (with new tags for IPTC/XMP) and appending this to the end-of-file, along with the IPTC/XMP data. PM then points to this new TIFF table by changing the "IFD" pointer at offset 4 in the file. The advantage with PM's method is that (other than the 4 bytes for the IFD pointer) the existing contents of the file remain completely untouched. When combined with some proprietary technology I invented, this lets PM "undo" any changes made by PM so that you can get back to the original RAW file (note: this applies only to TIFF-based RAW files which covers most RAW formats other than, for example, CRW and RAF).
Why is it important to get back to the original RAW file? Well, because not all software is smart enough to properly parse TIFF-based RAW files that have been modified (e.g. Mac OSX and therefore Aperture), even if the modifications are relatively minor. And theoretically, a RAW file is proprietary to the manufacturer and so there is no guarantee that modifications made to the RAW file will sit well with the manufacturer's own software. So far this hasn't been much of a problem with PM's method and, for example, Nikon Capture (and Capture NX) or with Photoshop. But other software that makes assumptions about the RAW format can get tripped-up by even the most basic changes made to a RAW file (e.g. assumptions like the TIFF table is still at the front of the file).
Unfortunately, by rewritting the RAW files as done by MPI, there is no guaranteed way to "undo" the addition of metadata if some software were to have problems reading the modified RAW file. Assuming there are no bugs with MPI or a codec employed by MPI, the modified RAW file should still be properly formatted and any errors reading the modified RAW file are because of improper parsing or assumptions about the RAW file's formatting. The exception here is the insidious private "maker note" used by manufacturers to include info about the photo that isn't expressable by Exif (or to hide info). If a maker note isn't properly "self-contained" such that it only references data inside itself and relative to itself (not the start of file), then moving the maker note in the file can break the maker note.
OK, now back to the RAW NEF file in question. What apparently is happening is that Nikon's codec not only rewrites the file in order to insert XMP (not IPTC), but it changes the byte ordering (endian) from big endian (Motorola) to little endian (Intel). I don't understand why it does this (since little endian sucks when it comes to examining a file in a hex editor), but technically there is nothing wrong with the byte order change. However, the maker note is left unchaged as big endian. Again, technically this is OK since Nikon's maker note is self-contained and internally specifies the byte ordering.
However, software that expects the endian of the maker note to be the same as the RAW TIFF wrapper (e.g. NEF) will obviously fail when trying to parse the maker note. This is apparently the case with Photoshop and Mac OSX (and other software no doubt). The file itself isn't corrupted in any way and is technically OK. It is surprising that Photoshop has a problem with these files since the DNG specification allows for maker notes to have a different byte ordering (endian) than the DNG container.
Photo Mechanic 4.4.3.3 itself has a small problem with this endian change, but it detects the problem and simply skips the parsing of the maker note. This means that you lose certain meta data (e.g. serial number), but PM will still show the photo OK. I have updated 4.5 to be able to handle this maker note endian change so that it can properly parse the maker note.
Given the problems with various RAW converters and operating systems, the safest solution is to leave the RAW file untouched and only create an XMP sidecar file. Photo Mechanic can be setup to operate this way. But some people who use software (e.g. Nikon Capture) that can properly handle embedded IPTC data prefer to configure PM to embed metadata into the RAW files. When encountering software that fails to read the modified RAW file, PM users can revert to the original file. Unfortunately this isn't an option with MPI which will always rewrite your RAW file. My advice: don't use Photo Info or software that relies upon the internal Windows Imaging Component to caption your RAW files until you have tested the modified RAW files with various RAW converters or browsers you use for compatibility.
It is really a shame that the state of RAW captioning continues to deteriorate rather than improve (outside of PM of course
).
--dennis
