PM, DxO, and metadata

[MikeA]

(Using PM 4.6.3 under Windows XP, SP2)

I am trying an evaluation version of the DxO RAW converter and have found that it can't open some images; some might have been run through Photo Mechanic first (though it's been long enough in some cases that I can't remember). In DxO's own forum, the claim was made that when Photo Mechanic updates metadata in RAW files, it alters the EXIF data in such a way that DxO can no longer read the data.

As DxO is heavily EXIF-based, this would certainly be a deal-killer, but the deal killed would likely be DxO. I'm not about to give up Photo Mechanic.

  [Edit: subsequent poking-around in the problem-child files revealed that it's unlikely
   Photo Mechanic had anything to do with the problem mentioned in this message.
   I'll post a reply with additional info. That aside, I'd love to know more about
   the IPTC/XMP-related questions posed in this message...]

If DxO can't read a file, this is apparently because it's having trouble with the EXIF data. But Photo Mechanic doesn't alter EXIF data -- only IPTC data. (Am I right about that?)

A message thread here on this topic included the following:

While adopting an XMP-only workflow when using Adobe products on RAW files sure is the easiest approach, you definitely can embed IPTC/XMP and  have everything work fine together (and is what I do).

You just have to be a bit more careful in this case and only update the IPTC/XMP info within PM. Well, even this is not completely true; you can even alter/add IPTC/XMP info with Adobe products just fine.  You then just have to have your PM XMP/IPTC preferences right (use the sidecar first), and always have PM "update" the IPTC/XMP afterwards to make sure the embedded data is synchronised with the sidecar info. If this sounds too complicated, stick to the sidecar only approach

There are quite a few PM user options w.r.t. IPTC data and XMP files -- I count seven separate drop-down menus or checkboxes that might have a bearing on this. The message quoted just above gives part of the story, but the dialog box in question has a lot more 'stuff' in it than is suggested above. What are the right combinations of settings allowing embedding of metadata in RAW files (Nikon .NEF in my case) such that both DxO and Adobe products won't have problems coping with these RAW files?

(The Adobe products in question are Lightroom 1.3, possibly 3.x in the future, and occasionally the current version of Camera RAW. Whether I go on to register DxO is an open question for me at the moment, but I would certainly like to be able to open files in it while I'm evaluating it.)

[MikeA]

A useful test might be to remove all PM-added metadata and then see if the files with the problems can be opened in DxO.

What is the correct procedure for restoring a given RAW (.NEF) file to its pre-Photo-Mechanic condition?

For a single image, I assume it would be to select the file in PM's contact-sheet view, click the "I" icon, select "Clear," and then click "Ok". Presumably that removes the IPTC data completely.

With multiple files, what would be the most efficient procedure for removing all IPTC data but without permanently altering the contents of the most recent IPTC stationery pad? It looks to me as if I could select multiple images, go to Image>IPTC Stationery Pad, clear the pad, then click "Apply Stationery to Selected." Then I would have to be careful to close the pad without saving changes. Maybe there's another approach, though.

[MikeA]

Poking around in these files with Phil Harvey's wonderful EXIFTOOL program, I find that I made entirely the wrong assumption about what happened.

The "Software" field for the problem-child files reads "http://www.idimager.com". I must have opened these files in a demo version of the program. (I don't own IDImager and don't plan to -- especially not now.)

The files that won't open in DxO have 100+ fewer EXIF fields compared with the files that do open in DxO. I never knowingly edit EXIF data. So I assume at least for argument's sake that IDImager removed some data that DxO needs. I don't know how to restore the missing data now. So much for tools capable of working with RAW files leaving files intact. The image data is untouched but some data critical at the least to DxO is now altered permanently. This might have happened not in IDImager but in a companion program for it, IDBatcher. IDBatcher is useful for basic save-for-web purposes but its editing features, last I looked, were pretty buggy. Maybe this is one of the bugs. Impossible to know at this point just what happened.

This certainly argues in favor of some other approach entirely to working with RAW files -- such as saving to DNG format first and despite the drive-space issues that entails. But I don't know yet how effectively DxO supports DNG format (if at all).

[Kirk Baker]

(Using PM 4.6.3 under Windows XP, SP2)

I am trying an evaluation version of the DxO RAW converter and have found that it can't open some images; some might have been run through Photo Mechanic first (though it's been long enough in some cases that I can't remember). In DxO's own forum, the claim was made that when Photo Mechanic updates metadata in RAW files, it alters the EXIF data in such a way that DxO can no longer read the data.

As DxO is heavily EXIF-based, this would certainly be a deal-killer, but the deal killed would likely be DxO. I'm not about to give up Photo Mechanic.

  [Edit: subsequent poking-around in the problem-child files revealed that it's unlikely
   Photo Mechanic had anything to do with the problem mentioned in this message.
   I'll post a reply with additional info. That aside, I'd love to know more about
   the IPTC/XMP-related questions posed in this message...]

If DxO can't read a file, this is apparently because it's having trouble with the EXIF data. But Photo Mechanic doesn't alter EXIF data -- only IPTC data. (Am I right about that?)

A message thread here on this topic included the following:

While adopting an XMP-only workflow when using Adobe products on RAW files sure is the easiest approach, you definitely can embed IPTC/XMP and  have everything work fine together (and is what I do).

You just have to be a bit more careful in this case and only update the IPTC/XMP info within PM. Well, even this is not completely true; you can even alter/add IPTC/XMP info with Adobe products just fine.  You then just have to have your PM XMP/IPTC preferences right (use the sidecar first), and always have PM "update" the IPTC/XMP afterwards to make sure the embedded data is synchronised with the sidecar info. If this sounds too complicated, stick to the sidecar only approach

There are quite a few PM user options w.r.t. IPTC data and XMP files -- I count seven separate drop-down menus or checkboxes that might have a bearing on this. The message quoted just above gives part of the story, but the dialog box in question has a lot more 'stuff' in it than is suggested above. What are the right combinations of settings allowing embedding of metadata in RAW files (Nikon .NEF in my case) such that both DxO and Adobe products won't have problems coping with these RAW files?

(The Adobe products in question are Lightroom 1.3, possibly 3.x in the future, and occasionally the current version of Camera RAW. Whether I go on to register DxO is an open question for me at the moment, but I would certainly like to be able to open files in it while I'm evaluating it.)

If you're going to use DxO you should never modify the RAW files.  Never embed anything.  From what I understand, DxO doesn't like it when the TIFF table is relocated.  They appear to have a fairly naïve parser and can't follow the link to the location of the TIFF table.  This modification happens when IPTC or XMP data is embedded into a RAW file.  EXIF data is not modified in this process.  Photo Mechanic will modify EXIF data if you use it to adjust Capture Times, or embed GPS coordinates.  But you can easily stay away from using those two features.

It's pretty easy to setup PM to have an XMP sidecar file only workflow.  I explained how in this thread:

http://forums.camerabits.com/index.php?topic=5377.0

-Kirk

[Kirk Baker]

A useful test might be to remove all PM-added metadata and then see if the files with the problems can be opened in DxO.

What is the correct procedure for restoring a given RAW (.NEF) file to its pre-Photo-Mechanic condition?

Use the 'Revert TIFF-based RAW to original...' command on the Tools menu.

For a single image, I assume it would be to select the file in PM's contact-sheet view, click the "I" icon, select "Clear," and then click "Ok". Presumably that removes the IPTC data completely.

With multiple files, what would be the most efficient procedure for removing all IPTC data but without permanently altering the contents of the most recent IPTC stationery pad? It looks to me as if I could select multiple images, go to Image>IPTC Stationery Pad, clear the pad, then click "Apply Stationery to Selected." Then I would have to be careful to close the pad without saving changes. Maybe there's another approach, though.

Clearing out the IPTC fields will just leave you with a fairly small IPTC record in your images.  It won't cause the removal of the IPTC data.  Use the method I described above.  It works for single images and batches of images.

-Kirk

[MikeA]

Clearing out the IPTC fields will just leave you with a fairly small IPTC record in your images.  It won't cause the removal of the IPTC data.  Use the method I described above.  It works for single images and batches of images.

Interesting result. It's certainly a fast approach, but it didn't work with two of the files I'm using here for testing.

They'd been processed in some fashion with a demo version of IDImager. For whatever the reason, PM can't delete their IPTC data. Maybe EXIFTOOL can. For whatever it's worth, PM also can't convert those two to DNG format, and sometimes it can't extract their preview images (but that problem appears intermittent). When the DNG converter is run from the command line, it won't save those two files, either. I can't be certain that IDImager or IDBatcher fouled up the files. But they're the only two of these test files with the EXIF "Software" tag's value having been changed to the URL for IDImager. Fairly reasonable circumstantial evidence, anyway.

[Kirk Baker]

Mike,

Clearing out the IPTC fields will just leave you with a fairly small IPTC record in your images.  It won't cause the removal of the IPTC data.  Use the method I described above.  It works for single images and batches of images.

Interesting result. It's certainly a fast approach, but it didn't work with two of the files I'm using here for testing.

They'd been processed in some fashion with a demo version of IDImager. For whatever the reason, PM can't delete their IPTC data. Maybe EXIFTOOL can. For whatever it's worth, PM also can't convert those two to DNG format, and sometimes it can't extract their preview images (but that problem appears intermittent). When the DNG converter is run from the command line, it won't save those two files, either. I can't be certain that IDImager or IDBatcher fouled up the files. But they're the only two of these test files with the EXIF "Software" tag's value having been changed to the URL for IDImager. Fairly reasonable circumstantial evidence, anyway.

I have no idea how IDImager captions RAW files, but apparently it doesn't do it the way Photo Mechanic does it which is completely undo-able.

-Kirk

[MikeA]

I have no idea how IDImager captions RAW files, but apparently it doesn't do it the way Photo Mechanic does it which is completely undo-able.

The only way they'll get Photo Mechanic away from me is to pry it out of my cold, dead fingers. :-)

I shouldn't be too quick to blame IDImager entirely. These particular files were used for testing and it might be that some other application trashed up the EXIF data in some of them. (At least they can still be opened in Photoshop.)

Back to the DxO hassles w.r.t. metadata: if I understood correctly, in embedding IPTC data Photo Mechanic moves the location of a table within the file, but the pointer to the table's location is also updated. But it seems DxO isn't finding the table's location because it is expecting some specific (hard-coded?) location. Do I have that right?

When PM reverts an image by removing the IPTC data, does it also relocate/revert the table and update the "here's where you'll find the table" data?

If so, at the very least the data could be removed temporarily via the 'revert' command, stored at that time in 'sidecar' files, and returned to the files -- again via embedding -- after they are processed in DxO. This seems like an awful lot of trouble to go through for one less-than-convenient application, meaning DxO itself, but the thing is: when DxO is good, it's really good...its lens de-blur and other automated corrections are like nothing I've seen before. Functionality from heaven; UI from hell. (sigh) So it'd be useful to have an approach that gets the job done with minimal pain. (Emphasis on minimal. Clearly a workflow involving DxO is not going to be a "no pain" situation.)

[MikeA]

This occurred to me after I saw your recommendation about using only 'sidecar' files when working with DxO...

A number of apps write small companion files containing settings. Some (Lightroom, ACR) appear to go entirely with the XMP standard. At least one I can think of (SilkyPix) writes some kind of proprietary binary file that nobody else reads or writes to. Others -- Bibble, for example -- write XMP-ish sidecars but for some reason add some extra stuff into the files' extensions; I guess it means that the programs have somehow extended the XMP standard to suit themselves.

When there's an XMP-sidecar-only workflow, do you ever find that some applications append data to the sidecars in a way that causes later problems if the files are viewed again in Photo Mechanic?

Let's say the metadata was added only to XMP files, and not embedded in the RAW files. Then the user opens the files in Lightroom and makes edits -- and finally, uses Lightroom's 'save settings to XMP files' command. Now we probably have a whole bunch of information in the sidecar file that Lightroom understands but that is not used by or understood by Photo Mechanic.

That you know of...in a case like that is any information first written to the sidecar file by Photo Mechanic ever removed or trashed by Lightroom? (Or whatever other 'sidecar-enabled' app we might be talking about.)

[Kirk Baker]

Mike,

This occurred to me after I saw your recommendation about using only 'sidecar' files when working with DxO...

A number of apps write small companion files containing settings. Some (Lightroom, ACR) appear to go entirely with the XMP standard. At least one I can think of (SilkyPix) writes some kind of proprietary binary file that nobody else reads or writes to. Others -- Bibble, for example -- write XMP-ish sidecars but for some reason add some extra stuff into the files' extensions; I guess it means that the programs have somehow extended the XMP standard to suit themselves.

When there's an XMP-sidecar-only workflow, do you ever find that some applications append data to the sidecars in a way that causes later problems if the files are viewed again in Photo Mechanic?

Let's say the metadata was added only to XMP files, and not embedded in the RAW files. Then the user opens the files in Lightroom and makes edits -- and finally, uses Lightroom's 'save settings to XMP files' command. Now we probably have a whole bunch of information in the sidecar file that Lightroom understands but that is not used by or understood by Photo Mechanic.

That you know of...in a case like that is any information first written to the sidecar file by Photo Mechanic ever removed or trashed by Lightroom? (Or whatever other 'sidecar-enabled' app we might be talking about.)

It's hard to say what all of these other programs might do, but Adobe's apps certainly don't delete the Photo Mechanic data in in XMP files.

-Kirk

[MikeA]

It's hard to say what all of these other programs might do, but Adobe's apps certainly don't delete the Photo Mechanic data in in XMP files.

Good to know.

In the DxO forum I am having a respectful disagreement with another member who believes that because DxO requires 'pristine' files ('factory RAW,' he calls it), the alteration of files via embedding of IPTC data indicates a 'deficiency' in any program that does so.

I'm having a bit of trouble wrapping my mind around the concept of embedding new data into a file without altering the file. (Perhaps there's something in String Theory that could explain it. :-)

(The idea of a sidecar-only workflow isn't entirely appealing. Grumble.)

[Kirk Baker]

Mike,

I have no idea how IDImager captions RAW files, but apparently it doesn't do it the way Photo Mechanic does it which is completely undo-able.

The only way they'll get Photo Mechanic away from me is to pry it out of my cold, dead fingers. :-)

I shouldn't be too quick to blame IDImager entirely. These particular files were used for testing and it might be that some other application trashed up the EXIF data in some of them. (At least they can still be opened in Photoshop.)

Back to the DxO hassles w.r.t. metadata: if I understood correctly, in embedding IPTC data Photo Mechanic moves the location of a table within the file, but the pointer to the table's location is also updated. But it seems DxO isn't finding the table's location because it is expecting some specific (hard-coded?) location. Do I have that right?

Correct.

When PM reverts an image by removing the IPTC data, does it also relocate/revert the table and update the "here's where you'll find the table" data?

Correct.

If so, at the very least the data could be removed temporarily via the 'revert' command, stored at that time in 'sidecar' files, and returned to the files -- again via embedding -- after they are processed in DxO. This seems like an awful lot of trouble to go through for one less-than-convenient application, meaning DxO itself, but the thing is: when DxO is good, it's really good...its lens de-blur and other automated corrections are like nothing I've seen before. Functionality from heaven; UI from hell. (sigh) So it'd be useful to have an approach that gets the job done with minimal pain. (Emphasis on minimal. Clearly a workflow involving DxO is not going to be a "no pain" situation.)

I'd say that rather than worry about those two specific files that appear to have had EXIF data removed, I suggest shooting some new images and then put them through your proposed workflow and see how things go.  Since you'd be working with throw-away images there would be no loss if one of the tools in the chain caused you issues and you would then know which tools to avoid.

-Kirk

[Kirk Baker]

Mike,

It's hard to say what all of these other programs might do, but Adobe's apps certainly don't delete the Photo Mechanic data in in XMP files.

Good to know.

In the DxO forum I am having a respectful disagreement with another member who believes that because DxO requires 'pristine' files ('factory RAW,' he calls it), the alteration of files via embedding of IPTC data indicates a 'deficiency' in any program that does so.

I'm having a bit of trouble wrapping my mind around the concept of embedding new data into a file without altering the file. (Perhaps there's something in String Theory that could explain it. :-)

(The idea of a sidecar-only workflow isn't entirely appealing. Grumble.)

It took a long time for users to convince us to add the embedding of IPTC and XMP into TIFF-based RAW files.  We resisted for quite a while but when you get feedback from hundreds of users that they want to have their metadata in their RAW files, we listened.

-Kirk

[theim]

Hi MikeA,

I had the same issue a few months back.  Below is my correspondence with DxO and Kirk's advice...

______________________________________________________

My message to Kirk on September 25, 2009:
I consider this a DxO issue, but thought you might be interested in this issue.  I just purchased a Sony A850 camera and am testing DxO for RAW conversion of the Sony files.  I have noticed that if I use PM to write metadata to the Sony RAW files, then DxO cannot open them.  If I then used PM to remove the metadata (Tools/Delete Metadata), then DxO CAN open them.

I sent the following [email] to DxO:

Hello,

I am evaluating this software for use with Sony A850 RAW files.
I also use Photo Mechanic (PM) for all metadata tagging. PM has become a crucial part of my workflow and integrates quite well with all my other applications (ie, Adobe Lightroom, Nikon Capture NX2, etc.).

I like DxO so far, but it will not open RAW files from the Sony A850 if those RAW files have first been tagged with metadata using PM.
DxO is able to open RAW files from my Nikon D3 if those RAW files have first been tagged with metadata using PM.

I am wondering if this is something that could be looked at? Why is DxO able to handle Nikon D3 files and not Sony A850 files (when both have been tagged with metadata using PM)?
Compatibility with PM is absolutely necessary in my case. This must be resolved before I am able to purchase DxO.

Thank you.
______________________________________________________

DxO reply:

Bonsoir Tom,

This would very much depend upon the specific metadata that is being changed by Photo Mechanic in the D3 and A850 Raw files, respectively. and how that EXIF data is used by DxO in processing A850 output files.

Generally speaking, changes made to anything contained in the Makers Notes is critical. And should neber be done.

Also, the A850 files likely have a substantively different EXIF data structure than D3 files.  We have had instances where the alteration of a single EXIF tag that is critical to DxO's recognition of the Raw file or something that is used for DxO's computations (functions are not done by Adobe Lightroom, Nikon Capture NX2, etc.) can render the Raw file unrecognizable by DxO or cause errors in it's extensive image corrections.
 
We would need to see and analyse specific examples to determine the cause in your particular case.  If you would be willing to provide such examples of unaltered and written to by PM D3 and A850 Raw files (same metadata changed by PM in both) the D3 and A850 Raw files), we can have a look to see what is causing the problem.

If you would like us to have a look at this problem, please upload two (2) repesentative sample sets of D3 and A850 Raws with and without PM's changes.

You can upload the files to our file server.

Then, please reply to this email to confirm that your data are available on our server for us to look at.

The problem here is that other Sony A850 owners may wish to do the same as you (with software other than PM).

So only if it turns out to be something very simple, will the results provide you with any relief.

There is very good reason why we insist that only directly out-of-camera Raw files be input to DxO.
______________________________________________________

Kirk's reply to me:
We do not modify the Maker Note data.  What we do is relocate the TIFF table from the beginning of the file to the end of the file, add one or more entries for the embedded IPTC and/or XMP data and adjust the TIFF table offset at the beginning of the file to point to the new location.  Some software applications do not follow these offsets properly and expect certain data to exist at exact offsets (they ignore the offsets specified in the file.)  This causes them problems.

There are two solutions:
1) Get the authors of that software to update their code to follow the offsets.
2) Instruct PM to use an XMP sidecar only workflow so that PM doesn't embed IPTC or XMP.

Regards,

-Kirk

______________________________________________________

How it ended up:

DxO was not willing to make any modifications.
I don't like to use sidecar files and I will not give up on PM.
So I replied to DxO that I would not be able to purchase their product.

Hope this helps,
-Tom

[MikeA]

How it ended up:

DxO was not willing to make any modifications.
I don't like to use sidecar files and I will not give up on PM.
So I replied to DxO that I would not be able to purchase their product.

Thanks for passing along the information about their response. The discussion in their forum (the company itself doesn't participate there, much) turned a wee bit nasty as one of the easily offended fanboy types insisted that it's a problem with PM alone and that DxO Labs shouldn't bother wasting time on other programs' problems. This is also someone who believes it "can't" read DNG files even if they contain the original RAW image fully embedded into the DNG file. This is just stupid. The offsets involved are documented and this is a parsing issue, not an "it can't be done" issue. But that's fanboys for ya. They can find any reason why The Precious "can't" do something they don't want it to bother doing. There are none so blind, etc.

I don't like the sidecar-only solution, either. I am trying to figure out how I might do something of a tap-dance around this problem to get what I need done with DxO, if I decide to register it. And the "roll eyes, sigh, decide not to use it, and move on" scenario is certainly one of the options.