DNG - Metadata sync with Capture One

[Kirk Baker]

...All data is expected to be in the DNG file, including the original RAW file used to produce the DNG file.

If this refers to original raw data from non-DNG, say a NEF, I believed embedding the original raw data in the DNG is optional, not mandatory.

True, it is optional.

PM also ignores XMP sidecar files for other formats like JPEG, PSD, PNG, etc. that provide the ability to embed metadata safely.

It follows, I think, that the DNG specification does not force PM to ignore DNG sidecars, it is more like a convention adopted by CB due to the fact that Adobe embeds metadata in DNG and expects it there, in contrast to other raw image formats.   

I think some of your users would appreciate an opening for CB applying Postel's law for all raw image files, including DNGs. 

I expect that if PM were to update XMP sidecar files on DNGs (to be more compatible with Phase One's software) then this would now cause incompatibility with Adobe products which we definitely wouldn't want.  Of course we could write to both the DNG and the XMP sidecar file, but other software won't do this (C1 for instance) and then the metadata in the XMP sidecar would become out of sync with the XMP in the DNG.  This is also undesirable.

It would be nice if C1 would have an option to modify DNGs (specifically, not for all RAW files) and then they would be more compatible with PM and Adobe products.

-Kirk

[Soizic]

What means CB ?

[Kirk Baker]

What means CB ?

CB = Camera Bits

[Odd Skjaeveland]

...Of course we could write to both the DNG and the XMP sidecar file, but other software won't do this (C1 for instance) and then the metadata in the XMP sidecar would become out of sync with the XMP in the DNG.

I probably need to state clearly that I use PM with Capture One without much trouble, if any at all. I maintain metadata in PM, even though it may require me to side track from C1 to PM for modifying metadata rather modifying metadata in C1 and be unable to pick up the changes in PM. It works ok for me the other way around, from PM to C1 (more in the next paragraph).  My reason for joining this thread was that I have seen this issue and hints of frustration among PM/C1 users in several forums.

If I modify metadata in PM, my C1 will read metadata from the DNG when I do "Reload Metadata" in C1. C1 prompts me with a "keep" or "overwrite" dialog if there is conflicting information in C1 and in the DNG. Overwrite means metadata from PM (in the DNG) enters C1, mission completed, I maintain metadata in PM and have them appear in C1 also for DNGs.

As far as I can tell, the problem is the other direction, from C1 to PM because C1 writes to XMP that PM ignores just because the XMP happens to be with a DNG rather than some other raw image file format. If C1 would honor an existing XMP sidecar, regardless who or what created it, C1 users should be able to modify metadata in C1 and have metadata appear in PM.

Suggestion for PM:
if raw file is DNG, write metadata to the DNG (like now)
if raw file is DNG and sidecar XML exists, read metadata from XML else read metadata from DNG (new feature)

I hope some other PM/C1 users will verify my description and that the suggested procedure would work for them.


A quick edit for the "read":

if raw file is DNG and newer XML exists then read metadata from XML else read metadata from DNG

[Soizic]

"If I modify metadata in PM, my C1 will read metadata from the DNG when I do "Reload Metadata" in C1. C1 prompts me with a "keep" or "overwrite" dialog if there is conflicting information in C1 and in the DNG. Overwrite means metadata from PM (in the DNG) enters C1, mission completed, I maintain metadata in PM and have them appear in C1 also for DNGs."

Excellent ! I work in the same way.

[dennis]

Hey Odd,

Thanks for your input!  You are correct that the DNG spec (1.4) does say "metadata may be embedded" (and also correct about it being an option to embed original RAW file).

I wanted to see how Photoshop handles this (since Adobe is the creator of that as well as DNG and XMP).  So I took a Leica M10 DNG file and "borrowed" an XMP file from a different photo (from an ARW).  Totally different metadata in DNG versus XMP sidecar.  I also made sure that the XMP sidecar file has a more recent modification time than the DNG file.  Then I opened the DNG file into PS (via ACR plug-in).  Guess what happens?

ACR uses the XMP metadata for the image loaded into PS (as we would hope), but then it also modified the DNG file to include this updated XMP, and then it deleted the XMP sidecar file!  So I could (jokingly) say that there is your fix for transferring XMP sidecar metadata from C1 into DNG so PM can read it: open the photo in Photoshop   Obviously this is not a solution.

If the DNG has a more recent modification date, then the XMP sidecar is ignored as far as what metadata is loaded into PS.  But PS/ACR still deletes the XMP sidecar!

This has a very unfortunate side effect if I put the DNG file (without an XMP sidecar) next to the ARW file (with an XMP sidecar), then make them have the same name, like this:

sample.DNG
sample.ARW    *-|   These belong together
sample.XMP    *-|

Guess what happens?  My ARW file now has no metadata after I open the DNG into PS/ACR, and if the ARW+XMP was more recently updated, then the DNG inherits the XMP of the wrong file.  Not good!

A similar issue could happen in PM if we associate XMP sidecars with DNG.  PM would have to figure out somehow whether the XMP belongs to the ARW or the DNG.  If it chose (incorrectly) the DNG, and I moved or deleted the DNG, then the ARW file would lose its metadata (and of course the DNG would have the wrong metadata).

So, although the DNG specification says that you may embed XMP metadata, I think the expectation is that you embed the XMP instead.

We certainly are NOT going to make any changes to PM 5, but we may do something about this in PM 6 but I'm not quite sure what to do that isn't a lot of work.  PM already has an issue where if you have, say, a NEF and a CR2 (etc) in the same folder with the same name, then an XMP sidecar file will be associated with both of them.  But this is a less likely situation than having a DNG alongside a NEF or CR2 (etc) with the same name.  For example I may choose to convert my NEF into a DNG with DNG Converter, and put the converted file into the same folder with the same name.

We will think this over carefully and try to fix this situation as best we can moving forward.

Regards,

--dennis

[Odd Skjaeveland]

Thank you very much for commenting on this issue, Dennis.

sample.DNG
sample.ARW    *-|   These belong together
sample.XMP     *-|

I guess even more entertaining things may pop up if I also put sample.JPG and sample.TIFF to the same folder with filename confusion enabled

It may well be the usual question of the lesser evil, and I trust you will decide on the sensible solution. PM could perhaps warn the user about conflicts and provide a list of options to rename/move, delete, overwrite etc. 

Like I said, I have few problems if any, not even with DNG files from my Leica. I will happily continue with PM and Capture One in my workflow.

[aram]

ACR uses the XMP metadata for the image loaded into PS (as we would hope), but then it also modified the DNG file to include this updated XMP, and then it deleted the XMP sidecar file!  [...] If the DNG has a more recent modification date, then the XMP sidecar is ignored as far as what metadata is loaded into PS.  But PS/ACR still deletes the XMP sidecar!

Hey Dennis,

This behavior can be prevented if you lock the DNG files (a.i. make them read-only). Some software like Iridient X-Transformer can be configured to produce locked DNGs by default, and then Adobe Camera Raw will do "the right thing". When you change settings, the XMP file is changed, not the DNG, the XMP is not deleted, etc.

Adobe Bridge has native support for locked files in the GUI. You can lock and unlock DNGs by right clicking on the file.

It would be great if PM would support read-only DNG workflow. My backup/version control system doesn't bode very well with large binary files that change constantly. It works very well when the RAW files never (or seldom) change, and all the edits are in a separate, smaller file.

Also, some of us we can't avoid this problem by avoiding DNGs in the first place. I use DNGs because Iridient X-Transformer can only generate DNGs, and I find ACR performs unacceptably with fuji RAF files.