Exchanging with darktable

[nwinspeare]

Hello,
I now use darktable as my raw developper as it has become in my view a very professional editor. My problem is exchanging tags...
Darktable uses xmp files stored in a <filename>.<ext>.xmp format whereas most apps (LR,C1...) use <filename>.xmp
Could it be possible to have the option for Photomechanic to read/write to <filename>.<ext>.xmp ?
Thanks,
Nicolas

[Kirk Baker]

Nicolas,

I now use darktable as my raw developper as it has become in my view a very professional editor. My problem is exchanging tags...
Darktable uses xmp files stored in a <filename>.<ext>.xmp format whereas most apps (LR,C1...) use <filename>.xmp
Could it be possible to have the option for Photomechanic to read/write to <filename>.<ext>.xmp ?

No.  That's completely non-standard.  The Darktable developers should follow the standard.

-Kirk

[LateJunction]

Kirk,
I have just, inadvertently, posted a Support Ticket on this very subject, not realising this Feature Request part of the forum is available. Your response seems awfully blunt, if you'll forgive my saying so. What is this 'standard'? Is it an ISO standard? If not, who defines and maintains it? What are the 'penalties' associated with not following the 'standard? Could you make a little more time available to consider if there is a work-around you could suggest?

Tony Hamilton

[Bob M]

Perhaps LateJunction might be interested in the thread https://discuss.pixls.us/t/free-software-and-photo-metadata-a-chance-to-engage-with-the-broader-photography-industry/23068 on the pixls.us forum.

[Kirk Baker]

Tony,

Your response seems awfully blunt, if you'll forgive my saying so. What is this 'standard'?

It is not an official standard.

Is it an ISO standard? If not, who defines and maintains it?

No, not ISO.  It is a de-facto standard from Adobe.

What are the 'penalties' associated with not following the 'standard?

Lack of interoperability with other software (you've discovered this already).  Lower performance when querying the file system for the existence of these additional files.  Duplication of data in both the standard basename.XMP and the basename.ext.XMP files.  More file system clutter.  Lack of synchronization of the files depending on what software is used.  I really could go on with how bad it is if I were to think about it some more.

Could you make a little more time available to consider if there is a work-around you could suggest?

I can't think of a workaround.  We would have to modify our software to accommodate this unusual use of XMP files.

-Kirk

[pll]

Hi,

I'm a little new here, and I see that this thread is a bit old, but I'm facing the exact same issue. While I understand it's Camerabits' position that the Darktable developers are "not following the standard", I find Kirk's position a little arrogant and off-putting.  First off, the XMP "standard" is not an official standard recognized by any official body. It's a "defacto" standard, also known as a "convention".  Additionally, I've read the standard. There is nothing in the standard dictating the filename format. There is merely one sentence, on Page 7 of Part 3 (https://github.com/adobe/XMP-Toolkit-SDK/blob/main/docs/XMPSpecificationPart3.pdf) which states:

For applications that need to find external XMP files, look in the same directory for a file with the same name as the main document but with an .xmp extension. (This is called a sidecar XMP file.)

So, one could argue that Darktable IS following the "standard", since the original name of the file is preserved, with a .xmp extension added to it, and PM is violating said "standard" by REPLACING the existing extension with .xmp instead. One could also argue that since Darktable is the only software doing something different, then "what everyone else does" is the defacto standard.  But that's kind of a lame argument when there is no official standard and the only document dictating a convention is ambiguous in it's definition of "proper filename".

Keep in mind, this "standard" is being put forth by Adobe. The very software company many of us are attempting to avoid. For Camberabits to essentially claim, "Sorry, we're following whatever our competitor dictates." doesn't really seem like the best business decision. (case in point, Word Perfect did this when Microsoft release MS Word in early 1990s. Look where WP is now...)

To claim "we'd have to change our software to something non-standard" is ridiculous. Camberabits and every other software company on the planet changes their software all the time, if they didn't, they wouldn't stay in business.

Additional PM ALREADY supports different naming conventions, just open up the preferences and look under the Files section and you'l see a drop-down clearly labelled "File Extensions". So really, how hard is it to add one more option to preserve the original filename and APPEND the .xmp extension instead of replacing it?

As a software developer myself, I find it ridiculous that that's the excuse being used. To add support for a filename preference when saving to disk is not rocket science. For someone who knows the code it's likely less than a day's worth of work to get it working, and at most a week's worth of QA to write and run all the proper unit and integration tests for the automated CI/CD pipeline to verify.

While I understand Kirk is probably not the official spokesperson for everything Camerabits says or does, it would be far more becoming of the company for someone to state something like, "Hey, this is a great idea. I'm not sure how soon we could support such a feature, given that it goes against the defacto standard, but I'll propose it to our Product Team and see what they say."

For the time being, and for those on Macs, a simple workaround would be to either copy or (hard or soft) link he Photomechanic XMP file to a Darktable-compatible filename. Sym (soft) Linking would likely be a better idea as it gives you more flexibility in that you end up with seemingly 2 different xmp files, but Darktable would access the real file using the symlink, and PM via it's own filename convention.

Sincerely,
Paul
A new PhotoMechanic user

[Kirk Baker]

Paul,

I'm a little new here, and I see that this thread is a bit old, but I'm facing the exact same issue. While I understand it's Camerabits' position that the Darktable developers are "not following the standard", I find Kirk's position a little arrogant and off-putting.

My apologies, I did not mean to offend the original poster or anyone else who read my response.

First off, the XMP "standard" is not an official standard recognized by any official body. It's a "defacto" standard, also known as a "convention".

Agreed, I admitted as much in my second post.

So, one could argue that Darktable IS following the "standard", since the original name of the file is preserved, with a .xmp extension added to it, and PM is violating said "standard" by REPLACING the existing extension with .xmp instead. One could also argue that since Darktable is the only software doing something different, then "what everyone else does" is the defacto standard.  But that's kind of a lame argument when there is no official standard and the only document dictating a convention is ambiguous in it's definition of "proper filename".

Keep in mind, this "standard" is being put forth by Adobe. The very software company many of us are attempting to avoid. For Camera Bits to essentially claim, "Sorry, we're following whatever our competitor dictates." doesn't really seem like the best business decision. (case in point, Word Perfect did this when Microsoft release MS Word in early 1990s. Look where WP is now...)

To claim "we'd have to change our software to something non-standard" is ridiculous. Camera Bits and every other software company on the planet changes their software all the time, if they didn't, they wouldn't stay in business.

Additional PM ALREADY supports different naming conventions, just open up the preferences and look under the Files section and you'l see a drop-down clearly labelled "File Extensions". So really, how hard is it to add one more option to preserve the original filename and APPEND the .xmp extension instead of replacing it?

As a software developer myself, I find it ridiculous that that's the excuse being used. To add support for a filename preference when saving to disk is not rocket science. For someone who knows the code it's likely less than a day's worth of work to get it working, and at most a week's worth of QA to write and run all the proper unit and integration tests for the automated CI/CD pipeline to verify.

While I understand Kirk is probably not the official spokesperson for everything Camera Bits says or does, it would be far more becoming of the company for someone to state something like, "Hey, this is a great idea. I'm not sure how soon we could support such a feature, given that it goes against the defacto standard, but I'll propose it to our Product Team and see what they say."

Much of what you said above has merit though I don't agree with some of your assumptions about implementation time, testing time, etc.  I'll bring it up with the team again for reconsideration.  Developer resources are finite and we have to make sure we are using them to maintain and improve our software that benefits our user base in the best ways possible.

-Kirk

[pll]

Hi Kirk,

Thank you for your response, I really appreciate it.

Much of what you said above has merit though I don't agree with some of your assumptions about implementation time, testing time, etc.  I'll bring it up with the team again for reconsideration.  Developer resources are finite and we have to make sure we are using them to maintain and improve our software that benefits our user base in the best ways possible.

-Kirk

You wouldn't be the first person to tell me I drastically underestimate the time and effort required to implement something    My team is constantly telling me the same thing!

My main point was simply that handling the ability to customize the output filename for the XMP sidecar file is not a monumental task, especially given that PM already handles 4 different cases, and some of them are conditional based on operating environment plus the duplicate name case, which has 5 different options.  Therefore, my assumption is that it shouldn't be overly difficult to implement the option of

Code: [Select]

<filename>.<orig ext>.xmp as another option .

Of course, you know the code, I don't. Therefore, I can't possibly estimate a reasonable implementation time complete with testing. My apologies for saying otherwise.

In the end, I think all the customers want is the ability to use a great program like Photo Mechanic in the most flexible way possible. Obviously dev and QA resources are finite, and there are a ton of features being requested, some with more merit than others, some with a lot of merit, but won't drive a lot of revenue, some with little merit, but might drive a ton of revenue. That's for the product people to weigh and balance. Our job as customers is to clearly articulate why we need/want a feature, hopefully with the software devs understanding what we're asking for, and lobbying to the product people on our behalf, or explaining to us why what we're asking for isn't viable.

Again, I thank you for your response, and hopefully you and the other developers can find a way to both deliver what I think would be a very welcome feature to many of us, as well as convince the product team that it's worth working on.

FWIW, I'm happy to be a guinea pig and beta test such a feature, if that's at all helpful.

Thanks.
--
Paul

[dennis]

My main point was simply that handling the ability to customize the output filename for the XMP sidecar file is not a monumental task, especially given that PM already handles 4 different cases, and some of them are conditional based on operating environment plus the duplicate name case, which has 5 different options.  Therefore, my assumption is that it shouldn't be overly difficult to implement the option of

Code: [Select]

<filename>.<orig ext>.xmp as another option .

Of course, you know the code, I don't. Therefore, I can't possibly estimate a reasonable implementation time complete with testing. My apologies for saying otherwise.

Hi Paul, thanks for your thoughts.  You are correct, you don't know our code.  Apologies accepted.

We will look into this and it may in fact be easy to accommodate the atypical "understanding" of how XMP sidecar files are named.  We already have a framework for this type of "attached" filename with Bibble I think.

But are you suggesting that we should do something completely non-standard like creating FOO.JPG.XMP files?  Or FOO.DNG.XMP?  Or FOO.HEIC.XMP?  IOW not embedding XMP into files that have a container format explicitly defined in Adobe XMP Standard to be embedded and not a sidecar XMP (regardless of how that is named)?  Not going to happen.

And are we supposed to "clean-up" the potential existence of FOO.XMP files paired to FOO.ARW when the user "flips the preference switch"?

Have you considered the amount of customer support headaches when someone "flips the preference switch" without any full understanding of what this means, and they use Adobe software?

And what about WAV files?  Are these supposed to be FOO.JPG.WAV?  Although cameras don't write XMP sidecar files (if they did it would be FOO.XMP I can guarantee you), they do write WAV files for audio annotation.  This HAS been a problem with PM when you pair a RAW+JPEG and there is also a WAV file.  If you separate RAW & JPG they both have WAV file "attached" and so if you rename/delete etc one of these RAW or JPG files, it renames/deletes the WAV file leaving the other partner "uninformed" shall we say.  But are cameras going to write FOO.JPG.WAV and also FOO.ARW.WAV files to distinguish?  No.

So I appreciate the logic on this.  But it does have a lot of real world implications with a software like Photo Mechanic that has been around for almost 25 years.

Again, we will look into this, but as Kirk says we "have a lot of eggs to fry and fewer cooks" so we have to consider the effort involved, the disruption of existing user base, and delays on future PM features that affect the masses.  I may give this a whack and send you a private build to test (but it would be a local PMDEBUG.TXT mod no UI setting).

Regards,

--dennis

[pll]

Hi Dennis,

Thank you for your detailed reply. And before I go any further, please let me apologize again for any erroneous assumptions, wild exaggerations, or baseless claims I may have made previously. As I've stated, I don't know the code at all, and, after the exchanges with Kirk and having had some time to mull things over, it seems quite arrogant of me to assume anything at all about the complexity or ease of modification of a piece of software like Photo Mechanic which works with countless devices and operating systems currently available as well as those of the past 20+ years, never mind the countless ones constantly flooding the market. It was quite ignorant and naive of me to assume anything at all.  So again, I apologize.

But are you suggesting that we should do something completely non-standard like creating FOO.JPG.XMP files?  Or FOO.DNG.XMP?  Or FOO.HEIC.XMP?  IOW not embedding XMP into files that have a container format explicitly defined in Adobe XMP Standard to be embedded and not a sidecar XMP (regardless of how that is named)?

Not at all! I am merely asking for the ability to set the filename format of a normal XMP sidecar file. The issue at hand seems to be this:

There is merely one sentence, on Page 7 of Part 3 of the XMP standard (https://github.com/adobe/XMP-Toolkit-SDK/blob/main/docs/XMPSpecificationPart3.pdf) which dictates what the filename format of a sidecar file should be:

For applications that need to find external XMP files, look in the same directory for a file with the same name as the main document but with an .xmp extension. (This is called a sidecar XMP file.)

Apparently most of the rest of the industry has interpreted this mean that:

If the original filename is: foo.CR2
Then the sidecar filename will be: foo.xmp

Technically, for this to be true, the standard should read "...look in the same directory for a file with the same name as the main document with the original extension replaced with .xmp".

Apparently, the above sentence is ambiguous enough such that the developers of Darktable have interpreted to mean that:

If the original filename is: foo.CR2
Then the sidecar filename will be: foo.CR2.xmp

The end result is, that when I use Photo Mechanic and create an XMP sidecar file, Darktable can't find it, because it's looking for <filename base>.<original ext>.xmp and Photo Mechanic is writing out <filename>.xmp.

All I want, and what seems to be what the originator of this thread wanted, was the simple ability in the settings configure the output filename format. For users of Darktable, simply specifying a format of <filename base>.<orig ext>.xmp would be good enough.

Have you considered the amount of customer support headaches when someone "flips the preference switch" without any full understanding of what this means, and they use Adobe software?

Apparently, not at all.  I'm a UNIX guy, and we tend to "point, shoot, aim" when it comes to things like this. After shooting ourselves in the foot a few times, we learn to be paranoid and overly cautious about maintaining lots of backups of everything.  From your perspective however, now that you point it out, I can appreciate that customers complaining about this without understanding what they're doing would be nightmare to support, especially if they changed this setting and suddenly Adobe products stopped working.

One possibility could be simply having a toggle in the preferences that says "Use Darktable sidecar naming" and "Use Adobe sidecar naming".  The end use doesn't have to necessarily know what this actually means as long as it does what they expect. Maybe this kind of option goes under an Advanced section, or the ITPC/XMP section of preferences, where I'd expect anyone messing with those options to be much more aware of what they're doing than the average user.  (this is me simply thinking out loud, so please don't take it to be me presuming how you should do this).

And lastly, there was never any presumption or expectation on my part that Camerabits would just pick this up and implement immediately. I understand the real-world business requirements of supporting software systems. The pressure and expectations from 1000s of customers, business partners, and all the other stakeholders out there is all too real.  My only "ask" in my original post was simply that it be considered and brought to the product team as a request. I was, and still am, prepared for the answer to be, "Sorry, ain't gonna happen!".

In the end, I want to use Photo Mechanic as the hub of my workflow. I want to maintain my inventory and catalog there, and avoid Adobe like the plague.  All I need for that to work is for both applications to support the same naming convention for sidecar files.

Anyway, thank you for your time and consideration in this matter. I apologize again for any hard feelings my earlier presumptuous statements may have caused.

--
Sincerely,
Paul Lussier

[pll]

In the spirit of transparency and mea-culpas, I also stated:

on Page 7 of Part 3 (https://github.com/adobe/XMP-Toolkit-SDK/blob/main/docs/XMPSpecificationPart3.pdf) which states:

For applications that need to find external XMP files, look in the same directory for a file with the same name as the main document but with an .xmp extension. (This is called a sidecar XMP file.)

and:

the standard should read "...look in the same directory for a file with the same name as the main document with the original extension replaced with .xmp".

Scanning through the above mentioned document a bit further, I found this on page 27:

XMP is not directly embedded within MPEG-2 files, but is specified as a sidecar file. This is a separate file containing just the XMP packet, which is stored at the same location as the MPEG-2 file, and uses the same file name, with the file extension .xmp replacing the original file extension.

So, they have apparently clarified their initial statement much deeper in the document when discussing MPEG-2 files.  So, I take back all the stuff I said about the file naming convention/definition being ambiguous, since technically, it is specified in the spec, just not as clearly as one might prefer, or called out in a specific section on what/how to name files.

Again, my apologies.
--
Paul

[Bob M]

Against my better judgement, I would like to insert a comment.  I have read threads on https://discuss.pixls.us/latest that make it very clear how strongly the naming convention that Darktable uses is supported.

To my mind the root of the issue was Darktable's decision to include the editing data of a particular version of an image within an xmp file.  To my mind the xmp file is appropriate for meta data that describes the original image -- content, location, copyright, etc. -- that is common to any and all versions of the image.  It is distinct from editing data that refers to a particular version of the image. 

I think that RawTherapee does it correctly by storing its editing data in a pp3 file and leaves the xmp files alone.  Photo Mechanic plays very nicely with RawTherapee.  Granted, I have two sidecar files that I need to carry around with me but Photo Mechanic takes care of that for me automatically in the background.  I have never had an issue of a sidecar getting lost. My fear though is that RawTherapee will follow Darktable's lead and abandon pp3 files in favour of xmp files, perhaps in version 6.0.

My apologies for interrupting this discussion to insert my very personal opinion.