New PM structured keywords not showing within Lightroom keyword list heirarchy

[Graham1]

I have an extensive library of hierarchical keywords in Lightroom.  I have exported my keywords from Lightroom to use with the PM structured keywords panel.  This all works as it should.

If, however, I add structured keywords via the PM structured keywords panel, not at the top level of the hierarchy (new child, not new parent), apply the new keywords to images in PM, and then update the images' metadata in Lightroom, the new keywords appear at the bottom of the Lightroom keyword list, as top level (or flat) keywords, rather than in the location/level within the hierarchy in which they were created in PM.

There are some old threads about incompatible Lightroom XMP standards for hierarchical keyword storage, which I only vaguely understand, but it seems (or seemed) to be the case that because of this, PM can read and understand the hierarchy created from keywording carried out in Lightroom, but that it does not work the other way round, namely being able to create and insert new keywords within PM at a level below top level in the hierarchy which will be recognised by Lightroom at that level when Lightroom reads the updated metadata.

Is this correct, or is there now any way of creating a lower level (child) keyword in PM which will show up at the same level once the metadata is updated in Lightroom? If this is not presently achievable, is this something that may be expected to change in version 6 of PM when it is released?

I prefer PM's way of dealing with keywords, particularly the ability to edit out some of the less necessary keywords/synonyms between the top and bottom levels of a multi-level hierarchy (which Lightroom cannot do on an image by image basis), but because I have so many images and such a large vocabulary of keywords, I need to be able to create/modify keywords at any level in either application interchangeably, with the same result on the overall hierarchy seen by the other application when it updates, whichever application I use to do the editing first.  Is this possible, or likely to become possible?

Graham

[Bob Russell]

Graham, do you have the Separator box checked at the bottom left of Photo Mechanic's Structured Keyword Panel?  If your box is checked, can you send a sample file with this problem and let me know what version of Lightroom you are using?

--Bob

[Graham1]

Bob, thank you for your reply.

I did not have the separator box checked.  I tried checking it, and then adding the new keywords.  When this happens, the new keywords do not show up in Lightroom at all, although other metadata such as the caption, colour and star rating are all changed as I make changes in PM and then update LR.  Uncheck the box, repeat, and again the new keywords added in PM are added as top level (flat) keywords in LR.

I also tried adding an existing keyword (i.e. existing within the Lightroom hierarchy) with the box checked and unchecked: in neither case was this keyword recognised as being applicable in Lightroom.  I am using the latest Lightroom Classic CC version.

I tried deleting all keywords from my test image and then adding new keywords with the separator box checked.  According to PM they exist, but the updated LR metadata shows no keywords at all.

I looked at the .XMP file, which showed no keywords (after the test in the paragraph above).  Therefore, I did the following.  (1) Opened the test file in PM, confirming there were no keywords.  (2) Checked the separator box.  (3) Added a few keywords using the structured keywords panel, some already in LR, some new to LR.  (4) Opened the .XMP file to verify that all the new keywords were there (they were).  (5) Opened LR and updated metadata, LR recognising that it had been changed externally (using Import Settings from Disk).  LR does not see any of the keywords just added by PM.  (6) Open the .XMP file again: the newly added keywords have disappeared.

Having gone through these steps, it seems to me that the problem lies with LR, not PM.  PM saves the new keywords into the .XMP sidecar file, then LR strips the added keyword metadata out again, even though LR is instructed to import settings from disk.

If I am right in this conclusion, I guess there is not a lot that Camerabits can do about this.  I will when I have a little more time search to see if there are other known instances of LR stripping out .XMP data even though it has been told to import the changes from disk.

Graham

[Graham1]

I have done a bit more research on this.

I made more or less the same changes to keywords in PM and in LR, then compared the .XMP sidecar files.

Make the changes in PM, and the new keywords appear only at the end of the file, within the tags starting: <lr:hierarchicalSubject>
    <rdf:Bag>

Make the changes in LR, and changes appear also within the earlier tags: <dc:subject>
    <rdf:Bag>

There are also a few changes in order of entries within earlier tags, but I assume these are immaterial.

It looks to me (as a non-programmer!) as if LR puts in extra entries to identify the hierarchy.  Presumably when it re-reads the altered metadata, if the earlier tags are absent, it deletes the later tags which make no sense to LR on their own.

To compare:

This is the relevant metadata as altered by PM:

<lr:hierarchicalSubject>
    <rdf:Bag>
     <rdf:li>[005_OBJECTS and THINGS_]|transport|aviation|airline|EasyJet</rdf:li>
     <rdf:li>aviation|airline|operator|operator|budget airline|cheap carrier</rdf:li>
     <rdf:li>aviation|airline|operator|operator|budget airline|cheap carrier|cheap carrier</rdf:li>
     <rdf:li>aviation|aircraft|aeroplane|airplane</rdf:li>
     <rdf:li>aviation|aircraft|aeroplane|airplane|aeroplane</rdf:li>
     <rdf:li>aviation|aircraft|aeroplane|airplane|airplane</rdf:li>
     <rdf:li>aviation|aircraft|aeroplane|airplane|jet</rdf:li>
    </rdf:Bag>
   </lr:hierarchicalSubject>

And this is the metadata for broadly similar keywords added in LR:

<dc:subject>
    <rdf:Bag>
     <rdf:li>EasyJet</rdf:li>
     <rdf:li>[005_OBJECTS and THINGS_]</rdf:li>
     <rdf:li>airlineaircraft</rdf:li>
     <rdf:li>airline</rdf:li>
     <rdf:li>aviation</rdf:li>
     <rdf:li>budget airline</rdf:li>
     <rdf:li>jet</rdf:li>
     <rdf:li>operator</rdf:li>
     <rdf:li>transport</rdf:li>
    </rdf:Bag>
   </dc:subject>

followed by:

<lr:hierarchicalSubject>
    <rdf:Bag>
     <rdf:li>[005_OBJECTS and THINGS_]|transport|aviation|aircraft|jet</rdf:li>
     <rdf:li>[005_OBJECTS and THINGS_]|transport|aviation|airline|EasyJet</rdf:li>
     <rdf:li>[005_OBJECTS and THINGS_]|transport|aviation|airline|operator</rdf:li>
     <rdf:li>[005_OBJECTS and THINGS_]|transport|aviation|airline|operator|budget airline</rdf:li>
    </rdf:Bag>
   </lr:hierarchicalSubject>

I hope this means enough to someone to understand what is going on, and hopefully to explain why changes made in PM are not being reflected in keywords visible within LR.  If this makes no sense, or is completely on the wrong track, please ignore!  That's all I plan to do for the time being, now I am going off to get on with my Christmas!  Merry Xmas to all.

Graham

[Hayo Baan]

First of all, LR adds all components of the hierarchical keywords to dc subject so in effect they are now also stand-alone keywords. If I'm not mistaken you can have PM do the same.
Then it looks like LR is simply cleaning up duplicates and adding the full path. But the data you show can not be from the same metadata set or otherwise LR would remove stuff, and I think it doesn't do that.

Does this help?

[Graham1]

Hayo, thank you for your reply.

You say that LR does not remove stuff, but that is demonstrably not the case.

I exported my test image from LR with the single lowest level keyword (in a hierarchy) of "easyjet".  I then opened the image in PM, and added the keyword "budget airline" within the same hierarchy, and saved the resulting .XMP file.  I then re-opened LR and updated the metadata for this image, and again saved the .XMP file.  I then compared the two .XMP files, as the image left PM compared to how it ended up read by LR.  Here is the relevant part of the changed .XMP file:

<lr:hierarchicalSubject>
    <rdf:Bag>
<rdf:li>aviation|budget airline</rdf:li>
     <rdf:li>[005_OBJECTS and THINGS_]|transport|aviation|airline|EasyJet</rdf:li>
    </rdf:Bag>
   </lr:hierarchicalSubject>

As you can see, LR has, without any warning, deleted the new "budget airline" keyword.  I cannot see any setting in PM which would alter this behaviour by LR.

Earlier in the .XMP file, LR had moved the order of the "easyjet" keyword and the head containing keyword of the hierarchy within which "easyjet" sits in LR, as below, although I do not think this would make any difference.

<dc:subject>
    <rdf:Bag>
     <rdf:li>EasyJet</rdf:li>
     <rdf:li>[005_OBJECTS and THINGS_]</rdf:li>
     <rdf:li>airline</rdf:li>
     <rdf:li>aviation</rdf:li>
     <rdf:li>EasyJet</rdf:li>
     <rdf:li>transport</rdf:li>
     <rdf:li>[005_OBJECTS and THINGS_]</rdf:li>
    </rdf:Bag>
   </dc:subject>

Does this help?

Graham

[Hayo Baan]

Hi Graham,

In LR, did you explicitly tell LR to update the metadata from the file? If not, than yes, you will lose information since then LR uses the metadata in its own database, not the data from the file. I wish you could tell LR to do this automatically, but I haven't yet found a way to do so. Since I only use LR for specific things (most of my editing is done with Camera Raw/Photoshop), I always run synchronise folders before doing anything. That makes sure all metadata is updated with the info in the files. You can also do that on individual files, e.g. right click, metadata, update.

Note: of course, I also have LR always write its updates to the file too, but that you already have set as you see the LR changes in PM.

[Graham1]

Hi, Hayo

Yes, LR shows a small icon at the top right of the thumbnail to indicate metadata has been changed.  It gives two options, either to import settings from disk (i.e. accept the external change), or to overwrite (i.e. use the metadata already in LR's database).  I have always, for these tests, chosen the former (import from disk), so LR should update using the metadata from the external file; but it does not in relation to hierarchical keywords added by PM, giving the results I have reported above.

You are correct that I have set LR to write its updates to file. I use .XMP sidecar files for everything. They update as I go along, so that nothing gets missed (and LR will warn about unsaved metadata if I try to close before it has finished writing metadata to file).  PM sees all the changes made and saved in LR; LR sees some of the changes made by PM (such as changes to the caption, label colours and ratings) when LR updates its metadata from file, but not these additional hierarchical keywords.

It would be impractical for me to keep on synchronising folders as I go along, unless perhaps on a folder-by-folder basis, since my LR database covers approaching 500,000 images. I do, however, have a smart collection which flags images with changed metadata, so I can usually pick up changes this way if I need to.

Graham

[Hayo Baan]

Strange, I just tried to reproduce the behaviour you are seeing and I can't; for me it looks like LR is doing the correct things.

Here's what I did:
1. Apply structured keyword (animals | laboratory animal | test animal) to a test image (raw with xmp) in PM
2. Open LR, apply changes from disk, verify keywords set (test animal < laboratory animal)
3. Add keyword testing 123 in LR
4. Quit LR, verify keywords in PM (animals, laboratory animal, test animal, testing 123, animals | laboratory animal | test animal)
5. Check xmp: matches keywords shown in PM – and are as expected
6. Add structured keyword (animals | pets | cat | kitten) in PM
7. Keywords in PM now: animals, laboratory animal, test animal, testing 123, animals | laboratory animal | test animal, animals | pets | cat | kitten
8. Open LR, apply changes from disk, verify keywords set (kitten, test animal < laboratory animal, testing 123)
9. Add keyword testing 345 in LR
10. Quit LR, verify keywords in PM (animals, cat, kitten, laboratory animal, pets, test animal, testing 123, testing 345, animals | laboratory animal | test animal, animals | pets | cat | kitten)
11. Check xmp: matches keywords shown in PM – and are as expected

So, I can't reproduce your problem... Could you provide a step by step similar to mine that reproduces the issue you're having?

[Graham1]

Thanks, Hayo

I will try on another computer when I am back home.  Given the steps you have outlined, it seems that I should next try to eliminate the possibility of some kind of system error on the computer I have been using to date. It may be a day or two before I get the chance to do this.

Thanks for your interest and efforts.

Graham

[msbc]

Hi,

Interesting topic - prompted me to do some testing. I used latest PM and LR Classic together with exiftool to test metadata in the XMP sidecar. I found:
PM will write structured keywords to lr:hierarchicalSubject
LR will write structured keywords to lr:hierarchicalSubject AND keywords to dc.subject

If you try to alter the structured keywords in PM after LR then the dc.subject and lr:hierarchicalSubject will be "out of sync". I am seeing that LR deletes all keywords when you import metadata under these circumstances.

So, it seems you can start keywording in PM and update the keywords in LR but you can't then return to PM to alter the keywords.

[Hayo Baan]

Interesting, but that’s not what I saw happening; all keywords stayed in place. Even after updating in PM (see e.g. step 6 in my example). What version of LR are you using?  Mine is the latest LR CC classic.

[msbc]

Current versions of all software.

Try this:
1. Open image in PM and add structured keyword: a | b | c
2. Import image to LR
3. Add keyword hierarchy: d | e | f
4. View Info Panel in PM to see all keywords and you should see: a | b | c, d | e | f, d, e, f

[Hayo Baan]

Isn’t that what you are getting? (Actually, I think you should also see a, b, c). Note that I hadn’t tried to add a structured keyword myself, but I don’t think that should matter.

[Graham1]

Note that I hadn’t tried to add a structured keyword myself, but I don’t think that should matter.

I have been carrying out lots of tests, with and without the "Use separator between keywords in the path" ticked.  I am getting inconsistent results - operator error, I am sure, but I am presently not sure what that is.  I need to do more tests when I have time, to determine exactly when it does work, and when it does not.  I repeat what I think is the same test: sometimes no new keywords in LR, sometimes all is as it should be and sometimes the new keywords appear but only as top level keywords, not in the hierarchy.  I think that a proper understanding of what is going one does require testing by adding structured keywords, however.

Graham