Multiple XMP Segments

[afengler]

In my workflow I preview all new images with PM, select the 'keepers' and rate them using the star system. I ran into problems transferring those ratings to my image management software (IMatch) with some older JPG files. After looking at these files with a hex editor I found out that they contain two (2) XMP segments, each beginning with an APP1 marker. PM adds the rating, tags, etc. to the second XMP block, whereas IMatch checks only the first XMP segment it encounters.
My newer images (starting about 2007?) contain only one (1) XMP segment, so the synchronization with IMatch is not a problem.

So there are two querstions:
1. Did PM change the way embedding rating, tags, etc. and other XMP data or did the camera manufacturer (Pentax) do that? I seem to recall a camera software update some time ago.
2. Can I somehow use PM to correct the older files or do I need to fiddle with 'exiftool' to transfer the PM XMP data to the first XMP segment?

Alf

[Kirk Baker]

Alf,

In my workflow I preview all new images with PM, select the 'keepers' and rate them using the star system. I ran into problems transferring those ratings to my image management software (IMatch) with some older JPG files. After looking at these files with a hex editor I found out that they contain two (2) XMP segments, each beginning with an APP1 marker. PM adds the rating, tags, etc. to the second XMP block, whereas IMatch checks only the first XMP segment it encounters.
My newer images (starting about 2007?) contain only one (1) XMP segment, so the synchronization with IMatch is not a problem.

So there are two querstions:
1. Did PM change the way embedding rating, tags, etc. and other XMP data or did the camera manufacturer (Pentax) do that? I seem to recall a camera software update some time ago.

It would be difficult to figure out what entity would have been responsible, though I doubt it very much that Photo Mechanic had written a second XMP chunk into your JPEGs.

2. Can I somehow use PM to correct the older files or do I need to fiddle with 'exiftool' to transfer the PM XMP data to the first XMP segment?

I think you'll have to use a different utility (like ExifTool) to remove the second XMP chunk and replace the first XMP chunk with the second chunk.

It is not uncommon to see multiple XMP chunks in a JPEG (XMP is fairly verbose and JPEGs have a chunk size limit just under 64KB, so really large XMP data need to be spread over two or more chunks.)

-Kirk

[afengler]

Kirk,
I just sent the following information to the IMatch forum where I also was looking for advice on this issue.

In case other users run into similar problems when using both PhotoMechanic and IMatch, the following notes might be useful.

1. Both of my cameras (Pentax Optio 43WR and Nikon D80) create JPEG files with the standard EXIF APP1 segment plus two (2) XMP APP1 segments, where certain but not all of the metadata are duplicated.

2. PhotoMechanic adds its tags, ratings, etc, to the second (last?) XMP segment.

3. IMatch looks only at the first XMP segment it encounters.

4. Therefore, synchronization between these applications does not work correctly.

5. When 'ingesting' using Breezesystem's DownLoader Pro and adding some IPTC data, the complete XMP section is rewritten, resulting in a single (combined) XMP APP1 segment.

My Solution:
- Use DownLoader Pro to 'ingest' all new images.
- Use exiftool to convert all original two-segment JPEGS into a standard form with one (1) XMP APP1 segment. PM's tags, ratings, etc. are then automatically transferred and recognized by IMatch. I use the command

exiftool -m -xmp:all= -tagsfromfile @ -xmp:all *.jpg

to correct my old files.

Note:
I can recommend two free software tools that really helped getting to the bottom of this:
1. Cygnus Hex Editor - Free Edition - to quickly find the APP1 segments (look for hex: FF E1)
2. exiftool, a terrific piece of software, especially after you work your way through the documentation and the dozens of options. I especially recommend to run

exiftool -HTMLDump FILE >FILE.html

to take a quick look at your JPEGs to see whether your camera creates more or less "standard" XMP data blocks.

[afengler]

Please note the following correction to my post:
As Phil Harvey (creator of ExifTool) correctly pointed out to me, the cameras do not add the XMP segments to the JPEG image file. It must happen in a further processing step, perhaps during the download or the initial review phase. I will pursue this problem somewhat on the backburner, since I was able to correct all my old images using 'exiftool' in a batchfile and all new images are being handled correctly using Downloader Pro 2.2.2.

[Kirk Baker]

Please note the following correction to my post:
As Phil Harvey (creator of ExifTool) correctly pointed out to me, the cameras do not add the XMP segments to the JPEG image file. It must happen in a further processing step, perhaps during the download or the initial review phase. I will pursue this problem somewhat on the backburner, since I was able to correct all my old images using 'exiftool' in a batchfile and all new images are being handled correctly using Downloader Pro 2.2.2.

The only software that I know that would create multiple XMP segments in a JPEG would be Adobe's software.  Generally it happens when a RAW file is adjusted by ACR/Bridge/Lightroom which generates a large amount of RAW adjustment settings.  These settings are so large that when represented in XMP they exceed the 64KB JPEG chunk size limit and have to be broken up into multiple segments.  Why these settings need to be represented in a non-RAW file type is not understood by us.  They have no relevance to a JPEG whatsoever and should be omitted.  When PM saves a RAW file as a JPEG and maintains the XMP data it does indeed omit this additional information and thus the newly created JPEGs won't have multiple XMP segments.  However, if PM does encounter a JPEG with multiple XMP segments it does not remove any data, it only adds its data to the last segment since that segment is the only segment that has room for additional information.

IMatch needs to be modified to read all segments as if they are one really large segment or it will run into other issues.  It shouldn't be too hard to do.

-Kirk

[afengler]

Kirk,
I am not using any Adobe software and all my images are JPEGs. Only Downloader Pro, Breeze Browser Pro, PhotoMechanic and IMatch are touching my images. While going through my older pictures I found JPEGs on my system that contained
1 APP1 EXIF segment
4 APP1 XMP segments (!)
1 APP13 Photoshop segment.
None of the XMP segments are especially large, all about 8 kBytes or so, so definitely no requirement to split XMP into Standard and Extended XMP.

I had used PM to rate these images from 2007 but had always had problems with IMatch not recognizing the ratings - I can see the difficulty now.
ExifTool corrects the problem of course by deleting all XMP data and then rewriting a clean new XMP segment.
Unless another PM user experiences similar multiplying XMP segments, I think I will fix the remaining images with ExifTool and hope for the best!

[Kirk Baker]

Alf,

I am not using any Adobe software and all my images are JPEGs. Only Downloader Pro, Breeze Browser Pro, PhotoMechanic and IMatch are touching my images. While going through my older pictures I found JPEGs on my system that contained
1 APP1 EXIF segment
4 APP1 XMP segments (!)
1 APP13 Photoshop segment.
None of the XMP segments are especially large, all about 8 kBytes or so, so definitely no requirement to split XMP into Standard and Extended XMP.

I had used PM to rate these images from 2007 but had always had problems with IMatch not recognizing the ratings - I can see the difficulty now.
ExifTool corrects the problem of course by deleting all XMP data and then rewriting a clean new XMP segment.
Unless another PM user experiences similar multiplying XMP segments, I think I will fix the remaining images with ExifTool and hope for the best!

That would be really odd to have more than two of them, especially when all were not near the 64KB limit.  It would be good to find out which application is adding all of those XMP chunks.  Keep an eye on it.  Since it only happened to a small number of images, it may have been a particular version of one of the applications you listed that had a bug which was later corrected.  You may not be able to get this issue to happen anymore.

-Kirk

[afengler]

Kirk,
Yes, I will keep an eye out for future anomalies. While looking at the structure of some JPEGs with ExifTool I noticed that they contained a block of 2,060 bytes named 'PhotoMechanic Trailer" by ExifTool. Interestingly they were at the end of the file, even past the EOI marker (hex: FF D9). Did PM write this at this location or is that an ExifTool problem? It started out with (I guess) 2,048 bytes of binary data, then a 4 byte length (?) field containing the number 8 and then 8 bytes 'cbipcbbl' (ASCII).

[Kirk Baker]

Alf,

Yes, I will keep an eye out for future anomalies. While looking at the structure of some JPEGs with ExifTool I noticed that they contained a block of 2,060 bytes named 'PhotoMechanic Trailer" by ExifTool. Interestingly they were at the end of the file, even past the EOI marker (hex: FF D9). Did PM write this at this location or is that an ExifTool problem? It started out with (I guess) 2,048 bytes of binary data, then a 4 byte length (?) field containing the number 8 and then 8 bytes 'cbipcbbl' (ASCII).

It is not an error.  Photo Mechanic writes its preferences to the end of file on most image types.

-Kirk