Warning: Problem with NEF and Capture NX, even IPTC!

[perw]

Kirk,

PM 4.5.1.1
Mac OS X 10.4.9, Intel and PPC
(Capture NX 1.1, Intel and PPC)

Since my last post I have had some more bad experiences with trying to embed data into NEF. I think it is bothersome that the(approximately) same edit steps on the same file have different effect for me (Swedish OS and CNX) and other users (archer69) with other OS and language.

I have also tried this:

- Embed with PM -> edit with CNX -> revert to orig (PM)

just to find that CNX still finds IPTC data in the file, although modified! Is perhaps CNX shuffling the data around so that the revert step can't find it again? (I realize that the "revert ..." step naturally can't be expected to work on a modified file, but I had to try, to try to rescue the editing work done).

if there is anything else you need to know, please ask.

It would be nice if this matter was resolved, but I also understand that the problem very well could be with CNX.

/Per
(Now exclusively using XMP sidecar files, alas)

[dennis]

Per,

Here are some known issues with Capture NX and PM.

Capture NX doesn't properly format IPTC data when it writes a new NEF or TIFF file.  The IPTC data is supposed to be formatted as an array of 4-byte values (longs) but the actual data can be 2 bytes short in some circumstances (25% of the time depending on length of overall IPTC record).  In these circumstances, PM 4.4 would (naively) assume that the NEF or TIFF file was properly formatted, and its modifications made would cause the ICC profile (or whatever data followed the IPTC record) to be changed unintentionally.  This could cause a crash in PM 4.4 or loss of ICC profile for other applications reading a TIFF.

As far as I know all of this was fixed in 4.5 (in terms of PM handling these improperly formatted NEF and TIFF files from CNX).  CNX 1.1 still exhibits the problem.

So, it may be that the files you are having problems with were originally edited in PM 4.4 (e.g. 4.4.3.3) but now you are using PM 4.5 (e.g. 4.5.1.1).

In order to reproduce this to see if there is still a problem we can fix in 4.5.2, we need to have an original NEF file straight out of the camera, and a problem file that has been edited by CNX and/or PM 4.5.1.1.  However, since PM 4.5.1.1 has the smarts to recognize these improperly formatted NEF/TIFF files from CNX, it may be difficult to recreate this problem if it is indeed a problem with PM 4.5 embedding IPTC/XMP.  If it is a problem with CNX, then you may need to try adding IPTC info of varying lengths (e.g. add a caption "1", "12", "123", and "1234" - only one of these will be improperly formatted by CNX when it is (re)saved).

I hope this helps to explain the problem.  Let me know what you find...

--dennis

[Juerg]

Dennis,

As you know I worked with the files Per sent me and could dublicate the issue.

Based on your replay (if I understand correctly), CNX will modify the IPTC data and write incorrectlly when a NEF file is saved and may cause other applications to crash. But as I understand Per and as I can actually can doublicate in my system, you can not even save the file (version) in CNX if certain conditions are present in the active file version to be saved.

Do you have access to Capture NX? I can send you some test files for your review and to try out.

Regards,

[perw]

Dennis - Kirk,

I have a set of files that clearly demonstrates the problem.

1 - file straight out of camera.
2 - file ingested with PM, sidecar file
3 - file ingested with PM, embed IPTC/XMP

Files 2 and 3 are identically edited with CNX (I edited file 3, then made a copy/paste of all edit steps). File 3 is still openable (?), but no longer saveable, not even as plain JPG. File 2 is OK, works as always.

Do you want me to send you these files?

/Per

[Kirk Baker]

Per,

I have a set of files that clearly demonstrates the problem.

1 - file straight out of camera.
2 - file ingested with PM, sidecar file
3 - file ingested with PM, embed IPTC/XMP

Files 2 and 3 are identically edited with CNX (I edited file 3, then made a copy/paste of all edit steps). File 3 is still openable (?), but no longer saveable, not even as plain JPG. File 2 is OK, works as always.

Do you want me to send you these files?

Of course.  Please contact me directly by clicking on my name to the left of this message.  Send me a personal message.  I'll reply with server upload info.

Thanks,

-Kirk

[dennis]

Per,

I took a look at the files you uploaded and examined the NEF file that had embedded IPTC and XMP.  I opened this into Capture NX 1.1 (US) and sure enough it would not save as a JPEG.  So I looked at the file in a hex editor and it looks like this file was essentially written by CNX and otherwise not touched by Photo Mechanic.  In other words, an earlier version of the NEF photo may have been captioned by PM with both IPTC and XMP, but the file with the problem was saved by CNX thereby creating a brand new file.  Its like opening a photo into Photoshop, editing it, then saving it as a new file - even though the original photo may have been captioned by PM, the file that is written is essentially a new file untouched by PM.

So just for the heck of it, I decided to mask the IPTC data by changing its TIFF tag from 0x83bb to 0x83bc and guess what!  The file could then be saved as a JPEG by CNX.  I don't know why CNX would write a file it can't deal with, but it obviously has something to do with embedded IPTC data.  I suspect you could get CNX to exhibit the same problems by adding IPTC data only in CNX and never using PM.

Anyway, I'll contact Nikon about this problem and hopefully they can find out why it fails since it is isolated to the IPTC data.  I don't think it has anything to do with PM other than PM putting the IPTC data in an earlier version of the file (if this problem happened on a PM modified NEF then I could take a look and see if anything is wrong, but I doubt this would be the real problem).

PS - You might be able to duplicate this fix by using the mask/unmask command in PM to mask the IPTC data on a file that won't save.

--dennis

[perw]

I don't know why CNX would write a file it can't deal with, but it obviously has something to do with embedded IPTC data.  I suspect you could get CNX to exhibit the same problems by adding IPTC data only in CNX and never using PM.

No, I cannot. If I take the same file (before ingesting and tagging with PM) and manually add (almost) the same IPTC data, the problem does not occur (using the same edit steps).

Conclusion: The problem within CNX is triggered by adding IPTC with PM.

Note: The "almost" earlier referrs to the fact that there is no field for country code in CNX. That field is not tagged as being an IPTC4XMP-only field. I believe a entered all the other values present.

Can it be that CNX is only able to hande IPTC fields editable using CNX, and not others, although they are "correct"?

PS - You might be able to duplicate this fix by using the mask/unmask command in PM to mask the IPTC data on a file that won't save.

--dennis

Will try that later, I have right now not access to CNX or PM.


Dennis: Files illustrating this point have been uploaded.

[Hayo Baan]

I use the "Location" field myself too, and haven't seen a problem with CNX in this respect.

By the way, CNX does treat the IPTC info strangely and when it rewrites the file, it adds lots of (empty) IPTC fields and even duplicates a few (duplicates are empty)... 

I think it leaves the XMP data intact.

I'm now at work so I can't show you what fields it adds and duplicates, I'll post this later.

[Guidigio]

Hi,


I ingest D200 NEF files with PM4.5.2 using a XMP sitecar file. When I open the file with CNX 1.1 and wanted to show the IPTC data, all fields are empty.
By 'accident' I have made a few JPG photo's and when I open those  in CNX, I can see the IPTC data filled in.
I asked Nikon support: answer since PM4 isn't Nikon software I can't say anything about this behaviour.
I use Windows XP sp2, CNX English version.
By the way, IPTC info from XMP sitecar files, made with IdimagerPro 3.5 are also not shown in CNX.
Do I something not correct or is CNX behaving strange in this matter?   


regards,
Guido

[Kirk Baker]

Guido,

I ingest D200 NEF files with PM4.5.2 using a XMP sitecar file. When I open the file with CNX 1.1 and wanted to show the IPTC data, all fields are empty.
By 'accident' I have made a few JPG photo's and when I open those  in CNX, I can see the IPTC data filled in.
I asked Nikon support: answer since PM4 isn't Nikon software I can't say anything about this behaviour.
I use Windows XP sp2, CNX English version.
By the way, IPTC info from XMP sitecar files, made with IdimagerPro 3.5 are also not shown in CNX.
Do I something not correct or is CNX behaving strange in this matter?

Capture NX completely ignores XMP data in sidecar files.  It only can display (and update) IPTC data embedded in JPEGs, TIFFs, and NEFs.

-Kirk

[Hayo Baan]

I ingest D200 NEF files with PM4.5.2 using a XMP sitecar file. When I open the file with CNX 1.1 and wanted to show the IPTC data, all fields are empty.

This is correct behaviour; Capture ignores the XMP (sidecar) data and only reads the embedded IPTC data.

By 'accident' I have made a few JPG photo's and when I open those  in CNX, I can see the IPTC data filled in.
I asked Nikon support: answer since PM4 isn't Nikon software I can't say anything about this behaviour.
I use Windows XP sp2, CNX English version.
By the way, IPTC info from XMP sitecar files, made with IdimagerPro 3.5 are also not shown in CNX.
Do I something not correct or is CNX behaving strange in this matter?   

Nope, everything is behaving as designed ;-)

What you can do is tell PM to always update the embedded IPTC and XMP data and synchronise the data.  You can synchronise the xmp/iptc using the "Update IPTC/XMP" action in the Tools menu of PM.
Note: make sure you have PM read the XMP sidecar file as first in the preferences...

[dennis]

Conclusion: The problem within CNX is triggered by adding IPTC with PM.

Per,

No the problem is CNX handling legitimate IPTC data that just happened to be originally created by Photo Mechanic.  It could be another program inserting IPTC data.

First let me reiterate that this file was written by CNX, and if CNX cannot handle a file it created then that is a problem with CNX.

Second, the IPTC record is clearly formatted by CNX.  For some reason Nikon likes to write out a bunch of IPTC fields (e.g. several record 1 and 2 fields) with no values (len = 0).  Very strange and basically a waste of space.  Nikon also likes to put all the IPTC fields in order, something that isn't necessary according to the IPTC spec (other than all the record 1 fields need to come before record 2 fields).  I noticed that the (sub)location field, however, was not in order.  So I tried moving it so that it would be in order like the other fields but that didn't help.

So then I just tried removing the (sub)location field from the IPTC record (which has to be done very carefully) and this fixed the problem so the file could then be saved by CNX.  Interestingly, this field was the only field with non 7-bit ASCII: "Idtjärnsgatan 16".  So next I tried leaving the (sub)location field in but changing the 'ä' to 'a', and this also fixed the problem.  So, apparently CNX doesn't like high bit ASCII or is having an error trying to parse / re-encode this string.

I will post the modified file for you with just that one character changed and you can see for yourself.

I suppose you could argue that PM put this string into the IPTC so it is PM's fault, but that would be a stretch since it is a legitimate string (although the character encoding for IPTC data is a bit murky other than Unicode).  It would be interesting to see what happens when you have PM format the IPTC as Unicode (UTF-8).  Maybe this would work with CNX, but since CNX doesn't write out the record 1 field for "coded character set" (which is what identifies the text as UTF-8) I suspect it wouldn't help (not even Photoshop can handle UTF-8 IPTC unfortunately).

So for now, if you avoid the high-bit ASCII then maybe CNX will behave.

--dennis

[Hayo Baan]

Dennis, excellent problem analysis!  I have written a little program to analyse EXIF data myself and if I dump the data of a file written by Capture NX, these are the fields it produces.  Notice that it inserts empty fields and also notice the duplicate entries!

IPTCCountryCode                 NLD
IPTCCountry                     The Netherlands
IPTCTransmissionReference
IPTCHeadline
IPTCCredit
IPTCSource
IPTCCopyrightNotice             Copyright 2004, Hayo Baan
IPTCCaptionAbstract             Greebo
IPTCWriterEditor
IPTCLanguageIdentifier
IPTCCatagory
IPTCSupplementalCategories
IPTCFixtureIdentifier
IPTCPMPrefs                     Tagged=No, ColorClass=0, Rating=1, FrameNumber=-1
IPTCKeywords                    Greebo
IPTCReleaseDate
IPTCReleaseDate
IPTCReleaseTime
IPTCExpirationDate
IPTCExpirationTime
IPTCInstructions
IPTCInstructions
IPTCObjectName
IPTCObjectName
IPTCReferenceNumber
IPTCDateCreated                 20040508
IPTCTimeCreated                 232827+0100
IPTCTimeCreated
IPTCOriginatingProgram
IPTCProgramVersion
IPTCProgramVersion
IPTCAuthor                      Hayo Baan
IPTCAuthor
IPTCAuthorPosition
IPTCCity                        Kockengen
IPTCLocation                    Home
IPTCProvinceState               Utrecht

I'm thinking maybe the duplicate fields are caused by capture wanting to keep the numerical order?

[dennis]

Per,

More info: I tried opening the "fixed" NEF photo (with 'ä' changed to 'a') into CNX, then I added the same (sub)location field you had "Idtjärnsgatan 16" into the Caption field (by pasting), then "Apply" to save the IPTC, and for some reason this could be subsequently saved as a JPEG.  I examined the JPEG and it had the same formatting of "Idtjärnsgatan 16" in the IPTC caption field as was originally in the (sub)location field, but CNX wrote the IPTC without the (sub)location or other fields it doesn't recognize (such as our prefs).  So apparently CNX can deal with high-bit ASCII as long as it is a field it knows about.  Very strange.

I tried another experiment to hack your file by moving "Idtjärnsgatan 16" into the IPTC Caption field, leaving the (sub)location field empty, and this also fixes the problem.

So, DON'T put high-bit ASCII in the (sub)location field (and maybe other fields CNX doesn't recognize)!!!

Another workaround would be to simply open the IPTC pane in CNX, add something like a space in the Caption field (so that the "Apply" button is enabled), then Apply, then do your Save.  Unfortunately, if you do this you will lose all IPTC fields that CNX doesn't recognize such as (sub)location, but at least you will be able to Save.

--dennis

[dennis]

Notice that it inserts empty fields and also notice the duplicate entries!
...
I'm thinking maybe the duplicate fields are caused by capture wanting to keep the numerical order?

Hayo,

When I examined the actual IPTC binary data I didn't see any duplications like you mentioned.  Of course some fields are allowed to be repeated (e.g. keyword, caption writer, etc), but if non-repeatable fields are really being written twice (or more) this is a no-no (especially when the first occurence is valid and the second occurence is blank as with the time created in your example).

There is no reason to put blank fields in the actual IPTC record, and there is no reason to keep them in order (as you must do with TIFF tags - another idiocy).  One could argue that these are "place holders", but that doesn't make any sense either.  It is just dumb, especially when you consider that many of the fields Nikon inserts don't even appear in their own dialog (or have I actually encountered photos with many of these fields present).

It is equally disturbing that CNX will not preserve fields it doesn't recognize if you edit the IPTC in CNX.  Bad meta-data etiquette!

PS - I'm looking at a NEF file written by CNX, not a JPEG (and the data better not be in the Exif block!).

--dennis