Warning: Problem with NEF and Capture NX, even IPTC!

[perw]

Conclusion: The problem within CNX is triggered by adding IPTC with PM.

Per,

No the problem is CNX handling legitimate IPTC data that just happened to be originally created by Photo Mechanic.  It could be another program inserting IPTC data.

First let me reiterate that this file was written by CNX, and if CNX cannot handle a file it created then that is a problem with CNX.

My pardon if I have not been clear in my writing, this is not my first language. I do realize that the problem is within CNX, when it can't save a file it without complaint has written earlier. I have only stated that PM is triggering this bug, not that there necessarily is any error in the way PM works.

So, DON'T put high-bit ASCII in the (sub)location field (and maybe other fields CNX doesn't recognize)!!!

Brilliant!!

Dennis, we now have a description of what (sometimes, not always) makes CNX trip up. Putting 8-bit ASCII in fields CNX does not recognize is not recommended as of now!

For me, the outcome of this is an easy decision: Keep all IPTC/IPTC4XMP data in the sidecar file until this bug is fixed in CNX, then I can batch-import it all again sometime in the future.

The other thing that ought to be done is make sure Nikon knows of this so they can fix it. I am very new to the Nikon world, and have no idea how they respond to reports of this kind. If they are block-headed, they will just claim: "If you use CNX there is no problem, and we don't publish the internal format of NEF-files, and if third-party applications change a NEF-file, it is their problem".

My view, and yours as I can understand, is that NEF-files are based on public standards, and should follow them!

Dennis or somebody else: If you have a good way of making Nikon listen, please try to do so.

I will try to make this a case here in Sweden.

/Per

[Hayo Baan]

Notice that it inserts empty fields and also notice the duplicate entries!
...
I'm thinking maybe the duplicate fields are caused by capture wanting to keep the numerical order?

Hayo,

When I examined the actual IPTC binary data I didn't see any duplications like you mentioned.  Of course some fields are allowed to be repeated (e.g. keyword, caption writer, etc), but if non-repeatable fields are really being written twice (or more) this is a no-no (especially when the first occurence is valid and the second occurence is blank as with the time created in your example).

Hmm, I will double check the code I wrote to determine the IPTC data.  I'll get back to this when I have found some time...

There is no reason to put blank fields in the actual IPTC record, and there is no reason to keep them in order (as you must do with TIFF tags - another idiocy).  One could argue that these are "place holders", but that doesn't make any sense either.  It is just dumb, especially when you consider that many of the fields Nikon inserts don't even appear in their own dialog (or have I actually encountered photos with many of these fields present).

It is equally disturbing that CNX will not preserve fields it doesn't recognize if you edit the IPTC in CNX.  Bad meta-data etiquette!

I hadn't noticed this yet (don't use CNX for editing IPTC though), only that it translates '/' into '\' in most fields (very annoying).

PS - I'm looking at a NEF file written by CNX, not a JPEG (and the data better not be in the Exif block!).

--dennis

That's what I was doing too.  Anyway, I'll do some more research on the duplicate fields (I'm now thinking my code finds another block with IPTC data, written by CNX, which is then merged with the proper block).

Will keep you posted,
    Hayo

[perw]

Anyway, I'll do some more research on the duplicate fields (I'm now thinking my code finds another block with IPTC data, written by CNX, which is then merged with the proper block).

I have seen cases where some IPTC fields were corrupted when viewed using CNX, but that was with a file already "destroyed" by CNX, as it was no longer saveable.

Remember though, when you are looking at one of the problematic files, that it is corrupted by CNX! Don't assume it follows the ordinary rules for how fields are subbosed to be written.

/Per

[Hayo Baan]

Anyway, I'll do some more research on the duplicate fields (I'm now thinking my code finds another block with IPTC data, written by CNX, which is then merged with the proper block).

I have seen cases where some IPTC fields were corrupted when viewed using CNX, but that was with a file already "destroyed" by CNX, as it was no longer saveable.

Remember though, when you are looking at one of the problematic files, that it is corrupted by CNX! Don't assume it follows the ordinary rules for how fields are subbosed to be written.

Per, I only have my own (non-corrupt) files at my disposal so I will concentrate on those.
Please note that I don't work for Camerabits and that I'm just a user, like you :-)
What I'm trying to do here is being helpful as well as satisfy my own curiosity...

[Hayo Baan]

Dennis, I've added some debugging to my little EXIF script, and Capture NX is definitely added duplicate fields.  The duplicate fields seem to have been added in a second block within the IPTC data and it could therefore be that I should ignore the first block completely...  Here's an example of the debug output from a file rewritten by Capture.
I've marked the duplicate fields in the first block in red.  As you can see not all fields are duplicates in the first block...

Does this help to determine what Capture NX is doing?

Hex Offset to IPTC data within File (pointed to by the IPTC tag, nr 0x83BB): 0x1A74
Hex representation of the IPTC data (324 bytes): 0x1C0100000200041C010500001C0114000200001C0116000200001C011E00001C012800001C013200001C013C0001001C014600001C015000001C0200000200041C020500001C020F00001C0219000E547265652028506C616E746165291C021E00001C022300001C022500001C022600001C022800001C0237000832303035303433301C023C000B3135303334312B303130301C024100001C024600001C025000094861796F204261616E1C025500001C025A000D4C6167652056757572736368651C025F0007557472656368741C026400034E4C441C0265000F546865204E65746865726C616E64731C026700001C026900001C026E00001C027300001C02740019436F7079726967687420323030352C204861796F204261616E1C02780006466F726573741C027A00001C028700001C02DD000C303A303A313A2D30303030310000

Interpretation of the IPTC Data:
TAG: 0(IPTCBlockMarker) LEN=2 VAL= 
TAG: 5(IPTCObjectName) LEN=0 VAL=
TAG: 20(IPTCSupplementalCategories) LEN=2 VAL= 
TAG: 22(IPTCFixtureIdentifier) LEN=2 VAL= 
TAG: 30(IPTCReleaseDate) LEN=0 VAL=
TAG: 40(IPTCInstructions) LEN=0 VAL=
TAG: 50(IPTCReferenceNumber) LEN=0 VAL=
TAG: 60(IPTCTimeCreated) LEN=1 VAL=
TAG: 70(IPTCProgramVersion) LEN=0 VAL=
TAG: 80(IPTCAuthor) LEN=0 VAL=
TAG: 0(IPTCBlockMarker) LEN=2 VAL= 
TAG: 5(IPTCObjectName) LEN=0 VAL=
TAG: 15(IPTCCatagory) LEN=0 VAL=
TAG: 25(IPTCKeywords) LEN=14 VAL=Tree (Plantae)
TAG: 30(IPTCReleaseDate) LEN=0 VAL=
TAG: 35(IPTCReleaseTime) LEN=0 VAL=
TAG: 37(IPTCExpirationDate) LEN=0 VAL=
TAG: 38(IPTCExpirationTime) LEN=0 VAL=
TAG: 40(IPTCInstructions) LEN=0 VAL=
TAG: 55(IPTCDateCreated) LEN=8 VAL=20050430
TAG: 60(IPTCTimeCreated) LEN=11 VAL=150341+0100
TAG: 65(IPTCOriginatingProgram) LEN=0 VAL=
TAG: 70(IPTCProgramVersion) LEN=0 VAL=
TAG: 80(IPTCAuthor) LEN=9 VAL=Hayo Baan
TAG: 85(IPTCAuthorPosition) LEN=0 VAL=
TAG: 90(IPTCCity) LEN=13 VAL=Lage Vuursche
TAG: 95(IPTCProvinceState) LEN=7 VAL=Utrecht
TAG: 100(IPTCCountryCode) LEN=3 VAL=NLD
TAG: 101(IPTCCountry) LEN=15 VAL=The Netherlands
TAG: 103(IPTCTransmissionReference) LEN=0 VAL=
TAG: 105(IPTCHeadline) LEN=0 VAL=
TAG: 110(IPTCCredit) LEN=0 VAL=
TAG: 115(IPTCSource) LEN=0 VAL=
TAG: 116(IPTCCopyrightNotice) LEN=25 VAL=Copyright 2005, Hayo Baan
TAG: 120(IPTCCaptionAbstract) LEN=6 VAL=Forest
TAG: 122(IPTCWriterEditor) LEN=0 VAL=
TAG: 135(IPTCLanguageIdentifier) LEN=0 VAL=
TAG: 221(IPTCPMPrefs) LEN=12 VAL=0:0:1:-00001
TAG: 0(IPTCBlockMarker) LEN=0 VAL=

[dennis]

Dennis, I've added some debugging to my little EXIF script, and Capture NX is definitely added duplicate fields.

Well not quite...

The first block has record 1 fields and the second block has record 2 fields.  They are different from each other (IPTC also has other records up to record 9).  The byte after the 0x1c is the record number, and the next byte is the field in that record.

Here are the record 1 fields written by CNX (except the last one):

0x1c0105: Destination
0x1c0114: File Format
0x1c0116: File Format Version
0x1c011e: Service ID
0x1c0128: Envelope Number
0x1c0132: Product ID
0x1c013c: Envelope Priority
0x1c0146: Date Sent
0x1c0150: Time Sent
0x1c015a: Coded Character Set

The record 1 fields are hardly ever seen so it is strange that Nikon writes them out (e.g. Photoshop only handles record 2).  The only record 1 field that Photo Mechanic writes and reads is 0x5a for identifying Unicode (UTF-8).  If 0x5a is missing then the character set is essentially undefined and Photo Mechanic will use whatever encoding is set in the IPTC/XMP preferences (older versions of PM always assumed MacRoman).

--dennis

[Hayo Baan]

Dennis, ah, thanks for the correction! Thanks to your info I have been able to fix my script in this respect. :-)

(I developed my IPTC decoding without having a full description of the IPTC data structure and had to include a lot of guesswork in my solution...)