Warning: Problem with NEF and Capture NX, even IPTC!

[perw]

I can't really decide if this is a request for help, or a general warning, but probably it is a bit of both.

Scenario:

• Nikon D80 RAW (NEF)
• ingest using PM4 while embedding both IPTC and XMP
• edit using Nikon Capture NX 1.1

This editing often (but not always) produces NEF-files that are not usable once opened again. Capture NX (CNX) will not complain when opening them again, but you can never save them again, no matter what changes you do.

While tracking this problem down, I discovered (as problably many others have already done) that embedding XMP data into NEF files is not a safe option. When I initially tested PM4 and CNX this worked, but it is not safe! I started having problems when I edited in CNX and saved several different edits inside the same file (nice feature of CNX). CNX is probably trampling on the area where XMP is saved.

I used the "revert ..." tool to extract everything into an XMP file, and all was well again.


Now comes the scary part.

Working with a NEF file with several different edits stored within, I used PM4 to insert just the IPTC parts of the external XMP-file into the file itself (I wanted the IPTC parts to be viewable from within CNX). This broke the file!

I did believe that it was always safe to insert IPTC into NEF-files, but have found instances where it does not work.

If this is supposed to always work (even if NEF-files are proprietary), there might be a problem.

If this is not guaranteed to always work, it would be nice to somewhere put a note that says that you always should use only XMP sidecar files with NEF.

After changing to XMP sidecar files exclusively, I have had no problems. I will use this solution from now on, or until I hear of changes. I would be nice to have all the data within the file again ...

[Juerg]

I use a D70 and D200 and always embed IPTC into the file (and create always an XMP sidecar file for XMP data) and I had never a problem with Capture NX, even if I have 2 or 3 versions saved within the NEF.

So, I don't know whether there is something different with D80 files (wouldn't be the first time Nikon changes NEF file formate for a specific camera).

What are your preference settings for IPTC/XMP in PM?

Archer

[perw]

What are your preference settings for IPTC/XMP in PM?

What I have right now is of course not interesting, since everything works .

What I had when I tried to embed IPTC from sidecar file into file:

• Reading IPTC/XMP: XMP sidecar, embedded IPTC, embedded XMP
• Writing IPTC/XMP: Add embedded IPTC, Always create/update ..

This is from memory, but I can verify it ...

Now I have verified that it fails!!

Below are the steps. You might not recognize all of the names, but this is because I am using the Swedish version, and the translation is mine, I don't know the original labels.

• File with IPTC and XMP, no edits
• Revert to original, save in XMP
• Make edits on file:
     WB
     Exposure
     Crop
     Levels/Curves
     Save as internal version 1
     Import set of corrections:
        Fit photo
        Adobe RGB -> sRGB
        USM
     Save as internal version 2
     
     Save file
     Close file
     Quit App
  
• Tools -> Update IPTC/XMP
• (Temporarily rename XMP to verify that IPTC has been updated. Rename back.)
• Edit using CNX (IPTC is back according to CNX)
    Deselect USM
    Save
    Revert to version 1
    Save
    Revert to version 2
    Save
    -->> Could not save file !!!!

What I have just done is to prove to myself that in my setup inserting IPTC into NEF-files sometimes make later edits impossible. I will stick with keeping all IPTC/XMP in a sidecar file.

I can of course not know the cause of this problem, but if I refrain from inserting IPTC data, these problems don't appear.


I have saved the corrupted file I created above. Earlier tests show that it will not work on my other systems (Mac). I have also tried using it on Windows, and there are problems there as well, although not exactly the same ones.

I can send the corrupt file to anybody that is interested in seeing how it is damaged. I also have the original file (as it appeared right after ingestion).

[Juerg]

• Edit using CNX (IPTC is back according to CNX)
    Deselect USM
    Save
    Revert to version 1
    Save
    Revert to version 2
    Save
    -->> Could not save file !!!!

Perw,

After you reverted to version 2 you could not save the file again, but did you get a error message?

When you closed NX and started again, could you load the file and was version 1 active?

If yes, could you then reverse to version 2 and file it?

Reason for these questions are that I had instances that NX did hang both during opening NEF's or saving NEF's, sometimes after, sometimes before they were touched by PM. I always have to force NX closed and restart and then it works again, sometimes for days, without problem.

Anyway, as I always write IPTC to NEF files, but never had that behaviour, I would be very interested to see the corupted file and see what it does on my system.

Thanks,
Archer

[perw]

After you reverted to version 2 you could not save the file again, but did you get a error message?

The only message was "Could not save file" (although in swedish).

As for the rest of the questions, I will address them when I get back home. I will also email you the broken file at that time. Do you want the correct one as well, to try the steps outlined above?

/Per

[Hayo Baan]

Hmm, since using PM I always insert the full IPTC/XMP combo in all my files, TIFF, JPG, NEF (both from a d200 and a d70).  So far never had the problem I could not reopen the NEF.

I did however notice that while browsing/rating files, I sometimes had to press the rating key (e.g. two or three for the nr of stars) twice for the first time it did not seem to stick.   I normally perform the rating in the preview mode (with one image enlarged) and there I have not seen an error message, and after pressing the rating-key for the second time, it worked.  On the other hand, I did get a "could not save" error when I tried saving IPTC information from within the IPTC info screen.  Here retrying the save worked as well...

I have not been able to reproduce any of the problems so far, but I think it may have to do with the fact that I had edited the files in CNX just prior to rating/adding IPTC.  So maybe CNX keeps a (temporary) lock on the file, which goes away when you try saving again.

Phew, hard to describe in words what went on, so I hope it is clear 

By the way, my IPTC settings are to always add both IPTC and XMP and to also always add/update the rating/colour class information immediately.

[perw]

• Edit using CNX (IPTC is back according to CNX)
    Deselect USM
    Save
    Revert to version 1
    Save
    Revert to version 2
    Save
    -->> Could not save file !!!!

Perw,

After you reverted to version 2 you could not save the file again, but did you get a error message?

Already answered. The answer was:

The only message was "Could not save file" (although in swedish).

When you closed NX and started again, could you load the file and was version 1 active?

Yes, I can load the file again, and it is at version 1. The last save, that was reported as failed, obviously also did fail, and the previous one did succeed.

If yes, could you then reverse to version 2 and file it?

Yes, i can revert to version 2. No, I cannot save it.

Anyway, as I always write IPTC to NEF files, but never had that behaviour, I would be very interested to see the corupted file and see what it does on my system.

I will send you both the corrupted file and the original.

/Per

[Juerg]

Per,

Thanks for the pictures you sent. I am (almost 110%) sure that your problem is created by Capture NX and has nothing to do with Photomechanic or writing IPTC or XMP data to the NEF file.

Here are your Capture NX edit steps in the version you can't save:

1. Base adjustments
2. Crop
3. Levels/Curves
4. Fit photo
5. Color Profile
6. USM (Unsharp Mask)


I started to play around and found that the problem was somehow connected to the Color Profile edit step. First I thought it's because you convert to Nikon sRGB 4.0.0.3000, which I don't have in my version of NX, I only have Nikon sRGB 4.0.0.3001. But that's not it.

I found that if you either uncheck (or delete) step 5 (color profile) or step 6 (USM) you can save the version.

So I tried what happens if you change (reverse) the order of edit steps 5 and 6 as following:

1. Base adjustments
2. Crop
3. Levels/Curves
4. Fit photo
5. USM (Unsharp Mask)
6. Color Profile

And surprise, now it works and you can save the version.

I tried this on your original file, your problem file, several of my NEF files (both out of camera and with PM captions applied) and even with some JPG fiels, and it is true in all cased that if you have

the 'Color Profile' step followed by a 'USM' step,

Capture NX can not save the file anymore. I tried several other combinations of 'Color Profile' steps followed by other edits steps (not all possibilities of course) and could save the file every time without problem, so it seems that only if you apply 'USM' after 'Color Profile' it causes problems.

Check the other files you can't safe and see whether you have this particular order of edits. If yes, just reverse them (UMS before Color Profile) and it should work.

You may want also to contact Jason P. Odell (author of The Photographer's Guide to Capture NX, http://www.luminescentphoto.com/). He may know about this problem and give you additional information.

Let me know what you find and whether you got my sample pictures (3 total).

Regards,
Archer

[perw]

Archer, I got your files, I will look at them later today.

From your description, I also believe that this is a problem with CNX. I will try to reproduce it with a few simple steps, and if I succeed I will file a problem report with Nikon.

Once I have filed the problem report, I will change the order of all convert/USM to USM/convert, and them re-import IPTC/XMP into the files again.

I will post the result of my contacts with Nikon.

/Per

[perw]

Intermediate follow up:

This is not clearcut at all. In my environment, editing files directly taken from the camera will not trigger the "cannot save" problem. The problem appears with files tagged with IPTC and XMP (embedded) by PM4.

So, if there is a bug in CNX, it is triggered by adding IPTC and XMP with PM4.

Adding the same IPTC data (the entries reachable) with CNX does not trigger the bug.

I have not yet tried to add only IPTC (and not XMP) with PM4.

I fear I will have a hard time getting Nikon to accept this as a bug if I can only trigger it by embedding extra data in "their" NEF-files using a third party product.


I will be back later after some more experiments.

[Juerg]

Sorry to say, but whenever I have a file (NEF or JPG out of camera and not touched with PM) and have the color profile step followed by the USM step, I can't save the version. No matter what I do.

Archer

[Hayo Baan]

So it seems like this IS a CNX bug/anomaly.

[perw]

Sorry to say, but whenever I have a file (NEF or JPG out of camera and not touched with PM) and have the color profile step followed by the USM step, I can't save the version. No matter what I do.

Archer

The jury is back, and the verdict is: Not guilty   

I agree with a previous poster saying that this probably is a CNX bug/anomaly.

You (Archer) can easily reproduce the problem, I cannot. I can only reproduce it with PM4 entering IPTC/XMP into the file. As a reported earlier, it will be difficult for me to successfully register a bug with Nikon if it requires injecting data into the NEF-file with a third party program. I am using the Swedish version of CNX 1.1 (1.1 - 11736: SPM version 110.3) with a Swedish Mac OS X 1.4.9. You are probably using another version.

I will not register this bug/anomaly, it would be fruitless. It would be nice if you (Archer) did so, you have a much simpler case to demonstrate.

There are other bugs in CNX 1.1, although they are not publicly disclosed. We must register bugreports to have them fixed!

Too much ranting, this is a PM4 forum after all. I am satisfied, the blame is not with PM4 after all. I think this topic does not need much more discussion. I will, however, refrain from using a color profile change followed by a USM again, at least until CNX 1.2. (And why bother with color profile changes, PM4 can easily do them for you for JPEG files.)

Anyone know of a good forum to discuss Nikon Capture NX problems?

[Juerg]

Per,

Try the Nikonians Forum at http://www.nikonians.org/cgi-bin/dcforum/dcboard.cgi. It's specifically for Nikon users.

By the way, the one difference is that I run the english Window version of NX, so perhaps there are some problems.

Archer

[Kirk Baker]

I can't really decide if this is a request for help, or a general warning, but probably it is a bit of both.

Scenario:

• Nikon D80 RAW (NEF)
• ingest using PM4 while embedding both IPTC and XMP
• edit using Nikon Capture NX 1.1

This editing often (but not always) produces NEF-files that are not usable once opened again. Capture NX (CNX) will not complain when opening them again, but you can never save them again, no matter what changes you do.

While tracking this problem down, I discovered (as problably many others have already done) that embedding XMP data into NEF files is not a safe option. When I initially tested PM4 and CNX this worked, but it is not safe! I started having problems when I edited in CNX and saved several different edits inside the same file (nice feature of CNX). CNX is probably trampling on the area where XMP is saved.

I used the "revert ..." tool to extract everything into an XMP file, and all was well again.


Now comes the scary part.

Working with a NEF file with several different edits stored within, I used PM4 to insert just the IPTC parts of the external XMP-file into the file itself (I wanted the IPTC parts to be viewable from within CNX). This broke the file!

I did believe that it was always safe to insert IPTC into NEF-files, but have found instances where it does not work.

If this is supposed to always work (even if NEF-files are proprietary), there might be a problem.

If this is not guaranteed to always work, it would be nice to somewhere put a note that says that you always should use only XMP sidecar files with NEF.

After changing to XMP sidecar files exclusively, I have had no problems. I will use this solution from now on, or until I hear of changes. I would be nice to have all the data within the file again ...

Can you tell me what OS and what version of PM you are using?

Thanks,

-Kirk