HELP - Color Class (and more) missing/wrong on 1000's of pics

[Juerg]

HELP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Just went back to old pictures not touched in 2-3 years to find that on 1000's of them:
- color class is missing (most) or wrong (few)
- orientation wrong (few)
- rating wrong (few)

All files were exclusively captioned, color coded, rated using PM going back to 2007.

WHAT IS GOING ON??

Naturally, I'm affraid that this is not the only thing ... but some files may be corrupt. Files affected include NEF's, CR2's, DNG's, TIFF's, JPEG's.

Using PM 4.5.4 (a couple month back sorthly PM 4.6 on a trail lisence) on a WinXP SP3 system.

Software used for various steps in the editing Process:
- PM for captio exclusively caption, keyword, color code, rate
- Nikon ViewNX to geo-tag
- Nikon Capture NX2 to develop NEF's and save as 16-bit TIFF's
- Bride (CS5) with ACR to develop CR2's(or into DNG's) and save as 16-bit TIFF's
- CS5 for TIFF edits

Here a screen shot of the PM General  and IPTC/XMP tab.

Hope you can help.

Thanks
Juerg

[attachment deleted by admin]

[Kirk Baker]

Juerg,

HELP!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Just went back to old pictures not touched in 2-3 years to find that on 1000's of them:
- color class is missing (most) or wrong (few)
- orientation wrong (few)
- rating wrong (few)

All files were exclusively captioned, color coded, rated using PM going back to 2007.

WHAT IS GOING ON??

I really have no idea without having some of your images.  Can you share one or two with us?

Thanks,

-Kirk

[Juerg]

Hi Kirk,

Sure. What exactly do you need and where to send/upload.

Question: Is there a way PM installation is somehow corrupted. Reason I ask is that it also seems to take ages doing any operation ... sometimes to the extend that I have to force a shut down and restart. Isn't there a log file that shows things like that.

Or is something with my IPTC/XMP setting wrong that can cause the use of NX2 and CS5/Bridge to cause such things ... even if those files where never even touched for a loooong time (sometimes years)?

I'm REALLY FREAKED OUT at the moment and don't want to touch anything anymore not knowing what is going on.

[Kirk Baker]

Juerg,

Question: Is there a way PM installation is somehow corrupted. Reason I ask is that it also seems to take ages doing any operation ... sometimes to the extend that I have to force a shut down and restart. Isn't there a log file that shows things like that.

Unlikely.  More likely is that your preferences have changed since you first started using PM and PM is now finding your metadata in a different order than before.

Or is something with my IPTC/XMP setting wrong that can cause the use of NX2 and CS5/Bridge to cause such things ... even if those files where never even touched for a loooong time (sometimes years)?

This is likely.  If the order in which PM is instructed to search for metadata is different than when the images were last edited with PM and then some other application like Adobe's apps then added their metadata, PM would (depending on your settings) see Adobe's data (which doesn't maintain PM's older data format) instead and thus the edits appear to be lost.

I'm REALLY FREAKED OUT at the moment and don't want to touch anything anymore not knowing what is going on.

I suggest you don't make any changes until we figure out the cause of your issue.

-Kirk

[Juerg]

Kirk,

Question: Is there a way PM installation is somehow corrupted. Reason I ask is that it also seems to take ages doing any operation ... sometimes to the extend that I have to force a shut down and restart. Isn't there a log file that shows things like that.

Unlikely.  More likely is that your preferences have changed since you first started using PM and PM is now finding your metadata in a different order than before.

There is unfortunately no way to save the IPTC/XMP setting tab to see what and how I changed these over the years. I know we did somehting a while back due to peoblems using CaptureNX and Adobe. Would perhaps be a nice feature to have an automatic dated backup made of the of this tab every time something is changed ... easier to follow up what changes were made over time.

Or is something with my IPTC/XMP setting wrong that can cause the use of NX2 and CS5/Bridge to cause such things ... even if those files where never even touched for a loooong time (sometimes years)?

This is likely.  If the order in which PM is instructed to search for metadata is different than when the images were last edited with PM and then some other application like Adobe's apps then added their metadata, PM would (depending on your settings) see Adobe's data (which doesn't maintain PM's older data format) instead and thus the edits appear to be lost.

Well, one thing I noticed is that with Photoshop CS3/4, the color class would synchronice between PM and Bridge. Since switching to CS5 about 15 month ago, that's not the case any longer.

I'm REALLY FREAKED OUT at the moment and don't want to touch anything anymore not knowing what is going on.

I suggest you don't make any changes until we figure out the cause of your issue.

Oh, believe me ... at the moment I don't touch anything. If it's only the color classes, some ratings and some orientations that's wrong ... well, a pain in the butt and a lot of work to correct again, but duable. But I'm still affraid that I may find some corrupted files ...

The requested files are being uploaded as of this moment ... assume another 20-30 minutes (it's one very large TIFF file included).

Regards,
Juerg

[Kirk Baker]

Juerg,

I have received your files.  For the NEFs, changing your read search order to:

First: Read XMP Sidecar file
Second: Read embedded XMP
Third: Read embedded IPTC

For everything: on the General tab, turn off the 'Synchronize Color Class with IPTC/XMP Urgency field' checkbox, and for the Color Label text, you need to have a label with the name 'Green' because it is called out specifically in the TIF file.

Adobe uses label names which unfortunately don't work well when you decide to use different names.  PM uses color numbers (or indexes) for Color Classes so it doesn't matter what you change the label to, the color will match (unless you change the color values as well.)

It appears to me that none of your data was lost.  Your settings have just changed enough that some data is out of sync now that you've used Photoshop or Bridge on them.

HTH,

-Kirk

[Juerg]

Hi Kirk,

Thanks for looking into that.

I have received your files.  For the NEFs, changing your read search order to:

First: Read XMP Sidecar file
Second: Read embedded XMP
Third: Read embedded IPTC

For everything: on the General tab, turn off the 'Synchronize Color Class with IPTC/XMP Urgency field' checkbox, and for the Color Label text, you need to have a label with the name 'Green' because it is called out specifically in the TIF file.

Adobe uses label names which unfortunately don't work well when you decide to use different names.  PM uses color numbers (or indexes) for Color Classes so it doesn't matter what you change the label to, the color will match (unless you change the color values as well.)

It appears to me that none of your data was lost.  Your settings have just changed enough that some data is out of sync now that you've used Photoshop or Bridge on them.

Befor I change something a question:

The settings I have are as far as I know based on a discussion we had based on using a combination of PM with CaptureNX and Adobe applications. I don't remember exaclty when I changed it, but I know we specifically changed also the 'Synchronize Color Class with IPTC/XMP Urgency field' checkbox. If I now change things "back" like you propose, what happens to the files that I have worked on since? Will they get out of sync?

And what wonders me is that the files I sent were for sure never opened in an Adobe application other than CS5 (never browsed in Bridge), but most likely worked on before I changed the General and ITPC/XMP tab to the current setting. So is that the problem?

Also, this is from the last PM you sent (copied it here to make it easier):

Another thing that I also noticed over the last few weeks is that PM becomes more and more sluggish ... sometimes does not responde for 5-10 minutes when opening a folder (including sub-folder) with 500-1000 pics total ... especially folders that contain pictures that have above problems with missing/wrong information. As if PM must work very hard in the background ... like updating file information?

What sort type are you using?  Anything other than Filename sort?

PM does not update files unless you make it do so by editing (IPTC/Color Class/Rating/Tag) the file.  It certainly won't modify the files when scanning them.

In PM, I usually use capture time as sort filter (sometimes file type, sometimes copyright).

In Bridge, which I used only to browse and work on a few folders with recent CR2 / DNG files, I used color class as well as copyright as search filters. What I never used Bridge for is change anything (color class, star rating, or other metadata), same for Capture NX.

Thanks,
Juerg

[Kirk Baker]

Juerg,

I have received your files.  For the NEFs, changing your read search order to:

First: Read XMP Sidecar file
Second: Read embedded XMP
Third: Read embedded IPTC

For everything: on the General tab, turn off the 'Synchronize Color Class with IPTC/XMP Urgency field' checkbox, and for the Color Label text, you need to have a label with the name 'Green' because it is called out specifically in the TIF file.

Adobe uses label names which unfortunately don't work well when you decide to use different names.  PM uses color numbers (or indexes) for Color Classes so it doesn't matter what you change the label to, the color will match (unless you change the color values as well.)

It appears to me that none of your data was lost.  Your settings have just changed enough that some data is out of sync now that you've used Photoshop or Bridge on them.

Befor I change something a question:

The settings I have are as far as I know based on a discussion we had based on using a combination of PM with CaptureNX and Adobe applications. I don't remember exaclty when I changed it, but I know we specifically changed also the 'Synchronize Color Class with IPTC/XMP Urgency field' checkbox. If I now change things "back" like you propose, what happens to the files that I have worked on since? Will they get out of sync?

It's entirely possible.  Adobe's apps aren't going to synchronize the color labels with the IPTC Urgency field since they have no concept of a Label to index relationship.  So the answer is yes it is possible for them to get out of sync.

And what wonders me is that the files I sent were for sure never opened in an Adobe application other than CS5 (never browsed in Bridge), but most likely worked on before I changed the General and ITPC/XMP tab to the current setting. So is that the problem?

CS5 will also let the IPTC Urgency field get out of sync.  All Adobe apps will do this.

Also, this is from the last PM you sent (copied it here to make it easier):

Another thing that I also noticed over the last few weeks is that PM becomes more and more sluggish ... sometimes does not responde for 5-10 minutes when opening a folder (including sub-folder) with 500-1000 pics total ... especially folders that contain pictures that have above problems with missing/wrong information. As if PM must work very hard in the background ... like updating file information?

What sort type are you using?  Anything other than Filename sort?

PM does not update files unless you make it do so by editing (IPTC/Color Class/Rating/Tag) the file.  It certainly won't modify the files when scanning them.

In PM, I usually use capture time as sort filter (sometimes file type, sometimes copyright).

In Bridge, which I used only to browse and work on a few folders with recent CR2 / DNG files, I used color class as well as copyright as search filters. What I never used Bridge for is change anything (color class, star rating, or other metadata), same for Capture NX.

All sort methods other than Filename are slow in PM.  In order to complete the sort, the Capture Time or other metadata must be fetched for each file in the contact sheet before the sort can complete.

-Kirk

[Juerg]

Hi Kirk,

So, let's see where we stand:

- I have thousands of pic (mostly worked on before 2009) out of sync before the PM preferences were changed.
- I have thousands of pics worked on since with the current preferences in sync.
- I use PM to capture, rate, label.
- I use Capture NX to develop RAW (Nef) files and ViewNX to geo-tag.
- I use CS5 to final edit files (TIFF's, JPEG's, and CR2/DNG's).
- I started to use Bridge again because of CR2/DNG's.

How/what do I change without even get into a bigger mess AND to ensure that the files and the information within are best keept up to date, also in view of any future software that may come along.

Would deleting all XMP side-car files ALLWAYS when PM starts and scannes a folder help as I want to have IPTC/XMP data included in the file anyway? Could that be made a option to be checked in the XMP tab. Adobe applications (as well as Nikon applications) would still read the the embeded information. Adobe would then perhaps write a new XMP sidecar file, which then would be deleted again by PM (and thus ignored).

As for sorting, I used capture time sort since I started using PM, but only the last few month it became such slugish.

Another thing that you said is that the PM color labels and Adobe color labels need to have the same name to be recognized. Well, why are some working and others not with the same "color class" naming differences?

Juerg

[Kirk Baker]

Juerg,

So, let's see where we stand:

- I have thousands of pic (mostly worked on before 2009) out of sync before the PM preferences were changed.
- I have thousands of pics worked on since with the current preferences in sync.
- I use PM to capture, rate, label.
- I use Capture NX to develop RAW (Nef) files and ViewNX to geo-tag.
- I use CS5 to final edit files (TIFF's, JPEG's, and CR2/DNG's).
- I started to use Bridge again because of CR2/DNG's.

How/what do I change without even get into a bigger mess AND to ensure that the files and the information within are best keept up to date, also in view of any future software that may come along.

That last part is an impossibility.  Who knows what other software will come along in the future and make this more difficult?

Would deleting all XMP side-car files ALLWAYS when PM starts and scannes a folder help as I want to have IPTC/XMP data included in the file anyway? Could that be made a option to be checked in the XMP tab. Adobe applications (as well as Nikon applications) would still read the the embeded information. Adobe would then perhaps write a new XMP sidecar file, which then would be deleted again by PM (and thus ignored).

The main issue is that you're trying to use two programs (Capture NX2 and Bridge/ACR) that don't handle metadata in the same way.  CNX2 only works with embedded XMP and completely ignores XMP sidecar files and moreover needs to have the IPTC Urgency field be re-purposed for the use of color labels.  Adobe's apps have various issues when embedded XMP is present and won't update embedded XMP, thus things can get out of sync and do not have the ability to synchronize the label index (no Indexes for Label exist in Adobe's apps!) with the IPTC Urgency field.

PM isn't going to delete XMP sidecar files, no matter what your settings are.  The most you can do is get PM to load them only as a last resort.

As for sorting, I used capture time sort since I started using PM, but only the last few month it became such slugish.

OK, so since nothing in PM has changed (you're still using PM 4.5.4, right?) then what has changed about your system?  Is it low on memory?  If virtual memory swapping kicks in, you're going to get very slow performance.

Another thing that you said is that the PM color labels and Adobe color labels need to have the same name to be recognized.

Very true.

Well, why are some working and others not with the same "color class" naming differences?

I'm not sure I follow.  Please give me an example.

Thanks,

-Kirk

[Juerg]

Hi Kirk,

Just an update confirming (I assume at least) that the pure fact that PM is scanning a folder containing pics does apply changes to the files in the folder:

After I discovered the out-of-sync issue yesterday I opened a few folders that only contain JPEG's of final edited pictures. I know for sure that I have not touched those since the last system backup last week.

This morning I run a backup (incremental, using the Vice-Versa which compares the time stamp on files on the working HD with the already present files on the backup HD to determin in files have changed on the working HD and then backs them up).

ALL files that were scanned by PM yesterday, but not otherwise touched (not in an Adobe application, nor a Nikon application) were flaged this morning as being changed ... ?

So, let's see where we stand:

- I have thousands of pic (mostly worked on before 2009) out of sync before the PM preferences were changed.
- I have thousands of pics worked on since with the current preferences in sync.
- I use PM to capture, rate, label.
- I use Capture NX to develop RAW (Nef) files and ViewNX to geo-tag.
- I use CS5 to final edit files (TIFF's, JPEG's, and CR2/DNG's).
- I started to use Bridge again because of CR2/DNG's.

How/what do I change without even get into a bigger mess AND to ensure that the files and the information within are best keept up to date, also in view of any future software that may come along.

That last part is an impossibility.  Who knows what other software will come along in the future and make this more difficult?

Understandable.

Would deleting all XMP side-car files ALLWAYS when PM starts and scannes a folder help as I want to have IPTC/XMP data included in the file anyway? Could that be made a option to be checked in the XMP tab. Adobe applications (as well as Nikon applications) would still read the the embeded information. Adobe would then perhaps write a new XMP sidecar file, which then would be deleted again by PM (and thus ignored).

The main issue is that you're trying to use two programs (Capture NX2 and Bridge/ACR) that don't handle metadata in the same way.  CNX2 only works with embedded XMP and completely ignores XMP sidecar files and moreover needs to have the IPTC Urgency field be re-purposed for the use of color labels.  Adobe's apps have various issues when embedded XMP is present and won't update embedded XMP, thus things can get out of sync and do not have the ability to synchronize the label index (no Indexes for Label exist in Adobe's apps!) with the IPTC Urgency field.

So, what are your saying? That my settings as they are are the best and should stay like that?

PM isn't going to delete XMP sidecar files, no matter what your settings are.  The most you can do is get PM to load them only as a last resort.

Well, that would be a NEW feature to be added :-)

As for sorting, I used capture time sort since I started using PM, but only the last few month it became such slugish.

OK, so since nothing in PM has changed (you're still using PM 4.5.4, right?) then what has changed about your system?  Is it low on memory?  If virtual memory swapping kicks in, you're going to get very slow performance.

No, nothing changed on the system .... at least not that I am aware of.

Another thing that you said is that the PM color labels and Adobe color labels need to have the same name to be recognized.

Very true.

So, the green label in PM has to be called "Green", the yellow label "Yellow" and so on?
I always thought the labels are only for ID purposes in PM. Not actually written into the file itself.

Well, why are some working and others not with the same "color class" naming differences?

I'm not sure I follow.  Please give me an example.

Means that the files I sent are from a folder that contains also files that are correct, but were edited at the same time as the files that are wrong.

Thanks,
Juerg

[Kirk Baker]

Juerg,

Just an update confirming (I assume at least) that the pure fact that PM is scanning a folder containing pics does apply changes to the files in the folder:

After I discovered the out-of-sync issue yesterday I opened a few folders that only contain JPEG's of final edited pictures. I know for sure that I have not touched those since the last system backup last week.

This morning I run a backup (incremental, using the Vice-Versa which compares the time stamp on files on the working HD with the already present files on the backup HD to determin in files have changed on the working HD and then backs them up).

ALL files that were scanned by PM yesterday, but not otherwise touched (not in an Adobe application, nor a Nikon application) were flaged this morning as being changed ... ?

PM should not touch your files unless you make changes (tag, rating, color class, IPTC).  Are you 100% certain that you made absolutely no changes whatsoever from within PM?  What changes did this Vice-Versa application say were made?

-Kirk

[Kirk Baker]

The main issue is that you're trying to use two programs (Capture NX2 and Bridge/ACR) that don't handle metadata in the same way.  CNX2 only works with embedded XMP and completely ignores XMP sidecar files and moreover needs to have the IPTC Urgency field be re-purposed for the use of color labels.  Adobe's apps have various issues when embedded XMP is present and won't update embedded XMP, thus things can get out of sync and do not have the ability to synchronize the label index (no Indexes for Label exist in Adobe's apps!) with the IPTC Urgency field.

So, what are your saying? That my settings as they are are the best and should stay like that?

What I'm saying is that it is difficult to have your cake and eat it too.  You could take PM completely out of the picture and just use CNX2 and Adobe applications together and have all sorts of problems.

PM isn't going to delete XMP sidecar files, no matter what your settings are.  The most you can do is get PM to load them only as a last resort.

Well, that would be a NEW feature to be added :-)

I don't think we'd ever add a feature that strips XMP sidecar files when PM scans a folder.  Recipe for disaster.

Another thing that you said is that the PM color labels and Adobe color labels need to have the same name to be recognized.

Very true.

So, the green label in PM has to be called "Green", the yellow label "Yellow" and so on?
I always thought the labels are only for ID purposes in PM. Not actually written into the file itself.

Not sure where you got that information.  PM must write the label names into the XMP data in order to work with Adobe products.

Well, why are some working and others not with the same "color class" naming differences?

I'm not sure I follow.  Please give me an example.

Means that the files I sent are from a folder that contains also files that are correct, but were edited at the same time as the files that are wrong.

Honestly, I have no idea.  Unless you gave me every single photo in this folder and I had the time to analyze each and every one of them, I couldn't give you a definite answer.

-Kirk

[Juerg]

Juerg,

Just an update confirming (I assume at least) that the pure fact that PM is scanning a folder containing pics does apply changes to the files in the folder:

After I discovered the out-of-sync issue yesterday I opened a few folders that only contain JPEG's of final edited pictures. I know for sure that I have not touched those since the last system backup last week.

This morning I run a backup (incremental, using the Vice-Versa which compares the time stamp on files on the working HD with the already present files on the backup HD to determin in files have changed on the working HD and then backs them up).

ALL files that were scanned by PM yesterday, but not otherwise touched (not in an Adobe application, nor a Nikon application) were flaged this morning as being changed ... ?

PM should not touch your files unless you make changes (tag, rating, color class, IPTC).  Are you 100% certain that you made absolutely no changes whatsoever from within PM?  What changes did this Vice-Versa application say were made?

-Kirk

Yes, I'm absolut certain I did not touch the files in PM other than open the contact sheet, scroll though it, and close it.

Vice-Versa does not give details. But knowing that it compares the time tag on the files I look at a file on the main HD and a file on the backup. And it seems the "Modified" time is differn by 1 hour, and the some files size (not for all of them) is slightly larger on the copy of the working HD (perhaps 100-200 bites).

Juerg

[Juerg]

The main issue is that you're trying to use two programs (Capture NX2 and Bridge/ACR) that don't handle metadata in the same way.  CNX2 only works with embedded XMP and completely ignores XMP sidecar files and moreover needs to have the IPTC Urgency field be re-purposed for the use of color labels.  Adobe's apps have various issues when embedded XMP is present and won't update embedded XMP, thus things can get out of sync and do not have the ability to synchronize the label index (no Indexes for Label exist in Adobe's apps!) with the IPTC Urgency field.

So, what are your saying? That my settings as they are are the best and should stay like that?

What I'm saying is that it is difficult to have your cake and eat it too.  You could take PM completely out of the picture and just use CNX2 and Adobe applications together and have all sorts of problems.

No, one of the reasons I use PM is because neiter the Capture nor Adobe applications work well for me ... PM does (just wished the two other companies would ask you guys how to do things right). Anyway, what do I need to change. I don't want to screw up more. And if I have to go back and reset color class, ratings, orientation in the OLD files already screwed up, so I will (sometime). I don't want to screw up the at the moment correct ones. And from now on forward, I need a setting that works best (albeit, probably not perfect).

Taking the "General" Tab:

What do I need to change there?
- Changing color labels from descriptiv back to actuall collors (ie "FinalEdit_Originals" to "Green" and so on)?
- Unckeck the 'Synchronize Color Class with IPTC/XMP Urgency field' checkbox? (But then you said NX needs that ... I think based on a previous discussion we had here http://forums.camerabits.com/index.php?topic=3601.msg16256#msg16256 is why the options in both the "General" and "IPTC/XMP" tabs were set as they are).

And in the "IPTC/XMP" tab:

What do I need to change there?

Please note that there is no way around using NX or CS5, but, as I did for a while until I got a Canon camera, I can start NOT using Bridge again. I believe that where things really get messed up. So setting should be for maximum copadibility and data safety (synchronization).

PM isn't going to delete XMP sidecar files, no matter what your settings are.  The most you can do is get PM to load them only as a last resort.

Well, that would be a NEW feature to be added :-)

I don't think we'd ever add a feature that strips XMP sidecar files when PM scans a folder.  Recipe for disaster.

LOL ... well, that's me not knowing what would happen ...

So, the green label in PM has to be called "Green", the yellow label "Yellow" and so on?
I always thought the labels are only for ID purposes in PM. Not actually written into the file itself.

Not sure where you got that information.  PM must write the label names into the XMP data in order to work with Adobe products.

Hmm, I just thought so as you set the actuall color by a drop down box and the information of this box is actually written, not the field behind where I (thought) one can enter discriptive text. So, nobody told me that, I just assumed (... based on my limited but VERY dangerous knowledge level ).

Means that the files I sent are from a folder that contains also files that are correct, but were edited at the same time as the files that are wrong.

Honestly, I have no idea.  Unless you gave me every single photo in this folder and I had the time to analyze each and every one of them, I couldn't give you a definite answer.

Well, that's out of question ... unless I upload files for a couple of days ... and you find time to look at them all individually.

But as I said, all the files that are already screwed up I am gonna correct again.

IMPORTANT for me now is how to set preferences in the "General" and "IPTC/XMP" Tabs to not screw up more and still can use CaptureNX, ViewNX, CS5 and (if possible in any way) Bridge (if you say it's not possible with Bridge included, I skip Bridge, but the others I need to work with) without screwing up the files that are currently correct.

And, do you think that some of the issues would be helped by updating from version 4.5.4 to the current 4.6.7? I wanted to wait for the next major release, but if you say it IS important to update, I will do (of course I now I missed the special update deal you had earlier this year   )

Juerg