Something is very, very wrong

[rjh1007]

Background -
I discovered several blank thumbnails with a red filename in a folder I was browsing. I subsequently found over 600 of the blank red lettered thumbnails by doing an Organizer/New Filter/Filter Path and then selecting the top level of my catalog. This showed over 12000 files and by changing the sort order to Capture Time all 600+ of the blank red lettered thumbnails were now grouped. I can see the correctly displayed thumbnails in their folders in the navigator, windows explorer and Capture One Sessions.
I tried a catalog rescan, a re-index, individually including them into the catalog - things sort of changed and I thought everything was fixed, however, doing another new filter and they all reappeared blank red lettered.
Here is where it gets very weird. I tried selecting the blank thumbs and rt/clicked Show In context.  The thumbs would display properly in the folder in both navigator and an organizer browse and a filter with the file path to the particular folder(s) in the catalog.  All seemed good, however, if I created a NEW filter and a filter path to view all images ALL of the blank red lettered thumbs magically reappeared both in the filter and their respective folders undoing all my previous efforts.  These are not new blank thumbnails but the same ones.

Attachments:  These are an example using the same folder shown in successive order
1 Navigator - all thumbs display correctly
2 Organizer Browse - blank red lettered thumbs displayed
3 Organizer Filter - blank red lettered thumbs displayed
4 Filter and browse and navigator after selecting thumbs and showing in context
5 New filter to folder path

This leaves me very, very dubious about the veracity of PM+ and whether I trust using it into the future.

[rjh1007]

Here are the attachments that didn't up load
Bob

[Bill Kelly]

Hi Bob,

We'd like you to enable some debug logging so we can troubleshoot.

From the Help menu in Photo Mechanic Plus, choose "Edit Debug Settings..." and find the items titled CATALOG_DEBUG_LOGGING and PMCACHE_LOGGING and set their checkboxes. Then click the Close button. Then restart Photo Mechanic Plus and reproduce the problem. From the Help menu choose "Reveal Support Data..." and moments later a Finder window (or Windows Explorer window) will come forward with the zipped log file selected. Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

Bill

[rjh1007]

Bill
I have attached the pm_log.zip as requested.
I just opened a new organizer filter/folder path to the to level folder in the catalog. The blank red lettered files are scattered through out so by changing the sort to Capture time they all appear together at the ned of the sort.
I also opened a folder from this same filter and there are several of these blank thumbs in it.  Hopefully this is trackable in the log file.
My original post hopefully helps to also explain the situation.
One of your tech's - Andrew was on the phone with me yesterday morning - very helpful  - with team viewer he logged into my system - he was going to pass along his findings.
I am available if needed I can be reached by phone. I am in the Vancouver, BC area.
Thanks for your help
Bob

[Bill Kelly]

Hi Bob,

Thanks for the logfiles.

Unfortunately, the CATALOG_DEBUG_LOGGING and PMCACHE_LOGGING flags do not appear to have been enabled during this logging session. (Note that PM must be restarted after changing logging flags for them to take effect.)

Could you try again to enable those flags, restart PM, and reproduce the problem?

Thanks,

Bill

[rjh1007]

Bill
I was sure I followed your instructions last time.
This time I checked both he boxes, closed the dialog, closed and restarted PM+ and then double checked that the two boxes were checked.

To re create the issue I did exactly what I did in my previous reply.
Bob

[Bill Kelly]

Thanks, Bob. Debugging was now successfully enabled in the latest logs. Unfortunately there's nothing indicative of any errors occurring. We may have to add custom logging and send you a new build of PM. However, there is one more logging level we could try in the meantime:

If you choose "Edit Debug Settings" from the "Help" menu, paste the following into the edit field labeled "New Debug Setting":

  PMCACHE_VERBOSE_LOGGING

Then press the "Add" button.

Make sure the new debug setting is checked, and close PM.

Then restart PM and go through the steps to reproduce the problem. Please take a screenshot showing the contact sheet when the problem occurs. Then attach the screenshot and logs to your post here.

Hopefully we'll be able to spot what's causing the issue.

Thanks,

Bill

[rjh1007]

Bill
I have attached the new pm_log.zip file as requested using the  PMCACHE_VERBOSE_LOGGING.
I sent an email earlier today regarding a different but possible related issue  - the scan is hanging.  I spoke with Andrew and he should be up to speed with both the previous black placeholder issue and the new issue.

I needed to rename files in a number of folders to get the sequencing numbers correct.  I did this in Capture One so I wouldn't lose my reference links to my image adjustments.  I then removed the applicable folders from the catalog and then did a scan to catalog.  I noticed that a number of the blank placeholders now had the correct thumbnail previews so I continued scanning to catalog all the newly renamed files. The blank place holders has disappeared and have been replaced by the correct thumbnail previews. I did not import new files - these are the same files but only renamed.

The new issue is that after successfully scanning to catalog a number of newly imported folders the scan to catalog is now hanging. On trying to exit PM+ I receive the message "The catalog system is taking a while to stop. Continue waiting?" If I click Yes it just repeats over and over. If I click NO then after about 15 seconds a similar message appears with "pmdyn" in place of catalog and if I click No for waiting then a third message appears with "Dynamic UI" in place of pmdyn.  If I click No then no other messages appear.

I can then restart PM+.  If I try to scan to catalog exactly the same thing happens. So I am not able to scan to catalog.
Bob

[rjh1007]

Bill
I attached the PM_log.zip, however, it is constantly increasing in size. When I first looked it was 768kb, when tried to attached it the size was 36mb and I see it uploaded about 90mb and currently it is 342mb. 
I also received the attached error message
Bob

[Kirk Baker]

OK, so now that the problem has changed, can you turn off all of the debugging flags, but leave CATALOG_DEBUG_LOGGING on?  Then quit PM Plus, start it again, and reproduce the problem, then generate the zipped log and post it here, please?

Thanks,

-Kirk

[rjh1007]

Instructions followed and PM_log.zip
BTW the dyn.log file is growing before my eyes  - it went from about 700kb to 3GB in 3 minutes and continue to grow in size.  It was at 117gb this evening and had used all my free disk space.  PM+ was left running.
Bob

[Kirk Baker]

Bob,

Instructions followed and PM_log.zip
BTW the dyn.log file is growing before my eyes  - it went from about 700kb to 3GB in 3 minutes and continue to grow in size.  It was at 117gb this evening and had used all my free disk space.  PM+ was left running.

Thank you for the log.  The issue is that there are paths on your disk that are very long.  The one in particular is in a "CaptureOne" folder and contains some files that would eventually be ignored since they're of a type that Photo Mechanic Plus does not recognize.  But the directory scanning code is getting stuck on those files and keeps retrying.  And on each retry it logs the errors and because it happens so rapidly, the log file grows significantly in a short time.

We have added a feature to a development build that will allow you to ignore these CaptureOne folders or just parts of their hierarchy if you'd like.  I don't know if there are images in the CaptureOne folder that you'd want added to your catalog, but I expect that some parts of the files/folders would be unwanted.  Specifically, there is a folder in the CaptureOne folder named "Cache" which has a sub-folder named "Thumbnails".  I'm thinking that you wouldn't want those at all.

We're also adding a handler for long paths (that aren't otherwise ignored) that will simply skip them until we can arrive at a better solution for handling long paths without issue.  How likely is it that your disk contains deep folder hierarchies such that the length of the absolute path is greater than 260 characters?  If it is unlikely then I can make you a build soon that should scan your drive and build your catalog without much issue.

-Kirk

[rjh1007]

Kirk
Never thought of that.
A little background on  Capture One might help.
There are basically two ways of working with images in Capture One. Either as a catalog or in sessions. I can't see anyone using both a CO catalog and a PM + catalog. Sessions is folder based and stores all of the images adjustments, previews, thumbnails and metadata, etc in a series of subfolders under one parent subfolder named Capture One. This is similar to the way On1 does their so called catalog. There is one Capture One subfolder in every image folder regardless of how many images or the image types such as raw, jpeg, tif, etc. If there was a way to simply ignore this Capture One folder that might help.
As to to having file names that are over 260 characters long because of deep hierarchy should be of no concern. There should be the parent folder which is always a 4 digit folder like 2019, then possibly a sub parent folder such as New Zealand, then the image folder - longest one I can find is 70 characters, then the image filename which should be no more than 80 characters. There will be nothing below that if you completely ignore the Capture One folder.
So to reiterate, you can ignore the Capture One folder and everything below it when considering hierarchical filename length. I assume that Capture One is storing the metadata in some of those subfolders as they don't write metadata to raw files, however, I have no idea PM+ reads the metadata from the CO files.

If you need to be reading those subfolders and they have to be included in the hierarchical file naming I can always rename the longer ones.
Bob

[Kirk Baker]

Bob,

A little background on  Capture One might help.
There are basically two ways of working with images in Capture One. Either as a catalog or in sessions. I can't see anyone using both a CO catalog and a PM + catalog. Sessions is folder based and stores all of the images adjustments, previews, thumbnails and metadata, etc in a series of subfolders under one parent subfolder named Capture One. This is similar to the way On1 does their so called catalog. There is one Capture One subfolder in every image folder regardless of how many images or the image types such as raw, jpeg, tif, etc. If there was a way to simply ignore this Capture One folder that might help.
As to to having file names that are over 260 characters long because of deep hierarchy should be of no concern. There should be the parent folder which is always a 4 digit folder like 2019, then possibly a sub parent folder such as New Zealand, then the image folder - longest one I can find is 70 characters, then the image filename which should be no more than 80 characters. There will be nothing below that if you completely ignore the Capture One folder.
So to reiterate, you can ignore the Capture One folder and everything below it when considering hierarchical filename length. I assume that Capture One is storing the metadata in some of those subfolders as they don't write metadata to raw files, however, I have no idea PM+ reads the metadata from the CO files.

PM is only going to read metadata from images that end up being included in the catalog (or in an appropriate XMP sidecar file).  It doesn't know anything about the layout of CaptureOne folders or what might be useful alternative metadata.  If it doesn't follow the standards, we're going to ignore it.

Please try this build: https://www.camerabits.com/download/PMPlusSetupR5464_09adc63f.msi

When you do the Scan to Catalog please add a folder pattern to the Folders to Ignore section.  Entering:

*/CaptureOne

...and clicking the "Add Pattern..." button should skip the CaptureOne folder (that caused the hang).  You mentioned the folder being named "Capture One" but from your log, I see that there is no space in the name.

Please let us know how it goes.  If you run into any trouble, please post the logs.

Thanks,

-Kirk

[Soizic]

Hello Kirk

This is exactly what I want!
Is there a Mac version?
Thank you