Preview Generation Stuck

[Michael Naylor]

For more than 24 hours the Preview Generation has been stuck at 2555 batches remaining (0.2/sec, set. 2.9 hours).  If this is not a PM+ bug, could this have something to do with some of my images (size, corruption, etc) and how can I figure out which ones?

[Kirk Baker]

Michael,

For more than 24 hours the Preview Generation has been stuck at 2555 batches remaining (0.2/sec, set. 2.9 hours).  If this is not a PM+ bug, could this have something to do with some of my images (size, corruption, etc) and how can I figure out which ones?

Which build of Photo Mechanic Plus are you using, and on what OS?

-Kirk

[Michael Naylor]

Hi Kirk,

I'm using Photo Mechanic Plus Version 6.0, build 5438 (a729133),  on macOS Big Sur Version 11.1.

Its come down a little since yesterday...

[Kirk Baker]

Hi Kirk,

I'm using Photo Mechanic Plus Version 6.0, build 5438 (a729133),  on macOS Big Sur Version 11.1.

Its come down a little since yesterday...

I'd like for you to turn on CATALOG_DEBUG_LOGGING and then restart Photo Mechanic Plus.  From the Help menu, choose "Edit Debug Settings..." and find the item titled CATALOG_DEBUG_LOGGING and set its checkbox.  Then click the Close button.  Then restart Photo Mechanic Plus and let it run until either it finishes or it reproduces the problem.  From the Help menu choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

[Michael Naylor]

It did 3 or 4 batches then stopped.  I left it for about 10 minutes, before Reveal Support Data...  Here's the log.

[Kirk Baker]

It did 3 or 4 batches then stopped.  I left it for about 10 minutes, before Reveal Support Data...  Here's the log.

Thanks Michael.  Could you also do the following for me?

In the Finder, from the Go menu, choose "Go to Folder..." and enter:

~/Library/Logs/DiagnosticReports

Then select all of the files that start with "pm-task" or "pmruby" and then right-click on one of the selected files and choose "Compress N items" where N is the number of files you selected.  A file called Archive.zip will be created.  Could you then attach that file here, please?

Thanks,

-Kirk

[Michael Naylor]

There are no files beginning like that in this directory.

[SteveMattan]

I am experiencing the same issue, macOS Catalina 10.15.7, PM+ v 6.0, build 5438(a729133).

I have opened a support ticket (#295302 Catalog Image Preview Generation Stalls) and included the PM_log.zip file with that ticket.

I then found this thread, quit and restarted PM+ and I too saw the Preview Generation batch total go down for a while and then stall again.

I checked the CATALOG_DEBUG_LOGGING but I do not see any files which begin with "pm" in "~/Library/Logs/DiagnosticReports".

[Michael Naylor]

Hi Kirk,

You may be interested to know...  I stopped the Preview Generation and the Catalog Image Preview Copying tasks to see what would happen.  After reopening PM+, I now have 3 green bars.  I then ran a slide show on the largest files (all PSD) and everything appears to be working fine.  On closing PM+ no waning messages appear.

So perhaps the problem is that PM+ isn't counting the the Generation process correctly.  I hope this helps.

[SteveMattan]

I created a second catalog, of my originals, mostly DNGs but with some JPEGs, TIFFs, and others, 88364 images. The "Catalog Image Preview Generation" initially stalled as well. Then I unchecked the "Generate preview & thumbnail proxy images for off-line or remote use (recommended)" checkbox for the Derivatives catalog and restarted PM+. And the process completed successfully for the Originals catalog.

I then re-enabled preview generation and restarted PM+. And like Michael the task bars are green.

Curious.

[Kirk Baker]

I created a second catalog, of my originals, mostly DNGs but with some JPEGs, TIFFs, and others, 88364 images. The "Catalog Image Preview Generation" initially stalled as well. Then I unchecked the "Generate preview & thumbnail proxy images for off-line or remote use (recommended)" checkbox for the Derivatives catalog and restarted PM+. And the process completed successfully for the Originals catalog.

I then re-enabled preview generation and restarted PM+. Add like Michael the task bars are green.

Curious.

Turning it off stopped preview generation.  Turning it back on won't make it spontaneously create all of the proxies.  The work is not completed.  Doing a full Catalog Sync would restart the process and you may again encounter stalls.  Some of your images are either malformed, corrupted, or cause the proxy generation task to crash.  Quitting PM Plus and relaunching it causes the last batch that was being process to be skipped and proxy generation will resume with the next batch.  But you'll still have some missing proxies.

-Kirk

[Michael Naylor]

Perhaps some of my images are damaged in some way.  I'm fully aware that the Lightroom Classic catalogue is buggy.  In the past I've attempted to migrate to Media Pro and PhotoSupreme with similar issues which (for example) they displayed keywords and labels that had previously been deleted and removed from the Lightroom catalogue. The number of images imported also differed.

Migrating away from Lightroom is difficult to impossible.  To use PM+ I need to be confidence in knowing I'm not loosing any images and metadata.  I'm beginning to rant, sorry.

I'm willing accept that the PM team has run many tests with over a million images before releasing. So if Lightroom really is the culprit, the question is - how to identify and fix the mess Adobe has created?  Is there some magic third party utility which could examine and repair thousands of images?

Moving on, I thought I'd try forcing LR to Write Metadata to all the files again.  I then executed a Full Sync in PM+...

During Full Sync, if I clicked anywhere in the scrolling list, everything that follows becomes highlighted as if selected and it's not possible to stop this strange behaviour.

SO HERE IS THE RESULT...

[Wed 13:14:48] End Sync (0 errors)

• • • Sync summary (Full Sync) • • •

Catalog: LR-Live - 0 errors
    Total files in catalog: 25877
    Files missing or offline: 0
    Files scanned in folders: 25877
    New files added to catalog: 0
    Files updated in catalog: 25728
    Missing preview proxies to generate: 114

25877 - 25728 = 149 why not all?
114 why and which ones?

Lightroom claims there are 25892 images
25892 - 25877 = 15 missing from PM+

Therefore, I have 114, 149 and 15 problems and the number of Preview Generation batches has doubled to 5214 remaining.

Can PM+ provide a log of which files are missing and/or causing the problems?  It's obviously clever enough to know which files to reject, so it would be helpful if it could explain itself.

Oh, and there's a new problem.  A new File Type Format has appeared, with all 25728 images looking very blank.

[Kirk Baker]

Michael,

25877 - 25728 = 149 why not all?
114 why and which ones?

Lightroom claims there are 25892 images
25892 - 25877 = 15 missing from PM+

Therefore, I have 114, 149 and 15 problems and the number of Preview Generation batches has doubled to 5214 remaining.

Can PM+ provide a log of which files are missing and/or causing the problems?  It's obviously clever enough to know which files to reject, so it would be helpful if it could explain itself.

Error reporting will improve over time as development continues on Photo Mechanic Plus.

Oh, and there's a new problem.  A new File Type Format has appeared, with all 25728 images looking very blank.

Can you browse some of those images directly (via a folder-based contact sheet) and get images that render correctly or do they also fail to render?

-Kirk

[Michael Naylor]

Kirk,

Without going through all 25,728, it's fair to say none of them are rendering.

[SteveMattan]

I created a second catalog, of my originals, mostly DNGs but with some JPEGs, TIFFs, and others, 88364 images. The "Catalog Image Preview Generation" initially stalled as well. Then I unchecked the "Generate preview & thumbnail proxy images for off-line or remote use (recommended)" checkbox for the Derivatives catalog and restarted PM+. And the process completed successfully for the Originals catalog.

I then re-enabled preview generation and restarted PM+. Add like Michael the task bars are green.

Curious.

Turning it off stopped preview generation.  Turning it back on won't make it spontaneously create all of the proxies.  The work is not completed.  Doing a full Catalog Sync would restart the process and you may again encounter stalls.  Some of your images are either malformed, corrupted, or cause the proxy generation task to crash.  Quitting PM Plus and relaunching it causes the last batch that was being process to be skipped and proxy generation will resume with the next batch.  But you'll still have some missing proxies.

-Kirk

Ok, thanks that makes sense.

So I did a full sync on the Derivatives catalog, and forced it through to "Catalog Image Preview Generation" completion. This involved quite a few cycles of quitting and restarting PM+. I then did another full sync.

Here are the results:

------------------

[Thu 12:32:29] End Sync (0 errors)


• • • Sync summary (Full Sync) • • •

Catalog: Derivatives - 0 errors
    Total files in catalog: 34235
    Files missing or offline: 0
    Files scanned in folders: 34235
    New files added to catalog: 0
    Files updated in catalog: 0
    Missing preview proxies to generate: 1

------------------

I started with 2472 batches for preview generation after the first full sync. I found that the system would process between 10 to 30 batches before stalling. You can do the math for an approximate number of quit and restart cycles. Yet now that I've done the full sync there is but one image without a preview proxy. Quitting and restarting and doing another full sync still leaves that one missing preview.

Still curious. One would think that if there were malformed or corrupt images they would cause problems with each full sync.

Let me know if there is anything I can do should you want to investigate this further.

Thanks and stay well.