Scanning to multiple catalogs hangs

[Gary D.]

installed beta 19 and attempted to create and populate two new catalogs.  (one catalog already existed)
I kick off the scan off the first new catalog  Wait for the  Catalog Folder Scanning task to complete.
I create another catalog, and scan another folder of images. The second scan begins and then   Tasks Catalog Metadata Gathering and  Catalog Image Preview Generation  hang.
Let it run overnight (12 hours) and didn't see any progress.
I've deleted the catalogs   and  went through that again   with same results.
Cleared everything out again , and restarted a scan on one just one catalog 
Waited for Folder Scanning, Metadata Gathering and Metadata Updates to complete (only Preview Generation and Preview Copying run-in).
Kicked off anther scan to a different catalog .  Folder Scanning and Metadata updates complete but   Metadata Gathering and Preview Generation  hang.

The folders being scanned have 25K and 40K images in them.

Photo Mechanic Plus Version 6.0, build 4826 (4293d5e)
 iMac Info
macOS Catalina Version 10.15.5
iMac (21.5-inch, Late 2012)
Processor 2.9 GHz Intel core i5
Memory 16 GB 1600 MHz DDR3
Graphics NVIDIA GeForce GT 650M512 MB

[Gary D.]

Just some extra info..  It seems that I get that hang even if I scan to only one catalog.  Metadata gathering and Image Preview Generation are the tasks that never get to completion.

While those processes are hung I can go to a contact sheet, select all the images and right click  "Include In Catalog" and the Metadata Updates and Image Preview Copying run and complete.
I've go to Catalog Status, and Stopped both Catalog Import Tasks and Preview Generation tasks.

With all task GREEN and idle I can select a folder in the navigator and select "Open Folder and Subfolder In A New Contact Sheet". Select all the images in the contact sheet and choose  "Include Selected Items" from the Catalog Menu.  All tasks complete and show idle, and the items are in the catalog as expected.  Did several runs like that and the largest contact sheet had 40k images.

Haven't tried including images into two different catalogs at the same time yet. 
Gary

[Kirk Baker]

I'd like to see your Photo Mechanic log.  In Photo Mechanic go to the Help menu and choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

[Gary D.]

Hey Kirt   Here's what I did
Start PM6+
Navigate to a folder
Right click and select "Include when Scanning"
Scanning dialog appears and I choose the Manage Catalogs  button.
Create a new catalog and select it for adding.  All other catalogs unselected.
Hit start and the folder scanning task starts and completes within a minute or so  ( about 10k images)
The Catalog Metadata Gathering and Catalog Image Preview Generation start and almost immediately freeze
I waited a little over 5 min and created the attached logs.
Rest of the interface is responsive I can even "search" the new catalog but it only has 10 images in it.
I eventually just close PM6+  and at prompt to continue wanting on the catalog to close  I just say No

Gary

[Kirk Baker]

Gary,

Thanks for the log file.  We found the last batch of images that were processed before the image loading/scaling process died.  Can you share them with us?

I'll send you a personal message with the list of filenames (full path).

Thanks,

-Kirk

[Kirk Baker]

Gary,

Thank you for the sample images.  We were unable to reproduce the problem locally.

I'm going to have you turn on some additional logging.  You need to create a plain text file named PMDebug.txt and it has to be in PM's Preferences folder.

PM's Preferences folder will be in the following location:

~/Library/Preferences/com.camerabits.PhotoMechanic/

Unfortunately Apple decided a few OS versions ago to hide the Library folder within your home folder.  You can still get there though by using the "Go to Folder..." command on the Go menu in the Finder.  Enter the path above in the dialog that appears.

Create your PMDebug.txt file there and make it have the following contents:

CATALOG_DEBUG_LOGGING ON

Restart Photo Mechanic Plus.  Reproduce the problem again. You can then use the "Reveal Support Data..." command on the Help menu to get the updated zipped log file.  Please post it here.

Thanks,

-Kirk

[Gary D.]

Hey Kirt
Created the PMDebug.txt file and started PM6+
Chose a directory, and kicked off the scan.
It ran for a little while this time  (maybe a couple of min)  Catalog Metadata Update task started and stopped a few times. Probably did the other times i ran the scan process but didn't notice it.
Attached is the log data  collected after the process hung

thanks
Gary

[Kirk Baker]

Gary,

Created the PMDebug.txt file and started PM6+
Chose a directory, and kicked off the scan.
It ran for a little while this time  (maybe a couple of min)  Catalog Metadata Update task started and stopped a few times. Probably did the other times i ran the scan process but didn't notice it.
Attached is the log data  collected after the process hung

Somehow the logs were incomplete.  Could you please go to that folder where you got the zip file and select all files that end with .log and then right-click on one of them and choose "Compress N items" (N will be the number of items)?  Then post that new zip file here?

Thanks,

-Kirk

[Gary D.]

Kirt

Lots of mixed results today.
I saw the post about the logs not being complete so I ran a scan and it completed.
Ran a few more and they would hang BUT if I stopped the Catalog Import Task.. The Preview tasks would restart and complete. That didn't happen before.
The other behavior that is new is that if I exit PM6 with the tasks hung and then restart PM6 the tasks pick up where they left off.

Attached are the log files taken after all the attempts.

Earlier in the morning I had removed On1 2020.1. It was the only out of the ordinary thing I did before I got that successful run. I reinstalled and the scans hung again . Removed on1 it and I got some successes, some hangs and some success on restarts.  Can't see how its related but I mention it only to explain all the starts and stops in the logs


Gary

[Kirk Baker]

Thanks, Gary.

When it hangs, can you open up the Activity Monitor and click on either the CPU or Memory buttons and then in the search field on the right side enter:

pm-task

and let us know how many of them you see?

Also, can you add the following to your PMDebug.txt file and restart Photo Mechanic Plus and create the problem again?

CILA_RPC_MSG_LOGGING ON

Then send the logs again (manually like you did last time) please.

Thanks,

-Kirk

[Gary D.]

Hey Kirt
Here’s what I did. 

in pm6+  stop all tasks
edited the debug file
 CATALOG_DEBUG_LOGGING ON
 CILA_RPC_MSG_LOGGING ON

Delete all catalogs
exit and reboot iMac

Start PM6+  ~ 9:00am est
   Create a catalog  Set it for Search and Add ( unselected the default Catalog)

From navigator   selected a  directory with a small number of  images and did “Include when scanning to catalog”
It completed no problems , Checked the catalog and the 28 images were there. They were all JPEG

From Navigator selected a directory with a larger number of images and  did “Include when scanning to catalog”
Just a note but the scan dialog came up in reverse colors  IE  White text Black background.
This also completed correctly This directory had 20 TIFF files

From Navigator selected a directory with a larger number of images and  did “Include when scanning to catalog”  (9:10 AM)
This directory had roughly 41K images  of all kinds,  JPEG, TIFF, Canon Raw, Nikon Raw, Sony Raw   you name it.  By 9:11
 I get the hang of catalog Metadata Gathering.
It noted there are ‘About 11310 items remaining (0.2/sec)
saved off a ps -fu Gary and attached the PM related processes  (PS_914,txt)

  501   828     1   0  9:00AM ??         0:44.01 /Applications/Photo Mechanic Plus.app/Contents/MacOS/Photo Mechanic Plus
  501   834   828   0  9:00AM ??         0:57.72 ./pmruby/bin/pm-task --disable-gems -EUTF-8:UTF-8 -I. ./dyn/script/start_dyn_platform.rb --vis-port=49464
  501   835   834   0  9:00AM ??         0:08.31 ./pmruby/bin/pm-task --disable-gems -EUTF-8:UTF-8 ./archive/common/localsvc/camerabits_catalog.rb 127.0.0.1 49465
  501   836   834   0  9:00AM ??         0:00.00 (pm-task)
  501   838   834   0  9:00AM ??         0:04.11 ./pmruby/bin/pm-task --disable-gems -EUTF-8:UTF-8 ./dyn/common/localsvc/camerabits_image.rb 127.0.0.1 49465

With it still hung I exited PM6+ and at the dialog asking if I wanted to wait some more  I clicked ’No’
Manually created  the logs and did another ps -fu Gary (PS_917.txt)

Attached it all and went out to mow the lawn.

Thanks
Gary

[esambo]

My "Scan to Catalog" gets the "Catalog Metadata Gathering" task stuck too, and sometimes the "Catalog Image Preview Generation" too.
I was able to isolate it to only occur if the directory contains a HEIF image.

Steps:
1. Create a new catalog
2. Enable debugging with: echo "CATALOG_DEBUG_LOGGING ON\nCILA_RPC_MSG_LOGGING ON # very noisy in: ~/Library/Caches/Photo\ Mechanic/dyn.log" > ~/Library/Preferences/com.camerabits.PhotoMechanic/PMDebug.txt
3. Restart PM+
4. Scan to Catalog for a directory that has a HEIC image in it
5. See the task get stuck and just hang
6. Quit PM+
7. Move the HEIC image out of the directory
8. Restart PM+
9. See the tasks complete and go back to green
See attachment for the log file

BTW, the context menu "Include in Catalog" works just fine for the HEIC image

[Gary D.]

Im seeing the same behavior. 
Scanning to catalog works fine if there are no heic files and hangs if there are.   
And using "Include in Catalog" from the contact sheet works fine.

[Kirk Baker]

Gary and esambo,

I have a new build for you to try:

https://www.camerabits.com/download/PhotoMechanicPlusR4865_922e7715.dmg

I have tested "Scan to Catalog" with HEIC/HEIF images and it completes without failure.  Please let me know if it works for you.

Thanks,

-Kirk

[Gary D.]

Hi Kirk

Ran though several scans this morning and no hangs at all.

I removed that debug file and I'll be back to working with my catalogs.



Thanks
Gary