Very slow content updating - edited

[DavidHoffmanuk]

I've removed about 200k image files from my catalog and added about 120k. I've done a full sync' and a 'scan to catalog' for the new ones. The Tasks panel is showing:
'Catalog Metadata Gathering - preparing
'Catalog Metadata  Updates' - 4740 batches 0/sec 15.9 days - see attached image.

It's been showing this for about 6 hours without any change.
Activity monitor shows 4 x pm-task using ~98%, ~13%, 0% & 0%
I'm running a Mac Pro 5,1 10.14.6. There's 56GB ram & about 20GB is currently free.
PM+ is Version 6.0, build 6499 (b8c6a0f)

[Below edited re what's stored where 17:39:26 21 August 2022].
The catalog and most image files are on an NVME drive with about 80k RAW files on slower spinning disks. The app is on an ssd.

Is this to be expected? If not is there anything I can do to speed things up?

David Hoffman

[Kirk Baker]

David,

Is this to be expected? If not is there anything I can do to speed things up?

Definitely not expected, no.

I'd like to see your Photo Mechanic log.  In Photo Mechanic Plus go to the Help menu and choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

[DavidHoffmanuk]

I was hoping that would be your answer Kirk! I've attached the log.

I'll also turn on debug, quit and restart the Mac - but PM doesn't want to quit, I'm getting
"The Catalog system is taking awhile to stop. Continue waiting?"
and clicking Yes but it's been 5 minutes so I'll kill it. I expect this is the same problem.

David

[Kirk Baker]

David,

It looks like the Catalog server became unavailable.  Did restarting the app resume the tasks and begin making progress again?

-Kirk

[DavidHoffmanuk]

Hi Kirk

Restarting didn't help.

What's the 'Catalog server'? The catalog.pmdb is on an NVME drive that's always available and the app is on my startup ssd drive, again always available.

I tried to run a backup from before I made the changes to the image storage. I removed the folder with the pmdb files and replaced it with one from the backup but when I started PM+ the contact sheet was still showing the same 15.9 days etc as in the earlier screenshot so it seems that the pmdb file isn't where the problem lies. Are there other files, not in that folder (maybe in Library?) that I need to remove if I'm to get back to where I was a few days ago?

PM+ is central to my workflow so I had thought to begin a new catalog in case I can't restore the existing one but I can't run PM+ at all without it stalling with this problem.

David

[Kirk Baker]

Hi Kirk

Restarting didn't help.

What's the 'Catalog server'? The catalog.pmdb is on an NVME drive that's always available and the app is on my startup ssd drive, again always available.

The Catalog system is running in a different process on your system and Photo Mechanic Plus communicates with it via sockets (TCP/IP).  In that way, it is a server and Photo Mechanic Plus is a client.

I tried to run a backup from before I made the changes to the image storage. I removed the folder with the pmdb files and replaced it with one from the backup but when I started PM+ the contact sheet was still showing the same 15.9 days etc as in the earlier screenshot so it seems that the pmdb file isn't where the problem lies. Are there other files, not in that folder (maybe in Library?) that I need to remove if I'm to get back to where I was a few days ago?

The cat_state.pmdb file journals the batches of work left to complete.  It can be deleted and you could repeat the operations to begin the process again.

I suggest shutting down the application first before proceeding.  You can find the file at this location:

/Users/<your username>/Library/Application Support/Camera Bits, Inc/Photo Mechanic/catalog/state/cat_state.sq3

If you want, you can just rename it instead of deleting it should you want to keep it around.

-Kirk

[DavidHoffmanuk]

I renamed the cat_state.sq3 file and a new one was created when I started PM+ but when I opened a contact sheet my catalog was not shown, only a small test catalog. Catalog Management shows "failed to open catalog" but all the options under Maintenance are greyed out - screenshot attached. I wondered about trying to forget the catalog and then trying to reintegrate it but is that sensible?

Probably not connected but the window in the search tab is being drawn with the top missing - screenshot attached.

David

[Kirk Baker]

David,

I renamed the cat_state.sq3 file and a new one was created when I started PM+ but when I opened a contact sheet my catalog was not shown, only a small test catalog. Catalog Management shows "failed to open catalog" but all the options under Maintenance are greyed out - screenshot attached. I wondered about trying to forget the catalog and then trying to reintegrate it but is that sensible?

That would be the best course of action, but before you do that, can you send me an updated log, please?

Probably not connected but the window in the search tab is being drawn with the top missing - screenshot attached.

Dragging the horizontal divider between the top area and the tabs below, lower should make room for the additional content missing at the top.

-Kirk

[DavidHoffmanuk]

Hi Kirk

Log attached.

I never knew about that horizontal divider... Duh.

David

[Kirk Baker]

David,

Thanks.  The log indicates that it cannot find the catalog.  You may as well forget the catalog and locate it again, then do a reintegrate.

-Kirk

[DavidHoffmanuk]

Thanks Kirk. The forgetting worked but trying to Locate Existing Catalog only gives me this error:

open_catalog failed: ["open_catalog: RuntimeError/existing catalog not found: \"/Volumes/Mercury/PM6+ catalogs/All pics 2/catalog.pmdb\" attrs={\"create\"=>false, \"delete_existing\"=>false, \"machine_uuid\"=>\"9a174476-fe6a-4cd0-6721-1b26e4fd1691\", \"chan_idx\"=>0}"]

David

[DavidHoffmanuk]

I fished a backup copy out and have relocated that, it's reintegrating now. I'll let you know how it goes but probably tomorrow now.

David

[Kirk Baker]

David,

Thanks Kirk. The forgetting worked but trying to Locate Existing Catalog only gives me this error:

open_catalog failed: ["open_catalog: RuntimeError/existing catalog not found: \"/Volumes/Mercury/PM6+ catalogs/All pics 2/catalog.pmdb\" attrs={\"create\"=>false, \"delete_existing\"=>false, \"machine_uuid\"=>\"9a174476-fe6a-4cd0-6721-1b26e4fd1691\", \"chan_idx\"=>0}"]

Was there no catalog.pmdb file at that location?

-Kirk

[DavidHoffmanuk]

That was quick and easy! The backup copy is up and apparently running OK.

The previous copy died following a great deal of file & folder reorganising & deduping that had to be done outside of PM+.

After the reorganising I selected all missing files (about 200k I think) and made a collection with them where I sorted them by path to check that they were all from the places I expected. They were, so I selected all and clicked 'remove from catalog'. I planned to follow that with a full sync to add the moved files but that's when PM ground to a halt.

That's my preferred next step but, before I try it again, is there anything in that procedure that looks like it might cause a recurrence?

I've just seen your pmdb question - yes, there's a 5.62GB one there though it has a mod date of 2021.

David

[Kirk Baker]

David,

That was quick and easy! The backup copy is up and apparently running OK.

The previous copy died following a great deal of file & folder reorganising & deduping that had to be done outside of PM+.

After the reorganising I selected all missing files (about 200k I think) and made a collection with them where I sorted them by path to check that they were all from the places I expected. They were, so I selected all and clicked 'remove from catalog'. I planned to follow that with a full sync to add the moved files but that's when PM ground to a halt.

That's my preferred next step but, before I try it again, is there anything in that procedure that looks like it might cause a recurrence?

I don't expect any problems, but I don't know what caused the problem in the first place.  I suggest letting the remove operation complete fully (see that the tasks all become idle).  Then quit Photo Mechanic Plus and backup the db again, then do the full sync.  Normally I wouldn't suggest any of that but something is going on with the previous catalog and you may as well have a backup at a known good state.

I've just seen your pmdb question - yes, there's a 5.62GB one there though it has a mod date of 2021.

That's very strange.  Could a restore from a backup have occurred at some point during this process?

-Kirk