I still can't synchronise PM+ to reflect my folders

[davidgordon]

It's been years and I still can't get PM+ to reflect my folders.

My workflow involves ingesting files to a "Camera Dump" folder. Once I'm done processing they're moved to their to permanent folder on my archive hard drive. That move is initiated outwith PM+, my trouble is updating PM+ to reflect the move.

Today I have no folders in my Camera Dump. PM+ disagrees and sees many. These folders have long since moved to my archive. PM+ knows this and I can browse to them there.

So I've done a Full Catalog Sync and tried to limit the sync to the Camera Dump folder. In Collections I have a few 'missing' images, pictures I know have been deleted. But also there are 'online' images. I can't reveal these in Finder, only Collections. They all have green 'online' circles. These images should be marked as missing?

So these images - which do / do not exist - are online but I can't get PM+ to show me where.

Someone share me their workflow for moving pictures through PM+ from ingest to archive.

[Kirk Baker]

David,

My workflow involves ingesting files to a "Camera Dump" folder. Once I'm done processing they're moved to their to permanent folder on my archive hard drive. That move is initiated outwith PM+, my trouble is updating PM+ to reflect the move.

Today I have no folders in my Camera Dump. PM+ disagrees and sees many. These folders have long since moved to my archive. PM+ knows this and I can browse to them there.

So I've done a Full Catalog Sync and tried to limit the sync to the Camera Dump folder.

I suggest doing the sync on the locations where the images now exist.  Catalog Sync won't remove items it cannot find in the last known locations.  If, after doing a sync on the new locations, you have missing items from the Camera Dump folder, select them and remove them from the Catalog.

In Collections I have a few 'missing' images, pictures I know have been deleted. But also there are 'online' images. I can't reveal these in Finder, only Collections. They all have green 'online' circles. These images should be marked as missing?

What is the path of one of the green items that won't allow you to show the item in the Finder?

-Kirk

[davidgordon]

I suggest doing the sync on the locations where the images now exist.

OK, I've done that. Here's what PM+ reported, if of importance.

Code: [Select]

Starting Sync for Catalog: David Gordon Archive
Beginning scan for missing files.
Missing files scan failed: query_command_stream failed: ["query_command_stream: RuntimeError/Catalog \"David Gordon Archive\" - database connection closed"]
Beginning scan for files with newer metadata.
Total files checked: 141800
Total metadata updates queued: 0
Total missing preview proxies queued: 1
Some new files were added as expected and the final report

Code: [Select]

• • • Sync summary (Full Sync) • • •

    Catalog: David Gordon Archive - 0 errors
    Total files in catalog: 0
    Files missing: 0
    Files offline: 0
    Files scanned in folders: 9014
    New files added to catalog: 1875
    Files updated in catalog: 0
    Missing preview proxies to generate: 1
I've also synced the Camera Dump folder. I expected that to create collections with all the 'missing' files. But it doesn't.

What is the path of one of the green items that won't allow you to show the item in the Finder?

These are the files which were in the Camera Dump folder and have now been moved to the main archive folders. The path is ~/Pictures/Camera Dump according to PM+ but they are not there. Have I confused things by moving the files? The files do exist and PM+ knows they are in the main archive folder and can show me the file in Finder if asked. But at the same time PM+ says the same images are in the Camera Dump folder, with a green 'on-line' icon but can't show them to me via Finder. Because they're not there. And this despite doing a sync.

I guess I could just remove the Camera Dump folder files from PM+. But I should have to, should I? There's a danger I'll remove files which I shouldn't. They ought to be listed as 'Missing' in Collections, as I understand things.

Thanks for any further advice

[Bob Russell]

Hi David,

  The failed missing file scan message from above, is not a good sign. It's indicating the database can't be properly access. It could be easier and probably faster to create a new catalog for just the images on your archived drive. I'm suspecting adding 141800 files to a new catalog could be done in one day.
 
  If you want to try to fix your existing catalog, then I would try using the Forget option on the Catalog Management page. Then use the Locate Existing Catalog option and select the same catalog again. Next you will need to run the Reintegrate Forgotten Catalog utility in the Maintenance section.  When that is done, I would display all images in the catalog. Just go to the Organizer panel's Search Tab. In the Quick Search field, just enter a period and then hit Return. Next go to the Edit menu's Select Catalog Status and select the Source File Unavailable option.  This should select all of the offline images.  Now you can right click any of the selected images and choose the Remove from Catalog option.
 
  To help minimize these problems in the future, I recommend waiting until your images are in their final destination folder on your archive drive, before including them in the catalog.

Best regards,

--Bob

[davidgordon]

Bob

Thanks for the advice. I've tried your second suggestion to fix the catalog, that hasn't changed anything. I'd prefer not to have to create a new catalog from scratch as my experience - and that of others I know - is that will take a very long time. My attempt to fix the catalog took half an hour, including breakfast.

I followed your instructions:

1 - Forget option on the Catalog Management page.
2 - Locate Existing Catalog option and select the same catalog again.
3 - Run the Reintegrate Forgotten Catalog utility in the Maintenance section.
4 - Display all images in the catalog. (Go to the Organizer panel's Search Tab. In the Quick Search field, just enter a period and then hit Return).
5 - In the Edit menu's Select Catalog Status, select the Source File Unavailable option.  This should select all of the offline images.
6 - Right click any of the selected images and choose the Remove from Catalog option.

There are no selected images.

To me this is because my ~/Pictures/Camera Dump folder is on-line and always will be. So browsing to that folder using the Browse tab shows the images as on-line. Remember these are images which do not exist in that folder.

So no further on other than having the nuclear option.

  To help minimize these problems in the future, I recommend waiting until your images are in their final destination folder on your archive drive, before including them in the catalog.

Sound advice. I think the problem stems from having done a full catalog sync which would have added those files when they were in that folder. My expectation - and tell me if this is wrong and or unreasonable - is that the next time I synced the catalog PM+ would see those files are no longer in that folder and mark them as 'missing'. Why isn't that happening?

Thanks

[DavidHoffmanuk]

This may be too simplistic or something you have already tried but can you just remove the ghost images from the catalog and then add them back from their real location? They'd disappear from any collections though there may be (slow) ways to save any important sets but I don't see any other issues.
David

[davidgordon]

This may be too simplistic or something you have already tried but can you just remove the ghost images from the catalog and then add them back from their real location?

Never one for taking the easy option... Yes, I could. But actually these 'ghost' images show in PM+ with a green 'on-line' icon. So part of my trouble is I want to - need to  - trust the database. Yes, I could go through my Camera Dump folder and check every subfolder, remove those from PM+ which aren't really there. But why have a dog and bark yourself? How many other folders are actually out-of-sync?

Lightroom makes this an easy single control-click command - select the folder and 'Synchronise Folder...'. Done. Even following the PM+ four click plus navigating to the required folder I still don't have synchronisation.

Either I don't understand the concept of PM+ or haven't read the (quite limited) instructions correctly. Perhaps there are things which the new funding model will address. The actual easy option is to use PM for what PM does best - metadata. Then forget PM+ and use Lightroom or another solution for DAM.

David

[ahoward]

I think the problem stems from having done a full catalog sync which would have added those files when they were in that folder. My expectation - and tell me if this is wrong and or unreasonable - is that the next time I synced the catalog PM+ would see those files are no longer in that folder and mark them as 'missing'. Why isn't that happening?

It sounds like you have a good understanding of what should be happening. I'm wondering if there is a .pmshare file left behind from where these images originally were? This sounds like a similar situation to what DavidHoffmanUK was experiencing that was apparently mostly resolved when he located a one of these files: https://forums.camerabits.com/index.php?topic=15946.0

[DavidHoffmanuk]

A bit more detail may be helpful.

I've moved from an old Mac Pro to a new Mac Studio. The Mac Pro had an internal NVMe drive (Mercury) with my PM+ catalogs and about 30k images. There are around 180k images on other drives.

The Studio has the contents of the old NVMe drive in a subfolder of a new external NVMe drive named Mercury.

I found the hidden .pmshare file at the top level of the old Mercury drive and copied it to the *top level* of the new drive that contained the Mercury folder and that has largely resolved the problem of seeing ghost drives that seems similar to the problem that David Gordon describes.

I was surprised to see that the .pmshare file shows a last modified date in 2021.

However, I'm still seeing one ghost drive for Perky. Perky was an internal drive on the old Mac & is now a subfolder on an external drive on the new Mac. I'm seeing it shown in both locations in the volume tree in the browse tab but the entry showing it as a separate drive with 44 files fails to show any when clicked. It gives a message 'Select one or more items in the Browse tree to view items here.' This doesn't seem to be causing any problems though any such error is worrying. I've attached a screenshot.

[davidgordon]

I'm wondering if there is a .pmshare file left behind from where these images originally were?

The images were originally in ~/Pictures/CameraDump which is currently empty and has no hidden .pmshare file.
I do have number of .pmshare files in various places including the current correct location of these files in the volume Raw Image Files/2020s.

What is the purpose of the .pmshare file?

It's hard to say if my trouble is the same as David Hoffman's. I haven't had any problem moving files between hard drives (something I recently did and PM+'s Browse shows me two volumes where there was only one and has the correct files in the correct location).

So one final bash at syncing and I've come across another roadblock. I've accidentally added 'Include Existing Paths From Catalog' to the 'Folders to Include in Sync'. Now every single folder is included and I can't remove them all. Other than select a few at a time and there are hundreds of folders. How do you clear this selection?

Instead let's just remove the duplicate folders from PM+. I'm in Browse and selecting on of the folders which is or is not in ~/Pictures/Camera Dump. Control-Click bringup the options including 'Remove Folder'. That should do it. But no, it doesn't.

As I say, I really don't want to have to rebuild my catalogue from scratch. What are my options?

Thanks

[ahoward]

So one final bash at syncing and I've come across another roadblock. I've accidentally added 'Include Existing Paths From Catalog' to the 'Folders to Include in Sync'. Now every single folder is included and I can't remove them all. Other than select a few at a time and there are hundreds of folders. How do you clear this selection?

Click on one of the lines, hold the Shift key and click another line, and all lines in between will be selected, and clicking 'Remove' will remove all selected.

I'm not sure what to tell you to do in regards to the main problem. I'd be happy to work with you directly to see if there's anything we are missing.

[davidgordon]

Click on one of the lines, hold the Shift key and click another line, and all lines in between will be selected, and clicking 'Remove' will remove all selected.

Yes, thanks, the standard macOS way. I think my frustration was caused by my having a very long list within a very small window which only shows a dozen or so lines. I was shift-clicking but scrolling was slow. Anyway, I've walked away, returned and cleared the list. Maybe a 'Clear All' button to help clear that selection would be handy?

I've now done (another) Catalog Sync for the Camera Dump folder. This has resulted in one folder within that folder correctly being marked as having missing files. Or at least the 'Missing' Collection shows the files. I've removed all the files and that has resulted in the folder disappearing from Camera Dump. These images also exist in their new archive location. and PM+ knows this. This is all correct and expected. But still PM+ will not see that a number of other folders have been moved and are not in the Camera Dump folder.

What is different about the older images stuck (according to PM+) in the Camera Dump folder and the ones which I was able to move through as expected?

I'm not sure what to tell you to do in regards to the main problem. I'd be happy to work with you directly to see if there's anything we are missing.

Happy to work with you. Let me know if some screen grabs or logs would be of use but I'd rather provide these directly.

[ahoward]

Happy to work with you. Let me know if some screen grabs or logs would be of use but I'd rather provide these directly.

Generally in these situations I find that we cover a lot more ground during a phone call. We are available from 9am to 5pm Pacific Time, Monday-Friday (our number is 1-503-547-2800). If you'd prefer to schedule a specific time, or you need us to call you, send an email to support@camerabits.com and we can work something out.