Moved catalogue, but original location in 'Documents' is still being written to.

[JSW]

I've got over half a million photos and my catalog is 180Gb and still growing. I rebooted my PC and it took 20 mins to log in because the catalogue is in the user area "C:\Users\<user>\Documents\Photo Mechanic Catalogs". Really daft idea to put it there. I've moved to to the root of c:  "C:\Photo Mechanic Catalogs" and now I can log into Windows in 20 secs again.

I've used 'Catalog Management' and using 'Locate Existing Catalog', pointed the it at "C:\Photo Mechanic Catalogs".  However, now I'm scanning new photos, the catalog.pmdb is updating in "C:\Users\<user>\Documents\Photo Mechanic Catalogs".  It's 10Gb and growing. If I move the catalogue, I expect everything to move and stay moved. I don't want a large file in my personal profile, as it can cause issues starting up & shutting down Windows and is more liable to corrupt.

Am I doing something wrong, ie is there a way to permanently move the catalogue, or is this a program bug that it always writes to "C:\Users\<user>\Documents\Photo Mechanic Catalogs"? If it's a bug, or even a bad design decision, could this please be fixed. 

[Kirk Baker]

I've got over half a million photos and my catalog is 180Gb and still growing. I rebooted my PC and it took 20 mins to log in because the catalogue is in the user area "C:\Users\<user>\Documents\Photo Mechanic Catalogs". Really daft idea to put it there. I've moved to to the root of c:  "C:\Photo Mechanic Catalogs" and now I can log into Windows in 20 secs again.

I've used 'Catalog Management' and using 'Locate Existing Catalog', pointed the it at "C:\Photo Mechanic Catalogs".  However, now I'm scanning new photos, the catalog.pmdb is updating in "C:\Users\<user>\Documents\Photo Mechanic Catalogs".  It's 10Gb and growing. If I move the catalogue, I expect everything to move and stay moved. I don't want a large file in my personal profile, as it can cause issues starting up & shutting down Windows and is more liable to corrupt.

Am I doing something wrong, ie is there a way to permanently move the catalogue, or is this a program bug that it always writes to "C:\Users\<user>\Documents\Photo Mechanic Catalogs"? If it's a bug, or even a bad design decision, could this please be fixed.

Could you take a screen shot of the Catalog Management window showing the expanded interface in the Local Catalogs section?  Also, could you send us your log?  I'd like to see In Photo Mechanic Plus go to the Help menu and choose "Reveal Support Data..." and moments later a Windows Explorer window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file and your screen shot in JPEG format, please.

As for long startup times when a large amount of files/folders are in the Documents folder in the user account, I have not experienced this before.  I did some searches and didn't find much talking about that issue.  Is there some documentation you can point me to that explains what is going on?

Thanks,

-Kirk

[JSW]

It's changed the location back! I want the catalog permanently on the root of c:\, not in the user area.

A few links:
https://4sysops.com/archives/troubleshoot-slow-logon-part-1-profile-size/
https://www.tricerat.com/blog/top-four-reasons-for-slow-logon-times
https://www.watchingthenet.com/is-your-bloated-user-profile-slowing-windows-down.html

When I was doing tech support we had a lot of problems with users putting too much in their personal profiles, which were loaded then Windows starts up. We had lots of corruption issues as well as very long load times. It's less of a problem these days with faster SSDs, but 180Gb in a user profile will certainly do it! Databases should never be in profiles where there is an alternative.

I'd also like control how large the thumbnails are.

 

[Kirk Baker]

It's changed the location back! I want the catalog permanently on the root of c:\, not in the user area.

We don't know why your catalog location reverted.  As a workaround we suggest doing the following:

1. Create a new temporary dummy catalog somewhere, and make it the only one checked for search/target.
2. Quit PM Plus, and move your real catalog where you want it to be.
3. Start PM Plus, and use Locate Existing Catalog to reacquire the catalog that was moved to the new location.
4. Delete the temporary dummy catalog from within PM Plus.

A few links:
https://4sysops.com/archives/troubleshoot-slow-logon-part-1-profile-size/
https://www.tricerat.com/blog/top-four-reasons-for-slow-logon-times
https://www.watchingthenet.com/is-your-bloated-user-profile-slowing-windows-down.html

When I was doing tech support we had a lot of problems with users putting too much in their personal profiles, which were loaded then Windows starts up. We had lots of corruption issues as well as very long load times. It's less of a problem these days with faster SSDs, but 180Gb in a user profile will certainly do it! Databases should never be in profiles where there is an alternative.

Thank you for the links.  Can you suggest a better default location for catalogs?

I'd also like control how large the thumbnails are.

We'll be adding that in a future build of Photo Mechanic Plus.

-Kirk

[JSW]

We don't know why your catalog location reverted.  As a workaround we suggest doing the following:

Have done that and am now re-scanning the folders.

Thank you for the links.  Can you suggest a better default location for catalogs?

I'm a little rusty since I was using my MCSE (Microsoft Certified System Engineer), but for progams I've been involved in development, we've gone with "Create a top-level folder that will serve as the root storage folder for all user-created data (for example, C:\Data)." https://www.netwrix.com/ntfs_permissions_management.html

Ie. I've changed the location of mine to C:\Photo Mechanic Catalogs\Default Catalog. It's a good place for a large database to live.

I'd also like control how large the thumbnails are.

We'll be adding that in a future build of Photo Mechanic Plus.

Good. Appreciated. I'm curious just how large the catalog and previews of a 1 million photo collection are. I assume you've tested it and have the figures? An indication of just how large it's going to be would be good, as I was rather surprised just how much space it would take up. I appreciate that as a professional photographer I have more photos than many. Then again, I'm assuming your clients want PM Plus, exactly because they do have very large photo sets. 

[Kirk Baker]

I'd also like control how large the thumbnails are.

We'll be adding that in a future build of Photo Mechanic Plus.

Good. Appreciated. I'm curious just how large the catalog and previews of a 1 million photo collection are. I assume you've tested it and have the figures? An indication of just how large it's going to be would be good, as I was rather surprised just how much space it would take up. I appreciate that as a professional photographer I have more photos than many. Then again, I'm assuming your clients want PM Plus, exactly because they do have very large photo sets.

The size of the proxies will vary based on the complexity of the source images.  With the fixed image size they vary between 120KB to 800KB.  If we split that down the middle to pick an average and multiply that by a million, it's gonna be in the 3.3TB range.

-Kirk

[Bill Kelly]

The size of the proxies will vary based on the complexity of the source images.  With the fixed image size they vary between 120KB to 800KB.  If we split that down the middle to pick an average and multiply that by a million, it's gonna be in the 3.3TB range.

In practice we're seeing closer to the 400 to 500 GB range for the proxies folder on our million-item catalog.

But it could indeed vary greatly. The smallest and largest of the proxy files in our test catalog are both sports photos. One is almost a megabyte (1008 KB) while the other is merely 119 KB. The large one has an extremely complex background; the smaller has a simple background (uniformly gray sky.)

- 1008K proxies/de7/82180353b5622375535a3cbd6465b7f9f6ea09a84e39eb69c4e6a79f40819#st_000000282_jpg#01600.jpg
-  119K proxies/f78/70611b3c4eaac80a6b65126fd02b90cfaddbf5169c61a19064a4f996f0a27#st_000006771_jpg#01600.jpg

[JSW]

Thanks for the detailed stats. I have a 512Gb NVME drive and am a bit reluctant to take 200Gb of it for Photo Mechanic. I've had to take some programs off it to make room. I believe it's better to have the database on the solid state drive for performance reasons however. Correct? Do you have any stats as to how much longer searches take on spinning hard drives?

[Bill Kelly]

Do you have any stats as to how much longer searches take on spinning hard drives?

I don't have numbers available, though my recollection is the performance difference could be pretty significant.

One possible workaround (that I wouldn't recommend to everyone, but I gather you may be comfortable with the filesystem) might be to offload the "proxies" directory within your catalog folder to a larger, slower drive. (By way of creating a directory junction with mklink /J)

Your catalog folder (with its catalog.pmdb database file) would remain on your fast NVME drive.

e.g.

Directory of P:\catalog_test_files\cat-temp4

12/04/2020  12:15 PM       187,523,072 catalog.pmdb
10/29/2020  09:28 AM            28,672 catalog_undo.pmdb
12/04/2020  12:15 PM    <JUNCTION>     proxies [S:\cat-temp4-alt\proxies]

[JSW]

Yes, comfortable doing this, though it's a faff and a hack (ie not as the program is designed).

Could this be added as an advanced option in catalog settings? Ie able to select where to store the database and also where to store the image thumbnails? Not everyone has huge amounts of storage on fast sold state drives available.

My 512Gb NVME system drive actually filled up during the cataloging and I had 100k available. I ran into all kinds of problems trying to sort it. It still says there is 3 days more worth of metadata to gather. My computer has been on solidly for around a 4 days. I would have expected the cataloging to have finished by now.

It might be a good idea to check during cataloging that the system still has at least (say) 500Mb of space available and then stop cataloging and pop up a message warning of low storage and offering the user the chance to move the database/thumbnail locations. 

I'm sure I'm not the only one going to be caught out by this issue.

[JSW]

So, I've had PMP for 14 days. For the past week it's been saying that it has 2.6 days (or thereabouts) more updating to do, but it just keeps churning and has not finished yet.
There's some more weird stuff: (Photo attached)

In Tasks:

Catalog Metadata Scanning:
Scanning complete

Catalog Metadata gathering:
About 1062640 item remaining (5/sec, est 2.5 days) (This number reduces slowly)

Catalog Metadata Updates
(Flicks between 'idle' and 'updating' every few seconds)

Catalog Image preview Generation
155575 batches remaining (This number is going up?? Why?)

Catalog Image preview copying
Copying to 0 of 1 catalogs (some catalogs offline), 254509 items remaining (This number doesn't change and I only have 1 catalog, so how can it be offline?)

What is going on here?

[Bill Kelly]

What is going on here?

Forensically, I think the system became confused when the Default Catalog was initially moved, and, upon not finding it, a new Default Catalog was created automatically.

I believe some of the lingering jobs (indicated in your screenshot as "some catalogs offline") must have been queued to that new, unwanted Default Catalog (which may have since been deleted.)

On the development side, we'll need to consider how to make the software more resilient in this scenario.

As to how to get things back on track, I'd recommend the following:

1. In the Catalog Status window, click the "Stop Catalog Import Tasks" and "Stop Preview Generation Tasks" buttons. The various Catalog Tasks should settle to an Idle state. (If tasks do not return to Idle, let me know, and we can try a more low-level reset.)

2. Use the "Sync Catalogs…" feature to continue building your catalog. In the Catalog Sync window, click the name of your catalog to edit its settings. Verify the "Folders to Include in Sync" paths look correct, or edit as needed. Enable the catalog for Sync via its checkbox in the Catalogs to Sync pane. Select "Quick" for Sync Mode. Start the Sync.

This should resume building your catalog from where it left off.

Hope this helps,

Bill

[JSW]

I followed those instructions. After running it there was still a red mark by Catalog Metadata Updates. I went to Catalog status and there was one item (Unknown catalog) in 'Unreachable or missing Catalogs'. I clicked 'Forget' and the status went green. I then went to 'Scan to Catalog, checked the 'Folders to scan' were correct and clicked 'Start'. It found 435320 photos to scan and is going through them a lot faster. I don't understand why the 'Catalog Image Preview Generation' batches remaining is going up though.

(Minor issue that the launching the Catalog Status window for the first time gives cut off text and buttons (as shown in the screenshot below).

[Bill Kelly]

I don't understand why the 'Catalog Image Preview Generation' batches remaining is going up though.

Proxy generation batches will accumulate during the Metadata Gathering phase.

We used to allow proxy gen to occur in parallel, but found that "proxy gen seems to slow down metadata gathering in a non-linear way."

So proxy gen is deferred until Metadata Gathering completes. (If users wanted to try re-enabling this parallelism in case it performed OK on their systems, we could perhaps add a checkbox to Catalog Prefs.)


-Bill

[JSW]

Doh, of course. The text metadata can be gathered at a far faster pace than the complex image generation. Watching the image generation queue carefully, I can occasionally see it going down. So, if the system scans a large folder, the metadata will be gathered quickly and will move onto scan the next folder while the image generation is trundling away. If there's a small folder of images scanned, the image generation can catch up and the queue will drop. Makes sense to have the metadata gathering as a separate process to the image generation.

You're don't seem to be using my 9900K processor very effectively though. The total utilisation hovers around 7-9% (screenshot attached). That's even with me manually changing the priority of the threads to 'high' in task manager. I would have hoped to see the CPU stressed and doing the job a lot faster. I know that queueing jpg calculations is not as efficient as video encoding (where my CPU often sits at 100%, which is great), but is there a way to get the thumbnail generation running faster?