File reading behavior in large contact sheets

[jkm]

Thank you so, so much!!

[jkm]

I have added just one folder and subfolders from this year to my catalog now, so that it will be snappy when I open it instead of waiting a while to get loaded.

And as of last night everything was finished scanning and totally ready and fine.

But now, upon opening PM+ today, I just noticed that apparently everything I already added to the catalog from last year is now being scanned again, despite not being modified at all.

Why has the catalog not been kept up to date by PM, despite all the data being inside the same catalog folder and all the paths the same?

Everything already in the catalog was already scanned and stored when I added it from photos taken last year in a very long process already, and nothing has been added to the catalog since then except this one folder and subfolders that I added last night, from my hope to make opening the folder fast when I opened it today.

But now all that happened is when I opened PM today (in regular folder view, not even catalog view) it's just re-cataloging all million photos that I added from last year, that have all be unchanged since then.

If it needed to be kept up to date, why wasn't it doing that when I was using PM every day this whole year, instead of just waiting to do it now when I least can bear the delay?

I'm afraid I don't understand the benefit of the catalog for speed, if everything I already stored in it needs to be updated again despite not being modified since it was first added, and/or if nothing stored in it has been kept up to date automatically while I used PM.

Why is everything I added and scanned from last year all being scanned again now, and apparently every time I add one more folder to it like I did last night?

The catalog is supposed to make using PM+ go faster, correct?

By not needing to any more scanning once something has been added to the catalog and not modified since?

But it is definitely not functioning in that way.

Just adding one more folder from a shoot to it last night is now causing the entire million-photo existing catalog to be rescanned and recreated when I don't have the time and I have a due date coming up for the 250th anniversary of the Marine Corps photoshoot.

My computer is the same, my drive name/mount point is the same, etc.

[jkm]

I have not deleted, cleared, moved, or even tried to attempt anything that might cause this to happen.

It's just doing this automatically when I opened PM today. It did not do this last night when I added the new folder. Last night it said the catalog was finished and up to date. It took max 30 minutes, I think much less, to add the folder to the catalog last night, so as of then I was extremely impressed and looking forward to productivity today.

[jkm]

I assume that the "horse is already out of the barn" and it's too late to stop the process and revert back to the catalog that was fully stored and saved of those photos from last year?

Is it maybe a version update that is causing the whole catalog to be dumped and recreated?

I haven't done anything except right click on one folder last night and choose Include when scanning to catalog last night after finishing everything up in PM for the night, and then when it was totally finished and idle, quitting the program and going to bed.

Then coming back to it today and opening the same folder to rate/organize more photos, only to see a "Catalog Metadata Gathering" task popup in the lower left while my disks began crunching away with no input from me.

[jkm]

Once it's up to date, I think it will be the most amazing thing imaginable, though.

So I am so, so grateful for your replies to help me realize I haven't even been using the catalog perks yet.

Which just shows how fast PM is already even without the catalog caching speed.

[Kirk Baker]

Were all of the images always available over time?  Or were they on a drive that was disconnected for a long time?  The catalog doesn't constantly rescan for changes.  It will notice they've changed if you browse a folder that contains them.

I expect that something changed in the images file system data (size or modification date) or that the metadata was changed at some point.

This new folder that you added, was it anywhere near the folder hierarchy that the other million photos are in?

It is not possible to stop the metadata updates via the UI.  If you quit and resume, they'll pick up where they left off.

The purpose of the Catalog system isn't particularly for boosting speed.  It's purpose is to help you find your images based on their metadata and create virtual collections to better organize your images without having to change them (where they're located, what their filename is, etc.)

-Kirk

[jkm]

The only thing I can think of is that while rendering raw files from DxO Photo Lab, and rendering previews for my website at the same time, apparently my laptop ran out of watts (despite being plugged in to big Thunderbolt power sources from both sides) and lost connection to the drive yesterday and then it instantly reconnected.

I did a Disk first aid immediately as always when this happens, and the disk checked out fine. Then I just reopened the folder I was in in PM and kept going.

(Apple is horrible for making their plugged-in laptops barely have any bus power, and then shutting off devices if it thinks there's not enough power.)

But all of that is pretty normal, and that should have not triggered any kind of catalog update.

I never had a catalog update happen when I launched PM before, and similar things have happened in the past.

I've just been editing more raw files today, but I need to get back to PM.

Is there any harm in continuing culling/rating photos in a contact sheet in PM while it does the catalog metadata gathering process? Which still has 1.5 days left?

I appreciate all your help so much.

[jkm]

Is there any way to make the catalog things not run when I am actively using PM?

Or at least to set them to run only at night while I sleep if it can't auto detect when I am using it?

Update: I found where to stop the catalog processes and stopped them. Hopefully I didn't ruin something or make it need to start from the beginning again. I just couldn't deal with the half second pause after rating each photo before it moved to the next one.

[jkm]

I've indexed the folder of 22,000 photos (I think), and I was done with all the dance teams who wanted photos, then another team wanted photos.

But PM is still acting almost unresponsive in trying to rate/cull photos for that team, even 15 minutes after opening the folder.

My understanding is that PM doesn't need to read through all 22,000 photos again after indexing the photos into the PM6+ database, correct?

So why is it still doing that? Is that a bug?

Even without the database, it makes sense that PM should deprioritize reading photos beyond a "bubble" of chronological files around where I am currently culling, unless I am not actively typing keystrokes/ratings into the computer.

Whether in a big folder of 22,000 or a small one of just 1,000.

I assumed this was always what PM did to go fast anyway, since the first day I used it.

But if PM does not do it already, and now implemented this "bubble" of prioritized file reading behavior, just think of how much faster it could even be.

This would surely increase the speed by a big factor, like not just being the fastest program, but the fastest program by a factor of 10 or so.

[Kirk Baker]

How are you opening the folder of 22k images?  From a query in the Catalog interface?  (Search, Browse, Filter). Or are you opening the folder as a Contact Sheet?  (Favorites, Navigator, drag and drop, Open)

If you're doing the latter, rather than the former, you're not using the Catalog database at all.

-Kirk

[jkm]

I'm just opening the folder of photos like normal in PM.

I guess I need to do the query thing next time. But can I cull photos at full viewing size of the original files in that mode?

I still think it would be an incredible feature add-on to have PM read files only in a "bubble" around the current photo being acted upon when someone is culling photos in a folder.

The next 5 or so photos could be pre-loaded, pre-rendered to pop on the screen instantaneously, and so could the 5 or so before the current one being viewed.

And then after that, once any dead time happens beyond the reading of those 11 photos, other photos from the folder could be read further for whatever purpose.

The delay I am experiencing is not the delay in opening the folder, which is less than 5 seconds now.

It's just a delay in moving to the next photo after a rating key is pressed, which is usually instant, but instead it is about 0.5 to 0.75 seconds for the first 15 minutes or so of working on anything within the folder of 22,000 photos.

But once "whatever it is" is finished, then working within the folder of 22,000 becomes snappy again.

Not sure why whatever it is has to be done again and again every time I open the folder, because the sort cache is saved, and previews are saved, and I'm just going through the photos linearly one at a time in chronological order.

Next time I can try the catalog query idea, but I am basically just scrolling to the time on the program when a dance team performed, out of 35 teams in one of the biggest showcases in the state. So I'm not sure exactly how I would query for those photos, as I would only be scrolling to un-culled photos that haven't been marked or culled yet in any way.

The folder itself is part of what's tagging these photos with their appropriate unique ID in the news agency system, so I can't just start by moving each team to a different folder.

The teams have already been time-stamped for their performance times in an external news agency database system, but that system isn't a part of the catalog system in PM of course. In the news agency system, I can get to each group of photos just by clicking on one button for each team, but that doesn't show me the files I can rate, just a web page of the photos, not any sort of app that I can use to cull and rate the photos.

Each photo is already assigned a folder and a unique ID per photo by a system separate from any culling I do.

But rating photos is still up to individual people such as me.

[Kirk Baker]

I'm just opening the folder of photos like normal in PM.

Then you're getting no benefit from the catalog system.

I guess I need to do the query thing next time. But can I cull photos at full viewing size of the original files in that mode?

As long as the files are locally available, yes.

I still think it would be an incredible feature add-on to have PM read files only in a "bubble" around the current photo being acted upon when someone is culling photos in a folder.

The next 5 or so photos could be pre-loaded, pre-rendered to pop on the screen instantaneously, and so could the 5 or so before the current one being viewed.

This is almost exactly what the caching system does for the Preview window.

And then after that, once any dead time happens beyond the reading of those 11 photos, other photos from the folder could be read further for whatever purpose.

I'm not sure what that other purpose would be, but it doesn't do anything like that.

The delay I am experiencing is not the delay in opening the folder, which is less than 5 seconds now.

Good. Then either the sort cache has the items it needs to do your sort, or you're sorting by Modification Time or Filename which is fast even when there is no sort cache data.

It's just a delay in moving to the next photo after a rating key is pressed, which is usually instant, but instead it is about 0.5 to 0.75 seconds for the first 15 minutes or so of working on anything within the folder of 22,000 photos.

PM was not designed to handle much more than a few thousand photos in a single folder (or multiple folders opened together), but on today's systems with lots of RAM and fast SSDs, it appears to be coping.

But once "whatever it is" is finished, then working within the folder of 22,000 becomes snappy again.

OK.

Not sure why whatever it is has to be done again and again every time I open the folder, because the sort cache is saved, and previews are saved, and I'm just going through the photos linearly one at a time in chronological order.

I have no idea what would make setting a tag, rating, color label, etc. slower while you're first in a contact sheet other than the cache being busy generating thumbnails and previews, but I do know that in order to write to an image safely and not cause image corruption, the thread that updates the image must gain an exclusive lock on the file, then the file is modified, and then other things in PM can access the file.  If a cache worker thread is actively reading the file, then the update must wait until it is able to gain its exclusive lock.

Next time I can try the catalog query idea, but I am basically just scrolling to the time on the program when a dance team performed, out of 35 teams in one of the biggest showcases in the state. So I'm not sure exactly how I would query for those photos, as I would only be scrolling to un-culled photos that haven't been marked or culled yet in any way.

You could search for the capture time being between two known times and then only the images for that part of the program will be seen.

-Kirk

[jkm]

Thank you so much for your awesome help! Very much inspiring and above and beyond the call of duty!!