Maintaining/updating catalogs too difficult (depending...)

[bazography]

After spending some time with PM6+ I’d like to offer some input on how we go about maintaining/updating catalogs, which can be a very time-consuming process. I know that what I'm going to describe may not be an issue at all for some, but I’ve realized the process can be really easy or really difficult depending on how you have your image folders organized. If every folder you want to catalog is contained within a single main folder, then selecting just the one main folder to scan or rescan is simple. But what if there is one main folder, and inside that folder there are some folders and/or subfolders I want in Catalog X and some in Catalog Y? In that situation I’m finding catalog maintenance almost unworkable.

For example, assume that I have just added several dozen photos to various folders/subfolders in different locations, and now I need to update my catalogs. Unless I’m missing something, I see 2 choices. 1) For each folder I added photos to, I open a contact sheet for that folder, look for the new photos, select each photo, add them to the catalog, then repeat this for every other folder I’ve changed. Or 2) Open Scan To Catalog, locate a folder I’ve added photos to, add it to the “Folders To Scan” list, add any "folders to exclude" to the list, repeat this for each of the folders I’ve changed, then run the scan. And this needs to be done each and every time I add new images. Plus I actually have to remember which of the many folders I dropped new photos into, and hope I don’t miss any (yes, I've missed some). It's also all too easy to forget to check the right box and accidentally add images to the wrong catalog (done that too... lots).

Currently there is a single Scan To Catalog dialog with checkboxes to select which catalog to work with, and I think this is the root of all these difficulties. Instead, what if each catalog had a dedicated Scan To Catalog dialog? Within the “Manage Cataolgs” dialog, each catalog already has buttons for various tasks (Forget, Rename, Delete) so maybe each catalog could also have its own Scan To Catalog button as well. Clicking it would bring up that catalog’s very own dedicated Scan To Catalog dialog box that remembers all the folders previously scanned into that catalog, plus any exclusions you added etc. If you want to update Catalog X, just click on Catalog X’s Scan To Catalog button and the dialog box opens up ready to scan. And if you need to add a new folder to the list, you only need to do it once as it will be there for next time.

Also, I strongly believe this process needs to go one step further and become automated. I’d love to see an option to designate folders as “watched folders” so that dropping new images into any of those folders triggers an auto-scan. If I’ve added new photos to a cataloged folder, I will always want the new photos cataloged too, no question, so if you know you’re going to rescan 100% of the time anyway, it may as well be automatic. In fact, if I had “watched folders” set up, I can’t see any reason why I would have to manually scan anything ever again. It would be a huge time-saver. Plus, if everything inside the chosen folders was automatically cataloged, I would never have to worry about forgetting to update, forgetting to add one of the changed folders to the scan list, or forgetting to check off the right catalog.

Here's a quick mockup of what I was thinking. Clicking a "Scan To Catalog" button brings up the dialog box with everything set up for that catalog, ready to go...

[KeithRJ]

I like your thinking and I believe it would work well.

I think another option may be useful:  In the Ingest dialog there could be a Select List to select the catalog(s) to be updated with the ingested images.

[JD McDowell]

I agree as well. It's a cumbersome process as it currently stands.

If I create a few derivatives of a shot (for web, for print, etc.) I either have to manually select each of those images and add it to the catalog or rescan the entire thing (roughly 25k images currently) and wait for it to finish.

I like the fact that we can add or remove single images from a catalog, but for me I think it would be nice if Photo Mechanic+ was adding any shots I add anywhere within the catalog's defined folder structure as I go.

[bazography]

If I create a few derivatives of a shot (for web, for print, etc.) I either have to manually select each of those images and add it to the catalog or rescan the entire thing (roughly 25k images currently) and wait for it to finish.

Yes, for me too it's the derivatives that create the problem. My folder structure uses a main folder for each event, then subfolders with a code at the end signifying the various versions, such as the original RAW files, Edited TIFs, the JPGs I send to the client, etc.

>  My Main Archive
>>  20180608 Some Random Event
>>>  20180608 Some Random Event CLIENT JPG
>>>  20180608 Some Random Event EDITED
>>>  20180608 Some Random Event RAW
>>  20191031 Another Event
>>>  20191031 Another Event WEB JPG
>>>  20191031 Another Event EDITED
>>>  20191031 Another Event RAW

If I wanted every image in one big catalog it would be dead simple, I'd just have to scan the "My Main Archive" folder. But I want one catalog for all the finished images, and one catalog for the RAW files. That means I have to individually select each EDITED folder for scanning (hundreds), then repeat for each RAW folder. I fully expected that I would have to do this in order to set up and scan the catalog the first time. I did not expect that whenever these folders needed to be re-scanned, they would have to be manually selected all over again, every time. That's just not practical.

I realize that I could re-structure my archive to make this easier, but there are reasons I have it structured this way and changing it will just create other issues.

[KeithRJ]

Coming from an IT and Database background, I believe it would be dead simple to create a function to scan all folders already in the catalog.  Simply find all folders already scanned and then re-scan them.

This could be extended by showing a tree of all folders already in the catalog and providing a mechanism for the user to select folders to be re-scanned.  Not difficult to implement 

[carlseibert]

Funny thing - I just posted about keeping track of stuff we have discussed in the past as we near release. There was a thread or two about this topic.

Bazography - Another way to accomplish what you are seeking in the Scan to Catalog dialog would be with a Snapshot button, or a couple of them. (At the moment, I have a text file if directory paths that I paste from when I have to do a rescan.) I have pestered for such a Snapshot button.

I would like a configurable watched folders function because, for me, the process of adding assets is that I move folders into the various cataloged trees. It would be effortless if they just got scanned without my intervention.

And yes Keith - A "scan only for new and changed items" feature, to run manually or as a cron job is common enough. It would be good.

I do occasionally use the current method of committing assets in place in the folders where they reside. It's good, for example, if I want to commit something to a second catalog when it doesn't live in that catalog's directory tree. Let's say I have a catalog of cars and a catalog of people and I have a picture of a person with their car that has to live in one catalog's directory tree or the other. Perfect. But in terms of day-to-day workflow, that's an outlier. Generally, the catalogs just have their own directories and need to absorb new assets as they land there. So mostly, I rescan with "Scan to Catalog".

 

 

[bazography]

Bazography - Another way to accomplish what you are seeking in the Scan to Catalog dialog would be with a Snapshot button, or a couple of them. (At the moment, I have a text file if directory paths that I paste from when I have to do a rescan.) I have pestered for such a Snapshot button.

Clever workaround, thanks for that. I thought about Snapshots... I love them and use them everywhere. In this scenario though, you'd wouldn't be able to just create a Snapshot and reuse it every time. You'd have to save a new Snapshot each time you added a new folder to the "Scan To Catalog" list, so it becomes a 2-step process... eg: add new folder to the "Scan To Catalog" list, then save the new Snapshot. Ultimately, having "Scan To Catalog" remember the previously scanned folders would require much less intervention.

Completely agree with your comments on watched folders... no intervention is even better.

[carlseibert]

Scan to catalog is recursive, so normally there would be no need to add to the path.

Let's say you had a catalog for client work. And client work lives in ~/pictures/client_work. In that folder are subfolders for each client and in those, subfolders for jobs. You have one client whose jobs you exclude from the catalog, ~pictures/client_work/nda_client.

So you include ~/pictures/client_work and exclude ~pictures/client_work/nda_client

You shoot two jobs and add ~/pictures/client_work/nice_cient/today_assignment and ~pictures/client_work/nda_client/secret_stuff

No need to fiddle with the paths. Scan to Catalog will do fine.

But if you add ~company/logos, which is outside your existing directory tree, for that you would need to add a path.

Just as an aside for anybody who hasn't noticed, for some while now we've had the ability to update existing Snapshots. Just choose "Other Snapshots" in the Save Snapshot dialog, choose and confirm. So, if you have to do it, it's not as big a deal as it once was.

[bazography]

Scan to catalog is recursive, so normally there would be no need to add to the path.

But that doesn't help in my case. And that's the whole point for me really; it depends entirely on how you have your archive organized.

[carlseibert]

Ah! I missed the way you structured your directories. Taxonomies make my head hurt.

It seems like you'd still have to add a new set of paths whenever you add a new event folder, even if the program re-scanned all by itself.

Maybe something to try - and since you can have overlapping catalogs you can try this experimentally with no huge risk - would be to make a "big" catalog that would include all your events and then use field values to sort out Raws, Edits, deliverables, etc. A code in Keywords could work. Or maybe in some other field, as long as you're not abusing the field.

I've done stuff like this before in "big" DAMs, where there wasn't any folder structure at all and it's worked fine. If you make up clever code words that aren't normal language words, you can avoid code words returning in normal searches. "evenraw" for "Event RAW", for example.

Thus, a QuickSearch for "WidgetCo evendeliv" would return all of your delivered event images for the client WidgetCo. If you jump from the search to files and folder, it all becomes academic because your taxonomy is built in your directory structure.

Then, some thought would have to go into how would be best to apply your codewords, whether overwriting, appending, or find and replace would work when a new derivative file was created. That would depend mostly on what field you might decide to use.

Something like that might be worth a try, since there are probably disadvantages to having related versions of an asset in different catalogs.

[bazography]

Maybe something to try - and since you can have overlapping catalogs you can try this experimentally with no huge risk - would be to make a "big" catalog that would include all your events and then use field values to sort out Raws, Edits, deliverables, etc. A code in Keywords could work. Or maybe in some other field, as long as you're not abusing the field.

Very good suggestion, thanks. Definitely worth a test. I might also play around with different folder structures, but I'm not keen on revamping my entire archive unless I have no other option.
I think I'll do some more testing, but before I do anything crazy I'll wait to see what the final release PM+ looks like. Maybe these issues will be solved by then.