Author Topic: Batch operations on files on a NAS and other questions  (Read 3823 times)

Offline Dykam

  • Newcomer
  • *
  • Posts: 6
    • View Profile
Batch operations on files on a NAS and other questions
« on: August 13, 2022, 01:06:21 AM »
Hi,

After having used Daminion for years, I ran into limitations of their software, and my main point of friction was the slow import and gallery view. Which a friend of mine informed me of Photo Mechanic being really good at this. When looking at PM, I noticed that it also added a "catalog" feature, essentially adding what I needed to replace Daminion. And so far I've been loving it, and it has been able to replace the essential things I used in Daminion pretty much perfectly.

But there are some things which I either miss, or am simply used to doing differently, or am unaware of in PM, so maybe you can help me out finding out what the best way is. I'm aware that it seems the catalog feature is quite new, so I suppose it'll be improved in the future anyway.

Note that so far, this post might come off as negative, I'm not mentioning everything else that works great.

For context, my catalog lives on my internal fast SSD, and the photo files themselves on a NAS with 1Gb up/down.



I've noticed that many batch operations (adding keywords, renaming, etc) only work on selections. And only when those selections are fully loaded. Otherwise I get some pop-up saying "gathering metadata", which then always fails. So for me, the only way to perform a bulk operation, is to browse to some limited set of photos, scroll down and wait until every item has it's metadata loaded (only those in view are), and then perform the operation.

To make this more difficult, PM seems to completely reload the contact sheet (including the database query) whenever I tab back in after tabbing out of PM (I'm on Windows), so the waiting before prevents me from doing anything else on my PC. This might just me? But it happens consistently.



I cannot for the life of me get video tiles to render. Looking at the logs, it seems that it's due to file size. But even a 250MB video file fails:
Code: [Select]
NRFWinBitmap.cpp:1129 ERROR: NRFWinBitmap::LoadVideoFile() GstVideoRender returned exit code:-1.
NRFWinBitmap.cpp:1130 <data too large>
NRFWinBitmap.cpp:1131 stderr:
NRFBitmapBase.cpp:733 NRF failed to load '<snip>' with error 5
I've tried the gstreamer version bundled with the installer, and by installing 1.16.2 manually.



Thumbnails and metadata load fairly slowly. I thought the catalog with proxy previews was so initializing either was speedy, but loading a contact sheet doesn't get any faster than 10 thumbnails a second. If there's video tiles among it, it's worse. And until this has been loaded, I can also not jump into preview mode.



The browse/filter/search system is unusual to me. I'm used to Daminion, which combines browsing and filtering into a single view. Or rather, it has the filtering dialog, but with the interface of the browsing dialog, and it does everything I need. The browsing dialog feels a bit useless to me, as e.g. the question "I want to see photos takes one date X, and also photos taken with camera Y" never comes up, and the question "I want to see photos taken on date X and Y, with camera Z" frequently, which filtering does fine.

But might be just me being used to a different application, so in a month I might not even mind anymore.



Is there an option to have multiple filter/browse tabs?



Is there a way to batch-edit metadata? I can batch-update metadata, other than add-or-replace using the template feature? As every operation seems to require all metadata to be loaded either way, some aggregate view of the metadata would make it easy to remove tags, not just add or replace.



Is there a way to make PM rescan just a specific set of files for new metadata? Often I do an external edit to the metadata (e.g. with a different gallery app or something), and to have to do a catalog-wide rescan for a file or two is a bit much. It seems to do a rescan the moment the contact sheet loads, but this requires the image to be in view on the sheet.



Preview generation seems a bit inefficient, as in my specific case I sometimes ingest from my internal drive (fast) to my NAS, but then the preview generator downloads them back to the PC to generate a preview, throttling ingest. If it could be part of ingest it might help.



Related to the previous option, it'd be nice if I could pause tasks. Specifically preview generation, to speed up ingest, or give my network some breathing room. I can abort it, and then do a catalog rescan to add the missing pictures, but that's not ideal.



That said, I'm really liking Photo Mechanic. The ingest is great. I'd love a (kind of) preview of what it will do, and a ETA indicator on the task, but otherwise it works great. The fact the catalog is immediately updated is fantastic. The performance of the preview window is incredible, and very useful. I'd like to be able to automate a bit more metadata-wise (adding keywords etc), but tagging and rating does the job just as well.

I am currently in the trial period, and strongly considering buying the package, but the issues with the catalog's (contact-sheet) performance make the price quite steep for a hobby photographer who just takes pics for their local communities.
« Last Edit: August 13, 2022, 02:39:12 AM by Dykam »

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Batch operations on files on a NAS and other questions
« Reply #1 on: August 13, 2022, 12:14:43 PM »
After having used Daminion for years, I ran into limitations of their software, and my main point of friction was the slow import and gallery view. Which a friend of mine informed me of Photo Mechanic being really good at this. When looking at PM, I noticed that it also added a "catalog" feature, essentially adding what I needed to replace Daminion. And so far I've been loving it, and it has been able to replace the essential things I used in Daminion pretty much perfectly.

But there are some things which I either miss, or am simply used to doing differently, or am unaware of in PM, so maybe you can help me out finding out what the best way is. I'm aware that it seems the catalog feature is quite new, so I suppose it'll be improved in the future anyway.

Note that so far, this post might come off as negative, I'm not mentioning everything else that works great.

No worries, we're open to criticism, especially if it is constructive.

For context, my catalog lives on my internal fast SSD, and the photo files themselves on a NAS with 1Gb up/down.



I've noticed that many batch operations (adding keywords, renaming, etc) only work on selections. And only when those selections are fully loaded. Otherwise I get some pop-up saying "gathering metadata", which then always fails. So for me, the only way to perform a bulk operation, is to browse to some limited set of photos, scroll down and wait until every item has it's metadata loaded (only those in view are), and then perform the operation.

I've tried to make this better several times now and usually it works fine if the catalog is not also simultaneously updating its database.  Was your catalog fully built?  Were there any tasks saying that it was processing batches of metadata or previews?

To make this more difficult, PM seems to completely reload the contact sheet (including the database query) whenever I tab back in after tabbing out of PM (I'm on Windows), so the waiting before prevents me from doing anything else on my PC. This might just me? But it happens consistently.

This can be turned off in the Contact Sheet page of the Preferences dialog (Rescan on resume).

I cannot for the life of me get video tiles to render. Looking at the logs, it seems that it's due to file size. But even a 250MB video file fails:
Code: [Select]
NRFWinBitmap.cpp:1129 ERROR: NRFWinBitmap::LoadVideoFile() GstVideoRender returned exit code:-1.
NRFWinBitmap.cpp:1130 <data too large>
NRFWinBitmap.cpp:1131 stderr:
NRFBitmapBase.cpp:733 NRF failed to load '<snip>' with error 5
I've tried the gstreamer version bundled with the installer, and by installing 1.16.2 manually.

What codec do these videos use?

Thumbnails and metadata load fairly slowly. I thought the catalog with proxy previews was so initializing either was speedy, but loading a contact sheet doesn't get any faster than 10 thumbnails a second. If there's video tiles among it, it's worse. And until this has been loaded, I can also not jump into preview mode.

Since your NAS is online and connected, the thumbnails/previews are being generated from the files on the NAS.  This is something that can be improved by us in the future.

The browse/filter/search system is unusual to me. I'm used to Daminion, which combines browsing and filtering into a single view. Or rather, it has the filtering dialog, but with the interface of the browsing dialog, and it does everything I need. The browsing dialog feels a bit useless to me, as e.g. the question "I want to see photos takes one date X, and also photos taken with camera Y" never comes up, and the question "I want to see photos taken on date X and Y, with camera Z" frequently, which filtering does fine.

But might be just me being used to a different application, so in a month I might not even mind anymore.

Selecting multiple items in Browse increases the number of results you'll get.  Selecting multiple criteria in the Filter reduces the number of items you'll get.  I think for your purposes, Filter is what you should use primarily.

Is there an option to have multiple filter/browse tabs?

No.

Is there a way to batch-edit metadata? I can batch-update metadata, other than add-or-replace using the template feature? As every operation seems to require all metadata to be loaded either way, some aggregate view of the metadata would make it easy to remove tags, not just add or replace.

The Metadata Template dialog is the batch editor for the most part.  There are also Keywords and Structured Keywords panels that operate in batches.  You can also toggle Tag, set Ratings, and set Color Class in batch directly in the Contact Sheet.

Is there a way to make PM rescan just a specific set of files for new metadata? Often I do an external edit to the metadata (e.g. with a different gallery app or something), and to have to do a catalog-wide rescan for a file or two is a bit much. It seems to do a rescan the moment the contact sheet loads, but this requires the image to be in view on the sheet.

Catalog Sync in Full Mode can be set to scan a specific folder.

Preview generation seems a bit inefficient, as in my specific case I sometimes ingest from my internal drive (fast) to my NAS, but then the preview generator downloads them back to the PC to generate a preview, throttling ingest. If it could be part of ingest it might help.

The previews are not being used while your NAS is reachable. (mentioned earlier, same issue)

Related to the previous option, it'd be nice if I could pause tasks. Specifically preview generation, to speed up ingest, or give my network some breathing room. I can abort it, and then do a catalog rescan to add the missing pictures, but that's not ideal.

Using previews when available instead of hitting the NAS should make the need to pause unnecessary.

That said, I'm really liking Photo Mechanic. The ingest is great. I'd love a (kind of) preview of what it will do, and a ETA indicator on the task, but otherwise it works great. The fact the catalog is immediately updated is fantastic. The performance of the preview window is incredible, and very useful. I'd like to be able to automate a bit more metadata-wise (adding keywords etc), but tagging and rating does the job just as well.

I am currently in the trial period, and strongly considering buying the package, but the issues with the catalog's (contact-sheet) performance make the price quite steep for a hobby photographer who just takes pics for their local communities.

Understandable.  Your use case is not one we optimized for and will improve in the future.  I do not have a timetable for that improvement to give to you at this time however.

-Kirk

Offline Dykam

  • Newcomer
  • *
  • Posts: 6
    • View Profile
Re: Batch operations on files on a NAS and other questions
« Reply #2 on: August 14, 2022, 06:55:15 AM »
Quote
I've tried to make this better several times now and usually it works fine if the catalog is not also simultaneously updating its database.  Was your catalog fully built?  Were there any tasks saying that it was processing batches of metadata or previews?

None of that. However it looks like it fails when PM is still loading the metadata/thumbnails for the visible part of the contact sheet. Which with the next issue was a more significant problem. But it seems to work if I give PM some time to fully load the (visible part of) the contact sheet. It's still a bit iffy, the first time it might fail with that message, but the second time it succeeds. Afterwards when I scroll the contact sheet, the metadata for all the photos does indeed seem already loaded.

Quote
This can be turned off in the Contact Sheet page of the Preferences dialog (Rescan on resume).

Saving my life. I did see the option when checking out all the settings, but I didn't gather that that was what it does. Seems more useful for folder contact sheets than catalog contact sheets, isn't catalog view automatically updated on e.g. ingest anyway? Though that might've been the rescan.

The term scan is also used in "scan to catalog", which probably threw me off.

Quote
What codec do these videos use?

HEVC, I think that's the issue. Though I installed gstreamer 1.20, supposedly 1.18 includes intel's HEVC decoder, but no luck so far. I'll be converting the clips to h.264 anyway, so this will probably solve itself.

Quote
Since your NAS is online and connected, the thumbnails/previews are being generated from the files on the NAS.  This is something that can be improved by us in the future.

So for faster viewing, I'd need to disconnect? So the preview feature is purely for if the storage medium isn't available? Now it makes sense, though it'd be greatly appreciated if at least thumbnails can be fetched locally. The previews load plenty fast for me.

I guess the same counts for metadata, the most important stuff should already be in the catalog database anyway. But not a deal breaker, good to know what the feature does. The documentation is a bit fuzzy on this, as it specifies "offline and remote use", I gathered my use case was the latter. I guess for me, I can disable previews for now as my NAS is always available.

One thing I noticed is that "Generate high quality thumbnails" makes a big difference for me (obviously). If it were possible to make this option dynamic based on chosen thumbnail size, it'd be great. As the case where I have many thumbnails to load, is also the case they're very tiny and I don't need high quality ones.

Quote
Selecting multiple items in Browse increases the number of results you'll get.  Selecting multiple criteria in the Filter reduces the number of items you'll get.  I think for your purposes, Filter is what you should use primarily.

Right, I see.

Quote
The Metadata Template dialog is the batch editor for the most part.  There are also Keywords and Structured Keywords panels that operate in batches.  You can also toggle Tag, set Ratings, and set Color Class in batch directly in the Contact Sheet.

Right, I noticed that. My reference is Daminion, which allows you to load aggregate data into the editor for your selection, but I think it does so from the catalog, so it's decently fast. Or for single files, it just prepopulates everything. If I were to wager a guess, PM operates on a file-first method as that was what it was historically based on, rather than the catalog database. Which is useful, as it prevents desyncs, but a bit slower.

Quote
Catalog Sync in Full Mode can be set to scan a specific folder.

Oh, in my mind that was the list of "these folders are in the catalog", unaware that that setting doesn't really exist and it's a "sync this now" sheet. That helps a lot. "Include existing paths from catalog" makes sense now, though that button made the dialog freeze. And after closing and reopening PM it seems it was literally including all folders, rather than the root folder. The latter being my expectation. A way to quickly do this from a selection or contact sheet would be useful, but for now this solves my issue entirely.

Quote
Using previews when available instead of hitting the NAS should make the need to pause unnecessary.

I guess you're saying that as to suggest it might be a future improvement/toggle? That'd be great.



Something I just noticed, could it be that it remembers scroll position when switching filters? I guess it makes sense in some scenarios, but usually scroll-to-top would be what I need.



After these responses, my biggest issue right now is that batch operations other than those with a dedicated task, pretty much freezes PM for me, with only occasionally the interface updating. Which, to be fair, is a minor thing, but it makes it hard to check out progress.

Quote
Understandable.  Your use case is not one we optimized for and will improve in the future.  I do not have a timetable for that improvement to give to you at this time however.

Thanks for the extensive reply, I hope my post wasn't too negative, but rather felt constructive. For a first version of a catalog feature, it works very well, so I'm quite hopeful it'd works even better in the future. I'm going to finish the trial to fully evaluate it, but it might seem that PM will turn into something essential for me.

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Batch operations on files on a NAS and other questions
« Reply #3 on: August 16, 2022, 09:31:04 AM »
Quote
What codec do these videos use?

HEVC, I think that's the issue. Though I installed gstreamer 1.20, supposedly 1.18 includes intel's HEVC decoder, but no luck so far. I'll be converting the clips to h.264 anyway, so this will probably solve itself.

Using a newer version of GStreamer is not supported.  Please use the 1.16.2 version of GStreamer.

-Kirk

Offline Dykam

  • Newcomer
  • *
  • Posts: 6
    • View Profile
Re: Batch operations on files on a NAS and other questions
« Reply #4 on: October 04, 2022, 03:29:05 PM »
I ended up acquiring a license, as it serves 99% of my needs fairly well, especially when compared to alternatives.

I do have a follow up question.

In my collection there's a bunch of videos, which PM imports with varying degrees of success and failure.

I have to import .MTS files. For convenience and to experiment I converted those to MP4 as well, which keeps the stream data unchanged, as it's just a different container.

When importing the MTS files:
  • The path has the correct date, but "default" as camera variable.
  • No sidecar XMP is generated.
  • The .MTS extension disappears. In the database they're stored as `<filename>.`, and on the filesystem they're just `<filename>`, rather than `<filename>.<ext>`, which makes renaming from inside PM impossible. This might also be why there's no sidecar XMP.
  • In PM, afterwards, the files show with their modification date rather than the (imported) creation date, as opposed to e.g. JPEG's. I assume this is because the sidecar is missing.

When importing the MP4 files:
  • The path has "1970-01-01" (unix epoch) as date, and "default" as camera variable.
  • A sidecar XMP is generated, but with the aforementioned date of 1970-01-01.

Now, both formats appear to be a royal pain to parse for a tool like PM, but exiftool is able to grok the files. I'd love the option to set certain be able to use Exiftool to provide metadata for certain files. I understand it'll be possibly much slower and complicated, but it'd be a fallback to support more files by being able to override PM's import.

My current workaround is to import the MP4's, use Exiftool to update the sidecars, and perform a rename and relocate using the updated metadata. I can only not figure out how to make PM recognize the camera model, exiftool writes it as tiff:Model to the sidecar but PM doesn't pick that up.

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Batch operations on files on a NAS and other questions
« Reply #5 on: October 05, 2022, 12:13:59 PM »
I do have a follow up question.

In my collection there's a bunch of videos, which PM imports with varying degrees of success and failure.

I have to import .MTS files. For convenience and to experiment I converted those to MP4 as well, which keeps the stream data unchanged, as it's just a different container.

When importing the MTS files:
  • The path has the correct date, but "default" as camera variable.
  • No sidecar XMP is generated.
  • The .MTS extension disappears. In the database they're stored as `<filename>.`, and on the filesystem they're just `<filename>`, rather than `<filename>.<ext>`, which makes renaming from inside PM impossible. This might also be why there's no sidecar XMP.
  • In PM, afterwards, the files show with their modification date rather than the (imported) creation date, as opposed to e.g. JPEG's. I assume this is because the sidecar is missing.

MTS files are not fully supported by Photo Mechanic.  They're downloaded/preserved by Ingest to the destination folder, but that's basically it.

When importing the MP4 files:
  • The path has "1970-01-01" (unix epoch) as date, and "default" as camera variable.
  • A sidecar XMP is generated, but with the aforementioned date of 1970-01-01.

That could be improved.

Now, both formats appear to be a royal pain to parse for a tool like PM, but exiftool is able to grok the files. I'd love the option to set certain be able to use Exiftool to provide metadata for certain files. I understand it'll be possibly much slower and complicated, but it'd be a fallback to support more files by being able to override PM's import.

My current workaround is to import the MP4's, use Exiftool to update the sidecars, and perform a rename and relocate using the updated metadata. I can only not figure out how to make PM recognize the camera model, exiftool writes it as tiff:Model to the sidecar but PM doesn't pick that up.

While I don't see a specific question (no sentence ending in a question mark), I'm guessing you want to know how to make PM utilize ExifTool to provide metadata for PM.  It's not possible at this time.

-Kirk

Offline Dykam

  • Newcomer
  • *
  • Posts: 6
    • View Profile
Re: Batch operations on files on a NAS and other questions
« Reply #6 on: October 06, 2022, 03:25:48 AM »
MTS files are not fully supported by Photo Mechanic.  They're downloaded/preserved by Ingest to the destination folder, but that's basically it.

Ah, right. If at least the extension and a minimal XMP was generated, it would be already hugely useful. That said, I'm converting everything to MP4, so I personally don't need it.

That could be improved.

I did notice that for some MP4's it works, and the date is properly detected. It seems that in the successful case, the date metadata is in the mp4 container itself. In the other case, it's part of the h264 stream for some reason.

While I don't see a specific question (no sentence ending in a question mark), I'm guessing you want to know how to make PM utilize ExifTool to provide metadata for PM.  It's not possible at this time.

It wasn't a specific question, more something I noticed. But I've gotten a workaround for now using Exiftool to generate sidecars.

Regarding something you mentioned earlier:
Quote
Catalog Sync in Full Mode can be set to scan a specific folder.
While that seems to  work for checking for new files, it appears that it still scans everything when looking for new metadata. I might be seeing it wrong, but it seems there's no way to quickly rescan a specific folder to fully sync.

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Batch operations on files on a NAS and other questions
« Reply #7 on: October 06, 2022, 09:17:19 AM »
Regarding something you mentioned earlier:
Quote
Catalog Sync in Full Mode can be set to scan a specific folder.
While that seems to  work for checking for new files, it appears that it still scans everything when looking for new metadata. I might be seeing it wrong, but it seems there's no way to quickly rescan a specific folder to fully sync.

Can you tell me more about what you're trying to accomplish with a single folder and having the catalog update?  Just adding new files?  Detecting missing files?  Updating metadata?

-Kirk

Offline Dykam

  • Newcomer
  • *
  • Posts: 6
    • View Profile
Re: Batch operations on files on a NAS and other questions
« Reply #8 on: October 06, 2022, 09:40:49 AM »
I occasionally have external tools which update metadata or delete files, and I usually know what set of photo's is affected. Since my entire collection involves 100k photos over a network, doing a full scan can take a fair amount of time. I usually know, by folder, down to a few hundred photos which are affected, so that's quite a difference.

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: Batch operations on files on a NAS and other questions
« Reply #9 on: October 06, 2022, 10:00:39 AM »
I occasionally have external tools which update metadata or delete files, and I usually know what set of photo's is affected. Since my entire collection involves 100k photos over a network, doing a full scan can take a fair amount of time. I usually know, by folder, down to a few hundred photos which are affected, so that's quite a difference.

Sounds like you'd be better off just browsing that folder (the metadata changes will be detected and the catalog updated).  As for deletion, deleting inside Photo Mechanic Plus will take care of that issue as well.  No sync needed at all.

-Kirk

Offline Dykam

  • Newcomer
  • *
  • Posts: 6
    • View Profile
Re: Batch operations on files on a NAS and other questions
« Reply #10 on: October 06, 2022, 02:24:24 PM »
As browsing a folder isn't that convenient, if the images don't fit into view you have to keep scrolling down slowly while all the images load. If there's some videos in the folder that can be quite slow. But that's definitely the method I use right now.

But yeah, there's other ways to do it, but it'd be a nice feature eventually. But indeed definitely not a priority, just a wish.