OR variables during ingest

[ejhutch]

Hi,
Is it possible to OR variables when Ingesting (or whenever applying variables to metadata) depending on whether they are defined?

I'd like to set up a general ingest task that ingests images into folders based on the {artist} OR {id} OR {owner} OR {user} OR {comment} variables depending on which one is defined in the metadata of the image to be ingested.

From the Variables Window:

{artist} - TIFF Artist field written by some cameras. {id} - Either {owner} (Canon) or {comment} (Nikon). {owner} - Camera's owner (if defined; same as {user}). {user} - Camera's user (if defined; same as {owner}). {comment} - The comment (written by some cameras).

Can I do something like include {artist|id|owner|user|comment} in the Folder Name field in the ingest dialog window? The result would expand to whichever variable was able to be resolved first.

Thanks.
Ed

[Kirk Baker]

Ed,

Is it possible to OR variables when Ingesting (or whenever applying variables to metadata) depending on whether they are defined?

I'd like to set up a general ingest task that ingests images into folders based on the {artist} OR {id} OR {owner} OR {user} OR {comment} variables depending on which one is defined in the metadata of the image to be ingested.

From the Variables Window:

{artist} - TIFF Artist field written by some cameras. {id} - Either {owner} (Canon) or {comment} (Nikon). {owner} - Camera's owner (if defined; same as {user}). {user} - Camera's user (if defined; same as {owner}). {comment} - The comment (written by some cameras).

Can I do something like include {artist|id|owner|user|comment} in the Folder Name field in the ingest dialog window? The result would expand to whichever variable was able to be resolved first.

No, there's no such feature, but I think it's a good idea.  I'll move this to the Feature Requests board.

-Kirk

[ejhutch]

Would you want to kill me then for asking for full regular expression pattern matching support (Such as PCREs) when expanding variables? Because I am. Thanks!
Ed

[Kirk Baker]

Ed,

Would you want to kill me then for asking for full regular expression pattern matching support (Such as PCREs) when expanding variables? Because I am. Thanks!

I'm not against such a thing (we already have an engine in PM for this) but what would the syntax look like?

-Kirk

[ejhutch]

That... is a good question. And probably a long discussion as well. I'll have to work on some possible examples and use cases and get back to you. I know I've done stuff like this in various scripting and macro languages. So any examples I provide are more about what to do, rather then about how (syntax, etc) to do it.

I have a love/hate relationship with regular expressions.

 I can say that the first case that got me thinking about it was the way PM Plus works with variables and path names.

You've currently got {filename}, {folder}, {folderpath}, {path}, and {parentfolderpath} variables.

And I know we can use regexes in searching the catalog.  However, when exploring variable substitution, I found the current implementation a bit limiting for what I wanted to do, especially considering that substring extraction and translation are limited to fixed, defined lengths from the starting to ending character positions.

I had a copy/move case where I wanted to extract and utilize more levels of the full path than just up to the parent folder, and realized it would be pretty easy if the variable substring extraction capabilities were expanded to be able to search for say /an/arbitrarily/long/pathname/to/a/photo.jpg and implement something like  {folderpath:pathindexstart,pathindexend,pathdelimiter} such as {folderpath:1,/} for an (starting at the beginning of the folder path), or {folder path:-1,/} for the parent a (starting at the end and counting backwards), or even something like {folderpath:-1,-5,/} for arbitrarily/long/pathname/to/a (which could also be {folderpath:2,6,/}, or even other variations like {folderpath:,6,/} (the first six folders in the path starting at the beginning) or {folderpath:,-6,/} (the last 6 folders in the path). Of course, there are many ways to accomplish, this, and keep the scope limited, but I can also see incredibly powerful utility in broadly supporting a more complete implementation of regular expression pattern matching, grouping, search-and-replacing, etc. for variable substring manipulation.

However, I realize you've got limited resources, and regexes can get pretty complex quickly, and I'm not sure how much capability 80% of your user base would be able to make use of. You'd know that better than I.

What are your thoughts?

[Kirk Baker]

My thoughts are that this is an interesting direction and I'd be happy to discuss further.

-Kirk

[ejhutch]

I'd be happy to help discuss this further. Let me know how you'd like to proceed.

[Kirk Baker]

I'd be happy to help discuss this further. Let me know how you'd like to proceed.

I'd like more scenarios about what is trying to be accomplished and what solutions make sense.

-Kirk

[ejhutch]

Ok. I'll take some time to think about it and come up with more scenarios or use cases.  While I do that, can you give me any information about how your regex engine is currently implemented? From what I can tell, it is available for use with entering custom searches of the catalog. Which regex version/flavor do you implement? Is it done in Ruby, or if a different programming language or library, what do you use? Is it accessible anywhere else within PM or PM+? Is my assumption that it is used to process the search string, which is then converted into an SQL query, which is then sent to SQLite to be executed? Or is it done a different way? How much use do you think regex searching gets as a portion of your user base (how popular or desired is it)?

I'm happy that you're intrigued by this, but it could be a huge can of worms to open, as I'm sure you well know, and I'd like to learn more so I can judge how to tailor my suggestions to appropriately limit or expand the scope of the request based on how useful it (or parts of it) could be.

[Kirk Baker]

Ok. I'll take some time to think about it and come up with more scenarios or use cases.  While I do that, can you give me any information about how your regex engine is currently implemented? From what I can tell, it is available for use with entering custom searches of the catalog.

Catalog Search does not support full regex.  It has a few operators (like matching the start or end), but certainly not full regex support.

Find/Find and Replace does offer a 'Grep' option, and it uses the Boost regex system, but it doesn't find over the entire catalog database, it only works with the images that are in the current Contact Sheet.  If the Contact Sheet is a million item Search results, it's not effective at all since it would take ages to access the metadata of all of the images in the process of performing the Find operation.

Which regex version/flavor do you implement? Is it done in Ruby, or if a different programming language or library, what do you use? Is it accessible anywhere else within PM or PM+? Is my assumption that it is used to process the search string, which is then converted into an SQL query, which is then sent to SQLite to be executed? Or is it done a different way? How much use do you think regex searching gets as a portion of your user base (how popular or desired is it)?

The Catalog system is a client server architecture written mostly in Ruby but all of the client-side UI is managed in C++.
The query parser is written in Ruby and only offers a minor subset of regular expressions.  The query parser produces SQL queries and hands them off to SQLite.

Most of our user base does not understand regular expressions.

All that said, I think using Regexes on the output of an evaluated {variable} is entirely doable (just like it is in the Find/Find and Replace panels), but it has to make sense and not be horrible to parse and execute via Boost's regex system.
 

I'm happy that you're intrigued by this, but it could be a huge can of worms to open, as I'm sure you well know, and I'd like to learn more so I can judge how to tailor my suggestions to appropriately limit or expand the scope of the request based on how useful it (or parts of it) could be.

If limited in scope to {variables} I'm not expecting huge complications.

I don't know of any database query system capable of handling full regex support that produces sane SQL that would execute in reasonable time.  So let's not go there.

-Kirk

[ejhutch]

Great info there. From what you've told me so far, I completely agree with you on limiting the scope. I'll do some research and get back to you with some ideas.

[esambo]

I would love regular expressions or SQL. That would be an amazing and very powerful feature.