Author Topic: Search query update  (Read 6294 times)

Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Search query update
« on: February 28, 2021, 03:51:32 AM »
I type in a search query and press enter, then realise I've made an error - correct the error, then press enter again. The old query is still working away for some time so I close the search tab. A new untitled 1 tab appears. I press enter again to submit my query but nothing happens. Untitled 1 is the only tab showing.

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24731
    • View Profile
    • Camera Bits, Inc.
Re: Search query update
« Reply #1 on: February 28, 2021, 10:17:37 AM »
Adrian,

I type in a search query and press enter, then realise I've made an error - correct the error, then press enter again. The old query is still working away for some time so I close the search tab. A new untitled 1 tab appears. I press enter again to submit my query but nothing happens. Untitled 1 is the only tab showing.

Could I have you turn on Catalog debug logging and try again, please?  From the Help menu, choose "Edit Debug Settings..." and find the item titled CATALOG_DEBUG_LOGGING and set its checkbox.  Then click the Close button.  Then restart Photo Mechanic Plus and reproduce the problem.  From the Help menu choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Re: Search query update
« Reply #2 on: February 28, 2021, 03:15:14 PM »
Debug log attached

Offline Bill Kelly

  • Software Developer
  • Full Member
  • ***
  • Posts: 131
    • View Profile
    • Camera Bits, Inc.
Re: Search query update
« Reply #3 on: March 01, 2021, 12:00:32 PM »
Hi Adrian,

I type in a search query and press enter, then realise I've made an error - correct the error, then press enter again. The old query is still working away for some

Thanks for the debug logs.

It appears the missing closing double-quote on the first of the two paths in the following expression produced a very long-running search:

path "manchester" AND (path ^"/Volumes/ALP Archive 01/ OR path ^"/Volumes/ALP Archive 02")

Unfortunately, we don't currently have a solution, but we'll explore ways to identify when a search is taking a long time in the database engine so that we can provide an option to cancel it.

Thanks for bringing this to our attention.

Regards,

Bill

Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Re: Search query update
« Reply #4 on: March 01, 2021, 01:54:03 PM »
I'm seeing the same behaviour when I use browse or filter and choose a parent folder with a lot of images (90,000). I clear the browse or filter query, and try to make a new query but nothing happens. It appears to have hung. No notifications as to what might be happening or if there's an error.


Offline Bill Kelly

  • Software Developer
  • Full Member
  • ***
  • Posts: 131
    • View Profile
    • Camera Bits, Inc.
Re: Search query update
« Reply #5 on: March 01, 2021, 02:06:59 PM »
I'm seeing the same behaviour when I use browse or filter and choose a parent folder with a lot of images (90,000).

If you haven't already, please try the "Optimize SQL Query Planning" button in the Maintenance options for your catalog(s) in the Manage Catalogs pane. It should take a few minutes on a larger catalog.

If that doesn't help, we'll need to troubleshoot further.


Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Re: Search query update
« Reply #6 on: March 01, 2021, 05:05:09 PM »
About an hour into the SQL analyse now... (210,000+ images). Bit more than a few minutes. Sounding normal?

Offline Bill Kelly

  • Software Developer
  • Full Member
  • ***
  • Posts: 131
    • View Profile
    • Camera Bits, Inc.
Re: Search query update
« Reply #7 on: March 01, 2021, 05:56:05 PM »
About an hour into the SQL analyse now... (210,000+ images). Bit more than a few minutes. Sounding normal?

Strange.

The analyze function here on a million-item catalog takes 87 seconds. (Granted, on a very fast SSD.)

On what type of disk does your catalog reside?


Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Re: Search query update
« Reply #8 on: March 01, 2021, 05:59:12 PM »
It’s an SSD albeit a 2.5” model on the 2012 Mac mini quad i7 W 16gb ram. Catalog is on what used to be the scratch disk. It has lots of space free.

Offline Bill Kelly

  • Software Developer
  • Full Member
  • ***
  • Posts: 131
    • View Profile
    • Camera Bits, Inc.
Re: Search query update
« Reply #9 on: March 01, 2021, 06:27:52 PM »
I just ran the analyze function on a million-item catalog which is stored on an external spinning platter HDD over a USB interface, and even that completed in about 187 seconds.

So, something indeed peculiar is going on with the Catalog system on your computer.

Since you're on a Mac, it's likely the "sqlite3" command is built-in. We'll write up a follow-up post containing some diagnostic commands we'd like you to try running on your catalog from the Terminal app.


Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24731
    • View Profile
    • Camera Bits, Inc.
Re: Search query update
« Reply #10 on: March 01, 2021, 06:46:56 PM »
Adrian,

Here is what we need you to do.

First quit Photo Mechanic Plus.  Make sure there are no lingering pm-task processes running.  Open the Activity Monitor and click on the "CPU" tab.  Then in the search field at the right side of the toolbar enter: pm-task

If you see any listed, click on them one-by-one and click the Stop button (looks like a hexagon with an X inside it.)  You can now quit the Activity Monitor.

Now open a Terminal window.  The Terminal application is in the /Applications/Utilities folder.  When the window opens type 'cd ' without the quotes.  That's the letter c and d followed by a space.

Then in the Finder, find your catalog folder and drag and drop it onto the Terminal window.  Next click in the Terminal window and press enter.  Your terminal session will now have its current directory inside your catalog folder.

Next copy and paste the following into your Terminal window, and press enter:

time sqlite3 catalog.pmdb "pragma integrity_check"

When that finishes, copy and paste the following into your Terminal window, and press enter:

time sqlite3 catalog.pmdb "analyze"

When that finishes, copy the output of both commands and paste them in your reply to this post.

Thanks,

-Kirk

Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Re: Search query update
« Reply #11 on: March 02, 2021, 12:28:11 AM »
ALP-MacMini:~ adrian$ cd /Volumes/PS/PM_Catalogs/ALP
ALP-MacMini:ALP adrian$ time sqlite3 catalog.pmdb "pragma integrity_check"
ok

real   6m39.322s
user   3m8.062s
sys   0m56.727s
ALP-MacMini:ALP adrian$ time sqlite3 catalog.pmdb "analyse"
Error: near "analyse": syntax error

real   0m0.009s
user   0m0.002s
sys   0m0.006s
ALP-MacMini:ALP adrian$

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Search query update
« Reply #12 on: March 02, 2021, 02:32:34 AM »
To add to the comments about SQL analyze timing, mine (on a 660,000 image catalogue after a reboot to make sure there were no stray processes running) took 1284 seconds (over 21 minutes) on a fast Windows 10 system with the catalogue and cache on a fast internal SSD.

I reran the analysis a few hours later with other applications running.  The timing was almost identical (1281 seconds).

Graham
« Last Edit: March 02, 2021, 08:13:25 AM by Graham1 »

Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Re: Search query update
« Reply #13 on: March 02, 2021, 04:17:20 AM »
Thanks. You just made me actually read what I posted! I’ll rerun the commands and repost.

Offline adrianlambert

  • Full Member
  • ***
  • Posts: 105
    • View Profile
Re: Search query update
« Reply #14 on: March 02, 2021, 05:24:39 AM »
ALP-MacMini:~ adrian$ cd /Volumes/PS/PM_Catalogs/ALP
ALP-MacMini:ALP adrian$ time sqlite3 catalog.pmdb "pragma integrity_check"
ok

real   7m16.892s
user 3m25.443s
sys   0m58.508s
ALP-MacMini:ALP adrian$ time sqlite3 catalog.pmdb "analyze"

real   3m27.502s
user 0m55.285s
sys   0m28.521s