Beachball when switching apps, network storage

[Jarred]

Hi.  Our shoots are typically ~5,000 5D3 full raws, ~150gb.  We store them on a network drive and load them into PM from a few different workstations. 

We are able to work through the images in PM with no problem.  But, when we switch apps, e.g. edit in Ps, browse in a web browser, etc, when we return to PM, we get a beachball for anywhere from 3-~20 seconds.  It is quicker on hard wired desktops, but unbearably long on laptops, even with 802.11ac. 

What is causing this?  Is there anything we can do to reduce or eliminate this delay?

Any help would be appreciated, thanks.

[Kirk Baker]

Jarred,

How much free RAM does your system have?  When you switch to other apps and then back to PM, has PM's memory been swapped out due to virtual memory paging?

One thing that PM always does when it is resumed is to rescan all of the folders that are currently being browsed.  This can take time on slow drives.  There are some flags that can be turned on that can help.

From the release notes in build 16268:

Code: [Select]

• Optimized folder rescanning and added the ability to control what file associations should be searched. The following flags can be added to a text file named PMDebug.txt which should be placed in Photo Mechanic’s preferences folder:
   IGNORE_EXTENSION_THM (THM files are used with the CRW RAW format. If you don’t have CRW files then you can safely enable ignoring this extension.)
   IGNORE_EXTENSION_LZN (LZN files are for the open source RAW editor Lightzone. If you don’t use Lightzone, then you can safely enable ignoring this extension.)
   IGNORE_EXTENSION_FLT (FLT files are for the QImage printing application. If you don’t use QImage, then you can safely enable ignoring this extension.)
   IGNORE_EXTENSION_BIB (BIB files are used by the Bibble RAW editor application. If you don’t use Bibble, then you can safely enable ignoring this extension.)
   IGNORE_EXTENSION_RDSF (RDSF files are used by the RAW Developer application. If you don’t use this now extinct application, you can safely ignore this extension.)
   IGNORE_EXTENSION_NKS (NKS files are used by Nikon Capture NX-D. If you don’t use this application, you can safely ignore this extension.)
   IGNORE_EXTENSION_XIP (XIP files were generated by older versions of Photo Mechanic prior to the existence of XMP sidecar files. Unless you’re browsing folders that were edited by a much older version of Photo Mechanic, you can safely ignore this extension.)

   Ignoring these extensions will significantly reduce the amount of network traffic utilized by Photo Mechanic when browsing a network drive. If you want to ignore any of the above extensions, simply place them, one per line, in your PMDebug.txt file followed by ON, YES, or 1. Restart PM and those extensions set to be ignored will now be ignored and will no longer be associated with your files if they exist. This means that if you rename/move/copy an item, the ignored files will not be processed in the operation and will not be renamed/moved/copied.

If you wanted to ignore all of the optional extensions you would put the following into your PMDebug.txt file:

IGNORE_EXTENSION_THM ON
IGNORE_EXTENSION_LZN ON
IGNORE_EXTENSION_FLT ON
IGNORE_EXTENSION_BIB ON
IGNORE_EXTENSION_RDSF ON
IGNORE_EXTENSION_IDSF ON
IGNORE_EXTENSION_IDPRE ON
IGNORE_EXTENSION_NKS ON
IGNORE_EXTENSION_XIP ON
IGNORE_EXTENSION_ORI ON

Please let me know if this helps your situation.

-Kirk

[Jarred]

Thanks for the reply.  No memory swapping is taking place.  I added the ignores to the debug file and it did help, it cut the beachballing from 15-20 seconds to 8-10.  Which is a massive improvement.

[DaveRe]

We see the same issue here in our shop. I added those settings to PMDebug.txt, and it definitely seems to improve things a bit, but the delay is still really painful. Is there any way to tune the refresh behavior further, by any chance?

I did a couple of experiments, knowing that our network is pretty robust - with the settings in place specified in this thread, it takes about 7 seconds for PM to scan a directory containing 11,000 files (3914 images, RAW+JPG and XML files). An "ls -l" in that directory takes about 6 seconds. So, obviously, the majority of the time is just trying to get inode data from the server - I get that, and there's not much that PM can do about it. You need the inode file change info to determine if there's anything to scan, even in the best case scenario.

The only thing that I could see improving this situation is if you could tune PM (optionally) to either ignore refreshing the directory (aside from whatever it needs to do to keep up with changes that you're making to files in that PM session), or tune how frequently it might do the refresh. For instance, if I'm back and forth between PM, email, Photoshop, chat, etc, perhaps I don't need it to refresh each and every time I go back.

Obviously, the risk is that someone could change metadata (file ratings, etc), remove or add a file, etc, and you'll miss it because you didn't refresh, right? You have the same risk if you're working in PM when the same event occurs, but you don't do anything to trigger a refresh, right? (that is, I don't change tabs, don't do a "Command-/", and don't change focus to another window). If you know nothing's being ingested to the folder and can mitigate the risk of someone changing something in that directory at the same time, wouldn't this be a viable way to improve performance there? Quite possible I'm missing something entirely, of course :-)

Alternately, when you first open a directory into a contact sheet, PM lets you get to work right away, building out previews, and scanning metadata, etc, as it goes. Contrast that with the refresh you get when switching back to PM from another app, where you have the spinning beach ball of death until PM finishes the refresh. If it could do the refresh in the background, just like it's doing the initial scan, that would be totally acceptable from my standpoint?

Thanks in advance, Kirk, for any thoughts or info.

Dave

[Kirk Baker]

Dave,

Alternately, when you first open a directory into a contact sheet, PM lets you get to work right away, building out previews, and scanning metadata, etc, as it goes. Contrast that with the refresh you get when switching back to PM from another app, where you have the spinning beach ball of death until PM finishes the refresh. If it could do the refresh in the background, just like it's doing the initial scan, that would be totally acceptable from my standpoint?

But the initial scan (and all subsequent scans) all happen in the foreground.  None of the scanning or sorting happens on a background thread, ever.

-Kirk

[DaveRe]

Kirk,

I guess ignore the language a bit - foreground vs. background? It's how the behavior appears to me, but technically isn't what's actually going on, obviously.

There's a definite difference between the two situations, as I can perceive it from a user standpoint.

Upon the initial load, I can double click an image to load a bigger version, I can switch which image I have selected, I can rate an image, I can quit the application - all of that before PM has been able to load everything it's loading for the directory (in theory). Upon the refresh, I can't do any of that until the refresh finishes.

For instance, upon initial load of that same directory I was using yesterday (PM already running, I drag and drop the folder onto the PM icon in the dock), the contact sheet appears and thumbnails start populating within 1 second. If I hit Cmd-Q immediately, the application exits immediately. That's about 3-4 seconds total time from when I dropped the folder on the icon - about half the time it takes to do the refresh (see previous post).  If I load that folder the same way, let everything catch up (thumbnails, etc), then switch away from PM to some other app (web browser, email, Finder) for a bit and come back, I get the beachball. If I hit Cmd-Q immediately, nothing happens - I still get the beachball for a while until the app catches up and then promptly quits.

That behavior gives the impression that something is handled differently during initial load of a folder vs. the later refresh. Maybe that perception is incorrect - but there it is? Since it can be reproduced consistently, it would seem like it would be something intentional and programmatic that's different between the two cases?

[Kirk Baker]

Dave,

I guess ignore the language a bit - foreground vs. background? It's how the behavior appears to me, but technically isn't what's actually going on, obviously.

There's a definite difference between the two situations, as I can perceive it from a user standpoint.

Upon the initial load, I can double click an image to load a bigger version, I can switch which image I have selected, I can rate an image, I can quit the application - all of that before PM has been able to load everything it's loading for the directory (in theory). Upon the refresh, I can't do any of that until the refresh finishes.

For instance, upon initial load of that same directory I was using yesterday (PM already running, I drag and drop the folder onto the PM icon in the dock), the contact sheet appears and thumbnails start populating within 1 second. If I hit Cmd-Q immediately, the application exits immediately. That's about 3-4 seconds total time from when I dropped the folder on the icon - about half the time it takes to do the refresh (see previous post).  If I load that folder the same way, let everything catch up (thumbnails, etc), then switch away from PM to some other app (web browser, email, Finder) for a bit and come back, I get the beachball. If I hit Cmd-Q immediately, nothing happens - I still get the beachball for a while until the app catches up and then promptly quits.

That behavior gives the impression that something is handled differently during initial load of a folder vs. the later refresh. Maybe that perception is incorrect - but there it is? Since it can be reproduced consistently, it would seem like it would be something intentional and programmatic that's different between the two cases?

I suggest that we find out where the time is going then.  Are you up for doing some testing and getting me the logging information I need?

Since you already have a PMDebug.txt file, add these to it:

SHOW_DIRSCAN_TIME ON
SHOW_CSHEET_RESCAN ON

Then do your tests.  Please wait a minute or so before you switch back in so it is clear to me where the two tests are (each line in the log is time stamped.)  Then send me the log file.  Go to the Help menu and choose "Reveal Support Data..." and then post the resulting file here.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped log file

Thanks,

-Kirk

[DaveRe]

I'll give it a whirl today, Kirk. Thanks!

[DaveRe]

Kirk,

Apologies - got hung up on some things. Here's the test data.

[attachment deleted by admin]

[Kirk Baker]

Dave,

Each rescan took between nine and ten seconds to complete, including initial scanning of the folders.  Other than adding a preference setting to make rescan on resume optional, I'm not sure there is much I can do to help you.

-Kirk

[DaveRe]

Kirk,

I noticed that, too, looking at the log. I wonder if perhaps PM isn't actually displaying the contact sheet until after the initial scan is done, thus giving the impression that there's a difference between the two?

Honestly, that preference would help us out a great deal. There are obvious risks - we would mitigate those on our end through awareness and being careful to not leave the preference set when working in a collaborative folder. Is that perhaps something that could be done?

Thanks!

Dave

[Kirk Baker]

Dave,

I noticed that, too, looking at the log. I wonder if perhaps PM isn't actually displaying the contact sheet until after the initial scan is done, thus giving the impression that there's a difference between the two?

Yes, the contact sheet has nothing to show until the scan finds the files.

Honestly, that preference would help us out a great deal. There are obvious risks - we would mitigate those on our end through awareness and being careful to not leave the preference set when working in a collaborative folder. Is that perhaps something that could be done?

Certainly, but not for PM5.  New features are going into the next major version of Photo Mechanic.

-Kirk

[DaveRe]

Thanks, Kirk. It's understandable that a feature addition wouldn't show up in PM5.