Slow (recovery) from file operations

[keydet]

I'm increasingly frustrated by very slow "recovery" from file operations...copying/moving files, or saving photos.  PM seems to breeze quickly through the operation itself, but then the little Vista wheel spins for anywhere from 10-20 seconds, while the slider bar in the navigator goes crazy.  Eventually I can regain control of the app.  Doesn't seem to matter whether I'm copying one file or 100.  Had similar issues with Win XP.

I'm not running a marginal machine by any means, so I'm wondering if I've got things configured wrong or what...

[Kirk Baker]

I'm increasingly frustrated by very slow "recovery" from file operations...copying/moving files, or saving photos.  PM seems to breeze quickly through the operation itself, but then the little Vista wheel spins for anywhere from 10-20 seconds, while the slider bar in the navigator goes crazy.  Eventually I can regain control of the app.  Doesn't seem to matter whether I'm copying one file or 100.  Had similar issues with Win XP.

I'm not running a marginal machine by any means, so I'm wondering if I've got things configured wrong or what...

What version of PM are you running?

-Kirk

[Rich Gibson]

I encounter this in my MacPRo running both Tiger and Leopard running PC 4.5.2. (Please see my latest entry in a related thread about hanging).  I have a single master folder with about 1000 subfolders for each day's shoot.  When I click on the master folder it takes as much as 4-5 seconds for the folders to appear.  When I use finder or mucommander (a Mac dual frame file manager) the folders appear virtually instantaneously.  The freezing involves selecting and copying images to a Smugmug website window in Safari for upload.  I'm not a programmer but it appears that there is a particular way file and folders are maintained in PM's memory which really slows performance.  The is the only software which exhibits this slow file/folder display.

Thanks, Rich

[Primoz]

I would say that at least in Windows (Win XP) this is "normal" thing for long time. Personally I ingest photos to one sub-directory, so main directory contains unchecked "daily" directories (/main_directory/yyyymmdd/). After I pick photos I will send them forward, I copy them to different directory (/main_directory/send/). If PM has this send directory as last used directory, I just press enter fast and it copies them right away. But if I wait, that PM starts to "read" whole tree, it takes those 10-20sec until it allows me to press enter to confirm destination.
PS: This main_directory is just for my unchecked photos, so it maybe contains 20-30 sub directories with about 5000 photos, but root of main directory is clean. On the other side, I have feeling it's working faster on Mac, but I'm running PM only on my Macbook Pro, where I normally don't have much more then 2 or 3 days old things.
And my info... 4.5.3 on WinXP, and 4.5.2 on Mac (will be upgrading as soon as I have some time .

[Kirk Baker]

Rich,

I encounter this in my MacPRo running both Tiger and Leopard running PC 4.5.2. (Please see my latest entry in a related thread about hanging).  I have a single master folder with about 1000 subfolders for each day's shoot.  When I click on the master folder it takes as much as 4-5 seconds for the folders to appear.  When I use finder or mucommander (a Mac dual frame file manager) the folders appear virtually instantaneously.  The freezing involves selecting and copying images to a Smugmug website window in Safari for upload.  I'm not a programmer but it appears that there is a particular way file and folders are maintained in PM's memory which really slows performance.  The is the only software which exhibits this slow file/folder display.

Please download version 4.5.3 of Photo Mechanic.

http://www.camerabits.com/download/index.html

-Kirk

[keydet]

I'm using 4.5.3.

I'm ashamed to say it, but I've started using ViewNX.  Everything seems to take forever lately with PM.  Sorts...ingests.  I'm sure something *must* be wrong, but right now ViewNX runs circles around PM.

[Kirk Baker]

I'm using 4.5.3.

I'm ashamed to say it, but I've started using ViewNX.  Everything seems to take forever lately with PM.  Sorts...ingests.  I'm sure something *must* be wrong, but right now ViewNX runs circles around PM.

I would expect that something must be wrong then with the interaction of your particular system and Photo Mechanic.  Please contact the support folks and let them help you get down to what is going on.  If things were running slowly on my Windows systems, you can bet that I would get it fixed right away.  But Photo Mechanic absolutely flies on my systems, Mac and Windows.

Maybe you could post a screen shot of the main window of Photo Mechanic, showing the Navigator, Favorites, and a typical Contact Sheet.

What kind of "sorts" are you using?  What kind of drives are you working with?  Do you have network volumes?

I really need a lot more information to go on besides "it's just slow everywhere."

I'd be happy to get you a logging version of Photo Mechanic so we can see where time is being spent, if you're interested and have the time to interact with me.

-Kirk

[Kirk Baker]

Primoz,

I would say that at least in Windows (Win XP) this is "normal" thing for long time. Personally I ingest photos to one sub-directory, so main directory contains unchecked "daily" directories (/main_directory/yyyymmdd/). After I pick photos I will send them forward, I copy them to different directory (/main_directory/send/). If PM has this send directory as last used directory, I just press enter fast and it copies them right away. But if I wait, that PM starts to "read" whole tree, it takes those 10-20sec until it allows me to press enter to confirm destination.

In this case it sounds like this has nothing to do with the Navigator, but is the folder picker dialog that appears when you copy or save as.  That dialog is Microsoft's and has none of our code running in it.  So this would be a problem in general with the file system and Microsoft's folder picker dialog.

-Kirk

[Kirk Baker]

Rich,

I encounter this in my MacPRo running both Tiger and Leopard running PC 4.5.2. (Please see my latest entry in a related thread about hanging).  I have a single master folder with about 1000 subfolders for each day's shoot.  When I click on the master folder it takes as much as 4-5 seconds for the folders to appear.  When I use finder or mucommander (a Mac dual frame file manager) the folders appear virtually instantaneously.  The freezing involves selecting and copying images to a Smugmug website window in Safari for upload.  I'm not a programmer but it appears that there is a particular way file and folders are maintained in PM's memory which really slows performance.  The is the only software which exhibits this slow file/folder display.

It is the main purpose of the Finder (or things like mucommander) to provide best-in-class support for navigating the file system.  I would not be surprised at all that they do things like pre-cache directory information on a background thread to keep the main UI responsive.  Photo Mechanic does not do any of this.  When you expand a folder containing 1000 subfolders, the Navigator scans each and every one of them to see if they have any subfolders so that it can predetermine if it should show the disclosure triangle to the left of the folder icon.  Because there is no public API to ask the filesystem if there is at least one subfolder in a given folder, the  entire contents of the folder must be scanned.  So for your master folder this is done one thousand times.  4-5 seconds sounds pretty quick given the circumstances.  I agree that this is slower than one would like but at least you now know the reason why.

I could make it optional to scan for subfolders and always show a disclosure triangle until the folder is attempted to be expanded and then scan only that folder for subfolders, but it would often be misleading, making you think there are subfolders when in fact there may only be images there and no subfolders at all...

-Kirk

P.S. I took a look a mucommander (screenshots from their site only) and it looks like they make no attempts whatsoever to let you know if any folder has child files/folders inside them.  That would indeed make it quite fast.

[keydet]

Kirk,

Be glad to help with whatever's needed.  I was on a bit of a hiatus after my initial post, but I'm back at it now.

Some basic info I just noted -

Startup time: 17 seconds.

Time to sort & display folder of 858 JPG files (mostly 9MB or so D300 files): 2:05 (min:sec).

Normally I use Capture Time or Filename for sorts, very occasionally something like Camera model as a cust sort...but sort criteria don't seem to have noticeable impact.

I am running Vista Ultimate on an AMD Phenom 9500 system w/3.25 GB of RAM and Raptors (system & pagefile) SATA II data drives to hold the JPG files.

I have 1024MB set up for disk cache which is running on a 36GB Raptor drive dedicated to PS/PM caching.  Memory cache is 256MB - perhaps this should be higher?  (I recently upped my system RAM from 2GB.)

I'd be happy to install the 'logging' version of PM if you think that would help.

Thanks,

Chuck

[Kirk Baker]

Chuck,

Be glad to help with whatever's needed.  I was on a bit of a hiatus after my initial post, but I'm back at it now.

Some basic info I just noted -

Startup time: 17 seconds.

Time to sort & display folder of 858 JPG files (mostly 9MB or so D300 files): 2:05 (min:sec).

Normally I use Capture Time or Filename for sorts, very occasionally something like Camera model as a cust sort...but sort criteria don't seem to have noticeable impact.

Anything but Filename sort requires a full scan of each file for its metadata.  What kind of sort time do you get if you use Filename sort only?

I am running Vista Ultimate on an AMD Phenom 9500 system w/3.25 GB of RAM and Raptors (system & pagefile) SATA II data drives to hold the JPG files.

I have 1024MB set up for disk cache which is running on a 36GB Raptor drive dedicated to PS/PM caching.  Memory cache is 256MB - perhaps this should be higher?  (I recently upped my system RAM from 2GB.)

You could try setting your memory cache to 512 which would require less reloading of image data.  But this wouldn't help on rescanning.

Let's eliminate a few variables.  Close down both the Navigator and the Favorites from the View menu and restart PM.  Does it start up faster?  Do the sorts go quicker when you're using Filename sort?

I'd be happy to install the 'logging' version of PM if you think that would help.

It's the only way I will get real data on what is taking time on your system.  But it takes much effort on my part since I have to find the right parts of the program to log.

-Kirk

[keydet]

Time to open folder of 862 files

Capture sort: 1:22
Filename sort: Almost instantaneous

Startup time is cut to about 4 seconds with Navigator closed (I don't use Favorites)...but subsequently displaying the navigator takes another 16 seconds or so.

[Kirk Baker]

Time to open folder of 862 files

Capture sort: 1:22
Filename sort: Almost instantaneous

Startup time is cut to about 4 seconds with Navigator closed (I don't use Favorites)...but subsequently displaying the navigator takes another 16 seconds or so.

OK.  Well the sort time for "Capture Time" is dependent on the time it takes to scan each of your images for their EXIF data (that's where the Capture Date/Time information is stored.)

The speed of the Navigator is dependent on how long it takes to restore the file system tree to the state that it was when you last used Photo Mechanic.  If you had a lot of folders expanded or any of the folders have a lot of subfolders, then this will indeed take some time to complete.

Now you were saying that when doing various operations you would see the Navigator "flashing" about and sort times would continue to be long, is that right?  How would I reproduce this on my system?  What operations do I need to do in order to see this?  A concise set of steps would be most appreciated, including detailing the state of the sort popup menu, settings in the dialogs used to complete a task, etc.

-Kirk