Indexing slow

[Matthew]

Hi - using the latest release of PM plus and latest version of MacOS and I’m finding indexing a library of 340k images a lot slower than usual.

Catalog Image Preview Generation is at 0.5-1/sec
Catalog Image Preview Copying is at 0.5-1/sec

I’ve seen these numbers up at 30-40/sec before which makes me wonder if something is up.

Images are on a SSD and catalog process has been running for 2-3 days now. Usually I can get everything indexed within about 24 hours.

At its current rate Catalog Image Preview Copying is going to take 4 days.

[Matthew]

Just a quick update, spotted a whole load of these errors in catalog status. Not seen this problem before...


[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260620_Stock_018.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260620_Stock_019.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260620_Stock_021.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260620_Stock_022.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260620_Stock_023.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260620_Stock_343.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260720_Stock_045.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260720_Stock_046.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_260720_Stock_047.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}
[Mon 16:48:14] {"cmd"=>"proxy_gen", "file"=>"/Volumes/LIBRARY/Dropbox/Pictures/Stock/Sent/COVID/MH_270320_Stock_100.JPG", "msg"=>"image_resize: RuntimeError/bitmap_load_scaled: NRFError: 5"}

[Kirk Baker]

It looks like your files cannot be read.  I noticed that your path contains the name "Dropbox" which usually indicates that files can be copied there but not accessed once they're there.  If you were to open that path and browse in Photo Mechanic as a contact sheet, do any images show up?

-Kirk

[Matthew]

Hi Kirk,

I was able to view the files in Photo Mechanic.

I have since stopped the indexing, deleted the Catalog and then disabled Dropbox (quit the app) and then created a new Catalog and have now started the process again. I've always managed to index the drive with reasonable speed (24 hours) since the first PM Plus beta using this setup.

My understanding is if the Dropbox app isn't running it shouldn't interfere with the indexing process as they are just standard MacOS folders.

I've attached a screenshot of Catalog Status and system profile. There are no errors so far but it does seem much slower.

[Kirk Baker]

Matthew,

I was able to view the files in Photo Mechanic.

I have since stopped the indexing, deleted the Catalog and then disabled Dropbox (quit the app) and then created a new Catalog and have now started the process again. I've always managed to index the drive with reasonable speed (24 hours) since the first PM Plus beta using this setup.

My understanding is if the Dropbox app isn't running it shouldn't interfere with the indexing process as they are just standard MacOS folders.

I've attached a screenshot of Catalog Status and system profile. There are no errors so far but it does seem much slower.

I'd like to see your Photo Mechanic log.  In Photo Mechanic go to the Help menu and choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

[Matthew]

Attached. Thanks.

[Matthew]

Hi Kirk - did you spot any issues in the log?

I’ve now totally removed Dropbox and reindexed the drive and I’m getting the same slow speed.

[Matthew]

Screenshot attached of current situation.

[Matthew]

And this is after installing the latest update which I’ve just seen announced.

[Kirk Baker]

Please post a new log.

Thanks,

-Kirk

[Matthew]

Here you go...

http://incoming.matt-horwood.com/PM_log.zip

[Kirk Baker]

Matthew,

Here you go...

http://incoming.matt-horwood.com/PM_log.zip

Thanks for the log.  There are a great number of files that Photo Mechanic Plus is having trouble with.  Some are JPEGs, some are GIFs, and a bunch of them are movies in various formats: mov, mp4, avi, etc.

Do you have a lot of files that are damaged in some way?  You can have a look at the 'dyn.log' yourself and see the paths to the files that are not loading correctly.  Some of the errors are due to some sort of file error (5), and some are purely bad data (10).

-Kirk

[Kirk Baker]

Matthew,

In Photo Mechanic Plus, go to the Help menu and choose "Edit Debug Settings..." and then set the checkbox "CATALOG_DEBUG_LOGGING", then click the "Close" button.  Next restart Photo Mechanic Plus and let it continue to process images for a while (10 minutes) and then from the Help menu, choose "Reveal Support Data..." and then post the zipped log here, please.

Thanks,

-Kirk

[Matthew]

Running now.

I can see the error messages in the DYN log - there are a number of video files within the library. As for bad data, I'm not aware of a large number of bad files in the library particularly with the most recent indexing. There are possibly a few though from the earlier years. Is this likely to cause a slowing down of indexing?

I'll post the log shortly.

[Matthew]

http://incoming.matt-horwood.com/PM_log2.zip