speed of preview generation

[DavidHoffmanuk]

2012 Mac Pro 10.14.6

Yesterday I began a 'scan to catalog' of about 180,000 Nikon .NEF files. After the metadata gathering I could see the preview generation running at about 6 batches/second. At the end of the day about half of the previews had been made & I shut down the Mac for the night.

When I started it up this morning the preview generation was running at about 1 batch/second. I quit & restarted PM+ but that made no difference. Activity monitor shows 3 pm-task entries, one at 400-500% cpu.

What determines the rate of preview generation & why would it be so much slower today than yesterday? There's nothing much else running.

David Hoffman

[ahoward]

The speed of the disk where the proxies are being stored is generally the primary factor in how long it takes to do the process. You can send us your log file if you think there was an issue with it that you'd like us to look into.

[DavidHoffmanuk]

The speed of the disk where the proxies are being stored is generally the primary factor in how long it takes to do the process. You can send us your log file if you think there was an issue with it that you'd like us to look into.

The first preview generation, running at about 6 batches/second, was on the same drive & connected to the same port as the second preview generation running at about 1 batch/second. The other conditions were also much the same so this seems unlikely to have been related to the drive itself. Disk First Aid gives the drive a clean bill of health.

The reason that I was running this  'scan to catalog' of about 180,000 Nikon .NEF files (from an external drive) is that I was finding sort operations involving .NEF files on an internal 6TB WD drive were running slowly or (mostly) failing to complete. I wanted to eliminate the drive as the cause so took a backup of that drive and scanned it into a different (test) catalog. That’s when the preview generation speed anomaly appeared but that now seems to be just one aspect of a wider problem.

Once the backup .NEFs had been scanned into the test catalog I repeated some of the sort operations that had failed & they failed in the same way with this new setup (different catalog, different drive). The problem only occurs with .NEF files & with the numbered sort items shown in the dropdown. Filename, mod date and the others in the top part of the menu are fine. I assume that the data for those is readily accessible but that the other sort modes need to read the data from file. The .NEF data is a sidecar .xmp but all the other image formats have the data embedded so perhaps it’s reading the .xmp files that is problematic. My .DNG files have embedded metadata & they, like .TIF, .JPG, .PSD etc all sort quickly.

I tried to attach a couple of recent logs but I get an error that your upload folder is full! I'll try again later.

Let me know if there are any specific tests that might help clarify this problem.

David

[Kirk Baker]

David,

Once the backup .NEFs had been scanned into the test catalog I repeated some of the sort operations that had failed & they failed in the same way with this new setup (different catalog, different drive). The problem only occurs with .NEF files & with the numbered sort items shown in the dropdown. Filename, mod date and the others in the top part of the menu are fine. I assume that the data for those is readily accessible but that the other sort modes need to read the data from file. The .NEF data is a sidecar .xmp but all the other image formats have the data embedded so perhaps it’s reading the .xmp files that is problematic. My .DNG files have embedded metadata & they, like .TIF, .JPG, .PSD etc all sort quickly.

Custom sorts (the ones with numbers) all require sorting in the contact sheet itself and are not accelerated by the catalog database.  This requires fully loading all of the metadata from the images and XMP sidecar files before the sort can begin.  Once obtained, the sort should finish quickly.  The speed of the media where the files are stored is the limiting factor here.

-Kirk

[DavidHoffmanuk]

Custom sorts (the ones with numbers) all require sorting in the contact sheet itself and are not accelerated by the catalog database.  This requires fully loading all of the metadata from the images and XMP sidecar files before the sort can begin.  Once obtained, the sort should finish quickly.  The speed of the media where the files are stored is the limiting factor here.

I could have been more precise. As you write, the actual sort does finish quickly. It’s the preceding gathering of metadata where the problem arises.

When I sort files on that same drive that are not .NEF files the gathering of metadata runs fairly quickly & the sort completes. It's only when the sort involves both large numbers & they are .NEFs that the process fails.

I made a collection of 5,000 .NEFs on that drive & tried various sorts. All ran quickly. I increased the collection to 20,000 & it was still quick. I increased it again to 60,000. Sorts for filename, mod date etc all ran quickly but a sort by filesize started normally preloading metadata but after 4 minutes, at 60% it was clearly slowing. After another 2 minutes it had reached 80% but then it stuck. The notification showing % progress in the middle of the contact sheet disappeared & the progress bar at the bottom right stopped moving. After 30 minutes there had been no change. I selected sort by filename & PM+ responded instantly. From previous tests I think that if I had I left it longer PM+ would have stopped responding.

It appears that PM+ reads embedded metadata quickly but when reading from an .xmp, it takes a progressively longer time to do so as the numbers increase. As a result, with large numbers to read, the process fails. Is there a timeout or a limit to the amount of data that can be held during a sort?

David

[Kirk Baker]

David,

Custom sorts (the ones with numbers) all require sorting in the contact sheet itself and are not accelerated by the catalog database.  This requires fully loading all of the metadata from the images and XMP sidecar files before the sort can begin.  Once obtained, the sort should finish quickly.  The speed of the media where the files are stored is the limiting factor here.

I made a collection of 5,000 .NEFs on that drive & tried various sorts. All ran quickly. I increased the collection to 20,000 & it was still quick. I increased it again to 60,000. Sorts for filename, mod date etc all ran quickly but a sort by filesize started normally preloading metadata but after 4 minutes, at 60% it was clearly slowing. After another 2 minutes it had reached 80% but then it stuck. The notification showing % progress in the middle of the contact sheet disappeared & the progress bar at the bottom right stopped moving. After 30 minutes there had been no change. I selected sort by filename & PM+ responded instantly. From previous tests I think that if I had I left it longer PM+ would have stopped responding.

It appears that PM+ reads embedded metadata quickly but when reading from an .xmp, it takes a progressively longer time to do so as the numbers increase. As a result, with large numbers to read, the process fails. Is there a timeout or a limit to the amount of data that can be held during a sort?

This failure to load metadata is an issue that has been unsolved for some time now.  So far we have been unable to fix it but it is a concern for us.  Generally, when it slows down, you can cancel the preload, wait a few seconds and try again and it will resume with full speed and then the operation will complete.

-Kirk

[DavidHoffmanuk]

Generally, when it slows down, you can cancel the preload, wait a few seconds and try again and it will resume with full speed and then the operation will complete.

That does happen with some other operations that that are preceded by metadata gathering although needing many attempts when large numbers of files are involved. I managed to get a file size sort to work on the 60,000 .NEF set once after 6 or so attempts but not since. The full set has never got past 45%.

This failure to load metadata is an issue that has been unsolved for some time now.  So far we have been unable to fix it but it is a concern for us. 

There's a feature request from last summer for a 'known issues' page. Had that been implemented it would have saved me many hours trying to fix the problem at my end. Perhaps that might now be possible?

David

[Kirk Baker]

David,

There's a feature request from last summer for a 'known issues' page. Had that been implemented it would have saved me many hours trying to fix the problem at my end. Perhaps that might now be possible?

Can you please provide a link to the post?

Thanks,

-Kirk

[DavidHoffmanuk]

Can you please provide a link to the post?

It's in the PM feature request list, not the PM+ one but it's equally applicable.
https://forums.camerabits.com/index.php?topic=15182.0

David

[DavidHoffmanuk]

Hi Kirk

There's a post today 'colour rendering in slide show' that also shows how useful a 'known issues' page would be, perhaps with a pinned link in this forum. FritzImages (the post author) has spent valuable time trying to sort a problem that wasn't fixable - as did I & probably many other users. The same with the failure to load metadata unsolved issue that I spent hours pointlessly trying to fix on my setup.

If there was a page where these issues were listed it would demonstrate Camera Bits' respect for its users' time as well as help to build trust & loyalty.

David

[Kirk Baker]

David,

There's a post today 'colour rendering in slide show' that also shows how useful a 'known issues' page would be, perhaps with a pinned link in this forum. FritzImages (the post author) has spent valuable time trying to sort a problem that wasn't fixable - as did I & probably many other users. The same with the failure to load metadata unsolved issue that I spent hours pointlessly trying to fix on my setup.

If there was a page where these issues were listed it would demonstrate Camera Bits' respect for its users' time as well as help to build trust & loyalty.

I'm not sure we're willing to list every bug we know of publicly.  Some of them are very insignificant and only happen in very rare instances.

Instead, how about you create a thread (I'll pin it so it stays at the top) and the community can manage the list of bugs that are affecting them.

How does that sound?

-Kirk

[Odd Skjaeveland]

...create a thread ... and the community can manage the list of bugs that are affecting them.

Sounds messy to me.

[Kirk Baker]

...create a thread ... and the community can manage the list of bugs that are affecting them.

Sounds messy to me.

What do you suggest instead?

-Kirk

[DavidHoffmanuk]

As Odd Skjaeveland says, I can't see how it wouldn't turn into a long messy string of posts and rapidly become unusable.

Kirk:

What do you suggest instead?

There are very many examples online. These are the first ones that came up in a Google search but there are sure to be other, more appropriate, implementations that could be adapted to suit.

https://support.google.com/chromeosflex/answer/11541806?hl=en
https://www.amazongames.com/en-us/support/new-world/articles/known-issues
https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-21h2

David

[ahoward]

As Odd Skjaeveland says, I can't see how it wouldn't turn into a long messy string of posts and rapidly become unusable.

Kirk:

What do you suggest instead?

There are very many examples online. These are the first ones that came up in a Google search but there are sure to be other, more appropriate, implementations that could be adapted to suit.

https://support.google.com/chromeosflex/answer/11541806?hl=en
https://www.amazongames.com/en-us/support/new-world/articles/known-issues
https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-21h2

David

I've created a known issues page: https://docs.camerabits.com/support/solutions/articles/48001248753
I've tried to add issues have been reported to the support team frequently. Hopefully folks will find this useful.