Author Topic: Sync Collection Counts Incorrect  (Read 3732 times)

Offline DavidHoffmanuk

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Sync Collection Counts Incorrect
« on: February 28, 2022, 10:58:58 AM »
Hi Kirk

Apologies for droning on with this. I can keep working, even with these problems, but I'm hoping that more info will be helpful.

I just did a full sync (I was getting anomalous search results but that can wait) and the 'new' collection formed shows as 105 files but clicking it shows just 6. Screenshot.

David

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #1 on: March 01, 2022, 10:25:31 AM »
David,

I just did a full sync and the 'new' collection formed shows as 105 files but clicking it shows just 6. Screenshot.

Please try this build with CATALOG_DEBUG_LOGGING on:

https://www.camerabits.com/download/PhotoMechanicPlusR6331_44568991.dmg

If you're able to reproduce the problem, please generate the zipped log file and post it here, please.

Thanks,

-Kirk

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #2 on: March 03, 2022, 08:17:49 AM »
David,

Could you please try to get the Catalog Sync Collection counts incorrect issue to happen again?  The logging that I added was concentrated around the Sync feature.

Thanks,

-Kirk

Offline DavidHoffmanuk

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Re: Sync Collection Counts Incorrect
« Reply #3 on: March 04, 2022, 11:21:42 AM »
Hi Kirk

Right thread this time now but I apologise for the confused trail I've left.

You're right. The 'combined view' setting is the reason for the counts shown in my collection tab differing from the count in the contact sheet. The numbers do match when that box is unchecked. That's solved that mystery but with 2 test collections I made this morning I selected 2000 and 5000 and the collections that resulted contained 1998 and 4995 images. I made a new 5000 set and added a unique keyword before adding them to a collection. That made a collection containing 5000 as it should - not repeating the earlier behaviour. When I used find/exchange to remove the test keyword the dialog reported having removed it from 441 images, not 5000. A search found no files with that keyword. Doesn't seem to be a problem but I wanted to document it.

I still can't add large numbers of files to a collection.

This is an edit from my last post:
I did a new search for files with no keyword. That found 211996, I tried to make a new collection with them but despite several attempts that collection remained empty. I've seen long delays with large collections in  the past but this was still empty the next morning. I've tried that several times again and the collection remains empty. Making collections with smaller numbers (5000 is the largest I've tried) works as expected.

You asked if I was getting a metadata preload progress dialog before I see items being added to the collection. I frequently find that when I add to a collection either I see no metadata preload progress dialog or I see it but it vanishes after a few seconds. Either way the collection remains empty . A second attempt usually works. Sometimes it needs more attempts but, other than with large numbers of files, it always works in the end.

When I tried again with the 200k+ files this morning -
attempt 1 - dialog showed for about 2 seconds
attempt 2 - dialog showed & progressed to 10k before closing
attempt 3 - dialog showed & progressed to 16k before closing
attempt 4 - dialog opened showing 12k & progressed to 16k before closing
attempt 5 - dialog opened showing 18k & took 3 minutes to progress to 20k before closing.
This was about 7 hours ago and the collection is still empty.

>Could you please try to get the Catalog Sync Collection counts incorrect issue to happen again?

I've just done a full sync. The collection shows as containing 100 new files (I don't think that there are 100) but opening it shows just one - which is indeed a new file and probably the only one.

I'll FTP the log.

David

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #4 on: March 04, 2022, 11:48:25 AM »
David,

Right thread this time now but I apologise for the confused trail I've left.

Thanks.  I appreciate you helping me help you.

You're right. The 'combined view' setting is the reason for the counts shown in my collection tab differing from the count in the contact sheet. The numbers do match when that box is unchecked. That's solved that mystery but with 2 test collections I made this morning I selected 2000 and 5000 and the collections that resulted contained 1998 and 4995 images. I made a new 5000 set and added a unique keyword before adding them to a collection. That made a collection containing 5000 as it should - not repeating the earlier behaviour. When I used find/exchange to remove the test keyword the dialog reported having removed it from 441 images, not 5000. A search found no files with that keyword. Doesn't seem to be a problem but I wanted to document it.

Sounds like a problem to me, but I'm kind of confused as to what you're describing.

I still can't add large numbers of files to a collection.

The collection add process needs some serious optimization.  First, it is slow.  Second, it requires that the contact sheet have all of the metadata for all of the images preloaded before they can be added to a collection.

In the future, I will be looking for a way to make this a non-issue.  In the meantime, you'll get the best results with smaller sets of images being added at once, or by scrolling through all of the images until they load if you're going to add many thousands of images to a collection.

We never expected that users would create collections with thousands of images in them but looking back, we should have expected it.

This is an edit from my last post:
I did a new search for files with no keyword. That found 211996, I tried to make a new collection with them but despite several attempts that collection remained empty. I've seen long delays with large collections in  the past but this was still empty the next morning. I've tried that several times again and the collection remains empty. Making collections with smaller numbers (5000 is the largest I've tried) works as expected.

You asked if I was getting a metadata preload progress dialog before I see items being added to the collection. I frequently find that when I add to a collection either I see no metadata preload progress dialog or I see it but it vanishes after a few seconds. Either way the collection remains empty . A second attempt usually works. Sometimes it needs more attempts but, other than with large numbers of files, it always works in the end.

When I tried again with the 200k+ files this morning -
attempt 1 - dialog showed for about 2 seconds
attempt 2 - dialog showed & progressed to 10k before closing
attempt 3 - dialog showed & progressed to 16k before closing
attempt 4 - dialog opened showing 12k & progressed to 16k before closing
attempt 5 - dialog opened showing 18k & took 3 minutes to progress to 20k before closing.
This was about 7 hours ago and the collection is still empty.

Yes, this is the preload problem.  The thing is, in the catalog database, only the item's ID is needed for inclusion in a collection.  So it should be able to be made much more efficient and reliable.

>Could you please try to get the Catalog Sync Collection counts incorrect issue to happen again?

I've just done a full sync. The collection shows as containing 100 new files (I don't think that there are 100) but opening it shows just one - which is indeed a new file and probably the only one.

I'll FTP the log.

Thanks.

-Kirk

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #5 on: March 04, 2022, 12:26:11 PM »
David,

>Could you please try to get the Catalog Sync Collection counts incorrect issue to happen again?

I've just done a full sync. The collection shows as containing 100 new files (I don't think that there are 100) but opening it shows just one - which is indeed a new file and probably the only one.

I'll FTP the log.

I got the log, thanks.  I see the items being added to the collection, I also see the great majority of them having some issue that causes the retrieving of the metadata to fail. (I did not do an exhaustive search)  They're probably damaged in some way.  I can give you a list of full paths to the files that were added if you'd like.  If they can't be recovered, you may as well move them aside or delete them.

The issue is that they're added to the collection during the scan but then they fail to be added to the catalog completely and the collections are not adjusted to remove them.  When you try to open those collections, only the ones that were fully added to the catalog show up.

-Kirk

Offline DavidHoffmanuk

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Re: Sync Collection Counts Incorrect
« Reply #6 on: March 04, 2022, 11:51:09 PM »
Hi Kirk

Quote
I see the items being added to the collection, I also see the great majority of them having some issue that causes the retrieving of the metadata to fail. (I did not do an exhaustive search)  They're probably damaged in some way.  I can give you a list of full paths to the files that were added if you'd like.  If they can't be recovered, you may as well move them aside or delete them.

I've only been finding an occasional damaged file while working, maybe one every couple of months so, yes please, I'd like that path list. 

David
« Last Edit: March 05, 2022, 01:37:48 AM by DavidHoffmanuk »

Offline DavidHoffmanuk

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Re: Sync Collection Counts Incorrect
« Reply #7 on: March 05, 2022, 02:22:20 AM »
Quote from: DavidHoffmanuk Yesterday at 11:21:42 AM
Quote
You're right. The 'combined view' setting is the reason for the counts shown in my collection tab differing from the count in the contact sheet. The numbers do match when that box is unchecked. That's solved that mystery but with 2 test collections I made this morning I selected 2000 and 5000 and the collections that resulted contained 1998 and 4995 images. I made a new 5000 set and added a unique keyword before adding them to a collection. That made a collection containing 5000 as it should - not repeating the earlier behaviour. When I used find/exchange to remove the test keyword the dialog reported having removed it from 441 images, not 5000. A search found no files with that keyword. Doesn't seem to be a problem but I wanted to document it.

Kirk:
Quote
Sounds like a problem to me, but I'm kind of confused as to what you're describing.

I'm confused too! The behaviours are not consistent.
I made 2 collections from 2000 & 5000 selected images but they only contained 1998 & 4995 images. To try and see which images were in the selection but not in the collection I keyworded all the selected 5000 set with a unique keyword so that I could then remove it from the 4995 images that had made it into the collection leaving, I hoped, only the 5 missing files with that keyword so that I could find them. That didn't work because, this time, the collection I made from those 5000 images did actually contain 5000, not the 4995 I ended up with last time. Having failed to find the 5 files apparently missing in the previous collection I used find/exchange to delete that keyword. On completion that reported having removed it from just 441 images - however there were no images left with the keyword so it had actually been removed from all 5000.

Quote from: DavidHoffmanuk on Yesterday at 11:21:42 AM
Quote
I still can't add large numbers of files to a collection.

Kirk:
Quote
The collection add process needs some serious optimization.  First, it is slow.  Second, it requires that the contact sheet have all of the metadata for all of the images preloaded before they can be added to a collection.

In the future, I will be looking for a way to make this a non-issue.  In the meantime, you'll get the best results with smaller sets of images being added at once, or by scrolling through all of the images until they load if you're going to add many thousands of images to a collection.

It's not just making the collections that is slow. Most metadata operations with large numbers are painfully slow and often seem unnecessary. When PM+ encounters a dud file it abandons the operation. It would be better to skip & report it.

Making a collection with maybe 50 or 100k images in batches of a few thousand is not very practical. Each addition can be quite fast but often it can take quite a long time.

Quote from: DavidHoffmanuk on Yesterday at 11:21:42 AM
Quote
I did a new search for files with no keyword. That found 211996, I tried to make a new collection with them but despite several attempts that collection remained empty... [remainder of quote cut]

Kirk:
Quote
Yes, this is the preload problem.  The thing is, in the catalog database, only the item's ID is needed for inclusion in a collection.  So it should be able to be made much more efficient and reliable.

When you get to start working on the collections it would be very useful if it were possible also to implement some/all of the points in this feature request I made last year:
Quote
I use collections a lot but the forced alphabetical order makes navigation slow and efficient grouping impossible. Moving sub-collections to different parents is too slow and clunky to allow full use of this feature.

The way that this was implemented in Media Pro (called catalog fields there) was more user friendly in this respect and it would add to functionality and ease of use if something similar could be developed in PM+.

Media Pro highlighted the collections that contained selected images, one could ctrl-click a collection to select the contents of a collection in the contact sheet and cmd-click a collection to add it to those already in a contact sheet. I found these features useful and used them often. I hope I'm not alone!

David
« Last Edit: March 05, 2022, 02:41:46 AM by DavidHoffmanuk »

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #8 on: March 05, 2022, 07:06:40 AM »
David,

Quote
I see the items being added to the collection, I also see the great majority of them having some issue that causes the retrieving of the metadata to fail. (I did not do an exhaustive search)  They're probably damaged in some way.  I can give you a list of full paths to the files that were added if you'd like.  If they can't be recovered, you may as well move them aside or delete them.

I've only been finding an occasional damaged file while working, maybe one every couple of months so, yes please, I'd like that path list. 

I'll send you a personal message with the list of paths.

-Kirk

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #9 on: March 05, 2022, 07:10:51 AM »
David,

Quote
Sounds like a problem to me, but I'm kind of confused as to what you're describing.

I'm confused too! The behaviours are not consistent.
I made 2 collections from 2000 & 5000 selected images but they only contained 1998 & 4995 images. To try and see which images were in the selection but not in the collection I keyworded all the selected 5000 set with a unique keyword so that I could then remove it from the 4995 images that had made it into the collection leaving, I hoped, only the 5 missing files with that keyword so that I could find them. That didn't work because, this time, the collection I made from those 5000 images did actually contain 5000, not the 4995 I ended up with last time. Having failed to find the 5 files apparently missing in the previous collection I used find/exchange to delete that keyword. On completion that reported having removed it from just 441 images - however there were no images left with the keyword so it had actually been removed from all 5000.

I'd suggest using the "Export" feature, specifically the "Text Export" template and use the {path} variable, exporting to a single text file.  You'll then have a list of files as a text file.  Do this for the original selection and for the collection.  Post them both here and I can sort and compare them (or you can do that yourself.)

-Kirk

Offline DavidHoffmanuk

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Re: Sync Collection Counts Incorrect
« Reply #10 on: March 05, 2022, 08:40:38 AM »
Thanks Kirk - that was much quicker and easier than my failed keyword kludge. I've found the 2 files, they're 14 year old poor quality scans but that shouldn't matter. They show in PM+ as grey placeholders with a green dot, 'edit' and 'reveal in finder' work. The file is shown as 512x341 px but is a little over 3000x2000px. They have no icons to rotate, magnify or show info so it looks like PM+ knows they are there but can't read them. I can't see anything about them that would be a problem. I've attached one - can you see why PM+ doesn't like it?

In my earlier post I mentioned that I'd selected 5000 files to keyword but when I removed the keyword later the report showed only 441 files changed yet there were no files remaining with that keyword. I now think that the keywording process may have hit an error with one of the files that were absent from the collection and stopped keying. If that's the case it would be better if such a problem had PM+ just skip the problem file(s) and report that at the end rather than stopping with an unknown number of files changed.

David


Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #11 on: March 07, 2022, 09:19:01 AM »
David,

When I view your image with Photo Mechanic Plus, it shows up fine.  I can even preview it at full resolution.

I can't find that file name in the list of files that I sent you so it does not appear to be one of the images contributing the the Collection counts being incorrect.

-Kirk

Offline DavidHoffmanuk

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Re: Sync Collection Counts Incorrect
« Reply #12 on: March 07, 2022, 11:06:00 AM »
I've found about 90 images (I think the number has changed slightly but that may be my poor note making) that show as grey rectangles with size 512x341. They are online & PM+ can open, copy or reveal them. P'shop sees no problem. I can drag them from PM+ to a Media Pro window where they show up fine & behave normally. I can't rotate or zoom them or see any metadata. A batch that I resaved in a new folder with a changed profile & scanned back into PM+ seem OK.

I scanned the unreadable set into another PM+ catalog with the same result.

It's interesting that your system doesn't have this problem. I'm on an old Mac Pro running 10.14.6 is do you have a comparable setup to check with?

Previously when I've had a file show up as a grey blank with size 512x341 I've mostly just deleted it as it's no real use and I've thought it likely a leftover from some web design or something. I now know that at least some of those files were high res files, up to 7000 px long side & 500+ meg with layers. There will be backups but it'll only be if I need one & if I realise it's missing that I can start hunting. That's a loss.

I'll send you a log in case it sheds some light.

David

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24767
    • View Profile
    • Camera Bits, Inc.
Re: Sync Collection Counts Incorrect
« Reply #13 on: March 07, 2022, 02:08:44 PM »
David,

I've found about 90 images (I think the number has changed slightly but that may be my poor note making) that show as grey rectangles with size 512x341. They are online & PM+ can open, copy or reveal them. P'shop sees no problem. I can drag them from PM+ to a Media Pro window where they show up fine & behave normally. I can't rotate or zoom them or see any metadata. A batch that I resaved in a new folder with a changed profile & scanned back into PM+ seem OK.

I scanned the unreadable set into another PM+ catalog with the same result.

It's interesting that your system doesn't have this problem. I'm on an old Mac Pro running 10.14.6 is do you have a comparable setup to check with?

You've just described by development system to a 'T'.  Same computer, same OS version.

It's quite possible that the image you uploaded isn't an actual bit-for-bit copy of your image when downloaded.  You could put it in zip archive instead of posting it directly.

Previously when I've had a file show up as a grey blank with size 512x341 I've mostly just deleted it as it's no real use and I've thought it likely a leftover from some web design or something. I now know that at least some of those files were high res files, up to 7000 px long side & 500+ meg with layers. There will be backups but it'll only be if I need one & if I realise it's missing that I can start hunting. That's a loss.

Hmm. What is a loss?

-Kirk

Offline DavidHoffmanuk

  • Sr. Member
  • ****
  • Posts: 301
    • View Profile
Re: Sync Collection Counts Incorrect
« Reply #14 on: March 08, 2022, 01:35:35 AM »
Kirk:
Quote
It's quite possible that the image you uploaded isn't an actual bit-for-bit copy of your image when downloaded.  You could put it in zip archive instead of posting it directly.

Possibly - but those images also seem fine with all my other apps so I think there must be something about them that my installation of PM+ specifically doesn't like.

Quote from: DavidHoffmanuk on Yesterday at 11:06:00 AM
Quote
Previously when I've had a file show up as a grey blank with size 512x341 I've mostly just deleted it as it's no real use and I've thought it likely a leftover from some web design or something. I now know that at least some of those files were high res files, up to 7000 px long side & 500+ meg with layers. There will be backups but it'll only be if I need one & if I realise it's missing that I can start hunting. That's a loss.

Kirk:
Quote
Hmm. What is a loss?

It's having deleted a high res archive image that I should have kept because I thought it was 512x341 as shown in the contact sheet when it was really >3000x2000.

The way I work when making sets for clients is to search for specific keywords and then choose images from the found set. If an image is missing I may well not realise that and make the set from what I see so the loss is in not having the full set to choose from.

David