No images :-(

[DavidHoffmanuk]

Hi Kirk

I said I'd reply in detail today but it's quite complex and I want to keep it as short as possible. Hopefully tomorrow.

David

[DavidHoffmanuk]

Hi Kirk - sorry, this is long.

I didn’t use PM6+ for a few days. When I opened it (by double clicking an alias of the pmdb file if that’s relevant) nothing had changed. A search for “” found nothing. Other searches also found nothing. I looked at the collections tab and the collections were now visible. I opened a few successfully. Back in the search tab and it works as it should. I’ve made no changes to the Mac over the last few days.

The catalog is showing maintenance needed (me too), that window shows that it needs reintegration. The proxy storage window is blank (this is always the case now. I don’t know if this is new or normal). I reset it to ‘with catalog’ and start the reintegration. After a few minutes it gives a success message but it’s still showing that maintenance is needed. I repeat it and get the same result. I quit and restart the app, then the Mac but no change. Quit.

When I open the catalog the next day, once again there are no images or collections found. I try reintegration again to no effect despite a success message and a report showing the expected number of images. I see that no debug items are checked and check the CATALOG_DEBUG_LOGGING item.

I search “” again, find nothing – but the collections now are visible. I open a few and then when I return to the search it works! Quit, restart and it’s still working.

Since then, I’ve had search not working a few times when I open the catalog but collections have been present and after opening one the search works.

The initial ‘no images’ problem occurred while I was trying to remove 1,000 or so missing images that were in a -sync- collection. Nothing appeared to happen and the images remained. I waited for a few minutes then tried again. I tried on different sets and by different ways but the images remained and were still present when I gave up. After the catalog began working again most of those missing images had been removed so I’m wondering if my multiple attempts may have triggered a problem?

All the missing files that now remain despite my attempts to remove them are .NEFs and in subcollections of -sync-.

There seems to be a problem with the -sync- collections. The top level -sync- shows 589 files but opening it shows 236. There are similar large anomalies with its subcollections. Two other collections out of about 20 show one fewer files in the contact sheet than is listed.

I have run some full and quick syncs but that has not changed anything.

Current position

Catalog maintenance is called for but the warning remains despite a ‘success’ message.
Catalog is functional though sometimes I can only get the search to work after opening a collection.
I can’t remove some missing files.
Some collections show an incorrect count
Search fails occasionally either finding nothing from a search that has worked and will work on a future attempt or just taking so long that I have to abandon it and find the files by other means.

My apologies for the length of this but I didn’t want to miss anything that might be relevant.

Any insights or advice will be most welcome, particularly as to the maintenance position. I’ve attached the log.

David

[Kirk Baker]

David,

I didn’t use PM6+ for a few days. When I opened it (by double clicking an alias of the pmdb file if that’s relevant) nothing had changed.

When I double click on a .pmdb file, I get a warning that states that 'There is no application set to open the document "catalog.pmdb"'.  Did you make some file association by clicking on the "Choose Application..." button or by some other means?  I do not recommend doing this at all.

A search for “” found nothing. Other searches also found nothing. I looked at the collections tab and the collections were now visible. I opened a few successfully. Back in the search tab and it works as it should. I’ve made no changes to the Mac over the last few days.

You have so many collections from your previous sync operations that they're taking a significant amount of time to stream in from the Catalog to the application which is making it unresponsive.  Did you remove the "-sync-" collections as I had advised earlier?

The catalog is showing maintenance needed (me too)

I don't understand the 'me too' part.  Can you clarify?

The proxy storage window is blank (this is always the case now. I don’t know if this is new or normal). I reset it to ‘with catalog’ and start the reintegration.

That UI was a work in progress when I built the version you're now using.  It is incomplete and can be ignored.

After a few minutes it gives a success message but it’s still showing that maintenance is needed. I repeat it and get the same result. I quit and restart the app, then the Mac but no change. Quit.

When I open the catalog the next day, once again there are no images or collections found. I try reintegration again to no effect despite a success message and a report showing the expected number of images. I see that no debug items are checked and check the CATALOG_DEBUG_LOGGING item.

I search “” again, find nothing – but the collections now are visible. I open a few and then when I return to the search it works! Quit, restart and it’s still working.

Since then, I’ve had search not working a few times when I open the catalog but collections have been present and after opening one the search works.

The initial ‘no images’ problem occurred while I was trying to remove 1,000 or so missing images that were in a -sync- collection. Nothing appeared to happen and the images remained. I waited for a few minutes then tried again. I tried on different sets and by different ways but the images remained and were still present when I gave up. After the catalog began working again most of those missing images had been removed so I’m wondering if my multiple attempts may have triggered a problem?

Possibly.  But I think the root cause is the huge number of -sync- collections that you have.  Removing them should improve things significantly, I expect.

Catalog maintenance is called for but the warning remains despite a ‘success’ message.
Catalog is functional though sometimes I can only get the search to work after opening a collection.
I can’t remove some missing files.
Some collections show an incorrect count
Search fails occasionally either finding nothing from a search that has worked and will work on a future attempt or just taking so long that I have to abandon it and find the files by other means.

Any insights or advice will be most welcome, particularly as to the maintenance position. I’ve attached the log.

Thanks for the log.  I will analyze it and see if I can spot a definitive cause of the issues you're having.  Please remove all unnecessary -sync- collections.  If you continue to have problems with your catalog, I suggest creating a new one and building from there.

-Kirk

[DavidHoffmanuk]

When I double click on a .pmdb file, I get a warning that states that 'There is no application set to open the document "catalog.pmdb"'.  Did you make some file association by clicking on the "Choose Application..." button or by some other means?  I do not recommend doing this at all.

I don't think I changed anything. The info tab for the catalog.pmdb files shows Photo Mechanic Plus.app (6.0, build 6140 (5a7e829)) as the app to open it. All the other files show Photomechanic 6.app. What should these be?

A search for “” found nothing. Other searches also found nothing. I looked at the collections tab and the collections were now visible. I opened a few successfully. Back in the search tab and it works as it should. I’ve made no changes to the Mac over the last few days.

You have so many collections from your previous sync operations that they're taking a significant amount of time to stream in from the Catalog to the application which is making it unresponsive.  Did you remove the "-sync-" collections as I had advised earlier?

I deleted them when I read your message. It was when I was getting to the end of that process and the missing files weren't being removed (the others were) that PM suddenly failed to find any images. I hadn't known that the -sync- process would lead to a problem and need regular clearing.

The catalog is showing maintenance needed (me too)

I don't understand the 'me too' part.  Can you clarify?

A failed attempt at humour.

After a few minutes it gives a success message but it’s still showing that maintenance is needed. I repeat it and get the same result. I quit and restart the app, then the Mac but no change. Quit.

When I open the catalog the next day, once again there are no images or collections found. I try reintegration again to no effect despite a success message and a report showing the expected number of images. I see that no debug items are checked and check the CATALOG_DEBUG_LOGGING item.

I search “” again, find nothing – but the collections now are visible. I open a few and then when I return to the search it works! Quit, restart and it’s still working.

Since then, I’ve had search not working a few times when I open the catalog but collections have been present and after opening one the search works.

The initial ‘no images’ problem occurred while I was trying to remove 1,000 or so missing images that were in a -sync- collection. Nothing appeared to happen and the images remained. I waited for a few minutes then tried again. I tried on different sets and by different ways but the images remained and were still present when I gave up. After the catalog began working again most of those missing images had been removed so I’m wondering if my multiple attempts may have triggered a problem?

Possibly.  But I think the root cause is the huge number of -sync- collections that you have.  Removing them should improve things significantly, I expect.

The excessive buildup of -sync- collections was removed soon after you told me about it, several days ago. The problems listed at the end of my post are current.

There were about 60 or so unneeded -sync- collections, I'd not really taken any notice of them as they only show as a single collection. Is this a problem only with -sync- collections or is there a practical limit to the number of collections I can have? I was planning to have >100.

Catalog maintenance is called for but the warning remains despite a ‘success’ message.
Catalog is functional though sometimes I can only get the search to work after opening a collection.
I can’t remove some missing files.
Some collections show an incorrect count
Search fails occasionally either finding nothing from a search that has worked and will work on a future attempt or just taking so long that I have to abandon it and find the files by other means.

Any insights or advice will be most welcome, particularly as to the maintenance position. I’ve attached the log.

Thanks for the log.  I will analyze it and see if I can spot a definitive cause of the issues you're having.  Please remove all unnecessary -sync- collections.  If you continue to have problems with your catalog, I suggest creating a new one and building from there.

Building a new one would take several days so is a last resort. If I have to do that, is there a way to rebuild the collections? My 200k+ images are in many different folders scattered over a number of drives. Is there a way for me to make a record of the various folder paths so as to help me to repopulate a new catalog?


David

[Kirk Baker]

When I double click on a .pmdb file, I get a warning that states that 'There is no application set to open the document "catalog.pmdb"'.  Did you make some file association by clicking on the "Choose Application..." button or by some other means?  I do not recommend doing this at all.

I don't think I changed anything. The info tab for the catalog.pmdb files shows Photo Mechanic Plus.app (6.0, build 6140 (5a7e829)) as the app to open it. All the other files show Photomechanic 6.app. What should these be?

There should be no association at all.  I don't know how yours got one.

There were about 60 or so unneeded -sync- collections, I'd not really taken any notice of them as they only show as a single collection. Is this a problem only with -sync- collections or is there a practical limit to the number of collections I can have? I was planning to have >100.

There's no hard limit, but each entry in those trees takes time to fetch from the database and stream over to the application UI.  While it's doing that, the search, filter, and browse features will be less responsive.  Are you going to have many thousands of items in those 100 or so collections?

Building a new one would take several days so is a last resort. If I have to do that, is there a way to rebuild the collections? My 200k+ images are in many different folders scattered over a number of drives. Is there a way for me to make a record of the various folder paths so as to help me to repopulate a new catalog?

Collections are the only data in a catalog that cannot be rebuilt, everything else in a catalog comes from the filesystem data or the metadata in the images themselves.

-Kirk

[DavidHoffmanuk]

There should be no association at all.  I don't know how yours got one.

I can't see any way to have no association for the pmdb files. In the 'Open with' dropdown in the Info panel I can only choose other apps. There's no option to leave it blank. If this matters then how can I leave the file with no association?

Quote from: DavidHoffmanuk on Today at 11:21:10 AM
There were about 60 or so unneeded -sync- collections, I'd not really taken any notice of them as they only show as a single collection. Is this a problem only with -sync- collections or is there a practical limit to the number of collections I can have? I was planning to have >100.

There's no hard limit, but each entry in those trees takes time to fetch from the database and stream over to the application UI.  While it's doing that, the search, filter, and browse features will be less responsive.  Are you going to have many thousands of items in those 100 or so collections?

Probably in the low thousands. Some will have just a dozen or so but others can have many hundreds. Are these is going to be a problem or slow down responsiveness just by their presence, even when they are not specifically in use?

Quote from: DavidHoffmanuk on Today at 11:21:10 AM
Building a new one would take several days so is a last resort. If I have to do that, is there a way to rebuild the collections? My 200k+ images are in many different folders scattered over a number of drives. Is there a way for me to make a record of the various folder paths so as to help me to repopulate a new catalog?

Collections are the only data in a catalog that cannot be rebuilt, everything else in a catalog comes from the filesystem data or the metadata in the images themselves.

I don't quite follow yet. I can re-make the collections by adding an appropriate keyword (that is not itself a keyword eg #methane) to its contents but if I forget the catalog and create a new one then what would I need to do to repopulate it with the same content?

David

[Kirk Baker]

David,

There should be no association at all.  I don't know how yours got one.

I can't see any way to have no association for the pmdb files. In the 'Open with' dropdown in the Info panel I can only choose other apps. There's no option to leave it blank. If this matters then how can I leave the file with no association?

There's some control panel that can remove associations from file types but I've never used it.  I don't think there's any real problem with you having that association, however, I recommend that you never launch Photo Mechanic Plus by double-clicking on the catalog.pmdb file.  There is no specific code in the app for reacting to a catalog file being given as an 'open' argument.  Use the Catalog Management window if you need to open a catalog that you had previously forgotten or copied over from another system.

Quote from: DavidHoffmanuk on Today at 11:21:10 AM
There were about 60 or so unneeded -sync- collections, I'd not really taken any notice of them as they only show as a single collection. Is this a problem only with -sync- collections or is there a practical limit to the number of collections I can have? I was planning to have >100.

There's no hard limit, but each entry in those trees takes time to fetch from the database and stream over to the application UI.  While it's doing that, the search, filter, and browse features will be less responsive.  Are you going to have many thousands of items in those 100 or so collections?

Probably in the low thousands. Some will have just a dozen or so but others can have many hundreds. Are these is going to be a problem or slow down responsiveness just by their presence, even when they are not specifically in use?

Data transfer from the catalog database to the application UI takes time.  There may be some optimizations that can be made to make the effect of many thousands of collections have less performance impact.

Quote from: DavidHoffmanuk on Today at 11:21:10 AM
Building a new one would take several days so is a last resort. If I have to do that, is there a way to rebuild the collections? My 200k+ images are in many different folders scattered over a number of drives. Is there a way for me to make a record of the various folder paths so as to help me to repopulate a new catalog?

Collections are the only data in a catalog that cannot be rebuilt, everything else in a catalog comes from the filesystem data or the metadata in the images themselves.

I don't quite follow yet. I can re-make the collections by adding an appropriate keyword (that is not itself a keyword eg #methane) to its contents but if I forget the catalog and create a new one then what would I need to do to repopulate it with the same content?

You would have to create the collection tree and choose which images to add into the various levels of your collection tree.  We're going to need to be able to export/import this data so that the work you do to create them is not lost.  Meaning, new features will need to be added.

-Kirk

[DavidHoffmanuk]

Thanks for this information Kirk. It seems that several of the issues I raise do not currently have a solution but I can work round them and I will await further development of PM+.

However I am concerned that I have no solution to these issues below and that they may be more problematic:

1 Catalog maintenance is called for but the warning remains despite a ‘success’ message. I don't know if this is a spurious warning or a failure to do the required maintenance. What might be the consequences of not doing the maintenance when requested?

2 I can’t remove the older missing file collections.

3 Some collections show an incorrect count.

I've attached a log.

Thanks
David

[Kirk Baker]

David,

Thanks for this information Kirk. It seems that several of the issues I raise do not currently have a solution but I can work round them and I will await further development of PM+.

However I am concerned that I have no solution to these issues below and that they may be more problematic:

1 Catalog maintenance is called for but the warning remains despite a ‘success’ message. I don't know if this is a spurious warning or a failure to do the required maintenance. What might be the consequences of not doing the maintenance when requested?

It means that some of the images being tracked by the catalog may still be missing (unresolved) and those images will show as offline.

-Kirk

[DavidHoffmanuk]

Quote from: DavidHoffmanuk on November 16, 2021, 06:42:14 AM
Thanks for this information Kirk. It seems that several of the issues I raise do not currently have a solution but I can work round them and I will await further development of PM+.

However I am concerned that I have no solution to these issues below and that they may be more problematic:

1 Catalog maintenance is called for but the warning remains despite a ‘success’ message. I don't know if this is a spurious warning or a failure to do the required maintenance. What might be the consequences of not doing the maintenance when requested?

It means that some of the images being tracked by the catalog may still be missing (unresolved) and those images will show as offline.

Thanks Kirk

Should I take this as meaning that there is currently no solution for the maintenance failure or for the collections that I cannot delete other than scrapping the catalog and starting again?

David

[Kirk Baker]

David,

Quote from: DavidHoffmanuk on November 16, 2021, 06:42:14 AM
Thanks for this information Kirk. It seems that several of the issues I raise do not currently have a solution but I can work round them and I will await further development of PM+.

However I am concerned that I have no solution to these issues below and that they may be more problematic:

1 Catalog maintenance is called for but the warning remains despite a ‘success’ message. I don't know if this is a spurious warning or a failure to do the required maintenance. What might be the consequences of not doing the maintenance when requested?

It means that some of the images being tracked by the catalog may still be missing (unresolved) and those images will show as offline.

Should I take this as meaning that there is currently no solution for the maintenance failure or for the collections that I cannot delete other than scrapping the catalog and starting again?

You could try the Forget option in Catalog Maintenance and then use the Locate Existing Catalog to open it again.  It will require a Reintegrate, but perhaps it will do a better job when it starts with a clean slate.

Note: Reintegrate will be unable to resolve any images that are on media that is not available at the time of the reintegrate operation and while it will list the overall operation as a success, it will still show as needing a reintegrate because it was unable to resolve 100% of the images in the catalog.

-Kirk

[DavidHoffmanuk]

You could try the Forget option in Catalog Maintenance and then use the Locate Existing Catalog to open it again.  It will require a Reintegrate, but perhaps it will do a better job when it starts with a clean slate.

Note: Reintegrate will be unable to resolve any images that are on media that is not available at the time of the reintegrate operation and while it will list the overall operation as a success, it will still show as needing a reintegrate because it was unable to resolve 100% of the images in the catalog.

Forget/relocate didn't change anything. I think my best bet now is to carry on with this slightly broken catalog while I build a new catalog to replace it with. That'll take a couple of days for >200k files I think. I can add the collection name with a special character added as a keyword to the contents of each collection. That'll make it easy to remake the collections.

Is there a way to extract a list of the folder paths from the damaged catalog? The files are scattered over many drives and folders and it'd save time reconstructing the collection if I could quickly add those paths to ingest the files to the new catalog.

I have one drive with 20-30,000 files that is normally offline but that I want to include in the catalog. There will be metadata changes to those files made while they're offline and those files will be in collections that get changed too. Is that likely to be a problem?

David

[Kirk Baker]

David,

You could try the Forget option in Catalog Maintenance and then use the Locate Existing Catalog to open it again.  It will require a Reintegrate, but perhaps it will do a better job when it starts with a clean slate.

Note: Reintegrate will be unable to resolve any images that are on media that is not available at the time of the reintegrate operation and while it will list the overall operation as a success, it will still show as needing a reintegrate because it was unable to resolve 100% of the images in the catalog.

Forget/relocate didn't change anything. I think my best bet now is to carry on with this slightly broken catalog while I build a new catalog to replace it with. That'll take a couple of days for >200k files I think. I can add the collection name with a special character added as a keyword to the contents of each collection. That'll make it easy to remake the collections.

Is there a way to extract a list of the folder paths from the damaged catalog? The files are scattered over many drives and folders and it'd save time reconstructing the collection if I could quickly add those paths to ingest the files to the new catalog.

No, there is no way to extract a list of folder paths.

I have one drive with 20-30,000 files that is normally offline but that I want to include in the catalog. There will be metadata changes to those files made while they're offline and those files will be in collections that get changed too. Is that likely to be a problem?

I don't think so, no.  I think the problems with your catalog are due to either crashes or force quitting when the system became non-responsive due to the large number of sync collections created by repeated use of the Catalog Sync function.  I think that if you eliminate them after you use them then you won't encounter this problem.

-Kirk