Removed images remaining in catalog

[Ethan Welty]

Mac OS X 11.4 (20F71)
Photo Mechanic Plus 6245 (14de08a)

I have over one thousand images, all under one shared root directory (but across many child directories) that refuse to be removed. They appear in catalog searching, their paths appear in catalog browsing, but they have an empty status circle. Removing them does nothing. Adding one of these images to the catalog adds it, resulting in duplicate entries, both of which now have green status circles. Removing them both brings me back to the same single image with an empty status circle...

SQL query planning, Reintegrate forgotten catalog, Catalog sync don't seem to help. Haven't yet dared to try Reindex catalog, and would rather not start over from scratch since the catalog contains ~200,000 images.

Edit: Based on other bug reports I've seen, I'll add that the paths to the files are very plain (e.g. "/scans/041217 Mexico").

[Kirk Baker]

Ethan,

Mac OS X 11.4 (20F71)
Photo Mechanic Plus 6245 (14de08a)

I have over one thousand images, all under one shared root directory (but across many child directories) that refuse to be removed. They appear in catalog searching, their paths appear in catalog browsing, but they have an empty status circle. Removing them does nothing. Adding one of these images to the catalog adds it, resulting in duplicate entries, both of which now have green status circles. Removing them both brings me back to the same single image with an empty status circle...

SQL query planning, Reintegrate forgotten catalog, Catalog sync don't seem to help. Haven't yet dared to try Reindex catalog, and would rather not start over from scratch since the catalog contains ~200,000 images.

Edit: Based on other bug reports I've seen, I'll add that the paths to the files are very plain (e.g. "/scans/041217 Mexico").

Reindexing should do no harm.  I'd like to see your Photo Mechanic log.  In Photo Mechanic go to the Help menu and choose "Reveal Support Data..." and moments later a Finder window will come forward with the zipped log file selected.  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your zipped PM log file.

Thanks,

-Kirk

[Ethan Welty]

The log file is attached.

[Kirk Baker]

Ethan,

Thanks for the log.  It appears that your catalog state database is not tracking your catalog(s).  It can be deleted and rebuilt.

First quit Photo Mechanic Plus.
Then open a Finder window.  Use the Go to Folder... command on the Go menu.  Enter:

~/Library/Application Support/Camera Bits, Inc/Photo Mechanic/catalog/state/

You should now see a cat_state.sq3 file.  Rename it by prefixing it with "old_".

Start up Photo Mechanic Plus.  Each of your catalogs will require a "Reintegrate Forgotten Catalog..." operation.
Once that is completed, your catalog state database should be rebuilt.

Try removing those items that wouldn't remove before.

-Kirk

[Ethan Welty]

I followed your instructions. After renaming cat_state.sq3 and restarting Photo Mechanic, the images (in the /scans folder) appeared with a green dot and I was able to remove them from the catalog successfully. I then used Scan to Catalog to repopulate the catalog for the /scans folder. So far so good. However, when I reran Scan to Catalog, the images in the /scans folder were all seen as new and re-added, resulting in duplicates. Removing all these duplicated images with green circles from the catalog, I am back where I was before: single images with empty circles that I can't remove. And so on, in an endless cycle.

New log attached.

[Kirk Baker]

I followed your instructions. After renaming cat_state.sq3 and restarting Photo Mechanic, the images (in the /scans folder) appeared with a green dot and I was able to remove them from the catalog successfully. I then used Scan to Catalog to repopulate the catalog for the /scans folder. So far so good. However, when I reran Scan to Catalog, the images in the /scans folder were all seen as new and re-added, resulting in duplicates. Removing all these duplicated images with green circles from the catalog, I am back where I was before: single images with empty circles that I can't remove. And so on, in an endless cycle.

New log attached.

Very strange.  Unfortunately without logging turned on, I only see where the remove operations are failing.  The cause of the duplicates is not present.  If you wouldn't mind repeating the process, but with logging turned on, I'd appreciate it.  You can do this by going to the Help menu and choosing the "Edit Debug Settings..." command and then setting the checkbox next to CATALOG_DEBUG_LOGGING, clicking the Close button and restarting Photo Mechanic Plus.  Then repeat the process and then get me the log.

Thanks,

-Kirk

[Ethan Welty]

New logs attached.

[Kirk Baker]

Ethan,

Thank you for the updated log and your patience.  Can you give me the full path to a few images that have this behavior?  There's a lot of data in the log and having a few examples (just five would do) will help me work my way through the log.

Thanks,

-Kirk

[Ethan Welty]

The implicated files are all of the images in the top level /scans folder (specifically /volumes/photos/scans). I have not been able to reproduce it for other images yet, nor with just a subdirectory of /scans.

edit: I tried with just the 60 images in the /scans/[undated] folder, and although everything worked as expected (no weird duplication or unremovable images), I noticed that when I run "scan to catalog" a second time, it again triggers metadata and preview updates which takes minutes (!), even though the images are all already in the catalog. Same if I right-click "Include in Catalog" - progress bar "adding x of y images to catalog" for a couple minutes.

New log attached, for good measure.

[Kirk Baker]

Ethan,

Thanks for the updated log.  Is there anything special about the file: "Photos/Scans/130815 Seattle/130815_r0092_11_19.TIF" ?  Does it render?

I see it in the log many dozens of times, generating proxies.

If you were to create a separate catalog (Test) and browse the Scans folder on the Photos drive as a Contact Sheet (use the Navigator) and then select just one image and add it to the Test catalog and then add it again to the Test catalog, do you get a duplicate?  If you could do that (and do as little other operations as possible), I may be able to spot what is going on.

Thanks,

-Kirk

[Ethan Welty]

Nothing special about 130815_r0092_11_19 (it renders fine). That was a file I tested on to see if I could reproduce the issue on a single file.

I made a Test catalog. Added and removed Photos/Scans/130815 Seattle/130815_r0092_11_19.TIF a couple times. All good so far. But that also worked on the original catalog (see the edit in my previous post). So next I used Scan to Catalog on photos/scans. (I'm surprised metadata gathering slows to < 1 image per sec). Then I ran it a second time, and boom: duplicates again. I stopped the second update short. Then selected all the images and hit remove. boom: zombie images in the catalog.

Log again. But it will be very crowded again...

[Kirk Baker]

Ethan,

Nothing special about 130815_r0092_11_19 (it renders fine). That was a file I tested on to see if I could reproduce the issue on a single file.

I made a Test catalog. Added and removed Photos/Scans/130815 Seattle/130815_r0092_11_19.TIF a couple times. All good so far. But that also worked on the original catalog (see the edit in my previous post). So next I used Scan to Catalog on photos/scans. (I'm surprised metadata gathering slows to < 1 image per sec). Then I ran it a second time, and boom: duplicates again. I stopped the second update short. Then selected all the images and hit remove. boom: zombie images in the catalog.

Log again. But it will be very crowded again...

Thanks.  Still hard to tell what's going on (the log is 15000 lines long).  If it's not too much of a problem, could you:

1) Create a new Test catalog.
2) Uncheck any other catalogs that you have.
3) Quit PM Plus.
4) In the Photos/Scans folder, create a new folder, called "Hidden".  Then select all of the other folders except "130815 Seattle" in the Photos/Scans folder and move them into the Hidden folder.
5) Start PM Plus.
6) Add the one Photos/Scans/130815 Seattle/130815_r0092_11_19.TIF file to the Test catalog.  You can add it a few more times.
7) Use Scan to Catalog and add Photos/Scans to the Folders to Scan, add Photos/Scans/Hidden to the Folders to Exclude and do the scan.
8 ) Do the operations you discovered earlier such that you notice duplicates.
9) Generate the log and Quit PM Plus.  Post the log here.  If the Test catalog can be zipped up and posted as well, that would be a bonus.  Just the catalog.pmdb file should be needed.
10) Move your folders in the Hidden folder back out to the Photos/Scans folder.

Thanks,

-Kirk

[Ethan Welty]

Well now I'm really puzzled. With the /scans/hidden arrangement (with a new catalog "Test2"), in your step 8, I was not able to reproduce the duplicates. Scanning to catalog triggered metadata gathering (at 0.3/images/sec, so it took about an hour for 1488 images) and preview generation, even though all the images were already in the catalog, but this did not result in duplicates. So I switched back to the original catalog (named "Catalog"), and now I can't reproduce the duplicate-behavior there either. But now I have a new bug in "Catalog", again only for files in "Photos/Scans": If I remove an image from the catalog, it removes from the catalog (or at least appears to be gone), but still shows with a green circle in the file browser. See "/Photos/Scans/040400 Mexico/040400_r0003_23.TIF" for a few rounds of include/remove in the log. Running catalog reintegration did not help.

[Kirk Baker]

Well now I'm really puzzled. With the /scans/hidden arrangement (with a new catalog "Test2"), in your step 8, I was not able to reproduce the duplicates. Scanning to catalog triggered metadata gathering (at 0.3/images/sec, so it took about an hour for 1488 images) and preview generation, even though all the images were already in the catalog, but this did not result in duplicates.

That alone shouldn't happen and I think it should be investigated.

So I switched back to the original catalog (named "Catalog"), and now I can't reproduce the duplicate-behavior there either. But now I have a new bug in "Catalog", again only for files in "Photos/Scans": If I remove an image from the catalog, it removes from the catalog (or at least appears to be gone), but still shows with a green circle in the file browser. See "/Photos/Scans/040400 Mexico/040400_r0003_23.TIF" for a few rounds of include/remove in the log. Running catalog reintegration did not help.

You did remove (rename) the cat_state.sq3 file earlier, right?  The above indicates a lack of synchronization between the current path to the file and the last known location of the file in the catalog state database.  Are all of the subfolders that were previously in the Scans folder still in the "Hidden" folder?

-Kirk

[Ethan Welty]

Catalog to Sync skips over existing files, but Scan to Catalog seems to always trigger metadata gathering and preview generation. I haven't seen it not do so in all these tests.

For whatever reason, after moving most content of /scans to /scans/hidden, testing on a test catalog, and moving everything back, I can no longer replicate the duplication issue. Images are back in their original position under /scans. After one more time removing cat_state.sq3 and reintegrating, the main catalog seems to be working as expected with the /scans images: no duplicate images if I Scan to Catalog a second time, and I can add and remove images as expected. Unless the issue reappears, I won't be able to get you a new log.

A maybe (not) related issue regarding the files in /scans: Many are large (~200MB+ 48-bit RGB TIFFs), and despite them being in a catalog with presumably complete 1600 px preview proxies, their thumbnails/previews sometimes load very slowly (and closing the contact sheet prematurely results in a spinning wheel for several seconds). Does this sound like an issue with my catalog? Or are proxies only used when the source images are not connected? Is there a way of checking that the proxy for an image exists and is being used?

p.s. I noticed that proxies of small images are larger than the images themselves. It seems a proxy should be no larger than the source image?