Author Topic: How can I remove phantom offline files?  (Read 5729 times)

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
How can I remove phantom offline files?
« on: March 21, 2022, 12:36:37 AM »
All my catalogued images are always online, on internal hard disks.  I recently ran a full catalogue sync, to update changes in metadata and add new files.  The sync collection reported 470 images offline (which is impossible).  Missing images, deleted outside PM+, were correctly identified and I was able to remove them from the catalogue, and new files were correctly added.

All but one of these phantom offline files are in one folder.  The contact sheet has no thumbnails, but the little dot in the footer seems to indicate they are online.  The info panel in the viewer shows the following nonsense path:

File name: _MG_7543.CR3
Folder: 2021%20Fungi
Path: amdoc:\\d51f4aa9-4676-4009-5cce-269acb69ba19\@Images_19%20(2021)\2021\

The file is not corrupted and is visible via browsing the catalogue or the navigator.  There the viewer info reports (correctly):

File name: _MG_7543.CR3
Folder: night
Path: I:\@Images_19 (2021)\2021\2021 Fungi\

I cannot get rid of the phantom offline files from the catalogue.  The option to remove from catalogue from the contact sheet derived from the collection has the remove from catalog option greyed out.  I have run the Optimize SQL Query Planning for the catalogue, but it made no difference.

How can I get rid of these phantom entries and stop them coming up every time I run a catalogue sync?

I run Windows 10, the PM+ version is the latest 6375.

Thank you.

Graham




Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #1 on: March 21, 2022, 09:37:58 AM »
Graham,

All but one of these phantom offline files are in one folder.  The contact sheet has no thumbnails, but the little dot in the footer seems to indicate they are online.  The info panel in the viewer shows the following nonsense path:

File name: _MG_7543.CR3
Folder: 2021%20Fungi
Path: amdoc:\\d51f4aa9-4676-4009-5cce-269acb69ba19\@Images_19%20(2021)\2021\

How is it that you're getting to this contact sheet?

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: How can I remove phantom offline files?
« Reply #2 on: March 21, 2022, 12:41:53 PM »
I get the contact sheet by clicking on the numbers shown on the right in the sync Collection after the sync has completed. This is the number of so-called offline files reported in the Collection.

Graham
« Last Edit: March 21, 2022, 02:13:30 PM by Graham1 »

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #3 on: March 21, 2022, 02:48:32 PM »
Graham,

I get the contact sheet by clicking on the numbers shown on the right in the sync Collection after the sync has completed. This is the number of so-called offline files reported in the Collection.

I suggest trying a "Re-index Catalog" maintenance operation on your catalog.  If that doesn't correct the problem, please let me know.  The new 6375 build has some cleanup tasks that are performed as part of re-indexing a catalog.

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: How can I remove phantom offline files?
« Reply #4 on: March 22, 2022, 12:40:59 PM »
Ran a re-index.  A new Full Sync is still running, but I can see it is already still reporting 470 files offline.

Graham

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #5 on: March 22, 2022, 01:18:56 PM »
Graham,

Ran a re-index.  A new Full Sync is still running, but I can see it is already still reporting 470 files offline.

In order to look into this further, I'll need two databases from you.  One is the catalog database (catalog.pmdb) and the other is the catalog state database (cat_state.sq3).  They can be quite large.  Let me know if you'd like to proceed.

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: How can I remove phantom offline files?
« Reply #6 on: March 22, 2022, 04:28:41 PM »
Thanks, Kirk

Happy to send.  Even zipped, these come to 11.2 GB.  How is it best to send the zip file to you?  Perhaps WeTransfer?  If so, could you please confirm the email address I should use.

Graham

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #7 on: March 22, 2022, 05:18:30 PM »
Graham,

Happy to send.  Even zipped, these come to 11.2 GB.  How is it best to send the zip file to you?  Perhaps WeTransfer?  If so, could you please confirm the email address I should use.

I have sent you a personal message.

-Kirk

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #8 on: March 23, 2022, 11:07:42 AM »
Graham,

All my catalogued images are always online, on internal hard disks.  I recently ran a full catalogue sync, to update changes in metadata and add new files.  The sync collection reported 470 images offline (which is impossible).  Missing images, deleted outside PM+, were correctly identified and I was able to remove them from the catalogue, and new files were correctly added.

All but one of these phantom offline files are in one folder.  The contact sheet has no thumbnails, but the little dot in the footer seems to indicate they are online.  The info panel in the viewer shows the following nonsense path:

File name: _MG_7543.CR3
Folder: 2021%20Fungi
Path: amdoc:\\d51f4aa9-4676-4009-5cce-269acb69ba19\@Images_19%20(2021)\2021\

I now have your catalog and catalog state databases.  Can you generate a full list of the phantom offline files and get it to me, please?  I think you should be able to open the offline collection, select all items, then use the Export command on the File menu, then select the Text Exporter template.  Leave the header and footer blank.  In the central text field enter:

{folderpath}{filename}

(NOTE: no space needed, no new line needed.)

In the Saving box, make sure to Save all as one file.  Attach the file to your reply or give me a personal message and paste the contents of the file in your message.

Thanks,

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: How can I remove phantom offline files?
« Reply #9 on: March 23, 2022, 03:42:41 PM »
Kirk

Thank you for your reply.  I attach the export file.  I hope I have followed your instructions correctly, but if not, please let me know.

Thank you.

Graham

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #10 on: March 24, 2022, 06:31:49 PM »
Graham,

On your I:\ drive, there should be a ".pmshare" file.  It's just a text file.  Can you get it to me, please?  Use the 'Attachments and other options' link when you're composing your reply to this message and there you'll be able to upload your .pmshare file.

Thanks,

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: How can I remove phantom offline files?
« Reply #11 on: March 24, 2022, 10:28:27 PM »
Kirk

.pmshare file attached as requested.  Thanks.

Graham

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #12 on: March 25, 2022, 09:25:15 AM »
Graham,

Thanks for the file.  Can you tell me what kinds of operations have happened between when you first cataloged those phantom files and now?  At one point the .pmshare file had the value d51f4aa9-4676-4009-5cce-269acb69ba19 but now it has the value 587a6675-3808-4519-4235-3bacd968b1b4.  The phantom files have the share ID of d51f4aa9-4676-4009-5cce-269acb69ba19, and the duplicates (the ones that show in the catalog as green when they're online) have the new value.

The cat_state database (which contains a copy of these IDs) do not have the share ID of d51f4aa9-4676-4009-5cce-269acb69ba19 so the files with that share ID are considered unknown.

It seems like something deleted the original .pmshare file and then the catalog system made a new one when files from that disk were added/sync'd to the catalog.

-Kirk

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 25020
    • View Profile
    • Camera Bits, Inc.
Re: How can I remove phantom offline files?
« Reply #13 on: March 25, 2022, 11:05:27 AM »
Graham,

Could I have you try and remove the phantom items again with CATALOG_DEBUG_LOGGING turned on?  You can do this by going to the Help menu and choosing the "Edit Debug Settings..." command and then setting the checkbox next to CATALOG_DEBUG_LOGGING, clicking the Close button and restarting Photo Mechanic Plus.  Then after restart try to remove the phantom items from the catalog.

And instead of using the contextual menu, use the Catalog menu's "Remove from Catalog" menu item.

Then generate the zipped log file and post it here.

Thanks,

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: How can I remove phantom offline files?
« Reply #14 on: March 27, 2022, 04:07:09 AM »
Kirk,

Thank you for your continued efforts to look into this. I attach the PMDebug log file, but I suspect it will not tell you anything useful - it appears only to record debugging being switched off and on (before doing this today, it was already switched on).

I tried the menu removal command, and removal worked from that, according to the collection.  I will rerun a full sync and let you know if they return on a new sync. [Update: the new full sync returns no offline images.  It looks as if the menu command works even if the right click contextual menu is greyed out]

Replying to your previous post questions, I cannot think of anything that may have happened between the phantom files having been originally catalogued and now.  The I: drive is a 6 TB Seagate Barracuda SATA drive, it was the last new internal drive I installed a little under 3 years ago.  It contains image folders, 3 folders of Macrium daily incremental backup files and an old Capture One catalogue that I have not accessed for about 9 months and which does not contain the originals of the phantom files.  It does not store any system temporary or cache files.  It remains my current disk for storing new images as and when they are added.  I have not reorganised the images on this drive beyond the addition of new images as they have been taken, adding derivative Photoshop images made from the RAW files and deletions from occasional culling.   I often tweak keywords which will mean that the .xmp sidecar files will have often changed and also the colour labels, which I amend to group panorama files visually and to reflect what I have done with exported images.  I cannot think of any external operations that might have deleted the root level .pmshare file.  I have not re-organised or re-allocated the drive letter for this drive since it was originally installed.  Sorry that this is not very helpful, but as far as my use of it is concerned, it is only for image storage and backups. 

Graham
« Last Edit: March 27, 2022, 09:28:00 AM by Graham1 »