Unreadable File Format

[Graham1]

A new catalogue (small) gave no problems, despite including the same folder which has the .DNG files which are Unrecognized in the All Photos catalogue.  As it now stands, unfortunately the All Photos catalogue does not have any folder with both .CR2 and .DNG images.  I have therefore reverted to a catalogue prepared yesterday which has Unrecognized files, although this has the same limitation on file types (the Unrecognized files are all .CR3). 

I went through basically the same steps as before.  This time, 2 images have a message about Unsupported Format Variant in the Organizer ¦ Browser view, which I do not get with the other catalogue.  These are, however, the same as all the other .CR3 files, and are visible in the Navigator view, but not in the Organizer Browse view, even though the other Unrecognized .CR3 files have temporarily visible thumbnails until PM6+ is restarted.  The count of Unrecognised did not go down, but that is presumably because none of the Unrecognized files stayed recognised after shutdown and restarting (before, the .CR2 files became recognised, the .DNG files remained Unrecognized).

I have zipped up the same kind of series of screen grabs and attach the updated log.

I think that in order to get a catalogue with Unrecognized images in more than one format, I will need to prepare a larger new catalogue, which I will try to do overnight.

Thanks for your continuing help.

Graham

[Kirk Baker]

Graham,

I think that in order to get a catalogue with Unrecognized images in more than one format, I will need to prepare a larger new catalogue, which I will try to do overnight.

Just adding one file of any type to a new catalog and having it become an "Unrecognized File Type" will suffice, no need for there to be more than one format to become unrecognized.

The smaller the catalog can be that produces one file as "Unrecognized File Type", the better.  No need to make a big catalog if you can help it.  We may want you to share the catalog with us and smaller is better is true for that as well.

Thanks,

-Kirk

[Graham1]

I am afraid I am unable to reproduce the issue with a new catalogue created with the new build.  A smaller catalogue (which included the folder with a mixture of .CR2 and .DNG files which gave problems previously) created without error.  I therefore deleted the old 2020 year catalogue which had errors last time, and recreated it overnight (just over 45,000 images): again, no errors.  Is there anything in the new build which has changed, which might have affected this?  I think that all I can do now is to delete my old All Photos catalogue and recreate this, to see if this now completes without errors, but this will take a few days.  I will stress it by adding all my files at the same time: if it survives that without error, it will survive anything!  I will let you know the outcome. 

Thanks.

Graham

[Kirk Baker]

Graham,

I am afraid I am unable to reproduce the issue with a new catalogue created with the new build.  A smaller catalogue (which included the folder with a mixture of .CR2 and .DNG files which gave problems previously) created without error.  I therefore deleted the old 2020 year catalogue which had errors last time, and recreated it overnight (just over 45,000 images): again, no errors.  Is there anything in the new build which has changed, which might have affected this?  I think that all I can do now is to delete my old All Photos catalogue and recreate this, to see if this now completes without errors, but this will take a few days.  I will stress it by adding all my files at the same time: if it survives that without error, it will survive anything!  I will let you know the outcome. 

Thanks.  I look forward to your final results.

-Kirk

[Graham1]

After chugging away for several days, the new all-images catalogue with some 660,000+ files has completed without errors.  It will take another day or so for previews to be fully generated, but I am not expecting this to be a problem.  No repetition of the Unreadable files format files.  It also worked perfectly when I added some new images outside PM6+, then scanned to bring them in and update the catalogue.

Before I consigned the last version of this all-images catalogue to history, I selected then deleted from the catalogue all the unreadable files.  The count went down to zero, as per the attached screen grab, although with zero entries I would have expected the category itself to have disappeared.  With the latest version, there is thankfully no count of unreadable files.

Kirk, thank you for all your help.  Whatever you did to the new version seems to have done the trick.  I will leave logging enabled for the time being, in case the old behaviour reverts, but I do not expect that it will.

Graham

[Kirk Baker]

Graham,

After chugging away for several days, the new all-images catalogue with some 660,000+ files has completed without errors.  It will take another day or so for previews to be fully generated, but I am not expecting this to be a problem.  No repetition of the Unreadable files format files.  It also worked perfectly when I added some new images outside PM6+, then scanned to bring them in and update the catalogue.

Before I consigned the last version of this all-images catalogue to history, I selected then deleted from the catalogue all the unreadable files.  The count went down to zero, as per the attached screen grab, although with zero entries I would have expected the category itself to have disappeared.  With the latest version, there is thankfully no count of unreadable files.

Kirk, thank you for all your help.  Whatever you did to the new version seems to have done the trick.  I will leave logging enabled for the time being, in case the old behaviour reverts, but I do not expect that it will.

Thanks for following up.  I'm not sure what change fixed your problem but I'm glad it appears to be working for you now.

-Kirk

[Graham1]

Sadly, the Unreadable File Format has reared its ugly head again.  The catalogue has been fine for a few days since I created the new all images version with about 660k images.  I have been processing images in Lightroom, and in doing so I have rejected and deleted several: it makes sense to do this in Lightroom as I go along and I find an image that I do not want, or is not technically perfect, rather than go back to PM6+ and delete it from there.  I made a few metadata updates (keywords) in Lightroom to some images I was processing as I went along before exporting the processed images as JPG files: again it makes sense to do that as I go along and the changes occur to me, particularly if I need them for the images I am in the course of processing.  While I was processing only a relatively few files, because of the nature of the Lightroom catalogue the metadata changes would have affected a larger number of images which already had the same keywords in the Lightroom catalogue. 

The processed images are exported as JPG files from Lightroom and are not included in either of the Lightroom or PM6+ catalogues: they are archived off - there is no point cataloguing the JPG files in the main catalogue of either application, because it is simple to recreate the JPG if I need it because all the processing metadata has been saved in the Lightroom catalogue and the keywords and other metadata are in both the Lightroom and (when synced) PM6+ catalogues for the original RAW files.

I knew, therefore, that the PM6+ catalogue would need updating, and ran the Sync Catalogs command after I had quit Lightroom.  Having run this, the next time I looked at the File Type list in Browse, there were Unreadable File Format entries.  Fortunately I had not cleared the Sync Catalogs panel.  The Sync Catalogs panel showed that metadata had changed in 1118 files, which was exactly the same number of new Unreadable File Format files that had appeared in Browse.  I attach screen grabs of both which show this.  I also attach the latest support data log. 

It seems clear, therefore, that changing metadata in Lightroom somehow makes the existing images already catalogued in PM6+ thereafter unreadable in PM6+.  This is despite the metadata having been largely originally created in Lightroom, from use over many years, so one would not expect there to be any inherent incompatibility with PM6+: if PM6+ can read metadata and recognise the associated image files for files catalogued in Lightroom when the PM6+ catalogue is created, why does PM6+ not recognise the files after Lightroom has updated the metadata and PM6+ tries to sync it?

I hope this information helps you to get to the bottom of this.

Graham

[Kirk Baker]

Graham,

Thanks for the additional info.  We have an interim fix for you while we work on a permanent fix.  Please download, install and use this build:

https://www.camerabits.com/download/PMPlusSetupR5657_fb3c6c6a.msi

You will need to use the Browse file type subtree to select the "Unreadable File Format" items for display.  Select them all in the contact sheet view and Include in Catalog (from the Catalog menu).  The count should go to zero but the item in the Browse tree won't disappear.

Any additional Full Sync operations you do should not result in images becoming listed as "Unreadable File Format", though the speed of the Full Sync will decrease until we have the permanent fix in place.

Thanks,

-Kirk

[Graham1]

Many thanks, Kirk.

Graham

[Kirk Baker]

Graham,

This build should solve that problem and be efficient with Full Sync when images haven't changed.

https://www.camerabits.com/download/PMPlusSetupR5667_14372b53.msi

-Kirk

[Graham1]

Many thanks again, Kirk.  Much appreciated.

Graham

[Kirk Baker]

Graham,

Many thanks again, Kirk.  Much appreciated.

Would you mind putting it through its paces and follow the workflow you described earlier and let me know if the problem is fixed?

Thanks,

-Kirk

[Graham1]

Would you mind putting it through its paces and follow the workflow you described earlier and let me know if the problem is fixed?

I ran a Full Sync, and everything seems to be fine.  I attach in the zip file a screen grab which shows files deleted and added, but thankfully no new entries marked as Unreadable in Browse.  The zero length Unreadable entry is still there under File Type in Browse, but you did warn previously that this would still happen.  SQL Analyze (with proxies) is still slow: I ran it immediately after the Full Sync and it took 1356.6 seconds.  This is far less material to me than the catalogue returning the correct data, not least since it should only be required rarely. 

Thanks again for your help.

Graham

Edit: since writing the above, I searched for all (double quotes) which was almost instantaneous.  I then used Edit : Select missing, intending to remove them from the catalogue.  Some three hours later, I am having to terminate this: the progress bar for "Selecting items according to criteria" is still at less than 50%.  Task manager reports that PM+ is using about 50% of the 16GB memory on my desktop computer (no other programmes running other than Chrome to make this post).

[Graham1]

We are not quite out of the woods yet.  I ran Select Missing again overnight.  I do not know how long it took, but it had been running for several hours when I left it, but it did complete overnight.  There were 263 reported missing files.   I then selected Remove Selected Items.  The numbers in Browse | File Type updated accordingly, the removals being from my most recent CR3 file types, which would be as expected, some having been culled in Lightroom a little while ago. 

I then ran a Quick Sync.  I had done nothing with any of my images since the Full Sync run earlier in this series of tests (i.e. since install of the latest version).  Having run a Full Sync, followed by Select Missing then Remove Selected,  the Quick Sync run should have produced no results, but it did: it showed 13 images added.  But the file counts in Browse | File Type were unchanged.  I saved the Quick Sync output as a text file (attached in the zip file) in which I have added *** after the 13 images added for ease of reference.  All these were old images, mostly CR2 files on a disk which holds only old images.  None showed up in the Browse tab when I navigated to them via the Folder path.

On checking, the 13 added images appear to be corrupted.  They were not corrupted by PM+: they do not show up in Lightroom either.  Nevertheless, this leads to these questions:

1. If these 13 corrupt images were not picked up in the Full Sync, why were they picked up in the subsequent Quick Sync?
2. If the catalogue reports that it has added 13 files, why do none of the File Type totals update?  The Unreadable File Format total remains at zero.

It would be useful if newly added images could be shown in a tab: without the saved Quick Sync log and searching through it, I would not have been able to find the added corrupted images (or indeed check what other files had been added, had there been any proper new ones).  Hopefully the attachments will help to make sense of this.

I will now run another search of all the files and Select Missing, to see whether or not it reports the 13 images imported, but invisible to Browse, as missing.  I know this will take some time: I will edit this post with the result when I know it. 

Graham

Edit: cannot confirm result of latest Select Missing.  PM6+ crashed during the process: Bug Splat report sent.  Will try again later.

[Graham1]

I ran Select Missing again.  Approaching 8 hours later (just under 1 hour to read metadata, over 6 1/2 hours for selecting items), the 13 missing corrupt images are identified and removed from the catalogue.  A subsequent Quick Sync then adds them back in, together with a small number of photographs taken earlier today.  Unreadable File Format in Browse still reads zero.  I have attached the latest PM Log zip file in case this is of any help.

Graham