Author Topic: Unreadable File Format  (Read 10160 times)

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Unreadable File Format
« on: February 21, 2021, 03:04:11 PM »
Executive summary: my large all-images catalogue is coming up with a significant number of "Unreadable File Format" images. 

Detail: Over the last few days I have been re-creating a master catalogue for all my 650k+ images.

I created the catalogue by scanning the master folders (arranged by year) in the various disks (6, all local) on which the images are held on my computer, limiting the file types scanned to the usual RAW and graphics file formats that I have in my collection.  So as not to overstress anything, I scanned the folders one disk at a time, waiting until the previous one had indexed (but not waiting for all the thumbnails to generate or all the metadata to have been written). 

When scanning was complete, I found I had 1445 images listed in Browse as File Type: "Unreadable File Format (????)" [this should be a row of 4 question marks, but the first three are converted into an emoji].  These images are seemingly randomly scattered across 4 of my 6 hard disks, spread across various folders.  The file extensions (which I can see from the red file names under the blank thumbnails) vary, covering a mix of .CR2, .CR3 and .DNG files.

I tried re syncing, Optimize SQL, Reintegrate Forgotten Catalog and Re-index Catalog.  None made any difference: there were still all these Unreadable File Format files.  I tried to write a keyword such as "unreadable" to them using the Metadata (IPTC) Template, so that I could easily find them in Lightroom, but there was an error message and PM6+ would not write to the files.

I then did several things.  Firstly I selected the 1445 files from Browse and removed them from the catalog.  Strangely, Browse still showed a "Unreadable File Format (????)" entry, with zero images (I would have expected the entry to have disappeared if there are no matching images).

Then I ran a full Sync Catalog which re-imported the images, still reported as "Unreadable File Format (????)": nothing changed.

Having done that, I selected from Browse the same 1445 files and copied them to a new catalogue, which I called "Unreadable File Format".  The files remained unreadable in that new catalogue, even after all 3 Catalog Management tests. 

There is, however, nothing wrong with any of the files (I have not checked all of them, but I have checked enough to be sure that this is an accurate statement).  They are all fine in Lightroom, and indeed those reported as unreadable include some images which were only taken or processed a few days ago. 

As a further check, I produced new catalogues of just some of the year folders that contain "unreadable" files.  The images are fine in those separate, smaller catalogues, proving that the files really are readable by PM6+ and that there is nothing wrong with the files.

My PM6+ catalogues, thumbnails and cache are stored on a large internal SSD with plenty of free space.  I am running Windows 10, using the latest version (5560) of PM6+.  My computer is a fast desktop machine with plenty of RAM: I have been keeping an eye on Task Manager and neither CPU not RAM have been anywhere near being maxed out. My NVIDIA graphics card drivers are fully up to date.

Any idea what is happening, and how I can get my main catalogue to recognise the Unreadable File Format files correctly (I prefer to have one single catalogue for everything, not least for collections to work as I intend)?

I have one theory, that may be totally wrong, but I will include it for what it is worth.  Looking at the "unreadable" images in Lightroom, many of them have a colour label in Lightroom.  Some of them used to have a colour label (I colour code those I am considering for processing, then remove the colour label if I do not use them), but no longer have one.  It seemed to me that there may be a preponderance of presently or previously colour-labelled images amongst the "unreadable" images.  There are still many that have no colour label, and for the older ones I simply cannot remember whether or not they might once have had a colour label applied and then removed.  Tens of thousands of images with colour labels are, however, correctly imported, so it cannot be the mere existence of a colour label that is the determining factor.  This may, of course, be totally irrelevant, since they are correctly scanned into the smaller single year PM6+ catalogues.  All of the images have keywords and most have a caption.

Any thoughts welcome.  Thank you.

Graham


Offline Dub

  • Full Member
  • ***
  • Posts: 164
    • View Profile
    • instantsphotos
Re: Unreadable File Format
« Reply #1 on: February 21, 2021, 10:35:05 PM »
Perhaps make one of these unreadable files available, via WeTransfer, for example, to test on our systems ...

 ;)
French and "English" with DeepL ¯\_(ツ)_/¯
MPB 13" M1 / Monterey
https://instantsphotos.fr/

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Unreadable File Format
« Reply #2 on: February 22, 2021, 02:33:40 AM »
Perhaps make one of these unreadable files available, via WeTransfer, for example, to test on our systems ...

 ;)
I am happy to supply samples to Camerabits (only) if they request them, but the fact that the images are properly read in smaller catalogues proves that there is nothing wrong with them.

Graham

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Unreadable File Format
« Reply #3 on: February 22, 2021, 04:39:17 AM »
Unfortunately, it gets worse.   My much smaller 2021 (only) catalogue is now showing 118 out of 1517 images as unreadable.  It was fine when I created it yesterday.  Today I have been processing some of the files in Lightroom.  I did a full Sync Catalog rescan afterwards, to pick up new panoramas created and some changes in metadata, whereupon all these images showed up as unreadable.  Nor does Select Missing work to select the images I deleted while working in Lightroom earlier. 

The catalog, while not able to read the images, can read some of their metadata.  All of the images showing as unreadable are this time Color Class "None", which means my theory in my post yesterday about unreadability being linked to colour class is wrong.

Graham

Offline jose

  • Member
  • **
  • Posts: 52
    • View Profile
Re: Unreadable File Format
« Reply #4 on: February 22, 2021, 08:20:24 AM »
Can you recover any of the problematic images from any PREVIOUS backup to these issues to test with them?

jose.

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Unreadable File Format
« Reply #5 on: February 22, 2021, 09:09:21 AM »
Can you recover any of the problematic images from any PREVIOUS backup to these issues to test with them?

jose.
There is nothing wrong with the images.  This becomes increasingly strange.  I tried opening a contact sheet from the Navigator panel for one of the folders which has unreadable files according to the Catalog.  They are fully readable in that contact sheet, including their metadata.  When I return to my catalog and look at Browse, these files are now visible - but still listed as Unreadable File Format.  It is really strange that the Navigator can see them but the Catalog Browse cannot.

I just went back into my main all photos catalogue (which was not open when I went into a couple of the folders via Navigator, as described above).  Now if I Browse to Unreadable File Format in that catalog, the thumbnails from the previously unreadable images (incidentally, not the same ones as are said to be unreadable in my smaller 2021 catalog) in the folders I opened in Navigator are visible, but the rest remain blank, with just a red file name underneath.  But the newly read files remain part of the 1445 Unreadable File Format images.

I am now going to run a Quick Catalog Sync to see whether or not any more of the Unreadable File Format files are read.  I will report back once it has finished, although when I tried this yesterday, it crashed part way through.
Update: Quick Catalog Sync made no difference to reported unreadable files.

Graham

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24768
    • View Profile
    • Camera Bits, Inc.
Re: Unreadable File Format
« Reply #6 on: February 22, 2021, 09:51:09 AM »
Graham,

Executive summary: my large all-images catalogue is coming up with a significant number of "Unreadable File Format" images. 

Detail: Over the last few days I have been re-creating a master catalogue for all my 650k+ images.

I created the catalogue by scanning the master folders (arranged by year) in the various disks (6, all local) on which the images are held on my computer, limiting the file types scanned to the usual RAW and graphics file formats that I have in my collection.  So as not to overstress anything, I scanned the folders one disk at a time, waiting until the previous one had indexed (but not waiting for all the thumbnails to generate or all the metadata to have been written). 

When scanning was complete, I found I had 1445 images listed in Browse as File Type: "Unreadable File Format (????)" [this should be a row of 4 question marks, but the first three are converted into an emoji].  These images are seemingly randomly scattered across 4 of my 6 hard disks, spread across various folders.  The file extensions (which I can see from the red file names under the blank thumbnails) vary, covering a mix of .CR2, .CR3 and .DNG files.

I then did several things.  Firstly I selected the 1445 files from Browse and removed them from the catalog.  Strangely, Browse still showed a "Unreadable File Format (????)" entry, with zero images (I would have expected the entry to have disappeared if there are no matching images).

Having done that, I selected from Browse the same 1445 files and copied them to a new catalogue, which I called "Unreadable File Format".  The files remained unreadable in that new catalogue, even after all 3 Catalog Management tests. 

There is, however, nothing wrong with any of the files (I have not checked all of them, but I have checked enough to be sure that this is an accurate statement).  They are all fine in Lightroom, and indeed those reported as unreadable include some images which were only taken or processed a few days ago. 

As a further check, I produced new catalogues of just some of the year folders that contain "unreadable" files.  The images are fine in those separate, smaller catalogues, proving that the files really are readable by PM6+ and that there is nothing wrong with the files.

Any idea what is happening, and how I can get my main catalogue to recognise the Unreadable File Format files correctly (I prefer to have one single catalogue for everything, not least for collections to work as I intend)?

I don't have a solution for you at this time, but I'd like some more information from you.

For those images that show as "Unreadable File Format", if you select one of them and look at the status bar, what is the path that is shown?
If you open the folder containing that image via the Navigator and select that same image, what is the path that is shown in the status bar?

Thanks,

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Unreadable File Format
« Reply #7 on: February 22, 2021, 01:09:26 PM »
Graham,

For those images that show as "Unreadable File Format", if you select one of them and look at the status bar, what is the path that is shown?
If you open the folder containing that image via the Navigator and select that same image, what is the path that is shown in the status bar?

Thanks,

-Kirk
Hi, Kirk

I have chosen one at random.  That happens to be G:\@Images_12 (2014)\2014\2014 South America\1.Peru\1.1 Lima\IMG_8095.CR2, as per Organizer status bar.    If I then go to Navigator and open the 1.1 Lima sub-folder in a new contact sheet, the entry on the status bar is exactly the same.  The difference is that I can see the thumbnail and read the metadata in the Navigator contact sheet.  Having looked at it in Navigator, I go back to Organizer: it is no longer in Browse ¦ Unreadable File Format, but it can be found (complete with thumbnail and readable metadata) in Browse ¦ Canon CR2 (CCR2). 

All of the unreadable format images are accessible also by right clicking to show in Windows Explorer: failure to recognise the path does not seem to be the issue: that is how I was able initially to verify that the files themselves are not corrupt.  This path is pretty typical for all my files: they all follow the format: [drive]:\@Images[n] (YYYY)\YYYY\YYYY [subject]\[sub-folder division]\ ..\file name.ext.  The sub-folders may be up to 3 or 4 levels deep beyond this in extreme cases, but most are about this length.  The files themselves are spread over @Images_01 to _19, drives D: to I: 

For the sake of completeness, as regards paths, my PM catalogues reside in J:\PM Catalogues.  The PM cache is at J:\PM Cache.  The J: drive is a large internal SSD.  Temp files are written to the L: drive.  All drives C: to L: are internal drives.  C:, J:, K: and L: are SSDs, the remainder normal hard disks.  PM6+ was installed to its default location on the C: drive.

Graham
« Last Edit: February 23, 2021, 02:27:11 AM by Graham1 »

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24768
    • View Profile
    • Camera Bits, Inc.
Re: Unreadable File Format
« Reply #8 on: February 23, 2021, 09:13:08 AM »
Graham,

For those images that show as "Unreadable File Format", if you select one of them and look at the status bar, what is the path that is shown?
If you open the folder containing that image via the Navigator and select that same image, what is the path that is shown in the status bar?

I have chosen one at random.  That happens to be G:\@Images_12 (2014)\2014\2014 South America\1.Peru\1.1 Lima\IMG_8095.CR2, as per Organizer status bar.    If I then go to Navigator and open the 1.1 Lima sub-folder in a new contact sheet, the entry on the status bar is exactly the same.  The difference is that I can see the thumbnail and read the metadata in the Navigator contact sheet.  Having looked at it in Navigator, I go back to Organizer: it is no longer in Browse ¦ Unreadable File Format, but it can be found (complete with thumbnail and readable metadata) in Browse ¦ Canon CR2 (CCR2).

Now that those images have been 'corrected', if you restart Photo Mechanic Plus and view those images via Browse/Filter/Search, do they remain corrected or do they revert to being an Unreadable File Format?

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Unreadable File Format
« Reply #9 on: February 23, 2021, 03:11:59 PM »
Quote
Now that those images have been 'corrected', if you restart Photo Mechanic Plus and view those images via Browse/Filter/Search, do they remain corrected or do they revert to being an Unreadable File Format?
I am finding it difficult to be entirely sure about this.  Last night, I found a folder with lots of Unreadable files.  I selected the folder in Navigator and opened a contact sheet.  Then I went back to Browse ¦ Unreadable.  The previously Unreadable files were now visible as thumbnails in Browse ¦ Unreadable.  The count of Unreadable went down. 

Having read your message this morning, I repeated the exercise. The images that had ceased to be Unreadable were a mixture of .CR2 and .DNG images.  The .DNG images are typically HDR images created in Lightroom.  I was surprised to see that the .CR2 files no longer seemed to be in Unreadable, but the .DNG files were.  Having looked at the folder in Navigator and then gone back to Browse, the same as the above happened: I could see thumbnails for the .DNG files in Browse ¦ Unreadable.  I closed and restarted PM6+.  The .DNG files were again showing in Unreadable, with blank thumbnails.  In other words, the revealed .DNG images were reverting to Unreadable and losing their newly found thumbnails.  There is, however, no problem generally with HDR .DNG files: the catalogue in Browse File Type counts 1549 .DNG files, a mixture of panoramas and HDR files created in Lightroom.  A file name search indicates that 722 of these are HDR files (I do not rename files: the file names end in -HDR, so it is easy to check).  After my search for files named -HDR, again the the thumbnails are visible in Browse ¦ Unreadable while PM6+ remains open, but do not persist and are absent again when PM6+ has been closed and reopened.

I tried with a another folder that had a number of Unrecognised .CR2 files.  Having been seen in a Navigator contact sheet, their thumbnails persisted after restarting PM6+ (i.e. they were then in the .CR2 File Type, not Unrecognised in Browser, after restarting).  The treatment of .CR2 files and .DNG files therefore appears to differ.

I mentioned in a previous post that many (but far from all) of the Unrecognised files seem to be those with a colour label.  Every single .DNG file in my portfolio has been colour labelled in Lightroom: I colour label the images which are to be used to create the panorama or HDR file for future reference, and the resulting .DNG file accordingly has the same colour label.  Most of the Unreadable .CR2 files in the folder I have just tested were either currently or previously colour labelled.  Coincidence or not? - I have no idea.

Graham


Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24768
    • View Profile
    • Camera Bits, Inc.
Re: Unreadable File Format
« Reply #10 on: February 23, 2021, 06:59:27 PM »
Thanks for the additional information, Graham.

We've added some logging and we'd like you to try it.  You'll need to turn CATALOG_DEBUG_LOGGING on.  Before you quit your existing Photo Mechanic Plus, go to the Help menu, choose "Edit Debug Settings..." and then click on the checkbox next to CATALOG_DEBUG_LOGGING to set it to 'checked'.  Then click the Close button and quit Photo Mechanic Plus.

Next download and install this build: https://www.camerabits.com/download/PMPlusSetupR5621_58f10327.msi

Start it up and do the same things that have made the images go from being unreadable to being readable and back again.  Then go to the Help menu and choose "Reveal Support Data..." and post the zipped log file here.  If it is too large, contact me privately for another way to share the file.

Thanks,

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Unreadable File Format
« Reply #11 on: February 24, 2021, 01:12:31 AM »
Thank you Kirk.

The log file is attached.  I have also included 5 sequentially numbered screen grabs (1. to 4. during the initial run, 5. after PM6+ had been restarted), tracking the view of file numbered _MG_3033-HDR.DNG.  The names should make the steps clear, and I have included the status bar in some of them so that you can see that the file selected remains unchanged.

Hope this helps.

Graham

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24768
    • View Profile
    • Camera Bits, Inc.
Re: Unreadable File Format
« Reply #12 on: February 24, 2021, 09:45:49 AM »
Thanks Graham.

Unfortunately there isn't any issue that we can see captured in the logs.  Also we're not seeing the counts of "Unreadable File Format" change as you described earlier.  But I believe you mentioned earlier that you were able to get the issue to happen again on a smaller catalog that you built.  Can you leave the logging on and use the same build (5621) that I provided you yesterday to create a small catalog that reproduces the problem again?

Thanks,

-Kirk

Offline Graham1

  • Full Member
  • ***
  • Posts: 148
    • View Profile
Re: Unreadable File Format
« Reply #13 on: February 24, 2021, 10:06:50 AM »
Thanks Graham.

Unfortunately there isn't any issue that we can see captured in the logs.  Also we're not seeing the counts of "Unreadable File Format" change as you described earlier.  But I believe you mentioned earlier that you were able to get the issue to happen again on a smaller catalog that you built.  Can you leave the logging on and use the same build (5621) that I provided you yesterday to create a small catalog that reproduces the problem again?

Thanks,

-Kirk

Thanks, Kirk.  Will do.  I think the file count probably decreased as the .CR2 files were recognised.  Having used the same folder as yesterday, that folder may well have had only .DNG files left in it as Unrecognised, which is why the file count did not go down in the same way this time.  I will also try it with the existing catalogue if I can find a folder with both .CR2 and .DNG images: there is bound to be one.  Hopefully the screen grabs at least showed how the .DNG image thumbnails appear then disappear later.

Graham

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24768
    • View Profile
    • Camera Bits, Inc.
Re: Unreadable File Format
« Reply #14 on: February 24, 2021, 10:17:13 AM »
Graham,

Hopefully the screen grabs at least showed how the .DNG image thumbnails appear then disappear later.

Yes, definitely illustrative.

Thanks,

-Kirk