DNG file change after ingest

[jarle]

Having ingested DNG files (from a folder on an external SSD hard drive, if that matters), the copied files are a few bytes larger than the originals. In a random example the ingested (copied) file is 23 546 917 vs 23 539 133 bytes.

In most cases it shouldn't matter, but I discovered the issue when using a duplicate file finder tool. The "identical" files cannot be found, which makes sense since they are technically different. No metadata is added during ingest.

Is this a bug or a feature? Is there anything I can do to fix this? I need to find and delete a large number of duplicate files.

PM 6.0, build 7102 (796dd4a) running on Sonoma 14.5 (23F79).

Jarle

[Kirk Baker]

Jarle,

Having ingested DNG files (from a folder on an external SSD hard drive, if that matters), the copied files are a few bytes larger than the originals. In a random example the ingested (copied) file is 23 546 917 vs 23 539 133 bytes.

In most cases it shouldn't matter, but I discovered the issue when using a duplicate file finder tool. The "identical" files cannot be found, which makes sense since they are technically different. No metadata is added during ingest.

Is this a bug or a feature? Is there anything I can do to fix this? I need to find and delete a large number of duplicate files.

If you want to share a file (same one from card and ingested), I'd be happy to investigate.

-Kirk

[jarle]

If you want to share a file (same one from card and ingested), I'd be happy to investigate.

Thanks, Kirk. Sure, where do I send them?

UPDATE: After opening both files in Photoshop and exporting the File Info > Raw Data to a text file, I asked ChatGPT to summarize the differences. It turns out:

The primary differences between the two files are as follows:

Modification and Metadata Dates:

Original: <xmp:ModifyDate>2024-08-08T21:16:59+02:00</xmp:ModifyDate>
Ingested: <xmp:ModifyDate>2024-08-08T21:15:44+02:00</xmp:ModifyDate>
Original: <xmp:MetadataDate>2024-08-08T21:16:59+02:00</xmp:MetadataDate>
Ingested: <xmp:MetadataDate>2024-08-08T21:15:44+02:00</xmp:MetadataDate>

Document and Instance IDs:

Original:
<xmpMM:DocumentID>xmp.did:7f97098c-1a94-46fb-953c-13eff15f96f8</xmpMM:DocumentID>
<xmpMM:InstanceID>xmp.iid:7f97098c-1a94-46fb-953c-13eff15f96f8</xmpMM:InstanceID>
Ingested:
<xmpMM:DocumentID>xmp.did:ee521020-89bd-44d9-b88f-d0ee0e1b90fa</xmpMM:DocumentID>
<xmpMM:InstanceID>xmp.iid:ee521020-89bd-44d9-b88f-d0ee0e1b90fa</xmpMM:InstanceID>
XMP Rights and Photo Mechanic Metadata (Ingested file):

Ingested file contains additional metadata:

<xmpRights:Marked>True</xmpRights:Marked>
<photomechanic:ColorClass>0</photomechanic:ColorClass>
<photomechanic:Tagged>False</photomechanic:Tagged>
<photomechanic:Prefs>0:0:0:000125</photomechanic:Prefs>
<photomechanic:PMVersion>PM6</photomechanic:PMVersion>
Image Number (Ingested file):

Ingested file has an additional tag:

<aux:ImageNumber>125</aux:ImageNumber>
Legacy IPTC Digest (Ingested file):

Ingested file has:

<photoshop:LegacyIPTCDigest>CCC1624B1E5517D1211BDF04DFD0AA2E</photoshop:LegacyIPTCDigest>

These differences explain why a duplicate file finder tool might not recognize these files as identical, even though they are essentially the same. The metadata changes and additional tags result in different file signatures. 

Jarle

[FVlcek]

Can't say about the other tags or speak for CameraBits at all, but

<photomechanic:Tagged>False</photomechanic:Tagged>

is much of a feature, IMHO.

Any in‑camera 'tagged' photo historically relied on the camera actually tagging the file attributes in the FAT filesystem file allocation table's as Read Only. That's what the 'protect' button on any digital camera since the early 2000s does. Worked well enough on DOS FAT and early Windows (where all the file attributes that existed were either Executable, Read Only or Hidden, plus a date, basically). Fun fact – you could even set photos to Hidden on earlier Canon DSLRs via the Play menu, which would leave them on the card, just invisible to any casual view of the card in Windows or such (obviously any computer forensics would find it immediately, but the feature literally saved my arse once when being interrogated by one quite computer illiterate MENA dictatorship regime police member about my photos of some protests there).

Modern file systems have moved on, though. As I understand it, the old DOS (remember DOS? Basically before Win 3.1) FAT Read Only tag might not even be supported in any of the modern FS you are ingesting to.

But you still want to see inside PM which files were 'tagged' or not. And be able to edit and save those files. As I understand it, PM just reads the "Read Only" filesystem flag from the FAT32 or ExFAT filesystem of your card and saves it inside the file as a filesystem‑agnostic XMP tag.

Which is IMHO pretty much preferred. Nobody uses FAT32 or ExFAT anymore, outside of camera cards. So there needs to be a place to save your tagged picks outside of some obscure filesystem attributes that aren't supported on modern systems anymore or might differ from OS platform to another, and a place where they would actually be persistent like XMP. Which FAT attributes totally aren't.

My guess is that you had copied the files from the card to an ExFAT or FAT32 (or even NTFS) external drive through other means, which had might heave preserved the Read Only attribute of the tagged ones, and the PM Ingest just tried to make sense of which were actually tagged or not in‑camera, adding the XMP field to those who were (and those who weren't).

[jarle]

This is obviously a feature, but it also makes finding and cleaning out duplicate files more difficult, if not impossible. Probably not a big issue for most users, but in my particular scenario it renders the ingest feature almost unusable.

I need to organize some 100.000 files into a year/month folder structure. Using PM to ingest from folder and dynamically creating {year4}-{monthname3} folders is everything I need, but I also need to verify that all files are correctly copied and then delete the unorganized originals/duplicates. A simple "Move" option in the ingest dialog instead of just copying the files would work for me.

[Mick O (Camera Bits)]

Hi there jarle,

Would it be possible to browse the external SSD without ingesting the files, selecting them while they are on the external SSD and then using Photo Mechanic's "Copy/Move Photos" feature to get them where you needed?  File > Copy/Move Photos...  is where it is located (Or command-y on macOS)     You'd have the same renaming/foldering options with the ability to automatically delete the originals (making it a "move")

Just a thought. I hope something like this might help.

Editing to add: You can also SAVE the selection of files (Edit > Save selection) before you do the Move/Copy so that when it comes time to delete them after some sort of double-checking. you could reselect the same files for deletion.

Mick

[jarle]

Would it be possible to browse the external SSD without ingesting the files, selecting them while they are on the external SSD and then using Photo Mechanic's "Copy/Move Photos" feature to get them where you needed?

Thanks for the suggestion, Mick. Unfortunately, considering the number of files I'm dealing with, I don't think it's feasible to do this manually. I've found a few third-party file copy/organizer tools that looked promising, but there always seems to be a little something missing. I might end up writing a Python script or something similar instead.