Unable to Rename

[akibu]

I am using PM+ on a MacOS (Monterey).  I have one default catalog which contains multiple duplicate images, the result of bad file management on my part over the years.  I am attempting to isolate those duplicate images via renaming so that the renaming function can add letters (A,B,C, etc.) to the end of the filename, thus allowing me to search for those files and to delete them from the catalog and disk.  However, I am running into a problem with the renaming command.  The exact error message is: "Unable to rename, because many photos are being renamed to be the exact same name 20050508173948.JPG.  Please try using a sequence, or choose different renaming variables so that unique names can be generated for all photos when renaming."  I am attempting to rename using the following variables:  {year4}{month0}{day0}{hour24}{minute}{second}{frame4}.  It seems that I should not get this error given that the renaming is set to add letters for duplicates. My images (~150K) are spread across 2 attached ssd's.  My catalog is located on my local mac harddrive.  The folder containing the images I was attempting to rename has about 5,500 images and is labelled year 2005, however, it doesn't matter which folder I use to run the rename function on as I still get the same error message.  To rename, I select all images in that folder, and then Command M to rename.

Any assistance is appreciated.

Thanks,

Akibu

[Kirk Baker]

Akibu,

I am using PM+ on a MacOS (Monterey).  I have one default catalog which contains multiple duplicate images, the result of bad file management on my part over the years.  I am attempting to isolate those duplicate images via renaming so that the renaming function can add letters (A,B,C, etc.) to the end of the filename, thus allowing me to search for those files and to delete them from the catalog and disk.  However, I am running into a problem with the renaming command.  The exact error message is: "Unable to rename, because many photos are being renamed to be the exact same name 20050508173948.JPG.  Please try using a sequence, or choose different renaming variables so that unique names can be generated for all photos when renaming."  I am attempting to rename using the following variables:  {year4}{month0}{day0}{hour24}{minute}{second}{frame4}.  It seems that I should not get this error given that the renaming is set to add letters for duplicates. My images (~150K) are spread across 2 attached ssd's.  My catalog is located on my local mac harddrive.  The folder containing the images I was attempting to rename has about 5,500 images and is labelled year 2005, however, it doesn't matter which folder I use to run the rename function on as I still get the same error message.  To rename, I select all images in that folder, and then Command M to rename.

When you say "it doesn't matter which folder", is this an actual folder on your drive, or is it a Catalog based folder (from a search or browse) ?

The naming resolution (appending digits or letters) only occurs when filenames with the same name exist in the same folder.  If your images are spread out over a number of folders, that collision may not occur.  The Rename window runs through your names and checks to see if they're unique enough.  If more than eight filenames map to the same name (regardless of what folder they're in) then that message will appear.  It is telling you that you're likely to have problems (may run out of digits or letters to apply) and then the overall operation will fail and the renaming will be undone.

-Kirk

[akibu]

Thanks Kirk.

To answer your question, it is a catalog folder from a browse.  I only meant that if I were to run the rename over the year 2004 folder instead of 2005, I would still run across that error message. 

Note, my actual photos are already organized and renamed (thanks to PM+) by Year>Month and then named "Year4Month2Day2_sequence.type".  So a photo taken on April 24th 2004 would be contained in a folder 2004 subfolder April and the photo would be named something like 20040424_######.cr2.   The photos are spread out across two 2T SSD external drives.  One of the SSD drives is just about full so I have been having my ingests go to the other SSD drive with the same year/month naming structure and all for the same catalog.

Your analysis re more than eight filenames mapping to the same name may be correct as I attempted to rerun the renaming function on year 2005 photos and made a note of the file name it indicated in the error message (20050508173948.JPG) which presumably meant the photo was taken on May 8, 2005 at 5:39 pm and 48 seconds and there were indeed more than eight of those photos. 

Given the multiple duplicates, is there a suggested manner to get rid of the duplicates other than the one I attempted to employ?
On ingest, I had the tick box for PM+ to copy new photos only... why would it also ingest duplicates?  If it does ingest duplicates, shouldn't it append letters or numbers to the end as it does on rename?

Would re-ingesting (not my favorite solution, but I will do it) the photos help or would PM+ merely re-ingest the duplicates once again?

When PM+ is ingesting photos, how does it know that there is a duplicate?  Does it look at the metadata of the photo to be ingested and then compares it to the metadata of the photos already on the specific SSD to which the photo is intended to be ingested?, or does it look at the metadata of the photos in the catalog (irrespective of on which SSD the photo sits) and then says "hey, I already have this photo in the catalog so I will not re-ingest it to the 2nd SSD"?   The reason I ask this question is because alot of the photos which I ingested to my second SSD would have already been previously ingested to the first SSD (and be part of the catalog) but I didn't know which ones, but figured that PM+ would know that it already had that photo in the catalog so would skip it when ingesting the photos to the 2nd SSD.

Would you recommend that I use a 3rd party de-duplication program such as dupeGuru on the photos from both of the SSDs to remove the duplicates... if so, what would I then have to do in PM+ so that it recognizes that those photos are no longer there and should be removed from the catalog? 

Thanks in advance and sorry for the long post.

[Kirk Baker]

I'm still a bit confused.  Did you Ingest folders or disks?  If it was disks, were they camera cards?  If it was folders, is the source folder writable?

I expect Incremental Ingest to work on both folders or disks and it should report a warning in the Ingest Tasks window if it was unable to perform the Ingest incrementally.

These duplicates, are any of them in the same folder?  If so, what are their names?  I'm looking for a small sample list, not all of your duplicates.

-Kirk

[akibu]

See responses inline below.

I'm still a bit confused.  Did you Ingest folders or disks?  If it was disks, were they camera cards?  If it was folders, is the source folder writable?

--It varied.  Sometimes I ingested disks (camera cards or mounted drives shared over thunderbolt connection from a windows computer).  Or sometimes I ingested specific folders within mounted drives.  The source folders would have been writable.

I expect Incremental Ingest to work on both folders or disks and it should report a warning in the Ingest Tasks window if it was unable to perform the Ingest incrementally.

--I may have gotten "duplicates were found messages" once the ingests were complete, but I can't recall.  If I did, I would have probably interpreted it merely as a notification that it found duplicates and did not ingest them.

These duplicates, are any of them in the same folder?  If so, what are their names?  I'm looking for a small sample list, not all of your duplicates.

--Attached is a screenshot of duplicate photos.  The photos in the first row are from the first SSD /Volumes/Samsung_T5-2T/Photographs/2005/May/...
The bottom two rows are from the second SSD /Volumes/Samsung_T7-2T/Photographs/2005/May/...


-Kirk

[Kirk Baker]

See responses inline below.

I'm still a bit confused.  Did you Ingest folders or disks?  If it was disks, were they camera cards?  If it was folders, is the source folder writable?

--It varied.  Sometimes I ingested disks (camera cards or mounted drives shared over thunderbolt connection from a windows computer).  Or sometimes I ingested specific folders within mounted drives.  The source folders would have been writable.

I expect Incremental Ingest to work on both folders or disks and it should report a warning in the Ingest Tasks window if it was unable to perform the Ingest incrementally.

--I may have gotten "duplicates were found messages" once the ingests were complete, but I can't recall.  If I did, I would have probably interpreted it merely as a notification that it found duplicates and did not ingest them.

These duplicates, are any of them in the same folder?  If so, what are their names?  I'm looking for a small sample list, not all of your duplicates.

--Attached is a screenshot of duplicate photos.  The photos in the first row are from the first SSD /Volumes/Samsung_T5-2T/Photographs/2005/May/...
The bottom two rows are from the second SSD /Volumes/Samsung_T7-2T/Photographs/2005/May/...

OK, so the naming was {year4}{month0}{day0}_{seqn} ?  I'm guessing at what produced the name after the underscore.  If Ingest didn't think that it had previously copied those images (it tracks files by their source name, modification date/time, and file size), then if the naming doesn't produce the same name when evaluated, then there will be no collision and no appending of letters or digits.  Ingest does not analyze the image or any other metadata to make the determination that it has copied a file before.  Generally, original name/size/date works quite well.

Without witnessing how you did your ingests or any warnings it may or may not have provided, I can't say what happened.

If you want to use some utility app to find and remove duplicates, you can go ahead and use one.  When you're done, use Catalog Sync in "Full Mode" to update your catalog.  Have it generate collections for missing images and use the collection to select all of the images and remove them.

-Kirk

[akibu]

OK, so the naming was {year4}{month0}{day0}_{seqn} ?  I'm guessing at what produced the name after the underscore.  If Ingest didn't think that it had previously copied those images (it tracks files by their source name, modification date/time, and file size), then if the naming doesn't produce the same name when evaluated, then there will be no collision and no appending of letters or digits.  Ingest does not analyze the image or any other metadata to make the determination that it has copied a file before.  Generally, original name/size/date works quite well.

Thanks Kirk, this is helpful.  I will use a third party utility to dedupe and follow the instructions you outline. 

On the above quoted language, particularly the red highlighted part, does PM+ compare the "to be ingested files" to those in the catalog (or other database)? or does it compare the "to be ingested files" to those which are existing on an SSD to which the "to be ingested files" are to be copied.  In other words, if I add a blank third SSD to store photos and decided to re-ingest photos from a camera card containing photos which are already present in the catalog, would PM+ add those photos to the third SSD?  If the comparison is made to the third blank SSD, it should re-ingest those photos as it has determined that there are no existing photos which are identical to the "to be ingested photos".  However, if the comparison is to the catalog (or other database), then the ingest should not take place as obviously PM+ sees from the catalog that it already has the "to be ingested photos".  I assume that the latter is what happens, but wanted to make sure.

[Kirk Baker]

OK, so the naming was {year4}{month0}{day0}_{seqn} ?  I'm guessing at what produced the name after the underscore.  If Ingest didn't think that it had previously copied those images (it tracks files by their source name, modification date/time, and file size), then if the naming doesn't produce the same name when evaluated, then there will be no collision and no appending of letters or digits.  Ingest does not analyze the image or any other metadata to make the determination that it has copied a file before.  Generally, original name/size/date works quite well.

Thanks Kirk, this is helpful.  I will use a third party utility to dedupe and follow the instructions you outline. 

On the above quoted language, particularly the red highlighted part, does PM+ compare the "to be ingested files" to those in the catalog (or other database)?

Another database (it's a text file with tab-separated values).

If I add a blank third SSD to store photos and decided to re-ingest photos from a camera card containing photos which are already present in the catalog, would PM+ add those photos to the third SSD?

Ingest is independent of the catalog system.  Ingest also exists in Photo Mechanic 6 and also offers the Incremental Ingest feature.  The ingest data is consulted and if a file exists in the data and it hasn't changed its modification time or size then it isn't copied, no matter the target destination.

If the comparison is to the catalog (or other database), then the ingest should not take place as obviously PM+ sees from the catalog that it already has the "to be ingested photos".  I assume that the latter is what happens, but wanted to make sure.

It should not copy files that have already been copied if Incremental Ingest is on.  That said, if PM or PM Plus cannot write to its data files, or if it cannot write a .pmingest.dat file to the disk, or if something deletes the .pmingest.dat file from the source disk (or the disk is formatted), then files previously copied will continue to be copied.

-Kirk

[akibu]

Super!!!

Thanks for the clarity.