Author Topic: Merging several folders into one and eliminate duplicate photos  (Read 10103 times)

Offline flyboy

  • Newcomer
  • *
  • Posts: 7
    • View Profile
Merging several folders into one and eliminate duplicate photos
« on: January 12, 2010, 11:44:47 AM »
I am running Photo Mechanic version 4.6.2.1 and I run Win7 64 Ultimate on my Intel-based PC.

Would like to merge several folders into one folder with new name for all. In doing so need to eliminate any duplicates. New name would be like "Capture Date_Camera type_Seq or fram#".

New to PM and would appreciate any and all suggestions.

Also how do PM determine if photo is duplicate? What is it looking for? File name ? Capture time?

Thank you. Jim


Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24763
    • View Profile
    • Camera Bits, Inc.
Re: Merging several folders into one and eliminate duplicate photos
« Reply #1 on: January 12, 2010, 12:24:07 PM »
Jim,

I am running Photo Mechanic version 4.6.2.1 and I run Win7 64 Ultimate on my Intel-based PC.

Would like to merge several folders into one folder with new name for all. In doing so need to eliminate any duplicates. New name would be like "Capture Date_Camera type_Seq or fram#".

New to PM and would appreciate any and all suggestions.

Also how do PM determine if photo is duplicate? What is it looking for? File name ? Capture time?

Photo Mechanic does not have any facility for determining if files are duplicates.

-Kirk

Offline Hayo Baan

  • Uber Member
  • ******
  • Posts: 2552
  • Professional Photographer & Software Developer
    • View Profile
    • Hayo Baan - Photography
Re: Merging several folders into one and eliminate duplicate photos
« Reply #2 on: January 12, 2010, 12:42:38 PM »
Hmm, this is not a straightforward task at all.

PhotoMechanic does not have any method of determining whether or not files are identical. It does check to see if two files would end up with the same name (e.g. after a rename or when ingesting), and will solve this by appending a letter or digit to the filename. This is not what you are looking for though.

What I can suggest you do, however, is the following:

First open all the folders you want to merge into a single contact sheet. With an empty contact sheet, add all folders to it by e.g., right clicking the folder to add and then choosing "open in current contact sheet" (choose the "and subfolders" variant if you want to add whole trees of folders).

Once you've added all folders you want merged, optionally choose a sorting method (with lots of images, some sorting methods can take a LONG time). The sorting will determine the order in which they get renamed (and thus the order of any sequence number you add to the filename).

Now select all image files (cmd/ctrl-a) and choose file|copy/move photos. In the resulting dialogue, set the correct options (e.g., select "Move photos (delete originals)" if you want to actually move the files to a new location, or leave it unchecked if you want to copy first, leaving the originals intact. Certainly check "Rename copied/moved photos as" (more on this later). Now select the destination either by selecting "Copy/Move to this location" and browsing to the correct folder, or leave it at "Always pick destination" to defer this question to just prior the actual copy/move. In either case make sure you select a folder that is not yet part of the folders you actually try to merge (create a new one and call it "Merged" or somthing, you can always rename/move it later).

About the renaming. There are many possibilities here, but something like "{datesort}_{model}_{seqn}", will be a simple implementation of what you want to achieve. You can set the format of the seqn variable with the button that will popup after adding {seqn} to the rename string.
If and only if all your files are still in their original naming (e.g. DSC_####), you can also use the original frame number in your rename string. In this case use "{datesort}_{model}_{fnum}".

Couple of notes on the renaming used here:
{seqn} will continue to count up, and will not reset when e.g. the date or the camera model changes. Numbering will be following the order of the images in the contact sheet.
{model} generates the camera model with its full name, including the brand name. If you shoot Nikon, you can easily chop off this name by using {model:6} instead. With Canon, this works too, but in this case you might also want to chop of e.g. EOS, so use {model:10} for instance. In a mixed case, this is more tricky (advanced: use code replacements in this case).

There's a lot more possible with this, but above should get you started. Come back to us if you need more info.

Cheers,
    Hayo
Hayo Baan - Photography
Web: www.hayobaan.nl

Offline Hayo Baan

  • Uber Member
  • ******
  • Posts: 2552
  • Professional Photographer & Software Developer
    • View Profile
    • Hayo Baan - Photography
Re: Merging several folders into one and eliminate duplicate photos
« Reply #3 on: January 12, 2010, 12:45:00 PM »
Oh yeah, on the duplicate determination: you'll have to do this by hand. However, with the renaming you've suggested, this already becomes a bit easier as you can probably go over your files quickly and see neighbouring ones and compare them.

Good luck,
    Hayo
Hayo Baan - Photography
Web: www.hayobaan.nl

Offline jwvaughn

  • Full Member
  • ***
  • Posts: 143
  • Retired programmer
    • View Profile
Re: Merging several folders into one and eliminate duplicate photos
« Reply #4 on: January 12, 2010, 03:21:34 PM »
Since you are on Windows, here is a possible solution: http://www.clonecleaner.com/HCCdownload.html aka HCC Lite.  This one is the free version and I have had very good luck with it finding duplicates.  It can do so by several methods including a byte-to-byte comparison.

It can take awhile if there are lot of files and/or they are big but once it's done, you can go through and pick and chose what to keep.

Jerry

Offline nmf

  • Member
  • **
  • Posts: 71
    • View Profile
Re: Merging several folders into one and eliminate duplicate photos
« Reply #5 on: January 13, 2010, 03:25:28 AM »
To find duplicates go to Preferences => Files and ensure "Renaming Resolution" is set to "Append letter when needed (A, B, C)". Then when you have copied all the files across to the same folder using PM you should be able to select the duplicates by doing select all and then a "Find" for A. B. C. (yes capital letter then full stop / point ) etc

This is dependent on the duplicate files in the folders being the same and you not having capitals in your filename, though even in such a case only those images with a capital beside the .jpg tag will be selected. I would suggest you colour code the results of each search and then scan through the sheet to check the selected images agains the neighbouring images before deleting to ensure they are dupes.

If the filenames are different then hopefully copying them all to a single new folder and setting PM to sort by Capture Time will hopefully put all the duplicates together. This is a bit more time consuming than the above method as it will require you to scan through them by eye for the duplicates and it may take a while for PM to sort the images by Capture Time depending on how many images you are working with and your computer.

There may be a reason this will affect the renaming but I can't see one as I would have thought the copy will contain all the same info used to then rename the final set using the parameters Hayo has suggested.

Whichever method you choose I would definitely do a test and then copy the files to a new folder so keeping your original sets intact until the task has been completed.

Good luck.


Offline flyboy

  • Newcomer
  • *
  • Posts: 7
    • View Profile
Re: Merging several folders into one and eliminate duplicate photos
« Reply #6 on: January 13, 2010, 12:49:30 PM »
Thanks to ALL responses. It is appreciated.

Am I thinking correctly that when PM checks for duplicates, the S/W is comparing one file name to another file name to see if they match? Is this correct in that the S/W is not looking at other elements of file?

Thus rename and moving to other folders can lead to a mess. Example: Rename files in folder A, also folder B, then merge to one big folder with another name format, PM would not detect even if using Capture Date, or frame number etc.?

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24763
    • View Profile
    • Camera Bits, Inc.
Re: Merging several folders into one and eliminate duplicate photos
« Reply #7 on: January 13, 2010, 01:34:11 PM »
Thanks to ALL responses. It is appreciated.

Am I thinking correctly that when PM checks for duplicates, the S/W is comparing one file name to another file name to see if they match? Is this correct in that the S/W is not looking at other elements of file?

Thus rename and moving to other folders can lead to a mess. Example: Rename files in folder A, also folder B, then merge to one big folder with another name format, PM would not detect even if using Capture Date, or frame number etc.?


Renaming, copy, or move only care about collisions in file naming.  Two files with the same name cannot exist in the same directory so when this happens PM renames the file being operated on to avoid the file name collision.  It does not care if other information about the files (image data, capture date, etc.) differ or are the same.  Only the file name matters.

-Kirk

Offline Hayo Baan

  • Uber Member
  • ******
  • Posts: 2552
  • Professional Photographer & Software Developer
    • View Profile
    • Hayo Baan - Photography
Re: Merging several folders into one and eliminate duplicate photos
« Reply #8 on: January 13, 2010, 02:41:55 PM »
Thanks to ALL responses. It is appreciated.

Am I thinking correctly that when PM checks for duplicates, the S/W is comparing one file name to another file name to see if they match? Is this correct in that the S/W is not looking at other elements of file?

Thus rename and moving to other folders can lead to a mess. Example: Rename files in folder A, also folder B, then merge to one big folder with another name format, PM would not detect even if using Capture Date, or frame number etc.?

You understand correctly. That said, you can have PM "sort of" look at other attributes of a file. Simply use them as the key for your file renaming; any duplicates will then get the suffix added and these can easily be filtered-out by sorting the contact sheet on those additional characters. Pretty complex stuff, but certainly doable. Let me try to explain how to do this a bit...

Step 1: renaming the files so the duplicates are easily sorted (please also refer to my first reply with instructions)
Your proposed renaming scheme with the original frame number would already be a great way to determine duplicates, but, as I already mentioned, would only work if all your images still were named according to the original camera naming (e.g. DSC_####), otherwise PM has no way of determining the frame number as-shot.
However, using the rename string "{datesort}_{timesortlong}_{model}_" you will give you a list of files where files shot at exactly the same moment (matching up to the second, or millisecond for those camera's supporting it), and with the same camera model, get the suffix added. Unless you've shot with two camera's at exactly the same moment (if this is the case, you could try adding the camera serial number or something to the string), these will be your duplicates. These we will filter in the second step.

Step 2: filtering the duplicates
Now for the cunning part, finding those files that are duplicates. Here we abuse PM's custom sorting ability.
Click the sort by button/dropdown and select edit custom. Choose one of the custom sorts and change it to read {filenamebase:-1}. This will sort the files on the last character of the filename. This last character is either an "_" for a non duplicate file, or a suffix character (e.g. A) for the duplicates. So this sorting will place all non duplicates and duplicates nicely in a block together, easing the ability to identify them.

Note: You can of course add all sorts of additional fields to the rename string in step 1 if you like. Also note that different file types (e.g. .tiff, .jpeg) will not be seen as duplicates of the same file!
After using this method, you can of course rename the set again, to set their names to something you really want.

Hope this helps and makes sense to you  ;D

Cheers,
    Hayo
Hayo Baan - Photography
Web: www.hayobaan.nl

Offline flyboy

  • Newcomer
  • *
  • Posts: 7
    • View Profile
Re: Merging several folders into one and eliminate duplicate photos
« Reply #9 on: January 13, 2010, 07:31:55 PM »
You folks are fantastic. Thank you's to Kirk,Hayo and all others. I think I got it now and will fix Thur. BTW PM is blazingly fast However UI is slow to get a grip on. Thanks again. Jim

Offline Kirk Baker

  • Senior Software Engineer
  • Camera Bits Staff
  • Superhero Member
  • *****
  • Posts: 24763
    • View Profile
    • Camera Bits, Inc.
Re: Merging several folders into one and eliminate duplicate photos
« Reply #10 on: January 14, 2010, 12:25:12 AM »
Jim,

You folks are fantastic. Thank you's to Kirk,Hayo and all others. I think I got it now and will fix Thur. BTW PM is blazingly fast However UI is slow to get a grip on. Thanks again. Jim

Suggestions for improvement are welcome.

-Kirk