Converting 400,000 photos as quickly as possible

[ronmart]

I just downloaded the trial today and my goal is to do the following:

1. From a source set of 400,000 images on a NAS, I want to convert each one into a lower resolution image

2. Each converted image would be a maximum of 1920 px on the long edge

3. Source set includes raw files from at least 100 different camera models

4. I use Lightroom today so I'd like the output JPEG to be the LR/PS equivalent of 80 for the quality

5. The files are in different folders and filenames might be duplicated. Ideally, I'd like the output to preserve the file structure of the input but obviously using a new root directory

6. I want ZERO changes to the output images - it's simply a read, downsize the longest edge to 1920px (no upsize), maintain existing aspect ratio no matter what that is, and copy to the new location using the same file name and directory structure

7. If there's anyway it can work with Lightroom's XMP sidecar files to process the images according to my Lightroom changes that would be great, but its not the end of the world if that's not possible.

I don't care how long it takes (multiple days isn't an issue), but I just ideally one it done in one run without any headaches. Lightroom doesn't see to be up to the task for a job like this as the export dialog takes ages to show up and then the job never starts afterwards despite having a beefy i7 machine with 32 GB of RAM (and memory utilization at < 10%).

Thanks,
Ron

[Kirk Baker]

Ron,

I just downloaded the trial today and my goal is to do the following:

1. From a source set of 400,000 images on a NAS, I want to convert each one into a lower resolution image

2. Each converted image would be a maximum of 1920 px on the long edge

3. Source set includes raw files from at least 100 different camera models

4. I use Lightroom today so I'd like the output JPEG to be the LR/PS equivalent of 80 for the quality

5. The files are in different folders and filenames might be duplicated. Ideally, I'd like the output to preserve the file structure of the input but obviously using a new root directory

6. I want ZERO changes to the output images - it's simply a read, downsize the longest edge to 1920px (no upsize), maintain existing aspect ratio no matter what that is, and copy to the new location using the same file name and directory structure

7. If there's anyway it can work with Lightroom's XMP sidecar files to process the images according to my Lightroom changes that would be great, but its not the end of the world if that's not possible.

I don't care how long it takes (multiple days isn't an issue), but I just ideally one it done in one run without any headaches. Lightroom doesn't see to be up to the task for a job like this as the export dialog takes ages to show up and then the job never starts afterwards despite having a beefy i7 machine with 32 GB of RAM (and memory utilization at < 10%).

Photo Mechanic isn't a RAW converter.  No other software besides Adobe's software can render RAW files and use Adobe's proprietary settings to produce output.

Photo Mechanic is not able to meet all of your requirements.  Adobe does not license their rendering technology to other developers.  The closest thing would be to have Photo Mechanic use Adobe DNG Converter to render your files (it would take ages), but then the previews that PM can extract may be lower than 80 for the quality (the DNG Converter puts fairly low quality previews in the DNG files that it produces.)  And then you'd run into issues with opening 400,000 images in Photo Mechanic at once.  Photo Mechanic is not designed to have that many images in its Contact Sheet windows.

If some of Adobe's tools are scriptable (meaning you can direct them to open a file, render the image, scale the image, and then output in JPEG) then you could certainly go that way and eventually it would get through them all (barring any crashes that stop your script.)

I don't have a full solution for you.

-Kirk
 

-Kirk

[Bob]

Kirk, above you mention that PM isn't designed to open 400,00 images in a browser window, but if working with a PM Plus catalog of a million images, as you say you test, using "" to search for all would result in a browser window of a million images. So do you advise against that "search all" in that case? (sorry to highjack the thread)

[Kirk Baker]

Bob,

Kirk, above you mention that PM isn't designed to open 400,00 images in a browser window, but if working with a PM Plus catalog of a million images, as you say you test, using "" to search for all would result in a browser window of a million images. So do you advise against that "search all" in that case? (sorry to highjack the thread)

Photo Mechanic Plus can handle a search that produces a million or more images.  Opening a folder with a million images in it would bring it to its knees.

-Kirk

[Mick O (Camera Bits)]

One way that Photo Mechanic Plus could prove useful in this case is that if you created a catalog that had all 400,000 images in various folders, the Photo Mechanic export-> Text Exporter could be used to create a text file list of all those images with their full path.

This text list could then be used in any command-line image processor. ImageMagick (https://imagemagick.org/) is one free tool that I know of. 

If it were me, that is probably where I would start investigating.