Ingesting RAW/NEF images as DNG

[judophotos]

Kirk,

I'm waiting for my delivery of a Nikon D5 fitted with the double XQD card which I already have on my D4.

Ingesting XQD RAW images is blindingly fast and was wondering if it would be possible to somehow automatically ingest the RAW images as DNG files?

If that could be built into PM it would make life a lot easier and remove those horrible little sidecar files.

Is it feasible?

Best regards,
David

[Kirk Baker]

David,

I'm waiting for my delivery of a Nikon D5 fitted with the double XQD card which I already have on my D4.

Ingesting XQD RAW images is blindingly fast and was wondering if it would be possible to somehow automatically ingest the RAW images as DNG files?

If that could be built into PM it would make life a lot easier and remove those horrible little sidecar files.

Is it feasible?

It can be done, but conversion to DNG on ingest would add significant time to the ingest.  If you're always keeping all of your images then it would be time well spent, but if you tend to cull a lot of images then conversion after the cull would save time.

-Kirk

[judophotos]

Thanks for the quick reply Kirk.

I have been experimenting with converting NEF files to DNG using PM which is why I am a bit late in replying.

1) I do tend to retain most of my images culling just the near duplicates and out of focus images, reducing 2k+ from a day's event down to around 1200-1500 for archiving.  I also backup to a separate portable drive while ingesting so always have reserve NEF images.

The events I attend are invariably 2 or 4 days and as long as 7 days for the Olympics with each day around 10 to 12 hours per day.

Because of the long days, I usually ingest into my laptop every couple of hours leaving the process to run in the background while starting again with a fresh card.  If there is a break I will do a little selecting and then, after the medal ceremonies, send as many pictures as I can to the agency who require all submissions within 12 hours of the day's event finishing.

Between events, I have other small jobs that might be up to 100 images but they are not a problem and often require very little tinkering with.

If there was a PM option to ingest and convert to DNG at the same time with the much faster XQD card system then I would likely find that very beneficial for the big events even if the process was longer than normal. Obviously, this workflow may be peculiar to me but I would have thought that a DNG ingesting conversion option in PM would benefit a lot of other photographers too.

2) This may not be the right place for this question. In the past, when required, I've used the Adobe converter and set the preferences and it has worked very well compressing a DNG file as needed or not at all depending how the preferences were configured.

However, when using the converter via PM yesterday it resorts to the uncompressed DNG file increasing the file size to 30-40 MBs rather than matching the NEF size or less.  Am I doing something wrong? 

I am using the latest version of the converter with Windows 10.

Best regards,
David

[Kirk Baker]

David,

2) This may not be the right place for this question. In the past, when required, I've used the Adobe converter and set the preferences and it has worked very well compressing a DNG file as needed or not at all depending how the preferences were configured.

However, when using the converter via PM yesterday it resorts to the uncompressed DNG file increasing the file size to 30-40 MBs rather than matching the NEF size or less.  Am I doing something wrong? 

I am using the latest version of the converter with Windows 10.

What settings are you using in PM?  When instructed to do so, PM passes the 'compression' flag to the DNG Converter application.

-Kirk

[judophotos]

Four thumbnails are attached as a string of images and they are as follows:

1) The first is a TIFF from the original NEF at 17.3mb.
2) The second is the original NEF at 19.2mb.
3) The third is a DNG of the NEF converted with the Adobe converter at 6.8mb.
4) The fourth is the DNG conversion at 55.8mb via PM with the same settings as used in 3 above.

The 2 other snippets show the PM settings and the Adobe Converter settings as also used by PM.

I must add that some reason both the DNG files are at 72ppi and I don't know how to alter that to 300ppi.

Hopefully, I have made that easy to understand.

Best rgds,
David

[Hayo Baan]

This is strange... Looks almost like Adobe broke the command line interface to the DNG converter. Even without Lossy compression the files shouldn't be this big (I never use that and still my DNGs are usually smaller than the nef).

One thing I do notice in your settings is that you have the dng converter limit to 20mp, that is an option that is not available in PM. But still shouldn't explain the difference you are seeing...

Can you try an earlier version of the dng converter to see if the problem is present there too?

Note: I always perform my conversions from the Adobe DNG converter itself as I feel more confident about the options. Some options are simply not available in PM. Note too though that that really isn't CameraBits' fault; the command line interface to the dng converter simply doesn't offer the capabilities (I looked for those myself in the documentation as I wanted to build my own automation tool). It would be nice if Adobe updated that interface (and its documentation)

[judophotos]

Thanks Hayo.

Yes, it is very strange why the PM DNG conversion should be so large.

Also, I can't understand why the DNG files are at 72 ppi and 300 ppi. Are you able to convert at 300 ppi?

Or am I missing something?

David

[Hayo Baan]

Doh, I should have spotted this right away. You tell PM to have the DNG converter produce Linear DNG files. That is causing the huge increase. Turn that option off and you should see much smaller files.
Note: unless you have a specific need for this, you should never convert to Linear DNG! Linear DNGs aren't really raw files, but contain already demosaiced image data, thereby losing the ability to benefit from any future improvements in the demosaicing algorithms.

Regarding the resolution, I did a little digging and from what I found, the DNG converter does not set this at all. It simply leaves the resolution at what's already in the file (which is good as you are just converting the data to another format, not change the metadata).

What baffled me though was that in the file, the resolution seemed to be set at 300, but PM showed it as 72. Weirder still when I then instructed PM to set the resolution to 300, it showed it as 300, but the underlying metadata (as inspected with exiftool) hadn't changed at all…
But it gets more funny: I then quit and reopened PM, et voila: the resolution once again back at 72ppi (at least this does explain why exiftool didn't find any differences )

@Kirk, this seems like a bug to me. My guess is that PM does not correctly interpret the DNG resolution data (which is in the makernotes).

[judophotos]

Thanks, Detective Hayo!

I haven't got a clue what Linear DNGs are. Now I have a better idea and, yes, the images are appearing at the right MB size.

In Photoshop the Exif data is shown as     

<tiff:XResolution>300/1</tiff:XResolution>
<tiff:YResolution>300/1</tiff:YResolution>

Where as in PM it is 72 ppi.

Now I will use PM to convert the images as needed but would love to see an auto-convert option if not too difficult to setup.

That way I would keep my backed-up NEFs and use the DNG converted images for publishing via a JPEG after any post production work, avoiding those nasty little sidecar files.

Also, I would then have a similar workflow to when I used Nikon Capture NX2.

Thanks again.
David