Embedded JPEG quality, "extracted" vs "save as" for NX2 NEF files

[rjp]

Dear PM staff.

I have recently downloaded a trial version of PM and I am absolutely blown away with the performance of this product. I will certainly be purchasing it.

My questions pertains to the JPEG images embedded in Nikon NEF files and their suitability for final products.

As far as I can tell from extensive pixel peeping there is almost no difference whatsoever between the JPEG image "extracted" in PM compared to the JPEG image that results when "saved as" JPEG from NX2 directly. Since this seems to be contrary to what some others are saying on this forum, I was wondering if I could get a confirmation on this from you. To be clear, I am talking about NEFS that have been edited in NX and resaved, so the embedded JPEG has been created by NX rather than the camera.

This is perhaps the most important feature PM offers to my workflow because it apparently removes the need to batch process all my NEFs to JPEG after I'm done editing them.

Here are some details.

I am running mac OS X Leopard.
PM version is 4.6
Files are Nikon NEF produced by D40.
I have disabled RAW conversion by Apple so embedded JPEG is used for "save as" and previews (Nice!)

My understanding is that your "extract" feature simply extracts the existing JPEG from within the NEF exactly as it was produced by whatever application put it there (i.e., NX or the camera).

When I do this with D40 NEFS I am getting JPEGs around the 3-4 MB size range, which is also what I get from NX "save as JPEG".  However, if I instead use the PM "save as JPEG" feature then I get a quality slider from 1 to 100, and most times the size produced at 100 is slightly larger than the extracted JPEG itself.

(1) How can this be? Where is the extra data coming from?

(2) How does the "save as" feature differ from the "extract" feature in terms of quality of the result? i.e., if the resulting file sizes are the same are they essentially identical?

(3) If I use "save as" to reduce the JPEG file size does PM have do decode the original JPEG, then re-encode it at a lower quality or does if just discard less significant image bits from the original embedded JPEG? The reason I ask is I realize that a JPEG of a JPEG can sometimes lead to  poor quality.

Thanks for any info you can provide.

Rich

[Kirk Baker]

Rich,

I have recently downloaded a trial version of PM and I am absolutely blown away with the performance of this product. I will certainly be purchasing it.

My questions pertains to the JPEG images embedded in Nikon NEF files and their suitability for final products.

As far as I can tell from extensive pixel peeping there is almost no difference whatsoever between the JPEG image "extracted" in PM compared to the JPEG image that results when "saved as" JPEG from NX2 directly. Since this seems to be contrary to what some others are saying on this forum, I was wondering if I could get a confirmation on this from you. To be clear, I am talking about NEFS that have been edited in NX and resaved, so the embedded JPEG has been created by NX rather than the camera.

That is correct.

This is perhaps the most important feature PM offers to my workflow because it apparently removes the need to batch process all my NEFs to JPEG after I'm done editing them.

Here are some details.

I am running mac OS X Leopard.
PM version is 4.6
Files are Nikon NEF produced by D40.
I have disabled RAW conversion by Apple so embedded JPEG is used for "save as" and previews (Nice!)

My understanding is that your "extract" feature simply extracts the existing JPEG from within the NEF exactly as it was produced by whatever application put it there (i.e., NX or the camera).

When I do this with D40 NEFS I am getting JPEGs around the 3-4 MB size range, which is also what I get from NX "save as JPEG".  However, if I instead use the PM "save as JPEG" feature then I get a quality slider from 1 to 100, and most times the size produced at 100 is slightly larger than the extracted JPEG itself.

(1) How can this be? Where is the extra data coming from?

When you use Save As, the image is loaded and then saved.  This causes the data to be recompressed at the quality setting you've chosen.

(2) How does the "save as" feature differ from the "extract" feature in terms of quality of the result? i.e., if the resulting file sizes are the same are they essentially identical?

At quality 100 you would be hard pressed to notice any difference.  If you want the most pristine quality you'll have to use the Extract function instead of Save As, but of course you'll lose a lot of potential functionality that way (Scaling, Additional cropping, IPTC Stationery Pad, Sharpening, Convert to sRGB, Renaming, etc.)

(3) If I use "save as" to reduce the JPEG file size does PM have do decode the original JPEG, then re-encode it at a lower quality or does if just discard less significant image bits from the original embedded JPEG? The reason I ask is I realize that a JPEG of a JPEG can sometimes lead to  poor quality.

It will definitely re-JPEG the decoded JPEG data.  Any quality setting near 100 with no Chrominance Subsampling is going to be what we call near-lossless.

HTH,

-Kirk

[rjp]

Thanks Kirk,

It is good to know that NX embeds such high quality JPEGS in its NEFS and that PM can use these.

I was not aware that the file size could be increased by forming the JPEG of a JPEG. Certainly no new information was added, so to what do you attribute the increase? Perhaps just different JPEG standards being used in the two encoders?

It's not a big increase mind you, If the extracted JPEG is about 3 MB, then I've found that the "save as" JPEG from PM at 100% is around 3.5 MB. Just wondering why it's bigger.

[devenh]

Like you, I find the Extract Embedded Jpg feature extremely useful.

Be aware that the quality of the the jpg will vary depending on:

1. Has the image been edited in NX2 (or NX)?  If so, the jpg embedded in the nef file will be full resolution and excellent jpg compression.  Typically, a nef file size will increase after NX2 editing because the quality jpg compression has gone from good to excellent.  For a D40, this may be a 2-3mb file size increase.

2. For some older cameras, like the D2h, the original (unedited by NX2) jpg image is not full resolution.  After NX2 editing, the embedded jpg will be full resolution.

The file size expanding when using Save As can be explained as follows:  Let's say NX2 embeds a jpg compressed in excellent quality and that this coincides with a PM equivalent of 95 quality.  At this quality level, there is some minor (visually lossless) compression going on.  When you use PM to Save As at quality 100, the image is loaded from the nef (using the embedded jpg) and then saved without (or very, very little) compression.  The result is a slightly larger file size.

Deven

[Kirk Baker]

Thanks Kirk,

It is good to know that NX embeds such high quality JPEGS in its NEFS and that PM can use these.

I was not aware that the file size could be increased by forming the JPEG of a JPEG. Certainly no new information was added, so to what do you attribute the increase? Perhaps just different JPEG standards being used in the two encoders?

It's not a big increase mind you, If the extracted JPEG is about 3 MB, then I've found that the "save as" JPEG from PM at 100% is around 3.5 MB. Just wondering why it's bigger.

At quality 100 with Chrominance Subsampling turned off our JPEG encoding is nearly lossless.  Other applications have different compression ratios than what we use.  Thus the difference in sizes.

-Kirk

[rjp]

Like you, I find the Extract Embedded Jpg feature extremely useful.

Be aware that the quality of the the jpg will vary depending on:

1. Has the image been edited in NX2 (or NX)?  If so, the jpg embedded in the nef file will be full resolution and excellent jpg compression.  Typically, a nef file size will increase after NX2 editing because the quality jpg compression has gone from good to excellent.  For a D40, this may be a 2-3mb file size increase.

2. For some older cameras, like the D2h, the original (unedited by NX2) jpg image is not full resolution.  After NX2 editing, the embedded jpg will be full resolution.

The file size expanding when using Save As can be explained as follows:  Let's say NX2 embeds a jpg compressed in excellent quality and that this coincides with a PM equivalent of 95 quality.  At this quality level, there is some minor (visually lossless) compression going on.  When you use PM to Save As at quality 100, the image is loaded from the nef (using the embedded jpg) and then saved without (or very, very little) compression.  The result is a slightly larger file size.

Deven

Thanks Deven,
Yes, this agrees with my observations too. When I open an original D40 NEF straight from the camera into PM it is much smaller than the same NEF after being opened and resaved as NEF by NX2. I had assumed that this was due to NX2 replacing the original embedded JPEG (from the camera) with it own version which was higher quality. Now you have confirmed this. Thanks.

With regard to the JPEG size increasing I think the question has still not been answered, perhaps because I did not explain it carefully enough. Of course if we make two JPEG versions of the same source image at, say, 95 and 100% respectively we will get two different file sizes, but that analogy does not exactly fit what is going on here. In this case only NX has the original RAW data to create its JPEG rendering. PM takes this JPEG, decodes it back to an RGB image, and then re-encodes it. This decode/encode process can not possibly add information to the image that was not there in the original JPEG, but the bit count has increased. This means it is taking more bits to convey the same information. It is not a higher quality JPEG, just a higher bit-count JPEG.

Since posting my question I did some reading on JPEG and I did learn something that may explain the increased file size. It seems the JPEG standard uses quantizer step sizes that are set according to the target quality (i.e., 0 through 100). And it appears that all these levels may still be used even if there is no discernible information in the image to warrant them, which leads to wasted bits.

For example, say an original RGB (TIFF) image has 30MB of data. Suppose you encode this as a low quality JPEG (say 20%) and get a file that is 1MB. If you then decode this JPEG to an intermediate RGB image and then re-encode this image at a high quality setting (say 100%) you may get a JPEG that is 5MB in size, but the result will never look any better than the 1MB version because the original loss of information can never be recovered. On a smaller scale I assume this is what is happening in PM when "save as JPEG" leads to more bits than "extract JPEG".Fortunately, the quality is already so high that it is likely never to make any difference in the finished result. My interest is really just a matter of curiosity.

[Kirk Baker]

Rich,

Like you, I find the Extract Embedded Jpg feature extremely useful.

Be aware that the quality of the the jpg will vary depending on:

1. Has the image been edited in NX2 (or NX)?  If so, the jpg embedded in the nef file will be full resolution and excellent jpg compression.  Typically, a nef file size will increase after NX2 editing because the quality jpg compression has gone from good to excellent.  For a D40, this may be a 2-3mb file size increase.

2. For some older cameras, like the D2h, the original (unedited by NX2) jpg image is not full resolution.  After NX2 editing, the embedded jpg will be full resolution.

The file size expanding when using Save As can be explained as follows:  Let's say NX2 embeds a jpg compressed in excellent quality and that this coincides with a PM equivalent of 95 quality.  At this quality level, there is some minor (visually lossless) compression going on.  When you use PM to Save As at quality 100, the image is loaded from the nef (using the embedded jpg) and then saved without (or very, very little) compression.  The result is a slightly larger file size.

Deven

Thanks Deven,
Yes, this agrees with my observations too. When I open an original D40 NEF straight from the camera into PM it is much smaller than the same NEF after being opened and resaved as NEF by NX2. I had assumed that this was due to NX2 replacing the original embedded JPEG (from the camera) with it own version which was higher quality. Now you have confirmed this. Thanks.

With regard to the JPEG size increasing I think the question has still not been answered, perhaps because I did not explain it carefully enough. Of course if we make two JPEG versions of the same source image at, say, 95 and 100% respectively we will get two different file sizes, but that analogy does not exactly fit what is going on here. In this case only NX has the original RAW data to create its JPEG rendering. PM takes this JPEG, decodes it back to an RGB image, and then re-encodes it. This decode/encode process can not possibly add information to the image that was not there in the original JPEG, but the bit count has increased. This means it is taking more bits to convey the same information. It is not a higher quality JPEG, just a higher bit-count JPEG.

Since posting my question I did some reading on JPEG and I did learn something that may explain the increased file size. It seems the JPEG standard uses quantizer step sizes that are set according to the target quality (i.e., 0 through 100). And it appears that all these levels may still be used even if there is no discernible information in the image to warrant them, which leads to wasted bits.

For example, say an original RGB (TIFF) image has 30MB of data. Suppose you encode this as a low quality JPEG (say 20%) and get a file that is 1MB. If you then decode this JPEG to an intermediate RGB image and then re-encode this image at a high quality setting (say 100%) you may get a JPEG that is 5MB in size, but the result will never look any better than the 1MB version because the original loss of information can never be recovered. On a smaller scale I assume this is what is happening in PM when "save as JPEG" leads to more bits than "extract JPEG".Fortunately, the quality is already so high that it is likely never to make any difference in the finished result. My interest is really just a matter of curiosity.

It is entirely reasonable that different JPEG quality settings (which you have correctly described as quantization values) will produce different sized files.  Your last example of the quality 20 image being re-saved at quality 100 is a perfect example of this.  JPEG compresses the data.  The uncompressed file would be quite a lot larger.  Some applications always subsample chrominance (color information) even at quality 100 whereas in Photo Mechanic this is optional.  Turning off the subsampling increases the color detail preservation and results in a larger file size.  And yes it is definitely true that re-saving a JPEG at a higher quality won't improve the original JPEG quality, but it will allow it to degrade less.

HTH,

-Kirk

[rjp]

fwiw, I did confirm that (for an "average" scene) the JPEG embedded by the camera in the NEF file is approximately 770KB, and when the same NEF file is opened and re-saved as NEF by NX the embedded JPEG becomes about 3.5MB. The NEF file itself grows by the same amount as we would expect since a much higher quality JPEG is now within.

So nice that PM can read these!

The JPEG-to-JPEG conversion used by the PM "save as" feature when a smaller output file is desired seems quite good too.

The camera is a Nikon D40 in this case.