Histogram puzzle...

[Don]

Nikon LS30, NikonScan 3.2, Photoshop 6

While studying histograms of a number of different scans (multiple
sources) in Photoshop (NikonScan's is too inexact) I noticed a weird
spike pattern in the first 20 values of individual channels and, of
course, the master.

In all cases (different sources, different images), pixels with the
value of 0, 5, 10, 15 and 21 have counts in 1000s, while intermediary
pixels (1-4, 6-9, 11-14 and 16-20) are all at 0!?!? After that there
are occasional 0-count pixels the last one being at 38.

Have I inadvertently set some clipping in NikonScan or is my scanner
dying?

While I'm at it, in the Photoshop's Histogram dialog what are Mean,
Standard Deviation and Median, exactly? Standard Deviation of what,
etc?

Thanks in advance, as always!

Don.

 

[Kennedy McEwen]

Nikon LS30, NikonScan 3.2, Photoshop 6

While studying histograms of a number of different scans (multiple
sources) in Photoshop (NikonScan's is too inexact)

Inexact in what way? I actually find the NS histogram superior to
Photoshop and the scaling is just the same in both, being 256 pixels
wide. In NS, the histogram can be set to show the effect of the level
and gamma adjustments *before* they are implemented, which cannot be
done in PS - at least not in PS7.01.

I noticed a weird
spike pattern in the first 20 values of individual channels and, of
course, the master.

In all cases (different sources, different images), pixels with the
value of 0, 5, 10, 15 and 21 have counts in 1000s, while intermediary
pixels (1-4, 6-9, 11-14 and 16-20) are all at 0!?!? After that there
are occasional 0-count pixels the last one being at 38.

Have I inadvertently set some clipping in NikonScan or is my scanner
dying?

More likely to have set inappropriate gamma on the Curves or LCH
palette. The LS30 is only a 10bpc scanner, so heavy use of the gamma
control will result in missing codes. Reset all of the Curves to
default by clicking the bottom right icon with the three lines on it
(and the same in LCH) should output a linear histogram with no missing
codes. Also check you are using the same colour space model in both
Photoshop and NikonScan.

While I'm at it, in the Photoshop's Histogram dialog what are Mean,
Standard Deviation and Median, exactly? Standard Deviation of what,
etc?

The mean, median and standard deviation of the data in the image or the
selected area of the image. For the red, green and blue histograms
these are the values of the respective data. For the luminosity they
are the values of a weighted combination of red, green and blue.

The mean is the average level of all of the data: add all the data up
and divide the total by the number of data.

The median is the middle of the data range: put all of the data in order
from lowest to highest and the median is the value of the data in the
middle. This is just another type of average that is less influenced by
outlying data values.

The standard deviation is the spread of the data, the square root of the
mean of the square of the deviations from the mean: for each datum
calculate the difference from the mean and square that (thus weighting
negative deviations the same as positive). Calculate the mean of all of
these values and take the square root of the result (thus returning to
the same dimensions as the data). Technically this is a virtually
meaningless parameter on typical image data, however on a gaussian
distribution, 68.27% of the data will be within one standard deviation
of the mean and 99.73% will be within 3x the standard deviation of it.

 

[Don]

Inexact in what way? I actually find the NS histogram superior to
Photoshop and the scaling is just the same in both, being 256 pixels
wide. In NS, the histogram can be set to show the effect of the level
and gamma adjustments *before* they are implemented, which cannot be
done in PS - at least not in PS7.01.

NS histogram doesn't show the actual pixel count (at least I haven't
found a way to do that in NS) while in Photoshop (I have 6 here) if
you go to Image/Histogram you get pixel count as you move the mouse.

However, what I do find better in NS is that curves are superimposed
on top of a histogram which is very handy. I wish PS 6 did that.

More likely to have set inappropriate gamma on the Curves or LCH
palette. The LS30 is only a 10bpc scanner, so heavy use of the gamma
control will result in missing codes. Reset all of the Curves to
default by clicking the bottom right icon with the three lines on it
(and the same in LCH) should output a linear histogram with no missing
codes. Also check you are using the same colour space model in both
Photoshop and NikonScan.

All settings are at neutral or manual i.e., no color management, auto
exposure off, etc. I've since noticed that histograms show weird
"ghosting" too. The histogram appears repeated with an offset and
superimposed back on. Hard to explain but have a look at a few
examples here:

http://members.aol.com/tempdon100164833/nikon/Blue.jpg
http://members.aol.com/tempdon100164833/nikon/Green.jpg
http://members.aol.com/tempdon100164833/nikon/Red.jpg

I've reinstalled NikonScan and tried to upgrade the firmware again
(same version, just to "refresh") but updater refuses because it is
the same version.

As I said before, I tried different films and settings but the weird
spikes and gaps are always there and always at *same* pixel values
(the actual counts vary). I also checked scans of the same slides I
did a few of months ago and they don't show these spikes and gaps.

I have a real bad feeling about all this. I mean, first I get this
"coincidence" of Nikon and Kodachrome not being on speaking terms,
then 3 months ago my docking station dies (read no SCSI!) but
fortunately I had a SCSI PCMCIA card. And now it appears the scanner
itself is dying! That's after only a year of use and no more than a
couple of hundred scans (which I have to do again because of the
Kodachrome thing...)!

And now this just when I'm finally getting a grip and making headway.
It seems every time I wrestle one problem to the ground another one
pops up! A year down the drain and nothing to show for it...

If I were religious (I'm not!) I'd say somebody up there doesn't want
me to scan! ;-)

Don.

 

[Kennedy McEwen]

NS histogram doesn't show the actual pixel count (at least I haven't
found a way to do that in NS) while in Photoshop (I have 6 here) if
you go to Image/Histogram you get pixel count as you move the mouse.

NS scan doesn't show the total number of pixels in the histogram, but
that is pretty much irrelevant. It does show the pixel value (input) as
the mouse moves, as well as the output level (ie. the effect the curve
adjustment would have if moved to that point on the transfer function),
which PS doesn't.

All settings are at neutral or manual i.e., no color management, auto
exposure off, etc. I've since noticed that histograms show weird
"ghosting" too. The histogram appears repeated with an offset and
superimposed back on. Hard to explain but have a look at a few
examples here:

http://members.aol.com/tempdon100164833/nikon/Blue.jpg
http://members.aol.com/tempdon100164833/nikon/Green.jpg
http://members.aol.com/tempdon100164833/nikon/Red.jpg

This looks like arithmetic rounding - probably due to the colour
management used since it appear to be more significant at the lower end
of the scale, where a gamma mismatch would dominate. I know you have it
turned off in NS, but what are you doing on PS? What gamma are you
using with NS and what does PS expect?

As I recall, the LS-x0 series (eg. LS-10/20/30) only output 8-bit data
although they had a 10bits ADC internally. Under those conditions any
gamma mismatch between the scanner and the application software will
give this rounding effect. Presumably there has been a firmware upgrade
to make the LS-30 output full 10-bits to the application and bypass the
internal data conversion. If not, that could give the rounding effect
as well. Fairly obviously, applying a transfer function from one 8-bit
range to another will result in some unpopulated codes and some codes
which increase their population - just as your histogram shows. For a
full 10-bit range, the same effect would be visible but should require a
slightly more significant transfer function to achieve it.

 

[Don]

NS scan doesn't show the total number of pixels in the histogram, but
that is pretty much irrelevant. It does show the pixel value (input) as
the mouse moves, as well as the output level (ie. the effect the curve
adjustment would have if moved to that point on the transfer function),
which PS doesn't.

Normally, I wouldn't care for the actual pixel count either, but in
this particular case it was important to know that intermediate values
were actually 0 and not a number too small to show up on the graph.

The input/output values as you move the mouse is another user friendly
function (in addition to already mentioned histogram as a background
image) which makes NS curves much better than PS. I also like the
ability to show all three channels together!

This looks like arithmetic rounding - probably due to the colour
management used since it appear to be more significant at the lower end
of the scale, where a gamma mismatch would dominate. I know you have it
turned off in NS, but what are you doing on PS? What gamma are you
using with NS and what does PS expect?

I'm on a Windows machine so the gamma in both NS and PS is 2.2.

As I recall, the LS-x0 series (eg. LS-10/20/30) only output 8-bit data
although they had a 10bits ADC internally.

That's correct.

Under those conditions any
gamma mismatch between the scanner and the application software will
give this rounding effect. Presumably there has been a firmware upgrade
to make the LS-30 output full 10-bits to the application and bypass the
internal data conversion.

Alas, not. The only way to get the full 10 bits is to program it
yourself (or use VueScan). Hey, that's a test I can try! Get raw
10-bit output and see what that looks like. The only problem is
turning off all the automatic "favors" VueScan does for the user which
is not easy to find with its... erm... "interesting" user interface...
FX: runs and hides from angry hordes of VueScan users carrying
pitchforks and torches... ;o)

Seriously though, what worries me is that I don't see this on older
scans of the very same slides. And it's so extreme! Several pixels at
0 and then all of a sudden a 10,000-count spike followed again by
several 0-count pixels Also, what about those overlapping "recursive"
waves in the histogram? Weird...

Don.

 

[Kennedy McEwen]

Normally, I wouldn't care for the actual pixel count either, but in
this particular case it was important to know that intermediate values
were actually 0 and not a number too small to show up on the graph.

On your examples they are true zeros. Photoshop has rounded histogram
data up since at least version 4 and possibly earlier, so even a single
pixel will result in a base point for that level. I am sure this is
documented in the help file somewhere, although I can't find it at the
moment, but its easy enough to prove to yourself - just paint a single
pixel in the image with one of those missing codes. ;-)

The input/output values as you move the mouse is another user friendly
function (in addition to already mentioned histogram as a background
image) which makes NS curves much better than PS. I also like the
ability to show all three channels together!

The best part though is what happens when you press the little histogram
icon - the histogram is transformed to reflect the transfer function you
have defined. That way you can see what the effect on the histogram is
before you actually implement the change! One day, Photoshop will do it
this way too. ;-)

I'm on a Windows machine so the gamma in both NS and PS is 2.2.

Not necessarily. Using Adobe gamma or some other colour management
within Windows? Something is definitely transforming levels through a
non-linear transfer function. Check the Colour Settings in Photoshop.
With PS7 these are under the Edit menu or by Ctrl-Shift-K, I think they
were the same in PS6, but can't remember well enough to be sure.

That's correct.


Alas, not.

In which case you are actually implementing the post scan adjustments on
8-bit data only with NS3.

The only way to get the full 10 bits is to program it
yourself (or use VueScan).

Are you sure about that? As I recall, the data was limited in the
scanner hardware and VS couldn't override that either. However this was
several years ago and Ed might have found a work around that since.

Seriously though, what worries me is that I don't see this on older
scans of the very same slides.

Are you sure you didn't scan these using an earlier software package,
such as NikonScan 1.63? As I recall, that operated significantly
different from version 3.1 in that it was designed to function with
scanners which only output 8-bit data but had higher internal
resolution. ie. the scanner hardware itself implemented the data
scaling, level shifting and gamma internally, prior to outputting 8-bit
data on the SCSI bus.

According to the software manual for NikonScan 3.1.2, the LS-30 scanner
is only supported with firmware version 1.3 and above.

Several pixels at
0 and then all of a sudden a 10,000-count spike followed again by
several 0-count pixels

That is what you get applying a gamma curve greater than unity to low
bit resolution data.

You can prove that gamma conversion gives this type of histogram just by
creating a blank image and filling it with a gradient from black to
white. Check the histogram. Then apply a gamma (say 2.2) using the
levels adjustment and view the histogram again. Should look very
similar to those you posted.

Also, what about those overlapping "recursive"
waves in the histogram? Weird...

Rounding of data - just the same as the missing codes, but the level
transformation is not severe enough to miss several levels completely.
This type of histogram is exactly what I would expect under these
circumstances, hence I am confident that somewhere between the data
coming out of the scanner and into the Photoshop display there is an
8-bit to 8-bit gamma conversion. If the problem is occurring in
Photoshop itself, you can use the dither option for conversion between
colorspaces to minimise the effect - this effectively adds random noise
to the data thus filling the missing codes. However it would be better
to eliminate it at source.

 

[Don]

On your examples they are true zeros. Photoshop has rounded histogram
data up since at least version 4 and possibly earlier, so even a single
pixel will result in a base point for that level. I am sure this is
documented in the help file somewhere, although I can't find it at the
moment, but its easy enough to prove to yourself - just paint a single
pixel in the image with one of those missing codes. ;-)

Good point! Literally and figuratively...

In which case you are actually implementing the post scan adjustments on
8-bit data only with NS3.

I'm not sure about that. I think NS3 is receiving 10 bits from the
scanner for post-scan processing but refusing to save them for
marketing reasons (so people are "nudged" into buying more expensive
scanners).

Just like Nikon's (market-driven) policy regarding various film
adapters. I believe there are adapters for different scanners which
are physically and electrically identical, yet Nikon implemented a
software "protection" to prevent them being used on other scanners.
But, I digress...

Is there a way to test if NS receives the full 10 bits before applying
curves, etc and then saves the result as 8 bits? Because if that's
what's happening, then I'll have to re-think my workflow - yet again.
In that case, even though curves are applied post-scan, it's much
better to do them in NS on 10-bit data rather than in PS on 8-bit
data. Another circumstantial evidence that this is what happens is NS'
ability to import PS curve settings. In other words, people more
comfortable with PS can do the corrections in PS on the initial scan
and then import those settings into NS for the final scan and have
them applied to 10-bit data. But I'm only guessing...

Are you sure about that? As I recall, the data was limited in the
scanner hardware and VS couldn't override that either. However this was
several years ago and Ed might have found a work around that since.

I seem to remember reading that VS could do that as well as multiscan
which is also not possible on LS30. I don't have Nikon SDK (software
developer kit) so I don't know what low level commands there are. For
example, if there are commands to control the stepper motor and
scanning separately (as there should!) then implementing multiscan
would be elementary.

Anyway, I remember reading about VS raw data from the scan which does
save 10 bits but I can't find it anymore... I can't even find the raw
data save option in VS menus but that may be because I have an
unregistered version.

Are you sure you didn't scan these using an earlier software package,
such as NikonScan 1.63? As I recall, that operated significantly
different from version 3.1 in that it was designed to function with
scanners which only output 8-bit data but had higher internal
resolution. ie. the scanner hardware itself implemented the data
scaling, level shifting and gamma internally, prior to outputting 8-bit
data on the SCSI bus.

According to the software manual for NikonScan 3.1.2, the LS-30 scanner
is only supported with firmware version 1.3 and above.

The first thing I did when I unpacked the scanner is upgrade both the
firmware and NS. But I know now why old scans have no spikes and
gaps...

Since last email I've been selectively turning on various NS options
(ICE, color management, etc) and once Nikon Color Management (NCM) is
on the spikes and gaps disappear! So, those old scans were done with
NCM. I know that also because they are tagged with a color profile
while these new ones are not!

BTW, I discovered that NCM is very "sticky". It's easy to turn on, but
hard to turn off. Restarting NS alone is not enough! At least in my
setup. This is what I did:

NCM is off.
Start NS, turn on the scanner.
Scan (A)

Turn NCM on.
Re-start NS as per instructions to activate NCM.
Scan (B) and, indeed, NCM induced color changes appear

Turn NCM off.
Re-start NS as per instructions to deactivate NCM.
Scan (C)

Now, in theory, scans A and C should be identical, right? Well, they
are not! Scan C is between A and B, closer to B!!! Only after I turn
the scanner off, did the subsequent scan (D) match scan A!

So, it appears, NCM stores some settings within the scanner's own
internal memory and this is not reset after NS is restarted leading to
unpredictable results. Caveat: At least that's the case in my setup...

That is what you get applying a gamma curve greater than unity to low
bit resolution data.

You can prove that gamma conversion gives this type of histogram just by
creating a blank image and filling it with a gradient from black to
white. Check the histogram. Then apply a gamma (say 2.2) using the
levels adjustment and view the histogram again. Should look very
similar to those you posted.

You're right!!! Phew! That means the scanner is OK... I was really
getting very worried it wasn't.

Rounding of data ....
you can use the dither ....
However it would be better
to eliminate it at source.

Absolutely! I want to get to the bottom of this.

Not necessarily. Using Adobe gamma or some other colour management
within Windows? Something is definitely transforming levels through a
non-linear transfer function. Check the Colour Settings in Photoshop.
With PS7 these are under the Edit menu or by Ctrl-Shift-K, I think they
were the same in PS6, but can't remember well enough to be sure.

I'm using a notebook with an LCD screen so color management (CM) is of
limited use since I can't find an ICC color profile for my machine
(VAIO PCG-818). Also, since I won't be printing, the only CM use would
be for displaying on other monitors (which is still important).

Having said that, I went and checked all the settings and they are as
follows:

Windows color management (Control Panel/Display
Properties/Settings/Advanced/Color Management) is set to "sRGB Color
Space Profile".

Adobe Gamma is set to: sRGB IEC61966-2.1 with Gamma at 2.2. The
phosphors are at HDTV (CCIR 709) with White Point at 6500 K but I
doubt those makes much sense for my notebook LCD screen!?

In PS, the Edit/Color Settings dialog has RGB working space set to
sRGB IEC61966-2.1 with Color Management Policies at "Preserve Embedded
Profiles". All mismatch checkboxes are at "Ask when opening".
Conversion options are: Engine = Adobe (ACE) and Intent = Relative
colorimetric. "Use blackpoint" and "Dither" checkboxes are on.
Finally, Desaturate and Blend Gamma checkboxes are off, but the values
are 20% and 1.00 respectively.

When I open a scan (which is untagged because NCM is off) I do indeed
get a dialog with 3 options: Don't do CM, use working profile (sRGB
IEC61966-2.1) and Assign profile (sRGB IEC61966-2.1). Regardless of
which option I pick the histogram gaps and spikes are still there.

PS help file says that for "Preserve Embedded Profiles" option
"Existing untagged documents use the current working space for editing
but remain untagged". Does that mean this is where the conversion
happens which causes gaps and spikes in my histogram?

I seem to remember reading somewhere in the NS help that it's possible
to embed a color profile without NCM but I can't find it right now.
I'll have a look again later, but could that be the solution?

I hope all this is not "too much information"... ;-) but I figure,
better more than less. You can always discard the irrelevant stuff.

Anyway, thanks for all the help, as always!

Don.

 

[Kennedy McEwen]

I'm not sure about that. I think NS3 is receiving 10 bits from the
scanner for post-scan processing but refusing to save them for
marketing reasons (so people are "nudged" into buying more expensive
scanners).

I am not convinced, I tend to prefer evidence to conspiracy theories -
see below for some evidence though. ;-) Certainly as originally
shipped this scanner would only output 8-bits, despite having a 10bit
ADC.

Is there a way to test if NS receives the full 10 bits before applying
curves, etc and then saves the result as 8 bits?

Not without the SDK - unless you are into disassembling the SCSI
handling module of the software, which you shouldn't really do without
Nikon's agreement in any case.

I seem to remember reading that VS could do that as well as multiscan
which is also not possible on LS30.

While Vuescan can do multiscan with the LS-30 it is only by using a
multipass technique which is much less successful than the single pass
approach available in suitable hardware, due to the pixel position
tolerances between passes which results in a blurring of the image. Even
so, with only 8-bit data available, the benefits of multiscan is reduced
significantly in any case, since you are dealing with systematic
quantisation, not random noise. I have had private communication from
Ed Hamrick from several years ago, admittedly when Vuescan was in its
relative infancy, discussing this issue and the additional bits that
Nikon handles internally to see if it was possible for these to be made
available and indeed used by Vuescan in the multiscan operation. Ed was
quite specific that these bits could not be made available and the
truncation occurred in the hardware. As I said previously, he may have
discovered a work-around since then but I doubt it. Why don't you ask
him yourself, he is a very approachable guy - especially if you buy his
product. ;-)

The first thing I did when I unpacked the scanner is upgrade both the
firmware and NS. But I know now why old scans have no spikes and
gaps...

Since last email I've been selectively turning on various NS options
(ICE, color management, etc) and once Nikon Color Management (NCM) is
on the spikes and gaps disappear!

This is evidence which seems to support my theory about the data depth
you are getting from the scanner and where the processing is
implemented. If implemented in the scanner then the conversion to sRGB
for windows will use the full 10-bit ADC range. Switching NCM off will
mean you only have 8 bits from which to apply the standard colour
management. If you recall the experiment with the grey ramp in
Photoshop - a gamma of 2.2 looks very similar to the histograms you
posted. I am sure this is not coincidence, but see below again. ;-)

BTW, I discovered that NCM is very "sticky". It's easy to turn on, but
hard to turn off. Restarting NS alone is not enough! At least in my
setup.

So, it appears, NCM stores some settings within the scanner's own
internal memory and this is not reset after NS is restarted leading to
unpredictable results. Caveat: At least that's the case in my setup...

Quite possibly - NS3 was written with the intention of everything being
implemented by the computer software, not the scanner, and targeted to
high end scanners. It only became available when the LS-40/4000 devices
were introduced - up till then the highest version number was 1.63. I
am not convinced that the LS-30 was seriously considered as a target
platform for NS3, and this may be a consequential bug.

You're right!!! Phew! That means the scanner is OK... I was really
getting very worried it wasn't.

Good - but note how similar the histogram for a gamma of 2.2 is to your
posted examples. Now try the following, starting from the same blank
image and gradient from black to white. However, before implementing
any gamma adjustment, convert the image to 16 bits per channel. Then,
using the levels adjustment, scale the data output range to 0 to 63
instead of the default range of 0 to 255. That has the effect of
reducing your 8-bit gradient to 10-bit precision, although only a
quarter of the full scale. Now implement a gamma of 2.2 as before and
view the histogram. No missing codes - and far less significant
rounding effects. My guess is that this looks pretty similar to the old
histograms you got previously, although the rounding effect may be a
little less pronounced due to random noise on the ADC.

In short, with NCM you appear to be implementing the Windows standard
gamma adjustment on the full 10-bit range, internally in the scanner
itself, however without it you appear to be implementing that gamma on
the truncated 8-bit output of the scanner itself.

When not using NCM, have you set the gamma of Nikonscan to 2.2?

Absolutely! I want to get to the bottom of this.

Err.. I am pretty certain the "recursive waves" are because you actually
have got to the bottom of it, right down to the lowest available bits!
;-)

I'm using a notebook with an LCD screen so color management (CM) is of
limited use since I can't find an ICC color profile for my machine
(VAIO PCG-818). Also, since I won't be printing, the only CM use would
be for displaying on other monitors (which is still important).

Having said that, I went and checked all the settings and they are as
follows:

Windows color management (Control Panel/Display
Properties/Settings/Advanced/Color Management) is set to "sRGB Color
Space Profile".

Adobe Gamma is set to: sRGB IEC61966-2.1 with Gamma at 2.2. The
phosphors are at HDTV (CCIR 709) with White Point at 6500 K but I
doubt those makes much sense for my notebook LCD screen!?

In PS, the Edit/Color Settings dialog has RGB working space set to
sRGB IEC61966-2.1 with Color Management Policies at "Preserve Embedded
Profiles". All mismatch checkboxes are at "Ask when opening".
Conversion options are: Engine = Adobe (ACE) and Intent = Relative
colorimetric. "Use blackpoint" and "Dither" checkboxes are on.
Finally, Desaturate and Blend Gamma checkboxes are off, but the values
are 20% and 1.00 respectively.

When I open a scan (which is untagged because NCM is off) I do indeed
get a dialog with 3 options: Don't do CM, use working profile (sRGB
IEC61966-2.1) and Assign profile (sRGB IEC61966-2.1). Regardless of
which option I pick the histogram gaps and spikes are still there.

PS help file says that for "Preserve Embedded Profiles" option
"Existing untagged documents use the current working space for editing
but remain untagged". Does that mean this is where the conversion
happens which causes gaps and spikes in my histogram?

Yes - and with 8-bit data you get the results you are observing,
recursive waves and all.

 

[Don]

I am not convinced, I tend to prefer evidence to conspiracy theories -
see below for some evidence though. ;-) Certainly as originally
shipped this scanner would only output 8-bits, despite having a 10bit
ADC.

I think you're right, after all. This is circumstantial, but NEF
format is supposed to be Nikon's equivalent of (semi-) raw scanner
data and (judging by file size) NEF also appears to be 8-bit.

Not without the SDK - unless you are into disassembling the SCSI
handling module of the software, which you shouldn't really do without
Nikon's agreement in any case.

I've done that half-way when I was trying to turn the Kodachrome mode
on for LS30. However, after realizing that the SCSI module doesn't
even have a Kodachrome mode, I stopped.

I'm not concerned with breaking the agreement because after what I've
been through with Nikon - as far as I'm concerned - they broke all
sorts of things long ago. I know, I know... "But, your honor, they
started it!" is not exactly a solid legal position...

Quite possibly - NS3 was written with the intention of everything being
implemented by the computer software, not the scanner, and targeted to
high end scanners. It only became available when the LS-40/4000 devices
were introduced - up till then the highest version number was 1.63. I
am not convinced that the LS-30 was seriously considered as a target
platform for NS3, and this may be a consequential bug.

That's exactly what I've been thinking (also, see below the disconnect
between the NS help file and actual options I get when using SCSI!)!
After all, Nikon abandoned SCSI altogether in the subsequent NS
release. I suppose they just didn't feel like opening that can of
worms for a very limited number of contrarians (like me) who insist on
using an LS30... ;o)

Actually, if I weren't going through some pretty lean times right now,
I would have bought another scanner months ago... Unfortunately I
really have no choice but to continue wrestling with the LS30. :-(

Err.. I am pretty certain the "recursive waves" are because you actually
have got to the bottom of it, right down to the lowest available bits!
;-)

Again, literally and figuratively!

Good - but note how similar the histogram for a gamma of 2.2 is to your
posted examples. Now try the following, starting from the same blank
image and gradient from black to white. However, before implementing
any gamma adjustment, convert the image to 16 bits per channel.

That alone made the histogram look quite "funky"...

Then,
using the levels adjustment, scale the data output range to 0 to 63
instead of the default range of 0 to 255. That has the effect of
reducing your 8-bit gradient to 10-bit precision, although only a
quarter of the full scale. Now implement a gamma of 2.2 as before and
view the histogram. No missing codes - and far less significant
rounding effects. My guess is that this looks pretty similar to the old
histograms you got previously, although the rounding effect may be a
little less pronounced due to random noise on the ADC.

Yes, that's right. So, you got it in one!

In short, with NCM you appear to be implementing the Windows standard
gamma adjustment on the full 10-bit range, internally in the scanner
itself, however without it you appear to be implementing that gamma on
the truncated 8-bit output of the scanner itself.

And I have some more tests below...

Yes - and with 8-bit data you get the results you are observing,
recursive waves and all.
Also:

This is evidence which seems to support my theory about the data depth
you are getting from the scanner and where the processing is
implemented. If implemented in the scanner then the conversion to sRGB
for windows will use the full 10-bit ADC range. Switching NCM off will
mean you only have 8 bits from which to apply the standard colour
management. If you recall the experiment with the grey ramp in
Photoshop - a gamma of 2.2 looks very similar to the histograms you
posted. I am sure this is not coincidence, but see below again. ;-)

OK, in the meantime it's quite evident that all histogram "oddities"
happen within NS itself. The following has all been done within NS.

Given: Scanner Extras/Interpolation Method set to "Bilinear" (NCM is
still off).

I do a *preview* scan and the NS histogram looks fine. Then, I perform
the scan itself (with auto-save) without changing anything.

Next, I use NS to load this very scan back in and the NS histogram
goes "dodgy" again!

So, something happens between the preview and the actual scan! To me,
that looks like another major bug! If anything, I would have expected
the preview to produce artifacts - because of its low quality - not
the actual scan!? But, then again, I don't really know what's going on
underneath.


I also discovered something else. Under "Scanner Extras/Interpolation
Method" I have three options: Bilinear, Neighbor and None. (BTW, the
help file here is totally useless because it only mentions "Default"
and "Lo-con" options which I don't even have!!!! Another proof that
Nikon totally ignored SCSI module ramifications in NS3!)

When I select either Neighbor or None (NCM is still off), even the
preview shows a dodgy histogram! I'm not sure what the implications of
this are (other than it looks inconsistent to me) but I'm sure it will
make more sense to you.

BTW, what does this Interpolation Method refer to? Is it the
interpolation from 10 bits to 8?

Finally, I have a Calibrate button right next to it. What exactly is
being calibrated? The so-called help file is, again, totally useless.


Anyway, I'm at the horns of a dilemma. If I turn NCM on I get all
sorts of unwanted "favors" and "adjustments" in the image, but if I
turn NCM off I get a "dodgy" histogram.

Is there something (anything?) I can do to get a decent histogram
without using NCM?

Don.

 

[Jean]

| On Fri, 13 Feb 2004 20:07:44 +0000, Kennedy McEwen
|
.....snip, snip
|
|
|
| Anyway, I'm at the horns of a dilemma. If I turn NCM on I get all
| sorts of unwanted "favors" and "adjustments" in the image, but if I
| turn NCM off I get a "dodgy" histogram.
|
| Is there something (anything?) I can do to get a decent histogram
| without using NCM?
|
| Don.


Sorry if you've already answered this, but have you tried enabling Nikon
Color Management and using the "Scanner RGB" color space?


Jean

 

[Kennedy McEwen]

The following has all been done within NS.

Given: Scanner Extras/Interpolation Method set to "Bilinear" (NCM is
still off).

I do a *preview* scan and the NS histogram looks fine. Then, I perform
the scan itself (with auto-save) without changing anything.

Next, I use NS to load this very scan back in and the NS histogram
goes "dodgy" again!

So, something happens between the preview and the actual scan! To me,
that looks like another major bug! If anything, I would have expected
the preview to produce artifacts - because of its low quality - not
the actual scan!? But, then again, I don't really know what's going on
underneath.


I also discovered something else. Under "Scanner Extras/Interpolation
Method" I have three options: Bilinear, Neighbor and None. (BTW, the
help file here is totally useless because it only mentions "Default"
and "Lo-con" options which I don't even have!!!! Another proof that
Nikon totally ignored SCSI module ramifications in NS3!)

When I select either Neighbor or None (NCM is still off), even the
preview shows a dodgy histogram! I'm not sure what the implications of
this are (other than it looks inconsistent to me) but I'm sure it will
make more sense to you.

There does appear to be some inconsistency, but the whether the
resultant scan and the preview show rounding effects in the histogram
with any particular interpolation method will depend critically on the
size of both preview and final scan.

BTW, what does this Interpolation Method refer to? Is it the
interpolation from 10 bits to 8?

No - it is the interpolation from the scanner native resolution to the
resolution used in the preview and final scan. This is explained fully
on page 55 & 60 of the NikonScan 3 Software Manual, which you should
have in pdf format (file nsrmena1.pdf). This is a 30Mb file, otherwise
I would send you the entire document! However, quoting the relevant
section:

Page 55 Scanner Extras Options
"LS-2000/COOLSCAN III only
Option Description
Interpolation Controls how the scanner generates "extra" pixels
method not included in the information from the scanning
element.

Prescan mode Controls the autoexposure operation performed
before scanning negatives, producing excellent
results even with low-contrast originals."

Page 60
"Interpolation Method (LS-2000/COOLSCAN III only)
Your scanner can scan at 2,700 ppi and at any resolution below this
value that can be produced by dividing 2,700 by a whole number (1,350
ppi, 900 ppi, 675 dpi, etc.). At other resolutions, pixels must be
calculated by computer algorithm to make up the specified number of
pixels per inch. The interpolation method is the algorithm used in
generating these extra pixels. The following interpolation methods are
available:

Method Description

Bilinear Extra pixels are generated using a bilinear algorithm.

Neighbor Extra pixels are generated using a "nearest neighbor"
algorithm.

None No interpolation is performed. Note that this may result
in the output size for the scanned image being larger
than that specified in the Crop palette."

The "Lo-Cont" settings refer to the Prescan mode of the autoexposure,
not the interpolation method used. It would appear that your Help file
is corrupted - at least it is very different from mine. :-(

Finally, I have a Calibrate button right next to it. What exactly is
being calibrated? The so-called help file is, again, totally useless.

The dark current and response of each element in the CCD, whether in a
scanner, digital camera or Mars rover, requires to be matched to a
normalised output for the entire device. Whilst the response if fairly
stable, the dark current is a strong function of temperature for all
silicon devices, doubling roughly every 8-12degC. Several calibration
algorithms are available, however a simple linear response assumption is
usually adequate for silicon CCDs. So at any given time and temeprature,
the output, y, of a CCD element is a linear function of the
iillumination on it, x. Moathematially, each CCD element has a transfer
function of the form y(i) = m(i).x(i) + c(i), where i is the pixel
number, m(i) its response and c(i) the dark current. Similarly, the
average output of the array is Y = M.X + C. For most applications, the
mean dark current on the CCD is not required and the output required is
actually Y-C = M.X, such that zero illumination produces zero output.

By viewing uniform black and white references, fairly simple algebra
permits you to calculate correction terms for each CCD element, R(i) a
O(i) such that Y-C = R(i).y(i) + O(i). In short, the individual
transfer function of each element in the CCD is converted to a uniform
transfer function for the entire CCD. Since the response of the CCD
element is usually stable, the calculation of R(i) terms may be
implemented only at the original manufacturing stage and the only
calculation necessary in use is that of O(i). Irrespective of this, the
array of data R(i) and O(i) are the calibration coefficients of the CCD
for that particular temperature and the process of computing them,
including viewing reference surfaces is the calibration process.

When each slide or film is placed in it the scanner should automatically
implement a calibration process. If you leave the film in place for too
long then the temperature of the CCD may change and the calibration will
no longer be valid, resulting in lines along the scan direction. The
calibrate button simply forces the scanner to implement a calibration
when it is pressed, in addition to those it has already performed.

Anyway, I'm at the horns of a dilemma. If I turn NCM on I get all
sorts of unwanted "favors" and "adjustments" in the image, but if I
turn NCM off I get a "dodgy" histogram.

Is there something (anything?) I can do to get a decent histogram
without using NCM?

My best guess would be to set the scanner gamma to 2.2 (assuming a PC)
and interpolation to NONE. That should implement gamma correction on
the 10-bit data however, if not, you can try again with gamma set to 1.0
but, whilst that will give normal looking histograms, the problem is
only being shifted further downline, because you need to implement the
gamma at some stage. What is your concern about using NCM when it
clearly works and is giving the best results with your scanner?

 

[Don]

| On Fri, 13 Feb 2004 20:07:44 +0000, Kennedy McEwen
|
....snip, snip
|
|
|
| Anyway, I'm at the horns of a dilemma. If I turn NCM on I get all
| sorts of unwanted "favors" and "adjustments" in the image, but if I
| turn NCM off I get a "dodgy" histogram.
|
| Is there something (anything?) I can do to get a decent histogram
| without using NCM?
|

Sorry if you've already answered this, but have you tried enabling Nikon
Color Management and using the "Scanner RGB" color space?

No, we haven't touched on this yet, but I have tried the "Scanner RGB"
color space on my own. Unfortunately, the histogram goes funny as well
with all the gaps and spikes.

Nevertheless, thanks very much for the idea, Jean!! Much appreciated!

I didn't think anybody except Kennedy and I actually read all these
long messages to the very end... ;-)

Don.

 

[Don]

No - it is the interpolation from the scanner native resolution to the
resolution used in the preview and final scan. This is explained fully
on page 55 & 60 of the NikonScan 3 Software Manual, which you should
have in pdf format (file nsrmena1.pdf).

I don't have this! When I upgraded about a year ago all I got was a
help and a read me file... I just tried locating "nsrmena1.pdf" on the
Nikon's web site but it's nowhere to be found. The only manual I see
there is UG for NikonScam 4.

This is a 30Mb file, otherwise
I would send you the entire document! However, quoting the relevant
section:

....

Thanks very much, Kennedy!

Reading this I see it's not really applicable in my case since I only
scan at 2700. Even during testing I make sure that when I downsample I
avoid fractional resolutions - meaning I only use 1350 or 675.

The "Lo-Cont" settings refer to the Prescan mode of the autoexposure,
not the interpolation method used. It would appear that your Help file
is corrupted - at least it is very different from mine. :-(

I think they just didn't bother checking for LS30 users probably
thinking "nobody uses *that* scanner anymore"... :-(

The dark current and response of each element in the CCD, whether in a
scanner, digital camera or Mars rover, requires to be matched to a
normalised output for the entire device. Whilst the response if fairly
stable, the dark current is a strong function of temperature for all
silicon devices, doubling roughly every 8-12degC. Several calibration
algorithms are available, however a simple linear response assumption is
usually adequate for silicon CCDs. So at any given time and temeprature,
the output, y, of a CCD element is a linear function of the
iillumination on it, x. Moathematially, each CCD element has a transfer
function of the form y(i) = m(i).x(i) + c(i), where i is the pixel
number, m(i) its response and c(i) the dark current. Similarly, the
average output of the array is Y = M.X + C. For most applications, the
mean dark current on the CCD is not required and the output required is
actually Y-C = M.X, such that zero illumination produces zero output.

I thought that, because Nikon scanners use LEDs as the light source
(instead of a lamp), the temperature was not really an issue. After
all, that's why the whole range is called "CoolScan". On reflection,
now that you mention it, I suppose the temperature inside the scanner
would vary simply because of the transformer and other electronics.

When each slide or film is placed in it the scanner should automatically
implement a calibration process. If you leave the film in place for too
long then the temperature of the CCD may change and the calibration will
no longer be valid, resulting in lines along the scan direction. The
calibrate button simply forces the scanner to implement a calibration
when it is pressed, in addition to those it has already performed.

In a serendipitous way that explains something else! During the
intensive testing over the last few days I noticed a couple of scans
with lines along the scan direction. I attributed that to the buggy
NikonScan (it was when I discovered the NCM-off bug). Especially since
that occurred only on a couple of scans. But it appears that was a
calibration problem because I did indeed leave a slide in for a very
long time trying out various settings and so on. Another mystery
solved! Thanks again!

My best guess would be to set the scanner gamma to 2.2 (assuming a PC)
and interpolation to NONE. That should implement gamma correction on
the 10-bit data however, if not, you can try again with gamma set to 1.0
but, whilst that will give normal looking histograms, the problem is
only being shifted further downline, because you need to implement the
gamma at some stage. What is your concern about using NCM when it
clearly works and is giving the best results with your scanner?

I already tried both, setting the gamma to 1 and interpolation at NONE
(interpolation was at NONE all along until I went over all the options
the other day) but it didn't make any difference.

My main issue with NCM is that it amplifies problems rather than
ameliorating them. Here are a couple of scans with everything at
neutral, no interpolation, RGB AG at +2:

http://members.aol.com/tempdon100164833/nikon/NCMon.jpg
http://members.aol.com/tempdon100164833/nikon/NCMoff.jpg

As you can see the "NCM on" scan decreased red and increased blue
which is exactly the *opposite* of what's needed.

Even though the red is down by only a couple of points every little
bit counts considering the extremely limited dynamic range I have to
wrestle with. And that's at +2 AG! Not to mention the amplification of
the blue cast which was the original problem I started with.

Don.

 

[Kennedy McEwen]

I thought that, because Nikon scanners use LEDs as the light source
(instead of a lamp), the temperature was not really an issue. After
all, that's why the whole range is called "CoolScan". On reflection,
now that you mention it, I suppose the temperature inside the scanner
would vary simply because of the transformer and other electronics.

1. The LED light source is very stable in both wavelength and intensity.

2. The response of each element in the CCD is fairly stable.

3. The dark current (DC offset) of each element in the CCD is a strong
function of temperature.

1 does not require calibration but 3, and occasionally 2, does.
Calibration is an issue for the CCD, not the light source. All CCDs
(and CMOS sensors) require this.

Here are a couple of scans with everything at
neutral, no interpolation, RGB AG at +2:

http://members.aol.com/tempdon100164833/nikon/NCMon.jpg
http://members.aol.com/tempdon100164833/nikon/NCMoff.jpg

As you can see the "NCM on" scan decreased red and increased blue
which is exactly the *opposite* of what's needed.

Even though the red is down by only a couple of points every little
bit counts considering the extremely limited dynamic range I have to
wrestle with. And that's at +2 AG! Not to mention the amplification of
the blue cast which was the original problem I started with.

I assume that you have tried adjusting the AG of the colour channels
individually. Even at +2 on the master AG, the individual channels can
still be increased by another 2EV. Having said that however, the
highlights of your image are clearly at peak white, so it is simply the
dynamic range of the image on the film which is much too high for the
LS-30 to cope with. Which brings us back to where you came in - the
LS-30 really just isn't up to the job you are putting it to.

 

[Don]

I assume that you have tried adjusting the AG of the colour channels
individually. Even at +2 on the master AG, the individual channels can
still be increased by another 2EV.

Yes, indeed. And in this particular example that's what I had to do to
get a half-decent range. Even with red AG at +2 (i.e. +4 in total!!)
the red channel is still far away from 255.

To be fair, this is one of the extreme slides. There are many others
which are much brighter and easier to handle - even without AG (still
have to remove the blue cast, of course). But I've been focusing on
the hard ones because they are the weak link.

Having said that however, the
highlights of your image are clearly at peak white, so it is simply the
dynamic range of the image on the film which is much too high for the
LS-30 to cope with.

In this particular case (the reflection on the building) I don't mind
burning that out and losing detail if in exchange I get the rest of
the image reasonably adjusted.

In other cases, I played around with scanning twice with different AG
values and then combining best parts of both scans. In addition to the
obvious problem of alignment there is also a problem with different
color balances.

But, I soldier on...

Which brings us back to where you came in - the
LS-30 really just isn't up to the job you are putting it to.

Which is why I will have such a fantastic feeling when all this is
over - and I take the crowbar to the beast to get my just revenge! ;o)

But seriously, in the end it's a learning experience (the hard way).
After all this, working with a "real" scanner will be a walk in the
park.

Don.

 

[Kennedy McEwen]

Yes, indeed. And in this particular example that's what I had to do to
get a half-decent range. Even with red AG at +2 (i.e. +4 in total!!)
the red channel is still far away from 255.

To be fair, this is one of the extreme slides. There are many others
which are much brighter and easier to handle - even without AG (still
have to remove the blue cast, of course). But I've been focusing on
the hard ones because they are the weak link.

To be honest I don't follow the logic in that. You have a scanner which
you know has significant limitations, particularly Dmax and dynamic
range, yet you continue to struggle with images you know are beyond its
limits. Flagellation of deceased equines indeed! With all the analogue
gains turned up to max there is little more you can do within the limits
of the NS software.

Vuescan has an exposure lock facility which might help. I have never
used it but it may be possible to expose on a very dark, even totally
opaque, slide to get maximum exposure, lock that and then scan the
slides you want. You will still have the same overall dynamic range,
which is insufficient for the image, but at a range of deeper densities.

 

[Bart van der Wolf]

SNIP

Vuescan has an exposure lock facility which might help. I have never
used it but it may be possible to expose on a very dark, even totally
opaque, slide to get maximum exposure, lock that and then scan the
slides you want. You will still have the same overall dynamic range,
which is insufficient for the image, but at a range of deeper densities.

VueScan does indeed allow very different exposure times. VueScan also has a
"Long exposure" option, which takes two scans with different exposure (and
registration limitations) before blending the two results. Sometimes it
works, sometimes it fails.

Bart

 

[Don]

To be honest I don't follow the logic in that. You have a scanner which
you know has significant limitations, particularly Dmax and dynamic
range, yet you continue to struggle with images you know are beyond its
limits. Flagellation of deceased equines indeed! With all the analogue
gains turned up to max there is little more you can do within the limits
of the NS software.

It's very simple: I have no choice! I'm certainly not submitting to
this torture voluntarily!!! I can't afford a new scanner, so it's this
or nothing... Faced with that bleak "choice" the logical thing to do
is get the best out of it. Indeed, considering the shortcomings of the
hardware it's that much more important to squeeze the very last bit
(sic) out of it. And that's all I've been trying to do.

If that means "flagellation of deceased equines", then I'll just do
that until bovines return to their domicile... ;o)

Vuescan has an exposure lock facility which might help. I have never
used it but it may be possible to expose on a very dark, even totally
opaque, slide to get maximum exposure, lock that and then scan the
slides you want. You will still have the same overall dynamic range,
which is insufficient for the image, but at a range of deeper densities.

Well, that was my original idea but you said there is no one universal
setting. Unless (above) you're only referring to a specific subset of
similar slides (e.g. same film, same scene, etc).

Besides, locking any one AG setting is not really a problem and using
VueScan to do that would be overkill. Not to mention that I'm not a
big fan of VueScan. (This is no slur, just personal opinion.)

Better the devil you know...

Don.

 

[Don]

VueScan does indeed allow very different exposure times. VueScan also has a
"Long exposure" option, which takes two scans with different exposure (and
registration limitations) before blending the two results. Sometimes it
works, sometimes it fails.

That's essentially what I have been doing with analog gain, and then
combining two images getting the best out of each in the attempt to
extend the dynamic range. As you mention, this is tricky so I prefer
to do it myself and have full control at each stage.

Don.

 

[Bart van der Wolf]

That's essentially what I have been doing with analog gain, and then
combining two images getting the best out of each in the attempt to
extend the dynamic range. As you mention, this is tricky so I prefer
to do it myself and have full control at each stage.

Agreed, but you'll have to do it at a linear gamma 1.0 setting before
combining different exposures. The possibility of registration by hand is
probably more accurate than relying on the scans starting at exactly the
same (stepper motor and play) position, especially if you enlarge the image
first.

Bart