NOT scanning negatives as positives

[ljlbox]

Is there any particular reason why scanner drivers should be told to
scan negatives as negatives and not as positives, *besides* that they
set channel exposures appropriate for removing the orange mask in
negative mode?

I ask because my Epson scanner has, apparently, a very weird way of
letting the user control exposure: when the gamma table for a channel
is changed, the exposure changes, so that if I want a longer exposure
than normal, I must tell it to maps values from 0 to 255 to (for
example) 0 to 128.

When in negative mode, however, this seems to conflict with the
exposures that are automatically applied in order to remove the orange
mask, so that the more exposure I try to apply, the more the orange
mask shows up.

So, if I knew for sure that scanning as positive vs negative is only a
matter of exposure, I would simply scan as positive, with exposures
manually corrected to account for the orange mask.


Besides all this, does anyone know more about the strange behaviors
exhibited by my scanner (Epson RX500), or by Epson scanners with
similar characteristics? All this mixed exposure/gamma table/color
correction (oh yeah, it does also apply different color corrections for
negatives and positives) is giving me a headache.

by LjL
(e-mail address removed)

 

[Don]

Is there any particular reason why scanner drivers should be told to
scan negatives as negatives and not as positives, *besides* that they
set channel exposures appropriate for removing the orange mask in
negative mode?

Not really. Whether it's positive or negative the scanner still scans
the same way. The only difference is in post-processing. The negative
setting simply tells the scanner *software* to reverse the image
afterwards. This may include additional things like boosting contrast.

Some people are not happy with this as it may cause clipping so they
simply scan negatives as positive and then do the rest themselves,
thereby getting more control over the process.

I ask because my Epson scanner has, apparently, a very weird way of
letting the user control exposure: when the gamma table for a channel
is changed, the exposure changes, so that if I want a longer exposure
than normal, I must tell it to maps values from 0 to 255 to (for
example) 0 to 128.

It's not entirely clear what you're doing but gamma has nothing to do
with exposure. Gamma does *not* change either end of the histogram,
only the pixels in between. Therefore, it's simply physically
impossible to map values 0 to 255 to 0 to 128 using gamma.

You may perceive the image getting "brighter" after you change gamma
but all that is done *after* the scan. The exposure is only relevant
while you scan, anything that happens afterwards is simply image
editing. So even if you change the contrast of the image by setting
Levels or using Curves to map 255 to 128, all that does is process the
existing data *after* the scan.

If you need to do that, then Levels and Curves are the wrong way to go
about it. You should boost exposure until the histogram reaches the
right edge instead of stretching existing data of an underexposed
scan.

I don't have an Epson so I'll have to yield to someone who does with
practical advice as it appears your problems are not with negative
conversion but simply how to operate the software.

Don.

 

[Ken Weitzel]

Not really. Whether it's positive or negative the scanner still scans
the same way. The only difference is in post-processing. The negative
setting simply tells the scanner *software* to reverse the image
afterwards. This may include additional things like boosting contrast.

Some people are not happy with this as it may cause clipping so they
simply scan negatives as positive and then do the rest themselves,
thereby getting more control over the process.




It's not entirely clear what you're doing but gamma has nothing to do
with exposure. Gamma does *not* change either end of the histogram,
only the pixels in between. Therefore, it's simply physically
impossible to map values 0 to 255 to 0 to 128 using gamma.

You may perceive the image getting "brighter" after you change gamma
but all that is done *after* the scan. The exposure is only relevant
while you scan, anything that happens afterwards is simply image
editing. So even if you change the contrast of the image by setting
Levels or using Curves to map 255 to 128, all that does is process the
existing data *after* the scan.

If you need to do that, then Levels and Curves are the wrong way to go
about it. You should boost exposure until the histogram reaches the
right edge instead of stretching existing data of an underexposed
scan.

I don't have an Epson so I'll have to yield to someone who does with
practical advice as it appears your problems are not with negative
conversion but simply how to operate the software.

Don.

Hi...

Epson scan software does indeed have the capability of
adjusting exposure.

It also has another tab that will let you look at the
histogram.

The downside is that as you adjust exposure the preview
image updates nicely, but the histogram doesn't.

Perhaps if any Epson folks are reading this they might
add it to the wish list

Ken

 

[Don]

Hi...

Epson scan software does indeed have the capability of
adjusting exposure.

It also has another tab that will let you look at the
histogram.

The downside is that as you adjust exposure the preview
image updates nicely, but the histogram doesn't.

That's certainly not good!!! Isn't there a way to "update" the
histogram somehow?

What I'm getting at is this: In NikonScan it's sometimes necessary to
click the Redraw button (as well as bring forward the Processed
window) in order to see the various changes one makes to the settings.

Is there something similar in Epson software?

Perhaps if any Epson folks are reading this they might
add it to the wish list

Or add it to the "avoid" list, as the case may be... ;o)

Don.

 

[ljlbox]

Don ha scritto:

Not really. Whether it's positive or negative the scanner still scans
the same way. The only difference is in post-processing. The negative
setting simply tells the scanner *software* to reverse the image
afterwards. This may include additional things like boosting contrast.

Actually, the "Negative" setting on my scanner doesn't even
automatically reverse the image. All it seems to do is adjust the
exposure, but I just wanted a confirmation.
By the way, I'm using the SANE libraries from Epson Avasys (was Epson
Kowa), not the Windows scanning software. The Windows software (Epson
Scan) definitely does invert negatives.

Some people are not happy with this as it may cause clipping so they
simply scan negatives as positive and then do the rest themselves,
thereby getting more control over the process.

Doesn't *seem* to alter contrast, but I really can't be sure :-\ Of
course, even just changing the exposure may cause clipping, if the film
mask is brighter than the software thinks it is...

It's not entirely clear what you're doing but gamma has nothing to do
with exposure. Gamma does *not* change either end of the histogram,
only the pixels in between. Therefore, it's simply physically
impossible to map values 0 to 255 to 0 to 128 using gamma.

I'll explain. I used the term "gamma" simply because that's the name of
the option, i.e. "--red-gamma-table", "--green-gamma-table" and
"--blue-gamma-table".
However, what these options let me specify is a complete map of color
values, so that, for example,

0,2,4,6,8,10,12,14,...,254,255,255,...,255

gives me an image with twice the brightness (if brightness is the
correct term), without really changing *gamma*.

Now, the funny thing is that if I specify a gamma table like that, the
scanner also used to (*) change its exposure time accordingly; so, with
that specific table, it would make the exposure about twice as longer
than usual.

Of course, if there was an "--exposure" option, I'd use it, but there
isn't one.

(*) "Used to" because I just realized that it doesn't anymore. Now, all
it does is, well, change the gamma table. I don't know why the behavior
changed and I'm currently beating my head against the wall. Previously,
it was entirely obvious that the actual exposure had changed, just by
listening to the scanner noise and looking at the time needed for a
scan.

You may perceive the image getting "brighter" after you change gamma
but all that is done *after* the scan. The exposure is only relevant
while you scan, anything that happens afterwards is simply image
editing.
So even if you change the contrast of the image by setting
Levels or using Curves to map 255 to 128, all that does is process the
existing data *after* the scan.

Yeah, except that my scanner software is (or at least, was) so smart
that it adjusted its exposure automatically depending on the curves and
levels I specified to it *before* the scan.
I know it's confusing. I know, trust me. :-\

If you need to do that, then Levels and Curves are the wrong way to go
about it. You should boost exposure until the histogram reaches the
right edge instead of stretching existing data of an underexposed
scan.

I don't have an Epson so I'll have to yield to someone who does with
practical advice as it appears your problems are not with negative
conversion but simply how to operate the software.

Yep. But I do also have some doubts about negative conversion, although
there's probably little I can do until I can get the software to work
as I want it to...

But, follow me for a minute. *If* I could adjust exposure (on a
per-channel basis), this is what I would do:
1) Scan an un-exposed part of the negative, to extract film mask color
(say, R100 G50 B30)
2) Pre-scan the frame to extract the frame's maximum color value (say,
R80 G30 B25)
3) Expose so that mask color registers as (clipped) white, and the
brighest color in the frame barely reaches 255 (say, maximum color
value = R254 G95 B79, which is the previously mentioned valued
multiplied by 3.18, so that red touches value 255)

What do you think of this procedure?

Alternatively, I could certainly just expose so that the brightest
pixel in the frame is R255 G255 B255. This way, however, I would
"automatically" adjust the frame's white point, but are we sure this is
a good idea for *every* picture? It is probably for most, but can't we
have pictures containing no white, where this white-balancing wouldn't
do good?

But on the other hand, I'm not really convinced that my procedure
treats the orange mask correctly. Does it? I've often read about
"subtracting" the orange mask, but boosting the green/blue exposure, or
stretching the histogram, doesn't really "subtract" it... it just makes
it white.

And then, *is* boosting exposure equivalent to stretching the
histogram? I think it is, at least when working at gamma=1.0 (meaning
no in-scanner gamma correction), which I'm currently doing.


by LjL
(e-mail address removed)

 

[ljlbox]

Ken Weitzel ha scritto:

[snip]

[snip]

Hi...

Epson scan software does indeed have the capability of
adjusting exposure.

It also has another tab that will let you look at the
histogram.

The downside is that as you adjust exposure the preview
image updates nicely, but the histogram doesn't.

Perhaps if any Epson folks are reading this they might
add it to the wish list

Sorry, I should have mentioned that I'm not using the standard Epson
Scan software, but the free SANE driver for Linux developed by Epson
Avasys (formerly Epson Kowa).

Anyway, can you point me to the exposure adjustment control on Epson
Scan please? I can't try it right now, since to use Epson Scan I must
disconnect my scanner from the computer running Linux and connect it to
the one running Windows... but the last time I used it, I didn't find
it entirely clear how to adjust exposure.

I remember there is an "Auto Exposure" button on the left (in
"Professional Mode" - the button has an icon, you have to wait for the
tooltip to see the "Auto Exposure" text); then there are various
controls for histogram levels and curves, but I didn't find a specific
control for exposure -- even though I suspect the levels control really
also adjusts exposure, as it does (did) with the Linux driver.

by LjL
(e-mail address removed)

 

[ljlbox]

(e-mail address removed) ha scritto:

[snip]

Now, the funny thing is that if I specify a gamma table like that, the
scanner also used to (*) change its exposure time accordingly; so, with
that specific table, it would make the exposure about twice as longer
than usual.

Of course, if there was an "--exposure" option, I'd use it, but there
isn't one.

(*) "Used to" because I just realized that it doesn't anymore. Now, all
it does is, well, change the gamma table. I don't know why the behavior
changed and I'm currently beating my head against the wall. Previously,
it was entirely obvious that the actual exposure had changed, just by
listening to the scanner noise and looking at the time needed for a
scan.

This is solved. I had forgotten (or never noticed) that this
auto-exposure-on-levels-change only works at resolutions of 1200dpi or
higher.
You know, just to add some more confusion... (well, they probably
thought that if I want to scan at a low resolution then I'm probably
not willing to wait longer for a longer exposure scan; still, wouldn't
a stupid "--exposure" option have been simpler for both them and me?!)

I'll add that I've observed the following pattern:
1 inch x 1 inch scan, 2400dpi
levels 0..255 -> 5 seconds scan time
levels 0..128 -> 10 seconds scan time
levels 0..64 -> 15 seconds scan time
levels 0..32 -> 15 seconds scan time

Obviously, the 0..32 test should have taken more than 15 seconds if the
exposure had actually changed compared to the test with 0..64.

Also, the 0..64 test should have taken about 20 seconds, not 15,
correct?
I suppose that the maximum possible exposure is about 3 times the
normal exposure -- which IIRC is the exposure needed for the green or
the blue channel (don't remember which) to compensate for the orange
mask on films.

[snip]

by LjL
(e-mail address removed)

 

[Ken Weitzel]

(e-mail address removed) ha scritto:

[snip]

Now, the funny thing is that if I specify a gamma table like that, the
scanner also used to (*) change its exposure time accordingly; so, with
that specific table, it would make the exposure about twice as longer
than usual.

Of course, if there was an "--exposure" option, I'd use it, but there
isn't one.

(*) "Used to" because I just realized that it doesn't anymore. Now, all
it does is, well, change the gamma table. I don't know why the behavior
changed and I'm currently beating my head against the wall. Previously,
it was entirely obvious that the actual exposure had changed, just by
listening to the scanner noise and looking at the time needed for a
scan.

This is solved. I had forgotten (or never noticed) that this
auto-exposure-on-levels-change only works at resolutions of 1200dpi or
higher.
You know, just to add some more confusion... (well, they probably
thought that if I want to scan at a low resolution then I'm probably
not willing to wait longer for a longer exposure scan; still, wouldn't
a stupid "--exposure" option have been simpler for both them and me?!)

I'll add that I've observed the following pattern:
1 inch x 1 inch scan, 2400dpi
levels 0..255 -> 5 seconds scan time
levels 0..128 -> 10 seconds scan time
levels 0..64 -> 15 seconds scan time
levels 0..32 -> 15 seconds scan time

Obviously, the 0..32 test should have taken more than 15 seconds if the
exposure had actually changed compared to the test with 0..64.

Also, the 0..64 test should have taken about 20 seconds, not 15,
correct?
I suppose that the maximum possible exposure is about 3 times the
normal exposure -- which IIRC is the exposure needed for the green or
the blue channel (don't remember which) to compensate for the orange
mask on films.

Hi...

I wanna put my order in right now for the first one to
come out with a (reasonably affordable) variable intensity
light source Really, really, bright one

Ken

 

[ljlbox]

Ken Weitzel ha scritto:

[snip]

Hi...

I wanna put my order in right now for the first one to
come out with a (reasonably affordable) variable intensity
light source Really, really, bright one

That would certainly be a nice fix for lack of (enough) exposure
control... and I suppose it would actually be much better, since AFAIK
noise increases with exposure, but not with light intensity.

By the way, I was wondering, what is the reason why increased CCD
exposure gives less noise? Isn't CCD noise linear and additive, i.e.,
for every pixel, given an exposure time t, the noise will be a random
number between 0 and t*k, k being a scanner-dependent constant?

If so, with increased exposure, noise should increase just as much as
image data. What am I forgetting?


Also, I would like to order a (not necessarily variable) infrared light
source, since my attempts at emulating ICE without an infrared channel
haven't been *too* satisfying...


by LjL
(e-mail address removed)

 

[Bart van der Wolf]

SNIP

By the way, I was wondering, what is the reason why increased
CCD exposure gives less noise? Isn't CCD noise linear and
additive, i.e., for every pixel, given an exposure time t, the noise
will be a random number between 0 and t*k, k being a scanner-
dependent constant?

With fixed exposure, Photon shot noise has a Poisson distribution and
thus varies as a function of the level of exposure. The average noise
equals the square root of exposure level.

If so, with increased exposure, noise should increase just as much
as
image data. What am I forgetting?

See above. With varying exposure, longer exposure will increase the
dark noise level. There will be spontaneous noise building up over
time (as a function of temperature), and it is totally independent of
any signal reaching the CCD (hence dark noise).

There are also other sources of noise in the signal as finally
quantized, but dark noise *and* Photon shot noise both deteriorate
image quality of relatively dense/non-transparent film areas.

Bart

 

[ljlbox]

Bart van der Wolf ha scritto:

SNIP

Sorry, but I didn't understand a word you said :=)
Let me see how much I've grasped...

With fixed exposure, Photon shot noise has a Poisson distribution and
thus varies as a function of the level of exposure. The average noise
equals the square root of exposure level.

Here "level of exposure" refers to the *film's* level of exposure.
Correct?
And this noises increases when the picuture's "brightness" (exposure)
increases -- but not as much (just the square root), so that
effectively, if you vary the exposure *of the CCD*, the signal-to-noise
ratio improves. Correct?

See above. With varying exposure, longer exposure will increase the
dark noise level. There will be spontaneous noise building up over
time (as a function of temperature), and it is totally independent of
any signal reaching the CCD (hence dark noise).

Now you're talking about a different kind of noise, which increases
linearly with CCD exposure time.
This means that, given a certain CCD exposure, every pixel will be
"off" by a random amount between zero and n, where n is a linear
function of exposure time. Correct?

There are also other sources of noise in the signal as finally
quantized, but dark noise *and* Photon shot noise both deteriorate
image quality of relatively dense/non-transparent film areas.

However, photon shot noise increases less than dark noise with longer
CCD exposures; the net result is that, by exposing the CCD longer, you
don't gain anything with respect to dark noise, but you get less photon
shot noise (actually, you get *more* photon shot noise, but you also
get *much* more signal - precisely, with exposure time t, you get
k*Sqr(t) noise, but k*t signal).
Correct?

by LjL
(e-mail address removed)

 

[Don]

By the way, I'm using the SANE libraries from Epson Avasys (was Epson
Kowa), not the Windows scanning software. The Windows software (Epson
Scan) definitely does invert negatives.

Ah, that's different, then. I don't know what SANE does but as far as
I recall it's GNU so you should be able to inspect the sources.

It would appear SANE either mislabeled this option or does something
unconventional. The sole point of a negative setting in scanner
software is to invert the image and remove the orange mask.

Doesn't *seem* to alter contrast, but I really can't be sure :-\ Of
course, even just changing the exposure may cause clipping, if the film
mask is brighter than the software thinks it is...

You can easily check this by doing two scans and then comparing the
histograms afterwards in your image editing software. I guess you're
using GIMP, right?

I'll explain. I used the term "gamma" simply because that's the name of
the option, i.e. "--red-gamma-table", "--green-gamma-table" and
"--blue-gamma-table".
However, what these options let me specify is a complete map of color
values, so that, for example,

0,2,4,6,8,10,12,14,...,254,255,255,...,255

gives me an image with twice the brightness (if brightness is the
correct term), without really changing *gamma*.

OK, that's a bit clearer, now. The values you're referring to are
indeed gamma values. They're so-called "look up tables" (LUT) and when
they're applied the following happens (I'll use your examples).

Any color with value 0 is replaced with value 0.
Any color with value 1 is replaced with value 2.
Any color with value 2 is replaced with value 4.
....
Any color with value 253 is replaced with value 255.
Any color with value 254 is replaced with value 255.
Any color with value 255 is replaced with value 255.

Now, if you look at these tables you'll notice that regardless of the
gamma setting *all* of them start with a 0 and end with a 255. The
only thing that changes are the values in between. That's why it's
impossible to map values from 0 - 255 to 0 - 128 using gamma.

Now, the funny thing is that if I specify a gamma table like that, the
scanner also used to (*) change its exposure time accordingly; so, with
that specific table, it would make the exposure about twice as longer
than usual.

I suspect you may be mistaking the application of gamma for exposure.
It's probably because, superficially, both may appear to "brighten"
the image but the differences are major. Again, looking at the
histogram should throw more light.

Also, when you change exposure the scan takes longer (assuming all
other options are kept the same). That's another test you may try.
Changing gamma does not change the time it takes to scan.

Of course, if there was an "--exposure" option, I'd use it, but there
isn't one.

This is a wild guess but maybe SANE has auto-exposure and doesn't even
let you change exposure manually? Not very likely, but possible.

Again, a histogram is the way to get to the bottom of it. Scan raw a
few images (some underexposed, some overexposed) without making any
editing changes, and then check the right edge of the histogram. If
they all "touch" the right edge it suggests SANE does auto-exposure.

Later: A lateral thought... Try searching SANE docs for things like
"gain". Maybe SANE just uses a different term for exposure?

(*) "Used to" because I just realized that it doesn't anymore. Now, all
it does is, well, change the gamma table. I don't know why the behavior
changed and I'm currently beating my head against the wall. Previously,
it was entirely obvious that the actual exposure had changed, just by
listening to the scanner noise and looking at the time needed for a
scan.

That's indeed the way to do it!

Have you tried re-installing?

Yeah, except that my scanner software is (or at least, was) so smart
that it adjusted its exposure automatically depending on the curves and
levels I specified to it *before* the scan.
I know it's confusing. I know, trust me. :-\

Oh, I believe you!

The thing is Levels and Curves have nothing to do with exposure. Even
though you *specify* these settings before the scan they do not get
*applied* until the scanner has obtained (i.e. scanned) the image.

Also, Curves are non-linear while exposure is linear. Therefore, one
can *not* emulate curve settings using exposure! It's just impossible.

But, follow me for a minute. *If* I could adjust exposure (on a
per-channel basis), this is what I would do:
1) Scan an un-exposed part of the negative, to extract film mask color
(say, R100 G50 B30)
2) Pre-scan the frame to extract the frame's maximum color value (say,
R80 G30 B25)
3) Expose so that mask color registers as (clipped) white, and the
brighest color in the frame barely reaches 255 (say, maximum color
value = R254 G95 B79, which is the previously mentioned valued
multiplied by 3.18, so that red touches value 255)

That's basically how I got into all this 3 years ago only I tried to
correct the notorious Kodachrome cast! ;o) And I'm still at it!

What do you think of this procedure?

The trouble with optics is that it's a maze of ambushes and "hidden"
pitfalls. For example, see below for the "progression" of color values
as you change exposure.

In theory, you can remove a cast using exposure as long as the cast is
linear. That opens up a whole new can of worms such as film response
curves, etc but with that caveat:

Here's a formula you may find useful. This may be simplified a lot but
I'm including everything to see how it's arrived at.

OP = ((OldPix / 255) ^ Gamma) * 255
NP = ((NewPix / 255) ^ Gamma) * 255

ev = Log(1 / (OP / NP)) / Log(2)

where:
OldPix = old pixel value before exposure is applied
NewPix = new pixel value after exposure is applied
ev = standard exposure value

NOTE: This assumes 8-bit depth. For 16-bit replace 255 with 65535.

Alternatively, I could certainly just expose so that the brightest
pixel in the frame is R255 G255 B255. This way, however, I would
"automatically" adjust the frame's white point, but are we sure this is
a good idea for *every* picture? It is probably for most, but can't we
have pictures containing no white, where this white-balancing wouldn't
do good?

But on the other hand, I'm not really convinced that my procedure
treats the orange mask correctly. Does it? I've often read about
"subtracting" the orange mask, but boosting the green/blue exposure, or
stretching the histogram, doesn't really "subtract" it... it just makes
it white.

And then, *is* boosting exposure equivalent to stretching the
histogram?

Yes, but do note that different values will "move" at different
*absolute* rates even though the *relative* values are the same!

For example, assuming gamma=1.0 and doubling of exposure:

A color with value 10 will become 20 and a color with value of 100
will become 200. So far so good. But if you locate these values on the
histogram you'll *perceive* 10 as barely "moving" to reach 20, while
100 will make a huge jump to get to 200. That's because 10/255 is not
the same as 100/255!

That's why it's so easy to blow the highlights but the shadows barely
lighten.

Don.

 

[Don]

However, photon shot noise increases less than dark noise with longer
CCD exposures; the net result is that, by exposing the CCD longer, you
don't gain anything with respect to dark noise, but you get less photon
shot noise (actually, you get *more* photon shot noise, but you also
get *much* more signal - precisely, with exposure time t, you get
k*Sqr(t) noise, but k*t signal).
Correct?

Yes, as far as I understand it, and why I don't think much of
single-pass multi-scanning. It just "averages" the noise, rather than
eliminating it. The same (or *visually* very similar effect) can be
achieved much more quickly by simply applying a small amount of
Gaussian Blur to the noisy part of the image (below a certain
threshold).

That's why I favor twin-scanning. By boosting exposure the noise is
actually "drowned" in the sense that it's overwhelmed by a much
stronger signal.

Don.

 

[Bart van der Wolf]

Bart van der Wolf ha scritto: SNIP
Sorry, but I didn't understand a word you said :=)
Let me see how much I've grasped...


Here "level of exposure" refers to the *film's* level of exposure.
Correct?

No, film exposure is a given for the scanner.
The scanner's sensors can have a variable integration time, to
compensate for film density.

And this noises increases when the picuture's "brightness"
(exposure) increases -- but not as much (just the square
root), so that effectively, if you vary the exposure *of the
CCD*, the signal-to-noise ratio improves. Correct?

To put it in my own words, the darker parts of the film will cause
lower CCD exposure levels, but those levels cannot be increased
indefinitely by longer integration/exposure times because that would
cause clipping of the most transparent film areas. So some CCD sensors
will receive a high level of exposure, and some a very low level of
exposure.

The lower levels will be relatively more noisy due to the Poisson
distribution of photons (e.g. Sqrt(100) = 10 which gives a 100:10 or a
10:1 signal to noise, and say Sqrt(10000) = 100 which gives a
10000:100 or a 100:1 signal to noise which is 10 times better).
Therefore it will pay-off to increase the exposure level as high as
possible without clipping the most transparent film areas.

Now you're talking about a different kind of noise, which
increases linearly with CCD exposure time.

Correct, it is one of the contributing noise sources, one which
benefits from the shortest possible integration/exposure time.

This means that, given a certain CCD exposure, every pixel
will be "off" by a random amount between zero and n, where n
is a linear function of exposure time. Correct?
Yep.


However, photon shot noise increases less than dark noise with
longer
CCD exposures; the net result is that, by exposing the CCD longer,
you
don't gain anything with respect to dark noise, but you get less
photon
shot noise (actually, you get *more* photon shot noise, but you also
get *much* more signal - precisely, with exposure time t, you get
k*Sqr(t) noise, but k*t signal).
Correct?

You've got it! The net S/N gain from optimizing (maximizing without
clipping) the signal level is larger than the loss from higher dark
noise.

In addition, when multi-sampling the film, the random noise decreases
as the square root of the number of samples because noise cancels out,
while the signal is just the average so S/N improves again. This does
require perfect alignment of the samples (e.g. with a single-pass
multi-scanning enabled scanner or, much more error prone, by averaging
multiple scan-passes).

Bart

 

[ljlbox]

Don ha scritto:

Ah, that's different, then. I don't know what SANE does but as far as
I recall it's GNU so you should be able to inspect the sources.

Not all of them, since part of the Epson Avasys driver is given in
binary format only.
What I can inspect, however, I'm inspecting right now.
But what's funniest is that there is *no* mention of exposure (or gain)
in the whole source.

I think the exposure logic is actually in the firmware, and it reacts
automatically to the gamma lookup tables that are passed to it (see
below for why I still think this is the case).

It would appear SANE either mislabeled this option or does something
unconventional. The sole point of a negative setting in scanner
software is to invert the image and remove the orange mask.

What option are you talking about, "--film-type Negative" or the
"--color-gamma-table" set of options?

"--film-type Negative" does just about what it's supposed to do: it
adjusts the exposure in order to remove the orange mask, and... well,
nothing else that I can see, since it doesn't even invert.

About "--***-gamma-table", as I said, I don't think it's the option
itself that does something unconventional, but rather it is the scanner
firmware that thinks it's smart enough to change the exposure based on
the gamma table it's passed.

You can easily check this by doing two scans and then comparing the
histograms afterwards in your image editing software. I guess you're
using GIMP, right?

Mostly Photoshop (the computer with Linux is my "server", the one I use
normally has Windows).

The problem is that, to meaningfully compare the histograms, I need to
be sure the the scan taken with "--film-type Negative" and the one
taken with "--film-type Positive" are made with exactly the same RGB
exposures -- and as you now know, this isn't an easy thing do with my
scanner

But really, contrast doesn't seem to be an issue. As I said, "I can't
be sure", but for all I can tell, "--film-type Negative" just changes
the green and blue exposures.

[snip]

I'll explain. I used the term "gamma" simply because that's the name of
the option, i.e. "--red-gamma-table", "--green-gamma-table" and
"--blue-gamma-table".
However, what these options let me specify is a complete map of color
values, so that, for example,

0,2,4,6,8,10,12,14,...,254,255,255,...,255

gives me an image with twice the brightness (if brightness is the
correct term), without really changing *gamma*.

OK, that's a bit clearer, now. The values you're referring to are
indeed gamma values. They're so-called "look up tables" (LUT) and when
they're applied the following happens (I'll use your examples).

Any color with value 0 is replaced with value 0.
Any color with value 1 is replaced with value 2.
Any color with value 2 is replaced with value 4.
...
Any color with value 253 is replaced with value 255.
Any color with value 254 is replaced with value 255.
Any color with value 255 is replaced with value 255.

Now, if you look at these tables you'll notice that regardless of the
gamma setting *all* of them start with a 0 and end with a 255. The
only thing that changes are the values in between. That's why it's
impossible to map values from 0 - 255 to 0 - 128 using gamma.

Why do you say it's impossible? I can set up that table to anything I
like, for example

scanimage [...] --red-gamma-table
0,32,64,96,128,160,192,224,255,255,[...],255

Sure, it still ends with 255, but I don't see any problem with that.
It's *not* a gamma table, it's a linear table. It's just like moving
the whitepoint, for all I can see.

I suspect you may be mistaking the application of gamma for exposure.
It's probably because, superficially, both may appear to "brighten"
the image but the differences are major. Again, looking at the
histogram should throw more light.

Believe me, I'm not mistaking gamma for exposure. The scanner takes
longer to scan using the table above (though only when scanning above
600 dpi, as I now realized; when scanning at 600 dpi or less, it
apparently keeps the exposure fixed and just lets the software processe
the image according to the table given).

Of course, if I gave it a table like
0,255,0,255,0,255,[...]
it would have to let the software apply this table; same for a "real"
gamma table which actually changes the gamma; but the point is that,
when it is given a table that puts the whitepoint below 255, it uses
exposure to achieve that instead of color-mapping.

Also, when you change exposure the scan takes longer (assuming all
other options are kept the same). That's another test you may try.
Changing gamma does not change the time it takes to scan.

No, changing the gamma doesn't, but changing the gamma table so that
the last n values are filled with 255 apparently does.
I think this is really an important information for Epson owners,
although more careful scrutiny of what the scanner is actually doing is
needed, in order to understand what side-effects there are, if any.
(Actually, the fact that changing the gamma tables in a specific way
changes the exposures is itself one of the biggest side-effects I have
ever seen..)

I've started measuring the times taken to scan with various different
gamma-tables (all linear, but with the whitepoint set at different
values). For now, all I can say is that the time *does* change, but
I'll post numeric results when I've got enough of them.

This is a wild guess but maybe SANE has auto-exposure and doesn't even
let you change exposure manually? Not very likely, but possible.

Nope. SANE itself leaves it to the individual driver to decide which
options to offer (there aren't even fixed option names, for example
"--film-type Negative" could be called "--mary-had-a-little Lamb" with
another scanner driver -- though there are standards, fear not).

The specific driver of my scanner certainly doesn't allow to change the
exposure explicitely.

Again, a histogram is the way to get to the bottom of it. Scan raw a
few images (some underexposed, some overexposed) without making any
editing changes, and then check the right edge of the histogram. If
they all "touch" the right edge it suggests SANE does auto-exposure.

And it doesn't.
It doesn't even have a way to, since it has no concept of a "preview"
(a preview is simply a scan at a low resolution, which SANE doesn't
perform unless I tell it to).

The Windows software (Epson Scan), on the other hand, does something
similar to what you say: it forces the user to take a preview scan, and
then automatically computes levels and all that based on the preview.
*And* it computes exposure, although, as I said, I have a feeling that
even Epson Scan doesn't *directly* decide exposure, but just sets a
white point by sending appropriate lookup tables to the scanner
firmware.

Later: A lateral thought... Try searching SANE docs for things like
"gain". Maybe SANE just uses a different term for exposure?

Tried. Nope. :-\

That's indeed the way to do it!

Have you tried re-installing?

As posted in another article, I have solved this one: exposure gets
changed depending on the lookup tables passed *only* when scanning at
1200 dpi or higher.

However, I'm still very uncertain on the details: for example, if I
scan as positive using the whitepoint that should lead to orange-mask
removal, I do *not* get the same exposure times I get when scanning
directly as negative -- I just get the normal exposure time (let's call
it 1x).

However, if I set the whitepoint even lower, then the exposure does
change, up to about 3x (based on manual timing).

In other words, it's all a big mess.

[snip]

Also, Curves are non-linear while exposure is linear. Therefore, one
can *not* emulate curve settings using exposure! It's just impossible.

Well, a "linear curve" *can* be emulated using exposure
But yes, I realize that anything that is not linear can't be emulated
with (just) exposure.

That's basically how I got into all this 3 years ago only I tried to
correct the notorious Kodachrome cast! ;o) And I'm still at it!

Encouraging ;-)

The trouble with optics is that it's a maze of ambushes and "hidden"
pitfalls.

Yeah, I've been realizing this. At just about every time I take a scan.

For example, see below for the "progression" of color values
as you change exposure.

In theory, you can remove a cast using exposure as long as the cast is
linear. That opens up a whole new can of worms such as film response
curves, etc but with that caveat:

But is the cast given by the orange mask linear? That's the issue that
concerns me most at the moment.
Obviously I'll also have to deal with film characteristic curves and
all that, but I think that's better left for when I have resolved the
more fundamental issues I have posted about, do you agree?
(But I wouldn't mind if you could point me to a site that gives good
curves for the "average" film given an ISO rating, or even various
brands, in *numerical* format, not just drawings...)

Here's a formula you may find useful. This may be simplified a lot but
I'm including everything to see how it's arrived at.

OP = ((OldPix / 255) ^ Gamma) * 255
NP = ((NewPix / 255) ^ Gamma) * 255

ev = Log(1 / (OP / NP)) / Log(2)

where:
OldPix = old pixel value before exposure is applied
NewPix = new pixel value after exposure is applied
ev = standard exposure value

Thanks. So at gamma=1.0, ev = Log(1 / (OldPix / NewPix)) / Log(2) ?

I admit to not knowing what a "standard exposure value" actually is.
Just tell me, is it linear to the exposure time?

Now, you know that I can't give my scanner the ev value directly: I
must directly map every possible OldPix value to every possible NewPix
value. Am I right in assuming that, using your formula, I could set an
ev value by solving
NewPix = 2^ev * OldPix (with gamma=1.0)
or
NewPix = 2^(ev/Gamma) * (OldPix^Gamma)^(1/Gamma) (with
gamma<>1.0)
?

(Just ignore the second formula if you prefer, it's the result my
calculator gives, I can't solve that kind of things =)

[snip]

For example, assuming gamma=1.0 and doubling of exposure:

A color with value 10 will become 20 and a color with value of 100
will become 200. So far so good. But if you locate these values on the
histogram you'll *perceive* 10 as barely "moving" to reach 20, while
100 will make a huge jump to get to 200. That's because 10/255 is not
the same as 100/255!

That's why it's so easy to blow the highlights but the shadows barely
lighten.

Yes, I understand this.


by LjL
(e-mail address removed)

 

[ljlbox]

Don ha scritto:

Yes, as far as I understand it, and why I don't think much of
single-pass multi-scanning. It just "averages" the noise, rather than
eliminating it. The same (or *visually* very similar effect) can be
achieved much more quickly by simply applying a small amount of
Gaussian Blur to the noisy part of the image (below a certain
threshold).

That's why I favor twin-scanning. By boosting exposure the noise is
actually "drowned" in the sense that it's overwhelmed by a much
stronger signal.

Wait a moment... isn't 2x exposure perfectly equivalent to 2x
single-pass multi-scanning (except that long exposure clips whites
while multi-scanning doesn't, and of course the histogram stretches
with long exposure but not with multi-scanning)?

If I'm right, then your (1x exposure + 2x exposure) is perfectly
equivalent to (1x exposure + 1x exposure + 1x exposure). Perhaps the
latter needs more bit depth to be as effective as the former, but I
don't think we've reached the point where 16 bits per channel are a
bottleneck.

I don't have the knowledge to be *certain* that two short exposure are
equivalent to (or better than) one longer exposure, but just think of
it this way: when you take two 1x exposures, you're basically doing a
2x exposure, except that you split it into two "sessions" -- which also
has the advantage of not overflowing the sensors (i.e. not clipping
whites).

Now, with *multi-pass* multi-scanning it's different, because there's
the alignment problems, but we know that.

Besides, don't you think multi-pass multi-scanning should theoretically
be *more effective* than single-pass multi-scanning, as long as we can
guarantee perfect alignment?
I say this because single-pass can only improve the S/N ratio, while
multi-pass - thanks to the different alignments of the two or more
scans one gets - should actually allow to increase *resolution*!
Shouldn't it?

That's exactly what ALE claims to do, by the way. And it's also what me
me relate multi-scanning to "staggered CCDs", as I did in the other
thread.


by LjL
(e-mail address removed)

 

[Don]

Not all of them, since part of the Epson Avasys driver is given in
binary format only.

As someone who runs Linux ocassionally, and just out of curiousity,
did you have to download this driver separately? The reason I ask is
because some distributions refuse to include packages which have a
component which is not GNU.

What I can inspect, however, I'm inspecting right now.
But what's funniest is that there is *no* mention of exposure (or gain)
in the whole source.

That's very strange, indeed!

I think the exposure logic is actually in the firmware, and it reacts
automatically to the gamma lookup tables that are passed to it (see
below for why I still think this is the case).

I don't think that can be done. Gamma is not related to ev, so there's
no way for software/firmware to extrapolate the exposure from gamma.
The two are just not related.

What option are you talking about, "--film-type Negative" or the
"--color-gamma-table" set of options?

I'm not that familiar with SANE so I don't know, but "--film-type
Negative" on the face if it *looks* like the correct option.

"--film-type Negative" does just about what it's supposed to do: it
adjusts the exposure in order to remove the orange mask, and... well,
nothing else that I can see, since it doesn't even invert.

That, I don't understand. It appears to be something peculiar to SANE.
Negative mode usually means both, removing the orange mask and
inverting the image.

About "--***-gamma-table", as I said, I don't think it's the option
itself that does something unconventional, but rather it is the scanner
firmware that thinks it's smart enough to change the exposure based on
the gamma table it's passed.

As I say, that's not really possible because exposure is linear and
gamma is non-linear. The two just don't relate.

Mostly Photoshop (the computer with Linux is my "server", the one I use
normally has Windows).

OK, then it should be easy to check that with Photoshop's histogram.

The problem is that, to meaningfully compare the histograms, I need to
be sure the the scan taken with "--film-type Negative" and the one
taken with "--film-type Positive" are made with exactly the same RGB
exposures -- and as you now know, this isn't an easy thing do with my
scanner

That's OK. The contrast is the range between the brightest and darkest
points. Even if the exposure is different you should be able to
extrappolate the contrast by looking at the histograms. Although 8-bit
may not have enough accuracy.

But I think, that's a distraction. The key point is to find out how is
the exposure determined and then set.

Why do you say it's impossible? I can set up that table to anything I
like, for example

Yes, but then it's not gamma anymore, but just a plain lookup table.

If you do that and set an arbitrary lookup table it's the same as
using Curves in Photoshop. Now, you can set gamma using Curves, but
you can also do much more. Gamma, however, is a very specific term.

scanimage [...] --red-gamma-table
0,32,64,96,128,160,192,224,255,255,[...],255

Sure, it still ends with 255, but I don't see any problem with that.
It's *not* a gamma table, it's a linear table. It's just like moving
the whitepoint, for all I can see.

It appears SANE uses the term gamma unconventionally. What SANE calls
"gamma table" is just a plain lookup table. It can be used for gamma,
of course, but it can also be used for anything else as well.

However, "gamma" has a very specific definition. It's not just *any*
lookup table. Gamma curves are created using a very specific formula:

output = input ^ gamma

Another important thing to understand is that using a lookup table is
*image editing". That's post-processing and has nothing to do with
exposure. Exposure is something which happes as the CCD array passes
accross the image. Anything one does to the image after that, is image
editing or post-processing.

Believe me, I'm not mistaking gamma for exposure. The scanner takes
longer to scan using the table above (though only when scanning above
600 dpi, as I now realized; when scanning at 600 dpi or less, it
apparently keeps the exposure fixed and just lets the software processe
the image according to the table given).

Although the time is one indicator it can also be very ambiguous.
There are many reasons why a scan may take longer.

For example, your bus throughput. If scans > 600 dpi take longer it's
an indicator that your bus is choked with data.

Does the scanner "stop-and-go" or is the scan performed in a single,
uninterrupted flow?

The Windows software (Epson Scan), on the other hand, does something
similar to what you say: it forces the user to take a preview scan, and
then automatically computes levels and all that based on the preview.
*And* it computes exposure, although, as I said, I have a feeling that
even Epson Scan doesn't *directly* decide exposure, but just sets a
white point by sending appropriate lookup tables to the scanner
firmware.

That's very unlikely becuse it would cause banding. Stretching an
underexposed scan (which is what you describe) is very unlikely any
software would do because it would produce very poor results.

Anyway, that can also be tested but you would have to use 16-bit depth
and some heavy math.

As posted in another article, I have solved this one: exposure gets
changed depending on the lookup tables passed *only* when scanning at
1200 dpi or higher.

OK, that looks more and more to be a bus question (or some other
throughput problem) and not an exposure question. You're just simply
getting too much data and the resons it takes longer is because the
system has to catch up.

However, I'm still very uncertain on the details: for example, if I
scan as positive using the whitepoint that should lead to orange-mask
removal, I do *not* get the same exposure times I get when scanning
directly as negative -- I just get the normal exposure time (let's call
it 1x).

However, if I set the whitepoint even lower, then the exposure does
change, up to about 3x (based on manual timing).

In other words, it's all a big mess.

Yes, so we have to "divide and conquer" which means back to basics and
not make any rash assumptions. Instead confirm each assertion before
going further. Otherwise we just compound the problem by basing
conclusions on incorrect data.

The first test I would do is the above "stop-and-go".

But is the cast given by the orange mask linear? That's the issue that
concerns me most at the moment.

Yes and no... Nominally it is but the film response curve is not
(strictly speaking). So while you can get acceptable results,
theoretically, it's two different things.

Obviously I'll also have to deal with film characteristic curves and
all that, but I think that's better left for when I have resolved the
more fundamental issues I have posted about, do you agree?

Yes and no... ;o) It depends on your requirements and your
environment. But that would be another digression...

(But I wouldn't mind if you could point me to a site that gives good
curves for the "average" film given an ISO rating, or even various
brands, in *numerical* format, not just drawings...)

I've only been "obsessed" with Kodachromes and Kodak site does give
characteric curves for all of their film. However, it's all as graphs
rather than actual values.

Try googling for it and if that fails, get this:

http://www.color.org/membersonly/profileinspector.html

Next, locate the profile for your film and the above program will dump
the characteristic curves in that profile as data! However, it's raw
data and will need some massaging before it can be used. The Profile
Inspector does dump all the other data you need to do that (white
point, etc).

Thanks. So at gamma=1.0, ev = Log(1 / (OldPix / NewPix)) / Log(2) ?
Yes.

I admit to not knowing what a "standard exposure value" actually is.
Just tell me, is it linear to the exposure time?

The official definition involves 100 ASA film and I don't know it off
hand. However, the actual technical definition is not really
important. What is important is what it means: Each step doubles the
amount. So, ev=2 is twice the exposure of ev=1. And ev=3 is four times
the exposure or ev=1, etc.

However, this only applies in linear gamma (gamma = 1.0)! That's why I
included gamma in the above formula, just in case.

Now, you know that I can't give my scanner the ev value directly: I
must directly map every possible OldPix value to every possible NewPix
value. Am I right in assuming that, using your formula, I could set an
ev value by solving
NewPix = 2^ev * OldPix (with gamma=1.0)
or
NewPix = 2^(ev/Gamma) * (OldPix^Gamma)^(1/Gamma) (with
gamma<>1.0)
?

If you're working in linear gamma it's real simple. You just
double/halve the value for each ev step. If you plot these values they
will be straight lines.

Don.

 

[Don]

Wait a moment... isn't 2x exposure perfectly equivalent to 2x
single-pass multi-scanning

In theory - i.e. in the sense that both *nominally* increase the
dynamic range by the same amount - but in practice it's night and
day... The single-pass multi-scan image is essentially an average
which appears quite "muddy", while the twin scan actually penetrates
the shadows resolving much more detail.

(except that long exposure clips whites
while multi-scanning doesn't, and of course the histogram stretches
with long exposure but not with multi-scanning)?

That's why the boosted scan resolves more detail. The "width" of the
histogram covering shadows in multi-scanning remains the same. The
boosted scan's histogram is much wider in the corresponding shadow
part which means wider dynamic range.

If I'm right, then your (1x exposure + 2x exposure) is perfectly
equivalent to (1x exposure + 1x exposure + 1x exposure). Perhaps the
latter needs more bit depth to be as effective as the former, but I
don't think we've reached the point where 16 bits per channel are a
bottleneck.

You don't have to only use two exposures, you can use any number of
exposures. As you know, HDR images usually use a whole series of
exposures at 1 stop intervals often 10 or more images.

The only reason I use two is because of the misalignment problem. If
that were not an issue and I could change exposure at each scan line,
for example, I would certainly use many more!

Besides, don't you think multi-pass multi-scanning should theoretically
be *more effective* than single-pass multi-scanning, as long as we can
guarantee perfect alignment?

No, it's the same thing. However, due to other reasons (remember,
optics are messy!) in that case I think it's better to do single-pass
multi-scanning.

It's because even during the short time the scan moves from the start
to the end of the scan, the conditions in the scanner change!! From
temperature to calibration to everything else you can think of... ;o)

So by doing two scans of the same line immediately one after the other
there would be much less difference between them. However, there will
be much more difference between the first line and last line because
the whole procedure takes twice as long and the conditions keep
changing. But, on balance, that's less of a problem than multi-pass
multi-scan because there we have that same start/end difference *plus*
individual scan line difference. At least that's what I think...
Optics are messy... ;o)

Kennedy, where are you when we need you!

I say this because single-pass can only improve the S/N ratio, while
multi-pass - thanks to the different alignments of the two or more
scans one gets - should actually allow to increase *resolution*!
Shouldn't it?

Not resolution, but dynamic range.

That's exactly what ALE claims to do, by the way. And it's also what me
me relate multi-scanning to "staggered CCDs", as I did in the other
thread.

I still have to try ALE but I'm swamped right now... Maybe tomorrow...

Don.

 

[ljlbox]

Don ha scritto:

[snip]

May I scream and write in caps and use some words that are not
considered civil in the context of a newsgroup? Please?
Yesterday night I replied with an awfully lengthy message where I put a
lot of measurements I made while writing the message, and only wrote
down in the message, and GOOGLE DID NOT SAVE THE "#¤/&%#¤ MESSAGE!
Why ain't I getting a newsgroup program? Even Outlook, heck, at least
it saves my messages!

HEAD -> WALL -> BOUNCE -> HEAD -> WALL -> BOUNCE -> etc

But no, really, I'll write it all again, so don't open up that bottle
of Champagne, you won't be spared!

by LjL
(e-mail address removed)

 

[Don]

May I scream and write in caps and use some words that are not
considered civil in the context of a newsgroup? Please?
Yesterday night I replied with an awfully lengthy message where I put a
lot of measurements I made while writing the message, and only wrote
down in the message, and GOOGLE DID NOT SAVE THE "#¤/&%#¤ MESSAGE!

Oh, no! Sorry to hear that. :-(

Why ain't I getting a newsgroup program? Even Outlook, heck, at least
it saves my messages!

Yes, that's why I write all my stuff offline. A good offline reader is
also handy because all the messages are in a searchable data base.

HEAD -> WALL -> BOUNCE -> HEAD -> WALL -> BOUNCE -> etc

I really sympathize! Just mention "Nikon" and "Kodachrome" in the same
breath to me, not only is there screaming but there are scratch marks
on the walls (as I climb them) and lots of pulled hair around my
chair... ;o)

But no, really, I'll write it all again, so don't open up that bottle
of Champagne, you won't be spared!

LOL!

Don.