VueSCAM!

[Kennedy McEwen]

Yes, but I'd like to try some kind of post-processing fix: because for
other aspects, I really like Vuescan. Plus, I can only use Vuescan to
drive my Polaroid, since it's attached to a Linux box.

Kennedy, do you have any suggestion about a procedure (even an
abstract one!) that, working on Vuescan raw files and a "dark scan",
could re-equalize the output from the sensor?

I don't have a Minolta scanner to try this, but one suggestion
(something similar has been made before by someone else on this thread,
but I can't remember who and don't have time to read all the thread
again) is to lock the exposure and create a dark scan with the same
settings as the main image. The dark scan subject should be completely
opaque, perhaps some aluminium foil in one of the apertures of the film
holder. Simply subtracting the dark scan from the main image should
eliminate the lines if both are scanned linearly (gamma=1).

The initial problem with the dark scan is that it will have random noise
as well as the systematic miscalibration noise producing the line
structures. One simple way of reducing the random noise is to average
all of the pixels produced from each CCD cell. There are probably lots
of ways of doing this, but one suggestion because you know exactly what
is going on (assuming you use Photoshop) is to create a custom filter
(Filter/Other/Custom). This permits you to define a 5x5 convolution
matrix - so just make it all zero, put 5 1's in along the centre line in
the direction of the scan axis, set scale to 5 and offset to 0. Now,
each time this filter is applied you will effectively be averaging 5
image pixels to create a new one, which reduces the noise by around a
factor of 2.2 on the first step and progressively less on subsequent
steps. Run the filter on the dark image as many times as you have
patience for (Ctrl-F just repeats the application) and you should get a
random noise free dark scan.

A quicker, but less well defined (in that you have to rely on PS doing
the arithmetic correctly, and who knows what arithmetic precision that
uses internally) is to use the motion blur filter. Set the direction to
the scan axis (either 0 or 90deg) and then the number of pixels to the
maximum of 999. A single application of this filter *should* reduce the
random noise on the dark image by over 30x, which ought to reduce it to
the point where it is insignificant. I would repeat the filter a couple
of times as above just to be sure you get rid of it, since we don't know
exactly how PS calculates that motion blur. I should add a little
caution here because I suspect this might introduce rounding error noise
of the same level as you are trying to remove, so it might not be as
successful as the custom filter approach, but you will have to try it
and see.

Having created an essentially noise free dark scan of the line
structure, using whatever method you settle on, just subtract that scan
from the main scan and you *should* get rid of the single line artefacts
completely - if both scans have been made without any gamma
compensation.

Now the rub - this all requires 16-bit arithmetic, which generally means
you need to be using Photoshop-CS. If you are using PS7 or lower, or
any 8-bit image processing package, then you will be restricted to 8-bit
depth for these operations, which is a problem since the operation is
only accurate if applied in the linear domain. Reducing the image to
8-bit depth before applying gamma will cause posterisation when the
gamma is subsequently applied, so you can't go sown that route without
introducing worse problems than you started with.

One way round this is to work in gamma compensated space so that you
effectively have the full 16-bit range of the scanner compressed into
the 8-bit range with equal perceptual weighting to the levels in the
image. However, applying the dark image subtraction in 8-bits after
gamma is applied will introduce inverse lines in the mid-tones and
highlights, because you will be subtracting too much - again, worse
problems than you started with.

Again, this problem can be overcome if you create a mask based on the
levels of the original image to restrict the dark current subtraction to
only the shadow levels, without applying any correction in the mid tones
and highlights - or even several masks to subtract smaller proportions
(based on the gamma scaling) to those regions. I suspect that only the
one shadow mask would be enough but, without the problem to overcome
myself, that is a guess. Anyway, using masks in this way you should be
able to post process the lines completely out of the Vuescan image even
with 8-bit processing, provided that you make the scans with a gamma
compensation close to perceptual space, ie. around 2.2-2.5.

The above is purely hypothetical, since I can't try the solution out,
but I would be interested to hear if it helps or what problems you
encounter trying it.

The above applies only a dark correction of course and, given the
descriptions of the problem, that ought to be adequate. However it may
require a gain correction as well, using a "response scan". That should
be created from a near white scan, filtered the same way as the dark
scan, and from which the dark scan is subtracted in linear space (or
ignored in gamma space). The response scan can then be divided into the
original scan after the dark scan has been removed, to create the fully
corrected image. Again, this is just applicable in linear scans, and
might require masks to apply in gamma compensated scans if you are
restricted to 8-bit processing.

 

[Fernando]

I don't have a Minolta scanner to try this, but one suggestion

[...]

Thanks Kennedy, useful as always!

I always work in 16bit/channel within PS CS, so no problems about having
adequate color depth.
I can also write some software that performs the filtering using the
ImageMagick graphics libraries.

I think I'll try the Photoshop method first.

Thanks a lot!

Fernando

(if it works well, I could even attach a couple samples "before" and
"after" to Ed Hamrick... who knows!!)

 

[Don]

The lines I have seen correspond to 'always the same CCD cell'. On a
35mm frame viewed in landscape orientation, this is a horizontal line.
When viewed in portrait orientation (as you do, apparently), it is vertical.

Actually I don't view it portrait orientation, the *scanner* does. I
expected you would know this!? But, then again, you are a VueScan
user... ;o)

Seriously though, the scans are always done in portrait mode because
that way the manufacturers can get away with a narrower CCD array
(easier to produce and cheaper). When inserting strips of film there
is little choice, but if the array were wider you could insert the
slides in landscape orientation. However, you can't because the
scanner insists on portrait.

That's why I specifically used the CCD array for orientation (parallel
to it or, perpendicular to it).

Mostly, yes. Sometimes they seem to (re)appear at different locations,
which indicates that some of these lines may be caused by dust particles
inside the scanner obscuring individual cells.

I don't think you can draw such a conclusion. Dust would be likely to
produce totally black lines and they would certainly not be
consistently exactly one pixel wide.

Don.

 

[Don]

I always work in 16bit/channel within PS CS, so no problems about having
adequate color depth.
I can also write some software that performs the filtering using the
ImageMagick graphics libraries.

I don't know if this is applicable (or even if it makes any sense?)
but I would think you'd have to make sure the scanner does not
recalibrate between the dark scan and the regular scan, otherwise the
CCD response will be all reshuffled (?) and you may end up introducing
lines, rather than removing them.

BTW, on a tangent (I'm just curious). Have you tried WINE (or any
other Windows' emulators) and then run the native software on your
Linux box? The odds are it won't work, but...

I'm sure you're also aware of http://sane-project.org/ for attaching
scanners to Linux.

(if it works well, I could even attach a couple samples "before" and
"after" to Ed Hamrick... who knows!!)

That will just make him angry because he's been unable to fix it for
two years and he's a "professional"... ;o)

Don.

 

[Fernando]

I don't know if this is applicable (or even if it makes any sense?)
but I would think you'd have to make sure the scanner does not
recalibrate between the dark scan and the regular scan

I'm sure it doesn't.
The 5400 performs a (firmware-level) self-calibration when first
accessed after power-up, and it saves the result of this
self-calibration in an appropriate area inside its built-in memory.
Then, Vuescan performs another type of calibration when the user asks
for it (it does not perform it automatically, although it warns the
user that it's a necessary step). The result of this software-level
calibration is stored in a binary file (Vuescan.se5, for the 5400).

It's curious to note that Vuescan, for some odd reason, does not
perform software calibration (always possible, while of course
firmware calibration is only available on scanners that have the
necessary built-in routines) on all scanners. For example,
Scanner->Calibrate has no effect on my Polaroid SS120, Epson 2450,
Epson 4180.
I wonder if doing a simple dark-frame calibration could lessen the
streaks on those models, too... it should, in my opinion.
If the dark frame experiment would succeed for the 5400, I'll try it
on the other models, too. Will keep the board informed.

BTW, on a tangent (I'm just curious). Have you tried WINE (or any
other Windows' emulators) and then run the native software on your
Linux box? The odds are it won't work, but...

It does not work (can't find the SCSI-attached scanner),
unfortunately.

I'm sure you're also aware of http://sane-project.org/ for attaching
scanners to Linux.

Yes, but actually it is very feature-limited. The few times I tried it
on my 2450, it gave me just average picture quality (OK for reflective
scans).

Fernando

 

[Don]

I don't have a Minolta scanner to try this, but one suggestion
(something similar has been made before by someone else on this thread,
but I can't remember who and don't have time to read all the thread
again) is to lock the exposure and create a dark scan with the same
settings as the main image.

One thing I don't quite understand is this: I was under the impression
that dark current (at least for consumer type scanners) is actually
automatically corrected in firmware i.e., a user is not even given the
opportunity to get access to a scan with dark current still present.

This seems further reinforced by reading the docs (if I read them
correctly). According to them (at least as far as my LS-50 goes) "Dark
Voltage Data reading" and writing (writing? presumably a user-defined
correction) is not supported - although provisions are made because
the device will report whether it's capable of it. Oddly, this is a
fucntion of the type of adapter attached (SA-21, IA-20, etc)!?

However, there is a command to "Setup Dark Current Correction Data"
which "Performs the dark voltage measurement". Presumably, that's just
another way of saying "the scanner will calibrate itself" because all
that happens internally since the command has no parameters.

The docs seem to use "voltage" and "current" interchangeably - or do
they? I'm (uncomfortably) assuming they do and put it down to language
because the docs are written in "Japanese English" .

Anyway, could Minolta be different in this respect in that it actually
does supply a scan with the dark current still present?

Don.

 

[Don]

I'm sure it doesn't.
The 5400 performs a (firmware-level) self-calibration when first
accessed after power-up, and it saves the result of this
self-calibration in an appropriate area inside its built-in memory.

Of course, I don't know how Minolta works, but I just thought it's
something to keep in mind...

On my Nikon, the same type of calibration is performed at startup.
However, after the scanner has been on for a while it automatically
re-calibrates itself at regular intervals!! That's because the
temperature inside the device changes and CCDs are very sensitive to
this so the regular recalibration is necessary.

Automatic recalibration happens after the media is removed. There is
an explicit button in NikonScan to perform the calibration but the
docs warn that the film must first be removed.

So, what I meant above is the following scenario: You do a regular
scan, remove the film and then - before you get a chance to put in the
dark aluminium foil slide - the scanner recalibrates itself! In that
case - it seems to me - you would have to do the regular scan again
because you need both scans to be performed using the same internal
calibration data.

It does not work (can't find the SCSI-attached scanner),
unfortunately.

That's too bad, but not really surprising. My old scanner (Nikon
LS-30) is a SCSI scanner but I only run Linux very rarely so I haven't
tried it.

Yes, but actually it is very feature-limited. The few times I tried it
on my 2450, it gave me just average picture quality (OK for reflective
scans).

I like to keep my OSes segregated on different (removable) drives so
I'm hoping to give SANE a try after I get a new drive for my notebook
where I plan to install Linux. (My Linux is on the old desktop which I
boot up very rarely because my main machine these days is the
notebook).

BTW, if you do your post-processing in Photoshop (does it run in
WINE?) isn't a basic ("raw") scan all you need?

Don.

 

[Fernando]

On my Nikon, the same type of calibration is performed at startup.
However, after the scanner has been on for a while it automatically
re-calibrates itself at regular intervals!! That's because the
temperature inside the device changes and CCDs are very sensitive to
this so the regular recalibration is necessary.

It is a very sensible approach. No wonders, since it comes from Nikon.

I have the habit of ("manually") recalibrating my Minolta (within MSU
or Vuescan) each hour or so.

BTW, if you do your post-processing in Photoshop (does it run in
WINE?) isn't a basic ("raw") scan all you need?

In principle, yes, but actually, there are some operations that I find
simpler / more practical when done at scan time. For example: gamma
correction, IR correction, color profile correction, working
colorspace conversion. Since I have the Linux box connected to my XP
workstation via Ethernet, I save a lot of time by having the scanner
working while I do postprocessing steps with Photoshop on XP. So if
the scan software can do a bit more work, I save a bit more time.
Since I typically scan at maximum resolution and bit depth, it's even
more important.
That's another reason why I'm so disappointed by the streaks: I'd
like Vuescan to work smoothly with my 5400 so I could attach it to my
Linux box along with the Polaroid; while now, I have to drive it from
XP, slowing my workflow a lot (MSU is an incredible CPU hog).

Fernando

 

[Kennedy McEwen]

One thing I don't quite understand is this: I was under the impression
that dark current (at least for consumer type scanners) is actually
automatically corrected in firmware i.e., a user is not even given the
opportunity to get access to a scan with dark current still present.

I am talking about a dark reference scan in this case, not dark current
in total but almost residual dark current - what is left *after* the
calibrations (hardware and/or software) have attempted to equalise the
output of all of the CCD elements to zero when not illuminated. You
don't need access to the complete dark current signal to do this, only
what the system produces when it should be creating pure black.

 

[Kennedy McEwen]

So, what I meant above is the following scenario: You do a regular
scan, remove the film and then - before you get a chance to put in the
dark aluminium foil slide - the scanner recalibrates itself! In that
case - it seems to me - you would have to do the regular scan again
because you need both scans to be performed using the same internal
calibration data.

That's why I suggested putting the aluminium foil in an aperture in the
film strip holder - that way the media is not removed and the scanner
treats it just like another frame in the same film strip. I expect
Fernando will have some other problems to overcome before this works.
For example, trying this in the Nikon would require the autofocus to be
switched off, otherwise it would throw up a fault condition.

 

[Fernando]

It works!

http://gundam.srd.it/PhotoPages/images/5400_vuescan_darkframesub_01.jpg

I was too lazy to completely get rid of random noise from the dark
frame, so I just switched on 4x multisampling during the acquisition.
I guess this is one of the reasons why I get an higher black point on
the corrected scan, and slightly less saturated colors.
I'll investigate about this later (any suggestions?), for now I want
to share my results:

http://gundam.srd.it/PhotoPages/images/5400_vuescan_darkframesub_01.jpg

The unadjusted scan was taken with 4x multisampling on as well.
Scanner was calibrated just after firing up Vuescan (8.1.35).

Fernando

 

[Bart van der Wolf]

It works!

http://gundam.srd.it/PhotoPages/images/5400_vuescan_darkframesub_01.jpg

Great news.

Bart

 

[Fernando]

Great news.

I reported to Ed Hamrick... I hope this can encourage him to try a
little harder on the 5400 issues.
BTW, I think the DFS could be handy with other scanners, too. A quick
test with my SS120 suggested this could indeed be the case.

As for now, the method is too clumsy to be handy; it should really be
integrated in Vuescan to be useable.

Many thanks to Kennedy to suggest the proper workflow. In the past, I
already tried a dark frame subtraction, but being very silly I
worked on gamma-corrected final scans, and was not able to kill
streaks in the shadows without getting other streaks in the
highlights. '
This time I worked on 48bpp RAW scans; the "cleaned" frame was then
used as source for a "scan from file" task within Vuescan.

Fernando

 

[Bart van der Wolf]

I reported to Ed Hamrick... I hope this can encourage him to try a
little harder on the 5400 issues.

Indeed. It looks like he somehow took a shortcut for calibration if it
can be solved by DFS. Maybe he just needs to scan and average (or take
a median of) a few more lines, or samples per line, for calibration.

BTW, I think the DFS could be handy with other scanners, too. A
quick test with my SS120 suggested this could indeed be the case.

A dark frame subtraction is always useful, although it should be
implemented in software, if the additional time to acquire the
darkframe data isn't an issue.

As for now, the method is too clumsy to be handy; it should really
be integrated in Vuescan to be useable.

Yes, it should.

Bart

 

[rgbcmyk]

I'm sure it doesn't.
The 5400 performs a (firmware-level) self-calibration when first
accessed after power-up, and it saves the result of this
self-calibration in an appropriate area inside its built-in memory.

What exactly are the 5400 features being calibrated during
"self-calibration"? How did you learn that the results are stored in
"its built-in memory"? I don't think that these are mentioned in the
manual.

 

[rgbcmyk]

It is a very sensible approach. No wonders, since it comes from Nikon.

I have the habit of ("manually") recalibrating my Minolta (within MSU
or Vuescan) each hour or so.

Do you power cycle the 5400 to manually recalibrate it? If you are
concerned about the temperature build up, do you wait for the 5400 to
warm up before making the first scan? To carry this even further, the
film flatness can be affected by the temperature *during* a scan. In
that case, will you dry run a few scans to "warm up" the film before
making the final one?

In principle, yes, but actually, there are some operations that I find
simpler / more practical when done at scan time. For example: gamma
correction, IR correction, color profile correction, working
colorspace conversion.

In another thread, someone else also mentioned "gamma correction" in the
5400, but I don't see that mentioned in the manual. How do you make this
adjustment, and is it a hardware adjustment?

Thanks.

 

[Fernando]

What exactly are the 5400 features being calibrated during
"self-calibration"?

I'd guess it calibrates for sensor dark current, but it isn't
mentioned in details anywhere.

How did you learn that the results are stored in
"its built-in memory"?

Because Ed Hamrick, who reverse-engineered the 5400, said so.

Bye!

Fernando

 

[Fernando]

Do you power cycle the 5400 to manually recalibrate it?

With Minolta Scan Utility, yes, I power cycle the scanner and restart
MSU, because there is no way to "force" a recalibration.
With Vuescan, you can calibrate at any time with a specific command
(Scanner->Calibrate), in addition of power-cycling and restarting
Vuescan if you really want (I never do that). The kind of calibration
is different, though. When you power-cycle and restart the scanning
software, the scanner performs some kind of internal calibration (for
sensor dark current, probably). While the "Calibrate" command in
Vuescan performs some kind of dark-frame sampling that is stored in a
disk file. I don't think MSU does something similar, or at least, I
can't find any calibration-related disk file created by MSU.

concerned about the temperature build up, do you wait for the 5400 to
warm up before making the first scan?

Not really. Since I do lots of previews before performing final scans,
the scanner is already warmed up when I do the latters.

film flatness can be affected by the temperature *during* a scan. In
that case, will you dry run a few scans to "warm up" the film before
making the final one?

No, because I manually focus each frame just before the final scan.

In another thread, someone else also mentioned "gamma correction" in the
5400, but I don't see that mentioned in the manual. How do you make this
adjustment, and is it a hardware adjustment?

Gamma correction is a software adjustment; its value depends upon lots
of things; for instance, it depends upon the output colorspace you
select for your scans. If you select AdobeRGB or sRGB, for example,
the right Gamma setting is 2.2 (and it is automatically applied by the
scanning software). For AppleRGB, the right Gamma setting is 1.8.
You only need to adjust Gamma by yourself if you scan in Linear mode:
because this way, scans have a "flat" tonal response, with a Gamma
setting of 1.0, and you'll need a proper Gamma adjustment within your
editing software to bring the levels to the right values (otherwise
you get a screwed up tonal response, with a very dark output).
Unless you need Linear output for some reason, I'd stick with
(automatically) Gamma-corrected scans at the beginning. AdobeRGB is a
good compromise as the output colorspace, by the way, for it
encompasses a reasonably wide gamut.

Please note that I'm keeping things simple; the reasons for the need
of Gamma correction, and the justifications for different Gamma
settings, would take a long time, and were already explained in depth
by both Kennedy McEwen and Bart Van der Wolf (and others, for sure) in
past threads.

Fernando

 

[Don]

I am talking about a dark reference scan in this case, not dark current
in total but almost residual dark current - what is left *after* the
calibrations (hardware and/or software) have attempted to equalise the
output of all of the CCD elements to zero when not illuminated. You
don't need access to the complete dark current signal to do this, only
what the system produces when it should be creating pure black.

Ah! OK, now it makes sense!

But it does make VueScan look even worse if actually (and, I'm hoping,
inadvertently) it *amplifies* this *residual* noise which under normal
circumstances shouldn't even be visible!

It also confirms that VueScan does a lot of "secret" image fudging,
presumably in order to hide various other bugs. However, fudging one
thing usually exposes another e.g. amplifying residual noise...

Makes me wonder even more about the state of the underlying code. :-/

Don.

 

[Don]

Great news.

No, terrible news because it makes VueScan even worse!

If this turns out to be the cause it took Kennedy all of 5 minutes to
come up with a solution which eluded the VueScan's so-called
"programmer" (and I'm using this term very loosely!!!) for over *two
years*!

Doesn't that make you wonder what other similar *massive* problems
lurk underneath?

VueSCAM, indeed!

Don.