VueScan raw file is not a true raw file!

[Henk de Jong]

VueScan present version 7.6.73

From VueScan Help: "Raw files are good for archiving as they contain as much
data as the scanner was able to produce without any operations that modify
the scanner data"

According to VueScan Help the Raw file should be independent of any colour
setting in VueScan, but it is not!
When I create a Raw file from a slide and choose the Built-in profile the
colour of this Raw file is different then when I save the same scan with an
ICC profile and all other settings the same. (I have checked this in Paint
Shop Pro after applying a gamma of 2.2. It is clearly visible.)

I discovered this after I saved raw files from a complete slide film to
process them later. When I read the Raw files later in VueScan, it looks
like the ICC Profile correction was applied two times. It turns out this
correction happened first when creating the Raw file and later a second time
when creating the TIFF file.

When I tested further with Raw files created more then a year before with a
much older version of VueScan, they did not show this problem.

Ed would you check this please?

With kind regards,
Henk de Jong

 

[Erik Krause]

Hello, Henk de Jong
you wrote...

When I create a Raw file from a slide and choose the Built-in profile the
colour of this Raw file is different then when I save the same scan with an
ICC profile and all other settings the same. (I have checked this in Paint
Shop Pro after applying a gamma of 2.2. It is clearly visible.)

Did you use 'Raw output with: Scan' or 'Raw output with: Save'? If it
is with 'Save' at least IR-clean and grain reduction filters are
applied. May be this is the case for ICC profiles, too...

 

[Spoon2001]

Hi Henk!

Nice to see a fellow ACFer here in the scanners newsgroup.

 

[Henk de Jong]

Did you use 'Raw output with: Scan' or 'Raw output with: Save'? If it

is with 'Save' at least IR-clean and grain reduction filters are
applied. Maybe this is the case for ICC profiles, too...

Confirmed!

When using 'Raw output with: Save' not only the IR-Clean and Grain Reduction
filters, but also the colour profile will be applied.

I think this is not how it should work. Ed?

With kind regards,
Henk de Jong

 

[Henk de Jong]

Hi Henk!

Hi Spoon2001,

Nice to see a fellow ACFer here in the scanners newsgroup.

ACFer?


With kind regards,
Henk de Jong

 

[Mendel Leisk]

So Henk, did you save your raw file at scan, or after? Your statement
"confirmed" did not clarify for me.

 

[Henk de Jong]

So Henk, did you save your raw file at scan, or after? Your statement
"confirmed" did not clarify for me.

I choose "Raw output with: Save" and got the bad results.

When I save the Raw file with "Raw output with: Save" not only the IR-Clean
and Grain Reduction
filters, but also the colour profile will be applied.

When I save the Raw file with "Raw output with: Scan" my raw files are okay.

Personally I find the option "Raw output with:" not very clear, except that
the raw file is saved automatically after the scan or after clicking on the
Save button.

Still no comment from Ed Hamrick...


With kind regards,

 

[Mendel Leisk]

I choose "Raw output with: Save" and got the bad results.

When I save the Raw file with "Raw output with: Save" not only the IR-Clean
and Grain Reduction
filters, but also the colour profile will be applied.

When I save the Raw file with "Raw output with: Scan" my raw files are okay.

Personally I find the option "Raw output with:" not very clear, except that
the raw file is saved automatically after the scan or after clicking on the
Save button.

Still no comment from Ed Hamrick...


With kind regards,

I'm glad to hear the "saved at scan" files appear to have no
modification. I am doing dust and scratch cleaning directly on such
files, in photoshop, so these are my ultimate source files. I am
scanning silver based Tri-X, mostly, and in any event I do not have
infrared equiped scanner at present.

One modification I do believe is made to "saved at scan" raws, if you
set media type to color negative there is marked adjustment to the
color balance of the resultant raw file. I believe in this case
Vuescan adjusts the proportions of exposure of the red, green and blue
sensors, at the hardware level, making a rough adjustment for the
orange color mask. This is documented (somewhere) in the help file.
You can see the difference if you try scanning both ways, say as color
negative, then as black and white negative.

Cheers,

Mendel Leisk

 

[Erik Krause]

Hello, Henk de Jong
you wrote...

Confirmed!

When using 'Raw output with: Save' not only the IR-Clean and Grain Reduction
filters, but also the colour profile will be applied.

What if you switch profile to 'Built in'?

 

[Erik Krause]

Hello, Henk de Jong
you wrote...

Personally I find the option "Raw output with:" not very clear, except that
the raw file is saved automatically after the scan or after clicking on the
Save button.

Still no comment from Ed Hamrick...

'Raw output with save' was introduced many versions ago, as there was a
request to save infrared cleaned files that where neither gamma
corrected, white balanced nor cropped. ICC correction is applied
directly to the RAW data, too...

 

[Ronald Bruck]

Hello, Henk de Jong
you wrote...


'Raw output with save' was introduced many versions ago, as there was a
request to save infrared cleaned files that where neither gamma
corrected, white balanced nor cropped. ICC correction is applied
directly to the RAW data, too...

This raises the question: what is the difference between using ICE and
using Ed's algorithm? They're different, sure, otherwise Ed would be
rich and not practically donating his labor to us

My understanding of ICE is that the algorithm interacts with the
scanner. You can't just save a raw file (whatever that is) and then
apply the algorithm to the result. But this leaves me confused,
because I thought a "RAW" file was just that, the raw output of the
scanner, and that RAW is RAW--what you get from the hardware, which
can't be adjusted. Can the hardware be "tuned" mid-scan? (Not by us,
obviously, but by the manufacturer.)

--Ron Bruck

 

[Bart van der Wolf]

SNIP

This raises the question: what is the difference between using ICE and
using Ed's algorithm?

A different method of "filling in the blanks", all in postprocessing based
on the IR 'shadow' mask.

Bart

 

[Henk de Jong]

What if you switch profile to 'Built in'?

Sorry, I don't know :-(
I have the solution to my problem (thanks to you) and start scanning again.
I do not have the time to experiment further. I already lost too much time
in seeking a solution for my problem.

It would be nice if Ed changes the program (exluding the colour profile from
the Raw Scan file) or changed the Help files!

With kind regards,
Henk de Jong

 

[Henk de Jong]

'Raw output with save' was introduced many versions ago, as there was

a request to save infrared cleaned files that where neither gamma
corrected, white balanced nor cropped. ICC correction is applied
directly to the RAW data, too...

If that was the main reason for the option, then Ed should consider to give
it another name.
Under the Input Tab you can choose when you would save your output file:
after the scan (or preview) is finished or when you push the Save button. It
is logical if this setting applies to Raw Scans too. The "'Raw output with:"
option will be obsolete and can be replaced with an option called "Infrared,
grain reduction and colour profile applied: Yes/No" wich is self explaining!

Still no comment from Ed...

Henk de Jong

 

[Erik Krause]

Hello, Ronald Bruck
you wrote...

My understanding of ICE is that the algorithm interacts with the
scanner.

No. It just causes the scanner to provide additional infrared channel
data.

You can't just save a raw file (whatever that is) and then
apply the algorithm to the result.

The infrared channel is saved to a TIFF alpha channel.

But this leaves me confused,
because I thought a "RAW" file was just that, the raw output of the
scanner, and that RAW is RAW--what you get from the hardware, which
can't be adjusted.

It does not matter whether the data comes directly from the scanner or
from a previously saved alpha channel...

 

[Ronald Bruck]

Hello, Ronald Bruck
you wrote...


No. It just causes the scanner to provide additional infrared channel
data.


The infrared channel is saved to a TIFF alpha channel.


It does not matter whether the data comes directly from the scanner or
from a previously saved alpha channel...

OK, you're saying "RAW is RAW". When I use Vuescan to save the image
as raw, I get a 64-bit per pixel image off my Minolta Pro: 48 bits for
the RGB data, and 16 bits for the infrared.

But if that's the case, why are there no Photoshop plug-ins to run ICE
on this 64-bit image? GEM and ROC (the other two legs of the triumvir)
are available (GEM only on 8-bit images), but not ICE. It would seem
that the developers are missing a big profit.

ASF's website (Eastman Kodak, now) says that ICE is a mixture of
hardware and software; part of it is in the scanner's firmware. Why
would that be necessary if they're only using an extra infrared
channel? Does the patent cover the production of infrared data?

--Ron Bruck

 

[Wilfred van der Vegte]

Ronald Bruck wrote:

But if that's the case, why are there no Photoshop plug-ins to run ICE
on this 64-bit image? GEM and ROC (the other two legs of the triumvir)
are available (GEM only on 8-bit images), but not ICE. It would seem
that the developers are missing a big profit.

I have no idea, but it's relatively easy to use the IR channel and
remove dust and scratches by yourself in Photoshop. Select the IR
channel, adjust the levels so that the defects are black and the rest of
the channel is white. Then, select the combined RGB channel, choose
Select > Load Selection and select the IR channel; check 'invert'. Then,
there's several things you can do to this selection. Expanding it by 1
pixel and applying a one-pixel feather radius always seems to be a good
idea. Then you could apply a Gaussian Blur to the selection (several
pixels radius), or the Dust & Scratches filter, and you're done. Of
course, there are more sophisticated ways involving layer blendings,
etc., but what I described here is pretty effective and it's no big
deal. You can even record it as an action.

ASF's website (Eastman Kodak, now) says that ICE is a mixture of
hardware and software; part of it is in the scanner's firmware. Why
would that be necessary if they're only using an extra infrared
channel? Does the patent cover the production of infrared data?

I expect that to be the case. And I assume that it won't be easy to
circumvent the patent, although Canon seems to have found a way. They
may also have patented parts of the software but there are too many
alternative approaches that can also effectively remove the defects once
you know where they are.

 

[Ed Hamrick]

My understanding of ICE is that the algorithm interacts with the
scanner. You can't just save a raw file (whatever that is) and then
apply the algorithm to the result.

Yes, you actually can. ICE is just a type of software
dust removal using the infrared channel from the scanner.

Regards,
Ed Hamrick

 

[Erik Krause]

Hallo, Ronald Bruck
in Deinem Posting stand...

OK, you're saying "RAW is RAW". When I use Vuescan to save the image
as raw, I get a 64-bit per pixel image off my Minolta Pro: 48 bits for
the RGB data, and 16 bits for the infrared.

But if that's the case, why are there no Photoshop plug-ins to run ICE
on this 64-bit image? GEM and ROC (the other two legs of the triumvir)
are available (GEM only on 8-bit images), but not ICE. It would seem
that the developers are missing a big profit.

You will have to ask them, but most probably they won't answer. Most
likely it's all politics.

ASF's website (Eastman Kodak, now) says that ICE is a mixture of
hardware and software; part of it is in the scanner's firmware. Why
would that be necessary if they're only using an extra infrared
channel?

What magic should be done in the firmware? You have an image with
defects. You have the possibility to detect this defects. You got to
find an algorithm to replace these defects with something that looks
like real image data.

Does the patent cover the production of infrared data?

What patent? They don't mention any. Ed would surely not have dared to
use infrared data if it was patented. BTW.: on
http://www.asf.com/products/ice/FilmICESpecs.shtml
they show the infrared channel themselves.

And they don't state anywhere that there is any processing made in the
scanner firmware. In their block diagram ICE algorithm communicates to
the TWAIN driver. It must be implemented in the hardware and firmware
of course to deliver infrared data...

 

[Ronald Bruck]

Ronald Bruck wrote:



I have no idea, but it's relatively easy to use the IR channel and
remove dust and scratches by yourself in Photoshop. Select the IR
channel, adjust the levels so that the defects are black and the rest of
the channel is white. Then, select the combined RGB channel, choose
Select > Load Selection and select the IR channel; check 'invert'. Then,
there's several things you can do to this selection. Expanding it by 1
pixel and applying a one-pixel feather radius always seems to be a good
idea. Then you could apply a Gaussian Blur to the selection (several
pixels radius), or the Dust & Scratches filter, and you're done. Of
course, there are more sophisticated ways involving layer blendings,
etc., but what I described here is pretty effective and it's no big
deal. You can even record it as an action.


I expect that to be the case. And I assume that it won't be easy to
circumvent the patent, although Canon seems to have found a way. They
may also have patented parts of the software but there are too many
alternative approaches that can also effectively remove the defects once
you know where they are.

Yes, I've tried this technique. It works, although it's not as good as
ICE or Ed Hamrick's software.

The process of building the IR channel is not as obvious as it seems.
In fact, IBM has patented this process. See


<http://lists.alioth.debian.org/pipermail/sane-devel/2003-November/00947
8.html>

(watch the line breaks) for a discussion of implementing this; read the
follow-ups, especially

<http://lists.alioth.debian.org/pipermail/sane-devel/2003-November/00948
4.html>

(where he advocates using the L3 norm!! LOL, I have to bring this up
to my graduate real analysis class [which is studying L^p spaces right
now] as an application!). Seriously, though, the IR is built as a
non-linear combination of the RGB and "defect" channels. I have no
idea what the "defect" channel is natively. It's clear from this
discussion that it is NOT the IR data. This also explains why ICE
doesn't work well with Kodachrome (as ASF notes, some of the dyes in
Kodachrome fool ICE).

So what Ed has done is skirt the patent--that shouldn't be hard, there
are lots of ways of recombining the channels, and I see no reason an
L^3 norm should be used as opposed to, say an L^2 or L^4 norm (which
would be faster and easier to compute anyway). It sounds like those 64
bits aren't LITERALLY what comes out of the scanner (not even after
scaling to take into account the fact the scanner provides only 14 bits
per channel). There is some additional transform which is done to
DEDUCE what the IR channel is.

So: what the heck is the "defect" channel? What is the hardware
producing? Some sort of reflectance data? (But how can that be? The
light has to pass THROUGH the film. Sigh.)

Presumably the coolscan2 backend referred to in these posts is Open
Source. Now if I can only find it...

Anyway, this thread makes it clear that one CAN apply postprocessing to
the data coming out of the scanner, equivalently to the built-in ICE
technology. Presumably not to Vuescan's 64-bit RAW file, since it's
already done some processing on the data, and that's probably why ASF
doesn't provide an after-market ICE plugin for Photoshop: there's no
guarantee you'll be able to access a file with the true 56-bit data (4
* 14-bit) as it comes out of the scanner.

--Ron Bruck