Announcement: James Photography seeks participants for 2nd Annual Film Scanner Bake-Off

Last year's scanner bake-off was a runaway success, with almost 18,000
hits on the results page alone. It was based solely on scanner
resolution, measured scientifically with Norman Koren's Imatest
software. This year a panel of judges will rate the results on
sharpness, highlight/shadow accuracy, noise, as well as contrast,
saturation, and overall realism as compared to the original slide.
The subject will be a still life composition photographed with 35mm
equipment on Fuji Provia-F.

Participants must register before the February 28 deadline; final
results will be published in April.


Please visit http://www.jamesphotography.ca/ for details under
"scanner bake-off".

Thank you,


Jim Hutchison

Jim,

How about including a colour negative and a black and white negative in
the competition?

W

On 12 Jan 2005 19:32:17 -0800, winhag@yahoo.com wrote:

>Jim,
>
>How about including a colour negative and a black and white negative in
>the competition?
>
>W


Interesting suggestion for sure... I know B+W negs are problematic,
as are colour negs for less-than-perfect sharpness. The majority of
pros I know work solely with trannies, so that's why I've geared it
for that medium. Gotta keep it simple though, otherwise I'd risk
loosing participants.

Thanks for the feedback...


Cheers,

jim

Jim Hutchison wrote:
>
> Last year's scanner bake-off was a runaway success, with almost 18,000
> hits on the results page alone. It was based solely on scanner
> resolution, measured scientifically with Norman Koren's Imatest
> software. This year a panel of judges will rate the results on
> sharpness, highlight/shadow accuracy, noise, as well as contrast,
> saturation, and overall realism as compared to the original slide.
> The subject will be a still life composition photographed with 35mm
> equipment on Fuji Provia-F.
>
> Participants must register before the February 28 deadline; final
> results will be published in April.
>
> Please visit http://www.jamesphotography.ca/ for details under
> "scanner bake-off".
>
> Thank you,
>
> Jim Hutchison

In this kind of bake-of, it would be nice to be able to separate a
scanner's capability from an user's capability and the original source
material quality. For example in sharpness evaluation, it would be nice
to know how well a scanner can handle film curvature (i.e. dof), and how
well point auto focus or point manual focus works. If a submitted test
scan came from a film with a lot of film curl, or if the operator did
not point focus at the most contrasty area, the judges should not
attribute a scan's lack of sharpness to the scanner.

The results of a bake-off are only meaningful if these things are under
control. But it is easier said than done.

On Thu, 13 Jan 2005 18:51:55 GMT, HKY4482@att.net wrote:

>
>
>Jim Hutchison wrote:
>>
>> Last year's scanner bake-off was a runaway success, with almost 18,000
>> hits on the results page alone. It was based solely on scanner
>> resolution, measured scientifically with Norman Koren's Imatest
>> software. This year a panel of judges will rate the results on
>> sharpness, highlight/shadow accuracy, noise, as well as contrast,
>> saturation, and overall realism as compared to the original slide.
>> The subject will be a still life composition photographed with 35mm
>> equipment on Fuji Provia-F.
>>
>> Participants must register before the February 28 deadline; final
>> results will be published in April.
>>
>> Please visit http://www.jamesphotography.ca/ for details under
>> "scanner bake-off".
>>
>> Thank you,
>>
>> Jim Hutchison
>
>In this kind of bake-of, it would be nice to be able to separate a
>scanner's capability from an user's capability and the original source
>material quality. For example in sharpness evaluation, it would be nice
>to know how well a scanner can handle film curvature (i.e. dof), and how
>well point auto focus or point manual focus works. If a submitted test
>scan came from a film with a lot of film curl, or if the operator did
>not point focus at the most contrasty area, the judges should not
>attribute a scan's lack of sharpness to the scanner.
>
>The results of a bake-off are only meaningful if these things are under
>control. But it is easier said than done.


Easier said than done - absolutely. Last year's bake-off was based
on sharpness alone, and yet there were *SO* many variables such as
white and black point, gamma, focusing, etc. that the test cannot be
declared scientific. So much is dependant on the operator that I
decided to go the other way... let the output be optimized by the
users themselves to bring out the scanner's best.

I had tons of feedback last year asking that a real-world scan be
used. That way, each user can tweak and play to make the image as
good as possible... but, variances in output is still determined by
the scanner - you can't make a silk purse from a sow's ear.

Hopefully this year a large enough representation of each scanner
model will average out the results. Let's see!

jim h

On Wed, 12 Jan 2005 20:58:48 GMT, Jim Hutchison
<jim.hutchison@shaw.ca> wrote:

>Last year's scanner bake-off was a runaway success, with almost 18,000
>hits on the results page alone. It was based solely on scanner
>resolution, measured scientifically with Norman Koren's Imatest
>software.

Jim, I enjoyed partecipating in last year's bake-off, and I already
submitted for this one. :-)

Just one thought: as you sure already know, the target slide we used
was not very appropriate for Imatest MTF analysis: the black/white
transition was too soft for a proper Slanted Edge Test run, and as a
result, the MTF figures were, in absolute terms, quite odd. Some of us
(for example Bart Van der Wolf and me) repeated the Imatest MTF tests
with a proper Slanted Edge target, and the results were far more
accurate and reliable.
Nonetheless, the test was very interesting and gathered a lot of
informations.

Now, this year you seem to steer towards a more "subjective" analysis,
with a real-world scene slide, postprocessed images, and a jury.
A good idea, but still, I'd have like to see, as a complement to this
subjective evaluation, a more scientific MTF-type test, this time with
a proper target slide (a 5 degrees framed razorblade or something
similar, with an extremely uniform edge and abrupt transition from
opaque to transparent, would do). For this kind of test, instructions
would be the same of last year's (proper focus, no sharpening, no
clipping).

Is it viable in your opinion?

Thanks, and keep on the good work!

Fernando

HKY4482@att.net wrote:
> In this kind of bake-of, it would be nice to be able to separate a
> scanner's capability from an user's capability and the original source
> material quality. For example in sharpness evaluation, it would be nice
> to know how well a scanner can handle film curvature (i.e. dof), and how
> well point auto focus or point manual focus works. If a submitted test
> scan came from a film with a lot of film curl, or if the operator did
> not point focus at the most contrasty area, the judges should not
> attribute a scan's lack of sharpness to the scanner.

As Jim noted, hopefully there will be enough participants that there
will be several data points for each type of scanner. With averaging
over enough data points, the effect of any given user's ability will be
diminished, and so the results should be fairly representative of the
scanner's ability. Or, alternately, if skill in this mostly amounts to
"not messing up", then it may be reasonable to presume that with enough
people there will be one for each scanner (at least, each popular
scanner) who hasn't messed up.

Another thing we're likely to see is just how much difference the
individual operator makes; I suspect that someone who's been doing this
for years can do a much better job with my scanner than I can, but it
will be interesting to see how true that is.

- Brooks


--
The "bmoses-nospam" address is valid; no unmunging needed.

Fernando wrote:
> Now, this year you seem to steer towards a more "subjective" analysis,
> with a real-world scene slide, postprocessed images, and a jury.
> A good idea, but still, I'd have like to see, as a complement to this
> subjective evaluation, a more scientific MTF-type test, this time with
> a proper target slide (a 5 degrees framed razorblade or something
> similar, with an extremely uniform edge and abrupt transition from
> opaque to transparent, would do). For this kind of test, instructions
> would be the same of last year's (proper focus, no sharpening, no
> clipping).
>
> Is it viable in your opinion?

Speaking as a prospective participant, I'd be glad to also scan a target
slide of that sort along with the real-world slide. Thus, I suspect
it's perfectly viable if you're willing to send out the slides to
everyone.

- Brooks


--
The "bmoses-nospam" address is valid; no unmunging needed.

On Thu, 13 Jan 2005 23:27:13 -0800, Brooks Moses
<bmoses-nospam@cits1.stanford.edu> wrote:

>it's perfectly viable if you're willing to send out the slides to
>everyone.

It's not my bake-off, you see. I'd never had the organization skill
(and time to spend) that James already showed last year.
But for sure I can provide technical explanations about how to build
and scan such a target.

Fernando

Brooks Moses wrote:
>
> Fernando wrote:
> > Now, this year you seem to steer towards a more "subjective" analysis,
> > with a real-world scene slide, postprocessed images, and a jury.
> > A good idea, but still, I'd have like to see, as a complement to this
> > subjective evaluation, a more scientific MTF-type test, this time with
> > a proper target slide (a 5 degrees framed razorblade or something
> > similar, with an extremely uniform edge and abrupt transition from
> > opaque to transparent, would do). For this kind of test, instructions
> > would be the same of last year's (proper focus, no sharpening, no
> > clipping).
> >
> > Is it viable in your opinion?
>
> Speaking as a prospective participant, I'd be glad to also scan a target
> slide of that sort along with the real-world slide. Thus, I suspect
> it's perfectly viable if you're willing to send out the slides to
> everyone.
>
> - Brooks

I second this suggestion. It would certainly help separating the scanner
capability from the user capability.