cK-Gunslinger left a note on my windscreen which said:
Consider benchmarking a car. Say there is a common stretch of road
typically used to determine a car's fuel use (miles per gallon, mpg.)
If a certain manufacturer realized that a strong wind was blowing in a
favorable direction on Tuesdays afternoon at 3:45pm, and this wind would
raise fuel efficiency from 28mpg to 36mpg (yeah, it's a strong wind =P),
so this company *always* performed its tests at this time and reported
that the car gets 36MPG, would this be fair? No. Perhaps if you lived
on this particular stretch of road and had to commute each Tuesady at
3:45, then you might consider it. Otherwise, for all intents and
purposes, the car gets 28mpg, not 36, as they would have you believe.
Consider an alternative benchmark with cars. If a Ford had a switch in
the engine management system which didn't affect performance but
increased miles per gallon then why would it be a cheat to have it on?
It's a perfectly valid 'optimization' for comparing the cars.
Likewise, if all you play is Farcry, then the benchmark is valid, but
you cannot use the Farcry results to extrapolate how any future game
will perform. Therefore, the benchmark data is invalid and the compnay
is *cheating.*
Therefore you cannot use ANY benchmarking software to predict how a
future game will perform since that future game may also be optimized.
All you will be looking at for unoptimized benchmark data is a raw
performance figure which means nothing in terms of real world
performance.
If ATI cards ran unoptized code better than Nvidia, and you benched
unoptimized code you would conclude that ATI cards are better than
Nvidia's. But if Nvidia could optimize the code better for their cards
the real world performance of the games would be superior on Nvidia
hardware. Which puts us right back at the beginning with the concept
that all benchmarking is pretty worthless unless you are comparing for a
specific game. And as you've said - specifically this 'cheat' is valid
for FC.
If they want to enable an optimization for a particular game, then that
should be a *user option* in the drivers. That is, you should be able
to enable/disable the feature with a few mouse clicks. Benchmarking for
comparative purposed should be peformed with all
(quality-reducing/modifying) optimizations off. You have to at least
strive for apples-to-apples comparisons.
That's the way I see it, anyway.
In these terms however, I don't see how comparing performance for a
group of apps for future performance of as-yet unreleased apps is
comparing apples to apples.
Benchmarking has always been a rough indicator of performance. Just
because a card gets a great 3D Mark 03 score doesn't mean it's going to
beat another card in any particular game, including unreleased software.
If this 'cheat' doesn't reduce IQ then I think it's an optimization and
is valid. Especially for comparing Far Cry between cards. Benching
other software for performance figures in future software is still just
as unreliable as it's ever been.