In message news:(E-Mail Removed), (E-Mail Removed) wrote...

> Oh? Then why do the "super crud detectors" always show quite low and
> competitive false alarms rates in the so called "quality independent
> tests"? The fact is that the super crud detectors manage to avoid the
> kind of false positives that really count. I can run them year after
> year on my PCs and they never produce a "real" false alert (on
> perfectly good files). So whats the big deal about crud detectors?
> Whats so damn bad about them? See my point and my issues here?

You missed the real point. The anti-virus numbers game is just an endless dick-waving contest, with no rules. :-)

Ignoring other forms of malware and concentrating on virus detection only, here is a very simplified scenario:

Scanner 1 erroneously reports 2,000 non-viral files (scanner fodder, bootsector images, logic bombs, ASM binaries, debug files, and assorted other crud downloaded over the years from VX sites) as functional viruses.

Scanner 2 policy is to keep up with scanner 1 in the detection ratings, so it includes detection signatures for the same 2,000 crud files in its data base.

Scanner 3 policy is to beat all other scanners in the detection ratings at all costs, so it includes specific detection signatures for the same 2,000 crud files in its data base, plus specific detection signatures for another 200 crud files sourced from VX sites.

Scanners 4 and 5 policies are to ignore crud and report functional viruses only.

Tester A has a test set of 50,000 viruses, each individually executed and verified as a functional (infectious) virus by a qualified researcher. Scanners 1, 2, 3, 4, and 5 report 50,000/50,000 viruses in Test A as functional viruses. Tester A rates all 5 scanners as equal in detection. His report is accurate.

Tester B has a test set of 53,000 viruses. 52,000 are reported as a functional virus (verified, in his lexicon) by his favourite scanners 1, 2, and 3. The VX sites where he downloads his stuff list the other 1,000 as rare viruses that are undetectable by any scanner. Scanners 1 and 2 report 52,000/53,000 viruses in Test B as functional viruses. Scanner 3 reports 52,200/53,000 viruses in Test B as functional viruses. Scanners 4 and 5 report 50,000/53,000 viruses in Test B as functional viruses. Tester B reports Scanner 3 as the having best detection, Scanners 1 and 2 as close runners-up, and Scanners 4 and 5 as trailing a long way behind. His report is crap. Scanners 4 and 5 actually have the best detection.The ensuing fights on the Internet are entertaining for a while.

Scanner 1 programmers hate being beaten in detection ratings, so they add the 200 missed crud files to their data base, plus another 50.

Scanner 2 programmers are constantly monitoring Scanner 1, and they add those 250 crud files to their own data base.

Scanner 3 programmers go one better and download another 300 rare undetectable crud files from VX sites.

Tester B downloads another 500 even rarer undetectable crud files from VX sites.

Next month, the dick-waving contest starts all over again.

GeNeSiS

On Sat, 10 Jan 2004 14:42:25 +0100 (CET), Tarapia Tapioco
<(E-Mail Removed)> wrote:

>In message news:(E-Mail Removed), (E-Mail Removed) wrote...
>
>> Oh? Then why do the "super crud detectors" always show quite low and
>> competitive false alarms rates in the so called "quality independent
>> tests"? The fact is that the super crud detectors manage to avoid the
>> kind of false positives that really count. I can run them year after
>> year on my PCs and they never produce a "real" false alert (on
>> perfectly good files). So whats the big deal about crud detectors?
>> Whats so damn bad about them? See my point and my issues here?
>
>You missed the real point. The anti-virus numbers game is just an endless
dick-waving contest, with no rules. :-)

I've known that ever since I first discovered the cheat switch on
McAfee DOS and F-Prot's /COLLECT switch It was obvious from the
very start of my interest in scanners that games were being played.

One thing that might help is to test scanners with, say, several
thousand carefully selected crud samples. For every alert on a crud
sample, a penalty of -1 would be added to the number of alerts on
viable samples


Art
http://www.epix.net/~artnpeg

On Sat, 10 Jan 2004 18:08:33 GMT, (E-Mail Removed) wrote:

>One thing that might help is to test scanners with, say, several
>thousand carefully selected crud samples. For every alert on a crud
>sample, a penalty of -1 would be added to the number of alerts on
>viable samples

BTW, the above was merely a fleeting nasty thought and actually I
don't think it's a good idea at all. I do think a separate test using
a large and carefully selected set of crud files is a good idea. I'd
certainly be interested in seeing a objective "crud detection index"
for the scanners.


Art
http://www.epix.net/~artnpeg