Pretty interested to hear any comments. Although this link is from the May
2003 test,

http://www.virus.gr/english/fullxml/...p?id=59&mnu=59

it seems that the results from the most resent test will be along similar
lines:

www.livepublishing.co.uk/content/page_325.shtml

This 2nd link is the recent press release from the pub that sponsored the
tests.

I personally found the results rather disconcerting, as I use Symantec
Corporate (90%). I also have crash problems with Kaspersky (ranked 1), and
absolutely hate the way F-secure gets its updates.

Harry:

Please read the following URL:
http://www.perl.com/language/misc/virus.html

Dave

Thanks for the declension lesson Dave!

Virii is the term used by nerds for such collections. However in the future
I will not refer to my 2 chow-chow pups as my dogii.

regards

harry

"David H. Lipman" <DLipman~nospam~@Verizon.Net> wrote in message
news:TK0Lb.4888$(E-Mail Removed)...
> Harry:
>
> Please read the following URL:
> http://www.perl.com/language/misc/virus.html
>
> Dave
>
>

"harry wong" <(E-Mail Removed)> wrote:

> Pretty interested to hear any comments. Although this link is from the May
> 2003 test,
>
> http://www.virus.gr/english/fullxml/...p?id=59&mnu=59

Did you read how he selected "samples" for inclusion in the test?

By taking all files from some larger set of rubbish and throwing out
all those that none of his pre-selected "top four" scanners detected.

There are _many_ problems with such bullshit tests, but the two
biggest are the following...

Starting with a sufficiently shitty pile of original "suspect files",
the "pre-selection" scanners _should_ be the top four scoring (so
long as they have roughly equal detection rates among "only real
malware" sample sets). In fact, this is _EXACTLY_ what the test
results show!

No they don't I hear some moron object. They say:

1. F-Secure version 5.40 - 99.67%

2. Kaspersky version 4.0.5.37 - 99.55%

3. e-Scan Pro version 2.5.181.5 - 97.66%

4. McAfee version 7.00.5000 - 97.14%

5. RAV version 8.6.104 - 95.18%

6. F-Prot version 3.13 - 92.92%

True, but to accept that at surface value is very disingenuous if
your concern (and expertise!) is impeccable product testing (as
mine is). To make sense of those results and my claim that they
in fact prove the presence of precisely the kind of testing bias
logical analysis suggests the employed testing methodology should
have, you have to "correct" that list for a couple of product
quirks. Note that eScan is actually an OEM'ed version of KAV.
Further note that F-Secure is a dual-engined product _combining_
both the F-PROT and KAV detection engines. Therefore, as neither
of these products is really separate or different from the four
pre-selection scanners, we will eliminate their results from the
list to get a "top four" just as the logical analysis suggested:

1. Kaspersky version 4.0.5.37 - 99.55%

2. McAfee version 7.00.5000 - 97.14%

3. RAV version 8.6.104 - 95.18%

4. F-Prot version 3.13 - 92.92%

So, the worst of the expected detection biases is proven in the
results.

What is the second most likely bias in such a test?

Well, as the test producer clearly has no ability to determine
for himself what is a valid sample and what not, it also seems
highly likely that a scanner with what I have, in past, referred
to as a "high crud-factor" should score above its real ability
(and, conversely, that some "OK" scanners will score low because
although they do not have stellar virus detection rates, several
of these pride themselves in having _very_ low crud factors).

At the risk of being sued I won't name the scanners I suspect to
have to have unduly benefited in this test for being high crud
factor detectors (though will note that they are mostly from
Central and Eastern Europe, though the reverse does _not_ hold).
Further, it seem highly likely that KAV's rating highest (of the
single-engine scanners) is due to both a very high real detection
rate _AND_ its notoriously high crud detection rate. (Some
products are strongly suspected of deliberately chasing crud
detection for precisely the reason that ignorant testers doing
stupid tests such as this will greatly overrate these products
relative to products that actually have better detection but
that also actively avoid the crud factor. These are easy to
pick by profiling the results of high crud content tests against
more carefully run tests...)

Also, given that F-PROT was about 4% behind McAfee and 6% behind
KAV, I'd say it's a safe bet that _at least_ 6% of the "samples"
present in this "test" would not have been present in any
properly-run test.

> www.livepublishing.co.uk/content/page_325.shtml
>
> This 2nd link is the recent press release from the pub that sponsored the
> tests.

And it shows that Live Publishing in general, and even more
disconcertingly, the editor of the group's "PC Utilities"
magazine has is seriously devoid of clue when it comes to AV
(and especially AV testing) issues...

I found the following comment from Gavin Burrell, in that
release, especially telling:

Gavin concluded: “It’s important to note that none of the
antivirus programs tested proved 100% effective, and none
even managed 100% effectiveness in a single category. This
is partly because what some programs class as a virus,
others may class as harmless code.

I couldn't fabricate more "clueless about AV" copy were I ever
required to!

> I personally found the results rather disconcerting, as I use Symantec
> Corporate (90%). I also have crash problems with Kaspersky (ranked 1), and
> absolutely hate the way F-secure gets its updates.

I found the results very disconcerting, but not for the reasons
you did...


--
Nick FitzGerald

On Thu, 8 Jan 2004 17:46:27 +1300, "Nick FitzGerald"
<(E-Mail Removed)> wrote:

>"harry wong" <(E-Mail Removed)> wrote:
>
>> Pretty interested to hear any comments. Although this link is from the May
>> 2003 test,
>>
>> http://www.virus.gr/english/fullxml/...p?id=59&mnu=59
>
>Did you read how he selected "samples" for inclusion in the test?
>
>By taking all files from some larger set of rubbish and throwing out
>all those that none of his pre-selected "top four" scanners detected.

Would the results have been any different if he had included any and
all files that any scanner alerted on? I venture to say the results
would have been virtually identical.

>There are _many_ problems with such bullshit tests, but the two
>biggest are the following...
>
>Starting with a sufficiently shitty pile of original "suspect files",
>the "pre-selection" scanners _should_ be the top four scoring (so
>long as they have roughly equal detection rates among "only real
>malware" sample sets). In fact, this is _EXACTLY_ what the test
>results show!
>
>No they don't I hear some moron object. They say:
>
>1. F-Secure version 5.40 - 99.67%
>
>2. Kaspersky version 4.0.5.37 - 99.55%
>
>3. e-Scan Pro version 2.5.181.5 - 97.66%
>
>4. McAfee version 7.00.5000 - 97.14%
>
>5. RAV version 8.6.104 - 95.18%
>
>6. F-Prot version 3.13 - 92.92%
>
>True, but to accept that at surface value is very disingenuous if
>your concern (and expertise!) is impeccable product testing (as
>mine is). To make sense of those results and my claim that they
>in fact prove the presence of precisely the kind of testing bias
>logical analysis suggests the employed testing methodology should
>have, you have to "correct" that list for a couple of product
>quirks. Note that eScan is actually an OEM'ed version of KAV.
>Further note that F-Secure is a dual-engined product _combining_
>both the F-PROT and KAV detection engines. Therefore, as neither
>of these products is really separate or different from the four
>pre-selection scanners, we will eliminate their results from the
>list to get a "top four" just as the logical analysis suggested:
>
>1. Kaspersky version 4.0.5.37 - 99.55%
>
>2. McAfee version 7.00.5000 - 97.14%
>
>3. RAV version 8.6.104 - 95.18%
>
>4. F-Prot version 3.13 - 92.92%
>
>So, the worst of the expected detection biases is proven in the
>results.

No, that's far from proven. It's simply expected.

>What is the second most likely bias in such a test?
>
>Well, as the test producer clearly has no ability to determine
>for himself what is a valid sample and what not, it also seems
>highly likely that a scanner with what I have, in past, referred
>to as a "high crud-factor" should score above its real ability
>(and, conversely, that some "OK" scanners will score low because
>although they do not have stellar virus detection rates, several
>of these pride themselves in having _very_ low crud factors).

But do they claim that any malware they alert on is definitely viable?
I've never heard of any such claim. And until vendors can honestly
make such claims, demanding the use of only viable samples in tests is
actually quite irrational.

>At the risk of being sued I won't name the scanners I suspect to
>have to have unduly benefited in this test for being high crud
>factor detectors (though will note that they are mostly from
>Central and Eastern Europe, though the reverse does _not_ hold).

Whacha got against super crud detectors? Personally, I much prefer
them. I like being alerted on all crud. After all, malware is just
"crud" . Anything typical users don't care to have or want to have on
their hard drives is "crud". Whether the crud is a result of botched
disinfections or whatever kind of crud you have in mind, I'd prefer to
have my scanner alert. I couldn't care less whether or not the file in
question is viable. I want to know about it.

>Further, it seem highly likely that KAV's rating highest (of the
>single-engine scanners) is due to both a very high real detection
>rate _AND_ its notoriously high crud detection rate.

More power to it IMO.

>(Some
>products are strongly suspected of deliberately chasing crud
>detection for precisely the reason that ignorant testers doing
>stupid tests such as this will greatly overrate these products
>relative to products that actually have better detection but
>that also actively avoid the crud factor. These are easy to
>pick by profiling the results of high crud content tests against
>more carefully run tests...)

Or they may simply have the same attitude I have.

>Also, given that F-PROT was about 4% behind McAfee and 6% behind
>KAV, I'd say it's a safe bet that _at least_ 6% of the "samples"
>present in this "test" would not have been present in any
>properly-run test.

Are you suggesting that F-Prot detects as much as KAV when large scale
virus zoo and Trojan tests of a more "scientific" nature are
conducted? Please show me those test results. I've never seen them.


Art
http://www.epix.net/~artnpeg

(E-Mail Removed) a écrit :

>>Did you read how he selected "samples" for inclusion in the test?
>>
>>By taking all files from some larger set of rubbish and throwing out
>>all those that none of his pre-selected "top four" scanners detected.
>
> Would the results have been any different if he had included any and
> all files that any scanner alerted on? I venture to say the results
> would have been virtually identical.

That wouldn't have been much better. You don't select samples by using
virus scanners (since the goal is to test virus scanners), but by
analysing the files and determining whether or not they can act as
viable samples.

When I was in school I once wrote a program that used Pi to calculate
Pi. :-)

>>1. Kaspersky version 4.0.5.37 - 99.55%
>>
>>2. McAfee version 7.00.5000 - 97.14%
>>
>>3. RAV version 8.6.104 - 95.18%
>>
>>4. F-Prot version 3.13 - 92.92%
>>
>>So, the worst of the expected detection biases is proven in the
>>results.
>
> No, that's far from proven. It's simply expected.

Why is it not proven? What further evidence do you need? Don't you think
that this list, if it's not a proof, is at least a very very strong
indication that the testing method is flawed?

> But do they claim that any malware they alert on is definitely viable?
> I've never heard of any such claim. And until vendors can honestly
> make such claims, demanding the use of only viable samples in tests is
> actually quite irrational.

That depends on the kind of crud we are talking about, no? :-) Crud can
be "working crud" (in that it's able to cause damage of some sort) or it
can be "crud crud" that doesn't work at all. Why make any efforts to
detect the latter?

> Whacha got against super crud detectors? Personally, I much prefer
> them. I like being alerted on all crud.

I certainly would like AV scanners to detect working crud. But another
issue is naming: wouldn't it be great if virus scanners could be
modified to display *meaningful* messages and reasonably precise
designations of the malicious program they detect, so as to allow
differentiation between viable malware and crud?

> I couldn't care less whether or not the file in
> question is viable. I want to know about it.

So do I but doesn't it make sense to concentrate first and foremost on
the malware that actually works?

On Thu, 08 Jan 2004 20:22:51 +0100, Frederic Bonroy
<(E-Mail Removed)> wrote:

>(E-Mail Removed) a écrit :
>
>>>Did you read how he selected "samples" for inclusion in the test?
>>>
>>>By taking all files from some larger set of rubbish and throwing out
>>>all those that none of his pre-selected "top four" scanners detected.
>>
>> Would the results have been any different if he had included any and
>> all files that any scanner alerted on? I venture to say the results
>> would have been virtually identical.
>
>That wouldn't have been much better. You don't select samples by using
>virus scanners (since the goal is to test virus scanners), but by
>analysing the files and determining whether or not they can act as
>viable samples.

I know how it's allegedly done Frederic

>When I was in school I once wrote a program that used Pi to calculate
>Pi. :-)

Not a good analogy. You're missing my point.

>>>1. Kaspersky version 4.0.5.37 - 99.55%
>>>
>>>2. McAfee version 7.00.5000 - 97.14%
>>>
>>>3. RAV version 8.6.104 - 95.18%
>>>
>>>4. F-Prot version 3.13 - 92.92%
>>>
>>>So, the worst of the expected detection biases is proven in the
>>>results.
>>
>> No, that's far from proven. It's simply expected.
>
>Why is it not proven? What further evidence do you need? Don't you think
>that this list, if it's not a proof, is at least a very very strong
>indication that the testing method is flawed?

Indications are not proof. I was picking nits.

>> But do they claim that any malware they alert on is definitely viable?
>> I've never heard of any such claim. And until vendors can honestly
>> make such claims, demanding the use of only viable samples in tests is
>> actually quite irrational.
>
>That depends on the kind of crud we are talking about, no? :-) Crud can
>be "working crud" (in that it's able to cause damage of some sort) or it
>can be "crud crud" that doesn't work at all. Why make any efforts to
>detect the latter?

Why not? I don't care to have that crud on my hard drive. Do you? So
why not alert on it?

>> Whacha got against super crud detectors? Personally, I much prefer
>> them. I like being alerted on all crud.
>
>I certainly would like AV scanners to detect working crud. But another
>issue is naming: wouldn't it be great if virus scanners could be
>modified to display *meaningful* messages and reasonably precise
>designations of the malicious program they detect, so as to allow
>differentiation between viable malware and crud?

You're preaching to the choir there

>> I couldn't care less whether or not the file in
>> question is viable. I want to know about it.
>
>So do I but doesn't it make sense to concentrate first and foremost on
>the malware that actually works?

That's what "quality" testing agencies have claimed to do for ages. So
I don't know what you mean by "first and foremost" since it's been
done many times over.

What's missing are current published test results of a higher quality
than the frowned upon test in question. But I doubt if the quality
tests would show significant differences in the results. And that's my
point,


Art
http://www.epix.net/~artnpeg

(E-Mail Removed) a écrit :

> I know how it's allegedly done Frederic

I know but I was confused by what you said.

> Why not? I don't care to have that crud on my hard drive. Do you? So
> why not alert on it?

There are many things on my hard drive that I don't want to have. :-)

You would have to define precisely what crud is. You need to consider
that crud poses less of a problem than actually working malware. And
then you need to consider that working malware is already so abundant
that AV companies have a hard time trying to keep up.

> What's missing are current published test results of a higher quality
> than the frowned upon test in question. But I doubt if the quality
> tests would show significant differences in the results.

Maybe not. The fact is, it could be a coincidence. You can't expect
meaningful results if your methodology is flawed. You can obtain them if
you're lucky, but you can't expect them.

On Wed, 07 Jan 2004 22:03:21 GMT, "harry wong" <(E-Mail Removed)> wrote:

>Pretty interested to hear any comments. Although this link is from the May
>2003 test,
>
>http://www.virus.gr/english/fullxml/...p?id=59&mnu=59
>
> it seems that the results from the most resent test will be along similar
>lines:
>
>www.livepublishing.co.uk/content/page_325.shtml
>
>This 2nd link is the recent press release from the pub that sponsored the
>tests.
>
>I personally found the results rather disconcerting, as I use Symantec
>Corporate (90%). I also have crash problems with Kaspersky (ranked 1), and
>absolutely hate the way F-secure gets its updates.

This does not test only wild viruses. So even if an AV has a high
percentage that does not mean it protects against 100% of the current
wild virii.

On Thu, 08 Jan 2004 21:20:43 +0100, Frederic Bonroy
<(E-Mail Removed)> wrote:

>(E-Mail Removed) a écrit :
>
>> I know how it's allegedly done Frederic
>
>I know but I was confused by what you said.

Sometimes I get confused by the responses

>> Why not? I don't care to have that crud on my hard drive. Do you? So
>> why not alert on it?
>
>There are many things on my hard drive that I don't want to have. :-)
>
>You would have to define precisely what crud is. You need to consider
>that crud poses less of a problem than actually working malware. And
>then you need to consider that working malware is already so abundant
>that AV companies have a hard time trying to keep up.

Nick has sort of defined his infamous term "crud" through the years. I
know he includes the remnants of botched disinfections, for just one
example. You and I have discussed this before. Remember when I pointed
out how F-Prot with the /collect (for virus collection tests) switch
on will alert on some "crud" files I sent to you? Some other crud in
typical vxer collections include infected boot sector images, object
files, and even .ASM text files. Of course I've never seen an av alert
on a text file regardless of the file extension. I doubt if any av
product is that stupid.

But I view all malware as crud whether the crap is viable or not. I'm
not discriminating

Insofar as av vendors keeping up with crud as well as viable samples,
it seems that KAV and McAfee do a pretty good job of it. Others like
F-Prot also "know about" many vxer crud files but won't alert unless
asked to. I do prefer Frisk's approach.

>> What's missing are current published test results of a higher quality
>> than the frowned upon test in question. But I doubt if the quality
>> tests would show significant differences in the results.
>
>Maybe not. The fact is, it could be a coincidence.

Obviously it _could_ be. I made no claim that it's _necessarily_ that
way. Merely that such "coincidences" are not at all unusual.

>You can't expect
>meaningful results if your methodology is flawed. You can obtain them if
>you're lucky, but you can't expect them.

Obviously. I'm not defending flawed methodology, and you of all people
should know that.


Art
http://www.epix.net/~artnpeg