Re: AV products tested vs 50K virii

In message news:alvtvvkbqr5ebpqj2lno47s1vth30f23h0@4ax.com, null@zilch.com wrote...
> On Fri, 9 Jan 2004 10:06:54 -0800 (PST), Anonymous
> <anon@anon.itys.net> wrote:

> >"Crud" applies to any file a scanner reports as infected which is not a functional virus.
> >"False alarm" is more appropriate.
>
> Seems rather antiquated to me. Many "antivirus" scanners now deal with
> Trojans as well as viruses. Arent the terms "malware" (as far more
> inclusive than "virus") and "viable" (meaning the sample will function
> as intended) far more suitable nowdays?

I use a virus scanner for viruses, a trojan scanner for trojans, a spyware scanner for spyware, etc, etc, backed up by common sense and sensible computing. (And please dont tell me KAV detects more trojans than specialized trojan scanners. That is crap with a capital "C".)

> >Back doors, trojans, logic bombs, spyware, hijackers, bootsector images,
> >debug files, ASM files, etc, etc, are not functional viruses, and are regarded
> >as crud by purists like Nick FitzGerald if they are included in anti-virus tests.
>
> Dunno if Nick would agree with that or not. Again, it seems a awfully
> outdated POV. And I cant imagine Nick being so out of touch that he
> would include Backdoors, Trojans, logic bombs. spyware, and
> hijackers, in his "crud".

Nick and I were on opposite sides of the fence for years. I kept an eye on him in those days, and unless he has changed his position since then, he classes anything that is not a functional (viable, if you prefer) virus included in an anti-virus test as crud. Times change. Nick may have changed with them. I am sure he will correct me if I am wrong.

> Interesting history. Certainly, I would think that "scanner fodder" as
> you describe would fall into the "crud" category. Yet, as Ive said, I
> dont have a problem with scanners which might to this day alert on
> that sort of crap. And I definitely have a problem with calling
> detections of such crud "false postives" or false alerts. I do have a
> problem with reporting them as if theyre actually viable ... which,
> unfortunately, too many scanners do.

What else can reporting a non-viral file as a functional virus be, if not a false alarm?

> I tend to be rather forgiving on that score since the state of the art
> is such that scanners cant yet do a good job of determining
> viability.

Correct, and this is where slovenly anti-virus testers fall down. A segment of code is not guaranteed to be a functional virus simply because several scanners report it as a virus. Some scanners still have "scanner fodder" detection signatures in their data bases, but those files never have been and never will be viruses. "Eddie lives.... Somewhere in Time!" is just one of many text strings that fooled some anti-virus programs into false alarming on harmless files and reporting them as viruses. One anti-virus program even false alarmed on any Windows document containing the common text "AutoOpen".

Clueless anti-virus testers (VXers, newsgroup glory-seekers, etc, etc,) accept without question anything reported as infected by their favourite scanner or given to them by their favourite swap-partner as a virus, and add it to their test sets. Their tests are crap. Not quite so clueless anti-virus testers (University test labs, PC magazine test labs, etc, etc,) accept without question anything reported as infected by several of their favourite scanners or given to them by their favourite anti-virus vendor as a virus, and add it to their test sets. Their tests are crap. Careful anti-virus testers (there are not many of those) have verified that every virus in their test sets is functional by executing it on a goat computer, the only way to be certain the code is infectious.

> Sandboxing everything would take too much time, I suppose.

It is not hard to code a virus to detect a sandbox or a virtual machine and shut itself off in that environment, or even to use that environment to its own advantage. TBAV learned that the hard way years ago.

> Yet, F-prot does a reasonably good job of reporting crud it knows
> about.

Fridrik Skulason pioneered this, and has been doing it for years. All scanners should report crud as crud, or not at all.

A little more history: When I faded out of the scene, the Virus Bulletin was the undisputed 1 tester. The NCSA (now the ICSA) was still trying to shake off its undesirable reputation as the marketing arm of McAfee. (It was rumoured that NCSA founder David Stang resigned in disgust when he discovered the test sets had been rigged by John McAfee.) West Coast Labs (now Secure Computing Labs) was struggling to establish itself as serious opposition to Virus Bulletin with its Checkmark Certification. Morton Swimmer at the VTC Hamburg was doing a reasonable job. Vesselin Vladimirov (I am not Dark Avenger) Bontchev was also at the VTC Hamburg, but he was already showing strong bias towards F-Prot which lowered his credibility as an independent tester several notches. VSUM (a blatant shill for McAfee VirusScan) and PC magazine reviews were the only really crap tests around in those days. Nobody took much notice of them.

Times have changed. The Virus Bulletin is still regarded as the 1 tester. Secure Computing Labs and the ICSA now have good reputations. VTC Hamburg seems to have faded from popularity and become a bit of a joke. VSUM is dead and buried, thank God. Most PC magazines seem to have quit doing their own anti-virus testing and now buy their tests from AV-Test.org or the Virus Bulletin. (AV-Test.org is praised by Joe (WildList) Wells, but I thought he was a goose 10 years ago and I still do. It is regarded as lame by most of my old friends who stayed in the scene. I have not yet made up my mind.) It seems crap tests on giant crud-infested test sets have gained cult status with the hoi polloi, who dismiss far superior crud-free tests as "not comprehensive enough." 2004 looks like being a fun year.

GeNeSiS

Anonymous Sender wrote:

> (And please dont tell me KAV detects more trojans than specialized trojan scanners.
> That is crap with a capital "C".)

Interesting. The tests I have seen so far suggest just the opposite (and
no, those were not PC magazine or VX tests).

On Sun, 11 Jan 2004 13:17:35 +0000 (UTC), Anonymous Sender
<anonymous@remailer.metacolo.com> wrote:

>In message news:alvtvvkbqr5ebpqj2lno47s1vth30f23h0@4ax.com, null@zilch.com wrote...
>> On Fri, 9 Jan 2004 10:06:54 -0800 (PST), Anonymous
>> <anon@anon.itys.net> wrote:
>
>> >"Crud" applies to any file a scanner reports as infected which is not a functional virus.
>> >"False alarm" is more appropriate.
>>
>> Seems rather antiquated to me. Many "antivirus" scanners now deal with
>> Trojans as well as viruses. Arent the terms "malware" (as far more
>> inclusive than "virus") and "viable" (meaning the sample will function
>> as intended) far more suitable nowdays?
>
>I use a virus scanner for viruses, a trojan scanner for trojans, a spyware scanner
>for spyware, etc, etc, backed up by common sense and sensible computing.
>(And please dont tell me KAV detects more trojans than specialized trojan scanners.
>That is crap with a capital "C".)

All I can do there is refer you to a couple of av-test.org tests which
did show that several antivirus products, including KAV, were far
superior in both ITW and zoo Trojan testing than several Trojan
specific scanners. One is the old "short test" for which I would
supply a url if it wasn't so old. The other test I recall (I don't
have a url any more) was more recent ... I think it was done early
last year. In that test, eleven antivirus products (I think it was)
detected more than 99% of the over 6,000 Trojans used.

>> >Back doors, trojans, logic bombs, spyware, hijackers, bootsector images,
>> >debug files, ASM files, etc, etc, are not functional viruses, and are regarded
>> >as crud by purists like Nick FitzGerald if they are included in anti-virus tests.
>>
>> Dunno if Nick would agree with that or not. Again, it seems a awfully
>> outdated POV. And I cant imagine Nick being so out of touch that he
>> would include Backdoors, Trojans, logic bombs. spyware, and
>> hijackers, in his "crud".
>
>Nick and I were on opposite sides of the fence for years. I kept an eye on him
>in those days, and unless he has changed his position since then, he classes
>anything that is not a functional (viable, if you prefer) virus included in an
>anti-virus test as crud. Times change. Nick may have changed with them.
>I am sure he will correct me if I am wrong.
>
>> Interesting history. Certainly, I would think that "scanner fodder" as
>> you describe would fall into the "crud" category. Yet, as Ive said, I
>> dont have a problem with scanners which might to this day alert on
>> that sort of crap. And I definitely have a problem with calling
>> detections of such crud "false postives" or false alerts. I do have a
>> problem with reporting them as if theyre actually viable ... which,
>> unfortunately, too many scanners do.
>
>What else can reporting a non-viral file as a functional virus be, if not a false alarm?

Hey, that's an idea! Perhaps the independent test agencies could at
least pressure the vendors to report crud as crud. Maybe some have
been doing that, I dunno. But wouldn't you say the av product is not
false alarming if it spells it out for what it is rather than
reporting a infection? Wouln't you say it's all really a reporting
rather than a "false alarm" problem? I would.

>> I tend to be rather forgiving on that score since the state of the art
>> is such that scanners cant yet do a good job of determining
>> viability.
>
>Correct, and this is where slovenly anti-virus testers fall down.
>A segment of code is not guaranteed to be a functional virus simply
>because several scanners report it as a virus. Some scanners still have
>"scanner fodder" detection signatures in their data bases, but those files
>never have been and never will be viruses. "Eddie lives.... Somewhere in
>Time!" is just one of many text strings that fooled some anti-virus programs
>into false alarming on harmless files and reporting them as viruses. One
>anti-virus program even false alarmed on any Windows document containing
>the common text "AutoOpen".
>
>Clueless anti-virus testers (VXers, newsgroup glory-seekers, etc, etc,) accept
>without question anything reported as infected by their favourite scanner or
>given to them by their favourite swap-partner as a virus, and add it to their
>test sets. Their tests are crap. Not quite so clueless anti-virus testers
>(University test labs, PC magazine test labs, etc, etc,) accept without question
>anything reported as infected by several of their favourite scanners or given to
>them by their favourite anti-virus vendor as a virus, and add it to their test sets.
>Their tests are crap. Careful anti-virus testers (there are not many of those)
>have verified that every virus in their test sets is functional by executing it on a
>goat computer, the only way to be certain the code is infectious.
>
>> Sandboxing everything would take too much time, I suppose.
>
>It is not hard to code a virus to detect a sandbox or a virtual machine and shut
>itself off in that environment, or even to use that environment to its own advantage.
>TBAV learned that the hard way years ago.
>
>> Yet, F-prot does a reasonably good job of reporting crud it knows
>> about.
>
>Fridrik Skulason pioneered this, and has been doing it for years. All scanners
>should report crud as crud, or not at all.

I agree of course.

>A little more history: When I faded out of the scene, the Virus Bulletin was the
>undisputed 1 tester. The NCSA (now the ICSA) was still trying to shake off its
>undesirable reputation as the marketing arm of McAfee. (It was rumoured that
>NCSA founder David Stang resigned in disgust when he discovered the test
>sets had been rigged by John McAfee.) West Coast Labs (now Secure Computing
>Labs) was struggling to establish itself as serious opposition to Virus Bulletin with its
>Checkmark Certification. Morton Swimmer at the VTC Hamburg was doing a
>reasonable job. Vesselin Vladimirov (I am not Dark Avenger) Bontchev was also
>at the VTC Hamburg, but he was already showing strong bias towards F-Prot
>which lowered his credibility as an independent tester several notches. VSUM
>(a blatant shill for McAfee VirusScan) and PC magazine reviews were the only
>really crap tests around in those days. Nobody took much notice of them.
>
>Times have changed. The Virus Bulletin is still regarded as the 1 tester.
>Secure Computing Labs and the ICSA now have good reputations.
>VTC Hamburg seems to have faded from popularity and become a bit of a joke.

What's the score there? I still refer to the VTC tests. I detest the
publication of all pass/fail ITW tests since naive users are misled by
them.

>VSUM is dead and buried, thank God. Most PC magazines seem to have
>quit doing their own anti-virus testing and now buy their tests from
>AV-Test.org or the Virus Bulletin. (AV-Test.org is praised by Joe (WildList) Wells,
>but I thought he was a goose 10 years ago and I still do. It is regarded as lame
>by most of my old friends who stayed in the scene. I have not yet made up my
>mind.) It seems crap tests on giant crud-infested test sets have gained cult
>status with the hoi polloi, who dismiss far superior crud-free tests as "not
>comprehensive enough." 2004 looks like being a fun year.

Well, it's a sad state of affairs if you can't place any confidence in
or put much credence in those test agencies such as av-test.org and
the VTC which do some very interesting large scale tests of different
kinds.


Art
http://www.epix.net/~artnpeg

"Anonymous Sender" <anonymous@remailer.metacolo.com> to Art to "Anon":

> > >"Crud" applies to any file a scanner reports as infected which is not a functional virus.
> > >"False alarm" is more appropriate.
> >
> > Seems rather antiquated to me. Many "antivirus" scanners now deal with
> > Trojans as well as viruses. Arent the terms "malware" (as far more
> > inclusive than "virus") and "viable" (meaning the sample will function
> > as intended) far more suitable nowdays?
>
> I use a virus scanner for viruses, a trojan scanner for trojans, a spyware
> scanner for spyware, etc, etc, backed up by common sense and sensible
> computing. (And please dont tell me KAV detects more trojans than
> specialized trojan scanners. That is crap with a capital "C".)
>
> > >Back doors, trojans, logic bombs, spyware, hijackers, bootsector images,
> > >debug files, ASM files, etc, etc, are not functional viruses, and are regarded
> > >as crud by purists like Nick FitzGerald if they are included in anti-virus tests.
> >
> > Dunno if Nick would agree with that or not. Again, it seems a awfully
> > outdated POV. And I cant imagine Nick being so out of touch that he
> > would include Backdoors, Trojans, logic bombs. spyware, and
> > hijackers, in his "crud".
>
> Nick and I were on opposite sides of the fence for years. I kept an eye on
> him in those days, ...

I'm flattered... 8-)

> ... and unless he has changed his position since then, ...

I have a little/kind of, and of course, we can turn this into a semantic
game too...

> ... he classes anything that is not a functional (viable, if you prefer)
> virus included in an anti-virus test as crud. Times change. Nick may have
> changed with them. I am sure he will correct me if I am wrong.

I agree that _only_ affirmed viable virus samples should be in an _antivirus_
detection test. As soon as you decide some of the "grey area" stuff is
"acceptable" _in a virus detection test_ you automatically bias the test in
favour of some or other sub-set of products. In fact, doing so _may not_
have a significant effect on the rankings (and even the relative performance)
in the overall results _IF_ the detection testing is done in "mega detection
mode" (all fancy additional detection options enabled, no matter how unusably
slow (and unstable!) they make the scanner, heuristics set to high, etc, etc,
etc), __BUT__ the amount of work to determine that is so much greater than
taking the purist approach from the outset that _it is NEVER worth it.

Thus, for example, even though it is _clearly_ a virus by all more or less
well-accepted, roughly "scientific" definitions of "computer virus" I would
not add Trivial.71.A (aka ALREADY.COM) to the VB test-set. Why not? Because
some -- just a few, but enough -- of the "influential" AV researchers argued
just as strongly and implaccably that it should not be detected and would not
add detection of it to their products. As it was purely a historical oddity
and no-one had or would ever consider it a "real threat" (despite having been
quite widespread "in the wild" -- IIRC it was distributed by PC Magazine or
some similarly "respected" publication), not including it in the VB tests did
not devalue the tests (which are run with scanners on default installation
settings, other than for report generation and "action on detecting a virus").
In fact, the reverse would have been true. I chose not to include that known
virus because it is a major grey area (if you don't know the history of
ALREADY.COM please Google it -- I'm not going to recap it yet again...).

Hopefully the gentle readers will now start to understand that there is no
such thing as an easy day in the life of a virus scanner detection tester, at
least if s/he is interested in doing vaguely "quality" tests. If you don't
understand the massive complexity of _me_ deciding to not include a viable
sample of a real virus in a test-set, you cannot begin to draw many truly
significant conclusions about the testing methodology of any given test and
therefore are necessarily unable to begin to consider judging the quality of
any test.

"Virus" is something we have as near as "solid" a definition of as anything
these products can supposedly detect, yet I've just shown that even genuine,
"real viruses" can provide sufficient greay areas that it may be better to not
include certain ones in "truly quality tests". Imagine then how the real
experts would struggle over the much less defined -- even outright wooly --
stuff like "Trojan Horse", "spyware", "adware" (hell, there are huge questions
over whether some of these should ever be detected _at all_ that, depending on
position, make the "well, how precisely shall we define <whatever>ware??"
questions entirely moot).

Of course, some testers have _NO_ problem in deciding these issues, at all.

Why? Because, being blithely ignorant of the _existence_ of these issues,
they just churn on doing whatever crappy kind of test they set their mind to
then publish their results...

Anyway, back to my "I have [changed my mind] a little/kind of" comment...

Where my attitude is sliding a bit is the issue of the value of detection
tests _for so-called virus scanners_ against test-sets of non-viral malware.
Ignoring for the moment the fundamentally difficult issue of deciding how to
define, say Trojan Horses, and thus defining the bounds of the potential
sample set from which such a Trojan Horse test-set can be drawn, most "virus
scanners" do, now, claim to also detect "Trojans", and many claim to detect
other kinds of "malware" as well.

It is not incorrect to want to test those claims, but it will be orders of
magnitude harder to do so _well_ than it is to perform a quality virus
detection test. As precious few individuals or groups around the world have
shown themselves capable of the latter,I think it will be a while yet before
we see the former, but is OK for people to try (perhaps especially those who
have already shown they have a few clues about doing virus detection tests
well...).

What is not OK though is lump all the non-viral stuff (even if it is all
"properly" part of one or other of those target-type test-sets) into what is
claimed to be a "virus detection test" and then suggest that you have
measured the quality of a product's virus detection abilities.

> ... Careful anti-virus testers (there are not many of those) have verified
> that every virus in their test sets is functional by executing it on a goat
> computer, the only way to be certain the code is infectious.

And, to go one step further, truly cautious antivirus testers verify that the
viruses in their test-sets are not only functional, _but remain so for
multiple infection generations_ (two is good, three is better). (As a rule,
if you know little or nothing about how a virus works you should take the
initial sample, replicate it to multiple hosts if possible, take several of
those and replicate them, take several of each of their offspring and check
that at least one replicates further. You then throw out any of _YOUR_ first
generation replicants that had no viable offspring and use all of the rest of
your first generation, or some random selection thereof, as your test-set
samples. If you plan to (possibly) provide product developers with samples of
missed viruses you should keep at least one of the "good for the test-set"
samples aside as a "proof of variant" sample for supplying to developers (i.e.
you should not give actual test-set samples to developers ever, unless you
immediately replace the test-set sample with another never given to a
developer). If you're not doing at least this level of replicating and
viability testing, you're not doing a good job (though, admittedly if you know
well how the virus works, you may be able to reduce the generational testing
somewhat, but if you are doing the computer virus study equivalent of "bucket
chemistry" you need to be doing this level of checking).

Finally, if that doesn't sound like enough work note that ideally you should
repeat one more level of replication and only keep samples from your first
replication phase that produce at least one sample that is viable by this
extra generational test (if you think about what the "recursively", commonly
found in the definitions of "computer virus", means you will realize this
third generation test is technically necessary rather than optional but it can
be a huge ammount of extra work if you are trying to produce large numbers of
usable samples, say many hundreds or thousands of samples of a polymorphic or
metamorphic virus). You MUST NEVER include originating samples -- especially
if they come from a developer or other researcher or tester with "close" links
to a developer, as they may be "tainted" in all manner of ways that bias the
ability of that developer's product to detect them. The exception would be if
a special "germ" or non-morphed form has been released by the virus writer --
if you are sure that you have such a form you can include that as one of the
samples of that virus, but never as _THE ONLY_ sample.

I could go on, but it's late and only about three other people on the planet
need to know all this and they already do, so...


--
Nick FitzGerald