AV products tested vs 50K virii

[kurt wismer]

i'm turfing the entire discussion so far... i've misunderstood you,
you've misunderstood me... there's lots of sloppy terminology usage...
it's a mess...

question : "Does alerting on crud constitute a false positive?" (direct
quote)

answer : it depends on the wording of the alert and the nature of the
crud... if the crud is what the alert says it is, it's not a false
positive - otherwise it is a false positive...

implied (i thought) question : does crud belong in the false positive
test-bed of a conventional virus detection test?

answer: since crud is the set of things in a detection test test-bed
that don't belong, then it is in the complement of the set of things
we're looking to detect and therefore can justifiably be placed in the
false positive test bed (example: in a virus detection test-bed, all
non-viruses are crud and can be moved to the false positive test-bed -
by definition if we found a virus in the crud it would be a false
positive)... to intentionally exclude crud introduces unnecessary
sampling bias...

 

[null]

i'm turfing the entire discussion so far... i've misunderstood you,
you've misunderstood me... there's lots of sloppy terminology usage...
it's a mess...
Agreed.

question : "Does alerting on crud constitute a false positive?" (direct
quote)

answer : it depends on the wording of the alert and the nature of the
crud... if the crud is what the alert says it is, it's not a false
positive - otherwise it is a false positive...

I'll buy that. I still have problems with using crud that 99.9% of
users will never see on their PCs in a FP test though. It just doesn't
meet the common sense test IMO.


Art
http://www.epix.net/~artnpeg

 

[kurt wismer]

I'll buy that. I still have problems with using crud that 99.9% of
users will never see on their PCs in a FP test though. It just doesn't
meet the common sense test IMO.

it really shouldn't matter... if we're talking about a virus detection
test, the false positive test will be looking for virus alerts in the
non-virus files... regardless of what type of file it is, if it isn't a
virus an anti-virus shouldn't alert on it as if it were, and if the av
does it's a false positive (and probably an indication that more are
likely)...

further, if the false positive test-bed is statistically representative
then the crud in it should only account for an insignificant proportion
of the test-bed as a whole (ex. perhaps 1 file in a thousand, perhaps 1
in 10,000)...

i don't see how their inclusion in a good FP test could negatively
impact the test on a semantic or numeric level... their explicit
exclusion, on the other hand, would be a peculiar source of bias...

 

[null]

it really shouldn't matter... if we're talking about a virus detection
test, the false positive test will be looking for virus alerts in the
non-virus files... regardless of what type of file it is, if it isn't a
virus an anti-virus shouldn't alert on it as if it were, and if the av
does it's a false positive (and probably an indication that more are
likely)...

further, if the false positive test-bed is statistically representative
then the crud in it should only account for an insignificant proportion
of the test-bed as a whole (ex. perhaps 1 file in a thousand, perhaps 1
in 10,000)...

But statistics too often lie and represent nothing more than the
biases of those who would dream them up for a purpose such as this.
It's easy to pull numbers out of a hat just as I did when said that
99.9% of users will never have (certain kinds of) crud on their PCs.
The devil is in the details.

i don't see how their inclusion in a good FP test could negatively
impact the test on a semantic or numeric level... their explicit
exclusion, on the other hand, would be a peculiar source of bias...

Their explicit inclusion could very well be a even worse source of
bias.

IMO, the crud detection problem is best addressed by a special test,
as I've said. You would use a carefully selected set of crud files.
For each file there are three possible results.

1. The scanner produces a message similar to "File is a infected boot
sector image file". The scanner then gets a score of +1

2. The scanner doesn't alert. It gets a score of zero.

3. The scanner produces a message like "Infected by XYZ virus". It
receives a score of -1

In this way, a "crud detection" profile or index can be obtained for
the scanners by adding up and finding their cumulative scores. This
aproach is far more meaningful and information rich than including
crud in FP tests. And hopefully such tests would serve to goad vendors
into making product improvements.

If F-Prot can only score well in such a test with the /collect switch
on then it should be redesigned to, in effect, have that switch on all
the time. Let's quit playing games


Art
http://www.epix.net/~artnpeg

 

[kurt wismer]

On Thu, 15 Jan 2004 19:16:34 -0500, kurt wismer <[email protected]>
[snuip]

question : "Does alerting on crud constitute a false positive?" (direct
quote)

answer : it depends on the wording of the alert and the nature of the
crud... if the crud is what the alert says it is, it's not a false
positive - otherwise it is a false positive...

I'll buy that. I still have problems with using crud that 99.9% of
users will never see on their PCs in a FP test though. It just doesn't
meet the common sense test IMO.

it really shouldn't matter... if we're talking about a virus detection
test, the false positive test will be looking for virus alerts in the
non-virus files... regardless of what type of file it is, if it isn't a
virus an anti-virus shouldn't alert on it as if it were, and if the av
does it's a false positive (and probably an indication that more are
likely)...

further, if the false positive test-bed is statistically representative
then the crud in it should only account for an insignificant proportion
of the test-bed as a whole (ex. perhaps 1 file in a thousand, perhaps 1
in 10,000)...

But statistics too often lie[/QUOTE]

spoken like someone who doesn't understand statistics... statistics
don't lie, people lie and sometimes they use statistics to try and lend
credibility to their lies...

however understanding statistics helps one see through the lie...

and represent nothing more than the
biases of those who would dream them up for a purpose such as this.
It's easy to pull numbers out of a hat just as I did when said that
99.9% of users will never have (certain kinds of) crud on their PCs.
The devil is in the details.

the ease of saying that is exactly why you can't use that as a
justification for excluding crud from the false positive test... you
balk at the prospect of lying through statistics but you commit the
same sin...

you want to exclude part of the population from sampling, but you have
no scientific basis to do so... what you have is a suspicion that that
part doesn't belong and you misrepresent it as a scientifically arrived
at conclusion by quoting a made up number... your suspicion may be
correct, but until a study is done you can't know that...

Their explicit inclusion could very well be a even worse source of
bias.

explain how drawing samples from a more complete subset of the
population is a worse source of bias... the broader the sampling
population, the less sampling bias is involved...

IMO, the crud detection problem is best addressed by a special test,

read my phosphors, i'm talking about false positive detection, not crud
detection... they're different things... obviously crud detection
requires a special test, just as trojan detection requires a special
test and adware detection requires a special test, etc... but a false
positive test to complement a virus detection test must draw samples
from the entire set of non-viruses...

 

[null]

read my phosphors, i'm talking about false positive detection, not crud
detection...

Read mine. I'm talking about and interested in crud detection. And
this is my last response to you in this thread, Bye bye.


Art
http://www.epix.net/~artnpeg

 

[kurt wismer]

Read mine. I'm talking about and interested in crud detection.

while that may be your interest, when you're making comments about the
suitability of crud inclusion in false positive testing you're talking
about something other than crud detection...