Av with the lowest overhaed

J

Julian

kurt said:
we're talking about a very select group of people with highly
specialized knowledge/experience and a class of technology that, in the
commercial sector, has taken years of full-time development to evolve to
it's current state...

The same could be said of O/S development. But volunteers developed Linux.
because you don't understand the complexity involved or the resources
required...

By going open source you have access to a larger pool of resources than
any commercial organization that has to pay people. The only question is
whether a project can attract enough of these people, but that's a
different issue.
 
J

Julian

If AV companies can share samples with each other, why couldn't they
share some technology or contribute in some other way to an open source
AV, much as the likes of Sun and IBM do to Linux? It would be in the
interest if the computing community as a whole to have an effective open
source AV. One example: it would help to eradicate viruses altogether if
the low-cost consumer ISPs could run virus scanning on their servers,
which they can't afford to do at the moment because of the prohibitive
"per-user" cost of commercial network virus scanners. Another example:
OEMs producing cheap PCs could afford to install it as standard on every
new computer.

Perhaps it's because they don't want to kill the goose that lays the
golden egg. :) It's certainly not in the interests of commercial
anti-virus companies to succeed in effectively eliminating viruses
before they reach the customers.

Perhaps this is why many of those who have the expert knowledge, who
possibly have some connection with the commercial AV developers, prefer
to criticize rather than help the open source project?
 
J

Julian

I find it hard to believe that the required skills don't exist out there
in the pool of volunteer programmers. There is all kinds of clever
software out there, if you look for it. More likely that those who could
help have never heard of open source AV, or there are other kinds of
projects that are more appealing for them to get involved in.

This dependency on volunteers is both the strength and the weakness of
open source projects. It's true, I think, that companies like IBM have
helped projects like Linux overcome difficult technical hurdles by
contributing code and funding development effort. They presumably felt
that they would gain from this, ultimately.

Unfortunately it seems to me that the AV industry is profiting too much
from this scourge of viruses and malware, and has therefore no interest
in helping an open source project, only in ensuring that it never
becomes successful.
 
F

Frederic Bonroy

Julian a écrit :
If AV companies can share samples with each other, why couldn't they
share some technology or contribute in some other way to an open source
AV, much as the likes of Sun and IBM do to Linux?

By contributing their knowledge of AV engines to an open-source project,
they would indirectly and implicitly disclose details on how their own
scanners work and for obvious reasons this wouldn't be a good idea.
 
A

Art

The same could be said of O/S development. But volunteers developed Linux.


By going open source you have access to a larger pool of resources than
any commercial organization that has to pay people. The only question is
whether a project can attract enough of these people, but that's a
different issue.

Julian, you had told me that you didn't want to argue, yet you keep on
arguing :) So I'll present arguments that I haven't seen presented
yet.

1. There are several "free for personal use" antivirus products
available. AVG, AntiVir and Avast have all been exposed to testing
at various independent testing agencies, and their histories of
capabilities in many categories can be found and studied. Many home
users who insist on using only free security products are finding
"adequate protection" from these.

2. The annual cost of purchasing top notch av products is
peanuts. There is no need for free av in the first place. User
education is what's needed.

3. Young quaified people interested in developing av will often have
debts for their education, a desire to pay off the mortgage, and want
the ability ro put money into long term investments for their
retirement and other purposes. Since their time is precious, they will
not be contributing to lost causes such as free av development if they
have their heads screwed on. They will get paid for their efforts in
addition to doing something worthwhile.


http://home.epix.net/~artnpeg
 
F

Frederic Bonroy

Syncme a écrit :
I'm not a programmer or really understand the innerworkings of a virus or Av
engine.

Then keep in mind that many people erroneously believe that virus
scanning is about looking for a couple of character strings here and
there. This very naive view of how virus scanners operate is probably
what has led to an emergence of, well, useless amateur scanner projects
that will probably fail miserably when confronted with a truly complex
virus.
 
R

Roger Wilco

Linux - An operating system designed from scratch started by 1 student
(can't get more complex than that)

IIRC, the complexity was borrowed from another OS. The code was written
from scratch once the architecture of the OS (from an analysis team) was
passed to the writing team.
 
J

Julian

Frederic said:
By contributing their knowledge of AV engines to an open-source project,
they would indirectly and implicitly disclose details on how their own
scanners work and for obvious reasons this wouldn't be a good idea.

Well, it would be a good idea if the objective was to raise the level of
performance of virus scanners in general. I don't see why AV companies
shouldn't be philanthropic to at least a certain extent: there is
something wrong with profiting as much as they do from the work of these
malware writers. And of course, it would help to rebuff those sceptics
who believe that the AV companies actually collude with the virus
writers to ensure that there are always new threats to help keep the
money rolling in...

I'm not suggesting they should give away *all* their secrets. They could
still compete in areas like cost, the UI, level of support, being the
first to have new detection methods and so on. After all, Sophos has
been very successful in the corporate market despite having one of the
most expensive retail priced products yet fewer features and a plainer
Windows UI than, say, Norton.
 
F

Frederic Bonroy

Julian a écrit :
Well, it would be a good idea if the objective was to raise the level of
performance of virus scanners in general. I don't see why AV companies
shouldn't be philanthropic to at least a certain extent:

Because they are businesses, and their goal is to make profit.
there is
something wrong with profiting as much as they do from the work of these
malware writers.

Yes, it's a cruel world. Those poor malware writers work soooo hard, and
it's the AV guys who actually earn the money. ;-)

Seriously, most people could substantially reduce the cost of their AV
protection if they chose to pay more attention to how they behave on the
net. I don't remember the last time I used my virus scanner (which is
free anyway since I am perfectly happy with an old-fashioned DOS
scanner). But then again I don't use IE/OE and I don't click on
attachments, and that is probably asking too much of the average user.

It really hurts to see computers that are running dozens of anti-virus,
anti-spyware, anti-this and anti-that programs. I couldn't work with
that, I would lose my mind within minutes.
And of course, it would help to rebuff those sceptics
who believe that the AV companies actually collude with the virus
writers to ensure that there are always new threats to help keep the
money rolling in...

There are already sensible arguments against that.
I'm not suggesting they should give away *all* their secrets.

There was a time when I knew absolutely nothing about how virus scanners
work internally. Actually, I still don't really know because I have
never seen one from inside, but I think I now have a fairly good clue of
how they *probably* work and of the basic principles and concepts.
Why is that? Well, many virus analysts make technical papers freely
available, so people who are interested in how virus detection works can
learn a lot from that.

A couple of examples:

Memory scanning under Windows NT:
http://www.peterszor.com/memscannt.pdf

Emulation of the Zhengxi virus:
http://www.noh.ro/craiu.com/papers/papers/zhengxi.html

Heuristics:
http://securityresponse.symantec.com/avcenter/reference/heuristc.pdf

So it's not like they aren't giving away anything. If you combine the
knowledge contained within those papers with common sense and
programming skills, then you are certainly prepared to write something
much more sophisticated than a scanner based on simple string search.
 
J

Julian

Art said:
1. There are several "free for personal use" antivirus products
available. AVG, AntiVir and Avast have all been exposed to testing
at various independent testing agencies, and their histories of
capabilities in many categories can be found and studied. Many home
users who insist on using only free security products are finding
"adequate protection" from these.

I agree with this, with one caveat. All of these products are going down
the path of acquiring feature bloat, and many of them won't now run on
older systems. I mostly come across AVG, and I've seen a lot of people
running Win 98 with 128MB or less that have had trouble upgrading to AVG
7. So I think there's a need for a lean, simple product that the
marketing driven commercial companies choose to ignore.
2. The annual cost of purchasing top notch av products is
peanuts. There is no need for free av in the first place. User
education is what's needed.

Unfortunately there are many small businesses running on a shoestring
that can't or won't pay for AV. Not to mention all those businesses in
the third world and eastern europe where the price isn't "peanuts."

I know that many people who get called to help these people end up
installing "free for personal use" software on these business PCs,
because they feel they have to do something, but I'd prefer an option
that was actually legal.

As for user education - you can't educate these new home PC users
because they aren't interested. To them, the PC is an appliance like the
TV and the video, and just like those, they expect it to come with
everything it needs, out of the box. Most of the viruses I get to see
are on PCs that never had an AV installed on them. If there was a free,
basic AV that OEMs could install along with Windows, then I think it
would have a big impact.
3. Young quaified people interested in developing av will often have
debts for their education, a desire to pay off the mortgage, and want
the ability ro put money into long term investments for their
retirement and other purposes. Since their time is precious, they will
not be contributing to lost causes such as free av development if they
have their heads screwed on. They will get paid for their efforts in
addition to doing something worthwhile.

I think most of the people contributing to open source development are
in precisely this category, Art. Who can say why they do it, but perhaps
it's because the work they are doing pays well but isn't particularly
challenging or interesting, like developing payroll systems for their
employer, or because they want to add a new skill to their CV. It's
probably quite hard to get a job as a programmer with an AV company, but
if that's what you want to do, showing that you've written some relevant
code in your spare time might be a good way to start.
 
J

Julian

Frederic said:
Yes, it's a cruel world. Those poor malware writers work soooo hard, and
it's the AV guys who actually earn the money. ;-)

I was thinking more of the poor users, who are forced into paying for
protection. You *could* argue the same goes for the makers of burglar
alarms, but the difference is that this is a software problem, and
software costs almost nothing to distribute once it is developed, so the
industry *could* fix this problem and save end users a lot of money. The
reason they don't may well be that it saves the IT industry money, and
allows others to make money, by leaving things the way they are...
Seriously, most people could substantially reduce the cost of their AV
protection if they chose to pay more attention to how they behave on the
net. I don't remember the last time I used my virus scanner (which is
free anyway since I am perfectly happy with an old-fashioned DOS
scanner). But then again I don't use IE/OE and I don't click on
attachments, and that is probably asking too much of the average user.

Same here (except that I do pay for F-Prot for DOS, because someone has
to) but then it's not people like us who have the problem and need the
advice. But it was the desire for something as low-overhead as my DOS
scanner but more 32-bit compatible that set me off in the direction of
looking at open source AV...
There are already sensible arguments against that.

I'm sure there are. I'm sure there are also people who still believe the
earth is flat.
A couple of examples:

Memory scanning under Windows NT:
http://www.peterszor.com/memscannt.pdf

Emulation of the Zhengxi virus:
http://www.noh.ro/craiu.com/papers/papers/zhengxi.html

Heuristics:
http://securityresponse.symantec.com/avcenter/reference/heuristc.pdf

So it's not like they aren't giving away anything. If you combine the
knowledge contained within those papers with common sense and
programming skills, then you are certainly prepared to write something
much more sophisticated than a scanner based on simple string search.

Thanks, interesting links I'll follow up some time. When I'm retired, I
might try and develop something myself just out of interest, if senile
dementia hasn't rendered me incapable of programming at all by then, of
course. ;-)
 
N

Nigel Horne

I was thinking more of the poor users, who are forced into paying for
protection.

No they're not. (a) you could drop windows and go for a system that it's
so prone to malware (b) use AVG(windows) or clamAV(open source) - both
are free (AVG for personal use only).
 
F

Frederic Bonroy

Julian a écrit :
I agree with this, with one caveat. All of these products are going down
the path of acquiring feature bloat,

AntiVir and AVG are not so terribly bloated. You get the bloat you pay
for, I suppose. ;-)
 
F

Frederic Bonroy

Julian a écrit :
I was thinking more of the poor users, who are forced into paying for
protection. You *could* argue the same goes for the makers of burglar
alarms, but the difference is that this is a software problem, and
software costs almost nothing to distribute once it is developed,

Yes - but the development as such costs a lot of money. And don't forget
that there is a need for regular, frequent updates.
Same here (except that I do pay for F-Prot for DOS, because someone has
to) but then it's not people like us who have the problem and need the
advice. But it was the desire for something as low-overhead as my DOS
scanner but more 32-bit compatible that set me off in the direction of
looking at open source AV...

If you pay for F-Prot for DOS, then surely you wouldn't mind paying for
the Windows version? It comes with a 32 bit command line scanner. There
are also command line versions of McAfee's scanner.
I'm sure there are. I'm sure there are also people who still believe the
earth is flat.

Sure, but I doubt that they have good arguments. :)
 
N

Nick FitzGerald

Syncme said:
I'm not a programmer or really understand the innerworkings of a virus or Av
engine. I'm not saying I can do it or even know where one would start,
however I find it hard to believe that it wouldn't be possible.

Od course it is _possible_.

As an open source project though, it just ain't gonna happen for all the
complexity-related reasons we have been trying to explain to you.

Linux and Apache _started_ "small and simple". A known virus detection
engine started that way _today_ will not be able to catch up. Therefore,
to do a new engine from scratch today would require you to get some really
good programmers _AND_ a few really experienced and specifically
specialized, existing virus detection experts. Throw them in a nice work
environment, pay then huge amounts of money (a lot more than they are
already getting), pay lots of very expensive lawyers to tackle the court
cases claiming against the non-compete clauses from their prior employment
contracts and in a year or so (maybe two) they should have been able to
produce a fairly decent engine and QA procedures. During this process you
will also have to start employing virus analysts, start training them on
the inner workings of your engine and let them loose on part of the huge
backlog of malware that you'll need to detect to get acceptable results on
all the "standard tests".

At this point the engine will probably be able to detect around 80-85% of
known malware (most of that heuristically/generically -- not as a result of
the few months of useful virus analysis your new lab will have done) so
you will now have to hire even more virus analysts to tackle that huge
backlog -- 15% of 100,000 malwares plus much of the new stuff released
while your team got to this point -- count on 1500+ per month) is a
_humungous_ amount of virus analysis work. (An alternative approach would
be to have "looser" heuristic and/or generic detections and suffer the
higher FP rates...)

Now, that is a best case _commercial_ solution.

Now, think about mapping that to the open source approach and you'll see
that, although it is possible _in theory_, it is never going to happen in
the real world.
 
N

Nick FitzGerald

Julian said:
If AV companies can share samples with each other, why couldn't they
share some technology or contribute in some other way to an open source
AV, much as the likes of Sun and IBM do to Linux? ...

Well, for starters, all their engines work quite differently. This is
why occasionally some new kind of virus will come along and some vendors
have totally reliable detection available almost immediately and as part
of a normal detection update, whereas others may take weeks to months to
get a properly tested (major) engine revision ready to ship.

When you realize that simple grunt scanning is only a tiny part of a
contemporary virus scanner it should be obvious that there is, in fact,
very little _technology sharing_ possible between developers. What
matters is access to samples and, occasionally, sharing information
about arcane, poorly documented (or undocumented) file formats and the
like...
... It would be in the
interest if the computing community as a whole to have an effective open
source AV. One example: it would help to eradicate viruses altogether if
the low-cost consumer ISPs could run virus scanning on their servers,
which they can't afford to do at the moment because of the prohibitive
"per-user" cost of commercial network virus scanners. Another example:
OEMs producing cheap PCs could afford to install it as standard on every
new computer.

If this actually happened (and it kind of started to happen once, with
MSAV), you would see the "bad guys" deliberately target the standard,
default scanner. All scanners have weaknesses (despite all the marketing
hype, they are all far from perfect and a lot of the "science" of
designing some detection processes is really the art of making good trade-
offs in such a way as to not make it too obvious where the gaps are...) so
a very widely distributed scanner would, in its success, make itself a
target for exploitation. Now, MSAV was not too successful despite being
packaged with DOS (many folk disabled it) but even still, several of its
flaws were quickly exploited by new viruses. If a widely distributed,
high market penetration product was open source as well, it would just be
that much easier for the bad guys to find the holes and weaknesses.
Perhaps it's because they don't want to kill the goose that lays the
golden egg. :) It's certainly not in the interests of commercial
anti-virus companies to succeed in effectively eliminating viruses
before they reach the customers.

Of course it is not in their commercial interests, but apparently
"eliminating viruses before they reach the customers" is not what most of
those customers want either! Known virus scanning _CANNOT_ achieve that.
There is known technology (that is very similar to that in most existing
scanners) that can do much better than known virus scanning, but folk
aren't interested in using it (though that may be partly because no-one
actually ships such a product).
Perhaps this is why many of those who have the expert knowledge, who
possibly have some connection with the commercial AV developers, prefer
to criticize rather than help the open source project?

The "experts" at the core of the AV industry would easily find all manner
of other employment, due to their training, intellectual curiosity, skills,
experience, etc, etc, etc. They certainly do not need malware writers to
keep writing viruses to keep themselves in work. Of course, the addictive
update model required by the deeply flawed known virus scanning technology
everyone seems to prefer using does provide a "natural" business model that
keeps AV industry executives (and their shareholders) smiling...
 
R

richard

Another point of view.... (assuming I didn't miss a similar post.)

Open source developers tend to work on Linux or the BSD variants.
These operating systems don't really have a virus/worm problem, at least
nowhere near as bad as the Windows world suffers. They certainly don't
suffer anything like polymorphics and worms compressed in multiple ways,
both of which need to be untangled before an infector can be identified.

Why would anyone expect this community to develop an antivirus product for
an OS they don't use and in many cases they despise? What possible
motivation would they have to do this?

This seems to be far more relevant to the discussion than the complexity
of producing a product.
(I suspect that if the free/open source software world had a similar
problem, the solution would look very different to todays signature based
retrospective detection:)

Richard.
 
K

kurt wismer

Syncme wrote:
[snip]
I'm not a programmer or really understand the innerworkings of a virus or Av
engine. I'm not saying I can do it or even know where one would start,
however I find it hard to believe that it wouldn't be possible.

but you're not even a programmer... don't you worry that that kinda
means you're talking out of your ass?

by the way, have you ever considered trimming you quotes?
 
K

kurt wismer

Julian said:
I find it hard to believe that the required skills don't exist out there
in the pool of volunteer programmers. There is all kinds of clever
software out there, if you look for it. More likely that those who could
help have never heard of open source AV,

most certainly not... the specialized skills/experience required to
make a decent scanning engine can only be gotten in the anti-virus
field - searching for viruses is unlike searching for anything else so
they should be quite well aware of open source av efforts...
or there are other kinds of
projects that are more appealing for them to get involved in.

if they have the skills then chances are they're already employed in
the field and working on equivalent projects in a closed-source
commercial environment...
This dependency on volunteers is both the strength and the weakness of
open source projects. It's true, I think, that companies like IBM have
helped projects like Linux overcome difficult technical hurdles by
contributing code and funding development effort. They presumably felt
that they would gain from this, ultimately.

indeed... the OS market became pretty stagnant... linux is currently
the best hope of breaking microsoft's strangle-hold on it... does ibm
then make money off of linux? not directly - instead they change the
operating system business model in such a way that they can once again
compete with the microsofts of the world and make money off of services
they can perform for linux customers... it's all about making money,
they didn't do it out of the kindness of their hearts...

an equivalent state does not exist in the anti-virus market, however...
there is plenty of healthy competition between a number of mainstream
vendors and they make the bulk of their money off of services they
provide (support, developing updates, etc)...
Unfortunately it seems to me that the AV industry is profiting too much
from this scourge of viruses and malware, and has therefore no interest
in helping an open source project,

yes, indeed... it's called business...
only in ensuring that it never
becomes successful.

all they're doing is standing idly by... companies don't get very far
by helping the competition, so they don't help...
 
K

kurt wismer

Julian said:
The same could be said of O/S development. But volunteers developed Linux.

no, the same cannot be said of OS development... OS development is
taught in schools, it is knowledge that is widely disseminated because
of it's general utility... the same is most certainly not true of virus
scanner engine development...
By going open source you have access to a larger pool of resources than
any commercial organization that has to pay people.

this is extremely myopic... you apparently think resources == manpower,
it doesn't... the open source virus-related knowledge-base does not
compare with it's commercial equivalent and that is a resource... the
open source library of analyzed virus samples does not compare to it's
commercial equivalent and that is also a resource...

without those resources, you end up with an inferior scanner...
The only question is
whether a project can attract enough of these people, but that's a
different issue.

yes, and the legal barriers to that have already been explained... the
people who have the expertise already have jobs and would get sued if
they entered into competition with their employers...
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top