Review - 24 FirewallsTested with 8 Leak Tests (also Stealth) by PC Flank - August 2003

[BillR]

An interesting follow-up article by PC Flank in August 2003: test 24
firewalls with 8 leak tests. Note that default configurations were
used. (See 1. below).

PC Flank also tested 10 firewalls for "stealthed" ports in August 2002
(See 2. below).

1. Personal Firewalls vs. Leak Tests: Part II: "Leak Tests Win Again!"
August 7, 2003
http://www.pcflank.com/art41a.htm

(Site excerpts - Google translation)
..... 24 personal firewalls were pitted against 8 leak tests. All
firewalls were tested with their default configuration right after
install. We realize that some firewalls can pass some tests after
being reconfigured, but we believe that, since most rookie users do
not change default settings, "out of box" tests are more telling.

The leak tests we used are: TooLeaky, Thermite, LeakTest, CopyCat,
FireHole, pcAudit, Atelier Web Firewall Tester [6 tests], TooLeaky,
and YALTA.

Most of these leak tests act as a Trojan trying to send out
information from the user's PC to a remote computer bypassing all
firewall filters.

(excerpts from http://www.pcflank.com/art41c.htm)
Best Performers
.....
Outpost Firewall Pro 2.0.226
....version 2.0 is rock-solid....

Its new Component Control feature enables Outpost Pro to pass most of
the leak tests without extra configuring. All you need to do is block
suspicious DLLs when Outpost 2.0 reports them.

[Development version is supposed to block all tests.]....

Look'n'Stop 2.0.4
The winner of our previous tests proved its high rating again this
year. However, it failed [several tests].... [Developer says beta
version passses all but one test.]

Other firewalls

The other firewalls, such as Sygate and ZoneAlarm Pro, can do much
better if properly configured. So users of ZoneAlarm Pro/Plus should
enable the "High" level of control to pass more tests.
.....
Each firewall was given a point for each passed Leak test (including 6
separate tests of AWFT) and here are the standings [from August 2003]:

Firewall Points [17 possible; 10 from AWFT]
Outpost Firewall Pro 2.0.226 11 [AWFT 10]
Look'n'Stop 2.0.4 7 [AWFT 2]
EZ Firewall 3.7.179 5 [All others received AWFT 0 or
1]
Norman Personal Firewall 1.3 5
pcInternet Patrol 2.0.1.1 5
ZoneAlarm Free 3.7 5 <---
Kerio 2.1.5 4 <---
McAfee Firewall for WinXP 8.0 4
Outpost Free 1.0.1817 4 <---
Steganos Online Shield 1.52 4
Kaspersky Anti-Hacker 1.0 3
McAfee Firewall Plus 4.1 3
Sygate Pro 5.1.1615 3
Sygate Firewall 5.0.1175 3 <---
Tiny Firewall 4.5 3 <---
ZoneAlarm Pro & Plus 4.0.123.012 3
Tiny Firewall 4.5 3 [sic]
Norton Firewall 6.0.2.25 2
NIS Pro 6.0.2.23 2
TGB BOB 2
BlackIce 1
PrivateFirewall 3.0 1
GIS TermiNET XP Firewall 1.82.043 0
SecureUp Personal Firewall 2.0 0
Look'n'Stop Light 1.0.4 0 <---
VisNetic Firewall 2.0 0
["<---" added to designate free/free for personal use/etc. versions.
I'm sure someone will correct me if my memory failed me.]

According to these results, most firewalls are not protecting their
users against hacking techniques. However, some of the top firewalls,
such as ZoneAlarm and Sygate, will score better if you specifically
reconfigure them. We do not know the reason those developers did not
apply those settings by default. Perhaps they suppose those settings
can limit the user's activities on the Internet. However, without
those settings, their users are at a much higher risk!

What is even more serious is that none of the tested products can pass
all leaks tests even after reconfiguration. The leak tests won the
battle again. We hope the next one goes to the firewall developers.

-----

2. Personal firewalls vs. Stealth Test, part II (August 12, 2002)
http://www.pcflank.com/art27.htm

(Site excerpts - NB August 2002)
..... The "stealthed" system .... is harder for intruders to "detect"
.... and thus far harder to attack. ...[W]e should not overrate it, but
it is the first barrier made by firewall to stop intruders and it is
better if this barrier works.

The Stealth test uses five scanning techniques: TCP ping, TCP NULL
scanning, TCP FIN scanning, TCP XMAS scanning and UDP scanning....
.....
Then after the test each firewall was given a point for each
"stealthed" result, and here are the standings [from August 2002]:

Firewall Points
Kerio 5
Look'n'Stop Pro and Lite 5
McAfee 5
Outpost 5
Sygate 5
Tiny 5
ZoneAlarm Pro and Plus 5
Deerfield 2
Norman personal firewall 0
Norton personal firewall 0

 

[Aaron]

(e-mail address removed) (BillR) wrote in @posting.google.com:

An interesting follow-up article by PC Flank in August 2003: test 24
firewalls with 8 leak tests. Note that default configurations were
used. (See 1. below).

.... 24 personal firewalls were pitted against 8 leak tests. All
firewalls were tested with their default configuration right after
install. We realize that some firewalls can pass some tests after
being reconfigured, but we believe that, since most rookie users do
not change default settings, "out of box" tests are more telling.

If you are really interested in the performance of leak tests see
http://www.firewallleaktester.fr.st/ . covers more leak tests (10), and has
2 tests for each firewall. One "out of the box" test, and one for
"hardened" settings.

Very clear testing methodology stated on that site.

Pcflank test's lacks the performance of Kerio 4.0.4. So I have done the
test.

According to Pcflank test

Best Performers
....
Outpost Firewall Pro 2.0.226
...version 2.0 is rock-solid....

Its new Component Control feature enables Outpost Pro to pass most of
the leak tests without extra configuring. All you need to do is block
suspicious DLLs when Outpost 2.0 reports them.

[Development version is supposed to block all tests.]....

Look'n'Stop 2.0.4
The winner of our previous tests proved its high rating again this
year. However, it failed [several tests].... [Developer says beta
version passses all but one test.]

BTW these 2 firewall makers have decided that dealing with leak tests is a
top priority so naturally they do well. Suprising Kerio 4+ also has a
"system security module" which helps it perform better then Kerio 2.15.

Other firewalls

The other firewalls, such as Sygate and ZoneAlarm Pro, can do much
better if properly configured. So users of ZoneAlarm Pro/Plus should
enable the "High" level of control to pass more tests.
....
Each firewall was given a point for each passed Leak test (including 6
separate tests of AWFT) and here are the standings [from August 2003]:

Firewall Points [17 possible; 10 from AWFT]

Outpost Firewall Pro 2.0.226 11 [AWFT 10]
Look'n'Stop 2.0.4 7 [AWFT 2]

Kerio Personal Firewall 4.0.4 6 ====> My test , BTW according to my
tests and confirmed by http://www.firewallleaktester.fr.st/ , Kerio 4.0.4
and 2.15 both fail Pcaudit which contradicts pcflank test results. Kerio
4.0.4 picks up 2 extra points compared to Kerio 2.15 due to picking up of
tool-leaky and firehole.

ZoneAlarm Free 3.7 5 <---
Kerio 2.1.5 4 <---
Outpost Free 1.0.1817 4 <---
Sygate Firewall 5.0.1175 3 <---
Tiny Firewall 4.5 3 <---
Look'n'Stop Light 1.0.4 0 <---

["<---" added to designate free/free for personal use/etc. versions.

I'm sure someone will correct me if my memory failed me.]

Note version 4.0+ will test better (a little) then 2.15 which it's sytem
control module. It's far from perfect though and still can be fooled.

It can for example detect tool-leaky,firehole trying to launch Internet
explorer but still fails to detect Pcaudit,thermite trying to launch IE.

Of course if you don't have IE it won't work, but then it could very well
be Mozilla, Opera etc.

What is even more serious is that none of the tested products can pass
all leaks tests even after reconfiguration. The leak tests won the
battle again. We hope the next one goes to the firewall developers.

I highly doubt that. Once you have run malware on your computer the battle
is almost over already.




Aaron

 

[Frank Bohan]

An interesting follow-up article by PC Flank in August 2003: test 24
firewalls with 8 leak tests. Note that default configurations were
used. (See 1. below).

PC Flank also tested 10 firewalls for "stealthed" ports in August 2002
(See 2. below).

1. Personal Firewalls vs. Leak Tests: Part II: "Leak Tests Win Again!"
August 7, 2003
http://www.pcflank.com/art41a.htm

(Site excerpts - Google translation)
.... 24 personal firewalls were pitted against 8 leak tests. All
firewalls were tested with their default configuration right after
install. We realize that some firewalls can pass some tests after
being reconfigured, but we believe that, since most rookie users do
not change default settings, "out of box" tests are more telling.

The leak tests we used are: TooLeaky, Thermite, LeakTest, CopyCat,
FireHole, pcAudit, Atelier Web Firewall Tester [6 tests], TooLeaky,
and YALTA.

Most of these leak tests act as a Trojan trying to send out
information from the user's PC to a remote computer bypassing all
firewall filters.

(excerpts from http://www.pcflank.com/art41c.htm)
Best Performers
....
Outpost Firewall Pro 2.0.226
...version 2.0 is rock-solid....

Its new Component Control feature enables Outpost Pro to pass most of
the leak tests without extra configuring. All you need to do is block
suspicious DLLs when Outpost 2.0 reports them.

[Development version is supposed to block all tests.]....

Look'n'Stop 2.0.4
The winner of our previous tests proved its high rating again this
year. However, it failed [several tests].... [Developer says beta
version passses all but one test.]

Other firewalls

The other firewalls, such as Sygate and ZoneAlarm Pro, can do much
better if properly configured. So users of ZoneAlarm Pro/Plus should
enable the "High" level of control to pass more tests.
....
Each firewall was given a point for each passed Leak test (including 6
separate tests of AWFT) and here are the standings [from August 2003]:

Firewall Points [17 possible; 10 from AWFT]
Outpost Firewall Pro 2.0.226 11 [AWFT 10]
Look'n'Stop 2.0.4 7 [AWFT 2]
EZ Firewall 3.7.179 5 [All others received AWFT 0 or
1]
Norman Personal Firewall 1.3 5
pcInternet Patrol 2.0.1.1 5
ZoneAlarm Free 3.7 5 <---
Kerio 2.1.5 4 <---
McAfee Firewall for WinXP 8.0 4
Outpost Free 1.0.1817 4 <---
Steganos Online Shield 1.52 4
Kaspersky Anti-Hacker 1.0 3
McAfee Firewall Plus 4.1 3
Sygate Pro 5.1.1615 3
Sygate Firewall 5.0.1175 3 <---
Tiny Firewall 4.5 3 <---
ZoneAlarm Pro & Plus 4.0.123.012 3
Tiny Firewall 4.5 3 [sic]
Norton Firewall 6.0.2.25 2
NIS Pro 6.0.2.23 2
TGB BOB 2
BlackIce 1
PrivateFirewall 3.0 1
GIS TermiNET XP Firewall 1.82.043 0
SecureUp Personal Firewall 2.0 0
Look'n'Stop Light 1.0.4 0 <---
VisNetic Firewall 2.0 0

Strange that Zone Alarm Pro and Plus score less points than Zone Alarm Free!

===

Frank Bohan
¶ Circular Definition: see Definition, Circular.

 

[BillR]

Strange that Zone Alarm Pro and Plus score less points than Zone Alarm Free!

If memory serves, defaults changed with upgrade.

The standard problem with the test as reported is that _only_ the
default results are reported. I would have found this information
more useful if a subsequent review had included results after simple
mods (e.g., selecting high security default configuration instead of
moderate) -- with the required mods listed. Once everything was set
up for this test, I would have thought a follow-on article -- or
series of articles (review 5 every month?) -- would have been an
obvious, easy, and popular. But what do I know (as a few have so
stridently pointed out).

BillR

 

[Frank Bohan]

"Frank Bohan" <[email protected]> wrote in message
If memory serves, defaults changed with upgrade.

The standard problem with the test as reported is that _only_ the
default results are reported. I would have found this information
more useful if a subsequent review had included results after simple
mods (e.g., selecting high security default configuration instead of
moderate) -- with the required mods listed. Once everything was set
up for this test, I would have thought a follow-on article -- or
series of articles (review 5 every month?) -- would have been an
obvious, easy, and popular. But what do I know (as a few have so
stridently pointed out).

BillR

Perhaps an email to PCFlank (URL above somewhere) suggesting that your
requirements be added would elicit a favourable response.

Frank Bohan
¶ He laughs first who thought the joke was finished.