dymanic route table problem

[Brian E]

I have an issue with a few servers, the few I have found so far, that are basically learning routes dynamically.
Background:
CompanyA sits in an outsourced data center that uses their IP addressing scheme in order to function across all of their sites. Outsourcing company sits behind firewalls that allow specific access into companyA's network in order to provide support to these systems. The normal routing is local systems have d-gway that points to companyA's routers in order to route, no special tables or routes are need on local machines, core routers handle it all.
This has been like this for 2 years, no issues.
Suddenly I am seeing some 2000 servers that are adding routes to the local routing tables dynamically and they are causing problems. If you delete one of these routes it will come back within seconds or minutes.
None of the systems in question are running routing protocols, the routing remote access service is disabled.

My question is how in the world does 2000 dymanically learn routes without running a routing protocal?
Before anyone asks there is no scheduled task doing this and it is not user defined.
Any help is appreciated.

 

[Phillip Windell]

My question is how in the world does 2000 dymanically learn routes without

running a
routing protocal?

It won't. You need to re-examine what you are looking at.

 

[Brian E]

Okay, do you have any suggestions?
Because everything I see is that it is dynamic. If I delete one of those routes it comes right back, and the other two will follow close behind.

My question is how in the world does 2000 dymanically learn routes without
running a
routing protocal?

It won't. You need to re-examine what you are looking at.

 

[Phillip Windell]

(Please switch to "plain text" format)

The routing table may also do that if you have addional IP#s assigned to
Nics in the Advanced section of the TCP/IP Propterties that you may have
forgotten are there.

Virtual Adapters like modems, VPN, and some other types will also create
entries in the table for themselves. Anything that shows up as an Adapter
when you run "IPConfig /All" can potentially do this.

Routing Protocols exchange routing tables between devices,...you can not get
a route dynamically unless there is another device on the LAN with a routing
table that it wants to "pass on",...the routing protocols do not create that
stuff on their own. Enabling routing protocols on a single device sitting
on the LAN by itself will not produce anything.

So,...whatever route you think you are getting will,...itself,...be the key
to where it is getting it.

--
Phillip Windell [MCP, MVP, CCNA]
www.wandtv.com

Okay, do you have any suggestions?
Because everything I see is that it is dynamic. If I delete one of those
routes it comes right back, and the other two will follow close behind.

My question is how in the world does 2000 dymanically learn routes without
running a
routing protocal?

It won't. You need to re-examine what you are looking at.

 

[Brian E]

Phillip
I do appreciate your help, however, I have gone over all of that on the
system.
There are no other network devices on these systems.
There are no routing protocols being broadcasted by the pix firewall,
this was verified by the network team today, it is all static entries, no
route learning by the device, that kind of control is needed.
There are no virtual adapters or modems.
there are no extra ip addresses or gateways.
To even complicate this I have found that 20 different systems that live
on this one vlan with this pix firewall all have at least one entry in the
route table that should not be there.
So, the basic question is how the OS truly adds an ip route to the table
when it has no interface to that subnet.
This is also an enterprise class network, it is not a workgroup with
hubs.
thanks,

(Please switch to "plain text" format)

The routing table may also do that if you have addional IP#s assigned to
Nics in the Advanced section of the TCP/IP Propterties that you may have
forgotten are there.

Virtual Adapters like modems, VPN, and some other types will also create
entries in the table for themselves. Anything that shows up as an Adapter
when you run "IPConfig /All" can potentially do this.

Routing Protocols exchange routing tables between devices,...you can not get
a route dynamically unless there is another device on the LAN with a routing
table that it wants to "pass on",...the routing protocols do not create that
stuff on their own. Enabling routing protocols on a single device sitting
on the LAN by itself will not produce anything.

So,...whatever route you think you are getting will,...itself,...be the key
to where it is getting it.

--
Phillip Windell [MCP, MVP, CCNA]
www.wandtv.com

Okay, do you have any suggestions?
Because everything I see is that it is dynamic. If I delete one of those
routes it comes right back, and the other two will follow close behind.

My question is how in the world does 2000 dymanically learn routes without
running a
routing protocal?

It won't. You need to re-examine what you are looking at.

 

[Phillip Windell]

To even complicate this I have found that 20 different systems that
live on this one vlan with this pix firewall all have at least one entry
in the route table that should not be there.
So, the basic question is how the OS truly adds an ip route to the
table when it has no interface to that subnet.

I am effectively sitting here with a blindfold on. I would need to know the
"IPConfig /All" output of the problem machine and the output of Route Print
(with all the routes showing),...and would have to know which route you are
deleting that keeps comming back.

I still may not have answer,...but at least my chances are better.
The fact that you run a VLAN raises flags, but I don't know what to think
about that at the moment.

I'm about to leave for the day, so I might not be able to reply till
tomorrow. That depends on how quick you reply.

 

[Brian E]

2 minutes

To even complicate this I have found that 20 different systems that
live on this one vlan with this pix firewall all have at least one entry
in the route table that should not be there.
So, the basic question is how the OS truly adds an ip route to the
table when it has no interface to that subnet.

I am effectively sitting here with a blindfold on. I would need to know the
"IPConfig /All" output of the problem machine and the output of Route Print
(with all the routes showing),...and would have to know which route you are
deleting that keeps comming back.

I still may not have answer,...but at least my chances are better.
The fact that you run a VLAN raises flags, but I don't know what to think
about that at the moment.

I'm about to leave for the day, so I might not be able to reply till
tomorrow. That depends on how quick you reply.

 

[Kurt]

Brian, maybe one of the routers is sending redirects. That will usually
happen with an incorrect subnet mask, where the subnet mask of the router
interface is different than that of the workstation.

....kurt

Phillip
I do appreciate your help, however, I have gone over all of that on the
system.
There are no other network devices on these systems.
There are no routing protocols being broadcasted by the pix firewall,
this was verified by the network team today, it is all static entries, no
route learning by the device, that kind of control is needed.
There are no virtual adapters or modems.
there are no extra ip addresses or gateways.
To even complicate this I have found that 20 different systems that
live on this one vlan with this pix firewall all have at least one entry
in the route table that should not be there.
So, the basic question is how the OS truly adds an ip route to the
table when it has no interface to that subnet.
This is also an enterprise class network, it is not a workgroup with
hubs.
thanks,

(Please switch to "plain text" format)

The routing table may also do that if you have addional IP#s assigned to
Nics in the Advanced section of the TCP/IP Propterties that you may have
forgotten are there.

Virtual Adapters like modems, VPN, and some other types will also create
entries in the table for themselves. Anything that shows up as an Adapter
when you run "IPConfig /All" can potentially do this.

Routing Protocols exchange routing tables between devices,...you can not
get
a route dynamically unless there is another device on the LAN with a
routing
table that it wants to "pass on",...the routing protocols do not create
that
stuff on their own. Enabling routing protocols on a single device sitting
on the LAN by itself will not produce anything.

So,...whatever route you think you are getting will,...itself,...be the
key
to where it is getting it.

--
Phillip Windell [MCP, MVP, CCNA]
www.wandtv.com

Okay, do you have any suggestions?
Because everything I see is that it is dynamic. If I delete one of those
routes it comes right back, and the other two will follow close behind.

My question is how in the world does 2000 dymanically learn routes
without
running a
routing protocal?

It won't. You need to re-examine what you are looking at.

 

[Brian E]

Hmm, I have posed that question to the network admin, dont have access to the pix.

Phillip.
ipconfig
Ethernet adapter Prod1:



Connection-specific DNS Suffix . : am.hjheinz.net
Description . . . . . . . . . . . : Compaq NC7780 Gigabit Server Adapter
Physical Address. . . . . . . . . : 00-08-02-A1-97-8F

DHCP Enabled. . . . . . . . . . . : No

IP Address. . . . . . . . . . . . : 167.126.101.25

Subnet Mask . . . . . . . . . . . : 255.255.255.0

Default Gateway . . . . . . . . . : 167.126.101.1

DNS Servers . . . . . . . . . . . : 167.126.107.27
167.126.107.20
Primary WINS Server . . . . . . . : 10.193.130.10

Secondary WINS Server . . . . . . : 167.126.107.27


Ethernet adapter Backup1:



Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Compaq NC7780 Gigabit Server Adapter #2
Physical Address. . . . . . . . . : 00-08-02-A1-97-BE

DHCP Enabled. . . . . . . . . . . : No

IP Address. . . . . . . . . . . . : 192.168.0.30

Subnet Mask . . . . . . . . . . . : 255.255.254.0

Default Gateway . . . . . . . . . :

DNS Servers . . . . . . . . . . . :
NetBIOS over Tcpip. . . . . . . . : Disabled


Route print
Interface List
0x1 ........................... MS TCP Loopback interface
0x1000003 ...00 08 02 a1 97 be ...... Compaq NC7780 Gigabit Server Adapter
0x1000004 ...00 08 02 a1 97 8f ...... Compaq NC7780 Gigabit Server Adapter
===========================================================================
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 167.126.101.1 167.126.101.25 1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
167.126.101.0 255.255.255.0 167.126.101.25 167.126.101.25 1
167.126.101.25 255.255.255.255 127.0.0.1 127.0.0.1 1
167.126.255.255 255.255.255.255 167.126.101.25 167.126.101.25 1
192.168.0.0 255.255.254.0 192.168.0.30 192.168.0.30 1
192.168.0.30 255.255.255.255 127.0.0.1 127.0.0.1 1
192.168.0.255 255.255.255.255 192.168.0.30 192.168.0.30 1
198.182.130.120 255.255.255.255 167.126.101.101 167.126.101.25 1
224.0.0.0 224.0.0.0 167.126.101.25 167.126.101.25 1
224.0.0.0 224.0.0.0 192.168.0.30 192.168.0.30 1
255.255.255.255 255.255.255.255 192.168.0.30 192.168.0.30 1
Default Gateway: 167.126.101.1
===========================================================================
Persistent Routes:
None

This morning there is only the 198.182.130.120 entry that has no business in the route table, I am sure the other routes will show up as the day progresses. As you can see from the adapters on the system these routes should not be here.
Also found out this morning that the HP unix boxes that live on this subnet also have these entries and they were put there by the unix admin either. So something is not stirring the cool aid.


Brian, maybe one of the routers is sending redirects. That will usually
happen with an incorrect subnet mask, where the subnet mask of the router
interface is different than that of the workstation.

...kurt

Phillip
I do appreciate your help, however, I have gone over all of that on the
system.
There are no other network devices on these systems.
There are no routing protocols being broadcasted by the pix firewall,
this was verified by the network team today, it is all static entries, no
route learning by the device, that kind of control is needed.
There are no virtual adapters or modems.
there are no extra ip addresses or gateways.
To even complicate this I have found that 20 different systems that
live on this one vlan with this pix firewall all have at least one entry
in the route table that should not be there.
So, the basic question is how the OS truly adds an ip route to the
table when it has no interface to that subnet.
This is also an enterprise class network, it is not a workgroup with
hubs.
thanks,

(Please switch to "plain text" format)

The routing table may also do that if you have addional IP#s assigned to
Nics in the Advanced section of the TCP/IP Propterties that you may have
forgotten are there.

Virtual Adapters like modems, VPN, and some other types will also create
entries in the table for themselves. Anything that shows up as an Adapter
when you run "IPConfig /All" can potentially do this.

Routing Protocols exchange routing tables between devices,...you can not
get
a route dynamically unless there is another device on the LAN with a
routing
table that it wants to "pass on",...the routing protocols do not create
that
stuff on their own. Enabling routing protocols on a single device sitting
on the LAN by itself will not produce anything.

So,...whatever route you think you are getting will,...itself,...be the
key
to where it is getting it.

--
Phillip Windell [MCP, MVP, CCNA]
www.wandtv.com

Okay, do you have any suggestions?
Because everything I see is that it is dynamic. If I delete one of those
routes it comes right back, and the other two will follow close behind.

My question is how in the world does 2000 dymanically learn routes
without
running a
routing protocal?

It won't. You need to re-examine what you are looking at.

 

[Phillip Windell]

Switch your news reader to plain text to help replys format properly.

The 198.182.130.120 is a route to a specific machine,...notice the mask.
What machine is it?

Why is this machine the data is from a Duel-Homed machine? If it is not
acting as a Firewall, or Router, then it should be single homed and use a
LAN Router to get between Segments. That by itself would cut the routing
table in half.

--
Phillip Windell [MCP, MVP, CCNA]
www.wandtv.com

-------------------------------------------------------------------------------
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 167.126.101.1 167.126.101.25 1
127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1
167.126.101.0 255.255.255.0 167.126.101.25 167.126.101.25 1
167.126.101.25 255.255.255.255 127.0.0.1 127.0.0.1 1
167.126.255.255 255.255.255.255 167.126.101.25 167.126.101.25 1
192.168.0.0 255.255.254.0 192.168.0.30 192.168.0.30 1
192.168.0.30 255.255.255.255 127.0.0.1 127.0.0.1 1
192.168.0.255 255.255.255.255 192.168.0.30 192.168.0.30 1
198.182.130.120 255.255.255.255 167.126.101.101 167.126.101.25 1
224.0.0.0 224.0.0.0 167.126.101.25 167.126.101.25 1
224.0.0.0 224.0.0.0 192.168.0.30 192.168.0.30 1
255.255.255.255 255.255.255.255 192.168.0.30 192.168.0.30 1
Default Gateway: 167.126.101.1
===========================================================================
Persistent Routes:
None

This morning there is only the 198.182.130.120 entry that has no business in
the route table, I am sure the other routes will show up as the day
progresses. As you can see from the

 

[Phillip Windell]

Pick a few machines that are doing this stuff,...sounds like it won't be
hard to find some.
Watch the route tables.
Record and inventory these routes from all the examined machines into a
Table (maybe Excel) like this:

NetID or Host Mask Gateway
Interface
198.182.130.120 255.255.255.255 167.126.101.101 167.126.101.25

After building up a list of them,. let's look for patterns or some kind of
consistancy between them. It is also important to note routes to individual
Hosts as opposed to routes to networks or subnets. Routes to Hosts will
use an "All 255" mask.We also need to ask if these routes actually represent
a true working path to the NetId or Host and if these NetIDs or Hosts
actually exist on your over-all network somewhere.

 

[Brian E]

Well that I can answer, each of the entries I see do correspond to a host within the outsourcing companies internal networks, jump stations used for administration, monitoring servers, so they are real.
The issue is that all unknown routes are supposed to be sent to the default gateway to let the core routers handle the flow of traffic.
There is no NAT or PAT happening here.
198.182.130.31
198.182.130.120
198.182.130.26
These are three I see consistently and all are valid internal hosts to the outsourcer.
And is only the systems that live on the same vlan that pix firewall for the outsourcer sits on , 167.126.101.101 is the internal interface for that device.
So the question again is how are these OS learning about routes when they should just send to the dgatway?

Pick a few machines that are doing this stuff,...sounds like it won't be
hard to find some.
Watch the route tables.
Record and inventory these routes from all the examined machines into a
Table (maybe Excel) like this:

NetID or Host Mask Gateway
Interface
198.182.130.120 255.255.255.255 167.126.101.101 167.126.101.25

After building up a list of them,. let's look for patterns or some kind of
consistancy between them. It is also important to note routes to individual
Hosts as opposed to routes to networks or subnets. Routes to Hosts will
use an "All 255" mask.We also need to ask if these routes actually represent
a true working path to the NetId or Host and if these NetIDs or Hosts
actually exist on your over-all network somewhere.

 

[Phillip Windell]

These are three I see consistently and all are valid internal hosts to the
outsourcer.
And is only the systems that live on the same vlan that pix firewall for the
outsourcer sits on , 167.126.101.101 is the internal interface for that
device.
So the question again is how are these OS learning about routes when they
should just send to the dgatway?
------------------------------------


The "outsourcer" needs to get involved. They are probably the cause, or they
have software on your machines that is creating this. It is almost
blatantly obvious that they have at least something to do with this since
everyone of these routes tartgets one of their machines and it only happens
on machine that are on the same segment as their PIX.

I have 4 local segments here with VPN connecting us to about 40 other sites
across the United States and none of my machine do anything similar to this.

 

[Brian E]

I agree whole heartedly, the unfortunate part is that I am admin for the outsourcer, I all of the hundreds systems for this client and I know every piece of software that is running and there is nothing installed that could do this.
Would you happen to have any references on how 2000/2003 discovers routes on boot, I have a feeling this is something learned, something like spanning tree.

These are three I see consistently and all are valid internal hosts to the
outsourcer.
And is only the systems that live on the same vlan that pix firewall for the
outsourcer sits on , 167.126.101.101 is the internal interface for that
device.
So the question again is how are these OS learning about routes when they
should just send to the dgatway?
------------------------------------


The "outsourcer" needs to get involved. They are probably the cause, or they
have software on your machines that is creating this. It is almost
blatantly obvious that they have at least something to do with this since
everyone of these routes tartgets one of their machines and it only happens
on machine that are on the same segment as their PIX.

I have 4 local segments here with VPN connecting us to about 40 other sites
across the United States and none of my machine do anything similar to this.

 

[J.H]

Hi there,

It might have learned the routes from the Router's routing table.
You might check that routing table to see what exactly the defined routing
table on that router is

If there is a route defined on that router, even one ping to the far side
host of different segment,
the routing table on your machine will be updated automatically. (even with
ping, however in
your case, your machine is up and doing something else on the network that
might've been
talking to that far side host, so the route keep updating itself
automatically when you removed!!)

Hope it helps!!
Regards,
J.H

I agree whole heartedly, the unfortunate part is that I am admin for the
outsourcer, I all of the hundreds systems for this client and I know every
piece of software that is running and there is nothing installed that could
do this.
Would you happen to have any references on how 2000/2003 discovers routes on
boot, I have a feeling this is something learned, something like spanning
tree.

These are three I see consistently and all are valid internal hosts to the
outsourcer.
And is only the systems that live on the same vlan that pix firewall for the
outsourcer sits on , 167.126.101.101 is the internal interface for that
device.
So the question again is how are these OS learning about routes when they
should just send to the dgatway?
------------------------------------


The "outsourcer" needs to get involved. They are probably the cause, or they
have software on your machines that is creating this. It is almost
blatantly obvious that they have at least something to do with this since
everyone of these routes tartgets one of their machines and it only happens
on machine that are on the same segment as their PIX.

I have 4 local segments here with VPN connecting us to about 40 other sites
across the United States and none of my machine do anything similar to this.

 

[Phillip Windell]

Would you happen to have any references on how 2000/2003 discovers routes on
boot,

I'm not aware that such a mechanism even exists,...in fact I don't think
such a thing does exist. The routing table is built at bootime based on
the TCP/IP config of the Interfaces, so it is effectively *static* and not
"learned", which is why I asked earlier if there were additional IP#s
configured that might have been forgotten about.

You need to be looking at the PIX and any other router that "touches" that
segment. If you don't have access to those things, find someone who does and
dump this in their lap.

I have a feeling this is something learned, something like spanning tree.

Spanning Tree is Layer2 and only effects the Switch Fabric. In the end all
it really does is detect rudundant Switch Pathes and shuts down the slower
one and holds it in reserve in case the primary one goes down,...then it
brings up the reservered one. That is all it does,..it is Layer2 and only
functions within a single subnet. Routing Tables in the OS are Layer3, so
there is really no relationship at all,..they aren't even aware of each
other.

 

[Kurt]

I agree 100% with Phillip. If it only happens in segments where the default
gateway is pointed at the PIX, the PIX is responsible - it may not be doing
anything wrong based on it's configuration - but it is almost surely the
source of the information. Windows will not learn routes from another
router's routing table (unless a routing protocol is running on both), but
it will learn direct routes to hosts via an ip redirect.

...kurt

 

[Brian E]

Thanks guys, I believe that is where the problem lies as well.
The redirect has been mentioned by another person as well and that is surely what it seems like it is happening.

I agree 100% with Phillip. If it only happens in segments where the default
gateway is pointed at the PIX, the PIX is responsible - it may not be doing
anything wrong based on it's configuration - but it is almost surely the
source of the information. Windows will not learn routes from another
router's routing table (unless a routing protocol is running on both), but
it will learn direct routes to hosts via an ip redirect.

..kurt

 

[Phillip Windell]

Hi Kurt!

What is this "ip redirect"? This is the second time I've heard it mentioned,
but I went through all the Cisco CCNA when I got the Cert and never heard
anything about this. What is it exactly?

 

[Brian E]

Well I am not exactly sure what the real definition is, but the problem above was solved by the network guys I work with.
In this instance the when the local 2000 system sent packet to the router to get to 198.182, the router is smart enough to tell the OS that it lives on the subnet that has the gateway for this ip, so go there.
OS-167.127.101.? going to 198.182.130.?-router has static entry for 198.182.130.0/24 out 167.126.101.101, so instead of actually handling the routing it is telling the OS go to 101.101 since you live closer.
Hence the reason all of these systems on the 101 subnet have entries in their route tables.
It is a type of redirect, cisco may call it some thing else.
Hi Kurt!

What is this "ip redirect"? This is the second time I've heard it mentioned,
but I went through all the Cisco CCNA when I got the Cert and never heard
anything about this. What is it exactly?