Windows Server 2003 DNS Cache Issue

[Richard Schwartz]

All,

I apologize in advance for posting a W2K3 question here
but alas I have no other option!

We are in the process of testing a production build of
Windows Server 2003 and Exchange 2003 and have been
experiencing an issue with the DNS cache failing to dump
the resolver cache. We manually try and flush the cache
on all servers, not just the DC's, and they continue to
report stale cache entries. This is obviously causing
serious issues for mail delivery in Exchange 2003. Once
we reboot all the servers, the cache is reset. NOT
IDEAL!!! Anyone else having a similar issue? Any ideas?

Many thanks in advance,

Richard Schwartz

 

[Herb Martin]

[2003 questions should be just fine here -- there is not separate
2003 newsgroup and only a few changes to 2003 DNS.]

How are you dumping the DNS server cache?

Forgive me if I am wrong, but I have a suspicion you might be
dumping the server's OWN CLIENT cache with IPConfig /flushDNS
which has no effect on the DNS service.

Try DNSCmd unless they have replaced that in Win2003 (from Win2000).

dnscmd Server.Name.TLD /clearcache

(If that's gone, someone else will correct me -- or you might try NetSh.exe
as it does most of the other services except DNS & IPSec in Win2000.)

 

[Jonathan de Boyne Pollard]

RS> We [...] have been experiencing an issue with the DNS
RS> cache failing to dump the resolver cache.

That's probably in part because the last part of that sentence makes no
sense. There is no "the" DNS cache, and none of the relevant caches have a
"dump the resolver cache" operation.

Please explain more clearly what you think your problem is.

RS> We manually try and flush the cache on all servers, [...]

_Which_ cache ? There are two: the cache in the DNS Client and the cache in
the DNS Server.

RS> This is obviously causing serious issues for mail
RS> delivery in Exchange 2003.

If stale DNS data is causing mail delivery problems in this manner, then your
DNS database content is wrong. You've placed a TTL on the relevant resource
record sets that doesn't match the actual situation.

But one has to ask: Why are your DNS data for mail delivery changing so much
anyway ? Are you playing musical IP addresses with your Exchange servers ?
If so: why ? If not: what is the thing that _is_ changing ?

RS> Once we reboot all the servers, [...]

It is DOS Think to consider rebooting to be the cure-all that one immediately
resorts to. Before you try rebooting to fix a problem with a service, try
simply stopping and restarting the particular service _without_ rebooting.

 

[Ace Fekay [MVP]]

In

All,

I apologize in advance for posting a W2K3 question here
but alas I have no other option!

We are in the process of testing a production build of
Windows Server 2003 and Exchange 2003 and have been
experiencing an issue with the DNS cache failing to dump
the resolver cache. We manually try and flush the cache
on all servers, not just the DC's, and they continue to
report stale cache entries. This is obviously causing
serious issues for mail delivery in Exchange 2003. Once
we reboot all the servers, the cache is reset. NOT
IDEAL!!! Anyone else having a similar issue? Any ideas?

Many thanks in advance,

Richard Schwartz

Make sure you are NOT using ISP DNS servers in any of your machine's IP
properties. THIS will cause this problem too. Use a forwarder in your DNS
server to get external resolution.

--
Regards,
Ace

Please direct all replies to the newsgroup so all can benefit.

Ace Fekay, MCSE 2000, MCSE+I, MCSA, MCT, MVP
Microsoft Windows MVP - Active Directory

 

[Richard Schwartz]

First let me thank Herb and Ace for your informative
replies. That's what "community" is all about!

Again, to revisit why this action is necessary for those
who did not understand the first post, we are still in
the sandbox stage and some of our IP's do change as
architecture evolves.

The server cache was cleared by right-clicking the server
object in the DNS console and selecting "Clear Cache"
menu option. This should have cleared the name table
entries right? This did not resolve the issue.

To answer Ace's inquiry about "Secure Cache Against
Pollution ", this option is indeed checked. This is
obviously the default right? What's best practice when it
comes to this option?

Also in reply to Ace's post regarding forwarders, we are
employing forwarders to reconcile non-local names, no
ISP's.

I will post more information as it becomes availible.

Again thanks,

Richard

 

[Herb Martin]

The server cache was cleared by right-clicking the server

object in the DNS console and selecting "Clear Cache"
menu option. This should have cleared the name table
entries right? This did not resolve the issue.

It should clear the Server cache of all but the Root Hints.
Most aggressive is to re-start the server (normally not
necessary.)

Watch out for client-side caching if you are testing with
"ping" etc -- nslookup goes direct to the DNS server so
it bypasses client cache.

To answer Ace's inquiry about "Secure Cache Against
Pollution ", this option is indeed checked. This is
obviously the default right? What's best practice when it
comes to this option?

Check it - but it is largley about MALICIOUS pollution
(or perhaps subversion) of the cache and probably unrelated.

It prevents the server from caching "extra" responses from
one DNS server that don't relate to the original query; e.g.,
a hacker could rig his OWN DNS so that when you asked
it a question, it added a FAKE address record for
www.Microsoft.com that might get cached and subsequently
used for say, Windows Update?

%SystemRoot%\Help\DNSconcepts.chm::/sag_DNS_imp_TuningAdvancedParams.htm

Also in reply to Ace's post regarding forwarders, we are
employing forwarders to reconcile non-local names, no
ISP's.

That answer is unsettling -- what specifically are you doing here?

You could be in a "forwarding loop" or some such, but I don't
see how that relates to your inability to clear the cache.

When I clear cache (Win2000), everything disappears from the
tree EXCEPT the "root ." and ".net" for the root servers.

Try re-start, its a hack but you are in a sand-box still and will
seldom clear cache in production.

 

[Ace Fekay [MVP]]

In

That answer is unsettling -- what specifically are you doing here?

You could be in a "forwarding loop" or some such, but I don't
see how that relates to your inability to clear the cache.

I think he means that he's forwarding to the ISP's, which is what we would
like to see, unless there's a delegation with child domains, but his
sandbox, from what I can tell, is still in the first domain stages.

--
Regards,
Ace

Please direct all replies to the newsgroup so all can benefit.

Ace Fekay, MCSE 2000, MCSE+I, MCSA, MCT, MVP
Microsoft Windows MVP - Active Directory

 

[Herb Martin]

I think he means that he's forwarding to the ISP's, which is what we would

like to see, unless there's a delegation with child domains, but his
sandbox, from what I can tell, is still in the first domain stages.

And I don't see how any of this prevent him from
clearing cache.

That is such a simple thing to do and test that if I saw it
then I would be sorely tempted to call it a BUG in Win2003
DNS GUI or the server.

One last idea? Is he clearing the cache on a remote server
and perhaps the messages is either going astray (doesn't seem
likely since he can view the server) or it's not responding to
a remote request.

There is something similar in WINS remote backup/restore but
they grey that out so you won't even try.

It also seems wrong to prevent this working remotely -- but maybe
it's a permission/authentication issue.

 

[Ace Fekay [MVP]]

In

And I don't see how any of this prevent him from
clearing cache.

That is such a simple thing to do and test that if I saw it
then I would be sorely tempted to call it a BUG in Win2003
DNS GUI or the server.

One last idea? Is he clearing the cache on a remote server
and perhaps the messages is either going astray (doesn't seem
likely since he can view the server) or it's not responding to
a remote request.

Nah, doesn't seem likely, as you say.

There is something similar in WINS remote backup/restore but
they grey that out so you won't even try.

It also seems wrong to prevent this working remotely -- but maybe
it's a permission/authentication issue.

Possibly, not sure. This is unusual behavior in and DNS server. I'm at my
wit's end on this one.

--
Regards,
Ace

Please direct all replies to the newsgroup so all can benefit.

Ace Fekay, MCSE 2000, MCSE+I, MCSA, MCT, MVP
Microsoft Windows MVP - Active Directory

 

[NT Canuck]

Possibly, not sure. This is unusual behavior in and DNS server.
I'm at my wit's end on this one.

hmm...I smell rubber burning! *g*

Hi Ace, we've been picking up some kind of sticky points in (??)
remote server DNS caches/replies and will advise more when/if
we get a proper handle on it. Seems that just the last week or so
there is some type of errata happening in DNS land...servers
jamming or hanging, caches getting jammed, both in win32 and
'nix versions. We suspect "malformed" ipv6 implementations but
not yet time to nail it down. Just a note/fyi...

I doubt you'll see too many errors/errata unless a very_busy
DNS server, I'm keeping an eye on event logs just in case.
(I may be wrong...but somethings changed/happening out there)

Bonus...something to read if news to you/anyone:
http://news.com.com/2100-1033_3-5055803.html

 

[Ace Fekay [MVP]]

In

hmm...I smell rubber burning! *g*

Hi Ace, we've been picking up some kind of sticky points in (??)
remote server DNS caches/replies and will advise more when/if
we get a proper handle on it. Seems that just the last week or so
there is some type of errata happening in DNS land...servers
jamming or hanging, caches getting jammed, both in win32 and
'nix versions. We suspect "malformed" ipv6 implementations but
not yet time to nail it down. Just a note/fyi...

I doubt you'll see too many errors/errata unless a very_busy
DNS server, I'm keeping an eye on event logs just in case.
(I may be wrong...but somethings changed/happening out there)

Bonus...something to read if news to you/anyone:
http://news.com.com/2100-1033_3-5055803.html

--
'Seek and ye shall find'
NT Canuck
http://ntcanuck.com BIND-PE & DNS
http://ntcanuck.com/tq/ Tips & Tweaks

Yeah, lot's of rubber here. Glad I got a new pair of boots! But spinning
wheels ain't gonna help...

Interesting about the IPv6 stuff maybe causing this. I've noticed in a
couple posts that a 216.x.x.x. number comes back for many lookups, can't
remember the exact IP, but after setting Secure Cache Against Pollution
setting, seemed to have helped one poster.

Looking forward to what you find on this. Seems to be hitting alot of folks
here with something concerning something going awry.

--
Regards,
Ace

Please direct all replies to the newsgroup so all can benefit.

Ace Fekay, MCSE 2000, MCSE+I, MCSA, MCT, MVP
Microsoft Windows MVP - Active Directory

 

[Ace Fekay [MVP]]

In

Interesting about the IPv6 stuff maybe causing this. I've noticed in
a
couple posts that a 216.x.x.x. number comes back for many lookups,
can't
remember the exact IP, but after setting Secure Cache Against
Pollution
setting, seemed to have helped one poster.

We are rewriting the DNS engines we distribute to compensate for
some of this...the freeware edition will be done as well (time
permitting).
Fact is...securing your LOCAL cache may help a little bit...but it
will
not greatly help remote A units serving weirdness from their_cache....
I suspect (given publicly released reports) there is more to this.

I haven't specific info since we jumped on internet accessibility and
accuracy of resolutions bandwagon and haven't enough resources
todo "forensics" on DNS/remote NS properly.

Looking forward to what you find on this. Seems to be hitting alot
of folks
here with something concerning something going awry.

Well...I did nail down several probes FROM port 53 udp *remotes* that
scan (theoretically) for open server ports. I say this only since 3
other
folks (including a few alt roots and isp's) have had odd readings and
errata that fit in with what we saw...but they were unable to capture
or dump the data swiftly enough to critique it more thoroughly.

[I have little time to even parse/read logs...so was odd finding
above]

****
This came in today (july 31/2003):
''We're seeing an Internet-wide increase in probing that could be a
search for vulnerable computers,'' says Wray. ''It could be a
precursor and it bears continued watching... It certainly could be
serious. It could lead to the distribution of destructive, malicious
code and it could cause considerable disruption.''

http://www.esecurityplanet.com/trends/article.php/2242891

****

hth

--
'Seek and ye shall find'
NT Canuck
http://ntcanuck.com BIND-PE & DNS
http://ntcanuck.com/tq/ Tips & Tweaks

Thanks for the heads up. Interesting. Very interesting. Maybe we should post
this as a new post here?

I'm going to post this and reference this thread and your post NT.

Thanks


--
Regards,
Ace

Please direct all replies to the newsgroup so all can benefit.

Ace Fekay, MCSE 2000, MCSE+I, MCSA, MCT, MVP
Microsoft Windows MVP - Active Directory

 

[Jonathan de Boyne Pollard]

AF> I'm at my wit's end on this one.

There is no need to be. As I pointed out to him, right at the start, there
are two caches. And as Herb also implied, clearing the one cache and not the
other produces the very symptoms of old data still being returned from test
queries.

Given that Richard Schwartz' proffered means of stopping and restarting a
service was to reboot the machine, I'd say that this is far more likely to be
such pilot error on his part; than to be some obscure bug in the DNS service
controls that you cannot guess at let alone replicate, or some vague (and
possibly nonexistent) oddity with DNS traffic that is affecting NT Canuck's
DNS server software and that might be affecting Microsoft's DNS server as
well.

 

[Ace Fekay [MVP]]

In

There is no need to be. As I pointed out to him, right at the start,
there are two caches. And as Herb also implied, clearing the one
cache and not the other produces the very symptoms of old data still
being returned from test queries.

Given that Richard Schwartz' proffered means of stopping and
restarting a service was to reboot the machine, I'd say that this is
far more likely to be such pilot error on his part; than to be some
obscure bug in the DNS service controls that you cannot guess at let
alone replicate, or some vague (and possibly nonexistent) oddity with
DNS traffic that is affecting NT Canuck's DNS server software and
that might be affecting Microsoft's DNS server as well.

True, just stymied on this one. But many other similar issues occuring
lately seem to be too coincidental to the current vulnerabilty recently
discovered.




--
Regards,
Ace

Please direct all replies to the newsgroup so all can benefit.

Ace Fekay, MCSE 2000, MCSE+I, MCSA, MCT, MVP
Microsoft Windows MVP - Active Directory

 

[Jonathan de Boyne Pollard]

AF> But many other similar issues occuring lately seem to be too
AF> coincidental to the current vulnerabilty recently discovered.

You remarked offhand earlier that cache poisoning because security was turned
off appears to be the problème du jour. However, I suspect that this is just
a local peak in the fluctuation of occurrence of this misconfiguration, and
not related to any external activity.

The problème du jour to me appears to be the "I've just set up 'split horizon'
DNS service and now my machines are having trouble with 'www.example.com.' and
'mail.example.com' because I forgot to populate my 'internal' DNS database
with data." one. (-:

 

[Ace Fekay [MVP]]

In

You remarked offhand earlier that cache poisoning because security
was turned off appears to be the problème du jour. However, I
suspect that this is just a local peak in the fluctuation of
occurrence of this misconfiguration, and not related to any external
activity.

The problème du jour to me appears to be the "I've just set up 'split
horizon' DNS service and now my machines are having trouble with
'www.example.com.' and 'mail.example.com' because I forgot to
populate my 'internal' DNS database with data." one. (-:

Good point. I would say that's one problème du jour. The one I was speaking
of is installing SP4 on DCs with resultant errors, such as long logon times,
5781's, etc. As soon as rolled back to SP3, the issues go away. So not sure
what's up with that and hard to figure out. I forwarded one of the newsgroup
issues to my MVP Lead. Haven't heard anything back yet, but it was only
Sunday when I sent it. I'll keep everyone posted on what I find out.

--
Regards,
Ace

Please direct all replies to the newsgroup so all can benefit.

Ace Fekay, MCSE 2000, MCSE+I, MCSA, MCT, MVP
Microsoft Windows MVP - Active Directory

 

[Jonathan de Boyne Pollard]

AF> The one I was speaking of is installing SP4 on DCs with
AF> resultant errors, such as long logon times, 5781's, etc. As
AF> soon as rolled back to SP3, the issues go away.

Are the symptoms similar to those described in Q329405 ?