Sniffer information to track LSASS activity.

[Guest]

I have a W2K DC which is showing unusual LSASS cpu activity. I had previously
posted this request to the w2k.network group and it was suggested that the DC
in question likely was infected by a worm. I do not believe this to be true
for the following reasons.

-the unusual activity was first spotted on another DC and moved to this DC
after that first DC was rebooted (I rebooted the first DC to see if a reboot
would clear the unusual activity, I noted the activity moved to this DC
before the first had even finished coming back up).

-the unusual activity has not has not appeared on other DCs even though 6
other DCs sit in the same network subnet.

-no unusual network related activity is seen on the effected DC during each
event (in either direction, that includes increase in activity or sequential
port scanning activities).

-the activity appears more like a network re-try than a scan activity, that
is LSASS activity peaks the CPU at 98% for about 10sec (it was less on the
first DC because that DC has faster/mulitple cpus) then idles, repeating in
about 60sec intervals.

-all DCs in site are patched the same, that is within 48hours of any
security patch release (we wait 24hours just to see if the patch will be
changed or recalled).

-DCs are not used interactively (no mail, no web, nothing).

-there are other member servers in the same subnet with active firewalls and
yet no firewall or AV software on those machines have detected worm activity.

-I have disconnected the DC in question (not a reboot, a disconnect, process
stayed in memory) and the activity stopped for about 16hours.

-I have rebooted the DC in question and again the activity stopped over night.

This, could be caused by a worm infected client, but if so, then why don't
any other DCs show this activity. And why wouldn't any of the firewalled
member or standalone servers in the same subnet show netbios access attempts.

I think it's caused by a mis-configured piece of software attempting to
access LSASS inapproprately.

I have a sniffer capturing packets sent to this machine, however, since this
DC also provides DNS and WINS functionality besides being a DC, it's hard to
find anything useful in the captures. The DC as you expect talks to a lot of
machines a lot of ways.

So, what I was wondering is if anyone can tell me what to filter my sniffer
captures on to find out the start of transaction packet for any machine
attempting to hit the LSASS service. If I can find that, I can make a list
of machines which appear to be starting transactions every 60 secs against
that host.

 

[Steven L Umbach]

That will be tough to track down on a busy server. Assuming you have
auditing of account logon events enabled in Domain Controller Security
Policy look in the security logs of the domain controllers for anything
unusual such as failed logon attempts from a domain computer/user at the
time this is happening. On that domain controller enable auditing of logon
events for success and failure, and for object access and privilege user for
failure only. Be sure the size of the security log is at least 10MB. It is
normal to sometimes see failed access for object access and privilege use
but if you see a pattern that correlates to the high CPU usage it may give a
clue. Check Event Viewer for anything unusual and run the support tools
netdiag and dcdiag on that domain controller to check for domain
configuration health and network connectivity.

SysInternals makes a few tools that may help you. In particular Process
Explorer, TCPView, and possibly filemon. You can run Process Explorer and
TCPview and they will update the screen fairly frequently. Process Explorer
will show running processes and CPU usage for each. You can also check the
properties of a process to see what services are related to it and the
tcp/ip usage of it [you can use that info for netmon possibly]. You will
probably see that lsass is related to more than a few services. You may be
able to track something down and it will show you what ports are used. Maybe
you will see another process start when the CPU useage rises. TCPView will
display process to port activity in a GUI. This may be very difficult with a
lot of traffic to the domain controller but easy enough to try for a
ile. --- Steve

http://www.sysinternals.com/ntw2k/freeware/procexp.shtml

 

[Roger Abell]

If it is Lsass that cycles, and it moved from DC to DC, and you
work off assumption that this is triggered from outside (stopped
when DC isolated), then as a first guess why not just limit what
is captured to only the DC and GC ldap traffic?
DC ldap on tcp 389 ldap over ssl on 636
GC ldap on tcp 3268 ldap over ssl on 3269

Hunt in the dark method, but it also divides and conquers
DC/GC traffic fairly effectively, i.e. if these turn up not to
be involved, they you could focus on the other roles of a DC
(other things than the DSA running in the LSASS space).

 

[Karl Levinson, mvp]

I think it's caused by a mis-configured piece of software attempting to
access LSASS inapproprately.

I have a sniffer capturing packets sent to this machine, however, since this
DC also provides DNS and WINS functionality besides being a DC, it's hard to
find anything useful in the captures. The DC as you expect talks to a lot of
machines a lot of ways.

So, what I was wondering is if anyone can tell me what to filter my sniffer
captures on to find out the start of transaction packet for any machine
attempting to hit the LSASS service. If I can find that, I can make a list
of machines which appear to be starting transactions every 60 secs against
that host.

A good sniffer will let you write a filter on what traffic you capture
and/or what traffic it displays after the capture. I can't tell you exactly
how to do this because 1) you haven't said what sniffer you're running and
2) this is a Microsoft support newsgroup, and you're probably not using a
Microsoft sniffer.

www.ethereal.com is a good and popular sniffer for Windows. Writing filters
for ethereal can be a little esoteric, but there is plenty of information in
Google. In your case, I would suggest writing capture filters before
starting the capture to filter out traffic you don't want to see.. DNS
lookups are mostly done on UDP port 53, and I'm guessing the WINS traffic
you are seeing is mostly on UDP 137? Do a capture with that traffic
filtered out, and if you're still getting large amounts of "uninteresting"
traffic, write additional filters and re-capture.

Capturing the network traffic may be useful, but it is possible that you
might not be able to determine the cause of the problem from network traces.

 

[Guest]

First, Thanks to all of you for responding.

Per your recommendations I have done the following.
- Enabled specific auditing of failures for certain events on the DC in
question.
- Downloaded and tried both Process Explorer and TcpView.
- -Examined/modified my capture filter by selecting on specific MS TCP/UDP
ports.

Yesterday (Saturday) I rebooted the DC in question. So far (Sunday night)
more than 24hours later the unusual CPU activity associated with this event
has not re-occurred (which tells me that this event is not caused by any
systems normally running on the weekend (i.e. the 24x7 systems) so I have not
been able to use any of your advice yet.

Although at this point I have nothing to report per your recommendations
above, I do have an one additional question at this time. Per the response I
got to my original posting on this issue, do any of you believe it is good
practice to install Anti Virus software on a Domain Controller. And if yes,
are there any caveats to doing so.

Thanks again for your suggestions. - bill

 

[Ezra Herman]

do any of you believe it is good
practice to install Anti Virus software on a Domain Controller. And if yes,
are there any caveats to doing so.

I strongly encourage you to install AV software on a DC, or on any
other server for that matter. As far as caveats, this should go
without saying, but the AV package you use should be designed for a
server, not a workstation.

You shouldn't need all the components of a workstation AV package. For
example, you should not need an email-checking component. But the
package you install must be able to do a) real-time scans, b) full
system scans, c) update itself and d) report status and events.

-Ezra Herman

 

[Roger Abell]

Make sure the software is not just server but DC compliant.
Some AV has been known to raise heck with FRS and also
with the ntds.dit