Re: Slow boot at "Applying Computer Settings"

[Paul Williams [MVP]]

Any ideas what is going on here?

Not really. So I've crossposted this into the GPO groups as well.
Hopefully someone there can help...

That seems to be pausing when it hits the (a) WMI provider. Are you using
the new WMI Filters with this GPO? That is, are you using a WQL query to
filter this policy?

Sorry, if that's not much help.

 

[Will Gillen]

We are not really doing anything in the GPO. We are just using the
default domain policy since setting up Active Directory. Since we
migrated to AD, we have seen this slow bootup time at "Applying
Computer Settings", and have not modified the Policy hoping to keep it
as small as possible. The WMI provider is just the WMI service that XP
starts. We are not doing any specific WQL queries.

I would be willing to send someone a complete dump of the userenv log
file, but I don't want to post it publicly since it might contain
sensitive information about the network configuration.

Thank you for helping on this. By the way the tip about turning on the
userenv verbose logging was AWESOME! Thanks.

-- Will G.

 

[Ken B]

If it hangs at Applying Computer Settings... for a while, it may be a DNS
issue. Ensure you're not pointing any domain computers to external DNS
servers--they should just be pointing to your DC's, and set up forwarding on
the DC's.

Ken

 

[Will Gillen]

Ken --

Thank you for that information, but I am very sure that we have DNS
setup correctly. As I stated in the original post, we have 2 Internal
DNS servers running Windows 2003 and they are DCs. ALL of our XP
clients are configured by DHCP to point to those 2 DNS servers, and
only those 2 DNS servers. The 2 DNS servers are setup to forward to
our outside DNS servers. It is not a DNS issue (at least not a problem
of pointing our XP clients to external DNS servers).

Do you have any other ideas that would cause it to hang?

 

[Ken B]

That's really the only reason I've ever come across that it hangs at
Applying Computer Settings... Maybe one of the MSFTs or MVPs has a better
clue than me?

Ken

 

[Darren Mar-Elia]

Your best bet is to enable verbose userenv logging and check out that log to
see where in GP processing the hang is occurring. It should be very easy to
spot since each event in the log is timestamped.

--
Darren Mar-Elia
MS-MVP-Windows Server--Group Policy
Check out http://www.gpoguy.com -- The Windows Group Policy Information Hub:
FAQs, Whitepapers and Utilities for all things Group Policy-related

 

[Will Gillen]

Yes, we have turned on verbose userenv debugging, and here is the
result. But, I still have no idea where the actual problem is. Any
ideas:

USERENV(1ec.5cc) 13:44:36:391 ProcessGPO: Found file system path of:
<\\nsuok.edu\sysvol\nsuok.edu\Policies\{31B2F340-016D-11D2-945F-00C04FB984F9}>
USERENV(1ec.5cc) 13:45:58:362 ProcessGPO: Found common name of:
<{31B2F340-016D-11D2-945F-00C04FB984F9}>

Notice the over 1 minute wait between the "Found file system path..."
and "Found common name..." entries. What could be happening that is
slowing this down?

 

[Darren Mar-Elia]

I think at that point in the cycle its querying the GPC in AD for info about
the GPO. I'm not sure why AD would sit there for a minute, esp. since it
seems to be getting the gpcFileSysPath attribute. I think it might be a good
idea to run a network trace from this workstation and capture the requests
to AD. Maybe its getting a referral or something.

 

[Will Gillen]

I captured all packets to/from this client's IP during a boot. I
noticed some interesting things:

The client would properly query DNS looking for the DC, the DNS server
would return the correct DC and the client would initiate the normal
LDAP (port 389) communication. Then, later (say 1-2 minutes later in
the packet capture), there were instances where the client would lookup
in DNS the IP of a DC at a different SITE, then ping the DC, then (this
is the strange part), it would attempt to contact the DC on port 80????
Why in the world would the client attempt to contact other DCs on port
80? Especially since the client has already located the DC for it's
site in DNS and has already started communicating with it prior to
these packets?

This just keeps getting stranger and stranger by the minute. I think
I'm almost ready to throw in the towel and pay the 245.00 to get
Microsoft Product support on this issue, it is getting out of hand....

If anyone has anything to contribute or has any other suggestions, I
will be more than willing to keep trying. Please help.

Thank you for everyone's help so far. This is truly a strange one!

-- Will Gillen

 

[Darren Mar-Elia]

Sounds like its having a hard time binding to the LDAP server. Can you see
the bind success? Do you see any successful LDAP queries or does it just
not even try and the ping another site? It could be getting a referral--do
you see any reference to that in any of the response packets from the DC. If
its a Network Monitor capture, feel free to post the capture file here--I'd
be glad to take a look at it.

 

[Will Gillen]

I do see proper LDAP query and return from the DC. I don't believe it
is an LDAP problem, but I guess it could be? From what I can see the
LDAP communication looks good with send/recieve working as expected. I
would post the capture but I don't have a way to attach anything to
this message, plus it is not a netmon trace; it is a cisco ".enc"
packet capture. You would need a tool like "Observer" to view the
results. You can download an evaluation version at their website.
(http://www.networkinstruments.com/)

If you still want to view the file I would be more than happy to email
it too you. Just send me your address to:
gille001<@>nsuok.edu (take out the <>).


Thank you for continuing to help on this. I have been stumped for
almost 2 weeks now, and I'm getting a lot of pressure from many people
on our campus to get a resolution to this problem. I believe that
tomorrow morning I will be putting in a call to Microsoft. I already
have the approval (given the widespread nature of this problem on our
network). So, thank you again for helping. If (when) I get a
resolution, I will be more than happy to share it with everyone,
because I have been to many, many forums and listservs that have
sysadmins still fighting this problem and have been for months...

-- Will Gillen

 

[Will Gillen]

Darren -- I sent you an email with the packet capture. It looks like
the client (for some reason), queries DNS, then pings a DC, then
attempts some port 445 queries, then attempts some port 80
communication with the remote DC. I still can't figure out how the
client is even learning the DC name. It is not configured that way at
all in Sites and Services???

Let me know if you think you can help on this.

Thanks.

-- Will G.

 

[Darren Mar-Elia]

I'm looking through it now Will. Hopefully will have something soon.

 

[Guest]

I have just had a similiar problem...it turned out to be a problem with Sites
and Services. The script in our GPOs are mapped using the DFS share for our
domain. for ex. \\microsoft.msft.com\sysvol\domain name\scipts...clients can
access this share from any of the domain controllers....by default, clients
are to pick a DC from within its own site, then if it can't find one, try
another DC in another site.

what happened is we made some IP addressing changes to one of the remote
networks, the IP range began to conflict with the IP subnets for our site.
Machines in our site, began to think the the DC in the remote site was
actually in our site, and began to access the DFS sysvol share on the remote
DC when executing the GPO logon scripts. Because the connection to the remote
DC is slower, clients were hanging during GPO processing.

I'm not sure if any of this applies to you, but check it out.....

 

[Guest]

Gosh, you would think I was in your organization (or your
shoes) on this problem. I have the duplicate issue and
just started this job 3 weeks ago. They have been having
this problem for 3 months though (before I got there).
Will, I have also checked what seems like ~100 different
websites and spent countless hours poring over list-serv's,
etc. I am starting to think that this is a bug / nightmare
that Microsoft does not want to deal with until the next
SP. My XP machines are randomly taking from 4-10 minutes
to login but my 98 machines are taking 10 seconds.
Let me know if you find the fix, Will
I am desperate to fix the problem and get it off
my place.

Brent Carey
970-246-3486 (W) or 970-522-4420 (H)
(e-mail address removed) or

 

[ixeye]

Further addition: I changed the XP clients to only point to the
internal DNS
in network configuration but I had a funny feeling so I checked
regedt32 and did a search for the external DNS and it is still there. I
reboot the machine and
check for the IP again in the registry and it is still there.
How can I point the xp client to the local internal DNS ip only???

 

[ixeye]

Further addition: I changed the XP clients to only point to the
internal DNS
in network configuration but I had a funny feeling so I checked
regedt32 and did a search for the external DNS and it is still there. I
reboot the machine and
check for the IP again in the registry and it is still there.
How can I point the xp client to the local internal DNS ip only???

 

[Will Gillen]

How did you change the XP clients to point only to internal DNS? Do
you use DHCP, or did you change the DNS settings from within the client
TCP/IP properties pages?

When you used regedt32 and did a search for the external DNS, what
specific keys did you see the external DNS in? Because configuration
information is retained in keys like "ControlSet001", "ControlSet002"
for backup/LastKnownGoodProfile. The DNS settings you want are in
"CurrentControlSet".

If you manually change the DNS entries for primary and secondary DNS in
TCP/IP properties and reboot the client, those are the DNS servers your
client will use.

You can test that by making the changes, and then log into the client
and perform an NSLookup for some of your internal AD server like this:
at a command prompt, type "nslookup"
That will put you at the nslookup commandline, then type the following:
set q=srv
_ldap._tcp.gc._msdcs.yourdomain.com


You should see a list of DCs that provide LDAP authentication.

 

[Will Gillen]

Drew,

I think the sysvol access that you mention is related to our problem,
because when I did a packet sniff from the client, I see that it is
trying to contact DCs at other remote sites across our WAN during the
GPO processing. During the initial DNS request, it is finding the
correct DC at its local site, but only during the GPO processing does
it start trying to contact servers across the WAN. And when I say
servers, I mean server(s), it tries like 4 different servers on four
different Sites?????

I sent the packet trace to Darren Mar-Elia, and he mentioned a few
things that I'm looking into. For one: it looks like the client first
tries to setup and SMB connection to a server, then that fails, then it
looks up the server in WINS and tries to connect over NetBIOS (445),
then that fails, and if finally tries a NetBIOS broadcast (which as we
all know takes nearly forever to time-out). Finally, it even tries the
"WebClient/WebDAVRedirector"!!! over port 80 (http). When ALL of that
fails, it moves on to another server in the list.

So as you can see, if it tries more than one server at a remote site,
this could cause the process to easily take 4-10 minutes.

However, there is one slight problem... I have looked through our
Sites and Services and don't see any subnet conficts?? I'm going to
continue to double-check with a fine-tooth comb and make sure that
maybe my subnetmasks aren't misconfigured or something...

I'll keep you posted, and thank you very much for sharing this
information!

-- Will G.

 

[Will Gillen]

I think I'm really close to figuring out the root problem here (thanks
to everyone's help!):

I've been continuing to look at the packet trace from the client, and I
found the part of the process where the client asks the DNS for the
SYSVOL share so that it can find the policies and scripts to apply.
Now, I would believe that AD should return a list of servers hosting
the SYSVOL share that are in the client's SITE, but it looks like it
DOESN'T (at least in our environment). Whent he client requests
information about the location(s) of the SYSVOL share it goes something
like this:

Client: "Where is \\nsuok.edu\sysvol"?
ADServer: "Ok, here's a list of servers its on:
\\nsum-ad1.nsuok.edu\sysvol, \\nsuba-ad2.nsuok.edu\sysvol,
\\nsut-ad1.nsuok.edu\sysvol, etc., etc."
Client: "Hey, DNS can you tell me how to get to nsum-ad1.nsuok.edu?"
DNSServer: "Sure its 192.168.xxx.xxx (nsum-ad1.nsuok.edu".
Client: "Ping nsum-ad1.nsuok.edu?"
nsum-ad1.nsuok.edu: "I'm here!"
Client: "Can I connect to you through port 139"?
Firewall: "Sorry, that server is in a different site across a WAN, and
I don't allow traffic on port 139 to that site."
Client: "Ok, can I connect through port 445"?
Firewall: "Nope, I don't allow traffic over that port either. Any
other ideas?"
Client: "Hmmmm, let me broadcast and see if I can find that dang
server."
Firewall: "Hey, look buddy, I'm not doing broadcasts either, why don't
you just give up."
Client: "No way, I'm not giving up, I've got the WebDAV Redirector, let
me try port 80, HaHaHa!".
Firewall: "Hmmmm, port 80 looks innocent enough, ok 'move along'".
nsum-ad1.nsuok.edu: "Sorry charlie, I don't have port 80 open for
business. "
Client: "Crap, ok, what's next on the list. Oh, its:
nsuba-ad2.nsuok.edu, let me try that one. Hey, DNS can you tell me
where nsuba-ad2 is?"
DNS: "Sure......."
-- and this process continues through several servers at the wrong
sites, until alas: --

Client: "Whew this is getting tiring... Ok, DNS do you know where
nsut-ad1 is?"
DNS: "Yes, its 192.168.xxx.xxx".
Client: "Ping nsut-ad1".
nsut-ad1: "I'm Here!"
Client: "Can I connect through SMB or port 445, or 139, or what".
nsut-ad1: "Sure, there's no firewall between you and me cause we are
ACTUALLY in the SAME SITE!"
Client: "Finally!!!! Do you suppose I could access your sysvol share,
please?"
nsut-ad1: "Sure, here you go buddy".

The end.

Now, I just need to improve the story and figure out why the heck AD is
even telling the client about the servers hosting SYSVOL in other sites
in the first place. This I believe is what is taking the clients so
long.

-- Will Gillen
-- NSU System Admin