Problems after demoting DC's

[Marphyre]

The guy before me at this company was a little DC crazy. The original
setup was 14 domain controllers (this is a network distributed over a
VPN WAN at 14 offices), approximately 170 XP workstations. I had
installed SP1 and R2 on all of the servers, and have slowly demoted all
but 2 here at the main office (that much replication traffic was just
plain unneccessary). 7 of these demoted servers are running fine. The
other 5 are having two specific problems - they will not update Group
Policy and they will not let me log into Remote Desktop using any
domain credentials - administrator or otherwise. Everything else is
running fine (they are also file/print servers, and some run a few
other tasks). An interesting point that I noticed is that if I ping
the domain name itself, sometimes it reports back one of the older
DC's. If I then run ipconfig /displaydns, under the domain name there
will be 4 servers - 2 are my current DC's and 2 are 2 of the other
servers that have been demoted. This same point will happen on
workstations also - they report the same 4 servers under my domain
name. I have cleaned everything I can find on the old DC's out of DNS
and WINS. I have tried removing one of the demoted servers from the
domain and rejoining it, and have the same problems. I have ran many
tools on these 5 servers and made sure that they are able to
communicate with the current DC's with no problems. The only problems
that I can come up with are the Group Policy and Terminal Server issues
(with TS, I can log into these servers using local users, just not
domain users). Here are the event log messages I'm getting that relate
(taken from one of the 5 demoted servers giving problems):

Application log:
Event Type: Error
Event Source: Userenv
Event Category: None
Event ID: 1054
Date: 3/30/2006
Time: 12:25:57 PM
User: NT AUTHORITY\SYSTEM
Computer: {name}
Description:
Windows cannot obtain the domain controller name for your computer
network. (An unexpected network error occurred. ). Group Policy
processing aborted.

The following two occur when I attempt to log in with Remote Desktop:

Application log:
Event Type: Error
Event Source: Winlogon
Event Category: None
Event ID: 1219
Date: 3/30/2006
Time: 2:02:50 PM
User: N/A
Computer: {name}
Description:
Logon rejected for DOMAIN\user. Unable to obtain Terminal Server User
Configuration. Error: Access is denied.

System log:
Event Type: Warning
Event Source: LSASRV
Event Category: SPNEGO (Negotiator)
Event ID: 40960
Date: 3/29/2006
Time: 10:10:38 AM
User: N/A
Computer: {name}
Description:
The Security System detected an authentication error for the server
cifs/{dc name}. The failure code from authentication protocol Kerberos
was "There are currently no logon servers available to service the
logon request.
(0xc000005e)".


I've been working on this for a few days, and so far my buddy Google
hasn't found anything, so I'm hoping someone here can help! Thanks in
advance...

 

[Herb Martin]

The guy before me at this company was a little DC crazy. The original
setup was 14 domain controllers (this is a network distributed over a
VPN WAN at 14 offices), approximately 170 XP workstations.

Not NECESSARILY a bad thing.

I had
installed SP1 and R2 on all of the servers, and have slowly demoted all
but 2 here at the main office (that much replication traffic was just
plain unneccessary).

With so few machines there probably wasn't much
replication traffic anyway -- and you could schedule
it to only happen every day or even every week if you
wished.

7 of these demoted servers are running fine. The
other 5 are having two specific problems - they will not update Group
Policy and they will not let me log into Remote Desktop using any
domain credentials - administrator or otherwise.

There are ordinary Servers now (presumably if demoted properly)
so just RESET their COMPUTER account in AD.

If that doesn't work, then unjoin and rejoin them to the domain.
(But prefer the reset if it works and try at least one reboot before
you give up on that idea.)

Also before you unjoin/rejoin make absolutely sure your DNS
is correct. VPNs frequently play havoc with people getting the
remote DNS correct.

Everything else is
running fine (they are also file/print servers, and some run a few
other tasks). An interesting point that I noticed is that if I ping
the domain name itself, sometimes it reports back one of the older
DC's.

That makes sense if the DNS server it is using is NOT yet
updated/replicated.

By the way, if the DNS is running on these former-DCs at
every site then likely you would have been better off with
them as DCs and using AD integrated DNS than removing them.

If I then run ipconfig /displaydns, under the domain name there
will be 4 servers - 2 are my current DC's and 2 are 2 of the other
servers that have been demoted. This same point will happen on
workstations also - they report the same 4 servers under my domain
name. I have cleaned everything I can find on the old DC's out of DNS
and WINS. I have tried removing one of the demoted servers from the
domain and rejoining it, and have the same problems.

Then it is likely a remnant from the DNS not being updated.

I have ran many
tools on these 5 servers and made sure that they are able to
communicate with the current DC's with no problems. The only problems
that I can come up with are the Group Policy and Terminal Server issues
(with TS, I can log into these servers using local users, just not
domain users). Here are the event log messages I'm getting that relate
(taken from one of the 5 demoted servers giving problems):

 

[Christopher Owens]

I have removed DNS from all of the servers except for the 2 remaining
DC's, so all of the workstations are getting DNS over the VPN's. I
found the two records for the demoted DC's in DNS that I missed and
removed them. Definitely my own fault overlooking those records for
that instance. But unfortunately it seems that this had nothing to do
with my real problems anyway.

I tried your suggestion of resetting the account, then I restarted the
machine, and received errors such as the following:

Event Type: Warning
Event Source: LSASRV
Event Category: SPNEGO (Negotiator)
Event ID: 40960
Date: 3/31/2006
Time: 8:20:37 AM
User: N/A
Computer: {name}
Description:
The Security System detected an authentication error for the server
ldap/{DC IP}. The failure code from authentication protocol Kerberos
was "The attempted logon is invalid. This is either due to a bad
username or authentication information.
(0xc000006d)".


So I did the next step, joined the server to a workgroup, then deleted
the computer name from AD, then rejoined the computer to the domain,
rebooted, and back to getting the same original problems.

Strangely, one of these servers decided to magically start working
again - so I'm down to 4 servers having these same problems.

 

[Christopher Owens]

Slight update - I have been able to log into Remote Desktop on one of
these servers after renaming that domain user's profile on that server.
Thus it created a new profile, and that allowed it in. After logging
in with a domain admin account, I tried gpupdate and received this
error:

Event Type: Error
Event Source: Userenv
Event Category: None
Event ID: 1110
Date: 3/31/2006
Time: 10:17:31 AM
User: NT AUTHORITY\SYSTEM
Computer: {name}
Description:
Attempt to determine whether user and machine accounts are in the same
forest failed (The interface is unknown. ).

 

[Christopher Owens]

I don't know if this has anything to do with anything at all, but I
found that if I cause Net Logon service not to turn on at startup, I
can log in with Remote Desktop with domain credentials, however Group
Policy still fails (since Kerberos can't function without Net Logon I'm
assuming), though I'm not sure how it's authenticating my domain
username without Net Logon. However, if I allow Net Logon to turn on,
then restart, it will give me Access is Denied again, same event log
errors also.

 

[Herb Martin]

I don't know if this has anything to do with anything at all, but I
found that if I cause Net Logon service not to turn on at startup, I
can log in with Remote Desktop with domain credentials, however Group
Policy still fails (since Kerberos can't function without Net Logon I'm
assuming), though I'm not sure how it's authenticating my domain
username without Net Logon. However, if I allow Net Logon to turn on,
then restart, it will give me Access is Denied again, same event log
errors also.

Net Logon should ONLY run on a DC and it MUST
run on a (functioning) DC.

Chances are you still have some (serious) DNS issues.

What are you DCDiag results on every remaining DC?

Make sure you clients are using STRICTLY the (internal)
DNS which can resolve ALL DCs etc.

They must NOT be using a mixture of internal/external DNS.

DNS Clients include all DCs and even DNS servers.

Remove any leftover DNS records from the removed DCs.

 

[Paul Bergson]

Herb that is incorrect. NetLogon should run on all machines desktops and
dc's.

From:
http://technet2.microsoft.com/Windo...5eed7068-f0f8-4650-ad8a-5c74ca6479571033.mspx

Services\Netlogon
Product(s):
a.. Windows Server 2003 R2
b.. Windows Server 2003 with SP1
Updated: March 28, 2003
HKLM\SYSTEM\CurrentControlSet\Services

Description
The Netlogon subkey stores information for the Net Logon service.

The Net Logon service verifies NTLM logon requests, and it registers,
authenticates, and locates domain controllers. Also, to maintain backward
compatibility, Net Logon manages replication of the user account database to
back up domain controllers running Windows NT 4.0 and earlier.

Note

. For Windows Server 2003, Net Logon manages replication only when
replication involves servers running Windows NT 4.0 and earlier. Net Logon
for Windows Server 2003 does not manage replication between two servers
running Windows Server 2003.



--

Paul Bergson MCT, MCSE, MCSA, CNE, CNA, CCA
http://www.pbbergs.com

This posting is provided "AS IS" with no warranties, and confers no rights.

I don't know if this has anything to do with anything at all, but I
found that if I cause Net Logon service not to turn on at startup, I
can log in with Remote Desktop with domain credentials, however Group
Policy still fails (since Kerberos can't function without Net Logon I'm
assuming), though I'm not sure how it's authenticating my domain
username without Net Logon. However, if I allow Net Logon to turn on,
then restart, it will give me Access is Denied again, same event log
errors also.

Net Logon should ONLY run on a DC and it MUST
run on a (functioning) DC.

Chances are you still have some (serious) DNS issues.

What are you DCDiag results on every remaining DC?

Make sure you clients are using STRICTLY the (internal)
DNS which can resolve ALL DCs etc.

They must NOT be using a mixture of internal/external DNS.

DNS Clients include all DCs and even DNS servers.

Remove any leftover DNS records from the removed DCs.

--
Herb Martin, MCSE, MVP
Accelerated MCSE
http://www.LearnQuick.Com
[phone number on web site]

 

[Paul Bergson]

Open up DNS and look at the root of your domain and see if you see any
removed servers still defined as dns servers. These don;t go away when you
demote a server, you have to manually remove them. Right click on your
domain, select properties, select the name servers tab and remove any dns
servers that are no longer serving up dns.

--

Paul Bergson MCT, MCSE, MCSA, CNE, CNA, CCA
http://www.pbbergs.com

This posting is provided "AS IS" with no warranties, and confers no rights.

 

[Herb Martin]

Silly me -- I guess I was thinking of the NetLogon shares.

--
Herb Martin, MCSE, MVP
Accelerated MCSE
http://www.LearnQuick.Com
[phone number on web site]

Herb that is incorrect. NetLogon should run on all machines desktops and
dc's.

From:
http://technet2.microsoft.com/Windo...5eed7068-f0f8-4650-ad8a-5c74ca6479571033.mspx

Services\Netlogon
Product(s):
a.. Windows Server 2003 R2
b.. Windows Server 2003 with SP1
Updated: March 28, 2003
HKLM\SYSTEM\CurrentControlSet\Services

Description
The Netlogon subkey stores information for the Net Logon service.

The Net Logon service verifies NTLM logon requests, and it registers,
authenticates, and locates domain controllers. Also, to maintain backward
compatibility, Net Logon manages replication of the user account database
to back up domain controllers running Windows NT 4.0 and earlier.

Note

. For Windows Server 2003, Net Logon manages replication only when
replication involves servers running Windows NT 4.0 and earlier. Net Logon
for Windows Server 2003 does not manage replication between two servers
running Windows Server 2003.



--

Paul Bergson MCT, MCSE, MCSA, CNE, CNA, CCA
http://www.pbbergs.com

This posting is provided "AS IS" with no warranties, and confers no
rights.

I don't know if this has anything to do with anything at all, but I
found that if I cause Net Logon service not to turn on at startup, I
can log in with Remote Desktop with domain credentials, however Group
Policy still fails (since Kerberos can't function without Net Logon I'm
assuming), though I'm not sure how it's authenticating my domain
username without Net Logon. However, if I allow Net Logon to turn on,
then restart, it will give me Access is Denied again, same event log
errors also.

Net Logon should ONLY run on a DC and it MUST
run on a (functioning) DC.

Chances are you still have some (serious) DNS issues.

What are you DCDiag results on every remaining DC?

Make sure you clients are using STRICTLY the (internal)
DNS which can resolve ALL DCs etc.

They must NOT be using a mixture of internal/external DNS.

DNS Clients include all DCs and even DNS servers.

Remove any leftover DNS records from the removed DCs.

--
Herb Martin, MCSE, MVP
Accelerated MCSE
http://www.LearnQuick.Com
[phone number on web site]

 

[Christopher Owens]

I was convinced that it was a DNS error of some kind as well for a long
time, but I have been through each DNS "folder" at least 5 times
looking for something that should not belong. The only NS records
point to the correct DC's that are running DNS, and everything under
_msdcs, _sites, _tcp, _udp, DomainDnsZones, and ForectDnsZones looks
correct - the only 2 DC's that are pointed to are the ones that are
left. I also don't think that it's DNS anymore because it is still the
same 4 machines that are giving these errors - I would think if it was
DNS that other machines would give me problems also at some point.
Unfortunately no more of them have magically started working. DCDiag
returns everything passing for both of my DC's - even when looking at
it from one of the "broken" machines.
And just for the record - these machines do not have the NETLOGON
shares

 

[Christopher Owens]

Well, for anyone else who may be interested, I finally got the problems
resolved today. It turns out that the fixes weren't related to each
other, although a site I found that fixed one problem eventally led me
to the fix for the other. For the Remote Desktop issue, I had to force
Kerberos traffic to use TCP instead of UDP, which involved setting a
DWORD value at the registry subkey:
[HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters]
"MaxPacketSize"=dword:00000001
After a restart I could log in with Remote Desktop domain credentials
again.

The fix to Group Policy seems to be that it was detecting a slow link,
and therefore ignoring the policies. To fix this required these DWORDs
added to the registry:
[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\System]
"GroupPolicyMinTransferRate"=dword:00000000

[HKEY_CURRENT_USER\Software\Policies\Microsoft\Windows\System]
"GroupPolicyMinTransferRate"=dword:00000000
And after a restart group policy began working fine. Wish I knew why
this happened on these 4 computers and none of the rest, but I'm just
glad to have a solution now! Thanks to those who tried to help.