(another) DC replication problem

[Brandon McCombs]

Hey guys,

I've seen a few other posts about DC replication issues, the newest one
in my list being the error I get, however I know my problem isn't
related to DNS.
I'm actually having other issues and they may all be related but I don't
know. I have the replication issue due to a custom dc policy and it
also causes servers to not be able to grab their policies if I apply it
on the first DC before the other servers get their policy settings.

Basically my setup is that I have a 2 separate (separate networks)
domains with 2 domain controllers and 2 file/print servers ( as far as
servers are concerned) on each domain. I created a custom DC policy in
AD and it works fine for a single DC and when no other servers are in
the domain. WHen 2 DCs exist (as well as fileprint servers) I have to
make sure the 2nd DC and the fileprint servers get updated with their
respective policy setings or otherwise they are never allowed to access
the first DC. After they get their policy then I can run gpupdate on
the first DC so that it gets its settings. That problem exists on both
of thos domains b/c I used the same policies. On one of those domains I
also have ldap replication issues because of something in my custom DC
policy I believe. For some reason ldap replication using Sites and
Services generate access denied errors. File replication service errors
appear as well. Sometimes all 3 default parittions in the directory
fail to replicate, other times the schema and configuration parittions
replicate fine but the main dc component level of the directory gets
access denied (i noticed that when I was fiddling around, not sure what
made 2 out of 3 work compared to when they would sometimes all fail to
replicate).

I know it's not a DNS issue because if I use the default DC policy
everything works. Licenses are also not replicating across the DCs
(this licensing problem is on both domains, the ldap replication is only
on one oddly enough) which may be related to this issue. Basically, I
need to know where is the access denied coming from concerning the ldap
replication. I've looked at permissions on the partitions in AD (using
ADSI) that are being replicated and they are correct. Since the default
DC policy works fine I figured it's just something in my custom policy
but I haven't hit upon what it is yet. When I lookup this issue on
google and visit websites none of them ever mention an issue with a
setting in User Rights Assignment or Security Options in the DC policy
other than Enterprise Domain Controllers needs to be an entry for
"Access this computer from the network" and Administrators have to be
listed for "Give user and machine accounts delegation control" or
something like that. Both of those settings are correct in my policy so
I don't know why else I get access denied.

The relevant services that are running are :
File replication service
licensing logging (for the license issue I brought up)
DFS and
MS software shadow copy (which i just turned on and may have been the
issue but I haven't confirmed that yet).

Am I missing one , like Background Intelligent transfer service or
anything like that? Unneeded services are disabled but maybe i disabled
too many.

I've been testing by opening up AD Sites and Services and forcing a
replication. It fails mainly when the 2nd DC has to contact the 1st DC
to do a replication from 1 to 2 (although sometimes I was good enough to
make it fail both ways but i dont know how). I've tried to run repadmin
and replmon and they all show the access denied error so that has pretty
much been drilled into my head, except I don't know what is actualy
being accessed other than the schema and configuration CNs in LDAP and
their permissions are okay so I don't know what else the problem is. I
also know the 2nd DC had issues being added to the domain because of my
policy and we disabled it and was able to get the 2nd DC added. It
seemed to be fine once I got all the servers with their policy settings
in teh right order. I checked the userAccountControl value for the 2nd
DC and it is the correct one for a DC.

The replication issue is a big one and if we fix that then it may fix
the issue where servers can't get their policy updates unless the first
dc has the custom dc policy disabled. That second issue is a problem
due to the fact if a server has to be rebuilt then for a few minutes the
custom DC policy must be disabled and the default DC policy enabled so
that the new server can have its settings updated and then we have to
reapply the custom DC policy. That presents a security issue.

Oddly enough, we can add workstaions to the domains w/o having a policy
yet and when I run gpupdate on them they can grab the workstation policy
just fine. Sorry for the long post as well but I wanted to try to
include as much info as possible.


thanks for any extra input. this is driving me crazy
Brandon

 

[Herb Martin]

Hey guys,

I've seen a few other posts about DC replication issues, the newest one
in my list being the error I get, however I know my problem isn't
related to DNS.

DNS or Active Directory?

Once DNS is integrated into AD, then such problems are AD
issues.

BUT, almost all AD replication issues are really based on DNS
configuration problems unless you have a broken network or
some sort of firewall filtering that interferes.

Pure DNS replication issues are very uncommon -- again, unless
you have bad IP, routing, hardware, or firewall issues which
really aren't a "DNS" problem per se. DNS is then just a symptom
of a bigger problem.

I'm actually having other issues and they may all be related but I don't
know. I have the replication issue due to a custom dc policy and it
also causes servers to not be able to grab their policies if I apply it
on the first DC before the other servers get their policy settings.

That would be a very odd policy. What does it do?

Basically my setup is that I have a 2 separate (separate networks)
domains with 2 domain controllers and 2 file/print servers ( as far as
servers are concerned) on each domain.

What do you mean by "separate networks"? Obviously if they don't
route, then you cannot replicate.

If they do route, then this is one internetwork, which you might call
multiple Sites or Locations.


Instead of rampling on and one, please tell us precisely what the
symtoms are and if you really believe the GPO is interferring tell
us what you put into it.

Perform the obvious tests with DCDiag, ping (number AND name),
RepAdmin or ReplMon, DNSLint etc.

Here is DNS for AD:

--
DNS for AD
1) Dynamic for the zone supporting AD
2) All internal DNS clients NIC\IP properties must specify SOLELY
that internal, dynamic DNS server (set.)
3) DCs and even DNS servers are DNS clients too -- see #2
4) If you have more than one Domain, every DNS server must
be able to resolve ALL domains (either directly or indirectly)

netdiag /fix

....or maybe:

dcdiag /fix

(Win2003 can do this from Support tools):
nltest /dsregdns /serverC-ServerNameGoesHere
http://support.microsoft.com/kb/q260371/

Ensure that DNS zones/domains are fully replicated to all DNS
servers for that (internal) zone/domain.

Also useful may be running DCDiag on each DC, sending the
output to a text file, and searching for FAIL, ERROR, WARN.

Single Label domain zone names are a problem Google:
[ "SINGLE LABEL" domain names DNS 2000 | 2003 microsoft: ]

 

[Brandon McCombs]

DNS or Active Directory?

DNS. It's not DNS because if I use the default DC policy things work just
fine.

Once DNS is integrated into AD, then such problems are AD
issues.

Our DNS is hosted on Unix servers with dynamic updates turned off. All
machines were put into the dns records and the service records were entered by
taking the entries from netlogon.dns and putting them into the appropriate dns
records.

BUT, almost all AD replication issues are really based on DNS
configuration problems unless you have a broken network or
some sort of firewall filtering that interferes.

Pure DNS replication issues are very uncommon -- again, unless
you have bad IP, routing, hardware, or firewall issues which
really aren't a "DNS" problem per se. DNS is then just a symptom
of a bigger problem.

well it's not DNS replication. It's LDAP replication within ADS and the error
is Access denied so the DCs are finding each other just fine.

That would be a very odd policy. What does it do?

Uh, what do you mean? I'm "limited" to the options within the policy template
to begin with. It's not like I'm doing anything extra. I've set security
settings, disabled unneeded services, turned on a few things in the Admin
templates section, set restricted groups (but matched them up with what the
groups should have so I didn't lock anyone out like Administrator). That's
all I can think of right now w/o being at work to look at the policy.

What do you mean by "separate networks"? Obviously if they don't
route, then you cannot replicate.

I'm not replicating across the 2 domains. I'm replicating within the domains
which happen to be on differnt networks and don't communicate with each other.

The 2 DCs in 1 domain replicate with each other and the 2 DCs in the other
domain replicate with each other. That's all I meant.

If they do route, then this is one internetwork, which you might call
multiple Sites or Locations.

Instead of rampling on and one, please tell us precisely what the
symtoms are and if you really believe the GPO is interferring tell
us what you put into it.

I already stated the symptoms by running teh tools you already mentioned.
I've ran repadmin and replmon. Both state access denied when replication is
attempted. Sometimes I get acess denied on all 3 partitions that replication
occurs on (cn=schema; cn=configuration; and dc=mycompany,dc=com) and other
times the schema and configuration partitions replicate fine. As stated in my
original post, I'm not sure yet what I do when I'm trying to fix the problem
that cause 2 out of the 3 partitions to replicate fine sometimes and other
times fail. Also as stated before, DNS is working fine, the machiens can ping
each other. I can simply apply the default DC policy or change the order of
how I apply my policy by making the 2nd DC get the policy first and then the
first DC and things work fine so I know it's confined to something that I've
set in the custom DC policy.

Perform the obvious tests with DCDiag, ping (number AND name),
RepAdmin or ReplMon, DNSLint etc.

I already did use repadmin and replmon and reported the results in the first
message.

Here is DNS for AD:

--
DNS for AD
1) Dynamic for the zone supporting AD
2) All internal DNS clients NIC\IP properties must specify SOLELY
that internal, dynamic DNS server (set.)
3) DCs and even DNS servers are DNS clients too -- see #2
4) If you have more than one Domain, every DNS server must
be able to resolve ALL domains (either directly or indirectly)

netdiag /fix

...or maybe:

dcdiag /fix

(Win2003 can do this from Support tools):
nltest /dsregdns /serverC-ServerNameGoesHere
http://support.microsoft.com/kb/q260371/

Ensure that DNS zones/domains are fully replicated to all DNS
servers for that (internal) zone/domain.

Also useful may be running DCDiag on each DC, sending the
output to a text file, and searching for FAIL, ERROR, WARN.

I already know my error. If you had read the full post you may have known
that. I have a feeling you didn't read everything I included but maybe you
did. My policies were copied from one domain to another so that probably
explains why both domains are having similiar issues however I doubt DNS is
messed up the same exact way on both domains because different DNS servers are
used being that the domains are on different networks and we would have had to
mess up DNS the same way in both instances which isn't likely. I just need to
know what is being accessed during ldap replication so I know what to look at
to fix the access denied issues. Permissions within LDAP have not been
modified so I figured it's a service that I disabled or a security permission
in a policy that was set that is causing issues.

Single Label domain zone names are a problem Google:
[ "SINGLE LABEL" domain names DNS 2000 | 2003 microsoft: ]

I created a custom DC policy in
AD and it works fine for a single DC and when no other servers are in
the domain. WHen 2 DCs exist (as well as fileprint servers) I have to
make sure the 2nd DC and the fileprint servers get updated with their
respective policy setings or otherwise they are never allowed to access
the first DC. After they get their policy then I can run gpupdate on
the first DC so that it gets its settings. That problem exists on both
of thos domains b/c I used the same policies. On one of those domains I
also have ldap replication issues because of something in my custom DC
policy I believe. For some reason ldap replication using Sites and
Services generate access denied errors. File replication service errors
appear as well. Sometimes all 3 default parittions in the directory
fail to replicate, other times the schema and configuration parittions
replicate fine but the main dc component level of the directory gets
access denied (i noticed that when I was fiddling around, not sure what
made 2 out of 3 work compared to when they would sometimes all fail to
replicate).

I know it's not a DNS issue because if I use the default DC policy
everything works. Licenses are also not replicating across the DCs
(this licensing problem is on both domains, the ldap replication is only
on one oddly enough) which may be related to this issue. Basically, I
need to know where is the access denied coming from concerning the ldap
replication. I've looked at permissions on the partitions in AD (using
ADSI) that are being replicated and they are correct. Since the default
DC policy works fine I figured it's just something in my custom policy
but I haven't hit upon what it is yet. When I lookup this issue on
google and visit websites none of them ever mention an issue with a
setting in User Rights Assignment or Security Options in the DC policy
other than Enterprise Domain Controllers needs to be an entry for
"Access this computer from the network" and Administrators have to be
listed for "Give user and machine accounts delegation control" or
something like that. Both of those settings are correct in my policy so
I don't know why else I get access denied.

The relevant services that are running are :
File replication service
licensing logging (for the license issue I brought up)
DFS and
MS software shadow copy (which i just turned on and may have been the
issue but I haven't confirmed that yet).

Am I missing one , like Background Intelligent transfer service or
anything like that? Unneeded services are disabled but maybe i disabled
too many.

I've been testing by opening up AD Sites and Services and forcing a
replication. It fails mainly when the 2nd DC has to contact the 1st DC
to do a replication from 1 to 2 (although sometimes I was good enough to
make it fail both ways but i dont know how). I've tried to run repadmin
and replmon and they all show the access denied error so that has pretty
much been drilled into my head, except I don't know what is actualy
being accessed other than the schema and configuration CNs in LDAP and
their permissions are okay so I don't know what else the problem is. I
also know the 2nd DC had issues being added to the domain because of my
policy and we disabled it and was able to get the 2nd DC added. It
seemed to be fine once I got all the servers with their policy settings
in teh right order. I checked the userAccountControl value for the 2nd
DC and it is the correct one for a DC.

The replication issue is a big one and if we fix that then it may fix
the issue where servers can't get their policy updates unless the first
dc has the custom dc policy disabled. That second issue is a problem
due to the fact if a server has to be rebuilt then for a few minutes the
custom DC policy must be disabled and the default DC policy enabled so
that the new server can have its settings updated and then we have to
reapply the custom DC policy. That presents a security issue.

Oddly enough, we can add workstaions to the domains w/o having a policy
yet and when I run gpupdate on them they can grab the workstation policy
just fine. Sorry for the long post as well but I wanted to try to
include as much info as possible.


thanks for any extra input. this is driving me crazy
Brandon

 

[Herb Martin]

DNS. It's not DNS because if I use the default DC policy things work just
fine.

The above it very unclear. "DNS. It's not DNS...."

Our DNS is hosted on Unix servers with dynamic updates turned off. All

It is impractical (almost impossible) to run AD without
DNS Dynamic updates.

machines were put into the dns records and the service records were entered by
taking the entries from netlogon.dns and putting them into the appropriate dns
records.

That is a very shaky practice.

What does DCDiag show.

And or DNSLint....

well it's not DNS replication. It's LDAP replication within ADS and the error
is Access denied so the DCs are finding each other just fine.

Ok, then why are you worried about replication?

What does ReplMon or RepAdmin tell you?

Uh, what do you mean? I'm "limited" to the options within the policy template
to begin with. It's not like I'm doing anything extra. I've set security
settings, disabled unneeded services, turned on a few things in the Admin
templates section, set restricted groups (but matched them up with what the
groups should have so I didn't lock anyone out like Administrator). That's
all I can think of right now w/o being at work to look at the policy.


I'm not replicating across the 2 domains. I'm replicating within the domains
which happen to be on differnt networks and don't communicate with each

other.

If this is ONE forest then some data DOES NEED to replicate.

If this is two forests in two networks then there is little point
in talking about them or troubleshooting them together.

The 2 DCs in 1 domain replicate with each other and the 2 DCs in the other
domain replicate with each other. That's all I meant.

One forest, or two?


Most "access denied" are really authentication problem which
are ALSO usually DNS problems.

Do you get those access denied problems both when you run
the tools from the actual DC and when you run them from
elsewhere on the network. (Run them locally on the DC.)

Also DNSLint should help and not require much privelege.

Here are the basics of DNS for AD:

1) Dynamic for the zone supporting AD
2) All internal DNS clients NIC\IP properties must specify SOLELY
that internal, dynamic DNS server (set.)
3) DCs and even DNS servers are DNS clients too -- see #2
4) If you have more than one Domain, every DNS server must
be able to resolve ALL domains (either directly or indirectly)

netdiag /fix

....or maybe:

dcdiag /fix

(Win2003 can do this from Support tools):
nltest /dsregdns /serverC-ServerNameGoesHere
http://support.microsoft.com/kb/q260371/

Ensure that DNS zones/domains are fully replicated to all DNS
servers for that (internal) zone/domain.

Also useful may be running DCDiag on each DC, sending the
output to a text file, and searching for FAIL, ERROR, WARN.

Single Label domain zone names are a problem Google:
[ "SINGLE LABEL" domain names DNS 2000 | 2003 microsoft: ]

DNSLint Tool: Microsoft utility to diagnose DNS diagnose
common DNS name resolution issues:
http://support.microsoft.com/?kbid=321045

How To Use DNSLint to Troubleshoot Active Directory Replication Issues
http://support.microsoft.com/default.aspx?scid=kb;en-us;q321046

 

[Brandon McCombs]

The above it very unclear. "DNS. It's not DNS...."

I answered your question with "DNS" becaused you asked "DNS or AD?" when I said
the problem isn't DNS. THen I elaborated by stating why DNS isn't the issue.

It is impractical (almost impossible) to run AD without
DNS Dynamic updates.

Why? Once all our servers and workstations are on the domain nothing will be
added. It is a secure gov't environment and things just won't be added willy
nilly. All the machines will be entered statically into DNS as well as the
servers. If if its impractical or impossible to do that then Microsoft
shouldn't make an option in the TCP/IP Properties of XP Pro (Home can't be on a
domain) to disable dynamic dns updates, right? Anyway, that part is working
fine.

That is a very shaky practice.

What does DCDiag show.

can't tell you b/c I don't remember the output. I'll run it agin when I go into
work on Monday but if i had to guess i'd say it reported 'access denied' just
like the other tools i ran reported.

And or DNSLint....


Ok, then why are you worried about replication?

Well one obvious reason might be because I GET ERRORS. A slightly better
question would be "then why are you still having replication issues?" and I've
have to say I don't know and that's why I'm here.

Just because the DCs find each other does not necessarily mean replication is
occuring because replication relies on more than just the fact that the DCs can
do a DNS lookup on each other. Replication also needs correct permissions on
each DC in order for the DCs to grab data from each other which is the problem
I'm running into right now.

What does ReplMon or RepAdmin tell you?

other.

If this is ONE forest then some data DOES NEED to replicate.

It is not one forest. Each domain is a separate forest. I have a feeling
mentioning the other domain to you was a mistake b/c it seems to have gotten you
waayyy lost. I only mentioned it b/c I was having my problems on both domains as
far as license replication is concerned and as far as having issues with other
servers being allowd to access the first DC if the first DC already updated its
policy settings with my custom DC policy. Something in the policy prevents other
servers from grabbing their policies and that's what I'm trying to figure out.
That problem may or may not be related to the replication issue.

2 main issues: replication and "access" I guess u could say. Replication can be
further dissected into LDAP replication issues and license replication issues.

If this is two forests in two networks then there is little point
in talking about them or troubleshooting them together.

I only mentioned both of them because they are running into the SAME PROBLEM
because I'm using the same policies on both. One domain is a soon to be
operational environment and the other is a test environment and that is why I
can use the same policies (because I knew you would ask how I can use the same
on both). If I fix one domain most likely I fix the other because they are
using the same policies. That is why there is a big point in troubleshooting
them together.

One forest, or two?
2


Most "access denied" are really authentication problem which
are ALSO usually DNS problems.

I already stated I can leave DNS totally untouched and just not use my custom DC
policy but instead use the default one and everything works fine. If somehow
that still means DNS is the issue then by all means tell me how but I've already
isolated where the problem is and now I'm searching for a solution.

Do you get those access denied problems both when you run
the tools from the actual DC and when you run them from
elsewhere on the network. (Run them locally on the DC.)

I never thought about running them from another server that wasn't a DC but I
can try. I've ran the commands from both DCs and repadmin works fine for
inbound replication when dc1 is pushing to dc2 but when dc2 has to connect to
dc1 replication fails.

Also DNSLint should help and not require much privelege.

I can try it but DNS isn't an issue right now. At one point when i noticed
replication issues it was due to missing dns entries when one of the admins
somehow didn't grab all teh service record entries in netlogon.dns when he put
them into the BIND dns files. He fixed that and we got past the first set of
errors we were seeing where File Rep. Service stated an nslookup on the GUID of
the second DC couldn't be done correctly. Now though I want to be able to fix
not only having to temporarily load the default DC policy(as stated in the
original post) if a server has to be rebuilt (otherwise the new server won't be
allowd to connect to the DCs and grab its policy) as well as the replication
issues I'm seeing. Both of those might be related to access issues but I
definitely want to sort out th replication problem.

I don't see why Dynamic DNS is so important if all the needed records are
already in DNS and service records can be looked up. If a lookup on some
service or host is needed then I should be getting more o fthose errors beyond
the first one we got (which i mentioned in the above paragraph).
The errors I'm seeing now deal with KCC and the lookup issue we ran into
initially came from FRS.

 

[Brandon McCombs]

I noticed someone else having replication issues on a website and saw you were
helping him too. It was from March 23rd. I wanted to add a comment about his
post:
---------------------------------from Chris Gradden on RE: AD Repl problem with
two servers at
http://forums.wugnet.com/Active-Dir...m-servers-ftopict347880.html-----------------

I generated a status report from the first server. This all looked ok until
I got to the Enterprise Data section.

The first server shows the same GUID for the Server GUID (used for DNS) and
the Replication Database GUID (used to identify partner in replication).
However, the second server has differing GUID's for each of these entries.
Is this normal or could this be the problem?

I have tried running dcpromo again to demote then promote the server back to
a DC but it wont let me as it wants to replicate the changes... aargh.

So I'm stuck for the moment.
-------------------------------------------------------
I noticed the same thing on the 2nd DC I believe in my soon to be operational
domain where the GUID was the same for the server and for the rep db but I
didn't know if it was a problem or not. Also, we tried to demote our 2nd DC
that is having replication issues and since replication was our problem we
couldn't demote it since it wanted to f**king replicate in order to do that.

 

[Herb Martin]

It is impractical (almost impossible) to run AD without

Why? Once all our servers and workstations are on the domain nothing will be
added. It is a secure gov't environment and things just won't be added willy
nilly. All the machines will be entered statically into DNS as well as the
servers. If if its impractical or impossible to do that then Microsoft
shouldn't make an option in the TCP/IP Properties of XP Pro (Home can't be on a
domain) to disable dynamic dns updates, right? Anyway, that part is working
fine.

It is possible, it just isn't practical and it is not
really due to the XP or any other client machines.

It is due to the DCs.

It everything it truly static and you get it right
initially, then of course it will work until you
change something.

Like moving a DC from one site to another.

 

[Brandon McCombs]

It is possible, it just isn't practical and it is not
really due to the XP or any other client machines.

It is due to the DCs.

It everything it truly static and you get it right
initially, then of course it will work until you
change something.

The other domains work fine being as they only have a single DC. We don't use
dynamic dns anywhere. Everything is static, no dhcp.

Like moving a DC from one site to another.

I ran dcdiag /fix when I got to work on both DCs and on a non DC (by using /s of
course).

I got access denied again for replication messages on all the partitions and for
the frsevent test I got an error 0x800034c4 I believe was the code.
I also got warnings about sysvol issues possibly arising even though the test
frssysvol passed just fine and I also saw a new message about a replication link
(error 0x8000051c) not existing but go figure since at one time today
replication worked both ways without changing anything (described in next
paragraph).

Oddly enough, I went ahead and did a gpupdate on the first DC and rebooted and
then both DCs could replicate both ways. As soon as I did a gpupdate on the
second DC and rebooted though I lost replication going from dc2 to dc1. I don't
know what the **** causes that to happen when I didn't touch the policies in
between doing that and the 2 DCs are in the same OU and thus have the same
policies being applied. dcdiag did not report any other errors other than
frsevent test failing so DNS is not an issue. I'd like to know why FRS is
failing b/c I'm not doing anything with it in my policies and it IS running so I
don't know what else to check. I didn't get a chance to run dcdiag again once
things were working before I ended up breaking it again after running gpupdate
on the 2nd DC. Hopefully I can get it all working again tomorrow and run dcdiag
to see if it reports any warnings or errors even when replication is working.

Any ideas?

 

[Brandon McCombs]

I fixed the replication issue by resetting the machine account password for dc2
and restarting both KDC and netlogon services (don't know if that was needed or
not). I still get a warning about errors in the last 24 hours concerning
frssysvol but i think after more waiting the next 24 hours will past and that
will go away too. Licenses still aren't being replicated across the domain
controllers though.