Restoring Ghost Image to W2K AD = Lost Synchronisation of accounts

L

Luke

Hi all w2k experts,

I am a newbie to w2k AD and also to this NG.
My AD has 2 domain controllers, Svr1 and Svr2. Svr1 is also a exch5.5
and DNS server. Svr2 is also a DNS server. I ghost Svr1 quarterly as a
form of backup, using norton ghost 2002. System state backups are also
done daily.

Situation:
Svr1 hdd crashed and i got a new hdd to replace it, dump back the
ghost image and restore the mailbox data. All the while when the
restoring was done on svr1, svr2's configuration was changed, ie the
fsmo roles wasnt transfered over to it. After the restoration was
done, svr1 was popped back to into the domain, and all was working
fine. However the problem was that any changes that i make in the AD
in svr1 was not replicated to svr2. Changes made in svr2 can be
replicated to svr1.

There is an error in the event viewer:

Event Type: Error
Event Source: NETLOGON
Event Category: None
Event ID: 5774
Date: 9/23/2003
Time: 12:28:20 PM
User: N/A
Computer: SVR01
Description:
Registration of the DNS record
'104e245c-1bd9-48cf-bd6f-623a5caa6377._msdcs.<domain>. 600 IN CNAME
svr01.<domain>.' failed with the following error:
DNS name does not exist.
Data:
0000: 2b 23 00 00 +#..

I have checked both DNS servers(Svr1 and Svr2) and that entry is in.
Also Svr1's dns is pointing to svr2 and svr2 is pointing to itself.

Svr1 is also rejecting any connections from Svr2. In the DNS msc, it
reports "Access is denied". Browsing to svr1 using WinExplorer reports
"Logon failure: Target Account name is incorrect.". Even manually
forcing replication from svr2 to svr1 fails with the error "Target
Account name is incorrect."

I am suspecting that the problem could be of the fsmo roles, but all
the error in the event log seems to be pointing in the direction of
DNS. Any one has any ideas to point me in the correct direction?

Thanx
Luke
 
J

Jason Robarts [MSFT]

I'd check the DNS records again. Make sure the A records have the right IP
addresses. What I was wondering was if the Ghosted image was too old. If
the error messages reported are the most up to date then DNS configuration
looks to be the problem. Another approach to get information would be to
force replication using Active Directory Sites and Services snap-in or using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last failure
times as well as an error code. You may find information on the error code
on support.microsoft.com or by posting it again here. If DNS is configured
correctly the error message may give a clue as to the problem.


Luke said:
Hi Jason,

the ghosted image is about 3 weeks old.


"Jason Robarts [MSFT]" <[email protected]> wrote in message
How old is the Ghosted image?
 
L

Luke

Hi Jason,
thanx for the reply. Checked my dns and the entries are correct.
I have tried forcing a replication but it fails with the message
target prinicpal name is incorrect and doing a repadmin /showreps
shows that the last successful replication is before the crash happen.
the error msg that i see from doing repadmin is "access is denied."

i was wondering if i need to do a netdom command between teh 2 DCs to
have their accounts synchronised, but i am not experienced enough to
know if there will be any side effects that this will have on the
otehr servers and clients taht are in the network.
Thanx.


Jason Robarts said:
I'd check the DNS records again. Make sure the A records have the right IP
addresses. What I was wondering was if the Ghosted image was too old. If
the error messages reported are the most up to date then DNS configuration
looks to be the problem. Another approach to get information would be to
force replication using Active Directory Sites and Services snap-in or using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last failure
times as well as an error code. You may find information on the error code
on support.microsoft.com or by posting it again here. If DNS is configured
correctly the error message may give a clue as to the problem.


Luke said:
Hi Jason,

the ghosted image is about 3 weeks old.


"Jason Robarts [MSFT]" <[email protected]> wrote in message
How old is the Ghosted image?


Anybody got any ideas to this or have experienced this before?
 
B

Brett Shirley [MSFT]

Luke,

Could you please describe how you got the "Ghosted image"? AD doesn't
support any sort of disk or file imaging that doesn't go through the
official backup/restore APIs. I believe, but am not really that sure, that
Norton Ghost does this. If you've done this, it is very very bad, and if
we're lucky, the only thing it's done is not replicate. Please stop trying
to get them to replicate, it's a good idea to shutdown this DC down or
unplug it from the network for now.

Is this the only DC for this domain? Is it plausible to wipe this DC and
bring it back online as a new replica DC?

Thanks,
Brett Shirley
SDE, AD Replication & AD Backup/Restore

Disclaimer: ( DmitriG posts alot, so I defer to his disclaimer ... )

I've CC'd Sorin, b/c from a quick web search, I saw he was trying to ghost a
DC too.
http://www.freelists.org/archives/windows2000/12-2002/msg00418.html


Luke said:
Hi Jason,
thanx for the reply. Checked my dns and the entries are correct.
I have tried forcing a replication but it fails with the message
target prinicpal name is incorrect and doing a repadmin /showreps
shows that the last successful replication is before the crash happen.
the error msg that i see from doing repadmin is "access is denied."

i was wondering if i need to do a netdom command between teh 2 DCs to
have their accounts synchronised, but i am not experienced enough to
know if there will be any side effects that this will have on the
otehr servers and clients taht are in the network.
Thanx.


"Jason Robarts [MSFT]" <[email protected]> wrote in message
I'd check the DNS records again. Make sure the A records have the right IP
addresses. What I was wondering was if the Ghosted image was too old. If
the error messages reported are the most up to date then DNS configuration
looks to be the problem. Another approach to get information would be to
force replication using Active Directory Sites and Services snap-in or using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last failure
times as well as an error code. You may find information on the error code
on support.microsoft.com or by posting it again here. If DNS is configured
correctly the error message may give a clue as to the problem.


Luke said:
Hi Jason,

the ghosted image is about 3 weeks old.


"Jason Robarts [MSFT]" <[email protected]> wrote in
message
How old is the Ghosted image?


Anybody got any ideas to this or have experienced this before?
 
L

Luke

Brett,

I formated a floppy using norton ghost wizard and the ghost program
was copied onto the floppy. i booted the DC from the floppy, loaded
the ghost program and do the ghost. After that i booted the DC back
into w2k.

After the crash the image was dumped back and it booted up fine with
the exception of not replicating and having a de-synchorisation of the
accounts with another DC. Looks like i got lucky this time. Just out
of curiousity, what could be the wrost thing that could happen if i
was unlucky?

I cannot shut down this DC as it is also an exchange server, and this
is the only mail server that we have, and thus i am not able to wipe
the DC out.

Will the Netdom command help in having the accounts synchronised? and
wil it have any negative effects on the network? What i want is just
to have the least disruption to the network.

thanx.
Luke

Brett Shirley said:
Luke,

Could you please describe how you got the "Ghosted image"? AD doesn't
support any sort of disk or file imaging that doesn't go through the
official backup/restore APIs. I believe, but am not really that sure, that
Norton Ghost does this. If you've done this, it is very very bad, and if
we're lucky, the only thing it's done is not replicate. Please stop trying
to get them to replicate, it's a good idea to shutdown this DC down or
unplug it from the network for now.

Is this the only DC for this domain? Is it plausible to wipe this DC and
bring it back online as a new replica DC?

Thanks,
Brett Shirley
SDE, AD Replication & AD Backup/Restore

Disclaimer: ( DmitriG posts alot, so I defer to his disclaimer ... )

I've CC'd Sorin, b/c from a quick web search, I saw he was trying to ghost a
DC too.
http://www.freelists.org/archives/windows2000/12-2002/msg00418.html


Luke said:
Hi Jason,
thanx for the reply. Checked my dns and the entries are correct.
I have tried forcing a replication but it fails with the message
target prinicpal name is incorrect and doing a repadmin /showreps
shows that the last successful replication is before the crash happen.
the error msg that i see from doing repadmin is "access is denied."

i was wondering if i need to do a netdom command between teh 2 DCs to
have their accounts synchronised, but i am not experienced enough to
know if there will be any side effects that this will have on the
otehr servers and clients taht are in the network.
Thanx.


"Jason Robarts [MSFT]" <[email protected]> wrote in message
I'd check the DNS records again. Make sure the A records have the right IP
addresses. What I was wondering was if the Ghosted image was too old. If
the error messages reported are the most up to date then DNS configuration
looks to be the problem. Another approach to get information would be to
force replication using Active Directory Sites and Services snap-in or using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last failure
times as well as an error code. You may find information on the error code
on support.microsoft.com or by posting it again here. If DNS is configured
correctly the error message may give a clue as to the problem.


Hi Jason,

the ghosted image is about 3 weeks old.


"Jason Robarts [MSFT]" <[email protected]> wrote in
message
How old is the Ghosted image?


Anybody got any ideas to this or have experienced this before?
 
B

Brett Shirley [MSFT]

Luke,

First My Suggestion:
I am still verifying with experts whether Norton Ghost does take an image of
the disk, w/o the proper AD backup methods. If it does, you're in a very
dangerous situation. At some point replication could go from bad to worse.
I think the unlucky case if they start replicating again, is actually just
at some point in the future for you.

My suggestion would be to migrate this Exchange server to your other server,
or backup the Exchange server, rebuild the AD replica DC from scratch, and
then restore the Exchange server. I'm not sure how feasible these
suggestions are b/c I'm not well an Exchange guru, just an AD
replication/backup/restore guru. However, I'm SURE that you will have AD
problems in the future if this DC remains on.

You _may_ be able to mitigate the likelyhood that the two systems start
replicating again, by pausing the netlogon service on the ghost restored DC.
This will ?hopefully? stop it from taking changes.


You asked, "What is the worst case?":
I'm glad you asked ... this is complicated, so I hope I can explain it
clearly. The potential problem relates to how we track replication
meta-data, and perform replication. Basically, a "first DC" tracks whether
it has seen a given replicated change from a 2nd DC by storing the
InvocationID (really a GUID) and a USN pair. The USN is a serial 64 bit
number and ever single transacted change has a USN stamped with it to
represent that change. If you imagine the USN always increasing for each
new change on a DC, you can see how every change in your AD forest, has an
InvocationID (i.e. a DC) and USN that represents that change.

Lets use some specific examples to make it clear, say the "highest USN" is
537 when the ghost image is taken of the 2nd DC, meaning their are changes
('changes' can be add users, password changes, new exch mailboxes, etc)
associated with USNs 1 through 536. After the ghost image is taken, changes
continue to trickle in, and each change replicates off to the first DC of
course. Each time replication happens, the first DC saves the "highest USN"
of the 2nd DC through the end of the changes it replicated from the 2nd DC.
Say USN 673 is the last "highest USN" of the last change the first DC
replicated from the 2nd DC right before the ghost image was restored.

Now, trouble ... when you now restore that ghost image, you take the 2nd DC
back in time, and the DC will have the USN 537 from the ghost image, while
the first DC will think it's seen the 2nd DC's changes through USN 673. New
changes by the 2nd DC will now get USNs 537, 538, 539, etc through 672, and
eventually beyond 672 as well. The first DC however will never get the 2nd
set of changes made at USNs 537-672, b/c it thinks it's seen all changes
before 673. So to start with you won't have the same data on the two DCs,
and this will never fix itself. Now things get really bad (as if that
wasn't bad enough ;) ... imagine that change USN 675 on the 2nd DC (which
will get replicated b/c it's after 673), depends on (like is a child object
of) a change that was made slightly earlier at USN 671. The first DC will
ask for changes 673 and up, and when it sees change 675 it will freak out,
b/c it claims to be above a child that couldn't possibly be there when the
first DC hasn't and won't replicate change 671. Other weird things can
happen to things like DN references (like a group's member attribute) that
reference objects that don't exist on one DC, etc.

I'm not sure if that made sense, but believe me; the basic data consistency
AD has tried to ensure is shot, and other apps that run on AD (like
Exchange) can start to misbehave. Cleaning up these inconsistencies could
be very very difficult, as you've taken AD very far outside it's normal
operating parameters. Microsoft PSS doesn't even have tools that I know of
to deal with this kind of thing.


Back to my first suggestion:
For this reason, it's vitally important to stop taking changes on this 2nd
DC, the ghost restored DC. WELL, of course this is all a mute point if Nort
on Ghost performs a regular AD backup somehow, then you wouldn't be
afflicted by the above horror. But I think this is unlikely to be the case,
but I'll repost again one way or the other, when I get a response from some
authoritative Norton/Symantec people.


Thanks,
Brett Shirley
AD Replication and AD Backup/Restore, SDE (that means Programmer)

This posting is provided "AS IS" with no warranties, and confers no rights.


Luke said:
Brett,

I formated a floppy using norton ghost wizard and the ghost program
was copied onto the floppy. i booted the DC from the floppy, loaded
the ghost program and do the ghost. After that i booted the DC back
into w2k.

After the crash the image was dumped back and it booted up fine with
the exception of not replicating and having a de-synchorisation of the
accounts with another DC. Looks like i got lucky this time. Just out
of curiousity, what could be the wrost thing that could happen if i
was unlucky?

I cannot shut down this DC as it is also an exchange server, and this
is the only mail server that we have, and thus i am not able to wipe
the DC out.

Will the Netdom command help in having the accounts synchronised? and
wil it have any negative effects on the network? What i want is just
to have the least disruption to the network.

thanx.
Luke

"Brett Shirley [MSFT]" <[email protected]> wrote in message
Luke,

Could you please describe how you got the "Ghosted image"? AD doesn't
support any sort of disk or file imaging that doesn't go through the
official backup/restore APIs. I believe, but am not really that sure, that
Norton Ghost does this. If you've done this, it is very very bad, and if
we're lucky, the only thing it's done is not replicate. Please stop trying
to get them to replicate, it's a good idea to shutdown this DC down or
unplug it from the network for now.

Is this the only DC for this domain? Is it plausible to wipe this DC and
bring it back online as a new replica DC?

Thanks,
Brett Shirley
SDE, AD Replication & AD Backup/Restore

Disclaimer: ( DmitriG posts alot, so I defer to his disclaimer ... )

I've CC'd Sorin, b/c from a quick web search, I saw he was trying to ghost a
DC too.
http://www.freelists.org/archives/windows2000/12-2002/msg00418.html


Luke said:
Hi Jason,
thanx for the reply. Checked my dns and the entries are correct.
I have tried forcing a replication but it fails with the message
target prinicpal name is incorrect and doing a repadmin /showreps
shows that the last successful replication is before the crash happen.
the error msg that i see from doing repadmin is "access is denied."

i was wondering if i need to do a netdom command between teh 2 DCs to
have their accounts synchronised, but i am not experienced enough to
know if there will be any side effects that this will have on the
otehr servers and clients taht are in the network.
Thanx.


"Jason Robarts [MSFT]" <[email protected]> wrote in
message
I'd check the DNS records again. Make sure the A records have the
right
IP
addresses. What I was wondering was if the Ghosted image was too
old.
If
the error messages reported are the most up to date then DNS configuration
looks to be the problem. Another approach to get information would
be
to
force replication using Active Directory Sites and Services snap-in
or
using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last failure
times as well as an error code. You may find information on the
error
code
on support.microsoft.com or by posting it again here. If DNS is configured
correctly the error message may give a clue as to the problem.


Hi Jason,

the ghosted image is about 3 weeks old.


"Jason Robarts [MSFT]" <[email protected]> wrote in
message
How old is the Ghosted image?


Anybody got any ideas to this or have experienced this before?
 
L

Luke

Brett,

thanx for the comprehensive reply. But i just discovered one strange
thing that is happening in my domain. The dc that didnt crash, dc2,is
able to replicate changes to the crashed dc, dc1, via its normal
schedule of replication but not the other way round, from dc1 to dc2.
It is sort of like a one-way street. However it fails to do the
replication if i were to force the replication from the AD sites adn
services from dc2. Even doing the repadmin /showreps tells me that
replication fails for 10000+ consecutive times. So currently, i am
just making changes to dc2 and wait for it to be replicated over, but
i dont know if there will be any chances that it will suddenly stop
replicating over to dc1.

As for creating a temp excahnge server, i guess that is pretty much
out as the company is on a very tight budget and there arent any spare
pcs around that i can make use of, but still i will try and look out
for any chance to get one for disaster recovery.

Pardon me for my ignorance, but if i pause the netlogon on dc1 and i
havnt setup a different exchange server then will my users be able to
logon to the exchange server to retrieve mails? I am not sure what
effects that pausing netlogon will have on the network. Are there any
sources/websites/books out there that can enlighten me on what effects
stopping/pausing any services in windows 2000 will have.

Thanx and regards,
Luke


Brett Shirley said:
Luke,

First My Suggestion:
I am still verifying with experts whether Norton Ghost does take an image of
the disk, w/o the proper AD backup methods. If it does, you're in a very
dangerous situation. At some point replication could go from bad to worse.
I think the unlucky case if they start replicating again, is actually just
at some point in the future for you.

My suggestion would be to migrate this Exchange server to your other server,
or backup the Exchange server, rebuild the AD replica DC from scratch, and
then restore the Exchange server. I'm not sure how feasible these
suggestions are b/c I'm not well an Exchange guru, just an AD
replication/backup/restore guru. However, I'm SURE that you will have AD
problems in the future if this DC remains on.

You _may_ be able to mitigate the likelyhood that the two systems start
replicating again, by pausing the netlogon service on the ghost restored DC.
This will ?hopefully? stop it from taking changes.


You asked, "What is the worst case?":
I'm glad you asked ... this is complicated, so I hope I can explain it
clearly. The potential problem relates to how we track replication
meta-data, and perform replication. Basically, a "first DC" tracks whether
it has seen a given replicated change from a 2nd DC by storing the
InvocationID (really a GUID) and a USN pair. The USN is a serial 64 bit
number and ever single transacted change has a USN stamped with it to
represent that change. If you imagine the USN always increasing for each
new change on a DC, you can see how every change in your AD forest, has an
InvocationID (i.e. a DC) and USN that represents that change.

Lets use some specific examples to make it clear, say the "highest USN" is
537 when the ghost image is taken of the 2nd DC, meaning their are changes
('changes' can be add users, password changes, new exch mailboxes, etc)
associated with USNs 1 through 536. After the ghost image is taken, changes
continue to trickle in, and each change replicates off to the first DC of
course. Each time replication happens, the first DC saves the "highest USN"
of the 2nd DC through the end of the changes it replicated from the 2nd DC.
Say USN 673 is the last "highest USN" of the last change the first DC
replicated from the 2nd DC right before the ghost image was restored.

Now, trouble ... when you now restore that ghost image, you take the 2nd DC
back in time, and the DC will have the USN 537 from the ghost image, while
the first DC will think it's seen the 2nd DC's changes through USN 673. New
changes by the 2nd DC will now get USNs 537, 538, 539, etc through 672, and
eventually beyond 672 as well. The first DC however will never get the 2nd
set of changes made at USNs 537-672, b/c it thinks it's seen all changes
before 673. So to start with you won't have the same data on the two DCs,
and this will never fix itself. Now things get really bad (as if that
wasn't bad enough ;) ... imagine that change USN 675 on the 2nd DC (which
will get replicated b/c it's after 673), depends on (like is a child object
of) a change that was made slightly earlier at USN 671. The first DC will
ask for changes 673 and up, and when it sees change 675 it will freak out,
b/c it claims to be above a child that couldn't possibly be there when the
first DC hasn't and won't replicate change 671. Other weird things can
happen to things like DN references (like a group's member attribute) that
reference objects that don't exist on one DC, etc.

I'm not sure if that made sense, but believe me; the basic data consistency
AD has tried to ensure is shot, and other apps that run on AD (like
Exchange) can start to misbehave. Cleaning up these inconsistencies could
be very very difficult, as you've taken AD very far outside it's normal
operating parameters. Microsoft PSS doesn't even have tools that I know of
to deal with this kind of thing.


Back to my first suggestion:
For this reason, it's vitally important to stop taking changes on this 2nd
DC, the ghost restored DC. WELL, of course this is all a mute point if Nort
on Ghost performs a regular AD backup somehow, then you wouldn't be
afflicted by the above horror. But I think this is unlikely to be the case,
but I'll repost again one way or the other, when I get a response from some
authoritative Norton/Symantec people.


Thanks,
Brett Shirley
AD Replication and AD Backup/Restore, SDE (that means Programmer)

This posting is provided "AS IS" with no warranties, and confers no rights.


Luke said:
Brett,

I formated a floppy using norton ghost wizard and the ghost program
was copied onto the floppy. i booted the DC from the floppy, loaded
the ghost program and do the ghost. After that i booted the DC back
into w2k.

After the crash the image was dumped back and it booted up fine with
the exception of not replicating and having a de-synchorisation of the
accounts with another DC. Looks like i got lucky this time. Just out
of curiousity, what could be the wrost thing that could happen if i
was unlucky?

I cannot shut down this DC as it is also an exchange server, and this
is the only mail server that we have, and thus i am not able to wipe
the DC out.

Will the Netdom command help in having the accounts synchronised? and
wil it have any negative effects on the network? What i want is just
to have the least disruption to the network.

thanx.
Luke

"Brett Shirley [MSFT]" <[email protected]> wrote in message
Luke,

Could you please describe how you got the "Ghosted image"? AD doesn't
support any sort of disk or file imaging that doesn't go through the
official backup/restore APIs. I believe, but am not really that sure, that
Norton Ghost does this. If you've done this, it is very very bad, and if
we're lucky, the only thing it's done is not replicate. Please stop trying
to get them to replicate, it's a good idea to shutdown this DC down or
unplug it from the network for now.

Is this the only DC for this domain? Is it plausible to wipe this DC and
bring it back online as a new replica DC?

Thanks,
Brett Shirley
SDE, AD Replication & AD Backup/Restore

Disclaimer: ( DmitriG posts alot, so I defer to his disclaimer ... )

I've CC'd Sorin, b/c from a quick web search, I saw he was trying to ghost a
DC too.
http://www.freelists.org/archives/windows2000/12-2002/msg00418.html


Hi Jason,
thanx for the reply. Checked my dns and the entries are correct.
I have tried forcing a replication but it fails with the message
target prinicpal name is incorrect and doing a repadmin /showreps
shows that the last successful replication is before the crash happen.
the error msg that i see from doing repadmin is "access is denied."

i was wondering if i need to do a netdom command between teh 2 DCs to
have their accounts synchronised, but i am not experienced enough to
know if there will be any side effects that this will have on the
otehr servers and clients taht are in the network.
Thanx.


"Jason Robarts [MSFT]" <[email protected]> wrote in
message
I'd check the DNS records again. Make sure the A records have the
right
IP
addresses. What I was wondering was if the Ghosted image was too
old.
If
the error messages reported are the most up to date then DNS configuration
looks to be the problem. Another approach to get information would
be
to
force replication using Active Directory Sites and Services snap-in
or
using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last failure
times as well as an error code. You may find information on the
error
code
on support.microsoft.com or by posting it again here. If DNS is configured
correctly the error message may give a clue as to the problem.


Hi Jason,

the ghosted image is about 3 weeks old.


message
How old is the Ghosted image?


Anybody got any ideas to this or have experienced this before?
 
B

Brett Shirley [msft]

I managed to get a hold of the senior dev manager for Ghost at
Symantec. Ghosting AD DCs is not supported as the product exists
today. I suggest you follow my suggestions below.

Thanks,
BrettSh

SDE, AD Replication

Posting is provided "AS IS"



Luke,

First My Suggestion:
I am still verifying with experts whether Norton Ghost does take an image of
the disk, w/o the proper AD backup methods. If it does, you're in a very
dangerous situation. At some point replication could go from bad to worse.
I think the unlucky case if they start replicating again, is actually just
at some point in the future for you.

My suggestion would be to migrate this Exchange server to your other server,
or backup the Exchange server, rebuild the AD replica DC from scratch, and
then restore the Exchange server. I'm not sure how feasible these
suggestions are b/c I'm not well an Exchange guru, just an AD
replication/backup/restore guru. However, I'm SURE that you will have AD
problems in the future if this DC remains on.

You _may_ be able to mitigate the likelyhood that the two systems start
replicating again, by pausing the netlogon service on the ghost restored DC.
This will ?hopefully? stop it from taking changes.


You asked, "What is the worst case?":
I'm glad you asked ... this is complicated, so I hope I can explain it
clearly. The potential problem relates to how we track replication
meta-data, and perform replication. Basically, a "first DC" tracks whether
it has seen a given replicated change from a 2nd DC by storing the
InvocationID (really a GUID) and a USN pair. The USN is a serial 64 bit
number and ever single transacted change has a USN stamped with it to
represent that change. If you imagine the USN always increasing for each
new change on a DC, you can see how every change in your AD forest, has an
InvocationID (i.e. a DC) and USN that represents that change.

Lets use some specific examples to make it clear, say the "highest USN" is
537 when the ghost image is taken of the 2nd DC, meaning their are changes
('changes' can be add users, password changes, new exch mailboxes, etc)
associated with USNs 1 through 536. After the ghost image is taken, changes
continue to trickle in, and each change replicates off to the first DC of
course. Each time replication happens, the first DC saves the "highest USN"
of the 2nd DC through the end of the changes it replicated from the 2nd DC.
Say USN 673 is the last "highest USN" of the last change the first DC
replicated from the 2nd DC right before the ghost image was restored.

Now, trouble ... when you now restore that ghost image, you take the 2nd DC
back in time, and the DC will have the USN 537 from the ghost image, while
the first DC will think it's seen the 2nd DC's changes through USN 673. New
changes by the 2nd DC will now get USNs 537, 538, 539, etc through 672, and
eventually beyond 672 as well. The first DC however will never get the 2nd
set of changes made at USNs 537-672, b/c it thinks it's seen all changes
before 673. So to start with you won't have the same data on the two DCs,
and this will never fix itself. Now things get really bad (as if that
wasn't bad enough ;) ... imagine that change USN 675 on the 2nd DC (which
will get replicated b/c it's after 673), depends on (like is a child object
of) a change that was made slightly earlier at USN 671. The first DC will
ask for changes 673 and up, and when it sees change 675 it will freak out,
b/c it claims to be above a child that couldn't possibly be there when the
first DC hasn't and won't replicate change 671. Other weird things can
happen to things like DN references (like a group's member attribute) that
reference objects that don't exist on one DC, etc.

I'm not sure if that made sense, but believe me; the basic data consistency
AD has tried to ensure is shot, and other apps that run on AD (like
Exchange) can start to misbehave. Cleaning up these inconsistencies could
be very very difficult, as you've taken AD very far outside it's normal
operating parameters. Microsoft PSS doesn't even have tools that I know of
to deal with this kind of thing.


Back to my first suggestion:
For this reason, it's vitally important to stop taking changes on this 2nd
DC, the ghost restored DC. WELL, of course this is all a mute point if Nort
on Ghost performs a regular AD backup somehow, then you wouldn't be
afflicted by the above horror. But I think this is unlikely to be the case,
but I'll repost again one way or the other, when I get a response from some
authoritative Norton/Symantec people.


Thanks,
Brett Shirley
AD Replication and AD Backup/Restore, SDE (that means Programmer)

This posting is provided "AS IS" with no warranties, and confers no rights.


Luke said:
Brett,

I formated a floppy using norton ghost wizard and the ghost program
was copied onto the floppy. i booted the DC from the floppy, loaded
the ghost program and do the ghost. After that i booted the DC back
into w2k.

After the crash the image was dumped back and it booted up fine with
the exception of not replicating and having a de-synchorisation of the
accounts with another DC. Looks like i got lucky this time. Just out
of curiousity, what could be the wrost thing that could happen if i
was unlucky?

I cannot shut down this DC as it is also an exchange server, and this
is the only mail server that we have, and thus i am not able to wipe
the DC out.

Will the Netdom command help in having the accounts synchronised? and
wil it have any negative effects on the network? What i want is just
to have the least disruption to the network.

thanx.
Luke

"Brett Shirley [MSFT]" <[email protected]> wrote in message
Luke,

Could you please describe how you got the "Ghosted image"? AD doesn't
support any sort of disk or file imaging that doesn't go through the
official backup/restore APIs. I believe, but am not really that sure, that
Norton Ghost does this. If you've done this, it is very very bad, and if
we're lucky, the only thing it's done is not replicate. Please stop trying
to get them to replicate, it's a good idea to shutdown this DC down or
unplug it from the network for now.

Is this the only DC for this domain? Is it plausible to wipe this DC and
bring it back online as a new replica DC?

Thanks,
Brett Shirley
SDE, AD Replication & AD Backup/Restore

Disclaimer: ( DmitriG posts alot, so I defer to his disclaimer ... )

I've CC'd Sorin, b/c from a quick web search, I saw he was trying to ghost a
DC too.
http://www.freelists.org/archives/windows2000/12-2002/msg00418.html


Hi Jason,
thanx for the reply. Checked my dns and the entries are correct.
I have tried forcing a replication but it fails with the message
target prinicpal name is incorrect and doing a repadmin /showreps
shows that the last successful replication is before the crash happen.
the error msg that i see from doing repadmin is "access is denied."

i was wondering if i need to do a netdom command between teh 2 DCs to
have their accounts synchronised, but i am not experienced enough to
know if there will be any side effects that this will have on the
otehr servers and clients taht are in the network.
Thanx.


I'd check the DNS records again. Make sure the A records have the right
IP
addresses. What I was wondering was if the Ghosted image was too old.
If
the error messages reported are the most up to date then DNS
configuration
looks to be the problem. Another approach to get information would be
to
force replication using Active Directory Sites and Services snap-in or
using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last
failure
times as well as an error code. You may find information on the error
code
on support.microsoft.com or by posting it again here. If DNS is
configured
correctly the error message may give a clue as to the problem.


Hi Jason,

the ghosted image is about 3 weeks old.


message
How old is the Ghosted image?


Anybody got any ideas to this or have experienced this before?
 
B

Brett Shirley [msft]

Sorry, I haven't had a chance to check news groups in awhile, busy.

Both DCs 1 and 2 are DCs for the same domain, right? If so then I've
got an alternative plan that you will hopefully find acceptable:

Step 1) Simply demote dc1.
Note: You'll lose any objects that DC1 has created,
but haven't replicated to dc1.
Step 2) Rempromote dc1.

Simple, eh? Well probably not so simple, b/c step 1 will probably
fail. ;) :p So if step 1 fails, here are steps 1a through 1h ;)

Step 1a) Again, ensure that both DCs are in the same Domain! Very
important. Do not continue if this is not the case.

Step 1b) Read, but don't yet follow this article:
http://support.microsoft.com/default.aspx?scid=kb;en-us;332199
(_no one else_ ever check out this article, it's never a good idea to
use this option in dcpromo, its a dangerous way to around the death of
a DC) I didn't write the article, and its lacking some pertinent
steps, IMO.

Step 1c) Try to _transfer_ all 5 of the FSMOs this DC may hold to the
other DC. Some may fail, some may be on the other DC.

Step 1d) If your DNS is AD Integrated, make this 2nd DC a DNS server
as well.

Step 1e) Mark or ensure that the 2nd DC is a GC as well (doable
through AD Sites & Services). Normally, I'd also tell you to ensure
the GC promotion process is complete, but since you have only 1
domain, I think it's not really necessary.

Step 1f) Seize the PDC FSMO to the other DC. This is the only FSMO
that is every truly safe to seize.

Step 1g) Actually follow the above KB article.

Step 1h) Go back and continue with Step 2.

Does that sound reasonable? I'll try to remember to check back in a
couple days.

Oh yeah, Step 3, don't use Ghost for backup of DCs. ;)

Thanks,
-BrettSh

Posting "AS IS", no liability, etc ... see one of DmitriG's posts, he
always has the right advice ...

Brett,

thanx for the comprehensive reply. But i just discovered one strange
thing that is happening in my domain. The dc that didnt crash, dc2,is
able to replicate changes to the crashed dc, dc1, via its normal
schedule of replication but not the other way round, from dc1 to dc2.
It is sort of like a one-way street. However it fails to do the
replication if i were to force the replication from the AD sites adn
services from dc2. Even doing the repadmin /showreps tells me that
replication fails for 10000+ consecutive times. So currently, i am
just making changes to dc2 and wait for it to be replicated over, but
i dont know if there will be any chances that it will suddenly stop
replicating over to dc1.

As for creating a temp excahnge server, i guess that is pretty much
out as the company is on a very tight budget and there arent any spare
pcs around that i can make use of, but still i will try and look out
for any chance to get one for disaster recovery.

Pardon me for my ignorance, but if i pause the netlogon on dc1 and i
havnt setup a different exchange server then will my users be able to
logon to the exchange server to retrieve mails? I am not sure what
effects that pausing netlogon will have on the network. Are there any
sources/websites/books out there that can enlighten me on what effects
stopping/pausing any services in windows 2000 will have.

Thanx and regards,
Luke


Brett Shirley said:
Luke,

First My Suggestion:
I am still verifying with experts whether Norton Ghost does take an image of
the disk, w/o the proper AD backup methods. If it does, you're in a very
dangerous situation. At some point replication could go from bad to worse.
I think the unlucky case if they start replicating again, is actually just
at some point in the future for you.

My suggestion would be to migrate this Exchange server to your other server,
or backup the Exchange server, rebuild the AD replica DC from scratch, and
then restore the Exchange server. I'm not sure how feasible these
suggestions are b/c I'm not well an Exchange guru, just an AD
replication/backup/restore guru. However, I'm SURE that you will have AD
problems in the future if this DC remains on.

You _may_ be able to mitigate the likelyhood that the two systems start
replicating again, by pausing the netlogon service on the ghost restored DC.
This will ?hopefully? stop it from taking changes.


You asked, "What is the worst case?":
I'm glad you asked ... this is complicated, so I hope I can explain it
clearly. The potential problem relates to how we track replication
meta-data, and perform replication. Basically, a "first DC" tracks whether
it has seen a given replicated change from a 2nd DC by storing the
InvocationID (really a GUID) and a USN pair. The USN is a serial 64 bit
number and ever single transacted change has a USN stamped with it to
represent that change. If you imagine the USN always increasing for each
new change on a DC, you can see how every change in your AD forest, has an
InvocationID (i.e. a DC) and USN that represents that change.

Lets use some specific examples to make it clear, say the "highest USN" is
537 when the ghost image is taken of the 2nd DC, meaning their are changes
('changes' can be add users, password changes, new exch mailboxes, etc)
associated with USNs 1 through 536. After the ghost image is taken, changes
continue to trickle in, and each change replicates off to the first DC of
course. Each time replication happens, the first DC saves the "highest USN"
of the 2nd DC through the end of the changes it replicated from the 2nd DC.
Say USN 673 is the last "highest USN" of the last change the first DC
replicated from the 2nd DC right before the ghost image was restored.

Now, trouble ... when you now restore that ghost image, you take the 2nd DC
back in time, and the DC will have the USN 537 from the ghost image, while
the first DC will think it's seen the 2nd DC's changes through USN 673. New
changes by the 2nd DC will now get USNs 537, 538, 539, etc through 672, and
eventually beyond 672 as well. The first DC however will never get the 2nd
set of changes made at USNs 537-672, b/c it thinks it's seen all changes
before 673. So to start with you won't have the same data on the two DCs,
and this will never fix itself. Now things get really bad (as if that
wasn't bad enough ;) ... imagine that change USN 675 on the 2nd DC (which
will get replicated b/c it's after 673), depends on (like is a child object
of) a change that was made slightly earlier at USN 671. The first DC will
ask for changes 673 and up, and when it sees change 675 it will freak out,
b/c it claims to be above a child that couldn't possibly be there when the
first DC hasn't and won't replicate change 671. Other weird things can
happen to things like DN references (like a group's member attribute) that
reference objects that don't exist on one DC, etc.

I'm not sure if that made sense, but believe me; the basic data consistency
AD has tried to ensure is shot, and other apps that run on AD (like
Exchange) can start to misbehave. Cleaning up these inconsistencies could
be very very difficult, as you've taken AD very far outside it's normal
operating parameters. Microsoft PSS doesn't even have tools that I know of
to deal with this kind of thing.


Back to my first suggestion:
For this reason, it's vitally important to stop taking changes on this 2nd
DC, the ghost restored DC. WELL, of course this is all a mute point if Nort
on Ghost performs a regular AD backup somehow, then you wouldn't be
afflicted by the above horror. But I think this is unlikely to be the case,
but I'll repost again one way or the other, when I get a response from some
authoritative Norton/Symantec people.


Thanks,
Brett Shirley
AD Replication and AD Backup/Restore, SDE (that means Programmer)

This posting is provided "AS IS" with no warranties, and confers no rights.


Luke said:
Brett,

I formated a floppy using norton ghost wizard and the ghost program
was copied onto the floppy. i booted the DC from the floppy, loaded
the ghost program and do the ghost. After that i booted the DC back
into w2k.

After the crash the image was dumped back and it booted up fine with
the exception of not replicating and having a de-synchorisation of the
accounts with another DC. Looks like i got lucky this time. Just out
of curiousity, what could be the wrost thing that could happen if i
was unlucky?

I cannot shut down this DC as it is also an exchange server, and this
is the only mail server that we have, and thus i am not able to wipe
the DC out.

Will the Netdom command help in having the accounts synchronised? and
wil it have any negative effects on the network? What i want is just
to have the least disruption to the network.

thanx.
Luke

"Brett Shirley [MSFT]" <[email protected]> wrote in message
Luke,

Could you please describe how you got the "Ghosted image"? AD doesn't
support any sort of disk or file imaging that doesn't go through the
official backup/restore APIs. I believe, but am not really that sure, that
Norton Ghost does this. If you've done this, it is very very bad, and if
we're lucky, the only thing it's done is not replicate. Please stop trying
to get them to replicate, it's a good idea to shutdown this DC down or
unplug it from the network for now.

Is this the only DC for this domain? Is it plausible to wipe this DC and
bring it back online as a new replica DC?

Thanks,
Brett Shirley
SDE, AD Replication & AD Backup/Restore

Disclaimer: ( DmitriG posts alot, so I defer to his disclaimer ... )

I've CC'd Sorin, b/c from a quick web search, I saw he was trying to ghost a
DC too.
http://www.freelists.org/archives/windows2000/12-2002/msg00418.html


Hi Jason,
thanx for the reply. Checked my dns and the entries are correct.
I have tried forcing a replication but it fails with the message
target prinicpal name is incorrect and doing a repadmin /showreps
shows that the last successful replication is before the crash happen.
the error msg that i see from doing repadmin is "access is denied."

i was wondering if i need to do a netdom command between teh 2 DCs to
have their accounts synchronised, but i am not experienced enough to
know if there will be any side effects that this will have on the
otehr servers and clients taht are in the network.
Thanx.


"Jason Robarts [MSFT]" <[email protected]> wrote in
message
I'd check the DNS records again. Make sure the A records have the right
IP
addresses. What I was wondering was if the Ghosted image was too old.
If
the error messages reported are the most up to date then DNS configuration
looks to be the problem. Another approach to get information would be
to
force replication using Active Directory Sites and Services snap-in or
using
repadmin.exe. Then use repadmin /showreps to show the status of your
replication attempts. It will report last success times and last failure
times as well as an error code. You may find information on the error
code
on support.microsoft.com or by posting it again here. If DNS is configured
correctly the error message may give a clue as to the problem.


Hi Jason,

the ghosted image is about 3 weeks old.


message
How old is the Ghosted image?


Anybody got any ideas to this or have experienced this before?
 
B

Brett Shirley [msft]

Crud, forgot some steps ( also forgot to email you directly ;) ...

1h) After follwing the forced demotion from the KB article, seize the
roles that failed to transfer correctly in step 1c.

1i) Delete the metadata for DC1 (no longer a DC). (You can probably
find a KB article on how to do this properly, ntdsutil unfortunately
does an incomplete job)

.... then go on to step 2 ...

Thx,
BrettSh

<see below for disclaimer>
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top