RPC probs with Ntfrs through firewall

[Guest]

Hi all

This is one I've been working on for a few months now. I've got two IIS6 web servers running W2003. Both are member servers in a 2003 native AD. We are trying to "mirror" the content of one server to another using DFS to provide a "hot backup" site if the production one goes down. The production server is in Virginia on a firewalled DMZ. The backup server is in Boston on our internal network. Each site also has a domain controller. Both servers are talking to their respective DC's and can use Kerberos through the firewall. We have the firewall rules to allow the the following ports between the two servers: 445, 139, 135, 88, and random RPC endpoints. This is a Checkpoint firewall, so it also has a specialized RPC rule that's supposed to keep state and allow through the random high port tcp connections that RPC likes to use. You put in different COM application GUID's and it's supposed to track them. I thought I got them all, but maybe I didn't. Also SMB works fine. No problem copying files. Also, there's only one replication connection, from VA to Boston. All changes are made on the VA server, then copied to Boston

The problem is not that it doesn't replicate. It does, but not very well. It will work for a bit, then it will start building up a huge backlog. In the Ntfrs debug logs, we keep getting the following entries on the Virginia server (the server names have been changed to virginiaserver.testcompany.com and bostonserver.testcompany.com)

<FrsHashCalcString: 3076: 4832: S0: 12:52:25> Name = S-1-5-21-1957994488-746137067-839522115-4455
<SndCsMain: 3504: 883: S0: 12:52:25> ++ ERROR - EXCEPTION (000006bf) : WStatus: RPC_S_CALL_FAILED_DN
<SndCsMain: 3504: 884: S0: 12:52:25> :SR: Cmd 04764ba0, CxtG d9f50492, WS RPC_S_CALL_FAILED_DNE, To bostonserver.testcompany.com Len: (66022) [SndFail - rpc exception
<VvJoinBuildTables: 3568: 1746: S0: 12:52:25> :V: VVJOIN ABORTED; MISMATCHED JOIN GUID
<MainVvJoin: 3568: 2253: S0: 12:52:25> :V: ERROR - FrsEnumerateTable failed. JStatus: KeyDuplicat
<MainVvJoin: 3568: 2338: S1: 12:52:25> :V: Stop vvjoin for unjoining DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346
<RcsForceUnjoin: 1204: 2960: S0: 12:52:25> :X: ***** UNJOINED DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346} -> CONUS\VASPFQWEB2$ RemoteCx
<RcsCheckCxtionCommon: 1204: 982: S1: 12:52:25> ++ WARN - RcsUnJoinCxtion: wrong unjoin guid for 04e2194

I've done some snfiffing with Netmon, and I can see that VA is connecting to Boston and successfully makes a bunch of RPC Bind and Request calls on different endpoints like it's supposed to. But I'm no RPC expert so I may be wrong

Any help

Here's a few lines of capture for the network gurus. I've stripped out the TCP overhead for clarity

Frm Time Proto Details From T
1 22.067081 MSRPC c/o RPC Request: call 0x1A96 opnum 0x3 context 0x0 hint 0x84 virginiaserver.testcompany.com bostonserver.testcompany.com
2 22.067081 MSRPC c/o RPC Response: call 0x1A96 context 0x0 hint 0x80 cancels 0x0 bostonserver.testcompany.com virginiaserver.testcompany.com
3 22.067081 MSRPC c/o RPC Bind: UUID F5CC59B4-4264-101A-8C59-08002B2F8426 call 0 virginiaserver.testcompany.com bostonserver.testcompany.com
4 22.067081 MSRPC c/o RPC Bind Ack: call 0x1 assoc grp 0x53B2377 xmit 0x16D0 recv bostonserver.testcompany.com virginiaserver.testcompany.com
5 22.067081 MSRPC c/o RPC Alt-Cont: UUID F5CC59B4-4264-101A-8C59-08002B2F8426 call 0 virginiaserver.testcompany.com bostonserver.testcompany.com
6 22.067081 MSRPC c/o RPC Alt-Cont Rsp: call 0x1 assoc grp 0x53B2377 xmit 0x16D0 recv bostonserver.testcompany.com virginiaserver.testcompany.com
7 22.067081 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
8 22.067081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
9 22.067081 MSRPC c/o RPC Request: call 0x12C opnum 0x0 context 0x0 hint 0x23C bostonserver.testcompany.com virginiaserver.testcompany.com
10 22.067081 MSRPC c/o RPC Response: call 0x12C context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
11 22.113956 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
12 22.113956 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
13 22.113956 MSRPC c/o RPC Request: call 0x12D opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
14 22.113956 MSRPC c/o RPC Response: call 0x12D context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
15 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
16 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
17 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
18 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
19 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
20 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
21 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
22 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
23 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
24 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
25 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
26 22.145206 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
27 22.160831 MSRPC c/o RPC Request: call 0x12E opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
28 22.160831 MSRPC c/o RPC Response: call 0x12E context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com
29 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
30 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
31 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
32 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
33 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
34 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
35 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
36 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
37 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
38 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
39 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
40 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..) virginiaserver.testcompany.com bostonserver.testcompany.com
41 22.192081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..) bostonserver.testcompany.com virginiaserver.testcompany.com
42 22.192081 MSRPC c/o RPC Request: call 0x12F opnum 0x0 context 0x0 hint 0x574 bostonserver.testcompany.com virginiaserver.testcompany.com
43 22.192081 MSRPC c/o RPC Response: call 0x12F context 0x0 hint 0x4 cancels 0x0 virginiaserver.testcompany.com bostonserver.testcompany.com

 

[Tom]

Quick Question
at any point were both servers in SAME lan SAME segment?
at any point were these 2 servers doing exactly what they were supposed to?

i have a similar situation except i am not dealing with iis at the moment.
but i am dealing with replication and consistancy of data.
we too have site in dc and site in boston. we further have a isolated
segment which was replicated from Production and now isolated by firewall so
both server environments now holds the same data and the isolated segment is
used strictly for testing purposes before anything is done to production.

recently testing segment had to be rebuilt. so we installed w2k srv on
testing server and joined production domain. next day all replication was
complete and now we know that it all works atleast once. now we can take
that server and put it in isolated segment and IT STILL WORKS AS IF IT IS
NOT ISOLATED. so any change in testing will reflect in production.

of all this only thing i am saying is
IF both servers were on same lan in same segment atleast 1 time till
everything worked like it should, you may have a much better luck.

shiva

Hi all,

This is one I've been working on for a few months now. I've got two IIS6

web servers running W2003. Both are member servers in a 2003 native AD. We
are trying to "mirror" the content of one server to another using DFS to
provide a "hot backup" site if the production one goes down. The production
server is in Virginia on a firewalled DMZ. The backup server is in Boston on
our internal network. Each site also has a domain controller. Both servers
are talking to their respective DC's and can use Kerberos through the
firewall. We have the firewall rules to allow the the following ports
between the two servers: 445, 139, 135, 88, and random RPC endpoints. This
is a Checkpoint firewall, so it also has a specialized RPC rule that's
supposed to keep state and allow through the random high port tcp
connections that RPC likes to use. You put in different COM application
GUID's and it's supposed to track them. I thought I got them all, but maybe
I didn't. Also SMB works fine. No problem copying files. Also, there's only
one replication connection, from VA to Boston. All changes are made on the
VA server, then copied to Boston.

The problem is not that it doesn't replicate. It does, but not very well.

It will work for a bit, then it will start building up a huge backlog. In
the Ntfrs debug logs, we keep getting the following entries on the Virginia
server (the server names have been changed to virginiaserver.testcompany.com
and bostonserver.testcompany.com):

<FrsHashCalcString: 3076: 4832: S0: 12:52:25> Name = S-1-5-21-1957994488-746137067-839522115-44557
<SndCsMain: 3504: 883: S0: 12:52:25> ++ ERROR -

EXCEPTION (000006bf) : WStatus: RPC_S_CALL_FAILED_DNE

<SndCsMain: 3504: 884: S0: 12:52:25> :SR: Cmd

04764ba0, CxtG d9f50492, WS RPC_S_CALL_FAILED_DNE, To
bostonserver.testcompany.com Len: (66022) [SndFail - rpc exception]

<VvJoinBuildTables: 3568: 1746: S0: 12:52:25> :V: VVJOIN ABORTED; MISMATCHED JOIN GUIDS
<MainVvJoin: 3568: 2253: S0: 12:52:25> :V: ERROR -

FrsEnumerateTable failed. JStatus: KeyDuplicate

<MainVvJoin: 3568: 2338: S1: 12:52:25> :V: Stop vvjoin

for unjoining
DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F
11-CBC21F9D1346}

<RcsForceUnjoin: 1204: 2960: S0: 12:52:25> :X: *****

UNJOINED
DCMA_INTRANET$\{18114BB4-FA6F-46C6-9F11-CBC21F9D1346}\{18114BB4-FA6F-46C6-9F
11-CBC21F9D1346} -> CONUS\VASPFQWEB2$ RemoteCxt

<RcsCheckCxtionCommon: 1204: 982: S1: 12:52:25> ++ WARN -

RcsUnJoinCxtion: wrong unjoin guid for 04e21940

I've done some snfiffing with Netmon, and I can see that VA is connecting

to Boston and successfully makes a bunch of RPC Bind and Request calls on
different endpoints like it's supposed to. But I'm no RPC expert so I may be
wrong.

Any help?

Here's a few lines of capture for the network gurus. I've stripped out the TCP overhead for clarity:

Frm Time Proto Details

From To

1 22.067081 MSRPC c/o RPC Request: call 0x1A96 opnum 0x3 context

0x0 hint 0x84 virginiaserver.testcompany.com
bostonserver.testcompany.com

2 22.067081 MSRPC c/o RPC Response: call 0x1A96 context 0x0 hint

0x80 cancels 0x0 bostonserver.testcompany.com
virginiaserver.testcompany.com

3 22.067081 MSRPC c/o RPC Bind: UUID

F5CC59B4-4264-101A-8C59-08002B2F8426 call 0
virginiaserver.testcompany.com bostonserver.testcompany.com

4 22.067081 MSRPC c/o RPC Bind Ack: call 0x1 assoc grp 0x53B2377

xmit 0x16D0 recv bostonserver.testcompany.com
virginiaserver.testcompany.com

5 22.067081 MSRPC c/o RPC Alt-Cont: UUID

F5CC59B4-4264-101A-8C59-08002B2F8426 call 0
virginiaserver.testcompany.com bostonserver.testcompany.com

6 22.067081 MSRPC c/o RPC Alt-Cont Rsp: call 0x1 assoc grp

0x53B2377 xmit 0x16D0 recv bostonserver.testcompany.com
virginiaserver.testcompany.com

7 22.067081 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

8 22.067081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..)

bostonserver.testcompany.com virginiaserver.testcompany.com

9 22.067081 MSRPC c/o RPC Request: call 0x12C opnum 0x0 context

0x0 hint 0x23C bostonserver.testcompany.com
virginiaserver.testcompany.com

10 22.067081 MSRPC c/o RPC Response: call 0x12C context 0x0 hint

0x4 cancels 0x0 virginiaserver.testcompany.com
bostonserver.testcompany.com

11 22.113956 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

12 22.113956 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..)

bostonserver.testcompany.com virginiaserver.testcompany.com

13 22.113956 MSRPC c/o RPC Request: call 0x12D opnum 0x0 context

0x0 hint 0x574 bostonserver.testcompany.com
virginiaserver.testcompany.com

14 22.113956 MSRPC c/o RPC Response: call 0x12D context 0x0 hint

0x4 cancels 0x0 virginiaserver.testcompany.com
bostonserver.testcompany.com

15 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

16 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

17 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

18 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

19 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

20 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

21 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

22 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

23 22.129581 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

24 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

25 22.145206 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

26 22.145206 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..)

bostonserver.testcompany.com virginiaserver.testcompany.com

27 22.160831 MSRPC c/o RPC Request: call 0x12E opnum 0x0 context

0x0 hint 0x574 bostonserver.testcompany.com
virginiaserver.testcompany.com

28 22.160831 MSRPC c/o RPC Response: call 0x12E context 0x0 hint

0x4 cancels 0x0 virginiaserver.testcompany.com
bostonserver.testcompany.com

29 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

30 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

31 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

32 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

33 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

34 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

35 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

36 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

37 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

38 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

39 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

40 22.160831 R_DRSUAPI RPC Client call drsuapi:IDL_DRSBind(..)

virginiaserver.testcompany.com bostonserver.testcompany.com

41 22.192081 R_DRSUAPI RPC Server response drsuapi:IDL_DRSBind(..)

bostonserver.testcompany.com virginiaserver.testcompany.com

42 22.192081 MSRPC c/o RPC Request: call 0x12F opnum 0x0 context

0x0 hint 0x574 bostonserver.testcompany.com
virginiaserver.testcompany.com

43 22.192081 MSRPC c/o RPC Response: call 0x12F context 0x0 hint

0x4 cancels 0x0 virginiaserver.testcompany.com
bostonserver.testcompany.com