Error 768 - failure to encrypt

[Todd hobdey]

I'm setting up a Windows 2003 Server as a VPN server for L2TP/IPSec. If a
take some random XP PC and hook it to the server with a crossover cable I
can get an L2TP tunnel going just fine. But if I take my laptop (also XP)
and do the same thing with exactly the same settings I get:

Error 768: The connection attempt failed because of failure to encrypt data.

I'm using a preshared key instead of a certificate and I know that I don't
have a typo (it's pretty hard to mistype "1").

Now for the kicker - I have Microsoft Virtual PC installed on said laptop.
From both 98 and XP virtual sessions I CAN connect to the server, so it's
not anything intrinsic to the NIC or the cabling. And the laptop itself can
connect just fine if I revert to PPTP/MPPE.

I don't want to start rolling this out if I'm going to have seemingly random
problems with some machines. Anyone have any thoughts/suggestions?

 

[Todd hobdey]

A new twist that undoubtedly related - I'm trying to follow a Microsoft
troubleshooting article and when I launch the IP Security Monitor snap-in, I
get "The IPSec server is unavailable or incompatible with the IPSec
monitor". The only information on that message is that I'm running it on
Windows 2000, which I'm not.

 

[Kadirvel C Vanniarajan [MSFT]]

Are you sure the IPSec Services is running on your machine?

 

[Todd hobdey]

Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came back
from that okay, but I still can't launch the MMC snap-in.

Are you sure the IPSec Services is running on your machine?

--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

A new twist that undoubtedly related - I'm trying to follow a Microsoft
troubleshooting article and when I launch the IP Security Monitor

snap-in,

I
get "The IPSec server is unavailable or incompatible with the IPSec
monitor". The only information on that message is that I'm running it on
Windows 2000, which I'm not.

If

a

cable

I itself
can

 

[Kadirvel C Vanniarajan [MSFT]]

You might want to check the following details:
1. Check if there is a binary mismatch. Ipsecnp.dll, Oakley.dll,
winipsec.dll and ipsecsvc.dll must be from the same build. The MMC error
suggested that there may be mismatch between winipsec.dll and ipsecsvc.dll.

2. Is policy agent actually up and running? (sc query policyagent).
The netdiag test on W2k3 will not indicate if ipsec modules are initiliazed
and running.

If these are not going to help, enable the RAS tracing (using netsh ras set
trace * enable) and also enable oakley logging (details in
http://support.microsoft.com/default.aspx?scid=kb;en-us;257225) get the logs
where the IPSec Monitor is failing. The ras logs will be found under
%windir%\tracing and the oakley logs can be found under %windir%\debug.


--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came back
from that okay, but I still can't launch the MMC snap-in.

Are you sure the IPSec Services is running on your machine?

--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

A new twist that undoubtedly related - I'm trying to follow a Microsoft
troubleshooting article and when I launch the IP Security Monitor

snap-in,

I
get "The IPSec server is unavailable or incompatible with the IPSec
monitor". The only information on that message is that I'm running it on
Windows 2000, which I'm not.


I'm setting up a Windows 2003 Server as a VPN server for L2TP/IPSec.

If

a
take some random XP PC and hook it to the server with a crossover

cable

I
can get an L2TP tunnel going just fine. But if I take my laptop

(also
XP)

and do the same thing with exactly the same settings I get:

Error 768: The connection attempt failed because of failure to encrypt
data.

I'm using a preshared key instead of a certificate and I know that I don't
have a typo (it's pretty hard to mistype "1").

Now for the kicker - I have Microsoft Virtual PC installed on said laptop.
From both 98 and XP virtual sessions I CAN connect to the server, so it's
not anything intrinsic to the NIC or the cabling. And the laptop itself
can
connect just fine if I revert to PPTP/MPPE.

I don't want to start rolling this out if I'm going to have seemingly
random
problems with some machines. Anyone have any thoughts/suggestions?

 

[Todd hobdey]

When I posted this, the file versions were:
winipsec.dll 5.1.2600.0
ipsecsnp.dll 5.1.2600.0
oakley.dll 5.1.2600.1106

"sc query policyagent" shows the service running.

I came across article 818043 relating to NAT-T and applied it thinking that
might have some bearing on the issue. It did, in that it changed the
problem, but I still have a problem.

After the update, the file versions on all of the files are 5.1.2600.1240.
When I try to launch the IP Security snap-in, I now get "The binding handle
is invalid", instead of the original error.

The rasapi32.log shows:

[1260] 09:15:42: RasEnumConnectionsW
[1260] 09:15:57: RasEnumConnectionsW

The tapisrv.log shows

[1332] 09:15:39:380: [INFO ] Service control code=4

And the only entry in the Oakley log is from startup, and that is:

1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 10045
1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 0
1-19: 07:00:06:20:1f4 Startup failed. Shutting down
1-19: 07:00:06:20:1f4 Initialization failed. Exiting
1-19: 07:00:06:20:1f4 Wait is done
1-19: 07:00:06:20:1f4 Before send_deletes
1-19: 07:00:06:20:1f4 AFter send_deletes
1-19: 07:00:06:20:1f4 Begin Wait. isadb_clean_socket 0
1-19: 07:00:06:20:1f4 End Wait. isadb_clean_socket
1-19: 07:00:06:20:1f4 Begin Wait. isadb_kill_old 0
1-19: 07:00:06:20:1f4 End Wait. isadb_kill_old
1-19: 07:00:06:20:1f4 Begin Wait. ActiveRpcCalls 0
1-19: 07:00:06:20:1f4 End Wait. ActiveRpcCalls

You might want to check the following details:
1. Check if there is a binary mismatch. Ipsecnp.dll, Oakley.dll,
winipsec.dll and ipsecsvc.dll must be from the same build. The MMC error
suggested that there may be mismatch between winipsec.dll and ipsecsvc.dll.

2. Is policy agent actually up and running? (sc query policyagent).
The netdiag test on W2k3 will not indicate if ipsec modules are initiliazed
and running.

If these are not going to help, enable the RAS tracing (using netsh ras set
trace * enable) and also enable oakley logging (details in
http://support.microsoft.com/default.aspx?scid=kb;en-us;257225) get the logs
where the IPSec Monitor is failing. The ras logs will be found under
%windir%\tracing and the oakley logs can be found under %windir%\debug.


--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came back
from that okay, but I still can't launch the MMC snap-in.

Are you sure the IPSec Services is running on your machine?

--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

A new twist that undoubtedly related - I'm trying to follow a Microsoft
troubleshooting article and when I launch the IP Security Monitor snap-in,
I
get "The IPSec server is unavailable or incompatible with the IPSec
monitor". The only information on that message is that I'm running

it
on

Windows 2000, which I'm not.


I'm setting up a Windows 2003 Server as a VPN server for

L2TP/IPSec.
If

a
take some random XP PC and hook it to the server with a crossover cable
I
can get an L2TP tunnel going just fine. But if I take my laptop (also
XP)
and do the same thing with exactly the same settings I get:

Error 768: The connection attempt failed because of failure to encrypt
data.

I'm using a preshared key instead of a certificate and I know that I
don't
have a typo (it's pretty hard to mistype "1").

Now for the kicker - I have Microsoft Virtual PC installed on said
laptop.
From both 98 and XP virtual sessions I CAN connect to the server, so
it's
not anything intrinsic to the NIC or the cabling. And the laptop itself
can
connect just fine if I revert to PPTP/MPPE.

I don't want to start rolling this out if I'm going to have seemingly
random
problems with some machines. Anyone have any thoughts/suggestions?

 

[Kadirvel C Vanniarajan [MSFT]]

Todd,

Can you check if there are any event logs indicating a file version change?
Also, from the KB article, can you check if the file versions that are
present on your machine matches that of those present in the QFE package?

There still seems to be an issue with the fileversions.

--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

When I posted this, the file versions were:
winipsec.dll 5.1.2600.0
ipsecsnp.dll 5.1.2600.0
oakley.dll 5.1.2600.1106

"sc query policyagent" shows the service running.

I came across article 818043 relating to NAT-T and applied it thinking that
might have some bearing on the issue. It did, in that it changed the
problem, but I still have a problem.

After the update, the file versions on all of the files are 5.1.2600.1240.
When I try to launch the IP Security snap-in, I now get "The binding handle
is invalid", instead of the original error.

The rasapi32.log shows:

[1260] 09:15:42: RasEnumConnectionsW
[1260] 09:15:57: RasEnumConnectionsW

The tapisrv.log shows

[1332] 09:15:39:380: [INFO ] Service control code=4

And the only entry in the Oakley log is from startup, and that is:

1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 10045
1-19: 07:00:06:20:1f4 Obtain WSARecvMsg pointer failed 0
1-19: 07:00:06:20:1f4 Startup failed. Shutting down
1-19: 07:00:06:20:1f4 Initialization failed. Exiting
1-19: 07:00:06:20:1f4 Wait is done
1-19: 07:00:06:20:1f4 Before send_deletes
1-19: 07:00:06:20:1f4 AFter send_deletes
1-19: 07:00:06:20:1f4 Begin Wait. isadb_clean_socket 0
1-19: 07:00:06:20:1f4 End Wait. isadb_clean_socket
1-19: 07:00:06:20:1f4 Begin Wait. isadb_kill_old 0
1-19: 07:00:06:20:1f4 End Wait. isadb_kill_old
1-19: 07:00:06:20:1f4 Begin Wait. ActiveRpcCalls 0
1-19: 07:00:06:20:1f4 End Wait. ActiveRpcCalls

You might want to check the following details:
1. Check if there is a binary mismatch. Ipsecnp.dll, Oakley.dll,
winipsec.dll and ipsecsvc.dll must be from the same build. The MMC error
suggested that there may be mismatch between winipsec.dll and ipsecsvc.dll.

2. Is policy agent actually up and running? (sc query policyagent).
The netdiag test on W2k3 will not indicate if ipsec modules are initiliazed
and running.

If these are not going to help, enable the RAS tracing (using netsh ras set
trace * enable) and also enable oakley logging (details in
http://support.microsoft.com/default.aspx?scid=kb;en-us;257225) get the logs
where the IPSec Monitor is failing. The ras logs will be found under
%windir%\tracing and the oakley logs can be found under %windir%\debug.


--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

Yes it is. I ran "netdiag /test:ipsec /v /debug" and everything came back
from that okay, but I still can't launch the MMC snap-in.


message Are you sure the IPSec Services is running on your machine?

--
Kadir

(e-mail address removed) [MSFT]
This posting is provided "AS IS" with no warranties, and confers no
rights.

A new twist that undoubtedly related - I'm trying to follow a Microsoft
troubleshooting article and when I launch the IP Security Monitor
snap-in,
I
get "The IPSec server is unavailable or incompatible with the IPSec
monitor". The only information on that message is that I'm

running

it

that

I

server,

so