"An operations error occurred"

[Andy]

The anonymous auth is typically when there is either some sort of delegation
problem or a problem with Kerberos auth itself. What happens is that if the
service impersonating the user cannot authenticate via Kerberos for some
reason, it will try to authenticate via NTLM instead. However, it doesn't
have any NTLM credentials it can use for the impersonated user, so it
instead logs in the NT AUTHORITY\ANONYMOUS user.

Ok, I've looked at a good number of the anonymous logons; the
workstation generating them is either the other DC, my workstation,
the database server, my testing server and my development server.

The first thing I would do there is find out if Kerb auth to SQL itself is
broken or whether the problem is just a function of delegation not working.
Connecting to SQL with a known good client that can do Kerb auth and
checking that audit would be a good way to start. It is possible that you
have an SPN problem in your directory related to SQL. This is especially
likely if you run SQL under an account other than System or Network Service
and did not explicitly set an SPN in AD on the actual account you used.

Well, connecting with SSMS didn't seem to cause anonymous logons, but
my program doesn't seem to be either. So I think those other
anonymous logons were unrelated.

Sql Server is running as the Local System account, except for sql
agent which is running as Network Service.

Is it ok for you to be authenticating to SQL as anonymous? That sort of
begs the question of why try to delegate in the first place.

I don't think anything is authenticating to Sql as anonymous;
certainly not my application, which would fail to run properly.
Except for some low security views where [public] is allowed, all
procs and functions and other views are tied to specific roles only.

Andy

 

[Andy]

Thanks for sticking with me on this.. my comments inline.

- Is the IIS app pool account enabled for delegation in AD? If so, what
type of delegation?

The app pool is running under the preconfigured Network Service
account. I'm not sure where I would check that account for
delegation; I don't see a Network Service user in the AD users and
computers snap-in. None of the other AD snap-ins I know of show it
either.. I didn't see anything in Local Security Policy regarding
delegation on the server running IIS.

- In the case where you see the anonymous user authentications in SQL, did
the authentication that came in to IIS use Kerberos or NTLM?

I'm only seeing Anonymous user connections on the local DC here.
According to the logs, they are coming from one of the servers I
mentioned, or my own workstation. I don't see any anonymous
connections in the logs on the iis / sql box itself.

- Is the problem here just for this one particular user on this one
machine, or do you see it more often?

The problem is for a particular user on a particular machine. My
account (which is a member of Domain Admin) works fine on her
workstation. Additionally, I tried one of my other user's account on
the problem workstation, and the application runs fine there as well.
So this seems to be a combination of this one particular user on one
particular workstation.

- With successful auths in SQL, do they all use Kerberos at both the SQL
and IIS tier?

The only time I see NTLM as the authentication is when the ANONYMOUS
user comes into play; otherwise all Success audits always say
Kerberos.

It may be the case that using protocol transition can help you here. If the
problem is that your remoting client sometimes uses NTLM instead of
Kerberos, you can still get Kerberos delegation to work by enabling "allow
to delegate with any protocol" in AD. You will have to specify which
services you want to delegate to (just SQL and AD here unless there are
other things you didn't mention), but that is a good idea anyway.

Well so far I'm only having an issue reading from Active Directory;
the user with the problem does seem to be able to logon to the
database server fine via the remoting, but that might be because IIS
and Sql are both running on the exact same server.

When my application starts, it first tries to connect to the database
to see if an Employee record exists for the UserName specified by the
WindowsIdentity. If not, and only in this case, it will create the
record by reading values from the Active Directory. Of course because
of a new requirement, I'll soon always read from Active Directory, and
then updating the Employee record at each logon. So I'm hoping you
can help me work this out, because if it comes up again I'd like to
know how to fix it.

As far as the remoting going, I'm using Csla.Net to build my business
layer, and it has functionality to allow a configuration file switch
to change between talking to the database directly, or going through
some other channel... in my case .Net remoting.

Thanks!
Andy

 

[Andy]

- Is the problem here just for this one particular user on this one
machine, or do you see it more often?

Hmm... this is more interesting. I fired up Virtual PC and started
WinXPSP2, and logged on as the user having problems. The issue
reproduced itself there.. but for some reason did not on my Vista
workstation (all the other users are XP though). Additionally, the
same thig happens on my managers XP workstation. So the only place
her account has worked was on my Vista workstation...

Andy

 

[Andy]

- Is the problem here just for this one particular user on this one
machine, or do you see it more often?

I must have done something wrong when I last tested with Vista; using
my own workstation also reproduces the issue.. so at least that's
consistent on every workstation I've used her account on.

Sorry for the bad info.. but hopefully this means there's an easy
solution.

Andy

 

[Joe Kaplan]

Regarding the delegation setting, the System and Network Service local
accounts use the domain computer account when doing "network stuff" such as
Kerb auth, so you would find the delegation settings, you would look in AD
Users and Computers for the computer account for the IIS box and check its
delegation tab in the properties.

I'm assuming that is does have delegation enabled.

The other thing to check then is in the case of the failing machines, do you
see the authentication from the remoting call into the IIS box generating a
success audit that says Kerberos or NTLMssp? I would expect NTLM here. The
question would then be WHY Kerb isn't working for these particular clients.

Joe K.

 

[Andy]

Regarding the delegation setting, the System and Network Service local
accounts use the domain computer account when doing "network stuff" such as
Kerb auth, so you would find the delegation settings, you would look in AD
Users and Computers for the computer account for the IIS box and check its
delegation tab in the properties.

Ahh, ok. The database / IIS server has Do not trust this computer for
delegation checked.

The other thing to check then is in the case of the failing machines, do you
see the authentication from the remoting call into the IIS box generating a
success audit that says Kerberos or NTLMssp? I would expect NTLM here. The
question would then be WHY Kerb isn't working for these particular clients.

Well, yesterday and today, even workstation which I try the user's
account exhibits the same problem. Even on workstation of my other
users where it DOES work for them. Going back to your original idea..
where would an account get permissions to read from Active Directory?

On the database server, I see lots of successful logons from the
workstations I used and the user account who is having the problem.

On the DC, I see successful Kerberos logons from the user, some
anonymous connections, followed by more successful Kererous logons.

The anonymous logons are coming from the database / iis server. Other
anonymous logons come from my development server, which is running
SourceGear's Fortress, under a fixed domain account specified on the
Application Pool. But that seems to be the extend of anonymous
logons.

 

[Joe Kaplan]

Ok, this is more good info. It was not clear to me that SQL and IIS were on
the same box. Since they are, you should not need delegation in order to
get the authentication to flow from IIS to SQL. However, since AD is on a
different box, you DO need delegation in order to flow the credentials from
IIS to AD.

One thing you might consider to make that requirement go away is to NOT
impersonate while doing calls to AD. Since you really just need to read
some data to get user info, using the process account is probably fine for
that. So, you might consider undoing impersonation programmatically during
the LDAP call and then turning it back on. Alternately, you could disable
impersonation in web.config and then just programmatically impersonate while
making your SQL calls. A third option would be to supply fixed credentials
in your DirectoryEntry constructor. If you do that, then the current thread
identity is ignored and your impersonation settings won't matter (except if
you use a serverless binding string, in which case the server to use is
still bootstrapped from the security context of the current thread).

In general, Authenticated Users should have rights to read most stuff in AD.
The thing is that anonymous does not count as an authenticated user. By
default, AD 2003 does not allow anonymous searches, so it returns an
operations error when code attempts to perform a search when authenticated
as anonymous. The computer account of the IIS server probably has rights to
read all of the data in AD that you need to see though.

Joe K.

 

[Andy]

Ok, this is more good info. It was not clear to me that SQL and IIS were on
the same box. Since they are, you should not need delegation in order to
get the authentication to flow from IIS to SQL. However, since AD is on a
different box, you DO need delegation in order to flow the credentials from
IIS to AD.

Ya, I was trying to make that clear, I guess I didn't though. I
understand what you are saying and it makes sense but my question is,
why does this same setup work for all my other users?

One thing you might consider to make that requirement go away is to NOT
impersonate while doing calls to AD. Since you really just need to read
some data to get user info, using the process account is probably fine for
that. So, you might consider undoing impersonation programmatically during
the LDAP call and then turning it back on. Alternately, you could disable
impersonation in web.config and then just programmatically impersonate while
making your SQL calls. A third option would be to supply fixed credentials
in your DirectoryEntry constructor. If you do that, then the current thread
identity is ignored and your impersonation settings won't matter (except if
you use a serverless binding string, in which case the server to use is
still bootstrapped from the security context of the current thread).

If I have to, I suppose I'll supply credentials. Since I'd have to
embed the connection information somewhere, how would I create a user
account that ONLY has permission to read user objects from Active
Directory? No other logon rights should be allowed.

I'm still curious too as to why this setup works for all my other
users; domain\bob, domain\sue work fine on any workstation.. but domain
\stacy doesn't anywhere.

In general, Authenticated Users should have rights to read most stuff in AD.
The thing is that anonymous does not count as an authenticated user. By
default, AD 2003 does not allow anonymous searches, so it returns an
operations error when code attempts to perform a search when authenticated
as anonymous. The computer account of the IIS server probably has rights to
read all of the data in AD that you need to see though.

Well it seems like everyone else is connecting as an Authenticated
User, because it works for all my other users. Or maybe not.. as I
said, it only reads from AD when my system detects there's no user
information in the database. Let me test one of my other user's
accounts. Perhaps something has changed recently in a patch from MS?

 

[Andy]

Ok, this is more good info. It was not clear to me that SQL and IIS were on
the same box. Since they are, you should not need delegation in order to
get the authentication to flow from IIS to SQL. However, since AD is on a
different box, you DO need delegation in order to flow the credentials from
IIS to AD.

One thing you might consider to make that requirement go away is to NOT
impersonate while doing calls to AD. Since you really just need to read
some data to get user info, using the process account is probably fine for
that. So, you might consider undoing impersonation programmatically during
the LDAP call and then turning it back on. Alternately, you could disable
impersonation in web.config and then just programmatically impersonate while
making your SQL calls. A third option would be to supply fixed credentials
in your DirectoryEntry constructor. If you do that, then the current thread
identity is ignored and your impersonation settings won't matter (except if
you use a serverless binding string, in which case the server to use is
still bootstrapped from the security context of the current thread).

In general, Authenticated Users should have rights to read most stuff in AD.
The thing is that anonymous does not count as an authenticated user. By
default, AD 2003 does not allow anonymous searches, so it returns an
operations error when code attempts to perform a search when authenticated
as anonymous. The computer account of the IIS server probably has rights to
read all of the data in AD that you need to see though.

Ok, I removed the employee record for another user on my test system,
and tried her account. I now get the same error as user 0. The last
user created when my application starts up happened on 2/8/2008. So I
can only assume something has changed that causes my code to no longer
work as it did previously. I assume this change came in the form of a
patch, since we generally keep up with all patches as they are
released to our WSUS server, and neither myself nor my manager change
delegation settings in AD; mostly we create new computer and user
accounts, that's about it.

Andy

 

[Joe Kaplan]

This is fun, isn't it?

I doubt this is a patch issue as much as some sort of environment
difference. It is possible that at one point the IIS box WAS enabled for
delegation and then that got turned off for some reason. That could have
caused it to stop working. Who knows? Since we don't really know why it
was working before, it may be very difficult to figure this out.

Based on what I understand at this point, I think you have a couple of
options to get this to work. The first decision is whether to implement the
trusted subsystem security architecture for the AD query or the delegated
model. In the trusted subsystem design, a fixed service account queries AD
on behalf of the user. In the delegated model, the app uses the credentials
of the orginally authenticated user to perform the query.

IMO, trusted subsystem is fine here as you are only doing a read operation
and any user's credentials are essentially equivalent to any others' to do
the query. Delegation doesn't really buy you anything in your security
model like it does with SQL. Your SQL security model depends on delegation.
You could still delegate if you want to, but it isn't really required.

If you want to delegate, you should just need to enable the IIS computer
account as trusted for delegation and it SHOULD work. Assuming Kerberos
auth to IIS for the remoting clients, it will then be able to delegate to
other Kerberos services. Constrained delegation and protocol transition may
also be things to consider. Note that you will have to do this if you ever
need to move SQL to a different box.

If you don't want to delegate, you can either use one of the alternate
impersonation strategies I mentioned or use fixed credentials. If you are
uncomfortable with fixed credentials and don't want to create a special user
for this case, then undoing impersonation programmatically or disabling in
web.config and doing the impersonation programmatically during the SQL calls
may be the better option.

If you want to create a service account but restrict what it can do, I'd
suggest starting a new thread on that. As long as only you have the creds,
it should not be open to abuse anyway.

Best of luck!

 

[Andy]

This is fun, isn't it?

In a confusing, hair pulling kind of way, yes.

I doubt this is a patch issue as much as some sort of environment
difference. It is possible that at one point the IIS box WAS enabled for
delegation and then that got turned off for some reason. That could have
caused it to stop working. Who knows? Since we don't really know why it
was working before, it may be very difficult to figure this out.

Well, I'm at a loss then, because I know I didn't change anything, and
my manager said he didn't either.. and we're pretty much the only ones
that could.

Based on what I understand at this point, I think you have a couple of
options to get this to work. The first decision is whether to implement the
trusted subsystem security architecture for the AD query or the delegated
model. In the trusted subsystem design, a fixed service account queries AD
on behalf of the user. In the delegated model, the app uses the credentials
of the orginally authenticated user to perform the query.

IMO, trusted subsystem is fine here as you are only doing a read operation
and any user's credentials are essentially equivalent to any others' to do
the query. Delegation doesn't really buy you anything in your security
model like it does with SQL. Your SQL security model depends on delegation.
You could still delegate if you want to, but it isn't really required.

If you want to delegate, you should just need to enable the IIS computer
account as trusted for delegation and it SHOULD work. Assuming Kerberos
auth to IIS for the remoting clients, it will then be able to delegate to
other Kerberos services. Constrained delegation and protocol transition may
also be things to consider. Note that you will have to do this if you ever
need to move SQL to a different box.

If you don't want to delegate, you can either use one of the alternate
impersonation strategies I mentioned or use fixed credentials. If you are
uncomfortable with fixed credentials and don't want to create a special user
for this case, then undoing impersonation programmatically or disabling in
web.config and doing the impersonation programmatically during the SQL calls
may be the better option.

If you want to create a service account but restrict what it can do, I'd
suggest starting a new thread on that. As long as only you have the creds,
it should not be open to abuse anyway.

Ok, I'll do the trusted subsystem approach; I'll open a new thread.

Thanks!
Andy