Very Strange Citrix Problem going on 6 months HELP!

[Craig N.]

Ok, got a big problem here, so hopefully someone can
help,we've been at this for a loooong time.

Got a client, maybe 200 users or so, about 100 are
on Citrix using thin clients. I have nice servers, all HP
DL380's, Dual 3.06 Xeons, 2 gig ram, 4 36 gig hard
drives. My specific server configurations are:

2 domain controllers replicating to each other. Both are
running active directory, Win'2000 Service Pack 3, DNS,
and DHCP, which is giving out DNS addressing.

4- Citrix servers running Office, Lotus, ACT, and BPCS.

1- EMC SAN, clarion unit.
1- Print server
1- SQL server for terminal services licensing.
(also got lotus server, bpcs server, and etc)

OK, so, here is the problem. The client had a citrix
environment running on off-brand/generic servers, and
they upgraded to these new HP DL380's, a day later, they
got major network lag using Citrix, it hangs when opening
programs, typing gets real slow, and etc.

My team was brought in to repair this problem. We got
some rental servers, and put them on the live network,
and took down the current dl380 servers and rebuilt them
from scratch, everything was formatted and redone. Same
problem appeared when we tested it.

The specific problem that we have found is that it seems
to work fine, until someone opens up word or something,
and clicks on the little open document button (the little
folder icon near the file menu), then everyone hangs for
90-180 seconds. You dont get the problem if you go to
file explorer or My network places. Also, it slows down
as more people log in, and is eventually unusable, at
about 10 people, giving a near 4 minute login. No prolems
on any other servers with logging in, and browsing the
network, only the citrix boxes.

Now, if you log in onto the server itself, on the local
machine account, you dont get the error, even if all the
citrix users are hung up, your running great.

I am also running a login script, and changed it from
a .cmd to a kixtart script, and all it does it map about
4 drives, and copy BPCS user data to a local drive at the
very first login.

So, here is what I have done:
-rebuilt all the servers from scratch
-Applied all the patches
-removed printers
-removed local serial connections
-upped the paging size
-verified DNS, active directory, and etc with 2 other
engineers
-Changed the NIC card
-applied registry tweaks
-disaled all non-essential services
-changed and rewrote the login script
-reinstalled the storage (san)

The logs are not showing any errors, so I've got nothing
to go off of. The client is running a mix of swtiches and
hubs. Intel Express hubs connect all the workstations,
and the servers are on Cisco 2900 series switches and HP
2524 switches.

So, anyone have any idea? This system has been completely
rebuilt by completely different people, and we have the
same problem. We even have a Citrix expert that helped
write one of the first ICA clients that is working with
us, and he is baffled. It doesnt make any sense at all,
but i gotta get it working. Anyone?????

Could it be network trouble, maybe the switches? I dunno,
going nuts, its 3:00 am, and I am still working on this,
I'm going to end up in some looney bin soon.

 

[Vera Noest [MVP]]

Do you get the same problem when you connect to the Citrix server
with an rdp client in stead of an ica client?

When this problem occurs, how is CPU and memory use (in Task
Manager on the Citrix server)? Any processes that monopolize the
CPU?

Since you seem to be able to repeat the problem at will by opening
a document from within Word, have you tried to monitor the system
with Performance Monitor and Network Monitor when the problem
occurs?

Does the problem also occur when you doubleclick on a Word
document to open it? Does it happen with *any* application, also
notepad? What is the default search path for the Open document
dialog in Word?

You write that you "applied all the patches" to the servers. But
since you are still on SP3, I find that hard to believe. Do you
really mean that you applied *all* patches that were contained in
SP4 *as separate hotfixes* + *all post-SP4 hotfixes*???
Reason why I ask is that there are numerous performance problems
solved in SP4 and post-SP4 hotfixes. I can't really list them all,
it would be easier if you listed the hotfixes you have installed.

Which version of Citrix are you running?

 

[Guest]

I havent tried an RDP yet, but if I log a new user into
the server console on the domain and open my computer a
few times the system hangs, then after about 90 seconds
it stops hanging, and I can not recreate the problem. Let
me try the word thing through RDP though.

As far as processes and and CPU usage, nothing is
spiking, the only thing I had seen was a Compaq WBEM
service taking up a lot of memeory, but killed that.

If I go into explorer, I can actually highlight 200
documents, hit open, and they all start opening, no
problems. It seems to only occur through office.

I need to double check the default search path, but I am
pretty sure its the local my documents folder, it could
also be going to a mapped drive on the EMC that has a my
documents folder.

We have done everything up to service pack 3, the all the
citrix patches we can think of on the new network. The
old network that is having the SAME problem is at Service
pack 4. It wil be hard to list the post hotfixes also, we
have TONS of them installed, and none of it repaired it.

 

[Tim Schoch]

Is this problem only affecting the thin clients? What
brand of thin client are they? I have seen many cases
with WYSE Terminals that in office applications, they
perform very poorly. To be more specific, in MSACCESS a
database with a form causes serious lagtime. When a user
on a WYSE Terminal uses the mouse to select a field in the
form, it will lock the session and the cursor will blink
very rapidly for about 15-60sec. You may want to look
into that if the issue is similar.

 

[Guest]

Compaq Evo clients. I'd give it consideration, but we
also have people that are not on thin clients that have
the same problem.