Dual Adsl connection in W2003 server

We are experimenting with dual ADSL connections to spread the load of
Terminal Services.

Problems so far are connections in from ADSL1 are being redirected back out
of the other gateway and into oblivion.

I've re-worked the system and now have 2 lan cards on different sub nets.
Each has an ADSL router and each card has it's own gateway address. I can
now connect on both ADSL boxes but only for a short time after which the
connection is dropped.

I think that some packets are still going out on the wrong gateway. eg
ADSL1 sometimes routes back out onto ADSL2 ?

Any ideas, I've tried adding some static routes but it's not helping.