Failed Win XP Upgrade Wipes Out UK Government Agency

H

hermes

"Most of the desktop computers in the UK's Department for Work and
Pensions were paralyzed for four days on Monday, when a failed upgrade
took them off line. The outage, covering 75-80 per cent of the DWP's
80,000 PCs, is one of the largest in the UK Government's not entirely
impressive IT history."
http://politics.guardian.co.uk/homeaffairs/story/0,11026,1360163,00.html

"Pension and benefit payments face disruption after what is being
described as the biggest computer crash in government history left as
many as 80,000 civil servants staring at blank screens and reverting to
writing out giro cheques by hand in the latest blow to a hi-tech
Whitehall revolution.
A week-long crisis in the giant Department for Work and Pensions created
a backlog of unprocessed claims with up to 80% of the ministry's 100,000
desk machines disrupted or knocked out by a blunder during maintenance.
Engineers battling to fix the problem last night claimed 95% were
functioning fully again as they prepared to reboot the entire network
after offices closed to the public.
Alan Johnson, the work and pensions secretary, has ordered an internal
inquiry into the role of Microsoft and the American contractors EDS, who
run the ministry's network as part of a £2bn information technology deal.
The disruption is the latest in a line of government technology failures
and follows last week's resignation of the head of the Child Support
Agency, part of Mr Johnson's empire, after the disastrous introduction
of an EDS system contributed to only one in eight parents receiving the
correct amount.
The DWP said some new and amended benefit claims this week would be
delayed but it sought to play down the impact of the technology
problems, pointing out that the department's mainframe computers were
not affected.
But internal DWP correspondence seen by the Guardian, backed up by
interviews with staff, appears to contradict public assertions that the
disruption was minimal and most of the system continued operating normally.
A "major incident report" distributed on Monday warned of "major
problems", with hourly updates issued to senior managers by fax or
telephone because email on the department's intranet was blocked.
The DWP established a "crisis management centre" to resolve the
problems, with Microsoft troubleshooters flown in from mainland Europe
to join a high-level team including EDS technicians.
A memo circulated yesterday within jobcentres said 30% of problems could
be eased by today with "a full solution potentially taking another 24-48
hours", and the difficulties running into the weekend.
"At this point there is no known solution or ETA," said the memo. "We
are hopeful that some interim measures that are being considered may
release some users from their current deadlock."
A routine software upgrade on a small number of PCs last weekend is
believed to have gone disastrously wrong when an incompatible system was
downloaded on to the whole network.
The DWP said last night that progress had been swifter than expected,
insisting contingency plans had worked.
"Pens came back out and in many cases hot desking was used," said a
senior official. "The problem was quite random so you could potentially
be sitting down at one desk and your mate next door couldn't access the
system."
Trade union leaders called on ministers to drop plans to cut 40,000 jobs
in the DWP and a total of 104,000 civil servants across the government
following the computer crisis.
Mark Serwotka, general secretary of the Public and Commercial Services
Union, said: "Just when you thought it couldn't get any worse after the
experiences of the CSA we have what can only described as near-meltdown
with IT across the whole of the DWP.
"Yet again we are seeing thousands of hardworking staff, many of whom
face the axe, trying to deliver essential services with one hand tied
behind their back.
"The department and the government are hellbent on axing thousands of
civil and public servants, saying IT will enable them to do so, but yet
again we are seeing IT systems come to a grinding halt and fail.
"For the government and the department to contemplate axing thousands of
jobs when the IT clearly isn't delivering is not only irresponsible and
foolhardy, but some would argue pure madness."
Texas-based EDS has already seen £12.1m withheld by the government from
an ill-fated £450m CSA project. Mr Johnson is said to be considering
scrapping the contract.
Labour has spent some £1.5bn on over-budget or scrapped computer
projects, many inherited from the Conservatives, prompting National
Audit Office reports.
The Swanwick air traffic control centre, due to open in 1996 at a cost
of £475m, started six years late and £180m over-budget, while £300m was
spent on a scheme, later scrapped, to use plastic cards to pay benefits
via post offices.
EDS failed to get an Inland Revenue contract renewed after complaints
about its work.
Keith Wylie, a PCS national officer, said: "I cannot remember a crash
this big."
He added: "It's a massive failure and unless it's fixed quickly it's
going to result in significant delays in benefits being paid out. If it
was two years down the line and those 40,000 staff had been lost, there
would have been no one to write out the giros."
http://politics.guardian.co.uk/homeaffairs/story/0,11026,1360163,00.html

I wonder if the UK National Health Service is regretting awarding
Microsoft a £500 million contract now?

--
hermes
DRM sux! Treacherous Computing kills our virtual civil liberties!
http://protectfreedom.tripod.com/index.html
http://www.cl.cam.ac.uk/~rja14/tcpa-faq.html
http://anti-dmca.org/
http://www.eff.org/IP/DMCA/unintended_consequences.php

Windows XP crashed.
I am the Blue Screen of Death.
No one hears your screams
 
H

hermes

Bill said:
Please point out in this article where it say this was a Windows XP upgrade that caused the problem. Why are you intentionally trying to mislead with the subject line you used?

Why are you assuming that I am "intentionally trying to mislead with the
subject line" when I'm not? The subject doesn't say "failed win xp
upgrade failed because of win xp", it says "Failed Win XP Upgrade Wipes
Out UK Government Agency". Would you be happier with "Failed upgrade of
Microsoft's latest operating system..." ? I worded the subject the way
it is because that is where they had the problem. As a matter of fact,
they don't officially know what caused the problem yet if you read the
articles. My, but you seem touchy.

--
hermes
DRM sux! Treacherous Computing kills our virtual civil liberties!
http://protectfreedom.tripod.com/index.html
http://www.cl.cam.ac.uk/~rja14/tcpa-faq.html
http://anti-dmca.org/
http://www.eff.org/IP/DMCA/unintended_consequences.php

Windows XP crashed.
I am the Blue Screen of Death.
No one hears your screams
 
B

Bruce Chambers

hermes said:
Bill James wrote:
Why are you assuming that I am "intentionally trying to mislead with the
subject line" when I'm not? The subject doesn't say "failed win xp
upgrade failed because of win xp", it says "Failed Win XP Upgrade Wipes
Out UK Government Agency".

Because absolutely no where in the news article you cite does it state
(or even hint, for that matter) that the "routine software upgrade" that
caused the failure was an upgrade to WinXP, or to any other Microsoft
product.

--

Bruce Chambers

Help us help you:



You can have peace. Or you can have freedom. Don't ever count on having
both at once. - RAH
 
M

M

If someone really upgraded 80,000 computers without testing the upgrade
first then they have only themselves to blame!
 
T

tinyang

CS said:
It doesn't say anything at all about XP. Here's the article:

http://news.independent.co.uk/uk/politics/story.jsp?story=587262
My mistake, I did not post the link to the other article about it. Here
is the one which states that the upgrade was from win 2000 to winxp.
http://www.theregister.co.uk/2004/11/26/dwp_network_outage/

--
hermes
DRM sux! Treacherous Computing kills our virtual civil liberties!
http://protectfreedom.tripod.com/index.html
http://www.cl.cam.ac.uk/~rja14/tcpa-faq.html
http://anti-dmca.org/
http://www.eff.org/IP/DMCA/unintended_consequences.php

Windows XP crashed.
I am the Blue Screen of Death.
No one hears your screams
 
A

Alex Nichol

Bill said:
Please point out in this article where it say this was a Windows XP upgrade that caused the problem. Why are you intentionally trying to mislead with the subject line you used?

It has been mentioned in responsible news sources here (eg BBC), quoting
(and indeed interviewing) officials from the Ministry concerned, that it
arose when doing 'an update' to MS Windows. So installing SP2 is the
suspect; and installing it but leaving the firewall blocking the
networks is my best guess
 
L

Leythos

So installing SP2 is the
suspect; and installing it but leaving the firewall blocking the
networks is my best guess

Installing SP2 does indeed cause problems since the firewall is enabled
by default, and since it only permits File/Printer sharing and RD. MS
should have disabled the firewall for Domain base systems. When we
rolled out Sp2 to a client it caused all of our remote control apps to
be blocked, but they could still access their servers and work normally.

I suspect that any computer company worth it's salt would have know
about the SP2 firewall and that SP2 was not what caused the problem for
them. In any case, if it was SP2, it would have been easy to spot and
correct quickly.
 
B

Bruce Chambers

tinyang said:
My mistake, I did not post the link to the other article about it. Here
is the one which states that the upgrade was from win 2000 to winxp.
http://www.theregister.co.uk/2004/11/26/dwp_network_outage/

No, that's not really what the article says. This article reports that
a 3rd party IT contractor mistakenly applied a WinXP patch to Win2K
machines. With a monumental screw-up like that, is it any wonder that
machines blue-screened?

I do admit some surprize, however, as this is the first Register
article I've ever seen that wasn't rabidly anti-Microsoft; it actually
exonerates Microsoft.


--

Bruce Chambers

Help us help you:



You can have peace. Or you can have freedom. Don't ever count on having
both at once. - RAH
 
C

Chris

It has been mentioned in responsible news sources here (eg BBC), quoting
(and indeed interviewing) officials from the Ministry concerned, that it
arose when doing 'an update' to MS Windows. So installing SP2 is the
suspect; and installing it but leaving the firewall blocking the
networks is my best guess

Further information available here:

http://tinyurl.com/3slu4

- rgds, Chris
 
L

Leythos

Further information available here:

http://tinyurl.com/3slu4

according to the details "According to staff reports the outage occurred
on Monday afternoon, disconnecting staff email, benefits processing and
Internet and intranet connectivity. According to one, a limited network
upgrade from Windows 2000 to Windows XP was taking place, but instead of
this taking place on only a small number of the target machines, all the
clients connected to the network received a partial, but fatal,
'upgrade.'"

This appears to have nothing to do with Windows XP or Windows 2000, it's
a problem that would have happened under any OS based on what they
describe.
 
B

Bob I

I REALLY hope you're not planning on posting every article where some
idiot managed to hose a Windows PC!
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top