D
Daeron
Software Bug Contributed to Blackout
Kevin Poulsen Feb 11 2004
A previously-unknown software flaw in a widely-deployed General
Electric energy management system contributed to the devastating scope
of the August 14th northeastern U.S. blackout, industry officials
revealed this week.
[Unknown as it didn't exist until it was needed as a scapegoat in
order to distract from the real reason. Sounds to me like FirstEnergy
trying to deflect blame onto GE
]
The bug in GE Energy's XA/21 system was discovered in an intensive
code audit conducted by GE and a contractor in the weeks following the
blackout, according to FirstEnergy Corp., the Ohio utility where
investigators say the blackout began.
'It had never evidenced itself until that day," said spokesman Ralph
DiNicola. "This fault was so deeply embedded, it took them weeks of
poring through millions of lines of code and data to find it.'
[ Who is this contractor ? What code are they referring to here ? How
were the tests conducted ? Did they include any other systems that
were involved in the BlackOut ? How many of the SCADA units running on
FirstEnergy were Microsoft Windows ?
"FirstEnergy was aware the alarm system was broken, said company
spokesman Ralph DiNicola. A functioning backup alarm at the Midwest
Independent System Operator, a nonprofit power pool that oversees the
region's electrical grid, was in place," DiNicola said.
http://www.nipc.gov/dailyreports/2003/August/DHS_IAIP_Daily_2003-08-18.pdf
]
The flaw was responsible for the alarm system failure at FirstEnergy's
Akron, Ohio control center that was noted in a November report from
the U.S.-Canadian task force investigating the blackout. The report
blamed the then-unexplained computer failure for retarding
FirstEnergy's ability to respond to events that lead to the outage,
when quick action might have limited the blackout's spread.
Power system operators rely heavily on audible and on-screen alarms,
plus alarm logs, to reveal any significant changes in their system's
conditions," the report noted. FirstEnergy's operators "were working
under a significant handicap without these tools. However, they were
in further jeopardy because they did not know that they were operating
without alarms, so that they did not realize that system conditions
were changing.
[.. TRANSCRIPTS of telephone conversations .. include explicit
mention of some unknown 'computer problems' at FirstEnergy, the Ohio
utility thought to have triggered the regional power failures, in
those preceding hours.
Early on, a controller at the Midwest Independent System Operator
asked his counterpart at FirstEnergy why it hadn't reacted to a
transmission line outage. The utility's technician replied:
"We have no clue. Our computer is giving us fits, too. We don't even
know the status of some of the stuff around us."
"I called you guys like 10 minutes ago, and I thought you were
figuring out what was going on there."
"Well, we're trying to. Our computer is not happy. It's not
cooperating either."
]
The cascading blackout eventually cut off electricity to 50 million
people in eight states and Canada.
The blackout occurred at a time when the Blaster computer worm was
wreaking havoc across the Internet. The timing triggered some
speculation that the virus may have played a role in the outage -- a
theory that gained credence after SecurityFocus reported that two
systems at a nuclear power plant operated by FirstEnergy had been
impacted by the Slammer worm earlier in the year.
[ "On January 25, 2003, Davis-Besse nuclear power plant was infected
with the MS SQL Server 2000 worm. The infection caused data overload
in the site network, resulting in the inability of the computers to
communicate with each other."
http://www.nrc.gov/reading-rm/doc-collections/gen-comm/info-notices/2003/in200314.pdf
]
Instead, the XA/21 bug was triggered by a unique combination of events
and alarm conditions on the equipment it was monitoring, DiNicola
said. When a backup server kicked-in, it also failed, unable to handle
the accumulation of unprocessed events that had queued up since the
main system's failure.
Because the system failed silently, FirstEnergy's operators were
unaware for over an hour that they were looking at outdated
information on the status of their portion of the power grid,
according to the November report.
[What were these 'unique combination of events and alarm conditions'
? ... This is Poulson in an earlier article about the earlier systems
crash at a Nuclear Plant.
"In that article, Poulsen offers a detailed description of how another
Microsoft worm, Slammer, crashed two Unix-based control systems at the
Davis-Besse nuclear power plant in Northern Ohio also operated by
FirstEnergy.
Poulsen reported that FirstEnergy engineers had bridged the nuclear
plant's control network with FirstEnergy's corporate network -- a
practice that is increasingly common among utility companies,
according to industry and security experts."
http://www.newsforge.com/software/03/09/09/1526221.shtml?tid=78
What number of SCADA units on this system were running Windows ? What
effect on the total monitoring system would a Windows SCADA system
being contaminated with a virus.
]
"The root cause of the outage was linked to .. trees .. FirstEnergy
says .. its role in the outage is overstated in the interim report"
[shuffle .. shuffle]
from http://www.securityfocus.com/news/8016
[Retrospective ass covering is all. I guess General Electric can't
afford as much protection on Capitol Hill as MICROS~1. Get those
cheque books out guys. It's election year!!!
]
["Specifically, key personnel may not have been aware of the need to
take preventive measures at critical times, because an alarm system
was malfunctioning."
"The existence of both internal and external links from SCADA systems
to other systems introduced vulnerabilities."
https://reports.energy.gov/BlackoutReport-5.pdf
]
Reliable, Field-Proven & Adaptable
The XA/21 transmission management system controls generation and the
high voltage transmission network for optimal generation and
transmission of power.
One of the industry's most advanced EMS/SCADA systems, the XA/21
system combines advanced open systems architecture with full graphics,
power system application, historical information storage and retrieval
and relational database technology.
With well over one million hours of online operation, the XA/21 system
has improved utilities' bottom lines by helping to:
...
• Enhance operational efficiency
...
http://www.gepower.com/prod_serv/products/scada_software/en/xa21.htm
[note: 'Enhance operational efficiency'. That's management speak for
it takes less people to operate.
]
[What is SCADA]
http://www.hackfaq.org/data_networks-23.shtml
]
quote from Bill Gates, Feb 14 1998
" ... It would help me immensely to have a survey showing that 90
percent of developers believe that putting the browser into the OS
makes sense. ... Ideally, we would have a survey like this done before
I appear at the Senate on March 3rd."
http://www.internetwk.com/news0199/news011599-3.htm
Bill Gates Feb 2004 .. what he might have said :-D
'It would help me immensely to have a survey showing that the blackout
was caused by Unix ... Ideally, we would have a survey like this done
before I appear at the RSA Conference in Feb'
Kevin Poulsen Feb 11 2004
A previously-unknown software flaw in a widely-deployed General
Electric energy management system contributed to the devastating scope
of the August 14th northeastern U.S. blackout, industry officials
revealed this week.
[Unknown as it didn't exist until it was needed as a scapegoat in
order to distract from the real reason. Sounds to me like FirstEnergy
trying to deflect blame onto GE
]
The bug in GE Energy's XA/21 system was discovered in an intensive
code audit conducted by GE and a contractor in the weeks following the
blackout, according to FirstEnergy Corp., the Ohio utility where
investigators say the blackout began.
'It had never evidenced itself until that day," said spokesman Ralph
DiNicola. "This fault was so deeply embedded, it took them weeks of
poring through millions of lines of code and data to find it.'
[ Who is this contractor ? What code are they referring to here ? How
were the tests conducted ? Did they include any other systems that
were involved in the BlackOut ? How many of the SCADA units running on
FirstEnergy were Microsoft Windows ?
"FirstEnergy was aware the alarm system was broken, said company
spokesman Ralph DiNicola. A functioning backup alarm at the Midwest
Independent System Operator, a nonprofit power pool that oversees the
region's electrical grid, was in place," DiNicola said.
http://www.nipc.gov/dailyreports/2003/August/DHS_IAIP_Daily_2003-08-18.pdf
]
The flaw was responsible for the alarm system failure at FirstEnergy's
Akron, Ohio control center that was noted in a November report from
the U.S.-Canadian task force investigating the blackout. The report
blamed the then-unexplained computer failure for retarding
FirstEnergy's ability to respond to events that lead to the outage,
when quick action might have limited the blackout's spread.
Power system operators rely heavily on audible and on-screen alarms,
plus alarm logs, to reveal any significant changes in their system's
conditions," the report noted. FirstEnergy's operators "were working
under a significant handicap without these tools. However, they were
in further jeopardy because they did not know that they were operating
without alarms, so that they did not realize that system conditions
were changing.
[.. TRANSCRIPTS of telephone conversations .. include explicit
mention of some unknown 'computer problems' at FirstEnergy, the Ohio
utility thought to have triggered the regional power failures, in
those preceding hours.
Early on, a controller at the Midwest Independent System Operator
asked his counterpart at FirstEnergy why it hadn't reacted to a
transmission line outage. The utility's technician replied:
"We have no clue. Our computer is giving us fits, too. We don't even
know the status of some of the stuff around us."
"I called you guys like 10 minutes ago, and I thought you were
figuring out what was going on there."
"Well, we're trying to. Our computer is not happy. It's not
cooperating either."
]
The cascading blackout eventually cut off electricity to 50 million
people in eight states and Canada.
The blackout occurred at a time when the Blaster computer worm was
wreaking havoc across the Internet. The timing triggered some
speculation that the virus may have played a role in the outage -- a
theory that gained credence after SecurityFocus reported that two
systems at a nuclear power plant operated by FirstEnergy had been
impacted by the Slammer worm earlier in the year.
[ "On January 25, 2003, Davis-Besse nuclear power plant was infected
with the MS SQL Server 2000 worm. The infection caused data overload
in the site network, resulting in the inability of the computers to
communicate with each other."
http://www.nrc.gov/reading-rm/doc-collections/gen-comm/info-notices/2003/in200314.pdf
]
Instead, the XA/21 bug was triggered by a unique combination of events
and alarm conditions on the equipment it was monitoring, DiNicola
said. When a backup server kicked-in, it also failed, unable to handle
the accumulation of unprocessed events that had queued up since the
main system's failure.
Because the system failed silently, FirstEnergy's operators were
unaware for over an hour that they were looking at outdated
information on the status of their portion of the power grid,
according to the November report.
[What were these 'unique combination of events and alarm conditions'
? ... This is Poulson in an earlier article about the earlier systems
crash at a Nuclear Plant.
"In that article, Poulsen offers a detailed description of how another
Microsoft worm, Slammer, crashed two Unix-based control systems at the
Davis-Besse nuclear power plant in Northern Ohio also operated by
FirstEnergy.
Poulsen reported that FirstEnergy engineers had bridged the nuclear
plant's control network with FirstEnergy's corporate network -- a
practice that is increasingly common among utility companies,
according to industry and security experts."
http://www.newsforge.com/software/03/09/09/1526221.shtml?tid=78
What number of SCADA units on this system were running Windows ? What
effect on the total monitoring system would a Windows SCADA system
being contaminated with a virus.
]
"The root cause of the outage was linked to .. trees .. FirstEnergy
says .. its role in the outage is overstated in the interim report"
[shuffle .. shuffle]
from http://www.securityfocus.com/news/8016
[Retrospective ass covering is all. I guess General Electric can't
afford as much protection on Capitol Hill as MICROS~1. Get those
cheque books out guys. It's election year!!!
]
["Specifically, key personnel may not have been aware of the need to
take preventive measures at critical times, because an alarm system
was malfunctioning."
"The existence of both internal and external links from SCADA systems
to other systems introduced vulnerabilities."
https://reports.energy.gov/BlackoutReport-5.pdf
]
Reliable, Field-Proven & Adaptable
The XA/21 transmission management system controls generation and the
high voltage transmission network for optimal generation and
transmission of power.
One of the industry's most advanced EMS/SCADA systems, the XA/21
system combines advanced open systems architecture with full graphics,
power system application, historical information storage and retrieval
and relational database technology.
With well over one million hours of online operation, the XA/21 system
has improved utilities' bottom lines by helping to:
...
• Enhance operational efficiency
...
http://www.gepower.com/prod_serv/products/scada_software/en/xa21.htm
[note: 'Enhance operational efficiency'. That's management speak for
it takes less people to operate.
]
[What is SCADA]
http://www.hackfaq.org/data_networks-23.shtml
]
quote from Bill Gates, Feb 14 1998
" ... It would help me immensely to have a survey showing that 90
percent of developers believe that putting the browser into the OS
makes sense. ... Ideally, we would have a survey like this done before
I appear at the Senate on March 3rd."
http://www.internetwk.com/news0199/news011599-3.htm
Bill Gates Feb 2004 .. what he might have said :-D
'It would help me immensely to have a survey showing that the blackout
was caused by Unix ... Ideally, we would have a survey like this done
before I appear at the RSA Conference in Feb'