Here’s a reminder of the need for region-wide smart grid systems, both to show power companies what their neighbors are doing in the midst of a blackout event, and to help themselves make sure their safety switches aren’t actually making things worse.

Such are the findings from a federal report released Tuesday, which found that operator errors and lack of planning played a key role in the September 2011 blackout that left about 9 million Southern California residents in a region-wide blackout that lasted through the night for most residents, and for days afterwards for a few.

Consider it proof that the best-laid power grid protection systems can’t prevent massive blackouts, if the people in charge of them don’t keep them up to snuff. To prevent similar failures in the future, the report recommends that “bulk power system operators improve their situational awareness through improved communication, data sharing and the use of real-time tools.”

The 151-page report revealed that power operators are making similar mistakes to those that caused the 2003 blackout that left 50 million people in the Eastern U.S. and Canada without power for days. Those included not updating grid models for real-time conditions and new equipment, and not heeding contingency plans to dispatch generation or shed load to keep the system stable.

That, in turn, allowed the loss of a single transmission line in Arizona to cascade into a region-wide power outage, shutting down California’s San Onofre nuclear reactor and leaving an estimated 2.7 million power customers of San Diego Gas & Electric customers without power for up to a day.

The confusion over how the blackout had happened was clear in the hours and days afterward, with Arizona utility APS saying that it hadn’t been able to isolate the chain of events that led to it. Indeed, Tuesday’s report from the Federal Energy Regulatory Commission (FERC) and the North American Electrical Reliability Corporation (NERC) required some hardcore computer modeling and delved into more than 20 gigabytes of data to figure out what happened that Thursday afternoon.

The first step in that process would presumably be in giving power system operators the knowledge of what’s happening in their neighboring systems. The report found that power authorities in Arizona and California’s Imperial Valley couldn’t know what the other was doing during the blackout, which shunted power from the disabled line onto parallel transmission systems that couldn’t handle the strain.

That, in turn, cut power to the San Onofre nuclear power plant in Orange County, which went offline as a safety measure -- one of the instances in which the report found that automatic safety systems actually made the problem worse. Automatic switches that isolated lines without taking the system-wide conditions into account also exacerbated the problem, the report found.

Investigators recommended that the region’s transmission operators and the Western Electricity Coordinating Council (WECC), a utility-government body that manages the Western U.S. grid, “study the effects of special protection systems, remedial action schemes and safety nets, to understand how they affect reliability and to ensure that they do not have unintended or undesirable effects.”

NERC is an industry body that can order utilities to take actions or face costly fines, though the report didn’t rise to the level of such an action. Still, it’s likely that the utility industry is taking note of pronouncements like these.

Synchrophasors, which closely monitor transmission line performance to allow action to prevent such massive failures, are one technology being put to use on the grid. WECC is leading a federal grant-backed project to test them out on a region-wide basis, and Southern California Edison has taken the next step and automated some transmission operations based on what the synchrophasors are reporting.

Behind all this gear, we’re going to need massive computing power to manage the constant flow of data in real time. IT giants like IBM, Microsoft and Oracle are the kings of this kind of system-of-systems smart grid work, along with consultancies like Accenture and Capgemini, grid giants like Siemens and Alstom (via UISOL), contractors like Lockheed Martin and SAIC, and software vendors like Autodesk that own parts of the complex set of variables that have to be fed into a real-time, working grid model.

Transmission grids already have fast communications networks and lots of remote control capability, which makes the task easier. But getting them all to run in a coherent way, based on goals of the region rather than each individual operator, will also require a big shift in business model and regulatory thinking to succeed. The first step would be for FERC and NERC to back up their recommendations with some enforcement.

Tags: accenture, california, capgemini, cisco, demand response, doe, efficiency, energy efficiency, enernoc, ge, ibm, microsoft, policy, san diego, siemens