How to Protect Electric Power Delivery Systems?

These days we see a lot of discussions on security in the domain of electric power delivery systems. One thing is for sure: The power delivery infrastructure is under heavy stress ... just to list a few issues:

1. Aging equipment (primary and secondary).
2. Increasing cyber attacks.
3. Increasing physical attacks.
4. Aging Workforce.
5. Political objective to reduce the rate per kWh of electric power consumed.
6. ...

A lot has been discussed recently regarding these and other issues.

Today I would like to have a brief look on the third bullet "Physical Attack". The Wall Street Journal (WSJ) published the other day a report on physical attacks of substations in the US: "Grid Attack: How America Could Go Dark". After reading these news I decided not to post anything about that report. But: When I got up this morning I read the (bad) news about the tragic attack on humans in Nice (France) last night with 80 people on the death toll of 80, I said to myself, I have to talk about these physical attacks.

First of all, our prayers are for the French people in general and especially for those that have lost one of their loved one, for those that are insured, and those that have experienced this attack.

Second, please read the WSJ report to understand the situation of our - partly very unprotected - electric power delivery system:

Click HERE for the report.

More or less the same could be reported about many substations worldwide.

Next time we may see a truck driving into a major substation, power plant, or high voltage transmission tower, ... How can we protect ourselves and the technical systems that are needed every second in our life?

2. Timothy 3:1-5 says: "1 But understand this, that in the last days there will come times of difficulty. 2 For people will be lovers of self, lovers of money, proud, arrogant, abusive, disobedient to their parents, ungrateful, unholy, 3 heartless, unappeasable, slanderous, without self-control, brutal, not loving good, 4 treacherous, reckless, swollen with conceit, lovers of pleasure rather than lovers of God, 5 having the appearance of godliness, but denying its power."

It is unlikely that all humans will understand the importance of the electric power delivery system (and other critical infrastructures) and to control themselves NOT TO TOUCH the system (AND of course other humans)! So, we have to do our best to better physically protect the crucial stations - which is better than do nothing. Attacks will continue to happen - but we have to spent more resources to increase the physical security.

We all have to accept the increase in your electric power bill - if we want to continue using power whenever we need it - 24/7. I hope that we learn better what the real value of our electric power infrastructure is for our daily life!

Old School versus New School - The Change of Generations in the Power Industry

Recently there was a conference in Cologne (Germany) held on how to improve human resources developments in power utilities. It seems that many managers expect that they fall short of hiring the best engineers.
Why is the situation in the power industry so crucial? There are many reasons - according to my personal experience. One crucial fear is the change of generations: Old School versus New School.
It is a pity that maintaining and improving the largest and the most crucial machines ever (the interconnected power delivery systems) are not so attractive to young people. What would young people do if there would not enough power to send their messages and photos back and forth? What would hospitals do without electric power? Or ...

Many discussions in the public are centered around one issue: getting cheap and cheaper power. I have experienced this quite often when HR managers start discussing with me to offer cheaper training courses ... I guess many of them have to learn that the change of generations of engineers will become THE challenge in the power industry. Why: Because the complexity of the secondary power system solutions has reached a level that can't easily be managed by inexperienced and less-motivated engineers. These engineers have to be trained in power systems theory AND information systems theory.

We (older) engineers had a chance to learn new technologies step by step over a long period - sometimes 20 or more years. These days we see new approaches being popping-up over night - more or less. The pressure is coming, e.g., from Internet technologies, Industry 4.0, and Smart Phones. These new possibilities will have a big impact on the way how power is generated, transmitted, and (of course) how it is distributed and used.

How do we get prepared for the future power delivery system? Solid education is one of the key issues and expensive.

But many HR managers want to get the best for free. The best things in life may be free. But not the power delivery. The real cost of power delivery will likely only be understood after a longer huge black-out.

One objective of HR managers is to hire more young and motivated engineers - another should be to give the experienced and interested engineers more chances to receive the education they need for the years to come.

Ask for a permission to attend a training - good luck!

IT Staff Needs Education in IEC 61850 and Related Standards

Information Technology (IT) is quite well understood by many (often young) experts. Operations Technology (OT) in the power industry is quite well understood by many (often older) experts with some tens of years experience in the use of hardware, automation and information exchange software.

These days we see the integration of Information Technology into the Operation Technology. The migration happens quite fast … and more and more experienced OP engineers are part of the “aging workforce”. This requires that new staff (IT experts and engineers) in the IT and in the OP world to learn two technologies: “IT”, “OT”, and very crucial: “More IT in OT”.

I found a very nice publication that discusses some of these issues in detail:

Click HERE for the paper “Information Technology (IT) vs Operations Technology (OT): What the C-Suite Needs to Know”

Based on my long-term experience since the early 80s with “More IT in OT” I fully agree with the need of Cross-Department Training and other needs listed in the paper:

“… considerations to improve the IT/OT rift:

• Annual Staff Surveys – to measure cooperation between IT and OT and look at utilization of resources, trust/conflict, clarity of goals and objectives, etc.
• Cross-Department Training – for both technical training but also support values and behaviors expected to foster cooperation and communications
• Cross-Functional Teams – to develop policies, standards, projects, etc. with both IT and OT perspectives
• Reach Over the Wall – encourage the IT and OT teams to reach out to the other technology team and be willing to walk in their shoes. This is also supported by the executive team taking the same actions.”

In some cases of big utilities I see this already happen!

IT and OP people would benefit from our training in IEC 61850 and related standards. Be aware that even a small embedded device could support TCP/IP, higher level protocols like ftp and http(s), IEC 6070-5-104 and IEC 61850, and … it is “IT in OT”.

How do we reach more IT and OT people? Please forward this information to them …

Can Power Outages impact the application of IEC 61850?

Yes – it can. Why? More often we receive reports on power outages caused by aging components of the grid. Upgrading many aged insulators, transformers, lines, … costs a lot of money! This money is not available for new technologies! Yes!

Yesterday it was an insulator that broke. The 220 kV line dropped to the ground and caused a blackout in the Washington (DC) and other areas. They were surprised that they “did not know why the outage rippled to far from the Ryceville switching station.”

Click HERE for a news message on yesterdays event at Ryceville switching station.

A similar incident happened back to 2009 in Auckland, New Zealand, which had a direct impact on IEC 61850 applications. Excerpt from a report on February 13, 2009:

“A power lines company is getting flak from the government after one of its conductors collapsed on to houses, trapping people in south Auckland. Amazingly, no-one was hurt when Transpower's monster 220,000 volt line came crashing down late on Friday morning.

The incident comes just 10 days after a Transpower transformer failed and plunged 75,000 Auckland residents and businesses into darkness for two hours.
"It's just totally unacceptable," says John Key, Prime Minister. "So look, there is going to be a massive expenditure on the upgrading of Transpower's grid and I'm making sure that that network is not only more reliable, but safer," he says.
Transpower is investigating the latest incident and feeling the political heat. "It's fallen into a residential area that's been built under the lines since the line was constructed in the fifties and sixties. But no, it's not good enough," says Kieran Devine, Transpower operations manager.”

Click HERE for the (old) news on the Transpower incident.

The Transpower incident had a direct influence on a project to get experience with IEC 61850 in substations. The project was stopped and my consultancy contract with Transpower was canceled soon after the incident happened.

The aging infrastructure is about to “eat” a good part of the funding for new technologies … including implementation of cyber-security measures. So it is no surprise that in some cases in the U.S we see routable protocols being replaced by serial links! This saves a lot of dollars.

Excerpt from a GarretCom paper on non-routable protocols:

“… When only non-routable protocols are used, substations with critical assets are networked
without requiring the use of Critical Cyber Assets (CCAs) at remote substations, as defined in CIPstandard CIP-002. Avoidance of “CCAs” means that the other CIP-002 to CIP-009 requirements do not
apply at these substations, which will likely defer significant implementation costs and ongoing
administrative overhead associated with CIP compliance.”

Click HERE for the complete report. See also HERE or HERE for more details on the NERC CIP on non-routable protocols.

Water Infrastructure in Los Angeles under “pressure”

Eight to 10 million gallons of (fresh!) water had been lost after major water main ruptured in Los Angeles yesterday.

Hm, the more than 90-year-old riveted steel pipe carries an estimated 75,000 gallons per minute … that’s a bit.

Click HERE for a report and video by Reuters.

The un-thinkable happened! Which part of the world-wide infrastructures breaks next?

Utility cuts IT workforce, hires Indian outsourcers

Northeast Utilities (NU) in Connecticut (USA) confirmed plans to outsource IT work to India-based outsourcers; move will affect some 200 jobs

Northeast Utilities confirmed the other day that it plans to turn over part of its IT operations to two India-based outsourcing firms, despite a recent push by state lawmakers to keep it from doing so.

NU says it employs some 400 IT workers, and "will retain about half of those employees" after turning some operations over to outsourcers Infosys and Tata Consultancy Services, two of India's largest IT firms.

Click HERE for more details.

Could this be a solution for automation, control, monitoring … ?

Here is another question somebody brought up discussing the NU approach:

“Would you outsource the autopilot to some inexpensive pilot on the ground in some foreign country with cheap labor? No? Then why is anyone even discussing such options for SCADA? And yet, they do!

The problem is education. Many utilities are putting poorly educated, poorly trained staff at the controls of our infrastructure, and then because it fails all the time, are seriously contemplating outsourcing the whole operation to another country.

Outsourcing to the cloud doesn't work. It's not about the technology. It's the people.”

I would add to state that another crucial problem is the education of people that make these decisions to outsource managing our infrastructure.

Hannover (Germany) Black Out Caused by Aging Infrastructure

Some 600,000 people of the city of Hannover (Northern Germany) were hit by a blackout (up to 81 minutes) on July 13, 2011.

First a power plant was tripped due to some internal problems. Then it was expected that a 220 kV / 110 kV transformer substation would transform more power from the 220 kV network into the 110 kV network – this would have balanced the load in the city and provided power from outside.

The transformer was protected by protection functions that prevent overloading (overheating) the transformer – in order to not damage the transformer windings by too high temperatures … Due to the increase of power flow after the power plant outage in the city, the protection tripped, because it decided that the flow was above the configured limit. The limit was correctly set – but a capacitor (worth a few Euro) was defect and the result was that the protection tripped the circuit breaker even at a current below the configured limit! It would be interesting to know how old the protection relay was.

Due to the aging of the hardware (capacitor) it happened in Hannover recently. What city or region will be next hit by the aging infrastructure of our electrical delivery system?

More to come.

More maintenance (as decided by the utility in Hannover) is not an anti-aging means – refurbishment of aged components of the infrastructures may help … but costs a lot of money.

Click HERE for the press release of the enercity utility of Hannover [German only].
Click HERE for a discussion of aging infrastructures [German only]