How To Bring Plant Engineers To The Table When Cyber Issues Are Discussed?

In my career as electrical and IT engineer I have experienced that engineers are quite often not invited to discuss the measures and plans for critical infrastructure protection with IT personnel.

It is completely different compared to the world of electric power system protection - I mean the applications of protection relays. Protection engineers are (in my understanding) the most crucial engineers. They are very important for the reliable delivery of electric power. Protection engineers are likely to attend any meeting when it comes to the reliability of the power flows. Protection engineers know what to do ... software people may help to implement the "what" and the IT personnel may help to solve the communication issues ... but the crucial parts are dominated by protection engineers!

Mr. Vytautas Butrimas, a globally well known engineer involved in cyber security of control systems has briefly discussed the "Berlin wall" between IT personnel and plant engineers.  

Click HERE for the four page paper written by Mr. Butrimas.

Either of the groups involved believes that his or her group is the center of universe. There is little communication between the IT personnel and the engineers. 

There are so many semipermeable walls between, e.g., politicians, company lawyers, economists, IT experts, and plant engineers. There is usually no way that experts from any layer are allowed to talk to the experts from the other layers. In the end: Each layer feels independent of the other layers ... which leads to what we see these days ... and may be even more in the future. Have you heard of a discussion between a power protection engineer and a lawyer or even a medical doctor?

It would help medical doctors to understand the basics of electric power system reliability ... and so on. Because medical doctors (and all other people of a society) depend 100% on available power.

So in the end: (Electrical) Engineers should be honored by the society ... the problem may be that the engineers are not wearing white coats but wear safety boots, safety helmets, goggles,  protective gloves, ... a single doctor may harm a few people ... a protection engineer may harm millions of people during a blackout caused by a misconfiguration of protection equipment.

How Serious Are You About Cyber Security For Power Systems?

I know: A lot has been talked and written about Cyber Security for power delivery and many other systems.

BUT: What about insurance that specify coverage for cyber damage? 

You may figure out that your company has insurance covering cyber damage. So far - so good!
Be careful and read the latest development regarding the question, if all damages will be covered by your policy.
Please check the following report and ensure this article is made available to all senior managers and executives immediately ... a famous case (Merck) explains that there may be cases where the insurance companies may not pay at all ... in case of big bang attack ...

Click HERE for the wake-up call for everybody - from Bloomberg!!

How to Protect Electric Power Delivery Systems?

These days we see a lot of discussions on security in the domain of electric power delivery systems. One thing is for sure: The power delivery infrastructure is under heavy stress ... just to list a few issues:

1. Aging equipment (primary and secondary).
2. Increasing cyber attacks.
3. Increasing physical attacks.
4. Aging Workforce.
5. Political objective to reduce the rate per kWh of electric power consumed.
6. ...

A lot has been discussed recently regarding these and other issues.

Today I would like to have a brief look on the third bullet "Physical Attack". The Wall Street Journal (WSJ) published the other day a report on physical attacks of substations in the US: "Grid Attack: How America Could Go Dark". After reading these news I decided not to post anything about that report. But: When I got up this morning I read the (bad) news about the tragic attack on humans in Nice (France) last night with 80 people on the death toll of 80, I said to myself, I have to talk about these physical attacks.

First of all, our prayers are for the French people in general and especially for those that have lost one of their loved one, for those that are insured, and those that have experienced this attack.

Second, please read the WSJ report to understand the situation of our - partly very unprotected - electric power delivery system:

Click HERE for the report.

More or less the same could be reported about many substations worldwide.

Next time we may see a truck driving into a major substation, power plant, or high voltage transmission tower, ... How can we protect ourselves and the technical systems that are needed every second in our life?

2. Timothy 3:1-5 says: "1 But understand this, that in the last days there will come times of difficulty. 2 For people will be lovers of self, lovers of money, proud, arrogant, abusive, disobedient to their parents, ungrateful, unholy, 3 heartless, unappeasable, slanderous, without self-control, brutal, not loving good, 4 treacherous, reckless, swollen with conceit, lovers of pleasure rather than lovers of God, 5 having the appearance of godliness, but denying its power."

It is unlikely that all humans will understand the importance of the electric power delivery system (and other critical infrastructures) and to control themselves NOT TO TOUCH the system (AND of course other humans)! So, we have to do our best to better physically protect the crucial stations - which is better than do nothing. Attacks will continue to happen - but we have to spent more resources to increase the physical security.

We all have to accept the increase in your electric power bill - if we want to continue using power whenever we need it - 24/7. I hope that we learn better what the real value of our electric power infrastructure is for our daily life!

Should Power Grids Put Their Critical Digital Systems Off?

Power delivery systems worldwide are under heavy stress: physical stress and stress caused by shareholders and hackers and ... The stress is heavy often due to very limited resources that hinder engineers to improve the system very much.

Some people believe that the solution may be lying in going back to the Old Days! They want to spend USD 10.000.000+ for studying to go ,Back to analog and non-digital control systems, purpose-built control systems, and physical controls. Who has said this? ... some 20 years ago? No: this month!

The motto of some US congress man seems to be: Get rid of state-of-the-art technology.

A corresponding bill was assigned to a congressional committee on June 6, 2016:
Click HERE for more details.
Click HERE to download the text of the bill [pdf].

Click HERE for a discussion published under nextgov.com.

Does this mean the end of digital protection and automation systems? The end of communication according to IEC 61850, IEC 60870-5, DNP3, Modbus, ...?

What is needed? More well educated engineers that can use the digital technology in a way that the power delivery system can be managed securely and that become able to understand how the technology can be applied in order to re-start the power delivery system after a blackout.

I would like to see 10 per cent of the budget (USD 1.000.000) spent into education for protection, automation, SCADA and communications engineers. Have you ever tried to get approval for attending a training course for advanced protection, automation, SCADA, and communications like IEC 61850, or ...??

My experience after I run more than 230 courses worldwide and educated more than 4.100 engineers is this: many engineers that have asked me for a quote to conduct an in-house course or attending a public course had to give up due to budget restrictions!!

The other USD 10.000.000 could be spent for improving the digital based equipment as Cris Thomas, a security expert (see link to nextgov.com above), said: "Instead of spending two years and $10 million exploring ways to downgrade critical systems with even more outdated tech, we should instead invest that time and money into transforming security for the technology currently in place, and into building next-generation security features directly into future technology."

If utilities want to change the way the run their assets the management and stake holders should listen to the engineers!! And follow their recommendations! This is even more crucial in case of implementing more physical control done by human beings!

Badly educated engineers could do more harm than well operating machines.

Whatever we want to do to "ruggedize" our power delivery system, we need more well educated and experienced engineers. Retiring senior engineers with 52 years (to reduce costs) is not a real option! Or? We need you all! And we need young people to study electric power systems and information technology.

Click HERE for some additional discussion by myself (German version).

Draft IEC 62351-13 TR - Guidelines on what security topics should be covered in standards and specifications

IEC TC 57 just published a very interesting draft technical report (57/1678/DTR):
IEC 62351-13 TR: Power systems management and associated information exchange -
Data and communications security -
Part 13: Guidelines on what security topics should be covered in standards and specifications
Voting terminates on 2016-04-15

The draft covers the following topics:


Excerpt from the document:
"1.2 Purpose of this Document
The security requirements for human users and software applications are different from the purely
technical security requirements found in many communication and device standards. For user security standards, more emphasis must be on “policy and procedures” and “roles and authorization” rather than “bits and bytes” cryptographic technologies that should be included in Information and Communications Technology (ICT). In addition, engineering practices and system configurations must be taken into account, since no cryptography can compensate for poor design."

As an excerpt not this single bullet: "Validation of information input for format and reasonability, including that the input is in the correct format, that values are within limits, that the values are not beyond the capabilities of the automation system."

There is always something to better understand!

Want to Understand one of the Largest Machines - The Interconnected European Electric Power Grid?

The Interconnected European Electric Power Grid is one of the biggest machines built by humans. It has been developed over a period of about some 130 years. It is a miracle that it is still working very stable and more or less uninterrupted form many years.

The challenge for the future is this: How to keep the power flowing, the grass green and the sky blue. I met with a retired - but still very active - power engineer yesterday. We discussed how more information technologies can be used to support a very reliable automation system to provide 24x7 power flow all over in Europe. We have figured out that one of the key challenges in the discussions is to find the correct language in our discussions. I mean: When I talk about preventing any "remote control command", what does the recipient of that term understand? We figured out that we have discussed this term for years - but did have a different understanding in mind!

Fortunately we solved our disconnect and were happy that we have the same understanding. We will use a new (or just another) terms to make sure that other people will understand what we want to say.

A "remote control command" an mean:

1. Switch on the electric heater of heat storage system  or
2. Allow the local controller of the heat storage system to draw electric power when the local controller sees a need to heat the storage.

In the first case the electric power will immediately flow. In the second, it may or may not - depending on the local situation. Not all heater will start immediately at the same time to heat.

In case we use the term "remote control command" for the first application only, we will not be understood by many people. Because - I guess - most people would say: In both use cases we send a "remote control command" to the remote system.

What is the real underlying difference of the two use cases? The first one has a direct impact on the power flow, while in the second there is a local control system involved to decide what to do. Let´s assume we have 1000 heaters of a total power of 10 MW. In the first use case we have an immediate power flow rate of 10 MW per a few seconds. In the second case it is a stochastic situation where some may immediately draw the power others may draw power one hour later ...

Finally: If we would have smart systems, then the local controller would be situationell aware of the condition of the power system: if the frequency or voltage would be below specific set-points, then they would not draw power at all ...

If you would like to learn more about the huge machine "Interconnected Electric Power Delivery System":

Click HERE to watch a video [with English translation] which discusses some basics of the complexity ... enjoy.
Click HERE for the version in German.
Click HERE for more options.

What if Remote Control Fails?

The year 2015 is almost over ... here in Karlsruhe (Germany) we are just 13 h and 13 min away from 2016. Have you looked back to the many lucky and bad situations you have experienced or you have seen during the year 2015?

I guess we all understand that we need more serious engineers that take care of the many processes and systems we need in our modern life. Our generation sees a lot of good solutions going away ... replaced by modern technologies. There is a need to use more communication systems to keep the lights on, the grass green and the sky blue.

Volkswagen has demonstrated that adaptive closed loop control can take the situation (in which a car is) into account and react in different directions - to the good of the company and share holders ... not to the good of the environment.

I have just seen what happens, when a control system does not take the situation into account: The locomotive at the end of a long multiple unit train did not stop pushing when the driver of the leading locomotive decided to stop. The wireless communication to carry the stop-command via a radio channel failed to reach the control system of the locomotive at the end of train.

Click HERE to see how the spinning wheel dug into the tracks ... for hours I guess. The control system did not check the speed which was zero for hours and did not automatically stop the wheels spinning. Obviously there was a use-case that was not taken into account: What to do when the stop command does not make it through to the locomotive at the end of the train?

At the door steps to 2016 I wish everybody reading this post a successful year 2016 ... helping to keep the power flowing.

Stromausfälle und ihre (katastrophalen) Folgen

Strom kommt aus der Steckdose! Oder? Ja – natürlich! Wie kommt er in die Steckdose? Wen interessiert das schon! So ein paar Techniker – die sollten das wissen!

Strom ist nicht so unterhaltsam wie eine Oper oder ein Konzert! Wirklich? Allerdings! Nur - Strom unterhält uns doch alle: Ohne Strom keine Er- und Unterhaltung! In jeglicher Hinsicht. Was passiert, wenn bei der Aufführung einer Oper das Licht ausgeht, die Aufzüge und Rolltreppen stehen bleiben, die Lüftungs- und Klimaanlagen ausfallen, die Handynetze ihren Geist aufgeben, … und die Tankstellen keinen Kraftstoff mehr verkaufen können … laut einer Studie gibt es in Berlin ganze zwei Tankstellen mit einer Notstromversorgung!

In meiner Kindheit habe ich meine Mutter erlebt, als sie gerade beim Waschen war, fiel der Strom aus; sie sagte mir, dann kann ich ja in der Zeit (mit dem elektrischen Bügeleisen) bügeln! Während ihrer Kindheit hätte das wahrscheinlich funktioniert.

Was bei einem Blackout geschieht - Folgen eines langandauernden und großräumigen Stromausfalls. Unter diesem Titel wurde 2011 eine umfangreiche und interessante Studie veröffentlicht.

Zusammenfassung der Studie [3 Seiten, pdf]

Komplette Studie des Büros für Technikfolgen-Abschätzung beim Deutschen Bundestag [251 Seiten, pdf]

Anmerkungen von einem mir bekannten pensionierten Experten, der die Herausforderungen der Energieversorgung von Berufswegen sehr gut kennt:

Die Studie umfasst 261 Seiten. Allein die Zusammenfassung ist 31 Seiten lang. Der Text von Prof. Popp verdichtet dies auf 3 Seiten. Im Februar dieses Jahres war im gesamten europäischen Stromnetz nur noch eine Reserve von ca. 1000 MW verfügbar. Der Ausfall eines einzigen Kernkraftwerkblockes (z. B. ein Block in Philippsburg) hätte zum Totalausfall des gesamten Stromnetzes geführt. So knapp war es noch nie. Aber weil „wieder“ nichts passiert ist, wird diese Extremsituation von der Bevölkerung schlicht nicht wahrgenommen. Unser Umgang mit dieser so wichtigen Infrastruktur wie der Stromversorgung kann nur noch als total unverantwortlich bezeichnet werden.

Da die Kommunikationsmöglichkeiten bei einem totalen Stromausfall sehr rasch wegbrechen (selbst Notausgaben von Zeitungen entfallen, denn wie sollen Journalisten an verlässliche Informationen kommen, wie sollen Notausgaben hergestellt und wie verteilt werden), erreichen Informationen über die Folgen eines längeren Stromausfalles nur noch gerüchteweise und eher lokal die Bevölkerung. Diese kann den tatsächlichen Umfang an Beeinträchtigungen bis hin zu eingetretenen Schäden oder gar zu beklagenden Opfern höchstens erahnen. Eine Wiederherstellung einer dann wieder nutzbaren Infrastruktur ist bisher nicht überlegt, auch nicht geübt. Wie soll das dann funktionieren? Was wären die Folgen?

Wir haben und pflegen die Meinung, dass wir ein technisch hochstehendes Land sind. Mir wird schummrig. Der Vorfall kürzlich in München müsste doch einigen die Augen geöffnet haben. Der lange Ausfall in New York noch viel mehr. Aber wir "spielen" weiterhin mit unserer so wichtigen Infrastruktur und meinen, "der Markt" sorgt dafür (mit Geldbewegungen?!), dass das Gleichgewicht zwischen Erzeugung und Nutzung ständig eingehalten ist. Was ein Irrtum. Mich bringt die Ignoranz noch zum Wahnsinn (oder um).

Sein Fazit: Wann werden die notwendigen Schlussfolgerungen gezogen und tatsächlich konsequent umgesetzt?

Was fällt uns – neben dem Aufbringen von Markierungen für Fahrradwege in Innenstädten – noch alles ein, um die Energieversorgung nachhaltig zu sichern? Es reicht nicht aus, keine Ideen zu haben, man muss auch unfähig sein sie umzusetzen!

Heute ist dafür alles „smart“ (was übrigens auch „gerissen“ bedeutet). Wir sollten uns wieder auf das besinnen, was bisher zu der eigentlich unglaublich hohen Verlässlichkeit der elektrischen Stromversorgung geführt hat: Der Sachverstand, die Vernunft und die gekonnte Beachtung physikalischer Gesetzmäßigkeiten – nicht das Geld und nicht der Markt.

Intelligente und sichere elektrische Energieversorgungsnetze wurden bereits
zu Beginn der Elektrifizierung erfunden und bis heute weiterentwickelt.
Elektrische Sicherungen, Schutz- und Überwachungseinrichtungen sind seit
über 100 Jahren phänomenale Geräte zum Schutz von Leben und technischen
Einrichtungen. Ohne diese „smarten“ Geräte wäre ein fehlerfreies und
ausfallsicheres elektrisches Energieversorgungssystem undenkbar und die
Versorgung mit elektrischer Energie viel zu gefährlich. Siehe auch:

http://blog.iec61850.com/2012/03/smart-grids-19th-century-invention.html

Could more intelligent computers have prevented the ever biggest power outage in India?

Yes – and No! It all depends. Computers do what we want them to do. They don’t get tired, work 24*7, are reliably doing their job. They do what it has been told by a specific program and configuration. And then there are a lot more crucial aspects to take into account.

What is needed are “intelligent actions to correct problems” in due time. The intelligence may be implemented by “smart” humans or “smart” computer programs. More important: Very crucial requirements for a stable system are the various settings, build-in redundancies and the various reserves (generators, lines, head-room in power flow, transformers, …). These requirements are specified by humans! Depending on the level of risk responsible people are willing to accept, these requirements may vary greatly from one utility to another.

Depending on the settings used, build-in redundancies and the amount of the various reserves (generators, lines, transformers, …) the total system costs may be low, moderate or high! Reducing redundancies makes the system less stable – in general. Reserves and redundancies could be quite expensive.

Building a stable system is not problem in principle – it could be build if you have “unlimited” resources. The question is more: What is the maximum cost an utility is willing to spent to meet a certain risk level? Risk analysis and the level of risk accepted are key – and how and when the operators use the reserves. If an operator uses a reserve in normal operation, he cannot use that reserve in a critical situation again – you cannot eat the cake and have it.

The power delivery systems are very complex – most people do not care what it means to plan, design, operate, maintain, extend, and use such complex systems! Computers, high speed communication and even IEC 61850 are all just tools. Even a fool with a tool is a fool. And: A fool with a tool can foul up a system much faster than a fool without a tool.

The most crucial influence on power delivery systems is man-made! During a seminar an electrical engineer told me that they had a lot of serious discussions with the accountants and management on how many transformers they were allowed to replace per year. They agreed to replace two per year. Great! But: They utility had 300 (!) transformers in operation. That means: It wiould take 150 years to finish the replacement program! Unless …

Any question?

Smart Grids – A 19th century invention

Intelligent, safe electrical power distribution networks were invented at the start of electrification and have been further developed up to the present day. Electrical fuses, protective devices and monitoring devices have been phenomenal in the protection of life and technical installations for more than 100 years. Without these "smart" devices a fault-free, fail-safe electrical energy supply system would be inconceivable and the supply of electrical energy much too dangerous.

Since the 19th century engineers have developed, tested, used on a large-scale and continuously improved suitable solutions for the safe and reliable operation of the rapidly growing supply of ever more applications with electrical energy. During the sustained further development of the supply systems, it is necessary to handle the available resources (energy sources, technical installations and individuals with experience) as well as the laws of physics both responsibly and in a "smart" manner.
Smart grids help to make it possible to use physics safely and reliably for the benefit of man – in the past, today and in the future.

A new paper discusses some aspects of the development of Smart Grids.

Download the Smart Grid paper in English single sided or double sided.

Laden Sie das Smart Grid paper in Deutsch im Einseiten-Format herunter oder im Doppelseiten-Format.

The papers will be published by Bender (Gruenberg/Germany) in the Bender magazine MONITOR 01/2012.

Bender creates new technologies for safe handling of electrical power; to ensure the protection of people and the safe operation of machines, systems and manufacturing plants.

IEC standards are quite important for Bender! Mr Wolfgang Hofheinz (CTO of Bender) is President of the DKE (German national committee of the IEC international standardization organization) since 2010.

Check which IEC product standards and guidelines Bender applies