Dangerous Situation in the European Electric Power System Caused by "frozen" Measurements

Measurements of power flow (Watts in export or import) are very crucial for Load Frequency Controller … wrong (i.e., “frozen”!) measurements have caused almost a big blackout in Europe in January 2019.
What happened: the measurement of power of the lines between two transmission systems (Germany – Austria) were frozen when the export value of 723 MW from Germany to Austria was measured (which was a result of 34 GW wind power generation in Germany). Later the wind power generation decreased to 4 GW … and the measurement (as input to the controller) many hours later still used the input value of 723 MW !!! In such a meshed power network it is unlikely that such a value is constant …

Oops … something went absolutely wrong!

Report by exception (on a value change as used for the above measurement) is great … as long as there are changes figured out and reported. A frozen value does not cause a change and thus no new value will be reported … No receiver should expect that the export power is constant (723 MW) for days!! The sensors may have worked fine … but the software and communication failed … on both sides (sender and receiver). A receiver should not trust that the software and communication is working fine all time.
Here are some measures to monitor the communication (by the receiver) to figure out if the communication is OK:

1. Ping (in case of TCP/IP) (if no response after some time: raise flag)
2. TCP Keep-alive (if no keep-alive message in t bigger keep-alive: raise flag)
3. Polling by receiver (if no response after some time: raise flag)
4. Periodic reporting (if no report in t greater period: raise flag)
5. In case of no message received in a configured time period (in case of using IEC 61850 Reporting) the receiver should check if the report control block is enabled and is using the correct configuration values like trigger option, …
6. Check if the sequence of received values are plausible
7. Use redundant systems (comm, …)
8. …

Check out the official Entso-E report (with links to more details):

https://www.entsoe.eu/news/2019/05/28/entso-e-technical-report-on-the-january-2019-significant-frequency-deviations-in-continental-europe/

This reminds me on the Boeing 737 MAX disaster … maybe a programmer left the vendor of the load frequency controller and hired with Boeing … I am kidding.

How many programmer or people that configure power control systems and communication systems that lack experience with complex systems like a plane or a power system. Where are the “grey-hair” experts that would tell you in minutes how to … ? They may enjoy the beach with warm water and sun shine – relax and spend the pension for ...

It is not sufficient to have no ideas – one should also be unable to implement them.

I expect that more of these problems will hit the street once we have far more control, monitoring and communication in the smart(er) grids of medium and low voltage. Note that the problem in January 2019 occurred at transmission level!! … where more resources (higher budgets) are available (in the past).

Have a great weekend – with power.

The IEC 61850 Tissue Database is a Great Resource of Hints and Background Information

The IEC 61850 tissue database is a comprehensive resource of hints and background information that can help you to understand why a particular change or correction has been implemented:
https://iec61850.tissue-db.com

Many tissues are posted that end up in the category "blue" (This is a question only). The discussion of the experts is often very helpful ... here you find answers that you may not get from the original published documents.
Example of discussion:
https://iec61850.tissue-db.com/tissue/1649



Enjoy the tissue database. As a registered user you can post tissues and see more details.

Many IEC 61850 Projects Under Progress

37 new parts and parts under revision are under progress as listed in the following table:

As you can see here: IEC 61850 is a bit more than another protocol.

Other parts are published already (some of them are under revision - see above):
https://webstore.iec.ch/publication/6028



Unfortunately many "grey-hair experts" with a lot of experience regarding IEC 61850 have been retired and a lot of experience has been wasted ... politics cares more about wasting food than wasting "experience" good for human survival! It's really a pity when we take into account that the comprehensiveness and complexity of IEC 61850 is growing so fast ...

IEC just Published Amendments to Three Core Parts of IEC 61850 as FDIS

Hurrah, after long wait ... now they are published:

IEC just Published amendments of the following three Core Parts of IEC 61850 as FDIS:

IEC 61850-7-2 - Services (57/2100/FDIS) - 138 pages
IEC 61850-7-3 - Common data classes (57/2101/FDIS) - 108 pages
IEC 61850-7-4 - Logical Nodes and Data Objects (57/2102/FDIS) - 422 pages

These amendments will be merged into the previous editions and made available as consolidated documents after FDIS ballot (Edition 2.1).

The consolidated documents (published as INF documents) comprise:

IEC 61850-7-2 (181 pages)
IEC 61850-7-3 (135 pages)
IEC 61850-7-4 (440 pages)

One of the most crucial issue solved now is the harmonization of definitions used in different domains like substation, protection, wind, hydro, DER, batteries, schedules, ...

NOTE: The three new documents (and the many other parts published or underway, e.g., 80-x, 90-x) make the standard series IEC 61850 even more comprehensive and MORE COMPLEX.

I hope that there will be enough people responsible for the future of the digitalization in the energy world understanding that there is no free lunch!! Education and training is key!

IEC 61850-7-2

This edition includes the following significant technical changes with respect to the previous
edition, based on almost 100 tissues ... and other issues.

IEC 61850-7-3

Compared to the second edition, this first revision of the second edition provides mainly clarifications and corrections to the second edition of IEC 61850-7-3, based on almost 100 tissues ... and other issues.

IEC 61850-7-4 

The motivation and goal of the amendment is to improve consistency of the data model over
all application domains of IEC 61850. Data (Logical Nodes, Data Objects, Data Attributes)
with the same semantics shall have the same naming where this part of IEC 61850 refers to
Logical Nodes and Data Objects and IEC 61850-7-3 to the Data Attributes.
Therefore, the amendement complements and updates the second edition of this part of
IEC 61850, which was published in 2010. It constitutes editorial revisions for consistency and
technical corrections of bugs as far as interoperability is touched.
To reach this goal and to keep it for all future as common working source a comprehensive
back-office UML version was created and will be maintained for future standard development.
The published parts of IEC 61850 such as IEC 61850-7-4, on which the amendment is based,
are generated automatically from the UML version. This allows publishing, voting and reading
the various parts of IEC 61850-7 as in the past.
This amendment includes changes with respect to IEC 61850-7-4:2010 based on almost 200 Tissues ... and other issues.

Could the Utility Industry learn from the Boeing 737 Max Disaster?

The other day I read the IEEE article:

How the Boeing 737 Max Disaster Looks to a Software Developer

https://spectrum.ieee.org/aerospace/aviation/how-the-boeing-737-max-disaster-looks-to-a-software-developer 

Everybody working in the power utility world SHOULD study this paper in detail - and take some time thinking about it and plan and implement consequences ... by some means or other.

We are in danger to end up here:

I have slightly modified a quotation from the above paper:

“Long ago there was a joke that in the future control centers would control themselves, and the only thing in the control room would be an operator and a dog. The operator’s job was to make the customers comfortable that someone was in the control room. The dog’s job was to bite the operator if he tried to touch anything.”

This 737 Max problem is a symptom of a trend that is happening in many domains: power production, power delivery, ... even education (from Kindergarten to university).

A senior developer sent me the other day the following link with the subject:

When management thinks 100 trainees can do work of 3 fully qualified senior developers

https://youtu.be/JeB19gvJcxs

This statement and clip is very very true - in my training business with more than 4,500 attendees all over I have experienced that in some cases training (regards IEC 61850 and other subjects) had to be paid by the attendees ... using annual vocation! Unbelievable! HR has a big (often negative) impact on the training of the employees as well. The complexity of, e.g., IEC 61850 is usually totally underestimated! ... no need for a training ... read the myriad of papers and study slides ... that is quite often the recommendation of the management and HR.

This understanding is widespread in the utility domain, too ... students are hired (for low wages) to investigate and figure out how new technologies (especially digitalization) could be used ...

I hope that the utility industry will wake up and do a better job than the people at Boeing - but it will cost a lot of money ... shareholders and customers may not want to spent.

The European electric power system is under more stress since January 2019 ... see the following link (first German and second English text):

Switzerland was on the brink of a blackout on May 20, 2019:

https://gridradar.net/schweiz_blackout_entgangen.html

On the left side bar you can see that we faced three additional critical situations (two in January and one in April 2019)!

One reason behind all this is the "market driven" power delivery ... more and more relying on software that processes a lot of data to get forecasts and set schedules for the energy flow. Hope that this software is better than ...

We all rely 24/7 on electric power. Have you thought about the possibility of local, regional or total blackouts? How would your life change?

I am nursing my wife here at home. She needs ventilation 24/7. The two ventilators have each a battery good for four hours ... so a blackout of 10 hours would mean that my wife ... For that reason I have several batteries, two emergency generators and some 30 liter gasoline.

I hope that not many "old grey-hairs are sitting in the corner" - BUT walking around and helping the young people to understand why we have the electric power system as it is now - developed in more than 130 years. Senior experts like Gregory Travis (author of the IEEE article) are very rare ... and not well understood ... and maybe too expensive for the bean counters.

The cumulative experience of the "old grey-hairs" (many retired years ago) that has been collected between the 60s and 90s and that are still involved one way or the other should have reasonable influence on young engineers ... in order to keep the power flowing.

In some time down the road we may have 200 trainees on the play field ... and no "old grey-hair" left. In this case the "w" in the German "Energiewende" will drop - means Energie-Ende" ... end of energy.

I am one of these "old grey-hair" engineers (66). Just a few young people are listening to me - I have to stay at home 24/7 ... no travel anymore ... One of the young people listening to me is our grand-daughter (20, finishes her bachelor in EE and IT this year). She is really eager to learn from my experience with MAP, Fieldbus, UCA, IEC 61850, Modbus, ... power systems, renewables, ... As an intern at a distribution company she transfers part of my knowledge to the utility - it is cheaper for the utility than to renew the contract with me ... ;-)

Maybe they will contract with me for the second time - but then it may be too late.