Security is more than a buzzword these days. You should be very serious about the security of your substation protection and automation system.
Joe Weiss asked yesterday:
Are the Good Guys as Dangerous as the Bad Guys – an Almost Catastrophic Failure of the Transmission Grid
What happened? A port scanning tool in an IEC 61850 GOOSE based substation protection system had a very negative impact on the GOOSE publisher and subscriber: The Relays stopped to operate!! They had to be manually rebooted.
Port scanning may provide a lot (too much) of stress to the devices and communication system. Such a crucial load has to be taken into account during the design of the devices and of the whole system. Theoretically this payload should be taken into account as part of the system engineering ... part of the System Configuration Description (SCD). Any unexpected traffic avalanche may have a serious impact on the stability of the system!
Click HERE for Joe's report.
I guess that the GridEx network monitor would have raised the red flag seeing the message avalanche in the transmission substation.
Lesson to be learned:
Any non-operational load on a critical network should be treated very careful. IT and OT people have to work together and make sure that such test tools do not put too much stress onto the devices connected in a substation or any other system:
Teamwork makes the dream work - and keeps the power flowing!
Click HERE for a discussion of port scanning ... written long time ago (2001 !!)
Click HERE for a worth to read report on how to apply IEC 62443.
My friend Andrea Bonetti (FMTP) responded as follows:
Dear Karlheinz!
What you have described is unfortunately a known problem.
It is really not at all the first time that it happens in the last 10 years, but it is maybe the first time that it is presented to the public.
I would like to stress-out that this problem is NOT related to IEC 61850 but it is related to the correct usage of digital technology.
Similar situations happened also “before” when proprietary digital technology was used. Maybe they were just more difficult to disclose because also the tools were proprietary.
Regarding GridEx, it would have detected the loss of communication among the devices, as it performs the supervision of the GOOSE messages. This would have been written in its report.
GridEx performs also network load calculations, but in the case you have described this would not have helped probably. Anyway that information would also have been written in the report.
Let me point out that GridEx is an “IEC 61850 passive tool”.
GridEx does not talk to any device, does not send any IEC 61850 message…. it can only listen to what happens, without interacting with the system.
Also the time synchronization of GridEx can be performed completely independently from the system, with its own independent GPS receiver accessory.
Also GridEx works without a PC, so you do not connect the PC to the substation network system.
As GridEx doesn’t interact to the system where it is connected to, it cannot cause any damage and it can be connected to the network while the system is in service.
Joe Weiss asked yesterday:
Are the Good Guys as Dangerous as the Bad Guys – an Almost Catastrophic Failure of the Transmission Grid
What happened? A port scanning tool in an IEC 61850 GOOSE based substation protection system had a very negative impact on the GOOSE publisher and subscriber: The Relays stopped to operate!! They had to be manually rebooted.
Port scanning may provide a lot (too much) of stress to the devices and communication system. Such a crucial load has to be taken into account during the design of the devices and of the whole system. Theoretically this payload should be taken into account as part of the system engineering ... part of the System Configuration Description (SCD). Any unexpected traffic avalanche may have a serious impact on the stability of the system!
Click HERE for Joe's report.
I guess that the GridEx network monitor would have raised the red flag seeing the message avalanche in the transmission substation.
Lesson to be learned:
Any non-operational load on a critical network should be treated very careful. IT and OT people have to work together and make sure that such test tools do not put too much stress onto the devices connected in a substation or any other system:
Teamwork makes the dream work - and keeps the power flowing!
Click HERE for a discussion of port scanning ... written long time ago (2001 !!)
Click HERE for a worth to read report on how to apply IEC 62443.
My friend Andrea Bonetti (FMTP) responded as follows:
Dear Karlheinz!
What you have described is unfortunately a known problem.
It is really not at all the first time that it happens in the last 10 years, but it is maybe the first time that it is presented to the public.
I would like to stress-out that this problem is NOT related to IEC 61850 but it is related to the correct usage of digital technology.
Similar situations happened also “before” when proprietary digital technology was used. Maybe they were just more difficult to disclose because also the tools were proprietary.
Regarding GridEx, it would have detected the loss of communication among the devices, as it performs the supervision of the GOOSE messages. This would have been written in its report.
GridEx performs also network load calculations, but in the case you have described this would not have helped probably. Anyway that information would also have been written in the report.
Let me point out that GridEx is an “IEC 61850 passive tool”.
GridEx does not talk to any device, does not send any IEC 61850 message…. it can only listen to what happens, without interacting with the system.
Also the time synchronization of GridEx can be performed completely independently from the system, with its own independent GPS receiver accessory.
Also GridEx works without a PC, so you do not connect the PC to the substation network system.
As GridEx doesn’t interact to the system where it is connected to, it cannot cause any damage and it can be connected to the network while the system is in service.
Posts
