Have you heard about FUZZING?
Wikipedia explains:"Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. ..." Wow!
Is there any link to IEC 60870-5-104, OPC UA or IEC 61850? Yes there are people that have used the technique to test these and many other protocols.
The "State of Fuzzing 2017" report just published by SYNOPSIS (San Francisco) wants to make us belief that, e.g., the above mentioned protocols are weak and may crash easily. What?
The best is to read the report and my comments below. Other experts have commented similarly.
Click HERE to download the report.
Any kind of testing to improve IMPLEMENTATIONS of protocols is helpful. You can test implementations only – not the protocols or stacks per se.
One of the crucial questions I have with the fuzz testing report is: Which IMPLEMENTATION(s) did they test? Did they test 10 different or 100? Open source implementations only? New implementations or old? Or what?
Testing is always a good idea … more testing even a better approach. At the end of the day, customers have to pay for it (e.g., higher rates per kWh).
I would like to see more vendor-independent tests of any kind … but the user community must accept the higher costs. Are you ready to pay more? How much more would you accept to pay? 50%?
As long as vendors have the possibility to self-certify their products we will see more problems in the future.
Anyway: The best approach would be to use a different protocol for each IED … ;-)
What about testing the wide spectrum of application software? Not easy to automate … to fuzz.
You may have a protocol implementation without any error within one year … but an application that easily crashes … a holistic testing approach would be more helpful. IEC TC 57 WG 10 has discussed many times to define measures for functional tests … without any useful result so far. Utility experts from all over the world should contribute to that project – go and ask you manager to get approval for the next trips to New Orleans, Seoul, New York, Frankfurt, Brisbane, Tokyo, … to contribute to functional testing. In case you do not attend – don’t complain in the future when IEDs crash …
The more complex an application is, the more likely it is that there will be serious and hard to find problems.
Crashing the protocol handler and application is one thing - what if they don’t crash but bad data gets through?
Conclusion
The report is a nice promotion for the fuzzing tools offered by Synopsis.
The last page states: "Synopsys offers the most comprehensive solution for building integrity—security and quality—into your SDLC and supply chain. We’ve united leading testing technologies, automated analysis, and experts to create a robust portfolio of products and services. ... our platform will help ensure the integrity of the applications that power your business."
Testing is very crucial and very complex. I hope that users of devices applying well known protocols in power system automation will soon better understand HOW important testing is - require various tests for devices they purchase and are willing to pay for it!
Start with an education phase as soon as possible - before it is too late.
Wikipedia explains:"Fuzzing or fuzz testing is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program. The program is then monitored for exceptions such as crashes, or failing built-in code assertions or for finding potential memory leaks. ..." Wow!
Is there any link to IEC 60870-5-104, OPC UA or IEC 61850? Yes there are people that have used the technique to test these and many other protocols.
The "State of Fuzzing 2017" report just published by SYNOPSIS (San Francisco) wants to make us belief that, e.g., the above mentioned protocols are weak and may crash easily. What?
The best is to read the report and my comments below. Other experts have commented similarly.
Click HERE to download the report.
Any kind of testing to improve IMPLEMENTATIONS of protocols is helpful. You can test implementations only – not the protocols or stacks per se.
One of the crucial questions I have with the fuzz testing report is: Which IMPLEMENTATION(s) did they test? Did they test 10 different or 100? Open source implementations only? New implementations or old? Or what?
Testing is always a good idea … more testing even a better approach. At the end of the day, customers have to pay for it (e.g., higher rates per kWh).
I would like to see more vendor-independent tests of any kind … but the user community must accept the higher costs. Are you ready to pay more? How much more would you accept to pay? 50%?
As long as vendors have the possibility to self-certify their products we will see more problems in the future.
Anyway: The best approach would be to use a different protocol for each IED … ;-)
What about testing the wide spectrum of application software? Not easy to automate … to fuzz.
You may have a protocol implementation without any error within one year … but an application that easily crashes … a holistic testing approach would be more helpful. IEC TC 57 WG 10 has discussed many times to define measures for functional tests … without any useful result so far. Utility experts from all over the world should contribute to that project – go and ask you manager to get approval for the next trips to New Orleans, Seoul, New York, Frankfurt, Brisbane, Tokyo, … to contribute to functional testing. In case you do not attend – don’t complain in the future when IEDs crash …
The more complex an application is, the more likely it is that there will be serious and hard to find problems.
Crashing the protocol handler and application is one thing - what if they don’t crash but bad data gets through?
Conclusion
The report is a nice promotion for the fuzzing tools offered by Synopsis.
The last page states: "Synopsys offers the most comprehensive solution for building integrity—security and quality—into your SDLC and supply chain. We’ve united leading testing technologies, automated analysis, and experts to create a robust portfolio of products and services. ... our platform will help ensure the integrity of the applications that power your business."
Testing is very crucial and very complex. I hope that users of devices applying well known protocols in power system automation will soon better understand HOW important testing is - require various tests for devices they purchase and are willing to pay for it!
Start with an education phase as soon as possible - before it is too late.
Posts
No comments:
Post a Comment