A SiL and HiL test system that automates acceptance testing for every firmware version

The task

A power quality analyser measures voltage, current and power quality, and before release every firmware version has to show that its readings stay within the tolerances set by the standard. At the client, this acceptance testing was largely manual: set the calibrator, read the device, log the result. The effort grew with every new version and every new device type, and release became the bottleneck.

The client needed a framework that automates these acceptance tests and runs in its own infrastructure, with its existing instruments, its test management and its development tools. The firmware itself was not to be modified for this.

The solution

The framework is based on Robot Framework. Each interface of the device has its own keyword library: Modbus, REST, OPC UA and direct access to the firmware's memory. The client's high-precision calibrators are integrated through dedicated libraries and are controlled from within the test case. The test sets a waveform with harmonics or a modulation, takes the reading from the device under test and compares it with the tolerance.

The test cases from the standards for power quality instruments are stored as test cases of this kind. Some of them are generated directly from the tables in the standard, so that a changed tolerance does not have to be updated in dozens of places.

The same tests run at two levels. In the SiL system, the firmware runs without hardware against a simulated calibrator; this is the fast run after every change. In the HiL system, it runs on the device against the real calibrator; this is the acceptance test. The results of the Google Test unit tests, together with test coverage, go automatically into the client's test management system, where the release decision is made.

The framework was built over several years in very close collaboration with the client's development team, in its infrastructure and with its tools.