How we built a test bench for embedded system testing

A bit of context

A test bench is an environment designed to run tests (and in our case, automated tests). The concept is simple: have a set of reference equipment (our supported hardware targets) integrated into a test bank. On these targets, we automatically execute scenarios driven by our Continuous Integration (CI) pipeline.

Our goal was to validate ourLinux platform features, ensure stability, and detect regressions as soon as they appear.

Why it seemed important to us

Embedded systems sit at the intersection of software and physical hardware, making their testing inherently more difficult and requiring an environment that can handle both simultaneously.

A well-designed test bench offered several key advantages:

• Identify issues early by testing software and hardware together
• Enable repeatable, automated testing scenarios
• Allow hardware tests that would be challenging or impossible to conduct manually

Test bench set up: key requirements

When building our test bench we identified a few software and hardware requirements to meet.

Embedded software requirements

• Testing software

Choosing the right software to test your embedded system is critical. Automated testing tools like Pluma are flexible enough to handle a wide range of system features and testing scenarios, from low-level components like bootloaders to high-level applications. Your embedded testing tool should be adaptable and scalable as your embedded system evolves.

• Automation tools

Automating your tests is a must for efficient CI/CD workflows. Tools like GitLab CI allow you to automatically trigger tests based on a schedule (e.g., daily builds) or upon the introduction of new features. This ensures continuous validation of your code and hardware integration.

• Test reporting

Comprehensive test reports provide insight into the testing process, helping you identify and isolate issues quickly. Proper reporting saves time and avoids the risk of overlooking important errors.

Hardware requirements

• Test runner

A device like a Raspberry Pi4 can act as a runner, executing tests on demand. This runner must be configured to communicate effectively with the device under test (DUT).

• Power Distribution Units (PDUs)

Proper control over the power supply to the DUT is critical. PDUs allow you to remotely reset devices and return them to a known state before each test, ensuring consistency in your results.

• Communication interfaces

Not all board features are accessible during testing. For instance, testing a bootloader often means SSH isn’t available. Your test bench should support multiple communication types, ensuring that tests can proceed regardless of the board’s state.

• Network management

Managing multiple devices over a network introduces challenges, especially when you need to avoid data collisions. A well-organized, secure network setup is essential to keep your tests running smoothly.

How we implemented it

We built our test bench using two complementary tools:

• Pluma, a test automation framework from The Embedded Kit ecosystem,
• and GitLab CI, which orchestrates our continuous integration.

The process is simple and fully automated:

• Image compilation: At the end of a build, before a release, or during a merge request, the CI generates a software image.
• New test suite: Instructions go to the test environment.
• Test execution: Pluma downloads and deploys the new image on the target. It then runs the predefined test suite to validate functionalities.
• Result visualization: Results are displayed in GitLab dashboards, thanks to JUnit report format support in Pluma & Gitlab.

Each hardware target we want to test is paired with a companion Raspberry Pi. This Raspberry Pi hosts the GitLab Runner and Pluma, enabling smart test distribution. Each runner is configured with a specific GitLab tag that identifies the connected board or device. This way, the CI knows exactly which test suite trigger to which runner.

Once the image is deployed, Pluma executes a set of tests covering Welma Linux distro key features. This test suite is included with the distribution but is fully extensible: you can add your own scenarios, whether low-level tests (custom drivers, serial or network interfaces, sensors, …) or application-level tests simulating user interactions.

The test bench infrastructure includes Ethernet and serial connectivity, as well as a programmable power supply to reboot targets on demand. Wi-Fi connectivity is also supported by Pluma.

The test bench is essential to guarantee the quality of any Yocto-based product.

Using and evolving your test bench

As new hardware is regularly introduced in our Linux platform, our test bench regularly needs to adapt and evolve. This includes rewriting custom tests tailored to the unique requirements of this new board. Once functioning correctly, these tests are integrated into our existing test plans.

The beauty of a test bench is that it can grow with your project. Most modern setups, integrating embedded testing tools, allow for an extensible approach, meaning you can add new boards and configurations without overhauling the entire system.

Take the recent addition of x86 boards within our Linux distribution, which required integrating PXE booting to reset the Yocto image. This was seamlessly incorporated into our existing test bench, while still maintaining a clear distinction between ARM-based and x86-based hardware testing.