Goals and Purpose

The Debugger-based On-Target Testing 'new-gen' (DOTT.NG) framework is designed as an approach to simplify the on-target testing of firmware developed for Arm Cortex-M microcontrollers. It uses the debugger attached to the target device via SWD or JTAG to control, monitor and alter the execution of firmware on the target. Below is a summary of the goals addressed by the DOTT framework.

Primary goals for the Debugger-based On-Target Testing framework are to enable

firmware testing without modifications of the firmware for testing
firmware component testing without mocking of, e.g., peripherals
firmware testing with selective injection of data into the execution
firmware testing on the original target device
firmware testing using the original compiler

Secondary goals for DOTT are

execution of the test runner and tests on the host
simple interfacing of external equipment (e.g., for stimulus generation our output monitoring)

Architectural Overview

The figure below provides an overview of DOTT's architecture. On the test target on the bottom left the unmodified firmware under test is executed. Strictly speaking, the firmware is not entirely unmodified but a small test helper library is linked into the firmware (cp. Dev Guide). It provides a well defined entry point for the debugger. In practice, its effects on the firmware are negligible such that this entry point can be even left in place in the production version of the firmware.

On the right side, the debugger hardware is connected to the target via SWD or JTAG. A vital aspect for DOTT's approach is that the debug probe exposes a GNU Debugger (GDB) server interface for debugging clients to connect to. This is fulfilled by debuggers supported by the OpenOCD framework, PyOCD but also commercial debuggers such as Segger J-Link or PE micro debuggers.

‍Note: DOTT currently is developed and tested only with Segger J-Link debuggers. For J-Links it also supports auto-start of the GDB server.

On the next layer, a GDB client is used to connect to the GDB server. DOTT connects to the gdb client using the GDB machine interface (GDB MI). DOTT uses the PyTest unit test framework as test runner. The tests themselves, which are also implemented in Python, are executed by PyTest on the host where they have access to the target via GDB.

Optionally, the tests implemented in Python can also use additional libraries to interface with external test equipment to generate stimuli towards the target or monitor target outputs.

DOTT Architecture Overview.

Supported Test Techniques

DOTT supports different approached for testing which are described below. A fundamental feature of DOTT used for all tests types is its ability to automate the programming of the target's FLASH memory and to bring the target into a defined state (target reset).

Component Testing

DOTT supports the implementation of component (unit) tests where an isolated function (or set of functions) is tested by providing input vectors and checking output vectors. Input vectors are defined in the tests implemented on the host and are provided, via the debugger, to the function(s) on the target. Output vectors are transferred back to the host where they are checked against expected results. With DOTT it is not only possible to invoke functions implemented in the target's firmware but it is also possible to inject (faulty) data into the execution flow. E.g., if the function under test calls some other functions, the return values of these functions can be changed and the reaction of the function under test can be observed.

For component testing, the target is typically booted up and halted at a certain location (test hook). Functions under test are then called from the context of this test hook.

System / 'Free-Running' Testing

The free-running mode is meant for system testing where the target is running and stimuli are provided to the target via its external interfaces (e.g., I2C). In addition to observing and checking the response to the stimuli in the public interface DOTT allows to add breakpoints to the target such that, e.g., it can be verified that a given stimulus actually triggers expected on-target functionality. One can also check if if externally provided data was correctly written into target memory (e.g., think about checking for endianess issues when doing data deserialization). Another usage example would be a bootloader where FLASH erase/write operations are triggered via the target's external interface and DOTT allows to inspect the target's memory and verify if it was correctly written/erased.

Support

DOTT was originally developed internally at ams AG (later ams-OSRAM AG) to simplify automated testing of firmware for Arm Cortex-M microcontrollers. In spring 2021 it was decided to release DOTT via GitHub to the public.

In fall 2022, the development of DOTT was discontinued and a fork called DOTT-NG (aka new generation) was created on Github where development continues under the same open source (Apache 2) license. DOTT.NG is maintained as a privately run open source project and is no longer connected to ams-OSRAM AG. Contributions from industry and the open-source community are welcomed and highly encouraged as long they follow the guidelines in the included CONTRIBUTING.md. Please submit your feature requests, bug reports and support questions via the project's GitHub issue tracker

Documentation Overview

The documentation for the Debugger-Based On-Target Test Framework is structured as follows:

This section presents the goals of DOTT along with its high-level architecture and concepts.
The Release Notes list the features and know limitations of DOTT and provide a version history.
The Setup section describes how to install the framework and what target system is need to execute the included example tests.
The Component Tests section gives examples how to use DOTT for component tests. It compares firmware functions under test and corresponding test code on the host and gives readers an immediate understanding of the core concepts of DOTT.
The System Tests gives examples how to use DOTT for system tests. It explains the basic concepts of operating the target in free running mode while providing stimuli to the target's external interfaces.
The Dev Guide picks up where the getting started section left off. Specifically, it covers all the technical details required for test developers to leverage the full potential of DOTT. Furthermore, it outlines the steps for creating a new test project and provides a technical reference.
The Reference Platform section describes how to turn an off-the-shelf evaluation board into a reference and evaluation platform for DOTT. All examples shipped with DOTT are implemented to run on this reference board.
The External Test Equipment section discusses how to integrate external components into the DOTT test execution flow.

Build Information

This documentation was built based on the following docu repository version:

version number: 1.17.0
build date: 20240807
git commit hash: da5f95a2a5e9e97f39b5466d152e67e9f218f0a2
git tag: 1.17.0
contact: Thomas Winkler (thomas.winkler[at]gmail.com)