 |
Debugger-based On-Target Testing
|
|
Debugger-based On-Target Testing
|
Before reading this section, it is assumed that you have already read the Component and System Test sections. The developer's guide build on the information form these sections and provides additional technical background which shall enable DOTT users to apply and adapt DOTT for their projects.
As a general rule, the DOTT framework uses the notion of target firmware/functions/code/... to refer to all code that is executed on the embedded target device. Likewise, host functions/code/tests/... is used for all code executed on the test host. This separation manifests itself throughout the framework where components such as examples are divided into target and host parts (most often by dedicated target and host folders).
The typical sequence of steps when executing a test using DOTT is as follows:
Which actions are performed before a test is executed (e.g., target reset) are specified for individual tests by means of PyTest fixtures. In the following example, two fixtures are used: target_load and target_reset.
Note that in the PyTest configuration file conftest.py in the examples folder, the target_load and target_reset fixtures are re-directed to target_load_flash and target_reset_flash. This means that the target firmware is loaded into FLASH memory and, after reset, execution is performed from FLASH. Likewise, DOTT provides two fixtures called target_load_sram and target_reset_sram for SRAM firmware loading and execution. Additional, project-specific fixtures can easily be added to conftest.py. It is, for example, recommended to implement target access via external interfaces (e.g., I2C) via a fixture and to use this fixture in the tests. With such abstract communication fixtures, communication interface changes (e.g., using SPI instead of I2C) can easily be propagated to all tests without changing the tests themselves.
For component testing, DOTT is typically used in 'halt mode' where the target is reset, a binary is downloaded to the target, the target is booted up and then halted in a specific location (typically the DOTT 'test hook'). In this state, DOTT passes control to a test implemented in PyTest. The following figure shows the halt mode style test execution.
The 'free running' test mode starts the same way as a halt mode test by halting the target, downloading a firmware binary to the target and booting the target. It also hits the DOTT 'test hook' where it passes control to the test runner on the host. Instead of halting the target and then calling selected target functions as in halt mode, the target is allowed to run freely. Before resuming target execution, one or more halt points are set up. Once the target is running, stimuli are provided via the target's public communication interfaces (e.g., I2C, SPI). When one of the halt points is hit, the host-side test can inspect the target's internal state and compare it against the expected state (based on the stimuli provided via the external interface).
The recommended way to set up a new DOTT-based test project is to start with a template project which you then customize to your need. To do so please follow these steps:
Note: A full list of dott.ini file options is provided here.
To use DOTT in a firmware project the following steps have to be executed.
testhelpers.c with your firmware project.To simplify the steps outlined above, a basic template project (templates/standard) is included with DOTT. It includes minimal target firmware for a Cortex-M0 device where the DOTT test hook is called from the main. On the host side it includes a test which reads the Systick counter and checks if it properly increments.
Test execution is performed using PyTest as test runner. For basic test execution, open a Python command prompt (and activate the DOTT venv if using a virtual environment). In the command prompt navigate to the folder containing your tests and executed pytest. PyTest will then collect all tests in this folder (and its sub-folders) and execute them. To list all found tests use the –collect-only argument:
To execute only selected tests instead of all tests used the -k argument. The following example executes all tests which have 'Systick' in their name:
In addition to this basic string-based filtering, PyTest supports custom markers for tests using the @pytest.mark decorator. PyTest allows users to define arbitrary markers and individual tests can carry multiple such markers. For example, the following test has a marker irq_testing:
To execute all tests with the irw_testing marker you can now invoke pytesst with the -m argument:
To run DOTT-based tests in a Jenkins environment, activate the DOTT venv. In a Jekinsfile 'bat' section this can be done as follows:
DOTT requires a J-LINK GDB server to interact with the target. There are two options to start the GDB server. First, J-Link GDB server is started manually. When doing so you will see a window as shown below. In this cases you need to specify the IP address of your machine using the gdb_server_addr option in dott.ini. Note that this allows you also to configure DOTT to connect to a remote GDB server.
The second option is to use the GDB server 'auto-start' feature of DOTT. If there is no gdb_server_addr option specified in dott.ini (or DOTT Configuration Reference} "conftest.py") then DOTT tries to auto start the J-Link GDB server. If gdb_server_binary is not specified in dott.ini, DOTT assumes to find the J-Link GDB server in its default install location.
This section describes how to write tests using DOTT. Tests are implemented in PyTest and hence following the PyTest convention that test functions have to start with test_ to be discovered by PyTest:
In addition, tests can be grouped using classes and modules.
What you can also see in the above example is that the test takes two test fixtures. In contrast to other test framework where you have dedicated setup/teardown methods for controlling your test environment PyTest uses fixtures. DOTT comes with the following built-in fixtures:
Custom fixtures can be easily added in a project-specific conftest.py. For convenience, the conftest.py in the examples aliases target_load_flash to traget_load and target_reset_flash to target_reset. This convention is also used throughout the examples in this document.
When a test which uses target_load and target_reset fixtures is entered, the binary has already been loaded to the target, the target was reset and execution was started and finally the target was halted at the location of the DOTT test hook. It must be emphasized that the target is not running when entering a test. Target execution can be continued with cont() and halted again with halt() as shown below:
With target_is_running() you can check if the target is currently running or not.
DOTT allows you to call target functions while the target is halted:
If a function expects arguments which are not primitive types but pointers, structs, arrays etc. target memory needs to be allocated and filled with appropriate values before calling the function. DOTT supports this type of on-target memory allocation while the target is halted in the DOTT test hook. This is the case after entering the test as long as target execution was not resumed using cont().
In DOTT there are multiple ways to access target memory. As a general rule, target memory can only be accessed while the target is halted (with the exception of target live access). Using eval() individual (static) global target variables can be read or written as well as local variables which are int the scope where the target is currently halted. Depending on compiler optimization, some variables might no longer be accessible. Some tricks how to deal with such situations are given in the section on compiler optimization.
An alternative way to read memory is the target.mem interface. In contract to eval, it is designed for bulk memory access and hence should be preferred in larger amounts of memory need to be transferred to/from the target.
As shown in the previous section, DOTT offers two ways to access target memory: eval() and mem.read() / mem.write(). While eval() is mor convenient since it also allows direct access to, e.g., struct members its performance is relatively low since every eval() translates into a single USB transaction to the debug probe. In contrast to that mem.read/write() offers much better performance since memory access is performed in bulks. The following example shows how eval based access can be replaced with mem.read/write when dealing with a struct. Performance increase for this example is more than 8. The downside is that the target struct has to be replicated on the host using ctypes. If the declaration of the target struct changes, also the ctypes struct on the host needs to be updated. Note that also aspects such as struct packing and byte order need to be taken into account.
In addition to target memory access while the target is halted, DOTT allows basic access to target memory while the target is running. This can be useful to monitor (and sometimes plot) target variables/state which might be difficult to monitor (e.g., in a system heavily relying on interrupts halting and reading target memory could severely mess up timing).
To make use of live access, add the live_access fixture to the test as shown below. Currently, the live_access feature has a rather limited interface consisting of just two functions namely mem_read_u32 and mem_write_u32.
The following example code demonstrates how to read the SysTick counter using live access and generate a plot using matplotlib which is stored as a png file.
DOTT supports breakpoints to halt target execution when pre-defined locations are reached. More specifically, DOTT currently offers two different breakpoint variants which have specific use cases (and limitations): HaltPoints and InterceptPoints.
A halt point comes very close to the behavior of a traditional break point. It can be placed at any accessible symbol of the target binary. When the location specified by the halt point is reached, the target is halted. A test can wait for this event by calling the wait_complete() method of the halt point instance which blocks until the halt point is reached.
Note that it usually is a good practice to specify a timeout for wait_complete to avoid that a test gets stuck if a halt point is not reached as expected. A halt point also offers a reached() method which you can override:
In the reached() method you have full access to all dott().target functions including memory access and target control meaning that you can also continue target execution. However there is one important limitation of a halt point:
WARNING: If a HaltPoint is hit while a target function is executed which was started using
>dott().target.eval(), the execution of this function is interrupted and can not be resumed. In
situations where you want to use a breakpoint without altering the target's execution state use an InterceptPoint!
InerceptPoints are breakpoints which allow you to access and modify target memory when they are reached without changing the execution state of the target. Specifically, currently ongoing function calls initiated via dott().target.eval() are not interrupted. This feature of InterceptPoints comes with a limitation: In the reached() method of an InterceptPoint no calls to dott().target functions must be made!. Instead, use the eval() and ret() methods of the InterceptPoint class to access target memory or let the target return from its currently executed function. Other functionality such as bulk memory access (functions from dott().target.mem are also not available in InterceptPoints). The following code snippet shows how to use an InterceptPoint:
In the example above, intercept points are create for target methods example_GetA and example_GetB. When the intercept points are reached the values returned by the function are altered and execution continues. Note that the function call to example_AdditionSubcalls performed via eval() is not interrupted as it would be if halt points would have been used.
Halt- and InterceptPoints can be placed at every accessible symbol of the target binary and are triggered upon symbol invocation (i.e., function entry). A more versatile approach is to add DOTT_LABELs to you code which can also be used as Halt- and InterceptPoint locations. DOTT_LABELs can be placed anywhere in the code:
When creating a Halt- or InterceptPoint in the Python test code, the DOTT_LABEL can be specified:
When implementing system tests for heavily interrupt-driven firmware locations where Halt- and InterceptPoints are placed have to be chosen wisely as evey Halt- and InterceptPoint influences the target's timing behavior. Breakpoints should ideally only be placed after the events of interest have been fully handled by the target and when it is safe to halt the targeted.
If this is not possible or if it is desired to test functionality closely related to interrupts, a different approach can be employed. The basic idea is to disable the generation of interrupts by peripherals and instead control the generation of interrupts via DOTT. This way, the tests no longer have to deal with potentially (too) high interrupt frequency but interrupt generation can be adapted to a speed which can be handled by the tests.
The following example shows a simple test which checks if a timer on the target is incrementing:
If one now wants to test specific aspects of the interrupt handling code (e.g., by injecting data using an InterceptPoint) this would substantially change the target's timing behavior and hence, timer interrupts might be missed and results might be incorrect.
The following example shows an approach which overcomes this limitation by (1) disabling the timer and (2) manually generating timer interrupts by setting the timer's pending bit in the ISPR (interrupt set pending register). In addition, an InterceptPoint is set in the timer's interrupt handler which alters the timer count. The before generating the next manual interrupt, the test waits until the InterceptPoint is completed.
DOTT comes with its own logger with is using Python's logging facility. To access DOTT's logger import it as follows:
High compiler optimization levels (e.g., O3 or Oz) and link-time optimization can substantially change the binary's structure versus the code's structure. This can make setting breakpoints or inspecting variable content difficult (if, e.g., some variables are optimized out entirely). To somewhat reduce these problems (usually on the expense of code size), DOTT comes with some macros which might be helpful to tweak the source code to enable/simplify test development.
With the DOTT_NO_OPTIMIZE macro a function can be excluded entirely from optimization:
With the DOTT_NO_INLINE macro a function be prevented from being inlined:
With DOTT_VAR_KEEP a variable, which would otherwise be optimized out by the compiler, can be preserved:
Note that it is recommended to use the macros discussed above instead of directly using, .e.g., __attribute__((noinline)). The rationale is that DOTT comes with implementations of these macro for all compilers supported by DOTT.
If the target project is a C++ project, some extra care must be taken related to the scope resolution operator (::). When, e.g., calling a method of a static class called Bar in namespace Foo this is done like this:
Please note that single quotes are required around namespace, class name and function name.
If you need to customize the target connection sequence, DOTT provides hooks via the DootHooks class. Via this class you are able to specify callback hooks which are called by DOTT. To date, the following hooks ara available:
The Segger JLINK software caches content of memory regions such as FLASH which it assumes to not change during runtime. However, this assumption might not be correct if, for example, the target uses an on-device bootloader to updated FLASH content via, e.g., I2C. This flash cache can be disabled as follows:
The example firmware and the DOTT library for the target reference board is provided in binary and source code formats. The binaries enable users to directly run the examples without the need to first set up a build environment and compile the source code for the reference target. All native code comes with Makefiles and Arm Compiler 6 is used to compile the code.
To compile the code on the Windows command line, you need an environment which includes the GNU make, a shell such as busybox and the Arm Compiler 6 command line version. In such an environment, you can now navigate to the folder where you have unpacked the DOTT docu and examples zip archive. Build the examples and the DOTT library by calling make.
DOTT uses two locations for configuration. First, in a folder containing tests there usually is a dott.ini file which is described here. The content of the dott.ini is parsed on startup and stored in an internal data structure called DottConf. The names of the configuration options stored in the DottConf structure match those from the dott.ini. If no dott.ini file can be found, the DottConf is filled with default values.
In conftest.py (PyTest configuration) which usually is also located in the test folder or a parent folder, the content of the DottConf structure can be altered programmatically. Here you can override settings in DottConf or set options based on the host name the tests are executed on. This is quite handy to have one configuration file with different config options depending on if the file is executed on the developer PC or a Jenkins slave. The example and template projects coming with DOTT include conftest.py files for your reference.
1.9.5