Component Testing

Component testing with DOTT is also referred to as 'halt mode' testing. This essentially means DOTT is used to download the firmware to the target and boot it up and then halt it in a defined location called the DOTT test hook. From there, functions implemented in the firmware can be called in a component/unit testing style.

To execute the included component test examples please use the reference target platform describe in setup section.

Component (Halt Mode) Test Examples

This section shows how to run component (unit) test examples where software functions implemented in the target's firmware are called from tests implemented on the host (cp. DOTT architecture).

Executing Example Component Tests

First, open a Python command prompt (and activate the DOTT venv if using a virtual environment). In the command prompt now navigate to the dott_ng_doc_examples_xxx\examples\01_component_testing of the dott examples package downloaded from GitHub:

> cd dott_ng_doc_examples_xxx\examples\01_component_testing
> dir
29.11.2018  14:19    <DIR>          host
29.11.2018  14:19    <DIR>          target

As you can see in the snippet above, the folder contains host and a target folders. The target folder contains the firmware under test which is downloaded to the target device as part of test execution. The target folder contains both the source code as well as pre-compiled firmware binaries. The host folder contains the actual tests implemented in Python. Now lets change into the host folder and invoke pytest to start the test execution:

> cd host
> pytest

Upon first execution you will be prompted with the following license information dialog from the Segger J-Link debug probe. Please make sure to tick the checkbox on the bottom left (highlighted in green) and then click accept (highlighted in green). Please note that you will get this message only when using the reference board with an ST-Link converted to a J-Link. If you are using a standalone J-Link debug probe you will not get this message.

Segger J-Link / ST-Link license dialog.

As a result, you should see an output similar to this:

Example tests for functions implemented in the firmware of the reference board.

Understanding the Example Component Tests

The following sections show selected functions implemented in the firmware under test (executed on the reference board) and the corresponding tests implemented in Python on the host. The source files for the example functions under test and corresponding host-side tests are given in the table below. Note that not all examples in the source files are discussed here.

Content	Source File
Firmware functions on target	examples/01_component_testing/target/testexamples.c
Corresponding host-side tests	examples/01_component_testing/host/test_example_functions.py

Simple Function Calls

The following examples show how target functions are called by their symbol name from the host and how scalar parameters and return values are handled. Note that it is also no problem to call static functions.

• Target functions

  The following functions take either no argument or take scalars as arguments and return scalar results.

  /**
   * Function without any arguments.
   */
  uint32_t __attribute__((used)) example_NoArgs(void)
  {
      return 42;
  }
   
   
  /**
   * Static function without any arguments.
   */
  static uint32_t __attribute__((used)) example_NoArgsStatic(void)
  {
      volatile uint32_t val = 42;
      return val;
  }
   
   
  /**
   * Function with simple scalar arguments.
   */
  uint32_t __attribute__((used)) example_Addition(uint32_t a, uint32_t b)
  {
      return a + b;
  }

• Host-side tests

  On the host, the target functions are called in PyTest test cases shown below. Note that the tests take test fixtures as arguments. DOTT offers pre-defined test fixtures for re-occurring tasks such as downloading the firmware onto the target or resetting the target. Note that for most tests it is important to bring the target into a defined, well known state before starting with the test execution. More information on DOTT test fixtures can be found in the Developer's Guide.

  ##
  def test_example_NoArgs(self, target_load, target_reset):
      res = dott().target.eval('example_NoArgs()')
      assert(42 == res)
   
  ##
  def test_example_NoArgsStatic(self, target_load, target_reset):
      res = dott().target.eval('example_NoArgsStatic()')
      assert(42 == res)
   
  ##
  def test_example_Addition(self, target_load, target_reset):
      res = dott().target.eval('example_Addition(31, 11)')
      assert(42 == res)

Functions with Pointer and Struct Arguments

The following examples illustrate how to call target functions which take or return pointers or structs. In contrast to simple scalars, memory on the target must be allocated which is then loaded with the intended values. Note that this does not imply the existence of a heap on the target but a DOTT-specific mechanism is used for memory allocation. Details on that are provided in the Developer's Guide.

• Target functions with pointer arguments

  A target function example taking three pointers, one used for the result of the addition and two for the operands.

  /**
   * Function with pointer arguments and pointer-based return value.
   */
  uint32_t __attribute__((used)) example_AdditionPtrRet(uint32_t *const a, uint32_t *const b, uint32_t *sum)
  {
      *sum = *a + *b;
      return *sum;
  }

• Host-side tests with pointer arguments

  First, memory has to be allocated on the target. For memory allocation and initialization a DOTT function called mem.alloc_type is used. Note that this function does not require dynamic memory (heap) in the target. Details on on-target memory allocation are provided in the Developer's Guide. The allocated memory can be accessed either via GDB convenience variables (starting with a $ prefix) or via their addresses as returned by mem.alloc_type. The names of the GDB convenience variables can be set as optional arguments when calling mem.alloc_type.

  The following two, functionally identical, tests show how to call a target function taking pointer arguments. The first variant uses GDB convenience variables while the second one directly uses the memory addresses returned by mem.alloc_type in Python f-Strings.

  ## variant relying on "GDB convenience variables"
  def test_example_AdditionPtrRet(self, target_load, target_reset):
      dott().target.mem.alloc_type('uint32_t', val=10, var_name='$a') # allocate target memory for an uint32_t
      dott().target.mem.alloc_type('uint32_t', val=999, var_name='$b')
      dott().target.mem.alloc_type('uint32_t', val=0, var_name='$sum')
      res = dott().target.eval('example_AdditionPtrRet($a, $b, $sum)')
      my_sum = dott().target.eval('*$sum')
      assert(1009 == res), 'Unexpected return value'
      assert(1009 == my_sum), 'Unexpected return value'
   
  ## alternate variant without using "GDB convenience variables"
  def test_example_AdditionPtrRet_Alternate(self, target_load, target_reset):
      dt = dott().target
      p_a = dt.mem.alloc_type('uint32_t', val=10)
      p_b = dt.mem.alloc_type('uint32_t', val=999)
      p_sum = dt.mem.alloc_type('uint32_t', val=0)
      res = dt.eval(f'example_AdditionPtrRet({p_a}, {p_b}, {p_sum})')
      my_sum = dt.eval(f'*{p_sum}')
      assert(1009 == res), 'Unexpected return value'
      assert(1009 == my_sum), 'Unexpected return value'

• Target functions with struct arguments

  This on-target function takes a struct as argument and performs an addition of two of the struct elements and saves the result in another struct element.

  typedef struct {
      uint32_t a;
      uint8_t  paddB; // non-word size padding for testing purposes
      uint32_t b;
      uint32_t sum;
  } my_add_t;
   
   
  /**
   * Function with struct as argument.
   */
  uint32_t __attribute__((used)) example_AdditionStruct(my_add_t ms)
  {
      ms.sum = ms.a + ms.b;
      return ms.sum;
  }

• Host-side tests with struct arguments

  In the host-side test, again on-traget memory is allocated for the struct and the address is stored in $add_data. The struct elements are then filled and the The de-referenced struct pointer is then passed as argument to the target function example_AdditionStruct.

  ##
  def test_example_AdditionStruct_Alternate(self, target_load, target_reset):
      dt = dott().target
      p_dat = dt.mem.alloc_type('my_add_t')
      dt.eval(f'{p_dat}->a = 55')
      dt.eval(f'{p_dat}->b = 22')
      dt.eval(f'{p_dat}->sum = 0')
      res = dt.eval(f'example_AdditionStruct(*{p_dat})')
      assert(77 == res), 'Unexpected return value'

  Note that filling the (on-target) struct elements with desired values is a relatively slow process since every value setting operation translates to a single debugger memory access via GDB. To enhance performances, target memory can be also accessed in bulk operations which leads to a performance improvement. Details on this can be found in the Developer Guide.

Strings and Array Arguments

With DOTT is is also possible to pass strings and other array data to target functions which is illustrated in this example.

• Target functions

  The first example implements a function returning the length of a string. The second example takes an array of integers and returns the sum of the elements.

  /**
   * Function taking a string argument and returning the string's length.
   */
  int32_t __attribute__((used)) example_StringLen(char *msg)
  {
      return strlen(msg);
  }
   
   
  /**
   * Function which returns the sum of the elements in the provided array.
   */
  static int32_t __attribute__((used)) example_SumElements(uint16_t *elem, uint16_t elem_sz)
  {
      int32_t ret_val = 0;
      for (uint16_t i = 0; i < elem_sz; i++) {
          ret_val += elem[i];
      }
      return ret_val;
  }

• Host-side tests

  On the host, for both tests first memory has to be allocated on the target to hold the string/ array content. Next, the array on target has to be filled with content using the mem.write function provided by DOTT. As a convenience feature, initial initialization can also be performed when calling mem.alloc_type. Note that the second examples uses Python's ctypes module to convert the Python integer array into an array of uint16_t types. Thereafter, the example functions on the target can be called and the results can be checked against the expected ones.

  ##
  def test_example_ArgString(self, target_load, target_reset):
      msg = 'Sensing is life.\0'
   
      addr = dott().target.mem.alloc_type('char', cnt=len(msg), val=0x0)
      dott().target.mem.write(addr, msg.encode(encoding='ascii'))
      res = dott().target.eval(f'example_StringLen({addr})')
      assert(len(msg) - 1 == res), f'expected: {len(msg)}, is: {res}'
   
      # same as above but with direct setting of initial data (skipping mem.write)
      addr = dott().target.mem.alloc_type('char', cnt=len(msg), val=msg)
      res = dott().target.eval(f'example_StringLen({addr})')
      assert(len(msg) - 1 == res), f'expected: {len(msg)}, is: {res}'
   
  ##
  def test_example_SumElements(self, target_load, target_reset):
      elements = [0, 1, 2, 65535, 99]
      addr = dott().target.mem.alloc_type('uint16_t', cnt=len(elements), val=0x0, var_name='$elements')
      if dott().target.byte_order == 'little':
          arr = (ctypes.c_int16 * 5)(*elements)
      else:
          arr = (ctypes.c_int16.__ctype_be__ * 5)(*elements)
      dott().target.mem.write(addr, bytes(arr))
      res = dott().target.eval(f'example_SumElements($elements, {len(elements)})')
      assert (sum(elements) == res), f'expected: {sum(elements)}, is: {res}'

Function Pointer Arguments

This example demonstrates how to to deal with target functions which take function pointers as arguments.

• Target functions

  On the target, two functions are defined taking two integer arguments, performing and addition or subtraction, and returning the result. A third function, example_CustomOperation, takes three arguments: A function pointer and two integer operands. It calls the function pointed two by the function pointer with the two integers as arguments and returns the result.

  /**
   * Function taking two arguments, adding the second to the first and returning
   * the result.
   */
  int32_t __attribute__((used)) example_FunctorAdd(int32_t a, int32_t b)
  {
      return a + b;
  }
   
   
  /**
   * Function taking two arguments, subtracting the second from the first and
   * returning the result.
   */
  int32_t __attribute__((used)) example_FunctorSub(int32_t a, int32_t b)
  {
      return a - b;
  }
   
   
  /**
   * Function taking a function pointer and two integer arguments. It 'executes'
   * the function pointed to by the pointer on the two arguments and returns the
   * result.
   */
  int32_t __attribute__((used)) example_CustomOperation(int32_t (*func_ptr)(int32_t, int32_t), int32_t a, int32_t b)
  {
      return (*func_ptr)(a, b);
  }

• Host-side tests

  On the host, a test is implemented which first gets the address of subtraction target function (example_FunctorSub). Next, it calls the target's example_CustomOperation function and passes this address as first argument and checks the result. It then performs the same steps with example_FunctorAdd.

  ##
  def test_example_CustomOperation(self, target_load, target_reset):
      args = (44, 22)
      exp_diff = args[0] - args[1]
      exp_sum = sum(args)
   
      func_sub_ptr = dott().target.eval('&example_FunctorSub')
      res = dott().target.eval(f'example_CustomOperation({func_sub_ptr}, {args[0]}, {args[1]})')
      assert(exp_diff == res), f'expected: {exp_diff}, is: {res}'
   
      func_add_ptr = dott().target.eval('&example_FunctorAdd')
      res = dott().target.eval(f'example_CustomOperation({func_add_ptr}, {args[0]}, {args[1]})')
      assert(exp_sum == res), f'expected: {exp_sum}, is: {res}'

Data Injection in Sub-calls

An important part when testing is to inject well-defined data into the test execution. With DOTT this is possible by overriding the return values of functions which are called by other functions. Technically, this is done by setting breakpoints in these sub functions and altering the return values of these functions. This is illustrated in the following example.

• Target functions

  On the target, there is a function (example_AdditionSubcalls) performing an addition of two values which are supplied by two sub functions, example_GetA and example_GetB. Note that these two sub functions are marked with DOTT_NO_OPTIMIZE which prevents the compiler from optimizing out these two functions and hence making the example fail. More information on DOTT_NO_OPTIMIZE is provided in the Developer's Guide.

  /**
   * Local function returning an integer.
   */
  static uint32_t __attribute__((used)) DOTT_NO_OPTIMIZE example_GetA(void)
  {
      uint32_t a = 42;
      return a;
  }
   
   
   
  /**
   * Local function returning an integer via pointer argument.
   */
  static uint32_t  __attribute__((used)) DOTT_NO_OPTIMIZE example_GetB(uint32_t *const b)
  {
      *b = 21;
      return 0;
  }
   
   
  /**
   * Function which calls two local functions to get the input for the computation.
   */
  uint32_t __attribute__((used)) example_AdditionSubcalls(void)
  {
      uint32_t a = example_GetA();
      uint32_t b;
      example_GetB(&b);
   
      return a + b;
  }

• Host-side tests

  On the host, the first test calls the exmaple_AdditionSubcalls function without any injection. In the next test, two InterceptPoints are set up to intercept the two sub-functions example_GetA and example_GetB.

  ‍Note: An intercept point is a a special variant of a breakpoint. DOTT supports different types of breakpoints where not all of them can be used under all circumstances. Please check the Developer Guide for additional details.

  In the intercept points' reached methods the return values of the functions are altered. The intercept points are located at function entry. The calls to ret() let the functions return and hence the function bodies are not executed. Instances of the two intercept point classes are created and attached to the target functions. After calling example_AdditionSubcalls the return value reflects the data that was injected into the sub-functions.

  ##
  def test_example_AdditionSubcalls(self, target_load, target_reset):
      res = dott().target.eval('example_AdditionSubcalls()')
      assert(63 == res)
   
  ##
  def test_example_AdditionSubcallsExtIntercept(self, target_load, target_reset):
   
      class BpA(InterceptPoint):
          def reached(self):
              self.ret(10)
   
      class BpB(InterceptPoint):
          def reached(self):
              self.eval('*b = 89')
              self.eval('*b += 10')
              val = self.eval('*b')
              assert(val == 99)
              self.ret(0)
   
      bpa = BpA('example_GetA')
      bpb = BpB('example_GetB')
   
      res = dott().target.eval('example_AdditionSubcalls()')
   
      bpa.delete()
      bpb.delete()
   
      assert(109 == res)

  ‍Note: Using a DOTT_LABEL it is also be possible to perform the interception of the sub-functions not only on function entry but at arbitrary locations within the function. Please check the Developer Guide for additional details.