Hello everyone!

In this article, we will cover the main methods of simulation and firmware testing for AVR microcontrollers.

Why do we need to test AVR microcontrollers?

Any testing of the software is necessary in order to make sure that the program is working. Testing also ensures that the program or certain parts of it meet specific requirements. However, passed tests do not guarantee a complete absence of errors in the project. They only increase the probability of issue detection at early stages of development.

An error during compilation has zero cost. At the stage of initial testing, the error cost is equal to the cost of the developer’s time. At the stage of alpha or beta testing, the cost of hardware errors is increasing. The price of fixing errors found after launching a product in a series can cover the cost of the entire project. You are lucky if it’s just about releasing a hotfix (though it might be not so easy for hardware projects). In the worst case, you will have to revoke the whole batch. The main purpose of testing is to save money and time.

Comparison of testing in software and hardware projects

Software testing is performed on standardized hardware, at a high level of abstraction. The standard environment and ready-made test frameworks allow for unambiguous test results, regardless of the hardware on which the product is running.

The situation with the testing of microcontrollers (and hardware devices in general) is simpler but still quite confusing. Testing is easier because microcontrollers are mainly designed for simple tasks. The examples of such tasks are LED blinking, sensor data gathering, simple data processing, controlling display output, communication with other peripheral modules. The main difficulty is that the environment where the program executes depends on the model and manufacturer of the controller. Firmware runs at the low level. Given the stringent performance requirements, testing becomes a daunting task.

What kind of testing can I use depending on the task?

There are three main options for running tests on embedded systems. Let’s list them all:

1.Local testing (on the host machine).
Advantages: tests start and work quickly, no need to access the device.
Disadvantages: limited testing area. It will not work with external peripherals. A good example of using this method is the testing of a platform-independent computational algorithm that requires a dataset from sensors. The testing of such an algorithm with a real data source would be very inconvenient. It’s much better to prepare a dataset in advance. You can do this by using a small logger program on your unprocessed sensor data.

2.Testing the MCU in simulation.
Advantages: no device needed for testing. You can program your own environment for the MCU.
Disadvantages: limited accuracy of MCU and environmental simulations, difficulty in creating and configuring such an emulator.

3.Testing the firmware on real MCU.
Advantages: it is possible to work with the periphery of the MC, the firmware will work out the same way as in production.
Disadvantages: you need to have a ready-made device with all peripherals and electronic components. The test cycles will take a long time due as you will have to constantly reflash the MCU. It is very difficult to automate testing using this method.

In this article we will analyze the first two methods as the most promising for automation.

Local Testing

Local tests are very good for testing of the firmware parts that are not dependent on the environment. Examples can be any computing algorithms, various containers, lists, queues, trees, high-level exchange protocols, finite machines, etc.

Let’s consider an example for AVR microcontrollers with the possibility of local testing of the platform-independent firmware parts.

Testing firmware compatible with AVR MCU

For AVR MCU the most convenient and productive development environment is Atmel Studio. Unfortunately, this environment is not cross-platform and is only available for Windows.

For a clear comprehension of examples in this project, I use open-source tools. I rely on VSCode on Ubuntu for source code, AVR GNU Toolchain for compile, and link firmware, gcc for compiling tests and simulator.

The project build process (compile, link, firmware) is done with the make utility. This approach allows me to automate test execution and firmware upload to the target system.

For example, let’s consider the firmware for the Atmega1284 microcontroller that implements the functionality of a simple thermometer.

Temperature measurement is performed by reading the voltage on the voltage divider (which includes a thermistor), converting the ADC value into temperature values, and displaying it on the 1602 (hd44780) LCD screen.

The function of communication with the internal and external periphery can be performed via operating the microcontroller registers.

Nevertheless, our firmware has functionality which it is necessary to test (converting ADC value into temperature and then displaying it on the LED screen).

Let’s consider an example of native testing of the firmware functionality.

Repo link: https://github.com/maddevsio/AVR_Testing

You will need AVR GNU Toolchain to compile.

├── builds — executable files
│ ├── firmware
│ └── tests
└── src — sources files
 ├── inc — peripherials drivers, computing source
 │ ├── adc.c
 │ ├── adc.h
 │ ├── calculate.c
 │ ├── calculate.h
 │ ├── lcd.c
 │ └── lcd.h
 ├── main.c — entry point
 ├── Makefile
 ├── obj — objects for linking firmware
 ├── test_obj — objects for linking tests
 └── tests — test sources
    ├── test.c
    └── test.h

To create a project with tests, you need to separate sources by function. What runs only for MCU should not be mixed with source files that can run on local systems. That is not an easy task. Also, you should use two compilers, making some modules platform-dependent and others not. Let’s look at main.c for more details.

#include "inc/calculate.h" // Testing module
#include <stdio.h>
#ifdef TEST //If we define TEST directive, than use testing librarry
 #include "tests/test.h"
#else // Else we connect peripherials drivers, initialize global variables
#include "inc/adc.h"
#include "inc/lcd.h"
 #define ADC_PIN 1
 uint16_t adcValue;
 float temperature;
#endif
int main(void)
{
 #ifdef TEST // Run test if TEST directive is defines
 RUN_TESTS();

 #else // Else do what firmware needs to do
 adcInit();
 lcdInit();
 for(;;){
 char temp[10] = {0};
 adcValue = adcRead(ADC_PIN);
 temperature = adcToTemp(adcValue); 
 sprintf(temp, "%f", temperature); 
 lcdPrintLn(temp);
 }
 #endif
}

In local testing there are two sections of main.c:
1. For testing functionality
2. For implementing the main program cycle.
Let’s consider testing conversion of ADC values to temperature
(src/tests/test.c file):

struct CalcTestCase // Testcase structure, consist ADC value and calculated temperature value according to ADC
{
 uint16_t adcVal[100];
 double temperature[100];
};
struct CalcTestCase calcCase = {
.adcVal= 
 {
10, 20, 30, 40, 50, 60, 70, 80, 90, 100
 ,110, 120, 130, 140, 150, 160, 170, 180, 190, 200
// … … … 
 , 810, 820, 830, 840, 850, 860, 870, 880, 890, 900
 , 910, 920, 930, 940, 950, 960, 970, 980, 990 
 },
 .temperature = 
 {
-55.609, -45.834, -39.719, -35.168, 
-31.5, -28.4, -25.699, -23.293, -21.116,
// … … …
83.642, 87.215, 91.284, 95.869, 101.222, 
107.639, 115.620, 125.990 
}
 };
double temperatureDeviation = 0.12; // Maximum absolute deviation
void RUN_TESTS(void){
 test_AdcCalc(); 
}
void test_AdcCalc(void){ // Iterate over all testcase values
 uint8_t adc_caseSize = sizeof(calcCase.adcVal)/sizeof(calcCase.adcVal[0]);
 for (int i = 0; i<adc_caseSize-1;i++){
 double calcResult = adcToTemp(calcCase.adcVal[i]); // Using calculation value from firmeware modules
 assert(test_DivEqual(calcResult, temperatureDeviation, calcCase.temperature[i])); 
 }
}
bool test_DivEqual( double testVal, double diviation, double value){ // Comparsion actual and calculated values
 if (((testVal <= value + diviation) && (testVal >= value - diviation))) { 
 return true; 
 }
 else
 {
 return false;
 }
}

When implementing such a test, it is necessary to split the Makefile in two sections: the first section will use a compiler from AVR Toolchain — avr-gcc (compiler for AVR microcontrollers), and the second section will use gcc for compiling of the platform-independent executable files.

This example enables you to test the functionality of converting ADC values to temperature using calculated temperature values depending on the input voltage of the microcontroller. When you change the conversion functionality, you just need to run the tests and make sure that they have not detected any errors.

To compile and run the tests, do the following:

$ cd AVR_Testing/src
$ make tests

after a successful compilation, the executable test file will appear in the AVR_Testing/builds/tests/ directory.

Native test development is quite a huge task, and implementing it takes a lot of time. Therefore, it will be more productive to use one of the existing frameworks for testing.

Testing with Unity framework

Let’s run the same tests using Unity, a simple framework for testing of embedded systems.

In the same project repository (the unity-framework branch) you will find an example of the framework usage.

When using Unity, the compilation of tests and firmware is done separately, like in native tests. The testing of each module has its own entry point and can be run independently if you use Unity.

To start using the framework, simply clone the framework’s repository.

git clone https://github.com/ThrowTheSwitch/Unity.git

Ruby is used to generate sources of an executable file from a test case file. To install it, do the following:

sudo apt install ruby-full

Another remark: to use unity in your project, you need to specify the absolute path to it in the variable $(UNITY_ROOT) of your Makefile.

An example of a testcase source:

#include "calculate.h"
#include "unity.h"
double temperatureDeviation = 0.12;
struct CalcCase
{
 uint16_t adcVal[100];
 double temperature[100];
};

struct CalcCase calcTestCase = {
.adcVal= 
 {
10, 20, 30, 40, 50, 60, 70, 80, 90, 100
  // ...  ... ...
 , 910, 920, 930, 940, 950, 960, 970, 980, 990 
 },
 .temperature = 
 {
-55.609, -45.834, -39.719, -35.168, 
 // ...  ... ...
107.639, 115.620, 125.990 
}
 };

void setUp(void)
{

}
void tearDown(void)
{

}
void test_adcToTemp(void)
{
int8_t caseSize = sizeof(calcTestCase.adcVal)/sizeof(calcTestCase.adcVal[0]);
#ifdef SHOW_TEST_RESULT
printf("Adc to temp assertion. calculated value:actual value\n");
#endif
for (int i=0;i<caseSize - 1;i++){
 double calculatedTemp = adcToTemp(calcTestCase.adcVal[i]);
 #ifdef SHOW_TEST_RESULT
 printf("%f:%f\n",calculatedTemp,calcTestCase.temperature[i]); 
 #endif
 TEST_ASSERT_DOUBLE_WITHIN(temperatureDeviation,calcTestCase.temperature[i],calculatedTemp);
}
}

In this example, we only need to specify the header file of the sources of the module under test and implement the verification of the operation of converting ADC values to temperature with the acceptable value for the observational error.

Besides writing test cases for Unity and specifying the path to the framework, you should also add special compiler directives.

In this case, we use a comparison of numbers with a floating point of increased accuracy. Testing of double types is disabled in the framework by default, you need to add directives to enable the double types:

-DUNITY_INCLUDE_DOUBLE #Enables double type
-DUNITY_DOUBLE_PRECISION=0.001f — e #Set presiccion of double comparsion

After you hit make tests, Ruby will generate test entry point files, they will get compiled and run.

Simulator using testing

In most hardware projects, the microcontroller is used to interact with external and internal peripherals, whose behavior is very difficult and sometimes impossible to simulate during tests. Target system simulators are very helpful in solving this problem.

For microcontrollers with AVR architecture, there are several simulation systems listed below:

• Atmel Studio is the development environment for AVR microcontrollers that I have mentioned earlier. (The one that works only for Windows.) It includes a built-in simulator for almost all AVR microcontroller models and a handy toolkit for creating and controlling both internal and external peripherals. The drawback is the lack of cross-platformity and cumbersomeness — it’s hard to automate the simulation.

• Simulavr is an open-source cross-platform C/C++ project that offers its own AVR microcontroller simulation systems. There are several models of microcontrollers, the ability to write debug scripts on Tcl/Tk and Python. The disadvantages are the complicated mechanism of adding new MC models to the simulator, complex documentation, and the fact that the project remains unsupported for several years.

• simavr — relatively new (and at the same time quite flexible) cross-platform project for writing AVR MCU simulators in C/C++. Because of the similar name, it can be confused with the previous tool, however, these are completely different projects. Simavr has a large number of AVR MCU models, the architecture enables you to easily add new devices and models. It also has integration with PlatformIO (vscode extension for embedded systems development) and clear examples of how to use the tool with different kinds of peripherals. Disadvantages: the lack of a description of the project build and rather obscure documentation.

I chose the simavr as the most promising option.

To start using simavr, just clone and bulild the repository.

git clone https://github.com/buserror/simavr/
cd simavr
make

After a successful build, you can test the simulator’s performance by running tests or examples. To run the executable files of this simulator, you need to create a symbolic link to the sources in the firmware directory. For example, in the simavr/tests directory:

atmega88_timer16.axf - mcu firmware that teed to test
obj-%your_system_architecture%/test_atmega88_timer16.tst - compiled enviroment, peripherials and simulator

Here’s the symbolic link for executable file creation:

ln -s obj-%your_system_architecture%/test_atmega88_timer16.tst mega88timer
./mega88timer

In the simavr/examples you will find folders with sources of simulations, and the parts folder contains the sources of common peripherals.

For a more immersive effect, you can run examples with graphics — such as board_hd44780, board_ssd1306.

Simavr provides a wide range of tools for the following types of tasks:

• Development of custom virtual boards with microcontrollers and peripherals.
• Development of virtual electronic components.
• Managing simulation behavior on time segments up to microcontroller tact.
• Connection of avr-gdb debugger.

A full description of the simulator’s features can be found in the project repository.

Structurally any simulator on a simavr represents sources of board, peripherals, and compiled firmware (by the way, a simavr simulator enables you to load the same firmware like on a real device, changes are not required).

For the example of testing let’s look at the tests of the same thermometer while using a simulator (see the simavr-testing branch).

The structure of src/tests/sim directory is as follows:

├── adcToLcd.c - board sources
├── main_wrapper.c — wrapper for firmware entry point - main.c
├── Makefile
├── obj-x86_64-linux-gnu — objects folder(name depends of system achitecture)
└── parts — board peripherials (virtual electronic components)
 ├── hd44780.c (1602 lcd implementation)
 └── hd44780.h

Schematic of the virtual board:

Simavr has a very simple, though not quite obvious project structure.

Let us have a look at the most important moments of a board implementation:

main_wrapper.c is the wrapping of the entry point for firmware. It provides the compiler and the simulator with additional information about firmware, power supply voltage, and other parameters (full description of all parameters can be found in simavr/simavr/sim/avr/avr_mcu_section.h).

#undef F_CPU
#define F_CPU 8000000
#include "avr_mcu_section.h"
#define VCC 5000
#define AVCC 5000
#define AREF 5000
AVR_MCU(F_CPU, "atmega1284"); //Set MCU frequency and model
AVR_MCU_VOLTAGES(VCC, AVCC, AREF); //Set power supply, adc, aref voltage
#include "../../main.c"

adcToLcd.c contains sources of the board as well as descriptions for manipulations with the periphery, data ports and operating time intervals between actions of the periphery.

int main(int argc, char *argv[])
{
 firmwareInit(firmware,argv[0]); //MCU initializing, firmware flashing
 hd44780_init(avr, &hd44780, 16, 2); //LCD initializing
 setConnections();// Connecting all parts together
 avr_cycle_timer_register_usec(avr, lcdTimer, lcdDataGathering, NULL); //Timer registration. It will fire lcdDataGathering after lcdTimer usec
 while (!simulationCompleted){ //Wait until simulation is complete
 avr_run(avr);
 }
 if (run_test()) //Run test with simulation data
 {
 printf("TEST PASSED.\n");
 }
 else{
 printf("TEST FAILED.\n");
 }
}

When initializing firmware, the board searches for the specified firmware elf-file, then creates MCU and uploads the firmware to it.

To initialize and work with the display, in the parts directory find the sources of the LCD screen on the hd44780 controller (for the project I just took them from simavr/examples/parts and refactored the function of displaying the data on the screen for parsing and returning a double-value).

Next, the function of connecting peripheral parts to the setConnections controller is applied, which uses such methods as:

avr_io_getirq(avr_t *avr, uint32_t ctl, int index)

— it returns a pointer to the unique identifier of the I/O port PIN. Arguments are controller identifier, port, PIN.

(See simavr/simavr/sim/sim_io.h for details).

avr_connect_irq(avr_irq_t *src, avr_irq_t *dst)

— this is the function of connecting one PIN I/O (peripheral or microcontroller) to another.

The example of a use for this function:

avr_connect_irq(avr_io_getirq(avr, AVR_IOCTL_IOPORT_GETIRQ(‘D’), \ 4),hd44780.irq + IRQ_HD44780_RS);

— connecting PORTD PIN 4 of MCU to RS PIN of LCD.

You can raise signals to be sent to the I/O ports using the following function:

avr_raise_irq(avr_irq_t *irq, uint32_t value)

— it accepts the port identifier and the required value. Value can be equal 1 or 0 for digital inputs, or store a value in millivolts for analog inputs.

Simavr has a convenient mechanism for setting time for external events by registering timers. On triggering of the timer, a callback function is executed.

There is a function for registering the timer:

void avr_cycle_timer_register_usec(struct avr_t *avr, uint32_t when, avr_cycle_timer_t timer, void *param)

— it accepts the controller identifier, the actual time after which the timer will trigger, the callback which will trigger and its optional parameters.

In our board, the callback performs the function of reading the screen output, switching to the ADC output of the next voltage value and registering a new timer. For adequate reading of information from the screen, the board uses a timer setting the delay until the display is filled with new data.

To perform every tick of MCU please use:

avr_run(*mcu identifier)

— using this mechanism enables you to suspend the simulation for some time to verify the data or perform calculations. It also serves as a mechanism for launching the simulator in a separate thread.

In our example, we use a simple implementation with running the simulation in a single thread, collecting the simulation results, and testing the collected data.

In order not to overload the code with two testing methods, it was decided to put the test case into a separate file adc-temp_test.c which connects to the project both when compiling tests on Unity and when using the simulator.

To compile and run the simulation, just specify the absolute path to simavr in src/tests/sim/Makefile, in the $(SIMAVR) variable, and execute it in the src directory:

make sim-test
cd tests/sim
./adcToLcd

Unlike modular tests, tests using a simulator (in fact, these are integration tests) allow to fully evaluate the correctness of firmware operation.

Simulation of ADC conversion is a clear example of the quality of testing in our project. Due to the limitations of ADC discretization, when calculating low values the simulation goes with considerable observational error than it is found out during the unit tests.

For larger projects, you can easily integrate the simulator with the Unity framework and automate the testing process via CI/CD.

Testing the firmware on real hardware

The launch of hardware tests is mostly done manually, using on-chip debugging systems such as JTAG.

To automate hardware tests, it is necessary to emulate the behavior of the external periphery using another microcontroller. The adequacy of the behavior of the emulated periphery is questionable. In this case, test might even be more expensive to write than the firmware itself. Anyway, this approach ensures maximum accuracy and closeness to real conditions. Description of test preparation methods deserves a separate article, and we will tackle this topic in the next publications.

Conclusion

There is a lot more to tell about firmware testing. There is no standard method for testing a platform-dependent code. However, general recommendations when writing tests and simulators for specific models of AVR microcontrollers allow to fully automate the firmware testing process. I hope this article has helped you to acquire basic knowledge about the testing of AVR microcontrollers.

Previously published at maddevs.io