# Introduction This article is about **easy** and **compile-time configurable** implementation of binary communication protocols using C++11 programming language, with main focus on embedded systems (including bare-metal ones). Interested? Then buckle up and read on. # Background Almost every electronic device/component nowadays has to be able to communicate to other devices, components, or outside world over some I/O link. Such communication is implemented using various communication protocols, which are notorious for requiring a significant amount of boilerplate code to be written. The implementation of these protocols can be a tedious, time consuming and error-prone process. Therefore, there is a growing tendency among developers to use third party code generators for data (de)serialization. Usually such tools receive description of the protocol data structures in separate source file(s) with a custom grammar, and generate appropriate (de)serialization code and necessary abstractions to access the data. The main problem with the existing tools is that their major purpose is data structures serialization and/or facilitation of remote procedure calls (RPC). The binary data layout and how the transferred data is going to be used is of much lesser importance. Such tools focus on speed of data serialization and I/O link transfer **rather than** on safe handling of malformed data, compile time customization needed by various embedded systems and significantly reducing amount of boilerplate code, which needs to be written to integrate generated code into the product's code base. The binary communication protocols, which may serve as an API or control interface of the device, on the other hand, require a different approach. Their specification puts major emphasis on binary data layout, what values are being transferred (data units, scaling factor, special values, etc.) and how the other end is expected to behave on certain values (what values are considered to be valid and how to behave on reception of invalid values). It requires having some extra meta-information attached to described data structures, which needs to propagate and be accessible in the generated code. The existing tools either don't have means to specify such meta-information (other than in comments), or don't know what to do with it when provided. As a result, the developer still has to write a significant amount of boilerplate code in order to integrate the generated serialization focused code to be used in binary communication protocol handling. # Required Features As an embedded C++ developer, I require the following features to be available to me, at least to some extent. ## Polymorphic Interfaces Configuration Usually every message definition is implemented (or code generated) as a separate class. In many cases, there is a common code that is applicable to all such message classes. The proper implementation would be to introduce polymorphic behavior with virtual functions (such as reading message data, writing it, calculating serialization length, etc.). However, creation of full interface to be polymorphic may be impractical. Every application that might use the protocol definition code is different. For example, in case many messages are uni-directional, one side (**client**) will require polymorphic write (but not read) for such messages, the one side (**server**) will require the opposite - polymorphic read (but not write). Most of the virtual functions will end up being included in the final binary / image, even if they are unused. It can result in unnecessary code bloat, which may be a problem for embedded systems (especially bare-metal ones). There is a need for compile time, or at least code generation time configuration of the polymorphic interfaces that are going to be used. In the best case scenario, such configuration should be done per message class. ## Extra Meta Information There may be a significant amount of extra meta information that comes with the protocol definition. For example, one of the protocol fields in one of the messages needs to report a distance between two points. The protocol designer has decided to report the distance in **centimeters**. Such information will probably be written in plain text in protocol specification and if some third party code generator is used, then this information might end up being in the comment section describing the field or a message. However, in most cases, such information will not end up being in generated code. The developer that needs to integrate the generated code into its business logic needs to manually write some boilerplate code to do the math of conversion into the different distance units (such as **meters**) relevant to the application being developed. It is also not very uncommon for the binary protocol being specified at the same time the application that uses it being developed. Imagine that some specific use case pops up during the development, where distance in **centimeters** doesn't provide sufficient precision. Thanks to the fact that protocol specification hasn't been finalized yet, the developer decides to change the reported distance units from **centimeters** to **millimeters**. It means that other developers that might have written their boilerplate code with assumption of distance units being **centimeters** must be aware of the change and not forget to modify their code. Some protocol developers also dislike sending floating point numbers "as-is" over the communication link. In many such cases, floating point numbers are multiplied by some predefined value and sent over the I/O links as integers while the remaining fraction after the decimal point is dropped. The other side does the opposite operation on reception of the message, i.e., dividing the received integer by the same predefined value to get the floating point number. Which predefined value to use for such scaling is also meta information, which is not transferred "on the wire" and should be present in the generated or manually written code of the protocol definition. Also in many cases, the protocol may define some values with special meaning. Let's assume there is a need to communicate a delay in seconds before some event needs to happen. There should be some special value that indicates **infinite** duration. Usually it is either **0** or maximum possible value of the unsigned type being used. The developer that integrates the generated code into the application needs to know what value it is. There is a need to write at least one extra piece of boilerplate code that wraps the special value and gives it a name. Wouldn't it be better if the generated code contained such helper function already? Many protocols specify ranges of valid values and expect certain behavior on invalid values. There is an expectation that generated code would provide an ability to inquire that the received value is valid to avoid manually written boilerplate code that checks this information. The list of such examples with meta-information that is part of the protocol definition, but not transfered as data of the messages can go on and on. There is a need for the generated code to provide the required functionality to be used by the integration developer without any concern of what to do when the meta-information is modified. ## Customization of Data Types Most of the currently available solutions for the protocol code generation use hard-coded types for some particular data structures, such as `std::string` for strings and/or `std::vector` for lists. Also in many cases, the generated functions that perform serialization / deserialization receive their input / output as `std::istream` or `std::ostream`. These data structures may be unsuitable for some applications, especially embedded (including bare-metal) ones. There is a need to be able to substitute the default data structures with some equivalent replacements, ideally at compile time for selected fields / messages, but globally (during code generation for example) may also be an acceptable solution. ## Excluding Exceptions and Dynamic Memory Allocation Many constrained embedded environments make it difficult to use exceptions as well as dynamic memory allocation (especially bare-metal ones). There is a need for an ability to generate protocol definition code that don't use either of them. ## Efficient Built-In Mapping of Message ID to Type Usually, when new encoded message arrives over I/O link, encoded as raw data, it has some transport framing containing numeric message ID. Unfortunately, most (if not all) of the available protocol code generation solution leave the task of mapping numeric ID into the actual message type (or appropriate handling function) to be written by the developer who integrates the generated code into the business logic. Usually, such code is boilerplate, efficiency of which depends on the competence of the developer. ## Third Party Protocol Support Many of the available protocol generation solutions have their own encoding and framing without any ability to modify it. Using of such tools may be acceptable for new protocols being defined from scratch. However, there are heaps of already defined third party protocols, code for which cannot be generated with such tools. ## Injecting Custom Code This requirement complements the [Third Party Protocol Support](#third-party-protocol-support) one. Even if a chosen code generation solution supports definition of the third party protocol and its grammar for schema files is very rich, there will always be a protocol containing some small nuance, which cannot be represented correctly using the available grammar. As a result, the generated code may be incorrect and/or incomplete. Proper protocol code generation solution should allow injection of snippets of custom, manually written code. # Working Solution Unfortunately, I could not find any third party solution that implements most of the [required features](#required-features) listed above. I had no other choice but to implement something of my own. I'd like to present to you the [CommsChampion Ecosystem](https://commschamp.github.io) that implements all of the mentioned earlier features. It's been in development as my side project for the last several years. Now it has a stable API and is ready to be used by a wider public. Further below, I'll go through the list of the [required features](#required-features) again and give examples on how my solution provides the required functionality. At first, there was a **headers-only**, cross-platform, and very flexible [COMMS library](https://github.com/commschamp/comms_champion#comms-library), which allows implementation of the protocol messages, fields and transport framing using simple declarative statements of types and classes definition. Such statements define **WHAT** needs to be implemented, the library internals handle the **HOW** part. The internals of the [COMMS library](https://github.com/commschamp/comms_champion#comms-library) is mostly template, highly compile-time configurable classes which use multiple meta-programming techniques. As a result, the C++ compiler itself becomes a code generation tool, which brings in only the functionality required by the application, providing the best code size and speed performance. After the library, the [test tools](https://github.com/commschamp/comms_champion#commschampion-tools) followed, which allow analysis, debugging, and visualization of the developed protocol, that was implemented using the [COMMS library](https://github.com/commschamp/comms_champion#comms-library). The test tools are generic and provide a common testing environment for all the protocols. All the applications are plug-in based, i.e., plug-ins are used to define I/O socket, data filters, and the custom protocol itself. Such architecture allows easy assembly of various protocol communication stacks. The tools use Qt5 framework for GUI interfaces as well as loading and managing plug-ins. They are intended to be used on the development PC and are not part [COMMS library](https://github.com/commschamp/comms_champion#comms-library), although hosted in the same repository. With time, the [COMMS library](https://github.com/commschamp/comms_champion#comms-library) grew in features and started requiring more cognitive effort to remember things. It became easier to make mistakes and/or implement the protocol in not a very generic way. As a result, the [code generator]("https://github.com/commschamp/commsdsl) has followed. It allows definition of the protocol using easy [XML based domain specific language](https://github.com/commschamp/CommsDSL-Specification) (DSL) and generates **headers-only** library of protocol definition (which in turn uses the types and classes from the mentioned earlier [COMMS library](https://github.com/commschamp/comms_champion#comms-library)), as well as code required to build the protocol plugin for the [test tools](https://github.com/commschamp/comms_champion#commschampion-tools). Now let's repeat the [required features](#required-features) mentioned earlier and see the solution provided by the [CommsChampion Ecosystem](https://commschamp.github.io). ## Polymorphic Interfaces Configuration The protocol implementation library defines a common interface class for all the messages in the following way. **Note**: All the types and classes from the [COMMS library](https://github.com/commschamp/comms_champion#comms-library) reside in the `comms` namespace, while the definition of the custom protocol in the examples below will be defined in `my_prot` namespace, and application code that uses the protocol definition will be in global namespace. ```cpp namespace my_prot { // Numeric IDs of the protocol messages enum MsgId : std::uint8_t { MsgId_Message1, MsgId_Message2, MsgId_Message3, ... }; // Common interface class for the protocol messages template using Interface = comms::Message< comms::option::MsgIdType, TOptions... >; } // namespace my_prot ``` The default definition passes `comms::option::MsgIdType` to the `comms::Message` (class provided by the `COMMS` library). It is used to define type used for storing message ID. The code above is equivalent to having it defined like this: ```cpp namespace my_prot { class Interface { public: // Type used for message ID typedef MsgId MsgIdType; }; } // namespace my_prot ``` The definition of the common interface class uses variadic template parameters, which are used to pass various default functionality extension **options**. The internals of the `COMMS` library parse the provided options and generate extra requested functionality. For example, adding polymorphic retrieval of message ID, required for the custom application, may look like this: ```cpp using MyAppInterface = my_prot::Interface< comms::option::IdInfoInterface // Add extra polymorphic ID retrieval interface >; ``` The code above is equivalent to having the following definition: ```cpp class MyAppInterface { public: // Type used for message ID (defined by using comms::option::MsgIdType option) typedef MsgId MsgIdType; // NVI of polymorphic retrieval of ID (defined by using comms::option::IdInfoInterface option) MsgIdType getId() const { return getIdImpl(); } protected: // Polymorphic retrieval of ID (defined by using comms::option::IdInfoInterface option) virtual MsgIdType getIdImpl() const = 0; // implemented in derived class }; ``` Note that the `COMMS` library uses Non-Virtual Interface Idiom (NVI) to define polymorphic behavior. The non virtual interface wrapper function is there to check various pre- and post-conditions the virtual function might require. The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) provides multiple extension **options**, which allow definition of various other polymorphic functions. The (currently) full list is below: ```cpp using MyAppInterface = my_prot::Interface< comms::option::IdInfoInterface, // Add polymorphic ID retrieval interface comms::option::ReadIterator,// Add polymorphic read interface comms::option::WriteIterator, // Add polymorphic write interface comms::option::LengthInfoInterface, // Add polymorphic serialization // length retrieval comms::option::ValidCheckInterface, // Add polymorphic contents validity check comms::option::Handler, // Add polymorphic dispatch to // handling function interface comms::option::NameInterface // Add polymorphic retrieval of message name >; ``` The detailed explanation of every listed above option is a bit beyond the scope of this article. The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) has a very detailed documentation that lists and explains every one of them. The magic behind the extension options and how they end up generating extra types and functions is also outside the scope of this article. It is explained in detail in the free e-book I've written called [Guide to Implementing Communication Protocols in C++](https://alex-robenko.gitbook.io/comms-protocols-cpp/). The definition of an actual message may look like this: ```cpp namespace my_prot { // Definition of the fields used by Message1 struct Message1Fields { // 32 bit unsigned integer, initialized to 0 using F1 = comms::field::IntValue< comms::Field, // use big endian serialization std::uint32_t >; // 16 bit signed integer, initialized to 10 using F2 = comms::field::IntValue< comms::Field, // use big endian serialization std::int16_t, comms::option::DefaultNumValue<10> >; // All the message fields bundled in std::tuple. using All = std::tuple; }; // Definition of the actual Message1. // Template parameter TMsgBase is common interface class required by the application template class Message1 : public comms::MessageBase< TMsgBase, comms::option::StaticNumIdImpl, // Set message ID known at compile time comms::option::FieldsImpl, // Define fields of the message comms::option::MsgType > // Pass the actual message type // being defined > { public: ... // some small irrelevant at this moment code here }; } // namespace demo1 ``` The definition of the actual custom protocol message `my_prot::Message1` is completely generic. It receives the application specific interface definition class as its template parameter, and based on the required polymorphic interface the `comms::MessageBase`, provided by the `COMMS` library, does all the magic of determining the required polymorphic interface and implementing the necessary virtual functions, while also inheriting from the provided interface class. How such magic of determining the required polymorphic functionality is implemented is beyond the scope of this article and also explained in details in my free [Guide to Implementing Communication Protocols in C++](https://alex-robenko.gitbook.io/comms-protocols-cpp/) e-book. So, the usage of such class in the actual application may look like this: ```cpp // Define Message1 relevant to the application using MyMessage1 = my_prot::Message1; // Definition of smart pointer holding any protocol message using MyMessagePtr = std::unique_ptr; // Holding Message1 by the pointer to its interface MyMessagePtr msg(new MyMessage1); // Retrieving serialization length, works only if comms::option::LengthInfoInterface // was passed as option to interface definition, if not compilation will fail. std::size_t msgLen = msg->length(); assert(msgLen == 6U); // 4 byte of f1 + 2 bytes of f2 ``` The code the developer needs to write to integrate protocol definition into the business logic of the application is quite simple. All the complexity is handled by the C++ compiler behind the scenes. However, writing the generic code for the protocol definition may require a bit more of cognitive effort with a bit of knowledge about the [COMMS library](https://github.com/commschamp/comms_champion#comms-library) internals. That's the reason the separate [code generator]("https://github.com/commschamp/commsdsl) has been developed which takes schema file(s) as an input and generates proper generic protocol definition that is simple to customize and use. The definition of the `Message1` message from the example above in [CommsDSL](https://github.com/commschamp/CommsDSL-Specification) schema can look like this: ```xml ... ... ``` The protocol definition code generated out such schema which will look similar to the code example above. ## Extra Meta Information The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) has a built-in support for various units and conversion between them. To follow the example from above with distance, such field could be defined as below: ```cpp namespace my_prot { using Distance = comms::field::IntValue< comms::Field, // Big endian serialization std::uint32_t, // 4 bytes serialization comms::option::UnitsCentimeters // Contains centimeters >; } // namespace my_prot ``` When such field is deserialized and its value in **meters** is needed, the units retrieval functionality, provided by the [COMMS library](https://github.com/commschamp/comms_champion#comms-library) can be used. ```cpp my_prot::Distance distanceField = ...; // Some value assigned double distanceInMeters = comms::units::getMeters(distanceField); ``` In case the definition of the field needs to be changed to contain **millimeters** instead, the client code doesn't need to be changed, just recompiled. The compiler will change the generated math operation to convert between units. The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) also contains compile-time protection against conversion between incompatible units. For example, it is not possible to retrieve **seconds** out of field containing **millimeters**, there will be `static_assert` failure with appropriate error message. The language of [CommsDSL](https://github.com/commschamp/CommsDSL-Specification) schema also contains an ability to specify units, which will result in usage of appropriate option passed to the generated field definition. ```xml ... ``` Both [COMMS library](https://github.com/commschamp/comms_champion#comms-library) and [CommsDSL](https://github.com/commschamp/CommsDSL-Specification) support scaling floating point values and serializing them as integers. Such fields are defined as integers with scaling ratio option. ```cpp namespace my_prot { using ScaledField = comms::field::IntValue< comms::Field, // Big endian serialization std::int32_t, // 4 bytes serialization comms::option::ScalingRatio<1, 1000> // Divide by 1000 to get floating point value >; } // namespace my_prot ``` Get/set of the floating point value from/to such field looks like this: ```cpp my_prot::ScaledField scaledField; scaledField.setScaled(1.23f); float val = scaledField.getScaled(); ``` The definition of such field in [CommsDSL](https://github.com/commschamp/CommsDSL-Specification) may look like this: ```xml ... ``` Combining units with scaling is also possible. Let's define the distance in **1/10** of millimeters. ```cpp namespace my_prot { using NewDistance = comms::field::IntValue< comms::Field, std::uint32_t, comms::option::UnitsMilliimeters, comms::option::ScalingRatio<1, 10> >; } // namespace my_prot ``` The [CommsDSL](https://github.com/commschamp/CommsDSL-Specification) definition of such field looks like this: ```xml ... ``` The retrieval of distance in meters from such field still looks the same, the compiler generates appropriate math operations taking into account both units and scaling ratio. ```cpp my_prot::NewDistance distanceField = ...; // Some value assigned double distanceInMeters = comms::units::getMeters(distanceField); ``` Specification of special values also supported in [CommsDSL](https://github.com/commschamp/CommsDSL-Specification) schemas. Let's define duration in seconds where value `0` means **infinite**. ```xml ... ``` The code generated for such field will contain appropriate member functions to help check for the special value. ```cpp namespace my_prot { struct Duration : public comms::field::IntValue< comms::Field, std::uint32_t, comms::option::UnitsSeconds > { static constexpr std::uint32_t valueInfinite() { return 0U; } bool isInfinite() const { /* checks that the stored value is 0U */ } void setInfinite() { /* assigns 0 to the stored value */ } }; } // namespace my_prot ``` Specifying ranges of valid values is also supported. For example: ```xml ... ``` The example above defines field of 1 byte with unsigned value, where all the values between `0` and `20`, between `40` and `60`, as well as single value `70` are considered to be valid. Every field defined by the means of the [COMMS library](https://github.com/commschamp/comms_champion#comms-library) has `valid()` member function, which can be used to inquire whether the field contains a valid value. The example above will result in proper functionality of the `valid()` member function. It will return `true` for any value between `0` and `20`, between `40` and `60`, and for value `70`. All other values will result in return of `false`. ```cpp my_prot::SomeField field = ...; // some value assigned if (!field.valid()) { ... // do something } ``` ## Customization of Data Types The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) was designed and implemented with a built-in ability to customize default choices of potentially problematic data structures and/or default behavior. Although the default choices for storage types of strings and lists are `std::string` and `std::vector`, it is possible to substitute it with something else as part of compile-time configuration. For example, some `string` field may be defined like this: ```cpp namespace my_prot { template using SomeString = comms::field::String< comms::Field, TOptions... // Extra configuration options >; } // namespace my_prot ``` Such definition of `string` field allows passing extra options, which can be used by the application. For example, if the application is being developed for bare-metal platform, which does not allow usage of dynamic memory allocation, it is recommended to pass `comms::option::FixedSizeStorage` option to substitute usage of `std::string` by the `string` container implementation with fixed maximum size, provided by the [COMMS library](https://github.com/commschamp/comms_champion#comms-library) itself (called `comms::util::StaticString`). ```cpp using MyAppString = my_prot::SomeString >; ``` It is also possible to use any custom third party storage type, that has the same `public` interface as `std::string`. ```cpp using MyAppString = my_prot::SomeString >; ``` The [code generator]("https://github.com/commschamp/commsdsl) is capable (and does it by default) of generating protocol definition code, which allows such compile time customization of various potentially problematic types. ## Excluding Exceptions and Dynamic Memory Allocation The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) was designed specifically for environments with constrained resources. It does not use RTTI and/or exceptions: the violation of pre- and post-conditions are checked using assertions, while runtime errors are reported via returned status codes. The dynamic memory allocation **is** used by default when new serialized message arrives via I/O link and appropriate message object is created. However, there is an available **compile time** configuration **option** that replaces dynamic memory allocation with "in-place" one (uses placement new on pre-allocated area), ## Efficient Built-In Mapping of Message ID to Type The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) provides multiple built-in options and helper functions which allow efficient mapping of message ID to appropriate type, as well as dispatching the message object to its appropriate handling function. They are described and documented in detail in the available documentation. ## Third Party Protocol Support The [COMMS library](https://github.com/commschamp/comms_champion#comms-library) was designed specifically to provide building blocks and facilitate implementation of the third party protocols. Its architecture allows an easy way to complement the default behavior with protocol specific nuances. ## Injecting Custom Code The available [code generator]("https://github.com/commschamp/commsdsl) also allows injection of custom code snippets instead of or in addition to the code that might have been generated by default. # Summary The main intended audience of [CommsChampion Ecosystem](https://commschamp.github.io) is **embedded C++** developers, who have to implement third party (or their own) protocols to communicate to various sensors, other devices or outside world. The non-embedded C++ developers, who don't have to worry about constrained environment and who don't require sophisticated code customization, may still find the provided solution convenient and useful, thanks to available constructs for reducing amount of integration boilerplate code. Those developers who already have their own implementation of some protocol(s) integrated into their product, may still use the available [code generator]("https://github.com/commschamp/commsdsl) as well as [test tools](https://github.com/commschamp/comms_champion#commschampion-tools) to easily test and debug their work. # Where to Start If the available solution caught your interest, please start your learning process by reading [Where to Start](https://commschamp.github.io/#where-to-start) section of the [CommsChampion Ecosystem](https://commschamp.github.io) page. # Licensing All the available components of the [CommsChampion Ecosystem](https://commschamp.github.io) are free and open source, each one has a separate license. Please refer to the [LICENSES](https://commschamp.github.io/licenses/) page for the details.

arobenko

Implement Binary Communication Protocols in C++

Join Our Newsletter. No Spam, Only the good stuff.