Are you looking to evaluate the performance of your MQTT broker? Do you want to know the maximum number of messages that can be sent by the broker in a specific amount of time? If so, ThingsOn MQTT Bench might be the tool you need!

In this article, I will introduce ThingsOn MQTT Bench, a benchmark tool for MQTT brokers. I will explain why I created the tool and how it works.

Github repo: https://github.com/volkanalkilic/ThingsOn.MQTT.Bench

Why I Created ThingsOn MQTT Bench

As an IoT and messaging systems developer, I struggled to find a benchmark tool that met my needs when testing the performance of MQTT brokers for my IIoT Platform. Most tools I found were either too complicated or too basic, and lacked the level of customization and control I desired. This motivated me to create my own benchmark tool that is simple, lightweight, and flexible, allowing for easy adaptation to various scenarios and requirements. The end result is ThingsOn MQTT Bench.

How ThingsOn MQTT Bench Works

ThingsOn MQTT Bench is a command-line tool that measures the maximum number of messages that can be sent by an MQTT broker in a specific amount of time. The tool is built on top of the MQTTnet library, a high-performance .NET library for MQTT communication.

The heart of ThingsOn MQTT Bench is a simple C# code that sends a large number of messages to an MQTT broker and measures the time it takes to send them. The code creates a specified number of MQTT clients and generates a specified number of messages for each client.

The code then connects the clients to the broker, sends the messages, and disconnects the clients from the broker. While sending the messages, the code measures the time it takes to send each message and reports the progress in real-time.

After the messages are sent, the code calculates the number of messages sent, the total time taken to send the messages, the message throughput, the connect time, the disconnect time, the success rate, and the loss rate.

Here is a summary of the C# code used in ThingsOn MQTT Bench to measure the maximum number of messages that an MQTT broker can handle:

// Create MQTT clients
var clients = new List<IMqttClient>();
for (var i = 0; i < clientCount; i++) {
    var client = factory.CreateMqttClient();
    clients.Add(client);
}

// Connect to MQTT broker and measure connect time
var stopwatchConnect = new Stopwatch();
stopwatchConnect.Start();
var connectTasks = clients.Select(c => c.ConnectAsync(mqttOptions)).ToList();
await Task.WhenAll(connectTasks);
stopwatchConnect.Stop();
var elapsedConnect = stopwatchConnect.Elapsed;

// Generate messages
var messages = new List<MqttApplicationMessage>();
for (int i = 0; i < messageCount; i++) {
    var message = new MqttApplicationMessageBuilder()
        .WithTopic($"test/{i}")
        .WithPayload(new byte[messageSize])
        .WithQualityOfServiceLevel((MqttQualityOfServiceLevel) qos)
        .WithRetainFlag(retain)
        .Build();
       messages.Add(message);
}

After the clients are connected to the broker, the application generates messages that will be sent by each client during the benchmark. The messages are created using the MqttApplicationMessageBuilder class from the MQTTnet library. The builder allows us to specify the topic, payload, quality of service level, and retain flag of each message.

In this implementation, we create a list of messages with the number of messages specified in the configuration file. The payload of each message is a byte array with the size specified in the configuration file.

The next step is to send the messages to the MQTT broker. We use the PublishAsync method of the MQTTnet library to send each message to the broker. We create a list of tasks to send the messages using the SendMessagesAsync method:

// Send messages
    private var sendTasks = clients.Select(client => Task.Run(async () =>
    {
        var results = new List<MqttClientPublishResult>();
        var progressReport = 0;
        foreach (var message in messages)
        {
            var result = await client.PublishAsync(message);
            results.Add(result);
            var newProgressReport = results.Count * 100 / messages.Count;
            if (newProgressReport > progressReport)
            {
                progressReport = newProgressReport;
                Console.WriteLine($"Progress: {progressReport}%");
            }
        }

        return results;
    }})).ToList();

The SendMessagesAsync method sends a list of messages to an MQTT client and returns a list of results indicating whether each message was successfully sent.

We create a list of tasks to send messages for each client using the Task.Run method. The tasks are created using the SendMessagesAsync method, which sends a list of messages to an MQTT client.

In the SendMessagesAsync method, we iterate over the list of messages and call the PublishAsync method for each message. The method returns a MqttClientPublishResult that indicates whether the message was successfully sent to the MQTT broker.

We keep track of the progress of the benchmark by calculating the percentage of messages that have been sent and printing it to the console. We use the progressReport variable to keep track of the last reported progress percentage and only print a new progress message if the progress percentage has changed:

if (newProgressReport > progressReport)
{
    progressReport = newProgressReport;
    Console.WriteLine($"Progress: {progressReport}%");
}

Once all messages have been sent, the benchmark is complete, and we can calculate the results. We do this by measuring the time it took to send all the messages, the time it took to connect to and disconnect from the MQTT broker, and the number of messages that were successfully sent. We print the results to the console using the following code:

stopwatch.Stop();
var elapsed = stopwatch.Elapsed;
var messagesSent = clientCount * messageCount;
var throughput = messagesSent / elapsed.TotalSeconds;
var lossRate = (double) sendResults.SelectMany(r => r).Count(r => r.ReasonCode != MqttClientPublishReasonCode.Success) / messagesSent;
var successRate = 1.0 - lossRate;// Disconnect from MQTT broker and measure disconnect time
var stopwatchDisconnect = new Stopwatch();
stopwatchDisconnect.Start();var disconnectTasks = clients.Select(c => c.DisconnectAsync()).ToList();
await Task.WhenAll(disconnectTasks);stopwatchDisconnect.Stop();
var elapsedDisconnect = stopwatchDisconnect.Elapsed;

// Print results
Console.WriteLine($"Sent {messagesSent:N0} messages in {elapsed.TotalSeconds:F5} seconds. Throughput: {throughput:N1} messages/second.");
Console.WriteLine($"Connect time: {elapsedConnect.TotalSeconds:F5} seconds. Disconnect time: {elapsedDisconnect.TotalSeconds:F5} seconds.");
Console.WriteLine($"Success rate: {successRate:F3}. Loss rate: {lossRate:F3}.");

In the output, we print the following information:

• Number of messages sent

• Total time taken to send messages

• Message throughput (messages per second)

• Connect time (time taken to connect to the MQTT broker)

• Disconnect time (time taken to disconnect from the MQTT broker)

• Success rate (percentage of messages that were successfully sent)

• Loss rate (percentage of messages that were not successfully sent)

To generate the messages that will be sent to the MQTT broker, we create a list of MqttApplicationMessage objects using the specified messageCount and messageSize. Each message is given a unique topic that includes the message index (e.g. “test/0”, “test/1”, etc.), and the payload is a byte array of the specified size filled with zeros.

Once the messages have been generated, we create a list of tasks to send the messages using the SendMessagesAsync method. Each task is responsible for sending messages for a single client. The SendMessagesAsync method iterates over the messages and publishes each one using the client’s PublishAsync method. After all messages have been sent, the task returns a list of MqttClientPublishResult objects, which contains information about the publication of each message.

After all send tasks have completed, we calculate the benchmark results and print them to the console using the information that was collected during the benchmark.

Features

• Measures the maximum number of messages that can be sent to an MQTT broker in a specified amount of time.

• Supports MQTT brokers running MQTT protocol versions 3.1.1 (MQTTv3.1.1) and 5.0 (MQTTv5.0).

• Allows customization of the following benchmark settings:

• Number of MQTT clients to use for benchmarking

• Number of messages to send per client

• Message size in bytes

• Quality of Service (QoS) level for messages (0, 1, or 2)

• Whether messages should be retained by the broker

• Outputs progress information to the console during benchmarking.

• Outputs the following information upon benchmark completion:

• Number of messages sent

• Total time taken to send messages

• Message throughput (messages per second)

• Time taken to connect to the MQTT broker

• Time taken to disconnect from the MQTT broker

• Success rate (percentage of messages that were successfully sent)

• Loss rate (percentage of messages that were not successfully sent)

How to Use

To use ThingsOn MQTT Bench, first make sure that you have .NET 7 installed on your system. Then, download the latest release of ThingsOn MQTT Bench for your operating system from the releases page.

Once you have downloaded the tool, you can run it from the command line by navigating to the directory where the tool is located and running the following command:

dotnet ThingsOn.MQTT.Bench.dll

By default, the tool will read its settings from a TOML file named config.toml in the same directory as the tool. You can customize the settings by editing this file.

Contributing

Contributions to ThingsOn MQTT Bench are welcome! If you find a bug or would like to suggest a new feature, please open an issue on the GitHub repository.

If you would like to contribute code to ThingsOn MQTT Bench, please fork the repository and submit a pull request.

ThingsOn MQTT Bench is a powerful and customizable benchmark tool for MQTT brokers. It provides real-time progress and comprehensive performance metrics that give you a complete view of the performance of your MQTT broker. If you need to evaluate the performance of an MQTT broker, give ThingsOn MQTT Bench a try!