This is the first "technical" post about flyxc. I'm not going to describe all the fine details of the architecture but a simplified view on how flyxc works.

Yesterday I wrote some code to reduce the amount of data exchanged between the server and the web app. The server is a "computer" located in a data center somewhere in the US. It fetches the pilot positions from the different live tracking platforms (InReach, XContest live, flymaster, ...). The web app queries the server periodically to display the live tracks on the map. It runs in your browser.

We'll start by looking at some concepts before describing the code update.

Data format

Most live tracking apps/sites use JSON to send the positions from the server to the app/site. It looks like the following:

{
    "devices": [
        {
            "DeviceID": "0-2446---",
            "PilotID": "---",
            "Name": "Some Pilot",
            "Num": "",
            "Driver": "",
            "Messages": [
                {
                    "Timestamp": "1715450483",
                    "Type": "0",
                    "Lat": "34.792191",
                    "Lng": "-94.903687",
                    "Alt": "638",
                    "Elev": "9",
                    "Battery": "2",
                    "MsgText": ""
                },
                {
                    "Timestamp": "1715450739",
                    "Type": "0",
                    "Lat": "34.792469",
                    "Lng": "-94.903633",
                    "Alt": "635",
                    "Elev": "11.6",
                    "Battery": "2",
                    "MsgText": ""
                }, ...

One great thing with JSON is that the messages are "human readable". It's easy to peek at the message above (it's an actual message seen in the wild) and see that it is composed of the track points of Some Pilot.

But computers are not (yet) humans. There are much better at handling numbers than strings. Both the server and the app works with a binary format ("numbers" in the previous sentence).

So the server has to do extra work to convert the internal binary to JSON. In a similar way, the web app needs to convert the JSON back to a binary format. Those conversion incurs $ expenses and latency.

flyxc messages are based on protocol buffers ("protobuf" if you want to sound cool). They are not human readable but much more compact and faster for computers to work with:

Protobufs can still be converted to a human readable format when there is a need (i.e. to investigate a bug).

Let's compare the size needed to send a point for both the formats:

JSON

"Lat":"34.792191",
"Lng":"-94.903687",
"Alt":"638",

Protobuf

0NB32
1NB32
2NB32

Any character you see in the messages above is 1 Byte. The leading 0, 1, and 2 of the protobuf represent "Lat", "Lon", and "Alt" - remember computers are good at numbers. NB32 is a placeholder for a number encoded on 32 bits (= 4 bytes/characters).

The JSON message is 49 bytes vs 15 bytes for the protobuf. The protobuf is over 3x more compact - the ratio is actually higher when you consider all the protobuf features.

Incremental update

flyxc tracks pilots over the last 24h and fetches new positions every minutes.

A naive implementation (seen in the wild) would fetch the last 24h of data every minutes. Flyxc only fetches 1h worth of data every minutes because former positions never changes. That's 20x less data than the naive implementation - an easy win.

The recent optimization

You might wonder why flyxc fetches 1h worth of data every minutes. Fetching 1 minute worth of data (2min if you are conservative) should be enough, right?

There are actually a few good reasons why we need to fetch more. Let's look at a couple.

Latency

Let's say your InReach records your position at 1PM. It then needs to send it to the Garmin server via satellite. The garmin server will insert the position in a database. From there, the flyxc server will then read that position and insert it in a database before it can be transmitted to the web app. Latencies add up and are somehow unpredictable. But it is likely than the position will not be available to the web app before 1:03PM.

If the web app fetches 2 minutes worth of data at 1:03PM it will not get your position because it's already 3 min old.

History

When I (re-)added live tracking support in flyxc about 2.5 years ago, only InReach and Spot were supported. Those devices send a position every 10 minutes and there was only ~50 registered pilots. It explains why 1 hour was a sensible value at that time. Even if half of the pilots were flying on a given day, 1h worth of data is small enough.

Fast forward to May 2024. There are now over 1500 registered users and about 500 users using a high resolution tracking platform. XContest Live, OGN, Skylines, and Flymaster support a resolution of up to 1 track point every 5s. It's not unusual to have 150 pilots tracked on flyxc on good days.

The number of points in a 1h incremental update is in the order of 100 [pilots] * 3600 [1h in seconds] / 5 [points per sec] = 72000. That's why I reduced the incremental update to cover only 5 minutes, that's 3600 / (5 * 60) = 12 times less data.

Why it matters

On the server side, sending less data means less processing time and a decreased bandwidth usage. Thanks to that a single server is able to accommodate more devices and there should be no need to upgrade the server anytime soon, keeping the cost lost.

On the web app side, receiving less data means also means less processing time which translates to reduce battery usage and a more snappy app. One other benefit is that flyxc will work better on slow/spotty cell networks we often encounter in remote area.

Conclusion

Small updates can sometimes make a big difference. That's why we need to revisit some design decisions as flyxc evolves.

There are many things that could be optimized in flyxc but we have to balance the benefit vs the the time it takes to implement. I prefer adding functionalities rather than spending time optimizing things that wouldn't make much difference.

When offline maps will come to flyxc, the app will consume very little data and will work great even with a poor connection.

Let me know if you like reading about the internals of flyxc and if you have any particular topic of interest or if you don't care and prefer to see me spending time improving flyxc.

Happy flying!