When in startup mode, a very important thing is to be flexible. Your product isn’t defined well-enough yet and it can change frequently and drastically. We knew that and decided to use Python – it’s simple and allows very fast development.

Business grew and we started having clients. With every new client, our understanding of the product would expand. Some features turned out to be more relevant than others, as they produced more value for clients. We started to understand what was our core product and began cutting irrelevant functionalities out.

Changing focus

Next step was doing what we do – just better. How can we reduce our costs but deliver the same amount of value (or more). There are 3 components from the development aspect I think are relevant:

• cost of development which is most influenced by how hard it is to create a product using the platform
• cost of deployment defined by the cost to run the whole product. This relates to the code performance and technologies used,
• cost of maintenance depends on how much expertise does someone need to have in order to maintain a product on the platform.

Python is great, but it has its limits. It is designed to be easy-to-use, while languages like C, Java, and Go are designed to be performant.

Between those languages, Go is the new and cool kid and is loved by some of my colleagues, so we decided to check it out. The hypothesis was that it should easily beat Python performance-wise.

Research: comparing Python & Go performance

We’ve taken one service called Gatekeeper inside our platform which was our biggest bottleneck. It communicates with MongoDB and Kafka a lot, and those IO operations turned out to take the longest. That is the part we wanted to improve most.

Due to the nature of our product, the test was focused more on increasing throughput, with latency reduction being less important. We’ve identified 5 scenarios on Gatekeeper that should be tested. In Python, we implemented a Sync and Async version. The Async one used asyncio – Python’s newest library that allows implementing coroutines. Also, Go was limited to a single core to reproduce our production setup.

The results:

Go turned out to be a clear winner in all 5 scenarios. When considering the frequency of each scenario happening in real life, the expected throughput increase is around 450%. That is great news for our cost of deployment.

Is it worth switching?

For me, the formula to calculate if it’s worth switching is this:

y = A × x_dev + B × y_depl + C × z_main

x_dev, y_depl & z_main are values representing the impact of a solution to each cost component. A, B & C are weights – how important is it to reduce each cost. y is the end result, how much does a solution cost considering our priorities. We’re trying to keep that as low as possible.

For example, at the beginning, we didn’t care that much about the cost of deployment and maintenance, so our formula could look like this:

y = 0.75 × x_dev + 0.1 × y_depl + 0.15 × z_main

We could really focus on reducing the cost of development, not really caring about the other two.

Right now, I’d say things have balanced out a lot. The formula looks much more like this:

y = 0.50 × x_dev + 0.25 × y_depl + 0.25 × z_main

Our main driver is still the cost of development, but the other two have increased significantly relative to it. That gives us a lot of incentive to consider changing things up.

What are our options?

We could rewrite everything to Go. Unfortunately, it’s a bit unrealistic. It would take too long to learn Go properly and do the rewrite. It would also negatively affect our productivity. Our cost of development and cost of maintenance would skyrocket.

We could stay all the way in Python. We would keep cost of development and maintenance low, but it keeps cost of deployment too high.

The final option we could try is introducing Go where it matters most, while keeping the minimum impact on developers using the platform. Our micro-services architecture allows us to do just a partial rewrite in another language. We could try rewriting only the components that handle internal stuff not visible by developers. That should keep cost of development somewhat low, but cost of maintenance for the platform just increased, since now we have to maintain 2 languages.

It’s never black or white

We will try to do a partial rewrite. If things go great, we could gradually switch to Go. There could be more suitable languages or implementations that would help even more, but we have to move forward. We can only guess how things will turn out. The best we can do is making reality checks often to learn if we made the right choice early (and pivot if we didn’t).

This post was originally posted on my blog. Consider subscribing 😊