"Web is an I/O device"

...as Uncle Bob describes.

I/O, Input / Output, *i.e *user input and server output.

Users want to access some data through applications or websites.

They want it to be available throughout the world over.

If the requirements are so simple, why is backend development so challenging? That is a great question I want to hopefully answer in this post. But more than that, this post should be a grumbling space for hardworking backend developers.

Let's have a look at a simplistic diagram of a backend server.

Everything looks simple from the outside. But a lot is going on behind the scenes. In the end, it is called "backend" development for a reason.

The job of a backend developer is to make everything work seamlessly, at least from a user perspective.

So, here is my list of the top 5 struggles of backend developers.

Struggle 1: Crashing Servers!

Backend developers need to always remind themselves:

The server can go down at any point in time

It is not only the server, unhandled rejections, network problems, and countless other things can cause application crashes.

This is a list of fallacies of distributed systems:

• The network is reliable
• Latency is zero
• Bandwidth is infinite
• The network is secure
• Topology doesn't change
• There is one administrator
• Transport cost is zero
• The network is homogeneous

They are more than an inconvenience for backend development.

It is already hard to write correctly functioning programs. Now you also need to consider failure scenarios and make the application fault tolerant.

Struggle 2: Scalability

There are various aspects of scalability, but let's keep things simple for the sake of this post.

A scalable application can continue to work even if the load increases significantly.

Servers have limited capacity and they can handle only a limited amount of requests.

• If you add more resources to the server it can handle more requests ( vertical scaling ).
• If you add more servers to the system they can handle more requests ( horizontal scaling ).

Designing an application to be horizontally scalable, immediately changes the way things work.

As an example, most of the backend developers make race condition mistakes in scalable applications - *including myself. *In a scalable application, data can be read or modified by multiple servers. A typical example:

• Server A reads the data { foo: 1 }
• Server B reads the data { foo: 1 }
• Server A modifies the data and writes back: { foo: 1, bar: 2 }
• Server B modifies the data and writes back: { foo: 1, baz: 2  }

The result can be either { foo: 1, bar: 2 } or { foo: 1, baz: 2  } depending on which server writes back the data last. So, both servers are in a race condition!

This was just an example, but there are a lot of things to consider when developing a horizontally scalable application.

Struggle 3: Security

Remember, a public server is accessible to everyone on the internet.

And, there are a lot of bad actors out there 🥷.

Bad actors can craft special requests to extract information from a vulnerable server.

Go check out https://owasp.org/www-project-top-ten/ for what kind of sorcery those bad actors do to your servers.

Even legitimate users should be considered bad actors from a backend developer's perspective.

Legitimate users may abuse the service or send unexpected inputs to exploit the server's vulnerabilities. All user inputs should be sanitized to prevent unexpected inputs from legitimate users. Schema definitions, such as JSON Schema, are very helpful to sanitize user input.

Also, legitimate users are not allowed to do anything they want. The service needs to have a well-defined authorization system to prevent users from accessing forbidden resources.

The server might be vulnerable to a Denial of Service attack. Most DoS attacks can be prevented with rate limiting.

The list goes on, and the truth is no server can be 100% secure. Try to follow best practices as much as possible to secure your servers.

Struggle 4: Consistency

If a car is red, then it is red.

Other people should also tell you that car is red.

And, nobody should say that car isn't red.

Then tell me in the comments...
https://en.wikipedia.org/wiki/The_dress
...what is the color of this dress? Is it black & blue or white & gold? Whichever you choose, you would be surprised to see other people swear it is the other one!

You see, sometimes it is not very easy to come up with a single true answer to such a basic question.

In computer terms, this dress can be a document read from a cache, a slow replicate instance of a database, or an event-sourced database.

In all cases, the data might be obsolete. One server might read obsolete data while the other has access to a more recent version of the document.

Trade-off: Consistency and Performance

The problem with consistency is that retrieving the latest information may not be the optimum solution for most cases.

For example, if thousands of people try to access a popular tweet at the same time what do they see? The same tweet content, except the read counts, are probably way different from each other.

This is the result of a design decision. The engineers at Twitter decided to trade off between performance and consistency at the expense of showing obsolete read counts. This is a pretty good deal because the read counts are not a critical part of the tweets.

More About Consistency

Well, sometimes inconsistencies happen not because of design decisions but because of the way things work in software engineering.

Remember the previous example about the race condition? Well, how do you handle that? You might use locks but that can be costly ( in terms of time ). Some systems use optimistic locking to get better results. Some other systems use event sourcing and evade the problem completely.

Then there is the problem with client and backend consistency. The client database should be consistent with the actual data on the backend. Here is a nice article if you want to read more about consistency: https://medium.com/ssense-tech/handling-eventual-consistency-with-distributed-system-9235687ea5b3

Struggle 5: Performance

I think this one is pretty obvious. Everyone wants to develop blazingly fast backend applications.

But, that is not an easy task. From the language and framework selection to the algorithms used and the overall design of the system, almost everything has an impact on the performance.

I would say, the best course of action is:

1. Use a performance testing tool such as k6
2. Don't waste time ultra-optimizing the application if it meets the criteria

It is that simple...

...to give advice from a blog post 😅

In reality, measuring the application performance is somewhat complicated. Maybe you can easily measure the end-to-end performance of your endpoints etc.

But detecting the actual bottleneck in your application is hard without observability tools.

In RESTful backend applications, a for loop or a basic algorithm in the code is rarely the bottleneck. Optimizing the database queries, network configurations, or API design usually have greater benefits.

Summary

The struggle is real for the backend developers. Share your struggles in the comments and show others they are not alone!

I hope you enjoyed this post, subscribe for more like this.