I've recently decided to add protobufs to a project, and since HTTP v2 instead of HTTP v1 is used to send protobuf messages between applications I thought, "Let's learn http2 as well." It was the most frustrating learning process yet. Learning something new has levels to it. It's always useful to be able to reach out to someone for advice. When you can't, learning anything new in tech can either be a joy or drive you to tears.



You type into your browser "[whatever] tutorial," you click on the first result, you follow the steps and everything works straight away. Then you want to learn more things on the subject. You find the docs, everything is well documented, with examples. Working on this project is a breeze.



Found the tutorial, followed it to the letter, but a few small things are not 1:1 with your working environment. Operating System might be different, so the version of the tool you're using requires an extra couple of searches on how to make it work. Nevertheless, you get it to work.



The tutorial you've found that matches your working environment is maybe out of date. So, you have to make adjustments in the steps. Maybe the functions used in the tutorial code is deprecated and you have to find the newer function. Maybe the tutorial skipped a step or two and you have to figure out where a variable came from -- is it user-defined, is it built-in? You're scratching your head. You maybe search the web a few times. This is taking a little longer than expected, but you manage to build and run the initial code successfully.



This is where years of tinkering and experience with other languages/technologies might decide if this is going to take hours, resulting in your partner/kids/friends knocking on the door asking when you'll be joining them for that quality time you promised. The thing you want to learn has an active community, but not as big or as popular as others. Let's just say the thing you're learning is Erlang. You'll find a decent tutorial on sockets, the documentation gets you started, albeit a little lacking in examples and you have to "print out" some functions to see if it gives you the output you're expecting. You've now got a grasp of functional programming in the Erlang context, but now you're building an app with a supervisor/worker and you're casting and calling and still don't have the hang of where to pass the params to the handlers. You stumble along and you finally get the "hello world" sockets app running. You are doing simple client/server stuff, but still feel a sense of accomplishment.



From your years of HTTP v1 API-building experience you know simple concepts like GET requests and how to process it. It's plain text. What you see is what you get. Even a POST request with a file attached is not an issue. You recently heard about protobufs, and are curious to see how it can add value to your projects. You write your first protobuf file, build it, load it, process it, great. Then you start thinking about sending protobuf messages between applications. Google doesn't have official support for Erlang, but a couple of nice people have already done the work by creating compilers, clients and servers for that. You can just use that. Perfect. You're doing the lord's work now. But you're a curious little fella. You have just started using, not just one, but two technologies you have never used before. So, you set out to understand the first one: HTTP 2.

When a server gets a request from a client using HTTP 1, it establishes a new connection with the client, accepts the request, processes it, sends a response back to the client and closes the connection. HTTP 2, if you're familiar with WebSocket or database connections, can keep a connection open, so that the client doesn't open and close millions of connections to the server, and just sends data back and forth over one connection. But on top of that you're dealing with concepts like streams. Multiple streams. And you're also not dealing with plain, human-readable payloads anymore, it's encoded data called frames. At this point you get up to go take a walk. Preferably on the beach. You finally understand why people come to sit at the beach and just stare into the distance. They are making important life decisions. Like how much time you actually want to devote to reading through and deciphering the (specification) RFC on how HTTP 2 works.



It's questionable whether you actually got the gist from the RFC. You now go through the third-party libraries written for HTTP 2 communication. You kinda get the idea. With your socket code from before you whip out Postman and without really spending time with just HTTP 2, you move on to the second technology you haven't used before: gRPC. Google's framework for sending protobuf messages over HTTP 2. The Erlang code for the gRPC libraries is fairly straight-forward. They're not well documented, but you use your tinkering skills to see which functions are actually for processing incoming data. You manage to figure out the sequence in which to decode the incoming message. Now, you need to know not just what to send back, but in what order. The order for HTTP 1 is: Status, Headers, Payload; each separated by new lines so you know where what is. The headers tell you what the size/encoding/etc of the Payload is. Everything is clearly described and understood. The order and syntax for HTTP 2 is:

HEADERS (flags = END_HEADERS)
:status = 200
grpc-encoding = gzip
content-type = application/grpc+proto

DATA
<Length-Prefixed Message>

HEADERS (flags = END_STREAM, END_HEADERS)
grpc-status = 0 # OK
trace-proto-bin = jher831yy13JHy3hc

There are two headers. One for communicating in the HTTP 2 context and the other in the gRPC context. You send the payload to Postman. No dice. You look at how the third-party Erlang HTTP 2 libraries do it. You do it just like they do it. No dice. You end up going around in circles. Looking at the other official gRPC libraries for answers. You revisit the HTTP 2 RFC. Pointless. You finally think "Wait a minute. I'm sending data to my server. How do I see what the data should look like when sending back to Postman? Create a gRPC client and send it to an existing gRPC server." Brilliant. You'll figure this out in no time. But you don't. Days go by. You're taking walks on the beach, staring into the unknown.



You take a deep breath, you sit down. You ask yourself, "Did you really want to learn the inner workings of HTTP 2 and gRPC, or did you just want to brag about how you built your own version of a gRPC server 'from scratch' in Erlang?"

If you're teaching yourself, the key component is making sure you actually understand what you're trying to teach, and not just copying and pasting from a tutorial or someone else's code. For instance, while reading the HTTP 2 RFC, this diagram comes up:

It is supposed to be the layout of bits in a frame, but it doesn't make any sense visually and causes confusion when transferring it to code. What it is trying to convey is the order of bits from left to right sequentially:

So, with a revised outlook you start to Read the RFC. Once you understand the RFC, you start understanding the other developers' implementation of the protocol, you start understanding what your code should look like and you can start comparing it with their code and analysing the output of each to see how a request or a response should be composed. Read, write, test, adjust, until you not just get the desired outcome, but that you actually understand how it was achieved and why.

Conclusion

I don't have million dollar ideas, and I don't learn all this to build someone else's million dollar idea either. I learn it because I genuinely just have a fascination with making bits do my bidding. So whenever I feel like I'm going insane trying to figure something out or I want to throw my monitor into the wall, I just take a deep breath and remind my computer "You're just a machine. I push your buttons, not the other way around." But also, if things start feeling like it's too big or too much to digest, find the smallest possible starting point and try to understand just that. Then build on top of that.

I now know how HTTP v2 and gRPC works under the hood.