Building a HTTP Server from scratch: Understanding Request & Response

Oh, hey there!

I'm glad you made it to this second post of the "Build the System: HTTP server" series.
This post is dedicated to decoding HTTP requests and encoding the response. I will also, offer a reliable way to test
our code for a more resilient project.
If you haven't read the first post of the series yet, I think you might want to. Just click here to read it.
I'll wait patiently for your return.

This article is a transcript of a Youtube video I made.

Alright, now that I know we're all on the same page, let's write some code.
For this project, I will use JavaScript and Deno, but the concepts don't change no matter what language or runtime you
are using.
Also one last disclaimer: this project first aim is to educate it will in no way be complete or the most performant!
I will discuss specifically the improvements we can bring to make it more performant and I will go through various
iteration with that in mind. At the end of the project, if there are part worth salvaging, I will replace the essential
parts.
All that to say, just enjoy the ride.

The first thing that I need to do is to announce listening on a port.
The incoming connection will be represented by a Readable/Writable resource.
First, I will need to read from the resource a specific amount of bytes. For this example, I will read around a KB.
The variable xs is a Uint8Array. I already wrote an article about this but long story short, a Typed Array is an array
that can only hold a specific amount of bit per item. In this case we need 8 bits (or one byte) array because you need 8 bits
to encode a single UTF-8 character.

🐙 You will find the code for this post here: https://github.com/i-y-land/HTTP/tree/episode/02

As a convenience, I will decode the bytes to a string and log the result to the console.
Finally, I will encode a response and write it to the resource.

// scratch.js
for await (const connection of Deno.listen({ port: 8080 })) {
  const xs = new Uint8Array(1024);
  await Deno.read(connection.rid, xs);

  console.log(new TextDecoder().decode(xs));

  await Deno.write(
    connection.rid,
    new TextEncoder().encode(
      `HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`
    )
  );
}

Now, I will run the code:

deno run --allow-net="0.0.0.0:8080" scratch.js

On a different terminal session I can use curl to send an HTTP request.

curl localhost:8080

On the server's terminal, we can see the request, and on the client's terminal we can see the response's body:
"Hello, World"

Great!

To get this started on the right foot, I will refactor the code into a function named serve in a file called
server.js. This function will take a listener and a function that takes a Uint8Array and returns a Promise of a
Uint8Array!

// library/server.js
export const serve = async (listener, f) => {
  for await (const connection of listener) {
    const xs = new Uint8Array(1024);
    const n = await Deno.read(connection.rid, xs);

    const ys = await f(xs.subarray(0, n));
    await Deno.write(connection.rid, ys);
  }
};

Notice that the read function returns the number of byte that was read. So we can use the subarray method to pass
a lense on the appropriate sequence to the function.

// cli.js
import { serve } from "./server.js";

const $decoder = new TextDecoder();
const decode = $decoder.decode.bind($decoder);
const $encoder = new TextEncoder();
const encode = $encoder.encode.bind($encoder);

if (import.meta.main) {
  const port = Number(Deno.args[0]) || 8080;
  serve(
    Deno.listen({ port }),
    (xs) => {
      const request = decode(xs);
      const [requestLine, ...lines] = request.split("\r\n");
      const [method, path] = requestLine.split(" ");
      const separatorIndex = lines.findIndex((l) => l === "");
      const headers = lines
        .slice(0, separatorIndex)
        .map((l) => l.split(": "))
        .reduce(
          (hs, [key, value]) =>
            Object.defineProperty(
              hs,
              key.toLowerCase(),
              { enumerable: true, value, writable: false },
            ),
          {},
        );

      if (method === "GET" && path === "/") {
        if (
          headers.accept.includes("*/*") ||
          headers.accept.includes("plain/text")
        ) {
          return encode(
            `HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`,
          );
        } else {
          return encode(
            `HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n`,
          );
        }
      }

      return encode(
        `HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n`,
      );
    },
  )
    .catch((e) => console.error(e));
}

Now that I have an way to parse the headers, I think it's a good opportunity to officialize all of this and write a new
utility function and the appropriate tests.

// library/utilities.js

export const parseRequest = (xs) => {
  const request = decode(xs);
  const [h, body] = request.split("\r\n\r\n");
  const [requestLine, ...ls] = h.split("\r\n");
  const [method, path] = requestLine.split(" ");
  const headers = ls
    .map((l) => l.split(": "))
    .reduce(
      (hs, [key, value]) =>
        Object.defineProperty(
          hs,
          key.toLowerCase(),
          { enumerable: true, value, writable: false },
        ),
      {},
    );

  return { method, path, headers, body };
};

// library/utilities_test.js

Deno.test(
  "parseRequest",
  () => {
    const request = parseRequest(
      encode(`GET / HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n`),
    );

    assertEquals(request.method, "GET");
    assertEquals(request.path, "/");
    assertEquals(request.headers.host, "localhost:8080");
    assertEquals(request.headers.accept, "*/*");
  },
);

Deno.test(
  "parseRequest: with body",
  () => {
    const request = parseRequest(
      encode(
        `POST /users HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n{"fullName":"John Doe"}`,
      ),
    );

    assertEquals(request.method, "POST");
    assertEquals(request.path, "/users");
    assertEquals(request.headers.host, "localhost:8080");
    assertEquals(request.headers.accept, "*/*");
    assertEquals(request.body, `{"fullName":"John Doe"}`);
  },
);

Now that I have a parseRequest function, logically I need a new function to stringify the response...

// library/utilities.js

import { statusCodes } from "./status-codes.js";

export const normalizeHeaderKey = (key) =>
  key.replaceAll(/(?<=^|-)[a-z]/g, (x) => x.toUpperCase());

export const stringifyHeaders = (headers = {}) =>
  Object.entries(headers)
    .reduce(
      (hs, [key, value]) => `${hs}\r\n${normalizeHeaderKey(key)}: ${value}`,
      "",
    );

export const stringifyResponse = (response) =>
  `HTTP/1.1 ${statusCodes[response.statusCode]}${
    stringifyHeaders(response.headers)
  }\r\n\r\n${response.body || ""}`;

// library/utilities_test.js

Deno.test(
  "normalizeHeaderKey",
  () => {
    assertEquals(normalizeHeaderKey("link"), "Link");
    assertEquals(normalizeHeaderKey("Location"), "Location");
    assertEquals(normalizeHeaderKey("content-type"), "Content-Type");
    assertEquals(normalizeHeaderKey("cache-Control"), "Cache-Control");
  },
);

Deno.test(
  "stringifyResponse",
  () => {
    const body = JSON.stringify({ fullName: "John Doe" });
    const response = {
      body,
      headers: {
        ["content-type"]: "application/json",
        ["content-length"]: body.length,
      },
      statusCode: 200,
    };
    const r = stringifyResponse(response);

    assertEquals(
      r,
      `HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
    );
  },
);

So now, we have everything we need to refactor our handler function and make it more concise and declarative.

import { serve } from "./library/server.js";
import {
  encode,
  parseRequest,
  stringifyResponse,
} from "./library/utilities.js";

if (import.meta.main) {
  const port = Number(Deno.args[0]) || 8080;
  serve(
    Deno.listen({ port }),
    (xs) => {
      const request = parseRequest(xs);

      if (request.method === "GET" && request.path === "/") {
        if (
          request.headers.accept.includes("*/*") ||
          request.headers.accept.includes("plain/text")
        ) {
          return Promise.resolve(
            encode(
              stringifyResponse({
                body: "Hello, World",
                headers: {
                  "content-length": 12,
                  "content-type": "text/plain",
                },
                statusCode: 200,
              }),
            ),
          );
        } else {
          return Promise.resolve(
            encode(stringifyResponse({ statusCode: 204 })),
          );
        }
      }

      return Promise.resolve(
        encode(
          stringifyResponse({
            headers: {
              "content-length": 0,
            },
            statusCode: 404,
          }),
        ),
      );
    },
  )
    .catch((e) => console.error(e));
}

So at this we can deal with any simple request effectively. To wrap this up and prepare the project for future iteration,
I will add a test for the serve function. Obviously, this function is impossible to keep pure and to test without
complex integration tests -- which I keep for later.
An actual connection is a bit figety so I thought I could mock it using a file as the resource since files are
readable/wriatable.
The first thing I did is to write a function to factorize an async iterator and purposely make it break after the first
iteration.
After that, I create a file with read/write permissions. With that, I can write the HTTP request, then move the cursor
back to the beginning of the file for the serve function to read back. Within the handler function, I make some
assertions on the request for sanity's sake, then flush the content and move the cursor back to the beginning before
writing a response.
Finally, I can move the cursor back to the beginning one last time, to read the response, make one last assertion then
cleanup to complete the test.

// library/server_test.js

import { assertEquals } from "https://deno.land/std@0.97.0/testing/asserts.ts";

import { decode, encode, parseRequest } from "./utilities.js";
import { serve } from "./server.js";

const factorizeConnectionMock = (p) => {
  let i = 0;

  return {
    p,
    rid: p.rid,
    [Symbol.asyncIterator]() {
      return {
        next() {
          if (i > 0) {
            return Promise.resolve({ done: true });
          }
          i++;
          return Promise.resolve({ value: p, done: false });
        },
        values: null,
      };
    },
  };
};

Deno.test(
  "serve",
  async () => {
    const r = await Deno.open(`${Deno.cwd()}/.buffer`, {
      create: true,
      read: true,
      write: true,
    });

    const xs = encode(`GET /users/1 HTTP/1.1\r\nAccept: */*\r\n\r\n`);

    await Deno.write(r.rid, xs);

    await Deno.seek(r.rid, 0, Deno.SeekMode.Start);

    const connectionMock = await factorizeConnectionMock(r);

    await serve(
      connectionMock,
      async (ys) => {
        const request = parseRequest(ys);

        assertEquals(
          request.method,
          "GET",
          `The request method was expected to be \`GET\`. Got \`${request.method}\``,
        );
        assertEquals(
          request.path,
          "/users/1",
          `The request path was expected to be \`/users/1\`. Got \`${request.path}\``,
        );
        assertEquals(
          request.headers.accept,
          "*/*",
        );

        await Deno.ftruncate(r.rid, 0);
        await Deno.seek(r.rid, 0, Deno.SeekMode.Start);

        const body = JSON.stringify({ "fullName": "John Doe" });

        return encode(
          `HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: ${body.length}\r\n\r\n${body}`,
        );
      },
    );

    await Deno.seek(r.rid, 0, Deno.SeekMode.Start);

    const zs = new Uint8Array(1024);
    const n = await Deno.read(r.rid, zs);

    assertEquals(
      decode(zs.subarray(0, n)),
      `HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
    );

    Deno.remove(`${Deno.cwd()}/.buffer`);
    Deno.close(r.rid);
  },
);

At this point we have a good base to work from. Unfortunately our server is a bit limitted, for example, if a request
is larger than a KB, we'd be missing part of the message, that means no upload or download of medium size files.
That's what I plan to cover on the next post. This will force us to be a little bit more familiar with
manipulation of binary bytes.

At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!

Ok bye now...

Blog

Building a HTTP Server from scratch: Understanding Request & Response

Sebastien Filion

Join Our Newsletter. No Spam, Only the good stuff.

Related