HTTP Request and Response streams. This is a story how I used postman to understand HTTP streams and their chunking.

First, the source code

// The source: https://nodejs.org/api/stream.html#api-for-stream-consumers
const http = require('http');

const server = http.createServer(
  function toUnderstandRequestAndResponseStreamRoleInDataFetching(req, res) {
    let body = '';
    let chunksReceived = 0;
    req.setEncoding('utf8');

    req.on('data', function saveChunkDataAndChunkCount(chunk) {
      body += chunk;
      chunksReceived += 1;
    });

    req.on('end', function sendReceivedChunkDataStatistic() {
      try {
        const chunkStatistic = JSON.stringify({
          chunkCount: chunksReceived,
          received: body,
        });

        res.writeHead(200, { 'Content-Type': 'application/json' });
        res.write(chunkStatistic);
        res.end();
      } catch (error) {
        // treat all errors as internal server error
        res.statusCode = 500;
        // let the user know what happened
        return res.end(error.message);
      }
    });
  }
);

const port = 3001;
server.listen(port);

Chunk collecting

The listener receives data in chunks. Because the server receives data in chunks, the data is split.

req.on('data', function saveChunkDataAndChunkCount(chunk) {
  body += chunk;
  chunksReceived += 1;
});

How large can each chunk be?

How I used postman

I opened postman and started sending get requests. I added more characters to each request. I printed each chunk and incremented chunk counter. I observed that it took 50kb or more to increase chunk count. I started sending requests with more characters. I observed that request with 88 chunks took 'a while'.

const chunkStatistic = JSON.stringify({
  chunkCount: chunksReceived,
  received: body,
});

All previous observations draw the following conclusions

• Request and Response are streams.
• Request is a readable stream.
• Response is a writable stream.

Reading and writing to Request and Response streams is kind of hidden from the user. But there is possibility to manually fetch and write data to streams. Manual control of reading and writing allows us to control the fetch priority.

References

• NextJS - Streaming SSR

I wanted to add a link to the document by Dan Abramov about the streaming SSR, but I couldn't find it. Still, I wanted to note that it exists.

Relevant reads

• MDN - Streams API
• MDN - Using readable streams
• MDN - Using writable streams
• MDN - ReadableStream
• MDN - Response
• MDN - WritableStream
• MDN - Streams API concepts