Lessons from opensource: How to use pipeline in Nodejs?

These lessons are picked from next.js/create-next-app open source code. In this article, you will learn what a pipeline is and how it is used in create-next-app to download a Github repo as tarball.

What is a pipeline?

As you can see from the above image, pipeline is import from stream/promises. You might be wondering what is a stream? read further.

What is a stream?

A stream is an abstract interface for working with streaming data in Node.js. The node:stream module provides an API for implementing the stream interface. There are many stream objects provided by Node.js. For instance, a request to an HTTP server and process.stdout are both stream instances. Streams can be readable, writable, or both. All streams are instances of EventEmitter. - Source

When you make a request to an HTTP server and want to handle the response as a stream, it usually involves using the concept of streaming in the context of web requests and responses. Streaming is useful when dealing with large amounts of data or when you want to process data incrementally as it arrives, rather than waiting for the entire response to be received.

One of the usecases to use stream api is when you are downloading large files. downloadAndExtractRepo helper function in create-next-app source code uses pipeline from stream/promises API.

How to use a pipeline?

The above image shows how to use pipeline. It basically pipes the output stream into another. In this case, Github tarball is downloaded as a stream and is piped into next stream where tar.x is used to extract the downloaded tarball.

Refer to the official documentation for more information.

Conclusion:

Now you know how to handle downloading large files. You are looking to handle this via Stream API in NodeJs. The implementations you find in popular open source codebase can be safely replicated unless you know more than original authors about the code you are implementing.

If you are looking to improve/learn frontend, checkout my website: https://tthroo.com/ where I teach project based tutorials.