Mastering Node.js Streams and Pipelines

Streams are Node.js's superpower for handling large datasets efficiently. Let's dive into streams and pipelines.

Why Streams?

Memory efficiency
Time efficiency
Composability

Types of Streams

Readable
Writable
Duplex
Transform

Basic Stream Usage

const fs = require('fs');

const readStream = fs.createReadStream('big.file');
const writeStream = fs.createWriteStream('output.file');

readStream.on('data', (chunk) => {
  writeStream.write(chunk);
});

readStream.on('end', () => {
  writeStream.end();
});

Enter Pipelines

Pipelines simplify stream composition and error handling.

const { pipeline } = require('stream/promises');
const fs = require('fs');
const zlib = require('zlib');

async function compressFile(input, output) {
  await pipeline(
    fs.createReadStream(input),
    zlib.createGzip(),
    fs.createWriteStream(output)
  );
  console.log('Compression complete');
}

compressFile('big.file', 'big.file.gz').catch(console.error);

Custom Transform Streams

const { Transform } = require('stream');

const upperCaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

pipeline(
  process.stdin,
  upperCaseTransform,
  process.stdout
).catch(console.error);

Performance Tips

Use highWaterMark to control buffering
Implement stream.Readable.from() for async iterables
Leverage stream.finished() for cleanup

Common Pitfalls

Ignoring backpressure
Mishandling errors
Neglecting to end writable streams

Streams shine with large datasets or real-time data processing. Master them for scalable Node.js applications.

cheers🥂

Blog