Zipadeedoodah šŸ¤ - Download Multiple Files To Zip On Client Browser

cmcnicholas

Craig McNicholas

Posted on November 14, 2024

Zipadeedoodah šŸ¤ - Download Multiple Files To Zip On Client Browser

If you've ever had to set up a backend API just to zip and download files (Google Drive, OneDrive etc.), you know it can be a pain in the nacellesā€”and it costs you additional infrastructure. This article shows you a simpler way - how to let your users zip files right in their browser. No backend required. šŸŒ•šŸŽļø "moon lambo" your infrastructure cost saving.


Why did I write this? We're developing a new product over at Vismo that deals with a lot of files and a subset of users would like "Download as Zip" functionality for their GeoSpatial data. We didn't want to stand up more infrastructure to support this so we have an effective solution for our customer base. Be sure to check your users and especially browser support


A Traditional Approach

Let's say you've got a bunch of files sitting in an S3 bucket in AWS. Normally, to zip these files for download, you would:

  1. Fetch the file records from a database.
  2. Read them from S3.
  3. Zip them up on the backend.
  4. Stream the zipped file back to S3.
  5. Expose a pre-signed URL (or similar) to the user to download.

All that works but hereā€™s the thing: every time someone requests a zip, you pay for compute and you pay for data egress when it leaves AWS. You can of course just try to download files individually and struggle with the browser blocking repeated downloads or a users download folder with 100's of files in it.

With client-side zipping, we can change all that. By handling the zipping right in the user's browser, we keep things simple(ish), cheaper, and easier to show progress to the user directly in page. Let's get into the pros and cons.


Pros of Client-Side Zipping

  • No Additional Infrastructure Cost ā€“ Skip the long-lived backend API and save on compute resources.
  • No Zip Hosting Cost ā€“ Since thereā€™s no server-side zipping, you donā€™t have to worry about hosting temporary zip files.
  • Better User Feedback ā€“ Show the download progress directly in the browserā€”great for the user experience.

Cons of Client-Side Zipping

  • Browser Compatibility ā€“ This approach relies on the File System Access API, which only works in modern browsers.
  • No Compression ā€“ Files download in their original size unless you've already got them compressed on the server.
  • File Availability ā€“ You need a way to serve the files securely to the browser, like with presigned URLs or similar.

How It Works

In a compatible browser, we can progressively enhance our app to zip and download files directly on the client. Hereā€™s the idea:

  1. Request Files Individually ā€“ The client requests each file directly.
  2. Stream Files to Zip Archive ā€“ Using the File System Access API, the browser writes the files into a zip archive on the user's device.
  3. No Storage/Infra Cost ā€“ Since the zipping happens locally, you're not paying to store the zip files or the compute required to support this.

For cases where data transfer is free (there are providers other than AWS out there, shocking I know), this method is a big win. No backend processing, no zip storage, no hassle. Itā€™s fast, itā€™s efficient, and your users get their files in one clean download.

So let's dive in to what this looks like.


Tooling Up: Zip.js

To make this all work smoothly, weā€™re going to use one package: the excellent Zip.js. Why Zip.js? Zip.js can handle a response buffer in chunks, flushing data to disk periodically as it streams. This keeps memory consumption low because it doesn't try to load the entire file into memory before writing. Some libraries do that, if you're working with large files it can crash your browser tab as you overconsume.

With Zip.js, though, we get efficient zipping that plays nice with the browser.


Step 1: Getting a Writable File Handle

First we need to get a writable file handle from the browser to save our zip content. To do this, weā€™ll use showSaveFilePicker. Now, fair warning: showSaveFilePicker doesnā€™t have built-in typing in TypeScript yet. So, weā€™ll get a little creative and opt out of type safety for this part.

To check if a browser supports this, use the following compatibility check:

const compatible = typeof window['showSaveFilePicker'] === 'function';
Enter fullscreen mode Exit fullscreen mode

If compatible is true, youā€™re good to go! If not, youā€™ll want to fall back to a more basic download approach, where files are downloaded individually.

This check lets you provide client-side zipping as an optional, progressive enhancement for browsers that support it. For those that donā€™t, users still get their files, just without the one-click zip download.

// showSaveFilePicker is not part of the standard types yet 
// (hence the comment of using a modern browser)
const showSaveFilePicker = (window as any).showSaveFilePicker;

// get the file handle for the zip file
const handle: FileSystemFileHandle = await showSaveFilePicker({
  suggestedName: 'download.zip',
  startIn: 'downloads',
  types: [{
    description: 'ZIP Files',
    accept: { 'application/zip': ['.zip'] },
  }],
});

// create a writable stream to the file handle
const writable: FileSystemWritableFileStream = await handle.createWritable({
  keepExistingData: false,
});
Enter fullscreen mode Exit fullscreen mode

Step 2: Setting Up the Zip Archive

Now letā€™s set up the zip archive. Weā€™re going to configure it for low memory use and fast processing. Hereā€™s how it works.

  1. Set Chunk Size ā€“ The chunkSize option determines how much data is downloaded and processed at once. Smaller chunks take a bit longer to download, but if a chunk fails, the smaller size makes it easier to retry without starting over.

  2. Configure Compression ā€“ Weā€™ll set compression to zero and enable passThrough: true (also known as STORE mode) on the zip instance. This disables compression. Why? Because compressing files uses a lot of memory and we don't want to keep large chunks of the file in memory to avoid crashing the page, we want to offload and stream to disk quickly.

This setup means we can keep memory use low. Hereā€™s an example of what this looks like in code:

// configure the zip js library to use a chunk size of 5MB
configure({ chunkSize: 5000 * 1024 });

// create a new zip writer
const zipWriter = new ZipWriter(writable, {
  compressionMethod: 0,
  passThrough: true,
  bufferedWrite: true,
});
Enter fullscreen mode Exit fullscreen mode

Step 3: Download & Zip the Files

Now, letā€™s gather the files we need to download and zip. Youā€™ll typically fetch file metadata from your API e.g. file name, URL, and size. Your API should return the data in a format you can process like the example below:

const files = [
  {
    name: 'myfilebigfile.mp4',
    url: 'https://myproduct.com/path/to/myfilebigfile.mp4',
    size: 1073741824, // 1GB
  },
  {
    name: 'anotherfile.jpg',
    url: 'https://myproduct.com/path/to/anotherfile.jpg',
    size: 10485760, // 10MB
  }
];
Enter fullscreen mode Exit fullscreen mode

Adding Files to the Zip Archive

With an array of file models in hand, we can now loop through each file and add it to our zip archive. Zip.js makes this process efficient by handling each file as a stream and sending requests with Range headers. This avoids loading the entire file into memory.

Hereā€™s how the code looks:

for (const file of files) { 
  // Add each file to the zip archive
  await zipWriter.add(file.name, new HttpRangeReader(file.url), { 
    uncompressedSize: file.size, 
    async onprogress(progress, total) { 
      // Update the progress for the current file
      // (could be displayed in a progress bar)
      console.log('entry progress:', file.name, progress, total); 
    }, 
  }); 
}
Enter fullscreen mode Exit fullscreen mode

In this example, each add call waits until the previous file completes streaming otherwise we risk saturating the network (you could have a worker pool here to process multiple but that's beyond this article). Weā€™re also updating progress in real-timeā€”great for giving users feedback if you want to display progress bars or status indicators.

With this, your users can download a custom zip of files without touching a backend. Just efficient, on-the-fly zipping directly in the browser.


Step 4: Cleaning It All Up

To wrap things up, make sure you properly close the zip archive. This final step ensures all data is written correctly, producing a valid zip file for download.

Simply call:

await zipWriter.close();
Enter fullscreen mode Exit fullscreen mode

This command flushes the zip data to the file, ensuring there are no loose ends. Without it, the zip file might end up incomplete or corrupted, which isnā€™t ideal for your users!


The Full Code

If you're ready to see it all together, hereā€™s the complete code sample for client-side zipping:

import {
  HttpRangeReader,
  TextReader,
  ZipWriter,
  configure,
} from '@zip.js/zip.js';

// this is the file we want to download, it should be returned from your
// api/database or whatever you are using to store references to files
type DownloadFile = {
  url: string;
  name: string;
  size: number;
};

// configure the zip js library to use a chunk size of 5MB
// you can tweak this if you want, a larger size means more 
// risk of network failure and having to retry the chunk, 
// a smaller size means more chunks and longer time
configure({ chunkSize: 5000 * 1024 });

export async function downloadAsZip(files: DownloadFile[]) {
  // check if the browser supports `showSaveFilePicker`
  const compatible = typeof window['showSaveFilePicker'] === 'function';
  if (!compatible) {
    // handle your fallback here, e.g. window.open each file
    return;
  }

  // showSaveFilePicker is not part of the standard types yet 
  // (hence the comment of using a modern browser)
  const showSaveFilePicker = (window as any).showSaveFilePicker;

  // get the file handle for the zip file
  const handle: FileSystemFileHandle = await showSaveFilePicker({
    suggestedName: 'download.zip',
    startIn: 'downloads',
    types: [{
      description: 'ZIP Files',
      accept: { 'application/zip': ['.zip'] },
    }],
  });

  // create a writable stream to the file handle
  const writable: FileSystemWritableFileStream = await handle.createWritable({
    keepExistingData: false,
  });

  // create a new zip writer
  const zipWriter = new ZipWriter(writable, {
    compressionMethod: 0,
    passThrough: true,
    bufferedWrite: true,
  });

  for (const file of files) {
    // add each file to the zip
    await zipWriter.add(file.name, new HttpRangeReader(file.url), {
      uncompressedSize: file.size,
      async onprogress(progress, total) {
        // log the progress of the file download to the user 
        // e.g. in a progress bar
        console.log('entry progress:', file.name, progress, total);
      },
    });
  }

  // cleanup the zip
  await zipWriter.close();
}
Enter fullscreen mode Exit fullscreen mode
šŸ’– šŸ’Ŗ šŸ™… šŸš©
cmcnicholas
Craig McNicholas

Posted on November 14, 2024

Join Our Newsletter. No Spam, Only the good stuff.

Sign up to receive the latest update from our blog.

Related