🏋🏽 Uploading Video and other Large File to S3 Compatible Storage

This post on SvelteKit S3 multipart upload follows on from the earlier post on uploading small files to S3 compatible storage. We will see how to upload large video files to cloud storage. In that earlier post we saw using an S3 compatible API (even while using Backblaze, Cloudflare R2, Supabase or another cloud storage provider) makes your code more flexible than using the provider's native API. We also saw the benefits of using presigned URLs for file upload and download. We level up the code from that tutorial here and introduce multipart uploads with presigned URLs. Sticking with an S3 compatible API, we will still leverage the flexibility benefits that brings. I hope you find this a useful and interesting extension to the previous tutorial.

⚙️ SvelteKit S3 Multipart Upload: Getting Started

Instead of building everything from scratch, we will use the previous tutorial on SvelteKit S3 Compatible Storage Uploads as a starting point. You can start here and check out the other tutorial another day, although multipart S3 uploads might make more sense if you start with the other tutorial. If you did work through the presigned URL upload tutorial, you can create a new branch in your repo and carry on from your existing code. Otherwise, clone the following repo to get going:

git clone https://github.com/rodneylab/sveltekit-s3-compatible-storage.git sveltekit-s3-multipart-upload
cd sveltekit-s3-multipart-upload
pnpm install

We won't need to add any extra packages beyond the ones we used last time.

🔨 Utility Functions

With multipart uploads, the presigned URL part works much as it did for a single upload. The workflow is a little different though. We will still keep the single file upload code and only use this when the file is small. With a multipart upload, we need to create a signed URL for each part we need to upload. Another difference is that once we have uploaded all of the parts to their respective URLs, we then we need to tell the provider we are done. This is so that they can combine the pieces at their end. For this to work we need to add a few more utility functions to our src/lib/utilities.js file. On top we will be restructuring our app slightly, so need to export some of the existing functions.

To get going let us import a few extra functions from the S3 SDK. Remember, although we are using the S3 SDK, we can expect our code to work with any S3 compatible provider (recalling only the initial authorisation step will vary from provider to provider).

import {
  CompleteMultipartUploadCommand,
  CreateMultipartUploadCommand,
  GetObjectCommand,
  PutObjectCommand,
  S3,
  UploadPartCommand,
} from '@aws-sdk/client-s3';

Continuing, in line 18, export the authoriseAccount function because we will want to access it from our SvelteKit endpoint:

export async function authoriseAccount() {

Multipart Upload Functions

Next we have to create the function which tells the provider we are done uploading. Add this code to the same file:

export async function completeMultipartUpload({ parts, client, key, uploadId }) {
  try {
    const { VersionId: id } = await client.send(
      new CompleteMultipartUploadCommand({
        Key: key,
        Bucket: S3_COMPATIBLE_BUCKET,
        MultipartUpload: { Parts: parts },
        UploadId: uploadId,
      }),
    );
    if (id) {
      return { successful: true, id };
    }
  } catch (error) {
    console.error('Error in completing multipart upload: ', error);
  }
  return { successful: false };
}

As with authoriseAccount, we will need to export getS3Client:

export function getS3Client({ s3ApiUrl }) {  `}

Next we want a function to generate presigned URLs. This works just like the function we had for single file upload presigned URLs:

export async function generatePresignedPartUrls({ client, key, uploadId, partCount }) {
  const signer = new S3RequestPresigner({ ...client.config });
  const createRequestPromises = [];

  for (let index = 0; index < partCount; index += 1) {
    createRequestPromises.push(
      createRequest(
        client,
        new UploadPartCommand({
          Key: key,
          Bucket: S3_COMPATIBLE_BUCKET,
          UploadId: uploadId,
          PartNumber: index + 1,
        }),
      ),
    );
  }

  const uploadPartRequestResults = await Promise.all(createRequestPromises);

  const presignPromises = [];
  uploadPartRequestResults.forEach((element) => presignPromises.push(signer.presign(element)));
  const presignPromiseResults = await Promise.all(presignPromises);
  return presignPromiseResults.map((element) => formatUrl(element));
}

Talking of the single upload, the generatePresignedUrls function needs exporting too:

export async function generatePresignedUrls({ key, s3ApiUrl }) {</CodeFragment>

Lastly, we will create a function to initiate a multipart upload using the S3 SDK:

export const initiateMultipartUpload = async ({ client, key }) => {
  const { UploadId: uploadId } = await client.send(
    new CreateMultipartUploadCommand({ Key: key, Bucket: S3_COMPATIBLE_BUCKET }),
  );
  return uploadId;
};

That was a lot of pasting! Do not worry if it is not 100% clear what we are doing yet, We will start to pull everything together in the next section where we call these functions from our endpoint.

📹 Multipart Presigned Upload Endpoint

You might remember from our SvelteKit frontend, we called an endpoint to tell us the presigned URL to upload the file to. Once we had that URL back, we proceeded with the upload directly from the frontend to the cloud provider. With multipart uploads, our ambition is again to upload directly from the frontend to our provider. For this to work we will change the logic in the endpoint.

We will pass the file size to the endpoint when we request the presigned upload URLs. Based on the file size, our logic will decide whether we will do a single file or multipart upload. When we create an S3 client object, we get back some parameters from the provider which give us minimum, maximum and recommended file part size. So to look at a concrete example. Let's say we want to upload a 16 MB video and the recommended part size is 5 MB. In this case we will need four parts: the first 3 parts will be 5 MB and the final one, 1 MB. Typically, the minimum part size is not enforced by the provider for the final part in a multipart upload.

Now we know what we are doing let's get coding!

SvelteKit S3 Multipart Upload: presigned-urls.json Endpoint Code

This is a substantial refactor on the previous code for the file at src/routes/api/presigned-urls.json:

import {
  authoriseAccount,
  generatePresignedPartUrls,
  getS3Client,
  initiateMultipartUpload,
  presignedUrls,
} from '$lib/utilities/storage';

export async function post({ body }) {
  const { key, size } = body;

  try {
    const { absoluteMinimumPartSize, recommendedPartSize, s3ApiUrl } = await authoriseAccount();
    if (s3ApiUrl) {
      const client = getS3Client({ s3ApiUrl });
      if (absoluteMinimumPartSize && size > absoluteMinimumPartSize) {
        const uploadId = await initiateMultipartUpload({ client, key });
        if (recommendedPartSize) {
          const partSize =
            size < recommendedPartSize ? absoluteMinimumPartSize : recommendedPartSize;
          const partCount = Math.ceil(size / partSize);
          if (uploadId) {
            const multipartUploadUrls = await generatePresignedPartUrls({
              client,
              key,
              uploadId,
              partCount,
            });

            const { readSignedUrl, writeSignedUrl } = await presignedUrls(key);

            return {
              body: JSON.stringify({
                multipartUploadUrls,
                partCount,
                partSize,
                readSignedUrl,
                writeSignedUrl,
                uploadId,
              }),
              status: 200,
              headers: {
                'Content-Type': 'application/json',
              },
            };
          }
        }
      }

      const { readSignedUrl, writeSignedUrl } = await presignedUrls(key);

      return {
        body: JSON.stringify({ partCount: 1, readSignedUrl, writeSignedUrl }),
        status: 200,
        headers: {
          'Content-Type': 'application/json',
        },
      };
    }
  } catch (error) {
    console.error(`Error in route api/presigned-urls.json: ${error}`);
  }
}

At the top of the file, you can see we now import the functions we have just exported from the utilities file. In line 13, we get the file size parameters we talked about. We use them in line 16 to work out if we will do a multipart upload or single. For a single upload we jump to line 50 and the code is not too different to what we had last time. We just add a partCount field in the response, to let the front end code know we only have one part (line 53).

For multipart uploads, we work out how big each of the parts is based on the recommendedPartSize provided by our authorisation response. Once we have that it is just a case of generating the presigned URLs and returning these to the frontend with some extra meta we will find handy.

🚚 Complete Multipart Upload Endpoint

Once the the parts have been uploaded, we need to let the provider know so they can piece the parts together. We will have a separate endpoint for this. Let's create the file now at src/routes/api/complete-multipart-upload.json.js, pasting in the content below:

import { authoriseAccount, completeMultipartUpload, getS3Client } from '$lib/utilities/storage';

export async function post({ body }) {
  const { key, parts, uploadId } = body;

  try {
    const { s3ApiUrl } = await authoriseAccount();
    if (s3ApiUrl) {
      const client = getS3Client({ s3ApiUrl });
      await completeMultipartUpload({ parts, client, key, uploadId });

      return {
        status: 200,
      };
    }
    return {
      body: JSON.stringify({ message: 'unauthorised' }),
      status: 400,
      headers: {
        'Content-Type': 'application/json',
      },
    };
  } catch (error) {
    console.error(`Error in route api/complete-multipart-upload.json: ${error}`);
  }
}

That's all the endpoint code in place now. Let's move on to the client page next.

🧑🏽 Client Homepage Svelte Code

There's not too much to change vs. the single file upload code. We'll start by adding a completeMultipartUpload function which calls that last endpoint we created. Add this block to src/routes/index.svelte:

  async function completeMultipartUpload({ key, parts, uploadId }) {
    try {
      const response = await fetch('/api/complete-multipart-upload.json', {
        method: 'POST',
        credentials: 'omit',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ key, parts, uploadId }),
      });
    } catch (error) {
      console.error(`Error in completeMultipartUpload on / route: ${error}`);
    }
  }

Handle Submit

Next we need to check in handleSubmit whether we have a single or multipart upload. If you are using this code in your own new project, you will probably want to refactor the block into separate functions, possibly in different files. Anyway, for now paste in this block:

  const handleSubmit = async () => {
    try {
      if (files.length === 0) {
        errors.files = 'Select a file to upload first';
        return;
      }

      isSubmitting = true;
      const { name: key, size, type } = files[0];

      // get signed upload URL
      const response = await fetch('/api/presigned-urls.json', {
        method: 'POST',
        credentials: 'omit',
        headers: {
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ key, size }),
      });
      const json = await response.json();
      const { multipartUploadUrls, partCount, partSize, readSignedUrl, writeSignedUrl, uploadId } =
        json;
      const reader = new FileReader();
      if (partCount === 1) {
        downloadUrl = readSignedUrl;

        // Upload (single part) file
        reader.onloadend = async () => {
          await fetch(writeSignedUrl, {
            method: 'PUT',
            body: reader.result,
            headers: {
              'Content-Type': type,
            },
          });
          uploadComplete = true;
          isSubmitting = false;
        };
        reader.readAsArrayBuffer(files[0]);
      } else {
        downloadUrl = readSignedUrl;
        const lastIndex = multipartUploadUrls.length - 1;

        // Upload (multipartpart) file
        reader.onloadend = async () => {
          const uploadPromises = multipartUploadUrls.map((element, index) =>
            fetch(element, {
              method: 'PUT',
              body:
                index !== lastIndex
                  ? reader.result.slice(index * partSize, (index + 1) * partSize)
                  : reader.result.slice(index * partSize),
              headers: {
                'Content-Type': type,
                'Content-Length': index !== lastIndex ? partSize : size - index * partSize,
              },
            }),
          );
          const uploadResults = await Promise.all(uploadPromises);
          const parts = uploadResults.map((element, index) => ({
            ETag: element.headers.get('etag'),
            PartNumber: index + 1,
          }));
          await completeMultipartUpload({ parts, key, uploadId });
          uploadComplete = true;
          isSubmitting = false;
        };
        reader.readAsArrayBuffer(files[0]);
      }
    } catch (error) {
      console.error(`Error in handleSubmit on / route: ${error}`);
    }
  };
</script>

Notice in line 49 we now get the file size, so we can pass that to the presigned URL endpoint. The value we have is in bytes. For single part uploads, nothing really changes. So let's jump to the reader.onloadend block for multipart uploads starting at line 85.

We use JavaScript's Promise API. That way we do not need to wait for one part to finish uploading before we start on the next one. This allows for faster uploads. For larger files, where there will be dozens of parts, it would make sense to extend this code to throttle the downloads so we only upload say three or four parts simultaneously and wait for one of those to finish before starting to upload a new part. We won't look at the detail of doing that here.

The code in lines 90–92 splits the file into chunks of the right size. We compute the part length and send it in the Content-Length header in line 95.

Multipart Upload Completion

When we complete the multipart upload, to help piece together the parts, we send an ID to identify each part. That ID comes in the form an ETag which is included in the multipart upload response header sent from our provider. We collate this data in lines 100–103 into the parts variable.

That parts object is passed to our completeMultipartUpload in this file and subsequently passed to the endpoint and the utility function.

Allowing Video Upload

The final change is to update the user interface to accept video as well as image files:

          <input
            id="file"
            aria-invalid={errors.files != null}
            aria-describedby={errors.files != null ? 'files-error' : null}
            type="file"
            multiple
            formenctype="multipart/form-data"
            accept="image/*,video/*"
            title="File"
            on:change={handleChange}
          />  

Remember you can change this to be more restrictive or, in fact, allow other types based your own needs.

⛔️ CORS Update

Because we want to look at a new header (the ETag header) from the client browser, we will need to update the bucket CORS policy. Check how to do this with your storage provider. If you are using Backblaze, you can update the backblaze-bucket-cors-rules.json file we introduced in the previous tutorial and submit this to Backblaze using the CLI.

[
  {
    "corsRuleName": "development",
    "allowedOrigins": ["https://test.localhost.com:3000"],
    "allowedHeaders": ["content-type", "range"],
    "allowedOperations": ["s3_put"],
    "exposeHeaders": ["etag", "x-amz-version-id"],
    "maxAgeSeconds": 300
  },
  {
    "corsRuleName": "production",
    "allowedOrigins": ["https://example.com"],
    "allowedHeaders": ["content-type", "range"],
    "allowedOperations": ["s3_put"],
    "exposeHeaders": ["etag", "x-amz-version-id"],
    "maxAgeSeconds": 3600
  }
]

🙌🏽 SvelteKit S3 Multipart Upload: What we Learned

In this post we looked at:

• how you can upload larger files to S3 compatible cloud storage,

• generating presigned URLs for multipart upload,

• how you can determine whether to use single or multipart upload and also calculate part size when choosing multipart upload.

I do hope there is at least one thing in this article which you can use in your work or a side project. As an extension you might consider throttling uploads, especially when uploading very large files with many parts. You can also extend the UI to show existing uploads in the bucket and even generate download presigned links with custom parameters, like link validity. On top consider adding code to abandon failed multipart uploads. This can potentially reduce costs.

You can see the full completed code for this tutorial on the Rodney Lab Git Hub repo.

🙏🏽 SvelteKit S3 Multipart Upload: Feedback

Have you found the post useful? Would you prefer to see posts on another topic instead? Get in touch with ideas for new posts. Also if you like my writing style, get in touch if I can write some posts for your company site on a consultancy basis. Read on to find ways to get in touch, further below. If you want to support posts similar to this one and can spare a few dollars, euros or pounds, please consider supporting me through Buy me a Coffee.

Finally, feel free to share the post on your social media accounts for all your followers who will find it useful. As well as leaving a comment below, you can get in touch via @askRodney on Twitter and also askRodney on Telegram. Also, see further ways to get in touch with Rodney Lab. I post regularly on SvelteKit as well as other topics. Also subscribe to the newsletter to keep up-to-date with our latest projects.