NestJS GraphQL image upload into a S3 bucket

Intro

In this tutorial I will explain how to upload images to a S3-compatible object storage (AWS S3, DigitalOcean Spaces, Linode Object Storage, etc.) bucket using NestJS and GraphQL for both Apollo and Mercurius drivers.

Common

Most of the set-up as well as the entirety of the upload process is the same for both drivers.

Set Up

Start by installing the class-transformer and class-validator packages for dto validation (see more in the docs), sharp for image optimization, and the S3 client packages.

Note: I assume you have UUID installed, however I still added it to the following command

$ yarn add class-transformer class-validator sharp @aws-sdk/client-s3 @aws-sdk/signature-v4-crt uuid
$ yarn add -D @types/sharp @types/uuid

Configuration

Assuming you are using nestjs ConfigModule start by creating an interfaces folder.

Bucket Data:

// bucked-data.interface.ts

export interface IBucketData {
  // name of the bucket
  name: string;
  // folder of the service
  folder: string;
  // uuid for the app
  appUuid: string;
  // the bucket url to create the key
  url: string;
}

Upload Middleware Options:

// uploader-middleware-options.interface.ts

export interface IUploaderMiddlewareOptions {
  maxFieldSize?: number;
  maxFileSize?: number;
  maxFiles?: number;
}

Uploader Options:

// uploader-options.interface.ts
import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';
import type { IUploaderMiddlewareOptions } from './uploader-middleware-options.interface';

export interface IUploaderOptions {
  clientConfig: S3ClientConfig;
  bucketData: IBucketData;
  middleware: IUploaderMiddlewareOptions;
}

On the config.ts function add the following:

import { IConfig } from './interfaces/config.interface';

export function config(): IConfig {
  const bucketBase = `${process.env.BUCKET_REGION}.${process.env.BUCKET_HOST}.com`;

  return {
    // ...
    uploader: {
      clientConfig: {
        forcePathStyle: false,
        region: process.env.BUCKET_REGION,
        endpoint: `https://${bucketBase}`,
        credentials: {
          accessKeyId: process.env.BUCKET_ACCESS_KEY,
          secretAccessKey: process.env.BUCKET_SECRET_KEY,
        },
      },
      bucketData: {
        name: process.env.BUCKET_NAME,
        folder: process.env.FILE_FOLDER,
        appUuid: process.env.SERVICE_ID,
        url: `https://${process.env.BUCKET_NAME}.${bucketBase}/`,
      },
      middleware: {
        maxFileSize: parseInt(process.env.MAX_FILE_SIZE, 10),
        maxFiles: parseInt(process.env.MAX_FILES, 10),
      },
    },
    // ...
  };
}

Module Creation

The uploader module set up changes depending if it is a library on a nestjs (or nx) mono-repo, or a monolith NestJS GraphQL API.

Monolith API

Create the uploader module and service:

$ nest g mo uploader
$ nest g s uploader

On the uploader.service.ts file add the S3 Client, the bucket data and the logger:

import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';

@Injectable()
export class UploaderService {
  private readonly client: S3Client;
  private readonly bucketData: IBucketData;
  private readonly loggerService: LoggerService;

  constructor(
    private readonly configService: ConfigService,
  ) {
    this.client = new S3Client(
      this.configService.get<S3ClientConfig>('uploader.clientConfig'),
    );
    this.bucketData = this.configService.get<IBucketData>('uploader.bucketData');
    this.loggerService = new Logger(UploaderService.name);
  }

  // ...
}

On the uploader.module.ts file add the global decorator:

import { Global, Module } from '@nestjs/common';
import { UploaderService } from './uploader.service';

@Global()
@Module({
  providers: [UploaderService],
  exports: [UploaderService],
})
export class UploaderModule {}

Library

Create the uploader library:

NestJS mono-repo:
```
$ nest g lib uploader
```

NX mono-repo:

$ npx nx @nrwl/nest:library uploader --global --service --strict --no-interactive

To be able to create a DynamicModule, we need to create a library options, start by creating an interfaces folder and move the bucket-data.interface.ts, and add an options.interface.ts:

import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';

export interface IOptions {
  clientConfig: S3ClientConfig;
  bucketData: IBucketData;
}

Create a new constants folder and add the options constant:

// options.constant.ts

export const UPLOADER_OPTIONS = 'UPLOADER_OPTIONS';

Unlike a normal module the class parameters come from the options passed to the module and not from the ConfigService:

import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IBucketData } from './interfaces/bucket-data.interface';
import { IOptions } from './interfaces/options.interface';

@Injectable()
export class UploaderService {
  private readonly client: S3Client;
  private readonly bucketData: IBucketData;
  private readonly loggerService: LoggerService;

  constructor(@Inject(UPLOADER_OPTIONS) options: IOptions) {
    this.client = new S3Client(options.clientConfig);
    this.bucketData = options.bucketData;
    this.loggerService = new Logger(UploaderService.name);
  }

  // ...
}

Turn the module into a dynamic global module:

import { DynamicModule, Global, Module } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IOptions } from './interfaces/options.interface';
import { UploaderService } from './uploader.service';

@Global()
@Module({})
export class UploaderModule {
  public static forRoot(options: IOptions): DynamicModule {
    return {
      global: true,
      module: UploaderModule,
      providers: [
        {
          provide: UPLOADER_OPTIONS,
          useValue: options,
        },
        UploaderService,
      ],
      exports: [UploaderService],
    };
  }
}

Image Uploading

Upload Scalar DTO

The graphql-upload package Upload scalar will be process into the following DTO, so add it into a dtos folder:

import { IsMimeType, IsString } from 'class-validator';
import { ReadStream } from 'fs';

export abstract class FileUploadDto {
  @IsString()
  public filename!: string;

  @IsString()
  @IsMimeType()
  public mimetype!: string;

  @IsString()
  public encoding!: string;

  public createReadStream: () => ReadStream;
}

File Validation

Since we want to upload only images the first thing we need to check is the mimetype:

//...

@Injectable()
export class UploaderService {
  // ...

  private static validateImage(mimetype: string): string | false {
    const val = mimetype.split('/');
    if (val[0] !== 'image') return false;

    return val[1] ?? false;
  }

  // ...
}

Stream Processing

As you saw the Upload scalar returns a read stream that we need to transform into a buffer:

//...
import { Readable } from 'stream';

@Injectable()
export class UploaderService {
  // ...

  private static async streamToBuffer(stream: Readable): Promise<Buffer> {
    const buffer: Uint8Array[] = [];

    return new Promise((resolve, reject) =>
      stream
        .on('error', (error) => reject(error))
        .on('data', (data) => buffer.push(data))
        .on('end', () => resolve(Buffer.concat(buffer))),
    );
  }

  // ...
}

Image Compression and Conversion

Before uploading the image we need to optimize it by compressing the image. Start by creating the constants necessary for compression, therefore on a constants folder add a file named uploader.constant.ts:

// The max width an image can have
// reducing the pixels will reduce the size of the image
export const MAX_WIDTH = 2160;

// an array with the percentage of quality ranging from 90 to 10%
export const QUALITY_ARRAY = [
  90, 80, 75, 70, 65, 60, 55, 50, 45, 40, 35, 30, 25, 20, 15, 10,
];

// Max image size, I recommend 250kb
export const IMAGE_SIZE = 256000;

I also recommend converting all images into a lighter format such as jpeg or webp, in this tutorial I will use jpeg:

// ...
import sharp from 'sharp';
import {
  IMAGE_SIZE,
  MAX_WIDTH,
  QUALITY_ARRAY,
} from './constants/uploader.constant';

@Injectable()
export class UploaderService {
  // ...

  private static async compressImage(
    buffer: Buffer,
    ratio?: number,
  ): Promise<Buffer> {
    let compressBuffer: sharp.Sharp | Buffer = sharp(buffer).jpeg({
      mozjpeg: true,
      chromaSubsampling: '4:4:4',
    });

    if (ratio) {
      compressBuffer.resize({
        width: MAX_WIDTH,
        height: Math.round(MAX_WIDTH * ratio),
        fit: 'cover',
      });
    }

    compressBuffer = await compressBuffer.toBuffer();

    if (compressBuffer.length > IMAGE_SIZE) {
      for (let i = 0; i < QUALITY_ARRAY.length; i++) {
        const quality = QUALITY_ARRAY[i];
        const smallerBuffer = await sharp(compressBuffer)
          .jpeg({
            quality,
            chromaSubsampling: '4:4:4',
          })
          .toBuffer();

        if (smallerBuffer.length <= IMAGE_SIZE || quality === 10) {
          compressBuffer = smallerBuffer;
          break;
        }
      }
    }

    return compressBuffer;
  }

  // ...
}

The ratio although not mandatory is a nice to have, so optionally you can create an enum with all common image ratios:

// ratio.enum.ts

export enum RatioEnum {
  SQUARE = 1, // 192 x 192
  MODERN = 9 / 16, // 1920 x 1080
  MODERN_PORTRAIT = 16 / 9 // 1080 x 1920
  OLD = 3 / 4, // 1400 x 1050
  OLD_PORTRAIT = 4 / 3 // 1050 x 1400
  BANNER = 8 / 47, // 1128 x 192
  ULTRA_WIDE = 9 / 21, // 2560 x 1080
  SUPER_WIDE = 9 / 32, // 3840 x 1080
}

This will add consistent with all image sizes on your application.

File upload

With the file buffer created, we just need to add a key to our file, and do a PutObjectCommand to our bucket:

import {
  // ...
  PutObjectCommand,
  // ...
} from '@aws-sdk/client-s3';
import {
  // ...
  InternalServerErrorException,
  // ...
} from '@nestjs/common';
import { v4 as uuidV4, v5 as uuidV5 } from 'uuid';
// ...

@Injectable()
export class UploaderService {
  // ...

  private async uploadFile(
    userId: number,
    fileBuffer: Buffer,
    fileExt: string,
  ): Promise<string> {
    const key =
      this.bucketData.folder +
      '/' +
      uuidV5(userId.toString(), this.bucketData.appUuid) +
      '/' +
      uuidV4() +
      fileExt;

    try {
      await this.client.send(
        new PutObjectCommand({
          Bucket: this.bucketData.name,
          Body: fileBuffer,
          Key: key,
          ACL: 'public-read',
        }),
      );
    } catch (error) {
      this.loggerService.error(error);
      throw new InternalServerErrorException('Error uploading file');
    }

    return this.bucketData.url + key;
  }

  // ...
}

I highly recommend using UUIDs both for the user ID and filename therefore if there is ever a breach on your bucket you are the only one that know which person owns which images.

I assumed that users are saved in a SQL database, if the user ID is not an int but a UUID or a MongoID, just change it to a string and it should work as expected.

Image upload

Putting all the previous private method together, we are able to create a public method that only accepts and uploads optimized images to our bucket:

// ...
import {
  BadRequestException,
  // ...
  InternalServerErrorException,
  // ...
} from '@nestjs/common';
import { RatioEnum } from './enums/ratio.enum';
// ...

@Injectable()
export class UploaderService {
  // ...

  /**
   * Upload Image
   *
   * Converts an image to jpeg and uploads it to the bucket
   */
  public async uploadImage(
    userId: number,
    file: Promise<FileUploadDto>,
    ratio?: RatioEnum,
  ): Promise<string> {
    const { mimetype, createReadStream } = await file;
    const imageType = UploaderService.validateImage(mimetype);

    if (!imageType) {
      throw new BadRequestException('Please upload a valid image');
    }

    try {
      return await this.uploadFile(
        userId,
        await UploaderService.compressImage(
          await UploaderService.streamToBuffer(createReadStream()),
          ratio,
        ),
        '.jpg',
      );
    } catch (error) {
      this.loggerService.error(error);
      throw new InternalServerErrorException('Error uploading image');
    }
  }

  // ...
}

File deletion

Files should be deleted asynchronously as it is often not a main event and can be done in the background:

import {
  DeleteObjectCommand,
  // ...
} from '@aws-sdk/client-s3';
// ...

@Injectable()
export class UploaderService {
  // ...

  /**
   * Delete File
   *
   * Takes a file url and deletes the file from the bucket
   */
  public deleteFile(url: string): void {
    const keyArr = url.split('.com/');

    if (keyArr.length !== 2 || !this.bucketData.url.includes(keyArr[0])) {
      this.loggerService.error('Invalid url to delete file');
    }

    this.client
      .send(
        new DeleteObjectCommand({
          Bucket: this.bucketData.name,
          Key: keyArr[1],
        }),
      )
      .then(() => this.loggerService.log('File deleted successfully'))
      .catch((error) => this.loggerService.error(error));
  }

  // ...
}

Driver specific

Apollo Driver

Using the latest version of graphql-upload with NestJS can be quite tricky since it uses mjs, this means we need to use dynamic imports, alternative you can use graphql-upload-minimal, or graphql-upload version 13.

In order to cover both approaches I will divide this section in 2 parts, one for version 16 and one for version 13.

Version 16

Using version 16 is not the best approach for production projects as it forces us to use a lot of experimental features through our app.

Start by installing graphql-upload:

$ yarn add graphql-upload

Middleware Set up

On your main file dynamically import the graphql-upload/graphqlUploadExpress.mjs:

import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  const configService = app.get(ConfigService);
  // ...

  const { default: graphqlUploadExpress } = await import(
    'graphql-upload/graphqlUploadExpress.mjs'
  );
  app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));

  // ...
  app.useGlobalPipes(new ValidationPipe());
  await app.listen(configService.get<number>('port'));
}

bootstrap();

Asynchronous Scalar

The scalar now will be a function that you add to your Field and Args decorators:

// upload-scalar.util.ts
import { GraphQLScalarType } from 'graphql/type';

let GraphQLUpload: GraphQLScalarType;

import('graphql-upload/GraphQLUpload.mjs').then(({ default: Upload }) => {
  GraphQLUpload = Upload;
});

export const uploadScalar = () => GraphQLUpload;

In the end a DTO (ArgsType) would look something like this:

import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { FileUploadDto } from './file-upload.dto';
import { uploadScalar } from '../utils/upload-scalar.util';

@ArgsType()
export abstract class PictureDto {
  @Field(uploadScalar)
  @ValidatePromise()
  @Type(() => FileUploadDto)
  public picture: Promise<FileUploadDto>;
}

Testing

Using mjs will break your tests, so to be able to use mjs on your app, add the --experimental-vm-modules flag to jest:

{
  "...": "...",
  "scripts": {
    "...": "...",
    "test": "yarn node --experimental-vm-modules $(yarn bin jest)",
    "test:watch": "yarn node --experimental-vm-modules $(yarn bin jest)  --watch",
    "test:cov": "yarn node --experimental-vm-modules $(yarn bin jest)  --coverage",
    "test:e2e": "yarn node --experimental-vm-modules $(yarn bin jest)  --config ./test/jest-e2e.json",
    "...": "..."
  },
  "...": "..."
}

Version 13

Using version 13 is what I recommend, start by installing the package:

$ yarn add graphql-upload@13
$ yarn add -D @types/graphql-upload

Middleware Set up

With version 13 we can just do a normal import:

import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { graphqlUploadExpress } from 'graphql-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';


async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  const configService = app.get(ConfigService);
  // ...
  app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));

  // ...
  app.useGlobalPipes(new ValidationPipe());
  await app.listen(configService.get<number>('port'));
}

bootstrap();

DTOs

The GraphQLUpload scalar will just be an export from graphql-upload:

import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { GraphQLUpload } from 'graphql-upload';
import { FileUploadDto } from './file-upload.dto';

@ArgsType()
export abstract class PictureDto {
  @Field(() => GraphQLUpload)
  @ValidatePromise()
  @Type(() => FileUploadDto)
  public picture: Promise<FileUploadDto>;
}

Mercurius

Mercurius has its own adaptation of graphql-upload, start by installing the mercurius-upload package:

$ yarn add mercurius-upload

Middleware Set up

As any fastify middleware you just need to import the default value of the library:

Note: do not forget to set "esModuleInterop": true on your tsconfig.json file

import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import {
  FastifyAdapter,
  NestFastifyApplication,
} from '@nestjs/platform-fastify';
import mercuriusUpload from 'mercurius-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';

async function bootstrap() {
  const app = await NestFactory.create<NestFastifyApplication>(
    AppModule,
    new FastifyAdapter(),
  );
  const configService = app.get(ConfigService);
  // ...

  app.register(
    mercuriusUpload,
    configService.get<IUploaderMiddlewareOptions>('uploader.middleware'),
  );

  // ...
  app.useGlobalPipes(new ValidationPipe());
  await app.listen(
    configService.get<number>('port'),
    '0.0.0.0',
  );
}

bootstrap();

DTOs

mercurius-upload is dependent on graphql-upload version 15 so you need to import default the scalar from graphql-upload/GraphQLUpload.js:

import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import GraphQLUpload from 'graphql-upload/GraphQLUpload.js';
import { FileUploadDto } from './file-upload.dto';

@ArgsType()
export abstract class PictureDto {
  @Field(() => GraphQLUpload)
  @ValidatePromise()
  @Type(() => FileUploadDto)
  public picture: Promise<FileUploadDto>;
}

Conclusion

A complete version of this code can be found in this repository.

About the Author

Hey there my name is Afonso Barracha, I am a Econometrician made back-end developer that has a passion for GraphQL.

I try to do post once a week here on Dev about Back-End APIs and related topics.

If you do not want to lose any of my posts follow me here on dev, or on LinkedIn.

Blog

NestJS GraphQL image upload into a S3 bucket

Afonso Barracha

Intro

Common

Set Up

Configuration

Module Creation

Image Uploading

Driver specific

Apollo Driver

Version 16

Version 13

Mercurius

Conclusion

About the Author

Join Our Newsletter. No Spam, Only the good stuff.

Related