Going down the rabbit hole of the canvas

Recently, I encountered an issue on ChatCraft that required me to adjust the size of an image. If it needs to be done on the frontend (browser), it involves the use of canvas. In simple terms, this process can be explained using the example below.



//an example of how you might use a canvas to 
//compress an image using JavaScript:
function compressImage(image, outputFormat, quality, callback) {
  // Create a canvas element
  const canvas = document.createElement('canvas');
  const ctx = canvas.getContext('2d');

  // Set the canvas size to the desired output size
  canvas.width = image.width / 2; // Example: reduce width by half
  canvas.height = image.height / 2; // Example: reduce height by half

  // Draw the image onto the canvas
  ctx.drawImage(image, 0, 0, canvas.width, canvas.height);

  // Convert the canvas to a Blob (binary large object)
  canvas.toBlob(function(blob) {
    // The blob is a compressed version of the image
    callback(blob);
  }, outputFormat, quality);
}

// Usage example
const image = new Image();
image.src = 'path/to/your/image.jpg'; // Load your image here
image.onload = function() {
  compressImage(image, 'image/jpeg', 0.7, function(compressedBlob) {
    // Do something with the compressed image blob
    console.log('Compressed image size:', compressedBlob.size);
  });
};

I was fascinated by this approach, prompting me to commence my exploration into the depths of canvas.

In the example above, let's see these two lines:



const ctx = canvas.getContext('2d');
ctx.drawImage(image, 0, 0, canvas.width, canvas.height);

First, the canvas.getContext('2d') is using the HTMLCanvasElement's method: HTMLCanvasElement.getContext(), :

A drawing context lets you draw on the canvas. Calling getContext with "2d" returns a CanvasRenderingContext2D object, whereas calling it with "webgl" (or "experimental-webgl") returns a WebGLRenderingContext object. This context is only available on browsers that implement WebGL.

So, ctx.drawImage(image, 0, 0, canvas.width, canvas.height) is using the CanvasRenderingContext2D's method drawImage()



drawImage(image, dx, dy)
drawImage(image, dx, dy, dWidth, dHeight)
drawImage(image, sx, sy, sWidth, sHeight, dx, dy, dWidth, dHeight)

ctx.drawImage(image, dx, dy) - Draws the image at its natural size at the specified position on the canvas.
ctx.drawImage(image, dx, dy, dWidth, dHeight) - Draws the image, scaling it to fit the specified width and height at the specified position on the canvas.
ctx.drawImage(image, sx, sy, sWidth, sHeight, dx, dy, dWidth, dHeight) - Draws a specified area of the source image to a specified area on the canvas, allowing for both cropping and scaling.

Then, I found this gem: an article The HTML5 Canvas Handbook Although it was written in 2013, I found it still applicable today. It is easier for beginners to understand various aspects compared to directly reading the MDN. I highly recommend it.

In the 2.2.11 Drawing images section, you can find straightforward examples. The author provides a series of images to demonstrate the usage of drawImage.

Next, I tried to find the source code for drawImage, and I found the CanvasRenderingContext2D.cpp from the Blink, which is the browser engine developed as part of the Chromium project.



void CanvasRenderingContext2D::drawImage(const CanvasImageSourceUnion& imageSource, float x, float y, ExceptionState& exceptionState)
{
    CanvasImageSource* imageSourceInternal = toImageSourceInternal(imageSource);
    FloatSize sourceRectSize = imageSourceInternal->elementSize();
    FloatSize destRectSize = imageSourceInternal->defaultDestinationSize();
    drawImage(imageSourceInternal, 0, 0, sourceRectSize.width(), sourceRectSize.height(), x, y, destRectSize.width(), destRectSize.height(), exceptionState);
}
void CanvasRenderingContext2D::drawImage(const CanvasImageSourceUnion& imageSource,
    float x, float y, float width, float height, ExceptionState& exceptionState)
{
    CanvasImageSource* imageSourceInternal = toImageSourceInternal(imageSource);
    FloatSize sourceRectSize = imageSourceInternal->elementSize();
    drawImage(imageSourceInternal, 0, 0, sourceRectSize.width(), sourceRectSize.height(), x, y, width, height, exceptionState);
}
void CanvasRenderingContext2D::drawImage(const CanvasImageSourceUnion& imageSource,
    float sx, float sy, float sw, float sh,
    float dx, float dy, float dw, float dh, ExceptionState& exceptionState)
{
    CanvasImageSource* imageSourceInternal = toImageSourceInternal(imageSource);
    drawImage(imageSourceInternal, sx, sy, sw, sh, dx, dy, dw, dh, exceptionState);
}

And all three functions will be calling:



void CanvasRenderingContext2D::drawImage(CanvasImageSource* imageSource,
    float sx, float sy, float sw, float sh,
    float dx, float dy, float dw, float dh, ExceptionState& exceptionState)
{
    if (!drawingCanvas())
        return;
    RefPtr<Image> image;
    SourceImageStatus sourceImageStatus = InvalidSourceImageStatus;
    if (!imageSource->isVideoElement()) {
        image = imageSource->getSourceImageForCanvas(&sourceImageStatus);
        if (sourceImageStatus == UndecodableSourceImageStatus)
            exceptionState.throwDOMException(InvalidStateError, "The HTMLImageElement provided is in the 'broken' state.");
        if (!image || !image->width() || !image->height())
            return;
    } else {
        if (!static_cast<HTMLVideoElement*>(imageSource)->hasAvailableVideoFrame())
            return;
    }
    if (!std::isfinite(dx) || !std::isfinite(dy) || !std::isfinite(dw) || !std::isfinite(dh)
        || !std::isfinite(sx) || !std::isfinite(sy) || !std::isfinite(sw) || !std::isfinite(sh)
        || !dw || !dh || !sw || !sh)
        return;
    FloatRect srcRect = normalizeRect(FloatRect(sx, sy, sw, sh));
    FloatRect dstRect = normalizeRect(FloatRect(dx, dy, dw, dh));
    clipRectsToImageRect(FloatRect(FloatPoint(), imageSource->elementSize()), &srcRect, &dstRect);
    imageSource->adjustDrawRects(&srcRect, &dstRect);
    if (srcRect.isEmpty())
        return;
    if (shouldDisableDeferral(imageSource) || image->imageForCurrentFrame()->isTextureBacked())
        canvas()->disableDeferral();
    validateStateStack();
    draw(
        [this, &imageSource, &image, &srcRect, dstRect](SkCanvas* c, const SkPaint* paint) // draw lambda
        {
            drawImageInternal(c, imageSource, image.get(), srcRect, dstRect, paint);
        },
        [this, &dstRect](const SkIRect& clipBounds) // overdraw test lambda
        {
            return rectContainsTransformedRect(dstRect, clipBounds);
        }, dstRect, CanvasRenderingContext2DState::ImagePaintType,
        imageSource->isOpaque() ? CanvasRenderingContext2DState::OpaqueImage : CanvasRenderingContext2DState::NonOpaqueImage);
    validateStateStack();
    bool isExpensive = false;
    if (ExpensiveCanvasHeuristicParameters::SVGImageSourcesAreExpensive && image && image->isSVGImage())
        isExpensive = true;
    if (imageSource->elementSize().width() * imageSource->elementSize().height() > canvas()->width() * canvas()->height() * ExpensiveCanvasHeuristicParameters::ExpensiveImageSizeRatio)
        isExpensive = true;
    if (isExpensive) {
        ImageBuffer* buffer = canvas()->buffer();
        if (buffer)
            buffer->setHasExpensiveOp();
    }
    if (imageSource->isCanvasElement() && static_cast<HTMLCanvasElement*>(imageSource)->is3D()) {
        // WebGL to 2D canvas: must flush graphics context to prevent a race
        // FIXME: crbug.com/516331 Fix the underlying synchronization issue so this flush can be eliminated.
        canvas()->buffer()->flushGpu();
    }
    if (canvas()->originClean() && wouldTaintOrigin(imageSource))
        canvas()->setOriginTainted();
}

This is a lengthy function, but we can observe one crucial call:



 drawImageInternal(c, imageSource, image.get(), srcRect, dstRect, paint);

And in drawImageInternal function, we can see:



void CanvasRenderingContext2D::drawImageInternal(SkCanvas* c, CanvasImageSource* imageSource, Image* image, const FloatRect& srcRect, const FloatRect& dstRect, const SkPaint* paint)
{
    int initialSaveCount = c->getSaveCount();
    SkPaint imagePaint = *paint;
    if (paint->getImageFilter()) {
        SkMatrix ctm = c->getTotalMatrix();
        SkMatrix invCtm;
        if (!ctm.invert(&invCtm)) {
            // There is an earlier check for invertibility, but the arithmetic
            // in AffineTransform is not exactly identical, so it is possible
            // for SkMatrix to find the transform to be non-invertible at this stage.
            // crbug.com/504687
            return;
        }
        c->save();
        c->concat(invCtm);
        SkRect bounds = dstRect;
        ctm.mapRect(&bounds);
        SkRect filteredBounds;
        paint->getImageFilter()->computeFastBounds(bounds, &filteredBounds);
        SkPaint layerPaint;
        layerPaint.setXfermode(paint->getXfermode());
        layerPaint.setImageFilter(paint->getImageFilter());
        c->saveLayer(&filteredBounds, &layerPaint);
        c->concat(ctm);
        imagePaint.setXfermodeMode(SkXfermode::kSrcOver_Mode);
        imagePaint.setImageFilter(nullptr);
    }
    if (!imageSource->isVideoElement()) {
        imagePaint.setAntiAlias(shouldDrawImageAntialiased(dstRect));
        image->draw(c, imagePaint, dstRect, srcRect, DoNotRespectImageOrientation, Image::DoNotClampImageToSourceRect);
    } else {
        c->save();
        c->clipRect(dstRect);
        c->translate(dstRect.x(), dstRect.y());
        c->scale(dstRect.width() / srcRect.width(), dstRect.height() / srcRect.height());
        c->translate(-srcRect.x(), -srcRect.y());
        HTMLVideoElement* video = static_cast<HTMLVideoElement*>(imageSource);
        video->paintCurrentFrame(c, IntRect(IntPoint(), IntSize(video->videoWidth(), video->videoHeight())), &imagePaint);
    }
    c->restoreToCount(initialSaveCount);
}

In the above function, this line



        c->scale(dstRect.width() / srcRect.width(), dstRect.height() / srcRect.height());

tells us it will use SkCanvas's scale method. You will notice that the SkCanvas is actually coming from third party lib Skia



//CanvasRenderingContext2D.cpp
#include "third_party/skia/include/core/SkCanvas.h"

Skia is an open-source 2D graphics library written in C++ and serves as the graphics engine for Google Chrome and ChromeOS, Android, Flutter, and many other products. Here is the source code of scale function:



void SkCanvas::scale(SkScalar sx, SkScalar sy) {
    if (sx != 1 || sy != 1) {
        this->checkForDeferredSave();
        fMCRec->fMatrix.preScale(sx, sy);

        this->topDevice()->setGlobalCTM(fMCRec->fMatrix);

        this->didScale(sx, sy);
    }
}

If you like, you can continue dig out how the scaling transformation is applied to the matrix in preScale function.

In terms of the pixel grid on a computer screen or canvas, the scaling transformation affects how the drawing commands are mapped to the actual pixels. Simply terms in general:

Original Coordinates: Each shape or image has coordinates that determine its position and size on the canvas. These coordinates are based on a logical coordinate system, not directly tied to the physical pixels.

Transformation Matrix: When a scaling transformation is applied, the transformation matrix is updated to include the scaling factors. This matrix is used to transform the logical coordinates of the shapes or images.

Rasterization: During rasterization, the transformed logical coordinates are converted to pixel coordinates. The rendering engine uses the transformation matrix to determine where and how to place each shape or image on the pixel grid.

Pixel Mapping: If an object is scaled up, more pixels are used to represent it, making it appear larger on the screen. Conversely, if an object is scaled down, fewer pixels represent it, making it appear smaller.

Interpolation: When scaling images, interpolation algorithms (like nearest-neighbor, bilinear, or bicubic interpolation) determine the color values of the new pixels based on the original pixels to create a smooth transition and minimize artifacts.

I stopped further tracking of the source code. As an end of this blog post, I would like to share my finding of the React Native Skia which brings the Skia Graphics Library to React Native. You can get more information from a Shopify Engineering blog Getting Started with React Native Skia. Hope it helps, see you next post.

Blog

Going down the rabbit hole of the canvas

Mingming Ma

Join Our Newsletter. No Spam, Only the good stuff.

Related