User Interactive background color remover using HTML Canvas

Intro

For my personal project, I needed to implement frontend side image background remover and color remover using javascript. In this post, I would like to share how I did it.
You can see a full demo in below link.

https://swimmingkiim.github.io/color-background-remover/

index.html settings

First, you need to set your index.html and import necessery packages from unpkg.

<body>
  Color Remover
  <button id="drawing-mode">remove</button>
  <button id="remove-background">remove background</button>
  <div id="color-box"></div>
  <canvas id="canvas"></canvas>
  <script src="https://unpkg.com/@tensorflow/tfjs-core@3.3.0/dist/tf-core.js"></script>
  <script src="https://unpkg.com/@tensorflow/tfjs-converter@3.3.0/dist/tf-converter.js"></script>
  <script src="https://unpkg.com/@tensorflow/tfjs-backend-webgl@3.3.0/dist/tf-backend-webgl.js"></script>
  <script src="https://unpkg.com/@tensorflow-models/deeplab@0.2.1/dist/deeplab.js"></script>
  <script src="script.js"></script>
</body>

#drawing-mode → mode changing button
#remove-background → remove image’s background button
#color-box → display the color that current mouse cursor points
tfjs packages → need for removing image’s background
deeplab package → need for using ready-to-use model for masking image

1. Remove image background with deeplab(tfjs)

This part, I got a lot of help from the first link in References. I didn’t changed much. So I’ll just explain one by one.

1. set variables


let model;
const segmentImageButton = document.getElementById("remove-background");
segmentImageButton.style.display = "none";
const canvas = document.getElementById("canvas");
const originalImageCanvas = document.createElement("canvas");
const maskCanvas = document.createElement("canvas");
const ctx = canvas.getContext('2d');
const originalCtx = originalImageCanvas.getContext('2d');
const maskCtx = maskCanvas.getContext('2d');

In order to remove image’s background, you need a model from deeplab. Deeplab is providing a model to segment image. Also, you need to prepare a mask canvas that which is not displayed on the screen, but helper canvas to record a background data.

2. load ready-to-use model from deeplab

async function loadModel(modelName) {
    model = await deeplab.load({ "base": modelName, "quantizationBytes": 2 });
    segmentImageButton.style.display = "inline-block";
}

In deeplab, there’re three possible mode for image segmentation. For this case, I would suggest you a ‘pascal’ mode. The model name is ‘pascal’ in all lowcase. Once you load a model, then you can display remove background botton. You need to do it like this because if the user trying to remove background before model is loaded, there’ll be an error.

3. predict(segment image to get the mask)

async function predict() {
    let prediction = await model.segment(image);
    renderPrediction(prediction);
}

function renderPrediction(prediction) {
    const { height, width, segmentationMap } = prediction;

    const segmentationMapData = new ImageData(segmentationMap, width, height);
    canvas.width = width;
    canvas.height = height;
    ctx.drawImage(image, 0, 0, width, height);
    maskCanvas.width = width;
    maskCanvas.height = height;
    maskCtx.putImageData(segmentationMapData, 0, 0);
    removeBackground([0,0,0], width, height);
}

Now, you can segment your image. The type of image is HTMLImage. The segment function returns a result data. In that data, you need width, height and segmentationMap fields. Width and height are the size of segmented image. The segmentationMap refers to the image data of segmented image. If you draw this by using putImageData method on a canvas, you’ll see a combinations of colors representing different objects. In pascal mode, background is black color. So you can use this to make a mask. First, draw a segmentationMap on the maskCanvas.

4. remove background using segment data

function removeBackground(color, width, height){
  image.width = width;
  image.height = height;
  originalImage.width = width;
  originalImage.height = height;
  canvas.width =width;
  canvas.height =height;
  originalImageCanvas.width =width;
  originalImageCanvas.height =height;
  ctx.drawImage(image, 0,0,width,height);
    var canvasData = ctx.getImageData(0, 0, width, height),
        pix = canvasData.data;
  var maskCanvasData = maskCtx.getImageData(0, 0, width, height),
        maskPix = maskCanvasData.data;

    for (var i = 0, n = maskPix.length; i <n; i += 4) {
        if(maskPix[i] === color[0] && maskPix[i+1] === color[1] && maskPix[i+2] === color[2]){
             maskPix[i+3] = 0;   
             pix[i+3] = 0;
        }
    }

    ctx.putImageData(canvasData, 0, 0);
    maskCtx.putImageData(maskCanvasData, 0, 0);
    const base64 = canvas.toDataURL();
    image.src = base64
    originalImage.src = originalImage.src;
}

In this part, I got help from second link in the References.

First, draw a current image on the displayed canvas, original image to other canvas. Since the size of an image will be changed after segmentation, you need to redraw your original(for backup) image to update.

Second, get Image data from canvas and mask canvas. And loop all mask canvas’s pixels. If you find black pixel on a mask canvas make it transparent. the pixel data array is an array of rgba number in each pixel, so you need to jump 4 at a time.

Third, redraw updated image on the canvas and mask canvas. Retrieve data url from canvas and update current image’s src.

2. Select color part according to mouse position

Now, you need to implement interactive color remover.

1. set variables & image

const canvas = document.getElementById("canvas");
const originalImageCanvas = document.createElement("canvas");
const modeButton = document.getElementById("drawing-mode");
const ctx = canvas.getContext('2d');
const originalCtx = originalImageCanvas.getContext('2d');
const colorBox = document.getElementById("color-box");
let mode = "remove";
colorBox.style.width = "20px";
colorBox.style.height = "20px";
colorBox.style.backgroundColor = "black";
const selectColorData = [19, 163, 188];
const removeColorData = [255,255,255, 0];
let originalImage;
let image;
let imageSize = {
  width: 500,
  height: 500,
}
const brushSize = 20;
let circle = new Path2D();
circle.stroke = `rgb(${selectColorData.join(",")})`;
const imageSrc = "https://images.unsplash.com/photo-1648199840917-9e4749a5047e?ixlib=rb-1.2.1&ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8&auto=format&fit=crop&w=687&q=80";

const canvasOffset = {
  x: canvas.getBoundingClientRect().left,
  y: canvas.getBoundingClientRect().top,
}

colorBox → for displaying current color of mouse cursor points at.
selectColorData → single rgb value array for display selected area.
removeColorData → single rgba value array with alpha to 0. (for removing background)
imageSize → Maxium size of initial image size.
brushSize → used when you remove or recover(heal) original area
cicle → for brush
canvasOffset → important when calculating cursor position relative to canvas itself.

const setCanvas = () => {
  const _image = new Image();
  _image.onload = () => {
    image = _image.cloneNode();
    image.onload = null;
    originalImage = _image.cloneNode();
    imageSize = {
      width: Math.min(_image.width, imageSize.width),
      height: Math.min(_image.width, imageSize.width) * (_image.height /_image.width)
    }
    canvas.width = imageSize.width;
    canvas.height = imageSize.height;
    originalImageCanvas.width = imageSize.width;
    originalImageCanvas.height = imageSize.height;
    image.width = imageSize.width;
    image.height = imageSize.height;
    _image.width = imageSize.width;
    _image.height = imageSize.height;
    originalImage = _image.cloneNode();
    ctx.drawImage(image, 0,0, image.width, image.height);
    originalCtx.drawImage(_image, 0,0, _image.width, _image.height);
    console.log(originalImageCanvas)
  }
  _image.crossOrigin = "anonymous";
  _image.src = imageSrc;
}

When your image is loaded, there’re some works to be done. You need to initialize canvas and original canvas(not change after this, it’s a reference for recovering image) and their src(HTML Image element). Finally, draw the image on canvas and original canvas.

3. Remove certain color from HTML Canvas

const isInColorRange = (targetColor, compareTo, i) => {
  return (
    targetColor[i] >= compareTo[0] - 10
    && targetColor[i] <= compareTo[0] + 10
    && targetColor[i+1] >= compareTo[1] - 10
    && targetColor[i+1] <= compareTo[1] + 10
    && targetColor[i+2] >= compareTo[2] - 10
    && targetColor[i+2] <= compareTo[2] + 10
  )
}

const selectColor = (colorData) => {
    let canvasData = ctx.getImageData(0, 0, canvas.width, canvas.height),
        pix = canvasData.data; 
    for (let i = 0, n = pix.length; i <n; i += 4) {
        if(isInColorRange(pix, colorData, i)){ 
          pix[i] = selectColorData[0];
          pix[i+1] = selectColorData[1];
          pix[i+2] = selectColorData[2];
        }
    }
    ctx.putImageData(canvasData, 0, 0);
}

const removeColor = (colorData) => {
    let canvasData = ctx.getImageData(0, 0, canvas.width, canvas.height),
        pix = canvasData.data;  
    for (let i = 0, n = pix.length; i <n; i += 4) {
        if(isInColorRange(pix, colorData, i)){ 
             pix[i] = removeColorData[0];
             pix[i+1] = removeColorData[1];
             pix[i+2] = removeColorData[2];
             pix[i+3] = removeColorData[3];
        }
    }
  ctx.putImageData(canvasData, 0, 0);
}

It’s just like removing background with tfjs(deeplab). Get pixel data and loop, find, change them. Note that isInColorRange is customizable you can change 10 to other number and adjust for you project.

4. Recover original image to HTML Canvas

const healArea = (position) => {
  ctx.clearRect(0,0, image.width, image.height);
  ctx.drawImage(originalImage, 0,0, originalImage.width, originalImage.height);
  ctx.globalCompositeOperation = "destination-in";
  ctx.moveTo(position.x, position.y);
  circle.moveTo(position.x, position.y);
  circle.arc(position.x, position.y, brushSize/2, 0, 2 * Math.PI);
  ctx.fill(circle);
  ctx.globalCompositeOperation = "source-over";
  ctx.drawImage(image, 0,0, image.width, image.height);
}

const removeArea = (position) => {
  ctx.clearRect(0,0, image.width, image.height);
  ctx.drawImage(image, 0,0, image.width, image.height);
  ctx.globalCompositeOperation = "destination-out";
  ctx.moveTo(position.x, position.y);
  circle.moveTo(position.x, position.y);
  circle.arc(position.x, position.y, brushSize/2, 0, 2 * Math.PI);
  ctx.fill(circle);
  ctx.globalCompositeOperation = "source-over";
}

Also, above two fuction’s logic is very simular too. The difference is that there’re using different globalcCompositionOperation. Removing area is using “destination-out” to remove the area of its own, and healing area is using “destination-in” to draw only area of circle itself and remove else.

5. Register mouse events

Finally, combine all those functions properly in mouse move, donw and up events. I won’t go on detail. I don’t think the code below contains any special new approch.

1. utilities

const detectColor = (position) => {
  const colorData = ctx.getImageData(
    position.x, 
    position.y,
    1,
    1
  ).data;
  return colorData;
}

const changeColorBoxColor = (colorData) => {
  const rgba = `rgba(${colorData.join(",")})`;
  colorBox.style.backgroundColor = rgba;
}

2. onMouseMove(without mouse down)

const onMouseMoveSelectColor = (e) => {
    ctx.drawImage(image, 0,0, image.width, image.height);
    const position = {
      x: e.clientX - canvasOffset.x,
      y: e.clientY - canvasOffset.y,
    }
    const color = detectColor(position);
    changeColorBoxColor(color);
    if (mode === "remove by color") {
      selectColor(
        color, 
        position.x, 
        position.y
      );
    } else {
      selectArea(
        position.x, 
        position.y
      );
    }
  }

3. onMouseMove(with mouse down)

const onMouseDownAndMoveRemoveColor = (e) => {
    const callback  = (e) => {
    const position = {
      x: e.clientX - canvasOffset.x,
      y: e.clientY - canvasOffset.y,
    }
    const color = detectColor(position);
  if (mode === "remove by color") {
    removeColor(
      color, 
      position.x, 
      position.y
    )
  } else if (mode === "heal area") {
    healColor(
    position,
  );
  } else {
    removeArea(position);
  }
  }
    canvas.onmousemove = callback;
    callback(e);
}

4. Register listeners

const toggleMode = (e) => {
  if (e.target.innerText === "remove by color") {
    mode = "heal area";
  } else if (e.target.innerText === "heal area") {
    mode = "remove area";
  } else {
    mode = "remove by color";
  }
  e.target.innerText = mode;
}

const registerListener = () => {
  canvas.onmousemove = onMouseMoveSelectColor;
  canvas.onmousedown = onMouseDownAndMoveRemoveColor;
  canvas.onmouseup = (e) => {
    canvas.onmousemove = null;
    canvas.onmousemove = onMouseMoveSelectColor;
    image.src = canvas.toDataURL();
  };
  canvas.onmouseleave = (e) => {
    canvas.onmousemove = null;
    canvas.onmousemove = onMouseMoveSelectColor;
    ctx.drawImage(image, 0,0, image.width, image.height);
  }
  modeButton.onclick = toggleMode;
}

Conclusion

I’m planning to implement these in my image editor project. If anyone interested to see my project implementation, please checkout below link.

https://online-image-maker.com/

Cheers!

Blog