[NOTE: The live web app that encompasses this functionality can be found here: https://www.paintmap.studio. All of the underlying code for that site can be found here: https://github.com/bytebodger/color-map.]

In the first few articles in this series, I discussed the challenges of capturing the "true" color of real-world objects and I showed how to use React/JavaScript to capture and render images. But now that we have an image loaded, we need to start the process of manipulating that image to our needs.

The first step in that process is to pixelate the image.

Why pixelate??

You don't have to pixelate an image if you're going to convert a digital image into a painting. But you probably want to.

First, with regard to my particular style of painting, I "grid-out" my panels so I know where I'm at on the painting at any given point in time. Specifically, I work on 3'x 3' panels, divided into quarter-inch squares. This means that each painting consists of 144x144 squares. This works particularly well for me because my style of painting is much more abstract. I'm not creating true-to-life images. Even though my style has evolved to look ever-less pixelated over time, I'm still working with big clunky blobs of paint on the panel. So there's no need to get down to pixel-level accuracy when my chosen style of painting is far "chunkier". As an example, this is what one of my panels looks like when I'm just about to start painting:

But even if you're aiming to do a more realistic style of painting, you almost certainly don't want to capture ALL of the colors that exist in a given digital image.

I work from images that are cropped to a height and width of 1440 pixels. This means that the source image contains more than 2 million pixels. Furthermore, there are more than 16 million potential colors available in the RGB color space. So in a 1440x1440 digital image, it's theoretically possible that every single one of those 2+ million pixels is a unique color.

Of course, many of those "unique" colors would be indistinguishable to the human eye. And no painter wants to set about mixing up 2 million different colors. Even the most realistic of painters needs only to get a general idea of the color palette that will be necessary for the whole painting, or at least, for a particular section of the painting. So it's far more practical to take a digital image, that is rich with many millions of technically-distinct colors, and "collapse" it down to a handful of reference colors. We can accomplish this task by pixelating the image.

Manipulating the loaded image

In the last article, we loaded a user's image and rendered it on a canvas. We started building a useImage Hook that will do most of our image processing. But where we left off, that hook does nothing but render the original image on the screen.

The previous state of the useImage Hook looked like this:

/* useImage.js */

export const useImage = () => {
   const canvas = useRef(null);
   const context = useRef(null);
   const image = useRef(null);

   useEffect(() => {
      canvas.current = document.getElementById('canvas');
   }, []);

   const create = (src = '') => {
      const source = src === '' ? image.current.src : src;
      const newImage = new Image();
      newImage.src = source;
      newImage.onload = () => {
         image.current = newImage;
         canvas.current.width = newImage.width;
         canvas.current.height = newImage.height;
         context.current = canvas.current.getContext('2d', {alpha: false, willReadFrequently: true});
         context.current.drawImage(newImage, 0, 0);
         let stats = pixelate();
         uiState.setStats(stats);
      }
   };

   return {
      create,
   };
};

To be clear, the code above doesn't precisely match the useImage Hook that I showed in the last article. Specifically, I've added a call to a new function - pixelate() - and I'm logging the resulting stats from the image back into our global context at the end of the onload event.

Also, for reference purposes, I'm going to show you what the new calculations will do to a target image. For the sake of this article, I'm going to use this image:

I've chosen this image because skin tones are particularly difficult to get "right" when doing algorithmic color matching. But we won't have to worry about that in this article. For the time being, I'm only concerned with pixelating the image so that it can be processed in larger chunks.

So let's take a look at the code in our new function, pixelate():

const pixelate = () => {
   // get dimensions from the image that we just put into the canvas
   const { height, width } = canvas.current;
   const stats = {
      colorCounts: {},
      colors: [],
      map: [],
   };
   const blockSize = 10;
   // looping through every new "block" that will be created with the pixelation
   for (let y = 0; y < height; y += blockSize) {
      const row = [];
      for (let x = 0; x < width; x += blockSize) {
         const remainingX = width - x;
         const remainingY = height - y;
         const blockX = remainingX > blockSize ? blockSize : remainingX;
         const blockY = remainingY > blockSize ? blockSize : remainingY;
         // get the image data for the current block and calculate its average color
         const averageColor = calculateAverageColor(context.current.getImageData(x, y, blockX, blockY));
         averageColor.name = `${averageColor.red}_${averageColor.green}_${averageColor.blue}`;
         // add this color to the row array so it can be added to the image stats
         row.push(averageColor);
         if (Object.hasOwn(stats.colorCounts, averageColor.name))
            stats.colorCounts[averageColor.name]++;
         else {
            stats.colorCounts[averageColor.name] = 1;
            stats.colors.push(averageColor);
         }
         // draw the new block over the top of the existing image that exists in the canvas
         context.current.fillStyle = `rgb(${averageColor.red}, ${averageColor.green}, ${averageColor.blue})`;
         context.current.fillRect(x, y, blockX, blockY);
      }
      stats.map.push(row);
   }
   return stats;
};

This function performs the following steps:

1. Grabs the height/width parameters from the existing <canvas> element.
   

2. Loops through every "block" that we're gonna create with our new pixelation. (I don't call them "pixels" because that gets rather confusing. The image already has pixels. 2,073,600 of them, to be exact. But for us to "pixelate" the image, we're essentially creating new, larger blocks that will be painted over the top of the original image.)
   

3. Calls another function to determine the average color in each block.
   

4. Adds the block info to a stats object so we can reference it in future operations.
   

5. Paints the new "block" over the top of the old image.
   

Essentially, what we're doing is tiling over the old image with new, chunkier blocks that represent the original colors beneath them. We do this by looping through the image data. The image data is organized on an x/y grid, with x representing the horizontal position of a pixel, and y representing the vertical position of that same pixel.

We already have the dimensions of our canvas because they reside in the canvas ref. So we can use those to instantiate our loops.

You may notice that getImageData() allows us to get the canvas information for all the pixels that reside in any given region on the canvas. The getImageData() function works like this:

context.getImageData(
   xCoordinateToStartCapture,
   yCoordinateToStartCapture,
   widthOfTheRegionToBeCaptured,
   heightOfTheRegionToBeCaptured,

So if you already have an image loaded into a canvas and you wanted to grab the data for a region that's 50 pixels wide and 75 pixels tall, starting from the pixel that's 20 pixels from the left and 90 pixels from the top, you'd do this:

context.getImageData(20, 90, 50, 75);

That line of code would return an array containing the data for all of the pixels in that particular region of the image.

In the pixelate() function, the regions we're grabbing are always squares. The size of those regions is determined by the blockSize variable.

[NOTE: In the live app, blockSize is a configurable variable that's defined by the user through the UI. I've only hardcoded it to 10 in this example for simplicity.]

So now that we see how to loop through the image data and repaint regions of the canvas, we need to dive into that calculateAverageColor() function...

Color averaging

Now that our pixelate() loop is parsing through the image in 10x10 blocks, we'll have a lot of individual blocks. For example, the first block in the image (meaning: the block in the far upper-left corner), looks like this:

Obviously, I enlarged that original 10x10 block by quite a bit so you don't have to squint to look at a 10-pixel-by-10-pixel image on your screen. But you get the idea.

This particular block of pixels is quite homogenous. Not a lot of color variation there. But there are still multiple different colors in this block of 100 pixels. For example, in this block alone, here are a few of the colors I can find with my color picker:

• rgb(180, 183, 168)
• rgb(182, 185, 170)
• rgb(179, 181, 167)
• rgb(187, 189, 175)
• and many more...

Even though the block looks quite unicolor, it actually contains many slight variations. So how do we get the average color for all pixels in the block?

Thankfully, this is fairly straightforward. Because for every pixel, we can extract the red, green, and blue values. Once we have the red, green, and blue values, you can simply average them all together to get the average red, green, and blue value for the entire block. Then we use the new, averaged RGB values to paint the new block on our canvas.

The calculateAverageColor() code looks like this:

const calculateAverageColor = (imageData = {}) => {
   let redSum = 0;
   let redCounter = 0;
   let greenSum = 0;
   let greenCounter = 0;
   let blueSum = 0;
   let blueCounter = 0;
   for (let x = 0; x < imageData.width; x++) {
      for (let y = 0; y < imageData.height; y++) {
         const {red, green, blue} = getRgbFromImageData(imageData, x, y);
         redSum += red;
         redCounter++;
         greenSum += green;
         greenCounter++;
         blueSum += blue;
         blueCounter++;
      }
   }
   return {
      red: Math.round(redSum / redCounter),
      green: Math.round(greenSum / greenCounter),
      blue: Math.round(blueSum / blueCounter),
   };
};

We pass in the region of image data that was captured from the original canvas. Then we calculate the average RGB values and return them in a convenient object. Our new, averaged block will look like this (again, enlarged for convenience):

All 100 pixels in the image are now the exact same color.

[Full Disclosure: If you take a color picker and hover it over the newly-averaged block shown above, it will still show you a myriad of slightly different colors. This isn't a bug in my averaging algorithm. It's a side effect from my desktop image editor that happens when you take a unicolor 10x10 image and blow it up to 500x500.]

Also, you may notice that in the function above, it makes a call to another custom function: getRgbFromImageData(). That's basically a helper function, which in turn calls getPixelFromImageData(), which in turn calls getPixelIndex(). I'm not going to bother illustrating all of those smaller helper functions here. But if you'd like to see them all - and all the rest of the code that runs https://paintmap.studio, you can feel free to dig through it in the GitHub repo: https://github.com/bytebodger/adam-davis-codes.

Once the pixelate() function has averaged every one of the blocks, it will render a new image that looks like this:

Obviously, you're going to lose some image acuity. After all, we've just reduced the color density by a factor of 10. But you can't pixelate an image and expect the resulting likeness to not have a few quirks.

And of course, the blockSize variable plays a key role in how "accurate" our processed image appears. Bigger blocks lead to rougher images. Smaller blocks produce images that are much closer to the original. For example, this is the same image pixelated with blockSize = 20:

And it's much more accurate with blockSize = 5:

So it may be tempting to ratchet blockSize down to its lowest possible number. But the closer that blockSize gets to 1, the more colors (and, for my purposes, paints) you'll need to produce to a replica.

In the next installment...

This feels like pretty good progress. We're now capturing images, rendering them onscreen, and transforming them by pixelating them. But there's still a lot of work to do, because we still have no way of knowing how any of the colors in the pixelated image would map to our inventory of real-world paints.

In the next installment, I'm gonna dive into the hairy world of digital color matching. Things get much more intense from that point forward...