WebGL Engine from Scratch 15: Normal Maps

Last time we went through the process of using bump-maps. Using normal maps is very similar. So we'll see if we can convert our work from last time to utilize normal maps.

Converting a bump-map to a normal map

We'll just start with the code operating on 2 canvases. The input bump-map canvas and the output normal canvas:

//utility functions
function normalize(vec) {
    const mag = Math.sqrt(vec[0] ** 2 + vec[1] ** 2 + vec[2] ** 2);
    return [
        vec[0] / mag,
        vec[1] / mag,
        vec[2] / mag
    ];
}
function getPx(imageData, col, row) {
    col = clamp(col, 0, imageData.width);
    row = clamp(row, 0, imageData.height);
    const offset = (row * imageData.width * 4) + (col * 4);
    return [
        imageData.data[offset + 0] / 255,
        imageData.data[offset + 1] / 255,
        imageData.data[offset + 2] / 255,
        imageData.data[offset + 3] / 255
    ]
}

function setPx(imageData, col, row, val) {
    col = clamp(col, 0, imageData.width);
    row = clamp(row, 0, imageData.height);
    const offset = (row * imageData.width * 4) + (col * 4);
    return [
        imageData.data[offset + 0] = val[0] * 255,
        imageData.data[offset + 1] = val[1] * 255,
        imageData.data[offset + 2] = val[2] * 255,
        imageData.data[offset + 3] = val[3] * 255
    ]
}

//conversion code

const bumpMapData = bumpMapCtx.getImageData(0,0, width, height);
const normalMapData = normalMapCtx.getImageData(0,0, width, height);
for(let row = 0; row < height; row++){
  for(let col = 0; col < width; col++){
    const positiveX = getPx(bumpMapData, col + 1, row)[0];
    const negativeX = getPx(bumpMapData, col - 1, row)[0];
    const positiveY = getPx(bumpMapData, col, row - 1)[0];
    const negativeY = getPx(bumpMapData, col, row + 1)[0];

    const changeX = negativeX - positiveX;
    const changeY = negativeY - positiveY;
    const tangentSpaceNormal = normalize([changeX, changeY, 1.0/10]);

    const colorSpaceNormal = [
            (tangentSpaceNormal[0] + 1) / 2,
            (tangentSpaceNormal[1] + 1) / 2,
            (tangentSpaceNormal[2] + 1) / 2,
        ];

    setPx(normalMapData, col, row, [
      colorSpaceNormal[0], 
      colorSpaceNormal[1], 
      colorSpaceNormal[2],
      1.0
    ]);
  }

This is almost the same as the shader code with 2 key exceptions:

The Y direction start at the top and moves downward (the inverse of the V direction of a texture)
We need to normalize the values into a range that can be understood. Before we had negative values which cannot be represented in RGB-space, they'll just get clamped to 0. The exact range was -1.0 - 1.0. So we need to shift the values over by +1 and divide by 2 to get them in the range from 0 - 1.0

If you tried without converting spaces you might get a result like this:

Input:

Output:

The blues are high because they are facing the camera but the reds and greens can't be seen because half of them are facing the negative direction.

const colorSpaceNormal = [
    (tangentSpaceNormal[0] + 1) / 2,
    (tangentSpaceNormal[1] + 1) / 2,
    (tangentSpaceNormal[2] + 1) / 2,
];
//export this instead!

We also need to know the strength, that is how much elevation does 1.0 represent? Unlike the bump-map this needs to be computed up-front. We can still change it at run-time by applying a scale value in the shader but we need to pick something to export and it should probably be what we expect so we don't need to scale it. A strength value 1 means that the pixel extrudes 1 unit and with a texture size of 512 or more this is very small, you won't even see it show up in the image. For my example I picked 10. The point here is what you choose will make a difference in how the normal maps looks visually.

Strength 1:

Strength 10:

Strength 50:

You can really see the colors at 50.

A different conversion

Digging more into this problem the artifacting is because we are using a very naive algorithm. There are better ones but first we need to talk about filters, convolution and kernels.

Image filtering

This probably deserves a post all to itself but image filtering is pretty much what you might expect. We take an input image and apply some sort of effect to it. In image processing, especially for things like computer vision we apply a "kernel" to each pixel. You can think of this like a simple pixel shader where we're applying it over an existing image and we use neighboring pixels to determine how to transform it. How many neighboring pixels is the kernel size. Our naive algorithm is a 3x3, we take a pixel to the left, right, top and bottom. While we don't technically look at the diagonals they are present in the kernel just zeroed out.

Kernels are represented by matrices. For example our filter for the X direct takes the pixel from X-1 and subtracts the pixel value at X + 1. We can represent that as:

const dxKernel = [
    [0, 0, 0],
    [1, 0, -1],
    [0, 0, 0]
];
const dyKernel = [
    [0, -1, 0],
    [0, 0, 0],
    [0, 1, 0]   
];

The way this works is that we multiply each pixel value by the number in that position with the matrix centered at the pixel we're modifying. So the top-left, top-middle and top-right are multiplied by 0. The middle-left is multiplied by 1, the middle-right is multiplied by -1 and all the rest are also multiplied by 0. We then add all the terms together. This gives us px[x-1, 0] - px[x+1, 0] the same thing as const changeX = negativeX - positiveX;. Note that when I say "pixel value" this is 1-dimensional, you'd repeat per color channel. Let's create a function to generalize this operation which is referred to as "convolution":

//image-helper.js
import { clamp } from "./math-helpers.js";

export function sample(imageData, row, col, oobBehavior = { x: "clamp", y: "clamp" }) {
    let sampleCol = col;
    if (typeof (oobBehavior.x) === "string") {
        switch (oobBehavior.x) {
            case "clamp": {
                sampleCol = clamp(sampleCol, 0, imageData.width);
                break;
            }
            case "repeat": {
                sampleCol = sampleCol % imageData.width;
                break;
            }
        }
    } else if (sampleCol < 0 || sampleCol > imageData.width) return oobBehavior.x;

    let sampleRow = row;
    if (typeof (oobBehavior.y) === "string") {
        switch (oobBehavior.y) {
            case "clamp": {
                sampleRow = clamp(sampleRow, 0, imageData.height);
                break;
            }
            case "repeat": {
                sampleRow = sampleRow % imageData.height;
                break;
            }
        }
    } else if (sampleRow < 0 || sampleRow > imageData.height) return oobBehavior.y;

    const offset = (sampleRow * imageData.width * 4) + (sampleCol * 4);
    return [
        imageData.data[offset + 0] / 255,
        imageData.data[offset + 1] / 255,
        imageData.data[offset + 2] / 255,
        imageData.data[offset + 3] / 255
    ]
}

export function setPx(imageData, row, col, val) {
    col = clamp(col, 0, imageData.width);
    row = clamp(row, 0, imageData.height);
    const offset = (row * imageData.width * 4) + (col * 4);
    return [
        imageData.data[offset + 0] = val[0] * 255,
        imageData.data[offset + 1] = val[1] * 255,
        imageData.data[offset + 2] = val[2] * 255,
        imageData.data[offset + 3] = val[3] * 255
    ]
}

export function convolute(imageData, kernel, oobBehavior = { x: "clamp", y: "clamp" }) {
    const output = new ImageData(imageData.width, imageData.height);
    const kRowMid = (kernel.length - 1) / 2; //kernels should have odd dimensions
    const kColMid = (kernel[0].length - 1) / 2;

    for (let row = 0; row < imageData.height; row++) {
        for (let col = 0; col < imageData.width; col++) {

            const sum = [0,0,0];
            for (let kRow = 0; kRow < kernel.length; kRow++) {
                for (let kCol = 0; kCol < kernel[kRow].length; kCol++) {
                    const sampleRow = row + (-kRowMid + kRow);
                    const sampleCol = col + (-kColMid + kCol);
                    const color = sample(imageData, sampleRow, sampleCol, oobBehavior);
                    sum[0] += color[0] * kernel[kRow][kCol];
                    sum[1] += color[1] * kernel[kRow][kCol];
                    sum[2] += color[2] * kernel[kRow][kCol];
                }
            }

            setPx(output, row, col, [...sum, 1.0]);
        }
    }

    return output;
}

The sample function allows us to sample imageData normalized with out-of-bounds behavior. convolute works as expected over all 3 color channels. setPx is the same. We can use it like this:

const bumpMapData = bumpMapCtx.getImageData(0, 0, width, height);
const normalMapData = normalMapCtx.getImageData(0, 0, width, height);
const strength = this.dom.strength.valueAsNumber;
const xNormals = convolute(bumpMapData, [
    [0, 0, 0 ],
    [1, 0, -1],
    [0, 0, 0]
], { x: "clamp", y: "clamp" })
const yNormals = convolute(bumpMapData, [
    [0, -1, 0],
    [0, 0, 0],
    [0, 1, 0]
], { x: "clamp", y: "clamp" })
for (let row = 0; row < height; row++) {
    for (let col = 0; col < width; col++) {
        const valX = sample(xNormals, row, col);
        const valY = sample(yNormals, row, col);
        const tangentSpaceNormal = normalizeVector([valX[0], valY[0], 1.0/strength]);
        const colorSpaceNormal = [
            (tangentSpaceNormal[0] + 1) / 2,
            (tangentSpaceNormal[1] + 1) / 2,
            (tangentSpaceNormal[2] + 1) / 2,
        ];
        setPx(normalMapData, row, col, [
            colorSpaceNormal[0],
            colorSpaceNormal[1],
            colorSpaceNormal[2],
            1.0
        ]);
    }
}
normalMapCtx.putImageData(normalMapData, 0, 0);

We still need the normalization and color space conversion as a separate mapping but we can replace the change calculation with 2 kernel convolutions.

Now that we have this, we can try out some other kernels. There are some others approximate the same data, and probably a bit better than what we are doing currently because they take the diagonal into account as well. It likely depends on the exact bump map data though as my test texture doesn't always look great, it's likely a noisier texture might have very different results.

What these filters are actually doing is edge detection. If you ran these over an image with lots of visual edges these would produce peaks where contrast is the highest. This is probably why the results aren't super good for a gradient.

Keep in mind that you might find these kernels are flipped from other sources, this is so that the normals are pointing the right direction.

The first the Sobel kernel:

//Sobel
const sobelDxKernel = [
    [-1, 0, 1],
    [-2, 0, 2],
    [-1, 0, 1]
];
const sobelDyKernel = [
    [-1, -2, -1],
    [-2, 0, 2],
    [1, 2, 1]
];

You can tell that this does about the same thing but the corner contribute a bit and the middle contributes more.

It looks like this (at strength 10):

The overall slopes are higher than the naive way.

Another is the Prewitt kernel:

//Prewitt
const prewittDyKernel = [
    [-1, -1, -1],
    [0, 0, 0],
    [1, 1, 1]
];
const prewittDxKernel = [
    [1, 0, -1],
    [1, 0, -1],
    [1, 0, -1]
];

This is almost the same as naive but allowing the diagonals some weight. It looks like this:

Another is the Scharr kernel:

Note: In the code it's called scharr but I found out later that this is actually called Sobel-Feldmen, Scharr is an even more pronounced

//Scharr
const scharrDxKernel = [
    [3, 0, -3],
    [10, 0, -10],
    [3, 0, -3]
];
const scharrDyKernel = [
    [3, 10, 3],
    [0, 0, 0],
    [-3, -10, -3]
];

Same sort of thing but different ratios. It looks like this:

This has even higher slopes so we might need to play with the strength to calibrate them. For example strength 1.0:

Unfortunately, this one seems to produce many more banding artifacts than the naive filter calibrated to a similar level. At this point I'm not a texture artist and I don't really have enough of an intuition on how deal with the different parameters. I'm also sure there are other filters we can add to fix things like banding. But at this point it's probably up to experimentation, we can at least build the normal maps from bump maps. So let's actually try to implement that so we can actually see the difference in 3d.

Implementing normal maps

This is actually pretty easy. It's the exact same setup as bump-mapping but with some lines removed:

//fragment.glsl
#version 300 es

precision mediump float;

in vec2 uv;
in vec3 position;
in vec3 normal;
in vec3 tangent;
in vec3 bitangent;
in vec4 color;

out vec4 fragColor;

uniform sampler2D uSampler0;
uniform mat4x4 uLight1;

void main() {

    //for normal maps
    //blue normal (UP)
    //green V (Y)
    //red U (x)
    mat3x3 tangentToWorld = mat3x3(
        tangent.x, bitangent.x, normal.x, 
        tangent.y, bitangent.y, normal.y, 
        tangent.z, bitangent.z, normal.z
    );

    vec3 colorSpaceNormal = texture(uSampler0, uv).xyz;
    vec3 tangentSpaceNormal = (colorSpaceNormal * 2.0) - 1.0;
    vec3 worldSpaceNormal = tangentToWorld * tangentSpaceNormal;

    bool isPoint = uLight1[3][3] == 1.0;
    if(isPoint) {
        //point light + color
        vec3 toLight = normalize(uLight1[0].xyz - position);
        float light = dot(worldSpaceNormal, toLight);
        fragColor = color * uLight1[2] * vec4(light, light, light, 1);
    } else {
        //directional light + color
        float light = dot(worldSpaceNormal, uLight1[1].xyz);
        fragColor = color * uLight1[2] * vec4(light, light, light, 1);
    }
}

We read the texture into the normal making sure to convert back from color-space (0 - 1) to tangent-space (-1 - 1). Then we just apply it and convert to world space like we did before.

Here's the naive:

Here's Sobel:

Here's Scharr:

Here's Prewitt:

These all look about the same for this height-map pattern, just different intensities and artifacting.

Let's make a normal map

Let's try something a little closer to real. I'll be taking the Wolfenstien 3D slime wall texture we used for specular maps and make a bump map out of it. A simple way to do this is just to convert to greyscale and threshold some values. I made a simple app using the previously created wc-glsl-shader-canvas (https://dev.to/ndesmic/exploring-color-math-through-color-blindness-2-partial-deficiency-2gbb) that let me apply a pixel shader to it:

//fragment.glsl
#version 300 es
precision highp float;
uniform sampler2D u_image;
in vec2 vTextureCoordinate;
out vec4 fragColor;
void main() {
    mat4 greyscale = mat4(
        0.5, 0.5, 0.5, 0.0, 
        0.5, 0.5, 0.5, 0.0, 
        0.5, 0.5, 0.5, 0.0, 
        0.0, 0.0, 0.0, 1.0
    );
    vec4 source = texture(u_image, vTextureCoordinate);
    vec4 target = greyscale * source;
    if(target.r > 0.5) {
        target.rgb = vec3(1.0, 1.0, 1.0);
    }
    fragColor = target;
}

This converts to rgb-space greyscale (not preceptural greyscale) and then converts anything above 0.5 to white. Because of the way the texture looks the darker areas grout areas become low and the lighter brick becomes high.

We can convert this to a normal map using our conversion:

And then apply it along with the texture:

#version 300 es

precision mediump float;

in vec2 uv;
in vec3 position;
in vec3 normal;
in vec3 tangent;
in vec3 bitangent;

out vec4 fragColor;

uniform sampler2D uSampler0;
uniform sampler2D uSampler1;
uniform mat4x4 uLight1;

void main() {
    //for normal maps
    //blue normal (UP)
    //green V (Y)
    //red U (x)
    mat3x3 tangentToWorld = mat3x3(
        tangent.x, bitangent.x, normal.x, 
        tangent.y, bitangent.y, normal.y, 
        tangent.z, bitangent.z, normal.z
    );

    vec3 colorSpaceNormal = texture(uSampler1, uv).xyz;
    vec3 tangentSpaceNormal = (colorSpaceNormal * 2.0) - 1.0;
    vec3 worldSpaceNormal = tangentToWorld * tangentSpaceNormal;

    vec4 tex = texture(uSampler0, uv);

    bool isPoint = uLight1[3][3] == 1.0;
    if(isPoint) {
        //point light + color
        vec3 toLight = normalize(uLight1[0].xyz - position);
        float light = dot(worldSpaceNormal, toLight);
        fragColor = tex * uLight1[2] * vec4(light, light, light, 1);
    } else {
        //directional light + color
        float light = dot(worldSpaceNormal, uLight1[1].xyz);
        fragColor = tex * uLight1[2] * vec4(light, light, light, 1);
    }
}

To get

Unfortunately it doesn't show up will in a low-res gif and admittedly not even that well likely due to the low resolution texture but there is some illusion that the slime and bricks are lifted off the surface especially at steeper angles.

Perhaps next time we can try with some textures authored by someone who knows what they are doing.

You can find the source code for this post here: https://github.com/ndesmic/geogl/tree/v11

Resources

https://pyimagesearch.com/2021/05/12/image-gradients-with-opencv-sobel-and-scharr/

Blog