Recently, I'm working with Varjo VR headset and had a chance to learn how to convert a fish-eye camera image to a rectangle image. The concept is very simple but I struggled for hours. So I hope this article helps other people who will do similar things.

The final result will be like this.

the photo from this page

The physics of light paths

First, we have to know the physics of light paths when we take a photo with Fish-eye lends. I searched around the web and found that the light paths are like the next image.

The blue line is a plane to take in a photo and the red line is a taken photo plane. The hemisphere which we can say as a virtual lends helps us to understand the light paths. Light paths going to the center of the hemisphere will be captured as a photo. When the light beams touches the surface of the hemisphere, they are refracted and goes to a photo image detector (red line). The angle from the vertical axis and the distance at the photo plane (red line) should be proportional.

Let's do some Math for conversion. The blue line is the result plane of the conversion. The red plane is the taken camera image. The radius of the hemisphere is R and the angle from the vertical axis is Φ (capital phi). The blue circle point on the converted image should correspond to the square red point on the camera image like the next image. Φ is the angle when the blue point is on the edge of the conversion result image. Wherever the blue point moves to, the angle should not be larger than Φ.

Let's go further. So far, I explained in 2D but we have to think in 3D. Please see the next image. Most of points are like this case. The blue point and red point should be on a line with the angle θ (theta). It makes the thing complicated but it's not too difficult.

The blue and the red points have the same θ (theta) so if we know the distance of the points from the center of each plane, the x / y position can be calculated.

For the blue point, will be like this.

For the red point, will be like this.

The distance of the blue point from its horizontal plane (r) is

The cosine and sine of θ are

The φ (lower case phi) is

The code

The code we have to write is finding the red point from the position of blue point. The final code will be like this in TypeScript. Take it easy, I will explain the details later.

• NOTE: The R in the previous images are set as 1 in my code so the R does not appear. Please try thinking why this assumption is valid by yourself!

/**
 * @param sourceArray Source image data array
 * @param sw Source image width
 * @param sh Source image height
 * @param captureAngle Catupure angle of image. Depends on camera spec. Usually it's 2 * pi.
 * @param rw Result image width
 * @param rh Result image height
 * @param resultAngle View angle of result image
 */
convert(
  sourceArray: Uint32Array,
  sw: number,
  sh: number,
  captureAngle: number,
  rw: number,
  rh: number,
  resultAngle: number
) {
    const resultArray = new Uint32Array(rw * rh)
    for (let i = 0; i < rw; i++) {
      for (let j = 0; j < rh; j++) {
        const halfRectLen = Math.tan(resultAngle / 2)
        // Result position
        const x = ((i + 0.5) / rw * 2 - 1) * halfRectLen
        const y = ((j + 0.5) / rh * 2 - 1) * halfRectLen
        const r = Math.sqrt(x * x + y * y)
        // cos(theta)
        const ct = x / r
        // sin(theta)
        const st = y / r
        // Angle from vertical axis
        const phi = Math.atan(r)
        // Source position
        const sr = phi / (captureAngle / 2)
        const sx = sr * ct
        const sy = sr * st
        // Sorce image pixel index
        const si = (sx * sw + sw) / 2
        const sj = (sy * sh + sh) / 2

        let color = 0x000000FF // Black
        if (si >= 0 && si < sw && sj >= 0 && sj < sh) {
          color = sourceArray[si * sw + sj]
        }
        result[i * rw + j] = color
      }
    }
    return resultArray
  }

I used 1D Uint32Array to store image data (sourceArray parameter and resultArray variable). (RGBA format is in integer not an array of 4 elements)

const resultArray = new Uint32Array(rw * rh)

The length of the result array should be the result of multiplication of the width and height in pixels. For source image array, the length should be relevant to the source image pixels.

To get all pixel colors of result image, using nested loops.

for (let i = 0; i < rw; i++) {
  for (let j = 0; j < rh; j++) {
    // calculation
  }
}

halfRectLen represents the half length of the result image. From the index of the pixels, the blue point (x, y) is calculated. r is the distance from the center.

const halfRectLen = Math.tan(resultAngle / 2)
const x = (i / rw * 2 - 1) * halfRectLen
const y = (j / rh * 2 - 1) * halfRectLen
const r = Math.sqrt(x * x + y * y)

The -1 is required because we assumed (x, y) if the position relative to the center but the indexes of the pixels starts from left top.

const x = (i / rw * 2 - 1) * halfRectLen

Calculates the corresponding point on the source image (x', y') in the explanation image.

// cos(theta)
const ct = x / r
// sin(theta)
const st = y / r
// Angle from vertical axis
const phi = Math.atan(r)
// Source position
const sr = phi / (captureAngle / 2)
const sx = sr * ct
const sy = sr * st

Finally, converts the source point to the indexes of source image pixels and store the color value in the result array. If the capture angle parameter is smaller than result angle, the result image should contain pixels that is not mapped in the source image. So checking if the pixel indexes are in the valid range.

const si = (sx * sw + sw) / 2
const sj = (sy * sh + sh) / 2

let color = 0x000000FF // Black
if (si >= 0 && si < sw && sj >= 0 && sj < sh) {
  color = sourceArray[si * sw + sj]
}
result[i * rw + j] = color

For getting pixel data from the camera image file and saving result image as a file, please use your own way. I'm using Jimp.

I'm not confident my explanation is easy to understand but hope this helps you just a bit.

Here's my github repo. It also contains WASM implementation written in AssemblyScript. Please check if you are interested in it. 😉