Monocular calibration

This document outlines the steps needed to calibrate a monocular camera using the library.

At a high level, our goal in camera calibration is to estimate a model that can transform points between the camera space (3D) into the image space (2D). This is often represented by a camera matrix and a distortion coefficients vector, but the compas_camcal library is extensible to other models.

In the standard pinhole Camera model, the camera matrix A and distortion coefficients d are defined as:

\[ A = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} \]

\[ d = \begin{bmatrix} k_1 & k_2 & p_1 & p_2 & k_3 & k_4 & k_5 & k_6 & s_1 & s_2 & s_3 & s_4 & \tau_x & \tau_y \end{bmatrix} \]

Note

This document assumes you have a basic understanding of camera calibration. If you’re new to the topic, we recommend reading the OpenCV camera calibration tutorial first.

Collect a calibration dataset

The first step of the calibration process is to collect a set of images of a target of known size and structure. Typically, this is a uniform, rectangular black & white “chessboard” pattern. The extracted structure is the corners between the chessboard squares. For monocular calibration, the size of the squares (distance between corners) does not matter, but for stereo calibration the square size is required to estimate the camera baseline.

The calibration dataset should “excite” the calibration optimization by providing a good variety of viewpoints. The viewpoints should vary both the position and orientation of the target relative to the camera. As a rule of thumb, 12-20 images should be sufficient for this purpose. These images are the input for the next step: finding the chessboard patterns.

Find correspondences

Next, we need to establish a correspondence between the 3D object points (the chessboard “structure”) and the seen 2D image points. So, we need to:

establish the 3D object structure, and
find the 2D image points in our images.

The end result of this process should be a 1-to-1 correspondence between a set of 3D object points and 2D image points as perceived by the camera, from which the calibration procedure can estimate the intrinsic parameters of a given camera model.

Using `cv-find-chessboards`

A command-line utility script is included in the compas_camcal Python package for finding these correspondences in images of chessboard patterns: cv-find-chessboards.

To invoke the script, we pass the chessboard structure, size, and a collection of images to process:

cv-find-chessboards [-h] [-o OUTPUT] [-j JOBS] [--overwrite] [-v] rows columns size [image_path ...]

The positional arguments are given as:

rows: The number of rows in the chessboard by (interior corners).
columns: The number of columns in the chessboard (interior corners).
size: The size (side length) of the chessboard squares.
image_path: ****The path to an image.

Any number of images can be provided.

The options are:

-h/--help: Show a help message and exit
-o OUTPUT/--output OUTPUT: The directory to save the correspondence files to. Default: correspondences
-j JOBS/--jobs JOBS: The number of worker processes to use. Default: 1
--overwrite: Whether to overwrite existing correspondence files or skip them.
-v/--verbose: Increase the verbosity level.

For large collections of images, it’s recommended to parallelize with several worker processes by specifying the -j/--jobs option.

The output correspondences are named by matching the image filename and appending the .npz extension. For example, an image image.png will have its correspondence saved to correspondences/image.png.npz by default. This file simply contains a serialized Numpy array of the 3D object points and corresponding 2D image points.

Note that this script can be interrupted and resumed. The --overwrite flag can be specified if you want to overwrite existing correspondences. Otherwise, the default behavior skips images with existing saved correspondences (matched by filename) to save time.

Info

Under the hood, this process is using cv::findChessboardCorners to detect the chessboard corners and cv::cornerSubpix to refine corners.

Using the library

The cv-find-chessboards source code is a good reference on the process of correspondence finding. Expand the following section to see the full source.

cv-find-chessboards source code

Create a `CorrespondenceFinder`

First, we need to create a CorrespondenceFinder object. This object is responsible for finding the 3D object points and 2D image points in a given image. The ChessboardCorrespondenceFinder is initialized with the chessboard structure and size:

finder = ChessboardCorrespondenceFinder(
    rows, 
    columns, 
    size, 
    flags=cv2.CALIB_CB_ADAPTIVE_THRESH | cv2.CALIB_CB_NORMALIZE_IMAGE
)

The flags argument is optional and is passed to cv::findChessboardCorners. The recommended value is cv2.CALIB_CB_ADAPTIVE_THRESH | cv2.CALIB_CB_NORMALIZE_IMAGE.

Find a correspondence

With a given image, invoke the find method to find the 3D object points to 2D image points Correspondence:

correspondence = finder.find(image)

Loading images

Images are assumed to be Numpy arrays. The find method will automatically convert images to grayscale, so it’s not necessary to do so beforehand.

Typically, images are loaded using cv2.imread:

image = cv2.imread(image_path)

If the chessboard pattern is not found in the image, None is returned.

Save the correspondence

The Correspondence object can be serialized to a file using the save method:

correspondence.save(output_path)

Calibrate the camera intrinsics

Once we have a collection of object-image point correspondences established for the dataset, we need to estimate the camera intrinsic parameters.

Using `cv-mono-calibrate`

A separate command-line utility script is included in the compas_camcal Python package for calibrating a monocular camera from saved correspondences: cv-mono-calibrate.

To invoke the script, we pass the image width, image height, and a collection of saved correspondence patterns to use:

cv-mono-calibrate [-h] [-k K] [-r] [-p] [-t] [-f {json,human}] [-v] width height [correspondence_path ...]

The positional arguments are given as:

width: The width of the image in pixels.
height: The height of the image in pixels.
correspondence_path: The path to a correspondence file

Any number of correspondences can be provided.

The options are:

-h/--help: Show a help message and exit
-k K: The number of correspondences to use. If negative, use all correspondences.
-r/--rational: Enable the cv::CALIB_RATIONAL_MODEL flag.
-p/--thin-prism: Enable the cv::CALIB_THIN_PRISM_MODEL flag.
-t/--tilted: Enable the cv::CALIB_TILTED_MOEL flag.
-f {json,human}/--format {json,human}: The format to print the calibration in. Default: human.
-v/--verbose: Increase the verbosity level.

For large collections of correspondences, it’s recommended to use K-means correspondence selection (based on extracted convex hull features; read more in the docs) by specifying the -K option.

Info

Under the hood, this process is using cv::calibrateCamera to perform the optimization.

Using the library

The cv-mono-calibrate source code is a good reference on the process of monocular camera calibration. Expand the following section to see the full source.

cv-mono-calibrate source code

Create a `MonocularCalibrator`

First, we need to create a MonocularCalibrator object. This object is responsible for performing the optimization to estimate the camera intrinsic parameters. The OpenCVMonocularCalibrator is initialized with the image width and height, as well as the flags to pass to cv::calibrateCamera:

flags = cv2.CALIB_RATIONAL_MODEL  # e.g. for a rational model

calibrator = OpenCVMonocularCalibrator(width, height, flags)

Calibrate the camera

With a given collection of correspondences, invoke the calibrate method to estimate the camera intrinsic parameters:

camera, stats = calibrator.calibrate(correspondences)

Loading correspondences

Correspondences can be loaded from saved .npz files using Correspondence.load. For example:

correspondence = Correspondence.load(path)

The resulting Camera instance contains the estimated camera intrinsic parameters. The calibrate method also produces a MonocularCalibrationStats instance containing summary calibration statistics and the estimated correspondence extrinsics.

Serialize the calibrated camera

The Camera object can be serialized to a file using the to_dict and to_json methods:

print(camera.to_dict())
...
{'intrinsics': {'fx': 2782.982470742835, 'fy': 2485.6043327516977, 'cx': 510.7105362901761, 'cy': 502.68586206232476, 's': 0.0}, 'distortion': {'k1': -2.152013767252389, 'k2': -23.465448935340486, 'p1': 0.12324709201954602, 'p2': -0.03774260720592705, 'k3': 338.5223638845042, 'k4': 0.0, 'k5': 0.0, 'k6': 0.0, 's1': 0.0, 's2': 0.0, 's3': 0.0, 's4': 0.0, 'tx': 0.0, 'ty': 0.0}, 'extrinsics': {'rotation': {'matrix': [[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]]}, 'translation': {'x': 0.0, 'y': 0.0, 'z': 0.0}}}

print(camera.to_json(indent=2))

{
  "intrinsics": {
    "fx": 2782.982470742835,
    "fy": 2485.6043327516977,
    "cx": 510.7105362901761,
    "cy": 502.68586206232476,
    "s": 0.0
  },
  "distortion": {
    "k1": -2.152013767252389,
    "k2": -23.465448935340486,
    "p1": 0.12324709201954602,
    "p2": -0.03774260720592705,
    "k3": 338.5223638845042,
    "k4": 0.0,
    "k5": 0.0,
    "k6": 0.0,
    "s1": 0.0,
    "s2": 0.0,
    "s3": 0.0,
    "s4": 0.0,
    "tx": 0.0,
    "ty": 0.0
  },
  "extrinsics": {
    "rotation": {
      "matrix": [
        [
          1.0,
          0.0,
          0.0
        ],
        [
          0.0,
          1.0,
          0.0
        ],
        [
          0.0,
          0.0,
          1.0
        ]
      ]
    },
    "translation": {
      "x": 0.0,
      "y": 0.0,
      "z": 0.0
    }
  }
}

Monocular calibration

Collect a calibration dataset

Find correspondences

Using cv-find-chessboards

Using the library

Create a CorrespondenceFinder

Find a correspondence

Save the correspondence

Calibrate the camera intrinsics

Using cv-mono-calibrate

Using the library

Create a MonocularCalibrator

Calibrate the camera

Serialize the calibrated camera

Using `cv-find-chessboards`

Create a `CorrespondenceFinder`

Using `cv-mono-calibrate`

Create a `MonocularCalibrator`