Image-based 3D reconstruction of "generalized box" with user sketches

Overview

This project deals with reconstructing of 3D models of real-world objects from a number of images. The shape of the object is called "generalized box" and describes objects with hexahedron shape, which edges are polylines or curves and faces are curved surfaces. The class of objects described as generalized box is rather wide. It includes technical equipment, machinery or their parts, architectural buildings, a host of common everyday objects.

Modeling process

We have developed a 3D modeling application, which guides the user through the modeling process.

The whole modeling framework consists of three steps: camera calibration, 3D modeling with user sketches, and texturing. Camera calibration data can be imported or computed by producing automatic pattern-based calibration.

Figure 1: GUI of our application - Calibration Step.

In modeling step user should outline at least three edges of the generalized box on at least two images. 3D model is interactively constructed if possible. All twelve edges are numbered, thus user should select edge number first. It is done by clicking at the corresponding edge of the help picture.

Each edge is represented as polyline. All editing functions for polylines are provided. A smart auto selection of next edge for editing is developed. It is based on typical order of drawing edges on the current image. It tries to select next edge in the same face as the last edited edge. In case of collision the edge is selected in the face with the largest number of already outlined edges.

Figure 2: GUI of our application - Modeling Step.

Algorithms

First step of reconstruction is building a curved 3D edge, corresponding to input 2D edges. We have proposed a new approach for estimating the correspondence between points of several 2D polylines or curves and constructing the according 3D edge.

If user outlined only several edges, we should construct missing edges. It is done with shifting and interpolating existing edges in faces with two or three already constructed 3D edges.

After all edges are constructed the full polygonal model can be built. It is done independently for each of six faces by discrete coons patches algorithm.