Image registration and mosaicing
Automatic real time image mosaicing
Registration and mosaicing of images started to be considered at the half
of the Nineteenth century to overlap aerial photographs. When the first satellites
started sending Earth images back, mosaicing became a need. Improvements in
computer technology have given a boost to research in mosaicing. Mosaicing
techniques have many application fields, ranging from computer graphics to
medical imaging and, in general, whenever enhancing either image resolution
or field of view is required.
Our research addresses real time methods that are fully image-based and
self-calibrating, meanwhile achieving high quality results typical of global
registrations. The solution we propose does not exploit any priori information
regarding scene geometry, acquisition device properties or feedback signals,
thus resulting in a fully image based, hence general purpose, solution.
Fig. 1: Mosaic build with (right) and without (left) our back registration.
The algorithm implemented performs in real time independently of the scenes being mosaiced,
therefore we have focussed our attention on the quality of the outcome, that could depend
on the scene being shot. To this purpose, several image sequences have been considered to
assess the performance of our approach and Fig. 2 reports the outcome concerning three different environments.
Fig. 2: First row: a classic panorama, achieved by mosaicing a few tens frames. Second row: mosaic of our lab, indoor (left), the plan highlighting the camera's field of view (middle), a color mosaic with a narrower field of view (right). Third row: the outside of our lab (left) and the sequence of frames to generate the mosaic (right).In the first row, a panorama probably represents the easiest of the test cases considered, since parallax effects are usually absent. However, although a long sequence of frame is considered, the mosaic is excellent with no drift errors. In the second row, the inside of our lab achieved in gray levels with pure pan (left) and in colours (right) with pan and tilt is acquired in quite a uniform light environment. However, reconstructing the set of highly changing depths (in the middle, the camera's field of view) represents a strong challenge that is won using back registration. Finally, the third row shows the most challenging environment, acquired using the PTZ of the video surveillance system we set up. The scene is outdoors, with a high dynamic range and highly structured. Actually, the mosaic is built using about 250 frames, overalapping more than 90% (the frames map is on the right). Although the entrance door jamb (on the left) and the gate (on the right) are placed at 1 and 30 meters, respectively, from the camera, our back registration approach together with our joint spatial and tonal registration [M1] prevents any error drift in the mosaic, that appears of an excellent quality.
Joint spatial and tonal registration
Real time mosaicing techniques often deal with spatial registration only,
while tonal registration is been often left out of consideration, because
of the absence of reliable algorithms compliant with the real time requirements.
However, mostly in case that the mosaic must to be kept up-to-date, global
scene illumination changes due to either environmental causes or camera
integrated Automatic Gain Control (AGC), autoiris or autoshutter yield
corresponding regions, framed at subsequent time, to look as being different.
Accordingly, in order to be used effectively, general purpose image mosaicing
requires to consider the overall alignment problem in a higher dimensional
space rather than the geometric 2D space, so that the frames making up the
mosaic undergo the same illumination.
Fig. 1: Example of images (at the farthest left, the original ones) histogram specified using our method with synthetic target histograms (linear, Gaussian and logarithmic, from left to right, respectively).
METHOD In this research we have developed an innovative exact specification method that is compliant with real time requirements. Practically speaking, our method offers the following features:
Jointly to spatial registration, this constitutes the core of our algorithm to achieve automatically a background mosaic without exploiting any prior information regarding the scene or the acquisition device, despite of the presence of moving objects in the scene [M1].
RESULTS We have carried out comparative experiments using the well known Baboon and Boat images. Fig. 2 reports the results of quasi-exact methods that are "more similar" to our, in terms of speed performance.
Fig. 2: At the top left, the original source PDF. From top to bottom, results with the classic method, Heramian's and our's when target PDFs are linear, Gaussian and logarithmic (from left to right, respectively).In order to assess quality and time performance of our method we propose the indicators we have implemented, that is computational speed, histogram distortion and image distortion. Although, here we just use 1-dim feature (i.e. gray levels) for the source histogram, our algorithm attains the same quality performance as the reference exact methods. Besides, it always achieves exact target histograms where other methods fail. Nevertheless, our algorithm runs two orders of magnitude faster than the reference exact methods and more than one order if compared with the other quasi-exact approaches. We can thus conclude that our method shows a very low image distortion (see Fig. 1), while achieving the exact specification (Fig. 2) faster than the other methods do. More details are reported in [M2].
In image registration, the effects of our methods are more clearly perceptible.
The lack of tonal registration is visible in Fig. 3, left, indoors (top) and
Indoors, one can see how the joint registration (right) can improve
mainly the geometric properties of the mosaic, adjusting its deformation.
Outdoors, when the changing lighting conditions could be heavy, the lack of our tonal registration makes
a mosaic unusable, where single frames keep their original illumination (left). On the contrary,
our approach faces the heavy lighting differences, thus permitting to build a mosaic
even outdoors (right), with very impressive results. A more extensive analysis is reported
Fig. 3: Mosaic with spatial registration only (left) and with joint tonal and spatial registration (right). Indoors (top), the even illumination bounds the effects of the tonal registration on the deformation adjustments. Outdoors, the changing lighting condition are visible in the single frames composing the mosaic on the left. Here, our method permits to build an effective seamless mosaic (right).
Motion Detection with Pan-Tilt-Zoom Cameras using Background Mosaics
Scene segmentation between background and foreground (moving) regions represents
the first stage of many applications such as visual surveillance, traffic monitoring
and human activity understanding. The background subtraction technique (BST) is known
as being the one yielding the highest quality of the detected moving masks when using
one stationary camera. Using this approach basically means comparing the current frame
with a reference scene (a previously computed background). Moving "blobs"
(aggregate of pixels) are identified by thresholding these differences.
Our group proposes an innovative solution to achieve a real time colour [M3] mosaic background
apt to work with existing colour background subtraction algorithms to yield excellent foreground
object masks [M4]. The two-stage real time algorithm we have developed for joint spatial and
tonal registration [M1], in order to account even for photometric misalignments induced
by varying light conditions and exposure, prevents error accumulation and permits to construct
globally coherent mosaics, leading to near optimal results even without resorting to computing
intensive global adjustment procedures.
Some indoor and outdoor panning and tilting challenging case studies have been considered.
In Fig. 1, three different scenes depict one person walking indoors (first row), outdoors
(second row) and two people tracked outdoors (third row).
Fig. 1: Walking people are detected using a background mosaic. First row: Motion detection in our lab, panning and tilting with small angles. Second row: a person is detected outdoors, with a high accurate mask, while panning manually. Third row: example of automatic people tracking, in our surveillance system, with wide pan and tilt angles.People are detected through background subtraction, where the background is the mosaic built using our joint spatial and tonal approach. In the first row, the camera is moved according to narrow pan and tilt movements while in the second row the camera manually pans the scene. The moving masks thus detected have been compared with those achieved by a fixed camera employing a static background, and substantially they have shown same quality. The third row shows two people tracked in our surveillance system (in the right frame the person comes into the scene after exiting). As a conclusion, we can state that the quality of the mosaics attained permits a novel approach in motion detection, based on background difference even with PTZ cameras. Moreover, this approach is the first in literature used even for automatic tracking.
[M1] P. Azzari, A. Bevilacqua, Joint Spatial and Tonal Mosaic Alignment for Motion Detection with PTZ Camera, Lecture Notes in Computer Science (LNCS), Vol. 4142 (2006) 764-775
[M2] A. Bevilacqua, P. Azzari, A High Performance Exact Histogram Specification Algorithm, 14th International Conference on Image Analysis and Processing (ICIAP 2007), September 10-14, 2007, Modena, Italy, pp.623-628
[M3] A. Bevilacqua, P. Azzari, High-quality real time motion detection using PTZ cameras, IEEE International Conference on Advanced Video and Signal based Surveillance (AVSS 2006), November 22-24, 2006, Sidney, NSW, Australia, pp.23-29
[M4] A. Bevilacqua, P. Azzari, A Fast and Reliable Image Mosaicing Technique with Application to Wide Area Motion Detection, Lecture Notes in Computer Science (LNCS), Vol. 4633 (2007) 501-512
Copyright © 2008, A.G. - All Rights Reserved