<script> window.addEventListener('load', function () { var w = window.outerWidth; if (w > 560) document.getElementById("intro_movie").width = "560"; document.getElementById("intro_movie").height = "315"; end }); </script>

<iframe id="intro_movie" width="256" height="144" src="https://www.youtube.com/embed/SubN2Nralt8" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

This is a project page dedicated to our CVPR 2020 paper.

For more information see the pre-print version of the paper here

Motivation

The majority of cameras use rolling shutter to capture images and videos today. When a rolling shutter camera moves relative to a scene, such as during hand held video capture or a camera mounted on a vehicle, the images become distorted. These distortions cause visual as well as computational issues.

• Despite almost two decades of research, image distortions caused by camera motion remain an issue.
• It is common to have multiple cameras on a single device - smartphones, cars, head trackers.
• Few methods focus on using images from two or more cameras mounted on the same device.
• No previous method is capable of utilizing two RS images captured by cameras that are mounted close together - such as on a smartphone.

Contribution

We present a new way to

• Deal robustly with even extreme RS distortions on multi-camera devices
• Utilize those distortions to compute the motion of the device
• Remove the RS distortions caused by any type of motion
• Improve RS SfM, generate depth maps

All this is facilitated by using different readout directions for each camera. This is easily achievable e.g. by rotating one of the sensors or changing the direction of the electronical shutter.

The idea

If a device ( e.g. a smartphone) with two RS cameras moves, the images contain distortions. In current devices the cameras both use identical readout directions, which causes the distortions to look identical.

Our idea is to roll the shutters in the opposite directions.

Having such differences in the distortion allows us to compute the motion of the device from a few sparse correspondences.

The motion parameters can then be used to e.g. undistort the image:

Examples

Here is another example of identical RS readout directions:

and opposite directions:

We can warp both input images and then combine them to obtain a more complete undistorted image:

For smartphones it is typical that one camera has a different FOV due to e.g. a telephoto lens. We show that even such combination can be efficiently used:

Wide	Zoom	Undistorted

If the motion contains also translation:

We can obtain dense correspondences using e.g. optical flow and using the motion parameters we can compute the depth for each pixel:

And backproject to create and undistorted image:

Note that the quality of the result will depend on the quality of the correspondences. The few artefacts are caused by having imperfect optical flow. We did not use any post-processing methods to remove those artefacts.

We can also correct the sparse correspondences and use them in a traditional SfM pipeline to obtain a better 3D reconstruction than with the RS images.

Sparse features image 1	Sparse features image 2	Undistorted sparse features

And here is the result of the 3D reconstruction for the original RS images, our undistorted ones and images captured by a GS camera:

The method

Here is a diagram briefly describing the workflows depending on the motion we want to model.

    

License

Patent pending. For more information please contact: [email protected] .