Depth perception in robotics (RGB-D). VSLAM use-case included

Is depth perception expensive?

Depth perception is the core component of modern robotic systems regarding path planning, obstacle avoidance, localization, mapping or more general scene understanding. It is challenging because it requires specialized active sensors like, e.g., Light Detection and Ranging (LiDAR) modules or multiple passive Red Green Blue (RGB) cameras organized in one system to estimate the depth from color images. However, while LiDARs have a lot of advantages, they also suffer from some drawbacks, i.e., they can be too expensive for amateur robotics enthusiasts and are ineffective in challenging environmental conditions (e.g., fog, dust, or rain). On the other hand, estimating the depth using multiple RGB cameras holds the drawbacks of these sensors, e.g., it requires a tracking system of some key points in images, and they cannot be registered on plain textures. Hence, depth estimation would break on such surfaces. 

But what if we could overcome problems with RGB cameras while maintaining affordable prices below the LiDAR level? The answer is RGBD camera, where the D stands for depth. It is a type of sensor that can utilize multiple techniques to estimate the depth, like mentioned stereo vision and structured light or could work in a way LiDARs work - using the time of flight of the emitted light beam. All that is affordable, increasing the accessibility of robotics sensors for all robotics enthusiasts, not only professionals. The following post will explain depth cameras, what technology they use to estimate the depth and how we can use them in robotics, namely speaking in Visual Simultaneous Localization and Mapping (VSLAM) (see our post about SLAM). Additionally, we demonstrate such a SLAM algorithm based on RGBD data in a real-world example in our office in Poznań. 

What is a depth map?

A depth map represents the distance between a scene and the camera that captured it in the form of an image, where each pixel has an associated value of how far it lies in the 3D space. Depth maps are typically used in robotics and computer vision for autonomous navigation and related tasks, like object detection or segmentation. As mentioned, we can create them using specialized hardware such as depth sensors or estimates from multiple images. Often, depth images are represented as a single channel grayscale image, where every pixel represents the object's distance to a camera, and the most popular format is a 16-bit unsigned integer. However, it is also possible to visualize a depth map using color maps where different colors represent different distances from the camera, e.g., close objects are bluer, while the background goes towards red. Using color to represent depth can make it easier to interpret the depth map, especially when working with complex 3D scenes. When dealing with depth images, it's essential to recognize a proper numerical format in which pixels are encoded because it affects our perception of distance - we have to know which pixel values represent what distance. The picture below presents an example of the depth map represented as a grayscale image with distances encoded in a colormap.