黑料福利网

Enabling Navigation with Indoor Guidance using Map-based Situational Awareness
Objectives: Soldiers entering buildings in hostile environments expose themselves and their team to potentially dangerous situations. Such an action imposes risks and requires good cooperation and continuous communication with the commander and within the team. At present, military troops in unknown indoor environments are coordinated by radio communication, where at best case, situation assessment is based not only on voice descriptions over radio channels but also on some visual observations. This limited communication provides very low global situational awareness for coordinators, as well as for the individual soldiers, which lowers the efficiency of operations and may cause even unexpected accidents. For more information: .

Within the Enigma NTP project, together with Ministry of Defense NL and Vinotion, we investigate the use of a wearable intelligent sensor that enables the automatic creation of a map of the building, while a soldier navigates this unknown indoor environment. To this end, we develop a wearable Visual Simultaneous Localization and Mapping (SLAM) system that streams scene information, such as on-the-fly created floorplans, to the coordinator on a remote location. This improves the situational awareness during operation and enables the coordinator to make more informed decisions. 

The goal of the Enigma project is the development of a prototype that uses wearable sensors that are capable of sensing and reconstructing the 3D indoor environment in which the soldier is moving. Moreover, the 3D reconstruction of the environment will be extended automatically while the soldier is exploring the building and penetrates further into the building. Although it is possible to automatically merge the reconstructed 3D environment acquired by multiple soldiers, this has been considered outside the scope of the current project. Apart from the reconstruction of the 3D spaces, also doors and windows will be automatically detected and classified as such. 

Passive Sensor harness mount: To enable the soldier to carry the camera sensor and processing unit (laptop), we have created a mount that holds the camera at the height of the soldier head while not obstructing his view. The mount is made from iron and is integrated in the military vest for stability.

It allows variations in height and width in terms of sensor location. It enables mounting of one or more cameras and has been used to make the recordings at the Harskamp military terrain. 
 

In the Enigma project, we use ZED stereo camera as an imaging sensor. The computed depth images serve as an input to the SLAM system. Actually, the depth pixels are the 3D points that will be integrated into the global 3D point cloud frame-by-frame. Besides this, we integrate the internal ZED IMU sensor data into the SLAM process to recover from the situations when visual odometry fails due to sharp movements or insufficient depth map details.

System Architecture: Together with Vinotion, we have designed the system architecture that targets real-time 3D reconstruction of unknown environments with ZED stereo cameras. The architecture incorporates multiple on-body harness mounts, the SLAM (localization and mapping) components, the data transmission and the commander site. The reconstruction pipeline shown below the outputs of each of the main components. 

3D semantic reconstruction pipeline for indoor environments: There are three major important aspects that prevent using the obtained 3D point clouds as is for the visualization on the coordinator鈥� device. First, the depth noise generated by any passive 3D sensor is producing high volumes of noisy points in the reconstructed 3D maps. The noisy points are the points located in the area with no real physical objects (i.e. in empty areas). Second, the depth noise leads to reconstruction of improper shapes of real physical objects on the 3D map. The surfaces of walls, floors, ceilings, doors, etcetera, are smeared out and therefore the objects become very thick. As an example, thickness of floors in the reconstructed models varies from 0.5 to 1.5 meter. Wall thickness usually varies from 0.3 to 1 meter. Finally, the dense point cloud representations of ceiling and floors on the 3D model occlude the actual spaces inside the rooms from viewing and do not allow to see the actual structure of the building internals. 

These three aspects lead to an incorrect 3D point cloud model, where the internal structures of a building (rooms, corridors, doors) are hidden behind the ceilings, external walls and floors. It is impossible for a coordinator to navigate through a building and understand the actual shapes and dimensions of the objects and rooms inside. To solve this problem, we have integrated semantic information and developed a vertical filtering method for the original 3D point cloud. 

Step 1: Depth generation from stereo pair for each frame: ZED cameras have been designed for indoor sequences and specifically also for use in/on virtual reality headsets. The cameras itself only deliver video output (synchronized left/right images) but provides a Software Development Kit (SDK) for depth computation and 3D reconstruction.

Step 2: 2D semantic segmentation:To perform an accurate semantic segmentation of indoor scenes we have adopted and re-trained on defense imagery the ResNeSt-based DeeplabV3 model. We have recorded and annotated two datasets, one at the university campus and one at the Harskamp military base, using the camera that is used for the project, to provide the most realistic training data.

Step 3: Semantic information integration: Provides a semantic label to each point in the dense point cloud (we call this points cloud 鈥渞aw_PC鈥�). Next, we extract semantic information from a sparse point cloud (we call this points cloud 鈥渟eg_PC鈥�), which is the output of the 鈥渟emantic segmentation鈥� component. We notice that the semantic labels in 鈥渟eg_PC鈥� are noisy. Moreover, the wrong classified labels influence the result of semantic integration. Therefore, semantic labels in 鈥渟eg_PC鈥� need refinement. The idea is to refine the label of a point by the majority label of its neighbors. We compare four possible implementations: a) Octree data structure with K nearest neighbors (KNN) search. b) KDtree data structure with KNN search. c) Octree data structure with radius search. d) KDtree data structure with radius search. After comparing the execution time, we decide to use the KDtree structure with radius search. 

Step 4: Point cloud filtering: Clean up of the raw noisy point-cloud data targets detection of columns that represent vertical structures of the building: walls, doors, windows. When we detect such columns, the points in the other columns can be safely omitted from the 3D point cloud, thereby leaving only the points that define the structural map of the building. The detection of these structural vertical columns is based on a simple 3D-to-2D filtering concept. Each 3D column holds a specific number of points. We compute the amount of belonging point for each column. Then we apply a sliding mask of 5x5 cells onto the cell grid. The mask is moving from left to right and from top to bottom over the cell grid and computes a value for its central cell located at the cell coordinates [3,3]. If, according to specified filtering criteria, the computed value is higher than a specified threshold, then the cell is kept remaining as a part of the building structure (if not, the point located inside the cell are removed from the point cloud). We apply various filtering criteria 鈥� mean-based filtering, standard deviation filtering, even convolutional filtering can be applied. 

Step 5. Generation of line-based floorplan: The source for the floor mapping is the resulting vertically filtered 3D semantic point cloud. For generation of accurate and clean floor maps, we perform the following processing steps:

• Automatically extract lines from an uneven 2D grid map to represent vertical structures (walls, windows and doors).

• Remove unexpected lines as much as possible.

• Integrate semantic information into detected lines.

• Define 3D planes based on detected 2D semantic lines. 

• Connect individual wall segments that are actually represent one continuous wall (fill in the holes).

• Make walls as straight as possible, when possible, even specify a wall by a simple line.

The resulting real-time reconstruction pipeline is depicted in the resulting steps shown in the images.