ºÚÁϸ£ÀûÍø

Global project objectives: The ELEVATION project addresses challenges in the field of Systems-of-Systems within live television and security.  Today most multi-camera networks are still CCTV and LAN based. However, more and more open network infrastructures will be used to realize true cloud-based operation. As the processing power in the cloud is virtually 'unlimited', new applications based on the transmitted image data will also be possible. Transmission of meta data from the imaging sources, securing data authenticity and keeping load factors to networks limited through smart compression are key focus points to address in order to realize cloud-based systems-of-systems for High-End Security and Broadcast. In ELEVATION, Adimec, Grass Valley, ºÚÁϸ£ÀûÍø, IntoPIX and Phoenix AI, establish and demonstrate a reference architecture with standardized connectivity for audio-video into the cloud that is useable for scalable cloud based SoS solutions in both broadcast and security.

AIMS lab innovations

The AIMS team is focusing on efficient AI-based multi-camera algorithms that build upon the secure recording and transmission technologies being developed in this project. Additionally, we are exploring efficient mapping of the multi-camera applications onto edge/cloud infrastructure for energy efficiency and real-time performance. In more details:

• Automated real-time camera calibration is required to map each image pixel to ground-plane coordinates. We are researching an automated calibration (camera pose estimation) algorithms that calculate intrinsic and extrinsic camera parameters in real time based on incoming image sequences. A combination of a Graph Neural Network or transformer-based calibration (Dust3r, VGGT) are serving as the foundation for this innovation. The automated calibration algorithms is to be validated for both the Grass Valley broadcast cameras and the Teledyne Adimec micro-cameras. Additionally, we are exploring optimal algorithmic mapping to cloud hardware.

• Video generation from user-defined viewpoints. The novel free-viewpoint generation for multi-camera setups is of high interest for broadcast and high-end security industry since it allows a scene to be viewed from a user-selected, dynamically changing perspective. The objective for us is to achieve high pixel-level accuracy, photorealistic image generation, and robust real-time performance. AIMS lab explores such AI techniques as avatar-based SMPL models, dynamic 4D gaussian splatting, frame interpolation and VGGT to create novel video feeds from user-selected viewpoints.

• 3D area mapping with passive sensors. Multi-camera based 3D reconstruction of complex dynamic scenes has proven to be the least expensive solution (e.g. compared to LiDAR or ToF-based) but also an extremely difficult one, as existing solutions often produce incomplete or geometrically-incorrect results, since moving objects, uniform and repetitive textures, and optical degradations pose significant algorithmic challenges. ºÚÁϸ£ÀûÍø develops AI models for 3D dynamic area mapping from moving vehicles, that generates a geometrically-accurate 3D model of vehicle’s surroundings in a near-real-time mode. The input data for the algorithm are feeds from several visual cameras mounted around a vehicle(s).The goals are multifold: enable near-real-time 3D reconstruction locally on a mid-range GPU mounted inside a vehicle; reach the accuracy able to provide acceptable signal-to-noise ratio of reconstruction as well as correct geometric dimensions of objects; provide algorithmic robustness to scenes with varying and unpredictable lighting conditions.