AI Multi-modal Sensing

The AIMS lab researches and develops AI models for systems equipped with sensors of multiple different modalities. We foster expertise in AI analysis of RGB, thermal, depth, LiDAR, acoustic, sonar and radar sensor data. When the multi-modal sensors are combined in a sensor suite, they often provide capabilities similar to the human 鈥�5-sense system鈥�, which bring the desired full situational awareness. This awareness is vital in our industrial partners in public safety & security, smart cities, defense, critical infrastructure inspection and intelligent transportation.

Research Profile

The AIMS lab specializes in advanced artificial intelligence methods for multi-modal sensor systems, enabling machines to perceive and interpret complex environments in real-time through the fusion of heterogeneous data sources. The core objective of the lab is to advance the real-time fusion and interpretation of multi-modal data through cutting-edge AI techniques. The research focuses on few-shot, unsupervised and self-supervised learning, casual reasoning, 4D gaussian splatting, embodied AI, and efficient edge deployment. By integrating these approaches, the lab develops methods capable of detecting events across modalities, localizing threats and objects in 3D space, and identifying abnormal patterns without requiring extensive labeled datasets.

AIMS group is involved as an initiator and research coordinator in several large European research projects, such as STRUCTURE, MariSens, SINTRA, ELEVATION, ADVISOR, etc. We have organized valorization and exploitation of the AI Models developed in such projects at ST Microelectronics (France), ViNotion (Netherlands), INRIA (France) and Port of Moerdijk (Netherlands). Our collaboration network includes 120+ companies and governmental organizations, such as Fugro, Avular, Antea Group, NXP, ASML, Bosch, Demcon, Esri Canada, TAV Turkey, Ministry of Defense, Rijkswaterstaat.

In academia, we conduct our research in close collaboration with Mechanical Engineering, Computer Science and Industrial Engineering departments at 黑料福利网. Externally, we work together with such research institutions as Reality Labs at Meta, MARIN, Inria, TNO, SIRRIS, as well as with Universities of Munich, Delft, Maastricht, Liege, Birmingham and Gent.

Multi-sense Perception for Situational Awareness

Combination of sensors of different modalities enables robust perception and high utility in many application areas. A system able to analyze sound sources, detect events at night and day, in rain/fog conditions, and localize objects of interest in 3D space is a dream for owners of critical infrastructure, transportation systems, defense and public safety systems.

The objective of the AIMS lab is to explore and learn how the multi-modal data can be processed and fused together by AI technologies to enable situational awareness in real-time. For this, the lab pushes the frontiers in unsupervised machine learning, Video Language Models (VLM), 3D scene reconstruction, anomaly analysis and edge AI. Our grand challenges in multi-modal sensor fusion are:

Automation in spatio-temporal registration of different modality data;
Distillation and fusion of relevant data from multiple sensor types;
Detection of anomalies without training data on such anomalies;
Holistic AI analysis of 3D area as a whole, instead of individual image/ signal analysis;
Enabling explainability in AI models.