黑料福利网

SINTRA aims to improve the resilience and protection of critical infrastructure (harbours, airports, power plants) by developing an open data-streaming AI platform that enables interoperability, information sharing, and privacy protection. Using multi-modal sensing and AI-powered data analysis, it will provide a comprehensive view of the infrastructure鈥檚 safety and security and detect complex anomalies.

The Dutch SINTRA consortium focuses on safety monitoring and threat detection of specific critical infrastructure objects: logistic hubs and harbours. The main end-user in the Netherlandsis the Port of Moerdijk.

The following vital threats are addressed in the project: drugs smuggling, people smuggling, cargo burglary, and environmental attacks. These threats bring the highest impact on harbour鈥檚 operations and, at the same time, are the most challenging for detection. Besides these, threats that cause costly disruptions are also in the project target: intrusions, violence and infrastructure attacks.

Concept-wise, the project objectives of the local consortium are formulated as follows:

• Provide real-time awareness about security and safety threats on premises for the control room operators, security officers and stakeholders of logistic hubs and harbours.   
• Enable robust and accurate detection of well-hidden and context-dependent criminal actions, such as drug smuggling and people trafficking.
• Overcome the privacy-related legislation barriers on data collection/usage for successful deployment of innovative AI-based threat analysis technologies.
• Facilitate public-private cooperation, coordination and data sharing between the infrastructure owners, external stakeholders and other harbours/hubs.

SPS-AIMS group objectives

In this project, 黑料福利网 AIMS group contributes with analysis of data from the sensors of the following modalities: video, thermal, acoustic and hyperspectral. We develop AI models that enable early detection of safety and security threats based on this data. Advanced data fusion techniques are researched to improve robustness and accuracy of the analysis. Furthermore, in cooperation with partners, AIMS group is creating video and acoustic data collections that would speed up research in the AI-based security of critical infrastructure. Also, we design techniques that would enable the privacy-by-design concept in the video processing and behavioural anomaly detection. This concept will allow processing video data without breaching the privacy preservation legislation.

Our research lines

Detection of anomalous behaviour  and criminal activity. The activity examples are drug snuggling, human trafficking, infrastructure attacks (explosion, fire, digging, fence cutting), hidden unloading from small boats, loitering around, burglary. Absence of sufficient training samples for such events does not allow applying standard deep learning classification techniques. While detection of rare anomalies for a pre-trained camera view is currently successfully tackled by autoencoders, the challenge of generalization of AI model to detect anomalies in any camera views and to perform reasoning on anomalies is still a large field of research.

To address the anomaly classification problem, we research a combination of the following techniques:

• Efficient feature embedding techniques integrated with Multi Instance Learning (MIL).
• Enhancing the RGB modality with additional pseudo-modalities, such as 3D human pose, depth and textual data.
• Knowledge distillation from different pseudo-modalities.
• Integration of Video Language Models (VLM) with conventional models for anomaly detection.
• Spatial localization of anomalous events in videos.
• 3D scene-aware learning, that considers not only the pixel data from cameras, but also the geo-spatial movements of the objects of interest in the scene.

Combined analysis of multi-modal data for crime detection. Fusion of multi-source data and combined AI analysis may bring higher accuracy and robustness in detection of complex, context-dependent and well-hidden behaviour, such as subversive crime actions. For example, the drugs and people smuggling is impossible to detect by just a video-based surveillance, while with addition of directional acoustic sensors, multispectral sensors, GIS and police data, these activities can be potentially recognized. Think of a detection of migrants in a closed container by combination of directional acoustic and video sensors, complemented with logistics timetable. If visible human activity (potential organizers) around the closed container is happening within an unscheduled period and the data is complemented with an abnormal noise from the container, the AI model can compute a reliable classification score for this event.

Combined analysis of multi-modal sensor data exposes challenges on data representation, translation, alignment, fusion, and co-learning. For the sensor/data set foreseen in SINTRA, a very limited literature on the data fusion is available. However, recent joint research of 黑料福利网, Sorama and ViNotion of novel fusion approaches, based on transformers, cross-attention and mixture-of-experts, show their high potential. In SINTRA, we further research on how integrate the AIS, radar, video, acoustic, GIS data and external streams from security stakeholders into a single transformer-based AI model for detection and classification of the target threats.