TY - INPR A1 - Song, Rui A1 - Liang, Chenwei A1 - Cao, Hu A1 - Yan, Zhiran A1 - Zimmer, Walter A1 - Gross, Markus A1 - Festag, Andreas A1 - Knoll, Alois T1 - Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles N2 - Collaborative perception in automated vehicles leverages the exchange of information between agents, aiming to elevate perception results. Previous camera-based collaborative 3D perception methods typically employ 3D bounding boxes or bird's eye views as representations of the environment. However, these approaches fall short in offering a comprehensive 3D environmental prediction. To bridge this gap, we introduce the first method for collaborative 3D semantic occupancy prediction. Particularly, it improves local 3D semantic occupancy predictions by hybrid fusion of (i) semantic and occupancy task features, and (ii) compressed orthogonal attention features shared between vehicles. Additionally, due to the lack of a collaborative perception dataset designed for semantic occupancy prediction, we augment a current collaborative perception dataset to include 3D collaborative semantic occupancy labels for a more robust evaluation. The experimental findings highlight that: (i) our collaborative semantic occupancy predictions excel above the results from single vehicles by over 30%, and (ii) models anchored on semantic occupancy outpace state-of-the-art collaborative 3D detection techniques in subsequent perception applications, showcasing enhanced accuracy and enriched semantic-awareness in road environments. UR - https://doi.org/10.48550/arXiv.2402.07635 Y1 - 2024 UR - https://doi.org/10.48550/arXiv.2402.07635 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-46035 PB - arXiv CY - Ithaca ER - TY - JOUR A1 - Song, Rui A1 - Xu, Runsheng A1 - Festag, Andreas A1 - Ma, Jiaqi A1 - Knoll, Alois T1 - FedBEVT: Federated Learning Bird's Eye View Perception Transformer in Road Traffic Systems JF - IEEE Transactions on Intelligent Vehicles N2 - Bird's eye view (BEV) perception is becoming increasingly important in the field of autonomous driving. It uses multi-view camera data to learn a transformer model that directly projects the perception of the road environment onto the BEV perspective. However, training a transformer model often requires a large amount of data, and as camera data for road traffic are often private, they are typically not shared. Federated learning offers a solution that enables clients to collaborate and train models without exchanging data but model parameters. In this paper, we introduce FedBEVT, a federated transformer learning approach for BEV perception. In order to address two common data heterogeneity issues in FedBEVT: (i) diverse sensor poses, and (ii) varying sensor numbers in perception systems, we propose two approaches - Federated Learning with Camera-Attentive Personalization (FedCaP) and Adaptive Multi-Camera Masking (AMCM), respectively. To evaluate our method in real-world settings, we create a dataset consisting of four typical federated use cases. Our findings suggest that FedBEVT outperforms the baseline approaches in all four use cases, demonstrating the potential of our approach for improving BEV perception in autonomous driving. UR - https://doi.org/10.1109/TIV.2023.3310674 Y1 - 2023 UR - https://doi.org/10.1109/TIV.2023.3310674 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-42789 SN - 2379-8904 SN - 2379-8858 VL - 9 IS - 1 SP - 958 EP - 969 PB - IEEE CY - New York ER - TY - CHAP A1 - Agrawal, Shiva A1 - Song, Rui A1 - Kohli, Akhil A1 - Korb, Andreas A1 - Andre, Maximilian A1 - Holzinger, Erik A1 - Elger, Gordon ED - Ploeg, Jeroen ED - Helfert, Markus ED - Berns, Karsten ED - Gusikhin, Oleg T1 - Concept of Smart Infrastructure for Connected Vehicle Assist and Traffic Flow Optimization T2 - Proceedings of the 8th International Conference on Vehicle Technology and Intelligent Transport Systems N2 - The smart infrastructure units can play a vital role to develop smart cities of the future and in assisting automated vehicles on the road by providing extended perception and timely warnings to avoid accidents. This paper focuses on the development of such an infrastructure unit, that is specifically designed for a pedestrian crossing junction. It can control traffic lights at the junction by real-time environment perception through its sensors and can optimize the flow of vehicles and passing vulnerable road users (VRUs). Moreover, it can assist on-road vehicles by providing real-time information and critical warnings via a v2x module. This paper further describes different use-cases of the work, all major hardware components involved in the development of smart infrastructure unit, referred to as an edge, different sensor fusion approaches using the camera, radar, and lidar mounted on the edge for environment perception, various modes of communication including v2x, system design UR - https://doi.org/10.5220/0011068800003191 KW - Road Side Unit (RSU) KW - Smart Infrastructure KW - Radar KW - LiDAR KW - Camera KW - Perception KW - Sensor Data Fusion KW - Vehicle-to-everything (V2X) KW - Communication KW - Simulation KW - ITS Y1 - 2022 UR - https://doi.org/10.5220/0011068800003191 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-30926 SN - 978-989-758-573-9 SN - 2184-495X SP - 360 EP - 367 PB - SciTePress CY - Setúbal ER - TY - INPR A1 - Song, Rui A1 - Lyu, Lingjuan A1 - Jiang, Wei A1 - Festag, Andreas A1 - Knoll, Alois T1 - V2X-Boosted Federated Learning for Cooperative Intelligent Transportation Systems with Contextual Client Selection N2 - Machine learning (ML) has revolutionized transportation systems, enabling autonomous driving and smart traffic services. Federated learning (FL) overcomes privacy constraints by training ML models in distributed systems, exchanging model parameters instead of raw data. However, the dynamic states of connected vehicles affect the network connection quality and influence the FL performance. To tackle this challenge, we propose a contextual client selection pipeline that uses Vehicle-to-Everything (V2X) messages to select clients based on the predicted communication latency. The pipeline includes: (i) fusing V2X messages, (ii) predicting future traffic topology, (iii) pre-clustering clients based on local data distribution similarity, and (iv) selecting clients with minimal latency for future model aggregation. Experiments show that our pipeline outperforms baselines on various datasets, particularly in non-iid settings. UR - https://doi.org/10.48550/arXiv.2305.11654 Y1 - 2023 UR - https://doi.org/10.48550/arXiv.2305.11654 UR - http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:bvb:573-41441 PB - arXiv CY - Ithaca ER - TY - CHAP A1 - Jagtap, Abhishek Dinkar A1 - Song, Rui A1 - Sadashivaiah, Sanath Tiptur A1 - Festag, Andreas T1 - V2X-Gaussians: Gaussian Splatting for Multi-Agent Cooperative Dynamic Scene Reconstruction T2 - 2025 IEEE Intelligent Vehicles Symposium (IV) UR - https://doi.org/10.1109/IV64158.2025.11097436 Y1 - 2025 UR - https://doi.org/10.1109/IV64158.2025.11097436 SN - 979-8-3315-3803-3 SP - 1033 EP - 1039 PB - IEEE CY - Piscataway ER -