- We present an approach to audio-visual speaker recognition that consists of two parts. First, we extend an audio-only ECAPA-TDNN model with an audio-visual (AV) fusion module that fuses both modalities to process the results in the ECAPA model. Second, we extend a ResNet that processes individual frames for face recognition with an attention block that processes a sequence of embeddings to focus on meaningful embeddings and thus increase the robustness towards quality differences in images. The fusion of the individual model’s results further improves the performance and reduces the error rates. Comparing our models to the baselines reveals that the additional visual information and attention-based processing of image sequences result in relative improvements of up to 56%.
