Improving Multimodal Speech-To-Slide Alignment for Academic Lectures with Vision LLMs
- We enhance the MaViLS multimodal algorithm to improve speech-to-slide alignment for lecture podcasts. Our approach integrates vision large language models for optical character recognition and automatic generation of lecture transcripts from slide content, coupled with a multilingual multimodal embedding model for text and image alignment. By combining slideextracted text with automatically generated lecture transcripts and captioned slide images, we generate enhanced audio features that better capture speech-slide mapping. Our method improves the average F1 score for audio feature alignment on the English MaViLS dataset from 0.51 to 0.71 and on a newly created German podcast lectures dataset from 0.65 to 0.84.
| Author: | Thomas RanzenbergerORCiD, Dominik Wagner, Steffen Freisinger, Tobias BockletORCiD, Korbinian RiedhammerORCiD |
|---|---|
| DOI: | https://doi.org/10.1109/ASRU65441.2025.11434615 |
| Parent Title (English): | 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) |
| Publisher: | IEEE |
| Place of publication: | Honolulu, HI, USA |
| Document Type: | conference proceeding (article) |
| Language: | English |
| Reviewed: | Begutachtet/Reviewed |
| Release Date: | 2026/06/11 |
| Tag: | feature embeddings; large language models; lecture accessibility; multimodal retrieval; slide matching; vision |
| Volume: | 2025 |
| Pagenumber: | 7 |
| First Page: | 1 |
| Last Page: | 7 |
| institutes: | Fakultät Informatik |
| Zentrum für Künstliche Intelligenz (KIZ) | |
| Research Themes: | Digitalisierung & Künstliche Intelligenz |
