Multimodal Learning Advances

The field of multimodal learning is rapidly advancing, with a focus on developing models that can seamlessly integrate and process multiple forms of data, such as text, images, and audio. Recent developments have seen the emergence of end-to-end approaches that fuse multimodal foundation models with dedicated translation models, enabling joint training and improving performance. Another notable trend is the use of emergent alignment learning, which allows for the efficient integration of new modalities and languages into existing models without requiring full retraining. Additionally, there have been significant improvements in speech-to-text translation, with models now capable of handling many-to-many translation across multiple languages. Noteworthy papers include: OmniFusion, which proposes a novel fusion strategy for multimodal translation and achieves a 1-second latency reduction in simultaneous speech translation. CACARA, which demonstrates a text-centric approach for cost-effective multimodal and multilingual learning, achieving up to a 14.24 percentage points improvement in audio-to-text retrieval. MCAT, which introduces a framework for scaling many-to-many speech-to-text translation to 70 languages, surpassing state-of-the-art models on the FLEURS dataset. BOOM, which presents a multimodal multilingual lecture companion that jointly translates lecture audio and slides, providing an accessible learning experience. Cross-Lingual Interleaving for Speech Language Models, which enables robust cross-lingual continuation and strengthens cross-lingual hidden-state alignment. RFOP and Shared Multi-modal Embedding Space for Face-Voice Association, which achieve high performance in the FAME 2026 challenge, with the latter ranking first place.

Sources

OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion

CACARA: Cross-Modal Alignment Leveraging a Text-Centric Approach for Cost-Effective Multimodal and Multilingual Learning

MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages

Cross-Lingual Interleaving for Speech Language Models

BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion

RFOP: Rethinking Fusion and Orthogonal Projection for Face-Voice Association

Shared Multi-modal Embedding Space for Face-Voice Association

Built with on top of