The field of multimodal speech recognition and document analysis is witnessing significant advancements with the integration of large language models, reinforcement learning, and multimodal fusion techniques. Researchers are exploring novel approaches to improve the accuracy and robustness of speech recognition systems in challenging environments, such as cocktail-party scenarios. The development of large-scale datasets, like SARD for Arabic OCR and MegaHan97K for mega-category Chinese character recognition, is bridging critical gaps in data scarcity and enabling substantial improvements in model performance. Noteworthy papers include QARI-OCR, which achieves state-of-the-art results in Arabic OCR, and MonkeyOCR, which introduces a Structure-Recognition-Relation triplet paradigm for document parsing, outperforming existing models. Additionally, UniCUE proposes a unified framework for Chinese Cued Speech Video-to-Speech generation, significantly reducing Word Error Rate and improving lip-speech synchronization.
Advances in Multimodal Speech Recognition and Document Analysis
Sources
SARD: A Large-Scale Synthetic Arabic OCR Dataset for Book-Style Text Recognition
CNVSRC 2024: The Second Chinese Continuous Visual Speech Recognition Challenge
Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishing
Cocktail-Party Audio-Visual Speech Recognition
PAIR-Net: Enhancing Egocentric Speaker Detection via Pretrained Audio-Visual Fusion and Alignment Loss
QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation
DnR-nonverbal: Cinematic Audio Source Separation Dataset Containing Non-Verbal Sounds
Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
Voice Activity Projection Model with Multimodal Encoders
UniCUE: Unified Recognition and Generation Framework for Chinese Cued Speech Video-to-Speech Generation
ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition
MegaHan97K: A Large-Scale Dataset for Mega-Category Chinese Character Recognition with over 97K Categories
A Survey on Vietnamese Document Analysis and Recognition: Challenges and Future Directions
MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm