Multimodal machine learning (MMML) is a vibrant interdisciplinary research area that advances core goals of artificial intelligence by integrating and modeling multiple communicative modalities, including language, speech, vision, and other sensory signals. Originating from early work in audio-visual speech recognition and evolving through modern language-and-vision systems such as image and video captioning, MMML addresses unique challenges arising from heterogeneous data sources and complex interdependencies between modalities.

This course introduces the fundamental mathematical and algorithmic concepts in machine learning and deep learning that underpin the six central challenges of multimodal machine learning:

  1. Representation
  2. Alignment
  3. Reasoning
  4. Generation
  5. Transference
  6. Quantification

Students will study both foundational and modern approaches to these challenges, including multimodal autoencoders, deep canonical correlation analysis, multimodal tensor fusion, multi-kernel learning, attention mechanisms, multimodal recurrent models, multimodal transformers, neuro-symbolic methods, mutual information–based objectives, and multimodal graph neural networks.

The course also examines state-of-the-art probabilistic models and computational algorithms through readings of recent research literature, with attention to open problems and emerging directions in the field. Applications discussed throughout the course include multimodal affect recognition, language grounding, language-vision navigation, image and video captioning, and cross-modal multimedia retrieval.

  • Time: Tuesday/Thursday 9:30 AM-10:50 AM
  • Location: POS :: A35
  • Discussion: Piazza
  • HW submission: Canvas
  • Online lectures: The lectures will be live-streamed through Panopto , recorded, and made available on YouTube .
  • Contact: Students should ask all course-related questions on Piazza , where you will also find announcements. For external enquiries, personal matters, or emergencies, create a private post in Piazza .