11-777 MMML | Schedule

Date	Lecture	Topics	Deadlines
8/25	Lecture 1.1: Introduction [ slides \| video ]	Multimodal core challenges Course syllabus
8/27	Lecture 1.2: Multimodal Research Task [ slides \| video ]	Designing research questions Datasets and research task
9/1	Lecture 2.1: Unimodal representations [ slides \| video ]	Language representations Self-attention transformer models
9/3	Lecture 2.2: Unimodal representations [ slides \| video ]	Visual and signal representations Dimensions of heterogeneity	Read due: 9/4
9/8	Lecture 3.1: Multimodal representations [ slides \| video ]	Cross-modal interactions Multimodal fusion	Proj. due: 9/9
9/10	Lecture 3.2: Multimodal representations [ slides \| video ]	Coordinated representations Multimodal fission	Read due: 9/11
9/15	Lecture 4.1: Multimodal alignment and grounding [ slides \| video ]	Discrete and continuous alignment Multimodal grounding
9/17	Lecture 4.2: Transference [ slides \| video ]	Cross-modal distillation, co-learning Pretraining, fine-tuning, post-training	Proj. due: 9/20
9/22	Lecture 5.1: Project presentations (proposal) – No lectures
9/24	Lecture 5.2: Project presentations (proposal) – No lectures		Read due: 9/25
9/29	Lecture 6.1: Visual-Language Models [ slides \| video ]	Visual-language alignment Multimodal LLMs
10/1	Lecture 6.2: Video and Unified Spaces [ slides \| video ]	Binding multiple modalities Video transformers	Proj. due: 10/4
10/6	Lecture 7.1: Guest lecture [ slides \| video ]
10/8	Lecture 7.2: Guest lecture – Alex Wilf [ slides \| video ]	Web agents – LLMs and MLLMs Self-learning and verification	Read due: 10/9
10/13	Lecture 8.1: Fall Break – No lectures
10/15	Lecture 8.2: Fall Break – No lectures
10/20	Lecture 9.1: Multimodal Reasoning [ slides \| video ]	Modular and neuro-symbolic models Graph and structured models
10/22	Lecture 9.2: Multimodal Reasoning [ slides \| video ]	Test-time computation Multimodal chain-of-thought	Proj. due: 10/25
10/27	Lecture 10.1: Project presentations (midterm) – No lectures
10/29	Lecture 10.2: Project presentations (midterm) – No lectures
11/3	Lecture 11.1: Democracy Day – No Class – No lectures
11/5	Lecture 11.2: Multimodal Generation [ slides \| video ]	Autoregressive generation	Read due: 11/6
11/10	Lecture 12.1: Multimodal Generation [ slides \| video ]	Diffusion and flow matching Omni any-to-any models
11/12	Lecture 12.2: Visual-Language-Action [ slides \| video ]	Embodied representations VLA foundation models	Read due: 11/13
11/17	Lecture 13.1: Quantification and Bias [ slides \| video ]	Heterogeneity and bias Evaluation and learning
11/19	Lecture 13.2: New Research Directions [ slides \| video ]	Recent research in multimodal ML
11/24	Lecture 14.1: Thanksgiving Week – No Class – No lectures
11/26	Lecture 14.2: Thanksgiving Week – No Class – No lectures
12/1	Lecture 15.1: Project presentations (final) – No lectures
12/3	Lecture 15.2: Project presentations (final) – No lectures		Proj. due: 12/6