11-777 MMML | Schedule

Date	Lecture	Topics
8/29	Lecture 1.1: Course introduction [ slides \| video ]	Multimodal core challenges Course syllabus
8/31	Lecture 1.2: Multimodal applications [ slides \| video ]	Research tasks Multimodal datasets Team projects
9/5	Lecture 2.1: Unimodal representations [ slides \| video ]	Dimensions of heterogeneity Visual representations
9/7	Lecture 2.2: Unimodal representations [ slides \| video ]	Language representations Signals representations Graphs representations Other modality representations
9/12	Lecture 3.1: Multimodal representations [ slides \| video ]	Cross-modal interactions Multimodal fusion
9/14	Lecture 3.2: Multimodal representations [ slides \| video ]	Coordinated representations Multimodal fission
9/19	Lecture 4.1: Alignment and grounding [ slides \| video ]	Explicit alignment Multimodal training Multimodal grounding
9/21	Lecture 4.2: Aligned representations [ slides \| video ]	Self-attention transformer models Masking and self-supervised learning
9/26	Lecture 5.1: Multimodal transformers [ slides \| video ]	Language pretraining Multimodal Transformer Transformer architecture
9/28	Lecture 5.2: Structured Representation and Reasoning [ slides \| video ]	Structured and hierarchical models Memory models
10/3	Lecture 6.1: Multimodal transformers [ slides \| video ]	Vision transformer Video transformer Vision-language transformer
10/5	Lecture 6.2: Guest Talk [ slides not available \| video not available ]
10/10	Lecture 7.1: Multimodal Interaction [ slides \| video ]	Language and reinforcement learning Interactive learning Q-learning Policy-based methods
10/12	Lecture 7.2: Multimodal Inference and Knowledge [ slides \| video ]	Language and discrete concepts Causal inference External knowledge Reasoning
10/17	Lecture 8.1: Fall Break – No lectures
10/19	Lecture 8.2: Fall Break – No lectures
10/24	Lecture 9.1: Multimodal Generation [ slides \| video ]	Translation, summarization, creation Generative models Auto-regressive language models
10/26	Lecture 9.2: New Generation Models [ slides \| video ]	Xixture of Guassians, VAE, diffusion models Open generation challenges
10/31	Lecture 10.1: Midterm presentations – No lectures
11/2	Lecture 10.2: Midterm presentations – No lectures
11/7	Lecture 11.1: Democracy Day – No lectures
11/9	Lecture 11.2: Transference [ slides \| video ]	Multimodal co-learning Co-training and self-training
11/14	Lecture 12.1: New research directions [ slides \| video ]	Recent approaches in multimodal ML State-of-the-art multimodal models Understudied modalities
11/16	Lecture 12.2: Quantification [ slides \| video ]	Mathematical framework for interaction Multimodal interaction quantification Evaluating quantification Optimization challenge
11/21	Lecture 13.1: Thanksgiving Week – No lectures
11/23	Lecture 13.2: Thanksgiving Week – No lectures
11/29	Lecture 14.1: Guest lecture [ slides \| video not available ]
12/1	Lecture 14.2: Guest lecture [ slides \| video not available ]
12/5	Lecture 15.1: Final project presentations – No lectures
12/7	Lecture 15.2: Final project presentations – No lectures

8/31

Lecture 1.2: Multimodal applications
[ slides | video ]

Research tasks
Multimodal datasets
Team projects

9/5

Lecture 2.1: Unimodal representations
[ slides | video ]

Dimensions of heterogeneity
Visual representations

9/7

Lecture 2.2: Unimodal representations
[ slides | video ]

Language representations
Signals representations
Graphs representations
Other modality representations

9/12

Lecture 3.1: Multimodal representations
[ slides | video ]

Cross-modal interactions
Multimodal fusion

9/14

Lecture 3.2: Multimodal representations
[ slides | video ]

Coordinated representations
Multimodal fission

9/19

Lecture 4.1: Alignment and grounding
[ slides | video ]

Explicit alignment
Multimodal training
Multimodal grounding

9/21

Lecture 4.2: Aligned representations
[ slides | video ]

Self-attention transformer models
Masking and self-supervised learning

9/26

Lecture 5.1: Multimodal transformers
[ slides | video ]

Language pretraining
Multimodal Transformer
Transformer architecture

9/28

Lecture 5.2: Structured Representation and Reasoning
[ slides | video ]

Structured and hierarchical models
Memory models

10/3

Lecture 6.1: Multimodal transformers
[ slides | video ]

Vision transformer
Video transformer
Vision-language transformer

10/5

Lecture 6.2: Guest Talk
[ slides not available | video not available ]

10/10

Lecture 7.1: Multimodal Interaction
[ slides | video ]

Language and reinforcement learning
Interactive learning
Q-learning
Policy-based methods

10/12

Lecture 7.2: Multimodal Inference and Knowledge
[ slides | video ]

Language and discrete concepts
Causal inference
External knowledge
Reasoning

10/17

Lecture 8.1: Fall Break – No lectures

10/19

Lecture 8.2: Fall Break – No lectures

10/24

Lecture 9.1: Multimodal Generation
[ slides | video ]

Translation, summarization, creation
Generative models
Auto-regressive language models

10/26

Lecture 9.2: New Generation Models
[ slides | video ]

Xixture of Guassians, VAE, diffusion models
Open generation challenges

10/31

Lecture 10.1: Midterm presentations – No lectures

11/2

Lecture 10.2: Midterm presentations – No lectures

11/7

Lecture 11.1: Democracy Day – No lectures

11/9

Lecture 11.2: Transference
[ slides | video ]

Multimodal co-learning
Co-training and self-training

11/14

Lecture 12.1: New research directions
[ slides | video ]

Recent approaches in multimodal ML
State-of-the-art multimodal models
Understudied modalities

11/16

Lecture 12.2: Quantification
[ slides | video ]

Mathematical framework for interaction
Multimodal interaction quantification
Evaluating quantification
Optimization challenge

11/21

Lecture 13.1: Thanksgiving Week – No lectures

11/23

Lecture 13.2: Thanksgiving Week – No lectures

11/29

Lecture 14.1: Guest lecture
[ slides | video not available ]

12/1

Lecture 14.2: Guest lecture
[ slides | video not available ]

12/5

Lecture 15.1: Final project presentations – No lectures

12/7

Lecture 15.2: Final project presentations – No lectures