MML Tutorial | Schedule

** Exact topics and schedule subject to change, based on interests and time. **

Part	Topics	Readings
1	Introduction [slides] [video] What is Multimodal? Definitions, dimensions of heterogeneity and cross-modal interactions. Historical view and multimodal research tasks. Core technical challenges: representation, alignment, transference, reasoning, generation, and quantification.	Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions Multimodal Machine Learning: A Survey and Taxonomy Representation Learning: A Review and New Perspectives
2	Representation [slides] [video] Representation fusion: additive, multiplicative, non-linear, complex fusion strategies. Representation coordination: contrastive learning, vector-space models, canonical correlation analysis. Representation fission: factorization, component analysis, clustering.	Issues in the Classification of Multimodal Communication Signals Nonnegative Decomposition of Multivariate Information Language and Image Interaction in Cartoons: Towards a Multimodal Theory of Humor Multimodality and Reading: The Construction of Meaning Through Image-text Interaction Examining the Redundancy of Multimodal Input Quantifying & Modeling Feature Interactions: An Information Decomposition Framework Multiplicative Interactions and Where to Find Them Tensor Fusion Network for Multimodal Sentiment Analysis Does My Multimodal Model Learn Cross-modal Interactions? It’s Harder to Tell Than You Might Think! On Deep Multi-View Representation Learning: Objectives and Optimization Learning Transferable Visual Models From Natural Language Supervision Learning Factorized Multimodal Representations
3	Alignment [slides] [video] Discrete alignment: grounding, optimal transport, distribution matching. Continuous alignment: time warping, CTC, temporal alignment, clustering. Aligned representations: attention models, multimodal transformers.	What Makes for Good Views for Contrastive Learning? Characterization and Classification of Semantic Image-text Relations A Taxonomy of Relationships Between Images and Text Deep Canonical Correlation Analysis Graph Optimal Transport for Cross-domain Alignment Deep Canonical Time Warping for Simultaneous Alignment and Representation Learning of Sequences Deep Visual-semantic Alignments for Generating Image Descriptions ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Multimodal Transformer for Unaligned Multimodal Language Sequences
4	Reasoning [slides] [video] Structure: hierarchical, graphical, temporal, and interactive structure, structure discovery. Concepts: dense and neuro-symbolic. Inference: logical and causal inference. Knowledge: external knowledge bases, commonsense reasoning.	Learning to Compose and Reason with Language Tree Structures for Visual Grounding The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision A Survey of Reinforcement Learning Informed by Natural Language Dynamic Memory Networks for Visual and Textual Question Answering Multimodal Memory Modelling for Video Captioning VQA-LOL: Visual Question Answering Under the Lens of Logic Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
5	Generation [slides] [video] Summarization, translation, and creation. Model evaluation and ethical concerns.	VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning DALL·E: Creating Images from Text and DALL·E 2 On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? The Social Impact of Deepfakes What a machine learning tool that turns Obama white can (and can’t) tell us about AI bias Lessons from the PULSE Model and Discussion Multimodal Abstractive Summarization for How2 Videos Video Generation From Text
6	Transference [slides] [video] Transfer via pre-trained models: pre-trained models, prefix tuning, representation tuning, multitask models. Co-learning: co-learning via representation and generation.	Vokenization: Improving Language Understanding via Contextualized, Visually-Grounded Supervision Does Vision-and-Language Pretraining Improve Lexical Grounding? Foundations of Multimodal Co-learning Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities Zero-Shot Learning Through Cross-Modal Transfer 12-in-1: Multi-Task Vision and Language Representation Learning Analyzing the Effectiveness and Applicability of Co-training An Information Theoretic Framework for Multi-view Learning
7	Quantification [slides] [video] Dimensions of heterogenity: modality importance, dataset biases, social biases, noise topologies and robustness. Cross-modal interactions: interpreting cross-model connections and interactions. Learning: learning and optimization challenges.	MultiBench: Multiscale Benchmarks for Multimodal Representation Learning HighMMT: Towards Modality and Task Generalization for High-Modality Representation Learning VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers Do explanations make VQA models more predictable to a human? Women also Snowboard: Overcoming Bias in Captioning Models Measuring Social Biases in Grounded Vision and Language Embeddings Multimodal datasets: misogyny, pornography, and malignant stereotypes