+

Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with + intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, + including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, + embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, + multimodality has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity + of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research + has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application + domains and theoretical frameworks from both historical and recent perspectives, this tutorial is designed to provide an overview + of the computational and theoretical foundations of multimodal machine learning.
+
+ Building upon a new edition of our survey paper on multimodal ML and academic courses at CMU, this tutorial will cover three topics: + (1) what is multimodal: the principles in learning from heterogeneous, connected, and interacting data, (2) why is it hard: + a taxonomy of six core technical challenges faced in multimodal ML but understudied in unimodal ML, and (3) what is next: + major directions for future research as identified by our taxonomy. +

+
+ +
    +
  • Time: Monday, 7/24/2022, 9:30am - 12:00pm HST.
    • +
  • Location: ICML 2023, Honolulu, Hawaii, USA. Recorded videos will also be uploaded here soon.
    • +
  • Contact: Presenters can be contacted at pliang@cs.cmu.edu and morency@cs.cmu.edu.
    • +
+ +
+ +