|
Abstract: |
| Recent research has offered insights into the extraordinary capabilities of Multimodal Large Multimodal Models (MLMMs) in various general vision and language tasks. There is a growing interest in exploring how MLMMs perform in specialized domains. Social media content, which is inherently multimodal, combines text, images, videos, and sometimes audio, presenting a challenge for models to interpret the interactions between these diverse communication modalities and their impact on the message being conveyed. To assess the capabilities of MLMMs in social multimedia analysis, we selected five key tasks, including sentiment analysis, hate speech detection, fake news identification, demographic inference, and political ideology detection. Our investigation began with a preliminary quantitative analysis of each task using existing benchmark datasets, followed by a review of the results and the selection of qualitative samples that illustrate GPT-4V’s potential in understanding multimodal social media content. We present several strategies to improve the performance on selected tasks. Our findings underscore a promising future for MLMMs in advancing our understanding of social media content and its users through the analysis of multimodal information. |
Biography： |
|
Professor Jiebo Luo is the Albert Arendt Hopeman Professor of Engineering and Professor of Computer Science at the University of Rochester, which he joined in 2011 after a prolific career of 15 years at Kodak Research Laboratories. He has authored over 700 technical papers and holds 98 U.S. patents. His research interests include computer vision, NLP, machine learning, data mining, computational social science, and digital health. He has been involved in numerous technical conferences, including serving as program co-chair of ACM Multimedia 2010, IEEE CVPR 2012, and IEEE ICIP 2017, and general co-chair of ACM Multimedia 2018 and IEEE ICME 2024. Professor Luo has served on the editorial boards of the IEEE TPAMI, IEEE TMM, IEEE TCSVT, IEEE TBD, ACM TIST, Pattern Recognition, Health Data Science, and Intelligent Medicine, and so on. He was the Editor-in-Chief of the IEEE Transactions on Multimedia from 2020 to 2022. He is a Fellow of ACM, AAAI, IEEE, AIMBE, IAPR, and SPIE, as well as a Member of Academia Europaea and the US National Academy of Inventors (NAI). He received the ACM SIGMM Technical Achievement Award in 2021 and the IEEE Computer Society Edward J. McCluskey Technical Achievement Award in 2025. |