Zhe Gan

I am a Principal Researcher at Microsoft Azure AI, primarily working on Vision-and-Language Multimodal Intelligence, a research area that sits at the nexus of computer vision and natural language processing, such as Vision-and-Language Pre-training (VLP), Visual Question Answering (VQA), image captioning, and video-text modeling. I also have broad interests on other machine learning topics, such as sparse neural networks, adversarial training, and self-supervised visual representation learning. I received my Ph.D. degree from Duke University in Spring 2018. Before that, I received my Master's and B.Sc. from Peking University in 2013 and 2010, respectively. My Ph.D. advisor is Lawrence Carin. I can be reached at zhe.gan@microsoft.com.

I am serving (or, has served) as an Area Chair for NeurIPS 2021/2020/2019, ICML 2021, ICLR 2021, ACL 2021, AAAI 2022, and a Senior Program Committee (SPC) member for AAAI 2021/2020, and received AAAI-20 Outstanding SPC Award.

Research Highlights:

[2021/09] We all know GPT-3 is a strong few-shot learner for NLP problems, but can it also benefit multimodal tasks? In this new work, we provide an empirical study of GPT-3 for knowledge-based VQA, and show that prompting GPT-3 via the use of image captions with only 16 examples surpasses supervised sota by an absolute +8.6 points on the OK-VQA dataset (from 39.4 to 48.0).

[2021/07] Our Adversarial VQA work is accepted by ICCV 2021 as an Oral paper (top 3% among all submissions). Our VALUE benchmark is accepted by NeurIPS 2021.

[2021/06] Our ClipBERT paper wins the Best Student Paper Honorable Mention Award at CVPR 2021.

[2021/06] 4 pieces of updates on our recent vision-and-language efforts: (i) Our CVPR 2021 tutorial will happen on 6/20; (ii) Our VALUE benchmark and competition has been launched; (iii) The arXiv version of our Adversarial VQA benchmark has been released; (iv) We are the winner of TextCaps Challenge 2021.

[2021/05] Two papers accepted by ACL 2021: one long paper in the Main Conference track and the other in the Findings track. Topics include (i) EarlyBERT for efficient BERT training, and (ii) Cluster-Former as an efficient transformer variant for question answering.

[2021/03] Two papers accepted by CVPR 2021, and one paper accepted to NAACL 2021. Topics include (i) ClipBERT for video-and-language learning (Oral with 3 strong accepts), (ii) enhancing contrastive knowledge distillation with Wasserstein learning, and (iii) text generation from hyperbolic space.

[2021/02] We will host a tutorial at CVPR 2021: From VQA to VLN: Recent Advances in Vision-and-Language Research. This year, we decide to extend our half-day event to a full one with enriched contents. Organizers include Peter Anderson, Yoav Artzi, Xiaodong He, Linjie Li, Jingjing Liu, Xin (Eric) Wang, Qi Wu, Luowei Zhou, and me.

[2021/01] Two papers accepted by ICLR 2021. Topics include using information-theoretic tools for (i) improved robustness of language models (BERT and RoBERTa) and (ii) zero-shot voice style transfer.

[2021/01] Our Meta Module Network wins the Best Student Paper Honorable Mention Award at WACV 2021.

[2020/09] Our VILLA paper got accepted to NeurIPS 2020 as a Spotlight paper with review scores 8887. It is the first known effort that studies large-scale adversarial training for vision-and-langauge representation learning in both pre-training and finetuning stages.

[2020/09] 7 long papers accepted by EMNLP 2020: (i) 6 of them accepted to the main conference, and (ii) 1 of them accepted to Findings of EMNLP 2020. Topics include: (i) Large-scale LM compression; (ii) Sentence embedding pre-training; (iii) Video+language pre-training; (iv) Constrained text generation via pre-training; (v) Summarization; (vi) Multi-hop reasoning for QA; and (vii) text style transfer.

[2020/09] We achieve #1 on the XTREME and XGLUE leaderboards for cross-lingual language understanding. Welcome to check our FILTER paper.

[2020/07] Two papers got accepted to ECCV 2020. (1) UNITER: a state-of-the-art pre-trained Vision+Language (V+L) model; (2) VALUE (ECCV Spotlight): the first work on probing pre-trained V+L models.

[2020/06] Two papers got accepted to ICML 2020. (i) CLUB: a novel upper bound of mutual information that is deeply connected with contrastive learning. (ii) GOT: a graph optimal transport framework for cross-domain alignment that can be used for V+L and NLP problems, such as VQA and NMT.

[2020/06] At this year's CVPR, we will give a tutorial on "Recent Advances in Vision-and-Language Research", covering the recent popular multi-modal pre-training methods and other topics. More details are provided in the website here.