Zhe Gan

Publications

arXiv preprints

Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, Jingjing Liu “EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets”, 2021. PDF
Linjie Li, Zhe Gan and Jingjing Liu “A Closer Look at the Robustness of Vision-and-Language Pre-trained Models”, 2020. PDF (SOTA on 7 VQA robustness benchmarks as of Dec. 15, 2020)
Liqun Chen, Zhe Gan, Dong Wang, Jingjing Liu, Ricardo Henao and Lawrence Carin “Wasserstein Contrastive Representation Distillation”, 2020. PDF
Jinghui Chen, Yu Cheng, Zhe Gan, Quanquan Gu and Jingjing Liu “Efficient Robust Training via Backward Smoothing”, 2020. PDF
Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng and Jingjing Liu “Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding”, 2020. PDF (Leaderboard #1 on NaturalQuestions as of Sep. 27, 2020)
Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun and Jingjing Liu “Accelerating Real-Time Question Answering via Question Generation”, 2020. PDF
Chen Zhu, Yu Cheng, Zhe Gan, Furong Huang, Jingjing Liu and Tom Goldstein “Adaptive Learning Rates with Maximum Variation Averaging”, 2020. PDF / Code
Shuyang Dai, Zhe Gan, Yu Cheng, Chenyang Tao, Lawrence Carin and Jingjing Liu “APo-VAE: Text Generation in Hyperbolic Space”, 2020. PDF

2021

Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li and Jingjing Liu “InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective”, Int. Conf. Learning Representations (ICLR), 2021. PDF (Leaderboard #1 on Adversarial NLI as of Oct. 9, 2020)
Siyang Yuan*, Pengyu Cheng*, Ruiyi Zhang, Weituo Hao, Zhe Gan and Lawrence Carin “Improving Zero-Shot Voice Style Transfer via Disentangled Representation Learning”, Int. Conf. Learning Representations (ICLR), 2021.
Yuwei Fang*, Shuohang Wang*, Zhe Gan, Siqi Sun and Jingjing Liu “FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding”, Proc. American Association of Artificial Intelligence (AAAI), 2021. PDF / Code / Slides / Blog (Leaderboard #1 on XTREME and XGLUE as of Sep. 8, 2020)
Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang and Jingjing Liu “Meta Module Network for Compositional Visual Reasoning”, Winter Conf. on Applications of Computer Vision (WACV), 2021. PDF / Code (Best Student Paper Honorable Mention)

2020

Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng and Jingjing Liu “Large-Scale Adversarial Training for Vision-and-Language Representation Learning”, Neural Information Processing Systems (NeurIPS), 2020. PDF / Code-I / Code-II / Slides / Poster / Blog (Spotlight) Top 4% among all submissions, SOTA on 6 Vision+Language tasks
Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang and Jingjing Liu “Contrastive Distillation on Intermediate Representations for Language Model Compression”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2020. PDF / Blog / Code
Shuohang Wang, Yuwei Fang, Siqi Sun, Zhe Gan, Yu Cheng, Jing Jiang and Jingjing Liu “Cross-Thought for Sentence Encoder Pre-training”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2020. PDF
Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung and Jingjing Liu “Multi-Fact Correction in Abstractive Text Summarization”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2020. PDF / Slides / Blog
Linjie Li*, Yen-Chun Chen*, Yu Cheng, Zhe Gan, Licheng Yu and Jingjing Liu “HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2020. PDF / Code / Blog (SOTA on 8 Video+Language datasets, Leaderboard #1 on TVR and TVC as of Sep. 15, 2020)
Yizhe Zhang*, Guoyin Wang*, Chunyuan Li, Zhe Gan, Chris Brockett and Bill Dolan “POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2020. PDF / Code / Demo
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang and Jingjing Liu “Hierarchical Graph Network for Multi-hop Question Answering”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2020. PDF / Code (Leaderboard #1 on HotpotQA as of Dec. 1st, 2019)
Yu Cheng, Zhe Gan, Yizhe Zhang, Oussama Elachqar, Dianqi Li and Jingjing Liu “Contextual Text Style Transfer”, Findings of Empirical Methods in Natural Language Processing (Findings of EMNLP), 2020. PDF
Yi Wei, Zhe Gan, Wenbo Li, Siwei Lyu, Ming-Ching Chang, Lei Zhang, Jianfeng Gao and Pengchuan Zhang “MagGAN: High-Resolution Face Attribute Editing with Mask-Guided Generative Adversarial Network”, Asian Conf. on Computer Vision (ACCV), 2020. PDF
Shuyang Dai, Yu Cheng, Yizhe Zhang, Zhe Gan, Jingjing Liu and Lawrence Carin “Contrastively Smoothed Class Alignment for Unsupervised Domain Adaptation”, Asian Conf. on Computer Vision (ACCV), 2020. PDF
Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen and Jingjing Liu “Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models”, European Conf. on Computer Vision (ECCV), 2020. PDF (Spotlight) Top 5% among all submissions
Yen-Chun Chen*, Linjie Li*, Licheng Yu*, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng and Jingjing Liu “UNITER: UNiversal Image-TExt Representation Learning”, European Conf. on Computer Vision (ECCV), 2020. PDF / Code (SOTA on 13 Vision+Language Datasets/Tasks, No. 1 on VCR and NLVR2 leaderboards as of Sep. 2019)
Yu Cheng, Zhe Gan, Yitong Li, Jingjing Liu and Jianfeng Gao “Sequential Attention GAN for Interactive Image Editing”, ACM International Conference on Multimedia (ACMMM), 2020. PDF
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan and Lawrence Carin “CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information”, Int. Conf. Machine Learning (ICML), 2020. PDF / Code
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin and Jingjing Liu “Graph Optimal Transport for Cross-Domain Alignment”, Int. Conf. Machine Learning (ICML), 2020. PDF / Code
Jiacheng Xu, Zhe Gan, Yu Cheng and Jingjing Liu “Discourse-Aware Neural Extractive Text Summarization”, Association for Computational Linguistics (ACL), 2020. PDF / Blog / Code
Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu and Jingjing Liu “Distilling Knowledge Learned in BERT for Text Generation”, Association for Computational Linguistics (ACL), 2020. PDF / Slides / Blog / Code
Ruiyi Zhang, Changyou Chen, Zhe Gan, Wenlin Wang, Dinghan Shen, Guoyin Wang, Zheng Wen and Lawrence Carin “Improving Adversarial Text Generation by Modeling the Distant Future”, Association for Computational Linguistics (ACL), 2020. PDF / Old
Yandong Li, Yu Cheng, Zhe Gan, Licheng Yu, Liqiang Wang, Jingjing Liu “BachGAN: High-Resolution Image Synthesis from Salient Object Layout”, Computer Vision and Pattern Recognition (CVPR), 2020. PDF / Code
Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, Jingjing Liu “VIOLIN: A Large-Scale Dataset for Video-and-Language Inference”, Computer Vision and Pattern Recognition (CVPR), 2020. PDF / Code
Ruiyi Zhang, Changyou Chen, Zhe Gan, Zheng Wen, Wenlin Wang, Lawrence Carin “Nested-Wasserstein Self-Imitation Learning for Sequence Generation”, Artificial Intelligence and Statistics (AISTATS), 2020. PDF
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein and Jingjing Liu “FreeLB: Enhanced Adversarial Training for Natural Language Understanding”, Int. Conf. Learning Representations (ICLR), 2020. PDF / Code (Spotlight) Leaderboard #1 on GLUE, ARC Easy/Challenge and Commonsense QA as of Sep. 2019
Wenlin Wang, Hongteng Xu, Zhe Gan, Bai Li, Guoyin Wang, Liqun Chen, Qian Yang, Wenqi Wang and Lawrence Carin “Graph-Driven Generative Models for Heterogeneous Multi-Task Learning”, Proc. American Association of Artificial Intelligence (AAAI), 2020. PDF / Poster / Slides (Spotlight)
Junjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu, Jianfeng Gao and Graham Neubig “What Makes A Good Story? Designing Composite Rewards for Visual Storytelling”, Proc. American Association of Artificial Intelligence (AAAI), 2020. PDF / Poster / Code (Spotlight)

2019

Wenlin Wang, Chenyang Tao, Zhe Gan, Guoyin Wang, Liqun Chen, Xinyuan Zhang, Ruiyi Zhang, Qian Yang, Ricardo Henao and Lawrence Carin “Improving Textual Network Learning with Variational Homophilic Embeddings”, Neural Information Processing Systems (NeurIPS), 2019. PDF
Ruiyi Zhang, Changyou Chen, Zhe Gan, Zheng Wen, Wenlin Wang, Lawrence Carin “Nested-Wasserstein Distance for Sequence Generation”, Workshop on Bayesian Deep Learning, NeurIPS 2019. PDF
Siqi Sun, Yu Cheng, Zhe Gan and Jingjing Liu “Patient Knowledge Distillation for BERT Model Compression”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2019. PDF / Poster / Code
Huazheng Wang, Zhe Gan, Xiaodong Liu, Jingjing Liu, Jianfeng Gao and Hongning Wang “Adversarial Domain Adaptation for Machine Reading Comprehension”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2019. PDF / Poster / Code
Dianqi Li, Yizhe Zhang, Zhe Gan, Yu Cheng, Chris Brockett, Ming-Ting Sun and Bill Dolan “Domain Adaptive Text Style Transfer”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2019. PDF / Code
Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner and Jianfeng Gao “TIGEr: Text-to-Image Grounding for Image Caption Evaluation”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2019. PDF / Poster / Code
Linjie Li, Zhe Gan, Yu Cheng and Jingjing Liu “Relation-Aware Graph Attention Network for Visual Question Answering”, Int. Conf. on Computer Vision (ICCV), 2019. PDF / Supp / Poster / Code
Zhe Gan, Yu Cheng, Ahmed El Kholy, Linjie Li, Jingjing Liu and Jianfeng Gao “Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog”, Association for Computational Linguistics (ACL), 2019. PDF / Poster
Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi and Siddhartha Srinivasa “Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation”, Computer Vision and Pattern Recognition (CVPR), 2019. PDF / YouTube / Code (Oral)
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson and Jianfeng Gao “StoryGAN: A Sequential Conditional GAN for Story Visualization”, Computer Vision and Pattern Recognition (CVPR), 2019. PDF / Slides / Poster / Code
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen and Lawrence Carin “Topic-Guided Variational Autoencoders for Text Generation”, North American Chapter of the Association for Computational Linguistics (NAACL), 2019. PDF (Oral)
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen and Lawrence Carin “Improving Sequence-to-Sequence Learning via Optimal Transport”, Int. Conf. Learning Representations (ICLR), 2019. PDF / Code
Qiuyuan Huang*, Zhe Gan*, Asli Celikyilmaz, Dapeng Wu, Jianfeng Wang and Xiaodong He “Hierarchically Structured Reinforcement Learning for Topically Coherent Visual Story Generation”, Proc. American Association of Artificial Intelligence (AAAI), 2019. PDF (Spotlight)

2018

Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett and Bill Dolan “Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization”, Neural Information Processing Systems (NeurIPS), 2018. PDF / Blog / Code
Liqun Chen, Shuyang Dai, Chenyang Tao, Dinghan Shen, Zhe Gan, Haichao Zhang, Yizhe Zhang and Lawrence Carin “Adversarial Text Generation via Feature-Mover's Distance”, Neural Information Processing Systems (NeurIPS), 2018. PDF / Code
Xinyuan Zhang, Ricardo Henao, Zhe Gan, Yitong Li and Lawrence Carin “Multi-Label Learning from Medical Plain Text with Convolutional Residual Models”, Machine Learning for Healthcare (MLHC), 2018. PDF (Spotlight)
Yunchen Pu, Shuyang Dai, Zhe Gan, Weiyao Wang, Guoyin Wang, Yizhe Zhang, Ricardo Henao and Lawrence Carin “JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets”, Int. Conf. Machine Learning (ICML), 2018. PDF / Supp / Poster / Slides / Code
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang and Xiaodong He “AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks”, Computer Vision and Pattern Recognition (CVPR), 2018. PDF / Poster / Code / Blog
Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh and Lawrence Carin “Topic Compositional Neural Language Model”, Artificial Intelligence and Statistics (AISTATS), 2018. PDF
Yunchen Pu, Martin Renqiang Min, Zhe Gan and Lawrence Carin “Adaptive Feature Abstraction for Translating Video to Text”, Proc. American Association of Artificial Intelligence (AAAI), 2018. PDF

2017

Zhe Gan*, Liqun Chen*, Weiyao Wang, Yunchen Pu, Yizhe Zhang, Hao Liu, Chunyuan Li and Lawrence Carin “Triangle Generative Adversarial Networks”, Neural Information Processing Systems (NeurIPS), 2017. PDF / Poster / Code
Yunchen Pu, Weiyao Wang, Ricardo Henao, Liqun Chen, Zhe Gan, Chunyuan Li and Lawrence Carin “Adversarial Symmetric Variational Autoencoder”, Neural Information Processing Systems (NeurIPS), 2017. PDF
Yunchen Pu, Zhe Gan, Ricardo Henao, Chunyuan Li, Shaobo Han and Lawrence Carin “VAE Learning via Stein Variational Gradient Descent”, Neural Information Processing Systems (NeurIPS), 2017. PDF
Yizhe Zhang, Dinghan Shen, Guoying Wang, Zhe Gan, Ricardo Henao and Lawrence Carin “Deconvolutional Paragraph Representation Learning”, Neural Information Processing Systems (NeurIPS), 2017. PDF / Code
Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He and Lawrence Carin “Learning Generic Sentence Representations Using Convolutional Neural Networks”, Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2017. PDF / Slides / Code (Oral)
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen and Lawrence Carin “Adversarial Feature Matching for Text Generation”, Int. Conf. Machine Learning (ICML), 2017. PDF / Supp / Slides / Poster / Code
Yizhe Zhang, Changyou Chen, Zhe Gan, Ricardo Henao and Lawrence Carin “Stochastic Gradient Monomial Gamma Sampler”, Int. Conf. Machine Learning (ICML), 2017. PDF / Supp / Slides / Poster / Code
Zhe Gan*, Chunyuan Li*, Changyou Chen, Yunchen Pu, Qinliang Su and Lawrence Carin “Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling”, Association for Computational Linguistics (ACL), 2017. PDF / Supp / Slides / Code (Oral)
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin and Li Deng “Semantic Compositional Networks for Visual Captioning”, Computer Vision and Pattern Recognition (CVPR), 2017. PDF / Slides / Slides2 / Poster / Poster2 / Video / Code (Spotlight)
Chuang Gan, Zhe Gan, Xiaodong He, Jianfeng Gao and Li Deng “StyleNet: Generating Attractive Visual Captions with Styles”, Computer Vision and Pattern Recognition (CVPR), 2017. PDF / Data
Zhe Gan, P.D. Singh, Ameet Joshi, Xiaodong He, Jianshu Chen, Jianfeng Gao and Li Deng “Character-level Deep Conflation for Business Data Analytics”, Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2017. PDF / Slides / Code
Yin Xian, Yunchen Pu, Zhe Gan, Liang Lu and Andrew Thompson “Adaptive DCTNet for Audio Signal Classification”, Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2017. PDF
Qinliang Su, Xuejun Liao, Chunyuan Li, Zhe Gan and Lawrence Carin “Unsupervised Learning with Truncated Gaussian Graphical Models”, Proc. American Association of Artificial Intelligence (AAAI), 2017. PDF (Oral)

2016

Yizhe Zhang, Zhe Gan and Lawrence Carin “Generating Text via Adversarial Training”, Workshop on Adversarial Training, NeurIPS 2016. PDF / Code
Yin Xian, Yunchen Pu, Zhe Gan, Liang Lu and Andrew Thompson “Modified DCTNet for Audio Signals Classification”, Journal of the Acoustical Society of America, 2016. Link
Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew Stevens and Lawrence Carin “Variational Autoencoder for Deep Learning of Images, Labels and Captions”, Neural Information Processing Systems (NeurIPS), 2016. PDF / Poster
Jiaming Song, Zhe Gan and Lawrence Carin “Factored Temporal Sigmoid Belief Networks for Sequence Learning”, Int. Conf. Machine Learning (ICML), 2016. PDF / Supp / Poster / Slides
Chunyuan Li, Andrew Stevens, Changyou Chen, Yunchen Pu, Zhe Gan and Lawrence Carin "Learning Weight Uncertainty with Stochastic Gradient MCMC for Shape Classification", Computer Vision and Pattern Recognition (CVPR), 2016. PDF / Supp / Poster / Slides (Spotlight)
Changyou Chen, David Carlson, Zhe Gan, Chunyuan Li and Lawrence Carin “Bridging the Gap between Stochastic Gradient MCMC and Stochastic Optimization”, Artificial Intelligence and Statistics (AISTATS), 2016. PDF / Poster / Slides / Code (Oral)

2015

Ricardo Henao, Zhe Gan, James Lu and Lawrence Carin "Deep Poisson Factor Modeling", Neural Information Processing Systems (NeurIPS), 2015. PDF / Supp / Poster / Code
Zhe Gan, Chunyuan Li, Ricardo Henao, David Carlson and Lawrence Carin "Deep Temporal Sigmoid Belief Networks for Sequence Modeling", Neural Information Processing Systems (NeurIPS), 2015. PDF / Poster / Slides / Code
Zhe Gan, Changyou Chen, Ricardo Henao, David Carlson and Lawrence Carin "Scalable Deep Poisson Factor Analysis for Topic Modeling", Int. Conf. Machine Learning (ICML), 2015. PDF / Supp / Poster / Slides / Code
Zhe Gan, Ricardo Henao, David Carlson and Lawrence Carin "Learning Deep Sigmoid Belief Networks with Data Augmentation", Artificial Intelligence and Statistics (AISTATS), 2015. PDF / Supp / Poster / Code

Tutorial and Workshop

Zhe Gan, Licheng Yu, Yu Cheng, Luowei Zhou, Linjie Li, Yen-Chun Chen, Jingjing Liu and Xiaodong He "Recent Advances in Vision-and-Language Research", Computer Vision and Pattern Recognition (CVPR), 2020. Tutorial Website
Peter Knees and Zhe Gan "The ACM Multimedia 2020 Interactive Arts Exhibition". Website

Book Chapter

Zhe Gan, Xin Yuan, Ricardo Henao, Ephraim Tsalik and Lawrence Carin "Inference of Gene Networks Associated with the Host Response to Infectious Disease", Chapter 13 of the Book "Big Data Over Networks". Cambridge University Press. In Press. Link / PDF / Code

PhD Dissertation

Zhe Gan "Deep Generative Models for Vision and Language Intelligence", Duke University. PDF