Time/Location

Instructors

Stefano Ermon
Aditya Grover

Course Assistants

Yang Song
Jiaming Song
Rui Shu
Casey Chu
Nishith Khandwala
Kristy Choi

Course Description

Generative models are widely used in many subfields of AI and Machine Learning. Recent advances in parameterizing these models using deep neural networks, combined with progress in stochastic optimization methods, have enabled scalable modeling of complex, high-dimensional data including images, text, and speech. In this course, we will study the probabilistic foundations and learning algorithms for deep generative models, including Variational Autoencoders (VAE), Generative Adversarial Networks (GAN), autoregressive models, and normalizing flow models. The course will also discuss application areas that have benefitted from deep generative models, including computer vision, speech and natural language processing, graph mining, and reinforcement learning.

Prerequisites

Basic knowledge about machine learning from at least one of CS 221, 228, 229 or 230. Students will work with computational and mathematical models and should have a basic knowledge of probabilities and calculus. Proficiency in some programming language, preferably Python, required.

Piazza
Course notes

Suggested Reading

There is no required textbook. Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville. Online version available free here.

Grading Policy

Project Guidelines

The course project will give the students a chance to explore deep generative modeling in greater detail. Course projects will be done in groups of up to 3 students and can fall into one or more of the following categories:

Collaboration Policy and Honor Code

You are free to form study groups and discuss homeworks and projects. However, you must write up homeworks and code from scratch independently without referring to any notes from the joint session. You should not copy, refer to, or look at the solutions in preparing their answers from previous years’ homeworks. It is an honor code violation to intentionally refer to a previous year’s solutions, either official or written up by another student. Anybody violating the honor code will be referred to the Office of Judicial Affairs.

Submission Instructions

We will be using the GradeScope online submission system. All students (non-SCPD and SCPD) should submit their assignments electronically via GradeScope. Students can typeset or scan their homeworks.

To register for GradeScope,

Written Assignments: Homeworks should be written up clearly and succinctly; you may lose points if your answers are unclear or unnecessarily complicated. You are encouraged to use LaTeX to writeup your homeworks (here is a template), but this is not a requirement.

Late Homework: You have 6 late days which you can use at any time during the term without penalty. For a particular homework, you can use only two late days. Once you run out of late days, you will incur in a 25% penalty for each extra late day you use. Each late homework should be clearly marked as “Late” on the first page. No late days is allowed for deadlines related to the course project.

Regrade Policy: You may submit a regrade request if you believe that the course staff made an error in grading. Any regrade requests should be submitted through Gradescope within one week of receiving your grade. Please try to be as specific as possible with your regrade request.

Syllabus (tentative, periodically updated throughout the quarter)

Week Starting date Lecture topics Coursework Sections
1 Sep 24 Introduction and Background (slides 1, slides 2)    
2 Oct 1 Autoregressive Models (slides 3, slides 4) HW 1 released Probability and linear algebra, PyTorch
3 Oct 8 Variational Autoencoders (slides 5, slides 6)    
4 Oct 15 Normalizing Flow Models (slides 7, slides 8) HW 1 due, HW 2 released CNNs and RNNs (slides)
5 Oct 22 Generative Adversarial Networks (slides 9, slides 10) Project Proposal due  
6 Oct 29 Guest lecture 1: Tengyu Ma on Monday,
Evaluation of Generative Models on Wednesday (slides 11)
HW 2 due  
7 Nov 5 Combining generative model variants (slides 12), Energy-based models (slides 13) Mid term  
8 Nov 12 Guest lecture 2: Diederik P. Kingma on Monday,
Discreteness in Latent Variable Modeling on Wednesday (slides 14)
Project Progress Report due, HW 3 released  
9 Nov 19 - Thanksgiving break -    
10 Nov 26 Applications: Vision, Speech, Language, Graphs, Reinforcement learning Poster presentation on Friday  
11 Dec 3 Generative Adversarial Imitation Learning. No class on Wednesday. HW 3 due  
12 Dec 10 - Finals week - Final project reports due on
Dec 12th (no late days)
 

Additional Reading: Surveys and tutorials

  1. Tutorial on Deep Generative Models. Aditya Grover and Stefano Ermon. International Joint Conference on Artificial Intelligence, July 2018.
  2. Tutorial on Generative Adversarial Networks. Computer Vision and Pattern Recognition, June 2018.
  3. Tutorial on Deep Generative Models. Shakir Mohamed and Danilo Rezende. Uncertainty in Artificial Intelligence, July 2017.
  4. Tutorial on Generative Adversarial Networks. Ian Goodfellow. Neural Information Processing Systems, December 2016.
  5. Learning deep generative models. Ruslan Salakhutdinov. Annual Review of Statistics and Its Application, April 2015.

Poster Session

Time & Location

Friday, Nov. 30, 2018.
AT&T Patio, Gates Computer Science Building.

Session 1: 9:30 - 11:30 am
Session 2: 12:30 - 2:30 pm
Session 3: 2:30 - 4:30 pm

Project Teams and Titles

Session 1: 9:30 - 11:30 am

Team Title
Suraj Nair Learning Visual Representations for Robot Manipulation Using Stuctured Generation
Masha Itkina A Generative Approach to Urban Environment Prediction
Margaret Guo, Zhenqin Wu Molecule Generation with Iterative Refining Graph Variational Autoencoder
Marcella Cindy Prasetio Unsupervised Face-to-Manga Translation with CycleGAN
Abubakar Abid Deep Generative Models with Contrast
Aaron Effron, Abhijeet Shenoi, Alexia Wu Progressive Flow Models
Rewa Sood, Jing Lim, Ashwini Ramamoorthy Super Resolution of MR images using GANs
Samir Sen, Trevor Tsue Deep Generative Voice Style Transfer
Kunal Menda, Patrick Slade Learned Observation Models For Real-Time State Estimation
Ryan Holmdahl Generating Novel Molecules with Iterative Refining Graph Autoencoders
Christopher Yeh, Sherrie Wang, Rose Rustowicz Learning disentangled features for satellite imagery
Xuerong Xiao GAN-based segmentation of dendritic spines
Ruohan Zhan, Ruilin Li Knockoff GAN
Kun Ho Kim Dynamics Model Learning via Generative Adversarial Imitation Learning
Andrew Deveau, Youkow Homma Question-Answer Joke Generation
Oskar Triebe, Tymor Hamamsy, Will Lauer Modeling Electrical Loads With Generative Models
Meng Tang, Yimin Liu Deep Learning based proxy model for fluid flowthrough porous media
Panagiotis Lolas Time-series anomaly detection
Manon Romain Adversarial Object Enhancement
Animesh Koratana Learned Divergence Metrics for Knowledge Distillation
Yanbing Zhu GANs for Stable Materials Generation
Carlos Gomez Learning Variational Autoencoders Through The Laplace Approximation
Xizhi Han, Zhaoheng Guo Variational Monte Carlo Search for Quantum Many-body Ground States
Anchit Gupta, Jonathan Booher sim2real Domain Adaptation for robotic manipulation using GAN
Liyue Shen Representation Learning based on Deep Generative Model for Abnormally Detection

Session 2: 12:30 - 2:30 pm

Team. Title
Laetitia Shao Unrestricted Adversarial Training
Gael Colas, Rafael Rafailov Video Colorization using GAN
Malik Boudiaf, Ianis Boigdal-Lambert Imitating driving behavior in an urban environment
Jacob Hoffman GAN-Based Data Augmentation for Brain Leision Segmentation
Arnaud Autef, Amaury Sabran, Benjamin Petit Neural Processes for Image Completion
Pedro Garzon, Noa Glaser, Swetava Gangli GeoGAN: GeoSpatial Feature Learning Using Generative Adversarial Networks
Alex Poms Generative Models for General 3D Scenes
Shuvam Chakraborty, Vineet Kosaraju, Zaid Nabulsi Poverty Prediction from Learned Satellite Representations through Multiclass cGANs
Ramon Iglesias Latent Controllable Dynamics
Ramtin Keramati Exploration with controllable agent in Reinfrocement learning
Christopher Lazarus, Ines Chami, Marc Thibault Hallucigraph
Cody Kala, Owen Wang, Todd Macdonald RinneGAN: Frame Interpolation on Animated Videos using Generative Adversarial Networks
Kyle Julian Wildfire Image Generation Through Generative Adversarial Networks
Eric Zelikman Adversarial Models of Reality (AMoRe)
Davis Rempe Generating Physical Dynamics of 3D Rigid Objects
Akhila Yerukola, Amita Kamath Text Generation using GANs
Alex Haigh, Sam Schwager, Frits van Paasschen Deep Autoregressive Models for Conditional Graph Generation
Aldo Gael Carranza Structured Variational Autoencoders
Megha Jhunjhunwala, Shreyash Pandey, Vivekkumar Patel Fine grained text to image generation
Ana-Maria Istrate Text to Image using Stacked GANs
Josh Singer, Sean O’Bannon, Tyler Smith Frame Interpolation using Generative Adversarial Networks
Chaofei Fan Optimal Decoder for VAE
Will Deaderick, Abhishek Roushan AutoDoodler: Style Infusion for Automatic Doodle Artwork Generation
Marco Monteiro lip2lip

Session 3: 2:30 - 4:30 pm

Team Title
David Tagliamonti, Akihiro Matsukawa, Benjamin Goeing Image-to-Image Translation with Deep Invertible Generators
Nathan Dalal, Nikhil Sardana, Zhilin Jiang Towards Fast Generative Image Compression
Cagan Alkan, Beliz Gunel Generative Adversarial Network for Compressed Sensing MRI
Andrew Bylard Generating Coarse Seed Trajectories for Refinement
Eric Wang + Hugh Zhang A new method for chit-chat dialogue generation
Stefania Moroianu, Charbel Saad Voice conversion with Flow-GAN + CycleGAN
Jonathan Gomes Selman, Woody Wang Intuitive Teleoperation via VAEs
Apoorva Dornadula, Ashwini Pokle Improving Model Fairness…One Dataset At A Time
Lisa Lei Unpaired GANs for MR Parallel Imaging Reconstruction
Mark Sabini, Zahra Abdullah, Darrith Phan GAN-stronomy: Generative Cooking with Conditional DCGANs
Yuxing Chen, Zhefan Wang An Analysis of Various Image Inpainting Methods
Stephen Pfohl, Daisy Ding, Tony Duan Latent-Variable Models for Counterfactual Fairness in Cardiovascular Disease Risk Prediction
Shuyang Shi, Yongshang Wu Change Point Detection with Deep Generative Models
Toby Generating chord progressions with recurrent VAEeeeeeeeees
Chris Chute, Kelly Shen CycleFlow: Unpaired Image-to-image Translation with Normalizing Flows
Loren Amdahl-Culleton Neural FM Synthesis
Mansheej Paul, Phillip Etter Deep Tunes
Ji Won Park Classification of Astronomical Time Series with Representation Learning
Karen Leung Latent MDP
Alex Kolchinski, Andrew Kondrich CycleFlow
Xiao Chen, Thomas Navidi A GAN based method for fairness and privacy
Jen Weng Cycle Consistency in Normal Flow Models
Arjun Arora, William Zeng Frame Interpolation on Dance Covers using Generative Adversarial Networks
Ben Sorscher Simulating Gravitational Lenses with Deep Generative Models
Abhi Kulgod, Neel Yerneni, Varun Nambakrishnan MuseVAE: Mood Based Music Generation
Victoria Tsai, Sagnik Majumder, Patrick DeMIchele FlowGAN
Zhen Qin, Kuangcong Liu, Haoshen Hong Travelling Salesmen Problem Approximation with Probabilistic Generations
Peter Zachares, Michelle A Lee Multimodal Representation Models in Robotics
Xiaobai Ma Novel View Synthesis by Geometric Transformation and Image Completion Neural Network
Jiaqi Jiang, Chenzhuo Zhu GAN-enabled metasurface design
Dian Ang Yap Audio joint-source channel coding
Justin Dieter, Brian Lui Dual Repulsive BiGANs for Anomaly Detection

We gratefully acknowledge funding from the Stanford Computer Forum for sponsoring the poster session.