Time/Location
- Lectures: Mon/Wed 4:30-5:50 pm, Location: Room 320-105.
- Office Hours and Sections: See course calendar.
- Mid-term: November 9 (Friday), 6-9 pm, Location: Hewlett 200.
Instructors
Course Assistants
Yang Song
Jiaming Song
Rui Shu
Casey Chu
Nishith Khandwala
Kristy Choi
Course Description
Generative models are widely used in many subfields of AI and Machine Learning. Recent advances in parameterizing these models using deep neural networks, combined with progress in stochastic optimization methods, have enabled scalable modeling of complex, high-dimensional data including images, text, and speech. In this course, we will study the probabilistic foundations and learning algorithms for deep generative models, including Variational Autoencoders (VAE), Generative Adversarial Networks (GAN), autoregressive models, and normalizing flow models. The course will also discuss application areas that have benefitted from deep generative models, including computer vision, speech and natural language processing, graph mining, and reinforcement learning.
Prerequisites
Basic knowledge about machine learning from at least one of CS 221, 228, 229 or 230. Students will work with computational and mathematical models and should have a basic knowledge of probabilities and calculus. Proficiency in some programming language, preferably Python, required.
Useful links
Suggested Reading
There is no required textbook. Deep Learning by Ian Goodfellow, Yoshua Bengio, Aaron Courville. Online version available free here.
Grading Policy
- Three homeworks (15% each): mix of conceptual and programming based questions
- Midterm: 15%
- Course Project: 40%
- Proposal: 5%
- Progress Report: 10%
- Poster Presentation: 10%
- Final Report: 15%
Project Guidelines
The course project will give the students a chance to explore deep generative modeling in greater detail. Course projects will be done in groups of up to 3 students and can fall into one or more of the following categories:
- Application of deep generative models on a novel task/dataset
- Algorithmic improvements into the evaluation, learning and/or inference of deep generative models
- Theoretical analysis of any aspect of existing deep generative models
Collaboration Policy and Honor Code
You are free to form study groups and discuss homeworks and projects. However, you must write up homeworks and code from scratch independently without referring to any notes from the joint session. You should not copy, refer to, or look at the solutions in preparing their answers from previous years’ homeworks. It is an honor code violation to intentionally refer to a previous year’s solutions, either official or written up by another student. Anybody violating the honor code will be referred to the Office of Judicial Affairs.
Submission Instructions
We will be using the GradeScope online submission system. All students (non-SCPD and SCPD) should submit their assignments electronically via GradeScope. Students can typeset or scan their homeworks.
To register for GradeScope,
- Create an account on GradeScope if you don’t have one already.
- Here are some tips for submitting through Gradescope.
Written Assignments: Homeworks should be written up clearly and succinctly; you may lose points if your answers are unclear or unnecessarily complicated. You are encouraged to use LaTeX to writeup your homeworks (here is a template), but this is not a requirement.
Late Homework: You have 6 late days which you can use at any time during the term without penalty. For a particular homework, you can use only two late days. Once you run out of late days, you will incur in a 25% penalty for each extra late day you use. Each late homework should be clearly marked as “Late” on the first page. No late days is allowed for deadlines related to the course project.
Regrade Policy: You may submit a regrade request if you believe that the course staff made an error in grading. Any regrade requests should be submitted through Gradescope within one week of receiving your grade. Please try to be as specific as possible with your regrade request.
Syllabus (tentative, periodically updated throughout the quarter)
| Week | Starting date | Lecture topics | Coursework | Sections |
|---|---|---|---|---|
| 1 | Sep 24 | Introduction and Background (slides 1, slides 2) | ||
| 2 | Oct 1 | Autoregressive Models (slides 3, slides 4) | HW 1 released | Probability and linear algebra, PyTorch |
| 3 | Oct 8 | Variational Autoencoders (slides 5, slides 6) | ||
| 4 | Oct 15 | Normalizing Flow Models (slides 7, slides 8) | HW 1 due, HW 2 released | CNNs and RNNs (slides) |
| 5 | Oct 22 | Generative Adversarial Networks (slides 9, slides 10) | Project Proposal due | |
| 6 | Oct 29 | Guest lecture 1: Tengyu Ma on Monday, Evaluation of Generative Models on Wednesday (slides 11) |
HW 2 due | |
| 7 | Nov 5 | Combining generative model variants (slides 12), Energy-based models (slides 13) | Mid term | |
| 8 | Nov 12 | Guest lecture 2: Diederik P. Kingma on Monday, Discreteness in Latent Variable Modeling on Wednesday (slides 14) |
Project Progress Report due, HW 3 released | |
| 9 | Nov 19 | - Thanksgiving break - | ||
| 10 | Nov 26 | Applications: Vision, Speech, Language, Graphs, Reinforcement learning | Poster presentation on Friday | |
| 11 | Dec 3 | Generative Adversarial Imitation Learning. No class on Wednesday. | HW 3 due | |
| 12 | Dec 10 | - Finals week - | Final project reports due on Dec 12th (no late days) |
Additional Reading: Surveys and tutorials
- Tutorial on Deep Generative Models. Aditya Grover and Stefano Ermon. International Joint Conference on Artificial Intelligence, July 2018.
- Tutorial on Generative Adversarial Networks. Computer Vision and Pattern Recognition, June 2018.
- Tutorial on Deep Generative Models. Shakir Mohamed and Danilo Rezende. Uncertainty in Artificial Intelligence, July 2017.
- Tutorial on Generative Adversarial Networks. Ian Goodfellow. Neural Information Processing Systems, December 2016.
- Learning deep generative models. Ruslan Salakhutdinov. Annual Review of Statistics and Its Application, April 2015.
Poster Session
Time & Location
Friday, Nov. 30, 2018.
AT&T Patio, Gates Computer Science Building.
Session 1: 9:30 - 11:30 am
Session 2: 12:30 - 2:30 pm
Session 3: 2:30 - 4:30 pm
Project Teams and Titles
Session 1: 9:30 - 11:30 am
| Team | Title |
|---|---|
| Suraj Nair | Learning Visual Representations for Robot Manipulation Using Stuctured Generation |
| Masha Itkina | A Generative Approach to Urban Environment Prediction |
| Margaret Guo, Zhenqin Wu | Molecule Generation with Iterative Refining Graph Variational Autoencoder |
| Marcella Cindy Prasetio | Unsupervised Face-to-Manga Translation with CycleGAN |
| Abubakar Abid | Deep Generative Models with Contrast |
| Aaron Effron, Abhijeet Shenoi, Alexia Wu | Progressive Flow Models |
| Rewa Sood, Jing Lim, Ashwini Ramamoorthy | Super Resolution of MR images using GANs |
| Samir Sen, Trevor Tsue | Deep Generative Voice Style Transfer |
| Kunal Menda, Patrick Slade | Learned Observation Models For Real-Time State Estimation |
| Ryan Holmdahl | Generating Novel Molecules with Iterative Refining Graph Autoencoders |
| Christopher Yeh, Sherrie Wang, Rose Rustowicz | Learning disentangled features for satellite imagery |
| Xuerong Xiao | GAN-based segmentation of dendritic spines |
| Ruohan Zhan, Ruilin Li | Knockoff GAN |
| Kun Ho Kim | Dynamics Model Learning via Generative Adversarial Imitation Learning |
| Andrew Deveau, Youkow Homma | Question-Answer Joke Generation |
| Oskar Triebe, Tymor Hamamsy, Will Lauer | Modeling Electrical Loads With Generative Models |
| Meng Tang, Yimin Liu | Deep Learning based proxy model for fluid flowthrough porous media |
| Panagiotis Lolas | Time-series anomaly detection |
| Manon Romain | Adversarial Object Enhancement |
| Animesh Koratana | Learned Divergence Metrics for Knowledge Distillation |
| Yanbing Zhu | GANs for Stable Materials Generation |
| Carlos Gomez | Learning Variational Autoencoders Through The Laplace Approximation |
| Xizhi Han, Zhaoheng Guo | Variational Monte Carlo Search for Quantum Many-body Ground States |
| Anchit Gupta, Jonathan Booher | sim2real Domain Adaptation for robotic manipulation using GAN |
| Liyue Shen | Representation Learning based on Deep Generative Model for Abnormally Detection |
Session 2: 12:30 - 2:30 pm
| Team. | Title |
|---|---|
| Laetitia Shao | Unrestricted Adversarial Training |
| Gael Colas, Rafael Rafailov | Video Colorization using GAN |
| Malik Boudiaf, Ianis Boigdal-Lambert | Imitating driving behavior in an urban environment |
| Jacob Hoffman | GAN-Based Data Augmentation for Brain Leision Segmentation |
| Arnaud Autef, Amaury Sabran, Benjamin Petit | Neural Processes for Image Completion |
| Pedro Garzon, Noa Glaser, Swetava Gangli | GeoGAN: GeoSpatial Feature Learning Using Generative Adversarial Networks |
| Alex Poms | Generative Models for General 3D Scenes |
| Shuvam Chakraborty, Vineet Kosaraju, Zaid Nabulsi | Poverty Prediction from Learned Satellite Representations through Multiclass cGANs |
| Ramon Iglesias | Latent Controllable Dynamics |
| Ramtin Keramati | Exploration with controllable agent in Reinfrocement learning |
| Christopher Lazarus, Ines Chami, Marc Thibault | Hallucigraph |
| Cody Kala, Owen Wang, Todd Macdonald | RinneGAN: Frame Interpolation on Animated Videos using Generative Adversarial Networks |
| Kyle Julian | Wildfire Image Generation Through Generative Adversarial Networks |
| Eric Zelikman | Adversarial Models of Reality (AMoRe) |
| Davis Rempe | Generating Physical Dynamics of 3D Rigid Objects |
| Akhila Yerukola, Amita Kamath | Text Generation using GANs |
| Alex Haigh, Sam Schwager, Frits van Paasschen | Deep Autoregressive Models for Conditional Graph Generation |
| Aldo Gael Carranza | Structured Variational Autoencoders |
| Megha Jhunjhunwala, Shreyash Pandey, Vivekkumar Patel | Fine grained text to image generation |
| Ana-Maria Istrate | Text to Image using Stacked GANs |
| Josh Singer, Sean O’Bannon, Tyler Smith | Frame Interpolation using Generative Adversarial Networks |
| Chaofei Fan | Optimal Decoder for VAE |
| Will Deaderick, Abhishek Roushan | AutoDoodler: Style Infusion for Automatic Doodle Artwork Generation |
| Marco Monteiro | lip2lip |
Session 3: 2:30 - 4:30 pm
| Team | Title |
|---|---|
| David Tagliamonti, Akihiro Matsukawa, Benjamin Goeing | Image-to-Image Translation with Deep Invertible Generators |
| Nathan Dalal, Nikhil Sardana, Zhilin Jiang | Towards Fast Generative Image Compression |
| Cagan Alkan, Beliz Gunel | Generative Adversarial Network for Compressed Sensing MRI |
| Andrew Bylard | Generating Coarse Seed Trajectories for Refinement |
| Eric Wang + Hugh Zhang | A new method for chit-chat dialogue generation |
| Stefania Moroianu, Charbel Saad | Voice conversion with Flow-GAN + CycleGAN |
| Jonathan Gomes Selman, Woody Wang | Intuitive Teleoperation via VAEs |
| Apoorva Dornadula, Ashwini Pokle | Improving Model Fairness…One Dataset At A Time |
| Lisa Lei | Unpaired GANs for MR Parallel Imaging Reconstruction |
| Mark Sabini, Zahra Abdullah, Darrith Phan | GAN-stronomy: Generative Cooking with Conditional DCGANs |
| Yuxing Chen, Zhefan Wang | An Analysis of Various Image Inpainting Methods |
| Stephen Pfohl, Daisy Ding, Tony Duan | Latent-Variable Models for Counterfactual Fairness in Cardiovascular Disease Risk Prediction |
| Shuyang Shi, Yongshang Wu | Change Point Detection with Deep Generative Models |
| Toby | Generating chord progressions with recurrent VAEeeeeeeeees |
| Chris Chute, Kelly Shen | CycleFlow: Unpaired Image-to-image Translation with Normalizing Flows |
| Loren Amdahl-Culleton | Neural FM Synthesis |
| Mansheej Paul, Phillip Etter | Deep Tunes |
| Ji Won Park | Classification of Astronomical Time Series with Representation Learning |
| Karen Leung | Latent MDP |
| Alex Kolchinski, Andrew Kondrich | CycleFlow |
| Xiao Chen, Thomas Navidi | A GAN based method for fairness and privacy |
| Jen Weng | Cycle Consistency in Normal Flow Models |
| Arjun Arora, William Zeng | Frame Interpolation on Dance Covers using Generative Adversarial Networks |
| Ben Sorscher | Simulating Gravitational Lenses with Deep Generative Models |
| Abhi Kulgod, Neel Yerneni, Varun Nambakrishnan | MuseVAE: Mood Based Music Generation |
| Victoria Tsai, Sagnik Majumder, Patrick DeMIchele | FlowGAN |
| Zhen Qin, Kuangcong Liu, Haoshen Hong | Travelling Salesmen Problem Approximation with Probabilistic Generations |
| Peter Zachares, Michelle A Lee | Multimodal Representation Models in Robotics |
| Xiaobai Ma | Novel View Synthesis by Geometric Transformation and Image Completion Neural Network |
| Jiaqi Jiang, Chenzhuo Zhu | GAN-enabled metasurface design |
| Dian Ang Yap | Audio joint-source channel coding |
| Justin Dieter, Brian Lui | Dual Repulsive BiGANs for Anomaly Detection |
We gratefully acknowledge funding from the Stanford Computer Forum for sponsoring the poster session.