Pre-requisites | Evaluation | Logistics | Schedule | Quizzes/Assignments | What Next?

Note: The video lectures for this course are now available on youtube

Pre-requisites

---

There are no official pre-requisites for this course but it would help if you have done the following courses (preferably in the order mentioned below) :

• Calculus [Online course from MIT]

• Linear Algebra [CS6015 or equivalent] | [Online course from MIT]

• Probability Theory [CS6015 or equivalent] | [Online course from MIT]

• Non-linear Optimization [CS5020 or equivalent] | [First Course in Optimization by Prof. Soman (IITB) available on CDEEP]

• Pattern Recognition and Machine Learning [CS5691 or equivalent] | [Andrew Ng's ML course]

If you can solve most of this assignment then you are ready for this course!

Logistics

---

• Instructor: Mitesh M. Khapra

• When: Jan-May 2019

• Lectures: Slot C

• Where: CS24, CS Building, First Floor

• Teaching Assistants

  Name	Lab	Email	Working hours	Days
  Shweta Bhardwaj	RBCDSAI	shwetabhardwaj44@gmail.com	2-4 pm	Wed,Fri
  Ananya Sai B	RBCDSAI	cs16m037@smail.iitm.ac.in	2-4 pm	Tue, Fri
  Shubham Patel	RBCDSAI	cs17m051@smail.iitm.ac.in	4-5 pm	Thu,Fri
  Jaya Ingle	RBCDSAI	cs17m060@smail.iitm.ac.in	2-3 pm	Mon, Tue
  Vamsi Dikkala	CV Lab	dikkalavamsi@gmail.com	10-12 am	Tue, Wed, Thu
  Golla Satish Kumar Yadav	CSE Library	satti20417608@gmail.com	2-4 pm	Wed, Thu
  Harsh Kumar Rai	AIDB Lab	cs17m015@smail.iitm.ac.in	4-6 pm	Tue, Wed, Thu

Evaluation

---

60% Assignments | 10% Quiz 1 | 30% Endsem

Schedule

---

*T = Teaching mode - this version of the slides contains animations (is good for first time viewing)
*H = Handout mode - this version of the slides will be available soon and does not contain animations (is good for revision before exams and for printing)
*M = Module (each lecture is broken down into smaller modules)

Lecture#	Contents	Lecture Slides	Lecture Videos	Extra Reading Material
Lecture 0	Jargon Busting	Slides	-	-
Lecture 1	(Partial) History of Deep Learning, Deep Learning Success Stories	T | H	M1 | M2| M3 | M4 | M5| M6 | M7 | M8| M9	Juergen Schmidhuber, Deep Learning in Neural Networks: An Overview. 2014
Lecture 2	McCulloch Pitts Neuron, Thresholding Logic, Perceptrons, Perceptron Learning Algorithm and Convergence, Multilayer Perceptrons (MLPs), Representation Power of MLPs	T | H	M1 | M2| M3 | M4 | M5| M6 | M7 | M8	Chapters 1,2,3,4 from Neural Networks by Rojas
Lecture 3	Sigmoid Neurons, Gradient Descent, Feedforward Neural Networks, Representation Power of Feedforward Neural Networks	T | H	M1 | M2| M3 | M4 | M5	http://neuralnetworksanddeeplearning.com/chap4.html
Lecture 4	Feedforward Neural Networks, Backpropagation	T | H	M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10	See Lecture 2 [Training Neural Networks] by Hugo Larochelle
Lecture 5	Gradient Descent (GD), Momentum Based GD, Nesterov Accelerated GD, Stochastic GD, AdaGrad, RMSProp, Adam	T | H	M1 and M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M9 part 2	http://ruder.io/optimizing-gradient-descent/ http://cs231n.github.io/neural-networks-3/ AdaGrad RMSProp
Lecture 6	Eigenvalues and eigenvectors, Eigenvalue Decomposition, Basis, Principal Component Analysis and its interpretations, Singular Value Decomposition	T | H	M1 | M2| M3 | M4 | M5| M6 | M6 part 2 | M7| M8	PCA Tutorial 1 PCA tutorial 2 SVD Tutorial 1 SVD Tutorial 2
Lecture 7	Autoencoders and relation to PCA, Regularization in autoencoders, Denoising autoencoders, Sparse autoencoders, Contractive autoencoders	T | H	M1 | M2| M3 | M4 | M5| M6	Lecture 6 from Hugo Larochelle's course Denoising Autoencoders Sparse Autoencoders Contractive Autoencoders
Lecture 8	Bias Variance Tradeoff, L2 regularization, Early stopping, Dataset augmentation, Parameter sharing and tying, Injecting noise at input, Ensemble methods, Dropout	T | H	M1 | M2| M2 part 2 | M3 | M4| M5 | M6 | M7 | M8 | M9 | M10| M11	Chapter 7 from Deep Learning book Dropout Ali Ghodsi's Video Lectures on Regularization
Lecture 9	Greedy Layerwise Pre-training, Better activation functions, Better weight initialization methods, Batch Normalization	T | H	M1 | M2| M3 | M4 | M5	Batch Normalization Greedy Layerwise Pre-training Reducing the Dimensionality of Data with Networks Exploring Strategies for Training Deep Neural Networks The difficulty of training deep architectures and effect of unsupervised pre-training Why does Unsupervised Pre-Training Help?
Lecture 10	Learning Vectorial Representations Of Words	T | H	M1 | M2| M3 | M3 part 2 | M4| M5 | M5 part 2 | M6 | M7 | M8 | M9 | M10	Deriving Mikiolov's word2vec word2vec parameter learning explained Blog 1 Blog 2 Blog 3
Lecture 11	Convolutional Neural Networks, LeNet, AlexNet, ZF-Net, VGGNet, GoogLeNet, ResNet	T | H	M1 | M2| M3 | M3 part 2 | M4| M4 part 2 | M5	Chapter 9 AlexNet ZFNet VGGNet 9 Deep Learning papers GooLeNet ResNet
Lecture 12	Object Detection, RCNN, Fast RCNN, Faster RCNN, YOLO	T | H	Will be available soon	RCNN Fast RCNN Faster RCNN YOLO
Lecture 13	Visualizing Convolutional Neural Networks, Guided Backpropagation, Deep Dream, Deep Art, Fooling Convolutional Neural Networks	T | H	M1 | M2| M3 | M4 | M5| M6 | M7 | M8 | M9 | M10	RCNN(See the section on visualizations) ZFNet(see the section on experiments) Striving for simplicity(Guided Backpropagation) DeepVis Deep Inside Convolutional Networks Understanding Convolutional Networks Understanding Deep Image Representations DeepDream A Neural Algorithm for Artistic Style Neural style Intriguing properties of neural networks Deep Neural Networks are easily fooled
Lecture 14	Recurrent Neural Networks, Backpropagation Through Time (BPTT), Vanishing and Exploding Gradients, Truncated BPTT	T | H	M1 | M2| M3 | M4 | M5	On the difficulty of training RNNs
Lecture 15	Gated Recurrent Units (GRUs), Long Short Term Memory (LSTM) Cells, Solving the vanidhing gradient problem with LSTMs	T | H	M1 | M2| M3 | M3 part 2	LSTM: A Search Space Odyssey Understanding Deriving and Extending the LSTM Understanding LSTM Networks (Nice blog) Paper which introduced Gated Recurrent Units Paper which introduced LSTMs
Lecture 16	Encoder Decoder Models, Attention Mechanism, Attention over images, Hierarchical Attention	T | H	M1 | M2| M3 | M3 part 2 | M4| M5	Attention-based Encoder–Decoder Networks
Lecture 17	Directed Graphical Models	T | H	Will be available soon	Graphical Models in a Nutshell
Lecture 18	Markov Networks	T | H	Will be available soon
Lecture 19	Using joint distributions for classification and sampling, Latent Variables, Restricted Boltzmann Machines, Unsupervised Learning, Motivation for Sampling	T | H	Will be available soon	An Introduction to Restricted Boltzmann Machines
Lecture 20	Markov Chains, Gibbs Sampling for training RBMs, Contrastive Divergence for training RBMs	T | H	Will be available soon	A Practical Guide to Training RBMs Training RBMs
Lecture 21	Variational autoencoders	T | H	Will be available soon	Tutorial on Variational Autoencoders by Carl Doersch Blog by Jaan Altosaar
Lecture 22	Autoregressive Models: NADE, MADE, PixelRNN	T | H	Will be available soon	NADE paper MADE paper PixelRNN paper
Lecture 23	Generative Adversarial Networks (GANs)	T | H	Will be available soon

Quizzes/Assigments

---

	Topics	Resources	Release Date	Submission Date
Assignment 1	Derivatives + Probability [Theory]	PDF | Source Code	21-Jan-2019	27-Jan-2019
Assignment 2	Linear Algebra [Theory]	PDF | Source Code	24-Jan-2019	31-Jan-2019
Assignment 3	Backpropagation [Programming]	-	31-Jan-2019	12-Feb-2019
Quiz 1	Lectures 1-7	-	20-Feb-2019
Assignment 4	Convolutional Neural Networks [Programming]	-	28-Feb-2019	15-Mar-2019
Quiz II	Lectures 8-15	-	27-Apr-2019
Assignment 5	Recurrent Neural Networks [Programming]	-	15-Mar-2019	10-Apr-2019
Assignment 6	Probability Refresher [Theory]	-	10-Apr-2019	15-Apr-2019
Assignment 7	Variational Autoencoders [Programming]	-	10-Apr-2019	26-Apr-2019
End Sem	Lectures 1-23	-	01-May-2019

What Next?

---

You may find the following courses to be useful:

• Deep Learning for Computer Vision [from Stanford]

• Deep Learning for NLP [from Stanford]