Course Details
Subject {L-T-P / C} : CS6510 : Deep Learning { 3-0-0 / 3}
Subject Nature : Theory
Coordinator : Dr. Puneet Kumar Jain
Syllabus
Module 1:
Mathematical Foundation: Geometry, Linear Algebra, Calculus, Probability and Statistics for Deep Learning.
Introduction to Neural Networks: Biological nervous system, artificial neuron and neural network, basic terminologies, activation functions (sigmoid, tanh, ReLU and its variants), single layer neural network, linear regression and Softmax regression as single layer neural network,
multilayer perceptron, Backpropagation Algorithm, training a single neuron model, training multilayer perceptron using Backpropagation algorithm.
Module 2:
Introduction to Deep Learning: Introduction to Deep Learning, Relationship between Artificial Intelligence, Machine Learning and Deep Learning. Exploding and Vanishing Gradient Problem, Dropout, Regularization, Weight Initialization (Xavier/Glorot Initializer, He Initializer, etc.),
Batch Normalization, Optimizers: Gradient Descent, Stochastic Gradient Descent (SGD), Minibatch Stochastic Gradient Descent, SGD with Momentum, Nesterov Accelerated Gradient (NAG), AdaGrad, AdaDelta, RMSProp, ADAM, Implementation of Deep Multilayer Perceptron
using Tensorflow / PyTorch.
Convolutional Neural Network (CNN): Introduction to CNN, Convolution: Padding and Strides, Convolution over RGB Images, Convolution Layer, Pooling, Data Augmentation, Popular CNN Models: LeNet, AlexNet, VGG, GoogLeNet, ResNet, DenseNet, Designing Convolutional Network Architectures, Transfer Learning, Fusion: Early Fusion (Feature Level Fusion), Late Fusion (Decision Level Fusion) and Intermediate Fusion.
Module 3:
Recurrent Neural Networks (RNN): Introduction and Basics of RNN, Backpropagation through time, Long Short-Term Memory (LSTM), Gated Recurrent Units (GRUs), Deep RNN, Bidirectional RNNs, Machine Translation, Encoder-Decoder Architecture, Sequence to
Sequence Learning for machine translation.
Nonparametric Statistical Tests: Wilcoxon Signed-Rank Test, Friedman Test, Kruskal-Wallis Test.
Attention Mechanism and Transformers: Sequence-to-Sequence Learning, Attention Mechanism, Self-Attention, Multi-Head Attention Representation Learning Transformer Architecture, Transformers for Vision.
Module 4:
Deep Generative Models: Generative Modelling, Variational Autoencoders (VAE), Generative Adversarial Networks (GANs), Combining VAEs and GANs.
Hyperparameter Optimization: Grid Search, Bayesian Optimization, Swarm and Evolutionary Algorithm for Hyperparameter Optimization.
Computational Performance: Compilers and Interpreters, Asynchronous Computation, Automatic Parallelism, Concise Implementation and Training on multiple GPUs
Course Objectives
- Understand the fundamentals of feedforward, recurrent and convolutional neural networks and apply them to solve various real-world problems.
- Develop a deeper understanding in different optimization algorithms gradient descent, stochastic gradient descent and recent developments like ADAM and RMSprop.
- Explore state-of-the-art architectures (CNN, LSTM, GRU, Bidirectional Models and Deep Generative Models) and learn transfer learning with feature and decision level fusion.
- Learn to improve the computational efficiency and performance of DL models for a specific problem by optimizing hyperparameters and employing GPUs
Course Outcomes
1. Develop in depth understanding of the key deep learning models and concepts. <br />2. Gain practical skills and theoretical knowledge necessary to apply deep learning techniques to a wide range of problems. <br />3. Improve the performance (tuning hyperparameters) and computational efficiency (using GPUs) of deep learning models for real-world problems. <br />4. Determining an appropriate deep learning model for an application.
Essential Reading
- A. Zhang, Z.C. Lipton, M. Li, A.J. Smola, Dive into Deep Learning, Cambridge University Press , Website: https://d2l.ai
- Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning, MIT Press , Website: https://www.deeplearningbook.org/
Supplementary Reading
- Francois Chollet, Deep Learning with Python, Manning Publishers
- E. Stevens, L. Antiga, T. Viehmann, Deep Learning with PyTorch, Manning Publishers
Journal and Conferences
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- AAAI Conference on Artificial Intelligence