Syllabus

1. Preliminaries: Implementing Data Manipulation, Data Preprocessing, Linear Algebra, Calculus, and Automatic Differentiation Operations.
2. Implement Linear Neural Networks for Regression and Classification Problems from scratch with a constant/epoch-varying learning rate.
3. Implementing a nonlinear Neural Network (Multilayer Perceptron) for Regression and Classification Problems from scratch with a constant/epoch-varying learning rate. (Consider implementing it for Stochastic Gradient Descent, Gradient Descent, and MiniBatch Gradient Descent). Consider using object-oriented programming concepts.
4. Implement a nonlinear Neural Network for Regression and Classification Problems from scratch, trained using Gradient Descent (GD), Momentum-Based GD, Nesterov Accelerated GD, and make a comparative analysis of their convergence.
5. Implement a nonlinear Neural Network for Regression and Classification Problems from scratch, trained using AdaGrad, RMSProp, and Adam, and make a comparative analysis of their convergence.
6. Implement a nonlinear Neural Network for Regression and Classification Problems trained using GD, Momentum-Based GD, Nesterov Accelerated GD, AdaGrad, RMSProp, and Adam, implemented using Tensorflow/Keras and PyTorch packages. Make a comparative analysis of their convergence. Consider different design alternatives for weight initialization, activation functions, dropout, batch normalization, and learning rate, and conduct a comparative analysis.
7. Tune the hyperparameters of deep learning models (implemented using Tensorflow/Keras and PyTorch packages) for Regression and Classification Problems using Grid Search, Random Search, and Bayesian Optimization.
8. Implement Convolutional Neural Networks (implemented using Tensorflow/Keras and PyTorch packages with hyperparameter optimization done) for Image Classification.
9. Implement different types of Transfer Learning (with early, late/intermediate fusions) for Image Classification.
10. Implement Recurrent Neural Network (RNNs), LSTM, GRU, Bidirectional Models from Scratch.
11. Implementing Attention Mechanism, Multi-Head Attention, Self-Attention in DL models for Regression and Classification Problems.
12. Implementing Vision Transformers and Large Language Models for related problems.

Understand and implement the fundamentals of feedforward, recurrent, and convolutional neural networks and apply them to solve various real-world problems.

Develop a deeper understanding of different optimization algorithms, such as gradient descent, stochastic gradient descent, and recent developments like ADAM and RMSprop, and Implement them from scratch.

Implement state-of-the-art architectures (CNNs, LSTMs, GRUs, Bidirectional Models, and Transformer Models) and learn to implement transfer learning with feature- and decision-level fusion.

Learn to implement improving the performance of DL models for a specific problem by optimizing hyperparameters.

Develop in depth understanding of the key deep learning models and concepts.

Gain practical skills and theoretical knowledge necessary to apply deep learning techniques to a wide range of problems.

Improve the performance (tuning hyperparameters) and computational efficiency of deep learning models for real-world problems.

Implementing an appropriate deep learning model from scratch for an application.

A. Zhang, Z.C. Lipton, M. Li, A.J. Smola, Dive into Deep Learning, Cambridge University Press , https://d2l.ai

Ian Goodfellow, Yoshua Bengio, Aaron Courville, Deep Learning, MIT Press , https://www.deeplearningbook.org/

Charu C. Aggarwal, Neural Networks and Deep Learning, Springer