Syllabus

Module 1: Basics of Probability, Discrete and Continuous Random Variables, Proability Mass and Density Functions, Conditional Probability, Independence. Monte Carlo simulation. (6 hours)

Module 2: Stochastic processes - Stationary and non-stationary process, Random walk, Wiener process, Gaussian process, Markov process, White noise. Discrete time system - Stochastic Difference Equation and solution, Continuous time system - Stochastic Differential Equation, basic stochastic calculus. (10 hours)

Module 3: Prediction and Optimization - State estimation for discrete time systems, Kalman - Bucy filter, Separation principle and LQG Control, Loop transfer recovery. (8 hours)

Module 4: Stochastic optimal control - Stochastic linear quadratic control, Finite horizon control, Markov chains, Markov Decision Process (MDP), Markov Reward Process (MRP), Dynamic Programming method, Value iteration, Policy iteration, Monte Carlo methods. (10 hours)

Module 5: Introduction to learning - Temporal Difference learning, SARSA, Q-Learning. (6 hours)

The course will provide an understanding of stochastic processes.

The course will provide an understanding of control and prediction problems under uncertainty.

The course will provide an understanding of the main results in stochastic optimal control and how they are used in various applications.

At the end of the course, students will be able to
CO1: Demonstrate basic knowledge in modeling stochastic processes.
CO2: Formulate and solve prediction problems in the presence of stochasticity.
CO3: Formulate and solve stochastic optimal control problems.
CO4: Use the dynamic programming method to determine best policy under uncertainty.
CO5: Extend the ideas of stochastic control to model-free methods.
CO6: Use computational tools such as MATLAB or Python to implement the methods learned in the course.

K. J. Astrom, Introduction to Stochastic Control Theory, Dover , 2004

D. Bertsekas, Dynamic programming and Optimal Control, Athena Scientific , 2007

R. S. Sutton and A. G. Barto, Reinforcement Learning, MIT Press , 2018

H. Kwakernaak and R. Sivan, Linear Optimal Control, John Wiley , 1972