Stochastic Processes: Definition and examples of stochastic processes, Classifications of stochastic processes, Markov chains: Definition and examples, Transition Probability matrices, Chapman-Kolmogorov equations, Random walk, Classification of states of a Markov chain, Determination of higher-order transition probabilities, Graph-theoretic approach, Markov chains with a denumerable number of states, Reducible Markov chains, Markov Chains with continuous state spaces, Markov chains in continuous time: General pure birth and death processes, Yule-Furry Process, Chapman-Kolmogorov forward and backward differential equations for continuous-time Markov chain. Renewal processes: Renewal processes in continuous time, renewal equation, Renewal theorems, Residual and excess lifetime, Stochastic processes in queuing and reliability: General concepts of queuing systems, Steady-state and transient behavior, Birth and death process in queuing theory, (M/M/1) and (M/M/s) queuing models. Introduction to Brownian motion: Wiener processes, Differential equations for a Wiener process, Kolmogorov equations. Numerical solutions of stochastic differential equations.

The course will consider Markov processes in discrete and continuous time. The theory is illustrated with examples from operation research, biology and economy.

After completed the course, the students are expected to be able to:

Carry out derivations involving conditional probability distributions and conditional expectations.
Define basic concepts from the theory of Markov chains and present proofs for the most important theorems.
Compute probabilities of transition between states and return to the initial state after long time intervals in Markov chains.
Identify classes of states in Markov chains and characterize the classes.
Determine limit probabilities in Markov chains after an infinitely long period.
Derive differential equations for time-continuous Markov processes with a discrete state space.
Solve differential equations for distributions and expectations in time-continuous processes and determine corresponding limit distributions.

J Medhi, Stochastic Processes, New Age Publishers , Second Edition, Reprint 2007.

S. Karlin and H. M. Taylor, A First Course in Stochastic Processes, Academic Press , Academic Press, 1975.

Sheldon M. Ross, Stochastic Processes, Wiley india Pvt. Ltd , Wiley india Pvt. Ltd, 2008.

V K Rohatgi and A. K. Md. E. Saleh, An Introduction to Probability and Statistics, Wiley and Sons, 2001 , Second Edition