# EE6150: Stochastic Modeling and the Theory of Queues

## Course Content

• Poisson processes

• Discrete-time Markov chains (DTMCs)

• Continuous-time Markov chains (CTMCs)

• Queueing models

• Introduction to Markov reward processes (if time permits)

• Mid-term: 25%

• Final exam: 50%

• Assignments: 25%

## Textbooks

• Vidyadhar Kulkarni, Modeling and analysis of stochastic systems.Third edition, Crc Press, 2016.

• Geoffrey Grimmett and David Stirzaker, Probability and random processes, Oxford university press, 2020.

• Robert Gallager, Stochastic processes: theory for applications. Cambridge University Press, 2013.

## Schedule

Lecture number Topics Covered Section reference
(Kulkarni's book)
Lecture 1 Exponential distribution 5.1
Lecture 2 Poisson processes: Definition
Shifted Poisson processes
5.2
Lecture 3 Poisson processes: Finite dimensional distributions
Alternate characterization
5.2
Lecture 4 Event times in a Poisson process 5.3
Lecture 5 Superposition and splitting of Poisson processes 5.4
Lecture 6 Non-homogeneous Poisson processes 5.5
Lecture 7 Problem solving session
Lecture 8 Discrete time Markov chain (DTMC): Definition, Examples
Chapman-Kolmogorov equations
2.1, 2.2, 2.4
Lecture 9 DTMC: Occupancy times 2.5
Lecture 10 Computing matrix powers (diagonalizable case)
Application to DTMCs
First passage time - definition
2.6, 3.1
Lecture 11 DTMC - First passage times
CDF, absorption probabilities
3.2, 3.3
Lecture 12 DTMC - First passage time
Analysis of random walks
Expectation of the first passage time
3.3, 3.4
Lecture 13 DTMC - Expectation of the first passage time 3.4
Lecture 14 DTMCs: Limiting behavior
A study using examples
4.1
Lecture 15 DTMC - limiting behavior: Classification of states
4.2
Lecture 16 Problem solving session
Lecture 17 Communicating classes, recurrence and transience 4.3
Lecture 18 Periodicity 4.3
Lecture 19 Determining transience/recurrence: infinite DTMCs 4.4
Lecture 20 Limiting behavior: transient DTMC 4.5
Lecture 21 Stationary distributions: Construction Grimmett-Stirzaker - Sec 6.4
Lecture 22 Stationary distributions: Existence Grimmett-Stirzaker - Sec 6.4
Lecture 23 Convergence to stationary distribution: Aperiodic case Grimmett-Stirzaker - Sec 6.4
Lecture 24 Convergence to stationary distribution: Cesaro sense Grimmett-Stirzaker - Sec 6.4
Lecture 25 Examples: limiting behavior 4.6
Lecture 26 DTMCs with costs 4.8
Lecture 27 Continuous time Markov chain (CTMC): Definition, alternate characterization 6.1
Lecture 28 CTMCs: Examples 6.2
Lecture 29 CTMC: Transient behavior, Chapman-Kolmogorov equations
Forward and backward equations
6.4
Lecture 30 Examples: Applications of forward equations
Occupancy times
6.4, 6.5
Lecture 31 Computation of P(t): Finite/infinite state space 6.6, 6.7
Lecture 32 Absorption probabilities, First-passage times 6.8
Lecture 33 CTMCs: Limiting behavior 6.9-6.11