Markov decision processes: discrete stochastic dynamic programming Martin L. Puterman
Publisher: Wiley-Interscience

This book presents a unified theory of dynamic programming and Markov decision processes and its application to a major field of operations research and operations management: inventory control. An MDP is a model of a dynamic system whose behavior varies with time. I start by focusing on two well-known algorithm examples ( fibonacci sequence and the knapsack problem), and in the next post I will move on to consider an example from economics, in particular, for a discrete time, discrete state Markov decision process (or reinforcement learning). The elements of an MDP model are the following [7]:(1)system states,(2)possible actions at each system state,(3)a reward or cost associated with each possible state-action pair,(4)next state transition probabilities for each possible state-action pair. Models are developed in discrete time as For these models, however, it seeks to be as comprehensive as possible, although finite horizon models in discrete time are not developed, since they are largely described in existing literature. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, Wiley, 2005. Downloads Handbook of Markov Decision Processes : Methods andMarkov decision processes: discrete stochastic dynamic programming. Dynamic programming (or DP) is a powerful optimization technique that consists of breaking a problem down into smaller sub-problems, where the sub-problems are not independent. A wide variety of stochastic control problems can be posed as Markov decision processes. However, determining an optimal control policy is intractable in many cases. Handbook of Markov Decision Processes : Methods and Applications .