This Article Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Google Scholar Articles by Zadorojniy, A. Articles by Shwartz, A.

A Strongly Polynomial Algorithm for Controlled Queues

School of Electrical Engineering, Tel-Aviv University, Tel-Aviv 69978, Israel
School of Electrical Engineering, Tel-Aviv University, Tel-Aviv 69978, Israel
Department of Electrical Engineering,Technion, Haifa 32000, Israel
sasha{at}eng.tau.ac.il, http://www.eng.tau.ac.il/sasha/
guy{at}eng.tau.ac.il, http://www.eng.tau.ac.il/guy/
adam{at}ee.technion.ac.il, http://www.ee.technion.ac.il/adam/

We consider the problem of computing optimal policies of finite-state finite-action Markov decision processes (MDPs). A reduction to a continuum of constrained MDPs (CMDPs) is presented such that the optimal policies for these CMDPs constitute a path in a graph defined over the deterministic policies. This path contains, in particular, an optimal policy of the original MDP. We present an algorithm based on this new approach that finds this path, and thus an optimal policy. In the general case, this path might be exponentially long in the number of states and actions. We prove that the length of this path is polynomial if the MDP satisfies a coupling property. Thus we obtain a strongly polynomial algorithm for MDPs that satisfies the coupling property. We prove that discrete time versions of controlled M/M/1 queues induce MDPs that satisfy the coupling property. The only previously known polynomial algorithm for controlled M/M/1 queues in the expected average cost model is based on linear programming (and is not known to be strongly polynomial). Our algorithm works both for the discounted and expected average cost models, and the running time does not depend on the discount factor.

Key Words: Markov decision process; constrained Markov decision process; controlled queues; linear programming; M/M/1 queue; optimization
History: Received: July 16, 2008; revision received: April 19, 2009;