Computing a numerical solution to a periodic optimal control problem can be difficult, especially when the period is unknown. A method of approximating a solution to a stochastic optimal control problem using Markov chains was developed in [Krawczyk, J. B. (2001). A Markovian approximated solution to a portfolio management problem. Information Technology for Economics and Management, 1, http://www.item.woiz.polsl.pl/issue/journal1.htm]. This paper describes the application of that method to a periodic optimal control problem formulated in [Gaitsgory, V. & Rossomakhine, S. (2006). Linear programming approach to deterministic long run average problems of optimal control. SIAM Journal on Control and Optimization, 44(6), 2006–2037]. As a result, approximately optimal feedback rules are computed that can control the system both on and off the optimal orbit.
Gaitsgory and Rossomakhine (2006), develop a specialized linear programming technique for approximating solutions to long-run average optimal control problems. They illustrate this technique using a pair of un-discounted periodic optimal control problems, for which they obtain feedback rules for states belonging to the optimal orbit.
Periodic optimal control is especially applicable to situations requiring sensible long-term management of resources or externalities. For example, sustainable harvesting of an ecological resource is often best performed cyclically (He et al., 1995 and Xiao et al., 2006). Similarly, the cost of controlling an established invasive species may be reduced by periodic use of countermeasures. Power generation and a variety of other industrial processes are either necessarily periodic or most efficient when operated periodically (Bausa and Tsatsaronis, 2001, Maurer et al., 1998 and van Noorden et al., 2003). The improvements in efficiency yielded by periodic operation appear increasingly valuable in the current world.
The uncertainty intrinsic to such problems requires optimal solutions which allow for stochastic perturbations. This paper proposes a practical feedback approach for dealing with these.
It was probably in the 1970s and early 1980s that applying the maximum principle became a standard approach to periodic optimal control (see Bittanti and Guardabassi (1985), Colonius (1985) and Noldus (1974)). A series of papers were published in which optimal open-loop solutions were computed and the methods yielding such solutions extensively discussed (see, for example, Han, Feichtinger, and Hartl (1994) and Maurer et al. (1998)). These solutions remain popular and can be found in papers from a variety of disciplines (see, for example, Bausa and Tsatsaronis (2001), He et al. (1995), Mombaur, Bock, Schlöder, and Longman (2005), Xiao et al. (2006), and Zhao (2004)).
As feedback solutions are more robust1 than their open-loop counterparts (and thus of particular interest to control practitioners), research on the former has been growing. For example, Grüne and Semmler (2004)’s method is amenable to periodic controls. It produces feedback rules for states not necessarily confined to the optimal orbit. However, as the method is based on policy improvement, it requires that the optimization problem be discounted.2
Value iteration is a solution method applicable to finite-horizon (stochastic, discounted or otherwise) optimal control problems. In this paper, we use value iteration to compute a feedback solution to one of Gaitsgory and Rossomakhine (2006)’s periodic optimal control problems. This solution is used to obtain approximately optimal decision rules for controlling the system both on and off the optimal orbit. There is a very high degree of coincidence between the optimal performance indices and periods computed by our method and that of Gaitsgory and Rossomakhine (2006), and we conjecture that the former can be successfully applied to problems of this type.3 Further pursuit of this thread of research (see also Gaitsgory and Rossomakhine (2006) and Grüne and Semmler (2004)) appears worthwhile.
The importance of the numerical methods proposed here and in Gaitsgory and Rossomakhine (2006) (also in Bausa and Tsatsaronis (2001), Han et al. (1994) and Grüne and Semmler (2004) and several other “discipline oriented” publications — see below) stems from the role that long-run average optimal control problems and their periodic solutions play in optimization. These problems arise from models finding application in such diverse areas as chemical reaction engineering (van Noorden et al., 2003), ecology management (He et al., 1995 and Xiao et al., 2006), flight path planning (Speyer, 1996 and Zhao, 2004), production planning (Maurer et al., 1998), robotics (Mombaur et al., 2005) and vibration damping (Kasturi & Dupont, 1998). The mathematical model discussed here and in Gaitsgory and Rossomakhine (2006) is representative of this class of problems.
This paper is organized as follows. In Section 2, we describe a useful class of long-run average control problems having periodic optimal solutions. A problem representative of this class is then defined and discussed in Section 3. In Section 5, we show how SOCSol4L–specialized software that converts an optimal control problem into a Markov decision chain (see Azzato and Krawczyk (2006) and Krawczyk (2001); for a brief introduction to SOCSol4L see the Appendix)–should be used to solve this periodic optimal control problem. The method used by SOCSol4L is outlined in Section 4, while Section 6 summarizes our findings.
We have demonstrated that value iteration can deliver an approximately optimal periodic solution to a long-run average optimal control problem. The obtained solution is in feedback form and can control the periodic process both on and off the optimal orbit, in contrast to the existing solutions commented on in the Introduction. It follows that a more general optimization method than that presented in Gaitsgory and Rossomakhine (2006) (and, in some other sense, more general than that in Grüne and Semmler (2004)) has been made available to managers interested in maximization of the “mean” state given limited resources. Moreover, we have seen that the feedback rules generated by our method provide an easy means for determining a suitable control for the optimization problem at hand from any point in the state space within the chosen state bounds.
