برنامه نویسی پویای شبکه عصبی برای طراحی سیاست های مدیریت شبکه مخزن آب

Stochastic dynamic programming (SDP) can improve the management of a multipurpose water reservoir by generating management policies which are efficient with respect to the management objectives (flood protection, water supply for irrigation, hydropower generation, etc.). The improvement in efficiency is even more remarkable for networks of reservoirs. Unfortunately, SDP is affected by the well-known ‘curse of dimensionality’, i.e. computational time and computer memory occupation increase exponentially with the dimension of the problem (number of reservoirs), and the problem rapidly becomes intractable. Neuro-dynamic programming (NDP) can sensibly mitigate this limitation by approximating Bellman functions with artificial neural networks (ANNs). In this paper the application of NDP to the problem of the management of reservoir networks is introduced. Results obtained in a real-world case study are finally presented.

Water reservoirs provide us with plenty of resources: they are a source of renewable and clean energy, they provide storage for drinking and irrigation water, and they are also a source of leisure. Their importance requires a careful management and the problem of designing optimal management policies has attracted the research community since the pioneering work of Rippl (1883). Historically, the first mathematical tool that was proposed to manage a reservoir was the release scheme (i.e., a sequence m0,m1,…mTm0,m1,…mT of water volumes to be released every time step—week, day, hour, ……—in a year, under ‘reference’ conditions). It was a very simple open-loop control scheme, which was partially improved in the following decades by associating it with a reference storage trajectory (rule curve), generally defined as the trajectory followed by the reservoir storage during an ‘average’ hydrologic year. Maas (Maas et al., 1962) was the first to criticise this approach noting that, by following this management approach, the reservoir manager ‘spills water when the storage exceeds the quantity specified by the rule curve and hopes for rain when it falls below’. Only towards the end of the 1960s, under the influence of new studies in the field of Control Theory and Operations Research, it was understood that the rational solution to the reservoir manager's problem was to automate the release decision through a feedback controller: given the reservoir storage, the release decision is provided by a control law, which is a monotone non-decreasing function of the reservoir storage (Piccardi & Soncini-Sessa, 1991). The time sequence of (usually periodic) control laws constitutes the management policy, which is, therefore, the solution of an optimal control problem. One of the most successful approach to the solution of this problem, whose intrinsic complexity lies in the contemporary occurrence of uncertainty and nonlinear dynamics, is stochastic dynamic programming (SDP) (Bellman, 1957; Bellman & Dreyfus, 1962). Although the SDP's underlying idea of a sequential decision-making process had already been suggested by Masse (1946), the first known application of dynamic programming to reservoir management, in the deterministic version, is due to Hall and Buras (1961). Since then, the popularity of the SDP approach has been increasing, and in 1989 it was found to be the most common optimisation technique for reservoir networks (Esogbue, 1989). Up to the 1980s, almost all the documented works dealt with the operation of single hydropower reservoirs and totally ignored stochasticity (see among the others Fults and Hancock, 1972, Hall, 1968, Heidari et al., 1971, Trott and Yeh, 1973 and Turgeon, 1981). Only later on, thanks to the rapid increase in computer performances, stochasticity and multipurpose reservoir networks were considered (see the review papers of Yakowitz, 1982 and Yeh, 1985, and the contributions of Gilbert and Shane, 1982, Read, 1989, Hooper et al., 1991, Vasiliadis and Karamouz, 1994, Tejada-Guibert et al., 1995 and Soncini-Sessa et al., 2001). Unfortunately, SDP is very sensible to increases in the dimensionality of the problem, i.e. adding reservoirs generates an exponential increase in the time required to find a solution to the recursive Bellman equation, which is the core of any SDP algorithm. This problem was named the ‘curse of dimensionality’ by Bellman himself and it prevented the application of the methodology to real-world water systems consisting of more than two or three reservoirs. This was only a partial limitation of the methodology, since most reservoirs were managed independently, given that the main objective was the maximisation of power production. Sometimes, irrigation was accounted for, but other objectives, such as environmental flows, rarely entered in the picture. Recently, Integrated Water Resources Management (Global Water Partnership, 2003 and Global Water Partnership, 2000) has been proposed as an emerging alternative to the reductionist and top-down approach that was central to water resource management in the last century. One of the main aims of the IWRM is to efficiently manage the water resource, while caring for the interests of all involved parties: from the socio-economical entities (economical sectors, local population, resource managers), administrative entities (countries, states, provinces, districts), to hydrological and ecological entities (catchments and ecosystems). Public participation and stakeholder involvement across sectors and administrative levels at the catchment scale are inherently connected to river basin and reservoir management: there is a wide consensus that managing water resources as if they were watertight compartments, without sharing information and decisions, is inefficient and it might result unsustainable on the long term. The implication of IWRM is that SDP is confronted with a new class of problems which it cannot solve and new tools and methodologies are therefore required, since it is no longer possible to design the optimal management policy by considering the optimisation of single reservoirs as independent units, the entire basin must be considered instead (World Commission on Dams, 2000). In the past decades many methods have been devised in order to overcome such SDP's limitations. Turgeon (1981) proposed the decomposition of a n-reservoirs control problem in n subproblems of two reservoirs, of which one is a reservoir of the original problem, while the other represents all the downstream reservoirs. In this way, the computational time grows up only linearly with the number of reservoirs. Saad, Turgeon, Bigras, and Duquette (1994) aggregated the whole reservoir network in a single storage unit, controlled by a vector of controls. Archibald, McKinnon, and Thomas (1997) suggested again a decomposition technique, where each subproblem included three reservoirs: a reservoir of the original problem and two other reservoirs representing the remaining downstream and upstream reservoirs, respectively. Thanks to this decomposition, the computational complexity turns out to be a quadratic function of the state dimension. Yet, all these approaches drastically simplify the system topology and, therefore, they may be applied only in a limited number of circumstances. Moreover, by implicitly assuming the reservoir units as the only responsible of the state dimension, they are practically useless when the problem includes other system components (catchment, irrigation district, etc.), for which a dynamic description would be the best option. The most conceptually flexible approach has been proposed by Nardini, Piccardi, and Soncini-Sessa (1994), who developed an heuristic control scheme (partial open-loop feedback control) based on the approximation of the off-line control problem with a sequence of simpler on-line problems, for which, however, a detailed and accurate description of the stochastic parts of the system is required. Another approach to tackle the ‘curse of dimensionality’ has been to remove nonlinearities and thus to reduce the original problem to a linear one, for which more efficient solution algorithms exist. Unfortunately, this approach is mainly valid for very large reservoirs, where saturation effects (reservoir full or empty) are extremely rare. Georgakakos and Marks (1987) tried to avoid this proposing the extended linear quadratic Gaussian method, an approach based on Pontriagyn's maximum principle, which is not affected by the dimensionality problem, but it requires the cost function to be quadratic. However, when dealing with multiple uses and stakeholders this hypothesis becomes often impossible to satisfy because of the difference in the shapes of the objectives associated with the different water uses and stakeholders. In conclusion, SDP is still the approach with the greatest flexibility, since it does not require stringent hypotheses on the shape of the objective functions, but it is now confronted with the requirement of the increase in the dimension of the problems. In this paper we propose a solution based on the application of neuro dynamic programming (NDP), an approach recently proposed by Bertsekas and Tsitsiklis (1996) and based on the functional approximation of the Bellman functions by means of artificial neural networks (ANNs). NDP has the same nice properties of SDP, but, thanks to the use of ANNs as approximations of the Bellman function, it is possible to find sub-optimal solutions for the water management problem for large-scale reservoir networks. The paper is organised as follows: in the next section the optimal control problem of a water reservoir network is formulated and its solution computed using SDP is illustrated in the following one. The fourth section proposes a NDP version of the successive approximation algorithm (SAA) (White, 1963), which was already adapted to the case of reservoir management policy design by Piccardi and Soncini-Sessa (1991). Finally, the results obtained by applying this approach to a real-world case study are presented.

An approach to the multipurpose water reservoir network management based on neuro-dynamic programming has been presented. Neuro-dynamic programming considerably reduces the computation time with respect to stochastic dynamic programming when the state space, used as training pattern, is sampled with a coarser grid, while the ANN, which approximates the Bellman function, still manages to maintain a good approximation performance. NDP also allows to reduce the amount of memory needed to store the Bellman functions during the solution of the optimal control problem. The results obtained both on test case studies and on a real-world case application are promising, but there is space for more research, especially on the efficient sampling of the discretised state space, trying to obtain the most efficient approximation of the Bellman function, and on the reduction of time required for the training of the ANN.