System dynamics based simulation models are becoming increasingly popular in the analysis of important energy policy issues including global warming, deregulation, conservation and efficiency. The usefulness of these models is predicated on their ability to link observable patterns of behavior of a system to micro-level structures. This paper argues that the structural validity of the simulation model—right behavior for the right reasons—is a stringent measure to build confidence in a simulation model regardless of how well the model passes behavior validity tests. That leads to an outline of formal structural validity procedures available but less explored in system dynamics modeling ‘repertoire’. An illustration of a set of six tests for structural validity of a system dynamics model for energy policy analysis follows. Then using Theil inequality statistics, the behavior validity of the model is also tested. Finally, some conclusions on the increased appeal for simulation models for energy policy analysis and design are presented.
Simulation models have been developed and applied to both operational problems and policy issues in energy sector. However, the need and the evaluation criteria of model validation differs for each case. For instance, in the case of operational problems, the results of a model can be accepted or rejected by exposing the results to a face validity test (Hermann, 1967; Emshoff and Sission, 1970; Zebda, 2002). In a face validity test, experts assess how the model and its results are close to the real system. Model solutions can be tested in real world environments: e.g., efficiency of the oil refinery can be enhanced under the recommended actions (Finlay and Wilson, 1987; Gass, 1983).
In contrast, the majority of policy models such as system dynamics (SD) type model and agent-based models are built for the analysis of policy, exploration of possible future scenarios, and management purposes (Gass, 1983; Sterman, 1984; Oliva, 2003; Scholl, 2001). Especially, researchers from SD community have found system dynamics modeling to be useful in energy policy domain. For instance, system dynamics models have been developed and applied to national energy policy evaluation (Ford, 1983; Naill, 1992; Qudrat-Ullah and Davidsen, 2001), investments and uncertainty (Ford, 1985), conservation policy analysis (Ford and Bull, 1989), effects of agents on utility performance (Geraphty and Lyneis, 1985), inter-fuel substitution in OECED-European electricity production (Moxnes, 1990), privatization of electricity industry (Bunn and Larsen, 1992; Bunn et al., 1997), and energy efficiency analysis (Dyner et al., 1995). These models are also able to link observable patterns of behavior of a system (e.g., global warming) to micro-level structure (e.g., stock of production capital) and decision making processes (e.g., Kyoto Protocols). Despite these relevant modeling strengths, the overall appeal of system dynamics modeling approach to the wider community of energy policy modelers is limited at best.
All models are built with a purpose. The ultimate test and utility of any model depends on how successfully it has accomplished the purposed. MDESRAP was built with a purpose to assess the “1990–91 energy policy” of Pakistan (Pey, 1993). The validated version of MDESRAP was used to evaluate this policy. The key features of this policy were (i) the government's guarantee for payment of WAPDA and KESC power purchase obligations for 30 years, (ii) the government's guarantee for fuel supply from public sector entities, and (iii) attractive bulk power tariffs (6.5 cents per kilowatt-hour). Based on MDESRAP analysis, the key findings were (i) a major shift from coal and hydro power plants to oil and gas power plants, (ii) 6,287 millions barrels, of the cumulative oil imports for electricity generation, a 40% above the pre-policy imports, and (iii) the cumulative CO2 emissions level that is 48% above the pre-policy case (Qudrat-Ullah, 2005). These finding were shared with the energy policy makers who played a fundamental role in shaping the new “1996–96 energy policy” by the Government of Pakistan (Pey, 1997).
Moreover, there exist strong similarities between SD and agent-based modeling approaches: (i) both are unique in modeling nonlinear, complex systems such as energy systems, (ii) both assume that micro-structures of a system are responsible for its behavior, and (iii) both aim at discovering leverage points in complex systems. Modelers of agent-based models seek leverage points in rules and agents, while SD modelers do so in the feedback structure of a system (Scholl, 2001). Therefore, arguments made and the validity procedures illustrated in this article should equally benefit agent-based modeling community.
Although structural validity tests constitute one of the two general types of tests required to build confidence in an SD type simulation model, these tests nevertheless are the core of SD modeling validation process and have temporal precedence over behavior validity tests. Illustrations provided through the applications of six tests for the structural validity of MDESRAP can help the modelers (and users) in energy policy domain to lend an effective and tangible support to the process of building confidence in their simulation models.
