ارزیابی نظریه بازی، ایفای نقش و قضاوت بدون پشتوانه

Green’s study [Int. J. Forecasting (forthcoming)] on the accuracy of forecasting methods for conflicts does well against traditional scientific criteria. Moreover, it is useful, as it examines actual problems by comparing forecasting methods as they would be used in practice. Some biases exist in the design of the study and they favor game theory. As a result, the accuracy gain of game theory over unaided judgment may be illusory, and the advantage of role playing over game theory is likely to be greater than the 44% error reduction found by Green. The improved accuracy of role playing over game theory was consistent across situations. For those cases that simulated interactions among people with conflicting roles, game theory was no better than chance (28% correct), whereas role-playing was correct in 61% of the predictions.

In Armstrong (1997a), I reviewed Co-opetition by Brandenburger and Nalebuff (1996). Their use of game theory to analyze real-world situations seemed compelling. I concluded that it was unfortunate that the decision makers had not engaged the help of game theorists before they made their decisions. I had some misgivings about the book, however. For example, was there any evidence that game theory had led to better decisions or predictions in conflicts? So I contacted the authors. Brandenburger responded that he was not aware of any studies of the predictive validity of game theory, and I was unable to find any such studies. Many hundreds of academics have been working on game theory for half a century. Thus, it seems strange that finding evidence on its predictive validity is difficult. Imagine that hundreds of medical researchers spent half a century developing drugs without testing whether they worked as predicted. They would not be allowed to market their drugs. Kesten Green sent me an early draft of his paper in July 2000 (Green, 2002). I thought it was an important contribution because he: (1) described an important problem, (2) challenged existing beliefs, (3) obtained surprising results, (4) used simple methods, (5) provided full disclosure, and (6) explained it all clearly. In short, Green violated all the rules in the ‘Author’s Formula’ (Armstrong, 1982). That formula, based on a review of empirical research, was updated in Armstrong (1997b). Given Green’s violations of the formula, I expected that reviewers would reject the paper. To avoid rejection, with the permission of Jan deGooijer, Editor of the International Journal of Forecasting, I informed Green that his paper would be accepted, subject to reasonable responses to any substantive reviewers’ concerns. Green had been systematic in his own evaluation of his study. He rated the study on the 32 principles for the evaluation of forecasting methods from Armstrong (2001c). His study did well on 28 of the principles, poorly on three, with one judged as not relevant. I have reviewed these ratings and am in agreement. The ratings are at View the MathML source. I discuss whether: (1) the problem is important, (2) the findings are important, and (3) the study was done in a competent manner. I then provide suggestions for further research.

6. Conclusions The game theorists’ predictions were slightly more accurate than those from participants using unaided judgment, although the advantage may have arisen from biases in the design, such as the lack of experience on the part of the participants using unaided judgment. This advantage does not apply however, when the situations are restricted to conflicts involving a series of interactions among the parties in conflict. Role playing was substantially more accurate than game theory despite biases favoring game theory. In general, an examination of the procedures used in Green’s comparative study of forecasting methods supports his findings. That said, much can be learned from further study of these issues. Hopefully, game theory researchers (and others) will conduct empirical studies that would assess the value of game theory relative to other approaches. More important, despite the substantial benefits identified in research to date, little research has been done on the use of role playing as a forecasting technique. In particular, research is needed for simulated interactions in cases involving conflicts. We know little about how to best implement role playing and about the conditions under which it is most effective relative to other methods.