مطالعه موردی از دقت و صحت دلفی در بلند مدت

We evaluated the long-term accuracy of predictions from a Delphic poll of the future of the mental health profession [1]. Six hundred participants provided predictions about the likely occurrence and probable time courses for 18 scenarios that could possibly have occurred over the 30-year interval between 1981 to present. Each of the panelists was polled twice with distributional feedback from the first poll provided on the second poll questionnaires. The data from the second polling was used to make predictions regarding the future of various issues that may have affected the mental health profession over the forecast interval. It has now been 30 years since the original poll was conducted; the purpose of this study was to evaluate the accuracy of the original group predictions. Results indicated that the Delphi panelists correctly predicted the occurrence of 14/18 scenarios. For those scenarios that did occur, the time course predictions were accurate within approximately 1–5 years.

According to Wikipedia, the on-line encyclopedia, “The Delphi method [pron: delfi] is a structured communication technique, originally developed as a systematic, interactive forecasting method which relies on a panel of experts” [2]. The Delphi technology involves extracting a consensus opinion from a panel of experts regarding the likelihood of future scenarios, probability judgments, or for developing likely scenarios. Consensus, in the form of decreased variability of panelists' responses, usually increases with iterative polling over two or more rounds; it is assumed that decreased variability of the panelists' predictions with iteration increases the accuracy of the group forecasts on the later rounds. Usually a facilitator provides group consensus feedback regarding the various scenarios to the members of the group along with justifications for outlying opinions and then encourages the panel members to revise their predictions in light of this feedback. The procedure terminates after consensus is reached and the final group predictions are offered as an educated guess about the outcome of future events [3]. Although there are several versions of Delphi, [3], [4], [5] and [6] the basic technology involves four characteristics: Amalgamation of a group consensus opinion, iterative polling between rounds that include feedback, anonymity of participation, and statistical predictions of the probability of occurrence and likely time course for the scenarios. This combination of procedures usually produces predictions that are more accurate than chance or the average forecast from an individual group member [3]. Indeed, some published studies have shown that the structure of the Delphic polling procedure produces predictions that are more accurate than those obtained from unstructured groups such as group meetings, opinion polls, or focus groups [7] and [5]. Why does the Delphi Method produce accurate forecasts? Perhaps the most widely accepted explanation of the effect is the “Theory of Errors” proposed by Dalkey [8]. The basic assumption follows from the old adage; two heads are better than one. Extending this wisdom; N heads are better than two. In other words, forecasts (or decisions) that derive from the consensus of a structured group of individuals will be more accurate than those obtained from at least half of the group. Most groups provide a distribution of responses that may or may not contain the actual event when it occurs. However, when the true answer falls outside this range, either above or below it, then the median response of the group will be closer to the true answer than half of the group. When the true answer falls within the distribution of responses, the median response will usually be closer to the true answer than most of the group. In general, regardless of where the true answer falls along the time line, the group consensus prediction will usually be more accurate than at least half of the individuals in the group. There is no guarantee, however, that the group median response will be more accurate than the best individual predictions. The effect of providing feedback has a somewhat less obvious effect on accuracy. Feedback may move the response on later rounds in the direction of the true answer or towards the median response of the group that receives feedback during subsequent rounds. In most Delphi studies, providing feedback on subsequent rounds reduced variability of the panelists' responses. It was assumed that decreased variability was an index of consensus and that consensus was correlated with accuracy. Unfortunately, there have been few studies that have experimentally validated this assumption. Although some early research demonstrated that decreased variability of responding on iterated polls correlated with increased accuracy [9] and [10], later studies that separated out the effects of iterated polling with and without feedback did not find a consistent relationship between accuracy and variability [11] and [12]. Rowe and Wright [13] provided an excellent summary of Delphi accuracy in published studies up through 1999. Of the 21 published studies these authors reviewed, more than half of the studies reported accuracy levels after iterative polling that were higher than those obtained from an initial round of polling. Only two of the published studies reported the reverse effect. However, most of the early studies investigated accuracy of almanac predictions or short-term general interest events that could be evaluated within a constrained time frame, for example, everyday events like gas prices or election outcomes. Other studies have looked at accuracy of economic forecasts, weather predictions, or technological developments. Although most studies report improve accuracy of the Delphi Method relative to Face-To-Face discussions or more modern techniques like Prediction Markets [7], it is difficult to compare accuracy measures with such divergent measures. Although there are very good summaries of the Delphi procedure, [12] there are very few studies of long-term predictive accuracy of the Delphi Method. Indeed, a Google search of published studies of Delphi accuracy that included the keywords: Delphi, Prediction, Forecast, Long-Term, and Accuracy did not return a single study that spanned more than 10 years. The lack of research in this area is probably due to the lengthy time frame necessary to validate the outcomes. However, it is important to study long-term accuracy of the Delphi Method because it may be the only means of forecasting technological breakthroughs or unique inventions that have no sequential history (e.g., manned landing on the moon) or geopolitical events that are difficult to predict (e.g., terrorist attacks). It is therefore important to determine if the technique actually produces accurate forecasts for these types of events. Although there are several studies that have documented short-term Delphi accuracy, [2] the data presented here are quite possibly the only study of Delphi accuracy that extends over a 30-year interval.

When reviewing the literature on Delphi accuracy, our internet searches did not reveal any published studies of Delphic Polling accuracy that extended over an interval of 20 years or more. It is therefore safe to say that there are very few studies of long-term Delphi accuracy. Indeed, this study may be the only one in existence that has studied the accuracy of Delphic Polling over a 30-year time frame. In this study, the overall accuracy of long-term Delphic polling is approximately 14/18 for IF type predictions. This figure is the sum of the two types of accuracy, Hit and Correct Rejections. Nine of 14 scenarios that did not occur were correctly predicted; all of the scenarios that did occur were correctly predicted. Although there are several limitations to our evaluation of long-term Delphi accuracy presented here, these results suggest that the Delphi Method can produce a level of IF type accuracy that exceeds chance levels. Our best estimate of Delphi WHEN type forecasts for those events that did occur was approximately 1–5 years, depending on how one defines the time of occurrence. Although these professional groups made slightly different predictions of when the occurring scenarios would happen, the median prediction was slightly biased towards underestimating the time of occurrence which suggests that Delphi panelists may tend to predict that events will occur before they actually do occur. However, this tendency may be beneficial because it may motivate decision-makers to implement proactive measures before a problem occurs. The original Anderson, et al. [1] article did not provide a comparison of predictions from the first and second poll because although variability decreased on the second poll, the forecasts did not change. Another measure of consensus, the mean rank difference of predictions among the groups computed for each scenario was not significantly correlated with accuracy of the IF predictions in this study. In addition, there was also no obvious relationship between WHEN accuracy and consensus. It is therefore reasonable to suggest that there is little relationship between variability of responding and Delphi accuracy in these data. However, this conclusion deserves further consideration. Traditionally, the purpose of iterated polling in the Delphi Method has been to promote consensus among the panelist predictions over repeated polls; it has been generally assumed that consensus that increases over iterations is correlated with accuracy [15]. However, this assumption has not been consistently validated in this study or in previous studies of short-term forecast accuracy [16] and [12]. Although there was a small overall decrease in the variability of predictions in this study with repeated polling, there was no corresponding increase in accuracy. However, this does not mean that iterated polling is unnecessary for improving accuracy. It simply means that providing quantitative feedback to promote convergence of opinion may not be the most effective technique for improving accuracy. Perhaps a better emphasis would be to create conditions during the polling sessions that enhance the panelists' active cognitive processing of the scenarios. For example, simply polling panelists two or more times without feedback may increase processing of the scenarios free of any “follow the group” effects. Providing individual panelists' justifications for their predictions or incentives for accurate predictions may also improve accuracy. Graefe and Armstrong (2011) provided an excellent recent review and comparison of these procedures [17]. There are also different types of consensus and ways for achieving it. For example, consensus that derives from panelists with different perspectives or different types of expertise may be related to accuracy in a Delphic poll. Although in this study, the level of agreement among participant groups' predictions was not significantly correlated with accuracy across the 18 scenarios, a greater divergence of expertise may be necessary to increase accuracy.