مقایسه جلسات چهره به چهره ، گروه های اسمی، دلفی و بازارهای پیش بینی در تخمین کار

We conducted laboratory experiments for analyzing the accuracy of three structured approaches (nominal groups, Delphi, and prediction markets) relative to traditional face-to-face meetings (FTF). We recruited 227 participants (11 groups per method) who were required to solve a quantitative judgment task that did not involve distributed knowledge. This task consisted of ten factual questions, which required percentage estimates. While we did not find statistically significant differences in accuracy between the four methods overall, the results differed somewhat at the individual question level. Delphi was as accurate as FTF for eight questions and outperformed FTF for two questions. By comparison, prediction markets did not outperform FTF for any of the questions and were inferior for three questions. The relative performances of nominal groups and FTF were mixed and the differences were small. We also compared the results from the three structured approaches to prior individual estimates and staticized groups. The three structured approaches were more accurate than participants’ prior individual estimates. Delphi was also more accurate than staticized groups. Nominal groups and prediction markets provided little additional value relative to a simple average of the forecasts. In addition, we examined participants’ perceptions of the group and the group process. The participants rated personal communications more favorably than computer-mediated interactions. The group interactions in FTF and nominal groups were perceived as being highly cooperative and effective. Prediction markets were rated least favourably: prediction market participants were least satisfied with the group process and perceived their method as the most difficult.

In situations where a lack of appropriate or available information precludes one from using quantitative methods, it can be helpful to incorporate human judgment in order to improve the forecast. But how does one get the best forecast when aggregating information from a group of people? While organizations most commonly rely on unstructured face-to-face meetings, it is difficult to find evidence to support the use of this strategy (Armstrong, 2006). The literature suggests that structured approaches such as nominal groups or Delphi provide more accurate forecasts than traditional meetings. In recent years, there has been increasing interest in prediction markets as an approach for eliciting information from people, and a number of organizations have started to experiment with them. However, to date, we still do not know much about the performance of prediction markets. The available studies are limited and are often of a small scale. In particular, we do not know of any study that has analyzed their performance relative to either meetings or other structured group techniques for aggregating the knowledge in groups. Since the emergence of the field, there has been no meta-analysis that has analyzed the accuracy of prediction markets. To address this deficiency, we conducted laboratory experiments which compared unstructured meetings, nominal groups, Delphi, and prediction markets. We analyzed the relative accuracies of the four group techniques on a quantitative judgment task and examined the participants’ perceptions of their group and their group process.

We compared the relative accuracy of traditional FTF to that of three structured approaches (NGT, Delphi, and prediction markets) on a quantitative judgment task that required percentage estimates for ten factual questions. Over the ten questions, we did not find any statistically significant differences in accuracy between the four methods. However, the method results differed somewhat at the individual question level. Delphi was as accurate as FTF for eight questions and outperformed FTF for two questions. By comparison, prediction markets were unable to outperform FTF for any of the questions and were inferior for three questions. The relative performances of NGT and FTF were mixed and the differences were generally small. The three structured approaches were more accurate than the participants’ prior individual estimates. Delphi was also more accurate than staticized groups, in contrast to NGT and prediction markets. The results suggest that NGT and prediction markets provide little additional value over simple averages of forecasts in situations where the participants are all drawing upon similar information. Future research should evaluate the relative performances of the methods for more complex problems in more realistic environments. We also analyzed the participants’ perceptions of the group and of the group process as a whole. The participants rated methods involving personal communication (FTF and NGT) more favorably than the computer-mediated Delphi and prediction markets. In particular, the FTF and NGT participants experienced higher levels of cooperation among their groups and perceived the group interaction as more effective. Prediction markets were rated least favorably: the prediction market participants were least satisfied with the group process and rated their method highest in terms of difficulty of participation.