اعتبارسنجی روش دلفی نوآورانه ی زمان واقعی - مقایسه روش شناختی بین مطالعات متعارف دلفی و زمان واقعی

A novel and innovative real-time Delphi technique is introduced in order to address previously identified weaknesses of the conventional Delphi method, such as complicated facilitator tasks, lack of real-time presentation of results, and difficulties in tracking progress over time. We demonstrate how the real-time (computer-based) method increases the efficiency of the process, accommodates expert availability, and reduces drop-out-rates. Modifications in the Delphi procedure (e.g. change of iteration principle) not only increase efficiency but also change the nature and process of the survey technique itself. By identifying and analysing three individual effects (initial condition effect, feedback effect, and iteration effect) we examine whether the modifications in the survey process cause deviations to the survey results. Empirical data obtained from both conventional as well as real-time Delphi studies is analysed based on multiple statistical analyses. The research findings indicate that significant differences between the two Delphi survey formats do not exist and final survey results are not affected by changes in the survey procedure.

Since its first application in the 1950s by the US RAND Corporation, Delphi has become a widely accepted and frequently used research method, especially for futures oriented research. The conventional Delphi can be defined as a method that aims at a consensus on a particular topic among a group of experts [1], while the procedure (paper-and-pencil version) follows an anonymous, multistage communication process based on several survey rounds. Previous publications prove that the technique is an established method for foresight activities and that Delphi outperforms other group formats such as statistical groups or standard interacting groups in terms of effectiveness [2] and [3]. Whilst the method itself has become more prevalent and mature, a number of researchers have focused on improving the procedure and tasks of Delphi [2], [4], [5], [6], [7], [8], [9], [10], [11], [12] and [13] due to critique on the method in the past. Related to the process and task characteristics of a conventional Delphi survey, Tapio [14] and Gordon and Pease [5] identify several major areas of improvement. They address the challenge of increasing efficiency in order to shorten the time to perform a Delphi survey. Moreover, the availability of experts and the drop-out-rate are identified as major methodological challenges: the repetitive and multiple feedback character of Delphi studies demands considerable time until such studies are completed, which can increase the likelihood of drop-outs [6] and [15]. In line with other researchers [e.g. [6], [16] and [17]], Gordon and Pease [5] therefore attempted to increase the efficiency of the Delphi survey procedure. By omitting sequential rounds and thereby shortening the time frame needed to perform such studies, Gordon and Pease improved the traditional Delphi survey process by calculating experts’ responses online in “real-time”. Furthermore, they addressed the issues of expert participation and simplifying the process by providing experts the opportunity to participate in the survey via the Internet. Although Gordon and Pease describe their innovative survey method and its benefits as successful, they also concluded that their online, real-time survey method exhibits some weaknesses such as the ease-of-use for the facilitator of the Delphi survey, the real-time presentation of results, and the tracking of progress over time, which all require further research and development. Moreover, modifications to the Delphi technique (e.g. round-less, real-time approach) not only increase efficiency in terms of procedure and questionnaire handling, they also change the nature and process of the survey technique itself. Delphi's principle of iteration is affected by immediate provision of feedback and by experts’ ability to reassess and adjust their responses as often as they want to. Consequently, the results of a real-time Delphi study could deviate from those of a conventional Delphi study. However, these possible effects have not been studied yet. This paper contributes to research on Delphi techniques in two ways. First, we introduce a novel real-time Delphi technique that overcomes the shortfalls identified by Gordon and Pease. This real-time Delphi method increases efficiency more than other online-based Delphi surveys and is more convenient for participants. Even though the use of the Internet may help to reduce the time required to conduct a Delphi survey in general, the conventional idea of having several survey rounds is still present. Consequently, experts have to be available and willing to participate in the conventional survey process for at least two survey rounds (assuming that the survey is based on only two survey rounds) and thus dedicate considerable resources into the survey process. If experts do not participate in a later survey round, the facilitator of the Delphi survey would register drop-out rates. In order to improve and simplify the process of conducting a Delphi study, we developed an internet-based, real-time Delphi survey format which operates without the necessity of having sequential rounds. Once an expert provides his/her initial assessment (which would correspond to survey round number 1 in a conventional Delphi study), he/she receives immediate feedback and is able to reassess the initial estimate (which would correspond to any later survey round in a conventional Delphi study). Consequently, experts need to dedicate fewer resources for participating in the Delphi survey process. In addition, we specifically illustrate how an internet-based, real-time Delphi survey tool can be equipped with innovative features such as easy-to-use facilitator portal, consensus portal, and graphical real-time feedback to make it easier for experts to participate and for survey moderators to facilitate. Second, we investigate whether modifying the survey procedure of a real-time Delphi study causes differences in study results. We identify three individual effects that may impact the results of the Delphi study: initial condition effect, feedback effect, and iteration effect. In the preparation phase of conducting real-time Delphi studies, it is necessary to provide a set of initial conditions so that even early respondents can receive feedback [5]. We examine whether these initial conditions adversely affect experts’ behaviour in the survey process. While experts participating in conventional Delphi surveys do not receive feedback until a certain amount of peers have finished the survey, real-time Delphi participants receive immediate feedback after their responses. Therefore, we compare the effect of feedback provided real-time versus the effect of feedback provided in sequential survey rounds on responses. Furthermore, we analyse whether the convergence processes (i.e. how consensus is achieved) between conventional, round-based Delphi surveys differ (significantly) from real-time Delphi formats. We conduct multiple statistical analyses based on data obtained from a Delphi study that utilizes our new real time Delphi technique and data from a conventional Delphi study to analyse the impact of the ‘initial condition effect’, ‘feedback effect’, and ‘iteration effect’. The remainder of the paper is organized as follows: Section 2 outlines the process and task characteristics of the newly developed real-time Delphi survey tool. By referring to two concrete Delphi studies, Section 3 compares the differences of the real-time Delphi technique with the conventional Delphi technique and the potential consequences on final results. The final section summarizes the methodological contribution of the paper and addresses limitations as well as future research areas.

The objective of this paper was twofold: We aimed to extend Gordon and Pease's [5] innovative idea of an internet-based, almost real-time Delphi survey. In order to do so, we developed additional features, as suggested by the authors to eliminate the weaknesses previously identified in the online Delphi method. Thereby, we refined the graphical layout and feedback provided to experts. By adhering to the four main characteristics of Delphi, we outlined how an internet-based, real-time Delphi survey tool equipped with these elements increases the validity and reliability, as well as usability considerably. Furthermore, we developed a consensus portal and graphical real-time feedback features to make participation more convenient for participants. In addition, the extended scope of operation of the internet-based, real-time Delphi study reduces the complexity and resources required to conduct this type of survey. By comparing a conventional Delphi survey with a real-time Delphi survey, it was shown that the internet-based, real-time Delphi works as effectively as the conventional, round-based Delphi method. Due to the modified iteration and feedback processes of the real-time Delphi method, several comparisons were made. By using a robust linear regression model, we studied the initial condition effect and feedback effect. In addition, we studied whether a potential iteration effect could affect the convergence building processes (i.e. the way consensus among participants is achieved) by applying an independent-samples t-test. Finally, by analysing the results of three identical projections in both studies, we demonstrated that the results of both studies do not differ. Thus, we were able to address the lack of replication of results, which has been one of the major criticisms about Delphi surveys in the past [42]. In summary, the comparison analyses showed no significant differences between conventional and real-time Delphi survey methods. Therefore, we argue that the results of the real-time Delphi survey tool are as robust as the results generated by a conventional Delphi survey. However, our research results are not exempt from some limitations. The comparison between conventional and real-time Delphi methods was based on two studies, whereby it might be beneficial to compare more than just two surveys. In addition, the comparisons are based on Delphi formats in which future projections had to be evaluated in terms of expected probability, impact, and desirability. Further research is required to study whether differences between conventional and real-time Delphi emerge if other types of questions, such as the estimation of times of occurrences, feasibility or urgency would have to be evaluated. In addition, we call attention to the fact that we used ‘alarming’ colour-codes in our feedback processes. Future research should address how the use of these alarming colours may influence experts’ correction behaviour. Moreover, we cannot rule out that the two survey methods do not differ due to the fact that adequate initial conditions were provided in the real-time Delphi study. Future analyses could study whether differences between conventional and real-time Delphi methods occur if other initial conditions were provided. In essence, Gordon and Pease's [5] idea of an online-based, real-time Delphi study method represents a considerable contribution to the scientific community and reduces the complexity of the conventional Delphi approach. Our research results support the validity and effectiveness of such an innovative approach and thus constitute a valid basis for additional applications of the real-time Delphi survey format