The results of Delphi studies are generally considered to be reliable and valid end products. However, in many research programs the nature of the results actually makes it difficult to test this assumption. In experimental research programs, long-term forecasts have therefore been replaced by short-term forecasts or almanac items that can be assessed more rapidly [1]. Other research programs implement Delphi studies for a variety of goals, such as the development of guidelines, the formulation of priorities, the specification of educational content, or the identification of promising technologies. These research programs have the potential to investigate the findings further, via parallel and/or follow-up research. Yet, few authors capitalize on this potential and therefore miss an opportunity to enhance and support their Delphi findings [2] and [3]. Based on our own use of Delphi in applied research programs [3] and [4], we will argue that further inquiry of Delphi findings is both theoretically and practically meaningful and can substantially improve the quality of the end product.
We shall first present a conceptual and theoretical rationale for the refinement of Delphi findings via a multiple-method, multiple-informant approach. We shall then discuss some published examples of Delphi studies that involved additional inquiry. Next, we shall detail our program of research in family-focused prevention and present a comparison of the findings of the initial Delphi study with follow-up research findings. Finally, we shall draw some conclusions regarding the benefits of further research focusing on enhancing the credibility of Delphi results.
In this paper we presented an example of how Delphi findings can be enhanced through further inquiry. The FIT framework that we developed in the initial Delphi study subsequently guided the program implementation in four communities. We evaluated this implementation, as well as the experiences of two communities that could not start the program, via follow-up research using individual interviews and focus group interviews. The analysis of these interviews allowed for a comparison of the findings with those of the initial Delphi study. It appears to us that the unique combination of implementation and evaluation of the FIT framework enhanced the credibility and applicability of our findings in several respects. Moreover, our methodological approach may inform future enhancement of Delphi research findings.
Each of the five key Delphi themes was enhanced by the follow-up inquiry. One of the most remarkable elements of support is the appreciation expressed both by the practitioner networks and by the parents for the inclusion of adolescents in the program. Throughout the Delphi study this approach was the subject of intense debate, presumably because it is conspicuously absent from youth prevention [50]. However, the inclusion of adolescents appeared to facilitate the organization of the program and the engagement of parents and adolescents once sufficient families were recruited. It would appear that the inclusion of adolescents is not only theoretically, but also organizationally and personally meaningful (cf. [51]). Similarly, practitioners and families supported a flexible approach to the program content and techniques as proposed in the Delphi study. The reasons for doing so were organizational and personal, because flexibility allowed for the adaptation of the program to group and personal needs and concerns (cf. [38]). In addition, the interviews revealed that the adolescents particularly valued the integration of content and techniques, as that distinguished the program from other academic-type activities. This personal adolescent perspective did not come up in the Delphi study. Similar refinements to the FIT framework were identified in the parent interviews. The parents said that not only were the intakes organizationally informative, but that they also personally encouraged them to enroll in the program. The parents also attached considerable importance to the group leaders' sharing of personal experiences during the program, an aspect that was not included in the initial list of desirable characteristics of group leaders. The implementation strategies are a last aspect of the FIT framework that could be refined via the interviews. Here the practitioners adduced organizational motives to argue for groups of minimum seven families. The parents asked for one or two extra sessions in the parent and the adolescent modules. Overall, the further inquiries contributed substantially towards the consideration of multiple perspectives in the FIT framework. The interviews with practitioners and participants particularly strengthened the organizational and personal perspectives in the framework and thus enhanced its credibility.
The follow-up inquiries presented here are also relevant to the applicability of the FIT framework. A first check of the applicability occurred through the selection of the communities that attempted to implement the program. We presented our framework to potentially interested practitioners in nine communities. Based on a mixture of arguments, six of them decided they would attempt to implement the program. The second applicability check was the actual implementation of the program. Although implementation was successful in four communities, the program could not start in two other communities. Clearly, this can and should not be attributed to the framework per se, as there are many other key factors in program implementation [52]. Nevertheless, we should not preclude the possibility of a higher success rate via an enhanced framework. Only the third check of the applicability of the FIT framework was formal. Specifically, we asked the practitioner networks to comment on the usefulness of each aspect of the framework. They generally agreed with the framework and considered it to be highly applicable. Interestingly, the differences in perceived applicability between the practitioner networks that succeeded and those that did not succeed in implementing the FIT program were limited. The latter expressed more concerns over the feasibility of recruiting families and over the length of the program. However, they mainly invoked contextual factors to explain its failure, such as the specificity of their community and inadequate timing of the attempted implementation.
Some limitations of our findings should be noted. The evaluation study essentially depended on the self-selection of the practitioners and the families who engaged in the program implementation. Ethically, we could not select a random sample of practitioners who would engage in the organization of the program, but depended on their personal and their institutions' willingness to engage. Therefore, a follow-up inquiry with random groups of practitioners to whom the FIT program is unknown could yield divergent or additional findings. In particular, a formal judgment by practitioners who are independent of the program might yield interesting information regarding the applicability of the framework. Although such a formal check of the applicability of a framework takes place mostly before its implementation [2], it might still provide some interesting opinions now that implementation has succeeded. A similar critique applies to the self-selected sample of parents and adolescents in the evaluation study. As participation in the program was voluntary, everyone decided long before the evaluation study that the program would somehow improve their situation. Therefore, and in spite of the high participation rates (91%), the opinions of the parents and the adolescent can only be positively biased. Although an independent inquiry with independent practitioners might counterbalance such a bias, a similar independent inquiry with a random sample of parents and adolescents seems impossible to conduct. The experience of being a participant in the program seems a condition sine qua non for reflexivity upon it.
The second limitation of the study lies in the methods we used. We conducted individual and focus group interviews with a view to further investigating the Delphi results. Although this procedure clearly allowed us to enhance the FIT framework, it cannot be concluded that our use of methods was the best possible. In particular, others have used individual and group interviews prior to and concurrently with a Delphi study [21] and [30]. Perhaps a similar design might have reduced the—albeit limited—dropout rate in our Delphi study and might have improved the discussion of multiple perspectives [30] and [33]. Moreover, other methods of follow-up inquiry, such as narratives or observation [3], might reveal additional or divergent clarifications of the framework.
These limitations notwithstanding, this article illustrated the adequacy of conducting follow-up research to support and refine Delphi findings. The implementation and evaluation of the findings added to their credibility and applicability. Previously published follow-up inquiries were conducted to improve credibility or as a first test of the applicability of the findings [2], [3] and [36]. Here we moved a step further in that we actually implemented the findings and subsequently sought to support and refine the Delphi findings. Similar research designs can be implemented in projects concerned with, among other things, the development of frameworks, guidelines, protocols, and quality indicators. Follow-up research can also enhance the recommendations of technology Delphi studies. Often it is assumed that the forecasting nature of these studies necessitates the postponement of follow-up inquiries until their time horizon has elapsed [32]. In our view there exist other possibilities. Follow-up designs can, for example, be implemented in the case of technology Delphi studies that are conducted to identify potential regional or national growth niches. Such inquiries can involve policy makers, investors and/or consumers in order to enhance the Delphi findings, e.g., as additional perspectives are brought in. No inquiry is definitive, and a Delphi study is no exception.
