مسائل روش شناختی و مفهومی در برخورد با مطالعه دلفی بین کشوری برای ارزیابی برنامه آموزشی

Although the Delphi is widely used, research on certain methodological issues is somewhat limited. After a brief introduction to the strengths, limitations, and methodological challenges of the technique, we share our experiences (as well as problems encountered) with an electronic Delphi of educational program evaluation (EPE) in the Asia-Pacific region. The study is described followed by a discussion of the difficulties in participant recruitment and selection, sample size, instrumentation, data collection and analysis and attempts to resolve them. Some of these problems are generic to the Delphi whereas others related to the specifics of the investigation. What we learned should be useful for future Delphis with a similar focus.

The Delphi technique (referred to as the Delphi) has been recognized as an effective method for reaching consensus and/or forecasting future events. It is a good vehicle for collecting opinions from geographically dispersed experts who cannot meet face to face (Delbecg, Van de Ven, & Gustafson, 1975; Linstone & Turoff, 1975). Delphi has been widely used in numerous settings (industry, government, and academe) and in education the procedure has been employed for curriculum development, institutional planning, and other similar matters (Clayton, 1997). Delphi as key word generates about 188,000 records on Google scholar search and over 3600 on the EBSCO academic electronic search. The numbers indicate that it is well-accepted across many disciplines. While there are numerous Delphi studies that report answers to specific questions, there are fewer that deal with methodological issues such as recruiting the panel, survey administration, and other challenges as well as use in evaluation studies. Limited information pertaining to cross-country applications is available. Thus, the main purpose of this paper is to share our experiences with an electronic Delphi in the Asia-Pacific region. The focus is not on research findings, but on issues encountered and their resolution. We also describe some of the difficulties that arose in the researchers’ collaboration in conducting this research.

Methodological and conceptual issues encountered reflect aspects of the Delphi technique and the unique nature of the project. We believe that our experiences will be informative for others tackling analogous types of research. Lessons learned from this project may help future Delphi research with a similar focus. The problems related to recruitment, panel composition and size, and participant drop out are common in the Delphi technique. Locating participants in each country and uneven representation of samples per country are specific to the study. Professional connections proved to be an effective alternative in overcoming sampling challenges in an international investigation like this one. Recruiting those with an interest in the topic, building rapport with them, and tracking non-respondents and sending reminder letters to them seem to be useful ways for dealing with drop-out issues in Delphi. Some of the instrumentation problems are generic to the Delphi. For example, open-ended questions in the first round generate rich information, but take more time for analysis for the next survey. Others such as the content and the use of the double scale for identifying EPE discrepancies are related to the content and context of the current effort. Piloting the survey and attending to all aspects of its design enhanced the overall quality of the endeavor. The electronic mode of data collection has advantages and is a useful medium for studies targeting international participants. Access and time constraints, however, must be considered in choosing such a data collection approach. Defining consensus and the unit of analysis were problems in this study. Determining consensus is a common issue in Delphi research. In this regard, our experiences suggest using the median with the inter-quartile range and the spread of the responses as criteria. As for the unit of analysis, providing clearer instruction for participants as to their frame of reference would have been helpful. Several challenges related to the nature of international study and research collaboration. First, language is prominent and future research with international participants needs to consider language competency. Written communication skill for responding to open-ended questions is critical. Second, for international research collaboration keep in mind the time lag and communication issues if research team members reside in different time zones. Finally, a meaningful and strong relationship between the researchers is elemental for successful research collaboration (Stead & Harrington, 2000). Our experience reinforces this perception.