بهبود دقت در روش تحقیق دلفی

The cornerstone of good research is establishing integrity. However, identifying and gauging methodological rigour for the Delphi technique remains elusive. This is due to a number of reasons such as the ongoing epistemological debate, along with continual modifications. Consequently, the scant studies exploring rigour are mainly experimental, component specific and outdated. This paper discusses the literature on establishing rigour in Delphi studies, the methodological trinity of reliability, validity and trustworthiness. In addition it presents a discussion of the principal forms of establishing rigour, such as the application of rigour using both qualitative and quantitative measurements and corroborating results with relevant evidence in the field for each individual Delphi. Addressing such issues will help enhance the development and utilisation of rigour in the future.

The “holy grail” of research is establishing methodological rigour. This refers to a researcher's responsibility to ensure that procedures have been adhered to and confounding factors eliminated [where possible] to produce dependable results. Traditionally in quantitative research this process is based on the assessment of reliability and validity [1]. Reliability refers to “the consistency of measurement within a study” [2, p 28] and has been sub-divided into three distinct types which include: “(1) the degree to which a measurement given repeatedly remains the same (2) the stability of a measurement over time and (3) the similarity of measurements within a given time period” [3, p 41–42]. There are four main approaches to estimating reliability: firstly, test-retest which involves administering a test on two different occasions to the same sample; secondly internal consistency, which assesses the consistency of results across items within a test; thirdly, inter-observer which requires the rating of the same information and the recording of consistent results by different testers; finally, parallel form, also referred to as alternate [4], which is undertaken when two different instruments are designed to test the same information and produce the same results [5] and [6]. Validity is divided into external, which measures the generalisability of the findings and internal, which refers to the confidence placed in the cause and effect relationship, normally proven by experimental research. There are several ways in which validity can be measured including content, construct and criterion, each of which highlights different aspects of rigour testing: content validity, similar to face validity, assesses if an instrument provides adequate coverage of a topic under investigation; construct validity, whilst subjective, assesses the theoretical foundations of a scale or measurement and the adequacy of the test in its measuring [6]. Finally, criterion-related validity is established when a test is shown to be effective in predicting criterion or indicators of a construct. There are two different types, concurrent and predictive, with the difference lying in the timing. Concurrent validity can be demonstrated when a test, administered at the same time is correlated with a measurement that has been previously validated. In contrast predictive validity is where one measure occurs earlier and is meant to predict some later measure [7]. In contrast to the quantitative approach, rigour is measured in qualitative research by applying the elements of trustworthiness [8]. Trustworthiness is composed of credibility, dependability, confirmability and transferability [9] and [10]. According to Cornick [11] dependability, (in preference to reliability) refers to the stability of data collected. Credibility, (comparable to internal validity), relates to the degree to which data can be believed based on the ability of the researcher, whilst confirmability conveys neutrality, which is related to the concept of objectivity in quantitative research. Finally, transferability corresponds to external validity and reports the application of the findings to other settings [12]. Regardless of what research design is adopted, attention to rigour throughout the process is a vital aspect of research. This is true for many research methods including the Delphi technique. The Delphi technique was originally conceived as “a method used to obtain the most reliable consensus of opinion of a group of experts by a series of intensive questionnaires interspersed with controlled feedback” [13, p 458]. However, with increasing usage and modifications of the approach, there are now many different forms in existence, such as the ‘modified Delphi’ [14] and [15], the ‘policy Delphi’ [16], and the ‘real-time Delphi’ [17]. It is important to point out that not all Delphi techniques aspire to achieve consensus, for instance, the policy Delphi aims to support decisions by structuring and discussing the diverse views of the ‘preferred future’ [18]. Consequently, broader definitions have been put forward which attempt to encompass or highlight the ever-adapting Delphi process in one sentence, which has resulted in broad and varying interpretations of the technique. For instance Linstone and Turoff [19] regard it as “a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem” (p 3), whilst Reid [20] believes the Delphi is a method for the systematic collection and aggregation of informed judgement from a group of experts on specific questions and issues. In accepting these broad definitions, no matter which one is favoured, the generic aim of the approach is to determine, predict and explore group attitudes, needs and priorities. Nevertheless, the lack of precise definition of the technique places the purist of reliability and validity at a distinct disadvantage. As argued by Rowe and Frewer [21] “the more precise our definitions, the better (more reliability, validity) we can conduct research, the easier it is to interpret findings, and the greater the confidence we can have in our conclusions” [p 252, 2005]. In addition to the definition of the approach, the technique is plagued with other uncertainties such as the meaning of consensus, the criteria for defining an expert and the sheer abundance of the types of Delphi available. Unsurprisingly, whilst the Delphi technique has evolved dramatically in its use, application and development, the establishment of rigour within such studies has remained elusive, with attempts widely [and justifiably] criticised [see Woudenberg [22]; Rowe et al. [23], who both presented an in-depth analysis of early Delphi studies on this area]. Consequently the number of studies exploring rigour regarding the Delphi is scant, many are outdated and refer to experimental Delphi studies that attempt to isolate and test a particular component of the method [16]. Few Delphi practitioners attempt to address this issue, but instead commonly refer to this as a key limitation of the approach. This, however, leaves Delphi studies and the method itself, open to criticism and dismissal, with some viewing it as a method of last resort used only when no other scientific method can be deployed [24]. This paper does not claim to hold the definitive answer to this puzzle; instead it aims to bring this issue to the forefront and remind researchers that, whilst limitations exist, it is still an issue that needs to be addressed. This paper will discuss the literature on establishing rigour in Delphi studies, the challenges associated with it and the methodological trinity of reliability, validity and trustworthiness. In addition, it presents a discussion of the principle forms of establishing rigour.

Debate over establishing the methodological rigour of the Delphi is ongoing but it is one which should not be ignored. Traditional approaches to measure rigour overlook what the fundamental flaws of a Delphi are and what in reality it can achieve. Delphi practitioners need to address such issues and be explicit about how and why they choose such criteria. It is recommended that measures of rigour for both qualitative and quantitative be applied to each Delphi study as well as its findings to be confirmed and verified. Addressing such issues will help enhance the development and utilisation of the Delphi in the future.