Involving stakeholders can greatly impact evaluation results. The Delphi method, a consensus-building tool, is a promising process to promote and encourage involvement from all stakeholders during the evaluation framing process. The Delphi method removes geographic challenges and time boundaries allowing all stakeholders to participate. The Delphi method uses a series of surveys interspersed with controlled feedback designed to gather information and build consensus without requiring face-to-face meetings. Two different formats of the Delphi method, a paper-and-pencil, postal-mail version and a web-based, real-time computer version are compared in this study. Both versions of the Delphi were administered to a non-profit community based organization as part of framing an evaluation. Participation rates were better with the paper–pencil version. The quantity and quality of data collected were comparable in both versions.
Stakeholder involvement is crucial when evaluating organizations. Involving stakeholders in all phases of the process, including the framing of the evaluation, increases the attention paid to the findings (Cousins & Earl, 1992); helps ensure that relevant questions are asked (Fine, Thayer, & Coghlan, 2000); increases stakeholders’ understanding of the organization and the evaluation (Brandon, 1998; Cousins & Earl); promotes a participatory and collaborative relationship between the evaluator and stakeholders (Patton, 1997); and increases the validity of the evaluation findings (Brandon). Including stakeholders in the process also allows the researcher to ask evaluation questions in the shared terms and language of the stakeholders (Patton). In their Program Evaluation Standards, the Joint Committee on Standards provides another compelling reason to involve stakeholders.
The evaluation should be planned and conducted with anticipation of the different positions of various interest groups, so that their cooperation may be obtained, and so that possible attempts by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted or counteracted (Sanders, 1994, p. 71).
When conducting a participatory and collaborative evaluation, measures should be taken to involve stakeholders who have distinct perspectives on the program. The Joint Committee on Standards suggested “Interview[ing] representatives of major stakeholders to gain an understanding of their different and perhaps conflicting points of view and of their need for information” (Sanders, 1994, p. 38). Common methods used to communicate with stakeholders include focus groups, personal interviews, meetings, semi-structured interviews, and informal interactions (Brandon, 1998).
While meetings are often convenient for collecting information from stakeholders, attendance can be a limitation, as it is not always possible to have all stakeholders represented (Renger & Bourdeau, 2004). As Rossi, Freeman, and Lipsey (1999) pointed out, important groups may be left out of the process even when an evaluation is structured as explicitly participatory and collaborative.
An important time to include stakeholders is during the framing of an evaluation, when evaluators need to be able to assess evaluability (Smith, 1989 and Wholey, 1994), as well as efficiently develop and prioritize evaluation questions (Patton, 2002, Rossi et al., 1999 and Weiss, 1998). A promising prioritization method involves using a consensus-building tool called the Delphi method (Dalkey and Helmer, 1963 and Linstone and Turoff, 1975), where an evaluator or researcher can investigate stakeholders’ opinions on the current state of an organization long before a site visit. The Delphi method can be thought of as a series of sequential questionnaires interspersed by controlled feedback (Linstone and Turoff). This information-gathering tool is ideal for evaluation situations where it is difficult to get feedback from stakeholders due to geographical barriers and busy schedules.
Using the Delphi method to solicit input and feedback from stakeholders was found to be a promising technique for involving stakeholders in the framing of evaluations. The Delphi method allowed for all stakeholders to be involved, regardless of their geographic and time constraints. The following discussion compares the PP and the RT versions of the Delphi.
