This study examined whether mental health community service users completed outcome self-reports differently when assessments were supervised by internal vs. external staff. The examination of potential differences between the two has useful implications for mental health systems that take upon themselves the challenge of Routine Outcome Measurement (ROM), as it might impact allocation of public resources and managed care program planning. 73 consumers completed the Manchester Short Assessment of Quality of Life (MANSA), a shortened version of the Recovery Assessment Scale (RAS), and a functioning questionnaire. Questionnaires were administered, once using support provided by internal staff and once using support provided by external professional staff, with a one-month time interval and in random order. A MANOVA Repeated Measures showed no differences in outcomes of quality of life and recovery between internal and external support. Functioning scores were higher for the internal support when the internal assessments were performed first. Overall, except for the differences in functioning assessment, outcome scores were not determined by the supporting agency. This might indicate that when measuring quality of life and recovery, different supporting methods can be used to gather outcome measures and internal staff might be a good default agency to do this. Differences found in functioning assessment are discussed.
It has been well-established that mental health consumers should and can take part in Routine Outcome Measurement (ROM; Linhorst and Eckert, 2002 and Oakley et al., 2011; Trauer, 2010). Consumers’ involvement in the evaluation process is consistent with several models of program evaluation and can enrich evaluations. Furthermore, the actual process of participation can empower consumers (Linhorst & Eckert, 2002).
Since taking part in ROM requires cognitive and attentional abilities, consumers might benefit from support, namely, assistance tailored to their specific needs and requests (i.e., explaining the meaning of a certain word, reading the questions aloud, using the computer keyboard and mouse). However, it remains unclear whether the way in which the support is provided might affect the reporting of outcomes.
The use of internal service providers to aid data collection has several advantages. First, they usually know the clients and the context within which the evaluation is conducted (Fitzpatrick, Sanders, & Worthen, 2003Love, 1991 and Spaulding, 2008). Second, they might be sensitive to cognitive and attentional abilities and subtle cues regarding whether questions were understood. Third, internal supporters are likely to be cheaper and available, leading to better long-term implementation of ROM, thereby improving service provision (Australian Health Ministers, 1992).
Nonetheless, legitimate concerns can be raised as to whether providers, who might have a stake in the outcome of the evaluation, should be involved. Indeed, they might pressure consumers to answer questionnaires in a certain fashion, for economic, administrative, or personal reasons (Mathison, 1999). Previous research has also reported that clients tended to overstate psychopathology and rate themselves worse in aspects such as well-being when they wished to gain eligibility for services. Conversely, they might rate their mental health as better when seeking discharge or to please their service providers (Bilsker and Goldner, 2002 and Choi and Pak, 2005).
Although the literature has focused mainly on potential biases of internal service providers, bias might also occur with external support. For example, a client might use the opportunity to speak to an external administrator, sent by the “authorities,” to overstate grievances about service provision. Furthermore, the encounter between the client and an external supporter with whom the client is not familiar might cause suspicion and anxiety and therefore undermine the evaluation process (Linhorst & Eckert, 2002).
While ROM is increasingly becoming a requirement by planners of mental health services (Trauer, 2010), the examination of potential differences between internal and external support might have useful implications for service providers taking up the challenge of ROM.
Therefore, the purpose of this study was to test whether discrepancies existed in outcome data when support was procured by internal service providers vs. external professional assistance. As the literature suggests, bias might occur in cases of internal as well as external support. We hypothesized that no differences would be found between outcomes following internal and external support.
Outcome scores were generally not affected by the type of support received. This might have useful implications for mental health systems, as it allows flexibility in selecting the type of support used in the evaluation process. Nevertheless, in the measurement of functioning, internal support might be preferable although further studies are needed to (a) replicate the finding, and (b) elucidate why this might be the case.
If different forms of assessment yield equivalent responses, it may be concluded that fears of bias in internally supported ratings are exaggerated, and that the results are likely to be trustworthy. This might have important implications because internal assessments, conducted by staff who are directly involved in the consumer's care might be more likely to be used in the actual delivery of care in light of considerations of availability and cost (Graham et al., 2001). Furthermore, providers and policy makers have also made the point that outcome measures should be an integral part of the delivery of care (Jacobs and McDaid, 2010 and Kelley and Bickman, 2009). Therefore, the next step in the process of using internal support would be to make consumers an integral part of the feedback process to their personal case managers, the services and policy makers. This would help provide quality control as well as rational and evidence-based decision-making tools for recovery and service planning.
