ارزیابی کیفیت شواهد در تحقیقات تجربی مدیریت حسابداری: مورد بررسی مطالعات

This study addresses the quality of survey research in management accounting. Specifically, we use a framework containing the five key elements of a well-designed survey to assess the quality of all mail surveys in management accounting published in eight accounting journals over a 20-year period (1982–2001). Our analysis shows that over time survey research in management accounting has improved. However, more attention to improving the ways in which the survey method is deployed is essential if credibility of the data is desired.

Surveys are the most heavily criticized research method employed by management accounting researchers with the central concern being the reliability of the data obtained (Young, 1996). This criticism is noteworthy, since over the past 20 years, 30% of all published empirical management accounting research has used the mail survey method.1 Previous studies on the use of the survey method in management accounting have identified a central weakness—the failure on the part of many studies to adhere to the fundamental principles of survey design and administration (Mangione, 1995). If surveys are constructed and administered appropriately, then they can be a source of large-scale, high-quality data (Diamond, 2000, Dillman, 1978 and Dillman, 1999). The key issue with the survey method, then, centers more on how it is deployed, rather than with the method itself. The purpose of this paper is to review the evidence on the quality of survey data in management accounting research with the goal of providing insights to improve the use of the survey method. Specifically, we use a framework containing the five key elements of a well-designed survey to assess the quality of all mail surveys in management accounting published in eight accounting journals over a 20-year period (1982–2001). We address such issues as how sampling is conducted, how surveys are designed and administered, and how specific ‘norms’ within survey research in management accounting have perpetuated. The framework has been used previously by judges to determine the efficacy of surveys offered as evidence in court under Federal Rule of Evidence 703 (Diamond, 2000). The central point is that whether evidence is acceptable depends not upon the method per se, but upon how well the method is used. Throughout the paper, we provide recommendations to improve the quality and elevate the status of survey evidence in addressing management accounting questions. In management accounting research, surveys are most commonly employed for theory testing, but some are used for descriptive purposes. While the survey method often evokes controversy (Young, 1996 and Zimmerman, 2001a), even critics of the method recognize the power that collective opinions have on the behavior and functioning of individuals, organizations, and society. For example, despite criticism about the quality of their instruments, business school surveys of students and recruiters by the business press (e.g., Business Week, US News and World Report, The Financial Times, The Wall Street Journal) have had an enormous impact on the behavior of business school administrators, staff, faculty, students, and other constituents (Zimmerman, 2001b). Further, census takers, political opinion makers, and television raters all use survey data. In short, surveys form a central role in shaping everyday opinions. Why then are surveys so controversial? Mangione (1995, pp. 2–3) sums it up as follows: The world is filled with examples of poor quality mail surveys. Mail surveys are seductive in their apparent simplicity—type up some questions, reproduce them, address them to respondents, wait for returns to come in, and then analyze the answers. Any survey process, however, contains many important steps that need to be considered carefully and carried out in a particular sequence. Not knowing what these steps are leaves you vulnerable to producing a poor quality product. There are many reasons that people get pushed into doing poor quality work. Certainly time pressure is one reason, but perhaps the biggest cause is ignorance—not knowing the negative consequences of a decision. There is no doubt that “data collected from [poor quality] surveys suffer from well-known problems such as response and surveyor biases” (Zimmerman, 2001a, p. 420, brackets added). However, researchers can employ several techniques to curtail these problems. For example, some researchers attempt to improve survey construction by developing the ‘right’ questions and their wording for a given purpose. Others try to increase response rates by determining when follow-up letters should be sent, how to entice respondents to fill out a questionnaire, and how to obtain representative samples from a target population. Using our framework, we discuss these and other ways to improve the quality of survey data under title “Applying the framework to the data”.

The quality of survey data in management accounting is as weak as the weakest link in the survey data collection process. Hence, no feature of the survey data collection process should be so poor that it would undermine the researcher’s ability to use the data for the purpose at hand (Fowler, 1984). In presenting several issues of survey research in management accounting, researchers are in effect saying (adapted from Sapsford, 1999, p. 91): OK, I know I haven’t got a random sample, so I have trouble delineating exactly which population my results hold for. Also, I haven’t got a very good response rate, further complicating the generalizability of my results. And, some of my measures weren’t very good either, introducing error. However, my findings have support from the literature and look useful. It’s up to you, dear reader, to decide how much reliance you will place on my results. Perhaps you’ll think the results are important and will be able to replicate them without the sampling difficulties that I have had and have reported. Conducting high-quality survey research requires a set of conditions that are not all within the researcher’s control. It requires a population that has good access; that uses a common language; that is willing to discuss a wide range of subjects with strangers; and that trusts pledges of confidentiality (Groves, 1989). Such conditions appear increasingly difficult to find, not only in management accounting, but also in other areas of organizational research. In management accounting, as in many other fields, we tend to just duplicate the survey procedures employed in other articles because it is convenient, it provides justification for what we did (or failed to do), and it usually helps to ‘make the case’ (e.g., for low response rates) with editors and reviewers. Unfortunately, it precludes innovation and improvements in the application of the survey method in management accounting research. There are extensive, often normative, literatures with empirical evidence in a variety of fields on all key aspects of the survey method (which we often could not discuss in detail in this paper), such as about novel ways to improve survey response (other than doing follow-ups), analyze non-response bias (other than comparing early and late responses), or check measurement reliability (other than reporting Cronbach alphas). If future studies could begin to use some of these methods, we are confident that the quality of management accounting survey evidence produced, and hence, the collective knowledge created from it, would improve. In conclusion, we believe that survey research in management accounting would benefit if we started devoting more effort to studying the fundamental principles of the method and apply them accordingly. In this spirit, we hope that our paper will be viewed as food-for-thought to conduct survey research more consciously to overcome its weaknesses in innovative and creative ways. After all, the legal framework suggests that a well-designed and well-executed survey can be admitted as evidence in court (Diamond, 2000 and Morgan, 1990). We hope this drives home the idea that the key issue with the survey method lies more in how it is used rather than with the method per se.