رابطه بین سطوح قابلیت فرایند ISO/IEC 15504، گواهینامه ISO 9001 و اندازه سازمان: مطالعه تجربی

The gradual spread in the use of ISO 9001 and ISO/IEC 15504 (also known as software process improvement and capability determination (SPICE)) has raised questions such as “At what ISO/IEC 15504 capability level would one expect an ISO 9001 certified organization's processes to be?” and “Is there any significant difference between the ISO/IEC 15504 capability levels achieved by the processes of ISO 9001 certified organizations and those of non ISO 9001 certified organizations?”. This paper provides answers to those questions as well as to the following question “Is there any significant difference in the capability levels achieved by the ISO/IEC 15504 processes of organizations with a large information technology (IT) staff and those with a small IT staff?” In order to answer these questions, we analyzed a data set including 691 process instances (PIs) taken from 70 SPICE phase 2 trial assessments performed over the two years from September 1996 to June 1998. Results show that the ISO/IEC 15504 processes of the ISO 9001 certified organizations attained capability levels of around 1–2.3 in 15504 terms. Results also show differences between the capability levels achieved by ISO 9001 certified organizations and non ISO 9001 certified organizations, as well as between organizations with a large IT staff and those with a small IT staff.

The software process improvement and capability determination (SPICE) project is an ongoing project supporting ISO/IEC JTC1/SC7/WG102 in the development and trialing of the emerging standard ISO/IEC 15504 for software process assessment, capability determination, and software process improvement. ISO/IEC JTC1/SC7/WG10 has developed a technical report 2 (TR2) (ISO/IEC TR2, 1998) consisting of the document set shown in Table 1. In the development of these documents, the SPICE project has empirically evaluated successive versions of the document sets of the emerging international standard through a series of trials (El Emam and Goldenson, 1995; Goldenson and El Emam, 1996; Maclennan and Ostrolenk, 1995; Smith and El Emam, 1996; ISO/IEC, 1999; Woodman and Hunter, 1998; El Emam and Jung, 2000). More information about ISO/IEC 15504 may be found in ISO/IEC 15504 (1998) and in (El Emam et al., 1998).ISO 9001 (1997) contains 20 clauses (see Appendix A) that collectively provide the minimum requirements for setting up a quality management system for use in software development and maintenance, as well as in other industries. Satisfaction of all the requirements leads to ISO 9001 certification. ISO 9000-3 (1997) contains software specific guidelines for the use of ISO 9001, and TickIT (1999) is an initiative, which originated in the UK, to promote the application of ISO 9001 to software. Although ISO 9001 and ISO/IEC 15504 have different origins, i.e., ISO 9001 is a generic standard for quality management and assurance while 15504 was created solely for software process assessment, capability determination, and process improvement, the two standards are intuitively similar, as has been shown in a comparative study of the two standards (Hailey, 1998). To the best of our knowledge, there are no studies which assess the degree of similarity between the two standards such as is seen in a comparative study of ISO 9001 and capability maturity model (CMM) (Paulk et al., 1993) conducted by (Paulk, 1995). In the study, Paulk attempted to answer questions such as “At what level in the CMM would an ISO 9001 compliant organization be?” and “Can a CMM level 2 (or 3) organization be considered compliant with ISO 9001?”. The aim of this study is to consider and, if possible, to provide an answer to the questions relating to ISO 9001 and ISO/IEC 15504 • At what ISO/IEC 15504 capability level would one expect an ISO 9001 certified organization's processes to be? • Is there any significant difference in the SPICE capability levels achieved by the processes of ISO 9001 certified organizations and those of non ISO 9001 certified organizations? • Is there any significant difference in the capability levels achieved by the SPICE processes of organizations with a large information technology (IT) staff and those with a small IT staff? This study answers the three questions above empirically by analyzing a data set taken from 70 SPICE phase 2 trial assessments. The SPICE trials can be seen as a response to a statement by Pfleeger et al. (1994) that Standards have codified approaches whose effectiveness has not been rigorously and scientifically demonstrated. Rather, we have too often relied on anecdote, `gut feeling', the opinions of experts, or even flawed research. Fenton et al. (1993) and Fenton and Page (1993) have made similar arguments. The remainder of this paper is organized as follows: Section 2 provides an overview of the ISO/IEC 15504 architecture, Section 3 presents the research method and Section 4 presents the main results of the study and discusses their implications. Finally, Section 5 concludes with a summary of the paper and some final remarks.

The results of this study may be summarized as follows: • A SPICE process capability level of around 1.0–2.3 corresponds to an ISO 9001 compliant organization. • In almost all SPICE processes, the average capability level of the ISO 9001 certified organizations is greater than that of the non ISO 9001 certified organizations. In addition, in 11 processes out of 29, there was statistical evidence indicating that the capability level is greater for the ISO 9001 certified organizations than for the non ISO 9001 certified organizations. • The capability levels of the processes associated with organizations with a large IT staff was greater than those of the processes associated with organizations with a small IT staff. However, only two processes (ENG.7 and ORG.3) showed a statistical significant difference in the capability levels. In interpreting the results obtained in comparing SPICE and ISO 9001, the similarities between the requirements of the two standards should be borne in mind. In this sense the results are perhaps not too surprising, though it is encouraging to have empirical evidence to suggest that SPICE and ISO 9001 applied to software, produce broadly similar results. One limitation of this study should be made clear in interpreting our results. This limitation is not unique to our study, but is characteristic of most comparison studies. It is worth explaining here. Suppose that an experimenter is interested in investigating the effect of a specific binary factor (i.e., ISO 9001 certification and non ISO 9001 certification) to determine whether this factor has a significant effect on an observed quantity. In retrospective studies of this sort designed to “look into the past” (Agresti, 1996), observed data is obtained through the analysis of historical data concerning the system or process (Montgomery et al., 1998). Thus, retrospective–observational studies of this sort are limited in that they cannot consider possibilities such as non ISO 9001 certified companies not going through the certification process because it was not necessary for their business, but nonetheless having satisfied all the clauses of ISO 9001. This limitation can be solved by performing a randomized experiment which is a more appropriate way of investigating the effect of such factors (Montgomery et al., 1998). However, sometimes such randomized experiments are not possible due to cost, ethics, or for legal reasons. Cost is a major barrier preventing randomization studies in the SPICE trials data collection.