To accurately model software failure process with software reliability growth models, incorporating testing effort has shown to be important. In fact, testing effort allocation is also a difficult issue, and it directly affects the software release time when a reliability criteria has to be met. However, with an increasing number of parameters involved in these models, the uncertainty of parameters estimated from the failure data could greatly affect the decision. Hence, it is of importance to study the impact of these model parameters. In this paper, sensitivity of the software release time is investigated through various methods, including one-factor-at-a-time approach, design of experiments and global sensitivity analysis. It is shown that the results from the first two methods may not be accurate enough for the case of complex nonlinear model. Global sensitivity analysis performs better due to the consideration of the global parameter space. The limitations of different approaches are also discussed. Finally, to avoid further excessive adjustment of software release time, interval estimation is recommended for use and it can be obtained based on the results from global sensitivity analysis.
During the last three decades, a large number of models have been proposed for software failure process [1], [2], [3], [4], [5], [6], [7] and [8]. In the recent years, incorporating testing effort into software reliability growth models (SRGMs) has received a lot of attention, probably because testing effort is an essential process parameter for management. Huang et al. [9] showed that logistic testing effort function can be directly incorporated into both exponential-type and S-type non-homogeneous Poisson process (NHPP) models and the proposed models were also discussed under both ideal and imperfect debugging situations. Kapur et al. [10] discussed the optimization problem of allocating testing resources by using marginal testing effort function (MTEF). Later, Kapur et al. [11] studied the testing effort dependent learning process and faults were classified into two types by the amount of testing effort needed to remove them. In addition, some research incorporated change-point analysis in their models as the testing effort consumption may not be smooth over time [12], [13] and [14]. Specifically, Lin and Huang [14] incorporated multiple change-points into the flexible Weibull-type time dependent testing effort function. The proposed model seems to be more realistic and therefore it is selected in this paper.
As constructing model is not the end, to guide project managers to decide when to release the software is a typical application of the model. The optimal release time problem considering testing effort was also discussed [14], [15], [16] and [17]. However, most of the research assumes that parameters of the proposed models are known. In fact, there always exist estimation errors as parameters in testing effort function and SRGMs are generally estimated by least square estimation (LSE) method and maximum likelihood estimation (MLE) method respectively. It is necessary to conduct the sensitivity analysis to determine which parameter may have significant influence to the software release time. This is even more important when there are an increasing number of parameters involved in the model, such as the model proposed by Lin and Huang [14].
Sensitivity analysis can be used to determine how sensitive the software release time is. It helps to find parameters that could significantly affect the solution to the release time. By showing how the software release time reacts against the changes in parameter values, the model is also evaluated and validated. In this paper, sensitivity of the software release time is studied and different approaches are used, including one-factor-at-a-time approach, design of experiments and global sensitivity analysis.
After the sensitivity analysis, significant parameters can be determined and they should be estimated precisely. However, it may not be possible due to the limited amount of information available. Thus, conservative estimation of release time is needed to avoid releasing the software too optimistically [18]. To this end, interval estimation is recommended for use and the simulation results from global sensitivity analysis can just help in this.
The rest of the paper is organized as follows. Section 2 introduces the general model incorporating testing effort and formulates the software release time problem. Section 3 discusses procedures when using different approaches to sensitivity analysis. In Section 4, an application example is given and some interesting results are obtained. In Section 5, limitations of different approaches are highlighted. The interval estimation of optimal release time is discussed in Section 6 and it can be seen that results from global sensitivity analysis are very helpful in this. Concluding remarks are made in Section 7.
In this paper, different approaches to sensitivity analysis are adopted and properties of them are discussed. Especially, the assumptions are highlighted which can help practitioners better understand the limitations that need attention in the real application. Results from traditional methods like the one-factor-at-a-time approach and DOE may not be accurate enough. Thus, global sensitivity analysis is recommended for use due to the consideration of the global parameter space. Furthermore, global sensitivity analysis possesses another advantage that other methods do not have. Results from it not only help to determine the sensitive parameters, but also provide further information for management to decide when to release software under parameter uncertainty. With the use of the interval estimation for the optimal release time, the precision of the estimation can be measured and controlled.
