Simulation plays a crucial role in analyzing discrete-event systems, in particular, when comparing alternative system designs with a view to optimizing system performance (Chen, Chen, Yücesan & Dai, 1998). However, due to the slow convergence of performance, simulation efficiency is still a major concern especially when the number of competing designs to be compared is large. This explains the increasing popularity of research in ranking and selection (R&S): techniques that determine the number of simulation replications required for each design so that a certain measure of selection quality is guaranteed at a pre-specified level with the least possible computational expense. For a comprehensive review of this field, see Branke, Chick, and Schmidt (2007), Kim and Nelson, 2003 and Kim and Nelson, 2007 and Swisher, Jacobson, and Yücesan (2003).
To identify the best system from a set of alternatives, one first needs to specify by which criterion we compare the systems and select the best one, i.e., how to measure the evidence of correct selection. In most studies, the selection quality is measured by the probability of correct selection (PCS), which is defined as the probability that the selected best is the true best. Solution approaches using PCS as the measure of selection quality include the indifference-zone (IZ) ranking and selection (Nelson et al., 2001 and Pritsker, 1986), the decision theoretic methods (Chick, 1997), and the optimal computing budget allocation (OCBA). The IZ-based procedure allocates additional replications based on a least-favorable configuration (LFC) formulation, and causes the allocation efficiency to be more conservative. To improve on this, the OCBA framework follows a Bayesian methodology and allocates additional replications by solving the problem as an optimization problem, in which PCS is maximized with a given total computing budget available. The numerical results show that the OCBA is able to give better results in terms of using the least computing budget for a given level of PCS. For more details, refer to Chen, Chen, and Dai (1996), Chen, Chen, and Yücesan (2000), Chen, Dai, Chen, and Yücesan (1997), Chen, Donohue, Yücesan, and Lin (2003) and Chen, Lin, Yücesan and Chick (2000). Another measure of selection quality, known as the expected opportunity cost (EOC), aims to quantify how far away the selected system is from the true best system. This measure is important as it not only maximizes the chance of selecting the best design, but also if it fails to find the best design, by optimizing this measure, it will ensure the selected design will not be very far off from the best design. EOC is defined as the difference in means between the selected system and the true best. It has been applied in some Bayesian decision theoretic methods, in which additional replications are allocated in a way that the expected value of information gained from the replications is maximized (Chick & Inoue, 2001). In Chick (2003), a selection procedure which can provide an upper bound for the EOC in a frequentist sense was proposed. The paper bridged a gap between the IZ approach (with frequentist guarantees) and the Bayesian approach (where the EOC is considered) to the selection procedures. Within the OCBA framework, EOC is applied and studied in He, Chick, and Chen (2007), where the original OCBA based on maximization of PCS is extended to a new OCBA seeking minimization of EOC. The expected Net Present Value (NPV), as a measure of selection quality, was proposed in Chick and Gans (2006). NPV concerns not only the economic benefit from implementing the selected system, but also the marginal costs of simulation runs and discounting due to simulation analysis time. NPV is to be maximized to underline the financial significance of the selected systems when the system performance and simulation results are themselves financial measures.
In this paper, we studied the multi-objective computing budget allocation problem by considering different measures of selection quality. We have conducted numerical runs to compare these allocation rules. For the probability related measure, we found that the MOCBA using average error as the switching rule gives the best result. We also compare the allocation rule derived based on the probability measure with the one based on opportunity cost. The results indicate that when the performance measures are very close to each other, the OC allocation rule can give better results in term of PCS as it will not concentrate all its computing effort to differentiate designs which have close performances.
In this work, we have derived the allocation rules using the Bonferroni bounds. Even though these bounds are known to be loose, the numerical runs show that the allocation rules are robust when the computing budget is large (Lee et al., 2010). One possible research direction is to explore tighter bounds in deriving the allocation rules to further improve this work.
