یک مدل تجزیه و تحلیل پوششی داده های گروه فازی برای انتخاب پروژه فن آوری پیشرفته: یک مطالعه موردی در ناسا

The assessment and selection of high-technology projects is a difficult decision making process at the National Aeronautic and Space Administration (NASA). This difficulty is due to the multiple and often conflicting objectives in addition to the inherent technical complexities and valuation uncertainties involved in the assessment process. As such, a systematic and transparent decision making process is needed to guide the assessment process, shape the decision outcomes and enable confident choices to be made. Various methods have been proposed to assess and select high-technology projects. However, applying these methods has become increasingly difficult in the space industry because there are many emerging risks implying that decisions are subject to significant uncertainty. The source of uncertainty can be vagueness or ambiguity. While vague data are uncertain because they lack detail or precision, ambiguous data are uncertain because they are subject to multiple interpretations. We propose a data envelopment analysis (DEA) model with ambiguity and vagueness. The vagueness of the objective functions is modeled by means of multi-objective fuzzy linear programming. The ambiguity of the input and output data is modeled with fuzzy sets and a new α-cut based method. The proposed models are linear, independent of α-cut variables, and capable of maximizing the satisfaction level of the fuzzy objectives and efficiency scores, simultaneously. Moreover, these models are capable of generating a common set of multipliers for all projects in a single run. A case study involving high-technology project selection at NASA is used to demonstrate the applicability of the proposed models and the efficacy of the procedures and algorithms.

The recent global economic meltdown and the passage of austere budgets have focused critical attention on government agencies that support technology development. The public is concerned with the governance of these agencies and with obtaining the maximum return on public spending. Public pressure has forced Congress to mandate the National Aeronautic and Space Administration (NASA) to be more accountable in its evaluation of high-technology projects. The high-technology assessment process at NASA is intended to: (1) identify what technologies are needed and when they need to be available; (2) develop and implement a rigorous and objective technology prioritization process; and (3) develop technology investment recommendations about which existing projects should continue and which new projects should be established (NASA ESAS Final Report, 2005). The assessment process involves budget, schedule, safety, reliability, feasibility and reusability considerations needed to develop an optimal portfolio of high-technologies projects to facilitate more feasible future space missions. The role of the Ground System Working Team (GSWT) at the Kennedy Space Center (KSC) is to help determine the value of investing in a particular high-technology that will maintain NASA’s current space science capabilities, and enable safe and successful future space exploration missions. The GSWT is a carefully balanced panel of senior technology and systems experts from five different divisions at the KSC. The assessment and selection of projects is an important issue in technology management (Linton et al., 2002, Shehabuddeen et al., 2006 and Sun and Ma, 2005). The rapid development of technological changes, together with increasing complexity, has made the task of technology assessment and selection a difficult one (Shehabuddeen et al., 2006 and Solak et al., 2010). The literature on project selection contains hundreds of models, including: scoring methods, ad hoc methods, comparative methods, economic methods, portfolio methods, mathematical optimization methods and simulation methods. Scoring methods use a relatively small number of quantitative criteria to specify project desirability. Using these methods, the merit of each project is determined with respect to each criterion, and then scores are combined to yield an overall performance score for each project (Coldrick et al., 2005, DePiante and Jensen, 1999, Henriksen and Traynor, 1999 and Oh et al., 2012). Ad hoc methods are a special form of scoring methods. In these methods, limits are set for the various criteria levels, and then any projects which fail to meet these limits are eliminated. Comparative methods consider both quantitative and qualitative attributes. In these methods, the weights of different attributes are determined and alternatives are compared on the basis of their contributions to these attributes, and then a set of project benefit measures is computed. Once the projects have been arranged on a comparative scale, the decision makers (DMs) proceed from the top of the list and select projects until available resources are exhausted (Huang et al., 2008, Khaili-Damghani and Sadi-Nezhad, 2013 and Tiryaki and Ahlatcioglu, 2009). Economic methods use financial metrics and models to calculate the monetary payoff of each project under consideration. In these methods, two dimensions such as the expected monetary value and the likelihood of success are selected, and then a representative mix of projects with respect to these dimensions are selected (Eilat et al., 2006, Ho and Liao, 2011 and Zapata and Reklaitis, 2010). Mathematical optimization methods optimize various objective functions within the constraints of resources, project logic and dynamics, technology, and project-related strategies. They include a wide range of methods, such as linear, non-linear, integer, dynamic, goal and stochastic mathematical programming methods (Beaujon et al., 2001, Dickinson et al., 2001, Elazouni and Abido, 2011, Kester et al., 2009 and Khalili-Damghani et al., 2011). Simulation is the process of imitating the operation of a real-world process or system over time. Essentially, simulation consists of: (1) building a model that describes the behavior of a system; and (2) experimenting with this model to support decision making and problem solving. The purpose of simulation is to shed light on the underlying mechanisms that determine the behavior of a system. Simulation is also used when the real system cannot be engaged, because it may not be accessible, or it may be dangerous or unacceptable to engage, or it is being designed but not yet built, or it may simply not exist (Banks, Carson, Nelson, & Nicol, 2009). Simulation is a powerful tool for evaluating alternative designs, plans and/or policies without having to experiment on a real system, which may be prohibitively costly, time-consuming, or simply impractical. Almost any system which can be modeled using equations and/or rules can be simulated. Computer simulation is often used as an addition to, or substitution for, modeling systems for which simple closed form analytic solutions are not possible. Computer simulations are generally developed to generate a random sample of representative scenarios for which a complete enumeration of all possible states would be prohibitive or impossible. However using high level computers, it is often feasible to generate a sufficient number of cases in a relatively short amount of processing time to accurately describe the behavior of the system under a variety of conditions. In many cases the problem is not the total number of enumerations, but the inability to completely validate the equations and the rules that are used to generate the simulations. Multi-scenario optimization is used for problems where the number of alternatives and/or decision parameters is large and the system cannot be modeled with deterministic optimization methods. Multi-scenario optimization is a convenient way to formulate design optimization problems that are tolerant to uncertainties and/or need to operate under a variety of different conditions. The behavior of the parameters and the states of the system are estimated stochastically using probability distribution functions when the number of scenarios increases extensively. A stochastic process is one whose behavior is non-deterministic and is a sequence of random variables. By definition any system or process that can be analyzed using probability theory is stochastic. The stochastic component of the problem (uncertainty) is reproduced by simulation and the overall performance of the system is measured by calculating the expected values and variation of the output variables for a very large number of iterations. Statistical techniques can be used to test various hypotheses about the system performance. Sensitivity analysis can also be used as a major tool to support decision making and problem solving. Law and Kelton (2007) and Banks et al. (2009) provide excellent overviews of simulation modeling and techniques. Optimization methods are also a special form of decision analysis. In these methods the DMs select from the list of candidate projects a set that provides maximum benefit (e.g., maximum net present value). These methods employ mathematical programming to facilitate the optimization process and to take into account project interactions such as resource dependencies and constraints, technical and market interactions, or program considerations (Araúzo et al., 2010, Stamelos and Angelis, 2001 and Vithayasrichareon and MacGill, 2012). In this study we use data envelopment analysis (DEA) for project portfolio selection at NASA. DEA is a non-parametric approach to performance evaluation. With DEA, the efficient frontier is the benchmark against which the relative performance of projects is measured. Given a group of high-technology projects, all projects should be able to operate at an optimal efficiency level which is determined by the efficient projects in the group. These efficient projects are usually referred to as the “peer projects” and determine the efficient frontier. The projects that form the efficient frontier use a minimum quantity of inputs to produce the same quantity of outputs. The distance to the efficiency efficient frontier provides a measure for the efficiency or its lack thereof. We use DEA for project portfolio selection at NASA because DEA: (1) accommodates a multiplicity of inputs and outputs; (2) does not require a priori weights on inputs and outputs or explicitly hypothesized forms of relations between the various inputs and outputs; (3) inputs and outputs can be quantified using different units of measurement including imprecise data represented by fuzzy sets; and (4) can determine possible sources of inefficiency in each project. The remainder of this paper is organized as follows. In Section 2 we review the relevant literature on DEA. The details of the proposed DEA method are presented in Section 3. In Section 4 we illustrate a high-technology project portfolio selection study at NASA to demonstrate the applicability of the proposed framework and exhibit the efficacy of the procedures. We end the paper with our conclusions and future research directions in Section 5.

Austere economic times and major funding cuts in NASA’s budget have forced the agency to obtain the maximum return on public spending. In response, NASA is trying to improve its national space posture during times of budget austerity by carefully evaluating high-technology projects and getting the most return on public spending. However, the assessment and selection of high-technology projects is often a difficult decision making process due to the multiple and often conflicting objectives in addition to the inherent technical complexities and valuation uncertainties involved in the assessment process. We proposed a DEA model with ambiguity and vagueness. The vagueness of the objective functions was modeled with a multi-objective fuzzy linear programming method. The ambiguity of the input and output data was modeled with fuzzy sets and a new α-cut based method. The emerging models were linear, independent of α-cut variables, and able to maximize the satisfaction level of the fuzzy objectives and efficiency scores, simultaneously. Moreover, these models were capable of generating a common set of multipliers for all projects in a single run. We also demonstrated the applicability of the proposed models and the efficacy of the procedures through a case study involving high-technology project selection at NASA. Our model was intended to assist DMs in their judgment when evaluating and selecting high-technology projects. In fact, human judgment is the core input in the decision making process. Our approach helped the DMs to think systematically about complex problems and improved the quality of their decisions. We decomposed the high-technology project selection process at NASA into manageable steps and integrated the results to arrive at a solution consistent with managerial goals and objectives. This decomposition encouraged the DMs to carefully consider the elements of uncertainty. The proposed structured framework does not imply a deterministic approach in project evaluation. While our approach enabled DMs to assimilate the information and organize their beliefs in a formal systematic approach, it should be used in conjunction with management experience. Managerial judgment is an integral component of high-technology project evaluation and selection decisions; therefore, the effectiveness of the model relies heavily on the DM’s cognitive capabilities. The methodology and the data analysis technique adopted in this paper have provided mitigations against the drawback of the previous project evaluation and selection studies. These include: (1) the use of financial measures, leading to difficulties in quantifying non-financial variables; (2) the use of regression techniques for the data analysis requiring a prior linear functional relationship between the variables and an arbitrary assignment of weights to the variables. The contribution of the proposed project assessment and selection method is fivefold: (1) it addresses the abovementioned drawback of the previous project evaluation and selection studies; (2) it is grounded in the DEA method that accommodates a multiplicity of inputs and outputs, does not require a priori weights on inputs and outputs, allows inputs and outputs to be quantified using different units of measurement, and can determine possible sources of inefficiency in each project; (3) it considers ambiguity in decision making by using verbal expressions and linguistic variables for subjective judgments (Poyhonen, Hamalainen, & Salo, 1997); (4) it considers vagueness in decision making by using imprecise data due to lack of expertise, unavailability of data, or time constraints (Kim & Ahn, 1999); and (5) it considers meaningful and robust aggregation of subjective and objective judgments that affect the evaluation process (Valls & Torra, 2000). There are a number of challenges involved in the proposed research. These challenges provide a great deal of fruitful scope for future research. The practicality of this model can be further enhanced by developing the proposed framework into a decision support system to reduce the computation time and effort. Another future research direction, which could be an area of theoretical study, is investigating the similarities and differences between the model proposed in this study and other multi-criteria decision making models. Finally, a systematic investigation of different defuzzification and ranking methods can be carried out to see the effects on the final ranking of the high-technology projects. We hope that our study can inspire others to pursue further research.