This paper develops an analytical model aimed at evaluating research and development (R&D) projects in different stages of their life cycle. It may be applied to project proposals—as part of a project selection procedure, or to ongoing projects—during their initiation, planning, execution or closing stages. Based on the evaluation, management has to decide which project proposals should be selected, which ongoing projects should be continued, or which resource level should be associated with each selected or continued project. The evaluation of projects at their closing stages should allow the creation of a knowledge base of “best practices” and “lessons learned” that would be communicated throughout the organization for continuous learning.
The R&D project evaluation problem is a challenging decision-making problem faced by decision makers that deal with R&D management. The evaluation involves multiple criteria measuring rewards, relevance to the organization's mission and objectives, strategic leverage potential, probability of technical and commercial success, etc. Once the criteria are determined, they should also be weighted to reflect the preferred emphasis of the organization. The focus on future events and opportunities in a dynamic environment cause much of the information required to be at best uncertain and at worst unavailable. Opinions and judgments often have to substitute for data, and measures could be estimated only qualitatively. While quantitative measures like return-on-investment (ROI) are sometimes hard to estimate, qualitative metrics like market familiarity and customer satisfaction are potentially important. The lack of reliable quantitative measures is especially apparent in not-for-profit organizations, where qualitative measures usually take a larger share in the overall evaluation.
Despite these difficulties, projects should be evaluated and prioritized, as they compete for resources. The model we propose in this article tries to respond to these challenges by integrating two well-established managerial methodologies: balanced scorecard (BSC) [1] and data envelopment analysis (DEA) [2].
The BSC is a management tool composed of a collection of measures, arranged in groups, and denoted as cards. The measures are related to four major managerial perspectives, and are aimed at providing top managers with a comprehensive view of their business. The cards offer a balanced evaluation of the organizational performance along financial, marketing, operational and strategic dimensions. BSC combines financial and operational measures, and focuses both on the short- and long-term objectives of the organization. It was motivated by the realization that traditional financial measures by themselves are inadequate in providing a complete and useful overview of organizational performance. In [1], a number of different BSC structures are presented for different industries. Indeed, many organizations have adopted the BSC approach to accomplish critical management processes, clarify and translate their vision and strategy, communicate and link strategic objectives and measures, plan and align strategic initiatives, and enhance strategic feedback and learning. A specific BSC model for projects was first suggested by Stewart [3].
DEA [2] and [4] is a mathematical programming technique that calculates the relative efficiency of multiple decision-making units (DMUs) on the basis of observed inputs and outputs, which may be expressed with different types of metrics. The basic concept in DEA is to measure the efficiency of a particular DMU against a projected point on an “efficiency frontier”. The usefulness of DEA in evaluating multi-criteria systems and providing improvement targets for such systems is expressed in the large number of its reported applications, as described in [5]. Specific DEA models for the context of technology selection or R&D project evaluation were suggested by Oral et al. [6], Khouja [7], and Baker [8].
The method that we propose in this paper uses an extended DEA model, which quantifies some of the qualitative concepts embedded in the BSC approach. The integrated DEA–BSC model addresses three common goals that firms are trying to accomplish [9] and [10]: (1) achieving strategic objectives (effectiveness goal); (2) optimizing the usage of resources in generating desired outputs (efficiency goal); and (3a) and (3b) obtaining balance (balance goal). The model is applicable for evaluating R&D projects in for-profit, private organizations (e.g., venture capital funds), as well as in not-for-profit organizations, such as government agencies charged with selecting R&D projects.
The contribution of the model that is presented in this paper is both conceptual—the integration, for the first time, of the BSC into the DEA model through balance constraints, and theoretical—the introduction of a hierarchical structure of balancing constraints that restrict the proportions of the total output/input of any given DMU that are devoted to specific output/input measures. While traditional weight restriction techniques in DEA (see the literature review) focus on restricting the weight flexibility of the individual weights, the model presented here focuses on balancing the “importance” attached to groups of measures, and applies it within a hierarchical balance structure. The model is also practical because it supports the evaluation of projects throughout their life cycle—during the selection, planning, execution, and termination phases—while relying on the popular BSC measurement system. It also pays attention to the goals that concern managers—namely, effectiveness, efficiency, and balance.
The rest of the paper is organized as follows: Section 2 provides a literature review; Section 3 develops a specific BSC for R&D projects. The integrated DEA–BSC model is presented in Section 4, while its associated mathematical formulations are given in the appendix. Section 5 discusses a case study that applies the DEA–BSC model. Finally, Section 6 presents concluding remarks.
This paper presented a multi-criteria approach for R&D project evaluation based on the integration of two different innovative managerial methodologies. We combined concepts taken from data envelopment analysis (DEA) and balanced scorecard (BSC), which have proven to be useful measurement and analysis tools in many practical applications. These concepts were integrated into a single DEA–BSC model. Values obtained through this model account for “benefits” (outputs), “costs” (inputs), and preferences. The model discriminates projects according to desired characteristics and ranks them consistent with the organization's intended emphasis.
The DEA–BSC model advances the individual capabilities of DEA and BSC. From the viewpoint of DEA, the model generalizes the standard treatment of the data by splitting the inputs and outputs into subsets (cards), and adding constraints (balancing requirements) that reflect relationships among the cards. From the viewpoint of BSC, the model proposes a new approach to evaluate performance by applying quantitative analysis that combines the measures within each card into a single value. It also addresses some of the difficulties in existing BSC applications, namely, reliance on a known (sometimes arbitrarily chosen) baseline against which performance is evaluated and the fact that BSC does not produce a single, comprehensive measure of performance.
Relying on the cards’ structure, we introduced multi-level balance restrictions. We included these restrictions in the DEA model, creating the integrated DEA–BSC model. This model was initially developed for the simplified, single-level partition hierarchy and was extended to the multi-level hierarchy.
We illustrated the implementation of the model in the settings of an industrial research laboratory charged with R&D projects. In future work, we hope to include an interim project evaluation, as well as a retrospective productivity assessment and post-project impact analysis.
The model is consistent with the BSC methodology that has been applied in many companies, and can be used for the evaluation of project proposals, and ongoing projects—in all their progressive stages.
The model presented in this paper could be extended to deal with portfolio considerations that are different from those taken in evaluating individual projects (e.g., finding the proper mix of projects that reflects the desired tradeoff between risk and reward, balancing R&D efforts among different technologies, etc.). The challenge is to devise a method that would aggregate the inputs and outputs of individual projects while taking into account possible interactions among them. In this case, the DMUs would represent alternative portfolios that would be evaluated against each other. This approach will require a decomposition of the portfolio analysis problem into two parts. The first would deal with the generation of the portfolio (i.e., the combination of projects and their aggregate inputs and outputs), whereas the second would evaluate their relative efficiency. This extension is currently being pursued by the authors.
