Organizations compete for resources and customers and must somehow assess the results of their decisions and actions. Organizational performance is by no means a simple phenomenon; rather, it is a complex and multidimensional concept (Cameron, 1986, Chakravarthy, 1986 and Venkatraman and Ramanujam, 1986). Cameron (1986) states that organizational performance is inherently paradoxical because, while a given perspective may indicate good performance, another perspective may indicate the opposite. Oftentimes a given performance indicator can only be improved at the expense of another. Furthermore, individuals may have different preferences about which aspects are most relevant to define and assess performance (Zammuto, 1984) and, as a consequence, may disagree on which measures to use, the level of importance to assign indicators, and how to interpret results.
Given the complexity of the phenomenon, several researchers (e.g., Barney, 2010, Chakravarthy, 1986 and Venkatraman and Ramanujam, 1986) advocate the use of multiple perspectives and multiple measures of organizational performance. Kaplan and Norton (1996) advance a framework of performance conceptualization and measurement – Balance Scorecard (BSC) – that explicitly incorporates several dimensions of performance. Although multiple perspectives and measures may better represent the multifaceted nature of the phenomenon, they pose several difficulties: (i) possible need to assign non-equal priorities to perspectives and to performance indicators within each perspective; (ii) need to account for mutually inconsistent results; and (iii) need to design an aggregated metric that would somehow summarize the whole story of success (or failure thereof). One of the methods that can address the complex issues of a balanced system of performance assessment is the Analytic Hierarchy Process (AHP), developed by Saaty, 1980 and Saaty, 1990a. AHP relies on decision-makers’ knowledge and expressed opinions in order to build a structure of hierarchically-organized objectives, criteria and decision alternatives. Based on decision-makers' explicitly stated priorities across criteria, AHP builds a ranking of the alternatives with respect to the objective and indicates the extent to which each alternative is better (or worse) than each of the others.
The main objective of the present study is to critically illustrate the application in a real-life business setting of an integrated approach that combines two theoretical frameworks – the multidimensional perspective on performance measurement (in this case, BSC) and the multidimensional ranking of decision alternatives (in this case, AHP) – in order to comparatively assess the performance of functional subunits within an organization. A subsidiary objective is to extend the external validity of both BSC and AHP by applying those frameworks to a particular industry (telecommunications) and setting (Brazilian environment and managers), which, thus far, the literature has rarely examined. Thus the paper focuses on the practical implementation of an integrated framework, bringing together two well established mechanisms for performance analysis and decision making. Although other studies explore a joint application of AHP and BSC, they typically rely on illustrative hypothetical examples (e.g., Leung, Lam, & Cao, 2006) or on a theoretical discussion of potential uses (e.g., Jovanovic & Krivokapic, 2008). In contrast, this paper examines a real-case implementation of AHP-BSC. Although one can conceive of several levels of organizational performance (e.g., corporate, business unit, functional area, etc.), the present study assesses only the performance of three functional areas in the financial department of a Brazilian privately-held telecommunications company.
The contribution of this paper rests on the attempt to address the thorny issue (cf. Venkatraman & Ramanujam, 1986) of organizational performance measurement — in particular, how to (i) make sense of multiple (and often mutually conflicting) perspectives and measures of the phenomenon; (ii) prioritize among them; (iii) reach some synthesized assessment; and (iv) rank-order the level of performance of alternatives under comparative evaluation. A real-life example illustrates the procedure based on agreement-building among managers, which has the additional benefit of enlightening their knowledge as to performance outcomes, thereby developing a shared vision, and fostering a sense of esprit de corps. The study of a Latin American firm is relevant because, as argued by Brenes, Haar, and Requena (2009), the process of internationalization and openness of economies in this region has far-reaching impacts on management practices. In this context, formal structures related to strategy formulation and implementation emerge as fundamental elements for Latin American firms since the mid-1990s (Brenes et al., 2009). When the internal political and economic environment becomes more stable, as in the case of Brazil, formal policies and decision-making tools become more useful. In particular, the analysis of a real application of an integrated BSC and AHP framework to assess performance exemplifies the pattern of progress in Brazilian management practices.
The following sections demonstrate the rank-ordering of the overall performance of the functional areas of the financial department of this Brazilian telecommunications company, taking into consideration (i) the relative degree of importance of four distinct perspectives of organizational performance; (ii) the relative degree of importance of performance indicators within each perspective; and (iii) the comparative relative performance of three functional areas in light of a weighted combination of all performance indicators and dimensions.
This study integrates two tools, BSC and AHP, to provide a better assessment of the (relative) performance of three organizational units within a Brazilian telecommunications company. While BSC explicitly incorporates several perspectives (besides the usual financial viewpoint) to organizational performance evaluation, AHP handles multiple perspectives (criteria) and measures (sub-criteria) with distinct degrees of importance, and translates the overall result into a unified metric. AHP circumvents the pitfalls of having managers use a simple, ad hoc, weighting approach, and helps them make sense of the multiplicity of performance measures from a balance scorecard. AHP provides more than an ordinal ranking and informs the magnitude of the difference between alternatives, i.e., how a functional area performs better (worse) than another.
As noted above, the study seeks to strengthen the adequacy of the focus group-like approach of (i) eliciting the priority values of perspectives and indicators in a balanced scorecard framework and (ii) weighting performance outcomes. The particular numerical values of weights for perspectives and indicators – and thus final comparative analysis among the units – are specific to the case studied. This specificity of results reflects not only firm characteristics, but also telecommunication industry characteristics, country environment and temporal moment. The values themselves are not a theoretical issue, but rather, a managerial one. However, the procedure to derive relative weights and rankings of alternatives here presented is sufficiently generic to be of utility to other firms, regardless of industry or country.
The application of BSC and AHP methodologies in this particular case – where several rounds of discussion are necessary before managers can reach informed agreement – underscores the fact that organizational performance is a complex and multifaceted phenomenon. Managers must employ a sophisticated framework for the assessment of performance of organizational units – one that explicitly accounts for multidimensionality, incongruence among objectives, and mutually inconsistent evaluations.
The following paragraphs acknowledge some limitations of the study and describe how the authors address them. Firstly, the authors do not perform a sensitivity analysis to verify whether results are reasonably stable — for example, preservation of the same ranking among the current alternatives, with the addition of some other alternatives (functional units) or criteria (performance perspectives and measures). Also, recognizing that a distinct group of managers might provide a distinct evaluation picture, this study employs a multiplicity of managers (16) and hierarchical levels (four) to embrace explicitly different viewpoints and to minimize particularistic assessment. Additionally, the study checks convergent validity by comparing the rankings derived from AHP with the opinion of a senior insider expert (the Vice President of Finance), whose rankings the researchers elicited without his prior knowledge of the final rankings of the other managers. By working with participants from the same department and with similar backgrounds, the study aims to preserve internal validity in terms of “equivalence of groups on participant characteristics and control of extraneous experience or environment variables” (cf., Gliner & Morgan, 2000, p.88).
The risk of self-assessment bias might threaten internal validity and constitutes a limitation of this study: the employees who propose the performance indicators are the same individuals who determine the relative degree of importance of such variables and who assess the relative level of achievement in each indicator. The consequences (good or bad) of the ensuing results also have an effect on managers. Further studies could emphasize internal validity issues by attempting, for instance, to compare a self-assessment performance using the AHP-BSC framework with another measure of performance defined independently by others in the firm or by a third-party. A promising avenue would be Evidential Reasoning (ER), a more sophisticated multi-criteria decision analysis method for self-assessment (Xu and Yang, 2003, Yang and Singh, 1994 and Yang and Xu, 2002) that assumes that uncertain subjective judgments satisfy rationality assumptions. The approach borrows from decision science, artificial intelligence, statistical analysis, fuzzy theory and computer technology to reduce subjectivity in self-assessment analysis (Xu & Yang, 2003).
The dynamic characteristics of the external environment (changes in internal strategy and in external pressures) inherently represent an important obstacle for any performance assessment mechanism. Therefore, any tool for performance evaluation should consider the dynamism of both firm and environment. However, mechanisms based on comparison between prior definition of goals and posterior measurement of indicators usually fail to address the fact that needs and priorities change over time. However, adjustment of priorities over time was outside the scope of this study. Importantly, the joint procedure using AHP-BSC aims to identify the best performing unit in a period, taking into account current prioritization of indicators and perspectives. This fact underscores another (albeit polemic) advantage of the procedure, which is the evaluation of past performance in the context of current prioritization criteria. Thus, the time horizon gap avoids, for instance, a bad performing unit, evaluated under current standards, as being identified as a good performing unit simply because it met goals defined for a previous period. The AHP procedure is not a dynamic mechanism per se, but is able to adapt to the effects of a changing environment, which is the case in the volatile telecommunications industry.
Limitations notwithstanding, the study's contributions are important. Leung et al. (2006, p. 683) suggests that, “although the conceptual framework of the BSC has been widely accepted in the business community, the proper method of implementing the framework remains an issue.” Although the present study's contribution to the advancement of the theory is modest, the approach described can aid in the comprehension of AHP and in surmounting the challenges posed by its application as a multi-criteria decision-making tool in real-world BSC performance assessment contexts. Managers reported that the endeavor as a whole, although demanding effort and time, helped them see the big picture and identify synergies and possible areas for global improvements, rather than focusing narrowly on a personal agenda. In fact, the joint consideration of several department-wide goals and the contributions of each functional unit – along several performance perspectives and metrics – leads managers to organize common efforts better and identify corrective actions that look beyond short-term financial results. The focus-group design illuminated managers’ understanding of the business and its outcomes, while at the same time fostering teamwork.
The approach of using group discussions to seek consensus for the AHP input constitutes a contribution to better application of theoretical models in real-world situations. Furthermore, although several studies present theoretical arguments and foundations to AHP and BSC, few provide explicit advice when it comes to implementing such concepts in a real organization. This study is an attempt to combine theory with practice. A balanced approach (BSC) to the assessment of performance is important because multiple perspectives (dimensions) and multiple measures (indicators) exist via which the investigator can conceive of and rate the performance of functional areas. Given that managers may not attribute the same degrees of performance to all dimensions and to all indicators – and, besides, a functional area may not perform equally well (poorly) in all indicators (and dimensions) – a decision-making tool (AHP) that accounts for this complexity and helps rank-order the alternatives under evaluation appears useful. As demonstrated, the integrated approach (BSC and AHP) employed here presented convergent validity, providing a fine-grained picture of performance (through the in-depth discussions among managers) and overall assessment. The interactive and iterative process employed in this study has the additional advantage of enabling managers to apprehend the diverse perspectives of performance assessment and to understand possible trade-offs. Managers receive insights as to which functional areas are doing well and which require more attention. Results can serve as input, for example, to shape bonus distribution, incentive schemes, and discussion of possible reasons for performance gaps and successes.
