کشورهای خوب و یا پروژه های خوب؟ ارتباط خرد و کلان عملکرد پروژه بانک جهانی

This paper investigates macro and micro correlates of aid-financed development project outcomes, using data from over 6000 World Bank projects evaluated between 1983 and 2011. Country-level “macro” measures of the quality of policies and institutions are strongly correlated with project outcomes, consistent with the view that country-level performance matters for aid effectiveness. However, a striking feature of the data is that the success of individual development projects varies much more within countries than it does between countries. A large set of project-level “micro” variables, including project size, project length, the effort devoted to project preparation and supervision, and early-warning indicators that flag problematic projects during the implementation stage, accounts for some of this within-country variation in project outcomes. Measures of World Bank project manager quality also matter significantly for the ultimate project outcomes. We discuss the implications of these findings for donor policies aimed at aid effectiveness.

A vast empirical literature has sought to answer the question of when foreign aid is effective in achieving its desired objectives. One influential strand of this literature has focused on the aggregate country-level impact of aid, typically on GDP growth, and has necessarily also focused on country-level factors that determine the aggregate effects of development assistance.1 However, recognizing that most of foreign aid is provided in the form of individual aid-financed development projects,2 another influential strand of the literature focused on aid effectiveness at the project level. During the 1970s and 1980s, this typically took the form of calculations of economic rates of return in aid projects, either prospectively in order to justify financing particular projects, or to assess their effectiveness after the fact. More recently, a large literature has used more rigorous impact evaluation techniques, often in the form of randomized controlled trials, to understand the effects of particular aid-financed interventions at the level of individual projects. Out of necessity, this project-level literature has for the most part focused on project-level factors that matter for the success or failure of individual projects. In this paper, we use a very large dataset of over 6000 World Bank projects, implemented in 130 developing countries since the 1970s, to simultaneously investigate the relative importance of country-level “macro” factors and project-level “micro” factors in driving project-level outcomes. Our effort to bridge the gap between the country-level and project-level aid effectiveness literatures is motivated by the observation that, while country-level factors are important for the aid project outcomes, these outcomes vary much more across projects within countries than they do between countries. This implies that both project-level factors (which often are at least in part under the control of the aid agency implementing project), as well as country-level characteristics (which typically are beyond the control of aid donors), need to be taken into account when assessing project performance, and aid effectiveness more generally. Our measure of project-level success consists of a subjective assessment of the extent to which individual World Bank projects were able to attain their intended development objectives. These ratings are generated through internal World Bank project management and evaluation procedures, which we describe in more detail below. While we acknowledge upfront that these ratings are highly-imperfect indicators of the ultimate effects of projects, we will for terminological convenience refer to these ratings as “project outcomes”. In addition, we share with the rest of the project-level literature the important limitation that the average effectiveness of individual aid projects may well not coincide with the aggregate impacts of aid. For example, there may be complementarities between individual aid projects, or between aid- and non-aid-financed projects, that contribute to a greater aggregate impact than any individual aid project. Conversely, to the extent that aid money is fungible, even highly-successful aid-financed projects may have the side effect of freeing up resources for less beneficial forms of recipient-government spending, so that the aggregate impact of aid is less than the project-level evidence would suggest. With these qualifications in mind, we first document a set of robust partial correlations between project outcomes and basic measures of country-level policy and institutional quality observed over the life of the project. This echoes other findings in the literature on macro-level determinants of aid effectiveness, which emphasize the role of country-level proxies for the quality of policies and institutions in driving project outcomes. However, enthusiasm for this finding on the importance of country-level variables for project outcomes needs to be tempered by the observation that roughly 80 percent of the total variation in project outcomes in our sample occurs across projects within countries, rather than between countries. This basic observation suggests that there are large returns to gathering and studying potential project-level correlates of project outcomes, which have largely been overlooked in the cross-country literature on aid effectiveness. We draw extensively on the World Bank's internal databases to extract three categories of such project-level variables: (1) basic project characteristics such as the size and sector of the project, and the amount of resources devoted to its preparation and supervision, (2) potential early-warning indicators of project success retrieved from the World Bank's administrative processes for monitoring and implementing active projects; and (3) information on the identity of the World Bank staff member responsible for the project. We find that several project-level variables, such as project size, project length, the extent of preparation and supervision costs, delays in starting projects, and whether the project was restructured or was flagged as problematic early in the life of the project, are significant correlates of project-level outcomes. However, interpreting these partial correlations is complicated by the fact that many of the project-level characteristics we observe are not randomly assigned to projects, but rather reflect deliberate choices by those responsible for designing and implementing the project. For example, more challenging projects might require greater supervision by World Bank staff, and might also be more likely to result in unsatisfactory outcomes. While we lack a plausibly exogenous source of variation in project characteristics that can be used to pin down causal effects, we make an extensive effort to document and interpret the size of the likely biases due to this endogeneity problem. In the final section of the paper we explore the role of differences in the quality of World Bank staff assigned to manage projects (known as “task team leaders”) in explaining variation in project performance. We study this question in a reduced sample of projects where we have information on the identity of the task team leader, and we also have meaningful variation in project outcomes across both countries and task team leaders. Our main finding here is that task team leader fixed effects are of comparable importance to country fixed effects in accounting for the variation in project outcomes, suggesting a strong role for task team leader-specific characteristics in driving project outcomes. We also document that task team leader quality (as proxied by the average outcome rating on all the other projects managed by the same staff member) is a highly significant predictor of project outcomes. Our results are based on the analysis of projects of just one aid donor, the World Bank. Despite this particular institutional focus, we believe that the evidence in this paper has broader implications for aid effectiveness beyond the World Bank itself. The World Bank is one of the largest single aid donors in the world, its basic model of financing and implementing specific aid projects is by far the most common mode of aid delivery among all aid donors. While each aid donor has its own mechanisms for allocating resources across countries, for identifying specific aid projects to finance within countries, and for determining the management, implementation, supervision, and evaluation of these projects, a few implications of our findings are plausibly relevant to the wider aid community. The first is basic and not very new, though it is confirmed by the updated and expanded work in this paper: targeting aid to countries with better policies and institutions pays off, as rates of project success are significantly higher in countries with good policy, as measured by the CPIA ratings. However, the very large heterogeneity in project performance within countries suggests that policies to improve aid effectiveness could focus more on project-level factors in addition to country-level factors. These include those that make individual projects difficult to restructure or cancel outright even after early indications of problems arise, as well as those that underlie the large differences in project performance across task managers that we observe in the data. The rest of this paper proceeds as follows. In the next section we briefly summarize related literature that has also studied the World Bank project-level data we work with here. In Section 3 we describe the project-level outcome data in detail. 4 and 5 contain our main empirical results on the links between country and project-level characteristics and project outcomes. Section 6 addresses the problem of unobserved project characteristics in driving our results, while Section 7 documents the importance of task team leader characteristics in explaining project outcomes. Section 8 offers concluding remarks and a discussion of the implications of these findings for policies to improve aid effectiveness.

We have analyzed correlates of project outcomes for a very large set of World Bank projects since the early 1980s, distinguishing between country-level correlates of country-average project performance, and project-level correlates of the variation in project outcomes within countries. This distinction is important as roughly 80 percent of the variation in project outcomes occurs across projects within countries, rather than between countries. Consistent with existing literature, we find that country-level variables, most notably the CPIA measure of policy and institutional quality, are robust partial correlates of country-level variation in project performance. This basic finding underscores the importance of country-level selectivity in aid allocation. In the case of the World Bank, this country-level selectivity is primarily implemented through the Performance Based Allocation system for IDA resources, which emphasizes “macro” country-level measures of policy and institutional quality such as the CPIA in determining the cross-country allocation of aid. However, since most of the variation in project outcomes occurs within countries across projects, the bulk of our effort is devoted exploring project-level “micro” variables that could potentially account for some of this variation. For example, we find that restructured projects perform better-than-average following their restructuring, underscoring the effectiveness of this particular intervention to turn around underperforming projects. On the other hand, we consistently find a statistically-significant negative partial correlation between project preparation and supervision expenditures and project outcomes, as well as significant negative partial correlations between project effectiveness delays as well as early-warning indicators flagging “problem” and “potential problem” projects and ultimate project outcomes. Interpreting these negative correlations is complicated by the fact that difficult projects are both more likely to be unsuccessful in attaining their development objectives, and also more likely to require greater preparation and supervision and to trigger early warning flags and accompanying remedial actions. Convincingly addressing this endogeneity problem is difficult, but we have shown that only modestly-strong prior beliefs about the strength of the feedback effect from project quality to preparation and supervision, for example, are sufficient to generate an intuitively-plausible significant positive estimated effect of these interventions on project outcomes. However, these estimated effects are not sufficiently large that intrinsically “bad” projects can – on average – be turned around to yield successful outcomes through greater preparation or supervision. This suggests that there are returns to (a) improving the process of identifying and selecting projects at the very beginning of the project cycle, as well as (b) strengthening supervision and responses to problem project flags, including potentially more frequent use of project restructurings. Some of our findings also call into question some of the conventional wisdom regarding determinants of project outcomes. For example, a commonly-held view is that more complex projects are less likely to turn out to be successful. Yet, of the three proxies for project complexity that we have studied, we find only some evidence that larger–and so possibly more complex–projects are less likely to be successful. On the other hand, greater dispersion of a project across sectors is in fact significantly associated with better project outcomes, and whether a project is a “repeater” project or not does not seem to matter much for outcomes. Another finding with important policy implications is that task team leader characteristics are significantly correlated with project outcomes. Simple analysis of variance suggests that task team leader fixed effects are of comparable importance to country fixed effects in accounting for the variation in project outcomes observed in the data. More specifically, task team leader quality, as proxied by average project outcomes in the rest of a task team leader's portfolio, is strongly significantly correlated with project outcomes. One immediate policy implication of this finding is the importance of internal practices to develop and propagate task manager skills in order to ensure better project outcomes. More generally, there may be returns to taking task team leader characteristics into account when making aid allocation decisions, to complement the current practice of allocating aid based on country characteristics. One approach might be to consider strengthening incentives for task team leaders with a record of success to work in countries where average project performance has been poor. Another might be to consider allocating aid in part to task team leaders rather than countries. For example, a portion of total World Bank assistance could be allocated through a fund to which task team leaders could submit “proposals” for development projects, much in the same way that researchers submit grant proposals to finance research projects. The criteria for judging proposals could then explicitly consider not just the usual merits of the project itself (including the characteristics of the country in which the project is to be implemented, the consistency of the project with the country's development strategy, the degree of country “ownership” of the project, etc.), but also the track record of the task team leader proposing the project. This would give decision makers flexibility to consider an appropriate weighting of project and task team leader characteristics when selecting projects for financing. The final policy implication comes from the humbling fact that, even after accounting for a wide range of micro and macro variables, much of the variation in project performance remains unexplained. After all, in our core specifications we can account for between 13 and 16 percent of the variation in measured project outcomes. Part of this low explanatory power may simply be due to measurement error in the IEG assessments of project outcomes, pointing to the importance of developing more robust tools for capturing project performance. But at the same time, much of this variation is likely to be real, and it reflects a wide range of as-yet-unmeasured factors at both the country and project levels. Developing empirical proxies for these other factors, and thinking creatively about how to use them to design selectivity at both the country and project levels based on such factors, will ultimately help to improve overall aid effectiveness, not just for the World Bank, but for other aid donors that finance and implement project-based aid as well.