آیا اصلاحات بخش بهداشت و درمان،تاثیرات مورد نظر خود را دارد؟: پروژه بهداشت VIII بانک جهانی در استان گانسو، چین

This paper combines differences-in-differences with propensity score matching to estimate the impacts of a health reform project in China that combined supply-side interventions aimed at improving the effectiveness and quality of care with demand-side measures aimed at expanding health insurance and providing financial support to the very poor. Data from household, village and facility surveys suggest the project reduced out-of-pocket spending, and the incidence of catastrophic spending and impoverishment through health expenses. Little impact is detected on the use of services, and while the evidence points to the project reducing sickness days, the evidence on health outcomes is mixed.

Remarkably little is known about the impacts of developing-country health sector reforms on key outcomes, such as health status and protection against the financial risks associated with ill health. In part, this is simply a reflection of the fact that most policies and programs in the sector have not been subjected to rigorous evaluation.1 But it also reflects the fact that the rigorous health sector evaluations to date have, for the most part, not been concerned with broad health sector reforms but rather with the impacts of inputs in the health production function, or with the effects on health outcomes of policy changes outside the health sector. Examples of the former include the paper by Jalan and Ravallion (2003), which looks at the effects of piped water on diarrheal disease among Indian children, and the paper by Miguel and Kremer (2004), which looks at the effects of deworming treatment in Kenya. Examples of the latter include Case's (2002) study of the effect of South Africa's old age pension program on the health of members of the pensioner's household, and the study by Galiani et al. (2005) of the effect on child mortality of Argentina's privatization of water services. There are exceptions. Gertler (2004) reports the effects on health outcomes of a conditional cash transfer program in Mexico that required mothers to take their children for regular health checks to receive the cash supplement. Newman et al. (2002) report impacts on child mortality of health facility infrastructure investments in Bolivia. Saadah et al. (2001) report the impacts on utilization of Indonesia's health card introduced after the economic crisis of the late 1990s. Wagstaff and Pradhan (2005) examine the effects on health utilization and health outcomes of Vietnam's social health insurance program. And Yip and Eggleston, 2001 and Yip and Eggleston, 2004 examine the effects of provider payment reforms on Chinese hospitals. Such studies are, however, relatively few. Furthermore, all concern a relatively small policy adjustment—none looks at a system reform of the type where several changes are introduced together, possibly operating on the demand and supply sides simultaneously.2,3 And yet much of what national governments and donors do in the health sector involves making broad changes to health systems. Over the period 1995–2005, for example, 40% of the World Bank's health sector lending was classified as being directed at “[improving] health system performance”.4 One factor explaining the lack of impact evaluations of broad-brush health sector reforms is the fact that reforms are often implemented across the country simultaneously, substantially complicating the job of constructing a counterfactual. This paper reports the results of an impact evaluation of a World Bank-financed health sector reform project in China, known officially as the World Bank China Basic Health Service Project but more often referred to simply as ‘Health VIII’. In line with China's policy of piloting reforms locally before implementing them nationwide, Health VIII was implemented only in certain counties. China has nearly 3000 administrative units at county level, with an average of 450,000 people living in each.5 The Chinese county thus provides a reasonable sized population for a health reform pilot. The evaluation of this particular pilot reported in the paper is partial, unofficial and opportunistic: it covers just one of the seven provinces where the project operated, namely Gansu; it is not part of the official evaluation exercise being undertaken by the Government of China and the World Bank; and it is opportunistic in the sense it makes use of data collected by researchers who have not been involved in the project and for a purpose other than evaluating the project. Do these features reduce the attractiveness of the paper? We think not. The case of Gansu is not without interest. Gansu is home to 26 million people—larger in population terms than most countries. It is also China's second poorest province.6,7 Given the project's emphasis on improving health among China's rural poor, it is of some interest to know what its impacts have been in Gansu. The nature and timing of the official evaluation are still the subject of some debate; by applying modern impact evaluation methods to the project, an unofficial evaluation may be of some value in this debate. Finally, the data used in the present paper may prove – at least in some respects – to be better suited to the task than that collected for the official evaluation. The Health VIII project team collected baseline survey data only in project counties8 – in fact, in only 28 of Health VIII's 71 counties – and at present is undecided whether to collect follow-up data. The present paper, by contrast, uses data from a survey that serendipitously was fielded just as Health VIII was starting to be implemented and again 4 years later, and in both project and non-project counties.9 The organization of the paper is as follows. Section 2 provides a brief overview of the Health VIII project. Section 3 outlines the methods used in the paper. Section 4 presents the data used. Sections 5 and 6 present the empirical results. Finally, Section 7 presents the conclusions of the paper and provides a brief discussion.

Overall, the results of the paper point to a mixed scorecard for Health VIII. There is evidence that the project achieved the intended dampening effect on out-of-pocket expenses, reducing the incidence of catastrophic health spending, especially among the very poor, and reducing the rate of impoverishment due to out-of-pocket health spending. These changes were mostly the result of downward pressure on drug spending. The project appears, however, to have little impact on the use of services—one of its other key goals. There is some evidence of positive impacts on HepB immunization and on the incidence of non-testing of suspected TB patients. But on service utilization more generally, the household data point, if anything, to a negative impact, while the THC data suggest that the project had no statistically significant impact on the use of either outpatient or inpatient services. This could be because the project successfully reduced the ‘need’ for health care, or eliminated unnecessary visits. Ultimately, of course, it is health outcomes that any health reform project wishes to improve. Encouragingly, there are signs that the project may have reduced days of sickness, especially among the poor, which given the project's goals of reducing both the direct and indirect costs of illness, is a good outcome. But there is no robust evidence that other health indicators were impacted favorably by the project, and it is puzzling why non-project counties came close to achieving a statistically significant reduction in their rates of infant mortality but Health VIII counties saw no change in theirs. Beyond spending, use and health outcomes, the results point to some interesting impacts on the organization of health care delivery, with the project apparently encouraging a substitution towards technical staff at THC level, and a reduction in pharmacists and retirees. There is also evidence of the project leading to a consolidation of health personnel at township level. Surprisingly, the impacts on medical equipment ownership appear to have been somewhat muted, being confined to equipment worth less than 10,000 RMB. One would, of course, like to be able to say something about the relative contributions of the different project components to these impacts. Had the project pressed ahead equally quickly on all fronts in all project counties, this would have been impossible. In the event, the project did not, either in general or in the three Gansu project counties covered by this study. In the household data, there is no sign of any membership of any CMS, let alone one meeting the project's requirements for financial support. How well developed MFA was at the time in these counties cannot be ascertained from the survey data. None of the three counties were among those supported by the British government's complementary Health VIII support project, in which MFA was a key component. Despite this, the evidence of especially large impacts on spending among the very poor suggests that MFA was operating, and was probably having an impact even in these counties. It would almost certainly be a mistake, however, to credit MFA with all the reduction in out-of-pocket spending. Table 8 implies significant negative effects for three of the poorest four quintiles. MFA almost certainly added to the downward pressure coming from the supply-side measures (treatment protocols, essential drug lists, and two-way referrals, etc.). The fact that these effects appear to have been achieved without any reform to the way THCs are paid is noteworthy. One might have anticipated that such effects would not have been possible without a move away from fee-for-service, and without a change in the price schedule that allows providers to make profits on drugs and high-tech care (but not on ‘basic’ care) and hence encourages them to try to shift demand to these more lucrative services (cf. e.g., Liu and Mills, 1999). Finally, some observations on methods. The differences-in-differences results illustrate the dangers of relying on single differences, such as the pre–post comparison that is commonly encountered in evaluations and that has become something of a tradition in World Bank health projects in China.41 The impact on out-of-pocket spending in this study provides a nice example. It was not so much that spending in the Health VIII counties fell—it did slightly, but not significantly so. Rather, it was the fact that Health VIII counties managed to hold spending unchanged while it was rising elsewhere. The impact on equipment in THCs is another example, albeit for a different reason. Equipment increased significantly in THCs in Health VIII counties, but it also increased in non-project counties. The impact on equipment also provides a nice example of importance of matching. The unmatched difference-in-difference impact estimates for equipment are not significantly different from zero: equipment did not increase significantly more quickly in Health VIII counties than in non-project counties. But after matching, the impact for the cheapest category of equipment is significantly positive. The unmatched estimate puts too much weight on changes occurring in non-project counties that had the resources to invest in new equipment, and not enough weight on changes occurring in non-project counties that shared similar constraints to the Health VIII counties. These (and other) results point to the importance of having data not just on untreated cases before and after the intervention, but having data for comparable cases. Where a project is tightly targeted on a few poor jurisdictions, and where the data are from a representative random survey of all jurisdictions, the risk is that the candidates for control jurisdictions are too different to provide good matches for the treated ones. This problem was evident in this study, but not insuperable, because although the survey somewhat oversampled better-off counties, Health VIII was not in the event as tightly targeted on poorer counties as it might have been. In a sense, the analysis benefited from the requirement that local governments had sufficient capacity to implement the project and sufficient resources to pay back the loan. Another option, in addition to a more relaxed approach to targeting, would be to oversample the poorer untreated jurisdictions or units in the evaluation design, perhaps using a propensity score estimated across all units to identify suitable control units to survey. The higher the level of the unit, the more important this becomes. A more relaxed approach to targeting could also have the benefit that the lessons from pilot projects such as Health VIII would be more generalizeable: many of the ideas in Health VIII are potentially applicable beyond the low-income settings targeted in the project, 42 but impact estimates gained from the project's implementation in targeted counties might not provide a reliable guide as to their impact elsewhere, even in Gansu.