Background
There is an increasing number of health-care systems using economic evaluations to inform decisions about the reimbursement of health technologies. There are usually two separate elements of this process: assembling relevant evidence and undertaking analyses (technology assessment), and decision-making. In most systems, technology assessment is undertaken by the manufacturer of the technology. In a few, “third-party” assessment is used.
Methods
In the United Kingdom, the National Institute for Health and Clinical Excellence used a combination of third-party and manufacturer assessments between 1999 and 2005. After this point, a Single Technology Appraisal program (using manufacturer-based assessment) was instituted for some technologies. Here the role of third-party assessment is considered in this from of decision-making. The article reviews the requirements of economic evaluation to support decision-making, and considers the extent to which each type of assessment is likely to meet these requirements. It also attempts to address whether the two forms of assessment differ in their impact on decision-making using a comparison of the decisions made by National Institute for Health and Clinical Excellence (NICE) (under its multiple-technology appraisal system) and the Scottish Medicines Consortium (SMC), which relies on manufacturer assessment.
Results
The comparison is limited by the small number of technologies considered by both bodies. Nevertheless, it suggests that there are potentially important differences between the two bodies, with NICE generally placing more restrictions of the use of technologies.Conclusions
The article concludes that there are potential advantages to third-party assessment, but its cost and timing may preclude its use for all new technologies. A hybrid arrangement is suggested where third-party assessment is used in particular circumstances.
Many health-care systems now require economic evaluations as a
key input into a formal decision-making process about whether to
reimburse/fund/cover new health technologies [1,2]. There are
two distinct components to this use of economic evaluation. The
first is the technology assessment process which, here, is understood
as the process by which relevant evidence is identified,
assembled, and synthesized to provide the basis for a clinical
review and cost-effectiveness modeling. The second is decisionmaking,
into which technology assessment is an important input.
In most systems, the technology assessments are undertaken by the
manufacturer, although they are usually critically reviewed by
experts within the system or a third party. In a small subset of
systems, however, there is a role for third-party organizations
(usually academic groups) to undertake the assessments, usually in
addition to assessments submitted by the relevant manufacturers.
The use of such third-party assessments exists, for example, with
National Institute for Health and Clinical Excellence (NICE) in
the United Kingdom and the Medical Services Advisory Committee,
which considers nonpharmaceutical technologies for the
Australian health-care system.
The role of third-party technology assessment in this form of
decision-making has recently been brought to the fore following
NICE’s decision to limit its use in their technology appraisal
process. From its advent in 1999 until the end of 2005, NICE’s
standard process involved a third-party academic group undertaking
a technology assessment report (TAR) to input, together
with manufacturer and professional evidence and analyses, into
the Appraisal Committee’s deliberations. The TAR consists of a
review of relevant clinical and economic evidence, a critical
assessment of one or more manufacturer submissions (which
includes the manufacturers’ own reviews and models), and,
usually, the development of a cost-effectiveness model. These
arrangements still exist for some technologies as part of NICE’s
multiple-technology appraisal (MTA) process. Nevertheless,
since 2006, many technologies (particularly newly licensed
pharmaceuticals) enter a new single-technology appraisal (STA)
process [3]. These arrangements are similar to those in many
systems internationally in that the only reviews and analyses
informing decision-making are undertaken by the manufacturer,
although, as in some other systems, these are accompanied by
critical review of the latter’s submission by a third-party evidence
review group.
This article considers the role of third-party technology
assessments and their potential strengths and weaknesses. The
aim is to inform the important policy question of whether healthcare
systems should use third-party technology assessment when
developing new decision-making processes. It can also inform the
specific debate in the United Kingdom about the balance between
the MTA and STA processes at NICE.
A key issue to consider in addressing this question is the
evidence regarding whether third-party assessment makes a
difference to decisions—that is, are decision-making authorities
more or less likely to support the use of a technology if they have
access to a third-party assessment? Given the limited number of
examples of third-party assessment internationally and the differences
between decision-making processes in different systems,
which hampers comparison, there is a dearth of such evidence. In
time, some potentially valuable evidence will be provided by
studying the decisions of the NICE Appraisal Committee and
comparing those made under the MTA system with those from STAs. Nevertheless, such a comparison will inevitably be constrained
by the fact that different technologies have been considered
under the two sets of arrangements. Such a comparison is
provided here for those technologies and indications considered
by both organizations, and this updates and extends the comparison
undertaken by Cairns [4].
To assess different assessment arrangements, the next part of
the article describes the required features of economic evaluation
to inform decision-making about health-care technologies and
considers the possible strengths and weaknesses of third-party
assessment for the achievement of these required features. The
section called “Does Third-Party Assessment Make a Difference to
Decisions” presents the comparison of NICE (MTA) and SMC
decisions for a range of technologies. The last section provides a
discussion that considers implications for health-care systems
internationally.
There is evidence that some manufacturers may be overoptimistic
in analyses of the cost-effectiveness of their products [7–9]. Nevertheless,
the risks of inappropriate or misleading analyses in
manufacturer assessments can be reduced using thorough critical
reviews of submissions such as those adopted by many healthcare
systems, including NICE (STA) and SMC. The success of
careful review in identifying problems with manufacturer assessment
can be increased in two ways. The first is for the system to
produce detailed and prescriptive methods guidance that defines,
as fully as possible, how it wants manufacturers to undertake
economic evaluations, thus minimizing the room for judgment
on the part of companies. The critical review of the manufacturer’s
submission is then undertaken against this methods guidance.
This would be supported by the decision-maker being as
transparent as possible in describing its decisions and how they
relate to the (lack of) evidence. Currently, however, most
methods guidelines internationally are quite general and nonprescriptive
[2], and few systems provide complete details of how
their decisions are made. The second way to underpin the role of
the review of company assessments is for decision-makers to
encourage a relationship with the manufacturers that in the
context of a particular piece of analysis, facilitate an iterative line
of communication between the two about appropriate analytic
methods.
This article has argued that there may be situations when
third-party assessment has some important advantages. The first
is when an important element of the evidence base is under the
control of a manufacturer but not the one undertaking the
assessment. Here, it may be possible for the decision-maker to
negotiate access to this evidence and to make it available to a
third-party assessment team. Second, when the manufacturer is
apparently unable to implement appropriate methods, this could
be the case with a small manufacturer that does not have the
appropriate expertise or necessary resources. Perhaps more often
it will be when a manufacturer has failed to reflect all the uncertainties
in their economic evaluation or to assess how costeffectiveness
might vary between a fully specified set of patient
subgroups.
There are some important limitations to the comparison
between the SMC and NICE (pre-STA) in the earlier section.
First, relatively few comparisons of decisions were found (25,
including 22 drugs and 18 indications), so the differences in
decisions were also small in number. Hence, this sample may be
too small to drive any firm conclusion about the comparability
of the two approaches. Second, the timing of NICE and SMC
guidance is often different, and a gap in published appraisal of
more than 2 years was found for some products. Thus, new
evidence might have been available during this gap of time and
could partly explain some differences in the recommendations.
Third, although NICE publishes its full guidance on the
website, the SMC only provides a short version, and some
details may not be available to the public. Finally, there may be
some limits to the independence of decisions between SMC and
NICE; in particular, in the nine cases where the SMC’s decision
was taken after initial NICE guidance, the latter may have
influenced the former, thus limiting the comparison. Despite
these limitations, some cautious conclusions seem warranted.
In particular, the comparison suggests a trend for the NICE guidance to be more restrictive and to give more information
on patient subgroups for which a product is recommended. The
example of the two organizations’ decisions about treatments
for osteoporosis suggests that in terms of population net
benefits, a failure to reflect heterogeneity can impose significant
costs on health systems. The absolute value of these costs
should be considered cautiously given the uncertainty in their
estimation and the possibility of “indication creep” when
actual use of a product in routine practice does not adhere to
the initial guidance regarding the specific subgroups of patients
for which it was recommended. Nevertheless, the analysis
shows the broad scale of these potential costs, and this can be
set against the costs of third-party assessment.
Although there may be some advantages to third-party assessments,
as discussed above, the resource cost of these arrangements
cannot be ignored, particularly in smaller countries and in systems
that use technology assessment to inform centralized decisions
about all newly licensed pharmaceuticals. This might suggest that
systems could consider some form of hybrid assessment arrangements.
For many technologies, an adequate evidential and analytical
basis for decisions could be provided by manufacturer
assessment. This would be strengthened through high-quality
review of submissions against clearly defined and prescriptive
methods guidance. Nevertheless, there are likely to be situations
when third-party assessment (in addition to analyses submitted
by manufacturers) would be highly advantageous for reliable
decision-making. One may exist when important sources of evidence
are not available to a manufacturer, but could be accessed by
a third-party group.Agood example of this would be when two or
more new pharmaceuticals for a particular patient group are
launched at a similar time and are comparators to each other in an
assessment, but no single manufacturer has access to all manufacturers’
evidence. A third-party assessment (together with manufacturer
submissions) may be the best approach in this context,
although this would probably need to feed into a simultaneous
decision about all the technologies. This ability to choose between
manufacturer and third-party assessment in advance is a feature of
the existing arrangements at NICE where technologies can either
be channeled along the MTA or STA routes, although it should be
noted that since the advent of STA, relatively few pharmaceuticals
have been assessed using MTA.
There may, however, be situations where information suggesting
advantages for third-party assessment only emerges after a
manufacturer assessment has taken place. For example, lack of
access to important evidence sources by the manufacturer may
only become clear at this stage. Perhaps more likely is a manufacturer
assessment in which the decision-making body, informed
by a critical review, believes there are methodological weaknesses
that the manufacturer has been unable or unwilling to correct
and that have important implications for the decision. These
weaknesses may relate, for example, to the absence of a full
quantification of uncertainty or analysis of heterogeneity, or the
failure to incorporate appropriate comparators or relevant evidence.
In such situations, the decision-maker may wish to commission
a third-party assessment, particularly when the potential
costs to the health system (in terms of health forgone) from an
inappropriate decision are high (e.g., because the decision relates
to a large patient population or where there are large sunk costs
to be incurred in investing in a new technology). Under such
arrangements, manufacturer assessment effectively becomes a
screening process: when it is of good quality and the appropriate
decision is relatively clear, it can suffice, but when there are major
flaws that are likely directly to affect the decision and the costs of
a wrong decision are high, proceeding to a third-party assessment
may be appropriate.In conclusion, there are some important advantages to the use
of third-party technology assessment to support decisionmaking,
and there is some evidence that its availability may
influence decisions. Nevertheless, there are serious questions
about the practicality and efficiency of using such assessment for
decisions regarding all new technologies. The use of a hybrid
system where third-party assessment is used to support those
decisions for which is it most crucial may offer a sensible way
forward.
