مدلسازی رایانه ای از دیابت و عوارض آن: گزارشی در مورد چالش نشست پنجم مونت هود

Objectives The Mount Hood Challenge meetings provide a forum for computer modelers of diabetes to discuss and compare models, to assess predictions against data from clinical trials and other studies, and to identify key future developments in the field. This article reports the proceedings of the Fifth Mount Hood Challenge in 2010. Methods Eight modeling groups participated. Each group was given four modeling challenges to perform (in type 2 diabetes): to simulate a trial of a lipid-lowering intervention (The Atorvastatin Study for Prevention of Coronary Heart Disease Endpoints in Non-Insulin-Dependent Diabetes Mellitus [ASPEN]), to simulate a trial of a blood glucose–lowering intervention (Action in Diabetes and Vascular Disease: Preterax and Diamicron Modified Release Controlled Evaluation [ADVANCE]), to simulate a trial of a blood pressure–lowering intervention (Cardiovascular Risk in Diabetes [ACCORD]), and (optional) to simulate a second trial of blood glucose–lowering therapy (ACCORD). Model outcomes for each challenge were compared with the published findings of the respective trials. Results The results of the models varied from each other and, in some cases, from the published trial data in important ways. In general, the models performed well in terms of predicting the relative benefit of interventions, but performed less well in terms of quantifying the absolute risk of complications in patients with type 2 diabetes. Methodological challenges were highlighted including matching trial end-point definitions, the importance of assumptions concerning the progression of risk factors over time, and accurately matching the patient characteristics from each trial. Conclusions The Fifth Mount Hood Challenge allowed modelers, through systematic comparison and validation exercises, to identify important differences between models, address key methodological challenges, and discuss avenues of research to improve future diabetes models.

A decade after Jonathan Brown and Andrew Palmer met to compare the 20-year predictions of two computer simulation models of type 2 diabetes in the Timberline Lodge, high on the side of Mount Hood near Portland, Oregon, the fifth Mount Hood Challenge meeting was held in Malmö, Sweden, in September 2010 [1]. A total of eight modeling groups participated in the 2010 challenge, which followed a similar format to previous Mount Hood meetings whereby modelers were asked to use their prediction models to simulate the outcomes of clinical studies to inform debate on the challenges facing groups working in this area. Computer simulation models, in essence a series of mathematical equations combined in a structured framework, have many uses such as allowing data from clinical trials to be extrapolated over longer time periods and to other populations. By providing information for health care decision makers on long-term clinical outcomes and costs, such models allow informed choices to be made between available interventions. As the issue of cost containment becomes ever more pertinent for many health care decision makers, the reliance on computer simulation modeling is increasing. This is particularly true of chronic diseases such as type 2 diabetes, which develop over a long period of time and are associated with significant morbidity and mortality and a substantial economic burden [2]. Although cost-of-illness studies have taught us a great deal about the scale of the economic burden associated with diabetes, as well as the identity of the main cost drivers, they do little to help us understand the incremental value of new interventions in a given population. Clinical trials provide essential information on new interventions, but their limitations in terms of time frame (typically 1–3 years), tightly controlled designs, and often (highly) selected populations can make their findings difficult to generalize to other care settings or populations. Key parameters such as demographics, life expectancy, patient management/medical technology, treatment costs, and health budgets can vary widely between regions and between countries. Flexible computer models have the potential to overcome these problems and provide valuable information, such as assessments of long-term cost-effectiveness, for policymakers and reimbursement decision makers. To fulfill this role, models must be based on the best available evidence, and validated against clinical data (internal and external validation) as well as each other, and they must also be transparent, documented in detail, and open about their mechanisms and assumptions. The aim of this article was to report the proceedings of the Fifth Mount Hood Challenge held in Malmö, Sweden, in 2010, with a view to providing a summary of how eight current diabetes models match up to data from published clinical studies as well as to each other, to highlight differences between models, and to offer an insight into the challenges facing diabetes models a decade after the First Mount Hood Challenge.

The general consensus on the results presented at the Fifth Mount Hood Challenge was that, in general, the models performed reasonably well in terms of predicting the relative risk of interventions versus control treatments, but less well in terms of the estimation of absolute risk. While relative effects are important to many decision makers, predictions of absolute effects may be needed, not just the relative changes if absolute amounts of money spent or saved need to be predicted, or how many adverse clinical events will be prevented or caused. Absolute rates of events may also be important to people designing clinical trials, as they directly affect the power of the trial and therefore the sample size, duration, and cost. When discussing the difficulty of simulating the trials, it is important to distinguish between simulating the outcomes in the control group versus the outcomes in the intervention group. For some of the trials (e.g., ACCORD glucose lowering) modeling the effect of the intervention was challenging, whereas the outcome rates in the control groups can still provide very useful tests of the models’ accuracies in addressing risk factors and the progression of complications with current care. The selection of trials for the validation exercise at the Fifth Mount Hood Challenge was deliberately challenging. For example, the ASPEN study was specifically chosen because it produced a different result (i.e., no statistically significant benefit of atorvastatin treatment on the primary composite end point) from Collaborative Atorvastatin Diabetes Study (CARDS), which was used in a previous Mount Hood Challenge in 2004. The ADVANCE study reported data from a fairly atypical type 2 diabetes population: patients were generally old, with long-standing advanced diabetes but not on insulin, and with over one-third of the population recruited to study centers in Asia. Historically, there has been a paucity of data available to inform the modeling of “high-risk” populations such as those in ADVANCE and ACCORD. Moreover, the relative lack of data on the cardiovascular risk profile in Asian patients with type 2 diabetes (relative to their Western counterparts) has made accurate modeling of outcomes challenging in the ADVANCE population. Data from the ACCORD trial has challenged previously accepted wisdom on the role of aggressive treatment of risk factors in diabetes. Aggressive treatment of blood pressure did not lead to a statistically significant composite outcome of fatal and nonfatal major cardiovascular events in the ACCORD trial, and intensive therapy targeting Hb A1c value below 6% was shown to increase mortality without significantly reducing major cardiovascular events. Because many of the models rely primarily on data linking risk factors to hard end points from landmark studies that predate these more recent and perhaps atypical data sets, the validation exercise set for the Fifth Mount Hood Challenge was a very demanding one. Many of the models were not able to reproduce all the primary outcomes and many of the secondary outcomes of each of the trials. During the meeting discussion session, the modeling groups raised a number of issues that may have contributed to the discrepancies in absolute risk between the model predictions and the trial results. Matching population characteristics were cited by several groups as a testing aspect of the validation analyses (due to complicated inclusion and exclusion criteria in the trials). Difficulties around effectively modeling risk for a patient with a history of complications (adjusting for the risk of second events) and covariance (e.g., patient age with duration of diabetes and/or history of complications, or ethnicity with baseline blood pressure) were also cited as major challenges, particularly when modeling without patient-level data. Different interpretations of end points were widely acknowledged as a reason that the model results did not match trial results in several cases. (For example, very few of the models captured revascularization as an end point, and differing methods were used to define and measure the occurrence of retinopathy end points.) These difficulties can be compounded when local treatment practice influences the end point. (For example, the clinical decision on when to perform revascularization can vary widely between regions and countries.) Another limitation of most models highlighted during the discussion was their failure to report fatal and nonfatal events separately (even though this is an integral part of the modeling calculations going on in the background). This would seem to be an essential function of a model, and future improvements will need to address these shortcomings. Differing assumptions around the progression of risk factors over time in the modeling analyses were also raised as a barrier to matching the trial results. (For example, Hb A1c, SBP, or serum lipid level changes over time.) It was also clear that different modelers had used different assumptions about other risk factors and this may have been a source of discrepancy in the results presented. As part of this discussion, the paucity of data on risk factor progression in type 2 diabetes was acknowledged. At present the only published formulae for risk factor progression are those from the UKPDS [14]. Although some modern trials have reported data on Hb A1c progression (e.g., Fenofibrate Intervention and Event Lowering in Diabetes study provides data on Hb A1c progression on modern therapy over 5 years), more work is needed in this area. Methodological issues were also discussed at the Fifth Mount Hood Challenge, arising from the challenge simulations. In light of the data from the ACCORD trial, the following question was raised: What should the relationship between Hb A1c levels and mortality be in diabetes models? Although some trial results suggest that the relationship could be U-shaped, more evidence is clearly needed to provide a definitive answer. This is an epidemiological puzzle that may become clearer in the years ahead. Another issue raised concerned the modeling of risk via changing risk factors as opposed to the direct treatment effects on end points (e.g., the gliclazide stroke effect). Most models currently rely on physiological risk factors (and patient characteristics) to estimate the risk of events. Although it would be advantageous to include specific treatment effects on end points in the models, suitable data are seldom available at the time of launch of new agents (when cost-effectiveness analyses are required to support reimbursement decision making). Differences were acknowledged in the way models generate simulation cohorts. In some cases, simulation cohorts were directly generated from distributions based on the available published trial data. In others, an overall population was generated and then selected on the basis of clinical trial inclusion/exclusion criteria to create a simulation cohort. No consensus was reached on which of these approaches would be best, but it was agreed that access to patient-level data would improve the projections made by most of the models presented at the Fifth Mount Hood Challenge. Although access to patient-level data from trials is frequently highly restricted, and patient-level simulation places more demands on computing resources, such data do allow covariance between different risk factors to be captured, and this may have important implications for the accuracy of simulations. To this end, electronic medical registry data may prove to be a valuable resource for estimating covariance matrices. They may also contain a much more heterogeneous set of patients who are more representative of the general population than are trials with restrictive recruitment criteria. Such data sets, however, may also have other selection biases and frequently lack rigorous clinical adjudication of end-point events in comparison with clinical trials. The only models presented at the Mount Hood Meeting that captured the influence of covariance were the UKPDS Outcomes Model and the ECHO-T2DM Model, both of which use a covariance matrix developed from patient-level data. The influence of this matrix on model outcomes has not yet been fully investigated. It may, however, have been one of the reasons why the UKPDS Outcomes Model and the other models at the meeting that rely on individual elements of the same UKPDS regression equations to estimate risk produced different results in the validation exercises. The influence of ethnic characteristics on the risk of complications in patients with type 2 diabetes was also raised as a point of methodology. While it would be optimal to factor this fully into the modeling analyses, ethnic group information is often poorly recorded, confounded with socioeconomic status, and/or subject to high uncertainty because of small numbers. Although the Fifth Mount Hood Challenge Meeting centered primarily on validation exercises as a vehicle to compare and contrast modeling methodologies in different groups, it was clear from the meeting that there is no clear consensus on precisely what model validation means. Appropriate statistical approaches should be defined to assess correlation between model and clinical trial outcomes, and limits could be predefined for model accuracy and precision. A consensus group with appropriate statistical expertise may offer the best opportunity to resolve this long-standing issue. The Fifth Mount Hood Challenge Meeting included a session on dealing with statistical uncertainty in simulation models of type 2 diabetes. Prof. Andrew Briggs highlighted the importance and many of the challenges in terms of dealing with statistical uncertainty in complex disease models. The importance of capturing parameter uncertainty (and dealing with parameter estimation) was emphasized because it affects decision uncertainty. Structural uncertainty in models, an aspect that is often overlooked, is being tackled in part by the modeling comparison at the Mount Hood meetings. Validation of simulation models has an important role to play in improving modeling efforts and, similarly, meetings such as the Mount Hood challenges offer a unique environment to further this cause. The Fifth Mount Hood Challenge modelers have performed an additional set of challenge simulations designed to investigate statistical uncertainty within the individual models (as well as aspects of structural uncertainty), and this analysis will be the subject of a future collaborative article. Data from the UKPDS have revolutionized the modeling of type 2 diabetes. Although UKPDS patients continued to be followed up until 2007, there is clearly a need for additional patient-level data sets to better understand treatment innovations, novel risk factors, and different target populations. As the diabetes epidemic continues to grow, particularly in the developing world, an already complex environment for modelers will give rise to even more challenges. It could be that the future for diabetes modeling will rely on the development of country-specific models or at least country-specific/population-specific risk estimates. Socioeconomic status and the role of molecular genetic testing may have an important role to play in future. An alternative scenario could see collaborative methods, such as those in the field of climate change, where recommendations are made on the basis of averages from the results of more than 20 different models [25]. Regardless of these future avenues of research, meetings such as the Mount Hood challenges will have an important role to play as modelers seek to continually improve on the performance of their diabetes models to better meet the needs of health care decision makers around the world. Modelers in other disease areas might wish to consider whether adopting a similar process in their area would have similar benefits in identifying problems and accelerating improvements.