مقایسه استراتژی تجزیه و تحلیل داده ها برای تحلیل قصد درمان در طرح پیش آزمون-پس آزمون با نرخ قابل توجه ترک تحصیل

The pre-test–post-test design (PPD) is predominant in trials of psychotherapeutic treatments. Missing data due to withdrawals present an even bigger challenge in assessing treatment effectiveness under the PPD than under designs with more observations since dropout implies an absence of information about response to treatment. When confronted with missing data, often it is reasonable to assume that the mechanism underlying missingness is related to observed but not to unobserved outcomes (missing at random, MAR). Previous simulation and theoretical studies have shown that, under MAR, modern techniques such as maximum-likelihood (ML) based methods and multiple imputation (MI) can be used to produce unbiased estimates of treatment effects. In practice, however, ad hoc methods such as last observation carried forward (LOCF) imputation and complete-case (CC) analysis continue to be used. In order to better understand the behaviour of these methods in the PPD, we compare the performance of traditional approaches (LOCF, CC) and theoretically sound techniques (MI, ML), under various MAR mechanisms. We show that the LOCF method is seriously biased and conclude that its use should be abandoned. Complete-case analysis produces unbiased estimates only when the dropout mechanism does not depend on pre-test values even when dropout is related to fixed covariates including treatment group (covariate-dependent: CD). However, CC analysis is generally biased under MAR. The magnitude of the bias is largest when the correlation of post- and pre-test is relatively low.

Missing data are ubiquitous in clinical trials in psychiatry. While observations may be missing because participants are temporarily unavailable or fail to complete a scheduled assessment, withdrawal from a study is by far the most common cause of missingness and is also the cause of greatest concern. Withdrawal threatens the integrity of a trial because it breaks randomization (cf. Peto et al., 1976 and Peto et al., 1977): it can never be determined if attrition is related to an unobserved factor associated with outcome. Failure to take this into account appropriately may result in reaching erroneous conclusions about the effectiveness or ineffectiveness of an intervention. This reasoning underpins the intent(ion)-to-treat (ITT) principle (Guyatt and Rennie, 2001). The ITT principle requires that all participants are retained in analyses regardless of their path through the trial. Participants are retained in the treatment group they are randomized to (“as randomized”), rather than being grouped post hoc according to the actual treatment they received (“as treated”). In the terminology of Schwartz and Lellouch (1967), ITT analysis is a pragmatic approach, the objective of the analysis being to estimate treatment effectiveness rather than efficacy, which is the objective of an “as treated” analysis. The effectiveness concept is arguably of particular relevance in clinical and public health contexts, since a treatment may not be tolerated even if it is efficacious due to aversive side effects or because of the time and effort involved. These effects are reflected in an ITT analysis but not in an “as treated” analysis, where the focus is on efficacy in people who comply fully with treatment. When implementing ITT analysis, the missing measurements are required for participants who have withdrawn. These values are frequently imputed by assuming that the participant's status did not change from the last occasion on which he or she was observed to the end of the trial. This is referred to as last observation carried forward (LOCF) imputation. The origins of this approach are unclear. It is frequently portrayed as yielding a conservative estimate of the treatment effect and its statistical significance. When there is a natural declining time trend in the outcome variable as in degenerative conditions such as dementia, however, assuming stability may yield in an overestimation of treatment effect (Little and Yau, 1995). Even if the estimate of treatment effect is conservative, LOCF imputation may affect the variance and covariance of measures in ways that yield optimistic tests of statistical significance. Another frequently used but essentially ad hoc approach is complete-case (CC) analysis. In this approach, only subjects who complete the trial are retained for analysis. Although directly contradicting the principle of ITT, Graham and Donaldson (1993) showed that, in the case of linear regression, under covariate-dependent (CD) missingness mechanisms (see Section 2.1 for definitions), CC analysis produces an unbiased estimate of treatment effectiveness. This result was supported by Little (1995), who stated that CC is generally unbiased under the CD missingness mechanism. Molenberghs et al. (2004) show that CC is generally biased when the mechanism underlying the missingness is missing at random (MAR), but it is unbiased when the missingness mechanism is missing completely at random (MCAR). In contrast to LOCF and CC, multiple imputation (MI; Rubin, 1987 and Little and Rubin, 2002) and maximum-likelihood based methods (ML; e.g. Verbeke and Molenberghs, 2000) are principled and theoretically rigorous approaches to the problem of missing data in randomized trials. Schafer and Graham (2002) compared MI and ML approaches. They found that the performance of both methods was very similar. In particular, both methods are valid when the MAR missingness mechanism (see Section 2.1) holds, but the methods are biased when the mechanism is non-ignorable in the sense that the probability of withdrawals depends on the unobserved part of the outcome variable. Growing availability of software is seeing the increasing but far from universal application of MI and ML (see Gueorguieva and Krystal, 2004). In psychiatric research, a pervasive and crucial limitation of both simulation studies and investigations using real data has been the failure to explicitly consider trials using a pre-test–post-test design (PPD). For example, Houck et al. (2004) analysed a real dataset from a 12-week antidepressant drug trial using each of the four approaches (CC, LOCF, MI and ML) and found that each approach produced different conclusions. In contrast to pharmacological trials such as these, where measurements are taken at regular intervals, the pre-test–post-test design involves only two occasions of measurement: prior to treatment (pre-test) and after the complete intervention has been delivered (post-test). This design is predominant in trials of psychotherapies and in public health interventions. Its popularity may well reflect budgetary constraints on non-commercial trials. However, psychotherapeutic treatments and public health interventions are conceptualized as integrated packages. Hence it is perceived to be meaningful to take measurements only once the complete program has been delivered. Conceptually, the impact of withdrawal and missing data in the PPD is striking. Where measurements are taken regularly over a trial, available data may characterize individual trajectories reasonably well even for participants who subsequently withdraw. In the PPD, withdrawal implies complete absence of information about response. Applying LOCF in this design (as is frequently done) is actually carrying forward the first and only observation! The advantages of MI and ML over LOCF have been clearly demonstrated when multiple observations are taken over time and thus when some information about response is available (Mallinckrodt et al., 2001). In the context of PPD, Molenberghs et al. (2004) provide general formulae that demonstrate the fact that LOCF is biased under all missingness mechanisms, while CC is unbiased under MCAR but generally biased under MAR. While the formulae are general ones and analytically simple, the parameters governing the formulae are complex functions of the underlying parameters such as intra-individual correlation, rate of departure from differential dropouts and the magnitude of departure from MCAR assumption (the extent of pre-test score effect in governing the missingness mechanism). As a result, direct interpretation of effect of some of these factors is not available. Moreover, the practical examples given by Molenberghs et al. (2004) involve multiple occasions of measurement and subjects with no post-baseline observations (the only form of missingness in the PPD) are excluded. Understanding the behaviour of the bias in terms of these parameters whose interpretation are more ‘direct’ is more appealing and provides a more practical guideline to researchers analysing data with missing observations collected using a PPD. We sought to investigate bias by comparing the performance of MI and ML when applied explicitly to PPD data with that obtained with traditional methods of analysis using CC and LOCF. As far as possible, we sought to identify scenarios under which these traditional methods were valid. Because the MCAR mechanism is unlikely to apply to real applications, we concentrate our simulation on data under MAR and CD missingness mechanisms. Results obtained using these methods of estimations are compared. No data under non-ignorable missingness are simulated, although we discuss the possibility of non-ignorable missingness and suggest several possible approaches under this missingness mechanism (see Section 6.1). The overarching aim of this study was to determine the accuracy of estimates of treatment effectiveness (ITT estimates) obtained using different data-analytic strategies in the presence of participant withdrawals. In order that our results would be applicable to typical trials using PPD, we concentrated on scenarios involving substantial dropout rates. Because trials in psychiatry are often modest in size, we incorporated a range of sample sizes, including small samples, in simulations. Three parameters were used to measure the performance of each method: (i) bias in treatment effect, (ii) power to detect a significant treatment effect and (iii) the accuracy of estimates of the standard error of treatment effect. It should be noted that the emphasis of this study is on comparing the different strategies of tackling the missing data problem and not on the different ways to evaluate treatment effectiveness. See, for example, Rausch et al. (2003) on the comparative advantage of using the different types of analysis (ANOVA versus ANCOVA) and concepts (main treatment effect or treatment by time interaction) in measuring treatment effects in the PPD. The article is organized as follows: In Section 2, the data-analytic strategies are described briefly. In Section 3, we present a comparison of the performance of the different strategies using extensive simulation. In Section 4, results from the simulation studies are discussed. In Section 5, we illustrate the comparison using a real dataset from an e-mental health trial. Finally, in Section 6 we discuss some findings from the study and identify strengths and limitations of each method as applied to the PPD.