On 2016 Sep 27, Alem Matthees commented:
Post-hoc outcome switching inflated clinical response rates by several times
The primary outcomes as pre-specified in the published PACE trial protocol [1] were abandoned after the trial was over but before the unmasking of data [2] (although the trial was already open-label). While the new primary outcomes are useful, they were group-level and therefore not a direct replacement of the original primary outcomes, which were individual-level i.e. the proportion of participants meeting thresholds for fatigue and physical function.
The thresholds for individual-level improvement on the measures of fatigue and physical function were eventually replaced with post-hoc analyses after the unmasking of data [3], and it is unclear whether these changes were independently approved. The new thresholds have been criticised for being too lax, much less stringent than the pre-specified version, poorly calculated, and a possible example of "outcome reporting bias" [4,5].
On 8 September 2016, Queen Mary University of London (QMUL) released the PACE trial primary outcome results as defined in the published trial protocol [6]. This happened one day before the deadline to appeal the Information Tribunal’s ruling that QMUL must disclose a selection of de-identified participant-level data under the Freedom of Information Act (see EA/2015/0269).
Here is a comparison of the different rates: the clinically useful difference in both fatigue and physical function (Lancet 2011) [3] was 45% for SMC, 59% for CBT, and 61% for GET; the overall improvers or positive outcomes for both fatigue and physical function (QMUL 2016) [6] were 10% for SMC, 20% for CBT, and 21% for GET.
Without going into the issues with the nature and generalisability of the reported benefits, these new results provide a more realistic estimation of the clinical response rates of CBT and GET. However it also indicates that post-hoc outcome switching had inflated the estimated response rates in the PACE trial by several times. This exaggeration allowed proponents of CBT and GET to claim that 60% improved after these therapies (often without mentioning the SMC control rate of 45%). Outcome switching is recognised as a major problem in the research community [7].
There is also evidence that the PACE trial investigators changed the primary outcome measures in the trial registration entry (ISRCTN54285094 [8]). Using web.archive.org shows that an earlier version dated May 2005 [9] describes the primary outcomes with details about the pre-specified thresholds for improvement, but the current version only describes the primary hypotheses being tested or general objectives, without those additional details, allowing the investigators to change their primary outcomes without appearing to be deviating from the trial registration details.
References
1) White PD, Sharpe MC, Chalder T, DeCesare JC, Walwyn R; PACE trial group. Protocol for the PACE trial: a randomised controlled trial of adaptive pacing, cognitive behaviour therapy, and graded exercise, as supplements to standardised specialist medical care versus standardised specialist medical care alone for patients with the chronic fatigue syndrome/myalgic encephalomyelitis or encephalopathy. BMC Neurol. 2007 Mar 8;7:6. PMID: 17397525. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2147058/
2) Walwyn R, Potts L, McCrone P, Johnson AL, DeCesare JC, Baber H, Goldsmith K, Sharpe M, Chalder T, White PD. A randomised trial of adaptive pacing therapy, cognitive behaviour therapy, graded exercise, and specialist medical care for chronic fatigue syndrome (PACE): statistical analysis plan. Trials. 2013 Nov 13;14:386. doi: 10.1186/1745-6215-14-386. PMID: 24225069. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4226009/
3) White PD, Goldsmith KA, Johnson AL, Potts L, Walwyn R, DeCesare JC, Baber HL, Burgess M, Clark LV, Cox DL, Bavinton J, Angus BJ, Murphy G, Murphy M, O'Dowd H, Wilks D, McCrone P, Chalder T, Sharpe M; PACE trial management group. Comparison of adaptive pacing therapy, cognitive behaviour therapy, graded exercise therapy, and specialist medical care for chronic fatigue syndrome (PACE): a randomised trial. Lancet. 2011 Mar 5;377(9768):823-36. doi: 10.1016/S0140-6736(11)60096-2. Epub 2011 Feb 18. PMID: 21334061. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3065633/
4) Giakoumakis J. The PACE trial in chronic fatigue syndrome. Lancet. 2011 May 28;377(9780):1831; author reply 1834-5. doi: 10.1016/S0140-6736(11)60689-2. Epub 2011 May 16. PMID: 21592554. http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60689-2/fulltext
5) Stouten B, Goudsmit EM, Riley N. The PACE trial in chronic fatigue syndrome. Lancet. 2011 May 28;377(9780):1832-3; author reply 1834-5. doi: 10.1016/S0140-6736(11)60685-5. Epub 2011 May 16. PMID: 21592558. http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(11)60685-5/fulltext
6) Goldsmith KA, White PD, Chalder T, Johnson AL, Sharpe M. The PACE trial: analysis of primary outcomes using composite measures of improvement. 8 September 2016. http://www.wolfson.qmul.ac.uk/images/pdfs/pace/PACE_published_protocol_based_analysis_final_8th_Sept_2016.pdf
8) http://www.controlled-trials.com/ISRCTN54285094
9) <http://web.archive.org/web/20050524130106/http://www.controlled-trials.com/mrct/trial/CHRONIC FATIGUE SYNDROME/1042/40645.html>
This comment, imported by Hypothesis from PubMed Commons, is licensed under CC BY.