On 2018 Jan 12, Stefan Tino Kulnik commented:

Further to my comment from 13 October 2017:

In June 2017, we approached Journal of Physiotherapy and submitted a commentary, in which we pointed out the error in this meta-analysis and that the authors’ conclusion with respect to the impact of respiratory muscle training on respiratory complications is therefore unfounded. Unfortunately our commentary was rejected, and so we were denied the opportunity of entering a scientific exchange with the authors within the pages of the journal.

The journal did acknowledge the data extraction error we pointed out and promised a correction, but a correction has not been published to date.

It may be regarded as rather unfortunate that this systematic review and meta-analysis (a study design that many colleagues in clinical practice will view as highest level evidence) presents a strong clinical message in favour of implementing respiratory muscle training for the prevention of respiratory complications, based on an erroneous meta-analysis.

Burden of treatment and opportunity cost to stroke survivors should not be underestimated, and it is important to focus clinical resources on the most meaningful rehabilitation activities based on best evidence.

For members of the research community and colleagues in clinical practice who may be interested, I have uploaded the content of our rejected commentary on my ResearchGate page https://tinyurl.com/yb3wxmkf. In this we also present a re-calculated meta-analysis using Peto odds ratio, which is a more appropriate and statistically more powerful model of meta-analysis when events are rare, to demonstrate that even with this statistically more powerful method the meta-analysis still fails to reach statistical significance of the overall effect.

On 2017 Oct 13, Stefan Tino Kulnik commented:

I write to highlight an error in this systematic review and meta-analysis of respiratory muscle training after stroke. The authors present a meta-analysis, in which data from two randomised controlled trials of respiratory muscle training in stroke were pooled (Figure 6). Respiratory muscle training is reported to result in a statistically significant risk reduction for the outcome respiratory complications (overall risk ratio 0.38, 95%CI 0.15 to 0.96).

It is this journal’s policy to publish simplified forest plots in the main article, and to provide detailed forest plots in an online supplement. It is therefore not immediately evident that in this meta-analysis there has been a data extraction error from one of the included studies.

The detailed forest plot for this analysis (Figure 7 in the online supplement) shows that for meta-analysis two intervention groups in the study by Kulnik ST, 2015 were combined to a total of n=53. The number of respiratory complications in this combined group is given as n=5, but in fact it was n=9. The correct overall risk ratio in meta-analysis (Mantel-Haenszel random effects model in RevMan, Version 5.3, 2014) is therefore 0.49 (95%CI 0.09 to 2.65). While this re-calculated point estimate still favours the intervention, it is no longer statistically significant.

This demonstrates how in studies such as these, where sample sizes are small and events are rare, small inaccuracies can have a considerable influence on statistical results.

Further studies are required to demonstrate a statistically significant effect of respiratory muscle training on respiratory complications after stroke. There is a good theoretical rationale why respiratory muscle training might reduce respiratory complications after stroke, but definitive empirical evidence is lacking at the moment.

On 2017 Oct 13, Stefan Tino Kulnik commented:

I write to highlight an error in this systematic review and meta-analysis of respiratory muscle training after stroke. The authors present a meta-analysis, in which data from two randomised controlled trials of respiratory muscle training in stroke were pooled (Figure 6). Respiratory muscle training is reported to result in a statistically significant risk reduction for the outcome respiratory complications (overall risk ratio 0.38, 95%CI 0.15 to 0.96).

It is this journal’s policy to publish simplified forest plots in the main article, and to provide detailed forest plots in an online supplement. It is therefore not immediately evident that in this meta-analysis there has been a data extraction error from one of the included studies.

The detailed forest plot for this analysis (Figure 7 in the online supplement) shows that for meta-analysis two intervention groups in the study by Kulnik ST, 2015 were combined to a total of n=53. The number of respiratory complications in this combined group is given as n=5, but in fact it was n=9. The correct overall risk ratio in meta-analysis (Mantel-Haenszel random effects model in RevMan, Version 5.3, 2014) is therefore 0.49 (95%CI 0.09 to 2.65). While this re-calculated point estimate still favours the intervention, it is no longer statistically significant.

This demonstrates how in studies such as these, where sample sizes are small and events are rare, small inaccuracies can have a considerable influence on statistical results.

Further studies are required to demonstrate a statistically significant effect of respiratory muscle training on respiratory complications after stroke. There is a good theoretical rationale why respiratory muscle training might reduce respiratory complications after stroke, but definitive empirical evidence is lacking at the moment.

On 2018 Jan 12, Stefan Tino Kulnik commented:

Further to my comment from 13 October 2017:

In June 2017, we approached Journal of Physiotherapy and submitted a commentary, in which we pointed out the error in this meta-analysis and that the authors’ conclusion with respect to the impact of respiratory muscle training on respiratory complications is therefore unfounded. Unfortunately our commentary was rejected, and so we were denied the opportunity of entering a scientific exchange with the authors within the pages of the journal.

The journal did acknowledge the data extraction error we pointed out and promised a correction, but a correction has not been published to date.

It may be regarded as rather unfortunate that this systematic review and meta-analysis (a study design that many colleagues in clinical practice will view as highest level evidence) presents a strong clinical message in favour of implementing respiratory muscle training for the prevention of respiratory complications, based on an erroneous meta-analysis.

Burden of treatment and opportunity cost to stroke survivors should not be underestimated, and it is important to focus clinical resources on the most meaningful rehabilitation activities based on best evidence.

For members of the research community and colleagues in clinical practice who may be interested, I have uploaded the content of our rejected commentary on my ResearchGate page https://tinyurl.com/yb3wxmkf. In this we also present a re-calculated meta-analysis using Peto odds ratio, which is a more appropriate and statistically more powerful model of meta-analysis when events are rare, to demonstrate that even with this statistically more powerful method the meta-analysis still fails to reach statistical significance of the overall effect.