On 2017 Nov 16, David Keller commented:

Thank you, again, for your illuminating and scholarly reply to my comments and questions. Your field of causal inference theory may provide a much-needed bridge spanning the chasm between the land of rigor where mathematicians dwell, and the land of rigor mortis inhabited by clinicians and patients. I will continue to follow your work, and that of your colleagues, with great interest.

On 2017 Nov 16, Ian Shrier commented:

We completely agree on objectives and importance of estimating the per protocol effect. It is absolutely the effect that I am interested in as a patient, and therefore as a physician who wants to communicate important information to the patient.

I do think we have different experiences on how people interpret the words "per protocol analysis". Historically, this term has been used to mean an analysis that does not estimate the per protocol effect except under unusual contexts. More recently, some have used it to refer to a different type of analysis that does estimate the per protocol effect. The field of causal inference is still relatively new and there are other examples of changing terminology. I expect the terminology will stabilize over the next 10 years, which will make it much easier for readers, authors and reviewers.

On 2017 Nov 15, David Keller commented:

Thank you for your thoughtful reply to my comment. In response, I would first remark that, while "it is sometimes difficult to disentangle the jargon", it is always worthwhile to clearly define our terms.

Commonly, an "analysis" is a method of processing experimental data, while an "effect" is a property of experimental data, observed after subjecting it to an "analysis".

As applied to our discussion, the "per protocol effect" would be observed by applying "per protocol analysis" to the experimental data.

My usage of "per protocol results" was meant to refer to the "per protocol effect" observed in a particular trial.

The above commonly-understood definitions may be a reason causal inference terminology "can get quite confusing to others who may not be used to reading this literature", for example, by defining "per protocol effect" differently than as "the effect of per protocol analysis".

Nevertheless, clinicians are interested in how to analyze clinical study data such that the resulting observed effects are most relevant to individual patients, especially those motivated to gain the maximal benefit from an intervention. For such patients, I want to know the average causal effect of the intervention protocol, assuming perfect adherence to protocol, and no intolerable side-effects or unacceptable toxicities. This tells the patient how much he can expect to benefit if he can adhere fully to the treatment protocol.

Of course, the patient must understand that his benefits will be diminished if he fails to adhere fully to treatment, or terminates it for any reason. Still, this "average expected benefit of treatment under ideal conditions" remains a useful goal-post and benchmark of therapy,despite any inherent bias it may harbor, compared with the results of intention-to-treat analysis.

On 2017 Oct 31, Ian Shrier commented:

Thank you for your comment. We seem to be in agreement on what many patients may be most interested in.

In your comment, you say "the per-protocol results of a treatment are, therefore, of interest to patients and their clinicians, and should be reported by clinical trials, along with appropriate statistical caveats and disclaimers."

The words "per protocol results" might mean different things to different people and I thought it important to clarify some of the terminology, which can get quite confusing to others who may not be used to reading this literature.

In the causal inference literature, it has been suggested that we use "per protocol analysis" to refer to an analysis that examines only those participants who follow their assigned treatment. This is different from the "per protocol effect" (also known as population average causal effect), which estimates the causal effect of what would be observed if the entire population received a treatment compared to the entire population not receiving a treatment.

Further, when we refer to the causal effect of “treatment”, we really mean the causal effect of a “treatment strategy”. For example, clinical practice would be to discontinue a medication if there is a serious side effect. In a trial, this would be part of the protocol. Therefore, a person with a serious side effect still counts as following the “treatment strategy” (i.e. the protocol of a per protocol effect) even though they are no longer on treatment.

In brief, the per protocol analysis and per protocol effect are only the same under certain conditions. Assume a randomized trial with the control group receiving usual care and also not having access to the active treatment. In this case, those who are assigned active treatment and do not take their active treatment still receive the same usual care as the control group. The per protocol analysis will be the same as the per protocol effect only if these non-adherent active treatment group participants receiving usual care have the same outcomes on average as those assigned to the control group receiving usual care. This is an assumption that many of us are reluctant to make because the reasons for non-adherence are often related to the probability of the outcome. This is why more sophisticated analyses are helpful in estimating the true population average causal effect.

I hope this makes sense. It is sometimes difficult to disentangle the jargon and still be 100% correct in statements.

On 2017 Oct 24, David Keller commented:

Patients considering an intervention want to know the actual effect of receiving it

Motivated patients are not very interested in how much benefit they can expect to receive from being assigned to a therapy; they want to know the benefits and risks of actually receiving the treatment. The average causal effect of treatment is, for these patients, more clinically relevant than the average causal effect of assignment to treatment.

Intention-to-treat analysis may be ideal for making public health decisions, but it is largely irrelevant for treatment decisions involving particular individuals. Patients want personalized medical advice. A patient's genetic and environmental history may modify his expected results of receiving treatment, and the estimated effects should be discussed.

Regardless of their inherent biases, the per-protocol results of a treatment are, therefore, of interest to patients and their clinicians, and should be reported by clinical trials, along with appropriate statistical caveats and disclaimers.

On 2017 Oct 24, David Keller commented:

Patients considering an intervention want to know the actual effect of receiving it

Motivated patients are not very interested in how much benefit they can expect to receive from being assigned to a therapy; they want to know the benefits and risks of actually receiving the treatment. The average causal effect of treatment is, for these patients, more clinically relevant than the average causal effect of assignment to treatment.

Intention-to-treat analysis may be ideal for making public health decisions, but it is largely irrelevant for treatment decisions involving particular individuals. Patients want personalized medical advice. A patient's genetic and environmental history may modify his expected results of receiving treatment, and the estimated effects should be discussed.

Regardless of their inherent biases, the per-protocol results of a treatment are, therefore, of interest to patients and their clinicians, and should be reported by clinical trials, along with appropriate statistical caveats and disclaimers.

On 2017 Oct 31, Ian Shrier commented:

Thank you for your comment. We seem to be in agreement on what many patients may be most interested in.

In your comment, you say "the per-protocol results of a treatment are, therefore, of interest to patients and their clinicians, and should be reported by clinical trials, along with appropriate statistical caveats and disclaimers."

The words "per protocol results" might mean different things to different people and I thought it important to clarify some of the terminology, which can get quite confusing to others who may not be used to reading this literature.

In the causal inference literature, it has been suggested that we use "per protocol analysis" to refer to an analysis that examines only those participants who follow their assigned treatment. This is different from the "per protocol effect" (also known as population average causal effect), which estimates the causal effect of what would be observed if the entire population received a treatment compared to the entire population not receiving a treatment.

Further, when we refer to the causal effect of “treatment”, we really mean the causal effect of a “treatment strategy”. For example, clinical practice would be to discontinue a medication if there is a serious side effect. In a trial, this would be part of the protocol. Therefore, a person with a serious side effect still counts as following the “treatment strategy” (i.e. the protocol of a per protocol effect) even though they are no longer on treatment.

In brief, the per protocol analysis and per protocol effect are only the same under certain conditions. Assume a randomized trial with the control group receiving usual care and also not having access to the active treatment. In this case, those who are assigned active treatment and do not take their active treatment still receive the same usual care as the control group. The per protocol analysis will be the same as the per protocol effect only if these non-adherent active treatment group participants receiving usual care have the same outcomes on average as those assigned to the control group receiving usual care. This is an assumption that many of us are reluctant to make because the reasons for non-adherence are often related to the probability of the outcome. This is why more sophisticated analyses are helpful in estimating the true population average causal effect.

I hope this makes sense. It is sometimes difficult to disentangle the jargon and still be 100% correct in statements.

On 2017 Nov 15, David Keller commented:

Thank you for your thoughtful reply to my comment. In response, I would first remark that, while "it is sometimes difficult to disentangle the jargon", it is always worthwhile to clearly define our terms.

Commonly, an "analysis" is a method of processing experimental data, while an "effect" is a property of experimental data, observed after subjecting it to an "analysis".

As applied to our discussion, the "per protocol effect" would be observed by applying "per protocol analysis" to the experimental data.

My usage of "per protocol results" was meant to refer to the "per protocol effect" observed in a particular trial.

The above commonly-understood definitions may be a reason causal inference terminology "can get quite confusing to others who may not be used to reading this literature", for example, by defining "per protocol effect" differently than as "the effect of per protocol analysis".

Nevertheless, clinicians are interested in how to analyze clinical study data such that the resulting observed effects are most relevant to individual patients, especially those motivated to gain the maximal benefit from an intervention. For such patients, I want to know the average causal effect of the intervention protocol, assuming perfect adherence to protocol, and no intolerable side-effects or unacceptable toxicities. This tells the patient how much he can expect to benefit if he can adhere fully to the treatment protocol.

Of course, the patient must understand that his benefits will be diminished if he fails to adhere fully to treatment, or terminates it for any reason. Still, this "average expected benefit of treatment under ideal conditions" remains a useful goal-post and benchmark of therapy,despite any inherent bias it may harbor, compared with the results of intention-to-treat analysis.

On 2017 Nov 16, Ian Shrier commented:

We completely agree on objectives and importance of estimating the per protocol effect. It is absolutely the effect that I am interested in as a patient, and therefore as a physician who wants to communicate important information to the patient.

I do think we have different experiences on how people interpret the words "per protocol analysis". Historically, this term has been used to mean an analysis that does not estimate the per protocol effect except under unusual contexts. More recently, some have used it to refer to a different type of analysis that does estimate the per protocol effect. The field of causal inference is still relatively new and there are other examples of changing terminology. I expect the terminology will stabilize over the next 10 years, which will make it much easier for readers, authors and reviewers.

On 2017 Nov 16, David Keller commented:

Thank you, again, for your illuminating and scholarly reply to my comments and questions. Your field of causal inference theory may provide a much-needed bridge spanning the chasm between the land of rigor where mathematicians dwell, and the land of rigor mortis inhabited by clinicians and patients. I will continue to follow your work, and that of your colleagues, with great interest.