Evidence Based Practice: Part 2 The Proof

The NQA Research and Education Committee is happy to present the second part of series on Evidence Based Practice to help qigong instructors understand and apply research in their practice.

Download the Paper.

Evidence-Based Practice (EBP) Part 2: The Proof

A report of the NQA Research and Education Committee

11 Dec 2016

Chief contributor: PJ Klein, PT, EdD

EBP evolves in a circular process linked to research.

Practice influences what research is conducted.

Research informs ‘best practice.’

Those in practice apply research to improve practice,

and seek further relevant knowledge.

This the second installment of a series of short papers on understanding Evidence-Based Practice for the Qigong instructor/clinician issued from the NQA Research and Education Committee. The purpose of the series is to better inform interested members on how to use emerging research to validate and market practice to potential clients, referring health providers, and third-party payers. This installment: The Proof, explores basic concepts of causal research within the scientific method.

Introduction to Part 2: the Proof,

Clinical experience and curiosity are foundational for relevant research rather than the proof itself. We practice and teach Qigong, because we have a belief it is effective. In life, this passion serves us well. We are motivated to promote the practice of Qigong. When we inform others who are naïve to Qigong, we advocate its use and list its many benefits in a persuasive way. This same passion may not be so helpful when we wish to objectively view and accurately convey emerging research evidence. Sometimes in our passion (bias), when we read research, we are influenced more by the question: ‘Does it confirm our beliefs?’ rather than ‘How valid or strong is the evidence?’. In evidence-based practice, one’s beliefs of therapeutic effect are established through unbiased scientific inquiry and proof of cause-and-effect obtained within controlled, reproducible study.

What is a causal relationship?

Causal research is research that asks a question about cause and effect. There is a temporal relationship. It tests whether an intervention (e.g., Qigong), once applied, causes an effect on a dependent variable (e.g., pain or sleep quality). For example, an individual has pain: a pain pill is ingested. After 20 minutes, the pain is found to be diminished. Or, an individual has poor sleep quality: Qigong is practiced prior to bedtime and that night the quality of sleep is found to be improved. These are observations suggesting a causal relationship for further study, not scientific proof.

What is scientific proof?

Scientific proof requires a study to control any and all extraneous factors, other than the intervention, that might influence outcome. For example, what if individuals in a study had complaints of both pain and poor sleep quality; and some individuals in the Qigong group decided to take pain pills as well as to initiate practice of Qigong. How would one be able to sort out effect of Qigong vs. pain pill vs. combination of pain pill and Qigong, unless use of pain medications was controlled during the study; or the study was sufficiently large to be able to view these individuals as a subgroup within the study.

If a study is set up that rigorously attempts to control all extraneous factors which might influence an outcome (dependent variable) such as pain or sleep quality, and it shows effect, then some conclusion about causation or therapeutic effect may be proffered. Even though this study implemented rigorous controls, this conclusion is still viewed with caution until additional, independent study repeats the experiment (same conditions, same Qigong protocol) and confirms (reproduces) similar results.

What is causal research validity?

Validity is a measure of a study’s integrity with regard to the truth or factualness of knowledge gained. There are two major categories of causal research validity: internal and external.

What is external validity within causal research?

External validity is the extent to which a study’s results can be generalized beyond that study’s event. This may sound clear cut and easily managed, but it is not. External validity can be challenged in several ways.

What are threats to External Validity?

The following are key questions to ask oneself, before giving clients the expectation that they too will experience the same benefits observed in a given study. Be careful what you promise.

Did the study control for extraneous factors so completely that results are not representative of the real world?

For example, a study conducted as a residential retreat would not directly generalize to once-a-week classes in the community.

How representative are the study subjects?

For example, let’s examine our previous context of pain and sleep management.

In this study, the selection for the study’s participation was limited to individuals who reported high levels of pain and very poor sleep quality. Therefore, one could not offer a promise of pain reduction and improved sleep to individuals in a community Qigong class where pain complaints and sleep disturbance might only be experienced as mild.

How closely is the study Qigong protocol being maintained when used in the real world?

This challenge of generalizability (external validity) within the context of internal Qigong practice raises many more questions than it answers. It will probably be continued to be debated for many years.

Was the study Qigong protocol sufficiently described so that it can be reproduced in real-world classes?
If a study used a traditional Tai Chi form as the major exercise, do results transfer to traditional internal Qigong exercises, or from one Qigong style to another?
Do frequency of classes influence generalizability?
Does support for home practice influence generalizability?
A final question in this list, although not fully answerable within a given study, is an important question of study generalizability. Do the skills and qualities of the Qigong instructor including emittance of Qi influence generalizability of effect? The ability to emit Qi is known as ‘operator effect.’

What is internal validity within causal research?

Internal validity is a measure of the level of certainty that study results and conclusions truly represent action of the independent variable (Qigong) on the dependent variable (e.g., pain, quality of sleep, sense of well-being, balance, vitality, longevity, immune support, mediation of inflammation, etc.) rather than other known or unknown factors that could influence outcome.

Study design controls to establish internal validity were briefly addressed in the preceding section on Scientific Proof. The following section discusses a selection of threats to internal validity deemed most relevant in studies addressing therapeutic effect of Qigong.

This information may be more than the casual reader wishes to know. And, is probably not information that you would want to spend valuable class time on. However, it is of value to those who scan new research to determine rigor and quality of the research to keenly determine relevance to practice. Be stalwart and forge on; knowledge is power.

Was there an active control group? Did the control group receive an intervention (e.g., playing cards, traditional exercise, stress management education, etc.). If they receive no sham or alternative intervention, then the attention given and expectations of benefit among those in the Qigong group might be the reasons for any observed benefit and not the Qigong.
Do we know if the comparison groups are similar? If the groups are not similar in age, education, socio-economics, ethnicity representation, clinical representation, then any effect between groups might be due to those differences and not Qigong.
Is there adequate sample size for statistical analysis? Statistical analysis determines the likelihood of any differences (effects) between group to be caused by the intervention (Qigong) as opposed to chance (like the difference of result heads-or-tails when flipping a coin). Understanding statistics will be addressed in a later paper in this series.
Were unknown extraneous variables controlled? Randomization is the accepted control for this threat.
Was influence of bias minimized? Bias is any influence exerted when one has an investment or preference for the outcome of a study (e.g., one wishes Qigong to work). Studies are typically protected from bias by a technique known as ‘blinding.’ In a study where Qigong is the intervention, the person allocating group assignment can be blinded to which group gets Qigong. The subject, naïve to Qigong, may be blinded (unknowing) of whether they are receiving true Qigong or sham (fake) Qigong or not. The person conducting evaluation testing may be blind to group assignment. This blinding of the evaluator becomes problematic when outcomes are measured by self-administered surveys, such as pain scales or quality of life surveys.

The last area of bias control addresses blinding of the person giving the intervention. It is easier to blind the person giving the intervention with pills or injections or a modality delivered through a machine. It is more difficult with Qigong, especially since a trained instructor usually has knowledge of the intervention. Studies of Qigong effect may only have partial blinding.

Was cross-group contamination avoided? e.g., Did individuals in the control group learn to regulate breath and meditate while doing sham gentle exercise? Or, did subjects from control vs qigong groups talk with each other during outcome testing sessions or external to study settings such that individuals in the control group were actually getting some knowledge of Qigong on their own. Sometimes study administrators will instruct subjects to not start meditation during the study period to minimize contamination.
Did the intervention group actually get Qigong? e.g., did subjects actually attend the Qigong classes. If home practice was involved in quantity of qigong practiced, were practice logs kept, and how truthful was log reporting?
Was there a placebo or ‘spotlight effect’? Spotlight effect – doing anything and testing for effect, is minimized by use of an active control – the control group gets some activity which engenders expectation of benefit, and randomization in group assignment. With placebo effect,some subjects perhaps as many as 30% if they believe that they are getting an intervention with benefit will perceive the benefit regardless of whether they were in the control group or the Qigong group. This effect is minimized by blinding as to group assignment. However, placebo effect may have unique interest to those interested in effects of mind/body Qigong. It may very well be that those who have the ability to enhance self-healing through intent have an innate ability that might be learned by others. We may find that we are studying this phenomenon more closely in the future rather than to dismiss its utility.
How many subjects failed to complete the study (Attrition)? If attrition is greater than 15-20% reasons for attrition and between group characteristics of dropouts need to be reported.
Was the passing of time a factor in pre- to post-intervention testing? e.g., there is a natural progression of healing in acute back complaints as many as 85% are resolved in 10-14 days without any intervention. Group comparisons minimize this influence.
How did length of study influence results? e.g., long-term effects cannot be measured in studies of short duration.
Was valid and reliable measurement conducted? Was the measurement instrument (like a balance test) one used by other researchers in other studies? Were evaluators who took measurements trained in the technique?
Was there a repeated testing effect? Sometimes, order of testing affects performance on succeeding tests: positively, if anxiety is assuaged or flexibility improved; negatively, if pain or fatigue are induced by preceding tests. These questions are difficult for the consumer of research to assess when reading a study, but of primary importance to researchers designing study measurement protocol.

How do I know the value of a study?

Key questions to ask are:

Is the research relevant to my practice?

Was it conducted with scientific rigor?

Is the summary of the research from a reputable source?

For questions of Qigong effect, the most rigorous study designs are large randomized trials with active control groups and minimization of bias through blinding of group assignment, subjects as to whether they actually received Qigong or not, and assessors who are blind to the group assignment of subjects being tested.

For source of information, one wants to use primary rather than secondary sources, and publication from peer-reviewed journals. Do not base your judgment of the value of a study from a secondary source such as a blog or newsletter. Minimally, get the study abstract. Ideally, get a copy of the full study. Also, rely on expert opinion of the quality of the study. Peer-reviewed journals provide the highest scrutiny of the quality of the study.

Finally, seek study validation. One study is interesting, perhaps promising; but it is not solid evidence of proof. A second independent study reproducing the same results is validation. Only good quality research is reproducible.

What is a peer-reviewed journal?

Research published on sites such as PubMed is from peer-reviewed journals. Peer review means that the study has been reviewed by a panel of independent experts. Expert reviewers have both knowledge of research design and knowledge within the area of practice. Thus giving us some external assurance of the rigor of the research.

What is ethics in research?

Sometimes, we collect outcome data for program evaluation, feedback, and marketing. This can be considered internal research. There is an expectation of privacy by our participants that any data collected is protected, and confidential. However, if one plans to collect program outcome with the intention to publish this information as research, then rules about ethics in research apply. A proposal of the research methods needs to be prepared, submitted, and subsequently approved by a regional Institutional review board (IRB) for the protection of human subjects. Training on protection of human subjects in research is available at the NIH Office for extramural research. To learn more about ethics in research, a free training course for prospective researchers can be accessed at https://phrp.nihtraining.com/users/login.php.

With limited knowledge, can we be confident in our assessment of quality research and standard of proof?

Reading a few papers on research basics will not make a person an expert. However, the more knowledge we have, the better informed we can become. It does get rather challenging to conduct quality research. So, kudos to those researchers in our field who actually have the skills, the determination and the resourcesto provide us with scientific proof of the therapeutic value of Qigong practice.

‍