The NQA Research and Education Committee is happy to present the second part of series on Evidence Based Practice to help qigong instructors understand and apply research in their practice.
Evidence-Based Practice (EBP) Part 2: The Proof
A report of the NQA Research and Education Committee
11 Dec 2016
Chief contributor: PJ Klein, PT, EdD
EBP evolves in a circular process linked to research.
Practice influences what research is conducted.
Research informs ‘best practice.’
Those in practice apply research to improve practice,
and seek further relevant knowledge.
This the second installment of a series of short papers on understanding Evidence-Based Practice for the Qigong instructor/clinician issued from the NQA Research and Education Committee. The purpose of the series is to better inform interested members on how to use emerging research to validate and market practice to potential clients, referring health providers, and third-party payers. This installment: The Proof, explores basic concepts of causal research within the scientific method.
Introduction to Part 2: the Proof,
Clinical experience and curiosity are foundational for relevant research rather than the proof itself. We practice and teach Qigong, because we have a belief it is effective. In life, this passion serves us well. We are motivated to promote the practice of Qigong. When we inform others who are naïve to Qigong, we advocate its use and list its many benefits in a persuasive way. This same passion may not be so helpful when we wish to objectively view and accurately convey emerging research evidence. Sometimes in our passion (bias), when we read research, we are influenced more by the question: ‘Does it confirm our beliefs?’ rather than ‘How valid or strong is the evidence?’. In evidence-based practice, one’s beliefs of therapeutic effect are established through unbiased scientific inquiry and proof of cause-and-effect obtained within controlled, reproducible study.
What is a causal relationship?
Causal research is research that asks a question about cause and effect. There is a temporal relationship. It tests whether an intervention (e.g., Qigong), once applied, causes an effect on a dependent variable (e.g., pain or sleep quality). For example, an individual has pain: a pain pill is ingested. After 20 minutes, the pain is found to be diminished. Or, an individual has poor sleep quality: Qigong is practiced prior to bedtime and that night the quality of sleep is found to be improved. These are observations suggesting a causal relationship for further study, not scientific proof.
What is scientific proof?
Scientific proof requires a study to control any and all extraneous factors, other than the intervention, that might influence outcome. For example, what if individuals in a study had complaints of both pain and poor sleep quality; and some individuals in the Qigong group decided to take pain pills as well as to initiate practice of Qigong. How would one be able to sort out effect of Qigong vs. pain pill vs. combination of pain pill and Qigong, unless use of pain medications was controlled during the study; or the study was sufficiently large to be able to view these individuals as a subgroup within the study.
If a study is set up that rigorously attempts to control all extraneous factors which might influence an outcome (dependent variable) such as pain or sleep quality, and it shows effect, then some conclusion about causation or therapeutic effect may be proffered. Even though this study implemented rigorous controls, this conclusion is still viewed with caution until additional, independent study repeats the experiment (same conditions, same Qigong protocol) and confirms (reproduces) similar results.
What is causal research validity?
Validity is a measure of a study’s integrity with regard to the truth or factualness of knowledge gained. There are two major categories of causal research validity: internal and external.
What is external validity within causal research?
External validity is the extent to which a study’s results can be generalized beyond that study’s event. This may sound clear cut and easily managed, but it is not. External validity can be challenged in several ways.
What are threats to External Validity?
The following are key questions to ask oneself, before giving clients the expectation that they too will experience the same benefits observed in a given study. Be careful what you promise.
Did the study control for extraneous factors so completely that results are not representative of the real world?
For example, a study conducted as a residential retreat would not directly generalize to once-a-week classes in the community.
How representative are the study subjects?
For example, let’s examine our previous context of pain and sleep management.
In this study, the selection for the study’s participation was limited to individuals who reported high levels of pain and very poor sleep quality. Therefore, one could not offer a promise of pain reduction and improved sleep to individuals in a community Qigong class where pain complaints and sleep disturbance might only be experienced as mild.
How closely is the study Qigong protocol being maintained when used in the real world?
This challenge of generalizability (external validity) within the context of internal Qigong practice raises many more questions than it answers. It will probably be continued to be debated for many years.
What is internal validity within causal research?
Internal validity is a measure of the level of certainty that study results and conclusions truly represent action of the independent variable (Qigong) on the dependent variable (e.g., pain, quality of sleep, sense of well-being, balance, vitality, longevity, immune support, mediation of inflammation, etc.) rather than other known or unknown factors that could influence outcome.
Study design controls to establish internal validity were briefly addressed in the preceding section on Scientific Proof. The following section discusses a selection of threats to internal validity deemed most relevant in studies addressing therapeutic effect of Qigong.
This information may be more than the casual reader wishes to know. And, is probably not information that you would want to spend valuable class time on. However, it is of value to those who scan new research to determine rigor and quality of the research to keenly determine relevance to practice. Be stalwart and forge on; knowledge is power.
The last area of bias control addresses blinding of the person giving the intervention. It is easier to blind the person giving the intervention with pills or injections or a modality delivered through a machine. It is more difficult with Qigong, especially since a trained instructor usually has knowledge of the intervention. Studies of Qigong effect may only have partial blinding.
How do I know the value of a study?
Key questions to ask are:
Is the research relevant to my practice?
Was it conducted with scientific rigor?
Is the summary of the research from a reputable source?
For questions of Qigong effect, the most rigorous study designs are large randomized trials with active control groups and minimization of bias through blinding of group assignment, subjects as to whether they actually received Qigong or not, and assessors who are blind to the group assignment of subjects being tested.
For source of information, one wants to use primary rather than secondary sources, and publication from peer-reviewed journals. Do not base your judgment of the value of a study from a secondary source such as a blog or newsletter. Minimally, get the study abstract. Ideally, get a copy of the full study. Also, rely on expert opinion of the quality of the study. Peer-reviewed journals provide the highest scrutiny of the quality of the study.
Finally, seek study validation. One study is interesting, perhaps promising; but it is not solid evidence of proof. A second independent study reproducing the same results is validation. Only good quality research is reproducible.
What is a peer-reviewed journal?
Research published on sites such as PubMed is from peer-reviewed journals. Peer review means that the study has been reviewed by a panel of independent experts. Expert reviewers have both knowledge of research design and knowledge within the area of practice. Thus giving us some external assurance of the rigor of the research.
What is ethics in research?
Sometimes, we collect outcome data for program evaluation, feedback, and marketing. This can be considered internal research. There is an expectation of privacy by our participants that any data collected is protected, and confidential. However, if one plans to collect program outcome with the intention to publish this information as research, then rules about ethics in research apply. A proposal of the research methods needs to be prepared, submitted, and subsequently approved by a regional Institutional review board (IRB) for the protection of human subjects. Training on protection of human subjects in research is available at the NIH Office for extramural research. To learn more about ethics in research, a free training course for prospective researchers can be accessed at https://phrp.nihtraining.com/users/login.php.
With limited knowledge, can we be confident in our assessment of quality research and standard of proof?
Reading a few papers on research basics will not make a person an expert. However, the more knowledge we have, the better informed we can become. It does get rather challenging to conduct quality research. So, kudos to those researchers in our field who actually have the skills, the determination and the resourcesto provide us with scientific proof of the therapeutic value of Qigong practice.