Even in health care, the loudest voice with the largest opinion can be the most persuasive – even with limited (or selective) use of scientific evidence. Sadly, fads exist in pain management too.
To counter our human biases we need to be critical of all research, and ask some serious questions about accepted practice as well. In most forms of allied health (as well as medical health) there are some ways of working that are based a lot more on ‘what we’ve always done’ and ‘it seems to work’, or even ‘but it works for this person’ or ‘I’ve seen it work for people like this’ than evidence from well-controlled trials.
Some people argue with me about this point saying ‘but if we only did what there was evidence for, we’d having nothing to offer’! Ummmm. That doesn’t mean that what you’re offering is doing any good!
So… what is critical appraisal? This link leads to a great pdf doc summary by Alison Hill and Claire Spittlehouse of just what questions you should consider if you’re reading a research article. I thought I’d summarise it briefly today, but I strongly encourage you to read the full article yourself.
Their definition of critical appraisal reads “Critical appraisal is the process of systematically examining research evidence to assess its validity, results and relevance before using it to inform a decision.”
They go on to say “Critical appraisal is an essential part of evidence-based clinical practice that includes the process of systematically finding, appraising and acting on evidence of effectiveness.”
They agree that sometimes carrying out a critical appraisal can be disheartening – research on clinical interventions can be flawed, have poor methodology, and can highlight just how little reliance we can have on what we do being helpful. We are sometimes working in the dark but putting on a good show to suggest that what we learned in our training ‘works’.
And I’m going to cut and paste the complete set of questions they recommend using when trying to appraise a piece of research. These questions are developed by Guyatt et al.
A. Are the results of the study valid?
1. Did the trial address a clearly focused research question?
Tip: a research question should be ‘focused’ in terms of:
l The population studied
l The intervention given
l The outcomes considered.
2. Did the authors use the right type of study?
Tip: the right type of study would:
l Address the research question
l Have an appropriate study design.
Is it worth continuing?
3. Was the assignment of patients to treatments randomised?
Tip: consider if this was done appropriately.
4. Were all of the patients who entered the trial properly accounted for at its conclusion?
Tip: look for:
l The completion of follow-up
l Whether patients were analysed in the groups to which they were randomised.
5. Were patients, health workers and study personnel ‘blind’ to treatment?
Tip: this is not always possible, but consider if it was possible – was every effort made to ensure ‘blinding’?
6. Were the groups similar at the start of the study?
Tip: think about other factors that might effect the outcome such as age, sex, social class.
7. Aside from the experimental intervention, were the groups treated equally?
Tip: for example, were they reviewed at the same time intervals.
B. What are the results?
8. How large was the treatment effect?
9. How precise was the estimate of the treatment effect?
Tip: look for the confidence limits.
C. Will the results help locally?
10. Can the results be applied to the local population?
Tip: consider whether the patients covered by the trial are likely to be very different from your population.
11. Were all clinically important outcomes considered?
12. Are the benefits worth the harms and costs?
© Critical Appraisal Skills Programme
The three questions to ask yourself when you read research?
Three broad issues need to be considered when appraising research:
A Are the results of the study valid?
B What are the results?
C Will the results help locally?
I leave you with that today – I think it’s quite enough for a Tuesday. Stop and think about the treatment you are using today. Have you read a systematic review of the treatment you’re using? If you have, do the patients you see look anything like those included in the studies? And were the results robust enough for you to justify the treatment of your patients?
Hard questions but fair: let’s not just treat people on the basis that we ‘know we’re right’, or because we are dogmatic.
More tomorrow! Don’t forget to make comments, to subscribe using the RSS feed or to bookmark this blog. And it’s always great to know you’ve visited! You can email me too – go to the ‘About’ page for my email address.
Guyatt GH, Sackett DL, Cook DJ. Users guides to the
medical literature. II: how to use an article about therapy or
prevention. JAMA 1993; 270: 2598Ð2601 and 271: 59Ð63.