A few posts ago I discussed the challenges of transferring research into practice, and discussed the examples of laterality training and graded exposure for CRPS. It’s difficult to know exactly what results to expect when moving from carefully selected participants to all-comers, and from highly detailed and prescribed protocols to more general principles and often varied application by a range of clinicians.
This report, then, by Morley, Williams and Hussein is a welcome review of the ‘real’ effects of ‘real’ therapy on ‘real’ patients in a typical clinical situation.
CBT-based pain management programmes have been implemented internationally and are probably the gold standard intervention for people who have completed all the biomedical interventions available, and have been told to ‘learn how to live with it’. CBT programmes are all about ‘how to live with it’!
This study reviews more than 1000 patients, over a 10 year period, accepted into a 4 week programme. ‘Data from more than 800 patients was available at pre-treatment and at one month post-treatment and for around 600 patients at pre-treatment and at 9 months follow-up. Measures reported in this analysis were pain experience and interference, psychological distress (depression and anxiety), self-efficacy, catastrophizing, and walking.’
As the authors of this study remark, quoting Barkham and Mellor-Clark (2003), ‘a model of research into efficacious and effective treatment should be based on a cycle in which evidence based practice (EBP) informs clinical practice which then generates evidence and questions (practice-based evidence) for testing under more controlled conditions of EBM.’
Now what is interesting about this study, apart from the realism of the setting and patients included, is the use of ‘reliable change index (RCI)/clinically significant change (CSC) methodology’ to evaluate the outcomes of the programme. This differs from the conventional inferential statistics (P values), the assumptions of which are routinely broken in clinical research – such as having a control group (the waiting list is not quite the same as a randomly assigned control group), and this type of statistic isn’t referenced to any external criteria – such as ‘does this change matter clinically?’, and it’s also sensitive to sample size.
I won’t review the programme content, measures used, or sample characteristics – to me, the programme content is ‘standard CBT’, and is based on Fordyce, Keefe and Turk’s work. The measures included the battery of pain intensity, Beck Depression Inventory, Coping Skills Questionnaire and Pain Self Efficacy Questionnaire. Nothing new in these outcome measures that have been used for many years! And the patients are very similar to those attending any chronic pain centre – mean pain duration of 113 months, 95% taking medication, few were employed (6%), average age of around 45 years, and just over half were women.
Using conventional statistical analyses, the results were significant. The authors reason that this may be because of the large sample size, and given that the effect sizes were small (.3–.49) to medium (.5–.8), the statistics cannot be used to support claims that any change was ‘above and beyond that produced by the measurement error inherent in the scales, or that the changes were clinically important.’
So, the authors set about using ‘Reliable Change Index’ and ‘Clinically Significant Change’, which are methods for determining whether change is about measurement error (RCI) or clinically significant (CSC). So, by predetermining ‘acceptable’ change criteria, the real value of the scores on various measures are made useful.
RCI is determined from obtaining the standard deviation of the measure (obtainable from the sample) and an estimate of the reliability of the measure, usually taken from the literature. CSC, using Jacobson’s determination employs statistical criteria to establish cut scores for continuous variables that essentially use the properties of the normal distribution.
Quoting directly from the article:
The criteria (a), (b) and (c) are defined
as follows: (a) is achieved when the post-treatment (or follow-
up) score lies outside of the range of the dysfunctional
population, where the range is defined as extending 2 SD units
beyond the mean for that population in the direction of a functioning
population; (b) is achieved when the post-treatment (or
follow-up) score lies within the range of the functioning population,
where the range is defined as within 2 SD units of the
mean of the functioning population; and (c) is defined as when
the post-treatment (or follow-up) score is statistically more
likely to be in the functional population – i.e. nearer to the
mean of the functional population than the mean of the dysfunctional
population. For practical reasons the sample of participants
was regarded as the population.
After using this approach, the following results were obtained:for measures of pain, emotional distress and self efficacy between one third and one fifth of patients achieved clinically significant outcome. A considerably smaller number (6%, or 1 in 17) achieved a clinically significant change on a measure of behavioural activity, the 5-minute walk test.
The RCI/CSC methodology explicitly separated patients above and below CSC cut points pre-treatment,
a feature which has not generally been reported in RCTs of psychological treatments for chronic pain. The results from this study show that factors other than statistics can be used to evaluate the effects of an intervention – but the method to derive the ‘clinically significant’ results are underpinned by sound statistics. Almost the best of both worlds!
Problems? Well, it is difficult to find treatment providers using the same clinical measures – a minimum data set would help so that programmes can review the effectiveness of their programme, and perhaps allow aggregation of data from different programmes as well.
Barkham M, Mellor-Clark J. Bridging evidence-based practice
and practice-based evidence: developing a rigorous and relevant
knowledge for the psychological therapies. Clin Psychol Psychother
Morley, S., Williams, A., Hussein, S. (2008). Estimating the clinical effectiveness of cognitive behavioural therapy in the clinic: Evaluation of a CBT informed pain management programme. Pain, epub