I wasn’t going to blog about this because meh, but I really think it’s important to point out the obvious flaws in this “study”.
A news item announced the results of an acupuncture study today, proclaiming in the headline “Acupuncture as effective as drugs in treating pain, trial shows.”
An acupuncture trial in four Melbourne emergency departments has found it is just as good as drugs in relieving lower-back pain and that from sprained ankles and migraines.
Emergency physicians at The Alfred, Northern, Cabrini and Epworth hospitals partnered with RMIT’s school of health sciences to see if acupuncture could relieve acute pain in hundreds of patients presenting to hospital with either lower-back pain, sprained ankles or migraines.
Ok, fine. But then this
“While data from the study is still being analysed and finalised for publication in a medical journal, one of the researchers, Dr Michael Ben-Meir, said it showed acupuncture offered the same level of pain relief as analgesic drugs when patients rated their pain one hour after treatment.
Wait a second….
“While data from the study is still being analysed and finalised for publication…”?
So it’s not been published yet? And the data is still being analysed? WTAF?!
Then why are these researchers apparently alerting the media to their findings when they haven’t even passed peer review? I have a few ideas on why this might be, which I’ll get to in a minute.
But first, it’s obviously difficult, nee impossible, to critique the findings effectively in lieu of a paper, but there are some obvious faults that stand out.
So we have three treatment groups:
1) standard pharmaceutical care alone (eg Endone, Panadeine Forte, Voltaren, Valium)
2) acupuncture alone
3) combination of acupuncture with standard pharmaceutical care
Findings: acupuncture alone delivered equivalent pain relief when compared to a combination of acupuncture with standard pharmaceutical care or standard care alone, reported 1 hr after treatment.
Conclusion: acupuncture is as effective as drugs for lower-back pain, sprained ankles and migraines
Well, the trouble with acupuncture is it’s hard to blind people to the treatment (and this study was not blinded) so researchers sometimes use what’s known as sham acupuncture. This means needles are still used but they’re either inserted in the wrong place (i.e., not along the magical meridian lines) or they are not inserted far enough to have an effect. This way the researcher can control whether the acupuncture is having a real effect or it’s the placebo effect. (sham acupuncture has it’s own issues too, however).
Since it’s not indicated if this control is used here, there’s no way to determine whether the effect from the acupuncture was real.
Patients reported they felt as much pain relief from acupuncture after one hour, as they did on pain killers.
Well sure, but we can’t draw any conclusions from this without an acupuncture control group. Pain is extremely subjective and is highly susceptible to placebo. Plus, people often feel better one hour after treatment anyway – it’s because they’ve been in hospital and been attended to. They’ve been looked after. Of course they feel better. Without effective blinding of treatments there is simply no way to tell.
In addition the placebo effect is a curious beast – the more invasive a procedure, the stonger the placebo can be, so you can imagine having needles stuck in you is likely to make you think your pain has reduced as much or even more than if you took a tiny white pill.
This claim also seems questionable, “..could open the doors to Australian hospitals offering the low-cost Chinese therapy, which is used by more than 1 billion people worldwide for pain relief.”
It’s not as simple as this surely. Traditional Chinese medicine practitioners now need to be registered with APHRA, and what’s so expensive about a packet of Panadeine Forte? (And on that note, why were apparently a random assortment or analgesics used? And if that is the case, does each type of analgesic constitute a separate treatment group?).
Look, all of this is speculation because in the absence of 1) the paper or 2) the correct controls, this study is meaningless.
Which leads me back to why it’s been announced in the press and not in the pages of peer review. I suspect it might not pass peer review without the sham controls. Also odd is that the research data Australia website says the trial began in 2008, and was due for completion in 2011. So why have the results not been published yet? Is it because they likely won’t be?
This would be a very big worry for the researchers, especially given that the NHMRC gave these guys $416,301. Are they announcing it in the press in an attempt to distract the NHMRC?
Even more of a worry though is the researchers apparent disregard for the Australian Code for the Responsible Conduct of Research. Section 4.12.1 says, “Discussing research findings in the public arena should not occur until the findings have been tested through peer review.”
By all means, study the efficacy of TCM, but at least do it with correctly designed experiments and try not to waste $400K of tax payers’ money (as if research doesn’t have enough money to go around). Until such time as that is done, approach me in A&E with an acupuncture needle, you’ll get a slap upside the head.
EDIT: A friend just suggested this on Facebook and I want to add it because it’s an important point that I should have included.
Wendy Wilkinson: “Id love to see the methods used for this research. If it ever gets published properly, that is. How did they recruit the participants? How were they randomised into the three groups? In busy EDs where this sort of thing is not an emergency. And most importantly – how long did they wait for the treatment? Because if they had been sitting in the waiting room for longer than that 1 hr how have they accounted for regression to the mean as far as pain goes.
Yep, regression to the mean may have also played a role in this study. VIP point.
UPDATE Monday March 31: Another issue with this study not being blinded is it may have selected for patients who are sympathetic to the efficacy of acupuncture. So if you rock up to A&E and someone says, “do you want to be in our acupuncture trial?”, if you’re me you’ll say “%^&^%&^ get me some drugs with a demonstrated mechanism of action please!!”. But those who think there’s something to acupuncture will more likely say yes. This then influences their pain perception – if you believe it, then it works (ie placebo).
Also, we must consider the role of “wanting to please the nice Dr”. Patients may report a reduction in pain (later found to be equivalent to the analgesics, well, ok, they haven’t finished analysing the data yet so who actually knows) because they have been fussed over and maybe taken to a private room to lie down and relax etc.
All of these things can introduce confounding variables into studies, which is why controls are critical. The appropriate controls were not used here. This study is worthless.
Monday March 31st. 11:51: Looks like there might be a few updates on this story.
Firstly, I want to clarify something re: 4.12 Responsibly communicating research findings in the public arena. I did not include this section of 4.12.1 (an oversight, not deliberate), “In discussing the outcomes of a research project, special care should be taken to explain the status of the project – for example, whether it is still in progress or has been finalised”.
So the researchers did make this clear – that being the results are still being analysed. Thus it might be fairer to say them releasing this is bad practice rather than a breach of the Australian Code for the Responsible Conduct of Research.
Further, I asked the Society and Science editor at The Age (where this was published) whether they had received this as a press release. He’s not answered this question yet.
- « How to avoid false balance and why you should
- » Acupuncture in emergency departments: bad science and bad practice