I wasn’t going to blog about this because meh, but I really think it’s important to point out the obvious flaws in this “study”.

A news item announced the results of an acupuncture study today, proclaiming in the headline “Acupuncture as effective as drugs in treating pain, trial shows.”

An acupuncture trial in four Melbourne emergency departments has found it is just as good as drugs in relieving lower-back pain and that from sprained ankles and migraines.

Emergency physicians at The Alfred, Northern, Cabrini and Epworth hospitals partnered with RMIT’s school of health sciences to see if acupuncture could relieve acute pain in hundreds of patients presenting to hospital with either lower-back pain, sprained ankles or migraines.

Ok, fine. But then this

“While data from the study is still being analysed and finalised for publication in a medical journal, one of the researchers, Dr Michael Ben-Meir, said it showed acupuncture offered the same level of pain relief as analgesic drugs when patients rated their pain one hour after treatment.

Wait a second….

“While data from the study is still being analysed and finalised for publication…”?

So it’s not been published yet? And the data is still being analysed? WTAF?!

Then why are these researchers apparently alerting the media to their findings when they haven’t even passed peer review? I have a few ideas on why this might be, which I’ll get to in a minute.

Do not pass step one, do not collect $200.

Do not pass step one, do not collect $200.

But first, it’s obviously difficult, nee impossible, to critique the findings effectively in lieu of a paper, but there are some obvious faults that stand out.

So we have three treatment groups:
1) standard pharmaceutical care alone (eg Endone, Panadeine Forte, Voltaren, Valium)
2) acupuncture alone
3) combination of acupuncture with standard pharmaceutical care

Findings: acupuncture alone delivered equivalent pain relief when compared to a combination of acupuncture with standard pharmaceutical care or standard care alone, reported 1 hr after treatment.

Conclusion: acupuncture is as effective as drugs for lower-back pain, sprained ankles and migraines

Well, the trouble with acupuncture is it’s hard to blind people to the treatment (and this study was not blinded) so researchers sometimes use what’s known as sham acupuncture. This means needles are still used but they’re either inserted in the wrong place (i.e., not along the magical meridian lines) or they are not inserted far enough to have an effect. This way the researcher can control whether the acupuncture is having a real effect or it’s the placebo effect. (sham acupuncture has it’s own issues too, however).

Since it’s not indicated if this control is used here, there’s no way to determine whether the effect from the acupuncture was real.

Patients reported they felt as much pain relief from acupuncture after one hour, as they did on pain killers.

Well sure, but we can’t draw any conclusions from this without an acupuncture control group. Pain is extremely subjective and is highly susceptible to placebo. Plus, people often feel better one hour after treatment anyway – it’s because they’ve been in hospital and been attended to. They’ve been looked after. Of course they feel better. Without effective blinding of treatments there is simply no way to tell.

In addition the placebo effect is a curious beast – the more invasive a procedure, the stonger the placebo can be, so you can imagine having needles stuck in you is likely to make you think your pain has reduced as much or even more than if you took a tiny white pill.

This claim also seems questionable, “..could open the doors to Australian hospitals offering the low-cost Chinese therapy, which is used by more than 1 billion people worldwide for pain relief.”

It’s not as simple as this surely. Traditional Chinese medicine practitioners now need to be registered with APHRA, and what’s so expensive about a packet of Panadeine Forte? (And on that note, why were apparently a random assortment or analgesics used? And if that is the case, does each type of analgesic constitute a separate treatment group?).

Look, all of this is speculation because in the absence of 1) the paper or 2) the correct controls, this study is meaningless.

Which leads me back to why it’s been announced in the press and not in the pages of peer review. I suspect it might not pass peer review without the sham controls. Also odd is that the research data Australia website says the trial began in 2008, and was due for completion in 2011. So why have the results not been published yet? Is it because they likely won’t be?

This would be a very big worry for the researchers, especially given that the NHMRC gave these guys $416,301. Are they announcing it in the press in an attempt to distract the NHMRC?

Even more of a worry though is the researchers apparent disregard for the Australian Code for the Responsible Conduct of Research. Section 4.12.1 says, “Discussing research findings in the public arena should not occur until the findings have been tested through peer review.”

By all means, study the efficacy of TCM, but at least do it with correctly designed experiments and try not to waste $400K of tax payers’ money (as if research doesn’t have enough money to go around). Until such time as that is done, approach me in A&E with an acupuncture needle, you’ll get a slap upside the head.

EDIT: A friend just suggested this on Facebook and I want to add it because it’s an important point that I should have included.

Wendy Wilkinson: “Id love to see the methods used for this research. If it ever gets published properly, that is. How did they recruit the participants? How were they randomised into the three groups? In busy EDs where this sort of thing is not an emergency. And most importantly – how long did they wait for the treatment? Because if they had been sitting in the waiting room for longer than that 1 hr how have they accounted for regression to the mean as far as pain goes.

Yep, regression to the mean may have also played a role in this study. VIP point.

UPDATE Monday March 31: Another issue with this study not being blinded is it may have selected for patients who are sympathetic to the efficacy of acupuncture. So if you rock up to A&E and someone says, “do you want to be in our acupuncture trial?”, if you’re me you’ll say “%^&^%&^ get me some drugs with a demonstrated mechanism of action please!!”. But those who think there’s something to acupuncture will more likely say yes. This then influences their pain perception – if you believe it, then it works (ie placebo).

Also, we must consider the role of “wanting to please the nice Dr”. Patients may report a reduction in pain (later found to be equivalent to the analgesics, well, ok, they haven’t finished analysing the data yet so who actually knows) because they have been fussed over and maybe taken to a private room to lie down and relax etc.

All of these things can introduce confounding variables into studies, which is why controls are critical. The appropriate controls were not used here. This study is worthless.

Monday March 31st. 11:51: Looks like there might be a few updates on this story.

Firstly, I want to clarify something re: 4.12 Responsibly communicating research findings in the public arena. I did not include this section of 4.12.1 (an oversight, not deliberate), “In discussing the outcomes of a research project, special care should be taken to explain the status of the project – for example, whether it is still in progress or has been finalised”.

So the researchers did make this clear – that being the results are still being analysed. Thus it might be fairer to say them releasing this is bad practice rather than a breach of the Australian Code for the Responsible Conduct of Research.

Further, I asked the Society and Science editor at The Age (where this was published) whether they had received this as a press release. He’s not answered this question yet.

Subscribe to comments Comment | Trackback |
Post Tags: , , , ,

Browse Timeline

  • Pingback: A skeptical look at a study of acupuncture delivered in emergency rooms | Quick Thoughts()

  • Absolutely agree.

  • Agreed Sam, and mentioned here “By all means, study the efficacy of TCM, but at least do it with correctly designed experiments and try not to waste $400K of tax payers’ money (as if research doesn’t have enough money to go around).”

  • Sam

    The trial you linked to states it received NHMRC funding – should we also be focusing on how NHMRC money is being used to fund an acupuncture study without a proper sham acupuncture control group so that the patient is blinded? The news story is bad, but from the perspective of a non-scientist journalist it appears to be a big government-funded trial in hospitals and therefore credible – surely the worse part is scarce NHMRC funding being diverted to fairly pseudoscientific research in the first place?

  • Pingback: bluzki wieczorowe()

  • DrRachie

    Thanks Volk.

  • Volk713

    If it is the same as this trial then it was only assessor blinded http://www.trialsjournal.com/content/12/1/241

  • DrRachie

    I’d like to see the actual study Wendy, not just a press release!

  • DrRachie

    I think the headline should be “scientists do dodgy study, can’t get it published because fail, get some media coverage in an attempt to cover arse” or something.

    Also, I’m surprised at Julia Medew who wrote this. She normally does good stuff. I’m surprised she didn’t stop at “data still being analysed for publication”. They got her a good one too.

  • Andy

    …and how would it be cheaper to have (presumably) some sort of specialist sit with you and carefully insert needles into specified locations, as opposed to a nurse giving you a pill in a cup?

    Do acupuncturists, like all other alt-med practitioners, work for free? Is their treatment complimentary as well as complementary?

  • Andy

    Did they also use placebo pills – or would that be unnecessary in order to test acupuncture in this way?

    Maybe the headline from this experiment should have been:
    “Gullible media untrustworthy, scientist shows”

  • Wendy Wilkinson

    I’d really like to see the participation consent form. Out of professional interest.

  • Wendy Wilkinson

    I’m also wondering how this was sold to the EDs that participated. And the hospital ethics committees.

  • Ilijasx

    Shameful. That $400K+ could have been put to much better use than this.

  • DrRachie

    Good point John. All together, seems like a waste of 400k to me

  • John Carney

    Another problem is that participants were almost certainly told the nature of the treatments being tested when asked whether they wanted to take part or not. This means that the cohort would have been heavily skewed towards people who believe acupuncture works.