Last week saw the publication of a new study in the BJSM entitled (highlighting mine), “Is interval training the magic bullet for fat loss? A systematic review and meta-analysis comparing moderate-intensity continuous training with high-intensity interval training (HIIT)“. Understandably intrigued given a prominent medical journal was suggesting there was a magic bullet for fat loss, I clicked through, and then reading the piece I learned that the amount of fat lost that the BJSM was calling a “magic bullet” was a 1 pound difference, one which the study’s abstract’s conclusion described as, “a 28.5% greater reductions in total absolute fat mass (kg)”. Duly surprised, I then took to Twitter to poke around and found that one of the study’s authors, James Steele, was tweeting out a corrective thread to his own study’s hype – hype which understandably and predictably led to an onslaught of media overreach. Intrigued, I approached him directly to ask about the discordance in tone between his tweets and his study’s title and conclusion, and he sent me such a thorough and thoughtful response (explaining how it was the BJSM’s editor who’d changed both), that I asked him if he’d mind my sharing his thoughts here as a guest post. Suffice to say, in my opinion, medical journals and their editors shouldn’t be in the business of clickbait hype, as it diminishes themselves, research, and furthers societal scientific illiteracy by suggesting that such things as “magic bullets” for weight or fat loss can conceivably exist.
I was first slightly concerned that the findings would be overhyped and potentially misrepresented when I saw the press release that was sent to the media. I was forwarded various requests by our institutions news team and saw the wording of the first line of which was
“Short bursts of high intensity exercise are better for weight loss than longer sessions in the gym, research suggests.”
My colleague James Fisher noted to me that he also thought the press release didn’t reflect the findings accurately and wondered whether the title change resulted in the perception of a different finding.
The original title in our submission to the journal was
“Comparing the effects of interval training versus moderate-intensity continuous training on body adiposity: is it possible to find a signal in the noise? A systematic review and meta-analysis“
which was chosen as an homage to Nate Silvers’ book and the use of meta-analysis to find the ‘signal’ from among the ‘noise’ of conflicting findings in smaller studies The paper underwent peer review as normal and we made changes suggested by the reviewers to improve the manuscript; but, none of the reviewers commented on the title if I recall. After the reviewers were happy with the paper and had no further changes they wanted we received a recommendation that it be published, but with minor revisions which were suggested by the editor. Most of the revisions suggested where helpful as they seemed to be aimed at improving readability of the manuscript. However, it was also suggested that the title was changed, as well as the addition of the percentage difference to the conclusion of the abstract. This was suggested to be intended to attract more attention to the article, make it seem more compelling, and ensure recognition was received for the work. I didn’t particularly like the newly suggested title, nor did some of my co-authors, but it was not strictly saying that anything ‘was’ a ‘magic bullet’ and so I did not push the issue. I must confess I did not at the time notice the seemingly minor change to the abstract conclusion though. I personally dislike the presentation of % values in this manner as to me they are often misleading and detract from whether the absolute values are really meaningful or not (a big problem in sport and exercise IMO wherein a lot of studies make interventions seem better than they are by reporting % values). The value is not inaccurate, but it does lead the less wary reader to potentially draw the wrong conclusions.
I did suspect that the changes were suggested because the paper would likely be selected for a press release which turned out to be correct. I’m glad the paper got some wide coverage, but wanted to make sure it was covered in a nuanced manner. So I tweeted a little thread to try and provide some balance and when I was interviewed about it on BBC World Service I also made sure to provide as balanced a commentary as I could in the time permitted.
It doesn’t surprise me that the media initially interpreted things to be saying that ‘HIIT’ (high-intensity interval training) was better than ‘MOD’ (moderate-intensity continuous training) for fat loss without considering all the nuance… that’s just how it goes sadly. I also can empathize with the journal and publisher in wanting to try and increase the reach of the work that they publish. To my mind if we can widen the reach of good science, and raise appreciation of its importance, then that’s a good thing. This is something I’d like to be able to do more of. But, though this is good in principle, in execution it proves to be difficult. It’s tough to get the nuance across because science is hard and most people aren’t really able to understand it. I guess it’s part of the media cycle though. The wider media wants ‘stories’ and just regular boring old science doesn’t make for a good story. So to get the wider media’s attention journals and academic publishers need to try and make things seem more exciting. In that process though nuance gets lost. However, I can’t think of any other way to communicate science more widely at the moment. I guess what we need to ensure is that, once the media get hold of a story and want to run it, the actual scientists themselves are the ones they speak to and interview so they end up with a platform and captive audience to explain the nuance and implications in an understandable manner. At least, that’s what I’ve tried to do and hope I achieved.
I think if I was able to ‘do over’ this example specifically then I would have likely pushed back more on the issues. I would like to have kept the original title and would have argued for this position as I suspect my co-authors likely would have too. I definitely would have pushed back on the change to the abstract conclusion and will be more vigilant to these issues in future. In likelihood this might have meant the paper would have been less ‘impactful’ as a story for the media. But it would have meant that the paper itself didn’t contribute to any potentially misleading publicity. The publisher could have still put out the press release as they desired… Can’t stop them from doing that. But at least the paper would have better reflected what we found in full. I think I would advise authors who face similar situations to make sure they think and have a conversation about this. We all want our work to reach the widest audience to hopefully have the biggest impact possible. But we don’t in the process want it to distort in terms of its message. Make sure to discuss it with your co-authors and the journal/publisher and find the right balance so that scientific integrity is retained, whilst reach is maximized. It’s tough to do, but worth striving for IMO.
Dr James Steele is the Principal Investigator at the ukactive Research Institute, and Associate Professor of Sport and Exercise Science at Solent University. James completed his BSc (Hons) in Applied Sport Science in 2010, and hid PhD examining the role of lumbar extensor resistance training in chronic low back pain in 2014. He has extensive experience of research and applied consultancy in the area of physical activity, exercise, and sport from over the past decade, working with a wide range of populations ranging from elite athletes across a range of sports, to the general population across the lifespan, and both those who are healthy and diseased. James has published numerous peer-reviewed articles and delivered several invited talks at international conferences on a variety of areas relating to sport, strength and conditioning, physical activity and exercise, health and fitness. He was appointed to the Expert Working Group revising the Chief Medical Officers Physical Activity Guidelines for the UK and is a Founding Member of the Strength and Conditioning Society, as well as the Society for Transparency, Openness, and Replication in Kinesiology, and member of both the British Association of Sport and Exercise Sciences and the American College of Sports Medicine.