‘But I’m a clinician! Why do I need to know about publication bias, isn’t that more for researchers?’.

If this was along the train of your first thoughts when you first saw this blog advertised then I am sorry to inform you, that publication bias should be at the forefront of clinicians’ minds when it comes to interpreting research and translating it in to practice. It is interfering with the ‘evidence-based practice’ ideal, BIG TIME! But well done for being intrigued enough to make it here to learn a bit more about it!

Evidence based practice for clinicians in an ideal world should be not the near impossible pursuit it is today. We are all aware of the hierarchy of evidence, with systematic reviews and meta-analyses sitting at the top with all of the collated, high-quality research sat in one neat paper with clear data and conclusions presented in an easy to read fashion. I, as a clinician, should be able to sit down with a cup of black coffee during a 30 minute period in work and read one paper to gain a reasonable insight into the current evidence base surrounding my topic of interest.

Alas, this is something that I cannot do if I actually want to be ‘evidence based’. Due to a myriad of issues that have resulted in an overwhelming amount of research (a ridiculous amount to be fair) being published, certainly not always of the highest quality and maybe not always asking the valuable questions that need to be asked. But perhaps one of the biggest issues with the evidence based is that the evidence base itself is essentially controlled by professional publishing companies that deem an article either suitable or not-suitable for their *prestigious* journals. This control over what gets published and what doesn’t gives the professional publishing companies an enormous influence over what we get to read and ‘know’, and an inordinate amount of power over researchers themselves.

THIS STUDY done in 2013 demonstrated that just 5 professional publishing companies now account for approximately 50% of ALL scientific papers published. That’s actually quite insane when you think about it!

Do these companies oversee the dissemination of knowledge pro bono, out of the goodness of their heart? Ha, no chance!

Information and knowledge has become monetised, people (clinicians for example) provide the demand for this and therefore there is money to be made! This is done in a multitude of ways:

• Instilling paywalls; having individuals, or institutions pay for access to the article

• Charging fees for open access; want your paper to be free to view for the public and therefore reach more people and have an impact? Well, you’re gonna have to cough up for that privilege! Recent figures suggest this could be $1595 for PLOSone or up to $5000 for the Lancet!!

• Charging less for open access but publishing literally any quality research; these are known as predatory publishers who don’t really care about advancement of any knowledge… just how much money they can get for as many papers as they publish.

You might think I am being quite pessimistic and harsh, and maybe I am, I am sure that the finance people and editors of journals do have good intentions about sharing valuable information on the whole… but they are a huge part of what seems like a broken system and it only ends up negatively impacting upon clinicians and consequently, patients.

Why can the Lancet charge $5000 for open access which is way more than other journals? Why would people pay that?

There are levels to the game and the Lancet (and it's family of journals) sits near the top when it comes to medical journals. This is measured in part by the ‘impact factor’ which is essentially the average number of citations for articles within the journal over the last 2 years. The higher the impact factor, the more ‘valuable’ the journal is to the advancement of knowledge.

Here lies part of the problem. Of course, if you are an editor of a journal, you want it to be a bloody good journal. One that people will pay to subscribe to and that will deliver the latest, highest quality information reliably with every issue.

What is one thing that journals think people don’t want to read? … Findings that fail to reject the null hypothesis or in other words, negative findings. These are FAR LESS LIKELY TO GET PUBLISHED. They aren’t sexy, people aren’t going to be talking about it or clapping over the results at a conference. But does this make them any less important to know? Of course not! In fact, the knowledge that something doesn’t ‘work’ or that there aren’t relationships between variables is actually pretty damn useful for us as clinicians! Take THIS STUDY for example, whilst it is not MSK rehabilitation it is representative of the problem that publication bias presents.

This certain type of anti-depressant medication, when explored in the literature had an overwhelmingly positive profile. 94% of trials demonstrated positive results. Take a moment to think what this would mean to you if you were looking to treat someone with depression. 94% positive findings, meta-analyses showing great treatment effects… it would be almost neglectful to not go down this route to treat them, right?!

That’s because the medical journals want to give the readers something great to read and impact positively on their practice. But in doing so probably reject the research that didn’t demonstrate a clinically positive result…

So, are we getting the whole picture or just what the people who have the power think we want to see?

When the literature on the anti-depressants was compared to the results of all trials on the FDA database, the percentage of positive results dropped dramatically to 51%!! Turner et al demonstrated further that there was a 32% increase in effect size when you look just at the published literature!! Depending on the treatment publication bias effects, this could be the difference between a large effect and a small effect size!

One of the biggest problems I think we have as clinicians, is that we will never know just exactly what has been rejected by the journal or what has not even been written up by researchers as the time spent writing up negative findings is just not worth it! It is a huge problem, and personally, it is a constant thought whenever I go to look at a meta-analysis.

What if I am just getting half the picture?

Thankfully, there are some partial ‘solutions’ or methods to evaluate if publication bias is present. One of these is a funnel plot. A funnel plot (pictured below) is a visual representation of studies included within a meta-analysis; the vertical black line demonstrates the average result and the black dots represent the individual study results. Publication bias is present when we see that there isn’t an even spread. Usually positive results will be included even when they lack strong statistical power, but negative results with strong statistical power are a struggle to publish, so the negative results with less statistical power have no hope in hell of making it into a journal!

There are also other methods such as egger equations which can calculate the presence of publication bias. Don’t fret about the specifics of either of these; researchers who look at this are usually pretty good at describing it in the text.

The difficulty with this appears to be getting researchers to include these measures in the first place!! I have found it to be very rare to actually get these measures included in the meta-analysis and taken into consideration with the results.

If we delve into the realm of psychology for a minute (psychology papers have highlighted a lot of issues within the literature base and is way ahead of us in physio / MSK); Shank et al went to conduct a meta-analysis of priming in risky behaviours, only, when they conducted the review they found a surprising lack of negative results published, have a look at the forest plot below.

See how all of the average effects sit on the right of the middle line implying a positive effect? Is it really likely that there is no conflicting data AT ALL?! Or is it more likely that the opposing data just hasn’t been written up for fear of not getting published… or just got rejected by the journal?

So here ends the start of the first section of this blog ranting about how publishers essentially dictate what we do and don’t see, and how this can impact on our knowledge.

And just to add, this is happening in our world of MSK rehab / sports medicine and orthopaedics. Buttner et al conducted a review of 129 journals in 2019 from fairly high ranking journals of British Journal of Sports Medicine, JOSPT and American Journal of Sports Medicine and found that 82.2% of original research papers had positive results…

Just how likely do you think it is that this is a representative figure of true results in our realm?

Onto the second problem; researchers kinda sorta need to publish in order to thrive and have a successful career! Science is a social construct amongst many other things. If you can’t communicate what you are doing to other researchers in your space then you don’t get recognised, you don’t get picked for the larger grants and your employer probably won’t be that happy with you if your job is to increase the research profile of the institution!

That makes non-significant findings a bit of a problem for researchers. Research itself takes A LOT of time, effort and skill so when all of that potentially goes to waste because a journal won’t publish then nobody wins.

The file-drawer effect is the name for what happens when researchers don’t bother with writing up their null results and it happens A LOT. In the social sciences it has been demonstrated that strong results are 60% more likely to get written up than null results! When questioned further, researchers gave different reasons for not writing up;

“I think this is an interesting null finding, but given the discipline’s strong preference for P < 0.05, I haven’t moved forward with it”

“There was no paper unfortunately. There still may be in future. The findings were pretty inconclusive”

The alternative, potentially more problematic issue, is when data gets a bit fudged in order to create a positive result (P-Hacking) or the researcher finds a significant relationship between variables and then creates a narrative around this to imply importance, when actually the hypothesis should be decided before even collecting any data. The latter of the two examples here is called HARKing which stands for ‘Hypothesis After Results are Known’.

This blog isn’t intended to go into the specifics of what these are and how they happen but whilst HARKing is probably self explanatory, p-hacking probably isn’t so here is a brief overview…

Imagine you are a researcher, you have a theory that a particular intervention might be beneficial for a certain patient group. You devote a huge amount of time to investigating the effects of this treatment, perhaps a company or institution have provided funding because they share your hypothesis. It comes down to crunching the numbers and… f**k, it hasn’t reached statistical significance.

But it’s close! Maybe if you added a few more participants it would reach it and you could prove that this intervention is worthwhile!

So to put this in an example, say you collected 20 participants and your P-Value was at 0.08 (close to 0.05) with a trend towards it becoming more significant. You add 3 more participants one at a time until you reach the magic 0.05 number which reaches statistical significance! SUCCESS! Right? Well, maybe not.

Are you just arbitrarily stopping at 23 participants? What if another participant has a negative response which would take it back below the significance threshold making the research suddenly support the null hypothesis?

This is an example of p-hacking. It might seem innocuous enough, it might even make sense to some people, I must admit I didn’t get it straight away. But this is actually manipulating data to try and reach a statistically significant result.

Is this happening in the MSK world? Well this paper HERE demonstrated statistical values that were JUST INSIDE the level of statistical significance were overwhelmingly more frequently reported than stronger relationships or weaker relationships… does this seem suspiscious to you bearing in mind everything else we have talked about so far?

What is the impact?

So what is the harm in questionable research practices? Hopefully this should be fairly obvious in the fact that we could be basing treatments off of invalid data which again doesn’t help anyone except the researcher who may have increased their profile.

Coming back to the psychology sphere, there was a huge paper published in 2015 from the ‘Open Science Collaborative’ which exactly replicated 100 studies that had been published.

Before going any further let’s just take a moment to appreciate the magnitude of that… 100 studies done in the exact same way, over the exact same time periods with the exact same methods!

With the original papers, 97% had reported statistically significant results, but when they were replicated only 39% of results were reproducible!!! Just imagine how much of a big hit that would have been to the scientific knowledge at the time. How much time and money has been spent building on research results that couldn’t be reproduced?

Camerer et al (2018) looked at articles published in very high impact journals of ‘Science’ and ‘Nature’ between 2010 to 2015 and replicated 21 studies. When they compared the results of the previous studies to the newly published studies it was found that 62% of the studies had similar, positive results; but the effect sizes were only 50% of those that were first reported! Of course, this could in part reflect the need to replicate studies regularly anyway, we shouldn’t just be relying on studies done once, that are often exploratory research to base a lot of our reasoning on. But with all the background surrounding publication bias, and the fact effect sizes were only half of the originally reported studies, my guess this is questionable research practices!

Questionable research practices, including P-Hacking and HARKing amoungst others were studied HERE by John et al in 2012 in the psychology sphere. Please see the picture below for more details and just look at how common these practices are!

Whilst we haven’t had this magnitude of research (yet) in our MSK world, I struggle to imagine a scenario that is hugely different and more positive. Constructs that are measured in psychology are messy and subjective… sound like anything that we deal with? Maybe pain, self reported function or kinesiophobia?!

So we don’t know the magnitude of the problem for us, and just how much of our research we shouldn’t trust. That’s a bummer. But there are certain things that can be done to increase our confidence in results. The first of these is called pre-trial registration.

It is now increasingly becoming a requirement for journals that researchers publish their intentions, methodology and outcome measures on a website BEFORE collecting any data. By doing this we have transparency on the intentions of the researcher from the outset, and any manipulation of data through methods of HARKing or sample size manipulation should be pretty obvious to us.

One of the things that really gets to me, in a way that it shouldn’t, is when papers are ‘pre-registered’ AFTER the study has actually ended! I mean seriously what is the point? It doesn’t fill me with confidence in results and potentially swings me in the opposite direction! Make sure you actually check the pre-registration page for details of when it was registered because I have found issues with this multiple times!

Also, a recent study HERE found prospectively registered trials in the physiotherapy sphere are likely really not that common (in this study only 22% were prospectively registered).

So pre-registration is a good step forwards… but there are still ways around it. If your data comes back not significant and therefore not publishable, then there are still ways to manipulate data if you really want to. Not suggesting this is the default for researchers… but I’m sure it is how some operate and you would never know!

What then? In comes the ‘Registered Report’

The process of a registered report actually makes a lot of sense. Researchers can submit their research methodology and reasoning to a journal BEFORE they collect any data, at this stage it undergoes a peer review process so that any changes that could potentially make the study stronger can be implemented. Once this process has been agreed, the journal will agree to publish the results regardless of the outcome because, well, the scientific methodology was sound and the results are important regardless of if they demonstrate statistical or clinical significance!

This is relatively new in MSK and I personally haven’t seen any examples of this yet, with only a couple of journals offering it. But there is a trend of registered reports seemingly improving the state of things within psychology literature.

Remember about 5 minutes ago when I mentioned Buttner et al found an implausibly high number of positive results reported in the MSK space? Well Scheel et al (2021) found very similar trends of 96% of results being positive in the published literature for their particular field! Again this is extremely unlikely and isn’t helpful for consumers of research. When they compared the number of positive results reported in the registered reports literature the number of positive results dropped to 44%!

44% is a much more realistic representation of true ‘positive’ results and when we can get to this stage in the MSK space, we will be able to trust our research base a lot more and maybe the pursuit of being ‘evidence based’ will become a more realistic goal!

Wrapping it up

That was a lot, let me summarise.

Publishing is not an easy process. It is biased by a overwhelming preference for ‘positive’ results that influence researchers massively on whether they will bother to write up a study or if they will adjust their methods in order to achieve more ‘interesting’ results in the eyes of publishers. This does not serve us well as clinicians as the results we see are not the entire picture and ‘negative’ and non-significant relationships still hold great clinical value to us.

Let’s keep a healthy dose of scepticism about everything and not tie our identities or beliefs too strongly to certain explanations or treatment options. Not until we have a more honest research base, once registered reports come into play a bit more!

Thanks for reading and as always reach out if you have any queries or comments!

Jeff