What Is Publication Bias? | Definition & Examples

Publication bias refers to the selective publication of research studies based on their results. Here, studies with positive findings are more likely to be published than studies with negative findings.

Positive findings are also likely to be published quicker than negative ones. As a consequence, bias is introduced: results from published studies differ systematically from results of unpublished studies.

Publication bias can affect any scientific field, leading to a biased understanding of the research topic.

What is publication bias?

Publication bias occurs when a study is published in a peer-reviewed journal based on the direction and strength of its results, rather than on individual merits or other factors

Researchers often formulate hypotheses to test an assumption regarding a population. They transform the research question into claims, the null and alternative hypotheses. The null hypothesis claims there is no effect in the population, while the alternative hypothesis claims the opposite.

• When researchers run statistical tests and find no effects, we say that these studies fail to reject the null hypothesis and the alternative hypothesis is not supported. Alternatively, these can be called negative studies.
• When they do find evidence of effects, we say that these studies reject the null hypothesis and thus the alternative hypothesis is supported. These are called positive studies because they have found evidence of a relationship, difference, or effect between variables. For example, researchers may find differences between experimental and control groups.

Because the academic community tends to view positive studies more favorably than negative ones, these are more likely to be published.

This means that a study’s findings determine whether it will be published, rather than the study design, relevance of the research question, or overall quality.

What causes publication bias?

There are a number of factors that can cause publication bias:

1. Researchers often do not submit their negative findings because they feel their research has “failed,” or that it’s not interesting enough.
2. In some cases, researchers may suppress negative results from clinical trials for fear of losing their funding. This can occur, for example, when for-profit companies sponsor medical research.
3. Researchers are themselves aware of publication bias. They know that if they submit positive results, they are more likely to see their work published in prestigious journals. This, in turn, can increase their reputation among their peers, the number of citations their articles generate, their chances of getting a grant, etc. This could lead them to not even submit other results.
4. The financial status of academic journals also depends on the number and frequency of citations that published studies generate. These are an indication of how much a journal is noticed or respected. Because studies with negative findings are less likely to be cited than studies with positive findings, it’s more attractive for journals to publish positive findings.

In other words, both researchers and editors introduce research bias into the process of determining which results are worth publishing.

Why is publication bias a problem?

Publication bias can cause problems in your research for a number of reasons:

• It increases the likelihood that published results reflect Type I errors. These bias effect sizes upwards and suggest stronger effects on future studies that may actually be due to chance. For example, this can lead to overestimation of the effectiveness of a new drug.
• Researchers may be wasting effort and resources conducting studies that have already been done but not published because the treatment or intervention didn’t prove to be effective.
• It affects the quality of literature reviews. A literature review that is limited to published studies is highly selective and may result in overestimated effects.
• Failure to publish null results because they “did not work” limits our ability to thoroughly understand all aspects of a scientific topic being studied. Even if strong results signify effective treatments or interventions, failure to publish null results means that a large portion of the topic remains hidden or unknown.
• It causes published studies to no longer be a representative sample of available knowledge. This bias can distort the results of systematic reviews using meta-analyses or statistical analyses combining results from multiple studies focused on the same topic. When not accounted for, publication bias compromises findings.

• It may lead some researchers to manipulate their results to ensure statistically significant results. One example of this is resorting to data dredging, or running statistical tests on a set of data until something statistically significant happens.

Publication bias example

Researchers also have biases regarding which results they consider worthy of publication. This is also sometimes known as the “file drawer problem.”

Ultimately, publication bias is especially damaging because it is compounded by the perception that journals have biases about which results are worth publishing. This in turn leads many researchers not to even submit their research projects for consideration.

How to avoid publication bias

Although individuals cannot really address publication bias on their own, there are preventive steps you can take. These include:

• Using registered reports. This is a form of journal article in which researchers may submit the first half of their paper—describing the hypothesis, planned research methods, and expected statistical power—rather than the whole thing at once. If found suitable, the journal provisionally accepts the article. If the researcher sticks to the reported plan, the results are then published regardless of the outcome. In this way, research papers can be evaluated on the quality of their methodology and the importance of the research question, not on the outcomes.
• Comparing the results of published and unpublished papers on the same research topic. By comparing results, you can establish whether there is bias towards positive results in that field of study. To do so, you can search through clinical trial registries and conference proceedings, or even contact researchers who haven’t published their results yet.

• Consulting journals that publish negative findings, such as Positively Negative (PLOS One), Journal of Articles in Support of the Null Hypothesis, or Journal of Pharmaceutical Negative Results.
• When conducting a systematic review, researchers often assess publication bias using Egger’s linear regression test, visualizing it using funnel plots. Here, a funnel plot is a scatterplot showing the relation between a study’s effect size and its precision. However, keep in mind that this visual inspection alone is not enough to establish publication bias.

Other types of research bias

Frequently asked questions

Why is there a publication bias against null effects?

Study results with null effects indicate that the result does not support the hypothesis. Researchers often consider these types of results unexciting or a sign of failure.

Journals also are more inclined to publish research with positive findings. Because both researchers and journals are biased against studies showing null effects, publication bias occurs.

How does publication bias affect research?

Publication bias affects research because it emphasizes results that do not represent the overall universe of evidence.

Because publication bias occurs when researchers withhold negative study results from publication, it’s implied that the visibility and potential impact of these studies is not worthy of publication. This can lead other researchers to pursue research that is redundant, misguided, or even a waste of resources.

How does a funnel plot measure publication bias?

A funnel plot shows the relation between a study’s effect size and its precision. It is a scatter plot of the treatment effects estimated from individual studies (horizontal axis) against sample size (vertical axis).

Asymmetry in the funnel plot, measured using regression analysis, is an indication of publication bias. In the absence of bias, results from small studies will scatter widely at the bottom of the graph, with the spread narrowing among larger studies.

The idea here is that small studies are more likely to remain unpublished if their results are nonsignificant or unfavorable, whereas larger studies get published regardless. This leads to asymmetry in the funnel plot.

What is the file drawer problem?

The file drawer problem (or publication bias) refers to the selective reporting of scientific findings. It describes the tendency of researchers to publish positive results much more readily than negative results, which “end up in the researcher’s drawer.”

What is data dredging?

Data dredging (also called p-hacking) is the statistical manipulation of data in order to find patterns which can be presented as statistically significant, when in reality there is no underlying effect.

This can be achieved in a number of ways, such as:

• Excluding certain participants
• Stopping data collection once a p value of 0.05 is reached
• Analyzing many outcomes, but only reporting those with p < 0.05

The reason for this practice is the widespread notion in the academic community that only statistically significant findings are noteworthy. This idea leads to publication bias.