Publication Bias in Hypnosis Research

Science has a file drawer problem. Across disciplines, studies with positive results are more likely to be published than studies that find nothing. In hypnosis research, this is a particularly important issue.

What publication bias is

Publication bias occurs when the likelihood of a study being published depends on its results [1]. Studies that find a treatment works are more interesting, more publishable and more rewarding for the researchers who conducted them. Studies that find no effect are often shelved, unpublished but not non-existent.

An empirical study that tracked pre-registered studies through to publication found that studies with strong, positive results were 40 percentage points more likely to be published than those with null findings [4]. This is not a subtle effect.

The consequence is that the published literature gives a systematically optimistic view of treatments. When a systematic review or meta-analysis searches only published studies, it is searching a biased sample.

Why hypnosis research is especially vulnerable

Several features of hypnosis research increase the risk of publication bias:

• Small studies. Most hypnosis trials have fewer than 50 participants. Small studies have more variable results, and when the variable results that happen to be positive are published preferentially, the distortion is larger.
• Researcher investment. Many hypnosis researchers are also practitioners with a professional interest in positive results. This is true in many fields, but it is worth noting.
• Few pre-registered protocols. Pre-registration, where the study plan is recorded before data collection, is increasing but was rare in older hypnosis research. Without pre-registration, it is impossible to know how many studies were planned but never published.
• Flexible analyses. With multiple outcome measures and analytic choices, researchers can (often unconsciously) select the analysis that gives the best result.

How to detect it

Funnel plots

A funnel plot graphs each study’s effect size against its precision (usually sample size or standard error). In the absence of bias, the plot should look like a symmetrical inverted funnel: large studies cluster near the average effect, and smaller studies scatter more widely but symmetrically. If small positive studies are over-represented, the funnel is asymmetric [2].

Several hypnosis meta-analyses have examined funnel plots. The surgical hypnosis meta-analysis noted funnel plot asymmetry for some outcomes [6]. The laboratory pain meta-analysis included a publication bias assessment [5].

Statistical tests

Tests such as Egger’s regression test formalize the funnel plot assessment. A significant result suggests asymmetry, though it can also be caused by genuine differences between large and small studies.

Trim and fill

This method estimates the number of “missing” studies needed to make the funnel symmetric and recalculates the pooled effect. The adjusted effect is usually smaller, sometimes substantially.

What this means for the evidence

If publication bias inflates effect sizes in the hypnosis literature, then the true effectiveness of hypnosis for most conditions is probably somewhat smaller than the published meta-analyses suggest. How much smaller is difficult to know without access to the unpublished studies.

This does not mean hypnosis does not work. It means that the strength of the evidence is probably weaker than it appears. An effect that looks moderate in a meta-analysis might be small after correcting for bias, and an effect that looks small might disappear.

Situation	Likely impact of publication bias
Many small positive studies, few null	Probably overestimates the effect
One or two large well-designed trials	Less affected
Meta-analysis with funnel asymmetry	Adjusted effect is likely smaller
Pre-registered studies with published null results	More trustworthy
Topic with strong researcher advocacy	Higher risk of bias

What would help

1. Pre-registration. Registering protocols on platforms like ClinicalTrials.gov or Open Science Framework before data collection.
2. Publishing null results. Journals that accept well-designed negative studies.
3. Registered reports. A format where journals accept or reject a study based on the protocol, before results are known.
4. Reporting all outcomes. Not just the ones that reached significance.
5. Independent replication. Studies conducted by researchers without a stake in the outcome.

What this means for reading research

When evaluating a claim about hypnosis effectiveness, consider not just the published evidence but the evidence that might be missing. If a meta-analysis includes only small positive studies and shows funnel plot asymmetry, the true effect is likely smaller. If it includes pre-registered studies and reports null results, the estimate is more trustworthy.

For how systematic reviews handle this issue, see What Is a Systematic Review?. For the broader challenges of studying hypnosis, see Blinding Problems in Hypnosis Trials.

A note on intellectual honesty. Acknowledging publication bias is not an attack on hypnosis research. It is a standard scientific concern that applies to every field. Researchers who address it directly in their work are doing better science, not undermining their own results.