Common Biases in Scientific Research
Cognitive Biases Affecting Researchers
1. Confirmation Bias
Description: Tendency to search for, interpret, and recall information that confirms preexisting beliefs.
Manifestations: - Designing studies that can only support the hypothesis - Interpreting ambiguous results as supportive - Remembering hits and forgetting misses - Selectively citing literature that agrees
Mitigation: - Preregister hypotheses and analysis plans - Actively seek disconfirming evidence - Use blinded data analysis - Consider alternative hypotheses
2. Hindsight Bias (I-Knew-It-All-Along Effect)
Description: After an event, people perceive it as having been more predictable than it actually was.
Manifestations: - HARKing (Hypothesizing After Results are Known) - Claiming predictions that weren't made - Underestimating surprise at results
Mitigation: - Document predictions before data collection - Preregister studies - Distinguish exploratory from confirmatory analyses
3. Publication Bias (File Drawer Problem)
Description: Positive/significant results are more likely to be published than negative/null results.
Manifestations: - Literature appears to support effects that don't exist - Overestimation of effect sizes - Inability to estimate true effects from published literature
Mitigation: - Publish null results - Use preregistration and registered reports - Conduct systematic reviews with grey literature - Check for funnel plot asymmetry in meta-analyses
4. Anchoring Bias
Description: Over-reliance on the first piece of information encountered.
Manifestations: - Initial hypotheses unduly influence interpretation - First studies in a field set expectations - Pilot data biases main study interpretation
Mitigation: - Consider multiple initial hypotheses - Evaluate evidence independently - Use structured decision-making
5. Availability Heuristic
Description: Overestimating likelihood of events based on how easily examples come to mind.
Manifestations: - Overemphasizing recent or dramatic findings - Neglecting base rates - Anecdotal evidence overshadowing statistics
Mitigation: - Consult systematic reviews, not memorable papers - Consider base rates explicitly - Use statistical thinking, not intuition
6. Bandwagon Effect
Description: Adopting beliefs because many others hold them.
Manifestations: - Following research trends without critical evaluation - Citing widely-cited papers without reading - Accepting "textbook knowledge" uncritically
Mitigation: - Evaluate evidence independently - Read original sources - Question assumptions
7. Belief Perseverance
Description: Maintaining beliefs even after evidence disproving them.
Manifestations: - Defending theories despite contradictory evidence - Finding ad hoc explanations for discrepant results - Dismissing replication failures
Mitigation: - Explicitly consider what evidence would change your mind - Update beliefs based on evidence - Distinguish between theories and ego
8. Outcome Bias
Description: Judging decisions based on outcomes rather than the quality of the decision at the time.
Manifestations: - Valuing lucky guesses over sound methodology - Dismissing good studies with null results - Rewarding sensational findings over rigorous methods
Mitigation: - Evaluate methodology independently of results - Value rigor and transparency - Recognize role of chance
Experimental and Methodological Biases
9. Selection Bias
Description: Systematic differences between those selected for study and those not selected.
Types: - Sampling bias: Non-random sample - Attrition bias: Systematic dropout - Volunteer bias: Self-selected participants differ - Berkson's bias: Hospital patients differ from general population - Survivorship bias: Only examining "survivors"
Detection: - Compare characteristics of participants vs. target population - Analyze dropout patterns - Consider who is missing from the sample
Mitigation: - Random sampling - Track and analyze non-responders - Use strategies to minimize dropout - Report participant flow diagrams
10. Observer Bias (Detection Bias)
Description: Researchers' expectations influence observations or measurements.
Manifestations: - Measuring outcomes differently across groups - Interpreting ambiguous results based on group assignment - Unconsciously cueing participants
Mitigation: - Blinding of observers/assessors - Objective, automated measurements - Standardized protocols - Inter-rater reliability checks
11. Performance Bias
Description: Systematic differences in care provided to comparison groups.
Manifestations: - Treating experimental group differently - Providing additional attention to one group - Differential adherence to protocols
Mitigation: - Standardize all procedures - Blind participants and providers - Use placebo controls - Monitor protocol adherence
12. Measurement Bias (Information Bias)
Description: Systematic errors in how variables are measured.
Types: - Recall bias: Systematic differences in accuracy of recall - Social desirability bias: Responding in socially acceptable ways - Interviewer bias: Interviewer's characteristics affect responses - Instrument bias: Measurement tools systematically err
Mitigation: - Use validated, objective measures - Standardize data collection - Blind participants to hypotheses - Verify self-reports with objective data
13. Confounding Bias
Description: Effect of extraneous variable mixed with the variable of interest.
Examples: - Age confounding relationship between exercise and health - Socioeconomic status confounding education and outcomes - Indication bias in treatment studies
Mitigation: - Randomization - Matching - Statistical adjustment - Stratification - Restriction
14. Reporting Bias
Description: Selective reporting of results.
Types: - Outcome reporting bias: Selectively reporting outcomes - Time-lag bias: Delayed publication of negative results - Language bias: Publishing positive results in English - Citation bias: Preferentially citing positive studies
Mitigation: - Preregister all outcomes - Report all planned analyses - Distinguish primary from secondary outcomes - Use study registries
15. Spectrum Bias
Description: Test performance varies depending on the spectrum of disease severity in the sample.
Manifestations: - Diagnostic tests appearing more accurate in extreme cases - Treatment effects differing by severity
Mitigation: - Test in representative samples - Report performance across disease spectrum - Avoid case-control designs for diagnostic studies
16. Lead-Time Bias
Description: Apparent survival benefit due to earlier detection, not improved outcomes.
Example: - Screening detecting disease earlier makes survival seem longer, even if death occurs at same age
Mitigation: - Measure mortality, not just survival from diagnosis - Use randomized screening trials - Consider length-time and overdiagnosis bias
17. Length-Time Bias
Description: Screening disproportionately detects slower-growing, less aggressive cases.
Example: - Slow-growing cancers detected more often than fast-growing ones, making screening appear beneficial
Mitigation: - Randomized trials with mortality endpoints - Consider disease natural history
18. Response Bias
Description: Systematic pattern in how participants respond.
Types: - Acquiescence bias: Tendency to agree - Extreme responding: Always choosing extreme options - Neutral responding: Avoiding extreme responses - Demand characteristics: Responding based on perceived expectations
Mitigation: - Mix positive and negative items - Use multiple response formats - Blind participants to hypotheses - Use behavioral measures
Statistical and Analysis Biases
19. P-Hacking (Data Dredging)
Description: Manipulating data or analyses until significant results emerge.
Manifestations: - Collecting data until significance reached - Testing multiple outcomes, reporting only significant ones - Trying multiple analysis methods - Excluding "outliers" to reach significance - Subgroup analyses until finding significance
Detection: - Suspiciously perfect p-values (just below .05) - Many researcher degrees of freedom - Undisclosed analyses - Fishing expeditions
Mitigation: - Preregister analysis plans - Report all analyses conducted - Correct for multiple comparisons - Distinguish exploratory from confirmatory
20. HARKing (Hypothesizing After Results are Known)
Description: Presenting post hoc hypotheses as if they were predicted a priori.
Why problematic: - Inflates apparent evidence - Conflates exploration with confirmation - Misrepresents the scientific process
Mitigation: - Preregister hypotheses - Clearly label exploratory analyses - Require replication of unexpected findings
21. Base Rate Neglect
Description: Ignoring prior probability when evaluating evidence.
Example: - Test with 95% accuracy in rare disease (1% prevalence): positive result only 16% likely to indicate disease
Mitigation: - Always consider base rates/prior probability - Use Bayesian reasoning - Report positive and negative predictive values
22. Regression to the Mean
Description: Extreme measurements tend to be followed by less extreme ones.
Manifestations: - Treatment effects in extreme groups may be regression artifacts - "Sophomore slump" in high performers
Mitigation: - Use control groups - Consider natural variation - Don't select based on extreme baseline values without controls
23. Texas Sharpshooter Fallacy
Description: Selecting data after seeing patterns, like shooting arrows then drawing targets around clusters.
Manifestations: - Finding patterns in random data - Subgroup analyses selected post hoc - Geographic clustering studies without correction
Mitigation: - Prespecify hypotheses - Correct for multiple comparisons - Replicate findings in independent data
Reducing Bias: Best Practices
Study Design
- Randomization
- Blinding (single, double, triple)
- Control groups
- Adequate sample size
- Preregistration
Data Collection
- Standardized protocols
- Validated instruments
- Objective measures when possible
- Multiple observers/raters
- Complete data collection
Analysis
- Intention-to-treat analysis
- Prespecified analyses
- Appropriate statistical tests
- Multiple comparison corrections
- Sensitivity analyses
Reporting
- Complete transparency
- CONSORT, PRISMA, or similar guidelines
- Report all outcomes
- Distinguish exploratory from confirmatory
- Share data and code
Meta-Level
- Adversarial collaboration
- Replication studies
- Open science practices
- Peer review
- Systematic reviews