Common Statistical Pitfalls

P-Value Misinterpretations

Pitfall 1: P-Value = Probability Hypothesis is True

Misconception: p = .05 means 5% chance the null hypothesis is true.

Reality: P-value is the probability of observing data this extreme (or more) if the null hypothesis is true. It says nothing about the probability the hypothesis is true.

Correct interpretation: "If there were truly no effect, we would observe data this extreme only 5% of the time."

Pitfall 2: Non-Significant = No Effect

Misconception: p > .05 proves there's no effect.

Reality: Absence of evidence ≠ evidence of absence. Non-significant results may indicate: - Insufficient statistical power - True effect too small to detect - High variability - Small sample size

Better approach: - Report confidence intervals - Conduct power analysis - Consider equivalence testing

Pitfall 3: Significant = Important

Misconception: Statistical significance means practical importance.

Reality: With large samples, trivial effects become "significant." A statistically significant 0.1 IQ point difference is meaningless in practice.

Better approach: - Report effect sizes - Consider practical significance - Use confidence intervals

Pitfall 4: P = .049 vs. P = .051

Misconception: These are meaningfully different because one crosses the .05 threshold.

Reality: These represent nearly identical evidence. The .05 threshold is arbitrary.

Better approach: - Treat p-values as continuous measures of evidence - Report exact p-values - Consider context and prior evidence

Pitfall 5: One-Tailed Tests Without Justification

Misconception: One-tailed tests are free extra power.

Reality: One-tailed tests assume effects can only go one direction, which is rarely true. They're often used to artificially boost significance.

When appropriate: Only when effects in one direction are theoretically impossible or equivalent to null.

Multiple Comparisons Problems

Pitfall 6: Multiple Testing Without Correction

Problem: Testing 20 hypotheses at p < .05 gives ~65% chance of at least one false positive.

Examples: - Testing many outcomes - Testing many subgroups - Conducting multiple interim analyses - Testing at multiple time points

Solutions: - Bonferroni correction (divide α by number of tests) - False Discovery Rate (FDR) control - Prespecify primary outcome - Treat exploratory analyses as hypothesis-generating

Pitfall 7: Subgroup Analysis Fishing

Problem: Testing many subgroups until finding significance.

Why problematic: - Inflates false positive rate - Often reported without disclosure - "Interaction was significant in women" may be random

Solutions: - Prespecify subgroups - Use interaction tests, not separate tests - Require replication - Correct for multiple comparisons

Pitfall 8: Outcome Switching

Problem: Analyzing many outcomes, reporting only significant ones.

Detection signs: - Secondary outcomes emphasized - Incomplete outcome reporting - Discrepancy between registration and publication

Solutions: - Preregister all outcomes - Report all planned outcomes - Distinguish primary from secondary

Sample Size and Power Issues

Pitfall 9: Underpowered Studies

Problem: Small samples have low probability of detecting true effects.

Consequences: - High false negative rate - Significant results more likely to be false positives - Overestimated effect sizes (when significant)

Solutions: - Conduct a priori power analysis - Aim for 80-90% power - Consider effect size from prior research

Pitfall 10: Post-Hoc Power Analysis

Problem: Calculating power after seeing results is circular and uninformative.

Why useless: - Non-significant results always have low "post-hoc power" - It recapitulates the p-value without new information

Better approach: - Calculate confidence intervals - Plan replication with adequate sample - Conduct prospective power analysis for future studies

Pitfall 11: Small Sample Fallacy

Problem: Trusting results from very small samples.

Issues: - High sampling variability - Outliers have large influence - Assumptions of tests violated - Confidence intervals very wide

Guidelines: - Be skeptical of n < 30 - Check assumptions carefully - Consider non-parametric tests - Replicate findings

Effect Size Misunderstandings

Pitfall 12: Ignoring Effect Size

Problem: Focusing only on significance, not magnitude.

Why problematic: - Significance ≠ importance - Can't compare across studies - Doesn't inform practical decisions

Solutions: - Always report effect sizes - Use standardized measures (Cohen's d, r, η²) - Interpret using field conventions - Consider minimum clinically important difference

Pitfall 13: Misinterpreting Standardized Effect Sizes

Problem: Treating Cohen's d = 0.5 as "medium" without context.

Reality: - Field-specific norms vary - Some fields have larger typical effects - Real-world importance depends on context

Better approach: - Compare to effects in same domain - Consider practical implications - Look at raw effect sizes too

Pitfall 14: Confusing Explained Variance with Importance

Problem: "Only explains 5% of variance" = unimportant.

Reality: - Height explains ~5% of variation in NBA player salary but is crucial - Complex phenomena have many small contributors - Predictive accuracy ≠ causal importance

Consideration: Context matters more than percentage alone.

Correlation and Causation

Pitfall 15: Correlation Implies Causation

Problem: Inferring causation from correlation.

Alternative explanations: - Reverse causation (B causes A, not A causes B) - Confounding (C causes both A and B) - Coincidence - Selection bias

Criteria for causation: - Temporal precedence - Covariation - No plausible alternatives - Ideally: experimental manipulation

Pitfall 16: Ecological Fallacy

Problem: Inferring individual-level relationships from group-level data.

Example: Countries with more chocolate consumption have more Nobel laureates doesn't mean eating chocolate makes you win Nobels.

Why problematic: Group-level correlations may not hold at individual level.

Pitfall 17: Simpson's Paradox

Problem: Trend appears in groups but reverses when combined (or vice versa).

Example: Treatment appears worse overall but better in every subgroup.

Cause: Confounding variable distributed differently across groups.

Solution: Consider confounders and look at appropriate level of analysis.

Regression and Modeling Pitfalls

Pitfall 18: Overfitting

Problem: Model fits sample data well but doesn't generalize.

Causes: - Too many predictors relative to sample size - Fitting noise rather than signal - No cross-validation

Solutions: - Use cross-validation - Penalized regression (LASSO, ridge) - Independent test set - Simpler models

Pitfall 19: Extrapolation Beyond Data Range

Problem: Predicting outside the range of observed data.

Why dangerous: - Relationships may not hold outside observed range - Increased uncertainty not reflected in predictions

Solution: Only interpolate; avoid extrapolation.

Pitfall 20: Ignoring Model Assumptions

Problem: Using statistical tests without checking assumptions.

Common violations: - Non-normality (for parametric tests) - Heteroscedasticity (unequal variances) - Non-independence - Linearity - No multicollinearity

Solutions: - Check assumptions with diagnostics - Use robust methods - Transform data - Use appropriate non-parametric alternatives

Pitfall 21: Treating Non-Significant Covariates as Eliminating Confounding

Problem: "We controlled for X and it wasn't significant, so it's not a confounder."

Reality: Non-significant covariates can still be important confounders. Significance ≠ confounding.

Solution: Include theoretically important covariates regardless of significance.

Pitfall 22: Collinearity Masking Effects

Problem: When predictors are highly correlated, true effects may appear non-significant.

Manifestations: - Large standard errors - Unstable coefficients - Sign changes when adding/removing variables

Detection: - Variance Inflation Factors (VIF) - Correlation matrices

Solutions: - Remove redundant predictors - Combine correlated variables - Use regularization methods

Specific Test Misuses

Pitfall 23: T-Test for Multiple Groups

Problem: Conducting multiple t-tests instead of ANOVA.

Why wrong: Inflates Type I error rate dramatically.

Correct approach: - Use ANOVA first - Follow with planned comparisons or post-hoc tests with correction

Pitfall 24: Pearson Correlation for Non-Linear Relationships

Problem: Using Pearson's r for curved relationships.

Why misleading: r measures linear relationships only.

Solutions: - Check scatterplots first - Use Spearman's ρ for monotonic relationships - Consider polynomial or non-linear models

Pitfall 25: Chi-Square with Small Expected Frequencies

Problem: Chi-square test with expected cell counts < 5.

Why wrong: Violates test assumptions, p-values inaccurate.

Solutions: - Fisher's exact test - Combine categories - Increase sample size

Pitfall 26: Paired vs. Independent Tests

Problem: Using independent samples test for paired data (or vice versa).

Why wrong: - Wastes power (paired data analyzed as independent) - Violates independence assumption (independent data analyzed as paired)

Solution: Match test to design.

Confidence Interval Misinterpretations

Pitfall 27: 95% CI = 95% Probability True Value Inside

Misconception: "95% chance the true value is in this interval."

Reality: The true value either is or isn't in this specific interval. If we repeated the study many times, 95% of resulting intervals would contain the true value.

Better interpretation: "We're 95% confident this interval contains the true value."

Pitfall 28: Overlapping CIs = No Difference

Problem: Assuming overlapping confidence intervals mean no significant difference.

Reality: Overlapping CIs are less stringent than difference tests. Two CIs can overlap while the difference between groups is significant.

Guideline: Overlap of point estimate with other CI is more relevant than overlap of intervals.

Pitfall 29: Ignoring CI Width

Problem: Focusing only on whether CI includes zero, not precision.

Why important: Wide CIs indicate high uncertainty. "Significant" effects with huge CIs are less convincing.

Consider: Both significance and precision.

Bayesian vs. Frequentist Confusions

Pitfall 30: Mixing Bayesian and Frequentist Interpretations

Problem: Making Bayesian statements from frequentist analyses.

Examples: - "Probability hypothesis is true" (Bayesian) from p-value (frequentist) - "Evidence for null" from non-significant result (frequentist can't support null)

Solution: - Be clear about framework - Use Bayesian methods for Bayesian questions - Use Bayes factors to compare hypotheses

Pitfall 31: Ignoring Prior Probability

Problem: Treating all hypotheses as equally likely initially.

Reality: Extraordinary claims need extraordinary evidence. Prior plausibility matters.

Consider: - Plausibility given existing knowledge - Mechanism plausibility - Base rates

Data Transformation Issues

Pitfall 32: Dichotomizing Continuous Variables

Problem: Splitting continuous variables at arbitrary cutoffs.

Consequences: - Loss of information and power - Arbitrary distinctions - Discarding individual differences

Exceptions: Clinically meaningful cutoffs with strong justification.

Better: Keep continuous or use multiple categories.

Pitfall 33: Trying Multiple Transformations

Problem: Testing many transformations until finding significance.

Why problematic: Inflates Type I error, is a form of p-hacking.

Better approach: - Prespecify transformations - Use theory-driven transformations - Correct for multiple testing if exploring

Missing Data Problems

Pitfall 34: Listwise Deletion by Default

Problem: Automatically deleting all cases with any missing data.

Consequences: - Reduced power - Potential bias if data not missing completely at random (MCAR)

Better approaches: - Multiple imputation - Maximum likelihood methods - Analyze missingness patterns

Pitfall 35: Ignoring Missing Data Mechanisms

Problem: Not considering why data are missing.

Types: - MCAR (Missing Completely at Random): Safe to delete - MAR (Missing at Random): Can impute - MNAR (Missing Not at Random): May bias results

Solution: Analyze patterns, use appropriate methods, consider sensitivity analyses.

Publication and Reporting Issues

Pitfall 36: Selective Reporting

Problem: Only reporting significant results or favorable analyses.

Consequences: - Literature appears more consistent than reality - Meta-analyses biased - Wasted research effort

Solutions: - Preregistration - Report all analyses - Use reporting guidelines (CONSORT, PRISMA, etc.)

Pitfall 37: Rounding to p < .05

Problem: Reporting exact p-values selectively (e.g., p = .049 but p < .05 for .051).

Why problematic: Obscures values near threshold, enables p-hacking detection evasion.

Better: Always report exact p-values.

Pitfall 38: No Data Sharing

Problem: Not making data available for verification or reanalysis.

Consequences: - Can't verify results - Can't include in meta-analyses - Hinders scientific progress

Best practice: Share data unless privacy concerns prohibit.

Cross-Validation and Generalization

Pitfall 39: No Cross-Validation

Problem: Testing model on same data used to build it.

Consequence: Overly optimistic performance estimates.

Solutions: - Split data (train/test) - K-fold cross-validation - Independent validation sample

Pitfall 40: Data Leakage

Problem: Information from test set leaking into training.

Examples: - Normalizing before splitting - Feature selection on full dataset - Including temporal information

Consequence: Inflated performance metrics.

Prevention: All preprocessing decisions made using only training data.

Meta-Analysis Pitfalls

Pitfall 41: Apples and Oranges

Problem: Combining studies with different designs, populations, or measures.

Balance: Need homogeneity but also comprehensiveness.

Solutions: - Clear inclusion criteria - Subgroup analyses - Meta-regression for moderators

Pitfall 42: Ignoring Publication Bias

Problem: Published studies overrepresent significant results.

Consequences: Overestimated effects in meta-analyses.

Detection: - Funnel plots - Trim-and-fill - PET-PEESE - P-curve analysis

Solutions: - Include unpublished studies - Register reviews - Use bias-correction methods

General Best Practices

1. Preregister studies - Distinguish confirmatory from exploratory
2. Report transparently - All analyses, not just significant ones
3. Check assumptions - Don't blindly apply tests
4. Use appropriate tests - Match test to data and design
5. Report effect sizes - Not just p-values
6. Consider practical significance - Not just statistical
7. Replicate findings - One study is rarely definitive
8. Share data and code - Enable verification
9. Use confidence intervals - Show uncertainty
10. Think causally carefully - Most research is correlational