Experimental Design Checklist

Research Question Formulation

Is the Question Well-Formed?

• [ ] Specific: Clearly defined variables and relationships
• [ ] Answerable: Can be addressed with available methods
• [ ] Relevant: Addresses a gap in knowledge or practical need
• [ ] Feasible: Resources, time, and ethical considerations allow it
• [ ] Falsifiable: Can be proven wrong if incorrect

Have You Reviewed the Literature?

• [ ] Identified what's already known
• [ ] Found gaps or contradictions to address
• [ ] Learned from methodological successes and failures
• [ ] Identified appropriate outcome measures
• [ ] Determined typical effect sizes in the field

Hypothesis Development

Is Your Hypothesis Testable?

• [ ] Makes specific, quantifiable predictions
• [ ] Variables are operationally defined
• [ ] Specifies direction/nature of expected relationships
• [ ] Can be falsified by potential observations

Types of Hypotheses

• [ ] Null hypothesis (H₀): No effect/relationship exists
• [ ] Alternative hypothesis (H₁): Effect/relationship exists
• [ ] Directional vs. non-directional: One-tailed vs. two-tailed tests

Study Design Selection

What Type of Study is Appropriate?

Experimental (Intervention) Studies: - [ ] Randomized Controlled Trial (RCT): Gold standard for causation - [ ] Quasi-experimental: Non-random assignment but manipulation - [ ] Within-subjects: Same participants in all conditions - [ ] Between-subjects: Different participants per condition - [ ] Factorial: Multiple independent variables - [ ] Crossover: Participants receive multiple interventions sequentially

Observational Studies: - [ ] Cohort: Follow groups over time - [ ] Case-control: Compare those with/without outcome - [ ] Cross-sectional: Snapshot at one time point - [ ] Ecological: Population-level data

Consider: - [ ] Can you randomly assign participants? - [ ] Can you manipulate the independent variable? - [ ] Is the outcome rare (favor case-control) or common? - [ ] Do you need to establish temporal sequence? - [ ] What's feasible given ethical, practical constraints?

Variables

Independent Variables (Manipulated/Predictor)

• [ ] Clearly defined and operationalized
• [ ] Appropriate levels/categories chosen
• [ ] Manipulation is sufficient to test hypothesis
• [ ] Manipulation check planned (if applicable)

Dependent Variables (Outcome/Response)

• [ ] Directly measures the construct of interest
• [ ] Validated and reliable measurement
• [ ] Sensitive enough to detect expected effects
• [ ] Appropriate for statistical analysis planned
• [ ] Primary outcome clearly designated

Control Variables

• [ ] Confounding variables identified:
• Variables that affect both IV and DV
• Alternative explanations for findings
• [ ] Strategy for control:
• Randomization
• Matching
• Stratification
• Statistical adjustment
• Restriction (inclusion/exclusion criteria)
• Blinding

Extraneous Variables

• [ ] Potential sources of noise identified
• [ ] Standardized procedures to minimize
• [ ] Environmental factors controlled
• [ ] Time of day, setting, equipment standardized

Sampling

Population Definition

• [ ] Target population: Who you want to generalize to
• [ ] Accessible population: Who you can actually sample from
• [ ] Sample: Who actually participates
• [ ] Difference between these documented

Sampling Method

• [ ] Probability sampling (preferred for generalizability):
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Systematic sampling
• [ ] Non-probability sampling (common but limits generalizability):
• Convenience sampling
• Purposive sampling
• Snowball sampling
• Quota sampling

Sample Size

• [ ] A priori power analysis conducted
• Expected effect size (from literature or pilot)
• Desired power (typically .80 or .90)
• Significance level (typically .05)
• Statistical test to be used
• [ ] Accounts for expected attrition/dropout
• [ ] Sufficient for planned subgroup analyses
• [ ] Practical constraints acknowledged

Inclusion/Exclusion Criteria

• [ ] Clearly defined and justified
• [ ] Not overly restrictive (limits generalizability)
• [ ] Based on theoretical or practical considerations
• [ ] Ethical considerations addressed
• [ ] Documented and applied consistently

Blinding and Randomization

Randomization

• [ ] What is randomized:
• Participant assignment to conditions
• Order of conditions (within-subjects)
• Stimuli/items presented
• [ ] Method of randomization:
• Computer-generated random numbers
• Random number tables
• Coin flips (for very small studies)
• [ ] Allocation concealment:
• Sequence generated before recruitment
• Allocation hidden until after enrollment
• Sequentially numbered, sealed envelopes (if needed)
• [ ] Stratified randomization:
• Balance important variables across groups
• Block randomization to ensure equal group sizes
• [ ] Check randomization:
• Compare groups at baseline
• Report any significant differences

Blinding

• [ ] Single-blind: Participants don't know group assignment
• [ ] Double-blind: Participants and researchers don't know
• [ ] Triple-blind: Participants, researchers, and data analysts don't know
• [ ] Blinding feasibility:
• Is true blinding possible?
• Placebo/sham controls needed?
• Identical appearance of interventions?
• [ ] Blinding check:
• Assess whether blinding maintained
• Ask participants/researchers to guess assignments

Control Groups and Conditions

What Type of Control?

• [ ] No treatment control: Natural course of condition
• [ ] Placebo control: Inert treatment for comparison
• [ ] Active control: Standard treatment comparison
• [ ] Wait-list control: Delayed treatment
• [ ] Attention control: Matches contact time without active ingredient

Multiple Conditions

• [ ] Factorial designs for multiple factors
• [ ] Dose-response relationship assessment
• [ ] Mechanism testing with component analyses

Procedures

Protocol Development

• [ ] Detailed, written protocol:
• Step-by-step procedures
• Scripts for standardized instructions
• Decision rules for handling issues
• Data collection forms
• [ ] Pilot tested before main study
• [ ] Staff trained to criterion
• [ ] Compliance monitoring planned

Standardization

• [ ] Same instructions for all participants
• [ ] Same equipment and materials
• [ ] Same environment/setting when possible
• [ ] Same assessment timing
• [ ] Deviations from protocol documented

Data Collection

• [ ] When collected:
• Baseline measurements
• Post-intervention
• Follow-up timepoints
• [ ] Who collects:
• Trained researchers
• Blinded when possible
• Inter-rater reliability established
• [ ] How collected:
• Valid, reliable instruments
• Standardized administration
• Multiple methods if possible (triangulation)

Measurement

Validity

• [ ] Face validity: Appears to measure construct
• [ ] Content validity: Covers all aspects of construct
• [ ] Criterion validity: Correlates with gold standard
• Concurrent validity
• Predictive validity
• [ ] Construct validity: Measures theoretical construct
• Convergent validity (correlates with related measures)
• Discriminant validity (doesn't correlate with unrelated measures)

Reliability

• [ ] Test-retest: Consistent over time
• [ ] Internal consistency: Items measure same construct (Cronbach's α)
• [ ] Inter-rater reliability: Agreement between raters (Cohen's κ, ICC)
• [ ] Parallel forms: Alternative versions consistent

Measurement Considerations

• [ ] Objective measures preferred when possible
• [ ] Validated instruments used when available
• [ ] Multiple measures of key constructs
• [ ] Sensitivity to change considered
• [ ] Floor/ceiling effects avoided
• [ ] Response formats appropriate
• [ ] Recall periods appropriate
• [ ] Cultural appropriateness considered

Bias Minimization

Selection Bias

• [ ] Random sampling when possible
• [ ] Clearly defined eligibility criteria
• [ ] Document who declines and why
• [ ] Minimize self-selection

Performance Bias

• [ ] Standardized protocols
• [ ] Blinding of providers
• [ ] Monitor protocol adherence
• [ ] Document deviations

Detection Bias

• [ ] Blinding of outcome assessors
• [ ] Objective measures when possible
• [ ] Standardized assessment procedures
• [ ] Multiple raters with reliability checks

Attrition Bias

• [ ] Strategies to minimize dropout
• [ ] Track reasons for dropout
• [ ] Compare dropouts to completers
• [ ] Intention-to-treat analysis planned

Reporting Bias

• [ ] Preregister study and analysis plan
• [ ] Designate primary vs. secondary outcomes
• [ ] Commit to reporting all outcomes
• [ ] Distinguish planned from exploratory analyses

Data Management

Data Collection

• [ ] Data collection forms designed and tested
• [ ] REDCap, Qualtrics, or similar platforms
• [ ] Range checks and validation rules
• [ ] Regular backups
• [ ] Secure storage (HIPAA/GDPR compliant if needed)

Data Quality

• [ ] Real-time data validation
• [ ] Regular quality checks
• [ ] Missing data patterns monitored
• [ ] Outliers identified and investigated
• [ ] Protocol deviations documented

Data Security

• [ ] De-identification procedures
• [ ] Access controls
• [ ] Audit trails
• [ ] Compliance with regulations (IRB, HIPAA, GDPR)

Statistical Analysis Planning

Analysis Plan (Prespecify Before Data Collection)

• [ ] Primary analysis:
• Statistical test(s) specified
• Hypothesis clearly stated
• Significance level set (usually α = .05)
• One-tailed or two-tailed
• [ ] Secondary analyses:
• Clearly designated as secondary
• Exploratory analyses labeled as such
• [ ] Multiple comparisons:
• Adjustment method specified (if needed)
• Primary outcome protects from inflation

Assumptions

• [ ] Assumptions of statistical tests identified
• [ ] Plan to check assumptions
• [ ] Backup non-parametric alternatives
• [ ] Transformation options considered

Missing Data

• [ ] Anticipated amount of missingness
• [ ] Missing data mechanism (MCAR, MAR, MNAR)
• [ ] Handling strategy:
• Complete case analysis
• Multiple imputation
• Maximum likelihood
• [ ] Sensitivity analyses planned

Effect Sizes

• [ ] Appropriate effect size measures identified
• [ ] Will be reported alongside p-values
• [ ] Confidence intervals planned

Statistical Software

• [ ] Software selected (R, SPSS, Stata, Python, etc.)
• [ ] Version documented
• [ ] Analysis scripts prepared in advance
• [ ] Will be made available (Open Science)

Ethical Considerations

Ethical Approval

• [ ] IRB/Ethics committee approval obtained
• [ ] Study registered (ClinicalTrials.gov, etc.) if applicable
• [ ] Protocol follows Declaration of Helsinki or equivalent

Informed Consent

• [ ] Voluntary participation
• [ ] Comprehensible explanation
• [ ] Risks and benefits disclosed
• [ ] Right to withdraw without penalty
• [ ] Privacy protections explained
• [ ] Compensation disclosed

Risk-Benefit Analysis

• [ ] Potential benefits outweigh risks
• [ ] Risks minimized
• [ ] Vulnerable populations protected
• [ ] Data safety monitoring (if high risk)

Confidentiality

• [ ] Data de-identified
• [ ] Secure storage
• [ ] Limited access
• [ ] Reporting doesn't allow re-identification

Validity Threats

Internal Validity (Causation)

• [ ] History: External events between measurements
• [ ] Maturation: Changes in participants over time
• [ ] Testing: Effects of repeated measurement
• [ ] Instrumentation: Changes in measurement over time
• [ ] Regression to mean: Extreme scores becoming less extreme
• [ ] Selection: Groups differ at baseline
• [ ] Attrition: Differential dropout
• [ ] Diffusion: Control group receives treatment elements

External Validity (Generalizability)

• [ ] Sample representative of population
• [ ] Setting realistic/natural
• [ ] Treatment typical of real-world implementation
• [ ] Outcome measures ecologically valid
• [ ] Time frame appropriate

Construct Validity (Measurement)

• [ ] Measures actually tap intended constructs
• [ ] Operations match theoretical definitions
• [ ] No confounding of constructs
• [ ] Adequate coverage of construct

Statistical Conclusion Validity

• [ ] Adequate statistical power
• [ ] Assumptions met
• [ ] Appropriate tests used
• [ ] Alpha level appropriate
• [ ] Multiple comparisons addressed

Reporting and Transparency

Preregistration

• [ ] Study preregistered (OSF, ClinicalTrials.gov, AsPredicted)
• [ ] Hypotheses stated a priori
• [ ] Analysis plan documented
• [ ] Distinguishes confirmatory from exploratory

Reporting Guidelines

• [ ] RCTs: CONSORT checklist
• [ ] Observational studies: STROBE checklist
• [ ] Systematic reviews: PRISMA checklist
• [ ] Diagnostic studies: STARD checklist
• [ ] Qualitative research: COREQ checklist
• [ ] Case reports: CARE guidelines

Transparency

• [ ] All measures reported
• [ ] All manipulations disclosed
• [ ] Sample size determination explained
• [ ] Exclusion criteria and numbers reported
• [ ] Attrition documented
• [ ] Deviations from protocol noted
• [ ] Conflicts of interest disclosed

Open Science

• [ ] Data sharing planned (when ethical)
• [ ] Analysis code shared
• [ ] Materials available
• [ ] Preprint posted
• [ ] Open access publication when possible

Post-Study Considerations

Data Analysis

• [ ] Follow preregistered plan
• [ ] Clearly label deviations and exploratory analyses
• [ ] Check assumptions
• [ ] Report all outcomes
• [ ] Report effect sizes and CIs, not just p-values

Interpretation

• [ ] Conclusions supported by data
• [ ] Limitations acknowledged
• [ ] Alternative explanations considered
• [ ] Generalizability discussed
• [ ] Clinical/practical significance addressed

Dissemination

• [ ] Publish regardless of results (reduce publication bias)
• [ ] Present at conferences
• [ ] Share findings with participants (when appropriate)
• [ ] Communicate to relevant stakeholders
• [ ] Plain language summaries

Next Steps

• [ ] Replication needed?
• [ ] Follow-up studies identified
• [ ] Mechanism studies planned
• [ ] Clinical applications considered

Common Pitfalls to Avoid

• [ ] No power analysis → underpowered study
• [ ] Hypothesis formed after seeing data (HARKing)
• [ ] No blinding when feasible → bias
• [ ] P-hacking (data fishing, optional stopping)
• [ ] Multiple testing without correction → false positives
• [ ] Inadequate control group
• [ ] Confounding not addressed
• [ ] Instruments not validated
• [ ] High attrition not addressed
• [ ] Cherry-picking results to report
• [ ] Causal language from correlational data
• [ ] Ignoring assumptions of statistical tests
• [ ] Not preregistering changes literature bias
• [ ] Conflicts of interest not disclosed

Final Checklist Before Starting

• [ ] Research question is clear and important
• [ ] Hypothesis is testable and specific
• [ ] Study design is appropriate
• [ ] Sample size is adequate (power analysis)
• [ ] Measures are valid and reliable
• [ ] Confounds are controlled
• [ ] Randomization and blinding implemented
• [ ] Data collection is standardized
• [ ] Analysis plan is prespecified
• [ ] Ethical approval obtained
• [ ] Study is preregistered
• [ ] Resources are sufficient
• [ ] Team is trained
• [ ] Protocol is documented
• [ ] Backup plans exist for problems

Remember

Good experimental design is about: - Asking clear questions - Minimizing bias - Maximizing validity - Appropriate inference - Transparency - Reproducibility

The best time to think about these issues is before collecting data, not after.