Level 3 · Undergraduate core teaching kit · First- and second-year university
Experimental design, inference, and reproducibility
Use the learner record during the live investigation, then use the instructor guide to facilitate comparison, address misconceptions, and assess evidence-bounded reasoning.
Learner lab record
Blinded residual and search-penalty analysis
When does a local residual remain credible after correlated uncertainty, injection recovery, and the declared search space are included?
Setup
Use the blinded-analysis laboratory. Freeze the model and cuts, inspect a baseline null ensemble, then test one hidden injection and apply the declared search adjustment.
Predict first
- 1. Predict whether more trials remove a fully correlated offset.
- 2. Predict how a larger declared search count changes global significance.
| Variable | Role | Unit |
|---|---|---|
| Trial count and independent noise | design inputs | count and signal unit |
| Correlated systematic | shared uncertainty input | signal unit |
| Injected signal | validation input | signal unit |
| Local and adjusted significance | dependent diagnostics | σ or probability |
Observation columns
Analyze
- 1. Which uncertainty term averages down?
- 2. Did the pipeline recover the known injection within tolerance?
- 3. Why can a local detection become a global non-detection?
- 4. What decision must be frozen before unblinding?
Conclusion frame
The pipeline produced local ___ and adjusted ___ across ___ searches; injection recovery was ___, so the preregistered decision is ___.
Instructor guide · 55–75 minutes
Teach the investigation, not the interface
Learning target: Learners treat blinding, correlated uncertainty, injection recovery, and multiplicity correction as one evidence pipeline rather than optional post hoc cautions.
Prepare
- • Define the analysis decision before revealing the modeled result.
- • Prepare one failed-injection case.
- • Distinguish local from family-wise probability.
Facilitation moves
- • Ask which choices were made before unblinding.
- • Do not allow trial count to erase shared systematics.
- • Require a decision statement even for an interesting residual.
Accessibility and participation
- • Translate probabilities into frequencies without overstating certainty.
- • Provide the decision tree in text and diagram form.
- • Allow spreadsheet or calculator support for uncertainty combination.
Evidence of learning
- • A frozen analysis decision
- • A correct correlated-error explanation
- • An injection and multiplicity-aware conclusion
Misconception checks
Enough repeated trials eliminate every uncertainty.
Independent noise averages down; correlated bias and model error require separate controls or bounds.
A small local p-value proves the preferred mechanism.
Search multiplicity changes surprise, and mechanism attribution requires discriminating predictions and controls.
Extension
Design a preregistered two-laboratory replication with a shared injection protocol and independent analysis teams.