Level 6 · Research preparation · advanced assessment
Dossier provenance and forecast audit
Reconstruct claims from dated sources and score forecasts without letting later knowledge leak backward.
Complete all three challenges. Show assumptions, units, uncertainty, and the boundary between calculation, model-dependent inference, observation, and unresolved claim.
Challenge 1 · derivation
Rebuild a source-to-claim graph
Select one bounded DIRD-related claim and reconstruct the inferential path from primary sources to present wording. Formalize direct, inferred, contextual, and contradictory edges and identify the weakest dependency.
Required deliverables
- • Dated claim graph with stable source labels
- • Edge-by-edge warrant table
- • A rewritten claim limited to the surviving evidence
Scoring criteria · 20 points
- • Source chronology and provenance: 8 points
- • Inference edges accurately classified: 7 points
- • Final claim matches the graph: 5 points
Challenge 2 · analysis
Calibrate the forecasts
Create at least ten date-locked forecasts from dossier claims, assign probabilities before inspecting outcomes, then score calibration and discrimination. Compare results with a simple base-rate forecaster.
Required deliverables
- • Forecast table with cutoff dates and resolution rules
- • Brier scores and calibration bins
- • Comparison with the base-rate benchmark
Scoring criteria · 20 points
- • No temporal leakage: 7 points
- • Scoring and benchmark calculation: 7 points
- • Calibration interpretation: 6 points
Challenge 3 · design
Run an adversarial dossier review
Design a red-team/blue-team review in which both sides receive the same frozen corpus and must publish claim graphs, confidence changes, and decisive missing evidence.
Required deliverables
- • Frozen-corpus and role-assignment protocol
- • Symmetric scoring rubric and adjudication plan
- • Public evidence packet with uncertainty log
Scoring criteria · 20 points
- • Teams operate on identical evidence: 7 points
- • Rubric rewards correction and calibrated uncertainty: 7 points
- • Adjudication and release are auditable: 6 points
Instructor solution outlines
Look for reasoning that survives inspection
1. Rebuild a source-to-claim graph
- • Separate what a document states from what later authors infer.
- • Apply the declared evidence cutoff before reading outcome information.
- • Downgrade the conclusion when a key edge is contextual or unsupported.
2. Calibrate the forecasts
- • Make each resolution criterion observable and time bounded.
- • Score the original probabilities without retrospective editing.
- • Treat poor calibration as evidence about forecasting performance, not automatically about every underlying mechanism.
3. Run an adversarial dossier review
- • Conceal author identity where practical and randomize which team argues each side.
- • Score source fidelity separately from rhetorical persuasiveness.
- • Publish disagreements and unresolved dependencies rather than forcing consensus.