In a conversation with a technical analyst, the author presses the distinction between explaining stored charts and predicting. The discussion of selected trend lines and recurring models exposes the flexibility of retrospective interpretation.
Reserve something you have not seen
Our research extension is to separate data used to formulate a rule from data used to evaluate it. If every revision is chosen because it improves the same historical result, the result also measures the researcher’s ability to fit that history.
Worked example
A fictional rule is adjusted ten times on January–June data. Testing the final rule on untouched July–December data asks a different and more demanding question.
Case connection
A fresh replication asks a different question from explaining the original sample. New observations challenge a fitted story.
Psychology’s replication project
Source-grounded facts
Researchers attempted to replicate 100 psychology findings; 36% reached statistical significance in the same direction.
Context
A published finding answers a question in one study. The Reproducibility Project: Psychology asked whether independent repetitions would support a selected group of published findings.
Outcome
The project exposed uncertainty about reproducibility. A failed significance test is not automatically proof that the original claim was false or fraudulent; interpretation requires examining methods, precision and conditions.
- Researchers attempted to replicate 100 psychology findings.
- They compared the new evidence with the original results, including direction, significance and effect sizes.
- The Center for Open Science reports that 36% reached statistical significance in the same direction; replication effect sizes were about half the originals.
Case analysis
Explaining an existing sample gives the researcher many opportunities to adjust the story. An independent repetition removes some of that flexibility because the result is not yet known. In your own study, freeze the rule before opening the evaluation data. A later failure should be recorded as evidence, not quietly absorbed into a redesigned historical example.
Try it
Design a study with a development sample and a later evaluation sample. Write what may change before evaluation and what must remain fixed.