Identical P values with very different interpretations

Experiments A and B have identical P values, but the scientific conclusion is very different. The interpretation depends upon the scientific context, but in most fields experiment A would be solid negative data proving that there either is no effect or that the effect is tiny. In contrast, experiment B has such a wide confidence interval as to be consistent with nearly any hypothesis. Those data simply do not help answer your scientific question. Similarly, experiments C and D have identical P values, but should be interpreted differently. In most experimental contexts, experiment C demonstrates convincingly that, although the difference is not zero, it is quite small. Experiment D provides convincing evidence that the effect is large.

Treatment 1Treatment 2Difference between MeansP Value95% CI of the Difference between Means
mean ± S.D. (n)
Experiment A1000 ± 100 (50)990.0 ± 100 (50)100.6−30 to 50
Experiment B1000 ± 100 (3)950.0 ± 100 (3)500.6−177 to 277
Experiment C100 ± 5.0 (135)102 ± 5.0 (135)20.0010.8 to 3.2
Experiment D100 ± 5.0 (3)135 ± 5.0 (3)350.00124 to 46
  • CI, confidence interval.