Skip to main content
Advertisement

Main menu

  • Home
  • Articles
    • Current Issue
    • Fast Forward
    • Latest Articles
    • Special Sections
    • Archive
  • Information
    • Instructions to Authors
    • Submit a Manuscript
    • FAQs
    • For Subscribers
    • Terms & Conditions of Use
    • Permissions
  • Editorial Board
  • Alerts
    • Alerts
    • RSS Feeds
  • Virtual Issues
  • Feedback
  • Submit
  • Other Publications
    • Drug Metabolism and Disposition
    • Journal of Pharmacology and Experimental Therapeutics
    • Molecular Pharmacology
    • Pharmacological Reviews
    • Pharmacology Research & Perspectives
    • ASPET

User menu

  • My alerts
  • Log in
  • My Cart

Search

  • Advanced search
Journal of Pharmacology and Experimental Therapeutics
  • Other Publications
    • Drug Metabolism and Disposition
    • Journal of Pharmacology and Experimental Therapeutics
    • Molecular Pharmacology
    • Pharmacological Reviews
    • Pharmacology Research & Perspectives
    • ASPET
  • My alerts
  • Log in
  • My Cart
Journal of Pharmacology and Experimental Therapeutics

Advanced Search

  • Home
  • Articles
    • Current Issue
    • Fast Forward
    • Latest Articles
    • Special Sections
    • Archive
  • Information
    • Instructions to Authors
    • Submit a Manuscript
    • FAQs
    • For Subscribers
    • Terms & Conditions of Use
    • Permissions
  • Editorial Board
  • Alerts
    • Alerts
    • RSS Feeds
  • Virtual Issues
  • Feedback
  • Submit
  • Visit jpet on Facebook
  • Follow jpet on Twitter
  • Follow jpet on LinkedIn
Article CommentaryCommentary

Common Misconceptions about Data Analysis and Statistics

Harvey J. Motulsky
Journal of Pharmacology and Experimental Therapeutics October 2014, 351 (1) 200-205; DOI: https://doi.org/10.1124/jpet.114.219170
Harvey J. Motulsky
GraphPad Software Inc., La Jolla, California
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • Info & Metrics
  • eLetters
  • PDF
Loading

Article Figures & Data

Figures

  • Tables
  • Fig. 1.
    • Download figure
    • Open in new tab
    • Download powerpoint
    Fig. 1.

    The many forms of P-hacking. When you P-hack, the results cannot be interpreted at face value. Not shown in the figure is that, after trying various forms of P-hacking without getting a small P value, you will eventually give up when you run out of time, funds, or curiosity.

  • Fig. 2.
    • Download figure
    • Open in new tab
    • Download powerpoint
    Fig. 2.

    The problem of ad-hoc sample size selection. I simulated 10,000 experiments sampling data from a Gaussian distribution with means of 5.0 and standard deviations of 1.0, and comparing two samples with n = 5 each using an unpaired t test. The first column shows the percentage of those experiments with a P value less than 0.05. Since both populations have the same mean, the null hypothesis is true, and so (as expected) about 5.0% of the simulations have P values less than 0.05. For the experiments where the P value was higher than 0.05, I added five more values to each group. The second column (n = 5 + 5) shows the fraction of P values where the P value was less than 0.05 either in the first analysis with n = 5 or after increasing the sample size to 10. For the third column, I added yet another 5 values to each group if the P value was greater than 0.05 for both of the first two analyses. Now 13% of the experiments (not 5%) have reached a P value less than 0.05. For the fourth column, I looked at all 10,000 of the simulated experiments with n = 15. As expected, very close to 5% of those experiments had P values less than 0.05. The higher fraction of “significant” findings in the n = 5 + 5 and n = 5 + 5 + 5 is due to the fact that I increased sample size only when the P value was high with smaller sample sizes. In many cases, when the P value was less than 0.05 with n = 5, the P value would have been higher than 0.05 with n = 10 or 15, but an experimenter seeing the small P value with the small sample size would not have increased sample size.

  • Fig. 3.
    • Download figure
    • Open in new tab
    • Download powerpoint
    Fig. 3.

    The problem of HARKing. (Reprinted from http://xkcd.com/882 under the CC BY-NC 2.5 license.)

  • Fig. 4.
    • Download figure
    • Open in new tab
    • Download powerpoint
    Fig. 4.

    P values depend upon sample size. This graph shows P values computed by unpaired t tests comparing two sets of data. The means of the two samples are 10 and 12. The S.D. of each sample is 5.0. I computed a t test using various sample sizes plotted on the x-axis. You can see that the P value depends on sample size. Note that both axes use a logarithmic scale.

  • Fig. 5.
    • Download figure
    • Open in new tab
    • Download powerpoint
    Fig. 5.

    Standard error bars do not show variability and do a poor job of showing precision. The figure plots one data set six ways. The left-most lane shows a scatter plot of every value, and so is the most informative. The next lane shows a box-and-whisker plot showing the range of the data, the quartiles, and the median (whiskers can be plotted in various ways, and do not always show the range). The third lane plots the median and quartiles. This shows less detail, but still demonstrates that the distribution is a bit asymmetric. The fourth lane plots mean with error bars showing plus or minus one standard deviation. Note that these error bars, by definition, are symmetrical and so give no hint about the asymmetry of the data. The next two lanes are different from the others as they do not show scatter. Instead they show how precisely we know the population mean, accounting for scatter and sample size. The fifth lane shows the mean with error bars showing the 95% confidence interval (CI) of the mean. The sixth (right-most) lane plots the mean plus or minus one standard error of the mean, which does not show variation and does a poor job of showing precision.

Tables

  • Figures
    • View popup
    TABLE 1

    Identical P values with very different interpretations

    Experiments A and B have identical P values, but the scientific conclusion is very different. The interpretation depends upon the scientific context, but in most fields experiment A would be solid negative data proving that there either is no effect or that the effect is tiny. In contrast, experiment B has such a wide confidence interval as to be consistent with nearly any hypothesis. Those data simply do not help answer your scientific question. Similarly, experiments C and D have identical P values, but should be interpreted differently. In most experimental contexts, experiment C demonstrates convincingly that, although the difference is not zero, it is quite small. Experiment D provides convincing evidence that the effect is large.

    Treatment 1Treatment 2Difference between MeansP Value95% CI of the Difference between Means
    mean ± S.D. (n)
    Experiment A1000 ± 100 (50)990.0 ± 100 (50)100.6−30 to 50
    Experiment B1000 ± 100 (3)950.0 ± 100 (3)500.6−177 to 277
    Experiment C100 ± 5.0 (135)102 ± 5.0 (135)20.0010.8 to 3.2
    Experiment D100 ± 5.0 (3)135 ± 5.0 (3)350.00124 to 46
    • CI, confidence interval.

    • View popup
    TABLE 2

    The false discovery rate when P < 0.05

    This table tabulates the theoretical results of 1000 experiments where the prior probability that the null hypothesis is false is 10%, the sample size is large enough so that the power is 80%, and the significance level is the traditional 5%. In 100 of the experiments (10%), there really is an effect (the null hypothesis is false), and you will obtain a “statistically significant” result (P < 0.05) in 80 of these (because the power is 80%). In 900 experiments, the null hypothesis is true, but you will obtain a statistically significant result in 45 of them (because the significance threshold is 5%, and 5% of 900 is 45). In total, you will obtain 80 + 45 = 125 statistically significant results, but 45/125 = 36% of these will be false positive. The proportion of conclusions of “statistical significance” that are false discoveries or false positives depends on the context of the experiment, as expressed by the prior probability (here, 10%). If you do obtain a small P value and reject the null hypothesis, you will conclude that the values in the two groups were sampled from different distributions. As noted earlier, there may be a high chance that you made a false-positive conclusion due to random sampling. But even if the conclusion is “true” from a statistical point of view and not a false positive due to random sampling, the effect may have occurred for a reason different from the one you hypothesized. When thinking about why an effect occurred, ignore the statistical calculations, and instead think about blinding, randomization, positive controls, negative controls, calibration, biases, and other aspects of experimental design.

    P < 0.05P > 0.05Total
    Really is an effect8020100
    No effect (null hypothesis true)45855900
    Total1258751000
PreviousNext
Back to top

In this issue

Journal of Pharmacology and Experimental Therapeutics: 351 (1)
Journal of Pharmacology and Experimental Therapeutics
Vol. 351, Issue 1
1 Oct 2014
  • Table of Contents
  • Table of Contents (PDF)
  • About the Cover
  • Index by author
  • Editorial Board (PDF)
  • Front Matter (PDF)
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for sharing this Journal of Pharmacology and Experimental Therapeutics article.

NOTE: We request your email address only to inform the recipient that it was you who recommended this article, and that it is not junk mail. We do not retain these email addresses.

Enter multiple addresses on separate lines or separate them with commas.
Common Misconceptions about Data Analysis and Statistics
(Your Name) has forwarded a page to you from Journal of Pharmacology and Experimental Therapeutics
(Your Name) thought you would be interested in this article in Journal of Pharmacology and Experimental Therapeutics.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Article CommentaryCommentary

Misconceptions about Data Analysis and Statistics

Harvey J. Motulsky
Journal of Pharmacology and Experimental Therapeutics October 1, 2014, 351 (1) 200-205; DOI: https://doi.org/10.1124/jpet.114.219170

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero

Share
Article CommentaryCommentary

Misconceptions about Data Analysis and Statistics

Harvey J. Motulsky
Journal of Pharmacology and Experimental Therapeutics October 1, 2014, 351 (1) 200-205; DOI: https://doi.org/10.1124/jpet.114.219170
del.icio.us logo Digg logo Reddit logo Twitter logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • Introduction
    • Misconception 1: P-Hacking Is OK
    • Misconception 2: P Values Convey Information about Effect Size
    • Misconception 3: Statistical Hypothesis Testing and Reports of Statistical Significance Are Necessary in Experimental Research
    • Misconception 4: The Standard Error of the Mean Quantifies Variability
    • Misconception 5: You Do Not Need to Report the Details
    • Summary
    • Footnotes
    • References
  • Figures & Data
  • Info & Metrics
  • eLetters
  • PDF

Related Articles

Cited By...

More in this TOC Section

  • Reporting Data Analysis and Statistical Methods
  • Editor’s Note
  • Editor’s Note
Show more Commentary

Similar Articles

Advertisement
  • Home
  • Alerts
Facebook   Twitter   LinkedIn   RSS

Navigate

  • Current Issue
  • Fast Forward by date
  • Fast Forward by section
  • Latest Articles
  • Archive
  • Search for Articles
  • Feedback
  • ASPET

More Information

  • About JPET
  • Editorial Board
  • Instructions to Authors
  • Submit a Manuscript
  • Customized Alerts
  • RSS Feeds
  • Subscriptions
  • Permissions
  • Terms & Conditions of Use

ASPET's Other Journals

  • Drug Metabolism and Disposition
  • Molecular Pharmacology
  • Pharmacological Reviews
  • Pharmacology Research & Perspectives
ISSN 1521-0103 (Online)

Copyright © 2023 by the American Society for Pharmacology and Experimental Therapeutics