What is the role of statistics in science?
Statistics are a very important component of scientific research. Statistical methods are used to design experiments and analyze data in order to draw meaningful interpretations of research findings. While statistics play an important role in research, the details of the statistical methods used in a study are often incomplete. Underspecified methods are a lot like providing only part of a recipe. To bake a loaf of bread, it isn’t enough to know what ingredients are needed. How much of each ingredient to add, in what order to add them, and the oven temperature are all important details needed to ensure a consistent outcome. When details aren’t clearly specified, the quality of the end-product will be variable. Baking a loaf of bread using an incomplete recipe may result in bread that’s overbaked, underdone or just right.
Like an incomplete recipe, methods that leave out important details of the analysis could significantly impact the results of future replication attempts and can make the findings difficult to interpret. Underspecified methods are a major factor contributing to irreproducible research. Irreproducible research is a major concern because invalid claims slow scientific progress, waste time and resources, and contribute to the public’s mistrust of science.
When reporting statistical analyses, it is critical to describe the statistical methods in enough detail so that other researchers can validate the study’s findings by precisely replicating the analyses in the original study. When reporting statistical results, it is important to provide enough detail so that readers can evaluate the credibility and usefulness of the information and make appropriate decisions. In this article, we discuss how to describe your statistical methods and results to ensure that other scientists are provided with the information needed to accurately repeat your analysis. Below we present a set of guidelines to consider when reporting your statistical methods and results.
Reporting statistical methods
- Sample size: The sample size tells us about the amount of information we have and, in part, determines the level of confidence that we have in our sample estimates. In the Methods section, provide and define the sample size used in the study and describe how it was determined. If the sample size was determined statistically (e.g., power analysis), describe the statistical method used. For sample size calculations, be sure to specify the inputs for power, effect size and alpha. If a sample size calculation was not performed, describe how the sample size was determined and provide a rationale for why the sample size was sufficient.
- Data transformations: If the raw data were transformed before analysis (e.g., normalization, log transformations, ratios), describe the procedure and provide a rationale for why the data were transformed.
- Statistical tests: Often in the Methods, you will see a list of statistical tests used, but the corresponding analysis for each test is not specified. It is impossible to know which test was used for each analysis when the methods are presented in this way. To ensure your methods can be precisely replicated, be sure that the methods include a detailed description of the statistical tests used for each analysis. Avoid just listing all statistical tests used in your study.
For each statistical analysis:
- Assumptions: Verify that the data conformed to the assumptions of the test used to analyze them. For instance, when using a parametric test (e.g. t test), demonstrate that the data are normally distributed.
- Directional tests: If applicable, indicate whether tests were one- or two-sided. When using a one-sided test, be sure to justify the use of a directional test. Imagine you want to test the hypothesis that an inexpensive drug is not less effective than a more expensive one. In this case, a one-tailed test would be justified because you only want to show that the drug is not less effective and do not care if it was more effective.
- Multiple comparisons: If the analysis involves multiple comparisons, indicate whether they were accounted for. If so, be sure to describe how adjustments for multiple comparisons were made (e.g., Bonferroni correction).
- Outliers: If relevant to your study, describe how outliers were treated. Unusual values within a dataset may distort statistical analyses, and it may be necessary to remove them. When removing outliers, be sure to describe how outliers were defined and explain why this procedure was legitimate.
- Alpha level: Always report the alpha level used to define statistical significance (e.g., p<0.05). Determining the statistical significance of a result depends on the alpha decided upon before you begin the experiment. Choosing the level of significance before you begin the experiment avoids any potential for biasing or statistical “cherry picking” of results.
- Software/tools: It is also important to name the statistical package or software used to perform the analyses in your study, specifying version number used (e.g., SPSS v. 21.0). The way analyses are implemented can vary across software packages, particularly for more complicated analyses. Thus, the results produced by one software package may not be exactly the same as those produced by another package. Precise software package information can be an important detail to include when providing a complete record of your statistical methods.
Reporting statistical results
Descriptive statistics: Descriptive statistics are used to summarize and describe the main features of a collection of data. Descriptive statistics allow us to visualize the data and present the data in a more meaningful way.
When presenting descriptive statistics:
- Sample size: Make sure the sample size (n) is indicated. Be sure to report the sample size for each analysis as a precise value, not a range.
- Percentages: Be sure to include the numerator and denominator when reporting all percentages.
- Measures of variability: Summarize the data using means and standard deviations when the data are normally distributed. The variability of the data should be reported using the standard deviation (SD), not the standard error (SE). The standard error is an estimate of how much the sample mean differs from the population mean and is therefore not an appropriate estimate of the variability among observations. The standard deviation is a measure of the amount of variation or dispersion of a set of values. For data that are not normally distributed, use medians and interpercentile ranges, ranges, or both.
Inferential statistics: Inferential statistics make inferences and predictions about a population based on a subset of sample observations, which are then used to make conclusions that extend beyond what can be shown with descriptive statistics alone.
When presenting inferential statistics:
- Hypothesis: Before presenting the results of a statistical test, be sure to state the hypothesis being tested and describe the purpose of the analysis.
- Variables: Identify the variables used in the analysis and summarize the data for each variable using the appropriate descriptive statistics.
- Sample size: Be sure to provide and describe the sample size. The sample size for each analysis should be reported as a precise value, not as a range. For example, if Group A consisted of 10 mice and Group B consisted of 13 mice, do not report the sample size as “n=10-13 mice”. Instead, clearly specify the precise sample size of each group (i.e., Group A, n=10 mice; Group B, n=13 mice).
- Statistical test: State the statistical test used. If applicable, indicate whether the test was one- or two-sided and describe any adjustments made for multiple comparisons.
- Results: Results derived from inferential statistics should include the test statistic, degrees of freedom, confidence interval, p value and effect size.
- P value: Whenever possible, report p values as precise values not a range (e.g. p=0.049 not p<0.05). Precise p values are especially important when reporting non-significant findings. Suppose the main results of a study are p=0.054 and p=0.54, but in the paper both results are simply reported as “not significant”. The interpretations of these two results differ, but this difference is not communicated to the readers because the precise p value was not provided. Report the precise p value of results that do not reach your established significance level; avoid reporting these results as p>0.05, “not significant” or NS.
Statistics are an important tool in scientific research. They are used to plan experiments, test hypotheses and interpret data. A detailed description of your study’s statistical methods and results presents research that is reproducible and provides your readers with a clear picture of what you did, why you did it and why it’s important.
For more information on statistical reporting, please refer to the SAMPL Guidelines https://www.equator-network.org/wp-content/uploads/2013/07/SAMPL-Guidelines-6-27-13.pdf
Summary
Statistics are a very important component of scientific research. While statistics play an important role in research, the details of the statistical methods used in a study are often incomplete. Underspecified methods are a lot like providing only part of a recipe. When details aren’t clearly specified, the quality of the end-product will be variable. In this article, we discuss how to describe your statistical methods and results to ensure that other scientists are provided with the information needed to accurately repeat your analysis.