Recently, I am working on a paper that based on an apprentice project I did before. I cannot remember how, but the concept of "power analysis" came into my mind. It seems that power analysis is very important in statistical analysis, but ignored by many, if not most, educational researchers. "[I]t is extremely surprising that very few researchers conduct and report power analyses for their studies (Brewer, 1972; Cohen, 1962, 1965, 1988, 1992; Keselman et al., 1998; Onwuegbuzie, 2002; Sherron, 1988) even though statistical power has been promoted actively since the 1960s (Cohen, 1962, 1965, 1969) and even though for many types of statistical analyses (e.g., r, z, F, ?2), tables have been provided by Cohen (1988, 1992) to determine the necessary sample size. Even when a priori power has been calculated, it is rarely reported (Wooley & Dawson, 1983)." (Onwuegbuzie & Leech, 2004, p. 207)
The "power analysis" concept was less or even not talked about in my previous research methods and statistics courses. Here are some reasons why power analyses were less used or reported (Onwuegbuzie & Leech, 2004).
- Researchers do not sufficiently understand the concept of statistical power.
- The concept and applications of power are not taught, or adequately covered, in many undergraduate- and graduate-level statistical courses. And, power is not recognized as important as other concepts.
- No sufficient information is provided on how to report statistical power.
- Research resource constraints do not allow research have enough sample size as required by the result of a priori power analysis.
- It is difficulty to estimate effect sizes and standard deviations before conducting a research because of the uncertainties involved.
- SPSS, SAS, and other software package do not have the function of conducting power analyses. Users need to use other software to do that. And, software for power analysis normally do not do other analysis.
- The power of a statistical test of a null hypothesis is the probability that it will lead to the rejection of the null hypothesis, i.e., the probability that it will result in the conclusion that the phenomenon exists. (Cohen, 1988, p. 4)
- Power is the probability of detecting an effect, given that the effect is really there. In other words, it is the probability of rejecting the null hypothesis when it is in fact false. (UCLA: Academic Technology Services, Statistical Consulting Group)
Why it should be reported
- It clearly represents a vital piece of information about a statistical test applied to research data. (Cohen, 1988, p. 4)
- APA publication manual requires reporting power analysis. (see APA publication manual 5th edition on page 24)
- Post hoc power analyses can be used to improve the design of independent replications (Onwuegbuzie & Leech, 2004, p. 225).
A priori or post hoc?
When I first touched the term "power analysis", I thought, aha, I can use the power analysis result to prove how confident (or correct) I was with the result of my statistical analysis, especially if I got a significant result (p < .05). However, this thought is wrong. First, many researchers did not like the idea of performing power analysis after the data has been collected and analyzed, they call it "post hoc power analysis". They propose a priori power analysis, meaning that power analysis should be performed as a part of research plan. Second, even post hoc power analysis is favored by some researchers, most of them suggest reporting post hoc power analysis only when there is a non-significant result.
- For the situations where significant statistical results were gained, what should we do? The answer is reporting effect size and confidence interval (CI) around effect size. (see Onwuegbuzie & Leech, 2004)
- A priori power analyses should be conducted and reported; post hoc analyses should never be used to replace a priori analyses (Onwuegbuzie & Leech, 2004).
- Post hoc power analyses should accompany statistically non-significant findings. Statistically non-significant results in a study with high power contribute to the body of knowledge because power can be ruled out as a threat to internal validity. (Onwuegbuzie & Leech, 2004, p. 219, p. 210)
How to Perform Power Analysis
- The most primary method is using the tables Cohen (1988) presented. This method sounds simple but actually complex, and not so "automaticly".
- Using Power Analysis Software
- How to conduct post hoc power analysis for multivariate tests such as multivariate analysis of variance and multivariate analysis of covariance? - Please see Onwuegbuzie and Leech, 2004.
- Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences(2nd Ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
- Onwuegbuzie, A. J., & Leech, N. L. (2004). Post hoc power: A concept whose time has come. Understanding Statistics, 3, 2001-230. - Available at UT online resources.
- UCLA: Academic Technology Services, Statistical Consulting Group. Statistical Computing Seminars: Introduction to Power Analysis. - The References section of this page has many useful resources.
- StatSoft, Inc.. Power Analysis.