Effect Size Statistics for Anova Tables

This vignettes demontrates those functions of the sjstats-package that deal with Anova tables. These functions report different effect size measures, which are useful beyond significance tests (p-values), because they estimate the magnitude of effects, independent from sample size. sjstats provides following functions:

• eta_sq()
• omega_sq()
• cohens_f()
• anova_stats()

Befor we start, we fit a simple model:

library(sjstats)
data(efc)

# fit linear model
fit <- aov(
c12hour ~ as.factor(e42dep) + as.factor(c172code) + c160age,
data = efc
)

All functions accept objects of class aov or anova, so you can also use model fits from the car package, which allows fitting Anova’s with different types of sum of squares. Other objects, like lm, will be coerced to anova internally.

The following functions return the effect size statistic as named numeric vector, using the model’s term names.

Eta-Squared

The eta-squared is the proportion of the total variability in the dependent variable that is accounted for by the variation in the independent variable. It is the ratio of the sum of squares for each group level to the total sum of squares. It can be interpreted as percentage of variance accounted for by a variable.

For variables with 1 degree of freedeom (in the numerator), the square root of eta-squared is equal to the correlation coefficient r. For variables with more than 1 degree of freedom, eta-squared equals R2. This makes eta-squared easily interpretable. Furthermore, these effect sizes can easily be converted into effect size measures that can be, for instance, further processed in meta-analyses.

Eta-squared can be computed simply with:

eta_sq(fit)
#> # A tibble: 3 x 2
#>   term                  etasq
#>   <chr>                 <dbl>
#> 1 as.factor(e42dep)   0.266
#> 2 as.factor(c172code) 0.00540
#> 3 c160age             0.0484

Partial Eta-Squared

The partial eta-squared value is the ratio of the sum of squares for each group level to the sum of squares for each group level plus the residual sum of squares. It is more difficult to interpret, because its value strongly depends on the variability of the residuals. Partial eta-squared values should be reported with caution, and Levine and Hullett (2002) recommend reporting eta- or omega-squared rather than partial eta-squared.

Use the partial-argument to compute partial eta-squared values:

eta_sq(fit, partial = TRUE)
#> # A tibble: 3 x 2
#>   term                partial.etasq
#>   <chr>                       <dbl>
#> 1 as.factor(e42dep)         0.281
#> 2 as.factor(c172code)       0.00788
#> 3 c160age                   0.0665

Omega-Squared

While eta-squared estimates tend to be biased in certain situations, e.g. when the sample size is small or the independent variables have many group levels, omega-squared estimates are corrected for this bias.

Omega-squared can be simply computed with:

omega_sq(fit)
#> # A tibble: 3 x 2
#>   term                omegasq
#>   <chr>                 <dbl>
#> 1 as.factor(e42dep)   0.263
#> 2 as.factor(c172code) 0.00377
#> 3 c160age             0.0476

Partial Omega-Squared

omega_sq() also has a partial-argument to compute partial omega-squared values. Computing the partial omega-squared statistics is based on bootstrapping. In this case, use n to define the number of samples (1000 by default.)

omega_sq(fit, partial = TRUE, n = 100)
#> # A tibble: 3 x 2
#>   term                partial.omegasq
#>   <chr>                         <dbl>
#> 1 as.factor(e42dep)           0.278
#> 2 as.factor(c172code)         0.00547
#> 3 c160age                     0.0649

Cohen’s F

Finally, cohens_f() computes Cohen’s F effect size for all independent variables in the model:

cohens_f(fit)
#> # A tibble: 3 x 2
#>   term                cohens.f
#>   <chr>                  <dbl>
#> 1 as.factor(e42dep)     0.626
#> 2 as.factor(c172code)   0.0891
#> 3 c160age               0.267

Complete Statistical Table Output

The anova_stats() function takes a model input and computes a comprehensive summary, including the above effect size measures, returned as tidy data frame (as tibble, to be exact):

anova_stats(fit)
#> # A tibble: 4 x 12
#>   term                   df    sumsq  meansq statistic p.value    etasq partial.etasq  omegasq partial.omegasq cohens.f  power
#>   <chr>               <dbl>    <dbl>   <dbl>     <dbl>   <dbl>    <dbl>         <dbl>    <dbl>           <dbl>    <dbl>  <dbl>
#> 1 as.factor(e42dep)      3.  577756. 192585.    109.    0.      0.266         0.281    0.263           0.278     0.626   1.00
#> 2 as.factor(c172code)    2.   11722.   5861.      3.31  0.0370  0.00500       0.00800  0.00400         0.00500   0.0890  0.630
#> 3 c160age                1.  105170. 105170.     59.4   0.      0.0480        0.0660   0.0480          0.0650    0.267   1.00
#> 4 Residuals            834. 1476436.   1770.     NA    NA      NA            NA       NA              NA        NA      NA

Like the other functions, the input may also be an object of class anova, so you can also use model fits from the car package, which allows fitting Anova’s with different types of sum of squares:

anova_stats(car::Anova(fit, type = 3))
#> # A tibble: 5 x 12
#>   term                   sumsq  meansq    df statistic p.value    etasq partial.etasq  omegasq partial.omegasq cohens.f  power
#>   <chr>                  <dbl>   <dbl> <dbl>     <dbl>   <dbl>    <dbl>         <dbl>    <dbl>           <dbl>    <dbl>  <dbl>
#> 1 (Intercept)           26851.  26851.    1.     15.2    0.     0.0130        0.0180   0.0120          0.0170    0.135   0.973
#> 2 as.factor(e42dep)    426462. 142154.    3.     80.3    0.     0.209         0.224    0.206           0.220     0.537   1.00
#> 3 as.factor(c172code)    7352.   3676.    2.      2.08   0.126  0.00400       0.00500  0.00200         0.00300   0.0710  0.429
#> 4 c160age              105170. 105170.    1.     59.4    0.     0.0510        0.0660   0.0510          0.0650    0.267   1.00
#> 5 Residuals           1476436.   1770.  834.     NA     NA     NA            NA       NA              NA        NA      NA

Confidence Intervals

eta_sq() and omega_sq() have a ci.lvl-argument, which - if not NULL - also computes a confidence interval.

For eta-squared, i.e. eta_sq() with partial = FALSE, due to non-symmetry, confidence intervals are based on bootstrap-methods. Confidence intervals for partial omega-squared, i.e. omega_sq() with partial = TRUE - is also based on bootstrapping. In these cases, n indicates the number of bootstrap samples to be drawn to compute the confidence intervals.

For partial eta-squared, confidence intervals are based on get.ci.partial.eta.squared (package apaTables) and for omega-squared, confidence intervals are based on conf.limits.ncf (package MBESS).

References

Levine TR, Hullet CR. Eta Squared, Partial Eta Squared, and Misreporting of Effect Size in Communication Research. Human Communication Research 28(4); 2002: 612-625