Description of Experiment and Data
Singmann and Klauer (2011) were interested in whether or not conditional reasoning can be explained by a single process or whether multiple processes are necessary to explain it. To provide evidence for multiple processes we aimed to establish a double dissociation of two variables: instruction type and problem type. Instruction type was manipulated betweensubjects, one group of participants received deductive instructions (i.e., to treat the premises as given and only draw necessary conclusions) and a second group of participants received probabilistic instructions (i.e., to reason as in an everyday situation; we called this “inductive instruction” in the manuscript). Problem type consisted of two different orthogonally crossed variables that were manipulated withinsubjects, validity of the problem (formally valid or formally invalid) and plausibility of the problem (inferences which were consisted with the background knowledge versus problems that were inconsistent with the background knowledge). The critical comparison across the two conditions was among problems which were valid and implausible with problems that were invalid and plausible. For example, the next problem was invalid and plausible:
If a person is wet, then the person fell into a swimming pool.
A person fell into a swimming pool.
How valid is the conclusion/How likely is it that the person is wet?
For those problems we predicted that under deductive instructions responses should be lower (as the conclusion does not necessarily follow from the premises) as under probabilistic instructions. For the valid but implausible problem, an example is presented next, we predicted the opposite pattern:
If a person is wet, then the person fell into a swimming pool.
A person is wet.
How valid is the conclusion/How likely is it that the person fell into a swimming pool?
Our study also included valid and plausible and invalid and implausible problems.
In contrast to the analysis reported in the manuscript, we initially do not separate the analysis into affirmation and denial problems, but first report an analysis on the full set of inferences, MP, MT, AC, and DA, where MP and MT are valid and AC and DA invalid. We report a reanalysis of our Experiment 1 only. Note that the factor plausibility
is not present in the original manuscript, there it is a results of a combination of other factors.
Data and R Preperation
require(afex) # needed for ANOVA, emmeans is loaded automatically.
require(multcomp) # for advanced control for multiple testing/Type 1 errors.
data(sk2011.1)
str(sk2011.1)
## 'data.frame': 640 obs. of 9 variables:
## $ id : Factor w/ 40 levels "8","9","10","12",..: 3 3 3 3 3 3 3 3 3 3 ...
## $ instruction : Factor w/ 2 levels "deductive","probabilistic": 2 2 2 2 2 2 2 2 2 2 ...
## $ plausibility: Factor w/ 2 levels "plausible","implausible": 1 2 2 1 2 1 1 2 1 2 ...
## $ inference : Factor w/ 4 levels "MP","MT","AC",..: 4 2 1 3 4 2 1 3 4 2 ...
## $ validity : Factor w/ 2 levels "valid","invalid": 2 1 1 2 2 1 1 2 2 1 ...
## $ what : Factor w/ 2 levels "affirmation",..: 2 2 1 1 2 2 1 1 2 2 ...
## $ type : Factor w/ 2 levels "original","reversed": 2 2 2 2 1 1 1 1 2 2 ...
## $ response : int 100 60 94 70 100 99 98 49 82 50 ...
## $ content : Factor w/ 4 levels "C1","C2","C3",..: 1 1 1 1 2 2 2 2 3 3 ...
An important feature in the data is that each participant provided two responses for each cell of the design (the content is different for each of those, each participant saw all four contents). These two data points will be aggregated automatically by afex
.
with(sk2011.1, table(inference, id, plausibility))
## , , plausibility = plausible
##
## id
## inference 8 9 10 12 13 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
## MP 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## MT 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## AC 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## DA 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## id
## inference 37 38 39 40 41 42 43 44 46 47 48 49 50
## MP 2 2 2 2 2 2 2 2 2 2 2 2 2
## MT 2 2 2 2 2 2 2 2 2 2 2 2 2
## AC 2 2 2 2 2 2 2 2 2 2 2 2 2
## DA 2 2 2 2 2 2 2 2 2 2 2 2 2
##
## , , plausibility = implausible
##
## id
## inference 8 9 10 12 13 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
## MP 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## MT 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## AC 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## DA 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
## id
## inference 37 38 39 40 41 42 43 44 46 47 48 49 50
## MP 2 2 2 2 2 2 2 2 2 2 2 2 2
## MT 2 2 2 2 2 2 2 2 2 2 2 2 2
## AC 2 2 2 2 2 2 2 2 2 2 2 2 2
## DA 2 2 2 2 2 2 2 2 2 2 2 2 2
ANOVA
To get the full ANOVA table for the model, we simply pass it to aov_ez
(aov_car
or aov4
would be alternatives producing the same results) using the design as described above. We save the returned object for further analysis.
a1 < aov_ez("id", "response", sk2011.1, between = "instruction",
within = c("inference", "plausibility"))
## Warning: More than one observation per cell, aggregating the data using mean (i.e,
## fun_aggregate = mean)!
## Contrasts set to contr.sum for the following variables: instruction
a1 # the default print method prints a data.frame produced by nice
## Anova Table (Type 3 tests)
##
## Response: response
## Effect df MSE F ges p.value
## 1 instruction 1, 38 2027.42 0.31 .003 .58
## 2 inference 2.66, 101.12 959.12 5.81 ** .06 .002
## 3 instruction:inference 2.66, 101.12 959.12 6.00 ** .07 .001
## 4 plausibility 1, 38 468.82 34.23 *** .07 <.0001
## 5 instruction:plausibility 1, 38 468.82 10.67 ** .02 .002
## 6 inference:plausibility 2.29, 87.11 318.91 2.87 + .009 .06
## 7 instruction:inference:plausibility 2.29, 87.11 318.91 3.98 * .01 .02
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1
##
## Sphericity correction method: GG
As mentioned before, the two responses per cell of the design and participants are aggregated for the analysis as indicated by the warning message. Furthermore, the degrees of freedom are GreenhouseGeisser corrected per default for all effects involving inference
, as inference
is a withinsubject factor with more than two levels (i.e., MP, MT, AC, & DA). In line with our expectations, the threeway interaction is significant.
The object printed per default for afex_aov
objects (produced by nice
) can also be printed nicely using knitr
:
instruction 
1, 38 
2027.42 
0.31 
.003 
.58 
inference 
2.66, 101.12 
959.12 
5.81 ** 
.06 
.002 
instruction:inference 
2.66, 101.12 
959.12 
6.00 ** 
.07 
.001 
plausibility 
1, 38 
468.82 
34.23 *** 
.07 
<.0001 
instruction:plausibility 
1, 38 
468.82 
10.67 ** 
.02 
.002 
inference:plausibility 
2.29, 87.11 
318.91 
2.87 + 
.009 
.06 
instruction:inference:plausibility 
2.29, 87.11 
318.91 
3.98 * 
.01 
.02 
Alternatively, the anova
method for afex_aov
objects returns a data.frame
of class anova
that can be passed to, for example, xtable
for nice formatting:
print(xtable::xtable(anova(a1), digits = c(rep(2, 5), 3, 4)), type = "html")

num Df

den Df

MSE

F

ges

Pr(>F)

instruction

1.00

38.00

2027.42

0.31

0.003

0.5830

inference

2.66

101.12

959.12

5.81

0.063

0.0016

instruction:inference

2.66

101.12

959.12

6.00

0.065

0.0013

plausibility

1.00

38.00

468.82

34.23

0.068

0.0000

instruction:plausibility

1.00

38.00

468.82

10.67

0.022

0.0023

inference:plausibility

2.29

87.11

318.91

2.87

0.009

0.0551

instruction:inference:plausibility

2.29

87.11

318.91

3.98

0.013

0.0177

PostHoc Contrasts and Plotting
To further analyze the data we need to pass it to package emmeans
, a package that offers great functionality for both plotting and contrasts of all kind. A lot of information on emmeans
can be obtained in its vignettes and faq. emmeans
can work with afex_aov
objects directly as afex comes with the necessary methods for the generic functions defined in emmeans
. emmeans
uses the ANOVA model estimated via base R’s aov
function that is part of an afex_aov
object.
Some First Contrasts
Main Effects Only
This object can now be passed to emmeans
, for example to obtain the marginal means of the four inferences:
m1 < emmeans(a1, ~ inference)
## NOTE: Results may be misleading due to involvement in interactions
## inference emmean SE df lower.CL upper.CL
## MP 87.51250 3.783074 126.87 80.02641 94.99859
## MT 76.68125 3.783074 126.87 69.19516 84.16734
## AC 69.41250 3.783074 126.87 61.92641 76.89859
## DA 82.95625 3.783074 126.87 75.47016 90.44234
##
## Results are averaged over the levels of: instruction, plausibility
## Confidence level used: 0.95
This object can now also be used to compare whether or not there are differences between the levels of the factor:
## contrast estimate SE df t.ratio p.value
## MP  MT 10.83125 4.611854 114 2.349 0.0933
## MP  AC 18.10000 4.611854 114 3.925 0.0008
## MP  DA 4.55625 4.611854 114 0.988 0.7566
## MT  AC 7.26875 4.611854 114 1.576 0.3963
## MT  DA 6.27500 4.611854 114 1.361 0.5266
## AC  DA 13.54375 4.611854 114 2.937 0.0206
##
## Results are averaged over the levels of: instruction, plausibility
## P value adjustment: tukey method for comparing a family of 4 estimates
To obtain more powerful pvalue adjustments, we can furthermore pass it to multcomp
(Bretz, Hothorn, & Westfall, 2011):
summary(as.glht(pairs(m1)), test=adjusted("free"))
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## MP  MT == 0 10.831 4.612 2.349 0.068970 .
## MP  AC == 0 18.100 4.612 3.925 0.000806 ***
## MP  DA == 0 4.556 4.612 0.988 0.325273
## MT  AC == 0 7.269 4.612 1.576 0.281756
## MT  DA == 0 6.275 4.612 1.361 0.296932
## AC  DA == 0 13.544 4.612 2.937 0.017639 *
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported  free method)
A Simple interaction
We could now also be interested in the marginal means of the inferences across the two instruction types. emmeans
offers two ways to do so. The first splits the contrasts across levels of the factor.
m2 < emmeans(a1, ~ inferenceinstruction)
## NOTE: Results may be misleading due to involvement in interactions
## instruction = deductive:
## inference emmean SE df lower.CL upper.CL
## MP 97.2875 5.350074 126.87 86.70057 107.87443
## MT 70.4000 5.350074 126.87 59.81307 80.98693
## AC 61.4875 5.350074 126.87 50.90057 72.07443
## DA 81.8125 5.350074 126.87 71.22557 92.39943
##
## instruction = probabilistic:
## inference emmean SE df lower.CL upper.CL
## MP 77.7375 5.350074 126.87 67.15057 88.32443
## MT 82.9625 5.350074 126.87 72.37557 93.54943
## AC 77.3375 5.350074 126.87 66.75057 87.92443
## DA 84.1000 5.350074 126.87 73.51307 94.68693
##
## Results are averaged over the levels of: plausibility
## Confidence level used: 0.95
Consequently test are also only performed within each level:
## instruction = deductive:
## contrast estimate SE df t.ratio p.value
## MP  MT 26.8875 6.522147 114 4.122 0.0004
## MP  AC 35.8000 6.522147 114 5.489 <.0001
## MP  DA 15.4750 6.522147 114 2.373 0.0882
## MT  AC 8.9125 6.522147 114 1.366 0.5229
## MT  DA 11.4125 6.522147 114 1.750 0.3031
## AC  DA 20.3250 6.522147 114 3.116 0.0122
##
## instruction = probabilistic:
## contrast estimate SE df t.ratio p.value
## MP  MT 5.2250 6.522147 114 0.801 0.8538
## MP  AC 0.4000 6.522147 114 0.061 0.9999
## MP  DA 6.3625 6.522147 114 0.976 0.7636
## MT  AC 5.6250 6.522147 114 0.862 0.8241
## MT  DA 1.1375 6.522147 114 0.174 0.9981
## AC  DA 6.7625 6.522147 114 1.037 0.7282
##
## Results are averaged over the levels of: plausibility
## P value adjustment: tukey method for comparing a family of 4 estimates
The second version treats all factor combinations together, producing a considerably larger number of pairwise comparisons:
m3 < emmeans(a1, ~ inference:instruction)
## NOTE: Results may be misleading due to involvement in interactions
## inference instruction emmean SE df lower.CL upper.CL
## MP deductive 97.2875 5.350074 126.87 86.70057 107.87443
## MT deductive 70.4000 5.350074 126.87 59.81307 80.98693
## AC deductive 61.4875 5.350074 126.87 50.90057 72.07443
## DA deductive 81.8125 5.350074 126.87 71.22557 92.39943
## MP probabilistic 77.7375 5.350074 126.87 67.15057 88.32443
## MT probabilistic 82.9625 5.350074 126.87 72.37557 93.54943
## AC probabilistic 77.3375 5.350074 126.87 66.75057 87.92443
## DA probabilistic 84.1000 5.350074 126.87 73.51307 94.68693
##
## Results are averaged over the levels of: plausibility
## Confidence level used: 0.95
## contrast estimate SE df t.ratio p.value
## MP,deductive  MT,deductive 26.8875 6.522147 114.00 4.122 0.0018
## MP,deductive  AC,deductive 35.8000 6.522147 114.00 5.489 <.0001
## MP,deductive  DA,deductive 15.4750 6.522147 114.00 2.373 0.2649
## MP,deductive  MP,probabilistic 19.5500 7.566147 126.87 2.584 0.1716
## MP,deductive  MT,probabilistic 14.3250 7.566147 126.87 1.893 0.5581
## MP,deductive  AC,probabilistic 19.9500 7.566147 126.87 2.637 0.1527
## MP,deductive  DA,probabilistic 13.1875 7.566147 126.87 1.743 0.6592
## MT,deductive  AC,deductive 8.9125 6.522147 114.00 1.366 0.8704
## MT,deductive  DA,deductive 11.4125 6.522147 114.00 1.750 0.6548
## MT,deductive  MP,probabilistic 7.3375 7.566147 126.87 0.970 0.9779
## MT,deductive  MT,probabilistic 12.5625 7.566147 126.87 1.660 0.7125
## MT,deductive  AC,probabilistic 6.9375 7.566147 126.87 0.917 0.9839
## MT,deductive  DA,probabilistic 13.7000 7.566147 126.87 1.811 0.6141
## AC,deductive  DA,deductive 20.3250 6.522147 114.00 3.116 0.0462
## AC,deductive  MP,probabilistic 16.2500 7.566147 126.87 2.148 0.3906
## AC,deductive  MT,probabilistic 21.4750 7.566147 126.87 2.838 0.0948
## AC,deductive  AC,probabilistic 15.8500 7.566147 126.87 2.095 0.4239
## AC,deductive  DA,probabilistic 22.6125 7.566147 126.87 2.989 0.0644
## DA,deductive  MP,probabilistic 4.0750 7.566147 126.87 0.539 0.9994
## DA,deductive  MT,probabilistic 1.1500 7.566147 126.87 0.152 1.0000
## DA,deductive  AC,probabilistic 4.4750 7.566147 126.87 0.591 0.9989
## DA,deductive  DA,probabilistic 2.2875 7.566147 126.87 0.302 1.0000
## MP,probabilistic  MT,probabilistic 5.2250 6.522147 114.00 0.801 0.9928
## MP,probabilistic  AC,probabilistic 0.4000 6.522147 114.00 0.061 1.0000
## MP,probabilistic  DA,probabilistic 6.3625 6.522147 114.00 0.976 0.9770
## MT,probabilistic  AC,probabilistic 5.6250 6.522147 114.00 0.862 0.9887
## MT,probabilistic  DA,probabilistic 1.1375 6.522147 114.00 0.174 1.0000
## AC,probabilistic  DA,probabilistic 6.7625 6.522147 114.00 1.037 0.9677
##
## Results are averaged over the levels of: plausibility
## P value adjustment: tukey method for comparing a family of 8 estimates
Running Custom Contrasts
Objects returned from emmeans
can also be used to test specific contrasts. For this, we can simply create a list, where each element corresponds to one contrasts. A contrast is defined as a vector of constants on the reference grid (i.e., the object returned from emmeans
, here m3
). For example, we might be interested in whether there is a difference between the valid and invalid inferences in each of the two conditions.
c1 < list(
v_i.ded = c(0.5, 0.5, 0.5, 0.5, 0, 0, 0, 0),
v_i.prob = c(0, 0, 0, 0, 0.5, 0.5, 0.5, 0.5)
)
contrast(m3, c1, adjust = "holm")
## contrast estimate SE df t.ratio p.value
## v_i.ded 12.19375 4.611854 114 2.644 0.0187
## v_i.prob 0.36875 4.611854 114 0.080 0.9364
##
## Results are averaged over the levels of: plausibility
## P value adjustment: holm method for 2 tests
summary(as.glht(contrast(m3, c1)), test =adjusted("free"))
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## v_i.ded == 0 12.1937 4.6119 2.644 0.0186 *
## v_i.prob == 0 0.3687 4.6119 0.080 0.9364
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported  free method)
The results can be interpreted as in line with expectations. Responses are larger for valid than invalid problems in the deductive, but not the probabilistic condition.
Plotting
Function emmip
from package emmeans
can be used for plotting the data directly from an afex_aov
object. As said initially, we are interested in the threeway interaction of instruction with inference, plausibility, and instruction. We show a few variants of this plot in the following. emmeans
supports plotting via either lattice
or ggplot2
. The default plot function uses ggplot2
, which also supports model based confidence intervals (i.e., these intervals cannot be used to gauge whether or not two conditions differ).
emmip(a1, instruction ~ inferenceplausibility, CIs = TRUE)
We can also pass a different theme for ggplot2
, if we do not like the default one:
emmip(a1, instruction ~ inferenceplausibility, CIs = TRUE) +
ggplot2::theme_light()
One downside of the ggplot2
version is that it appears to always distinguish the different lines by color and does not appear to support distinguishing the lines by linetype. For this, we can use lattice
if we set the appropriate options:
require(lattice) # for plots
## Loading required package: lattice
lattice.options(default.theme = standard.theme(color = FALSE)) # black and white
lattice.options(default.args = list(as.table = TRUE)) # better ordering
emmip(a1, instruction ~ inferenceplausibility, engine = "lattice")
Replicate Analysis from Singmann and Klauer (2011)
As this plot is not very helpful, we now fit a new ANOVA model in which we separate the data in affirmation and denial inferences, as done in the original manuscript and plot the data then a second time.
a2 < aov_ez("id", "response", sk2011.1, between = "instruction",
within = c("validity", "plausibility", "what"))
## Warning: More than one observation per cell, aggregating the data using mean (i.e,
## fun_aggregate = mean)!
## Contrasts set to contr.sum for the following variables: instruction
## Anova Table (Type 3 tests)
##
## Response: response
## Effect df MSE F ges p.value
## 1 instruction 1, 38 2027.42 0.31 .003 .58
## 2 validity 1, 38 678.65 4.12 * .01 .05
## 3 instruction:validity 1, 38 678.65 4.65 * .01 .04
## 4 plausibility 1, 38 468.82 34.23 *** .07 <.0001
## 5 instruction:plausibility 1, 38 468.82 10.67 ** .02 .002
## 6 what 1, 38 660.52 0.22 .0007 .64
## 7 instruction:what 1, 38 660.52 2.60 .008 .11
## 8 validity:plausibility 1, 38 371.87 0.14 .0002 .71
## 9 instruction:validity:plausibility 1, 38 371.87 4.78 * .008 .04
## 10 validity:what 1, 38 1213.14 9.80 ** .05 .003
## 11 instruction:validity:what 1, 38 1213.14 8.60 ** .05 .006
## 12 plausibility:what 1, 38 204.54 9.97 ** .009 .003
## 13 instruction:plausibility:what 1, 38 204.54 5.23 * .005 .03
## 14 validity:plausibility:what 1, 38 154.62 0.03 <.0001 .85
## 15 instruction:validity:plausibility:what 1, 38 154.62 0.42 .0003 .52
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '+' 0.1 ' ' 1
Then we plot the data from this ANOVA. Note that we again use the lattice
engine to pass corrected xaxis labels directly.
emmip(a2, ~instruction ~ plausibility+validitywhat,
engine = "lattice",
scales = list(x=list(
at = 1:4,
labels = c("pl:v", "im:v", "pl:i", "im:i")
)))
We see the critical predicted crossover interaction in the left of those two graphs. For valid but implausible problems (im:v
) deductive responses are larger than probabilistic responses. The opposite is true for invalid but plausible problems (pl:i
). We now tests these differences at each of the four xaxis ticks in each plot using custom contrasts (diff_1
to diff_4
). Furthermore, we test for a validity effect and plausibility effect in both conditions.
(m4 < emmeans(a2, ~instruction+plausibility+validitywhat))
## what = affirmation:
## instruction plausibility validity emmean SE df lower.CL upper.CL
## deductive plausible valid 99.475 6.01019 183.89 87.6172 111.3328
## probabilistic plausible valid 95.300 6.01019 183.89 83.4422 107.1578
## deductive implausible valid 95.100 6.01019 183.89 83.2422 106.9578
## probabilistic implausible valid 60.175 6.01019 183.89 48.3172 72.0328
## deductive plausible invalid 66.950 6.01019 183.89 55.0922 78.8078
## probabilistic plausible invalid 90.550 6.01019 183.89 78.6922 102.4078
## deductive implausible invalid 56.025 6.01019 183.89 44.1672 67.8828
## probabilistic implausible invalid 64.125 6.01019 183.89 52.2672 75.9828
##
## what = denial:
## instruction plausibility validity emmean SE df lower.CL upper.CL
## deductive plausible valid 70.550 6.01019 183.89 58.6922 82.4078
## probabilistic plausible valid 92.975 6.01019 183.89 81.1172 104.8328
## deductive implausible valid 70.250 6.01019 183.89 58.3922 82.1078
## probabilistic implausible valid 72.950 6.01019 183.89 61.0922 84.8078
## deductive plausible invalid 86.525 6.01019 183.89 74.6672 98.3828
## probabilistic plausible invalid 87.450 6.01019 183.89 75.5922 99.3078
## deductive implausible invalid 77.100 6.01019 183.89 65.2422 88.9578
## probabilistic implausible invalid 80.750 6.01019 183.89 68.8922 92.6078
##
## Confidence level used: 0.95
c2 < list(
diff_1 = c(1, 1, 0, 0, 0, 0, 0, 0),
diff_2 = c(0, 0, 1, 1, 0, 0, 0, 0),
diff_3 = c(0, 0, 0, 0, 1, 1, 0, 0),
diff_4 = c(0, 0, 0, 0, 0, 0, 1, 1),
val_ded = c(0.5, 0, 0.5, 0, 0.5, 0, 0.5, 0),
val_prob = c(0, 0.5, 0, 0.5, 0, 0.5, 0, 0.5),
plau_ded = c(0.5, 0, 0.5, 0, 0.5, 0, 0.5, 0),
plau_prob = c(0, 0.5, 0, 0.5, 0, 0.5, 0, 0.5)
)
contrast(m4, c2, adjust = "holm")
## what = affirmation:
## contrast estimate SE df t.ratio p.value
## diff_1 4.1750 8.499692 183.89 0.491 1.0000
## diff_2 34.9250 8.499692 183.89 4.109 0.0004
## diff_3 23.6000 8.499692 183.89 2.777 0.0303
## diff_4 8.1000 8.499692 183.89 0.953 1.0000
## val_ded 35.8000 6.877109 70.38 5.206 <.0001
## val_prob 0.4000 6.877109 70.38 0.058 1.0000
## plau_ded 3.2750 3.627998 64.94 0.903 1.0000
## plau_prob 30.7750 4.102924 65.86 7.501 <.0001
##
## what = denial:
## contrast estimate SE df t.ratio p.value
## diff_1 22.4250 8.499692 183.89 2.638 0.0633
## diff_2 2.7000 8.499692 183.89 0.318 1.0000
## diff_3 0.9250 8.499692 183.89 0.109 1.0000
## diff_4 3.6500 8.499692 183.89 0.429 1.0000
## val_ded 11.4125 6.877109 70.38 1.659 0.6088
## val_prob 1.1375 6.877109 70.38 0.165 1.0000
## plau_ded 4.5625 3.627998 64.94 1.258 1.0000
## plau_prob 13.3625 4.102924 65.86 3.257 0.0143
##
## P value adjustment: holm method for 8 tests
As the resulting eight contrasts have different numbers of degreesoffreedom, we can only pass them to multcomp
in small batches. This gives us more powerful Type 1 error corrections but overall a reduced correction as we now control for three families of tests (i.e., overall Type 1 error probability of .15).
summary(as.glht(contrast(m4, c2[1:4])), test =adjusted("free"))
## Note: df set to 184
## $`what = affirmation`
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## diff_1 == 0 4.175 8.500 0.491 0.623874
## diff_2 == 0 34.925 8.500 4.109 0.000212 ***
## diff_3 == 0 23.600 8.500 2.777 0.017403 *
## diff_4 == 0 8.100 8.500 0.953 0.564739
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported  free method)
##
##
## $`what = denial`
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## diff_1 == 0 22.425 8.500 2.638 0.0331 *
## diff_2 == 0 2.700 8.500 0.318 0.9554
## diff_3 == 0 0.925 8.500 0.109 0.9554
## diff_4 == 0 3.650 8.500 0.429 0.9554
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported  free method)
summary(as.glht(contrast(m4, c2[5:6])), test =adjusted("free"))
## Note: df set to 70
## $`what = affirmation`
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## val_ded == 0 35.800 6.877 5.206 3.69e06 ***
## val_prob == 0 0.400 6.877 0.058 0.954
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported  free method)
##
##
## $`what = denial`
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## val_ded == 0 11.412 6.877 1.659 0.192
## val_prob == 0 1.137 6.877 0.165 0.869
## (Adjusted p values reported  free method)
summary(as.glht(contrast(m4, c2[7:8])), test =adjusted("free"))
## Note: df set to 65
## $`what = affirmation`
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## plau_ded == 0 3.275 3.628 0.903 0.37
## plau_prob == 0 30.775 4.103 7.501 4.5e10 ***
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported  free method)
##
##
## $`what = denial`
##
## Simultaneous Tests for General Linear Hypotheses
##
## Linear Hypotheses:
## Estimate Std. Error t value Pr(>t)
## plau_ded == 0 4.562 3.628 1.258 0.21304
## plau_prob == 0 13.362 4.103 3.257 0.00358 **
## 
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported  free method)
The pattern for the affirmation problems is in line with the expectations: We find the predicted differences between the instruction types for valid and implausible (diff_2
) and invalid and plausible (diff_3
) and the predicted nondifferences for the other two problems (diff_1
and diff_4
). Furthermore, we find a validity effect in the deductive but not in the probabilistic condition. Likewise, we find a plausibility effect in the probabilistic but not in the deductive condition.