PSY 626: Bayesian Statistics for Psychological Science 7/2/2018 Bayesian ANOVA Greg Francis PSY 626: Bayesian Statistics for Psychological Science Fall 2016 Purdue University PSY200 Cognitive Psychology

ANOVA Most widely used hypothesis test in the NHST framework The Bayesian variation of ANOVA focuses (as it must) on model comparison Standard ANOVA often does the same kind of thing In violation of controlling the Type I error rate If there is really no difference between means, a 2x2 ANOVA has a 14% chance of reporting a significant difference for at least one of: Main effect 1 Main effect 2 Interaction Add in various contrasts, and the Type I error rate is much higher!

Search of memory How is memory searched? 7/2/2018 Search of memory How is memory searched? Explore by varying the number of items in memory set measure reaction time Sternberg (1969) NO 5 3 2 9 5 3 2 9 8 5 3 2 9 PSY200 Cognitive Psychology

7/2/2018 Search of memory Typical results: Parallel curves for “present” and “absent” targets Implications for how people search short term memory Average of 107 participants PSY200 Cognitive Psychology

Single subject Usually each (random) subject shows a similar pattern

ANOVA Use data from a single subject Condition: Present, Absent MemorySetSize: 1, 3, 5 Dependent Variable: RT 10 trials for each combination of Condition x MemorySetSize 60 trials overall SSdata<-read.csv(file="OneSubject.csv",header=TRUE,stringsAsFactors=FALSE) # Treat MemorySetSize as a factor SSdata$MemorySetSize = factor(SSdata$MemorySetSize) levels(SSdata$MemorySetSize) = c("Small", "Medium", "Large") # Treat Condition as a factor SSdata$Condition = factor(SSdata$Condition)

NHST ANOVA Nothing is significant Does not mean the effects are not there > summary(aov(RT ~ Condition*MemorySetSize, data=SSdata)) Df Sum Sq Mean Sq F value Pr(>F) Condition 1 81034 81034 1.043 0.312 MemorySetSize 2 325205 162603 2.092 0.133 Condition:MemorySetSize 2 32628 16314 0.210 0.811 Residuals 54 4197308 77728

BayesFactor Convenience function compares additive and interaction models to null # load the BayesFactor library library(BayesFactor) # run the BayesFactor ANOVA bf = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata) > bf Bayes factor analysis -------------- [1] MemorySetSize : 0.6504893 ±0.01% [2] Condition : 0.4044618 ±0% [3] MemorySetSize + Condition : 0.2673152 ±1.63% [4] MemorySetSize + Condition + MemorySetSize:Condition : 0.06960248 ±6.5% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

BayesFactor > bf Bayes factor analysis -------------- [1] MemorySetSize : 0.6504893 ±0.01% [2] Condition : 0.4044618 ±0% [3] MemorySetSize + Condition : 0.2673152 ±1.63% [4] MemorySetSize + Condition + MemorySetSize:Condition : 0.06960248 ±6.5% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS Convenience function compares additive and interaction models to null # load the BayesFactor library library(BayesFactor) # run the BayesFactor ANOVA bf = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata) Unlike for the t-tests, these calculations are based on sampling methods You get slightly different values for different “runs” bf = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata) |===========================================================================================================================================| 100% > bf Bayes factor analysis -------------- [1] MemorySetSize : 0.6504893 ±0.01% [2] Condition : 0.4044618 ±0% [3] MemorySetSize + Condition : 0.2721669 ±1.68% [4] MemorySetSize + Condition + MemorySetSize:Condition : 0.06359971 ±2.16% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

BayesFactor If you do not like this kind of varying response, you can “recompute” the models with more sampling > newbf = recompute(bf, iterations = 500000) Data anecdotally favors the null model compared to either main effect Data substantially favors the null model compared to additive models (without interaction) > newbf Bayes factor analysis -------------- [1] MemorySetSize : 0.6504893 ±0.01% [2] Condition : 0.4044618 ±0% [3] MemorySetSize + Condition : 0.2702162 ±0.15% [4] MemorySetSize + Condition + MemorySetSize:Condition : 0.06479797 ±0.42% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

BayesFactor You can compare and contrast different models Additive vs. additive with interaction Substantial support for the simpler (no interaction) model > newbf[3]/newbf[4] Bayes factor analysis -------------- [1] MemorySetSize + Condition : 4.170134 ±0.45% Against denominator: RT ~ MemorySetSize + Condition + MemorySetSize:Condition --- Bayes factor type: BFlinearModel, JZS

BayesFactor Again for convenience, you can compare simpler models to the “full” model by removing one term (factor or interaction) bf3 = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata, whichModels="top", iterations=500000) > bf3 Bayes factor top-down analysis -------------- When effect is omitted from MemorySetSize + Condition + MemorySetSize:Condition , BF is... [1] Omit Condition:MemorySetSize : 4.055128 ±1.37% [2] Omit Condition : 2.380806 ±1.37% [3] Omit MemorySetSize : 1.483591 ±1.32% Against denominator: RT ~ MemorySetSize + Condition + MemorySetSize:Condition --- Bayes factor type: BFlinearModel, JZS

BayesFactor Likewise, for convenience, you can compare more complicated models to the “null” model by adding one term (factor or interaction) bf4 = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata, whichModels=”bottom", iterations=500000) > bf4 Bayes factor analysis -------------- [1] MemorySetSize : 0.6504893 ±0.01% [2] Condition : 0.4044618 ±0% [3] MemorySetSize:Condition : 0.241134 ±0.04% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

Rethinking We can do the same kind of model comparisons with the rethinking package We just have to define each model in turn and then compare them library(rethinking) # load full data file SSdata<-read.csv(file="OneSubject.csv",header=TRUE,stringsAsFactors=FALSE) SSdata$TargetPresent <- ifelse(SSdata$Condition =="Present", 1, 0) SSdataNull <- data.frame(RT= SSdata$RT) SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a, a ~ dnorm(1000, 1000), sigma ~ dunif(0, 1000) ), data= SSdataNull ) > precis(SSmodelNull) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a 1106.72 37.06 1045.41 1161.17 619 1 sigma 286.12 26.00 248.08 328.40 629 1

Rethinking: MemorySetSize SSdataMSS <- data.frame(RT= SSdata$RT, MemorySetSize=SSdata$MemorySetSize) SSmodelMSS <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*MemorySetSize, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataMSS ) > precis(SSmodelMSS) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a 982.33 67.70 870.02 1086.58 340 1.00 b 41.80 19.31 16.46 77.49 330 1.00 sigma 278.59 24.48 238.20 312.02 386 1.01

Rethinking: Target condition SSdataCondition <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent) SSmodelCondition <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*Condition, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataCondition ) > precis(SSmodelCondition ) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a 1131.41 47.25 1062.45 1205.31 512 1 b -48.47 55.60 -134.77 40.37 566 1 sigma 286.37 27.77 244.52 328.87 593 1

Rethinking: Additive model SSdataAdditive <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize) SSmodelAdditive <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*MemorySetSize + c*Condition, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), c ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataAdditive ) > precis(SSmodelAdditive) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a 1005.56 72.52 882.75 1113.73 485 1 b 42.63 20.33 11.43 75.15 489 1 c -50.00 55.62 -143.98 31.56 666 1 sigma 279.05 25.45 244.45 323.63 559 1

Rethinking: Interaction model SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize) SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*MemorySetSize + c*Condition+ d*MemorySetSize*Condition, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), c ~ dnorm(0, 100), d ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteraction) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a 999.69 77.83 869.66 1112.78 457 1 b 42.96 24.17 3.09 78.99 427 1 c -49.63 77.60 -178.54 74.56 505 1 d 0.72 27.60 -46.16 42.22 460 1 sigma 279.18 25.66 242.75 316.75 604 1

WAIC > compare(SSmodelInteraction, SSmodelAdditive , SSmodelCondition, SSmodelMSS, SSmodelNull) WAIC pWAIC dWAIC weight SE dSE SSmodelMSS 846.9 2.6 0.0 0.37 11.50 NA SSmodelAdditive 847.3 3.2 0.4 0.31 11.81 1.53 SSmodelInteraction 848.7 3.9 1.8 0.15 12.08 1.60 SSmodelNull 849.5 1.9 2.6 0.10 11.31 4.00 SSmodelCondition 850.0 2.5 3.1 0.08 11.74 4.45 No clear favorite among the models > newbf Bayes factor analysis -------------- [1] MemorySetSize : 0.6504893 ±0.01% [2] Condition : 0.4044618 ±0% [3] MemorySetSize + Condition : 0.2702162 ±0.15% [4] MemorySetSize + Condition + MemorySetSize:Condition : 0.06479797 ±0.42% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

Mixed design ANOVA Sternberg search experiment 113 participants

NHST SSdata<-read.csv(file="SternbergSearch.csv",header=TRUE,stringsAsFactors=FALSE) # Treat MemorySetSize as a factor SSdata$MemorySetSize = factor(SSdata$MemorySetSize) levels(SSdata$MemorySetSize) = c("Small", "Medium", "Large") # Treat Condition as a factor SSdata$Condition = factor(SSdata$Condition) # Treat Participants as a factor SSdata$Participant = factor(SSdata$Participant) # NHST ANOVA summary(aov(RT ~ Condition*MemorySetSize + Error(Participant/(Condition*MemorySetSize)), data=SSdata)) > summary(aov(RT ~ Condition*MemorySetSize + Error(Participant/(Condition*MemorySetSize)), data=SSdata)) Error: Participant Df Sum Sq Mean Sq F value Pr(>F) Residuals 112 297873026 2659581 Error: Participant:Condition Df Sum Sq Mean Sq F value Pr(>F) Condition 1 441 441 0.003 0.954 Residuals 112 15105750 134873 Error: Participant:MemorySetSize Df Sum Sq Mean Sq F value Pr(>F) MemorySetSize 2 37548107 18774053 218.2 <2e-16 *** Residuals 224 19269553 86025 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Error: Participant:Condition:MemorySetSize Df Sum Sq Mean Sq F value Pr(>F) Condition:MemorySetSize 2 2221905 1110952 16.63 1.84e-07 *** Residuals 224 14962110 66795 Error: Within Residuals 6102 348154998 57056

BayesFactor bf = anovaBF(RT ~ Condition*MemorySetSize + Participant, data=SSdata, whichRandom="Participant") Definitive evidence for an effect of memory set size Definitive evidence for no effect of target condition (present/absent) Definitive evidence for different slopes (whoops!) > bf Bayes factor analysis -------------- [1] MemorySetSize + Participant : 5.633486e+126 ±1.27% [2] Condition + Participant : 0.0272587 ±1.51% [3] MemorySetSize + Condition + Participant : 1.517811e+125 ±2.43% [4] MemorySetSize + Condition + MemorySetSize:Condition + Participant : 5.143171e+130 ±1.78% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS

BayesFactor bf2 = anovaBF(RT ~ Condition*MemorySetSize + Participant, data=SSdata, whichRandom="Participant", whichModels="all") Definitive evidence for an effect of memory set size Definitive evidence for no effect of target condition (present/absent) Definitive evidence for interaction with different slopes (whoops!) > bf2 Bayes factor analysis -------------- [1] MemorySetSize + Participant : 5.519399e+126 ±0.59% [2] Condition + Participant : 0.02776611 ±1.88% [3] MemorySetSize:Condition + Participant : 70932.28 ±1.18% [4] MemorySetSize + Condition + Participant : 1.518957e+125 ±2.36% [5] MemorySetSize + MemorySetSize:Condition + Participant : 1.910749e+132 ±1.23% [6] Condition + MemorySetSize:Condition + Participant : 2186.301 ±14.32% [7] MemorySetSize + Condition + MemorySetSize:Condition + Participant : 5.138857e+130 ±1.94% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS

Rethinking Null model has different intercepts for each participant May as well use shrinkage, though (why not?) SSdataNull <- data.frame(RT= SSdata$RT, Participant=SSdata$Participant) SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant], a[Participant] ~ dnorm(grand_mu, grand_s), grand_mu ~ dnorm(1000, 1000), grand_s ~ dunif(0, 2000), sigma ~ dunif(0, 1000) ), data= SSdataNull ) > precis(SSmodelNull) 113 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mu 859.65 18.95 831.07 890.05 1000 1 grand_s 210.27 14.25 188.85 232.85 1000 1 sigma 256.06 2.29 252.63 259.71 1000 1

Rethinking Model with intercept and slope for Memory Set Size (with shrinkage) SSdataMSS <- data.frame(RT= SSdata$RT, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelMSS <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataMSS ) > precis(SSmodelMSS) 226 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua 723.28 20.85 690.60 755.65 1000 1.00 grand_sa 208.78 15.08 183.92 231.26 1000 1.00 grand_mub 45.26 2.41 41.77 49.44 1000 1.00 grand_sb 16.88 2.39 13.24 20.63 302 1.01 sigma 243.61 2.10 240.62 247.14 1000 1.00

Rethinking Model with intercept and slope for Target condition (with shrinkage) SSdataCondition <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, Participant=SSdata$Participant) SSmodelCondition <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataCondition ) > precis(SSmodelCondition) 226 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua 859.25 20.25 826.47 891.01 1000 1 grand_sa 210.06 13.92 189.57 234.19 1000 1 grand_mub -0.39 8.60 -13.63 13.20 733 1 grand_sb 67.24 9.02 51.96 81.24 232 1 sigma 253.97 2.20 250.83 257.55 1000 1

Rethinking Additive model (with shrinkage) SSdataAdditive <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelAdditive <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), c[Participant] ~ dnorm(grand_muc, grand_sc), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataAdditive ) > precis(SSmodelAdditive) 339 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua 723.82 20.80 692.90 758.66 1000 1 grand_sa 209.58 16.32 183.93 234.28 1000 1 grand_mub 45.21 2.40 41.30 48.99 811 1 grand_sb 17.01 2.60 12.96 21.03 253 1 grand_muc -0.33 9.10 -16.23 12.62 1000 1 grand_sc 70.72 8.48 57.02 83.09 533 1 sigma 240.91 2.10 237.87 244.57 1000 1

Rethinking > precis(SSmodelInteraction) Additive model and interaction (with shrinkage) SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), c[Participant] ~ dnorm(grand_muc, grand_sc), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteraction) 452 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua 755.47 21.49 718.58 787.80 1000 1.00 grand_sa 209.36 14.70 185.80 233.36 1000 1.00 grand_mub 34.79 2.95 29.64 39.11 114 1.00 grand_sb 16.12 2.58 12.22 20.26 118 1.00 grand_muc -61.93 13.50 -81.87 -40.32 49 1.00 grand_sc 64.40 8.94 51.03 79.23 152 1.01 grand_mud 20.53 3.81 14.49 26.42 43 1.00 grand_sd 9.99 3.55 4.38 15.34 24 1.03 sigma 240.17 2.17 236.23 243.20 1000 1.00

Rethinking Interaction only (and intercept) (with shrinkage) SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteractionOnly <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteractionOnly) 226 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua 828.91 20.66 795.75 860.36 1000 1.00 grand_sa 210.03 14.43 189.14 234.77 1000 1.00 grand_mud 20.57 2.25 16.67 23.87 597 1.01 grand_sd 16.94 2.24 13.34 20.26 227 1.00 sigma 251.16 2.28 247.82 254.99 1000 1.00

Rethinking Memory Set Size and Interaction (and intercept) (with shrinkage) SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteractionMMS <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteractionMMS) 339 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua 723.67 20.19 692.56 757.12 1000 1 grand_sa 209.02 15.12 186.33 233.31 1000 1 grand_mub 42.67 2.50 39.16 47.05 624 1 grand_sb 15.74 2.75 11.63 20.24 152 1 grand_mud 4.77 2.30 1.39 8.81 1000 1 grand_sd 17.42 2.37 13.75 21.10 287 1 sigma 241.48 2.20 237.93 244.82 1000 1

Rethinking Target Condition and Interaction (and intercept) (with shrinkage) SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteractionCondition <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), c[Participant] ~ dnorm(grand_muc, grand_sc), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteractionCondition) 339 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua 860.31 20.30 830.67 895.79 1000 1.00 grand_sa 211.12 14.97 188.58 235.92 1000 1.00 grand_muc -164.48 10.71 -182.50 -148.70 152 1.04 grand_sc 61.07 10.17 43.42 75.93 96 1.01 grand_mud 55.05 2.80 50.53 59.43 126 1.03 grand_sd 12.72 3.15 7.61 17.60 64 1.00 sigma 244.88 2.07 241.23 247.85 1000 1.00

> bf2 Bayes factor analysis -------------- [1] MemorySetSize + Participant : 5.519399e+126 ±0.59% [2] Condition + Participant : 0.02776611 ±1.88% [3] MemorySetSize:Condition + Participant : 70932.28 ±1.18% [4] MemorySetSize + Condition + Participant : 1.518957e+125 ±2.36% [5] MemorySetSize + MemorySetSize:Condition + Participant : 1.910749e+132 ±1.23% [6] Condition + MemorySetSize:Condition + Participant : 2186.301 ±14.32% [7] MemorySetSize + Condition + MemorySetSize:Condition + Participant : 5.138857e+130 ±1.94% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS Model comparison compare(SSmodelInteractionCondition, SSmodelInteractionMMS, SSmodelInteractionOnly, SSmodelInteraction, SSmodelAdditive , SSmodelCondition, SSmodelMSS, SSmodelNull) WAIC pWAIC dWAIC weight SE dSE SSmodelInteraction 93808.8 233.2 0.0 1 178.63 NA SSmodelAdditive 93841.9 224.7 33.0 0 177.89 12.57 SSmodelInteractionMMS 93864.5 214.7 55.7 0 177.91 15.18 SSmodelMSS 93919.9 160.5 111.0 0 176.06 23.33 SSmodelInteractionCondition 94022.5 192.4 213.6 0 174.29 30.12 SSmodelInteractionOnly 94341.3 167.0 532.5 0 168.19 47.85 SSmodelCondition 94493.2 168.2 684.3 0 165.55 55.33 SSmodelNull 94554.9 112.7 746.1 0 164.59 58.00 Strongly favors model with differences in Memory Set Size, Target Condition, and Interaction (varying slopes)

Shrinkage I re-did the rethinking analyses, but without shrinkage SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant], a[Participant] ~ dnorm(grand_mu, grand_s), grand_mu ~ dnorm(1000, 1000), grand_s ~ dunif(0, 2000), sigma ~ dunif(0, 1000) ), data= SSdataNull ) I re-did the rethinking analyses, but without shrinkage Null model SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant], a[Participant] ~ dnorm(1000, 1000), sigma ~ dunif(0, 1000) ), data= SSdataNull )

Shrinkage I re-did the rethinking analyses, but without shrinkage SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), c[Participant] ~ dnorm(grand_muc, grand_sc), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) I re-did the rethinking analyses, but without shrinkage Additive effects with interaction SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(1000, 1000), b[Participant] ~ dnorm(0, 100), c[Participant] ~ dnorm(0, 100), d[Participant] ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataInteraction )

> bf2 Bayes factor analysis -------------- [1] MemorySetSize + Participant : 5.519399e+126 ±0.59% [2] Condition + Participant : 0.02776611 ±1.88% [3] MemorySetSize:Condition + Participant : 70932.28 ±1.18% [4] MemorySetSize + Condition + Participant : 1.518957e+125 ±2.36% [5] MemorySetSize + MemorySetSize:Condition + Participant : 1.910749e+132 ±1.23% [6] Condition + MemorySetSize:Condition + Participant : 2186.301 ±14.32% [7] MemorySetSize + Condition + MemorySetSize:Condition + Participant : 5.138857e+130 ±1.94% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS Model comparison compare(SSmodelInteractionCondition, SSmodelInteractionMMS, SSmodelInteractionOnly, SSmodelInteraction, SSmodelAdditive , SSmodelCondition, SSmodelMSS, SSmodelNull) WAIC pWAIC dWAIC weight SE dSE SSmodelAdditive 93900.8 302.5 0.0 1 176.90 NA SSmodelInteraction 93929.4 365.5 28.6 0 176.99 17.76 SSmodelInteractionMMS 93955.0 325.4 54.1 0 176.49 19.92 SSmodelMSS 93974.8 220.8 74.0 0 174.86 25.94 SSmodelInteractionCondition 94161.9 278.8 261.1 0 172.10 45.21 SSmodelInteractionOnly 94403.1 224.6 502.2 0 168.76 55.18 SSmodelCondition 94500.0 190.9 599.1 0 165.76 56.60 SSmodelNull 94556.6 114.5 655.8 0 164.83 61.12 Strongly favors additive model (with no interaction) WAIC pWAIC dWAIC weight SE dSE SSmodelInteraction 93808.8 233.2 0.0 1 178.63 NA SSmodelAdditive 93841.9 224.7 33.0 0 177.89 12.57 SSmodelInteractionMMS 93864.5 214.7 55.7 0 177.91 15.18 SSmodelMSS 93919.9 160.5 111.0 0 176.06 23.33 SSmodelInteractionCondition 94022.5 192.4 213.6 0 174.29 30.12 SSmodelInteractionOnly 94341.3 167.0 532.5 0 168.19 47.85 SSmodelCondition 94493.2 168.2 684.3 0 165.55 55.33 SSmodelNull 94554.9 112.7 746.1 0 164.59 58.00

Model comparison Remember, all of these model comparisons are Estimates Based on the models you are considering “Small world” conclusions Your conclusions could change with New data (are you sure you measured exactly what you wanted?) Different models (does the model really represent your ideas?) Different priors (how robust are your conclusions to the priors?)

Making decisions It is not clear that you have to make a decision Many people want to use Bayes Factors (or WAIC) to guide decisions: Does the data favor one model or the other? Which model does best? It is not clear that you have to make a decision If you have to make a decision, then you probably need to consider other issues as well Next time, we will consider how to use models to make decisions