PSY 626: Bayesian Statistics for Psychological Science

Presentación del tema: "PSY 626: Bayesian Statistics for Psychological Science"— Transcripción de la presentación:

1 PSY 626: Bayesian Statistics for Psychological Science
7/2/2018 Bayesian ANOVA Greg Francis PSY 626: Bayesian Statistics for Psychological Science Fall 2016 Purdue University PSY200 Cognitive Psychology

2 ANOVA Most widely used hypothesis test in the NHST framework
The Bayesian variation of ANOVA focuses (as it must) on model comparison Standard ANOVA often does the same kind of thing In violation of controlling the Type I error rate If there is really no difference between means, a 2x2 ANOVA has a 14% chance of reporting a significant difference for at least one of: Main effect 1 Main effect 2 Interaction Add in various contrasts, and the Type I error rate is much higher!

3 Search of memory How is memory searched?
7/2/2018 Search of memory How is memory searched? Explore by varying the number of items in memory set measure reaction time Sternberg (1969) NO 8 PSY200 Cognitive Psychology

4 7/2/2018 Search of memory Typical results: Parallel curves for “present” and “absent” targets Implications for how people search short term memory Average of 107 participants PSY200 Cognitive Psychology

5 Single subject Usually each (random) subject shows a similar pattern

6 ANOVA Use data from a single subject
Condition: Present, Absent MemorySetSize: 1, 3, 5 Dependent Variable: RT 10 trials for each combination of Condition x MemorySetSize 60 trials overall SSdata<-read.csv(file="OneSubject.csv",header=TRUE,stringsAsFactors=FALSE) # Treat MemorySetSize as a factor SSdata$MemorySetSize = factor(SSdata$MemorySetSize) levels(SSdata$MemorySetSize) = c("Small", "Medium", "Large") # Treat Condition as a factor SSdata$Condition = factor(SSdata$Condition)

7 NHST ANOVA Nothing is significant
Does not mean the effects are not there > summary(aov(RT ~ Condition*MemorySetSize, data=SSdata)) Df Sum Sq Mean Sq F value Pr(>F) Condition MemorySetSize Condition:MemorySetSize Residuals

8 BayesFactor Convenience function compares additive and interaction models to null # load the BayesFactor library library(BayesFactor) # run the BayesFactor ANOVA bf = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata) > bf Bayes factor analysis [1] MemorySetSize : ±0.01% [2] Condition : ±0% [3] MemorySetSize + Condition : ±1.63% [4] MemorySetSize + Condition + MemorySetSize:Condition : ±6.5% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

9 BayesFactor > bf Bayes factor analysis [1] MemorySetSize : ±0.01% [2] Condition : ±0% [3] MemorySetSize + Condition : ±1.63% [4] MemorySetSize + Condition + MemorySetSize:Condition : ±6.5% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS Convenience function compares additive and interaction models to null # load the BayesFactor library library(BayesFactor) # run the BayesFactor ANOVA bf = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata) Unlike for the t-tests, these calculations are based on sampling methods You get slightly different values for different “runs” bf = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata) |===========================================================================================================================================| 100% > bf Bayes factor analysis [1] MemorySetSize : ±0.01% [2] Condition : ±0% [3] MemorySetSize + Condition : ±1.68% [4] MemorySetSize + Condition + MemorySetSize:Condition : ±2.16% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

10 BayesFactor If you do not like this kind of varying response, you can “recompute” the models with more sampling > newbf = recompute(bf, iterations = ) Data anecdotally favors the null model compared to either main effect Data substantially favors the null model compared to additive models (without interaction) > newbf Bayes factor analysis [1] MemorySetSize : ±0.01% [2] Condition : ±0% [3] MemorySetSize + Condition : ±0.15% [4] MemorySetSize + Condition + MemorySetSize:Condition : ±0.42% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

11 BayesFactor You can compare and contrast different models
Additive vs. additive with interaction Substantial support for the simpler (no interaction) model > newbf[3]/newbf[4] Bayes factor analysis [1] MemorySetSize + Condition : ±0.45% Against denominator: RT ~ MemorySetSize + Condition + MemorySetSize:Condition --- Bayes factor type: BFlinearModel, JZS

12 BayesFactor Again for convenience, you can compare simpler models to the “full” model by removing one term (factor or interaction) bf3 = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata, whichModels="top", iterations=500000) > bf3 Bayes factor top-down analysis When effect is omitted from MemorySetSize + Condition + MemorySetSize:Condition , BF is... [1] Omit Condition:MemorySetSize : ±1.37% [2] Omit Condition : ±1.37% [3] Omit MemorySetSize : ±1.32% Against denominator: RT ~ MemorySetSize + Condition + MemorySetSize:Condition --- Bayes factor type: BFlinearModel, JZS

13 BayesFactor Likewise, for convenience, you can compare more complicated models to the “null” model by adding one term (factor or interaction) bf4 = anovaBF(RT ~ Condition*MemorySetSize, data=SSdata, whichModels=”bottom", iterations=500000) > bf4 Bayes factor analysis [1] MemorySetSize : ±0.01% [2] Condition : ±0% [3] MemorySetSize:Condition : ±0.04% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

14 Rethinking We can do the same kind of model comparisons with the rethinking package We just have to define each model in turn and then compare them library(rethinking) # load full data file SSdata<-read.csv(file="OneSubject.csv",header=TRUE,stringsAsFactors=FALSE) SSdata$TargetPresent <- ifelse(SSdata$Condition =="Present", 1, 0) SSdataNull <- data.frame(RT= SSdata$RT) SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a, a ~ dnorm(1000, 1000), sigma ~ dunif(0, 1000) ), data= SSdataNull ) > precis(SSmodelNull) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a sigma

15 Rethinking: MemorySetSize
SSdataMSS <- data.frame(RT= SSdata$RT, MemorySetSize=SSdata$MemorySetSize) SSmodelMSS <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*MemorySetSize, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataMSS ) > precis(SSmodelMSS) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a b sigma

16 Rethinking: Target condition
SSdataCondition <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent) SSmodelCondition <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*Condition, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataCondition ) > precis(SSmodelCondition ) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a b sigma

17 Rethinking: Additive model
SSdataAdditive <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize) SSmodelAdditive <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*MemorySetSize + c*Condition, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), c ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataAdditive ) > precis(SSmodelAdditive) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a b c sigma

18 Rethinking: Interaction model
SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize) SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a + b*MemorySetSize + c*Condition+ d*MemorySetSize*Condition, a ~ dnorm(1000, 1000), b ~ dnorm(0, 100), c ~ dnorm(0, 100), d ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteraction) Mean StdDev lower 0.89 upper 0.89 n_eff Rhat a b c d sigma

19 WAIC > compare(SSmodelInteraction, SSmodelAdditive , SSmodelCondition, SSmodelMSS, SSmodelNull) WAIC pWAIC dWAIC weight SE dSE SSmodelMSS NA SSmodelAdditive SSmodelInteraction SSmodelNull SSmodelCondition No clear favorite among the models > newbf Bayes factor analysis [1] MemorySetSize : ±0.01% [2] Condition : ±0% [3] MemorySetSize + Condition : ±0.15% [4] MemorySetSize + Condition + MemorySetSize:Condition : ±0.42% Against denominator: Intercept only --- Bayes factor type: BFlinearModel, JZS

20 Mixed design ANOVA Sternberg search experiment 113 participants

21 NHST SSdata<-read.csv(file="SternbergSearch.csv",header=TRUE,stringsAsFactors=FALSE) # Treat MemorySetSize as a factor SSdata$MemorySetSize = factor(SSdata$MemorySetSize) levels(SSdata$MemorySetSize) = c("Small", "Medium", "Large") # Treat Condition as a factor SSdata$Condition = factor(SSdata$Condition) # Treat Participants as a factor SSdata$Participant = factor(SSdata$Participant) # NHST ANOVA summary(aov(RT ~ Condition*MemorySetSize + Error(Participant/(Condition*MemorySetSize)), data=SSdata)) > summary(aov(RT ~ Condition*MemorySetSize + Error(Participant/(Condition*MemorySetSize)), data=SSdata)) Error: Participant Df Sum Sq Mean Sq F value Pr(>F) Residuals Error: Participant:Condition Df Sum Sq Mean Sq F value Pr(>F) Condition Residuals Error: Participant:MemorySetSize Df Sum Sq Mean Sq F value Pr(>F) MemorySetSize <2e-16 *** Residuals --- Signif. codes: 0 ‘***’ ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Error: Participant:Condition:MemorySetSize Df Sum Sq Mean Sq F value Pr(>F) Condition:MemorySetSize e-07 *** Residuals Error: Within Residuals

22 BayesFactor bf = anovaBF(RT ~ Condition*MemorySetSize + Participant, data=SSdata, whichRandom="Participant") Definitive evidence for an effect of memory set size Definitive evidence for no effect of target condition (present/absent) Definitive evidence for different slopes (whoops!) > bf Bayes factor analysis [1] MemorySetSize + Participant : e+126 ±1.27% [2] Condition + Participant : ±1.51% [3] MemorySetSize + Condition + Participant : e+125 ±2.43% [4] MemorySetSize + Condition + MemorySetSize:Condition + Participant : e+130 ±1.78% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS

23 BayesFactor bf2 = anovaBF(RT ~ Condition*MemorySetSize + Participant, data=SSdata, whichRandom="Participant", whichModels="all") Definitive evidence for an effect of memory set size Definitive evidence for no effect of target condition (present/absent) Definitive evidence for interaction with different slopes (whoops!) > bf2 Bayes factor analysis [1] MemorySetSize + Participant : e+126 ±0.59% [2] Condition + Participant : ±1.88% [3] MemorySetSize:Condition + Participant : ±1.18% [4] MemorySetSize + Condition + Participant : e+125 ±2.36% [5] MemorySetSize + MemorySetSize:Condition + Participant : e+132 ±1.23% [6] Condition + MemorySetSize:Condition + Participant : ±14.32% [7] MemorySetSize + Condition + MemorySetSize:Condition + Participant : e+130 ±1.94% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS

24 Rethinking Null model has different intercepts for each participant
May as well use shrinkage, though (why not?) SSdataNull <- data.frame(RT= SSdata$RT, Participant=SSdata$Participant) SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant], a[Participant] ~ dnorm(grand_mu, grand_s), grand_mu ~ dnorm(1000, 1000), grand_s ~ dunif(0, 2000), sigma ~ dunif(0, 1000) ), data= SSdataNull ) > precis(SSmodelNull) 113 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mu grand_s sigma

25 Rethinking Model with intercept and slope for Memory Set Size (with shrinkage) SSdataMSS <- data.frame(RT= SSdata$RT, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelMSS <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataMSS ) > precis(SSmodelMSS) 226 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua grand_sa grand_mub grand_sb sigma

26 Rethinking Model with intercept and slope for Target condition (with shrinkage) SSdataCondition <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, Participant=SSdata$Participant) SSmodelCondition <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataCondition ) > precis(SSmodelCondition) 226 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua grand_sa grand_mub grand_sb sigma

27 Rethinking Additive model (with shrinkage)
SSdataAdditive <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelAdditive <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), c[Participant] ~ dnorm(grand_muc, grand_sc), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataAdditive ) > precis(SSmodelAdditive) 339 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua grand_sa grand_mub grand_sb grand_muc grand_sc sigma

28 Rethinking > precis(SSmodelInteraction)
Additive model and interaction (with shrinkage) SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), c[Participant] ~ dnorm(grand_muc, grand_sc), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteraction) 452 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua grand_sa grand_mub grand_sb grand_muc grand_sc grand_mud grand_sd sigma

29 Rethinking Interaction only (and intercept) (with shrinkage)
SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteractionOnly <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteractionOnly) 226 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua grand_sa grand_mud grand_sd sigma

30 Rethinking Memory Set Size and Interaction (and intercept) (with shrinkage) SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteractionMMS <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteractionMMS) 339 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua grand_sa grand_mub grand_sb grand_mud grand_sd sigma

31 Rethinking Target Condition and Interaction (and intercept) (with shrinkage) SSdataInteraction <- data.frame(RT= SSdata$RT, Condition=SSdata$TargetPresent, MemorySetSize=SSdata$MemorySetSize, Participant=SSdata$Participant) SSmodelInteractionCondition <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), c[Participant] ~ dnorm(grand_muc, grand_sc), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) > precis(SSmodelInteractionCondition) 339 vector or matrix parameters omitted in display. Use depth=2 to show them. Mean StdDev lower 0.89 upper 0.89 n_eff Rhat grand_mua grand_sa grand_muc grand_sc grand_mud grand_sd sigma

32 > bf2 Bayes factor analysis [1] MemorySetSize + Participant : e+126 ±0.59% [2] Condition + Participant : ±1.88% [3] MemorySetSize:Condition + Participant : ±1.18% [4] MemorySetSize + Condition + Participant : e+125 ±2.36% [5] MemorySetSize + MemorySetSize:Condition + Participant : e+132 ±1.23% [6] Condition + MemorySetSize:Condition + Participant : ±14.32% [7] MemorySetSize + Condition + MemorySetSize:Condition + Participant : e+130 ±1.94% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS Model comparison compare(SSmodelInteractionCondition, SSmodelInteractionMMS, SSmodelInteractionOnly, SSmodelInteraction, SSmodelAdditive , SSmodelCondition, SSmodelMSS, SSmodelNull) WAIC pWAIC dWAIC weight SE dSE SSmodelInteraction NA SSmodelAdditive SSmodelInteractionMMS SSmodelMSS SSmodelInteractionCondition SSmodelInteractionOnly SSmodelCondition SSmodelNull Strongly favors model with differences in Memory Set Size, Target Condition, and Interaction (varying slopes)

33 Shrinkage I re-did the rethinking analyses, but without shrinkage
SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant], a[Participant] ~ dnorm(grand_mu, grand_s), grand_mu ~ dnorm(1000, 1000), grand_s ~ dunif(0, 2000), sigma ~ dunif(0, 1000) ), data= SSdataNull ) I re-did the rethinking analyses, but without shrinkage Null model SSmodelNull <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant], a[Participant] ~ dnorm(1000, 1000), sigma ~ dunif(0, 1000) ), data= SSdataNull )

34 Shrinkage I re-did the rethinking analyses, but without shrinkage
SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(grand_mua, grand_sa), b[Participant] ~ dnorm(grand_mub, grand_sb), c[Participant] ~ dnorm(grand_muc, grand_sc), d[Participant] ~ dnorm(grand_mud, grand_sd), grand_mua ~ dnorm(1000, 1000), grand_sa ~ dunif(0, 2000), grand_mub ~ dnorm(0, 100), grand_sb ~ dunif(0, 200), grand_muc ~ dnorm(0, 100), grand_sc ~ dunif(0, 200), grand_mud ~ dnorm(0, 100), grand_sd ~ dunif(0, 200), sigma ~ dunif(0, 1000) ), data= SSdataInteraction ) I re-did the rethinking analyses, but without shrinkage Additive effects with interaction SSmodelInteraction <- map2stan( alist( RT ~ dnorm(mu, sigma), mu <- a[Participant] + b[Participant]*MemorySetSize + c[Participant]*Condition+ d[Participant]*MemorySetSize*Condition, a[Participant] ~ dnorm(1000, 1000), b[Participant] ~ dnorm(0, 100), c[Participant] ~ dnorm(0, 100), d[Participant] ~ dnorm(0, 100), sigma ~ dunif(0, 1000) ), data= SSdataInteraction )

35 > bf2 Bayes factor analysis [1] MemorySetSize + Participant : e+126 ±0.59% [2] Condition + Participant : ±1.88% [3] MemorySetSize:Condition + Participant : ±1.18% [4] MemorySetSize + Condition + Participant : e+125 ±2.36% [5] MemorySetSize + MemorySetSize:Condition + Participant : e+132 ±1.23% [6] Condition + MemorySetSize:Condition + Participant : ±14.32% [7] MemorySetSize + Condition + MemorySetSize:Condition + Participant : e+130 ±1.94% Against denominator: RT ~ Participant --- Bayes factor type: BFlinearModel, JZS Model comparison compare(SSmodelInteractionCondition, SSmodelInteractionMMS, SSmodelInteractionOnly, SSmodelInteraction, SSmodelAdditive , SSmodelCondition, SSmodelMSS, SSmodelNull) WAIC pWAIC dWAIC weight SE dSE SSmodelAdditive NA SSmodelInteraction SSmodelInteractionMMS SSmodelMSS SSmodelInteractionCondition SSmodelInteractionOnly SSmodelCondition SSmodelNull Strongly favors additive model (with no interaction) WAIC pWAIC dWAIC weight SE dSE SSmodelInteraction NA SSmodelAdditive SSmodelInteractionMMS SSmodelMSS SSmodelInteractionCondition SSmodelInteractionOnly SSmodelCondition SSmodelNull

36 Model comparison Remember, all of these model comparisons are
Estimates Based on the models you are considering “Small world” conclusions Your conclusions could change with New data (are you sure you measured exactly what you wanted?) Different models (does the model really represent your ideas?) Different priors (how robust are your conclusions to the priors?)

37 Making decisions It is not clear that you have to make a decision
Many people want to use Bayes Factors (or WAIC) to guide decisions: Does the data favor one model or the other? Which model does best? It is not clear that you have to make a decision If you have to make a decision, then you probably need to consider other issues as well Next time, we will consider how to use models to make decisions