MlWin gives different results to Stata/R

[KatrijnDelar]

I carried out a multilevel analysis in MlWin, using the Markov Chain Monte Carlo estimation procedure. The MCMC estimation procedure is used to take into account the cross-classified structure (see attachment).

I calculated the same model in R and Stata. Some of the fixed effects are quite similar, but the effects regarding temporal variation (age and cohort) and parental education differ (as you can see in the table: see attachment).

Does anyone have an explanation?
It is noteworthy that there is a very high correlation between cohort (level-2 characteristic) and age (level-1 characteristic): r = 0.97 . However, this is normally not a problem for the type of analysis I conducted, that is a hierarchical age-period-cohort analysis, as carried out by Andrew Bell (Life-course and cohort trajectories of mental health in the UK, 1991-2008 – A multilevel age-period-cohort analysis)

----------


(a) Stata – command: xtmixed health gender ib0.Parentaleducation c.education##c.cohort c.education##c.age || _all: R.cntry|| _all: R.cohorts|| periodcountry:, ml var
(b) R – command: model1_MCMC <- MCMCglmm (health ~1 + gender + factor(Parentaleducation) + education + age + cohort + education*cohort + education*age, random = ~ cntry + periodcountry + cohorts, nitt=20000, burnin = 2000, data = data)

[ChrisCharlton]

Which version of MLwiN are you using (there was a bug in 2.33 that would cause incorrect results with MCMC if the data contained missing values)? Have you checked the MCMC convergence criteria to ensure that the chains are stable and have been run for long enough? Could you provide further details of how the model is set up in MLwiN, for example providing a screenshot of the model along with information on the options that you have set. Alternatively syntax for runmlwin (http://www.bristol.ac.uk/cmm/software/runmlwin/) or R2MLWiN (http://www.bristol.ac.uk/cmm/software/r2mlwin/) would be useful.

[KatrijnDelar]

Dear Chris,

I’m using version 2.31 of MlWin and the data contained no missing values.
The MCMC convergence criteria were met (ESS > 400 and the trajectories look fine).
I added a screenshot and further information as attachment.

Many thanks,
Katrijn

[andrewjdbell]

Hello Katrijn,

Interesting. I don't have a simple answer I'm afraid, but a few things to check:

Have you centred the variables consistently across the model - might make a difference when the interactions go in if what zero means is different.

The exact zero on the education*age interaction is suspicious to me - if you increase the decimal places, is it still exactly zero? If so, that suggests it is exactly collinear with some of the other variables. Different software often deal with exact multicollinearity differently (although I would have thought R and Stata would give some kind of warning...).

I'd want to know a bit more about the data - at the moment are you modelling it like repeated-cross-sectional data, as opposed to panel data. Are the individuals followed between panels (panel data) or is there a new survey of individuals in each country each year (RCS data)? At present they are modelling the data like the latter; if the former they would need an additional observation level beneath the individual level.

Based on their code, it looks like the R model was only run fro 20k iterations, but the MLwiN model for 200k (or it might just be a typo...) Have they checked the R model for convergence?

That's all the ideas I have at the moment - happy to look at a dataset to see if I can work out the problem. Not used R much so may not be very helpful in that regard though.

Cheers,
Andy