Page 1 of 1

Comparing variance estimates from two MCMC models

Posted: Tue May 05, 2020 6:48 pm
by slavutich89
I'd like to compare variance components from two models estimated with MCMC. Model 1 is an unconditional model and Model 2 includes Level 1 predictors. I need to calculate the relative percentage change in Level 1 variance between two models, such that this change also contains a standard error and p-value (I saw it in one paper using MLwiN and would like to replicate it).

Using one of the MLwiN examples, here is what I intend doing (my model differs from the example below, but the logic is the same):

Code: Select all

use "http://www.bristol.ac.uk/cmm/media/runmlwin/tutorial.dta", clear
runmlwin normexam cons, level2(school: cons) level1(student: cons) nopause
runmlwin normexam cons, level2(school: cons) level1(student: cons) mcmc(on) initsprevious nopause
estimates store model1
runmlwin normexam cons girl standlrt, level2(school: cons) level1(student: cons) nopause
runmlwin normexam cons girl standlrt, level2(school: cons) level1(student: cons) mcmc(on) initsprevious nopause
estimates store model2
After estimating these two models, I'd like to run something like:

Code: Select all

nlcom ([model1][RP1]var(cons)-[model2][RP1]var(cons))/([model1][RP1]var(cons)+[model1][RP2]var(cons))
Normally, it could be done by first running "suest model1 model2", but it is not possible to use with runmlwin or other multilevel models.

Is it possible to simply store MCMC chains from both estimations (mcmcsum, getchains), manually merge them and then do the following?

Code: Select all

gen dif=(RP1_var_cons_model1-RP1_var_cons_model2)/(RP1_var_cons_model1+RP2_var_cons_model1)
mcmcsum dif, variables

Re: Comparing variance estimates from two MCMC models

Posted: Thu May 14, 2020 5:31 pm
by ChrisCharlton
I asked George about this and this was his response:
I don't think it is possible to do what they want as the percentage change in the level-1 variance involves two different models fitted to the sample data. So while you could mechanically do what is suggested, I don't think it would be statistically valid. That is, you could save a chain of 1000 values for the level-1 variance from each of your two models as two separate columns of data and then calculate a third variable as the percentage change in the level-1 variance, then report the mean and SD of this as the point estimate and SE. The problem is that you have assumed the sampling co-variability of the level-1 variance parameter process the two models is zero when it would be positive.

One approach which might be more fruitful would be to take a bootstrapping approach and estimate both models on each of the B bootstrapped samples and calculate the percentage change in the level-1 variance on each of the B bootstrapped samples and go about it that way.

Re: Comparing variance estimates from two MCMC models

Posted: Fri May 15, 2020 8:22 pm
by slavutich89
Dear Chris,

Thank you for your answer.

Would it be possible for you to share a Stata code how I can use a bootstrapping approach in my MCMC example from the previous post?

I know it could have been potentially done through comparing scalars in a similar way that is described here:
https://www.statalist.org/forums/forum/ ... ll-be-save
I have tried bootstrapping a difference in scalars as in this Statalist post by simply storing scalars for RP1:var(cons) from each models. But I cannot use "return scalar" after "runmlwin" because Stata returns the following error: "non r-class program may not set r()"

Re: Comparing variance estimates from two MCMC models

Posted: Thu May 21, 2020 12:49 pm
by ChrisCharlton
runmlwin is an estimation command, so you need need to retrieve the returned results using ereturn rather than return (see https://www.stata.com/help.cgi?ereturn). You can find the model estimates in the matrix e(b), and their covariance matrix in e(V).

I am not aware of any example code for the bootstrapping approach and runmlwin, but will check with George and post again if we do.

Re: Comparing variance estimates from two MCMC models

Posted: Thu May 21, 2020 1:53 pm
by ChrisCharlton
Unfortunately we don't have any examples, however George gave the following outline of how this would work:

You would need to set up a bootstrapping routine in Stata, which would call a program that you define. In this you would fit each model and store the quantities of interest as scalars. The bootstrapping routine would then monitor these scalars.

If you look in the bootstrapping topic in the Stata manuals (https://www.stata.com/manuals13/rbootstrap.pdf) you should find examples of how to do this. For your purposes you will need to do a clustered bootstrap.

Re: Comparing variance estimates from two MCMC models

Posted: Fri May 22, 2020 10:33 pm
by slavutich89
Dear Chris,

Thank you for your help with this. Greatly appreciated.

Re: Comparing variance estimates from two MCMC models

Posted: Tue Oct 27, 2020 5:31 pm
by slavutich89
Dear Chris,

I would like to follow up on this post, if it is fine with you. I have managed to write a bootstrap program in Stata and it worked fine for a 2-level model, as the one above.

I now have a question regarding having more levels in the model and using the same produce as you and George have recommended. If I have firms (Level 2) observed over several years (Level 1) and which are nested within subnational regions (Level 3) and these regions are nested within countries (Level 4), I am wondering whether bootstrap clustering should be done at the aggregate (country in this case) level? I only have 10 countries. Or perhaps clustering at the firm level would be sufficient? Thank you.

Re: Comparing variance estimates from two MCMC models

Posted: Thu Oct 29, 2020 9:51 pm
by ChrisCharlton
Sorry, I'm afraid that we don't know the answer to this. George thought that the following paper, along with the papers that it references may be of some help:

Austin, P. C., & Leckie, G. (2020). Bootstrapped inference for variance parameters, measures of heterogeneity and random effects in multilevel logistic regression models. Journal of Statistical Computation and Simulation, 1-25.

Re: Comparing variance estimates from two MCMC models

Posted: Mon Nov 02, 2020 1:57 pm
by LINDAOKEEFFE2020
Hi All,

I am examining change over time in SBP for females and males (2 level model with individual level random effects and occasion level for intercept and slopes) - models are sex-stratified. I want to formally compare estimates from the two models producing an estimate of the difference in SBP between females and males at different points on the trajectory. I have stored the estimation results for both the female and male model but can't seem to formally test whether (for example) _cons for model 1 is different from _cons for model 2. Slavutich89 I wondered if you could share your bootstrapping code as I think this may work for me?

Thanks,
Linda

Re: Comparing variance estimates from two MCMC models

Posted: Fri Nov 13, 2020 3:03 pm
by slavutich89
Hello Linda,

You could look how they do bootstrapping for a difference of two coefficients from two estimations:
https://www.statalist.org/forums/forum/ ... difference

For clustered bootstrap (panel data), see this link:
https://www.statalist.org/forums/forum/ ... s-together

Just use runmlwin commands instead of commands in these examples on Statalist. You will need to see how the estimate you need is called in the matrix produced by runmlwin. You can use matrix list e(b) after each estimation.

The explanation why clustered bootstrap should be done this way is here:
https://www.stata.com/support/faqs/stat ... anel-data/

Hope it helps.