3 level cross-classified model: how to set level 2 and 3?

Welcome to the forum for runmlwin users. Feel free to post your question about runmlwin here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Go to runmlwin: Running MLwiN from within Stata >> http://www.bristol.ac.uk/cmm/software/runmlwin/
Post Reply
scorpionwwj
Posts: 5
Joined: Fri Jun 07, 2019 3:22 pm

3 level cross-classified model: how to set level 2 and 3?

Post by scorpionwwj » Fri Jun 07, 2019 3:39 pm

Dear users,

I am fitting a 3 level cross-classified model: level 1 is patient ID (ca.40000), the other 2 levels are physician ID (ca.7000) and region ID where patients live in (ca.700). Outcome variable is binary so this is a logistic model. Below is the code:

Code: Select all

runmlwin y sex age cons, level3(region:cons) level2(physician:cons) level1(id) discrete(distribution(binomial) link(logit) denominator(cons)) or nopause
runmlwin y sex age cons, level3(region:cons) level2(physician:cons) level1(id) discrete(distribution(binomial) link(logit) denominator(cons)) or mcmc(on) initsprevious nopause nogroup
But if I switch the level 2 and level 3 like below:

Code: Select all

runmlwin y sex age cons, level3(physician:cons) level2(region:cons) level1(id) discrete(distribution(binomial) link(logit) denominator(cons)) or nopause
runmlwin y sex age cons, level3(physician:cons) level2(region:cons) level1(id) discrete(distribution(binomial) link(logit) denominator(cons)) or mcmc(on) initsprevious nopause nogroup
I got quite different results, especially the random effects.

It would be great if anyone could help. Thank you indeed!

Best,
wj

ChrisCharlton
Posts: 1099
Joined: Mon Oct 19, 2009 10:34 am

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by ChrisCharlton » Mon Jun 10, 2019 8:59 am

In order to fit a cross-classified model you need to add cc to the mcmc options, otherwise the model will be fitted as hierarchical. This should be reflected in group summary information in the model output. Note that even after doing so the results from switching the levels around, as in your example, will be slightly different as the IGLS results are used as starting values and these will still assume a hierarchical structure. You can find examples of specifying cross-classified models within runmlwin in chapter 15 of the MCMC manual examples, available at https://www.bristol.ac.uk/cmm/software/ ... /examples/.

scorpionwwj
Posts: 5
Joined: Fri Jun 07, 2019 3:22 pm

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by scorpionwwj » Fri Jun 14, 2019 4:38 pm

Dear Chris,

thank you indeed for your reply. It is really helpful.

Best regards,
wj

scorpionwwj
Posts: 5
Joined: Fri Jun 07, 2019 3:22 pm

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by scorpionwwj » Fri Jun 14, 2019 7:47 pm

ChrisCharlton wrote:
Mon Jun 10, 2019 8:59 am
In order to fit a cross-classified model you need to add cc to the mcmc options, otherwise the model will be fitted as hierarchical. This should be reflected in group summary information in the model output. Note that even after doing so the results from switching the levels around, as in your example, will be slightly different as the IGLS results are used as starting values and these will still assume a hierarchical structure. You can find examples of specifying cross-classified models within runmlwin in chapter 15 of the MCMC manual examples, available at https://www.bristol.ac.uk/cmm/software/ ... /examples/.
Dear Chris,

I have one further question. I have modified the code according to your reply:

Code: Select all

runmlwin y sex age cons, level3(region:cons) level2(physician:cons) level1(id) discrete(distribution(binomial) link(logit) denominator(cons)) or nopause
runmlwin y sex age cons, level3(region:cons) level2(physician:cons) level1(id) discrete(distribution(binomial) link(logit) denominator(cons)) or mcmc(cc on) initsprevious nopause nogroup


But the results of random-effect parameters are a bit weird: the higher level variation is 0.08 for level 3 (region), 1308 for level 2 (physician) which is too big. I have also tried two 2-level models, one with region as 2nd level, the other one with physician as 2nd level, the higher level variation for the 2 models are 0.08 and 0.18, respectively. I am not sure if the random effect estimates in the cross-classified model, especially for the physician level are correct.

Thank you in advance for your reply.

Best,
wj

ChrisCharlton
Posts: 1099
Joined: Mon Oct 19, 2009 10:34 am

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by ChrisCharlton » Mon Jun 17, 2019 9:11 am

Have you checked the MCMC chain and convergence diagnostics? It may be that you have to run the model for more iterations before it converges to a distribution. I would also suggest removing the nopause option to check within MLwiN via the Model->Hierarchy Viewer window that you have the expected number of units within each classification.

scorpionwwj
Posts: 5
Joined: Fri Jun 07, 2019 3:22 pm

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by scorpionwwj » Tue Jul 02, 2019 5:12 pm

ChrisCharlton wrote:
Mon Jun 17, 2019 9:11 am
Have you checked the MCMC chain and convergence diagnostics? It may be that you have to run the model for more iterations before it converges to a distribution. I would also suggest removing the nopause option to check within MLwiN via the Model->Hierarchy Viewer window that you have the expected number of units within each classification.
Dear Chris,

thank you very much for your reply.

First, I have removed nonpause option and checked in the "Hierarchy Viewer" and the number of units within each classification is correct.

Second, I tried to increase iterations with the option "mcmc(cc burnin(5000) chain(10000))", but the random effect at physician level was even bigger (ca.4500), then I increased further the iterations by "mcmc(cc burnin(100000) chain(500000))", the random effect at physician level turned to 26500. And the random effect at region level was always stable (0.08).

Then, I checked the trajectories within MLwin, attached are the figures for region level and physician level, respectively. The figure for region level looks fine, but for physician level it is odd. I noticed that the mean value is abnormally large, but the median value is quite reasonable (as same as in the 2-level model with patient and physician, ca.0.18). I am not sure about the reason, but do you think if I could use median instead of mean for random effect at physician level?

Thank you indeed.

Best,
wj
Attachments
region random effect.PNG
region level
region random effect.PNG (168.81 KiB) Viewed 1013 times
physician random effect.PNG
physician level
physician random effect.PNG (158.06 KiB) Viewed 1013 times

ChrisCharlton
Posts: 1099
Joined: Mon Oct 19, 2009 10:34 am

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by ChrisCharlton » Wed Jul 03, 2019 11:44 am

The spikes in the chains do look odd. It may be that there is something unusual in your data causing this (for example units with mostly or all zeros or ones). We would be willing to take a closer look if this data is something that you are able to share. If so could you send it to Bill Browne, who implemented the MCMC algorithm, using the email address on https://www.bristol.ac.uk/cmm/team/?

scorpionwwj
Posts: 5
Joined: Fri Jun 07, 2019 3:22 pm

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by scorpionwwj » Thu Jul 04, 2019 12:10 pm

Dear Chris,

thank you for your reply. Yes, I agree that the reason might be that both region and physician levels have many units and three might be quite a lot of units with zeros and ones, especially at the physician level, around 1500 of 7000 physicians have only one patient.

Unfortunately, I could not share the data as they are from the insurance company and it is not allowed to share it without permission.

What would you think if I use median instead of mean? Or do you have any further suggestions?

Thank you indeed.

Best,
wj

ChrisCharlton
Posts: 1099
Joined: Mon Oct 19, 2009 10:34 am

Re: 3 level cross-classified model: how to set level 2 and 3?

Post by ChrisCharlton » Fri Jul 05, 2019 8:37 am

I asked Bill whether he could give any further advice without seeing the data and the response was as follows:
This isn't behaviour that we have seen before and so therefore likely to be something to be specific to their dataset, given how often these models are fitted, which is hard to diagnose without seeing the dataset.

A couple of things
  1. Do they have any missing data as they could then listwise delete the missing data and see if that fixes things in case either runmlwin or MLwiN itself is getting confused by missing data.
  2. They could in MLwiN try the MLwiN->WinBUGS convertor and then (i) try running the model in WinBUGS and/or (ii) Look at the data generated for WinBUGS to see if there is anything unusual e.g. any weird values etc.

Post Reply