Brooks-Drapers diagnostic says 1,03e+7, do I need to rerun the model with 10 300 000 iterations?

Welcome to the forum for runmlwin users. Feel free to post your question about runmlwin here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Go to runmlwin: Running MLwiN from within Stata >> http://www.bristol.ac.uk/cmm/software/runmlwin/
Post Reply
ManuelDewez
Posts: 16
Joined: Tue Mar 09, 2021 5:54 pm

Brooks-Drapers diagnostic says 1,03e+7, do I need to rerun the model with 10 300 000 iterations?

Post by ManuelDewez »

Hello,

I am running the following model with MCMC for a binary outcome:
runmlwin dipstick_available_for_use cons private solo centered_distance_lab99 centered_pc_turna_time3, level1(id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) nopause
runmlwin dipstick_available_for_use cons private solo centered_distance_lab99 centered_pc_turna_time3, level1(id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) mcmc(burnin(1000) chain(10300000)) initsprevious nopause nogroup
(This is a first model with a single level, but I will follow on with a second model in which I will add a second level: country , to then compare the models and asses the contribution of country to the outcome.)

I understand that the trajectories and five ways graphs look all “healthy” ( attached), but in terms of Raftery-Lewis and Brooks-Draper diagnostics I get the following for the parameter [FP1] solo:

[FP1]solo

Percentiles
Mean 0.0635883 0.50% -0.7214065 Thinned Chain Length 10000
MCSE of Mean 0.0081857 2.50% -0.5319016 Effective Sample Size 1540
Std. Dev. 0.320784 5% -0.4458137 Raftery Lewis (2.5%) 15424
Mode 0.042611 25% -0.1491841 Raftery Lewis (97.5%) 16195
P(mean) 0.441 Brooks Draper (mean) 1.03E+07
P(mode) 0.441 50% 0.0536541
P(median) 0.441
75% 0.2672179
95% 0.6101967
97.50% 0.7130077
99.50% 0.9634089


I want to report the estimate with 2 significant figures. I understand that then I need to rerun the model with 1.03e+07 iterations, which I did and produce slightly different results but took much much longer.
How can I know the number of iterations needed when I get such extreme Brooks-Draper diagnostics?
Do I need to use thining:
mcmc(burnin(1000) chain(10300000) thinning())
If I do that I understand that the parameter means and standard deviations will then be based on all iterations, while the ESS's and 95% credible intervals will be based on the stored iterations depending on the specified thinning.

But how many thinning figures do you advise? 50, 500, 5000?

Or do I need to check diagnostics only for the next model, ie the multilevel model?

Thank you


Manuel
billb
Posts: 157
Joined: Fri May 21, 2010 1:21 pm

Re: Brooks-Drapers diagnostic says 1,03e+7, do I need to rerun the model with 10 300 000 iterations?

Post by billb »

Morning Manuel,
The purpose of thinning is largely just to make things feasible due to memory constraints etc. so ideally we wouldn't thin at all but clearly the computer then has to store a lot of numbers. As you rightly say we use running totals for ALL iterations to construct the point estimate and SD in MLwiN when using MCMC so you don't lose anything there but quantiles etc. will be calculated on the stored chains.
Hope that helps,
Bill.
billb
Posts: 157
Joined: Fri May 21, 2010 1:21 pm

Re: Brooks-Drapers diagnostic says 1,03e+7, do I need to rerun the model with 10 300 000 iterations?

Post by billb »

P.S. As you say the Brooks Draper diagnostic is how long is required to be confident to quote your estimate as in this case 0.064 rather than say 0.065 or 0.063 which given your effect is non-significant anyway i.e. the SD here is 0.321 and 95% CI is -0.532 to 0.713 might not be really that important? This is why the number is so big!
ManuelDewez
Posts: 16
Joined: Tue Mar 09, 2021 5:54 pm

Re: Brooks-Drapers diagnostic says 1,03e+7, do I need to rerun the model with 10 300 000 iterations?

Post by ManuelDewez »

Hello Bill,

Thanks for your reply.

I want to report the estimate as 0.06.
Then what should I do? can I be satisfied with the 10 000 iterations of the initial model, or do I need to increase iterations, even if no up to to 10 300 000 ?

Thank you.
billb
Posts: 157
Joined: Fri May 21, 2010 1:21 pm

Re: Brooks-Drapers diagnostic says 1,03e+7, do I need to rerun the model with 10 300 000 iterations?

Post by billb »

So 0.06 would be 1 significant figure (number of digits required after leading 0s) so plug that in and see what it says.
Best wishes,
Bill.
Post Reply