My name is Jillian and I am currently using threelevel multinomial logistic regression in MlwiN to look at student substance use outcomes. I am trying to see what I am actually powered to look at (in terms of how many variables I can have in my model and how I have to operationalize my outcomes).
I want to be able to look at "regular users" differently from "occasional users" but my n for regular users gets rather small.
I have found that MLwiN produces effective sample sizes in column diagnostics. I am wondering if this takes into consideration the threelevel structure of the data as it appears I am only selecting a single column without identifying my levels? I cannot find any documentation online that describes column diagnostics and this effective sample size calculation.
From my diagnostics, the Effective sample size for marijuana is 3387. I think this means that MLwiN is calculating the design effect as about 3 (10058/3387). Therefore, if we divide the n of regular users (~647) by 3 (design effect) we get 216 as the effective sample size for regular users.
Similarly for tobacco users, the effective sample size would be 148 (for n=400) and we would still be powered for our analyses (6 variables in the model).
I would truly appreciate if you would be able to let me know if I am interpreting this correctly, and if not, how I would go about calculating the effective sample size to determine power.
Power/Effective Sample Size
Re: Power/Effective Sample Size
Hi Jillian,
I think you have got yourself completely confused here due to the use of the term 'effective sample size' for two totally unrelated concepts. In MCMC estimation, being a simulationbased procedure, the user has to run the technique for a large number of iterations to get estimates that are sufficiently accurate (by reducing the amount of simulation/Monte Carlo noise). One way of judging how long to run is the concept of effective sample size which loosely means how many equivalent independent iterations your chains are equivalent to.
This has absolutely nothing to do with the size of your data or power calculations  you might look at MLPowSim for some background to power calculations in multilevel models but it doesn't cover multinomial models.
Hope that helps.
Bill.
I think you have got yourself completely confused here due to the use of the term 'effective sample size' for two totally unrelated concepts. In MCMC estimation, being a simulationbased procedure, the user has to run the technique for a large number of iterations to get estimates that are sufficiently accurate (by reducing the amount of simulation/Monte Carlo noise). One way of judging how long to run is the concept of effective sample size which loosely means how many equivalent independent iterations your chains are equivalent to.
This has absolutely nothing to do with the size of your data or power calculations  you might look at MLPowSim for some background to power calculations in multilevel models but it doesn't cover multinomial models.
Hope that helps.
Bill.
Re: Power/Effective Sample Size
Hi Divyamore,
Just to reiterate my reply to Jillian. MCMC has a concept called effective sample size which has nothing to do with power but is the number of effective independent iterations of an MCMC chain run. This is what MLwiN is reporting (and I should know as one of the programmers!) and although what you say is perfectly correct and good knowledge but not really answering Jillian's question.
Best wishes,
Bill.
Just to reiterate my reply to Jillian. MCMC has a concept called effective sample size which has nothing to do with power but is the number of effective independent iterations of an MCMC chain run. This is what MLwiN is reporting (and I should know as one of the programmers!) and although what you say is perfectly correct and good knowledge but not really answering Jillian's question.
Best wishes,
Bill.

 Posts: 3
 Joined: Mon May 13, 2019 7:32 am
Re: Power/Effective Sample Size
Hello. I am using the mcmc estimator to conduct crossclassified growth curve analyses. I am running the analyses using 100,000 iterations. An examination of the RafteryLewis figures in the Trajectories section indicates that this is enough. However, the 'Effective Sample Size' figure is often less than the RafteryLewis figures. Could you please tell me whether I have met the RafteryLewis criteria on the basis of running 100,000 iterations, or whether I should judge this on the basis of the 'Effective Sample Size' figure? Any help would be greatly appreciated.
Re: Power/Effective Sample Size
The effective sample size and Raftery Lewis are completely different diagnostics and in fact a low Raftery Lewis number is good as the diagnostic is a minimum number to run for. In contrast a high ESS is good as it gives an estimate of the equivalent number of independent estimates for the parameter. The RL should be compared with the actual iterations and not effective sample size. Otherwise you are accounting for correlation in the chains twice.
Hope that helps,
Bill.
Hope that helps,
Bill.

 Posts: 3
 Joined: Mon May 13, 2019 7:32 am
Re: Power/Effective Sample Size
That's very helpful. Thank you.