Power/Effective Sample Size

Welcome to the forum for MLwiN users. Feel free to post your question about MLwiN software here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Remember to check out our extensive software FAQs which may answer your question: http://www.bristol.ac.uk/cmm/software/s ... port-faqs/
Post Reply
Jillianeh
Posts: 4
Joined: Wed Mar 22, 2017 9:01 pm

Power/Effective Sample Size

Post by Jillianeh »

My name is Jillian and I am currently using three-level multinomial logistic regression in MlwiN to look at student substance use outcomes. I am trying to see what I am actually powered to look at (in terms of how many variables I can have in my model and how I have to operationalize my outcomes). 

I want to be able to look at "regular users" differently from "occasional users" but my n for regular users gets rather small. 

I have found that MLwiN produces effective sample sizes in column diagnostics. I am wondering if this takes into consideration the three-level structure of the data as it appears I am only selecting a single column without identifying my levels? I cannot find any documentation online that describes column diagnostics and this effective sample size calculation. 

From my diagnostics, the Effective sample size for marijuana  is 3387. I think this means that MLwiN is calculating the design effect as about 3 (10058/3387). Therefore, if we divide the n of regular users (~647) by 3 (design effect) we get 216 as the effective sample size for regular users. 

Similarly for tobacco users, the effective sample size would be 148 (for n=400) and we would still be powered for our analyses (6 variables in the model).

I would truly appreciate if you would be able to let me know if I am interpreting this correctly, and if not, how I would go about calculating the effective sample size to determine power. 
billb
Posts: 157
Joined: Fri May 21, 2010 1:21 pm

Re: Power/Effective Sample Size

Post by billb »

Hi Jillian,
I think you have got yourself completely confused here due to the use of the term 'effective sample size' for two totally unrelated concepts. In MCMC estimation, being a simulation-based procedure, the user has to run the technique for a large number of iterations to get estimates that are sufficiently accurate (by reducing the amount of simulation/Monte Carlo noise). One way of judging how long to run is the concept of effective sample size which loosely means how many equivalent independent iterations your chains are equivalent to.
This has absolutely nothing to do with the size of your data or power calculations - you might look at MLPowSim for some background to power calculations in multilevel models but it doesn't cover multinomial models.
Hope that helps.
Bill.
billb
Posts: 157
Joined: Fri May 21, 2010 1:21 pm

Re: Power/Effective Sample Size

Post by billb »

Hi Divyamore,
Just to reiterate my reply to Jillian. MCMC has a concept called effective sample size which has nothing to do with power but is the number of effective independent iterations of an MCMC chain run. This is what MLwiN is reporting (and I should know as one of the programmers!) and although what you say is perfectly correct and good knowledge but not really answering Jillian's question.
Best wishes,
Bill.
SeamusHarvey
Posts: 3
Joined: Mon May 13, 2019 7:32 am

Re: Power/Effective Sample Size

Post by SeamusHarvey »

Hello. I am using the mcmc estimator to conduct cross-classified growth curve analyses. I am running the analyses using 100,000 iterations. An examination of the Raftery-Lewis figures in the Trajectories section indicates that this is enough. However, the 'Effective Sample Size' figure is often less than the Raftery-Lewis figures. Could you please tell me whether I have met the Raftery-Lewis criteria on the basis of running 100,000 iterations, or whether I should judge this on the basis of the 'Effective Sample Size' figure? Any help would be greatly appreciated.
billb
Posts: 157
Joined: Fri May 21, 2010 1:21 pm

Re: Power/Effective Sample Size

Post by billb »

The effective sample size and Raftery Lewis are completely different diagnostics and in fact a low Raftery Lewis number is good as the diagnostic is a minimum number to run for. In contrast a high ESS is good as it gives an estimate of the equivalent number of independent estimates for the parameter. The R-L should be compared with the actual iterations and not effective sample size. Otherwise you are accounting for correlation in the chains twice.
Hope that helps,
Bill.
SeamusHarvey
Posts: 3
Joined: Mon May 13, 2019 7:32 am

Re: Power/Effective Sample Size

Post by SeamusHarvey »

That's very helpful. Thank you.
Post Reply