Page 1 of 1

Power/Effective Sample Size

Posted: Wed Nov 29, 2017 1:36 pm
by Jillianeh
My name is Jillian and I am currently using three-level multinomial logistic regression in MlwiN to look at student substance use outcomes. I am trying to see what I am actually powered to look at (in terms of how many variables I can have in my model and how I have to operationalize my outcomes). 

I want to be able to look at "regular users" differently from "occasional users" but my n for regular users gets rather small. 

I have found that MLwiN produces effective sample sizes in column diagnostics. I am wondering if this takes into consideration the three-level structure of the data as it appears I am only selecting a single column without identifying my levels? I cannot find any documentation online that describes column diagnostics and this effective sample size calculation. 

From my diagnostics, the Effective sample size for marijuana  is 3387. I think this means that MLwiN is calculating the design effect as about 3 (10058/3387). Therefore, if we divide the n of regular users (~647) by 3 (design effect) we get 216 as the effective sample size for regular users. 

Similarly for tobacco users, the effective sample size would be 148 (for n=400) and we would still be powered for our analyses (6 variables in the model).

I would truly appreciate if you would be able to let me know if I am interpreting this correctly, and if not, how I would go about calculating the effective sample size to determine power. 

Re: Power/Effective Sample Size

Posted: Thu Nov 30, 2017 1:01 pm
by billb
Hi Jillian,
I think you have got yourself completely confused here due to the use of the term 'effective sample size' for two totally unrelated concepts. In MCMC estimation, being a simulation-based procedure, the user has to run the technique for a large number of iterations to get estimates that are sufficiently accurate (by reducing the amount of simulation/Monte Carlo noise). One way of judging how long to run is the concept of effective sample size which loosely means how many equivalent independent iterations your chains are equivalent to.
This has absolutely nothing to do with the size of your data or power calculations - you might look at MLPowSim for some background to power calculations in multilevel models but it doesn't cover multinomial models.
Hope that helps.
Bill.

Re: Power/Effective Sample Size

Posted: Fri Aug 10, 2018 4:40 pm
by billb
Hi Divyamore,
Just to reiterate my reply to Jillian. MCMC has a concept called effective sample size which has nothing to do with power but is the number of effective independent iterations of an MCMC chain run. This is what MLwiN is reporting (and I should know as one of the programmers!) and although what you say is perfectly correct and good knowledge but not really answering Jillian's question.
Best wishes,
Bill.

Re: Power/Effective Sample Size

Posted: Mon Jun 29, 2020 11:05 am
by SeamusHarvey
Hello. I am using the mcmc estimator to conduct cross-classified growth curve analyses. I am running the analyses using 100,000 iterations. An examination of the Raftery-Lewis figures in the Trajectories section indicates that this is enough. However, the 'Effective Sample Size' figure is often less than the Raftery-Lewis figures. Could you please tell me whether I have met the Raftery-Lewis criteria on the basis of running 100,000 iterations, or whether I should judge this on the basis of the 'Effective Sample Size' figure? Any help would be greatly appreciated.

Re: Power/Effective Sample Size

Posted: Tue Jun 30, 2020 11:19 am
by billb
The effective sample size and Raftery Lewis are completely different diagnostics and in fact a low Raftery Lewis number is good as the diagnostic is a minimum number to run for. In contrast a high ESS is good as it gives an estimate of the equivalent number of independent estimates for the parameter. The R-L should be compared with the actual iterations and not effective sample size. Otherwise you are accounting for correlation in the chains twice.
Hope that helps,
Bill.

Re: Power/Effective Sample Size

Posted: Tue Jun 30, 2020 12:31 pm
by SeamusHarvey
That's very helpful. Thank you.