Power/Effective Sample Size

Welcome to the forum for MLwiN users. Feel free to post your question about MLwiN software here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Remember to check out our extensive software FAQs which may answer your question: http://www.bristol.ac.uk/cmm/software/s ... port-faqs/
Post Reply
Jillianeh
Posts: 4
Joined: Wed Mar 22, 2017 9:01 pm

Power/Effective Sample Size

Post by Jillianeh » Wed Nov 29, 2017 1:36 pm

My name is Jillian and I am currently using three-level multinomial logistic regression in MlwiN to look at student substance use outcomes. I am trying to see what I am actually powered to look at (in terms of how many variables I can have in my model and how I have to operationalize my outcomes). 

I want to be able to look at "regular users" differently from "occasional users" but my n for regular users gets rather small. 

I have found that MLwiN produces effective sample sizes in column diagnostics. I am wondering if this takes into consideration the three-level structure of the data as it appears I am only selecting a single column without identifying my levels? I cannot find any documentation online that describes column diagnostics and this effective sample size calculation. 

From my diagnostics, the Effective sample size for marijuana  is 3387. I think this means that MLwiN is calculating the design effect as about 3 (10058/3387). Therefore, if we divide the n of regular users (~647) by 3 (design effect) we get 216 as the effective sample size for regular users. 

Similarly for tobacco users, the effective sample size would be 148 (for n=400) and we would still be powered for our analyses (6 variables in the model).

I would truly appreciate if you would be able to let me know if I am interpreting this correctly, and if not, how I would go about calculating the effective sample size to determine power. 

billb
Posts: 112
Joined: Fri May 21, 2010 1:21 pm

Re: Power/Effective Sample Size

Post by billb » Thu Nov 30, 2017 1:01 pm

Hi Jillian,
I think you have got yourself completely confused here due to the use of the term 'effective sample size' for two totally unrelated concepts. In MCMC estimation, being a simulation-based procedure, the user has to run the technique for a large number of iterations to get estimates that are sufficiently accurate (by reducing the amount of simulation/Monte Carlo noise). One way of judging how long to run is the concept of effective sample size which loosely means how many equivalent independent iterations your chains are equivalent to.
This has absolutely nothing to do with the size of your data or power calculations - you might look at MLPowSim for some background to power calculations in multilevel models but it doesn't cover multinomial models.
Hope that helps.
Bill.

billb
Posts: 112
Joined: Fri May 21, 2010 1:21 pm

Re: Power/Effective Sample Size

Post by billb » Fri Aug 10, 2018 4:40 pm

Hi Divyamore,
Just to reiterate my reply to Jillian. MCMC has a concept called effective sample size which has nothing to do with power but is the number of effective independent iterations of an MCMC chain run. This is what MLwiN is reporting (and I should know as one of the programmers!) and although what you say is perfectly correct and good knowledge but not really answering Jillian's question.
Best wishes,
Bill.

Post Reply