Sample Sizes for Multilevel Models
See also MLPowSim
Fitting a 2 level multilevel model means that we are assuming that
- we have observations which contain values that vary randomly according to a distribution e.g. normal, Poisson etc. and
- these observations are contained in identifiable clusters (represented by a categorical variable) which themselves are in some sense randomly drawn from some population e.g. a sample of schools from a population of schools.
It is clearly wrong to fit some categorical variables e.g. gender as a level in a model, more because it doesn't make much sense to think of male and female as a sample of genders from a population of genders than because there are only 2 of them! This said (and to complicate things) people do fit random effects when they have the full population of units e.g. all schools in a district. In such cases we consider an underlying superpopulation.
Different software packages and estimation algorithms have more or less success in fitting a multilevel model depending on the size of the data including the numbers of level 2 units and level 1 units per level 2 unit. Problems will occur if
- the data or model is too large - memory problems, slow execution etc.
- the data is too small - not enough data to properly estimate the model.
This is not just a problem for multilevel models. If you only collected 10 observations for a regression you might not be overly confident of the estimated regression line and the same is true if in a multilevel model if you only collect data on 10 schools - you will not get a very accurate estimate of their variability.
This doesn't mean you cannot attempt to fit a multilevel model with 10 schools however you may find that the software estimates the variance as zero and if the variance is estimated when you look at the 10 residuals it will be hard to justify that they fit a normal model (as opposed to another distribution).
Rules of thumb such as only doing multilevel modelling with 15 or 30 or 50 level 2 units can be found and are often personal opinions based on personal experience and varying reasons e.g. getting a non zero variance, being able to check the normality assumption etc. For example, in Gelman's new book recently he will attempt multilevel modelling with as little as 3 level 2 units! (note he is generally using MCMC so will not get a 0 variance estimate). It is clear that, if you use IGLS in MLwiN and want the estimate not to be 0 then the more units the better, and how many will depend on how variable the cluster means are relative to the data.
There are also sample size considerations which you might like to consider (prior to collecting data) which will give you desirable numbers of level 1 and level 2 units. This is generally to achieve power for testing fixed effects, see
http://seis.bris.ac.uk/~frwjb/materials/ssize.ppt for some slides on this. Of course you could not worry about clustering by simply taking your sample from 1 cluster but this will mean your population and hence your conclusions also only correspond to the 1 cluster!
MLPowSim
We are pleased to make available a new free piece of software MLPowSim that is designed for performing sample size/power calculations in multilevel models via simulation.
The software package has been developed as part of a UK ESRC funded-project and is an 'old-fashioned' text input program that creates files that can be used in conjunction with MLwiN or R to perform the necessary computations to perform complex power calculations.
The program is available along with an extensive 150 page manual from my web page
We will describe the software currently as a Beta version as we have only had time to do preliminary testing and we haven't included much error trapping. The software is FREE and as such comes with no guarantees in terms of producing correct answers (although we hope it does!) and no guarantee of fast response to fixing of any bugs reported (although we hope there aren't many). We will however be genuinely pleased if people use it and let us know of any bugs they find or if they have a 'wish list' of additional features they might like.
If you do use the software and would like to give feedback, bug reports or a wish list please e-mail either Richard Parker (richard.parker@bristol.ac.uk) or me (william.browne@bristol.ac.uk).
Good luck with the software,