Inconsistent results from runmlwin and svy:mean
Posted: Thu Aug 22, 2019 7:07 pm
Hello,
Edit: I should provide more contexts. I am analyzing a complex survey (multistage sampling) design data. I wanted to compare a multilevel method using a MLwin and a svyset method in Stata. The actual data structure is three levels but below I am only showing as a level 1 structure to simplify. To incorporate the three level structure, I would use three levels in MLwiN and jackknife replicate weights in svyset of Stata.
I compared baseline statistics of RCT using runmlwin and svy:mean + estat sd, variance.
At the bottom of the post, you can see the results as an attachment. You see that for the unweighted version, means and the variances are the same (or very similar) for both methods. For the weighted version, you see that the variances are vastly different, while the means are the same. For example, svy: mean gives larger variances for the treatment group, while runmlwin gives the opposite.
Below are my codes for svy:mean. I am using the final weight w.
Below are my codes for runmlwin. It might be confusing because I had to modify the codes to post it here. I am using final weight w at level 1, which I assume to be the same approach with svy: mean. Also, I used separate coding, including trt (treatment), ctrl (control) in the fixed part and the random part, instead of using the conventional contrast coding which uses intercept. This shouldn't affect the result. And as you can see for the unweighted version, this was not the issue.
I cannot figure out why the variances are different after I used the weight... obviously, it may be hard to know without looking at the data but could anyone take a look at the codes and see if something is wrong there? Especially the weighted approach for svy: mean and runmlwin. Should the two blocks of codes do the same thing or did I miss something?
Thank you in advance,
Sun
Edit: I should provide more contexts. I am analyzing a complex survey (multistage sampling) design data. I wanted to compare a multilevel method using a MLwin and a svyset method in Stata. The actual data structure is three levels but below I am only showing as a level 1 structure to simplify. To incorporate the three level structure, I would use three levels in MLwiN and jackknife replicate weights in svyset of Stata.
I compared baseline statistics of RCT using runmlwin and svy:mean + estat sd, variance.
At the bottom of the post, you can see the results as an attachment. You see that for the unweighted version, means and the variances are the same (or very similar) for both methods. For the weighted version, you see that the variances are vastly different, while the means are the same. For example, svy: mean gives larger variances for the treatment group, while runmlwin gives the opposite.
Below are my codes for svy:mean. I am using the final weight w.
Code: Select all
**********************baseline statistics with descriptive analysis*****************
/////*unweighted*/////
local outcomes test1 test2 test3
foreach outcome in `outcomes' {
mean `outcome', over(treatment)
estat sd, variance
}
/////*weighted*/////
local outcomes test1 test2 test3
svyset [pweight=w]
foreach outcome in `outcomes' {
svy: mean `outcome', over(treatment)
estat sd, variance
}
Code: Select all
**********************baseline statistics with mlwin*****************
/////*unweighted*/////
***set a loop for a model for cognitive outcomes
eststo clear
local outcomes test1 test2 test3
local n : word count `outcomes'
local residuals a b c d e f g h i j k l m n o p q r s t u v w y z
forvalues i = 1/`n' {
local outcome = word("`outcomes'", `i')
eststo clear
*** treatment (F, R)
runmlwin `outcome' trt ctrl, ///
level1( HSIS_CHILDID: trt ctrl, diagonal residuals(`residual')) nopause
eststo
}
/////*weighted at level 1 only*/////
***set a loop for a model for cognitive outcomes
eststo clear
local outcomes test1 test2 test3
local n : word count `outcomes'
local residuals a b c d e f g h i j k l m n o p q r s t u v w y z
destring CHILDCOHORT, replace
forvalues i = 1/`n' {
local outcome = word("`outcomes'", `i')
eststo clear
*** treatment (F, R)
runmlwin `outcome' trt ctrl, ///
level1( HSIS_CHILDID: trt ctrl , weightvar(w) diagonal residuals(`residual')) nopause
eststo
}
Thank you in advance,
Sun