Using Stata commands lroc and lstat after runmlwin

Welcome to the forum for runmlwin users. Feel free to post your question about runmlwin here. The Centre for Multilevel Modelling take no responsibility for the accuracy of these posts, we are unable to monitor them closely. Do go ahead and post your question and thank you in advance if you find the time to post any answers!

Go to runmlwin: Running MLwiN from within Stata >> http://www.bristol.ac.uk/cmm/software/runmlwin/
Post Reply
sujust18
Posts: 6
Joined: Wed Dec 05, 2012 4:54 pm

Using Stata commands lroc and lstat after runmlwin

Post by sujust18 »

Hello everybody,

I'm running a multilevel logit model with crossed-random effects using runmlwin.

I would like to let Stata calculate the Area Under the ROC curve afterwards as explained here: http://www.ats.ucla.edu/stat/stata/faq/roc.htm
using the lroc, nograph command. Similiarly, there is a lstat command in stata giving helpful measures for the validation of a logit model.
However, it cannot be used after runmlwin with the error: "last estimates not found".

Is there a way to save my runmlwin estimates so that Stata can use its other implemented commands?

Thanks in advance for your help,

Best regards.
GeorgeLeckie
Site Admin
Posts: 432
Joined: Fri Apr 01, 2011 2:14 pm

Re: Using Stata commands lroc and lstat after runmlwin

Post by GeorgeLeckie »

Hi sujust18,

Am not familiar with ROC analysis, but I note that it doesn't look like you can do a ROC after xtmelogit, Stata's own multilevel modelling command.

http://www.stata.com/help.cgi?xtmelogit_postestimation

After fitting model using runmlwin, you can access all parameter estimates with the following commnads

Code: Select all

ereturn dir
ereturn list e(b)
ereturn list e(V)
In a multilevel logistic regression you should be able to retrieve the linear preditor as...

Code: Select all

predict xb1, xb
Best wishes

George
ManuelDewez
Posts: 16
Joined: Tue Mar 09, 2021 5:54 pm

Re: Using Stata commands lroc and lstat after runmlwin

Post by ManuelDewez »

Dear George,
I am also doing a series of multilevel analysis to identify the availability of diagnostic tests in primary care surgeries in different countries. Surgeries characteristics are level 1 and country level 2.
I would like to do it with MCMC, because the prevalence of my binary dependent variable is very high (>90%)or very low (<10%)
After using runmlwin I would like to plot ROC curves so I can compare visually the ROC of a single level model (with primary care surgeries characteristics only ) with the ROC of a 2 levels model in which I add country as the second level variable. This is to see visually how country improves the ability of model 2 to predict the outcome. The approach is based on this paper by Juan Merlo:https://europepmc.org/article/PMC/PMC4847925

When using
predict xb1, xb
, as suggested below, stata calculates the predicted mean based on a linear predictor
that includes the fixed effects only.
How can I have a calculation that includes both the fixed and the random effects?

After - melogit - , the syntax would be:
predict xb1, mu
mu being an option that allows to include both the fixed and random effects.

How can I do this with runmlwin?

Thank you

Manuel
ChrisCharlton
Posts: 1348
Joined: Mon Oct 19, 2009 10:34 am

Re: Using Stata commands lroc and lstat after runmlwin

Post by ChrisCharlton »

As you have found, running predict after runmlwin will only give you the fixed-part prediction. If you want a prediction with both fixed and random effects then you will need to manually request that the higher-level residuals are returned via the residuals() option and then use these to extend the prediction. George gives an example of this in his presentation: "8th International Amsterdam Multilevel Conference (17th March 2011)", available on the following page:

https://www.bristol.ac.uk/cmm/software/ ... entations/

You can find the syntax for this in the provided Stata do-file or on slides 19 and 26 of the presentation.
ManuelDewez
Posts: 16
Joined: Tue Mar 09, 2021 5:54 pm

Re: Using Stata commands lroc and lstat after runmlwin

Post by ManuelDewez »

Dear Chris,

Thank you very much for your prompt reply, much appreciated!

I am using the dataset provided in the paper by Merlo and colleagues to learn from their approach : https://doi.org/10.1371/journal.pone.0153778.s004


I generated the following syntax :
runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) nopause

and then
runmlwin Psycmed cons male agegroup2 agegroup3 agegroup4 agegroup5 agegroup6 poor, level2(Neigh: cons, residuals(u)) level1(Id:) discrete(distribution(binomial) link(logit) denom(cons) pql2) mcmc(burnin(1000) chain(10000)) initsprevious nopause nogroup
This gives the same outputs that are presented in the paper.

The higher level residuals and SE are provided back to stata as u0 and u0SE

I am trying to generate a variable which includes the fixed and random effects
I tried
predict newvar, xb +u0
but I get :
option + not allowed
r(198);
So I tried to adapt (probably wrongly) the syntax provided in the slides of the conference you mentioned
generate prediction = [FP1]cons + [FP1]male + [FP1]agegroup2 + [FP1]agegroup3 + [FP1]agegroup4 + [FP1]agegroup5 + [FP1]agegroup6 + u0
but when then I use
roctab Psycmed prediction, graph summary
to get the AUC and ROC curve, I get values that are smaller than in the paper, I get an UAC of 0.57, while in the paper is 0.630. I think it is because my new variable prediction in fact only contains the random effect residuals u0.
How can I generate a variable that includes both the fixed and the random effects, that I could then use with -roctab-?

Thank you very much.


Manuel
ChrisCharlton
Posts: 1348
Joined: Mon Oct 19, 2009 10:34 am

Re: Using Stata commands lroc and lstat after runmlwin

Post by ChrisCharlton »

I suspect that your problem is that you have forgotten to multiple the fixed-part parameters by their associated data, so you'll want something more like:

Code: Select all

generate prediction = [FP1]cons*cons + [FP1]male*male + [FP1]agegroup2*agegroup2 + [FP1]agegroup3*agegroup3 + [FP1]agegroup4*agegroup4 + [FP1]agegroup5*agegroup5 + [FP1]agegroup6*agegroup6 + u0
or more simply if you get Stata to do half the work:

Code: Select all

predict xb
generate prediction = xb + u0
You can find a couple of more detailed examples from George in the following topics:

Can you use Stata's predict command after runmlwin?
Predictions via the runmlwin interface: a clarification
ManuelDewez
Posts: 16
Joined: Tue Mar 09, 2021 5:54 pm

Re: Using Stata commands lroc and lstat after runmlwin

Post by ManuelDewez »

It works!
Thanks a lot Chris this is very useful!
Post Reply