www.cmm.bristol.ac.uk/forum

Posted: **Sat Oct 22, 2016 12:01 pm**

Hello all,

I need to interpret a significant p-value that has a confidence interval (CI) for the estimate that crosses 0 (see below).
Usually, I place more importance on CI.

Knowing that MCMC was conducted in this model, should I place more importance on the CI or the p-value?

MLwiN (version: 2.36) multilevel model (Binomial)
N min mean max N_complete min_complete mean_complete max_complete
iID 60 1 23.05 77 55 1 20.36364 74
Estimation algorithm: MCMC Elapsed time : 12.08s
Number of obs: 1120 (from total 1383) Number of iter.: 5000 Chains: 1 Burn-in: 500
Bayesian Deviance Information Criterion (DIC)
Dbar D(thetabar) pD DIC
1415.712 1401.830 13.881 1429.593

Coef. Std. Err. z Pr(>|z|) [95% Cred. Interval] ESS
Intercept 0.79868 0.35035 2.28 0.02263 * 0.16612 1.50846 28
...
MalnutritionPYes 0.31611 0.16009 1.97 0.04832 * -0.01020 0.62371 663
...

Thank you so much!
Adrian

Posted: **Mon Oct 24, 2016 10:00 am**

Note that one-sided Bayesian p-values are another option which can be helpful; these can be toggled on/off in R2MLwiN via z.ratio. There is an example in Section 6 of the R2MLwiN article (http://dx.doi.org/10.18637/jss.v072.i10) recently published in the Journal of Statistical Software; as the article states, these return the proportion of chain iterations which are of the opposite sign to the chain mean (the coefficient estimate). I.e. they use the same information as the credible intervals (insofar as they both use the values of the chain iterations), but just emphasise a different element of that information.

E.g.:

Code: Select all

library(R2MLwiN)
data("tutorial")
F3 <- normexam ~ 1 + standlrt + (1 | school) + (1 | student)
print(standlrtMCMC <- runMLwiN(Formula = F3, data = tutorial, estoptions = list(EstM = 1)), z.ratio = FALSE)

...would yield the following output (truncated):

Code: Select all

The fixed part estimates:  
              Coef.   Std. Err.  pMCMC(1-sided)  [95% Cred.   Interval]    ESS 
Intercept   0.00514     0.04246           0.444    -0.07933     0.08896    211 
standlrt    0.56321     0.01250               0     0.53862     0.58791   4060

So the proportion of chain iterations which were below zero for the Intercept was 0.444, whilst none of the 5000 iterations were below zero for standlrt.

Compare this with (note that z.ratio = TRUE is actually the default here, but just including it to be explicit):

Code: Select all

print(standlrtMCMC, z.ratio = TRUE)

...which would yield (again, truncated):

Code: Select all

The fixed part estimates:  
              Coef.   Std. Err.      z   Pr(>|z|)       [95% Cred.   Interval]    ESS 
Intercept   0.00514     0.04246   0.12     0.9036         -0.07933     0.08896    211 
standlrt    0.56321     0.01250  45.05          0  ***     0.53862     0.58791   4060 
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Best wishes,

Richard

www.cmm.bristol.ac.uk/forum

Confidence Intervals vs. Significant p-values

Confidence Intervals vs. Significant p-values

Re: Confidence Intervals vs. Significant p-values