Penalized Maximum Likelihood Logistic Regression

Joseph Coveney

Cobridge Co., Ltd.

Topics

• Separation in Logistic Regression

• Approaches to Separation

• Firth’s Bias-reduced GLMs

• firthlogit: syntax and examples

• Caveats and to-do’s

Separation in Logistic Regression

Dataset adapted from D. W. Hosmer and S. Lemeshow, Applied Logistic Regression Second Edition. (New York: John Wiley & Sons, 2000), pp. 138–39.

Complete Separation

Quasi-complete Separation

Dataset adapted from D. W. Hosmer and S. Lemeshow, Applied Logistic Regression Second Edition. (New York: John Wiley & Sons, 2000), pp. 138–39.

Approaches to Separation

• Remove predictors– Pool groups– Remove interaction terms

• Gather more data

• Use alternatives

Exact Logistic Regression

But . . .

Dataset from D. M. Potter. 2005. A permutation test for inference in logistic regression with small- and moderate-sized data sets. Statistics in Medicine 24:693–708.

[19] D. Firth. 1993. Bias reduction in maximum likelihood estimates. Biometrika 80:27–38.

firthlogit

But . . . redux

But . . . redux, continued

0

.2

.4

.6

.8

1A

ctua

l Pro

babi

lity

(LO

WE

SS

Sm

ooth

ed)

0 .2 .4 .6 .8 1Predicted Probability

bandwidth = .8

Profile Likelihood Ratio CIs

Caveats

• Profile Penalized Likelihood CIs

• Small-sample Behavior

G. Heinze and M. Ploner, A SAS macro, S-PLUS library and R package to perform logistic regression without convergence problems. Technical Report 2/2004. Medical University of Vienna. p. 36.

To-do’s

• Profile Penalized Likelihood CIs

• Modify ml d0