1

Hester van EerenErasmus Medical Centre, Rotterdam

Halsteren, August 23, 2010

2

The propensity score (1)The propensity score is “…the conditional

probability of assignment to a particular treatment given a vector of observed covariates.” (Rosenbaum en Rubin, 1983: 41).• Used in non-randomized studies to control for

selection bias• Balance observed pretreatment variables among

patient• Find an estimate of the average treatment effect

But, treatment effect can be different within subgroups

3

The propensity score (2)Univariate propensity score Multivariate propensity score

Bartak and colleagues (2009) Spreeuwenberg and colleagues (2010)

Used for 2 treatment categories Used for > 2 treatment categories

Propensity score used in: • Matching• Stratification• Regression• Inverse probability weight• Combinations of …

Methods in this studyTo find a treatment effect within subgroups, if the

propensity score is applied:• Method 1: Regression analysis with propensity score,

subgroups and interaction with treatment assignment;

• Method 2: Weighted regression analysis with inverse of the propensity score (to weight observations), subgroups and interaction with treatment assignment;

• Method 3: Propensity score applied for groups defined on treatment assignment and subgroups; then, regression analysis with propensity score and dummies for groups

Two treatment categories and two subgroups are used in this study

4

Variable selection for propensity scoreDoes the variable for subgroups has to be included?Discussion about variable selection for propensity score;

• Only variables related to outcome?• Only variables related to treatment assignment?• Both variables…?

In this study; 8 different propensity scores (PS) formulated, based

on: • Variables related to outcome, with and without subgroup• Variables related to treatment assigment, with and without

subgroup• Both variables…, with and without subgroup• Only variables related to both outcome and treatment

assignment, with and without subgroup5

How to test? (1)Real dataset not useful because effects

unknown beforehand; You cannot test whether the effect found is

accurate

Monte Carlo simulation study to test methods and different propensity scores:Simulate data with known treatment effectsEstimate different propensity scores for this dataApply different methods for different propensity scores,

for this data Repeat this process 1000 times

6

How to test? (2)What do you want to know?

If the treatment effect estimated is (almost) equal to the treatment effect you used to simulate the data

Bias of estimator: difference between estimated treatment effect and the true value of parameter

Want to have an unbiased estimate; Less bias indicates a more accurate estimate of the

treatment effect

Bias is estimated for overall treatment effect and for the treatment effect within subgroups

7

Results

8

N=250;ρ=0 N=250;ρ=0.3 N=250; ρ=0.7 N=500;ρ=0 N=500;ρ=0.3 N=500; ρ=0.7 N=1000;ρ=0 N=1000;ρ=0.3 N=1000; ρ=0.7

Row Method Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSERegression1 PS1, func 1 .0736 .0243 .0706 .0265 .0797 .0304 .0844 .0160 .0763 .0159 .0803 0.172 .0812 .0109 .0786 .0113 .0834 .0135

2 PS1, func 2 .0728 .0237 .0724 .0258 .0785 .0288 .0853 .0156 .0774 .0156 .0807 .0169 .0801 .0104 .0779 .0110 .0825 .0130

3 PS1, func 3 -.0060 .0312 -.0050 .0336 -.0014 .0345 .0042 .0149 -.0025 .0162 .0006 .0168 -.0001 .0071 -.0053 .0080 .0029 .0092

4 PS1, subgr. -.0035 .0820 -.0056 .0762 -.0008 .0719 .0030 .0414 .0007 .0382 .0000 .0398 .0006 .0218 .0085 .0193 -.0014 .0178

5 PS2, func 1 .0746 .0262 .0710 .0290 .0798 .0317 .0860 .0175 .0745 .0171 .0804 .0180 .0813 .0116 .0787 .0119 .0838 .01396 PS2, func 2 .0740 .0257 .0730 .0281 .0785 .0298 .0864 .0171 .0755 .0166 .0807 .0175 .0804 .0112 .0778 .0116 .0830 .0135

7 PS2, func 3 -.0049 .0343 -.0055 .0371 -.0023 .0368 .0070 .0169 -.0053 .0183 .0012 .0178 .0002 .0083 -.0061 .0088 .0033 .0098

8 PS2, subgr. -.0030 .0901 -.0025 .0892 .0020 .0765 -.0015 .0467 .0028 .0448 -.0014 .0433 .0006 .0243 .0105 .0220 -.0009 .0196

9 PS3, func 1 .0727 .0237 .0723 .0258 .0787 .0289 .0852 .0156 .0774 .0156 .0807 .0169 .0802 .0104 .0779 .0110 .0824 .0130

10 PS3, func 2 .0740 .0237 0730 .0258 .0785 .0289 .0864 .0156 .0755 .0156 .0807 .0169 .0804 .0104 .0778 .0110 .0830 .0130

11 PS3, func 3 -.0062 .0312 -.0053 .0337 -.0017 .0346 .0039 .0149 -.0028 .0162 .0005 .0168 -.0001 .0071 -.0053 .0080 .0027 .0091

12 PS3, subgr. -.0030 .0819 -.0047 .0763 .0001 .0719 .0034 .0414 .0012 .0383 .0003 .0399 .0008 .0218 .0088 .0190 -.0011 .0178

13 PS4, func 1 .0738 .0257 .0731 .0281 .0782 .0299 .0863 .0170 .0756 .0166 .0808 .0175 .0805 .0112 .0778 .0116 .0830 .0135

14 PS4, func 2 .0739 .0257 .0730 .0281 .0782 .0299 .0862 .0170 .0756 .0166 .0807 .0175 .0805 .0112 .0779 .0116 .0830 .0135

15 PS4, func 3 -.0051 .0343 -.0056 .0372 -.0029 .0370 .0068 .0169 -.0053 .0183 .0011 .0178 .0002 .0083 -.0061 .0088 .0030 .0098

16 PS4, subgr. -.0027 .0901 -.0020 .0894 .0027 .0766 -.0012 .0466 .0033 .0449 -.0012 .0433 .0007 .0243 .0107 .0220 -.0006 .0197

17 PS5, func 1 .0838 .0345 .0855 .0383 .0859 .0357 .0809 .0201 .0785 .0213 .0779 .0196 .0827 .0138 .0768 ..0126 .0812 .0130

18 PS5, func 2 .0836 .0332 .0850 .0370 .0872 .0345 .0806 .0193 .0769 .0202 .0778 .0189 .0824 .0136 .0766 .0123 .0800 .0126

19 PS5, func 3 .0047 .0407 .0027 .0440 .0129 .0396 -.0018 .0209 -.0056 .0209 -.0026 .0183 -.0002 .0104 -.0002 .0100 .0000 .0085

20 PS5, subgr. -.0023 .0898 .0061 .0895 -.0135 .0868 .0064 .0481 .0066 .0421 .0013 .0381 .0067 .0224 -.0081 .0220 -.0006 .0184

21 PS6, func 1 .0836 .0332 .0849 .0369 .0871 .0346 .0805 .0193 .0769 .0202 .0778 .0188 .0823 .0136 .0765 .0123 .0800 .0126

22 PS6, func 2 .0836 .0332 .0850 .0370 .0871 .0345 .0806 .0193 .0768 .0202 .0778 .0189 .0823 .0136 .0765 .0123 .0801 .0126

23 PS6, func 3 .0046 .0407 .0023 .0439 .0126 .0395 -.0019 .0210 -.0059 .0209 -.0027 .0183 -.0003 .0104 -.0004 .0100 -.0001 .0085

24 PS6, subgr. -.0021 .0898 .0069 .0834 -.0127 .0869 .0067 .0481 .0070 .0421 .0016 .0381 .0068 .0224 -.0079 .0220 -.0003 .0184

25 PS7, func 1 .0839 .0320 .1611 .0528 .1663 .0544 .0791 .0180 .1561 .0380 .1584 .0377 .0829 .0130 .1561 .0303 .1632 .0328

26 PS7, func 2 .0836 .0307 .1603 .0518 .1678 .0536 .0784 .0173 .1550 .0369 .1587 .0373 .0827 .0129 .1560 .0301 .1621 .0322

27 PS7, func 3 .0045 .0362 .0790 .0460 .0948 .0459 -.0037 .0180 .0724 .0242 .0786 .0235 .0017 .0093 .0794 .0154 .0827 .0152

28 PS7, subgr. -.0013 .0819 .0037 .0849 -.0162 .0828 .0055 .0435 .0068 .0402 .0005 .0379 .0029 .0207 -.0085 .0205 -.0022 .0185

29 PS8, func 1 .0837 .0307 .1602 .0517 .1679 .0537 .0783 .0173 .1551 .0369 .1587 .0373 .0827 .0129 .1560 .0301 .1621 .0323

30 PS8, func 2 .0836 .0307 .1603 .0517 .1680 .0538 .0784 .0173 .1550 .0369 .1587 .0373 .0827 .0129 .1560 .0301 .1621 .0323

31 PS8, func 3 .0043 .0362 .0787 .0458 .0947 .0459 -.0039 .0181 .0722 .0242 .0785 .0235 .0016 .0093 .0793 .0154 .0826 .0152

32 PS8, subgr. -.0008 .0819 .0046 .0848 .0059 .0436 .0073 .0402 .0009 .0379 .0009 .0379 .0031 .0207 -.0083 .0205 -.0019 .0185

N=250;ρ=0 N=250;ρ=0.3 N=250; ρ=0.7 N=500;ρ=0 N=500;ρ=0.3 N=500; ρ=0.7 N=1000;ρ=0 N=1000;ρ=0.3 N=1000; ρ=0.7

Row Method Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSERegression1 PS1, func 1 .0812 .0272 .1017 .0301 .0987 .0321 .0874 .0173 .1043 .0208 .0960 .0208 .0796 .0113 .1043 .0158 .1019 .0160

2 PS1, func 2 .0780 .0258 .0735 .0247 .0715 .0274 .0851 .0163 .0746 .0149 .0687 .0156 .0796 .0110 .0751 .0104 .0763 .0114

3 PS1, func 3 -.0011 .0322 -.0022 .0296 -.0131 .0358 .0094 .0157 -.0040 .0148 -.0138 .0175 -.0002 .0080 -.0041 .0075 -.0109 .0087

4 PS1, subgr. .0025 .0806 -.0087 .0797 .0253 .0913 -.0103 .0426 -.0009 .0375 .0186 .0441 -.0005 .0202 .0000 .0184 .0306 .0218

5 PS2, func 1 .0824 .0289 .0792 .0280 .0762 .0289 .0866 .0185 .0799 .0175 .0741 .0180 .0781 .0118 .0780 .0119 .0802 .0125

6 PS2, func 2 .0805 .0273 .0813 .0280 .0763 .0293 .0846 .0175 .0767 .0170 .0726 .0169 .0780 .0115 .0802 .0117 .0801 .0124

7 PS2, func 3 -.0009 .0341 .0035 .0347 -.0086 .0380 .0094 .0179 .0011 .0170 -.0102 .0185 -.0006 .0092 .0000 .0086 -.0072 .0098

8 PS2, subgr. .0030 .0863 -.0047 .0968 .0248 .0989 -.0112 .0475 -.0012 .0451 .0183 .0477 -.0031 .0224 .0020 .0208 .0294 .0246

9 PS3, func 1 .0798 .0258 .0790 .0254 .0769 .0280 .0852 .0163 .0733 .0158 .0735 .0163 .0795 .0110 .0803 .0112 .0807 .0121

10 PS3, func 2 .0805 .0258 .0813 .0253 .0763 .0279 .0846 .0163 .0767 .0158 .0726 .0162 .0780 .0110 .0802 .0112 .0801 .0120

11 PS3, func 3 -.0014 .0322 .0018 .0296 -.0117 .0359 .0093 .0157 -.0002 .0148 -.0129 .0175 -.0003 .0080 -.0003 .0075 -.0105 .0088

12 PS3, subgr. .0033 .0806 -.0050 .0806 .0352 .0930 -.0098 .0426 .0027 .0377 .0286 .0451 -.0002 .0202 .0034 .0186 .0400 .0228

13 PS4, func 1 .0804 .0273 .0810 .0277 .0764 .0289 .0847 .0175 .0798 .0170 .0727 .0170 .0780 .0115 .0801 .0117 .0801 .0124

14 PS4, func 2 .0805 .0273 .0810 .0276 .0760 .0287 .0847 .0175 .0797 .0170 .0728 .0169 .0780 .0115 .0800 .0117 .0800 .0124

15 PS4, func 3 -.0010 .0342 .0030 .0347 -.0088 .0376 .0093 .0179 .0010 .0170 -.0102 .0185 -0007 .0092 -.0003 .0085 -.0073 .0097

16 PS4, subgr. .0033 .0863 -.0042 .0972 .0252 .0992 -.0109 .0476 -.0009 .0450 .0189 .0478 -.0029 .0224 .0021 .0209 .0294 .0247

17 PS5, func 1 .0851 .0343 .0769 .0326 .0800 .0330 .0875 .0213 .0784 .0201 .0752 .0191 .0830 .0139 .0787 .0134 .0729 .0118

18 PS5, func 2 .0845 .0329 .0758 .0317 .0786 .0321 .0873 .0204 .0789 .0195 .0744 .0184 .0830 .0136 .0779 .0129 .0743 .0116

19 PS5, func 3 .0062 .0410 -.0024 .0430 -.0029 .0403 .0078 .0201 -.0016 .0208 -.0123 .0185 .0009 .0097 -.0038 .0105 -.0092 .0095

20 PS5, subgr. -.0046 .0945 -.0029 .0899 .0144 .0955 -.0010 .0465 .0031 .0442 .0288 .0450 .0054 .0243 .0063 .0252 .0189 .0236

21 PS6, func 1 .0844 .0329 .0756 .0317 .0786 .0322 .0872 .0204 .0791 .0196 .0743 .0184 .0830 .0136 .0781 .0129 .0742 .0116

22 PS6, func 2 .0844 .0330 .0756 .0317 .0783 .0321 .0872 .0204 .0790 .0195 .0741 .0183 .0830 .0136 .0780 .0129 .0741 .0115

23 PS6, func 3 .0059 .0410 -.0030 .0430 -.0034 .0405 .0075 .0201 -.0016 .0208 -.0128 .0185 .0009 .0098 -.0039 .0105 -.0091 .0095

24 PS6, subgr. -.0041 .0934 -.0019 .0899 .0153 .0958 -.0006 .0465 .0035 .0442 .0296 .0451 .0056 .0243 .0065 .0252 .0185 .0234

25 PS7, func 1 .0831 .0303 .0187 .0592 .1980 .0650 .0875 .0199 .1881 .0478 .1941 .0505 .0816 .0128 .1882 .0420 .1939 .0439

26 PS7, func 2 .0819 .0289 .1426 .0437 .1426 .0447 .0875 .0192 .1461 .0332 .1389 .0318 .0815 .0125 .1448 .0272 .1406 .0256

27 PS7, func 3 .0046 .0359 .0677 .0433 .0672 .0412 .0080 .0185 .0677 .0231 .0585 .0210 -.0006 .0087 .0645 .0137 .0640 .0131

28 PS7, subgr. -.0068 .0852 -.0101 .0820 .0002 .0883 -.0010 .0437 -.0018 .0410 .0136 .0434 .0053 .0220 .0031 .0229 .0026 .0220

29 PS8, func 1 .0818 .0289 .1428 .0438 .1437 .0450 .0876 .0192 .1464 .0333 .1402 .0322 .0815 .0124 .1451 .0273 .1419 .0259

30 PS8, func 2 .0818 .0289 .1429 .0438 .1434 .0450 .0876 .0192 .1464 .0333 .1398 .0320 .0815 .0125 .1451 .0273 .1416 .0258

31 PS8, func 3 .0043 .0358 .0660 .0433 .0626 .0409 .0078 .0185 .0665 .0230 .0537 .0206 -.0006 .0087 .0631 .0135 .0595 .0126

32 PS8, subgr. -.0062 .0853 -.0051 .0821 .0146 .0892 -.0005 .0437 .0022 .0411 .0284 .0444 .0055 .0220 .0071 .0230 .0165 .0224

N=250;ρ=0 N=250;ρ=0.3 N=250; ρ=0.7 N=500;ρ=0 N=500;ρ=0.3 N=500; ρ=0.7 N=1000;ρ=0 N=1000;ρ=0.3 N=1000; ρ=0.7

Row Method Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSE Bias MSEInverse PS1 PS1, func 1 .0831 .0298 .1116 0377 .1402 .0712 .0892 .0181 .1054 .0244 .1127 .0409 .0810 .0118 .1041 .0173 .1102 .0311

2 PS1, func 2 .0835 .0283 .0714 .0295 .0738 .0490 .0881 .0173 .0611 .0165 .0478 .0257 .0816 .0117 .0597 .0101 .0483 .0175

3 PS1, func 3 .0049 .0380 -.0027 .0492 -.0034 .0781 .0120 .0198 -.0171 .0267 -.0311 .0473 .0027 .0099 -.0191 .0136 -.0387 .0313

4 PS1, subgr. .0015 .1177 -.0018 .1667 .0181 .2239 -.0068 .0634 .0032 .0881 .0136 .1394 -.0016 .0307 .0008 .0411 .0279 .0811

5 PS2, func 1 .0921 .0380 .1012 .0524 .1384 .0801 .0940 .0236 .0872 .0356 .1012 .0504 .0831 .0147 .0838 .0207 .0933 .0471

6 PS2, func 2 .0922 .0354 .1058 .0467 .1239 .0646 .0940 .0227 .0889 .0303 .0959 .0376 .0837 .0143 .0856 .0193 .0917 .0338

7 PS2, func 3 .0120 .0494 .0362 .0667 .0450 .0866 .0182 .0282 .0120 .0376 .0154 .0556 .0032 .0140 .0044 .0225 .0011 .0474

8 PS2, subgr. .0076 .1496 -.0095 .2144 .0219 .2375 -.0058 .0818 .0038 .1280 .0181 .1561 .0038 .0418 .0086 .0596 .0391 .1124

9 PS3, func 1 .0818 .0285 .0903 .0329 .1257 .0649 .0871 .0171 .0821 .0196 .1008 .0360 .0811 .0116 .0810 .0130 .1007 .0269

10 PS3, func 2 .0820 .0283 .0905 .0313 .1157 .0496 .0875 .0172 .0823 .0187 .0963 .0286 .0812 .0116 .0812 .0127 .0961 .0214

11 PS3, func 3 .0033 .0384 .0159 .0476 .0391 .0701 .0111 .0198 .0043 .0251 .0120 .0392 .0023 .0099 .0029 .0124 .0117 .0246

12 PS3, subgr. .0014 .1187 -.0026 .1638 .0108 .2112 -.0064 .0635 .0016 .0855 .0024 .1268 -.0015 .0307 -.0016 .0399 .0177 .0716

13 PS4, func 1 .0895 .0368 .1025 .0526 .1384 .0821 .0920 .0229 .0868 .0348 .1006 .0498 .0830 .0143 .0842 .0207 .0931 .0467

14 PS4, func 2 .0903 .0359 .1026 .0455 .1225 .0592 .0932 .0227 .0879 .0293 .0981 .0360 .0833 .0143 .0849 .0187 .0914 .0321

15 PS4, func 3 .0100 .0510 .0327 .0669 .0429 .0815 .0171 .0288 .0109 .0376 .0180 .0525 .0027 .0141 .0035 .0218 .0013 .0449

16 PS4, subgr. .0076 .1534 -.0092 .2170 .0230 .2368 -.0052 .0828 .0038 .1285 .0162 .1545 .0040 .0421 .0087 .0595 .0377 .1125

17 PS5, func 1 .0957 .0458 .0994 .0595 .1335 .0816 .0918 .0273 .0914 .0350 .1102 .0612 .0823 .0162 .0830 .0219 .0972 .0316

18 PS5, func 2 .0983 .0440 .0985 .0523 .1210 .0651 .0936 .0254 .0930 .0312 .1025 .0417 .0830 .0158 .0843 .0199 .0960 .0249

19 PS5, func 3 .0298 .0619 .0213 .0845 .0455 .0861 .0182 .0295 .0075 .0407 .0211 .0487 .0012 .0147 .0027 .0225 .0172 .0256

20 PS5, subgr. -.0204 .1722 .0117 .2172 .0125 .2358 -.0063 .0885 .0235 .1119 .0204 .1491 .0074 .0437 .0092 .0614 .0047 .0745

21 PS6, func 1 .0959 .0446 .0970 .0587 .1326 .0812 .0914 .0262 .0912 .0350 .1097 .0599 .0823 .0160 .0822 .0216 .0974 .0308

22 PS6, func 2 .0972 .0438 .0963 .0523 .1221 .0604 .0925 .0254 .0914 .0307 .1034 .0391 .0826 .0158 .0839 .0198 .0934 .0231

23 PS6, func 3 .0287 .0621 .0191 .0861 .0469 .0813 .0173 .0296 .0058 .0407 .0230 .0450 .0006 .0148 .0024 .0223 .0142 .0245

24 PS6, subgr. -.0208 .1731 .0115 .2208 .0111 .2352 -.0068 .0894 .0237 .1130 .0176 .1505 .0076 .0439 .0088 .0621 .0059 .0749

25 PS7, func 1 .0843 .0318 .1923 .0677 .2242 .1002 .0889 .0204 .1925 .0520 .2136 .0765 .0809 .0131 .1889 .0436 .2088 .0585

26 PS7, func 2 .0851 .0306 .1346 .0473 .1166 .0568 .0901 .0198 .1359 .0325 .1082 .0370 .0813 .0128 .1301 .0245 .1063 .0233

27 PS7, func 3 .0116 .0434 .0576 .0604 .0425 .0760 .0114 .0219 .0520 .0302 .0337 .0395 -.0012 .0107 .0502 .0160 .0332 .0199

28 PS7, subgr. -.0114 .1236 .0041 .1558 .0087 .2113 -.0007 .0648 .0165 .0792 .0022 .1262 .0076 .0325 .0032 .0407 -.0091 .0603

29 PS8, func 1 .0836 .0304 .1499 .0528 .1819 .0787 .0889 .0197 .1519 .0376 .1730 .0571 .0809 .0127 .1466 .0292 .1706 .0418

30 PS8, func 2 .0840 .0304 .1502 .0510 .1746 .0658 .0832 .0197 .1515 .0366 .1670 .0479 .0810 .0127 .1469 .0290 .1646 .0368

31 PS8, func 3 .0102 .0435 .0729 .0610 .1027 .0722 .0103 .0220 .0674 .0313 .0939 .0400 -.0016 .0107 .0672 .0176 .0932 .0243

32 PS8, subgr. -.0111 .1241 .0026 .1530 -.0049 .1924 -.0003 .0651 .0154 .0782 -.0076 .1122 .0079 .0325 .0015 .400 -.0174 .0522

Results (1)Which propensity score is the most accurate within

each method tested (tested with ANOVA):

But, some values for bias per propensity score where not very different from each other…

9

General treatment effect

Treatment effect within subgroups

Method 1

PS with variables related to outcome

PS with variables related to outcome

Method 2

PS with variables related to outcome and variable for subgroups

PS with variables only related to outcome and treatment assignment and variables for subgroups

Method 3

PS with variables related to outcome

NA

Results (2)Which method is most accurate when the most

accurate propensity scores are compared?Decide on partial effect size of method in ANOVA*

For general treatment effect, the partial effect size is 0,028, where method 1 gives the lowest bias (followed by method 3)

For treatment effect within subgroups, the partial effect size is 0,051, where method 1 gives the lowest bias too

Although the effect sizes for method are not very large, regression analysis with treatment assignment, subgroup, interaction between these and the propensity score, which is estimated with variables related to outcome, seems to be the most accurate method to find treatment effects within subgroups

*Effect size – 0,010 = small; 0,059 = medium; 0,138 = large (Cohen, 1988) 10

Discussion (1)Data simulation is done for different settings:

Sample size, correlation between covariates and correlation with covariate for subgroups are changed over simulations

Results for most accurate propensity score are based on sum of bias over all these settings; comparisons between methods for all propensity scores could give more in depth results

The overall bias for different propensity scores was sometimes not very different

Model for simulation of data was simple, linear; the relation between variables and outcome in practice can be more complicated

…. 11

Discussion (2)

Questions?

12