Statistical Approaches to Assessing Single and Multiple Outcome Measures in Dry Eye Therapy and...

Innovative Techniques and Technology
GARY N. FOULKS, MD, EDITOR
Statistical Approaches to Assessing Singleand Multiple Outcome Measuresin Dry Eye Therapy and Diagnosis

ALAN TOMLINSON, PHD, DSC, 1 MARIO HAIR, M.PHIL, 2 AND ANGUS MCFADYEN, PHD, 2

ABSTRACT Dry eye is a multifactorial disease which wouldrequire a broad spectrum of test measures in the monitoringof its treatment and diagnosis. However, studies havetypically reported improvements in individual measureswith treatment. Alternative approaches involve multiple,combined outcomes being assessed by different statisticalanalyses. In order to assess the effect of various statisticalapproaches to the use of single and combined test measuresin dry eye, this review reanalyzed measures from two pre-vious studies (osmolarity, evaporation, tear turnover rate,and lipid film quality). These analyses assessed the measuresas single variables within groups, pre- and post-interventionwith a lubricant supplement, by creating combinations ofthese variables and by validating these combinations withthe combined sample of data from all groups of dry eyesubjects. The effectiveness of single measures and combi-nations in diagnosis of dry eye was also considered.

KEY WORDS dry eye, outcome measures, statisticalanalysis, tear physiology

Accepted for publication May 2013.

From 1Vision Sciences, Glasgow Caledonian University, Glasgow, UnitedKingdom and 2Independent Statistics Consultants, Glasgow.

This work was supported by a research grant from Pfizer Inc (IIRWS887596).

The authors have no commercial or proprietary interest in any concept orproduct discussed in this article.

Single-copy reprint requests to: Alan Tomlinson, PhD, DSc (Addressbelow).

Corresponding author: Professor Alan Tomlinson, PhD, DSc, Departmentof Vision Sciences, Glasgow Caledonian University, Cowcaddens Road,Glasgow G4 0BA, UK. Tel: 0141-331-3380. Fax: 0141-331-3387. E-mailaddress: [email protected]

� 2013 Published by Elsevier Inc. The Ocular Surface ISSN: 1542-0124.Tomlinson A, Hair M, McFadyen A. Statistical approaches to assessing sin-gle and multiple outcome measures in dry eye therapy and diagnosis.2013;11(4):267-284.

THE OCULAR SURFACE / OCTOBER 2013, VO

I. INTRODUCTION

S tudies attempting to define dry eye have variedwidely in the test measures employed and in theeffectiveness of the diagnosis. Criteria used for defi-

nition and classification have been based on symptom-atology, clinical test results (Schirmer wetting, tear filmbreakup time, ocular surface staining) and a range of tearphysiology tests that measure distribution, stability, evapo-ration, etc. The number and variety of tests used in variousstudies have made it difficult to assess prevalence levels ofdry eye and the efficacy of treatment regimens. Descriptionsof the assessment techniques and their advantages andshortcomings have been well described in recent litera-ture.1-7

The multifactorial etiology of dry eye disease makes itunlikely that a single test can provide a complete assessmentof the condition; for example, if the only study outcome is achange in the Schirmer test, this may be inadequate for theassessment of subtypes such as evaporative dry eye ormeibomian gland disease. The effectiveness of definitionand diagnosis can be improved if multiple, combined out-comes are assessed by different statistical approaches.Previously, we have shown the benefit of discriminant func-tion analysis in the diagnosis of dry eye,8,9 In recent inter-national workshops re-evaluating dry eye disease, multipletests have been advocated for diagnosis and evaluation oftherapeutic effects.3,6

There are significant problems in comparing the effec-tiveness of single and combined test measures in the pub-lished dry eye studies because of differences in studydesign, test technique, criteria (cut-off value) for diagnosis,and the ever-present problem of selection and spectrumbias. A comprehensive survey by Khanal et al of publishedstudies lists diagnostic sensitivities and specificities varyingfrom 10-99% and 49-100%, respectively.8 In the present re-view, data from two previous studies of the tear physiologyof dry eye subjects (osmolarity [OSM], evaporation [EVAP],tear turnover rate [TTR], and lipid film quality [TFI]) werereanalyzed to compare directly single versus multipleoutcome measures in the effectiveness of diagnosis and the

L. 11 NO. 4 / www.theocularsurface.com 267

mailto:[email protected]

http://www.theocularsurface.com

OUTLINE

I. Introduction

II. Statistical Methodology

III. Analyses of the Studies

A. Univariate Analysis of Measures

1. Descriptive Statistics

2. Evaporation Rate

3. Tear Turnover Rate

4. Osmolarity

5. Tear Film Structure

6. Summary

B. Multivariate Outcomes Measures: Separate Studies

1. Introduction

2. Factor Analysis Using Osmolarity and a Combi-nation of Evaporation and Tear Turnover Rate

3. The Effect of Different Measurement Units onthe Combinations

4. Using Indices to Compare the Effect of Treat-ment Groups over 30 Days

5. Summary

C. Multivariate Outcomes Measures: CombinedStudies

1. Introduction

2. Using Original Variables

3. Using Combination 1 and Combination 2 asDefined in Section III.B

4. Indices Comparing the Effectiveness ofTreatment

5. Comparing Multiple Outcome against IndividualMeasures of Tear Physiology in the Determina-tion of Effectiveness of Dry Eye Treatment

6. Diagnosing Dry Eye Patients at Baseline

7. Summary

IV. Summary and Conclusions

Abbreviations

ANCOVA Analysis of covarianceANOVA Analysis of varianceEVAP Evaporation rateHPMC Hydroxypropyl methylcelluloseKMO Kaiser-Meyer-Olkin (measure of sampling

adequacy)OSM OsmolarityTFI Lipid film qualityTTR Tear turnover rate

ASSESSING SINGLE AND MULTIPLE OUTCOME MEASURES / Tomlinson, et al

evaluation of therapeutic benefits of treatment. Thisapproach avoids many of the problems inherent incomparing results from different studies.5-7

In this review, we re-evaluated data from two studies ofdry eye conducted in our laboratory. In the first study,10 twogroups of subjects (n1¼27, n2¼26) were compared, and inthe second study,11 three groups (n3¼24, n4¼24, n5¼25)were evaluated. In both previous studies, the same fouraspects of tear physiology were measured as single variableswithin groups pre- and post-intervention with a lubricantsupplement. The techniques of measurement of these vari-ables, evaporimetry,12 interferometry,13 TTR,14 and tearOSM,10 have been well described in the literature.

The purpose of the current investigation was to evaluatethe effectiveness of single and multiple test measures indetermining the effectiveness of diagnosis and treatmentof dry eye disease. This was carried out by applying variousstatistical techniques to existing sets of data (a range oflaboratory measures of tear physiology) collected in ourlaboratory. By this means it was possible to avoid the biases

268 THE OCULAR SURFACE / OCTOBER 2013

and inconsistencies inherent in comparisons of the resultsfrom studies from different clinical sites. The statisticalmethods used ranged from Analysis of Variance (ANOVA)and Analysis of Covariance (ANCOVA) models for singlemeasure analyses, to multivariate statistical techniquessuch as factor analysis and discriminant analysis for multi-ple measures.

The difference between treatment groups on a singleoutcome measure is analyzed over time in Section III.A bythe commonly employed repeated measures ANOVAmodels. However, when the groups being compared havesignificantly different baseline levels, it is more appropriateto use ANCOVA with the baseline measures as the covari-ate, as illustrated in this section.

When considering multiple outcome measures statisti-cally as in Sections III.B and C, it is common practice toattempt to reduce the dimensionality, ie, number of vari-ables, while still retaining a large proportion of the variationof the data. A commonly used technique, factor analysis, isused in Sections III.B.1 and 2 and III.C.2 and 3 to createfactors (linear combinations of the original variables whichrepresent the original variables). These factors or indicesare usually independent of one another (orthogonal). Oncea number of indices have been developed, the differencebetween treatment groups on each index over time isassessed in Sections III.B.4 and III.C.5, using ANOVA andANCOVA.

It is also possible to assess the effectiveness of theseindices for the initial diagnosis of dry eye. Multivariatediscriminant analysis and logistic regression are commonlyused in such clinical scenarios when the predicted outcomeis dichotomous (dry eye or not dry eye). However, when asingle independent index as an outcome variable is analyzed,the clinical interpretation is simpler with use of discriminantanalysis. This is illustrated in Section III.C.6.

II. STATISTICAL METHODOLOGYThe present study reassessed the effects of artificial tears

on aspects of tear physiology (EVAP, TTR, OSM and TFI)by adopting a range of approaches to the statistical analysesof the effect of treatment for dry eye. The physiology mea-sures were assessed first as single variables, then by creatingcombinations of these variables, and finally by repeatingwith the combined sample of data from five groups of dry

, VOL. 11 NO. 4 / www.theocularsurface.com



eye subjects. Additionally, the effect of single and combinedtest measures were assessed in the diagnosis of the disease.

The analyses are discussed in three sections.In Section III.A, instead of taking a simple pre-post

method of analysis for each outcome measure within eachgroup, the single variables in each study were reanalyzedincorporating a within and between group analysis usingtwo-factor repeated measures ANOVA models,15 where theinteraction between groups over time was the key outcome.This analysis enabled more direct comparisons of the efficacyof the treatment methods.

In Section III.B, the variables were treated as a multivar-iate sample within each study. A factor analysis model16 wasdeveloped for each of the previous studies. This analysisreduced the dimensionality of each study sample by creatinglinear combinations of the existing variables known as fac-tors, ie, the combined variables. Any factors obtained wereanalyzed using the same two-factor repeated measuresANOVA procedure outlined in Section III.A.

In Section III.C, the five sets of data from the two studieswere combined for a sample of 126 cases for each of theoutcome measures, and the analysis of Section III.C wasreplicated with this larger sample. It was hoped that thisanalysis would produce stronger estimates of the factorstructure allowing a five-group repeated measures ANOVAmodel to be run. This was intended to make final compar-isons of the efficacy of each intervention using a multiplemeasurement scale.

All statistical analysis was carried out using SPSSversion18. The detailed description of the process andoutcome of each analysis is contained in the following sec-tions, together with a summary of that section. Each sectionalso highlights the advantages and disadvantages of employ-ing such statistical methods in analyzing outcomes measuresfrom the study of ocular surface disease. Section IV providesa review of all the analyses and the comparison of singleversus multiple measures.

III. ANALYSES OF THE STUDIESA. Univariate Analysis of Measures

Univariate analysis is a set of statistical techniquesemployed when a single outcome variable is studied in isola-tion. The assumption is that other outcome variables havelittle or no relationship with the variable analyzed. In clinicalresearch in ocular surface disease, such independence of asingle isolated variable has obvious disadvantages, giventhe links between several variables dependent on the multi-ple etiological factors in this complex disease. Conversely,the advantage is that the actual statistical analysis itself isrelatively simple and thus achievable by most researchers.However, results from univariate analysis may be misleadingand often lack the ability to be generalized.

McCann et al compared the efficacies of three differenttreatments for the management of lipid-deficient dry eye–sodium hyaluronate, hydroxypropyl methylcellulose (HPMC),and an oil-in-water emulsion.11 There were 24 subjects in


the sodium hyaluronate and emulsion groups and 25 subjectsin the HPMC group. Each treatment was assessed by fouraspects of tear physiology measured at baseline (baseline),30 days post-treatment, and 90 days post-treatment. Onlythe first two measures, baseline and 30 days, are used in thisstudy to allow comparison with the Khanal data, which hadmeasures out to 30 days.

Khanal et al compared the efficacies of two differenttreatments for the management of lipid-deficient dry eye–a conventional dry eye supplement (hypromellose) and anoil-in-water emulsion eye drop.10 There were 26 subjectsin the hypromellose group and 27 subjects in the emulsiongroup. Each treatment was assessed by four aspects of tearphysiology measured at baseline and again at 30 dayspost-treatment.

Throughout the text of this review, the studies will bereferred to as “McCann” or “Khanal” data. The variablesin both studies were EVAP rate, TTR, OSM, and TFI. Thelast variable was dichotomous (stable or unstable), whilethe other three were numeric and measured on an intervalscale.

In this section, we reanalyzed the baseline and 30-daynumeric data using a two-factor repeated measuresANOVA. By using the ANOVA model, single variableswere still being analyzed as in the original studies, but thetwo-factor model allowed the contribution of both the treat-ments and the time effects to be assessed at the same time.The repeated measures aspect was essential, given that thesame subjects had the clinical outcome response recordedon two occasions. One factor was a between-group 2 or 3level treatment factor (GROUP), the other factor was atwo-level (baseline and 30 days) within-group repeated fac-tor (TIME). TFI, which was a dichotomous variable, wastested using Fisher’s exact test17 on the overall time effectand the chi-square test17 on between-group data.

1. Descriptive StatisticsTable 1 gives the descriptive statistics for each measure

for each group for the McCann and Khanal studies. Meansand standard deviations are given for numeric data and per-centages for categorical data. It was noted that the baselineEVAP score in the emulsion group of the McCann studywas considerably higher than for any other group.

Each numeric variable was tested for normality at base-line and 30 days for each group in each study, using the Kol-morgorov Smirnov nonparametric test. None of thevariables were significantly nonnormal (the smallest p-valuewas .370). Outliers were checked when appropriate, but theyare discussed below only when they substantively affectedthe results. Researchers must, however, always consider out-liers with great care, given that they may be incorrectbecause of errors in measurement or data entry. Stray valuesmay also be due to poor inclusion or exclusion control orrepresent a group of individuals who should be includedbut are severely underrepresented in the sample. Outliersshould never simply be deleted and assumed to be errors.



Table

1.Descriptiv

estatisticsfortheMcC

annan

dKh

anal

data

Group

McC

anndata

Khan

aldata

HYALU

RONATE

(N[

24)

HPM

C(N

[25

)EM

ULS

ION

(N[

24)

Total

(N[

73)

EMULS

ION

(N[

27)

HYPR

OMEL

LOSE

(N[

26)

Total

(n[

53)

Mea

n(St.Dev

)Mea

n(St.Dev

)

EVAPat

baselin

e31

.99(15.43

)32

.79(15.17

)47

.84(21.46

)37

.47(18.80

)34

.19(17.21

)31

.13(16.09

)32

.69(16.58

)

EVAPat

30da

ys22

.98(9.53)

25.18(13.20

)23

.83(9.68)

24.01(10.85

)23

.35(11.45

)21

.96(7.13)

22.67(9.51)

TTRat

baselin

e9.43

(3.97)

9.92

(4.59)

8.68

(3.22)

9.36

(3.95)

8.72

(3.82)

8.73

(3.20)

8.73

(3.49)

TTRat

30da

ys9.83

(3.94)

9.90

(4.95)

9.30

(3.47)

9.68

(4.13)

10.46(3.59)

8.93

(1.72)

9.71

(2.91)

OSM

atba

selin

e31

2.78

(4.97)

311.92

(5.04)

314.46

(6.45)

313.04

(5.55)

311.49

(7.58

311.52

(5.14)

311.51

(6.44)

OSM

at30

days

311.97

(3.72)

313.38

(8.85)

311.10

(3.97)

312.16

(6.04)

310.67

(5.70)

310.48

(5.85)

310.58

(5.72)

TFI%

stab

leat

baselin

e0%

0%0%

0%44

.4%

57.7%

50.9%

TFI%

stab

leat

30da

ys20

.8%

12.0%

37.5%

23.3%

74.1%

61.5%

67.9%

Figure 1. Fall in evaporation rates over time by treatment group:McCann data.



2. Evaporation RateFor the McCann data, we carried out a mixed ANOVA

with a three-level between-groups factor (GROUP) and atwo-level (baseline and 30 day) within-groups repeated fac-tor (TIME). There was a significant time effect, with a signif-icant fall in mean EVAP rate across all three groups from37.47 at baseline to 24.01 at 30 days (P<.001). However,there was no significant group effect; there was no changein the mean “average EVAP rate” (averaging the baselineand 30-day scores) between the three groups (P¼.053). Asignificant interaction effect was found between TIME andGROUP (P<.001), with the emulsion group showing asignificantly larger fall in mean EVAP rate (24.01) comparedto either the sodium hyaluronate (9.01) or HPMC (7.61)group.

Figure 1 shows the change in mean EVAP rates acrossthe three groups over time. It was clear that the emulsiongroup has a much larger fall than either of the other groups.However, as was noted earlier, the emulsion group had amuch higher initial EVAP rate than either of the othergroups. There was the possibility that the large fall was sim-ply because the group had abnormally large EVAP rates atbaseline and that other treatments would also have regis-tered a similar fall in those circumstances.

Whenever there are two sets of readings, pre- and post-treatment, and especially when the pre-treatment scoresvary widely, it is now widely recognized that the mostpowerful form of analysis is an analysis of covariance(ANCOVA) with the pre-treatment score included as a co-variate.18,19 By employing an ANCOVA model, allowancecan be made for one or more variables, which vary widelybetween groups and affect the results and clinical interpreta-tions of the effect. Such models are often employed to allowthe effect of, for instance, prior treatment to be built into amodel if it has not been possible to randomize this effectacross treatment groups. For this analysis it makes no differ-ence whether the dependent variable is the post-treatmentscore or the change between pre- and post-treatment scores.



Figure 2. Mean tear turnover rate over time by group: Khanal data.

Table 2. Estimated mean scores at 30 days corrected for baseline

Evaporation Tear turnover rate Osmolarity TFI (% stable)

McCann Groups

Hyaluronate 24.71 9.75 311.95 20.8

HPMC 26.66 9.34 313.30 12.0

Emulsion 20.56 9.97 311.19 37.5

Effect size 0.25 0.19 0.14

Khanal Groups

Hypromellose 22.27 8.93 310.67 3.8

Emulsion 23.05 10.46 310.48 29.7

Effect size 0.04 0.35 0.02

TFI, lipid film quality.


Accordingly, a one-way between-groups ANCOVA wascarried out with the 30-day EVAP rate as the dependent var-iable and the baseline EVAP rate as a covariate. As expected,the ANCOVA showed a significant difference in baselineEVAP rates by group (P<.001). The model then estimatedthe EVAP rate at 30 days as if each of the groups had startedat the same mean baseline (these estimated means areshown in Table 2). Once allowance was made for this, therewas no significant difference between groups in the meanEVAP rate at 30 days (P¼.110).

The comparable mixed ANOVA model with a two-levelbetween-groups factor used to analyze the Khanal datafound a significant time effect (P<.001); the mean EVAPrate across both groups fell from 32.69 at baseline to 22.67at 30 days, but there was no significant group effect(P¼.463) or interaction effect (P¼.708).

As the baseline scores for each group were similar, anANCOVA analysis was not deemed necessary for this studydata.

3. Tear Turnover RateFor the McCann dataset, a mixed ANOVA was carried

out for measures of tear turnover. Overall, there was a sig-nificant time effect, with a significant increase in meanTTR across all three groups from 9.36 at baseline to 9.68at 30 days (P¼.040). There was no significant group effect,with no significant change in the mean ”average TTR” be-tween the three groups (P¼.716), and no significant interac-tion effect (P¼.249).

The comparable analysis for the Khanal data also found asignificant time effect (P¼.011); mean TTR increased acrossboth groups from 8.73 at baseline to 9.71 at 30 days, but therewas no significant group effect (P¼.347). There was, how-ever, a significant interaction effect (P¼0.039); the meanrise in TTR in the emulsion group over the 30 days wassignificantly higher than the mean rise in the hypromellosegroup (1.74 against 0.20 [Figure 2]). One subject in theemulsion group had a large rise in TTR (7.54 at baseline to


19.04 at 30 days), and when this subject was removed therewas no longer a significant interaction effect (P ¼.071), indi-cating that this result was not robust.

As the baseline scores for each group with each studywere similar an ANCOVA analysis was not deemednecessary.

4. OsmolarityA mixed ANOVA result for the McCann data yielded no

significant time effect (P¼.360); no significant change inmean OSM was found across the three groups from baseline(313.04) to 30 days (312.16). There was no significant groupeffect, with no change in the mean ”average OSM” betweenthe three groups (P¼.937), nor was there any significantinteraction effect (P¼.143); no group did any better overtime than another.

Overall, for the Khanal data, there was no significanttime effect (P¼.506), nor was there a significant group effect(P¼.931) or a significant interaction effect (P¼.937). In thisstudy, however, a large number of subjects had baselineOSM within ”normal” ranges20; hence, the change in OSM




after 30 days was generally not very large. This may accountfor the finding that there was no time effect.

As the baseline scores for each group were similar anANCOVA analysis was not deemed necessary.

5. Tear Film StructureTear film structure as an outcome measure is usually

measured on a categorical scale with various distinct valuesor, as in these studies, a simple dichotomous variable whereonly two values are possible, one representing a stablestructure and the other an unstable structure. By simply us-ing a dichotomous scale, any possible variation in tearinstability is lost and the analysis weakened. With only cat-egorical variables available (one representing the treatmentgroup and the other tear film structure as stable or not),statistical analysis was limited to chi-square and Fisher’sexact test.

For the McCann study none of the patients had a stablefilm structure at baseline, as tear film instability was one ofthe inclusion criteria. At 30 days, 9 (37.5%) of the subjects inthe emulsion group, 5 (20.8%) in the hyaluronate group, and3 (12.0%) in the HPMC group showed stable film structure.Overall, 23.3% of the sample had a stable film structure after30 days. A Fisher’s exact test on the overall change in tearfilm quality between baseline and 30 days showed a signifi-cant increase in the percentage of stable cases after 30 days(P<.001). However, a chi-square test comparing the changebetween groups showed no significant difference betweengroups (P¼.101).

For the Khanal data, 12 (44.4%) patients in the emulsiongroup had a stable film structure at baseline and 20 (74.1%)at 30 days. Fifteen (57.7%) patients in the hypromellosegroup had a stable film structure at baseline and 16(61.5%) at 30 days. Overall, 27 (50.9%) patients had stablefilm structure at baseline and 36 (67.9%) at 30 days. The sta-tistical analysis showed a reversal of the results from theMcCann data. There was no overall change in tear film qual-ity between baseline and 30 days (P¼.113), but there was asignificant difference between groups (P¼.006).

6. SummaryThe desired change with treatment of dry eye is a lower

EVAP rate and OSM and higher TTR and percentage of sta-ble tear films.

For both datasets, there was a significant decrease inEVAP rates after 30 days, but there was insufficient evidenceto suggest that any one group did better than another oncedifferences in baseline scores were taken into account.

For both datasets, there was a significant increase inTTRs after 30 days. For the McCann data, there was no ev-idence to suggest that any group did better than another,and, although for the Khanal data there was a significantgroup effect, this result was not robust and omitting a singleoutlier resulted in a nonsignificant group effect.

For both datasets, there was no significant change inOSM after 30 days, and there was also no evidence to sug-gest that any group did better than another.


For the McCann data, there was a significant increase instable tear film structure after 30 days, but there was insuf-ficient evidence to suggest that any one group did betterthan another. For the Khanal data, there was no overall in-crease, but one group showed a significant improvement intear film stability after 30 days while the other group did not.

For the McCann data, there was no clear evidence of anydifference between the three treatment groups on any of thefour measures (EVAP, TTR, OSM, and tear film structure).For two measures in the Khanal data, TTR and tear film sta-bility, there was some evidence of a difference between thetwo groups. However, the result for TTR relied on the inclu-sion of a single outlier. Only tear film stability gave clear ev-idence that the one group showed greater improvement over30 days than the other group.

The ANCOVA analysis is capable of estimating the ef-fect size and the mean 30-day value of each numeric mea-sure for each group after differences in baseline score aretaken into account. Effect size is an objective measure ofthe magnitude of the group differences observed. An effectsize of 0.1 is considered a small effect, 0.3 a medium effect,and 0.5 a large effect.21,22 Table 2 gives the adjusted mean30-day values for each measure and the group effect size.The effect sizes in this study were all small-to-moderate,ranging with a maximum of 0.35. Small effect sizes requirelarger samples in order to find significant differences.

For the McCann dataset, the effect sizes were small, butEVAP had the largest effect size of the tear physiology mea-sures. For the Khanal dataset, only tear turnover had a mod-erate effect size. Hence, different conclusions as to theeffectiveness of therapy would be reached for each study,depending on which single measure was considered. Howev-er, in the McCann study, the emulsion group had the lowestEVAP and OSM rates and the highest TTR and percentageof stable tear films. Therefore, while each measure analyzedin isolation may not differentiate between the treatmentgroups, the consistency of the McCann results across allthe measures indicates that there may be some potentialin examining if a composite measure may be more effectiveand robust in comparing treatments.

B. Multivariate Outcomes Measures: Separate Studies1. Introduction

Multivariate statistical methods were specificallydesigned to enable analysis of complex datasets whereseveral outcome variables are measured and known to berelated and to have an effect on each other. Although orig-inally developed in other research areas, such methods havebeen employed in clinical research and specifically in dry eyeresearch for many years. The obvious advantage of thesemethods is the ability to explain complex clinical situationsin simpler ways. Large datasets are, however, recommended,and analysis is usually possible only with use of specializedstatistical software. In such methods, the dimensionality ofthe data is often referred to and is simply a representationof the number of cases by the number of outcome variables.Sample sizes are often said to be appropriate when the




number of cases (subjects) is 5 or even 10 times the numberof outcome measures. In ocular surface research, whereEVAP, TTR, and OSM, or their surrogates are often thethree main outcomes measures, this implies that a minimumof 15 subjects should be available, but perhaps 30 would bemore appropriate.

A factor analysis model using principal component anal-ysis16 was developed for each dataset in order to reduce thedimensionality of the tear physiology variables down to oneor two factors combining these variables. Such reduction ispossible if the original variables are sufficiently related, asthey might be when differing symptoms of the same under-lying condition are measured. If any consistent factors couldbe found, then these would be used to try to differentiate be-tween treatments using discriminant analysis.

Categorical variables are not suitable for either factor ordiscriminant analysis, so TFI had to be omitted from theanalysis. The remaining three variables – EVAP, OSM,and TTR – were entered into the factor model. Separate fac-tor models were run for the three measures at baseline, at 30days, and, finally, on the change over 30 days; change wascalculated throughout as baseline reading minus 30-dayreading so that a fall over 30 days registers as a positivescore. Table 3 shows the important results from these factormodels. In all cases, two factors were extracted with varimaxrotations when appropriate.23,24

Factor analysis is traditionally carried out on larger data-sets than those available here. The Kaiser-Meyer-Olkin[KMO] measure of sampling adequacy25,26 was used to deter-mine if the sample size was adequate to provide reliable fac-tors, ie, whether the data is adequate for the method beingemployed. Values can range from 0 to 1 and the KMO for

Table 3. Factor analysis models using evaporation (EVAP), tearand change over 30 days

Time Data set KMO % variatio

Baseline Khanal 0.53 39

32

McCann 0.50 38

33

30 days Khanal 0.45 38

35

McCann 0.50 42

32

Change Over 30 days Khanal 0.51 39

32

McCann 0.55 43

30

KMO, Kaiser-Meyer-Olkin (measure of sampling adequacy).


all of these models was around 0.5, indicating that the samplesize was only just acceptable. Moreover, for each model, therewas a nonsignificant Bartlett’s test of sphericity,15 which indi-cates that theremay not be sufficient shared variability (corre-lation) between the variables for factorization. The %variation in the table indicates howmuch of the total variationin the threemeasures was explained by the each factor. Gener-ally, this was good with a combined total of over 70% for eachmodel. The factor score coefficients give theweights to be usedto calculate each factor.

At baseline, two factors were derived for the Khanaldataset:

Factor 1 ¼ 0:76 � EVAP � 0:07 � TTR � 0:55 � OSM

Factor 2 ¼ �0:23 � EVAP þ 0:93 � TTR � 0:26 � OSM

and for the McCann data

Factor 1 ¼ 0:04 � EVAP þ 0:66 � TTR � 0:66 � OSM

Factor 2 ¼ 0:99 � EVAP � 0:12 � TTR � 0:07 � OSM

It was clear that there was little consistency in the factorsacross the two datasets. One of the two factors had a large pos-itiveweighting for EVAP, with smaller negativeweightings forTTR and OSM, and the weightings themselves vary consider-ably. There was even less consistency in the second factor.

The conclusion from this analysis was that no factoriza-tion works consistently across both datasets at either base-line, 30 days, or on the change across 30 days.

turnover rate (TTR) and osmolarity (OSM) at baseline, 30 days

n explained FactorScore coefficients

EVAP TTR OSM

.9% 1 0.76 �0.07 �0.55

.0% 2 �0.23 0.93 �0.26

.3% 1 0.04 0.66 �0.66

.4% 2 0.99 �0.12 �0.07

.3% 1 0.67 �0.65 0.02

.5% 2 0.31 0.34 0.85

.0% 1 0.08 0.74 0.55

.9% 2 0.90 �0.25 �0.30

.5% 1 0.71 �0.03 �0.61

.9% 2 �0.21 0.95 �0.20

.6% 1 0.64 �0.64 �0.07

.6% 2 �0.05 0.05 1.01




2. Factor Analysis Using Osmolarity and a Combinationof Evaporation and Tear Turnover RateFactor analysis is restricted to searching for linear combina-

tions of variables (a multiple of one plus a multiple of another,and so on). In dry eye patients, improvement in the condition isgenerally accompanied by a reduction in EVAP rate, an in-crease in TTR, and a decrease in OSM. A number of authorshave suggested that a composite measure including EVAPand TTR may be able to distinguish dry eye states from thenorm.4,5,8,27,28 Hence, it was decided to investigate whether afactor analysis with OSM and a combination of EVAP andTTRwould givemore consistent results across the twodatasets.

A number of mathematical combinations were investi-gated, but the following two seemed to have themost promise.

Combination 1 : EVAP=TTR

Combination 2 : EVAP=ðEVAP þ TTRÞ:
Both of these combinations have been discussed in the
literature as possible measures of proportion of tear produc-tion lost in the dry eye condition.29-32

These two combination scores were calculated for each pa-tient in each dataset at baseline, at 30 days, and as change over30 days. So, for example, Combination 1 was calculated as:

Combination 1_ baseline ¼ EVAP at baseline=

TTR at baseline;

Combination 1_ 30 days ¼ EVAP at 30 days=

TTR at 30 days;

Combination 1_Change ¼ ðEVAP at baseline=

TTR at baselineÞ� ðEVAP at 30 days=

TTR at 30 daysÞ:

Separate factor models were run using OSM with eachcombination at baseline, at 30 days, and finally on the change

Table 4. Factor analysis models using Combination 1 (EVAP/TTRat baseline, 30 days, and change over 30 days

Time Data set % variation explainedScore coeffic

Combination 1

Baseline Khanal 54.21% 0.68

McCann 56.01% 0.67

30 days Khanal 50.56% 0.70

McCann 57.49% 0.66

Change Khanal 57.31% 0.66

McCann 57.78% 0.66


over 30 days. Table 4 shows the important results from thesefactor models. In all cases, a single factor was extracted.

The KMO value for all of these models was 0.5, indi-cating, as before, that the sample was only just acceptable,and for each model there was a nonsignificant Bartlett’stest of sphericity, indicating that there may not be sufficientshared correlation between the variables. However, the %variation explained was reasonable at between 50% and60% for each model.

The factor score coefficients giving the weights to beused to calculate each factor were derived, and at baselinethe factor for Combination 1 (EVAP/TTR) and OSM was:

0:68 � Combination 1_baseline

� 0:68 � OSM at baseline ðfor the Khanal datasetÞ; and

0:67 � Combination 1_baseline

þ 0:67 � OSM at baseline ðfor the McCann dataÞ:The percentage variation explained was a little higher for

Combination 1, and there was slightly better consistencyacross datasets in coefficient size; however, there were differ-ences in signs. The problem underlying this was related to thecorrelation between the combination andOSM. For factoriza-tion to be effective, there had to be some shared variationacross variables and, in this case, where there were only twovariables, there needed to be some correlation between thevariables. The correlations between OSM and the combina-tions of EVAP and TTRwere low for both datasets at baseline,at 30 days, and for change over 30 days ranging from �0.275toþ0.156. Results such as these are typical of data from sam-ples of subjects in ocular surface research.

Theoretically, at baseline a positive correlation betweenCombination 1 and OSM might be expected. Dry eye isassociated with higher than average EVAP and lower thanaverage TTR, which would combine to give high valuesfor Combination 1. OSM would also be higher than averageso that the correlation between the two would be positive. Inthe McCann dataset, the correlation between Combination 1and OSM at baseline was positive at þ0.12, while for the

) and Combination 2 (EVAP/EVAPþTTR) and osmolarity (OSM)

ients% variation explained

Score coefficients

OSM Combination 2 OSM

�0.68 50.14% 0.71 0.71

0.67 50.20% 0.71 0.71

�0.70 51.66% 0.70 �0.70

�0.66 63.76% 0.63 �0.63

�0.66 54.41% 0.68 �0.68

0.66 51.48% 0.70 �0.70




Khanal data it was negative at �0.08. Neither correlationwas very large, but because they differ in sign this had theeffect of giving different factor loadings. It was the Khanaldata that was not as expected and, as discussed in SectionIII.A, the OSM measures for this data are lower than wouldbe expected of dry eye subjects. A positive correlation wouldalso be expected between Combination 2 and OSM. In fact,the correlation between Combination 2 and OSM was veryweak for both datasets (0.004 for the McCann data and0.003 for the Khanal data) but positive, which gives consis-tent factor structures. However, the lower correlations,which can be typical in ocular surface research, lead toless variation being explained by this combination comparedwith the other one.

At 30 days, it was more difficult to predict the expectedcorrelation between OSM and Combinations 1 and 2, as anyrelationship would be confounded by treatment effects. Butin practical terms, the 30-day results were of less importancethan either the baseline or the effect of change over 30 days.The baseline scores could be used to differentiate betweennormal and dry eyes, while the change over 30 days canbe used to assess treatments.

For change over 30 days, a positive correlation betweenOSM and both combinations would also be expected, withtreatments reducing the values of both OSM and thecombinations over 30 days. In theMcCann dataset, the corre-lation between the change in Combination 1 and OSM over30 days was þ0.15, while for the Khanal data it was �0.15.Once again, it was the Khanal data that was not as expected.

The correlation between the change in Combination 2and OSM over 30 days was low and negative for both data-sets (�0.03 for the McCann data and �0.09 for the Khanaldata). While this gives consistent factor structures, it was notas expected, and the smaller correlations meant that less ofthe variation was explained by the Combination 2 factoriza-tion. In fact, the low negative correlation for the McCanndata was a consequence of the different units used to mea-sure EVAP and TTR; this is discussed in the next section.

3. The Effect of Different Measurement Units on theCombinationsIn both the McCann and Khanal studies, EVAP was

measured in units of g/m2/h and TTR was in %/min.When creating composite measures of EVAP and TTR,the use of different units can be problematic. It is possibleto transform both readings into a common unit of ml/minas long as some assumptions are made. EVAP in units ofg/m2/h can be transformed to units of 10�7g/cm2/sec bydividing by a factor of 3.6.29 The EVAP rate in ml/m in isnumerically equal to 1/100 of the value of the EVAP ratein units of 10�7g/cm2/sec when the area of the evaporatingocular surface is assumed to be 167mm2. Hence, under thisassumption the EVAP rate can be transformed from units ofg/m2/h to ml/min by dividing by 1/360. The TTR in ml/mincan be calculated by the product of the TTR in %/min andtear volume. If a tear volume of 7 ml is assumed, then TTRcan be transformed from units of %/min to ml/min by


dividing by a factor of 14. [x%/min x 7¼x/100 x 7 zx/14] This assumption of a similar volume of tears in thedry eye patient and in the normal is open to question, asthe volume varies in the two states.33,34

The use of Combination 1 as a composite measure doesnot depend on units of measurement, as the only effect is tointroduce a constant. EVAP in units of g/m2/h divided byTTR in %/min is simply about 0.04 times the equivalentmeasure in ml/min. The constant does not affect the factoranalysis or the correlation with OSM. However, use of Com-bination 2 is affected by the units used, as the effect of trans-forming from one set of unit to the other is not simplymultiplication by a constant.

The preceding analysis for Combination 2 (EVAP/EVAP þ TTR) was repeated after transforming both vari-ables to common units of ml/min. In general, the use ofthe common units brought the results for Combination 2more in line with those for Combination 1 using the originalunits. For example, the negative correlation between OSMand Combination 2 for the change over 30 days on theMcCann data was replaced by a stronger positive correlationof þ0.137 (much closer to the correlation of 0.156 betweenOSM and Combination 1 using original units) when com-mon units were used. This resulted in a factor that was apositive sum of the combination and OSM, which explaineda greater percentage of variation 57.56% compared to51.48% (very similar to the percentage found for Combina-tion 1 using original units). The transformation of EVAPand TTR into common units was not considered necessaryfor the simpler Combination 1. It made little substantive dif-ference to Combination 2 except to bring the results closerto those of Combination 1.

The conclusion from this and previous sections was thatwhile Combination 2 gave consistent factors for both data-sets, the correlation between this combination and OSMwas so weak that it was debatable whether an index madeup of Combination 2 and OSM would be any improvementover Combination 2 alone. By comparison, Combination 1did have more potential, even though it was problematicfor the Khanal dataset, giving a higher correlation withOSM that was not affected by the measurement units used.

4. Using Indices to Compare the Effect of TreatmentGroups over 30 DaysFor each subject in the dataset a simple index for the

change over 30 days was created from the factorization bymultiplying the combination of EVAP and TTR and OSMby the appropriate weighting. Ideally, both scores shouldbe standardized before the index is created. Standardizationinvolves adjusting each patient’s score by subtracting thesample mean and dividing by the sample standard deviation.This was necessary, as the variance of change in OSM over30 days was around 10 times higher than the variance of thecombination scores. As the following analyses were based onvariance, this imbalance could have biased results towardthe effect of OSM. Standardization would reduce the vari-ance of each score to 1, eliminating bias.




However, it is recognized that standardization is notpractical in a clinical setting. A cruder but simpler expedientwould be to use the percentage change in OSM over 30 days.This had a variance in line with the combinations and sowould act as a surrogate for standardization.

Two indices based on the results of the factorization forthe change over 30 days were created:

Index 1_change ¼ 0:66 � Combination 1_Change � 0:66 � percentage change in OSM ðfor the Khanal data onlyÞ:

Index 2_change ¼ 0:66 � Combination 1_Change þ 0:66 � percentage change in OSM ðfor the McCann data onlyÞ:

As before:

yCombination 1_ Change ¼ ðEVAP at baseline=TTR at baselineÞ � ðEVAP at 30 days=TTR at 30 daysÞ:

(yNote: for simplicity these will be abbreviated to Index 1, 2and Combination 1 in this section).

In all cases, higher values of the index indicatedimprovement in the dry eye state. Table 5 gives the meanand standard deviation of the Index scores by treatmentgroup.

A one-way ANOVA was run with the Index as thedependent variable and a single between-group factor con-sisting of the treatment groups in each dataset. For Index1 with the Khanal data, there was no significant differencein the mean Index scores between the two groups(P¼.389). For Index 2 with the McCann data, there was asignificant difference in the mean Index scores betweenthe three groups (P¼.002). The emulsion treatment grouphad a significantly higher mean Index score than eitherthe HPMC or the hyaluronate treatment groups (Bonferronipost hoc test, P<0.05). Figure 3 shows the mean differencesby group.

It was noted earlier that for the McCann dataset therewas a significant difference in mean change in EVAP bytreatment group, but once baseline EVAP scores were

Table 5. Mean and standard deviation of index scoresby treatment

Treatment Group NMean

(Std. Dev)

Index 1Khanal data

Emulsion 27 1.38 (3.68)

Hypromellose 26 0.67 (2.01)

Total 53 1.03 (2.98)

Index 2McCann data

Emulsion 24 3.07 (3.00)

HPMC 25 0.31 (2.97)

Hyaluronate 24 0.82 (2.04)

Total 73 1.38 (2.93)


included as a covariate, the effect was no longer significant.Using Index 2 not only gave a significant difference betweentreatment groups, but the difference remained significanteven when the baseline combination score was added as acovariate (P¼.035). Hence, Index 2 was an improvementon any single measure for this data.

5. SummaryThe initial intention of the work presented in this sec-

tion was to find consistent factors across both datasetsderived from the variables EVAP, TTR, TFI, and OSM.This proved not to be possible. We decided instead toconsider using two combinations of EVAP and TTR. Com-bination 1 was EVAP divided by TTR (EVAP/TTR) andCombination 2, EVAP divided by the EVAP plus TTR[EVAP/(EVAP þ TTR)]. Indices based on linear combina-tions of these measures and OSM were created.

Transforming EVAP and TTR into common units wasnot necessary for Combination 1. It did make a differenceto Combination 2 but only to bring results closer to thoseusing Combination 1, and the differences were not substan-tive. For this reason, it was recommended that the simplerCombination 1 with its original units be tried. Standardiza-tion of the variables was not necessary if the percentagechange in OSM was used instead of actual change.

Figure 3. Mean Index 2 scores (with 95% Confidence Intervals) bytreatment group: McCann data.




The conclusions from Section III.A, which looked ateach variable separately, suggested that a combination ofvariables was more likely to be useful in assessing treat-ment for the McCann data than the Khanal data. Thefindings from Section III.B are in line with these expecta-tions. Index 2 with Combination 1 and OSM was capableof differentiating between treatment groups for theMcCann data.

No significant differences were found for Index 1 for thetwo groups in the Khanal data. This may be because of theproblems with OSM in this dataset, or it may simply bebecause there were no real treatment effects present. Thedry eye treatments were different for the McCann and Kha-nal datasets.

Overall conclusions from Section III.B are:1) Combination 1 was superior to Combination 2, as itgave similar or better results and was not affected bythe units for EVAP and TTR.2) Index 2 change based on a linear sum of Combination1 change and percentage change in OSM was the betterindex.The results from this section are far from definitive. The

original goal of finding consistent factors across both data-sets was not achieved, and more work should be carriedout with different datasets in order to confirm the findingsof this section. This is attempted in the next section in whichthe findings were tested across the combined datasets andagainst new normative data.

C. Multivariate Outcomes Measures: CombinedStudies

1. IntroductionThe lack of consistent factors across the two separate

datasets in Section III.B suggested that a combination ofEVAP and TTR with OSM may give a better index for theeffectiveness of treatments. The analysis in Section III.B isnow replicated for the combined datasets [McCann andKhanal] totalling 126 cases. In addition to replicating theprevious analyses, the more promising indices werecompared against individual measures for effectiveness oftreatment and for the initial diagnosis of dry eye disease(employing dry eye baseline data and a small sample ofnon-dry eyes subjects [n¼32]). In this latter scenario, aformal discriminant analysis was employed in an attemptto discriminate between normal and dry eye subjects.Discriminant analysis has been employed in many clinicalresearch areas, including that of ocular surface,8,35-40 andis appropriate where groups of subjects are known to existand data from each are available. The profiling of suchgroups in terms of several outcome variables is oftenrequired, but the main aim of the technique is to obtain a

yCombination 1_Change ¼ ðEVAP at baseline=TTR at

yCombination 2_Change ¼ ðEVAP at baseline=EVAP þ TTR a


decision rule that will enable new cases to be allocated tothe appropriate group with a high probability of success.This ensures high sensitivity and specificity. Logistic regres-sion41 is an increasingly popular alternative to discriminantanalysis, especially when there is a mix of continuous,discrete, and dichotomous independent variables. However,we decided not to use it in this scenario, as there is only asingle independent variable, and, hence, discriminant anal-ysis is simpler to run and the output is easier to interpretclinically.

2. Using Original VariablesTable 6 shows the important results from factor models

on the combined data using the three variables [EVAP,TTR, OSM] with two factors extracted and Varimax rota-tions applied when appropriate.

The sample size [n¼126] was adequate for a three-variable analysis, as it exceeded the suggested value of tentimes the number of variables.42,43 The KMO values wereonly just adequate and, as in Section III.B, Bartlett’s test re-sults were nonsignificant. The cumulative % variationexplained was, however, consistently above 70%. The result-ing weights, ignoring weights less than 0.4, suggested thatComponent 1 was a combination of TTR and OSM, withvarying arithmetic signs, and Component 2 explainedEVAP. This was similar to the interpretation from SectionIII.B. As in that section, there was a lack of consistency ofthe weights. Given this, along with the fact that two measures,ie, two component calculations, would be require for an index,we decided that assessing the benefits of the two combinationsdevised in Section III.B would be a better approach.

3. Using Combination 1 and Combination 2 as Definedin Section III.BThe argument in support of forming Combinations 1

(EVAP/TTR) and 2 (EVAP/EVAPþTTR) were presentedin Section III.B. The results from the factor analysis forCombinations 1 and 2 with OSM as the input variables ateach of the three time points are given in Table 7. Foreach analysis only one component was extracted. The singlefactor in each explained over 50% of the variation in thedata though the KMO was minimally adequate at 0.5 andthe Bartlett results nonsignificant. The magnitude of eachweight was consistent at around 0.7.

The use of the percentage change in OSM instead ofactual change over 30 days gave very similar componentscore weights and percentage of variation explained.

4. Indices Comparing the Effectiveness of TreatmentIf, as in Section III.B,

baselineÞ � ðEVAP at 30 days=TTR at 30 daysÞ and

t baselineÞ � ðEVAP at 30 days=EVAP þ TTR at 30 daysÞ:



Table 6. Factor analysis models of evaporation (EVAP), tear turnover rate (TTR) and osmolarity (OSM) at baseline, 30 days, andchange over 30 days

Time Data set KMO % variationexplained FactorScore coefficients

EVAP TTR OSM

Baseline Combined 0.51 37.9% 1 0.33 0.60 �0.64

32.9% 2 0.92 �0.39 0.11

30 days Combined 0.52 39.7% 1 0.04 0.65 0.66

32.3% 2 1.00 0.01 0.04

Change Combined 0.49 36.9% 1 �0.05 �0.59 0.75

33.6% 2 0.89 �0.34 �0.26


(yNote : these will be referred to as Combination 1 &Combination 2 in this section), then, based on the resultsof the factorization for the change over 30 days, two indiceswere created as below (the reasons for preferring percentagechange in OSM were discussed in Section III.B):

Index A_Change ¼ 0:70 � Combination 1_Change

þ 0:70 � percentage change in OSM

and

Index B_Change ¼ 0:69 � Combination 2_Change

� 0:69 � percentage change in OSM

Table 8 gives the mean and standard deviation of the In-dex scores by treatment group. A one-way ANOVA was run
with the Index as the dependent variable and a singlebetween-group factor consisting of all the treatment groups.
a. Index A ChangeThere was a significant difference in the mean Index

scores between the five groups (P¼.005). Tukey’s post hoctest revealed significant differences only for the McCanntreatment groups, namely that the McCann emulsion grouphad a significantly higher mean Index score than either theHPMC (P¼.004) or the hyaluronate (P¼.032) groups.Figure 4 shows the mean differences by group. Recallingthe effect of baseline EVAP rate in Section III.A for theMcCann data (where a significant effect disappeared once

Table 7. Factor analysis models using Combination 1 (EVAP/TT

Time % variation explainedComponent coefficien

Combination 1 OSM

Baseline 52.2% 0.69 0.6

30 days 54.4% 0.68 �0.6

Change 51.4% 0.70 0.7


baseline EVAP was considered), an ANCOVA was carriedout using Index A change as the dependent variable and In-dex A baseline as a covariate; there was still a significant dif-ference between groups (P¼.029).

A one-way ANOVA with Combination 1 as the depen-dent variable also showed a significant difference in themean Index scores between the five groups (P¼.005). Howev-er, once baseline values of Combination 1 were added as a co-variate, the significant difference between groups disappeared(P¼.169). Hence, Index A was more effective than Combina-tion 1 on its own in identifying the effect of treatment.

b. Index B ChangeA one-way ANOVA showed no significant difference in

the mean Index B scores across the five treatment groups(P¼.552). A one-way ANOVA with Combination 2 as thedependent variable did show a significant difference betweenthe five groups (P<.05). However, once baseline values ofCombination 2 were added as a covariate, there was no sig-nificant difference between groups (P¼.144).

Conclusion from Section III.C: The composite measuresformed by both Combinations and Index A were capable ofassessing significant changes over 30 days as a result of thetherapeutic interventions in dry eye. However, as in SectionIII.B where similar indices were derived, the only significantchanges found once baseline values were taken into accountwas for Index A change. This suggested that Index A is themost promising composite measure for comparing the effec-tiveness of treatments.

R)

ts% variation explained

Component coefficients

Combination 2 OSM

9 50.4% 0.70 0.70

8 58.3% 0.66 �0.66

0 53.0% 0.69 �0.69



Table 8. Mean and standard deviation of index scoresby treatment

Treatment Group NMean (Std.Deviation)

Index AChange

Emulsion (Khanal) 27 1.72 (2.75)

Hypromellose 26 1.13 (2.70)

Emulsion (McCann) 24 3.25 (3.18)

HPMC 25 0.33 (3.15)

Hyaluronate 24 0.87 (2.16)

Total 126 1.45 (2.94)

Index BChange

Emulsion (Khanal) 27 �0.05 (2.59)

Hypromellose 26 �0.17 (1.82)

Emulsion (McCann) 24 �0.64 (1.50)

HPMC 25 0.37 (2.41)

Hyaluronate 24 �0.11 (1.53)

Total 126 �0.11 (2.03)

Figure 4. Mean Index A change scores (with 95% confidence intervals)by treatment group.


5. Comparing Multiple Outcome against IndividualMeasures of Tear Physiology in the Determinationof Effectiveness of Dry Eye TreatmentReports of the effectiveness of tear tests in assessing the

therapeutic value of interventions in dry eye are rare. Somestudies have reported improvement in OSM with the use ofartificial tears,44,45 but others have reported a lack ofimprovement over a short trial period with this test.10 Inconsidering the benefits of sodium hyaluronate and polyvi-nyl alcohol artificial tear preparations for dry eye, Nelsonand Farris evaluated disease status by means of tear filmOSM, tear film breakup time, rose bengal staining, Schirm-er’s test (without anesthesia), and ocular surface-impressioncytology.46 Tear film OSM and rose bengal staining scorewere found to improve over the study. Sullivan et al havesuggested that for the purposes of clinical trials, the use ofa composite index for inclusion criteria should be adopted.20

Further, because the longitudinal variability of each of theindividual tests has not been established, the valuesmeasured at a single snapshot in time may not be consistentduring follow-up visits.

The best composite score in the present review, Index Achange, was compared against the three individual measuresof tear physiology, EVAP, tear turnover, and OSM to assessthe effectiveness of treatment using ANCOVA with thechange over 30 days as the dependent variable, a singlebetween-group factor consisting of all the treatment groups,and the baseline score as a covariate. Table 9 gives theadjusted mean 30-day values for each measure and thegroup effect size.

There was no significant difference in the mean EVAP(P¼.423) nor in mean OSM over 30 days across the fivegroups (P¼.363). However, there was a significant difference


in mean tear turnover across the five groups (P¼.023).Tukey’s post hoc test revealed that the Khanal emulsiongroup had a significantly higher mean change in tear turn-over than either the hypromellose (P¼.047) or the HPMCgroup (P¼.017). However, when the outlier in the Khanalemulsion group (discussed in Section III.B) was omitted,the mean TTR fell to �1.30 for this group, and there wereno significant differences between groups (P¼.084).

By comparison, the Index A group effect was significantwhen baseline values were included (P¼.029) and was stillsignificant when the outlier in the Khanal dataset wasomitted (P¼.033 [the mean Index A score fell to 1.89 forthe Khanal emulsion group]).

Individual measures of tear physiology were found tohave lower effect sizes than Index A except for tear turnover,which was similar. However, when the outlier was omitted,the effect size for tear turnover fell to 0.26 while that for In-dex A remained unchanged. The Index A effect size wasmoderate and was robust to outliers. Therefore, the multipleoutcome measure out-performed any individual measure.

6. Diagnosing Dry Eye Patients at BaselineAs reported in the previous section, Index A was the best

discriminator of treatment effects over 30 days. As a furthervalidation, we decided to assess Index A in initial diagnosisof dry eye. The combined sample of the dry eye patients[n¼126] and a new sample of 32 non-dry eye subjectswere employed in this analysis.

Index A_baseline was created from the baseline charac-teristics of the two samples.

Index A_baseline ¼ 0:70 � Combination 1_baseline

þ 0:70 � OSM at baseline

(where � Combination 1_baseline¼EVAP at baseline/TTRat baseline)



Table 9. Estimated mean scores at 30 days corrected for baseline

Group Evaporation Tear turnover rate Tear Turnover (without outlier) Osmolarity Index A

Emulsion (Khanal) 11.77 �1.67 �1.30 1.89 1.93

Hypromellose 12.34 �0.14 �0.14 2.08 1.66

Emulsion (McCann) 14.97 �0.55 �0.55 0.91 2.02

HPMC 9.56 �0.12 �0.12 �0.89 0.59

Hyaluronate 11.55 �0.45 �0.45 0.35 1.02

Effect size 0.18 0.30 0.26 0.19 0.29


Table 10 gives the mean and standard deviation of IndexA at baseline for the two groups, and Figure 5 shows thenormal curves based on these figures.

A discriminant analysis was carried out and a significantdifference was found between the two groups of subjects(P<.001). The discriminant function was also significant(P<.001).

Table 11 gives the overall classification of patients andsubjects by resubstitution. The Index correctly classified 20of the non-dry eye cases (62.5%) and 87 of the dry eye cases(69%). Cross-validation23 in which each case was classifiedby functions derived from all cases other than that case,reduced the correct dry eye classification slightly to 86(68.3%) but did not alter the non-dry eye classification per-centage. Index A at baseline gave reasonable sensitivity andspecificity.47

In order to assess the advantages of multiple over singleoutcome measures discriminant analyses were performed oneach of the three baseline tear physiology variables and onthe two simple combinations measured at baseline.Table 12 summarizes the cross-validation results.

The results for Index A were overshadowed by those ofCombination 2. Combination 2 at baseline had better sensi-tivity and specificity than Index A or other single or com-pound measures for the present datasets. Table 13 givesthe mean and standard deviation of Combination 2 forthe two groups, and Figure 6 shows the normal curves basedon these figures. A discriminant analysis found a significantdifference between the two groups of subjects (P<.001). Thediscriminant function was also significant (P<.001).

In the diagnosis of dry eye, the results obtained with In-dex A were in the mid-range for values reported in the

Table 10. Mean and standard deviation of index A atbaseline for the two groups

Type of subject Mean Std. Dev.

Non-dry-eye [n¼32] 217.00 6.88

Dry-eye [n ¼ 126] 221.94 4.87

Total 220.94 5.68


literature6,48 and those obtained by other single and multiplemeasure combinations in this reanalysis study. However, se-lection bias may have been a factor in some studies reportedin the literature7,49 and spectrum bias in those studiesconfined to more severe cases.4 The dry eye subjects in thecurrent reanalysis study were confined to the mild-to-moderate category. Recently, Jacobi et al have reported asensitivity of 87% and specificity of 81% for diagnosis by asingle OSM measure.50 This study had, however, an elementof spectrum bias, with the dry eye patients a ”moderate”group with a median of 320 and an inter-quartile range of301-324; hence, distribution was skewed toward the upperlevels. The present study had a median of 311 and aninter-quartile range of 307-317 and comprised essentiallymilder cases of dry eye. If Jacobi et al had had a similarrange of cases, they would have had lower sensitivity andspecificity values.50 In all of the comparisons with the resultsreported in the literature, biases make comparison difficultbetween studies from different centers. Therefore, themost valid comparison of the effectiveness of single versusmultiple measures is in the same patient samples, recruitedand assessed under the same conditions. This is the compar-ison reported here.

Figure 5. Normal curves for Index A baseline scores for normal and dryeye subjects.



Table 11. Sensitivity and specificity of Index A at baseline

Index A baseline Number Percentage Cross-validation

Sensitivity (dry eye patients correctly classified) 87/126 69.0% 68.3%

Specificity (normal patients correctly classified) 20/32 62.5% 62.5%


7. SummaryIndex A formed from Combination 1(EVAP/TTR) and

OSM was the most robust measure in comparing the effectof treatments over 30 days. Index A was not as effective atdiagnosing dry eye patients as other combinations in thisstudy or others reported in the literature. However, someof the reported literature contains studies in which selectionand spectrum bias contributed to the results of diagnosis.The latter is unlikely to have been a factor here becausethe patients in the datasets in this re-analysis had mild-to-moderate dry eye. Another multiple measure, Combination2 (EVAP/[EVAP þ TTR]), was the best measure for distin-guishing between dry eyes and non-dry eyes at baseline,with 83% of dry eye cases and 72% or normal cases beingcorrectly classified.

Multiple outcomemeasures were found to bemore robustthan single outcomemeasures, both in diagnosing dry eye andin assessing the therapeutic value of interventions. Thesimplicity of both multiple outcome measures, Index A andCombination 2, is a distinct advantage in that no complicatedcalculations are required for its use in clinical diagnosis.

IV. SUMMARY AND CONCLUSIONSDry eye is a multifactorial disease classified into the ma-

jor subgroups of aqueous deficiency dry eye (ADDE) andevaporative dry eye (EDE).51 In its management, identifyingthe subgroups is beneficial to target first-line treatments.The multifactorial nature of dry eye makes essential a broadspectrum of tear physiology measures in diagnosis andmonitoring of treatment.

Studies of dry eye disease have typically involved analysesof individual aspects of tear physiology in diagnosis and in thedetermination of the effects of therapy.1,3-5,8,10,48,52 This has

Table 12. Comparisons of cross-validation sensitivity and speci

Index/Variable/Combination (all at baseline)Sensitivity

corre

Index A

EVAP

TTR

OSM

Combination 1 (EVAP/TTR)

Combination 2 [EVAP/(EVAPþTTR)]


been particularly so in investigational studies required bythe U.S. Food and Drug Administration, where singleoutcome measures have been preferred. However, such anapproach may lose the benefits of assessing comprehensivelythis multifactorial condition andmiss the measurement of as-pects of the etiology specifically relevant to the condition. Asall tests evaluate different characteristics of the tear film, it isunlikely that a single test can serve as a complete assessment ofdry eye.

The limitations of single-test evaluation can be avoidedif multiple, combined outcomes are assessed by different sta-tistical approaches. Previously we have shown the benefit ofdiscriminant function analysis in the diagnosis of dry eye.8,9

In recent international workshops reevaluating dry eye dis-ease, multiple tests have been advocated for diagnosis andevaluation of therapeutic effects.3,6

In the present review, data from two previous studies ofthe tear physiology in dry eye10,11 were extensively reana-lyzed using a number of statistical approaches. In bothstudies, the same four tear physiology variables (OSM,EVAP, TTR, and TFI), were measured as single variableswithin groups, pre- and post-intervention with a lubricantsupplement.

This review also reassessed the effects of artificial tearson aspects of tear physiology. These measures were analyzedfirst as single variables, then by creating combinations ofthese variables, and finally by validating these combinationswith the combined sample of data from all groups of dry eyesubjects. The effectiveness of combinations in diagnosis ofdry eye was also considered.

For single measures, instead of taking a simple pre-postmethod of analysis for each outcome measure within eachtreatment group, the single variables in each study were

ficity results based on discriminant analysis

[% Dry Eye patientsctly classified]

Specificity [% Normal patientscorrectly classified]

68.3% 62.5%

60.3% 68.8%

83.3% 68.8%

61.1% 53.1%

59.5% 90.6%

83.3% 71.9%



Table 13. Mean and standard deviation of Combination2 at baseline

Type of subject Mean Std. Dev.

Non-dry eye [n¼32] 0.56 0.16

Dry eye [n¼126] 0.77 0.12

Total 0.73 0.15

Figure 6. Normal curves for Combination 2 (baseline) scores fornormal and dry eye subjects.


reanalyzed incorporating a within- and between-group anal-ysis using two factor repeated measures ANOVA models,15

where the interaction between groups over time was the keyoutcome. Where baseline measures differed between groups,an ANCOVA18 model was used with baseline measure as acovariate. These analyses enabled more direct comparisonsof the efficacy of the treatment methods. Individual tearphysiology measures showed some improvements overtime but were not effective in defining differences betweendry eye treatments.

The individual variables were then treated as a multivar-iate sample within each study. A factor analysis model16 wasdeveloped for each of the previous studies. This analysisreduced the dimensionality of each study sample by creatinglinear combinations of the existing variables known as fac-tors, ie, the combined variables.

Combinations of the measures of tear physiology werethen considered for their ability to differentiate the effectsof dry eye treatments across the two separate study datasets.A combination of EVAPdivided by TTR (EVAP/TTR) was su-perior to EVAP divided by EVAP plus TTR (EVAP/

Index ¼ 0:7��

Evaporation at baselineTear Turnover at baseline

� Evaporation at 30 daysTear Turnover at 30 days

�þ 0:7

��Osmolarity at baseline� Osmolarity at 30 days

Osmolarity at baseline� 100

�

[EVAP þTTR]). The former had the advantage that it wasnot affected by the units of measurement for EVAP andTTR. The main factors obtained were analyzed using thesame ANOVA and ANCOVA procedures outlined above. Indetermining the effect of treatment, an index based on thelinear sum of the weighted change in EVAP/TTR, and theweighted, percentage OSM change offered the best potential.However, the results were not definitive and a goal of findingconsistent factors across both datasets was not achieved.

The data from the two studies were then combined toform a larger sample of cases for each of the outcome mea-sures, and the factor analysis was then replicated with thislarger sample. It was hoped that this analysis would producestronger estimates of the factor structure allowing a five-group repeated measures ANOVA model to be run. This


was intended to make final comparisons of the efficacy ofeach intervention using a multiple measurement scale.

Combinations of the measures of tear physiology inmonitoring treatment were considered for their ability todifferentiate the effects of dry eye treatments across theaggregated study datasets. An Index formed from a weightedcombination of EVAP/TTR and the weighted percentagechange in OSM appeared to be the most promising in thatit was able to effectively differentiate between treatments af-ter differences in baseline score are taken into account. ThisIndex was also more robust than any individual measure.The effect size for this measure over all groups was a mod-erate 0.29, and this was larger than for any individual mea-sure of tear physiology.

The Index formula for monitoring treatments:

As a further validation, it was decided to assess the Indexin diagnosis of dry eye for the combined sample of the dryeye patients and a new sample of non-dry eye subjects.Discriminant analysis was employed to assess the sensitivityand specificity when using the baseline data for each of thesingle outcome variables, the Index and both Combinations,as a diagnostic tool.

In the initial diagnosis of dry eye, the Index identified dryeye patients vs normal with good sensitivity and specificity.However, the combination of EVAP/[EVAP þ TTR] wasfound to be the most sensitive and specific measure, with83% of dry eye and 72% of normal cases being correctly clas-sified. This combination was generally better than singlemeasures in diagnosis, although similar to the performanceof the single measure, TTR, in sensitivity but lower in




specificity. Previously, the merit of using TTR to differen-tiate the subtypes (ADDE and EDE) of dry eye has beennoted.34

The combination formula for initial diagnosis of dry eyewas:

Combination

¼�

Evaporation at baselineEvaporation at baselineþ Tear Turnover at baseline

�

The simplicity of both multiple measures, Index andCombination, is a distinct advantage in that no complicated
calculations are required in clinical implementation.
The diagnosis and monitoring of dry eye with the algo-rithms derived from multiple measures of tear physiologyneed to be validated on larger and independent samples ofpatients. A determination of the clinically significant changein the compound measures, used for treatment monitoringand diagnosis in dry eye, also needs to be obtained in futurestudies. Additionally, consideration should be given to theevaluation of similar combinations of surrogate clinical mea-sures of tear EVAP, turnover, and OSM (eg, tear filmbreakup time and Schirmer or thread tests).

REFERENCES1. Lucca JA, Nunez JN, Farris RL. A comparison of diagnostic tests for

keratoconjunctivitis sicca: lactoplate, Schirmer, and tear OSM. CLAOJ 1990;16:109-12

2. Nichols KK, Mitchell GL, Zadnik K. The repeatability of clinical mea-surements of dry eye. Cornea 2004;23:272-85

3. Tomlinson A, Bron AJ, Korb DR, et al. The International Workshop onMeibomian Gland Dysfunction: Report of the diagnosis subcommittee.Invest Ophthalmol Vis Sci 2011;52:2006-49

4. Van Bijsterveld OP. Diagnostic tests in the sicca syndrome. Arch Oph-thalmol 1969;82:10-4

5. Goren MB, Goren SB. Diagnostic tests in patients with symptoms ofkeratoconjunctivitis sicca. Am J Ophthalmol 1988;106:570-4

6. (No authors listed). Methodologies to diagnose and monitor ry eeye dis-ease: report of the Diagnostic Methodology Subcommittee of the Inter-national Dry Eye WorkShop (2007). Ocul Surf 2007;5:108-52

7. Farris RL, Gilbard JP, Stuchell N, et al. Diagnostic tests in keratocon-junctivitis sicca. CLAO J 1983;9:23-8

8. Khanal S, Tomlinson A, McFadyen A, et al. Dry eye diagnosis. InvestOphthalmol Vis Sci 2008;49:1407-14

9. Craig JP, Tomlinson A. The effect of age on tear OSM. Optom Vis Sci1995;72:713-7

10. Khanal S, Tomlinson A, Pearce EI, Simmons PA. Effect of an oil-in-water emulsion on the tear physiology of patients with mild to moder-ate dry eye. Cornea 2007;26:175-81

11. McCann LC, Tomlinson A, Pearce EI, Papa V. Effectiveness of artificialtears in the management of evaporative dry eye. Cornea 2012;31:1-5

12. Trees GR, Tomlinson A. Effect of artificial tear solutions and saline ontear film evaporation. Optom Vis Sci 1990;67:886-90

13. Doane MG. An instrument for in vivo tear film interferometry. OptomVis Sci 1989;66:383-8

14. McCann LC, Tomlinson A, Pearce EI, et al. A clinical alternative to flu-orophotometry for measuring tear production in the diagnosis of dryeye. Cornea 2010;29:745-50

15. Field A. Discovering Statistics Using SPSS. ed 3. London, Sage Publica-tions Ltd, 2009


16. Everitt BS, Dunn G. Applied Multivariate Data Analysis. London,Edward Arnold, 1991

17. Daniel WW. Applied Nonparametric Statistics. ed 2. Boston, PWS-KENT, 1990

18. Van Breukelen GJP. ANCOVA versus change from baseline had morepower in randomized studies and more bias in nonrandomized studies.J Clin Epidemiol 2006;59:920-5

19. Senn S. Change from baseline and analysis of covariance revisited. StatMed 2006;25:4334-44

20. Sullivan BD, Whitmer D, Nichols KK, et al. An objective approachto dry eye disease severity. Invest Ophthalmol Vis Sci 2010;51:6125-30

21. Cohen J. Statistical Power Analysis for the Behavioural Sciences. ed 2.New York, Academic Press, 1988

22. Cohen J. A power primer. Psychol Bull 1992;112:155-923. Hair JF, Anderson RE, Tatham RL, Black WC. Multivariate Data Anal-

ysis with Readings. ed 4. New Jersey, Prentice-Hall International, 199524. Tabachnick BG, Fidell LS. Using Multivariate Statistics. ed 5. Allyn &

Bacon, Boston, 200725. Kaiser HF. A second generation little jiffy. Psychometrika 1970;35:

401-1526. Kaiser HF. An index of factorial simplicity. Psychometrika 1974;39:35-927. Vitali C, Moutsopoulos HM, Bombardieri S. The European Community

Study Group on diagnostic criteria for Sjogren’s syndrome: sensitivityand specificity of tests for ocular and oral involvement in Sjogren’s syn-drome. Ann Rheum Dis 1994;53:637-47

28. Farris RL. Tear osmolarity: a new gold standard? Adv Exp Med Biol1994;350:495-503

29. Tomlinson A, Khanal S. Assessment of tear film dynamics: quantifica-tion approach. Ocul Surf 2005;3:81-95

30. Tomlinson A, Doane MG, McFadyen A. Inputs and outputs of thelacrimal system: review of production and evaporative loss. Ocul Surf2009;7:186-98

31. Mathers WD. Why the eye becomes dry: a cornea and lacrimal glandfeedback model. CLAO J 2000;26:159-65

32. Xu KP, Yagi Y, Toda I, Tsubota K. Tear function index: a new measureof dry eye. Arch Ophthalmol 1995;113:84-8

33. Mathers WD, Lane JA, Sutphin JE, Zimmerman MB. Model for oculartear film function. Cornea 1996;15:110-9

34. Khanal S, Tomlinson A, Diaper CJ. Tear physiology of aqueous defi-ciency and evaporative dry eye. Optom Vis Sci 2009;86:1235-40

35. Douglass KH, Links JM, Chen DC, et al. Linear discriminant analysis ofregional ejection fractions in the diagnosis of coronary artery disease.Eur J Nucl Med 1987;12:602-4

36. Detrano R, Leatherman J, Salcedo EE, et al. Bayesian analysis versusdiscriminant function analysis: their relative utility in the diagnosis ofcoronary disease. Circulation 1986;73:970-7

37. Lansky A, Fleming PL, Byers Jr RH, et al. A method for classification ofHIV exposure category for women without HIV risk information.MMWR Recomm Rep 2001;50:31-40

38. Ellis G, Goldberg DM. Discriminant function analysis applied to labo-ratory tests in patients with hepatobiliary disease. Comput Biomed Res1979;12:483-501

39. Visser PJ, Verhey FR, Ponds RW, Jolles J. Diagnosis of preclinicalAlzheimer’s disease in a clinical setting. Int Psychogeriatr 2001;13:411-23

40. Ratei R, Karawajew L, Lacombe F, et al. Discriminant function analysisas decision support system for the diagnosis of acute leukemia with aminimal four color screening panel and multiparameter flow cytometryimmunophenotyping. Leukemia 2007;21:1204-11

41. Menard S. Applied Logistic Regression Analysis. ed 2. Thousand Oaks,CA, Sage Publications, 2001


http://refhub.elsevier.com/S1542-0124(13)00081-5/sref1



































































































42. Nunnally JC. Psychometric Theory. New York, McGraw-Hill, 197843. Kass RA, Tinsley HEA. Factor analysis. J Leisure Res 1979;11:

120-3844. Gilbard JP, Rossi SR, Azar DT, Heyda KG. Effect of punctal occlusion

by Freeman silicone plug insertion on tear osmolarity in dry eyedisorders. CLAO J 1989;15:216-8

45. Gilbard JP, Rossi SR, Heyda KG. Ophthalmic solutions, the ocularsurface, and a unique therapeutic artificial tear formulation. Am JOphthalmol 1989;107:348-55

46. Nelson JD, Farris RL. Sodium hyaluronate and polyvinyl alcohol artifi-cial tear preparations. A comparison in patients with keratoconjuncti-vitis sicca. Arch Ophthalmol 1988;106:484-7

47. Bland M. An Introduction to Medical Statistics. ed 3. Oxford, OxfordUniversity Press, 2000


48. Tomlinson A, Khanal S, Ramaesh K, et al. Tear film osmolarity: deter-mination of a referent for dry eye diagnosis. Invest Ophthalmol Vis Sci2006;47:4309-15

49. Gilbard JP, Farris RL. Tear osmolarity and ocular surface disease inkeratoconjunctivitis sicca. Arch Ophthalmol 1979;97:1642-6

50. Jacobi C, Jacobi A, Kruse FE, Cursiefen C. Tear film osmolarity mea-surements in dry eye disease using electrical impedance technology.Cornea 2011;30:1289-92

51. (No authors listed). The definition and classification of dry eye disease:report of the Definition and Classification Subcommittee of the Inter-national Dry Eye WorkShop (2007). Ocul Surf 2007;5:75-92

52. Pflugfelder SC, Tseng SC, Sanabria O, et al. Evaluation of subjectiveassessments and objective diagnostic tests for diagnosing tear-film dis-orders known to cause ocular irritation. Cornea 1998;17:38-56































Date post:	23-Dec-2016
Category:	Documents
Upload:	angus
View:	212 times
Download:	0 times

Statistical Approaches to Assessing Single and Multiple Outcome Measures in Dry Eye Therapy and...

Documents