JOURNAL OF RESEARCH IN PERSONALITY 17, 451-456 (1983)

Sense, Nonsense, and the Use of Personality Measures

ROBERT HOGAN, JOYCE HOGAN, STEPHEN BRIGGS, AND WARREN JONES

University of Tulsa

J. G. Nicholls, B. G. Licht, and R. A. Pearl (Psychological Bulletin, 1982, 92, 572-580) criticize measurement-based personality research by recounting a well-known limitation: indices of relationships are often inhated because of the similarity of content across measures of presumably different constructs. In this paper we endorse the major contention of Nicholls et al. that measurement- based research may be conducted thoughtlessly, and we extend their thinking by pointing out an important underlying issue-the problem of defining what constitutes a “syndrome.” In contrast to them, however, we argue that the problems in measurement they cite do not entail the consequences they foresee and, in addition, have limited application.

In measurement-based research investigators are often interested in the relationship between two constructs. The most convenient way to estimate such a relationship is to compute an index of association, usually a correlation coefficient of some kind, between a set of scores based on measures of the two constructs. The degree of association between the two measures enables one to make an inference about the relationship between the underlying constructs. In a recent paper, Nicholls, Licht, and Pearl (1982) point to certain problems latent in such procedures. The purpose of this paper is to comment on the Nicholls et al. essay, which we believe makes some useful points, but which is also misleading in certain ways. In what follows we mention what we regard as the most important features of the Nicholls et al. critique of measurement research, we then discuss some ways in which their paper is misleading, and we end by pointing out that their paper must be interpreted in the context of broader psychometric issues.

THOUGHTLESSNESS IN MEASUREMENT-BASED RESEARCH The Nicholls et al. paper is concerned with the use of questionnaires

in developing and testing personality theory. Their major argument is that the relationship between two constructs cannot be accurately de- termined if the constructs are operationalized in terms of measures that

Request for reprints should be sent to Robert Hogan, Department of Psychology, University of Tulsa, Tulsa, OK 74104.

4.51 0092-6566183 $3.00

Copyright 0 1983 by Academic Press, Jnc. All rights of reproduction in any form resewed.

452 HOGAN ET AL.

contain similar content. They support their argument with two kinds of examples. The first concerns the case where two constructs are defined by self-report measures. The relationship between the constructs (e.g., masculinity and self-esteem) cannot be precisely determined if items on a scale measuring one of the constructs (self-esteem) are similar to some of those on a scale measuring the other construct (masculinity). As Nicholls et al. point out, in this first case the simplest interpretation of a positive correlation is that the self-esteem items on one scale are related to the self-esteem items on the other, not that masculinity is related to self-esteem.

Their second example concerns evaluating the relationship between a self-report measure of a construct and a measure of behavior thought to be related to that construct. They argue that the relationship between a self-report measure (femininity) and behavior (nurturance) cannot be accurately determined if items on the scale refer to the behavior in question. Here, a positive correlation suggests only that nurturant items predict nurturant behavior, not that femininity and nurturance are related.

In general, we agree with these points. Two measures are likely to be related by definition if they are comprised of the same elements. The relationship is “built-in” to the measures. We should point out, however, that the potential for a relationship to be built-in is greater for two self- report measures than for a self-report measure and a behavioral index. In fact, certain writers (e.g., Mischel, 1968) maintain that large and significant correlations wil1 rarely be found between personality measures and nontest behavior, independent of the degree of similarity between items and criterion behavior. In the continuing effort to evaluate critically Mischel’s claims, any correlation between a measure and a nontest be- havior, even if there is semantic overlap between the scale and the criterion behavior, is important theoretically and, for some at least, is grounds for celebration rather than for discrediting the measure.

The foregoing points raise a more generic issue: researchers often use “self-report” measures and do correlational research in a naive and unreflective manner. This thoughtlessness takes several forms. For ex- ample, in selecting and using scales, researchers sometimes accept the names of measures at face value without examining the items that actually comprise them. Unfortunately, some tests are remarkably misnamed. Eysenck’s (Eysenck & Eysenck, 1968) extraversion scale, for instance, contains substantial elements of stimulus-seeking and impulsivity, and his psychoticism scale has a large planfulness component (cf. Hogan & Cheek, 1982). Often such scales are designed to measure clusters of traits rather than single or homogeneous dimensions. In selecting and using these multidimensional measures, researchers may be unaware of the fact that they include various components or how the components in- terrelate. The Self-Monitoring Scale (Snyder, 1974), for instance, pur- portedly measures a cohesive cluster of traits defined by the scale’s name.

PERSONALITY MEASURES 453

Analysis of the scale, however, shows that it is formed of components that bear little resemblance to the underlying concept or to the name of the scale itself (Briggs, Cheek, & Buss, 1980; Gabrenya & Arkin, 1980). Thus, researchers are well advised not to believe that the name given to a scale, or the way that it is described conceptually, will necessarily tell them what scores on the measure mean.

SOME MISLEADING POINTS Nicholls et al. make two points that we find more troublesome. First,

after correctly pointing out that correlations between measures with similar content overestimate the magnitude of covariation between the constructs they represent, they go on to assert that correcting for this item similarity results in an inexorable dilemma: similar items must be removed in order to test for a relationship, but dropping the items removes essential com- ponents of the scale, thereby making it a less valid measure of the construct. This dilemma is illusory. Notice first of all that the constructs in question (e.g., masculinity, achievement motivation, and extraversion) are broad and multifaceted; they are what might be called syndromes. Such syndromes consist of clusters of traits arranged in particular patterns, and some syndromes are more organic or cohesive than others. The crucial issue is not the dilemma of similar item content but rather the problem of knowing what should be included in the syndrome in the first place. This is a conceptual not a methodological question (see Lazarsfeld, 1959, for a discussion of the formal properties of syndromes). Furthermore, it should be clear that all personality measures are more or less imperfect representations of their underlying conceptual domains and a responsible investigator constantly amends his or her measures. It is hard to imagine a meaningful construct whose essence is defined by the presence or absence of three or four items on a scale.

Moreover, it is easy to estimate the extent to which item similarity accounts for a relationship between two measures. One simple solution is to examine correlations between the individual items (or components) underlying a syndrome (e.g., masculinity) and the total score of some other measure (e.g., a self-esteem scale or dominant behavior) which presumably involves one of the underlying components. If all of the masculinity items correlate at least moderately with the other measure, then the original correlation was not artifactual despite the item similarity.

The second point that we find troublesome is the Nicholls et al. claim that cognitive social learning theory provides an unassailable alternative to measurement-based personality research. In our view social learning theory is largely irrelevant to personality psychology for the following reasons. On the one hand, it is directed toward a set of problems (e.g., part processes, miniature theories of social perception, cognition, and action, etc.) that are quite distinct from the concerns of traditional mea- surement-based personality research (Hogan, 1982). In fact, social learning

454 HOGAN ET AL.

theorists question the very existence of syndromes and traits and thus have little to say about crucial issues in personality measurement-for example, what trait elements are involved in a syndrome and how to assess them. On the other hand, although cognitive social learning research is based on laboratory settings and manipulated variables, and seems, therefore, outside the purview of the Nicholls et al. critique, it can only by convention, but not in principle or practice, avoid the obligation to deal with standard questions of measurement-namely, the reliability and validity of its measures and its outcome variables. In assessing the effects of one’s experimental manipulation, one must, by definition, do measurement. With well-established psychological tests one is often dealing with instruments of limited and time-bound validity. Much laboratory research, however, uses even cruder measures whose psychometric prop- erties have never been evaluated. In their continued attack on measurement- based research, these critics are in danger of being hoisted upon their own petards.

PROVIDING CONTEXT

In evaluating the Nicholls et al. paper, one should recognize that their arguments encompass a limited domain. In this section, we put their paper in context by (a) distinguishing between the use of tests for theory testing and their use in applied settings and (b) noting briefly the importance of one’s theory of item responses.

In an extensive survey of the history of personality measurement, Goldberg (1971) distinguishes between two kinds of inventories: those developed to evaluate theoretical notions about the structure of personality (e.g., the 16-PF) and those developed for use in applied settings (e.g., the California Psychological Inventory). Nicholls et al. are concerned only with scales developed for theoretical purposes. We tend to agree that one cannot determine whether certain behaviors are part of a construct (syndrome) when those behaviors are already built into the measure of that construct. This does not mean, however, that nontest behaviors ought not be assessed using items reflecting these same behaviors, i.e., using items that are “face valid.” In applied research, face validity- does the test appear to measure what it is supposed to measure-is often crucial for reducing resistance on the test consumer’s part. But more importantly, in the area of personnel selection, one must explicitly dem- onstrate correlations between tests and criterion behavior even when using face valid tests; such documentation is required by the Uniform Guidelines for Employee Selection Procedures (Equal Employment Op- portunity Commission, Civil Service Commission, Department of Justice, and Department of Labor, 1978) and by the Standards for Educational and Psychological Test (American Psychological Association, American Educational Research Association, and National Council on Measurement

PERSONALITY MEASURES 455

in Education, 1974) and investigators who fail to provide these correlations risk professional disenfranchisement.

Throughout their article, Nicholls et al. use the term “self-report.” The word “self-report” implies that when people respond to items on a personality measure they are explicitly reporting their behavior, that personality measures are, in a sense, a second-best way of observing behavior, which measures are primarily justified by their ease, convenience, and brevity. It is important to understand that in describing item responses as self-reports-what Buchwald (1961) calls the correspondence approach- one is making a theoretical claim as opposed to giving a factual description of what may actually occur. Rather than going into a treatise on item- response theory, we will simply note that there are at least three other viewpoints that one can take on item-response dynamics: an empirical, a substantive, and a self-presentational view (see, respectively, Meehl, 1945; Loevinger, 19.57; Mills & Hogan, 1978). Moreover, from an ep- istemological perspective, all of these are preferable to the rather simple- minded self-report or correspondence conceptualization.

Finally, then, we agree with Nicholls et al. that “researchers using self- report measures of personality constructs need to keep the content of their measures in view.” But this is an old issue: How any construct is operationalized-whether by means of self-reports, behaviors, or cog- nitions-is fundamentally important. Moreover, the method of study must be matched to both the question being asked and the reason for asking it. These issues are important for all psychologists, not just for those who are concerned with measurement-based personality research.

REFERENCES American Psychological Association, American Educational Research Association, and

National Council on Measurement in Education. Standards for Educational and Psy- chological Tests (Rev. ed.). Washington, D.C.: American Psychological Association, 1974.

Briggs, S. R., Cheek, J. M., & Buss, A. H. An analysis of the Self-Monitoring Scale. Journal of Personality and Social Psychology, 1980, 38, 679-686.

Buchwald, A. M. Verbal utterances as data. In H. Feigl & G. Maxwell (Eds.), Current issues in the philosophy of science. New York: Holt, 1961. Pp. 461-468.

Equai Employment Opportunity Commission, Civil Service Commission, Department of Justice and Department of Labor. Adoption by four agencies of uniform guidelines on employee selection procedures (1978). Federal Register, 1978, 43, 38290-38315.

Eysenck, H. J., & Eysenck, S. B. G. Manual for the Eysenck Personality Inventory. San Diego: Educational and Industrial Testing Service, 1968.

Gabrenya, W. K., Jr., & Arkin, R. M. Factor structure and factor correlates of the Self- Monitoring Scale. Personality and Social Psychological Bulletin, 1980, 6, 13-22.

Goldberg, L. R. A historical survey of personality scales and inventories. In P. McReynolds (Ed.), Advances in psychological assessment. Palo Alto, California: Science & Behavior Books, 1971. Vol. 2.

Hogan, R. On adding apples and oranges (review of Personality, cognition, and socinl interaction by N. Cantor & J. F. Kihlstrom). Contemporary Psychology, 1982, 27, 851-852.

456 HOGAN ET AL.

Hogan, R., & Cheek, J. M. Identity, authenticity, and maturity. In T. R. Sarbin & K. E. Scheibe (Eds.), Studies in social identity. New York: Praeger, 1982.

Lazarsfeld, P. J. The logical and mathematical foundation of latent structure analysis. In S. A. Stouffer et al. (Eds.), Measurement and prediction. Princeton, New Jersey: Princeton Univ. Press, 1959. Pp. 263-412.

Loevinger, J. Objective tests as instruments of psychological theory. Psychological Reporfs, 1957. 3, 635-694 (Monograph No. 9).

Meehl, P. E. The dynamics of structured personality tests. Journal of Clinical Psychology. 1945, 1, 296-303.

Mills, C., & Hogan, R. A role theoretical interpretation of personality scale item responses. Journal of Personality, 1978, 46, 778-785.

Mischel, W. Personality and assessment. New York: Wiley, 1968. Nicholls, J. G., Licht, B. G., & Pearl, R. A. Some dangers of using personality questionnaires

to study personality. Psychological Bulletin, 1982, 92, 572-580. Snyder, M. The self-monitoring of expressive behavior. Journal of Personality and Social

Psychology, 1974, 30, 526-537. Standards for educational and psychological tests (Rev. ed.). Washington, D.C.: American

Psychological Association, 1974. Uniform guidelines on employee selection procedures. Washington, D.C.: Bureau of National

Affairs, FR43, Aug. 25, 1978, p. 38314. Request for reprints should be sent to Robert Hogan, Department of Psychology, University

of Tulsa, Tulsa, OK 74104.