The importance of replication to ensure the validity and reliability of research has beenwell established (Dewald, Thursby & Anderson, 1986; Feigenbaum & Levy, 1993; Hubbard,Vetter & Little, 1998; Mittelstaedt & Zorn, 1984; Tsang & Kwan, 1999). Replication servesthe fundamental role of protecting against the uncritical acceptance of empirical results.It is thus as important as the core academic practices of peer review and publication ofresearch, and is necessary for any stream of scientific inquiry to develop the requisite rigor
� A previous version of this paper was presented at the Academy of Management Annual Meetings in Chicagoin 1999.
534 K. Singh et al. / Journal of Management 2003 29(4) 533–549
of a science. For emerging fields of inquiry, particularly in the social sciences, replicationrepresents an essential core practice.
Though the need for and benefits of replication are essentially unchallenged, continuedcalls for more replication in business and strategy research (e.g.,Hubbard et al., 1998) indi-cate its inadequacy. The need for replication research is overshadowed by reward structuresthat are believed to favor original over replication research (Hubbard & Armstrong, 1994;Kerr, Tolliver & Petree, 1977; Neuliep & Crandall, 1991, 1993), and by perceived biasamong top journals for publishing original research (Tsang & Kwan, 1999). Consequently,relatively limited replication strategy research has been published (e.g.,Hubbard et al.,1998).
We highlight the role and importance of replication in strategy and propose three sets ofmeasures to increase the volume and value of such research. We first propose a re-concep-tualization of replication studies, to that of the good-enough replication. We then provide aframework to focus replications to improve understanding of the state of different sub-fieldsand to facilitate their theoretical advancement. Finally, we propose means for promoting andpublishing replication research. In presenting our proposals, we focus on conceptual andpractical means to facilitate and guide the development and conduct of replication researchin strategy.
Replication in Strategy Research
Replication is traditionally defined as the duplication of a previously published empiricalstudy to determine whether the findings of that study are repeatable. A replication withextension departs from the original study in some respect or employs different data whilelargely repeating the original study to evaluate the generalizability of earlier results (Brown& Coney, 1976; Reid, Soley & Wimmer, 1981).
Tsang and Kwan (1999)classify replications into six types along two dimensions: (1)whether the same methods of measuring constructs and analyzing data are employed; and(2) whether the sources of data are the same. The resulting six categories of replications(checking of analysis; exact replication; empirical generalization; reanalysis of data; con-ceptual extension; and generalization and extension) are useful for understanding differentapproaches for conducting replications. This classification largely makes redundant theexisting classification of replication, and replication with extension.
Replications are particularly useful for dealing with two problems that characterize re-search in an emerging field: protecting against Type 1 errors and enhancing the general-izability of empirical findings. The preference for works that reject the null hypothesis inthe behavioral sciences (Bakan, 1967; Hubbard & Armstrong, 1992; McNemar, 1960) mayproduce a bias where studies that find a statistically significant result are published, whilethose that fail to detect a result are not (Rosenthal, 1979). The generalizability argumentfor replication rests largely on the complexity of phenomena studied in the social sciences.The large number of forces that impact organizational and human behaviors in business,and the varying nature of their impact make the drawing of conclusions difficult. The de-velopment of findings into theories applicable beyond the specific context in which theyare first observed requires extensive testing in other contexts.
K. Singh et al. / Journal of Management 2003 29(4) 533–549 535
The importance of replication is illustrated by the recent replication byLane, Cannellaand Lubatkin (1998)of the landmark study byAmihud and Lev (1981). The latter studyestablished what has become a precept in the agency literature, that shareholder monitoringaffects the self-serving behavior of managers. However, there have been almost no attemptsto verify or generalize these findings through replication for many years.Lane et al. (1998)replicated Amihud and Lev’s study using a sample that closely matched the original, whileadopting more management-oriented conceptual frameworks, and with methodological andmeasurement improvements. Their results did not supportAmihud and Lev (1981), a findingverified in a second study they conducted concurrently on a different sample. Importantly,Lane et al. (1998)do not attribute their differing results to deficiencies of the original study,but to improved understanding of the phenomena being studied.
A single contradictory study is insufficient to completely overturn seminal findings, andwill therefore require verification through further replication. Nevertheless, the absence ofearlier efforts to replicateAmihud and Lev (1981)represents a significant deficiency thatshould have been addressed. There is clearly a need for studies to replicateAmihud and Lev(1981)andLane et al. (1998)to establish which of fundamentally contradictory findingsshould serve as the foundation for this stream of research. How researchers react to thefindings ofLane et al. (1998)represents an important test of this field’s progress towardsthe ideals of scientific inquiry.
Many studies have found relatively limited replication in several fields of business re-search, such as economics (Dewald et al., 1986; Feigenbaum & Levy, 1993), finance (Kane,1984), marketing (Hubbard & Armstrong, 1994; Raman, 1994), and management (Hubbardet al., 1998). In sampling 701 empirical papers published in nine first-, second- and third-tiermanagement-oriented journals over twenty years,Hubbard et al. (1998)found no strictreplications, and only 37 replications with extension in strategic management. Evaluationof these 37 studies found few differences in publication patterns across various tiers of jour-nals, other than that top-tier journals were more likely to publish supportive than conflictingreplication research. AsHubbard et al. (1998)included several leading strategy-orientedjournals in their sample, their study partially demonstrates the insufficiency of replication instrategy research. This result is a concern, in light of the search for rigor that has character-ized strategy research in recent years (Shrivastava, 1987) and the identification of replicationas a key aspect of this process (Schendel, 1995).
The conclusion of insufficient replication in strategy is based on the assumption thatreplication publications reflect the quantity of replication research actually undertaken. It ispossible that extensive replication research is actually undertaken but that these studies arenot published. This could occur if, as suggested byRosenthal (1979), replication studiesconsistently find supportive results or fail to find significant results. Alternatively, it ispossible that extensive replication research is not published because journals favor originalresearch and perceive replication to be less useful or creative (Madden, Easley & Dunn,1995; Neuliep & Crandall, 1991).
Reasons for Inadequate Replication in Strategy Research
Several reasons can be advanced for the limited number of published strategy repli-cation studies. Some of these reasons are common to other fields, and include (1) the
536 K. Singh et al. / Journal of Management 2003 29(4) 533–549
lack of publication outlets (Reid et al., 1981); (2) the belief that replications that sup-port original research do not make significant contributions and therefore are not submit-ted or accepted for publication (Dewald et al., 1986; Kane, 1984; Lindsay & Ehrenberg,1993); (3) reluctance to accept a replication that contradicts original findings on the groundsthat the results are an artifact of the replication (Hubbard, 1994); and (4) the belief thatreplication research is viewed and rewarded less positively by journals, peers, and in-stitutions (Hubbard & Armstrong, 1994; Kerr et al., 1977; Neuliep & Crandall, 1991,1993).
In addition, there are three strategy-related reasons that may explain why researchershave placed less emphasis on replication studies (e.g.,Bergh & Holbein, 1997; Bowen &Wiersema, 1999; Hoskisson, Hitt, Wan & Yiu, 1999; Schendel, 1995; Shrivastava, 1987).These are the relatively weak state of strategy theories, the complexity of strategy models,and the difficulty of obtaining data.
Relatively weak state of strategy theories. The immaturity and emerging nature ofstrategy theories relative to, for example, microeconomics, suggests greater need andreward for original discovery rather than for replication. The nascence of strategyresearch also provides greater opportunities for original discovery than in more estab-lished post-paradigmatic fields. The lack of consensus on the fundamentals of strategyand organizational theory (Pfeffer, 1993) also hinders replication efforts. Though emerg-ing fields might be expected to emphasize replication, the grounding of strategy theoriesin business, the rapid diffusion of these theories to practice, and the significant rewardsfor such diffusion may explain why strategy as a field has not emphasized replicationresearch.
Complexity of strategy models. Strategy research usually focuses on complex com-binations of variables and relationships that are difficult to isolate and study in varyingcontexts (Hoskisson et al., 1999; Mitchell, 1998). This complexity makes replications dif-ficult to undertake, since even a slightly different context is likely to introduce additionalfactors that need to be controlled. This may also explain why some studies that appearto replicate many aspects of previous research are not identified or viewed as replica-tions. It is also difficult to obtain strictly identical variables when data are specificallycollected from primary or archival sources. Therefore, replication studies are more difficultto undertake in strategy than in physical sciences or some other fields of the behavioralsciences.
Difficulty of obtaining data. Strategy research often focuses on firm-level issues, forwhich data are relatively difficult to obtain. Examples are studies based on surveys of seniormanagers or that use archival data of firm-level performance for an entire industry over anextended period. Data sets are often purpose collected and thus are kept proprietary untilthey have been extensively examined. This difficulty reduces the probability of replicationresearch, since the rewards for replicating a study on inaccessible and/or costly data areperceived to be less than for studies that evaluate new issues. Consequently, many of thelarge-scale, more rigorous empirical studies in strategy are based on databases that are notavailable for examination or analysis. When data are available, such as through on-line
K. Singh et al. / Journal of Management 2003 29(4) 533–549 537
databases, original studies are likely to have utilized all or most of the data, preventingreplications for the same time period.
Interestingly, the factors that explain limited replication research in strategy are preciselythose that dictate the need for such research. The relatively weak state of strategy theory,the complexity of strategy models, and the difficulty of obtaining data represent impor-tant arguments for conducting more replication in strategy. The fact that some data thattest theories are either unavailable or inaccessible must challenge the validity of strategyresearch, notwithstanding the rigor with which that research is undertaken. Similarly, thefield’s evolving nature should accentuate, rather than attenuate, the need for verification.As strategy research is at a pre-paradigmatic stage, replication can serve as critical meansfor adding depth to knowledge by affirming or disconfirming findings of previous research.Recognizing the problems that hinder replication research in strategy therefore sets theground for increasing such research.
We propose three sets of actions to translate the call for increased replication into morepublished studies: Re-conceptualising the notion of a replication, focusing replication re-search, and measures to promote the conduct and publication of replication studies.
Re-Conceptualizing Replication
We believe that in all but the most narrow and focused of studies, strict replicationsas conceptualized in the physical sciences cannot be conducted in strategy research. Thesources of variation in strategy studies are so great that it is improbable that any studycan precisely duplicate the context to meet the standards of experimental replication. Wesuggest two approaches for dealing with this issue. One approach is to plan for and conductinternal replications at the time the original study is conducted. While they do not solvethe problem of uncontrollable variance, these efforts provide tests of hypotheses usingmultiple research contexts (such as, for example, in different product-markets, industries,or countries). They also furnish stronger evidence of research results than conventionalstatistical tests of significance, delineate the likely boundaries for findings, and rule outalternate explanations. Relatedly, such efforts may also be more acceptable for researchfunding than exact replication research.
A second, complementary approach is to change our conceptualization of what consti-tutes a replication in strategy, in order to allow and encourage such studies. We proposethe adoption of a definition based on the good-enough principle (Leong, 1985; Serlin &Lapsley, 1985) to delineate what is a good-enough replication. The good-enough principleis a pragmatic approach to observing social phenomena which recognises that precise du-plication is almost always impossible. However, the existence of variation does not in itselfnecessarily amount to a fundamental or significant difference. Instead, variation within arange that does not amount to major deviation from the original may permit the phenomenato be viewed as a good-enough representation of the original. Since precise duplicationis not possible, the prior specification of acceptable deviation is necessary to delineate areplication that adequately approximates the original. Therefore, the good-enough approachrecognizes that variation in social phenomena is permissible, should be expected, and canbe planned for.
538 K. Singh et al. / Journal of Management 2003 29(4) 533–549
The Good-Enough Replication
We propose that a good-enough replication is one that conscientiously follows as closelyas possible, the stated methodology of the original study pertaining to design, procedure, datacollection, analysis, and reporting of results. If variations are made, the replication shoulddocument departures, provide sound theoretical or methodological rationale for these depar-tures, specify their likely impact on research findings, and account for the affected outcomesarising from them. Our definition encourages faithful adherence to original procedures soas to preserve methodological integrity. However, where departures are made, the onus is onthe replication researchers to furnish persuasive evidence permitting independent evaluationof their rationale and effects.
The appropriate application of the good-enough replication offers advantages that makeits adoption desirable. First, it explicitly recognizes the difficulty of exact replications andoffers a reasonable solution for overcoming this difficulty. Second, by requiring thea priorispecification of variation and resulting impact, the good-enough approach places additionaldemands on researchers that strengthens overall rigor, without being onerous. Third, itallows the conduct of replications within the context of research with the potential to advanceknowledge, albeit incrementally, in most circumstances.
The good-enough approach is feasible and realistic and will allow greater replicationresearch in strategy. We therefore propose moving away from the question “Is it possibleto replicate a study?” to focus on the question, “Is it possible to conduct a study thatapproximates the original sufficiently as to comprise a good-enough replication?” This shiftin conceptual frames can have significant impact on the nature and quantity of replication.
Though theTsang and Kwan (1999)six-category classification of replications is usefulfor understanding replications, their approach does not address the challenge of conductingreplications, which is the focus of our proposal. For example, a researcher seeking to conductan empirical generalization (employing the same measures and analysis on a differentpopulation) faces the same challenge of exactly replicating measurement and analyticalmethods in a different context and time. This researcher continues to face the question,“what degree of variation in constructs and their measures is permissible, so as to be viewedas being the same as in the original study?” Or, for a generalization and extension replication(which deviates from the original study in employing different methods for measuringconstructs on different populations), “how much variation will result in the study beingviewed as being fundamentally different from the original?” The good-enough approachcan be employed to evaluate these issues. Therefore, our specification of the good-enoughreplication is consistent withTsang and Kwan’s (1999)perspective and complements theirclassification.
Consider a situation where Researcher A proposes a theory based on an extensive casestudy of three organizations in the US. Researcher B wishes to test the generalizabilityof the original study by repeating it in Japan. As B is unable to obtain access to conducta case study (or perhaps lacks the training or interest required to conduct a case studyappropriately), a mail survey is conducted. As the industry used by A does not exist inJapan, B conducts the study in another industry. Though B defines and measures most ofthe same variables used by A, some variables are dropped and other controls added. Ingeneral, B’s study is based on more quantitative data but much less contextual information
K. Singh et al. / Journal of Management 2003 29(4) 533–549 539
and insight about processes. Clearly, B’s study is not an exact replication. Neither can it beconsidered to be a replication with extension, since it is inadequate relative to some aspectsof the original study. However, B’s study may be considered a good-enough replication ofA’s study on the grounds that (1) it supports or verifies an earlier study with significanttheoretical contributions and implications where a strict replication may not be feasible;(2) it employs identical dependent and key independent variables (though measurementsmay differ) at the same level of analysis, and (3) it studies the same phenomenon with anarguably superior research design and different population.
The good-enough replication also clarifies that departures in methodology need not nec-essarily be considered poor replication research or undesirable. For example, the emer-gence of new data analysis techniques or superior methodological approaches suggestedby the original effort may be instrumental in improving understanding of the issues understudy. Replication studies can therefore consciously depart from the methods of the orig-inal study by adopting improved processes, while constituting a good-enough replication.However, compliance with original methodological approaches is an important require-ment for any study to be considered a replication, so that departures should generally beundertaken when justifiably better alternatives exist and their impact on research outcomesdelineated beforehand. Even so, such methodological departures must also not be so sig-nificant as to be tantamount to the conduct of a different study. Nor does good-enoughreplication imply that researchers can pick and choose any issue from an original studyas long as they can justify doing so. Clearly, investigating an issue raised from a paststudy, however well justified, from a different theoretical perspective, with potentiallydifferent methodological imperatives, constitutes new research rather than good-enoughreplications.
Our definition of replication is an extension of the falsificationist demand that resear-chers establish what would be accepted as refuting instances by setting standards in-dicating the types of outcomes and procedures adopted that are in reasonably goodaccord with the original study. It should also be noted that our good-enough definitionprovides for replication research that aims to uncover evidence that differs from origi-nal results. In particular, a change in procedures may help provide additional informa-tion about the theory or phenomenon investigated. Consequently, the good-enough repli-cation can encourage more replication research, without compromising on the aims ofreplication.
However, it is necessary to exercise caution that the good-enough replication is not viewedas a lower, looser hurdle that replication research must satisfy, but is viewed as a reasonableand pragmatic solution to insurmountable data, methodological, and other research issues.These studies must therefore satisfy the standards of good original research and meet therequirements of good-enough replications, before they can in fact be viewed as good-enoughreplications.
Diversification research and studies employing the organizational ecology perspectiveare examples of streams in strategy that can be viewed as having undertaken extensivegood-enough replications. In fact, the adoption of the good-enough standard to evaluatethese areas suggests that they can be characterized as programs of study that have adheredto original methodologies and documented the rationale for deviations, while aiming toreplicate and extend findings.1 Significantly, though their findings are not unchallenged,
540 K. Singh et al. / Journal of Management 2003 29(4) 533–549
the diversification and ecology-based strategy literatures are viewed as providing amongthe more robust conclusions and rigorous methods of strategy research (Hoskisson & Hitt,1990; Pfeffer, 1993; Ramanujam & Varadarajan, 1989; Winter, 1990).
Extension of the Good-Enough Concept and an Illustration
Our discussion of the good-enough replication has assumeda priori specification of theintent to replicate. This requirement will ensure that researchers are aware of and attemptto replicate key aspects of the original study to the extent possible. However, an importantapplication of the good-enough principle allowsex post identification of studies that aregood-enough replications of original research, despite not having been conducted as such.Explicit acceptance of the good-enough principle would therefore make it possible to reviewa research stream to identify studies that have adequately complied with significant aspectsof a prior study, to replicate that study to a good-enough degree. In particular, reviews ormeta-analyses would represent contexts in whichex post good-enough replications can andshould be identified, to establish the validity, reliability, and generalizability of originalstudies and to facilitate theoretical development. Indeed, the validity of meta-analyses insocial studies is based on the assumption that sampled studies are good-enough replicationsof an original study or of each other. Explicit recognition of this assumption will inform theconduct of meta-analytic studies, and equally important, constrain researchers to includingonly appropriate data and studies.
We illustrate theex post good-enough replications inTable 1, using studies that evalu-ate a fundamental issue in strategy, the impact of industry and other factors on businessand firm-level performance.Schmalensee (1985)is a pioneer study of this area of research.Rumelt (1991)andWernerfelt and Montgomery (1988)extendSchmalensee’s (1985)studyby using longer data periods, different statistical methods, and added variables. The some-what controversial nature of Rumelt’s findings encouraged several researchers to re-visitthe issue.Brush, Bromiley and Hendrickx (1999), McGahan and Porter (1997), Powell(1996), andRoquebert, Phillips and Westfall (1996)follow in the same research stream,though each departs from preceding studies in substantial respects.Table 1summarizes keycharacteristics and findings of each study.
Rumelt (1991)andPowell (1996)explicitly indicated their intention to replicate and/orextend the earlier study. In contrast,Brush et al. (1999), McGahan and Porter (1997),and Roquebert et al. (1996)(all of which were published inSMJ, as were the Rumeltand Powell studies) did not statea priori that they were replicating and/or extending anearlier study, despite multiple references to the Rumelt study. Though not strict replications,Brush et al. (1999), McGahan and Porter (1997), Powell (1996), and Roquebert et al.(1996)meet the good-enough standard. The evaluation of these four papers collectively asgood-enough replications ofRumelt (1991)provides valuable information on the reliabilityof Rumelt’s findings and methods, offers greater confidence in consistent findings, andmuch greater potential for advancing theory.2 The value of the good-enough replication isfurther illustrated by our use ofTsang and Kwan’s (1999)six categories of replications toevaluate the studies inTable 1. By this standard, at leastBrush et al. (1999)andPowell(1996)would not be considered as replications, because of substantial methodological andtheoretical differences, and differences in perspectives. Therefore, the good-enough concept
K.Singh
etal./JournalofManagem
ent200329(4)
533–549541
Table 1Selected studies on “how much does industry matter?”
No. Significantlydifferent researchmethods and narrowerscope of study
542 K. Singh et al. / Journal of Management 2003 29(4) 533–549
can contribute importantly botha priori to the conduct of replication research, andex post,to the identification of good-enough replication studies and research streams.
Acceptance of the good-enough approach to replication will encourage more such studiesand will allow more research that seeks to build on established findings. As our example inTable 1suggests, it is likely that against the good-enough standard, the frequency of repli-cation research will be viewed as higher than detected byHubbard et al. (1998). However,we do not expect the frequency of replications would then be so great as to invalidate thecall for greater replication research.
The adoption of the good-enough replication will also help journals and readers to distin-guish research that makes incremental advances within an existing literature stream from re-search that is fundamentally different from an original study. We do not intend this to serve asa research quality indicator but to help deal with a problem that afflicts some fields of strategy,that of inadequate knowledge accumulation and consensus on fundamentals (Pfeffer, 1993).It would be particularly beneficial for authors of review papers or of meta-analyses to identifykey papers and to define research streams that follow these key papers using the good-enoughreplication standard. This would also facilitate the emergence of otherwise unrecognized“invisible colleges” or networks of researchers, with positive impact on the development ofconsensus on theoretical and conceptual fundamentals (Pfeffer, 1993). We propose that thediversification, organizational ecology, and alliances literatures are suitable choices for theinitial conduct of analyses aiming to identify good-enough replication streams.
Focusing Replication Research
Our second recommendation focuses on locating replication efforts where they can havethe largest impact. Strategy research has progressed at different rates with, for example,research on diversification having advanced more than that on knowledge management.Thus, replication may serve different purposes in different sub-fields, such as verificationof basic findings, and knowledge accumulation for theoretical development. What criteriashould be employed for deciding the purpose and location of replication research?Table 2presents our framework for this purpose.
Traditionally, emerging fields that have not developed theoretical and methodologicalfundamentals adequately (cell A), such as knowledge management, would not have at-tracted replication studies. Traditionalists may argue that replication of weak theory wouldnot add significant value, and should be deferred until adequate theoretical and method-ological development occurs. However, a broader perspective of the value of replicationresearch as contributing towards knowledge accumulation, the predictive powers of the-ory, and verification and falsification of theory, suggests a more aggressive approach. Asreplications serve to build a theoretical base, a confirmatory finding obtained under suitableconditions will provide quantum leaps in credibility (Tsang & Kwan, 1999). We believethat knowledge management would benefit significantly from replication efforts that focuson building theoretical understanding and on empirical generalizations. Replications thatseek to verify basic propositions and findings of an emerging field, and that seek to extendor falsify basic theory should therefore be encouraged. As theoretical and methodologicalunder-development may discourage or hinder exact replications, broader perspectives of
K. Singh et al. / Journal of Management 2003 29(4) 533–549 543
Table 2Focusing replication research
Degree of theory development
Limited Substantial
Degree ofmethodologicaldevelopment
Limited A B
e.g., knowledge management e.g., resource-based view of firmAction: Checking ofanalysis and data, exactreplications, empiricalgeneralizations andconceptual extension
Focus: Prediction andverification for theoreticaldevelopment
Focus: Verification andfalsification
Substantial C D
e.g., strategic groups e.g., diversification, alliancesAction: Conceptualextension andgeneralization
Action: Generalization andextension
Focus: Prediction,knowledge accumulationand falsification fortheoretical development
Focus: Broadenunderstanding, e.g., throughinter-disciplinaryreplications, unique methods,or radically different data
the value of replications and on the application of the good-enough principle will facilitatethe progress of the field.
At the other end of the spectrum are research streams with developed theoretical andmethodological foundations (cell D), such as diversification and alliances. It may be rela-tively easy to conduct replication studies in these fields, since most theoretical and method-ological issues have achieved broad support. However, these fields can be advanced throughreplications that adopt cross-disciplinary perspectives, radically different methodologies,or non-traditional data. Replications across national, institutional, and cultural contexts canprovide useful opportunities for such advancement.
The generalizability of research results and theories across cultural boundaries is of-ten questioned. Although the vast majority of strategy research has been conducted andpublished in North America, it has been offered and applied globally, usually without repli-cation (Ghoshal & Westney, 1993). The mirror image problem is equally serious: researchconducted outside North America or Europe is often dismissed as irrelevant or inapplica-ble or simply ignored, often without attempts to refute results through replication in othercontexts. Replication in various international contexts would help address both aspects ofthis problem.
Table 2also presents two other classes of research streams, one (cell B) that has madesubstantial progress in developing its theory but which has not progressed adequately interms of research methods (e.g., resource-based view of the firm), and the other (cell C) that
544 K. Singh et al. / Journal of Management 2003 29(4) 533–549
made some progress on research methodology, without adequate theoretical advancement(e.g., strategic groups). Replications in such fields should either be focused on advancingtheories or research methods.
It has been argued that research on strategic groups, for example, does not amount toa scientific body of knowledge because it has largely been driven by empirical methodsand data, while failing to develop theoretical building blocks (Barney & Hoskisson, 1990;Dranove, Peteraf & Shanley, 1998). If this view is accepted, replication efforts in thisarea are not productive and therefore should not be attempted until more fundamental is-sues are resolved. Against the good-enough standard, it is possible to argue that strategicgroup research has had too many replications that have aimed for empirical generaliza-tions of under-developed theory. Acceptance of this argument may help direct strategicgroup research towards more useful conceptual development and extension. Importantly,this argument indicates that scarce research resources can be preserved and channeled intomore productive uses with the adoption of the good-enough principle. A clearer focus forreplication research, as suggested inTable 2, may have similar positive effects.3
Promoting and Publishing Replication Research
It would be useful as an initial step to establish the number of replication studies sub-mitted to journals, and whether the rejection rates for these papers are higher than fornon-replication studies. An additional step would be to survey editors and editorial boardsto identify their perceptions of the importance, quality, and need for replication studies.These steps would provide additional evidence of the status and extent of replication re-search, and provide additional inputs on how this trend can be corrected.
Conferences with a replication theme or track, and special sections and issues in journalsdevoted to promoting replications can help build outlets for disseminating such works(Reid et al., 1981). Researchers other than the original author should be encouraged toreplicate, thus avoiding the problem of higher incidences of supportive results when authorsrather than independent investigators replicate their own work (Hubbard & Vetter, 1996).Some journals have treated replications as more than an auditing device. Responding to asuggestion byFeige (1975), the editors ofJournal of Political Economy (seeEditors, 1975)created a “Confirmations and Contradictions” section in the journal. In reaction toKane(1984), theQuarterly Journal of Business and Economics announced a new editorial policygiving priority to publishing replication studies. More recently, theJournal of ConsumerResearch (Mick, 2001) announced a new regular section, “Re-inquiries,” to encouragethe publication of replication studies. However, top journals have greater demands forjournal space and may believe that it is not productive to publish replications that only offer“marginal contributions.” This is especially so if the replication is not the first of the originalstudy (Tsang & Kwan, 1999). Under these conditions, top journals, concerned with the needto maintain reputations and encourage originality, may be less likely to publish replications.
Special issues of leading strategy journals to showcase quality replication research wouldrepresent a good start.4 These journals should also undertake electronic publication of repli-cations of original studies they have previously published. This approach would overcomespace limitations and would carry relatively limited costs. However, journals should ensure
K. Singh et al. / Journal of Management 2003 29(4) 533–549 545
that these electronic volumes adhere to all quality standards associated with regular issues.These outcomes are unlikely to result without the participation of the editors of leadingstrategy and management journals. We therefore urge these leaders to actively call for repli-cation studies. An initial step may be their commissioning studies that trace good-enoughreplications research streams of the ten most cited studies in strategy.
The promotion of replication by leading strategy researchers would set an example forothers to pursue replications and enhance the image of replication research. Though thesescholars would naturally be attracted to more intellectually challenging tasks, they maybe encouraged to pursue joint work with junior colleagues on replication efforts or withinternational researchers, to replicate their work in other country, cultural, or institutionalcontexts. Doctoral students should conduct replications (Reid et al., 1981) as part of theirtraining and to inculcate the value of such research.
All these efforts should be supplemented by researchers’ willingness to identify theirstudies as replications and/or extensions. This will accelerate consolidation of knowledge.An example is our illustration of the articles byBrush et al. (1999), McGahan and Porter(1997), and Roquebert et al. (1996), whose impact on the issue of industry impact onfirm performance may have been enhanced if they had been identified as replications ofpreceding studies. This practice is also important especially when replications appear indifferent publications in the same sub-field or in different sub-fields.
Greater discussion of the relevance of replication to the strategy research and publicationprocesses will help address concerns related to their conduct and publication, especially inrespect to perceived editorial bias. Ironically, a less rigorous original study may be easierto publish than a less rigorous replication, since a replication has to at least match the rigorof its original predecessor. A less rigorous original study may provide value-added in otherways that may offset weaknesses in its rigor. Hence, enhancing the rigor of replicationsmay correct editorial biases against such work. In addition, researchers should encouragereplications of their own studies, and undertake replications of major works within theirarea of interests. As part of these efforts, and with the advent of improved technology, itwould be useful to establish a central data warehouse or library to make available proprietarydatabases for replication research. This effort would be greatly boosted by donations byleading scholars of their well-researched datasets. We believe that making data availablewill encourage more replication research, and more original research that would otherwisenot have been conducted because of data limitations. We recognize the issue of intellectualproperty rights, but believe that this can be overcome by institutionalizing procedures fororiginal author approval and recognition, and maintaining publication embargoes till afterthe original research is published.
The implementation of these proposals will depend on the existence of reliable proce-dures, the provision of adequate oversight, and on the reputation of the organizing institu-tions. We therefore call on the leading academic institutions with an interest in strategy—theAcademy of Management, the Strategic Management Society, and the Academy of Interna-tional Business—to lead in implementing these suggestions. One or more of these institu-tions should establish, maintain, promote, and make available a central library of databasesfor replication studies. They may also consider electronic publication of replication studiesconducted on datasets borrowed from their library, with due regard to journal publicationrequirements.
546 K. Singh et al. / Journal of Management 2003 29(4) 533–549
Collectively, these efforts should see a rise in replication research. The question natu-rally arises: “How much replication is enough?”Neuliep and Crandall (1993)propose thatjournals allot 15% of their space to replication studies, significantly higher thanHubbardet al.’s (1998)estimate of the 2.6% of journal pages that they actually occupy. While thisbenchmark will almost certainly be sufficient, we believe the answer is not dependent onthe number of published replication papers meeting a quantitative hurdle. Instead, the suf-ficiency of replication research must depend on the field of strategy demonstrating thecharacteristics that replication research seeks to achieve, i.e., strategy becoming a coherentbody of knowledge based on theory that has been extensively tested and verified. We proposethe following criteria to indicate the sufficiency of replication research: (1) Major researchfindings are verified in replications studies and in studies that extend the original in otherimportant contexts, industries, and countries. (2) Replication studies that consistently pro-duce results contradicting original studies succeed in refuting the proposed relationships. (3)Controversial theories or research results achieve widespread acceptance and mainstreamstatus as a consequence of support provided by replication studies. (4) Replication researchis published in the leading strategy journals routinely, though not necessarily with greatfrequency. (5) Leading researchers conduct replications more often. (6) Authors willinglyidentify their research as replication and editors consider such research on equal footing.
Conclusion
The tradition of conducting and publishing replications has not been adequately estab-lished in strategy. As one of the solutions to this problem, we propose a more realisticand appropriate conceptualization of replication in strategy, the good-enough replication.This, with our other two recommendations—to focus and promote replication in strategyresearch—aims to encourage the conduct and publication of such studies. The adoption ofthese measures will facilitate knowledge accumulation and theory building in strategy.
Notes
1. One reviewer questioned why good-enough replications were more common in thesestreams of research. For organizational ecology research, we believe a combination ofthe following to be the main reasons: (1) The strong theoretical base the field startedwith from its adoption of concepts from ecology and the biological sciences. Thisprovided researchers with a wide and deep range of hypotheses for testing and so-phisticated methodological tools that advanced empirical research rapidly. (2) Mostorganizational ecology studies were based on industry analyses that incorporated sur-vival, density, and niche data. Though difficult to collect, once compiled, unrelateddatasets could be used for replication and verification of core concepts, which re-lied on similar data types. (3) Institutional factors, such as the early concentration oftop ecology researchers in a few schools, the appointment of leading ecologists totop journals in the field, and the willingness of these journals to publish papers thatwere closely related enough to be viewed as good-enough replications. The reasonsfor the diversification stream are somewhat different and involved a combination of
K. Singh et al. / Journal of Management 2003 29(4) 533–549 547
the following: (1) Diversification was possibly the first major issue central to firmstrategy that was theoretically important, empirically testable, and that had not beenextensively addressed by economists. It naturally attracted the depth of attention andinterest that other issues have not received, even with the development of the fieldand the explosion of researchers in strategy. (2) Some early follow-on papers gener-ated different results or employed improved techniques, giving the impetus to otherresearchers to examine the issue in greater depth. (3) Many diversification studieshave used tapes or other on-line datasets, easing data requirements and encouragingthe sharing of datasets. The methods employed for diversification research were alsoaccessible.
2. We evaluate these four studies againstRumelt (1991)rather than the pioneering work ofSchmalensee (1985)since these studies focused more on Rumelt’s study and findings,rather than on those of Schmalensee. The Social Science Citation Index identifies 152studies published before July 2002 which citeRumelt (1991), several of which (beyondthose identified here) are likely to be good-enough replications. The adoption of thegood-enough replication approach to view these papers will allow the identificationof replicated findings and more reliable conclusions about the issues addressed in thisresearch stream.
3. Referring to the issue of why replication may occur more frequently in some fieldsthan in others, a reviewer made the following comment, which we reproduce here withpermission: “. . . it may be interesting to examine why there is too much replicationin some areas and not enough in others. This examination may lead to a deeper under-standing of why some research topics are more successful (in attracting researchers’attention) as compared to others. It should be in our competence to pose and find an-swers to this question in replication research. Herein may lie the role of the paradigmused by the faculty members during a person’s doctoral program, candidate’s back-ground and goals, level of nurturing provided by the faculty members, requirementof time and other resources as well their availability, and tacit knowledge about thesub-area of research, etc. For example, meta-analysis may be used more in organi-zational behavior than strategy because of the number of studies available on strate-gic management topics and organizational behavior faculty members teaching mostmethods/meta-analysis courses. The newly emerging area of micro-strategy may berelevant in this investigation in the way how individuals make decisions and choicesthat result in an organization’s strategy or a field’s direction.” This comment illus-trates the myriad factors that influence research trends. In the context of replications,it reinforces our argument that action must be taken broadly and at fundamental levels.
4. It is interesting thatStrategic Management Journal’s ten- and five-year indices, pub-lished in 1990 and 1995, respectively, do not contain the subject category “replication.”
Acknowledgments
We thank Senior Associate Editor David Ketchen, and three anonymousJM reviewers,Steven W. Floyd, Fred S. Switzer and Chuck Williams, for their helpful comments, and theNational University of Singapore for funding this research.
548 K. Singh et al. / Journal of Management 2003 29(4) 533–549
References
Amihud, Y., & Lev, B. 1981. Risk reduction as a managerial motive for conglomerate mergers.Bell Journal ofEconomics, 12: 605–617.
Bakan, D. 1967.On method. San Francisco: Jossey-Bass.Barney, J. B., & Hoskisson, R. E. 1990. Strategic groups: Untested assertions and research proposals.Managerial
and Decision Economics, 11: 187–198.Bergh, D. D., & Holbein, G. F. 1997. Assessment and redirection of longitudinal analysis: Demonstration with a
study of the diversification and divestiture relationship.Strategic Management Journal, 18: 557–571.Bowen, H. P., & Wiersema, M. F. 1999. Matching method to paradigm in strategy research: Limitations of
cross-sectional analysis and some methodological alternatives.Strategic Management Journal, 20: 625–636.Brown, S. W., & Coney, K. A. 1976. Building a replication tradition in marketing. In K. L. Bernhardt (Ed.),
Marketing 1776–1976 and beyond: 622–625. Chicago, IL: American Marketing Association.Brush, T. H., Bromiley, P., & Hendrickx, M. 1999. The relative influence of industry and corporation on business
segment performance: An alternative estimate.Strategic Management Journal, 20: 519–547.Dewald, W. G., Thursby, J. G., & Anderson, R. G. 1986. Replication in empirical economics: The Journal of
Money, Credit, and Banking project.American Economic Review, 76: 587–603.Dranove, D., Peteraf, M., & Shanley, M. 1998. Do strategic groups exist? An economic framework for analysis.
Strategic Management Journal, 19: 1029–1044.Editors. 1975, December. Editorial comment.Journal of Political Economy, 83: 1295–1296.Feige, E. L. 1975, December. The consequences of journal editorial policies and a suggestion for revision.Journal
of Political Economy, 83: 1291–1295.Feigenbaum, S., & Levy, D. M. 1993, July–September. The market for (Ir)reproducible econometrics.Social
Epistemology, 7: 286–292.Ghoshal, S., & Westney, E. D. 1993. Introduction and overview. In S. Ghoshal & E. D. Westney (Eds.),Organization
theory and the multinational corporation: 1–23. NY: St. Martin’s Press.Hoskisson, R. E., & Hitt, M. A. 1990. Antecedents and performance outcomes of diversification: A review and
critique of theoretical perspectives.Journal of Management, 16(2): 461–509.Hoskisson, R. E., Hitt, M. A., Wan, W. P., & Yiu, D. 1999. Theory and research in strategic management: Swings
of a pendulum.Journal of Management, 25: 417–456.Hubbard, R. 1994, July. The dangers of generalizing from published marketing studies.Journal of the Market
Research Society, 36: 257–260.Hubbard, R., & Armstrong, J. S. 1992. Are null results becoming an endangered species in marketing?Marketing
Letters, 3(2): 127–136.Hubbard, R., & Armstrong, J. S. 1994, June. Replications and extensions in marketing: Rarely published but quite
contrary.International Journal of Research in Marketing, 11: 223–248.Hubbard, R., & Vetter, D. E. 1996. An empirical comparison of published replication research in accounting,
economics and finance.Journal of Business Research, 35(2): 153–164.Hubbard, R., Vetter, D. E., & Little, E. L. 1998. Replication in strategic management: Scientific testing for validity,
generalizability, and usefulness.Strategic Management Journal, 19(3): 243–254.Kane, E. J. 1984. Why journal editors should encourage the replication of applied econometric research.Quarterly
Journal of Business and Economics, 23(Winter): 3–8.Kerr, S., Tolliver, J., & Petree, E. L. 1977, March. Manuscript characteristics which influence acceptance for
management and social science journals.Academy of Management Journal, 20: 131–141.Lane, P. J., Cannella, A. A., & Lubatkin, M. H. 1998. Agency problems as antecedents to unrelated mergers and
diversification: Amihud and Lev reconsidered.Strategic Management Journal, 19: 555–578.Leong, S. M. 1985. Metatheory and metamethodology in marketing: A Lakatosian reconstruction.Journal of
Marketing, 49(Fall): 23–40.Lindsay, R. M., & Ehrenberg, A. S. C. 1993, August. The design of replicated studies.American Statistician, 47:
217–228.Madden, C. S., Easley, R. W., & Dunn, M. G. 1995. How journal editors view replication research.Journal of
Advertising, 24: 77–87.McGahan, A., & Porter, M. E. 1997. How much does industry matter, really?Strategic Management Journal,
18(Special Issue): 15–30.
K. Singh et al. / Journal of Management 2003 29(4) 533–549 549
McNemar, Q. 1960. At random: Sense and nonsense.American Psychologist, 15: 295–300.Mick, D. G. 2001. Announcing a new section in JCR: Re-enquiries.Journal of Consumer Research, 28(1):
un-numbered.Mitchell, W. 1998. Commentary on ‘Entry into new market segments in mature industries: Endogenous and
exogenous segmentation in the U.S. brewing industry’ by A. Swaminathan.Strategic Management Journal,19(Special Issue): 405–411.
Mittelstaedt, R. A., & Zorn, T. S. 1984. Econometric replication: Lessons from the experimental sciences.QuarterlyJournal of Business and Economics, 23(Winter): 9–15.
Neuliep, J. W., & Crandall, R. 1991. Editorial bias against replication research. In J. W. Neuliep (Ed.),Replicationresearch in the social sciences: 51–69. Newbury Park, CA: Sage Publications.
Neuliep, J. W., & Crandall, R. 1993. Reviewer bias against replication research.Journal of Social Behavior andPersonality, 8: 22–29.
Pfeffer, J. 1993. Barriers to the advancement of organizational science.Academy of Management Review, 18:599–620.
Powell, T. C. 1996. How much does industry matter? An alternative empirical test.Strategic Management Journal,17: 323–334.
Raman, K. 1994, March. Inductive inference and replications: A Bayesian perspective.Journal of ConsumerResearch, 20: 633–643.
Ramanujam, V., & Varadarajan, P. 1989. Research on corporate diversification: A synthesis.Strategic ManagementJournal, 10: 523–551.
Reid, L. N., Soley, L. C., & Wimmer, R. D. 1981. Replication in advertising research: 1977, 1978, 1979.Journalof Advertising, 10(1): 3–13.
Roquebert, J. A., Phillips, R. L., & Westfall, P. A. 1996. Markets vs. management: What ‘drives’ profitability?Strategic Management Journal, 17: 653–664.
Rosenthal, R. 1979, May. The “file drawer problem” and tolerance for null results.Psychological Bulletin, 86:638–641.
Rumelt, R. P. 1991. How much does industry matter?Strategic Management Journal, 12(3): 167–185.Schendel, D. 1995. Notes from the Editor-in-Chief.Strategic Management Journal, 16(1): 1–2.Schmalensee, R. 1985. Do markets differ much?American Economic Review, 75(3): 341–351.Serlin, R. C., & Lapsley, D. K. 1985, January. Rationality in psychological research: The good enough principle.
American Psychologist, 40: 73–83.Shrivastava, P. 1987. Rigor and practical usefulness of research in strategic management.Strategic Management
Journal, 8(1): 77–92.Tsang, E. W. K., & Kwan, K. M. 1999. Replication and theory development in organizational science: A critical
realist perspective.Academy of Management Review, 24(4): 759–780.Wernerfelt, B., & Montgomery, C. A. 1988. Tobin’s q and the importance of focus in firm performance.American
Economic Review, 78(1): 246–250.Winter, S. G. 1990. Survival, selection, and inheritance in evolutionary theories of organization. In J. V. Singh
(Ed.),Organizational evolution: New directions: 269–297. Newbury Park, CA: Sage Publications.
Kulwant Singh (Ph.D., University of Michigan, Ann Arbor) is Associate Professor of Busi-ness Policy at the NUS Business School. His research focuses on cooperative and compet-itive strategy in the context of technological and rapidly changing industries.
Siah Hwee Ang (Ph.D., National University of Singapore) is research fellow in the Facultyof Management at Cass Business School, London. His primary research interests relate totechnology strategy, alliances and reputation in knowledge-based and high-tech industries.
Siew Meng Leong (Ph.D., University of Wisconsin, Madison) is Professor of Marketingat the NUS Business School. His research interests relate to marketing theory, consumerbehavior, and Asian marketing management.
