The Role of Demonstrations in Federal R&D Policy

8/14/2019 The Role of Demonstrations in Federal R&D Policy

http://slidepdf.com/reader/full/the-role-of-demonstrations-in-federal-rd-policy 1/62

The Role of Demonstrations in Federal R&D Policy

July 1978

NTIS order #PB-284387



Library of Congress Catalog Card Number 78-600070

For sale by the Superintendent of Documents, U.S. Government Printing OfficeWashington, D.C. 20402 Stock No. 052-003-00557-3



FOREWORD

The Federal Government now spends about $28 billion per year on research anddevelopment (R&D) activities and facilities in the United States. With another $20 billionper year from the private sector, the total national investment in R&D approaches $50billion annually.

Large though this total is, it portrays only a small portion of the overall impact of

R&D on the economy and the quality of life in our society. Research and developmentis the engine that drives the currents of change in our civilization. From R&D stem the in-ventions, techniques, and processes that propel innovations through our economic andsocial system. Moreover, it has been estimated that, on the average, each person en-gaged in R&D eventually generates 6 to 10 other jobs throughout the economy. As aconsequence, the $48 billion annual national investment in R&D has a massivemultiplier effect on our entire socioeconomic system.

Therefore, it behooves Congress to consider this investment carefully and pay closeattention to the ways in which it is allocated and used, as well as to the framework of laws, regulations, incentives, and constraints whereby the fruits of scientific research anddevelopment are converted into operational results.

Furthermore, R&D and the process of innovation help to determine the options andestablish many of the parameters whereby specific technologies can be assessed for theirpotential impacts on society. In assessing a particular technology, the Office of Technology Assessment (OTA) compares its advantages and disadvantages with thoseof alternative technologies and assesses its impact on economic, social, environmental,and political factors within a perspective of probable future human needs, capabilities,and values.

To carry out its assessments effectively, OTA needs a thorough understanding of the Nation’s R&D effort and of the process whereby R&D results are converted intouseful innovations. While helping to strengthen and integrate OTA’S overall assessmentactivities, such understanding also enables OTA to assist Congress in better shaping thenational investment in R&D by developing more soundly based R&D policies andpriorities. Thus through such understanding, OTA can more effectively fulfill its mandate

to give Congress early indication of the impacts of technological change.

In response to these needs and the urging of a number of congressional committeesand individual Members, the OTA Board authorized a Program of R&D Policies andPriorities, which became operational in May 1976.

Recognizing that such an assessment cannot be carried out effectively through asingle, comprehensive project which attempts to address all facets of the problem, theProgram was designed to proceed through a series of manageable, interrelated studieswhich will help to build an understanding of how to maximize the beneficial impacts of our total R&D enterprise.

The Program has operated with the guidance of three interrelated Advisory Panelsmade up of distinguished leaders of science, technology, industry, labor, the profes-

sions, and the consumer, environmental, and public interest movements.The Panel on the Health of the Scientific and Technical Enterprise, chaired by Dr.

Harvey Brooks, Benjamin Peirce Professor of Technology and Public Policy at HarvardUniversity, has been concerned with ways we can maintain and enhance the health andvitality of the entire scientific and technical enterprise.

. . .

111



demonstrations which have

RUSSELL W. PETERSON Director

Office of Technology Assessment

iv



OTA R&D Policies and

Priorities Program Staff

Eltis Mottur, Assistant Director, OTA and Program Manager

John H. Young, Project DirectorJudith Angerman Dorothy S. Poole

Contractors

The body of this report was prepared by:

Thomas K. Glennan, Jr.William F. Hederman

Leland L. JohnsonRichard A. Rettig

of the RAND Corporation

OTA Publishing Staff

John C. Holmes, Publishing Officer

Kathie S. Boss Joanne Heming



Program on R&D Policies andPriorities Steering Committee

Russell W. PetersonDirectorOffice of Technology Assessment

Jerome B. WiesnerChairmanTechnology Assessment Advisory Council

Lewis M. BranscombChairmanPanel on the Applications of Science and

Technology

Application

Technology

Preston T. BankstonDeputy Director, Office of Science and TechnologyMississippi Fuel and Energy Management Commission

Harvey BrooksChairmanPanel on Health Of the Scientific and

Technical Enterprise

Gilbert F. WhiteChairmanPanel on Decision making on R&D Policies and

Priorities

Ellis MotturEx Officio

of Science andAdvisory Panel

Lewis M. Branscomb, ChairmanVice President and Chief Scientist, IBM Corporation

Alice Tepper MarlinExecutive DirectorCouncil on Economic Priorities

Barry R. BloomPresident, Central Research

Pfizer, Inc.

Irving BluestoneVice President, Director

United Auto WorkersGeneral Motors Division

Edward E. David, Jr.PresidentEXXON Research and Engineering Company

Charles J. HitchPresidentResources for the Future, Inc.

C. Lester HoganPresident and Chief Executive OfficerFairchild Camera and Instrument Corporation

Claire NaderIndependent Consultant

Arthur S. ObermayerPresidentMOLECULON Research

Robert M. SolowInstitute Professor

Corporation

Massachusetts Institute of Technology

Philip H. TreziseSenior FellowThe Brookings Institute

Herbert F. York Professor of Physics

University of California at San Diego

vii



SUMMARY

Demonstration projects have become increas-ingly popular as innovative responses to a broadspectrum of national problems. Federal expendi-tures for demonstration projects, including socialprogram demonstrations, have grown to over $1billion annually, and further growth appearslikely. Yet their effectiveness has been limited.

Demonstrations are frequently used in areas,such as energy, where there is controversy con-cerning what constitutes desirable and timely in-novations. To effectively evaluate proposed andongoing demonstrations in such areas, Congressmust be able to develop a set of realistic expecta-tions for a demonstration’s outcome. In order toprovide this capability, the present reportdevelops:

q A perspective that permits analysis of demonstrations in many policy contexts;and

q Guidelines to aid the evaluation of individ-ual proposals for demonstration projects.

The perspective and guidelines are derivedfrom analyzing the lessons learned in both socialand hardware demonstrations.

Perspective

The perspective developed here encompassesboth the different purposes of demonstrationprojects and the principal factors that determinesuccess.

What is a Demonstration?

The term “demonstration” is ambiguous. Itcan mean to test an innovation for the purpose of formulating national policy. Or, it can mean toshow others the relative advantage of an innova-tion for the purpose of persuading them to use it.We define a demonstration here as:

A project in which an innovation is operatedat or near full scale in a realistic environment inorder to (1) formulate national policy, and/or (2)promote the use of an innovation.

The term “innovation” may refer to a new pro-gram, product, or process.

3 0 - 7 8 - 2

Policy-Formulating Demonstrations

We refer to the first type of demonstrationproject as a p o l i c y -f o r m u l a t i n g d e m o n s t r a -

t i o n . Federal decisionmakers are its principal au-

dience. The income maintenance experimentsthat examine the administrative feasibility, costs,and impacts of a variety of income transfer pro-grams provide an example of this type of demon-stration. Demonstrations that provide the neces-sary technical or economic information for settingregulations and standards are also a type of policy-formulating demonstration. The Refanprogram to reduce noise from commercial jet air-craft and inform regulatory decisions of theFederal Aviation Administration is an example.Typically, policy-formulating demonstrations areintended to provide information to Federal deci-

sionmakers about:

q Technological and administrative feasibilityof instituting a policy or adopting an innova-tion.

q Expected economic, environmental, andsocial impacts of the policy or innovation.

Policy-Implementing Demonstrations

We refer to demonstrations to promote the useof an innovation as p o l i c y -i m p l e m e n t i n g

d e m o n s t r a t i o n s . Solar heating demonstrationsare an example. The criterion of success for thistype of demonstration is diffusion of the innova-tion from the demonstration site. Thus, thosenon-Federal decisionmakers who control the rateof diffusion of an innovation are the principal au-dience for these demonstrations. In addition tothe factors mentioned above for policy-formulat-ing demonstrations, policy-implementing dem-onstrations are typically intended to provide in-formation on:

q

q

q

q

Costs of adopting and using the innovation.Reliability of that innovation in use.Demand for the innovation.Feasibility of implementing the innovationat the adopter’s site.

The common denominator of both types of demonstrations is the generation of informationfor decisionmaking, and a single project may in-

ix



corporate elements of both types of demonstra-tions.

Who Expects What From a Demonstration?

Demonstrations often serve important politicalas well as information functions, such as pro-viding:

q A compromise between those who prefer a

large-scale operating program and thosewho prefer nothing.

q A means of expressing concern for a na-tional problem.

q A response by executive agencies topressures to show the usefulness of theirR&D program.

Furthermore, because demonstration projectsare an instrument of transition from R&D to use,an innovation may be simultaneously moving:

FROM

Small scale in the laboratoryControl by R&D personnel .

Technical criteria of success.

Federal management . . . . .

TO

. . Full scale in the field;

. . Control by operatingpersonnel;Inst i tut ional cri teria of .success;

. . Private sector managementor State and local manage-ment.

The range of different but limited perspectivesinvolved makes a transition such as this difficultto bring about and complicates the evaluation of proposed demonstrations.

All of these factors imply that various in-terested parties often have different objectivesand expectations from a demonstration project.Some may view it primarily as a test of an in-novation; others may view it as a promotion of an innovation; still others may view it primarily asa means of expressing concern for a nationalproblem.

These different objectives and expectationsmake the evaluation of a demonstration difficultand necessarily judgmental. Nonetheless, such

judgments can be informed by realistic expecta-tions of a demonstration’s outcome.

Institutional Environment, Technology,and Success

There are two essential requirements for a suc-cessful demonstration project: (1) clearly specify-

X

ing the relevant information to be generated, and(2) generating reliable information once speci-fied. The specification of the relevant informationdepends largely on the i n s t i t u t i o n a l e n v i r o n -

m e n t ; reliability of the information generateddepends largely on the t e c h n o l o g y (pp. 22-27).

An institutional environment is characterizedby the users of an innovation, its suppliers,

markets, and regulators of those markets. A well-developed institutional environment has twocharacteristics—a tradition of using the results of R&D, and an accepted Federal role. In such anenvironment there is generally consensus amongthe various participants as to what constitutes adesirable innovation and when a demonstrationis appropriate. An example of such a well-developed institutional environment is theagricultural sector with its system of experimentstations, land-grant colleges, and extensionagents. Conversely, the components of the in-stitutional environment in the elementary and

secondary education system are unevenlydeveloped, poorly linked, and frequently indisagreement over the process of change in theschools. In the public sector, there is often a lack of consensus—and even fundamental value andgoal conflicts—regarding the desirability of in-novations. In the private sector, the discipline of the marketplace tends to force a greater degreeof consensus as to what constitutes a desirable in-novation.

The term “technology” refers here toknowledge for the production and delivery of

goods and services. Some technologies, such ascentral-station electric power generation, are welldeveloped, with well-characterized inputs andoutputs and a good understanding of their rela-tionship; other technologies are poorly devel-oped, as in education and law enforcement.Demonstrations employing poorly developedtechnologies are less likely to be perceived asgenerating reproducible results than thoseemploying well-developed technologies.

The Federal role in a given policy sector deter-mines the scope of the Federal R&D effort in that

sector, and consequently, the role of demonstra-tions (pp. 9-15). In many areas of domesticpolicy a major Federal role is relatively new andoften controversial. In areas where the Federalrole is still controversial, and it is difficult either toreach consensus as to relevant outcomes or to



generate reliable information, the effectivenessand role of demonstrations remain to be clearlydetermined. The experience in agriculture sug-gests that an extended period of time is requiredfor an effective and accepted Federal role to beestablished.

Guidelines for Evaluating Proposed

Demonstration Projects

The following questions focus attention on keyfactors that influence success in demonstrationprojects (pp. 31-41). The brief discussions at-tempt to provide the necessary perspective bywhich one can evaluate critically the answers tothese questions.

1. Are the goals for a demonstration proj-ect clearly articulated and agreedupon?

A divergence of goals and expectations amongfunders, performers, and potential audiencesmay often be inevitable. Nonetheless, to avoidan inadvertent lack of clarity in goals, an effortshould be made to ensure that program mana-gers and performers of demonstrations share acommon understanding of the purposes of demonstrations. Specifying how the results of ademonstration are to be evaluated can be an ef-fective device for clarifying goals.

2. Given the purposes of a demonstra-t ion project , ‘have the information

needs of the demonstration’s au-dience been adequately considered?

A demonstration may be intended to inform acongressional debate on a welfare system. Or, ademonstration may be intended to promote thecommercialization of a new energy technology.In either case, the demonstration should bedesigned to address the critical issues upon whichthe decisions will turn, as perceived by the rele-vant decisionmakers themselves.

This assumes, of course, that the necessarydecisions can be effectively informed by ademonstration. For example, policy-implement-ing demonstrations by themselves have beenweak means for bringing about institutionalchange. In such cases, other measures, whereavailable, should be considered as alternatives or

complements to demonstration projects (pp.48-51) .

3. If the demonstration is intended topromote the diffusion of an innova-tion, have key actors in the institu-tional environment been involved?

It is not only important that an institutional en-

vironment be sufficiently well developed to gen-erate a consensus on the criteria for successful in-novations. Diffusion of an innovation is alsoenhanced by the actual involvement of an institu-tional environment’s key actors in the planning of a policy-implementing demonstration.

An illustrative example is provided by ademonstration of mechanized refuse collectionaimed at reducing labor costs through smallercollection crews. The mechanism was designedby city personnel, but no garbage truck outfittingfirm was involved in the demonstration. Despitestriking success at the demonstration site, no

commercial firm could be found to market the in-novation, and there has consequently been littlediffusion.

4. Is the technology incorporated in ademonstration sufficiently reproduci-ble to be credible?

Projects incorporating poorly developed tech-nologies seldom lead to similar operations inother locations. For example in education,significant innovation appears to occur at a siteonly when there are major adaptations in the in-

novation to meet local needs. The apparent re-quirement for a unique implementation of an in-novation at each potential adoption site tends topreclude its replication elsewhere.

Support for local problem-solving effortswhere diffusion is not a relevant success criterionmight be better termed “subsidized local develop-ment” than a demonstration project. Althoughfew such efforts have survived withdrawal of theFederal subsidy, such support might still be usefulwhere a genuine commitment to address realproblems can be distinguished from mere op-

portunism in response to available funds.Extensive adaptation of an innovation to meet

specific local needs may also occur wheretechnologies of substantial reproducibility areused in a larger “system .“ Examples include mass

xi



transit, law enforcement hardware, and mining.For technologies of low to intermediate repro-ducibility, successful replication at multiple sitesmay be necessary to stimulate diffusion.

5. Is the technology being demonstrated

well in hand?

Even though a technology may be potentiallyreproducible with great reliability, its develop-ment may not have proceeded to the pointwhere it is well in hand. There is persuasiveevidence that attempts t o demonstrat e atechnology that is not well in hand adversely af-fect diffusion.

For example, extensive development work

was required during the demonstration of adesalination process in Freeport, Tex. Althoughthis work led to important improvements in per-formance, it also caused interruptions in plantoperations. Potential adopters of desalinationplants mistakenly perceived these interruptionsas an indication that the desalination process wasunreliable, and there has been no diffusion.

This example illustrates the importance of con-sidering the alternative of a full-scale test at a test-bed facility when a technology is not yet well inhand. Such testing would avoid both conveyingthe adverse impression of unreliability to poten-

tial adopters and facilitate engineering solutionsto technical problems by removing real-worldoperating constraints.

6. Is there sufficient time and opera-tional flexibility for the demonstrationto meet its objectives?

Strict time constraints generated by policy orpolitical needs have seriously impaired demon-strations from achieving diffusion success.

, operat ional f lexibi l i ty is essentia l forcoping with unanticipated difficulties that fre-

** appear in such projects.

7. Is there sufficient evidence of commit-ment to the innovation by the per-former?

Cost sharing and initiative by non-Federal par-ticipants in a policy-implementing demonstrationprovide an important test of user need. It maybedifficult to distinguish opportunism as a responseto available funds from a commitment to addressreal problems without a significant measure of cost sharing.

Failure of the private sector to assume asubstantial portion of the costs and risks is itself informative, and may indicate the technological,market, institutional, or environmental uncertain-ties are too high. A demonstration would then bepremature without prior steps to reduce suchuncertainty.

Alternatively, the innovation may simply beuneconomic. Failure to commercialize would

then have nothing to do with the lack of knowledge that could be produced by ademonstration project. In such a case, policiesthat change the incentive structure facing thepotential innovator should be considered as analternative to a demonstration project, or thecommercialization effort should be delayed untilfurther R&D or changed economic conditionsmake it more attractive.

8. Does the design of the demonstrationproject reflect the experiences of pastdemonstrate ions?

Any well-conceived and well-executed dem-onstration project strives to reduce uncertainty inthe various dimensions of a problem. However,the failure of a demonstration may highlightdimensions of a problem whose significance wasnot adequately appreciated. The insights gainedcan be used to reformulate national policy or beincorporated into the design of future demonstra-tions.

For example, at the time of the Atomic EnergyCommission’s first power reactor demonstra-tions, electric utilities could not obtain liability in-surance against nuclear accidents. The recogni-tion of this barrier to the diffusion of nuclearpower led to passage of the Price-Anderson Actwhich imposed a legal limit on damages that

xii



could be claimed after a nuclear accident. By Conclusioncreating a more favorable institutional environ-ment, further demonstrations were able to suc- Despite their obvious potential, demonstra-ceed in promoting the diffusion of nuclear power. tions can be easily misused. The perspective andFurther demonstrations by themselves, however, guidelines developed here are to help Congresswould have been unable to overcome this barrier more effectively utilize this important policy in-which had nothing to do with the lack of informa- strument for pursuing national goals.tion that could be produced by a demonstrationproject.

acid



CONTENTS

Chapter Page

I.

II .

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Purposes of This Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........4

Conceptual Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........5Review of the Literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .....5Guide lines for Congress . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .....5

Format of theStudy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........6

The Historical Contex tof Dem ons t ra t ions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

The R&D Tradition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........9Agriculture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........9National Defense. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......10Basic and Applied Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11Domestic Public Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......11Regulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......12Energy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......13

TheSocialActionTradition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ..14Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......15

III. A Conceptual Frame work for the Consideration of DernonStrationProjects in Federal RaDPolicy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

The PurposesofDemonstrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......19Demonstrations intheR&D Tradition . . . . . . . . . . . . . . . . . . . . . . . . . . . .......19Demonstrations aslnstruments inthe political process . . . . . . . . . . . . . . . . . . . . ..21

General FactorsAffectingthe SuccessofDemonstrations . . . . . . . . . . . . . . . . . . . . 22

TheTechnology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......22The institutional Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .24

IV. Fact o rs Affect ing t he Succ ess o fDem ons tr a t ions . . . . . . . . . . . . . . . . . . . . . 31

Factors Affecting the Success of Policy-Implementing Demonstrations . . . . . . . . . , .31Technologies With Low Reproducibility. . . . . . . . . . . . . . . . . . . . . . . . . . .......31Technology ThatIsWellinHand . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......33Demonstrationsin Well-Developed Institutional Environments. . . . . . . . . . . . . . . .34Requirementsfor Cooperative Efforts . . . . . . . . . . . . . . . . . . . . . . . . . . . .......34User Need. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......35Time and Operational Flexibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......36

Factors Affecting the Success of Policy-Formulating Demonstrations . . . . . . . . . . ...37Perceived Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .......38

Perceived Relevance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. ....38Continuing CloseTiesWith Policymakers . . . . . . . . . . . . . . . . . . . . . . . . .......40Including PointsofConflict in Demonstration Design . . . . . . . . . . . . . . . . . . .....40

V. I m pl i ca t ion s f o r Co n gr es s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Reasons for Use ofDemonstration Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...45

xv





Chapter I

INTRODUCTION



Chapter I

INTRODUCTION

Demonstration projects have become increas-

ingly important in Federal Government researchand development (R&D) programs. They areused in such diverse areas as energy, transporta-tion, environmental quality, health, water re-sources, aeronautics, education, and incomemaintenance. The resources allocated to demon-strations are substantial. One study estimatedthat approximately $625 million of the Federalcivilian R&D budget was allocated to demonstra-tions in FY 1974;

1an update of that estimate

placed the funds for demonstrations at roughly$860 million for civilian R&D in FY 1977.

2

Another study estimated funds of $400 million

for social program demonstrations in FY 1976. 3

Reliable data for prior years are not available; it is

clear, however, that Federal support for demon-

strations has increased at a rapid rate and that

continued increases in funding are likely.

It might be reasonable to assume from theirapparent popularity that demonstrations are awell understood and highly effective instrumentof Federal policy. But this is not the case.Demonstrations are poorly understood and theireffectiveness is open to question.

One reason for our limited understanding of demonstrations is the lack of an agreed upon

‘Walter S. Baer, Leland L. Johnson, and Edward W.Merrow, Analysis of Federally Funded Demonstration Proj-ects: Final Report, The RAND Corporation, R- 1926 -DOC,April 1976, p. 2. Hereafter, this study will be cited asFederal Demonstrations. A supplementary volume of casestudies is Waker Baer, C. Johnston Conover, Cheryl Cook,Patricia Fleischauer, Bruce Goeller, William Hederman,Leland Johnson, Edward Merrow, Richard Rettig, and JohnWirt, Analysis of Federally Funded Demonstration Projects:Supporting Case Studies, The RAND Corporation, R-1927-DOC, April 1976. This volume will be cited as FederalDemonstrations: Case Studies.

‘See Walter S. Baer, Leland L. Johnson, and Edward W.Merrow, “Government-Sponsored Demonstrations of NewTechnologies.” Science, Vol. 196, May 27, 1977, p. 951.

3See the Report of the Study Committee on Social R&D,

The Federal investment in Knowledge of Social Problems,The National Academy of Science, Washington, D. C.,forthcoming, 1977.

definition. An analyst, several years ago, referred

to “the ‘demonstration-research’ project as themajor instrument for social planning in Americancommunities today. ”4 Another analyst of socialprograms distinguished among experimental, de-velopmental, and demonstration projects, defin-ing the latter as aimed at “showing administrativeand/or political feasibility. ”s A forthcoming studyof social research and development by the Na-tional Academy of Sciences (NAS) defines ademonstration as “a small-scale program under-taken in an operational setting for a finite periodof time to test the desirability of a proposedcourse of action .“s A recent study of energypolicy defined the purpose of energy technologydemonstrations as “providing hardware andnonhardware information with sufficient reliabili-ty and credibility to inform commercial utilizationdecisions.’” Last, the most extensive empiricalstudy to date of demonstrations referred to “ac-tivities undertaken at sufficient scale so thatresults can be easily translated into regular com-mercial operations” and distinguished this frompilot plant and field test activities that “involveoperation on a smaller scale to determine techni-cal feasibility, to identify major problem areas,

and to provide early estimates of costs. ”8The various usages of demonstrations reflect

semantic ambiguity about the term. There are,however, two principal meanings that underliemost discussions. The first is that demonstrationsare intended to prove: to test, validate, andprove the innovation under consideration. The

second is that demonstrations are intended toshow others the relative advantages of an in-

‘Martin Rein, Social Policy: Issues oj Choice andChange, Random House, New York, 1970.

‘Walter Williams, Social Policy Research and Analysis:

The Experience in the Federal SoCml Agenc~es, AmericanElsevier Publishing Company, New York, 1971, pp. 53-54.

‘The Federal Investment in Knowledge O) Socia) Prob-lems, op. cit.

‘Don E. Kash et al , Our Energy Future, The Universityof Oklahoma Press, Norman, Okla., 1976, p. 25

‘Federal Demonstrations, p. 19.

3



novation for the purpose of persuading them touse it. Incorporating both these meanings intoour analysis, we have adopted the followingdefinition:

A demonstration is a project, involving an in-novation and operated at or near full scale in arealistic environlnent, for the purpose of (1) for-mulating national policy or (2) promoting the useof the innovation.

The term “innovation, ” as used in this study,may refer to a new program, product, or process.The “use of an innovation” in our definition en-compasses the stages of adoption, implementa-tion, and incorporation.

The ways that Congress has provided statu-tory authorization for demonstrations indicate therange of their intended uses. Frequently, a broadauthorization for R&D activity, without specificreference to demonstrations, has provided thebasis for agency demonstration projects. TheMaritime Administration, for example, has con-

ducted its ship development and constructiondemonstration program under authority to con-duct “research and development activities. ” Asecond pattern has been congressional authoriza-tion of demonstrations for which the objectiveshave been broadly defined. The Bureau forEducation of the Handicapped, for instance, hasbroad authority to support “demonstrationsrelating to education of handicapped children”and “demonstrations relating to physical educa-tion or recreation for handicapped children. ”Congress has also authorized demonstrationsand specified their objectives. The solid wastedemonstration authority of the EnvironmentalProtection Agency is for “the development andapplication of new and improved methods of col-lecting and disposing of solid waste” as well as“processing and recovering materials and energyfrom solid waste. ” Congress has on occasion pro-vided specific authority for particular demonstra-tion projects. In 1956, for instance, Congressauthorized the “construction, outfitting, andpreparation for operation . . . of a nuclear-powered ship, ” which became the IV. S. Sauan-nah. Nevertheless, the two primary meanings of

proving and of showing are clear, even fromthese various statutory formulations.9

————.——.

Vhis material is drawn from an analysis by Christopher J.Conover of The RAND Corporation, “Federal Demonstra-tion Projects: Statutory Language to Fund DemonstrationProjects, ” prepared for the Federal Demonstrations study.

4

More important, perhaps, than the absence of clarity about the meaning of demonstrations isthat the results of their use have been discourag-ing. Some of the main conclusions of one recentstudy, for example, include the following:“demonstration projects have a narrow scope foreffective use; “ “demonstration projects appear tobe weak tools for tackling institutional andorganizational barriers to diffusion ;“ “large dem-

onstration projects with heavy Federal fundingare particularly prone to difficulty. ”lo From othersources, come additional criticisms.

11 Demon-

strations, it is argued, provide little generalized in-formation because they are often characterizedby poor research designs. They seldom arereplicated beyond their initial sites, nor do theyoften lead to commercialization. Many concludethat demonstrations provide little additional in-formation that could not be obtained more inex-pensively by other means.

Thus, Congress has good reason to be con-

cerned with demonstrations. First, because Con-gress appropriates funds for demonstrations, thesubstantial amount of resources invested in themis in itself reason for congressional attention.What is the Federal Government receiving for itsmoney? What might be better alternative uses of these funds? Second, because Congress providesauthorization for demonstrations, an understand-ing of their appropriate uses and limits is neededfor informed decisionmaking on new initiatives.Should authorization be broad or specific, pro-viding substantial or limited administrative discre-tion? Should demonstrations be authorized by

themselves or in relation to other policy in-struments and tools? Third, because statutoryauthority is reviewed regularly and performanceof programs assessed periodically through thereauthorization and oversight processes, criteriafor the review and evaluation of demonstrationscan be helpful. When should demonstrations beemployed? How should they be managed? Whatare the most likely predictors of success?

Purposes of This Study

q To develop a conceptual framework for the

analysis of demonstration projects.———

‘“Federal Demonstrations, pp. v, vi.“See, for example, Alice M. Rivlin and P. Michael Tim-

pane (eds.), Planned Variation in Education: Should WeGiue Up Or Try Harder?, The Brookings Institution,Washington, D. C., 1975.



q To review and synthesize the literature on

demonstrations in relation to this concep-tual framework.

q To draw out implications of the analysis forcongressional action.

Each of these purposes deserves further com-ment.

Conceptual Framework

Demonstrations, as is indicated above, have

been used both with physical technologies andsocial programs. Yet no study has attempted toanalyze hardware and social demonstrationstogether. One objective of the conceptual frame-work developed below is to permit the analysis of demonstrations in many policy contexts, in-cluding both physical and social technologies. Arelated objective is to determine whether thereare lessons that can be learned from the use of

hardware demonstrations that have utility forsocial demonstrations, and uice uersa. Last, anobjective of this study has been to understand thecontribution of demonstrations to the differentstages of the policy process by distinguishing be-tween those that generate information useful forformulating national policy and those undertakenfor the purpose of promoting the utilization of atechnology.

Review of the Literature

The literature that constitutes the basis for thisstudy is indicated in the bibliography. It falls intofour principal categories. First, there are twostudies that focus directly upon demonstrationsas policy instruments, both done by the RANDCorporation. One of these is the Federal Demon-strations study, cited above, performed for theDepartment of Commerce. The other is the“change agent” study of educational demonstra-tions conducted by Berman and McLaughlin forthe Office of Education. 12 The second category of literature consists of two retrospective analyses of specific social experiments, both sponsored by

“See Paul Berman and Milbrey Wallin McLaughlin et al.,Federal Programs Supporting Education Change, Vols. l-V,The RAND Corporation, R-1589 l-HEW, September 1974and R 1589 2-5-HEW, April 1975,

The Brookings Institution. ’3 One, by Pechmanand Timpane, reviews the New Jersey negativeincome tax experiment. 14 The other, by Rivlinand Timpane, reviews the experience withplanned variation in education. 15 Third, there is ageneral literature on R&D utilization and com-mercialization that is relevant to demonstrationsbut in an indirect way. The forthcoming NASreport on social R&D is a case of this type. 16

Finally, there are prescriptive analyses of R&Dissues. The analysis by Kash, et al., of U.S.energy options illustrates this type of analysis. 1 ’

In this report, we draw upon the literature to sup-port or challenge propositions that emerge fromour conceptual framework. Thus, this study isnot a conventional literature review but ananalytical interpretation and extension of theliterature.

Guidelines for Congress

Previous s tudies of demonst ra t ions havedirected their policy recommendations primarily

to the Federal agencies sponsoring demonstra-

tions. Congress, however, has responsibilities for

the authorization of demonstrations, appropria-tion of funds to support them, and review and

evaluation of program performance. An impor-

tant purpose of this study, then, is to analyze the

literature and experience of demonstrations i nthe context of congressional concerns.

‘3 For the purposes of this study, we do not distinguishbetween social experiments and social demonstrations. Themajor difference between these two types of projects lies inthe nature of the evaluation by which the effects are deter-mined rather than in the fundamental purposes they serve.See Henry W. Riecken and Robert F. Borouch (eds.),Social Experimentation: A Method for Planning andEvaluating Social lnteruention, Academic Press, New York,1974, chapter 1.

‘*Joseph A. Pechman and P. Michael Timpane (eds. ),Work incentives and Income Guarantees: The New JerseyNegatiue Income Tax Experiment, The Brookings Institu-tion, Washington, D. C., 1975.

“Alice M. Rivlin and P. Michael Timpane (eds. ), PlannedVariation in Education: Should We Give Up Or Try

Harder?, The Brookings Institution, Washington. D. C.,1975,

“See also Arthur D. Little, Inc., Federal Funding of Civilian Research and Development, Volume 1: Summary,Washington, D. C., February 1976.

“Don E. Kash et al., Our Energy Future, University of Oklahoma Press, Norman, Okla., 1976.

5



Format of the Study

Chapter 11 includes a brief overview of thehistorical context from which demonstrationshave evolved. Chapter 111 develops a conceptual

framework for analyzing policies for u s i n gdemonstrations. This framework is used in chap-

ter IV to develop several propositions concerningfactors likely to affect the success of demonstra-tions. In the concluding chapter, chapter V, we

trace the implications of the analysis for Con-gress.

As this report will make clear, experience andthe research literature provide no infallible guidesto good policy concerning demonstrations. Prob-lems that arise from using demonstrations are theresult of the institutional complexity of the publicand private sectors and the workings of a demo-

cratic government. We hope that a treatment of this complexity will be helpful to Congress andothers for future action.

6



THE

Chapter II

HISTORICAL CONTEXT

OF DEMONSTRATIONS



T H EI I

Demonstrations as an explicit policy instru-

ment have emerged from two distinct traditions.They have evolved, first, within the context of

the Federal Government’s support of R&D and,second, in the context of the Federal social actionprograms of the 1960’s. It is both appropriateand useful, therefore, to sketch the origins of demonstrations in these two traditions so that wecan better understand their contemporary mani-festations. And because the R&D tradition isolder, broader, deeper, and the source of muchof our thinking about demonstrations. it willreceive somewhat greater emphasis than thesocial action tradition.

The R&D Tradition

The Federal Government currently supportsR&D for a number of purposes at an annual levelof $23 billion. Historically, six major patterns of support can be identified, each of which isdiscussed briefly below. These patterns areagriculture, national security, basic and appliedresearch, domestic public policy, regulation, andenergy. (Though agriculture and energy arelogically subordinate to domestic policy, they are

historically distinct and of sufficient importance towarrant separate consideration. ) An analysis of these six patterns can reveal important similaritiesand differences in the purposes for which theFederal Government has supported R&D.Because demonstrations have been justifiedmainly within the context of Federal R&D pro-grams, then, an understanding of these patternscan contribute to an understanding of demonstrations.

Agriculture

Agriculture is often regarded as possessing acomprehensive system for generating practicalresearch results and transmitting them into appli-cation. This “system” includes a highly differen-tiated research establishment, supporting ac-tivities from basic research through development

Chapter II

HISTORICAL CONTEXTOF DEMONSTRATIONS

State, regional, and National laboratories. It

also includes educational programs, locatedmainly at State land-grant colleges, which pro-vide training for the next generation of farmersand agricultural scientists and continuing educa-tion for agricultural producers, processors, andeven consumers. In the State experiment sta-tions, the system has the capacity to test prom-ising new research results on the appropriatescale as well as exemplify the utility of validatedresults. Through the extension service and avariety of other communication channels, theagricultural system has the means for transmittingvaluable new information from the research

establishment to agricultural producers, proc-essors, and consumers.

Several factors should be kept in mindconcerning the agricultural system.

1First, as with

many of the other policy areas we will discuss,the development of Federal R&D policy was ac-companied by fundamental political conflict.There was no formally sanctioned Federal role inagriculture until the 1860’s. The Southern States,guided by the doctrine of States rights, suc-cessfully opposed creation of a Federal Depart-ment of Agriculture and a federally supported

system of agricultural schools prior to the CivilWar. This constitutional controversy was re-solved only when the South seceded from theUnion. The massive realignment of politicalpower and reorientation of political philosophythat then occurred in Congress resulted in sig-nificant new legislation in 1862: a Department of Agriculture was established, the Merrill Actauthorized land-grant colleges of “agriculture andmechanical arts, ” and the Homestead Actopened the lands of the West.

2

Within a sanctioned Federal role in agriculture,

‘The primary source for this section is A. Hunter J1.pee,Science in the Federal Government, Harvard UniversityPress, Cambridge, Mass., 1957, PP. 109-114. 149-183,348-350 et passim.

‘Ibid, pp. 149-151.

9



a second consideration is that the system of research, education, experimentation, and ex-tension developed slowly. The Federal Depart-ment itself went through a slow process of devel-opment which rejected a scientific discipline-oriented pattern of research organization in favorof an agricultural problem focus.

3There was a

concurrent, but also lengthy, development of State experiment stations in conjunction with theland-grant colleges,

but Federal funding andFederal efforts at coordination took 25 years todevelop. The extension service, moreover, wasnot authorized by Federal statute until 1914.Regional laboratories, and the national Beltsvillelaboratory, came even later. That a system thatevolved over at least three generations has notbeen readily duplicated by others should not besurprising.

Several other points are noteworthy for thisdiscussion. The institutional framework thatdeveloped in agriculture was much more theproduct of politicians and practical men than of

scientists.’ The framework itself, however, didnot dictate results and the “validation” of thesystem came only as scientific results in ento-mology, animal diseases, and botany began toshow practical results of the 1880’s and 1890’s.In this context, the experiment station providedthe logical site for the “demonstrations” of theday: laboratory results that appeared promisingcould be “proved” in the agricultural setting, andvalidated results could be demonstrated to skep-tical farmers.

National Defense

The major share of Federal R&D in the post-World War 11 period has been for national secur-ity purposes. Most of the national security R&Dhas been funded by the Department of Defense(DOD), though the Atomic Energy Commission(AEC) and its successor agencies have providedfunds for R&D on nuclear warhead develop-ment. Furthermore, the creation of the NationalAeronautics and Space Administration (NASA)in 1958 and the rationale for the Apollo-mannedlunar exploration mission in the 1960’s can be at-

3

This development alone took several decades (ibid., pp.157-169)

‘The deep involvement of the States through the land-grant colleges and the experiment stations practically in-sured attention to the system by political officials (see ibid.,PP. 169-176 et passim).

10

tributed to the international threat from the spaceprogram of the Union of Soviet SocialistRepublics.

The role of the Federal Government in na-tional security has never been an issue, inas-much as “the common defense” is a central partof the Constitution. Defense-related R&D, there-fore, has evolved as an integral component of DOD and the uniformed services. It has been

supported for the purposes of providing andmaintaining a technologically advanced militarycapability, generating technical options for thefuture, and hedging against technological sur-prise by international adversaries.

Demonstrations per se do not constitute partof the vocabulary of defense R&D. On the otherhand, certain R&D activities can be identified inthe defense context as quite similar to demon-strations as defined in this report. These includetesting a complex weapons system (usually at theprototype stage) at the level of systems integra-

tion of a number of established component tech-nologies in a simulation of a real-world opera-tional environment; and conducting competitionamong prototype weapons systems for the pur-pose of choosing the preferred system. These ac-tivities have their analogues in the policy-implementing and policy-formulating demonstra-tions discussed below.

The distinguishing characteristic of defenseR&D is that a strong relationship exists betweenthe R&D investment and the incorporation of theresults of that investment in advanced weaponssystems and component technologies. In institu-

tional terms, a single, integrated organizational

system is both responsible for generating newmilitary technology and for purchasing s u c htechnology. It should be emphasized, though,that the search for the appropriate organizationalarrangements to relate R&D, weapons systemsprocurement, and force posture needs has beena long and complicated one in the three decadessince World War 11.

5As complex as this institu-

tional system is, it remains simple in the fund-amental relationships that govern organizations

5A useful overview of the institutional evolution of defense R&D is presented in Herbert F. York and G. AllenGreb, “Military Research and Development: A PostwarHistory,” Bulletin of the Atomic Scientists, Vol. 33, No. 1,January 1977, pp. 13-26. York was the first Director of Defense Research and Engineering.



within it. Defense R&D and the equivalent of

demonstrations within the R&D context, there-

fore, are characterized by strong relationships

between R&D inputs and defense products and

by a fundamentally integrated, and relativelysimple, institutional system.

Basic and Applied Research

World War 11 had a powerful “demonstrationeffect” on conceptions of the Federal Govern-

ment’s role in R&D.b

Basic research becamerecognized as a critical contributing factor to ma-

jor technological advance. The connection wasmost obvious between nuclear physics andnuclear fission, but was also dramatically illus-trated in the case of penicillin. Consequently, inthe postwar era, several Federal agencies devel-oped with the support of basic research as aprimary mission.

The policy rationale for this basic research waslargely utilitarian (science for its applications) with

a strong aesthetic component (science for its ownsake). The rationale that emerged as welfareeconomists turned their attention to the area wasthat basic research generated “external benefits”that could not be fully captured by the sponsor.

7

This led to an assumption that systematic under-investment in basic research by the private sectorwould occur relative to the socially optimal levelof support and then to an a priori justification forGovernment support of basic research. Thisargument was extended to include appliedresearch whose results were not easily appro-priable by private firms for commercial applica-tion. The theoretical limit on the Government’sR&D activity was thus defined by appropriabilityof results: it was inappropriate for the Federal

‘See, for example, J. Merton England, “Dr. Bush Writesa Report: ‘Science—The Endless Frontier, ’ “ Science. Vol.191, Jan. 9, 1976, pp. 41-47, and Daniel J. Kevles, “TheNational Science Foundation and the Debate over PostwarResearch Policy, 1942-1945: A Political Interpretation of ‘Science—The Endless Frontier,’ “ 1S1S, Vol. 68, No. 241,1977, pp. 4-26.

‘See Richard R. Nelson, “The Economics of Invention: ASurvey of the Literature, ” Journal of Business Vol. XXXII,

No. 2, April 1959, pp. 101-127, and Kennetb J. Arrow,“Economic Welfare and the Allocation of Resources for In-vention, “ in Richard R, Nelson (ed, ), The Rate and Direc-tion of Inuentiue Activity: Economic and Social Factors, aReport of the National Bureau of Economic Research,Princeton University Press, Princeton, N. J., 1962, pp609-626,

Government to fund work appropriable by theprivate sector, since firms could be expected tofinance such R&D themselves. The operationallimit on Government’s R&D activity was essen-tially set by drawing a boundary between appliedresearch and development.

aIn this context, the

question of demonstrations did not arise becauseapplication of R&D results was left to the privatesector.

Domestic Public Policy

The Federal Government supports R&D in anumber of areas of domestic policy. Many of these R&D efforts include demonstrations.

The area of health and medicine, however,differs from many other policy areas in notableways. First, the rationale for Federal Governmentsupport of medical research, largely through theNational Institutes of Health (NIH), is a productof several factors—the successful applications of medical research results in World War 11 (e. g.,

penicillin, antimalarial drugs, treatment of burnand trauma),9 the limited resources of privatephilanthropy in sustaining an expanding medicalresearch enterprise after the war, 10 the lack of in

terest by the pharmaceutical industry in sup-porting fundamental research, and the lack of opposition by organized medicine.

11Second,

because the main element of NIH strategy overthe years has been to invest proportionately morein fundamental research than in applied orclinical research, interest in demonstrations hasbeen relatively weak.

12Third, in response to

‘See Richard R. Nelson, Merton J. Peck, and Edward D.Kalachek,Technology, Economic Growth, and PublicPolicy, The Brookings Institution, Washington. D. C.. 1967,pp. 171-211 et passim; George Eads and Richard R. Nelson,“Governmental Support of Advanced Civilian Technolo-gy—power Reactors and the Supersonic Transport, ” publicPolicy, Vol. 19, 1971, pp. 405-427. and George Eads,“U.S. Support for Civilian Technology Economic Theoryand Political Practice,” Research Policy, Vol. 3, 1974, pp.

2-16.9See E.C. Andrus et al. (eds. ), Aduances in Military

Medicine, Vols. 1 and II, Little, Brown and Company,Boston, Mass., 1948.

‘“See England op. cit., p. 44.*’See American Medical Association, Report oj the Comm-

ission on Medical Research, Chicago, 111., 1967.IZA very useful historical overview is presented in

Stephen P S t r i c k l a n d , Pol/tics, Science, and Dread

Disease: A Short History of United States Nledlcal ResearchPolicy, Harvard University Press, Cambridge, Mass., 1972.

11



more recent pressures to concern itself with theapplication of medical research results, NIH hassupported an increasing number of controlledclinical trials. These trials emphasize the vali-dation of research results in the clinical settingrather than promotion of widespread use of suchresults, and are thus analogous to demonstra-tions intended to prove a technology. There arecancer and heart-disease control programs, on

the other hand, that are more analogous to ex-emplification demonstrations. These control pro-grams, however, coexist rather uneasily with themore rigorous research-oriented activities of NIH.

In the mid-1960’s, Federal Government R&Dresources were increasingly allocated to anumber of domestic policy areas—urban masstransit, postal services, housing and urban affairs,education, manpower, and law enforcement andcriminal justice. The R&D efforts in these areastook place as two large secular shifts, sometimes

overlapping, occurred. One was that the im-portance of the public sector increased relative tothe private sector as, for example, in urban masstransit. The other was that the importance of theFederal Government increased relative to Stateand local government. In some policy areas, anexpanded Federal role paved the way for greaterFederal R&D, e.g., education; in others R&Dconstituted the “entering wedge” for an

expanded Federal role, e.g., criminal justice.

These domestic policy areas have two charac-teristics of consequence for demonstrations. Onthe one hand, the scientific understanding of cen-

tral phenomena and the technical base under-girding operational activity are not well

developed, thus the contribution of R&D is oftenproblematic. At the same time, the institutionalenvironment in these areas is highly complex,consisting of public and private organizationsfunctioning at all levels of government—Federal,State and local. ” The authority of Federal agen-cies is frequently limited, while responsibility isdistributed in a fragmented way throughout theinterorganizational network that constitutes theinstitutional environment. Thus, the FederalR&D agency seldom, if ever, has anything but aweak relationship with the agencies actually

“For a fascinating illustration of this point, see DonaldSchon, “The Blindness System, ” The Public fnterest, No.18, Winter 1970, pp. 25-38.

responsible for policy and program implementa-tion. Although demonstrations have become animportant part of R&D activity in these policyareas, their utility has been limited by bothtechnical and institutional factors, as discussed atgreater length in the next chapter.

Regulation

The scope of Federal Government regulatoryactivity has expanded markedly in recent yearsfrom its more traditional concerns with “markets,rates, and the obligation to serve” to include theconditions under which goods and services areproduced and the characteristics of thosegoods. This expansion has blurred, if noterased, the prior distinction between regulatedand unregulated industries. In addition, thescope of Federal regulation has also come to in-clude activity by lower levels of government, as inthe case of water pollution control standards ap-plied to municipal sewage disposal. This expan-

sion of regulatory activity has drawn the attentionof an increasing number of commentators; 15 it isnot our purpose to review this discussion here.

There is one aspect of the newer forms of regulation, however, that does bear stronglyupon the use of demonstrations as Federal policyinstruments. Regulations and standards are in-creasingly based upon technical, as distinct fromeconomic, criteria. This is true, for instance, inthe case of S0

2stack-gas scrubbers, fabric flam-

mability standards, and aircraft engine noisestandards.

l6The precise values of these stand-

ards are often a direct function of the technolog-

ical state of the art; statutes sometimes requirethe “best available technology” as an additionalcriterion. An increasing portion of Federal R&Dfunds is directed to generating the technical in-formation required for these new standard-setting activities. In this context, demonstrations

“See William Lilley, 111, and James C. Miller, III, “TheNew ‘Social’ Regulation, ” The Public Interest, No. 47, Spr-ing 1977, p. 53.

“See, for example, Charles L. Schultze, “The Public Useof the Private Interest,” Harper’s, Vol. 254, May 1977, pp.

43-62, and Regulation, a new journal initiated by theAmerican Enterprise Institute in summer 1977.l6 See the following cases in Federal Demonstrations:

Cale Studies–’’Refuse Firing Demonstration (Solid-Waste-to-Fuel-Conversion Plant), ” “Resource Recovery fromRefuse, ” “ Poultry Waste Processing, ” and the “Refan JetEngine Program .“

12



have begun to take on an important role in prov-ing or verifying given technologies. This repre-sents the use of demonstrations in a policy-formulating manner.

A concomitant of basing regulatory standardson technical criteria is that the regulations providea powerful force for diffusing technology. In-deed, the coercive quality of regulations probably

provides a more powerful incentive to adopt newtechnologies than do the usual market forces.While this naturally provides an important op-portunity for the Government to promote tech-nological change, it simultaneously creates aresponsibility to ensure that the change is, in fact,socially desirable. Demonstrations seem likely tobe an important aid in making judgments con-cerning the worth of proposed standards.

Energy

Federal Government R&D investment inenergy was, through the 1960’s, primarily di-rected to the development of nuclear powerthrough the programs of the AEC. A smallamount of coal research was supported by theDepartment of the Interior, but little R&D was in-vested in other energy sources or energy conser-vation.

The AEC initiated the Power Reactor Demon-stration Program in the mid-1950’s, and this pro-gram was instrumental in the adoption of nuclearpower by the private utilities. 17 The initial objec-tive of this program was to generate R&D in-formation and to involve commercial firms and

utilities in the construction of nuclear power-plants. Two successive rounds emphasized con-struction of small power reactors for use in ruralareas and foreign export and the developmentand exemplification of larger reactors.

The Federal Government’s role in energy wasmarkedly altered in 1973 by the action of theOrganization of Petroleum Exporting Countries(OPEC) in raising the price of OPEC-producedoil. The AEC gave way to the Energy Researchand Development Administration and the latter,in turn, to the new Department of Energy. R&D

activity was increased across all energy technol-ogies and, in the most significant policy depar-ture, “commercialization” of new technologies

“See Wendy Allen, Nuclear Reactors for GeneratingElectricity: U.S. Development from 1946 to 1963, TheRAND Corporation, R-21 16-NSF, June 1977.

became an important Federal goal.18

It is this altered Federal role with responsibilityfor “commercialization” that makes the energyarea distinct from other domestic policy areas.The Federal Government is now engaged in thecomplex task of relating public investments in thedevelopment of new energy technologies tostrategies for seeing those technologies used for

energy production and conservation in t h eprivate sector. ’9 The matching of public andprivate investment decisions, choice processes,and decision criteria is being worked out in a verycomplicated manner. Demonstrations, as it turnsout, happen to be one of the policy instruments,however imperfect, being used in the search foran improved match. 20

Conclusions

Several conclusions emerge from the abovediscussion. First, the Federal role in a givenpolicy area establishes the scope of the FederalR&D effort in that area. In national security, theFederal Government is primary and its role iscomprehensive in scope. In basic and appliedresearch, however, the Federal role is limited tothat research generating external benefits, andduplication of private sector development work isnot warranted. A second conclusion is that theseveral processes of generating scientific knowl-edge, incorporating that knowledge into usefultechnology, and building the associated institu-tional and professional capital nearly always re-quire a long period of time. This is clearly the

case in agriculture and national security, two of the most highly developed policy areas from anR&D perspective. There is no reason to supposeit less true in areas where scientific understandingand technological capability are less developed.Furthermore, if the Federal role authorizes ormandates that the R&D effort deal with the

“See MIT Energy Laboratory Policy Study Group,Government Support for the Commercialization of NewEnergy ‘Technologies: An Analysis and Exploration of theIssues, Cambridge, Mass., November 1976.

“See Leland L. Johnson, Edward W. Merrow, Walter S.

Baer and Arthur J. Alexander, Alternative institutional Ar-rangements for Developing and Commercializing BreederReactor Technology, The RAND Corporation, R-2069 -NSF, November 1976.

20 See Don E. Kash et al., Our Energy Future, Universityof Oklahoma Press, Norman, Okla., 1976, pp. 25-26 etpassim.

13



utilization of R&D results, the use of demonstra-

tions will be a derivative response. In agriculture

and defense, for instance, the comprehensive

Federal role led long ago to the equivalent of in-

tegrating demonstrations into the institutions and

procedures of these sectors. In basic and applied

research, on the other hand, because the Federal

role is limited, the question of demonstrations

has never been raised. In domestic policy, regu-

lation, and energy, where R&D is performed togenerate innovation, demonstrations are preva-lent. Even so, it is not surprising that demonstra-tions have yet to be firmly established in domesticpolicy, regulation, and energy. These areas arerelatively new, deal continuously with distributionof costs and benefits of policies among socialgroups, have complex institutional relationshipswith the private sector and with other levels of Government, and often manifest lack of consen-sus about the appropriate Federal role. Finally,the effectiveness of demonstrations will be deter-mined by the contribution of the R&D effort tothe technical problems of the operational worldand by the institutional environment in which thatrelationship is set. This point is elaborated analy -tically in chapter III.

The Social Action Tradition

Demonstrations as a policy instrument havetheir roots in a second tradition, one of social ac-tion,

z’ This tradition has as its concern those in-

dividuals, families, and groups deemed by soci-

ety to have too few of the necessities andamenities that constitute an acceptable physical,economic, and social standard of living. Specifi-cally, this tradition has championed the cause of the poor, the disadvantaged, and those who en-counter discrimination and is manifest in policieslike income maintenance, education of disad-vantaged children, provision of adequate hous-ing and health care, job training programs, andthe like. Demonstrations here frequently havebeen organized and directed by individuals withlittle scientific or technical training—socialworkers, city and regional planners, teachers and

school administrators, and public administrators.

“See Martin Rein, Social Policy: Issues of Choice andChange, Random House, New York, 1970, pp. 138-152 etpassim.

14

The mid-1960’s witnessed a significant andrapid expansion of Federal Government activitiesdirected to the alleviation of a number of socialills. In the early days of the War on Poverty, forinstance, “new programs to help the poor tum-bled out of the White House and Congress inrapid succession, and idealistic Government of-ficials worked frantically to get them started andconfidently looked forward to quick and visible

results. “22 A wide range of far-reaching sociallegislation was enacted in a similar manner dur-ing this era.

Demonstration projects were a prominentfeature of these social action programs. Often itwas the case that demonstrations funded by non-Federal sources preceded the emergence of Federal programs; they constituted “prototypes”in the sense of indicating in particular instancesthe main lines that a larger invention mightfollow. ” Demonstrations also were initially pro-posed as limited efforts to learn “what works, ”

only to be expanded to a broader program byCongress, as was the case for Model Cities. Athird instance was the use of demonstrations toinitiate a service provision program as in the com-munity mental health centers funded by the Na-tional Institute of Mental Health.

Although the term “demonstration” suggestedan effort to validate social intervention strategies,little about these demonstrations was systematic.They were described as “random innovation” byRivlin.

24(Try enough innovation approaches to a

problem and some are bound to workout; or so itwas thought. ) Seldom were these demonstra-tions subject to good program evaluation, so littleknowledge emerged of how effective programscould be used to make other projects more suc-cessful.

Even so, social action demonstrations com-mended themselves to policy makers for impor-

22Alice M. Rivlin and P. Michael Timpane, (eds. ), Plann-ed Variation in Education: Should We Giue Up Or TryHarder?, The Brookings Institution, Washington, D. C.,1975, p. 3..

“See, for instance, Terrance Keenan, “The HealthRecord of Private Foundations, ” Journal of Health Politics,

Po/icy and Law, Vol. 2, No. 1, Spring 1977, pp. 11-19, fordiscussion of the efforts of private philanthropic foundationsto “craft and test the prototypes upon which governmentprograms are modeled. ”

“Alice M. Rivlin, “Systematic Thinking Jor Social Action,The Brookings Institution, Washington, D. C., 1971, pp.87-90.



tant reasons. They were a readily availablemeans for making a prompt Federal response toa pressing social need, thus satisfying the “dosomething” injunction to policy makers. Beyondthis, their execution occurred at local govern-ment levels, thus indicating action “where itcounts. ” In addition, they implied a strategy of rational social change—first demonstrations,then full-scale programs, thus carrying promise

for Iongrun achievement. Furthermore, theywere an accommodation to the problem of scarceresources, requiring substantially fewer resourcesthan full-scale programs of intervention .25 Thisaccommodation had a dual appeal: liberals couldapprove this as a strategy of “getting a foot in thedoor, ” while conservatives could regard it as asecond-best solution where the preferred coursewas no Federal program. Finally, demonstrationsquite often were important in providing symbolicrecognition by the Federal Government of claims

for attention or resources by the important con-stituency group. These political uses of demon-strations, more prominent in the social actiontradition than in the R&D tradition, retain theirattractiveness in many situations today.

Gradually, for a combination of reasons,demonstrations in the social policy area began to

be drawn into a more formal R&D tradition. Thisdevelopment was supported by research person-nel in the agencies and by the Office of Manage-ment and Budget. A body of doctrine was devel-oped about good and bad practices in themanagement of demonstrations. Project goalsbecame more specific. The time duration of proj-

ects was more clearly indicated. Expectations

“See Rein, op cIt , pp 139-140.

were reduced. In some cases, demonstrationsbegan to be viewed as a stage in program R&Ddevelopment not unlike that characteristic of theprogram in the R&D tradition.

Conclusions

Demonstrations in the social policy area today

constitute one important means in the search bypolicy makers for ways to learn about “whatworks. ” These policy makers have been taught bythe experience of the 1960’s and 1970’s that“random innovation” leads to few generalizationsthat are widely applicable. They are generallyconvinced of the need for systematic accumula-tion of knowledge as a basis for the establishmentof sound social policy. But they are also increas-ingly aware of the limits of R&D programs ingeneral, and of demonstrations in particular. togenerate that knowledge. The rhetoric surround-ing demonstrations in the social policy area is

similar, if not identical, to that in areas involvinghardware demonstrations. But the confidencethat demonstrations are the logical next step tomove research results into operational uses is notas strong. Thus, we find demonstrations havedeveloped from two quite different traditions, butthat social demonstrations are increasingly beingguided by an R&D perspective. The fact that thelanguage and aspirations of social and hardwaredemonstrations are increasingly similar maymean that social policy and action are steadily be-ing undergirded by a systematically developedknowledge base. Alternatively, it may mean that

the language of the R&D tradition is only servingto mask temporarily the limited capacity of R&Dto generate an adequate knowledge base forsocial policy.

15



—.

Chapter Ill

A CONCEPTUAL FRAMEWORK

FOR THE CONSIDERATION OFDEMONSTRATION PROJECTS

IN FEDERAL R&D POLICY



Chapter Ill

A CONCEPTUAL FRAMEWORKFOR THE CONSIDERATION OF

DEMONSTRATION PROJECTS

IN FEDERAL R&D POLICY

Our approach to demonstrations is from anR&D perspective. It is not a narrowly technicalperspective, but one that explicitly recognizes theway that political and technical factors interact indemonstration projects. In this way, we believethat a conceptual framework can help in under-standing the uses and limits of demonstrations asa policy instrument.

This chapter sets forth a conceptual framework

within which to examine demonstration projects.We examine the purposes of demonstration proj-ects in meeting R&D objectives and other policygoals and review the general factors that affectthe probability that demonstration projects willachieve these purposes. Throughout we drawupon the literature to provide illustrations andsupport for the analysis.

The Purposes of Demonstrations

The purposes of demonstrations can be con-sidered from two quite distinct perspectives: pur-poses related to the development and applicationof knowledge through an R&D process, and theiruses within a political process.

Demonstrations in the R&D Tradition

Within the R&D tradition there are two fund-amental purposes for a demonstration project:

q To aid Federal policy makers in decidingwhether or not to adopt a policy.

q To promote the use of an innovation.

Demonstration projects in which the first pur-

‘We define innovation as a program, product. or produc-tion process in the particular existing context: innovation isnot necessarily new in the sense of “first instance. ”

pose predominates we refer to as p o l i c y -f o r m u -

l a t i n g d e m o n s t r a t i o n s ; projects in which thesecond purpose predominates we call p o l i c y -

i m p l e m e n t i n g C i e m o n s t r a t # o n s .

A good example of policy-formulating demon-strations is the income maintenance experimentsconducted by the Office of Economic Opportun-ity (OEO) and the Department of Health, Educa-tion, and Welfare. These experiments were be-

gun in the late 1960’s and continue at the presenttime. They have sought to provide information topolicy makers concerning the effects of incomemaintenance payments upon the work incentivesof the participants in the programs, the admini-strative feasibility of implementing such a pro-gram, and the difficulties of integrating incomemaintenance programs with other service pro-grams such as manpower training. The emphasisupon the work incentives reflected the beliefs of the demonstration’s sponsors concerning thelikely focus of a policy debate over welfarereform. The sponsors in OEO felt that a majorcongressional concern would be whether individ-uals who receive some form of guaranteed in-come would withdraw from the labor force andbecome permanent members of the welfarepopulation.2

An example of a policy-implementing demon-stration is those conducted by the ShipbuildingResearch, Development, and Demonstrationprogram; through this program, the MaritimeAdministration is attempting to stimulate innova-tion in the shipbuilding industry by organizing aninterindustry consortium of shipbuilding firms to

propose and assist in managing demonstrationprojects. The projects with this program invoke

‘Joseph A. Pechman and P. Michael Timpane (eds.),Work incentives and income Guarantees: The New JerseyNegatiue Income Tax Experiment, The Brookings Institu-tion, Washington, D. C., 1975, pp. 15-24.

19



the development and testing of a new technologi-cal device or production method, which is fol-lowed by a formal demonstration in a shipyard towhich all industry engineers are invited. Most of the projects are small and concern the pragmatic“nuts and bolts” problems of shipbuilding, notlarge-scale demonstrations of significantly newtechnologies. The projects have included new

techniques of welding, surface coating, materialshandling, and automation.3

In all cases in this paper, our perspective is thatof a Federal policy maker. This caveat is impor-tant. It is quite possible that the firm or institutioncarrying out a project the Federal Governmentviews as a policy-implementing demonstrationconceives the project as a means of formulatingits own “policies. ” But because we are examiningdemonstrations sponsored by the Federal Gov-ernment, our analysis is from the perspective of the Federal policy maker.

Moreover, individual demonstration projectscan possess elements of both policy-formulatingand policy-implementing purposes. A projectthat begins as a test of the worth of an innovationmay ultimately serve as an exemplary demon-stration for other sites that are funded under anew Federal program created as a result of thetest. Alternatively, a demonstration project maybe initiated to promote what is considered to be aproven concept but, as a result of adverse experi-ences in the early stages of the project, it be-comes a means of testing the worth of the con-cept. A project may also possess different pur-

poses, as perceived by the various actors con-cerned with its execution. The funder of a projectmay view the project as a test of a concept; themanager, as a means of promoting the use of aninnovation whose worth has already beenproved.

Although a demonstration may possesselements of both purposes, the distinction is im-portant. If policy makers do not specify the pur-poses of a demonstration, agencies conductingthe demonstration are free to view its purposesfrom the perspective that best suits their in-

dividual needs. This may result in the promotionof the use of innovations that Congress orexecutive policy makers do not feel are proved:demonstrations that these policy makers view aspolicy-formulating are viewed by the agencies as

3Federal Demonstrations: Case Studies, pp. E-1 - E-6.

20

policy-implementing. Alternatively, agenciesmay treat programs as experimental that Con-gress sees as a means of implementing desirablepolicies.

For either policy formulation or policy imple-mentation, the value of demonstrations shouldbe based upon their ability to provide more credi-ble information to either policy makers or poten-

tial users of innovations than would be availablefrom other R&D methods. A project’s scale andthe realistic environment within which it is triedare the major contributors to this credibility.

Po l i cy -fo rm u la t in g demonstrations shouldprovide information to Federal policy makersabout:

q

q

q

q

Technological and administrative feasibilityof instituting the policy or adopting the in-novation on a wider scale.Expected economic, environmental, andsocial impactsof the policy or innovation.

Relative merits of alternative policies or in-novations intended to meet the sameneeds.Unanticipated consequences of adopting aparticular policy or innovation.

In seeking to provide this information, demon-strations may help to clarify the nature of thepolicy problem itself. For example, the NewJersey income maintenance experiment high-lighted the sensitivity of program costs to the fre-quency and manner of assessment of a family’sincome, an issue that was given little attention in

early legislative planning. The experiment alsoprovided detailed information on the variability of the incomes of the working poor.

4

Negative information, information concerningthe shortcomings of an innovation, may also bean important output of policy-formulating dem-onstrations. Because these demonstrations takeplace before a commitment to a course of policy,negative results can be used to support a decisionto abandon consideration of the policy. ’

P o l i c y - i m p l e m e n t i n g demonstrationsshould provide many of the same types of in-

4Pechman and Timpane, op. cit, pp. 207-214.

50E0’s experiment with Educational Performance Con-

tracting is a good example. See Edward M. Gramlich andPatricia P. Koshel, Educational Performance Contracting,An Evaluation oj an Experiment, The Brookings Institution,Washington, D. C., 1975.



formation to potential users of information. How-ever, the emphasis is typically placed upon

q Costs of adopting and using the innovation.q Reliability of that innovation in use.q Demand for the innovation.

Feasibility of implementing the innovationat the adopter’s site.

Inherent in the definition of purposes of

demonstrations is a specification of audiences.For policy-formulating demonstrations, the au-dience is the Federal Government and the consti-tuencies that seek to influence Federal policy; forpolicy-implementing demonstrations, potentialusers of the innovation. Two classes of users canbe identified: “final” users—private consumers,local governmental units, or individual firms; and“intermediary” users—suppliers of equipment,for example. Planning for a demonstrationshould involve clear identification of the audienceand a determination of their information needs.

These two types of demonstrations should bedistinguished from two other types of activitiesthat share some characteristics of demonstrationprojects. We do not consider large-scale engi-neering tests of hardware components to bedemonstrations. Such projects, which typicallytest scaled-up components of an innovationprocess or product, neither test a full “system”nor take place in a realistic operating environ-ment. For example, aircraft engines may be runin large test cells or even mounted on test aircraftbefore being integrated into prototype systems.Such activities are potentially quite important to

the development of new products but should beconsidered part of development rather than aninstrument for promoting the transition fromdevelopment to use.

The second type of activity that we exclude isthe projects that are created as a result of Government subsidies for local development. Ina number of social programs, for example, fundsare provided to local political jurisdictions todevelop innovative activities. The Law Enforce-ment Assistance Administration (LEAA) has pro-vided such funds to police departments; Title IVCof the Elementary and Secondary Education Act

supports innovative projects that relate to local orState school needs. Projects such as these aregenerally not considered as means of promotingthe use of a particular innovation. Rather, theyplace the responsibility for innovating on the local

jurisdiction itself.

Demonstrations as Instrumentsi n the Political Process

The purposes of demonstrations can also beviewed in terms of their usefulness within ourpolitical system. Demonstration projects are ameans of showing political initiative at relativelylow budgetary cost. They are an attractive com-

promise between the advocates of sweepingpolicy or program change and the supporters of the status quo. They can help promoters of apolicy to “get a foot in the door” to build aconstituency. By demonstrating a concern for anissue or a particular group, demonstrations canbecome significant symbolic acts. They can assistFederal R&D managers to gain a political constit-uency. Demonstrations permit an R&D programmanager to maintain some momentum in hisprogram if the political support for expansiondoes not otherwise exist. Finally, demonstrationprojects can serve the needs of traditional pork

barrel politics.These are important functions within our

governmental system because there are manyissues on which compromises are needed. In theabsence of such compromises, we would oftenreach an impass between advocates and op-ponents of a policy. A demonstration projectmay permit concrete experience with a policy to

be obtained to settle the debate. It may provide avehicle through which a particular interest groupcan establish a dialogue with elements of theGovernment. A demonstration can provide an

important governmental gesture of concern forindividuals facing difficult problems when theGovernment does not know how to solve thoseproblems.

The fact that a demonstration is a compromiseposes substantial problems for a project withR&D objectives. The parties to the compromisehave a variety of objectives and frequently thecompromise itself will not specify which of theseobjectives should predominate. The F O1 1O W

Through Program illustrates this problem well.Follow Through was proposed as a program toprovide continuing compensatory educational

services to schools with disadvantaged childrencoming from Headstart preschool programs. Ad-vocates of the program argued that such con-tinued services would prevent the students fromlosing the gains they had achieved in their

21



Headstart programs. Opponents argued thatthere were insufficient funds to support such aprogram and in any case there was no evidencethat it would be effective. The compromise was a“planned variations” program to test the effec-tiveness of different program designs.

b

The advocates of an operating programsought quick implementation and, as a conse-quence, most of the programs were incompletely

designed. In addition, the experimental designnecessary to detect differences among programswas deemed inappropriate by the operating of-ficials. The absence of clearly defined goals ledthe evaluators to collect data having onlymarginal usefulness. While there have beensome useful outcomes, it seems unlikely thatthey are commensurate with the costs that wereincurred. An important cause of this low yieldwas the conflicting goals of the parties to thecompromise and the failure to resolve these con-flicts in the design of the program.

The political uses of demonstrations will leadto projects with multiple and often conflicting ob-

jectives. Those groups that are involved will fre-quently possess “hidden agendas” that may in-terfere with the R&D objectives of the project.The absence of a clear consensus about goalsmay weaken project management and lead to in-conclusive evaluations. A major task of both theexecutive branch and Congress should be torecognize both the political and R&D functions of demonstration projects and to design projects ina manner that minimizes the adverse interactionsof the goals.

In the next section we address the factors thataffect the success of demonstration projects aspart of the R&D tradition rather than as in-struments in the political process. Our concludingsection will consider how the political and R&Duses of demonstrations can usefully be meldedtogether.

General Factors Affecting theSuccess of Demonstrations

Many factors affect the success of a particular

demonstration project: the quality of the project

‘Alice M. Rivlin and P. Michael Timpane (eds. ), PlannedVariation in Education: Should We Give Up Or Try

Harder?, The Brookings Institution, Washington, D. C.,1975, chapter 1.

22

staff, the appropriateness of its goals, the ade-quacy of the funding, and the nature of the politi-cal and physical environment. Moreover, thesefactors will affect the outcome of demonstrationprojects in every policy area. Our interest,however, is not to investigate how an individualproject can be most successfully executed once adecision has been made to initiate it. Rather, weseek to understand the types of situations in

which well-executed demonstrations are mostlikely to be successful in reaching their objectives.

Our review of the literature leads us to suggestthat two factors are likely to be of prime im-portance in determining the success of a demon-stration project: first, the nature of the technologythat is being demonstrated; second, the nature of the institutional environment within which ademonstration must be carried out.

The Technology

As we have noted, we use the term

“technology” to denote the inputs, outputs, and theory relating them that are associated with theinnovation under consideration,

7 The technology

of the automobile is thus the inputs of personaltime and invested capital, the output of transpor-tation services, and the theories that relate the in-puts to outputs that are embodied in the auto-mobile itself. Two points deserve note. First, thedefinition of a technology is a matter of conven-ience. We can discuss the technology of an auto-mobile or we can enlarge the concept to discussthe technology of an urban transportationsystem. The choice of the scope of the technol-

ogy depends upon the purposes for which theconcept is used. In this report, the scope is asso-ciated with the innovation whose demonstrationis being contemplated. If the innovation is anelectric automobile, the technology would be theautomobile itself. If the innovation were a vancarpool system, however, the technology wouldextend beyond the vehicle to encompass the sys-tem of roads, the manner in which carpools areformed, the location of work and living sites as in-puts, and transportation services as outputs. Thetheories that relate these inputs are no longersolely embodied in physical entities but extend to

institutions and behavioral responses.

Second, the theories and inputs associated

‘Our use of the term “technology” is similar to that of economists who are concerned with production functions.



with differing technologies vary substantially in

quality. These variations are important to under-

standing the impact that the quality of a technol-

ogy has on the usefulness of demonstration proj-

ects as policy tools. Technologies differ in the

degree to which their inputs involve important

uncontrollable variability. Thus, the inputs of an

education technology such as a reading programinclude teachers and students whose capabilities

vary substantially. The technology associatedwith a large coal powerplant involves inputs of coal and water whose characteristics are predict-able and have little variance for a single plant.Technologies also differ in the degree to whichthe inputs interact and change in the course of the transportation to outputs. In education, thequalities of teachers can be modified as they work to change the skills and other attributes of thestudents. For example, teachers can becomemore or less directive in their instruction orchange their allocations of time among students.In the case of the powerplant, the coal does not

change its attributes in unpredictable ways as itenters the process of producing power. Finally,technologies differ in the nature of theirassociated theories. For a technology such aspower production, the theories that underlie thetransformation of the inputs to outputs and resid-uals are quite deterministic and provide high con-fidence that certain inputs will lead to predictableoutputs. In the case of education, the theoriesthat relate inputs to outputs either do not exist orcan predict outputs onlyranges.

For some technologies,proved through research,testing. While we may not

within very broad

theory can be im-development, andbe sure that a new

design for an internal combustion engine willhave high fuel efficiency and useful power out-put, we are confident that the developing andtesting will allow us to perfect the technology’stheory in the sense that we have used the term.In other cases, we are far less confident that we

can produce such theory through R&D proces-ses. For example, incentive systems for en-couraging the insulation of houses can be de-signed and tested but will probably never reach a

point where we may confidently predict that aparticular individual, faced with the incentives,will make a particular decision. Even without thecapability to make such exact predictions, how-ever, decisionmakers can reduce the uncertaintyconcerning the behaviors of groups of individ-

uals. Thus, while the theory may not be as deter-ministic as that associated with internal combus-tion engines, it is useful to a policy maker and it

can be improved through R&D. In other areas,however, theory may be at such a primitive stage

that it cannot be readily perfected through R&D.This may well be the case with police patrol pro-cedures, where the relationship between the fre-quency of patrols, the type of patrol, and the

type of site on the one hand and the occurrenceof crimes on the other seems to be beyond ourpresent capacities to discover.

8

These qualities of a technology can be conven-iently characterized in essentially two ways. Thefirst of these is the degree to which a technologyis reproducible from site to site. With what con-fidence can the policy maker or the industrial firmsay that the use of the innovation will result in thetransformation of inputs into desired outputs?This will depend upon the variability of the in-puts, their tendency to interact or modify them-

selves in the course of the transformation, andthe quality of the theory that exists. In general.technologies that are substantially embodied inhardware would be quite reproducible, whilethose that involve processes of human interactionwould be relatively nonreproducible. Sometechnologies occupy a middle ground. For exam-ple, transportation systems have some repro-ducibility but will vary substantially by site.

The second quality of a technology is thedegree to which research, development, and testing can reduce uncertainty about the likelyoutputs to be associated with a particular set of inputs. Designing and testing prototype modelsof automobile engines can yield high-confidenceestimates of the performance of the engine. Testsof curriculum materials, on the other hand, yieldonly low-confidence estimates concerning theperformance of those materials because of theweak theory upon which they are based and thevariability of the inputs to the education process.If the uncertainties surrounding the relationshipbetween the inputs and outputs of a technologyhave been substantially reduced prior to its in-corporation as a demonstration, we will refer tothe technology as being well in hand.

“Jan Chaiken,Police Activities,1977.

What is Known About Deterrent Effects of The RAND Corporation, P-5735-1, July

23



The Institutional Environment

A demonstration will result in a number of out-comes. Some of these will simply be what wehave referred to as outputs of the technologyassociated with an innovation; others will beassociated with the impacts that the use of the in-novation has upon the environment, the behav-iors of groups of citizens, or the employment in acommunity. A major reason for conducting a

demonstration project is to improve knowledgeconcerning the effect of using the innovation onthese outcomes.

The context in which the demonstration is car-ried out determines the scope of relevant out-comes. The needs and interests of the target au-diences determine the specific outcomes meas-ured. If, for example, the demonstration is in-tended to influence local police departments toadopt and use a new piece of equipment, theoutcomes examined are those that are importantto officials in these departments. The choice of asite or sites should include operational en-vironments that are similar to the sites of mostpotential adopters. If the demonstration isintended to aid in the congressional debate con-cerning a new welfare system, it should bedesigned to address questions that Congress andits constituents consider important.

Thus, the relevant scope of outcomes for ademonstration is determined by the institutionalenvironment in which it will be implemented.This environment encompasses the collection of organizations and the linkages among thoseorganizations that are involved in selecting and

implementing the innovation being demon-strated. The developers of the innovation,ultimate users, regulatory bodies, markets, andthe Federal agencies that fund the demonstrationare components of the institutional environment.The concept of a “technology delivery system”suggested by Wenk and that of a “selection en-vironment” used by Nelson and Winter areroughly equivalent concepts.

9

‘See Edward Wenk, Jr., “The Social Management of Technology,” in John E. Mock (cd.), Science for Society,proceedings of the National Science Conference held atAtlanta, Ga., Oct. 12-14, 1970; The Committee on Public

Engineering Policy, National Academy of Engineering,Priorities for Research Applicable to National Needs, reportto the National Science Foundation, Washington, D. C.,1973; Arthur A. Ezra, “Technology Utilization: Incentivesand Solar Energy, ” Science, Vol. 187, Feb. 28, 1975, pp.707-713; Richard T. Nelson and Sidney G. Winter, “InSearch of Useful Theory of Innovation, ” Research Policy,Vol. 6, 1977, pp. 36-76.

24

The institutional environment has importanceto Federal officials for two reasons. If the FederalGovernment were contemplating using a demon-stration to help determine the worth of a policy orthe desirability of promoting the use of a par-ticular innovation, the relevant scope of out-comes as set by the institutional environmentwould determine the inputs and outputs thatmust be measured to make credible inputs to the

policy process. If, on the other hand, the Gov-ernment were considering a demonstration topromote the use of an invention, the institutionalenvironment would be the medium in which theuse would take place; the important actors andcommunications links in this environment wouldbe the critical determinants of the way in whichthe innovation is adopted and used.

The basic proposition about institutional envir-onment that emerges from the literature is thatinnovation is strongly affected by the nature andthe workings of the institutions in a policy sector.There are some familiar examples that help toclarify this point. In the area of national security,it is widely believed that the development, pro-curement, and development of new weaponssystems is facilitated because the Federal Gov-ernment is the sponsor of the relevant R&D, thebuyer of the R&D product (usually a weaponssystem), and the user of that product. Eventhough the institutional environment in the na-tional security area is complex, the dominant po-sition of the Federal Government simplifies theprocess of innovation relative to other areas in-volving intergovernmental or public/private rela-

tionships. 10A second example is that of the Bell System.

Within the framework of the American Tele-phone and Telegraph Company are the BellLaboratories, the R&D organization; WesternElectric, the production organization; and theLong Lines Division and the operating telephonecompanies, the organizations providing service.Thus, institutions within the AT&T organizationare specialized in their functions, yet they arehighly integrated concerning the manner inwhich each plans its operations and the ways inwhich each transfers the information or products

to the others. It is argued by the Bell System, andbelieved by many, that this complex group of in-

‘“Herbert F. York and G. Allen Greb, “Military Researchand Development: A Postwar History, ” Bulletin Of theAtomic Scientists, Vol. 33, No. 1, January 1977, pp.13-26.



stitutions, tied together as they are, facilitates thegeneration and diffusion of new communicationstechnologies. 11

A third example is found in the agriculturalsector. The U.S. Department of Agriculture(USDA) has developed an institutional systemover an extended period of time that is thought tobe quite effective in ensuring the flow of innova-

tion in agriculture. The USDA directly supports anational laboratory, four regional laboratories,and satellite research laboratories through theAgriculture Research Service. Through the Co-operative State Research Service, it financesresearch jointly with the States in colleges of agriculture and State experiment stations acrossthe country. The Agricultural Extension Service,funded jointly by Federal, State, and countyfunds, supports the agricultural extension agentswho are responsible for the dissemination of R&D results to prospective users and the needsof users to the research system. Thus, a strong

system need not be centralized. 12

These three examples illustrate relatively well-developed institutional environments. In con-trast, consider the elementary and secondary ed-ucation system. The National Institute of Educa-tion, the National Science Foundation, and theOffice of Education provide funds at the nationallevel. Fifty State departments of educationassume a variety of stances toward promotingthe use of new innovations in local schooldistricts. Most education costs are for personnel.Training professionals in the school system is

mainly the province of a large number of teacherpreparation colleges that are not widely knownfor their interest in or conduct of R&D activities.Decisions concerning the adoption of new curric-ulum materials vary widely among States and lo-calities. A beleaguered and embryonic network of R&D centers established by the Office of Edu-cation over a decade ago has failed to achievenotable successes. A persistent hostility existsbetween the practitioner community and the re-search community, which makes it difficult forcommunications to occur between them. Thetextbook manufacturers place little reliance on

“See H.W. Bode, Synergy: Technical Integration andTechnological Innovation in the Bell System, BellLaboratories, Murray Hill, N. J., 1971.

‘2A. Hunter Dupree, Science in the Federal Government,Harvard University Press, Cambridge, Mass., 1957.

systematic materials development and testing.The components of the institutional environmentare unevenly developed, poorly linked, and fre-quently in disagreement over the process of change in the schools. 13

An important contributor to the nature of theinstitutional environment is the degree of consen-sus that exists concerning the appropriate Federal

role in a particular policy area. In chapter II, webriefly sketched the evolution of the Federal rolein a number of policy areas. In areas such as na-tional security or agriculture, a strong Federalrole is viewed as legitimate and is generally ac-cepted and understood by the elements of the in-stitutional environment. Similarly, there is fairlygeneral agreement concerning the limits of theFederal role in promoting the use of basic re-search. However, in many areas of domestic pol-icy and in energy policy there is considerabledebate over the appropriate Federal role.

The strength of the rationale for Federal in-volvement in these “mixed responsibility” sectorsvaries substantially; more importantly, it is sub-

ject to a great deal of controversy. The involve-ment of the Federal Government in these sectorshas come about because important political con-stituencies have felt that the performance of these sectors was inadequate for the needs of thecountry. For example, the Federal Governmententered the health care financing area becausethe health needs of the elderly and the poor werenot viewed as being adequately addressed by theexisting Federal policy in these areas. State and

local officials, whose operations are being cur-tailed by Federal activities, see Federal officials asinterfering in their legitimate policy responsibil-ities. When demonstrations become a tool to fur-ther controversial Federal policies, their legiti-macy is questioned.

This problem is exacerbated by increasingpressures from Congress and the public to takeresults of publicly supported research “off theshelf” and put them into practice. While we mayquestion whether sufficient R&D exists “on theshelf” to justify this concern, the pressure has ledFederal R&D agencies to promote the commer-

13For a description of the education R&D system and the

institutional environment which surrounds it, see The Statusof Education Research and Development in the UnitedStates: 1976 Databook, The National Institute of Education,Washington, DC., 1976.

25



cialization and use of R&D activities and to pro-vide demonstration projects with a promient rolein this effort. The Federal Government mustchoose specific technologies for demonstration.These demonstrations will be chosen to advanceFederal policy goals, ghals that maybe in conflictwith those of State and local authorities or theprivate sector. In the situation where such conflictexists, the probability of success in promoting theuse of a new innovation or technology will below.

Another general factor that shapes the institu-tional environment is the extent to which an R&Dtradition is a part of the operations of a policy sec-tor. For example, in the military sector, in civiliantechnology, in agriculture, or in parts of thehealth sector, there is a long tradition of usingR&D results to improve performance. In con-trast, R&D in support of the improvement of thecriminal justice or educational systems is rela-tively recent. In the absence of a tradition of using

R&D and of training emphasizing the utility of R&D to important actors within a sector, it isunlikely that R&D and demonstration will play amajor role in the near future.

J u d g i n g t h e D e v e l o p m e n t o f I n s t i t u -

t i o n a l E n v i r o n m e n t s . -The examples and dis-cussion suggest criteria by which to judge the de-gree of development of these institutional envi-ronments. A well-developed environment mighthave the following attributes:

A set of institutions necessary for the entireprocess of research, development, com-

mercialization, and application.Established, agreed-upon roles and respon-sibilities of these institutions (including theFederal role).Communication paths among these institu-tions along which information critical to theR&D planning, utilization, commercializa-tion, and innovation process can flow.Communications among institutions thatare sufficiently frequent and strong to facili-tate the movement of an innovation intoutilization.Well-developed criteria at each stage of theinnovation process for evaluating the desir-

ability of the movement from one stage to

the next.

These criteria for a developed institutional en-vironment emphasize qualities that are likely to

26

lead to routine and continuing innovative activi-ties. Equally important, sectors that have thedegree of development implied by these criteriaare likely to have developed a consensus aboutthe relevant scope of outcomes by which a pro-spective innovation should be evaluated; con-versely, in less developed institutional environ-ments, there is likely to be considerable disagree-ment over the outcomes of greatest importance.

T h e I n t e r a c t i o n Be t w e e n T e c h n o l og y a n d

t h e I n s t i t u t i o n a l E n v i r o n m e n t . - An appreci-

ation of these factors and the manner in whichthey affect the success of demonstrations can begained by an examination of several specificexamples.

Central power generation can be contrastedwith solar heating and cooling, a decentralizedform of “power” generation. In both cases, thetechnologies seem reproducible. We know agreat deal about their performance and are confi-

dent that we can learn more. Some aspects of these technologies are currently well in handwhile others require further development. How-ever, these two technologies are used in twoquite different institutional environments.

Central power station technologies are devel-oped and used in an environment where the sup-pliers are known; the regulatory apparatus is inplace; and the market is orderly. ” In contrast,solar heating and cooling is being developed andused in a less developed environment; its appli-cation is controlled by hundreds of buildingcodes. The industry that supplies the technology

is new and unstructured. Financial and tax poli-cies for installations are not yet routinized.Regular sources of maintenance are not yet ap-plicable. Suitable criteria for making decisionsabout whether or not to invest in the technologyare not yet widely agreed upon. Thus, while weare confident that we know or can know a greatdeal about the performance of the technology,the institutional environment for facilitating the

“This institutional environment is less well developednow than several years ago because of the advent of both

energy and environmental concerns. These concerns haveresulted in new and as yet incompletely specified roles forthe Federal Government. The regulatory structure govern-ing powerplants has been elaborated and new groups have

joined in powerplant design decisions Still. according to thecriteria presented in this section, central power generationoccurs in a developed institutional environment.



application of technology is not well developed. 15

Day care in industrial and community settingsprovides an example of how institutional envi-ronments of differing levels of development canshape the relevant scope of outcomes for dem-onstration projects. There is little agreementamong child care specialists concerning the levelsof inputs. competencies of care givers, and types

of activities that will produce various outcomes inchildren in day care settings. The standards thatshould govern day care financed by the FederalGovernment have beendebate for a number of considerable disagreementthe appropriate measurescare.

subject to vigorousyears. There is alsoover what constitutesof outcome for day

In the public sector, day care is typically pro-vided by a variety of institutions for the benefit of the children, as well as the parents. Accreditationfor day care centers varies substantially among

jurisdictions. The standards for training of caregivers are highly variable. Nationally, the level of subsidization for day care and the restrictionsplaced upon the operation of programs thatreceive subsidies are still matters of debate. Theinstitutional environment is poorly developed.The relevant scope of outcomes is broad. Thetechnology has low reproducibility.————.—

“For an interesting discussion of institutional en-vironments in general and that surrounding solar heating inparticular, see Arthur A. Ezra, “Technology Utilization: in-

centives and Solar Energy, ” science, VOI. 187. Feb. 28?1975, pp. 707-713.

‘6A brief history of the controversy surrounding Federal

support for day care is contained in Gilbert Y. Steiner, TheChildren Cause, The Brookings Institution, Washington,D.C. , 1976, pp. 14-35.

Several years ago, as a result of the generalconcern over extending day care to more parentsand the pressures from feminist and othergroups, a number of industrial firms experi-mented with day care. These firms provided daycare to small groups of their own employees on atrial basis .1’ The firms quickly decided that therelevant outcome of the day care they supportedwould be reduced absenteeism and lower turn-

over among their personnel. These day carecenters met existing local standards for good daycare but did not consider provision of educationto children as a relevant goal. The demonstra-tions suggested that relatively few parents foundday care provided at the work site as desirable orno improvement in turnover or absenteeism. Theidea seems to be largely abandoned. 18

In this case, within an industrial setting therewas agreement over the outcomes that wereimportant. Responsibility for the demonstrationwas clearly assigned. And there was general

agreement over the meaning of the outcomes of the demonstration among the potential providersof industrial day care. The institutional environ-ment was well developed. Moreover, thetechnology associated with these outcomes wasreproducible. While there may be no reproduci-ble means of providing day care that maximizescognitive growth in children, there is no problemin creating day care centers that meet specified(input) standards.

“Using our terminology, they conducted policy-

formulating demonstrations.‘*’’Minding the Kid, Frustrating for Companies,” New

York Times, Sept. 11, 1977, p.F9.

27



Chapter IV

FACTORS AFFECTING THE

SUCCESS OF DEMONSTRATIONS

.



Chapter IV

FACTORS AFFECTING THE

SUCCESS OF DEMONSTRATIONS

The previous section developed a conceptual

framework with which to discuss factors that af-

fect the success of both policy-formulating and

policy-implementing demonstrations. In this

chapter we advance several propositions con-

cerning the determinants of success of demon-

strations and assess the evidence contained in theliterature concerning these propositions. As inthe last chapter, our emphasis is on the use of demonstrations as a logical step in an R&D proc-ess. Because the experience with policy-imple-menting demonstrations is rich, we begin our

discussion there.

Factors Affecting the Success of

Policy-implementing Demonstrations

Given the manner in which we have definedpolicy-implementing demonstrations, the criteriaby which to judge their success is straightforward:did the demonstration result in the diffusion anduse of the demonstrated innovation in othersites? An innovation may be an industrial proc-

ess, a new product, or a program design. In thecourse of the innovation’s diffusion to other sites,it may be expected to change somewhat in re-sponse to the needs and interests of that site. It isa matter of judgment as to when the innovationhas sufficiently changed that it should no longerbe viewed as the same innovation.

The analysis above together with the literaturethat we have reviewed suggest that a number of propositions concerning the use of demonstra-tions that should guide policy makers at theFederal level:

q Implementation demonstrations havingtechnologies with low reproducibility areunlikely to lead to successful diffusion.

q If a technology is well in hand, a policy-implementing demonstration has a higher

q

q

q

q

probability of success than if the technologyis not well in hand.Implementation demonstrations that areconducted in a well-developed institutionalenvironment where there is a clearly ac-cepted Federal role are more likely t o

achieve diffusion success.An innovation requiring cooperative actionamong elements of an institutional environ-ment will be less likely to diffuse than onewith an application within the scope of asingle institution.

User need, as exemplified by non-Federalinitiative and cost/risk sharing, is an impor-tant factor in the success of policy-implementing demonstrations.Lack of time constraints and operationalflexibility are important to the success of policy-implementing demonstrations.

Each of these propositions is discussed in detailin the following,

Technologies With Low Reproducibility

If the technology is perceived as being

nonreproducible, a demonstration incorporatingthat technology will not be likely to lead to suc-cessful widespread diffusion of similarly perform-ing processes or projects because potential userswill not be able to reproduce the demonstratedprocess. Projects incorporating nonreproducibletechnologies may provide ideas to other potentialusers; they may suggest pitfalls to avoid in 1plementing a project; they may even provitequipment and materials that can be used b

others. But they seem unlikely to lead to similaroperations in other locations.

Available evidence is consistent with this prop-osition. Education, for example, possesses tech-nology with low reproducibility. In their study of Federal programs promoting educationalchange, Berman and McLaughlin found no sig-nificant diffusion of the innovative projects sup-

31



ported under the programs. Their study showedthat (1) local staff preferred to develop their ownmaterial, even when prepared material was avail-able; (2) successful projects that differed substan-tially from usual practice were viewed as nonrep-licable by staff from other locations; (3) ratherthan search for alternatives, local staff preferredto solve problems by using information alreadyknown to local personnel; and (4) local personnel

tended to view their own site as unique. As aconsequence of these characteristics, the ChangeAgent study hypothesized that significant innova-tion could be implemented only through a pro-cess of “mutual adaptation. ” It reported:

The initial design of an innovative projectmust be adapted to the particular organizationalsetting of the school, classroom, or other institu-tional hosts, and, at the same time, the organiza-tion and its members must adapt to the demandsof the project. Many educational innovationsmay fail to have desirable effects because theproject is not adapted to the institutional setting

or vice versa during the implementation stage.2

This model of change requires unique interac-tion of the innovation with staff at each potentialadoption site. Given such a requirement, itseems unlikely that exemplary projects atselected demonstration sites will be replicatedelsewhere.

If diffusion is no longer a relevant successcriterion for such demonstrations because aunique implementation of a technology with lowreproducibility is required at each site, Federalpolicy makers should consider redefining the

goals of a program supporting such projects.Perhaps these programs should consciously seek site-specific development for use mainly at theproject site. In such cases, decisions to initiate aproject should be evaluated on the basis of thecosts and benefits associated with an individualadoption. The use of the term “demonstrationproject” would appear misleading for this type of project. ’ We have referred to this type of pro-

‘Paul Berman and Milbrey Wallin McLaughlin, FederalPrograms Supporting Educational Change, The Findings inReuiew: Vol. IV, The RAND Corporation, R-1589/4-HEW, April 1975, p. 1, hereafter referred to as the Change

Agent study.‘Ibid, p. 6.‘It should be noted, however, that experiences with this

type of program are not encouraging. The Change Agentstudy found that only 5 percent of a sample of such localdevelopment projects resulted in permanent incorporationof the project.

32

gram as subsidized local development.

In a number of areas, innovative technologiesof substantial reproducibility are used in a larger“system, ” where adaptation of the innovationmay occur. Examples include mass transit, lawenforcement hardware, and mining. Success of demonstrations where such adaptation can takeplace depends in part upon the perceivedsimilarity of the environment in which the dem-onstration is conducted to that of the potentialadapter.

Two examples from mass transit illustrate thispoint. The Shirley Highway Express Bus Lanesin the Washington, D .C., area used conventionalbuses and highway and street lanes exclusivelydesignated for buses. The project measured ac-tual ridership and thus avoided the theoreticaldebates in the mass transportation communityabout the relative importance to potential com-muters of trip time, waiting time, transfers, orcomfort. The success of the system in attracting

commuters and in showing that the demonstra-tion results are relevant to other metropolitanareas with radial commuting corridors has led todiffusion of the concept to other cities. ”

In contrast, the Personal Rapid Transit (PRT)demonstration which, as we shall note, experi-enced many technical difficulties, was demon-strated in such a specialized situation that itseems unlikely that it would have been per-suasive even in the absence of technical prob-lems. This system, involving unattended smallvehicles operating on a fixed track, was used to

connect parts of The University of West Virginiain Morgantown. It seems unlikely that the condi-tions prevailing at this university campus wouldbe sufficiently similar to other contexts for this in-novation to be persuasively demonstrated to cityplanners. The demonstration did not providecredible predictions of the relevant outcomes.

The existence of replicated projects may alsobe important to the diffusion of innovations usingtechnologies with low reproducibility. Yin

discusses this issue with respect to innovation inlocal law enforcement. In considering the Kansas

City Preventive Patrol Experiment as an exam-

‘Cheryl D. Hayes, “Toward a Conceptualization of theFunctions of Demonstrations, ” in Thomas K. Glennan, Jr.,(cd.), Studies in the Managment of Social Research andDevelopment, The National Academy of Sciences,Washington, D. C., forthcoming.



pie, he cautions that “findings . . . are n o tfacts.’” Unless successful results are experienced

at multiple sites, they may not be credible to

potential adopters. Diffusion of intermediate

strength technology innovations in the absence of such replications is not likely.

Technology That Is Well in Hand

The degree to which a technology is in hand

will affect the success of policy-implementing

demonstrations. In projects using technologiesthat are not well in hand, managers must concen-trate on developing and improving the technol-ogy. The rhythm of the project will reflect thetechnical needs of development activities. On theother hand, the concerns of the adopters will fre-quently focus on questions such as the reliabilityof the process, the administrative feasibility of im-plementing the process with the m anpow er

available at local sites, and the operating costs of

processes under routine conditions. If the tech-nology is not well in hand, evidence concerningthese latter issues will be more speculative andless clear. The demonstration may well be lessthan effective in persuading adopters to make adecision to implement a new innovation.

The Federal Demonstrations study providespersuasive evidence that the lack of technologywell in hand adversely affects diffusion. Thisstudy found little or no diffusion in any of thedemonstrations for which there was high techni-cal uncertainty. Uncertainty was considered high

“when the technology has not previously beenprototype or field tested; or when basic prob-lems with the technology are known to exist, andtechniques for dealing with them are not clear atthe time of the demonstrations. 6 With one ex-ception, all of the examined demonstrations of technologies with medium or low uncertaintyachieved at least some diffusion success unlessthere were institutional problems.

7A possible ex-

5Robert K. Yin, R&D Utilization by Local Services: Prob-

lems and Proposals ~or Further Research, The RAND Cor-poration, R-2020 -DOJ, December 1976, p. vii.

‘Federal Demonstrations. p. 31.‘Medium technological uncertainty was considered to ex-

ist when technology had been tested at a lower scale butuncertainties remained about performance at near-commercial size or where uncertainties remained concern-ing performance of a new configuration of components.Low technological uncertainty characterized projects usingexisting components in configurations similar to previoususe. See Federal Demonstrations, p. 31.

planation for this finding is that the lack of uncer-tainty enables a realistic appraisal of the pros-pects for successful implementation. As a conse-quence, technologies that do not possess someadvantage over the status quo are not demon-strated.

The exception to this pattern—a demonstra-tion of a seawater desalination process in

Freeport, Tex.—is instructive. The demonstra-tion involved testing a large-scale version of adesalination process previously tested only in apilot plant. The project was classified as havingmedium preproject technological uncertainty.The process required significant developmentwork during the course of the demonstration,which resulted in important improvements in per-formance but also in interruptions in plant opera-tions. At the end of the demonstration, thetechnology could be considered well in hand.However, potential adopters of desalinationplants mistakenly perceived these instructions at

the Freeport plant as an indication that thedesalination process was unreliable. These po-tential adopters failed to understand that the in-terruptions were necessary for technologicaldevelopment work to occur.

8

The PRT demonstration illustrates how the im-plementation of an undeveloped technology af-fects the operational results and the innovation’sdiffusion. The major leap in technology incorpor-ated in the project led to rapid escalations in cost,ultimately to a cost of over $60 million. To keepcosts down, the system was modified and cur-

tailed, and the resulting system provided littleuseful data on the possible spacing of vehicles,the relative attractiveness of PRT to privatetransportation, or other important measures.

9

The Freeport desalination and the PRT casesillustrate the importance of considering an alter-native to a demonstration when a technologycontains some elements of uncertainty. Whenhigh technological uncertainty exists, a full-scaletest at a test-bed facility should be considered.This test of the technology at the development-oriented facility would ease the development of

engineering solutions to technical problems byremoving real-world operating constraints.

“William F. Hederman, “Saline Water Conversion Plant,”in Federal Demonstrations: Case Studies, pp. G 1-G46.

‘Federal Demonstrations, pp. 137-138.

33



Demonstrations in Well-DevelopedInstitutional Environments

Areas with well-developed institutional en-vironments pose relatively minor problems forFederal policies governing the use of demonstra-tions. Their very development means that the in-novation system is working well and probably ona continuing basis. Mutual agreement exists con-cerning when demonstrations are appropriate.Sponsors and performers of demonstrations willknow which actors to involve and those actorswill understand the needs and patterns of successwithin the policy area.

Poorly developed institutional environmentsseem more likely to give rise to the type of goaland value conflicts among the actors that havebeen mentioned frequently in earlier sections of this report. The absence of close ties among theactors, lack of important institutional entities, orthe lack of consensus over the appropriate meansof bringing about and diffusing innovations all

lead to the possibility of such disagreements overgoals and values.

The empirical literature tends to support thisproposition. The Federal Demonstrations studyfound that all demonstrations that examinedwhat had taken place in well-developed institu-tional environments had achieved at least somediffusion success.

Moreover, the evidence indicates that the par-ticipation of the components of the institutionalenvironment is also important to diffusion suc-cess. The Federal Demonstration study found

that only one demonstration project thatexcluded some active components of the institu-tional environment was diffused.

The exception to the observed pattern illumi-nates this proposition. A demonstration of mech-anized refuse collection aimed at reducing laborcosts through smaller collection crews was a strik-ing application success. The service, provision of a mechanism to improve truck refuse collection,was cost-effective and received strong public ap-proval (exceeding 90 percent). However, themechanism was designed by city personnel, and

no garbage-truck outfitting firm could be found to

‘“Federal Demonstrations, pp. 51-53. The one exceptionto this finding appears to have been examined too soon afterthe demonstration for any meaningful assessment of diffu-sion to have been made.

market the new truck. Consequently, the diffu-sion observed was piecemeal and not self-sustaining, a disappointing result for an innova-tion based upon a reproducible technology thatwas successful in its initial application.11

The innovative behavior of local governmentssuggests the existence of goals other than thosethat would normally be emphasized in the plan-ning and execution of a demonstration project.For example, in an investigation of technologicalinnovation in State and local services, Yin andhis colleagues found that evidence on the trialand adoption or rejection of new technologiessupported two differing interpretations of howlocal agencies operate, One model, termed aproblem-solving model, involves local identifica-tion of problems and a search for means to solvethe problems. The second emphasizes conditionsof bureaucratic self-interest that govern the in-novations to be incorporated. Yin et al., in a

study of innovation, suggest that both models of innovative behavior may operate at the State and

local level.

The important point to note in these findings isthat demonstrating a new technology is not an ef-fective way to influence local problem solvingand the bureaucratic process. The character of local innovation seems to spring from the charac-teristics of each locality and the needs that theyperceive, rather than from the opportunities pre-sented by a demonstrattion project. This discrep-ancy may be part of the reason that Yin’s analysisfound no statistically significant relationship be-tween a variety of Federal policies and successful

incorporation in local innovation. *3

Requirements for Cooperative Efforts

An innovation requiring cooperative actionamong elements of an institutional environmentwill be less likely to diffuse than one where its ap-plication is within the scope of a single institution.Innovations can be significantly mismatched tothe institutional environment to which they areintended to apply. In such instance, even if theenvironment is developed and the technology isfree of uncertainty, diffusion may not take place.

“Federal Demonstrations, p. 96.‘zRobert K. Yin, Karen A. Heald and Mary E. Vogel,

Tinkering with the System, Chapter 5, Lexington Books,Lexington, Mass., 1977.

‘31 bid., Chapter 6.

34



The regional emergency medical service is acase in point. The capability of specialized sys-tems of emergency medical care to save lives hasbeen demonstrated. Necessary equipment,personnel, and organizational procedures areknown and local governments and communitiesseem to appreciate the need for such services.Nonetheless, the development of these systemshas been sporadic.

A major reason for this sporadic developmentis that the appropriate scale for such operationsexceeds the scale of most local political jurisdic-tions and hence requires cooperation across juris-dictional boundaries. Funds must be raised froma variety of jurisdictions, central control andmanagement must be agreed upon, and a varietyof hospitals and other medical providers must co-operate. Because this type of joint effort is dif-ficult to bring about, the diffusion of the innova-tion is slow and uncertain even though there hasbeen Federal funding to support it.

The use of cable telecommunications to pro-vide social services seems likely to face similarproblems. Effective use of this medium will re-quire cooperation among a number of socialservice and regulatory agencies as well as severallevels of government. These agencies have dis-parate and conflicting goals and frequently resistefforts of other agencies to enter their areas of specialization. Diffusion of cable technology forthese uses will not occur easily.

User Need

A serious commitment to the innovation by the

performer is important to the successful opera-tion of a demonstration. Both the Change Agentstudy and the Federal Demonstrations studyfound non-Federal initiative to be important todemonstration success. The Change Agentstudy’s description of the two types of project ini-tiation encountered—opportunism and problemsolving—illustrates the problem well:

“The Comptroller General of the United States, Prog-ress, but Problems, in Developing Emergency Medical Serv-ice Systems, Washington, D. C., July 13, 1976.

“For an interesting discussion of this point see William A.Lucas, “Social Service Applications of Electronic Abun-dance, ” in Forrest Chisman and Glen Robinson (eds. ),Communications for Tomorrow: Policy Perspectives for theFuture (tentative title), Aspen Institute, Washington, DC.,forthcoming.

Projects generated essentially by opportunismseemed to be a response to available funds andwere characterized by a lack of interest and com-mitment on the part of local participants—fromdistrict administrations to classroom teachers. Asa result, participants were often indifferent toproject activities and outcomes, and little in theway of serious change was ever attempted—oroccurred.

The problem-solving motive for projects

emerged primarily in response to locally iden-tified needs and was associated with a strongcommitment to address these needs. Federalfunds were viewed as a way to support the localsolution—one which often broke new ground inlocal educational practice. 16

Cost sharing by non-Federal participants is anindication of interest. The Federal Demonstra-tions study found that demonstrations with largeshares of Federal funding (more than 90 percent)had a poor chance of diffusion success.

The form of the cost sharing is also important.

For example, the share of costs for projects at theState and local levels is frequently small and inthe form of contributed space and services. Insuch instances, the type of opportunism that theChange Agent study notes can easily occur. Itseems likely that larger contributions, involvingstaff and financial resources, will assure that atleast some consideration is given to whether thedemonstration is consistent with the interests of the local jurisdiction. For example, the mecha-nized refuse collection demonstration that wascited earlier involved contributions of nearly 40percent of the cost by the city of Scottsdale,

A r i z .

When the intent of the demonstration is topromote the commercialization of a technologyby the private sector, a variety of forms of costsharing are possible. A close-ended contributioncan be made as was the case with the AtomicEnergy Commision’s Power Demonstration Re-actor Program. 18 Loan guarantees can be provid-ed as was contemplated in the proposed syn-thetic fuels legislation. 19 The demonstrating firm

“Change Agent study, p. 9“Federal Demonstrations, p. 96.

18 Federal Demonstrations. Case Studies, pp I-55 and1-56.

“U.S. Congress, House. 94th Cong., 2d Sess., Commit-tee on Banking, Currency and Housing, Loan Guarantees

jor Demonstration of New Energy Technologies, report toaccompany H.R. 12112, Washington, D. C,, June 1976.

35



(or consortium of firms) can make a fixed con-tribution and the Government can support the re-mainder, as was proposed in the Clinch RiverBreeder Reactor Demonstration .’” Tax subsidiesmay be provided for specific types of invest-ments. The Government may guarantee theprice of the output of a particular plant. Detailedexamination of these various methods of financeis beyond the scope of this report, and in any

case must be made on a case-by-case basis.Nonetheless, the literature provides a few guidingprinciples.

The MIT Energy Laboratory Policy Group, inexamining policies for commercialization of energy technologies, suggests three guidelines:

1. Subsidies for demonstration projects shouldsimulate the workings of the normal marketand should be as small as possible.

2. The mechanism for providing the subsidyshould ideally provide managers with a cir-cumstance much the same as that in the un-

subsidized case. In particular, it should beneutral with respect to the choice of inputssuch as capital, labor, transportation, andmaintenance.

3. The financing mechanism should reveal thefull costs of the program and the detailedcost performance of the individualtechnologies. Loan guarantees are cited aspoor means of achieving this objectivebecause the degree of subsidy is not ob-vious. 21

These guidelines suggest simple, obvious cost

arrangements designed to elicit a commitment tothe effort by the private-sector firm.

Federal Demonstrations suggests that at thebeginning of planning for a demonstration proj-ect, a survey of possible participants should bemade to determine their willingness to participatein the demonstration. Lack of willingness tomake substantial contributions to the cost maywell indicate that technological uncertainty is too

20Leland L. Johnson, Edward W. Merrow, Walter S.

Baer, and Arthur J. Alexander, Alternative Institutional Ar-rangements for Developing and Commercializing Breeder

Reactor Technology, The RAND Corporation, R-2069 -NSF, November 1976, pp. 117-120.

“Policy Study Group, Energy Laboratory, GovernmentSupport for the Commercialization of New EnergyTechnologies, An Analysis and Exploration of the Issues,MIT, Cambridge, Mass., November 1976, p. 11.

high to proceed with a demonstration, thatmarket demand is weak, that costs are high, orthat institutional factors or side effects are likely toinhibit the use of the innovation .22 Thus, theFederal Demonstrations study suggests that thewillingness to share costs may be an appropriate“market test” for the demonstration of a newtechnology.

This willingness cannot always be used as an

indicator of commitment. If the problem to whichthe demonstration is addressed arises because of State or local inattention or indifference, a strongcost-sharing requirement may merely provide aconvenient excuse for that Government to avoidresponse. There is no easy solution to thisdilemma.

Time and Operational Flexibility

In the Federal Demonstrations study, not onedemonstration that proceeded under strict timeconstraints achieved any diffusion success. It is

also worth noting that none of these demonstra-tions produced the information necessary tomake an adoption decision, and only one wassuccessfully adopted at the demonstration site.These time constraints had little to do with in-terest in diffusing the innovations. They wereprimarily generated by other policy and politicalneeds .23

Operational flexibility allows the project per-formers to react to unanticipated events—eitherto avoid or recover from negative events or totake advantage of positive ones. Every demon-

stration encounters some unexpected situationsof varying importance. One reading project ex-amined in the Change Agent study encountereda freeze on the use of outside contractors whowere to print a locally developed test, the printingof the wrong version of the test by studentprinters, and a citywide teachers’ strike .24 Inanother case, an attempt to extend a classroomorganization project to a junior high failed. Bycompletely redesigning the program consistentwith the elementary project but using differenttechniques, a “significant degree of success” wasachieved .25

“Federal Demonstrations, p. 71.231bid, pp. 53-54.

“Peter Greenwood et al., Federal Programs SupportingEducational Change: The Process of Change, Vol. III, TheRAND Corporation, R-1589/3-HEW, April 1975, p. 43.

“Ibid, p. 44.

36



The Change Agent study found that attemptsto prescribe management techniques were “usu-ally counterproductive, leading to nonimplemen-tation or cooptation. ”

26 The Federal Demonstra-tions study does not dwell on detailed manage-ment procedures for the hardware demonstra-tions except to state that “onsite managementwas generally effective, ” meaning that projectmanagement was not a major source of trouble in

the cases studied .27

Factors Affecting the Success ofPolicy-Formulating Demonstrations

In comparison with policy-implementingdemonstrations, there has been relatively littleexamination of the success or failure of policy-formulating demonstrations. In part, this isbecause there have not been many such demon-strations. More importantly, however, the com-plexity of the policymaking process makes it dif-ficult to trace the use of information produced bysuch demonstrations.

The making of policy seldom conforms to themodels of rational decisionmaking that are pre-sented in the literature. At the Federal level,policymaking extends over a long period of timeand often includes loosely coupled actions atseveral levels of Government. For example,debate over the Nation’s welfare system has beengoing on for decades, with additions from time totime of new information or new calls for nationalconcern. Perceptions concerning the fiscal health

of cities have shaped the nature of the programsolutions that have been advanced. The almostinadvertent expansion of programs such as foodstamps changes the need for welfare reform.Determining the role that a particular piece of in-formation plays in shaping a particular policy isvirtually impossible,

Policymaking consists of identifying problems,searching for possible courses of action, ar-ticulating and evaluating these courses of action,and choosing one or a combination of the op-tions. These are not necessarily sequential

events, however. The definition of the problemmay take place as a committee of Congress or alegislative drafting group articulates and evalu-

2’Change Agent study, p. 26,“Federal Demonstrations, p. 58,

ates proposed programs. A possible solution maybe shaped by policy actions in quite differentproblem areas. The identification and articulationof policy alternatives may occur in adversary pro-ceedings or as the result of patient analytical staff work.

Demonstration projects may make contribu-tions to virtually all steps of the policymakingprocess. The most obvious contribution is the ar-

ticulation of the consequences of adopting a par-ticular policy. Estimates of the costs, perform-ance, and unanticipated effects of a policy or pro-gram may be obtained. However, the process of designing the project may help to clarify thenature of the problem and therefore to define theobjectives of a potential policy. Conducting ademonstration may be a means of advocating theconsideration of a policy alternative by groups orprograms that have the interest and resources todo so. It is even possible to imagine that an arrayof projects which embody the various policy or

program alternatives could provide decisive in-formation for choosing among policies.

Of course, many factors affect the outcome of a policy debate besides the predicted per-formance of the policy itself. Distribution of bene-fits and costs among groups, regions, or indus-tries may be important. The particular interestsand values of policy makers in important positionswill shape both the alternatives considered andthe choices made. Crises or events that requireimmediate action may make quick decisionsimperative. Consequently, it will be very difficultto provide sharp assessments of the success or

failure of demonstrations that seek to contributeto the policymaking process.

The distinction between policy-formulatingand policy-implementing demonstrations wasoriginally made to categorize the purposes of social demonstrations. 28 Social experiments suchas those dealing with housing allowances, in-come maintenance, health insurance, and utilityrate structures fit the description well. Othersocial service and education projects can be iden-tified that are tested in realistic environments todetermine if they should be put into effect in

large-scale programs.“Cheryl D. Hayes, “Toward a Conceptualization of the

Function of Demonstrations, “ in Thomas K. Glennan, Jr.,(cd.), Studies in the Management OJ Social Research andDevelopment, the National Academy of Sciences,Washington, D. C., forthcoming.

37





debates this issue was important. However, thedebates have now continued for some 8 yearsand other concerns have become more impor-tant. Today concerns are expressed about thedistribution of benefits among different classes of recipients and different areas of the country, themanner in which States will participate in the ad-ministration of the program, the degree to whichbenefit levels will vary by area of the country, andthe effect the program will have on employmentprograms. In part, this shift can be attributed toCongress’s recognition that political issues sur-rounding welfare reform relate to the distribution

of costs and benefits. In part, however, the

reason for the receding relevance of the informa-

tion provided by the income maintenance ex-

periments may be that their findings were such asto reduce the concern on the part of the law-makers and the public that there would bemassive withdrawals from the labor force.

The success of a demonstration in aiding a

policy debate should depend upon the joint ac-ceptance of the validity of the information by par-ticipants in that debate. If, for example, one goalof a demonstration is to determine the en-vironmental effects of a new process for the pro-duction of synthetic fuel and if both the pro-moters of the innovation and the interested en-vironmental groups do not agree upon thepollutants to be measured and the means of measuring them, the evidence produced by thedemonstration and the policy debate is likely tobe less useful than would otherwise be the case.In another context, one problem—although not

necessarily the most significant one—in deter-mining the efficacy of Laetrile in the treatment of cancer patients is that agreement among pro-ponents and opponents as to what constitutes afair test is difficult to conceive.

The timing of demonstrations also posessignificant problems. Many policy decisions aremade as a result of forces quite independent of the development and evolution of a technology.When a crisis arises, such as the OPEC oil em-bargo or the urban riots, policies will be for-mulated and implemented whether or not there

is information from demonstrations or otherclasses of R&D that would affect those policies.Politicians in both executive and legislativebranches have short time horizons. They standfor reelection at regular intervals or possesscareer progression patterns that place them in

particular policy positions for short periods of time. Policies are often made in response topressure from various interest groups. Because of the fluidity of the situation and because of lengthof time usually involved in the conduct of dem-onstrations, matching the results of demonstra-tion activities to the policymaking process istroublesome.

The difficulty in matching the timing of dem-onstrations to decisonmaking varies amongpolicies. Where the question is one of adoptingand promoting a major technological device suchas a new weapons system or a specific energy-related process such as the breeder reactor, thesensitivity of the policy to evidence concerningthe performance of the device may be so widelyappreciated that the policymaking will be pacedby the evolution of the technological work. Apolicy-formulation demonstration in this casewould set the timing for policymaking.

In contrast, in cases where the technology isclearly not perceived as central to the decison-making process, other imperatives that governthe policymaking process will take over. Con-sider again the income maintenance experiment:the underlying technology, the transfer of fundsto individuals contingent upon their other sourcesof income, was perceived as being feasible fromthe beginning. The demonstrations focused onthe work incentives, unanticipated outcomes,and administrative processes. These issues werenot suffiently central to the policy debates thatthe executive branch or Congress felt the debate

should await the completion of the demonstra-tion.

This discussion leads us to advance two ten-tative propositions concerning factors affectingthe success of policy-implementing demonstra-tions:

q

q

The perceived relevance and usefulness of policy-formulating demonstrations will beenhanced when they are initiated and per-formed by agencies with close continuingties with policy makers.

Policy-formulating projects that give con-scious attention to expected points of con-flict in the policy debate and address theseconflicts in their design will be more suc-cessful than those that are developed alongthe interests of just one party.



Continuing Close Ties With Policymakers

One theme pervades our discussion of factorsaffecting the success of policy-formulating dem-onstrations. The success of such demonstrationsdepends upon the styles and interests of relevantpolicy makers. Demonstrations initiated andmanaged by organizations that are closely linkedto the policy maker are likely to reflect those in-

terests.All the cited examples of policy-formulating

demonstrations have been initiated and managedby organizations that bore close relationships toexecutive branch policy makers. The social exper-iments have been sponsored and monitored byAssistant Secretary level offices of policy plan-ning and evaluation. The regulatory demonstra-tions were carried out by the cognizant regulatoryagency. We have reviewed no examples of pol-icy-formulating demonstrations that have beensupported by relatively independent researchagencies such as the National Science Founda-tion or the National Institutes of Health. Thus, wecannot provide evidence to test this proposition.

Although a major portion of the policymakingprocess concerning the issues examined in theseexperiments will be associated with the develop-ment of legislation, Congress has generallyplayed a minor role in the design of the ex-periments. Sponsors of demonstrations frequent-ly do not relate closely to Congress. The housingexperiments, although mandated by law, wereimplemented with little input from legislators.The income maintenance experiments were de-

signed on the basis of executive branch percep-tions of what the policy issues would be. Thesame was true of the health insurance experi-ment.

We have found two instances in which Con-gress mandated demonstrations to understandthe effects of a proposed policy change. The1972 amendments to the Social Security Actauthorized a group of experiments anddemonstrations: 30

.

native methods for classifying providers, forestablishing prospective rates of payment, andfor implementing on a gradual, selective, orother basis the establishment of a prospectivepayment system.

Under this authorization, a number of demon-strations have been mounted; in addition, severalexisting State efforts to limit the rate of increaseof hospital costs have been evaluated, whichessentially made these efforts policy-formulationdemonstrations .3*

In 1974, Congress authorized up to 20 ex-perimental programs as a part of the Study of theEffectiveness of Compensatory Education Pro-grams. The primary purpose of these programswas to determine the effects of changing thebases upon which compensatory education fundswere allocated within a local education agency.Under this authorization, 13 districts are e x -

perimenting with policies such as using academic

achievement rather than poverty cri teria to

allocate compensatory education funds. 32

Including Points of Conflict inDemonstration Design

The making of policy frequently involvesmelding together a variety of conflicting views.This is particularly true for the policies made byCongress. As a consequence, we might hypoth-esize: policy-formulating demonstrations thatgive conscious attention to the expected points of conflict in the policy debate and address theseconflicts in their design will be more successful

than those that are developed along the interestsof just one party.

Explicit and participatory processes for seekingthe conflicting views were not apparent in theplanning of the demonstrations we examined.However, in some cases, particularly those asso-ciated with the regulatory demonstrations, manyof the interested parties directly participated inthe conduct of the demonstration. In the case of the Refan engine, manufacturers, the regulatory

to determine the relative advantages anddisadvantages of various alternative methods of

making payment on a prospective basis tohospitals, skilled nursing facilities, and other pro-viders of services for care and services providedby them under Title XVII . . . including alter-

‘“Sec. 22 of Social Security Amendments of 1972.

31U. S. Department of Health, Education, and Welfare,

Social Security Administration, Office of Research and

Statistics, Research in Health Care Reimbursement,Spring/Summer 1976, Publication No. (SSA) 77-11901,Washington, D. C., 1976.

32National Institute of Education, Evaluating Compen-

satory Education, an interim Report on the NIE Compen-satory Education Study, Washington, D. C., Dec.30, 1976,Appendixes A and B.

40



agency, and the airlines all participated, and theevidence from the demonstration does not seemto have been a matter of dispute.

Kash et al., in a study of needed energy R&D,proposed that many of the demonstrations of energy supply technologies should be viewed asmeans of obtaining evidence to resolve disputesamong concerned groups such as the en-

vironmentalists and industry .33 It certainly seemsdesirable to assure that the concerns of variousparties in a dispute over the desirability of acourse of action should be reflected in the designof the demonstration but it must be rememberedthat some part—perhaps the major part—of thedispute is over values possessed by various par-ties in a decision, and value conflict will not beresolved by a demonstration project.

Even in cases where there are importantdisputes over potentially knowable facts such asthe level of pollution or costs associated with aparticular production process, groups that expectultimately to oppose the adoption of an innova-

3’Don E, Kash et al., Our Energy Future, University of

Oklahoma Press, Norman, Okla.. 1976, pp. 25-26.

tion (whatever its outcome) may oppose the con-duct of a policy-formulating demonstration. Suchdemonstrations are frequently seen as a meansby which a Government agency or industrial cor-poration that has already made its decision ini-tiates action. These groups tend to doubt the

Government’s assurance that a demonstration is

focused primarily on obtaining information about

the desirability of pursuing a course of action.

This problem is heightened by the nature of Government-industry relationships that seemlikely to lead to successful commercializationefforts—i.e., close and sympathetic ties that pro-mote a good exchange of reliable information.

If the two propositions advanced here are cor-rect, Congress has important contributions tomake to the conduct of many policy-formulatingdemonstrations. Where legislative action is ex-pected to follow a demonstration, Congressneeds to assure that the design of the demonstra-tion is relevant to its deliberations. In particular, it

must seek to ascertain that issues in which impor-tant political constituencies are likely to disagreeare treated in a manner that will help to resolvethe debate.

41







stances, the importance of the problem maydictate its use.

The Demonstration as an Instrument ofTransition: Inherent Difficulties

These qualities of demonstrations pose dif-ficulties for Congress as it performs its legislativefunctions. Moreover, the role of demonstrationprojects as instruments of transition means that

Congress must proceed in the face of poor andprobably biased information. Ideas imbedded inthe project are moving from R&D to use;technical criteria are being supplanted by institu-tional criteria of success. Projects are evolvingfrom the small scale typical of a laboratory to fullscale in the field. Control is shifting from R&Dpersonnel to operating personnel. In many in-stances, this transition is associated with a move-ment from public to private-sector sponsorshipand management. Alternatively, the innovationmay shift from Federal emphasis and spon-sorship to State and local use. Not only is a tran-

sition such as this difficult to bring about, but itgives rise to strong advocacy by individuals andgroups with a stake in the innovation.

R&D personnel who have developed a newtechnology often wish to proceed to the nextlogical step, a demonstration in the field. In-dustry, particularly that part associated withearlier development work, may seek a subsidyfor continuation into the demonstration phase.Final users such as State and local officials maywell see the innovation as a means for enlargedpolitical power or career advancement. Because

of the transitional nature of the demonstration,however, most actors have only limited perspec-tives concerning the worth of the innovation.

The developer may not be aware of all the in-stitutional impediments to the application of theinnovation. The final users may have insufficientunderstanding of the nature of the technologyand may have a large personal stake in its ap-plication. The industrial firm may see the op-portunity to obtain a subsidy for an effort that itmight otherwise have to fund itself. In any case,Congress or the senior executive branch officialswill be faced with conflicting information.

Incentives for Government Agenciesto Use Demonstrations

When considering policies to promote thecommercialization of new technologies by the

46

private sector, Congressnote of the nature of thethe behavior of Federal

should take particularincentives that governR&D agencies. These

agencies are under continual pressure from thepublic and Congress to show that their programshave resulted in technologies and knowledge thatare being used. One of the most straightforwardmeans of obtaining an example of use is to createa demonstration project. A demonstration proj-

ect may thus be viewed as a simple means of placating critics rather than as a component of awell-developed strategy to promote the use of the results of an R&D program.

Even in public bureaucracies where there is athoughtful policy for promoting the use of tech-nology, there are some undesirable incentives.As we have noted, it is the reality of the R&Dprocess that there is both uncertainty concerningthe worth of an idea that is being developed anda likely division of opinion about the most likelyoutcome of its application. This should result in

dropping or sharply redirecting a substantial pro-portion of R&D efforts before they reach fruition.In the private sector, decisions to continue orcease development efforts are disciplined by therealities of the market, which provides a measureof outcome to which all members of the firm mustadhere. In general, the structure of a businessfirm also makes it clear who will be responsiblefor various levels of decision. If a member of thefirm does not agree with the decision that ismade, he has little recourse except to leave thefirm.

In the bureaucracy of the public sector, the in-centives are different. Frequently, the bureauc-racy will possess multiple goals. In energy, for ex-ample, program goals encompass both the devel-opment of a technological base and the commer-cialization of new technologies. More impor-tantly, decisons concerning projects to be ini-tiated or continued are subject to considerablepublic scrutiny. Public officials do not have theluxury of simply making a decision because theyhave the responsibility and authority to make thatdecision; they also must justify those decisionsbefore a considerable number of outsiders, many

of whom have quite different views concerningthe prospects for a technology, the goals of a pro-gram, or the appropriate policy tools to be used.Employees of the agency or their friends outsidethe agency can frequently appeal the decisions tohigher agency officials or to Congress itself. The



debate over the demonstration of the Clinch

River breeder reactor shows that even a Cabinet

Secretary and the President are unable to make

decisions that cannot be overturned.

The incentives of the Government programmanager may also be somewhat different fromthose of his counterpart in the private sector. Thelatter program manager will frequently report tohigher level corporate officers who have had con-siderable experience with the development of new products and processes. This program man-ager can easily be rewarded for knowing when tocut off a development as well as for promotingpromising ventures because the firm’s manage-ment appreciates the risks associated with newproduct development. In contrast, the public offi-cial may frequently have to face the scrutiny of anarray of elected officials and citizens who do notshare his views of the uncertainty and who takethe cessation of a project as an indication of failure on the part of the program manager. As aconsequence, he may push for a demonstrationproject despite its low probability of success inpromoting the use of an innovation.

These qualities of a Government agency seemlikely to inhibit its effective participation in makingsound decisions concerning the commercializa-tion of products and processes developed. It willtake such bureaucracies a long period of time tomake decisions because so many points of viewmust be considered. Similarly, it will take time toimplement decisions when they are controversialbecause of the several levels at which the deci-sions can be overturned. It may well be that the

necessary absence of profit-oriented goals withinpublic bureaucracies will increase the influence of personal career-oriented incentives of Govern-ment officials in the decision process.

Implications for Congress

The narrow scope of usefulness of demonstra-tions in achieving R&D objectives, the political at-tractiveness of those demonstrations, the diffi-culty of achieving the variety of transitions thatthey seek to make, and the nature of the incen-

tives for Federal R&D agencies suggest four char-acteristics of programs containing demonstrationprojects that Congress might encounter.

1. Congress should expect a low rate ofsuccess with demonstrations as a

means of promoting the use of atechnology.

The transition of an innovation from develop-ment to use is difficult under any circumstances.It is likely that Federal officials promote technol-ogies having objectives that are not fully sharedby either the private sector or by State and localauthorities. Demonstrations may frequently notbe the policy instrument of first choice but ratherthe only politically feasible instrument available.Each of these circumstances makes success dif-ficult.

2. Congress should expect that the in-formation it receives concerning thepotential of a proposed demonstrationwiII probably be biased and imperfect.

Again, the goals of transition mean that thereare few, if any, experts that have appropriate ex-perience or are in institutional settings that do notbias their judgments. Moreover, the fact that

demonstrations are frequently the product of apolitical decision process means that the partiesto that process will have different goals for the ac-tivity and hence different assessments of likelysuccess.

3. Congress should expect that there wbe frequent confusion over the goaof a demonstration project.

The inherent semantic confusion over the ter

II

s

n“demonstration” constitutes the first problem.Demonstrations can be used to prove as well as

to display a concept. In addition, the perspectivesof the actors involved in the demonstration proc-ess may be quite different. Some may be certainthat the innovation has been proved and seek only to promote its use; some may be dubiousand seek to discover its worth; still others maysee the demonstration as an easy way to disposeof a political problem and thus will be primarilyconcerned with satisfying important constituen -cies.

4. Congress should expect that the eval-uation of the success or failure of a

demonstration will be difficult and judgmental.

As policies are formulated, a demonstrationcan provide important but rarely decisive infor-

47



mation. Moreover, the distribution of policy-making through time and among different insti-tutions and levels of Government makes tracingthe value of the outcome of demonstrations near-ly impossible.

While at first glance, evaluation of policy-im-plementing demonstrations appears to be simple,our analysis suggests that it is not. The possibilityof goal conflict between Federal and non-Federal

sectors, the strength of a technology, or thequality of an institutional environment all affectthe outcome. A particular demonstration projectmay not have spawned a large group of replica-tions but may have illuminated the institutionalproblems so that future efforts to promote the useof new technologies will be improved. The proj-ect may have brought a policy problem to the at-tention of local government and resulted in con-tinued attempts at that level to deal with theproblem. It may have sustained an R&D capabil-ity in one or more private-sector firms that willlead to successful new innovations. It may have

forced the resoution of uncertainties over en-vironmental or other regulations that will makesubsequent investment planning easier.

In light of these expectations, Congress should

consider whether other types of policy or pro-

gram actions will better serve their ends or willcomplement and enhance the possibilities of suc-cessful demonstration projects.

Alternative Strategies toPromote the Use of R&D

Five suggested strategies for improving thechances of successful demonstration projects are:

q Conduct engineering tests.q Change market incentives.q Modify the institutional environment.q Subsidize local development.qUtilize existing projects.

The Congress, and the Federal Government ingeneral, can benefit by considering these strate-gies as either alternatives or complements to

demonstration projects.

Conduct Engineering Tests

The evidence suggests that if there is a high

degree of uncertainty associated with technol-ogy, a policy-implementing demonstration willhave a high probability of failing to achieve its ob-

jectives. If this is thought to be the case, a large-scale prototype test or test-bed experiment maybe needed. It may be less expensive, quicker,and more decisive than a demonstration projectthat has to deal with the problems of a real-worldsetting. If a major barrier to commercial use of anew technology is technological uncertainty, theresolution of that uncertainty may be a sufficientmeans of promoting the use of the technology.

In the development of large-scale technologythere frequently is pressure both to advancetechnology and to demonstrate its usefulness in asingle project; this appproach can have unfortu-nate outcomes. Experience with the develop-ment of major weapons systems is instructive,although the problems arising in those systemsare not fully analogous to the policy problem

discussed here. The military has often attemptedto compress development times of weapons

systems by initially purchasing a sufficientnumber of completed weapons systems permit-ting realistic operational testing. In many cases,the production of these test systems has begunbefore all the technological uncertainties havebeen resolved. The result has been increasedcosts, lengthened time schedules, and erodedlevels of performance.

Despite this experience, pressures for earlytest and demonstration continue. Enthusiasticsupporters of a weapons system do not want it be discarded and thus seek the maximum com-

mitment to the system. The industrial producerswant to minimize the risk of cancellation. Mostimportantly, developers and sponsors of newtechnologies perpetually seem to underestimatethe amount of uncertainty that exists with a newtechnology. Thus,

q T h e Congress s h o u l d s e e k t o e n s u r e

t h a t i t d o e s n o t a d d t o t h e i n c e n t i ve s

f o r p r e m a t u r e d e m o n s t r a t i o n o f a n

u n d e v e l o p e d t e c h n o l o g y b y p r e s s i n g

t o t u r n e n g i n e e r i n g t e s t s i n t o f u l l -

s c a l e d e m o n s t r a t i o n s .

IR. L. Perry, Reforms inCorporation, P-5482, July

System Acquisition, The RAND1975.

4 8





improvements by bringing elements of State gov-ernments together to seek common perceptionsof social problems and solutions. Federal author-ities can also foster the development of new cur-riculum and support the training of personnel inthe skills necessary to improve the innovativecapabilities of the State and local government.

As with any policy, such attempts to improvethe institutional environment should be precededby efforts to determine the real nature of theproblem. For example, it has been argued thatlocal schools lack the capacity for useful problemsolving. Consequently, proposals have beenmade for educational extension agents, teachercenters, technical assistance agencies, and otherforms of assistance. It is quite possible, however,that the real reasons for the lack of desired im-provements stem from the incentive structuresthat face teachers and administrators at the locallevel, from the type of training that is provided inteacher’s colleges, or from the pressures thatchanging societal norms are imposing upon the

schools. Developing better problem-solving skillsmay accomplish little. In the absence of goodproblem diagnosis, attempts to improve the in-stitutional environment may simply lead to addi-tional expensive complexity in the environment.To paraphrase,

q C o n g r e s s s h o u l d r e g a r d p o l i c i e s t h a t

s e e k t o d e v e lo p i n s t i t u t i o n a l e n v i ro n -

m e n t s a s co m p le m e n t s t o R & D

po l ic ies .

Subsidize Local Development

When a technology of low reproducibility or apoorly developed institutional environment iscoupled with a demonstration, Congress maywant to subsidize local development efforts tofocus attention on a problem of national concern.Such subsidies permit local units of governmentto implement the forms of new technologies thatthey want and to modify them in ways that seemappropriate to their particular situations.

There is a growing literature on these types of programs. ’ In general, it has been found that a

‘See, for example, Robert K. Yin, Karen A. Heald and

Mary E. Vogel, Tinkering with the System, LexingtonBooks, Lexington, Mass., Chapter 5, 1977; and Paul Ber-man and Milbrey Wallin McLaughlin, Federal ProgramsSupporting Education Change, Vols. I-IV, The RAND Cor-poration, R-1589 1-4-HEW, April 1975.

disappointingly small number of programs sur-vive the withdrawal of the Federal subsidy. Par-tof this low-survival rate reflects the opportunisticbehavior that characterizes some local govern-ment and educational institutions. Also, thebureaucratic needs of local governments may notbe served by many of the developments that areattempted. Innovation is never easy, There is awide range of plausible explanations for the lowrate of incorporation of results, and it would be

premature to suggest that the observed rate of in-corporation is the best that can be obtained.Thus,

q Th e Congress s h o u l d s e e k t h e t y p e o f

e x p e r i m e n t a t i o n a n d e v a l u a t i o n t h a t

w i l l c a s t l i g h t o n t h e b e s t m e a n s t o

p ro v id e su b s id ies fo r lo ca l d eve lo p -

m e n t a n d t o i n c o rp o ra t e t h e f i n d i n g s

i n n ew o r r ev i s ed l eg i s l a t i o n .

Utilize Existing Projects

For some social policies, existing projects orprograms may provide information that can onlybe marginally improved by introducing a newpolicy-formulat ing demonstrat ion . In thecongressionally mandated program examiningmeans to provide incentives for efficient deliveryof health care, such an approach was taken.Health care reimbursement systems alreadyunder development or in use by several Stateswere evaluated to provide information about thepotential effects of several different approachesto the problem.

There are many advantages to using existingprojects. Since they already exist, informationcan be more quickly obtained than would be thecase with a new project. Many of the costs of starting a project will already have been incurred.It is possible that the information on implementa-tion problems will be more reliable than would bethe case with a federally funded demonstration.On the other hand, the existing project may inad-equately reflect important policy options; theproject managers may resist being evaluated byFederal agencies; or the project may be struc-tured in ways that make determination of out-comes difficult or impossible. The last case wouldoccur if several policies had been simultaneouslyimplemented in a way that prevented the separa-tion of outcomes attributable to each policy. Tosummarize,

50





proposed innovation, the purposes of the dem-onstration can be compromised. Costs willescalate. Potential users of the innovation mayperceive it as unreliable. Compromises in theconduct of the demonstration will then have to bemade to accommodate resolution of the technicalproblems. Congress should seek to ensure thatthe technology associated with the demonstra-tion is sufficiently well in hand to preclude

technological problems from dominating the out-come of the demonstrations.

q Is there sufficient evidence of commit-ment to the demonstration by the per-former?

The difficulties and the complex array of in-centives associated with any demonstration proj-ect make commitment on the part of the par-ticipants an important factor in its success. In theprivate sector, the best indicator of such commit-ment is the willingness to assume a substantial

share of the costs of the demonstration. Ex-perience also suggests that projects that are con-ceived by the private or State and local sectorsthemselves are more likely to have this commit-ment than projects developed in response todetailed solicitation by Federal agencies.

In a few cases, this type of indicator maybe in-appropriate. A willingness to invest funds impliesa belief that the conduct of the demonstration isclearly consistent with the goals of a firm ororganization. For some innovations, however,there will be Federal interest in promoting goalsthat are not currently consistent with those of

private firms or public agencies. For example, theuse of cable television to deliver social services tothe aged and the poor may require changes inthe behaviors of social service agencies and coststo cable television franchises that are unlikely tobe recovered through fees. It may be impossibleto implement an innovation like this in theabsence of almost total support from the FederalGovernment. Demonstrations of programs deal-ing with a new and difficult clientele face similarproblems. Much of the Federal involvement inelementary and secondary education wasbrought about by the failure of the Nation’sschool systems to deal adequately with the needsof disadvantaged students. Given that theseschools did not consider this problem to be im-portant, it is unlikely that they would have beenwilling to make a substantial financial commit-

52

ment to demonstration projectsproblems of the disadvantaged.

dealing with the

In general, however, Congress should aban-don cost sharing as a measure of commitmentonly if the goals of the demonstration are of significant national importance, and if the poten-tial gains from the demonstration outweigh thehigher probability of its failure when the per-formers are unwilling to make financial com-mitments.

q Does the design of the demonstrationproject reflect the experiences of pastdemonstrations?

The failure of a demonstration project to con-tribute to formulating a policy or promoting theuse of an innovation is not necessarily an indica-tion of wasted public funds. As we haverepeatedly noted, the task that a demonstrationproject seeks to perform is difficult. Apparentproject failures may well reflect problems with theinstitutional environment, conflicting goals at dif-fering levels of government, or other factorsdiscussed in this report rather than poor manage-ment or inadequate funding.

These failures are likely to contain importantlessons and may point the way to better futurepolicy and program design. For example, whilethe ambitious demonstration of educational tele-vision has left comparatively little in the way of significantly changed patterns of instruction, it

has heightened our knowledge concerning theprocess for changing these instructional patterns.Cable television demonstrations and experimentshave clarified the conflict between the incentivesdriving the expansion of commercial markets forcable systems and the desire to use such systemsto enhance the access to services by disadvan-taged groups in society. Demonstrations of solarheating and cooling have served to highlight as-pects of the institutional environment to bedeveloped.

The lessons learned from these demonstra-tions can lead to policies to encourage the devel-opment of the institutional environment. Theycan provide guidance for future technologicaldevelopments. Congress should promote effortsto learn these lessons, and, where appropriate,encourage new demonstrations that reflect them.



—

The Future Use of sponsors and performers. But demonstrations

Demonstration Projects are also easily misused. They can lead to waste,frustration, and discouragement. We hope that

Demonstrations constitute fascinating policy the concepts and guidelines developed in thistools. They provide opportunities to try innova- report will help Congress and others to improvetions; they can be used to promote important the use of demonstrations in pursuing nationalcauses; they are exciting experiences, for both goals.

53



BIBLIOGRAPHY



—.

BIBLIOGRAPHY

Allen, Wendy, Nuclear Reactors for GeneratingElectricity: U.S. Development from 1946 to1963, The RAND Corporation, R-21 16-NSF,June 1977.

American Medical Association, Report of theCommission on Medical Research, Chicago,111., 1967.

Andrus, E. C., et al. (eds. ), Advances in Military Medicine, Vols. I and II, Little, Brown andCompany, Boston, Mass., 1948.

Arrow, Kenneth J., “Economic Welfare and theAllocation of Resources for Invention, ” inRichard R. Nelson (cd.), The Rate and Direc-

tion of Inuentiue Actiuity: Economic and Social Factors, a Report of the NationalBureau of Economic Research, PrincetonUniversity Press, Princeton, N. J., 1962.

Arthur D. Little, Inc., Federal Funding of Civilian Research and Deuelopment, Volume 1: Sum-mary, Washington, D. C., February 1976.

Baer, Walter, et al., Analysis of FederallyFunded Demonstration Projects: SupportingCase Studies, The RAND Corporation, R-1927 -DOC, April 1976.

Baer, Walter S., Leland L. Johnson, and Ed-ward W. Merrow, Analysis of FederallyFunded Demonstration Projects: Final Report,The RAND Corporation, R-1926 -DOC. April1976.

Baer, Walter S., Leland L. Johnson, and Ed-

ward W. Merrow, “Government-SponsoredDemonstrations of New Technologies, ”Science, Vol. 196, May 27, 1977.

Berman, Paul, Milbrey Wallin McLaughlin, et al.,Federal Programs Supporting EducationChange, Vols. I-V, The RAND Corporation,R-1589/l-HEW, September 1974 and R-1589/2-5-HEW, April 1975.

Bode, H. W., Synergy: Technical Integration and Technological innovation in the Bell System,Bell Laboratories, Murray Hill, N. J., 1971.

Chaiken, Jan, What is Known About Deterrent Effects of Police Activities, The RAND Cor-

poration, P-5735-1, July 1977.Committee on Public Engineering Policy. Na-

tional Academy of Engineering, Priorities for Research Applicable to National Needs, reportto the National Science Foundation, National

Academy of Engineering, Washington, D. C.,1973.

Comptroller General of the United States, Prog-ress, but Problems, in Developing Emergency

Medical Service Systems, Washington, D. C.,July 13, 1976.

Dupree, A. Hunter, Science in the FederalGovernment, Harvard University Press, Cam-bridge, Mass., 1957.

Eads, George, “U.S. Support for Civilian Tech-nology, Economic Theory, and Political Prac-tice, ” Research Policy, Vol. 3, 1974.

Eads, George, and Richard R. Nelson, “Govern-mental Support of Advanced Civilian Tech-nology —Power Reactors and the SupersonicTransport, ” Public Policy, Vol. 19, 1971.

England, J. Merton, “Dr. Bush Writes a Report:

‘Science—the Endless Frontier, ’ “ Science,Vol. 191, Jan. 9, 1976.Ezra, Arthur A., “Technology Utilization: Incen-

tives and Solar Energy, ” Science, Vol. 187,Feb. 28, 1975.

Gramlich, Edward M., and Patricia P. Koshel, Educational Performance Contracting, An Evaluation of an Experiment, The BrookingsInstitution, Washington, D. C., 1975.

Hayes, Cheryl D., “Toward a Conceptualizationof the Functions of Demonstrations, ” inThomas K. Glennan, Jr. (cd.), Studies in the

Management of Social Research and Develop-

ment, The National Academy of Sciences,Washington, D. C., forthcoming.

House, Peter W., and David W. Jones, Getting It Off the Shelf: A Methodology for implement-ing Federal Research, West View press,

Boulder, Colo., 1977.

Johnson, Leland L., Edward W. Merrow, Walter

S. Baer, and Arthur J. Alexander, Alternativeinstitutional Arrangements for Developing andCommercializing Breeder Reactor Technol-ogy, The RAND Corporation, R-2069-NSF,November 1976.

Kash, Don E., et al., Our Energy Future, Univer-

sity of Oklahoma Press, Norman, Okla.,1976.

Keenan, Terrance, “The Health Record of Pri-vate Foundations, ” Journal of Health Politics,Policy and Law, “ Vol. 2, No. 1, Spring 1977.

57



Kevles, Daniel J., “The National Science Foun-dation and the Debate over Postwar ResearchPolicy, 1942-1945: A Political Interpretationof ‘Science—The Endless Frontier, ’ “ /S1S,Vol. 68, No. 241, 1977.

Lilley, William, III, and James C. Miller, 111, “TheNew ‘Social’ Regulation, ” The Public Interest,No. 47, Spring 1977.

Lucas, William A., “Social Service Applications

of Electronic Abundance, ” in Forrest Chismanand Glen Robinson (eds. ), Communications Jor Tomorrow: Policy Perspectives for the Fu-ture (tentative title), Aspen Institute, Washing-ton, D. C., forthcoming.

Merrow, Edward W., Constraints on the Com-mercialization OJ Oil Shale, The RAND Cor-poration, forthcoming.

‘&Minding the Kid. Frustrating for Companies, ”New York Times, Sept. 11, 1977, p. F9.

MIT Energy Laboratory Policy Study Group,Government Support for the Commercializa-tion of New Energy Technoogies: An Analysis

and Exploration of the Issues, N ovem ber1976.National institute of Education, Evaluating Com-

pensatory Education, an interim report on theNIE Compensatory Education Study, Wash-ington, D. C., Dec. 30, 1976, Appendixes Aand B.

Nelson, Richard R., “The Economics of Inven-tion: A Survey of the Literature, ” Journal of Business, Vol. XXXII, No. 2, April 1959.

Nelson, Richard R., Merton J. Peck, and EdwardD. Kalachek, Technology, Economic Growth,and Public Policy, The Brookings Institution,

Washington, D. C., 1967.Nelson, Richard R., and Sidney G. Winter, “In

Search of Useful Theory of Innovation, ”Research Policy, Vol. 6, 1977.

Pechman, Joseph A., and P. Michael Timpane(eds. ), Work incent ives and IncomeGuarantees: The New Jersey Negative in -

come Tax Experiment, The Brookings Institu-tion, Washington, D. C., 1975.

Perry, R. L., Reforms in System Acquisition, TheRAND Corporation, P-5482, July 1975.

Riecken, Henry W., and Robert F. Borouch(eds. ), Social Experimentation: A Method for

Planning and Evaluating Social lntervention,Academic Press, New York, 1974.

Rein, Martin, Social Policy: issues of Choice and Change, Random House, New York, 1970.

Rivlin, Alice M., Systematic Thinking for Social Action, The Brookings Institution, Washing-ton, D. C., 1971.

58

Rivlin, Alice M., and P. Michael Timpane (eds.),Planned Variation in Education: Should WeGive Up or Try l-larder?, The Brookings In-stitution, Washington, D. C., 1975.

Schon, Donald, “The Blindness System, ” ThePublic Interest, No. 18, Winter 1970.

Schultze, Charles L., “The Public Use of the Pri-vate Interest, ” Harper’s, Vol. 254, May 1977.

Social Security Administration, Office of Re-

search and Statistics, Research in Health CareReimbursement, Spring/Summer 1976, Pub-lication No. (SSA)77-11901, Washington,D.C. , 1976.

Steiner, Gilbert Y., The Children’s Cause, TheBrookings Institution, Washington, D. C.,1976.

Stricland, Stephen P., Politics, Science, andDread Disease: A Short History of UnitedStates Medical Research Policy, HarvardUniversity Press, Cambridge, Mass. 1972.

Study Committee on Social R&D, The FederalInvestment in Knowledge of Social Problems,

The National Academy of Sciences, Washing-ton, D. C., forthcoming.The Status of Education Research and Develop-

ment in the United States: 1976 Databook,National Institute of Education, Washington,D. C., 1976.

U.S. Congress, House, 94th Congs., 2d Sess.,Committee on Banking, Currency and Hous-ing, Loan Guarantees for Demonstration of New Energy Technologies, report to ac-company H.R. 12112, Washington, D. C.,June 1976.

Wenk, Edward, Jr., “The Social Management of

Technology,” in John E. Mock (cd.), Sciencefor Society, proceedings of the NationalScience Conference held in Atlanta, Ga., Oct.12-14, 1970.

Williams, Walter, Social Policy Research and Analysis: The Experience in the Federal SocialAgencies, American Elsevier Publishing Com-pany, New York, 1971.

Yin, Robert K., R&D Utilization by Local Seru-ices: Problems and Proposals for FurtherResearch, The RAND Corporation, R-2020 -DOJ, December 1976.

Yin, Robert K., Karen A. Heald, and Mary E.

Vogel, Tinkering with the System, LexingtonBooks, Lexington, Mass., 1977.

York, Herbert F., and G, Allen Greb, “Military

Research and Development: A Postwar His-tory, ” Bulletin oj the Atomic Scientists, Vol.33, No. 1, January 1977.



Date post:	31-May-2018
Category:	Documents
Upload:	chris-nash
View:	215 times
Download:	0 times