1

PaperfortheOECDBlueSkyForum2016–finalversion,15July2016Title:MonitoringtheEvolutionandBenefitsofResponsibleResearchandInnovation(MoRRI)–apreliminaryframeworkforRRIdimensions&indicatorsAuthors:IngeborgMeijer,NielsMejlgaard,RalfLindner,RichardWoolley,IsmaelRafols,ErichGriesler,AngelaWroblewski,SusanneBuehrer,JackStilgoe,LenaTsipouri,NikosMaroulis&ViolaPeter,representingtheMoRRIconsortium

AbstractTherehasbeenagrowingfocus,amongbothpolicymakersandresearchers,onideasofresponsibleresearchandinnovation,asawaytoensurethatnewsourcesofpublicvaluearecaptured.TheECdefinesRRIas“aprocesswhereallsocietalactors(researchers,citizens,policymakers,business,thirdsectororganisationsetc)worktogetherduringthewholeR&IprocessinordertobetteralignR&Ioutcomestothevalues,needsandexpectationsofEuropeansociety”.Inoperationalterms,itbringstogetherundertheRRIumbrelladimensionsofengagement,gender,education,openaccess,ethicsandgovernance.RRIisahighlyrelevantareaforpolicy.ThereforetheEChascommissionedastudyon‘MonitoringtheEvolutionandBenefitsofResponsibleResearchandInnovation’(MoRRI).ItcontributesconceptualworkonRRI,providesextensiveexplorationofexistingmetricscapturingRRI,anddevelopsnewindicatorsrequiringprimarydatacollection.Indicatordevelopmentisbaseduponaninterventionlogicthatbuildsmainlyonthefivedimensionsmentionedcomplementedbythehorizontaldimensionofgovernance.Monitoringissuggestedtobuilduponasetof6x6indicatorsthatwereselectedfromabroadernumberofpotentialindicatorsthatwerevalidatedinamulti-levelprocess.Initialdatacollectionwasusedtotestingtheseindicators,wherebytheemphasisisputonusingbothquantitativeandqualitativemethodsforallareas.ThelackofadequatemeasuresofresponsibilityinresearchandinnovationhampersthemainstreamingofRRI.While‘pairing’responsibilityandscientificexcellenceisanexplicitaimfortheRRIagenda,theyareinrealityoftenperceivedascontradictorydemandsbytheindividualscientistsorviewedasunequallyimportantconcernsbytheresearchperforming-andresearchfunding–organisations.IdentificationofusefulindicatorsandmetricsforRRImightcontributetobringingissuesofresponsibilityfromaperipheralpositionandclosertothecenterofactivity.Thispaperconcludeswithareflectiononevidenceof‘benefits’orimpactsofRRI.BeingabletoidentifysuchbenefitsgoesbeyondtherathersimplisticideaofmonitoringRRIthroughindicators.Identifiablebenefitswillfurtherhelpinshapingandrefiningindicators.AndfindingacommondenominatorinabroaddiversityofbenefitshelpspolicyinstructuringfurthermainstreamingofRRI.

2

1.IntroductionInrecentyears,thenotionofResponsibleResearchandInnovation(RRI)hasemergedinEuropeanpolicymaking,asawaytoensurethatnewsourcesofpublicvaluearecaptured.Fromtheperspectiveofthe‘Sciencewithandforsociety’schemeoftheEuropeanCommission(EC),thepurposeofpromotingRRIis“tobuildeffectivecooperationbetweenscienceandsociety,torecruitnewtalentforscienceandtopairscientificexcellencewithsocialawarenessandresponsibility”.BuildingonworkbyVonSchomberg(2013)andothers(Stilgoeetal.,2013;Owenetal.,2013;PandzaandEllwood,2013;andRip2014),theECdefinesRRIas“aprocesswhereallsocietalactors(researchers,citizens,policymakers,business,thirdsectororganisationsetc)worktogetherduringthewholeR&IprocessinordertobetteralignR&Ioutcomestothevalues,needsandexpectationsofEuropeansociety”1.ConceptuallyRRIreflects(previous)strandsofactivitiessuchasanticipatorygovernance(KarinenandGuston,2010),Constructive,Real-Timeandotherformsoftechnologyassessment(Ripetal.,1995;GustonandSarewitz,2002;GrinandGrunwald,2000),upstreamengagement(WilsdonandWillis,2004),value-sensitivedesign(Friedman,1996)socio-technicalintegration(Fisheretal.,2006),socialresponsibility(GlerupandHorst,2014)andtransdisciplinarity(WicksonandCarew,2014).TheRRIdefinitionisfirmlyanchoredinEuropeanpolicyprocessesandvalues.Startingfromaratherconfinedacademicdebatecallingforresponsibleinnovation(Hellström,2013),theideaisnowpartoftheEU’sresearchandinnovationpolicyasacross-cuttingthemeinthecurrentframeworkprogrammeHorizon2020andtheEuropeanUnion’sambitiousgoaltobecomeatrueInnovationUnion,inwhichresearchandinnovationarethemaindriversofcompetitiveness,jobs,sustainablegrowthandsocialprogress.Atthesametimeanumberofgrandsocietalchallengesneedtobeaddressedaswellsuchasclimatechangeorhealthandwellbeing.OrientatingresearchandinnovationtowardssocietalgoalsisbeingreinforcedbythedebateonRRIsinceRRIemphasizesspecificqualitiesofresearchandinnovationpractices,andaimstoredefinetherolesandresponsibilitiesatscience-societyinterfaces(Nielsenetal.,2015).WhileRRIissupportingthenotionhowresearchandinnovationcancontributetoourfuturesociety,itlikewiseraisesthequestionhowtotakeintoaccountandmitigatetheunanticipated,unintendedandundesirableconsequencesofemergingscienceandinnovation.Inoperationalterms,however,theECbringsdimensionsofpublicengagement,genderequality,scienceliteracyandscienceeducation,openaccess,ethicsandgovernanceundertheRRIumbrella.ThesesixRRI‘keys’haveinformedthecompositionoftheworkprogramforSwafSandthefeaturingofRRIasacross-cuttingissueofHorizon2020,intendedtobeembeddedacrosstheprioritiesofthefundingprogramme.Sofar,effortsto‘mainstream’RRIacrosstheEuropeanresearchareahavebeenmodestlysuccessful(MejlgaardandGriessler,2016).Studiesindicatesignificantobstacles,pertainingnotleasttodisincentivizingrewardstructuresatbothorganisationalandindividuallevel2.While‘pairing’responsibilityandscientific1https://ec.europa.eu/programmes/horizon2020/en/h2020-section/science-and-society.2Moreinformationavailableathttp://www.rri-tools.eu/documents/10184/107098/RRITools_D2.2-AnalysisNeeds+ConstraintsStakeholderGroupsRRI.pdf/83c55909-118c-4cad-b7e4-74d5a770c8a1

3

excellenceisanexplicitaimfortheRRIagenda,theyareinrealityoftenperceivedascontradictorydemandsbytheindividualscientistsorviewedasunequallyimportantconcernsbytheresearchperforming-andresearchfunding-organisations.Whiletheproductionofhighimpactpublicationsis,e.g.,consideredacoreacademicactivityclearlycarryingmerit,engaginginpublicoutreachorstakeholderdialoguesmighteasilybeconsideredperipheralactivitieswithoutstraightforwardvaluefortheindividualscientists.Moreover,thelackofadequatemeasuresofresponsibilityinresearchandinnovationfurtherhampersthemainstreamingofRRI.Inabilitytoevaluate,compare,andbenchmark‘performance’intermsofRRIatthenationalaswellasdisaggregatedlevels,constitutesabarriertoanyrevisionofrewardschemesanddilutesthepotentialvitalityoftheorganisationalornational‘horserace’forhighperformanceinthisarea.IdentificationofusefulindicatorsandmetricsforRRImightthencontributetobringingissuesofresponsibilityfromaperipheralpositionandclosertothecenterofactivity.Finally,evidenceof‘benefits’orimpactsofRRIarelikelytofurthercontributetomainstreamingofRRIactivitiesatthelevelofmemberstatesandR&Iorganisations,andpossiblyalsotoincreasedattentiontothiscross-cuttingissueacrosstheH2020programmes.Againstthisbackdrop,theEChascommissionedastudyon‘MonitoringtheEvolutionandBenefitsofResponsibleResearchandInnovation’(MoRRI).ItcontributesconceptualworkonRRI,providesextensiveexplorationofexistingmetricscapturingRRI,anddevelopsnewindicatorsrequiringprimarydatacollection.Inthenextparagraphs,thispaperelaboratesonfurtherconceptuallydefiningwhatis(andwhatisnot)RRI,inordertolaythefoundationsforabroad-basedagreementforpolicyintervention;andthekey’sandareastobemonitored(par2);theinterventionlogicformonitoringandevaluationofRRIkeys(par.3);apreliminaryselectionofindicatorsacrossallkeysandreflectionondatacollection(par.4);andfinallyaconcludingparagraphofferingacriticalreflectionon,indicatorsandthe‘benefits’ofRRIandtheuseofRRIindicatorsformainstreamingofRRI(par5.).WhilethisprojectputsthingsinaEuropeanperspective,theaimisnotlosingsightofglobalevolution.Thepaperbuildsuponpreviousprojectdeliverables(MoRRI,2015)

2.TheSixRRIkeysThefirstphaseoftheindicatordevelopmentworkintheprojecthasbeencarriedoutandconsistedofreviewoftheliteratureonthesixRRIkeys.ThefirststepwastoidentifytherelevantliteratureanddocumentsdealingwithRRIissues.AsystematicreviewcoveringavarietyofRRIrelateddocuments,includingacademicliterature,ECandotherpolicydocuments,conferencesandon-goingprojects,providedcentralinsightsintoeachofthesixRRIkeydimensions,theirpolicycontextandmaindefinitionalelements.Informedbythesetheoreticalandconceptualexplorations,themainnextstephasbeentodevelopafunctionalvocabularycoveringeachofthesixRRIkeys.ThisproceduralstephasinvolvedastocktakingandassessmentofallexistingdatasourcesconsideredrelevantinthemonitoringofthesixRRIdimensions,includingreflectionsondatagapsandassessmentsoftheneedforprimarydatacollectioninthesubsequenttasksoftheproject.Thesixkeysare:

4

2.1Citizenengagementandparticipationofsocietalactorsinresearchandinnovation,inshortPublicEngagement(PE)Thesynthesisofcoreliteratureshowsthat‘citizenengagementandparticipationofsocietalactorsinresearchandinnovation’,orPE,istodayarichanddiversifiedfieldofpracticeandacademicstudies,andtheconceptofPEismultifaceted.ThePEfieldhasbeenreframedandtransformedwithinthelastdecadesandageneralturnfromone-wayandtop-downmodelsofcommunicationtowardsincreasedfocuson‘new’dialogue-basedapproachescharacterizesthedevelopmentofthefield(Baueretal.,2007).In1985,TheRoyalSocietyinLondonissuedareporton‘ThePublicUnderstandingofScience’,whichhasbeenaninfluentialdocumentinthemodernhistoryofpublicengagementwithscience.Themainmessageofthereportwasthatgivenconsiderablepublicinvestmentsinscienceandtechnology,scienceshouldbetransparentandscientistsshouldaccountforthesocietalconsequencesoftheirwork.Thereporthighlightedtheintimateconnectionbetweennationalprosperity,science,andtechnologicalprogress,andthepervasivenessofscienceandtechnologyintheeverydaylifeofcitizens.Itidentifiedaneedtostrengthensciencecommunicationeffortsfortwopurposes:first,becauseitbasicallyisnotpossibletonavigatesuccessfullyinmodernsocietieswithoutanoverallunderstandingofscience.Second,scienceandtechnologyarefundamentalforcesinthebroaderinnovationsystem,whichgeneratesprogressandprosperity.Thereisaneedtotellthisstorytothepublicinordertogenerateabroaderappreciationofscienceandtechnology(Wynne,1995).Inthisrespect,‘understanding’scienceisnotmerelyaquestionofbeinginterestedandknowledgeable,butratheraquestionofappreciatingandacknowledgingtheimportanceofscienceandtechnologyasmaindriversofeconomicandsocietalprogress.Buildingontheresultsoftheliteraturereview,afunctionalvocabularyofPEwasdevelopedthatpresentsthedefinitionsandterminologyrelatedtoPEthatwillallowanempiricalandpracticalapproachtotheconceptofPE.Fromthis,itwasconcludedthatthereisnosingularconceptionof‘engagement’andnosinglemodelofitsimplementation.RoweandFrewer(2005)developatypologyofPEmechanismsbasedonthedirectionoftheflowofinformationbetweenrepresentativesofthepublicontheonehandandthesponsorsofengagementinitiativesontheotherhand,resultinginadifferentiationbetween‘publiccommunication’,‘publicconsultation’,and‘publicparticipation’.Ravn,MejlgaardandRask(2014),informedbyaninventoryingof250specificengagementinitiativesacrossEuropeandbeyond,addtothisPEmechanismslike‘publicactivism’and‘publicdeliberation’.Together,intheMoRRIproject,theoperationalunderstandingofPEwhilerecognizingthecomplexityofobjectivesforPEandthevariationinmechanismsforengagement,distinguishesfivemaincategoriesofPE,namely‘publiccommunication’,‘publicactivism’,‘publicconsultation’,‘publicdeliberation’,and‘publicparticipation’.ThiscategorisationwasoriginallydevelopedbythePE2020project3.

3Moreinformationontheprojectavailableatwww.pe2020.eu

5

Publiccommunication–theaimistoinformand/oreducatecitizens.Theflowofinformationconstitutesone-waycommunicationfromsponsorstopublicrepresentatives,andnospecificmechanismsexisttohandlepublicfeedback(examplesincludepublichearings,publicmeetingsandawarenessraisingactivities).Publicactivism–theaimistoinformdecision-makersandcreateawarenesstoinfluencedecision-makingprocesses.Theinformationflowisconveyedinone-waycommunicationfromcitizenstosponsorsbutnotontheinitiativeofthesponsors,whichcharacterisethe‘publicconsultation’category(examplesincludedemonstrationsandprotests).Publicconsultation–theaimistoinformdecision-makersaboutpublicopinionsoncertaintopics.TheseopinionsaresoughtfromthesponsorsofthePEinitiativeandnodialogueisimplemented.Thus,inthiscase,theone-waycommunicationisconveyedfromcitizenstosponsorsontheinitiativeofsponsors(examplesincludecitizens’panels,planningforreal,focusgroupsandscienceshops).Publicdeliberation–theaimistofacilitategroupdeliberationonpolicyissues,wheretheoutcomemayimpactdecision-making.Informationisexchangedbetweensponsorsandpublicrepresentativesandadialogueisfacilitated.Theflowofinformationconstitutestwo-waycommunication(examplesinclude‘minipublics’suchasconsensusconferences,citizenjuries,deliberativeopinionpolling).Publicparticipation–theaimistoassignpartlyorfulldecision-making-powertocitizensonpolicyissues.Informationisexchangedbetweensponsorsandpublicrepresentativesandadialogueisfacilitated.Theflowofinformationconstitutestwo-waycommunication(examplesincludeco-governanceanddirectdemocracymechanismssuchasparticipatorybudgeting,youthcouncilsandbindingreferendums).

2.2Scienceliteracyandscientificeducation(SLSE)Scienceliteracyandscientificeducationhavebeentopicsofacademicandpublicdiscussionsforalongtimeandcontinuestodoso,especiallyinthelightofthechallengesofmodernsocieties.Givenitslonghistorythefieldis,ononehand,wellresearched.Ontheotherhand,substantialquestionsstillremaintobeanswered.Thesubjectofscienceliteracyonlybecameatopicofsystematicstudyinthe1930swhenJohnDeweyarguedthat“theresponsibilityofsciencecannotbefulfilledbymethodsthatarechieflyconcernedwithself-perpetuationofspecializedsciencetotheneglectofinfluencingthemuchlargernumbertoadoptintotheverymake-upoftheirmindsthoseattitudesofopen-mindedness,intellectualintegrity,observationandinterestintestingtheiropinionsandbeliefs,thatarecharacteristicsofthescientificattitude.”(Dewey,1934)Overthepastfivedecades,thefieldofscienceliteracyandscienceeducationhasexperiencedatleasttwomajorshifts,leadingtothecurrentco-existenceofthreeparadigms:thedeficitmodel(i.e.theideathatscienceis“toocomplicated”forthegeneralpublictounderstand),thedialoguemodel(i.elaypeoplehaveknowledgeandcompetencieswhichenhanceandcompletethoseofscientistsandspecialists”(Bucchi,2008;Callon,1999)andtheparticipationmodel(orco-productionofknowledge)wheretheneedandrightofthepublictoparticipateinthediscussionhasbeenputtotheforefront.ThedialoguemodeldevelopedfollowingconclusionsintheinfluentialRoyalSocietyreportonPublicengagement(1985).Thereportarguedthatthedeficitofthepublicwasderivinglessfromknowledgebutratherfromattitudes.Itwasclaimedthatthepublic’sattitudetowardsscienceandtechnologyisnotsufficientlypositiveandthatdangersexistedthatcitizenswouldbecomenegativeorevenespouseanti-science

6

attitudes(Baueretal,2007).Inordertorecoverthedeficits,activitiesinthisperiodfocusedonchangingattitudesandmarketingtheimageofscience.Forthepurposeofthisprojectwedefinescienceliteracyastheabilityofcitizenstoreadabout,comprehendandexpressopinionaboutscience,aswellastheabilitytocontributeto“doingscience”.Bybuildingonthisidea,thefocusofourunderstandingofscienceliteracyisputontheideaofdevelopingcapacitiesforscienceandinnovation.Scienceliteracyisgeneratedthroughactivitiesaimingtoprovidecitizenswithadeeperunderstandingofscience,toshapetheirattitudestowardsscienceandtodeveloptheirabilitiestocontributetoscienceandscience-relatedpolicy-making.TheoperationalunderstandingofSLSEappliesatripartitecategorisationforthemultifacetedfieldofscienceliteracy.Scienceliteracycanbegeneratedthroughthreemainmechanisms:Scienceeducationaimsateducating(especiallyyoung)citizensaboutscientificfacts(textbookknowledge),thenormsofscienceandthewayscienceis‘done’aswellasatconveyingapositive‘image’ofsciences.However,italsoprovidestheopportunitytoreflectandquestionscienceandthe‘truths’itproducescritically.Ittakesplaceininstitutionsinearlychildhoodeducationandcare,theschoolsystem,highereducation,vocationaleducationandtrainingaswellasinlifelong-learning.Scienceeducationisthebasisforscienceliteracy.Sciencecommunicationactivitiesaimateducatingcitizensofallagesaboutscienceaswellasatgeneratingawarenessofscience-relatedissuesandapositiveimageof/attitudetowardsscience.Theseactivitiescantakedirectforms(forinstancethroughopendays,museumsorsciencecentres)orbemoreindirectwithmediatorsbetweenthescientistsandthepublic(e.g.viasciencejournalistsandtheirproductssuchasTVprogrammesormediaarticlesetc).Generally,alargenumberofdifferentinstitutionsareinvolvedinsciencecommunication.Co-productionofknowledgeischaracterisedbyaco-creationofknowledgethroughcooperationofscientificexpertsandnon-experts.Includingtheco-productionofknowledgeinthedimensionofSLSEaltersthewaywethinkaboutthepublicanditsroleinscienceandinnovation,fromamerereceiverandcustomertoanactiveagentofchange.Citizensco-producescientificdata,possiblyhelpintheirinterpretationandanalysisandframeresearchquestions.Theaspectofco-productionofknowledgeisclearlyinterlinkedwithmechanismsandactivitiescarriedoutwithinthefieldofpublicengagement.

2.3Genderequality(GE)Sincetheturnofthecentury,thetopicofgenderequalityinscienceandresearchhasbeenintensivelydiscussed.Accordingly,abroadrangeofliterature,pilotprojectsandempiricalevidenceisavailablewhichdealswithgenderinequalitiesinthisarea.Thisprovidesthestartingpointforthediscussionongenderwithinresponsibleresearchandinnovation(RRI)andthedevelopmentofindicatorsforthegenderdimensioninRRI.Sincetheearly1990s,thepresenceofwomeninsciencehasgainedincreasinginterestinpoliticalaswellasscientificdebate.Thisdebatewasinitiallysupportedby callsforsocialjusticeandwasembeddedinthedevelopmentofgeneralantidiscriminationpoliciesatbothnationalandEuropeanlevelaimedatestablishingequalrightsfor

7

womeninemployment.Correspondingresearchfocusedonthecareerpathsofmenandwomenaswellasonthecomplexinterplaybetweentheinstitutionalarrangementsandpersonalpreferencesthatmightservetoexplaintheunderrepresentationofwomen,especiallyatthetoplevels(Caprileetal.,2012).Sincetheturnofthecentury,economicargumentshavealsobeenusedincreasinglytojustifygenderequalitypolicies:- IntheEuropeanCommission’s(EC)view,realizingEurope’sambitiontoachievea

competitiveknowledge-basedsocietywillrequireanincreaseinthenumberofresearchers(EuropeanCommission:TheWake-UpCallforEuropeanIndustry2003).Inordertoachievegoalslike‘competitiveness’,‘innovation’anda‘knowledge-basedsociety’,itisevidentthatthetalentsandpotentialofwomenhavetobedeveloped,mobilized,leveragedandusedmoreactively,deeplyandcompletely.

- Fromthescienceandtechnologyperspective,’genderedinnovations‘enhanceexcellenceinscience,medicineandengineeringbothintermsofknowledgeandpersonnel.‘Genderedinnovation’isdefinedastheprocessthatintegratessexandgenderanalysisintoallphasesofbasicandappliedresearchtoassureexcellenceandqualityinoutcomes(SchiebingerandSchraudner,2011).

- Inthebusinesssector’sview,thereasonswhygenderdiversityshouldbetakenintoseriousconsiderationlieinwomen’stalents,theeconomicpowerofwomen,thechangingmarketstructureandthepositiveimpactofwomenonorganizationalexcellenceandfinancialperformance(Catalyst,2004;McKinsey,2007).

Theissueoftheunder-representationofwomenintoppositionsbothinacademiaandinthebusinesssectoriswidelydiscussed.Theliteratureinterpretsandexplainsthispersistentgendersegregationatthreelevels:theindividuallevel,theinstitutionallevelandthesocial/culturallevel.In2007,theEuropeanCommissionchangeditspolicyapproachfrom‘fixingthewomen’to‘fixingtheinstitutions’inlinewiththeprocessrelatedapproachofgendermainstreaming(Lipinsky,2014).Asaconsequenceofalltheabove,abroadpolicymixhasbeendevelopedtosupportwomenandovercomegenderedstructures.Thisincludesboththeprovisionofcareersupportforwomenaswellasinstitutionalmeasures.Followingtherecentpoliticalandscientificdiscourse,theoperationalunderstandingofgenderequalityisdefinedasathree-dimensionalconstructaddressing1)the(under-)representationofwomeninresearchandinnovationwiththeobjectivetoreducegendersegregation,2)thestructuralandorganisationalchangesinresearchinstitutionswiththeaimtobreakdownstructuralgenderbarriersbymeansofactionplans,genderbudgeting,amongothersactions,and3)theinclusionofgenderinR&Icontent,therebyaimingat:• Integrationofwomeninallfieldsandatalllevelsinresearchandinnovation

(reductionofhorizontalandverticalsegregation);Thiscomprisesmeasurestopromotewomeninfields,wheretheyareunderrepresentedaswellastoincreasefemaleparticipationinmanagementanddecision-makingpositions.Thegoalhereistoreducegendersegregation.

• Structuralchangeinresearchinstitutionsinordertoabolishstructuralbarriersforwomen.Thiscomprisesstructuralmeasuresaimedatrevisingexistingorganisationalarrangementstoprogressivelyeliminatebarriersimpedingwomen’sadvancementtotoppositionsandfactorsinducingwomentodropoutofscience.

• Integrationofgenderinthecontentofresearchandinnovationtoensurethatwomen’sneedsandinterestsareadequatelyaddressed.Explicitgenderissuesarerarelyincludedinthecontent,anditisarguedthatresearchresultsarenotvalidor

8

reliableiftheyonlyconsidermaleresearchsubjects.Thisislegitimisedbythegendermainstreamingstrategyontheonehandandbyqualitystandardsinscienceandresearchontheother(Caprileetal.,2012).

Thegenderequalitykeyiscloselyconnectedwiththeethicsandgovernancedimension.

2.4OpenAccess(OA)Historically,opensciencerelatestotheneedtobuildapubliclyrecognisedreputation.Thescholarlytraditionofopenknowledgewasturnedintoaprocedureforestablishingknowledgeclaimsthatcouldbeevaluatedandrecognisedbypeersandthenutilizedbyothers.Knowledgewasconsideredapublicgood,andlikewiseapublication(anykind)aswell.Sincethen,propertisationofknowledgeoccurredthroughcopyrightimposedbytheacademicpublishingmarket.Alackofpolicycoordinationand/orframeworkconditionsimpededthefreemovementofresearchactivitiesandknowledge,hinderingaccesstopubliclyfundedresearchresultsandknowledgetransfer.Adiversityofnationalpolicies,legalrequirementsandpracticesregardingknowledgetransferaswellasopenaccesstoscientificpublicationsandscientificdataadverselyaffectedthewiderdissemination,accesstoanduseofknowledgecreatedwithpublicfunds.Open,however,shouldapplytoallcomponentsoftheresearchprocess,andnotberestrictedtotheoutcomesonly.Openwillthenneedtobeembeddedintheresearchprocessfromstarttofinish.Suchchangesmayimpacttheentireresearchcycleanditsorganisation,fromtheinceptionofresearchtoitspublication.Researchorganisations,fundingbodies,careerpathsofresearchersareaffectedaswellasthesciencesystem,e.g.theriseofnewscientificdisciplines,newpathwaysinpublishing,anddifferentscientificreputationsystems,andhowthequalityandimpactofresearchisevaluated.Openaccess(OA)istheideaofmakingresearchresultsfreelyavailabletoanyonewhowantstoaccessandre-usethem.Nowadays,itiswidelypropagatedthatmakingresearchresultsmoreaccessiblecontributestobetterandmoreefficientscience,andtoinnovationinthepublicandprivatesectors(EC2012,2014).In2003theBerlinDeclarationonOpenAccesstoKnowledgeintheSciencesandHumanitiessignaledthestartofopenaccesspolicies,followedbytheOpenAccesspilotinitiativewithinthemainstreamFrameworkProgrammeitself(inadditiontotheERC)whichwaslaunchedbytheECin2008.Subsequently,thepolicymovementtoenhanceOpenAccesstoresearchoutputsledtothelaunchofaseriesofprojects;oneofthem,fundedundertheECResearchInfrastructuresprogramme,istheOpenAIREpilotaction4.In2012,theEuropeanCommissionencouragedallEUMemberStatestoputpublic-fundedresearchresultsinthepublicsphereinordertomakesciencebetterandstrengthentheirknowledge-basedeconomy.Asotherchallengesneedtobeaddressedsuchasinfrastructure,intellectualpropertyrights,content-miningandalternativemetrics,butalsointer-institutional,inter-disciplinaryandinternationalcollaborationamongallactorsinresearchandinnovation,theEuropeanCommissionisnowmovingdecisivelyfrom‘Openaccess’intothebroaderpictureof‘Openscience’inorderto“enhanceopencirculationofknowledgeacrossnationalborders,includingknowledgetransfer”.TheassumedbenefitsofOpenSciencewillbemoretransparency,openness,andnetworkedcollaboration.Inthelongterm,Europeanpolicyhopestomaymakesciencemore

4Moreinformationontheprojectavailableathttps://www.openaire.eu/

9

efficient,reliableandresponsivetotheEuropeanGrandChallengesofourtimesaswellasfosterOpeninnovation.TheoperationalunderstandingofOAinthecontextoftheMoRRIproject isreviewedfromthreeseparateangles:Startingfromthegeneralpolicyconceptofopensciencethatfocusesonalackofpolicycoordinationand/orframeworkconditionsimpedingthefreemovementofresearchactivitiesandknowledge,hinderingaccesstopubliclyfundedresearchresultsandknowledgetransfer(vandenEyndenandBishop,2014),twospecificaspectsareofinterest:‘TheOpenAccessinstrumentforpublications’and‘DevelopmentsinOpendata’.• OpenAccesspublications:ThetermOpenAccessoriginallyreferredtothe

provisionoffreeaccesstopeer-reviewedacademicpublications.The‘AssessmentofOpenAccesspublishing’iscomplicatedbythegrowingdiversityofwhatcountsasOpenAccess,thecopyrightrestrictionsforwhenapublicationcanbemadeopenlyaccessible,andthelackofclearandconsistentidentificationofOpenAccesspublicationsinbibliographicdata.TherearetwoformaloperationalpathstoaccessthroughOpenAccessjournalsandself-archivinginrepositories,subsequentlyreferredtoasGoldOpenAccess(openaccessjournals),andGreenOpenAccess(alsocalled‘self-archiving’).Theimportantandon-goingdebateintheassessmentofOAiswhetherOApublishingyieldsincreasedcitationimpact.WhilethereareconflictingreportsinthedebateaboutanOpenAccesscitationadvantage(OACA),heightenedattentiontothisissuehasincreasedourunderstandingaboutOpenAccesspublishingmoregenerally(Björketal.,2009;Archambaultetal.,2014).

• OpenResearchData(OD)isarelativelynewandemergingfieldofscholarship,andsystematizeddatasourcesarestillfairlyscarcecomparedtothedataavailabilityonissuesrelatedtoOpenAccess(Costasetal.,2013;Farhanetal.,2013).Dataanddatasetsarecentralforempiricallyorientedscienceandscholarship.Theycanbeverydiverse(e.g.archaeological,biological,genetic,economic,mathematical,astronomic,etc.)andoncecollected,thesamedatacanbeusedbyavarietyofresearchersfromdifferentinstitutes,disciplinesandorganisationstoproducenewresults.Opendataanddatasharinghaveproventocreatemanybenefits,suchasstrongeropenscience;ahigherefficiencyintheuse(andreuse)ofscientificresources;thepossibilitiesofexpandingnewresearchlines.Despitethesemainbenefits,therearecultural(e.g.trustandcontrol)andtechnical(e.g.standardsandmetadata)issuesthathamperwidespreaddatacuration,sharinganduse.

TheotherbigchallengeforOAisre-unitingpublicationswithpublicfundingandtointerlinkpublicationswithresearchdata.Suchimprovedinterlinkingwouldallowfortheinvestigationofresearchresults,beyondthelimitsofproject-basedfunding,andprovidethedataneededfortheexplorationoflonger-termresultsofpublicresearchfunding.

2.5EthicsTheterm“ethics”israrelydefinedcomprehensively;itseemsthatmostauthorsassumethatthereaderalreadyknowswhatethicsinthecontextofresearchandinnovationisabout(Brometal.,2015).Oneexampleofadefinitionis“Acommonplatformfordeliberationanddiscussionofvaluesinsociety,thatisbasedonperceptionsofrightandwrong,isinfluencedbyculturalnorms,andaimsatinformingpolicymaking”(Ladikasetal.,2015).Othersmakeadistinctionbetweenethicsasascientificdisciplineandmoral.

10

GriesslerandLittig(2006)followtheliteratureinstatingthat:“Ethicsasascientificdisciplineisconcernedwithnormativerulesforeverybody,whichotherthanmoralshouldbeusedtoevaluateandnottoguideactions”Schicktanzetal.(2012)definemoralityas“asetofrulesandvaluesactuallyguidingindividuallifeandsocialinteraction–andethicsasitsnormativereflection,justificationorcritiqueinviewofvalidity,desirabilityandlegitimacy”.Ethicsisdiscussedindifferentpoliticalcontexts(global,European,national),reflectingtheinstitutionalizationofethicsinscienceandinnovation.Butethicscanbediscussedattheinformallevelaswell(Feltetal.,2009),e.g.byengagingexpertsandlaypeopleinasymmetricaldiscussionoftheethicsofparticularbiomedicalfieldsofresearch.KeyqualitiesofethicsinthecontextofR&Idemandedbymanyauthorsareprocessqualitiessuchas:1.Opennesstowardsstakeholdersandthepublic;2.Publicparticipation(includinginformation,consultationof,andwithdeliberationpublic);3.Transparencyandaccountabilityofprocesses;4.Thematicopennessintermsofwhichquestionscanberaised;5.Systematicargumentationintermsofapriorityofarguingover(political)bargaining(thisalsoincludesscholarlyintegrity).Theoperationalunderstandingofethicsinresearchandinnovationanddelineationoftheinstitutionalizationofethicsiscategorizedin:‘ethicalgovernance’,‘ethicaldeliberation’,and‘ethicalreflection’.Ethicalgovernance:I.e.“institutionalisingethicsdebateintermsoftheimplementationofstandardsinresearchethicsinscience,technologyandinnovationpolicies”(Brometal.2015).Intheareaofethicalgovernance,ethicscommissionsareamajorgovernanceinstrument,advisinge.g.nationalgovernmentonpolicymaking.Anotherimportantinstrumenttogovernethicsinscienceandtechnologyareethicalcodesandsoftlaw.Ethicaldeliberation:I.e.“institutionalisingethicsdebatethatraiseissuesinscienceandtechnologicaldevelopmentsinscience,technologyandinnovationpolicies”.Oneimportantinstrumentforethicaldeliberationistechnologyassessment(TA).Ethicalreflection:I.e.“institutionalisingethicsdebatethatsupportcriticalreflectionandengagementindebatesonresearchstandards,emergingtechnologyissuesandsocialjusticeinscience,technologyandinnovationpolicies”.Insteadoffocusingonadvisorycommitteesethicalreflectionisadvocatedthroughroundtableformatstobringtogetherexpertsandlaypeople.

2.6GovernanceTherelationshipbetweengovernance,researchandinnovationisfarfromsimple,andfarfromlinear.Notonlyarescienceandinnovationgovernedinvariousways,someofwhichmaybeconsideredmoreresponsiblethanothers,butalsoscienceandinnovationareavitalandincreasingpartofourgovernanceregimes.‘Governance’herereferstocontrolormanagement.Itcanbefoundnotjustinthestate,butalsoinbusinessesoranysocialorganization.Inthissense,governancegoessubstantiallybeyond‘government’.HerewetakeadefinitionfromBorrásandEdler(2014),whodefinegovernanceinrelationtoSTIasthe"[...]wayinwhichsocietalandstateactorsintentionallyinteractinordertotransformST&Isystems,byregulatingissuesofsocietalconcern,definingprocessesanddirectionofhowtechnologicalartefactsandinnovations

11

areproduced,andshapinghowtheseareintroduced,absorbed,diffusedandusedwithinsocietyandeconomy."Governanceinvolvestheestablishmentofgoals,settingupofmeans,andverificationofperformance.Thisincludesproviding,distributing,andregulating.Whenappliedtoscience,governanceisofteninterpretedtomean‘regulation’,suggestingarestrictionoffreedom.Regulationisasubsetofgovernancefocusedonsteeringandconfirmingtrajectories,asopposedtoprovidinganddistributing.Thegovernanceofscienceisconcernedwithhowknowledgeisproducedandhowitisdistributed.Thegovernanceofinnovationisfarbroader.Theconcernwiththelattergoesbeyondconcernsabouttechnologyanditsregulation.Scientistsgovernhowknowledgeisproduced,certified,madecredibleandcommunicated.RRI,then,isclearlyalignedwithparticularideasofgovernanceandasetofinnovationstowardsthisend,suchaspublicdeliberation,laymembershipofexpertcommittees,transparencyandmultidisciplinarycollaboration.Inparticular,RRIgovernanceshouldbeseeninthelightofrecentmovesandframeworksaimingtowards‘anticipatorygovernance’(Barbenetal.,2008;Guston,2014).Theoperationalunderstandingofthehorizontaldimensionofgovernanceisdefinedassteeringinnovationthroughtheestablishmentofgoals,theestablishmentofmeansandtheverificationofperformance.Particularlyimportantinthecaseofthegovernanceofscienceistherealisationthatmuchgovernancehappenswithinandisdonebythescientificcommunityitself.Belowwecategoriseatypologyofgovernanceapproachesthathelpstostructurediscussionsaboutchanginggovernance(adaptedfromHagendijkandIrwin,2006):Discretionarygovernance:Policiesinthiscategoryaremadewithoutexplicitinteractionwith‘thepublic’.Governanceispresentedprimarilyasamatterforgovernment,whichisseenasservinguniversalgoalsofprogress.Corporatistgovernance:Thisinvolvesaformalrecognitionofdifferencesofinterestasaninputtonegotiation.Asnegotiationtakesplacewithinaclosedorhighlyregulatedspace,thedecisivefeatureofthismodeistheadmissionofstakeholders.Educationalgovernance:Thisassumesthatpoliciesforscienceandtechnologyhavefounderedontheshoalsofpublicignorance.Hence,itisnecessarytocreateaninformedcitizenry.Marketgovernance:Scienceandtechnologyarebestregulatedbydemandandsupply.Thevalueofsciencecomesfromthesurplusvaluecreatedthroughitscommercializationandcontributiontothegenerationofwealth.Thepublicparticipatesascustomersandconsumers.Agonisticgovernance:Thisformofgovernanceoccursinacontextofconfrontationandadversity.ThestorageofnuclearwasteintheUKisacasewherepolicyseemstohavestalledinthefaceofpublicopposition:oppositiontoGMfoodshasalsotakenagonisticform.Deliberativegovernance:Thisrestsontheassumptionthatopendebateandengagementcancreateasatisfactoryfoundationfordecision-making.Inthismode,thepublicarenotconsumersofscience,butrather‘scientificcitizens’.

12

3.TheinterventionlogicformonitoringandevaluationofRRIkeysIndicatordevelopmentisbaseduponaninterventionlogicthatbuildsmainlyonthesixkeysmentioned.Monitoringissuggestedtobuilduponasetofappropriateindicatorsthatareselectedfromabroadernumberofpotentialindicatorsthatwerevalidatedinamulti-levelprocess.Initialdatacollectionisusedtotestingtheseindicators,wherebytheemphasisisputonusingbothquantitativeandqualitativemethodsforallareas.Thefirstphaseoftheindicatordevelopmentworkintheprojecthasbeencarriedoutandconsistedof:• Assessmentsoffinishedandongoing(inter)nationalandECprojectstomakean

inventoryonusedandpotentialindicatorsforthesixkeys;• Theextentofdataavailabilitywithinseparatekeycategorisations/typologiesas

describedabovewasassessedthroughacross-readofdataexplorationsvis-à-visfunctionalvocabularies

Theliteraturereviewtogetherwiththeassessmentofprojectsanddataavailabilityhaveresultedinatentativelistofaltogether98existingindicators/datasourcesthatwereconsideredpotentiallyusefulinmeasuringandcapturingcoreaspectsofRRI.Inthenextstep,thesepromising98existingindicatorsanddatasourceswereclassifiedandassessedwithrespecttotheirrelevance/proximityandrobustness/qualityasindividualmeasuresofRRIatspecifiedkeysandlevelsofanalysis.Moreover,anaggregateassessmentandclassificationoftheoverallrelevance/proximity,robustness/quality,anddatarichnessoftheavailableindicatorshavebeenconductedaswell.TodevelopasolidconceptualframeworkcapableofaddressingthecomplexnatureofRRIinthebestpossiblemanner,theMoRRIprojectborrowsideasfromtheevaluationliterature.Itintroducesthe‘interventionlogicmodel’asastartingpointformonitoringvarioustypesofimpactsandbenefitsofRRI.Theinterventionlogicmodelisbasedontheexplanatoryideathatcomplexpolicyproblemsarecharacterisedbyaseriesofissuesorproblemsthatneedtobeaddressed,asetofinputswhichareappliedtoaseriesofactivities,whichgenerateoutputswhichinturnleadtooutcomesortheresolutionoftheproblems.Thislogicinformsandrelatestoa‘theoryofchange’,i.e.anassumptionorhypothesisofwhyaninterventionwillsucceedinproducingdesiredoutcomesandimpact(s).A‘theoryofchange’specifieshowactivitiesareexpectedtoleadtointerimandlongertermoutcomes.Theelaborationofthetheoryofchangecanstrengthenthecaseforattributingobservedchangestoanactionandisthusanimportantaspectofthemethodofapproachtotheevaluation.Animportantaspectofthelogicmodelistheidentificationanddescriptionofkeycontextual,externalfactorsthatcouldinfluencetheinterventioneitherpositivelyornegatively.ThemaincharacteristicsoftheelementsoftheinterventionlogicmodelappliedintheMoRRIprojectcanthusbesummarisedasfollows:• Contextindicatorsprovideinformationontheenvironmentandoverallsituationina

countryandacrosscountries;• Inputindicatorsconcerntheactivitiescarriedout,measurestaken,structures

createdandresourcesallocatedtopromoteRRI.ThistypeofindicatorsfocusesonthesystemandconsistencyoftheRRIrelatedinitiatives.

• Outputindicatorsaddresstheimmediateanddirectresultsoftheseactivities,whileindicatorsofoutcomescrutinizethemorefar-reachingandlong-termachievementsandperceivedbenefitsoftheRRIwork.

13

Amainconclusionderivedfromthesynthesisandassessmentofthe98promisingRRIindicators,isthattheinputandoutputdimensionsoftheinterventionlogicmodelcomprisethemostcomprehensiveandsaturatedindicatorsofRRI.ParticularlytheRRIdimensionsofPublicEngagementandGenderEqualitycontainhighlyrelevant,robustanddatarichinputlevelindicators,whereastheavailableOpenAccessandEthicsindicators,whilerelevant,aremoredifficulttoassessintermsofrobustnessandrichnessofdata.Moreover,adatagapexistsfortheScienceLiteracyandScienceEducationdimensionattheinputlevel.Figure1:Existingandpotentialinterlinkages/overlapsbetweenRRI(sub-)categories(Source:MoRRIdeliverable3.1)

InterlinkagesamongRRI(sub-)keys/categoriesWhilethesixkeysindividuallyprovidedanumberofpotentialindicators,thekeysarenotindependentofeachother.Asdescribedabove,interlinkagesexistamongandbetweentheconceptualcategoriesofeachRRIkey.Inordertoaccountfortheanalysisofsuchinterlinkages,thefollowingquestionswereaddressed:•Aretheanalyticalsub-categorieswithineachRRIkeysufficientlycovered?•Whenconsideringinternaloverlapsamongsub-categories,andtherelevanceofeachsub-categories,doallcategoriesneedtobeequallywellcovered?•Dosub-categoriesneedtobeequallywell-representedatalllevelsintheinterventionlogicmodelandacrossaggregationlevels?•TowhichextentdoRRIkeysandtheirrespectivesub-dimensionsintersect,andhowdosuchintersectionsinfluencedatacoverage?Tofurtherguidethecollectionofprimaryandsecondarydata,whileensuringconsistencyacrossthesixRRIkeysatdifferentintervention-anddatalevels,existing

14

andpotentialinterlinkagesbetweenRRIdimensionsandsub-dimensionsaredepictedinfigure1.Thecolouredarrowsreflectwhetherinterlinkagesaredirectlyaddressed(green)intheexistingsetofindicators,whereasthepurplearrowsdisplaypotentialinterlinkagesthatcouldbefurtherexplored.Thedirectionofthearrowsindicateswhethertheinterrelationsarereciprocalornonreciprocal.SeveraloftheindicatorsaddressingthePublicCommunicationaspectofPE,forinstance,bearclearrelevanceto,andrevealactualoverlapswith,theSLSEsub-categoryofsciencecommunication.Similarly,arangeofotherpotentialinterlinkagescouldfruitfullybefurtherexplicatedandexploredwhenidentifyingandestablishingthefinalsetofindicators.

4.ApreliminaryselectionofindicatorsacrossallkeysandreflectionondatacollectionThemoderatedlistofindicatorsandtherelatedassessmentprocessgaverisetocardinalquestionsconcerningtheadequacyandaccuracyoftheindividualandcompiledsetofindicatorsincoveringthegeneralconceptofRRIanditssixkeys.Inajointworkshopin2015amongtheprojectpartners,themaincriteriaforselectingindicatorswerediscussedaswellasthefollowingissues:[a]doesthemoderatedlistofindicatorsadequatelycoverpotentialindicatorcandidatesforsustaineddatacollection(i.e.timeseriesdata)?[b]Willitbepossibletocompileasetofdatacollectionmethodsthroughwhichseveralmetricsacrosskeyscanbeharvested?[c]DoesthemoderatedlistofindicatorsfullycaptureEU28whilealsoadequatelycoveringtheconceptofRRI?Thejointdeliberationsresultedinacommonunderstandingoftheframeworkforthefinalindicatordesign,takingintoaccounttheviewofanumberofECrecommendations.Themaincriteriatoguidetheidentificationandconstructionofthefinalsetofindicatorsare:•EU28coverage:Ideally,thesetofRRIindicatorsshouldadequatelycoverEU28forthesakeofrepresentativeness.Inapolicyperspective,thecomparativeelementalsoremainssignificantasameanstogaininsightsintocountryperformancevis-à-visresponsibleresearchandinnovation.•Possibilityforsustaineddatacollection:Indicatorsthatcouldbecandidatesforlong-term,repetitivedatacollection(beyondthelifespanoftheproject)shouldbepursuedandtakenintoconsiderationinthedesignoftheprimarydatacollection.Afewindicatorsarenotconsideredwellsuitedforsustaineddatacollection.Inmostcases,however,theseindicatorsarebasedonsecondarydata,whicharefairlyeasytoharvest.•RRIconceptualcoverage:Theindicatorsshouldadequatelycoverallsixkeys.Ideally,anumberofindicatorsshouldbe‘multi-dimensional’incharacter,sincesuchindicatorsareconsideredbetteratcapturingthecoreRRInotion.Itshouldbenotedthatitwillbeempiricallytestedhowindicatorsclustertogetheroncethedataisavailable.•Focusoninputandoutputindicators:Basedontheassessmentofexistingindicators/secondarydata,itisrecommendedthatMoRRIshouldfocusontheactuallyimplementedRRIactivities(inputs)aswellastheoutputsgenerated(output).InlinewiththeRRIexpertgroup(EC2015),theargumentissupportedofmaintainingafocusonthe‘shortandmidtermeffects’,whileunderliningthat‘impactevaluationisshiftingfrom(end)producttoprocess,andfromverdicts/judgmentstolearningandimproving’.

15

•Balancebetweentargetedsocialactors:LikethedifferentRRIkeysneedtobeadequatelyrepresentedinthefinalsetofindicators,sodothevarioussocietalactorsandstakeholders,bothindividuallyandcollectively.Forinstance,iftheprojectistoreachasetofreasonablycomprehensiveRRIindicators,keysocialactorsrepresentingcivilsociety,researchcommunities,governmentalinstitutions,businessandindustry,amongothers,needtobeaddressedwithregardtotheirvariousrolesinresearchandinnovationprocesses.•Qualitativeandquantitativedata:Consideringtheunchartedfieldofresponsibleresearchandinnovation,andconsequentlythedearthofcomprehensivesetsofRRIindicators,qualitativedataenablingin-depthunderstandingsofthecomplexandmulti-dimensionalRRIfieldareseenasvitalintheidentificationandconstructionofindicators.Suchapproachesdohowever‘rarelyprovidedatathatarestraight-forwardlyapplicableintermsofbenchmarking,andforthepurposesofthemonitoringstudy,itwillbenecessarytotranslatequalitativematerialinto‘quantitative’indicatorsandmeasuresthatallowforcomparisonsacrosscountries’(Davisetal.,2012).•Generalqualityassessmentcriteria:Thecriteriaofrobustness/qualityconcernthevalidityandreliabilityofindicatorsinmeasuringspecifiedkeysandanalyticallevelsofRRI.TheissuesofvalidityandreliabilitydohoweveralsoconstitutemoregenericqualitycriteriathatneedtobetakenintoconsiderationintheconstructionofRRIindicatorsingeneral.Themostimportantaspectsstatedbelow:-Contentvalidityconcernstheextenttowhichthecontentortheoreticalconstructoftheindicatormatchesthecontentdomainithasbeendefinedtomeasure(Hertogetal.2014).Theissueofindicatorattributionconstitutesacrucialelementinensuringthecontentvalidityoftheidentifiedmeasures.-Reliabilityconcernsthequality,consistencyandcomparabilityoftheunderlyingdataformingthebasisoftheidentifiedindicators.AsmentionedbyDenHertogetal.(2014),onemajorissuewhendrawingonsecondarydataisthat“eachcountryusesitsownspecificdatasets,measurementmethods,anddefinitions.AlthoughsupranationalorganisationssuchastheOECDandEUhavemadegreatprogressinunifyinginternationaldatacollection,substantialdifferencesexistbetweencountries.Thebasicproblemisthatitissometimesdifficulttotellwhether(ortowhatextent)thedifferencesinamodelbetweencountriesarerealorratherconstructsduetodifferencesinmeasurements”.Inotherwords,itseemscrucialtoaccountfortheactualconsistencyandcomparabilityoftheavailableaggregatedataderivedfromcountry-specificdata-sets.-Indicatorcoveragebiasaimstoclarifywhetherabiasexistsinthestructureofthedataitself(DenHertogetal.,2014).ThelackofcoverageofthehumanitiesandpartsofthesocialsciencesinThomsonReuters’WebofScience,mayforinstanceleadtostructuralbiasintheotherwisehighlyrelevantmeasuresofdevelopmentsinOpenAccesspublicationsacrossscientificdisciplinesandcountries.-Externalvalidityaddressestheextenttowhichthedatacollectedonthebasisoftheindicatorsareprovidinginformationthatisgeneralizabletoabroaderpopulationofcases,situationsorpeople.Thelistof98promisingindicatorsoutlinedforthekeyswasassessedintermsofrobustness,richnessandRRIrelevance–thelatterasdefinedwithinthecontextofMoRRI,withtheaimoffurtherdelineationandselectionofalimitedsetofindicators.Tothisend,thefollowingconsiderationsapplied:

16

• Dataquality,coherenceandavailabilitywereconsideredaccordingtotheinterventionlogicmodelaswellasthelevelsofaggregationofthesepromisingindicators.Morespecifically,datacoveragewasconsideredintermsoftheavailabilityofdataacrossdimensionsrelatingtotheircharacteristicsintermsoftheinterventionlogicmodel,i.e.datadescribingthecontext,input,output,andoutcome.Contextrelatestotheenvironmentandoverallsituationinacountry;inputtotheactivitiescarriedout,measurestaken,structurescreatedorresourcesprovidedtoaddresswhatisdoneinordertoaddressissuesofRRIandwhetheritisdoneinasystematicmanner;outputstotheimmediateordirectresultsofactivitiesandoutcomesrelatetotheachievements.Additionally,availabilityofdataisdescribedaccordingtothelevelofaggregationofthesedata,distinguishingdatathatdescribethegloballevel,thenationallevel,theregionallevel,theinstitutionallevel,theprogramme/projectlevelandtheindividuallevel.

• Adiscussionofdatagapsandrequireddatacollectionatdifferentinterventionanddatalevelstoguideanddirecttheprimarydataspecification;

• IntermsofthepillarsofRRI,thekeyofgovernance–whilestillconstitutingaseparatekey–alsofunctionsasanoverarchingdimensionor‘umbrella’conceptfortheremainingkeysInthisregard,agreatnumberofindicatorsidentifiedwithintheotherfivekeysrelatedirectlytothegovernanceofresearchandinnovation,furtherindicatingthatthiskeycanbetreated‘asanoverarchingconsiderationacrosstheotherkeysofresponsibleresearchandinnovation’.(EuropeanUnion2012).

Theconsiderationsintermsofrobustness,richnessandRRIrelevance,andwithregardtoEU28coverageandpotentialforsustaineddatacollection,resultedintheconstruction,identification,andspecificationofrelevantmetricsandindicatorstobeusedinthesubsequentRRImonitoring.Basedupontheprevioussynthesisandassessmentofexistingindicatorsandsecondarydata,wehave[a]pinneddownthefinal,coresetto6indicatorsforeachofthe6keysofRRI,[b]providedasdetaileddescriptionsofeachindicatoraspossible,and[c]specifiedtheprimaryandsecondarydataandproceduresguidingthedata-collection.ItshouldbenotedthattheindicatorsselectedtapintoRRIpracticesandcouldpotentiallybeusedformonitoringtheevolutionofRRIacrossEurope.The36indicatorsarelistedinTable1below.Table1:IndicatorscapturingaspectsofRRIDimension Indicator IndicatorTitleGenderequality

GE1 ShareofRPOswithgenderequalityplansGE2 Shareoffemaleresearchersbysector(secondarydata)GE3 ShareofRFOspromotinggendercontentinresearchGE4 DissimilarityIndex(secondarydata)GE5 ShareofRPOswithpoliciestopromotegenderinresearchcontentGE6 GlassCeilingIndex(secondarydata)GE7 GenderPayGap(secondarydata)GE8 ShareoffemaleheadsofRPOsGE9 Shareofgender-balancedrecruitmentcommitteesatRPOsGE10 Numberandshareoffemaleinventorsandauthors

Scienceliteracyandeducation

SLSE1 SciencecurriculaSLSE2 RRIrelatedtrainingSLSE3 Sciencecommunication(secondarydata)SLSE4 Citizenscience

Ethics E1 EthicsatthelevelofUniversitiesE2 NationalEthicsCommittees(secondarydata)E3 ResearchFundingOrganizationsEthicsIndex

Public PE1 ModelsofpublicinvolvementinS&Tdecisionmaking(secondarydata)

17

engagement PE2 Policy-orientedengagementwithscience(secondarydata)PE3 CitizenpreferencesforactiveparticipationinS&Tdecisionmaking(secondarydata)PE4 Activeinformationsearchaboutcontroversialtechnology(secondarydata)PE5 PublicengagementperformancemechanismsatthelevelofresearchinstitutionsPE6 DedicatedresourcesforPEPE7 EmbedmentofPEactivitiesinthefundingstructureofkeypublicresearchfundingagenciesPE8 PublicengagementelementsasevaluativecriteriainresearchproposalevaluationsPE9 R&IdemocratizationindexPE10 Nationalinfrastructureforinvolvementofcitizensandsocietalactorsinresearchandinnovation

Openaccess OA1 OpenAccessLiteratureOA2 DatapublicationsandcitationspercountryOA3 SocialmediaoutreachOA4 PublicperceptionofOpenAccess(secondarydata)OA5 FunderMandates(secondarydata)OA6 RPOsupportstructuresforresearchersasregardsincentivesandbarriersfordatasharing

Governance GOV1 Compositeindicator:Governanceforresponsibleresearchandinnovation(secondarydata)GOV2 ExistenceofformalgovernancestructuresforRRIwithinRFandRPorganisationsGOV3 ShareofresearchfundingandperformingorganisationspromotingRRI

Outofthe36selectedindicators,13exploitsecondarydata(markedinredinthetable)whiletheremaining23requireprimarydatacollection.Multiplemethodswillbeappliedforthispurpose.Table2providesanoverview.Table2:MethodsforprimarydatacollectionMethodsforcollectingprimarydata Indicators‘ScienceinSociety’actorsurvey PE9,PE10Researchperformingorganisationssurvey GE1,GE5,GE8,GE9,SLSE2,SLSE4,PE5,PE6,OA6,E1,GOV2,GOV3Researchfundingorganisationssurvey GE3,PE7,PE8,E3,GOV2,GOV3Register-baseddata GE10,OA1,OA2,OA3Qualitativedata,interviews/desk-research SLSE1

5.Acriticalreflectiononthe‘benefits’ofRRIandtheuseofRRIindicatorsformainstreamingRRIAsdescribedin‘theMetricTidereport(Wilsdonetal.,2015),therearegrowingpressuresforaccountabilityandevaluationofpublicspendingonhighereducationandresearch,suchasdemandsbypolicymakersformorestrategicintelligenceonresearchqualityandimpact,andtheneedforinstitutionstomanageanddeveloptheirstrategiesforresearch.Againstthisbackdrop,thedevelopment,description,andoperationalizationof‘metrics’remainscontestedandopentomisunderstandings,especiallywhenthesemetricsareaboutconceptsthatarecomplex,suchasRRI,orsocietalimpactingeneral.Althoughwideruseof(quantitative)indicatorscouldsupportthetransitiontoamoreopen,accountableandoutward-facingresearchsystem,inappropriateindicatorscouldeasilycreateperverseincentives.Hence,theconsequencesofanRRIScoreboardwithanarrowsetofRRIindicatorsneedtobeidentified,Theriseinproductionanduseofindicatorsinglobalgovernancehasnotbeenaccompaniedbysystematicstudyofandreflectionontheimplications,possibilitiesandpitfallsofthispractice.Thereisnoagreedmeaningof“indicator,”butDavisetal.,(2012)defineindicatorsas‘anamedcollectionofrank-ordereddatathatpurportstorepresentthepastorprojectedperformanceofdifferentunits.Thedataaregeneratedthroughaprocessthatsimplifiesrawdataaboutacomplexsocialphenomenon.Thedata,inthissimplifiedandprocessedform,arecapableofbeingusedtocompareparticularunitsof

18

analysis(suchascountriesorinstitutionsorcorporations),synchronicallyorovertime,andtoevaluatetheirperformancebyreferencetooneormorestandards’.Indicatorsoftentaketheformof,orcanbetransformedinto,numericaldata.Akeychallengeiswhetherandhowindicatorsoughttobedistinguishedfromothercompilationsofnumericallyrendereddata.Thedifferenceslieinhowindicatorssimplify“raw”dataandthennametheresultingproduct.Thatsimplificationcaninvolveaggregationofdatafrommultiplesources,orfilteringthatexcludescertaindata,includingforinstanceoutliers.Theysimplifylife,andwhencomparingdifferentvaluesystemsoneshouldnotcompareapplesandoranges.Theuseofanyindicatorisfineaslongaswealsodiscusswhatitdoesnotcover,whatitmayhideornotconsider,orwhatotherobjectionscouldbetheretomakeclearthatindicatorsdoesnotcapturetherealityofRRI.Inessenceindicatorsaretheultimatediscrepancybetweenthe‘systemsworld’and‘lifeworld’(Habermas,1987).Theworldofsciencehasitsownrulesofengagementwherescientificpublicationsandacquiringfundingarecrucial‘currencies’(Polanyi,2000).Criteriaandindicatorsfore.g.engagementwithsocietythatareimposedbygovernmentforreasonsofaccountabilityorsteeringdonotplaysucharoleinthe‘lifeworld’ofscience.InaccordancewithHabermas(1987),the‘systemworld’istrying(orevennecessary)togovernthesciencesystem.Inordertoactintheacademicworld,governmentintendstobeaccountablefortheorganisationandexecutionofscience.Theinformationthatisnecessarytofulfillsuchtaskrequiresstructuredinformationgatheringandcriteria(orindicators).Inturnthisleadstoadynamicofmorespecificinformationalongdifferent(RRI)keysthatcollidewiththelifeworldofscience.WehavetobeawarethatwedonotinvadetheactuallifeworldofscienceandinnovationinsuchawaythatRRIindicatorsbecomecounterproductiveandcreateunintendedandperverseeffects.TheideaofmonitoringRRI(throughtheRRIindicators)aremechanismsfromthesystemsworldas.IfwewanttomainstreamRRI,wehavetomakesurethatthelifeworldofscienceacceptsRRIinthesamewayastheyaccepttheactualcurrenciesofscience.FromthevisioningworkshopthatwasorganizedbytheMoRRIconsortiumwithanumberofstakeholdersin2015,afutureRRIrealityhasbeendescribedinLindneretal.(2015).Itstatesthat:-RRIisinyourDNA,embeddedindailyactivitiesacrossallactors-ThereisamultipleanddiverseunderstandingofexcellenceinRRI-ThereisameritsandincentivestructuretosupportRRIatalllevels-Inallstepsoftheresearchprocess–fromresearchagendasettingtoevaluation–societyisactivelyinvolvedintheimplementation-Andfinally,RRIisacreativeactivity/opportunityinsteadofaburdenWhatthisvisionactuallyshowsisthatRRIisnotyetmainstreaminthelifeworldofscience!HowthencouldwethinkoffurthermainstreamingofRRIpoliciesandpractices?Whataresuitablemodelsthatdescribehowthelifeworldofsciencecouldinteractwiththewidersociety?TheRRIpolicydevelopmentalignswellwiththedebateaboutsocialresponsibilityofscience,i.e.fortheuseoftheoutcomesoftheirscience(Shapin,2008).GelrupandHorst(2014)conceptualizedifferentformsofrationalitiesinscience.Theyidentifyfourkindsofresponsibilities(seefigure2)alongtwodimensions:Thefirst

19

dimensiondescribeswhethertheregulationofscienceisinternalorexternal,andtheseconddimensiondescribeswhetherissuesofresponsibilityrelatetotheprocessortheoutcomeofscience.TheDemarcationandReflexivitydimensionsrepresentinternalregulationofscience;theContributionandIntegrationrationalitiesrepresenttheexternalregulationofscience,buteitheraimforoutcomesthatcanworkassolutions(contribution)ortoorganizingtheprocess(integration).Figure2.Overviewoffourpoliticalrationalities(source:GelrupandHorst,2014)

(1)Demarcationrationality:departsfromaprideinsciences’endeavors,andconsideritanhonorableprofession,whichisincreasinglysufferingfromfraudandmisconduct.Thisrationalityarticulatesscienceasaprofessionthatshouldhaveahighautonomyfromotheractors:outsiderstothescientificrealmshouldnotinterferewiththediscussionsaboutscientists’responsibility.(2)Reflexivityrationality:acknowledgesthatsciencehassolvedmanyproblemsforsocietybutworriesthatscientistdonotassumeresponsibilityforthemisapplicationsofscience.Scientistsneedtomakeanefforttoforeseehowtheirresearchmayaffecttheirsurroundings.Andthereflexivityrationalitydescribesvarioustechniquestohelpscientistsgetbetteratreflectingontheirownpractices.(3)Contributionrationality:articulatesscienceasasocialinstitution;itispartofsocietyandservescertainsocietalgoals.Avisionofwhatisgoodforsocietyisinherentinthesespecificgoalsthatsciencepursues,andlikewisesocietyhasadecisiveroleinshapingthesevisionsandgoals.Accordingtothecontributionrationality,theproblemwithresponsibilityinscienceisthatscientistsdonotseethemselvesas‘workingforthepublicgood’,andhencethisrationalityproposestoenhanceoutside(societal)controloverscientists,inordertomakesurethatscientiststakeresponsibilitytodeliverresultsthatareneededbysociety.(4)Integrationrationality:articulatesthatscienceissupposedtobefirmlyrootedwithinsociety,butitiscenteredaroundthevisionthatactorsfromscienceandsocietyneedtoworktogetherasequalpartnersinordertoproducebetterresults.Thereisalackofintegrationbetweenscienceandotheractorsinsociety,andthereforeadialoguebetweenscientistsandotheractorsshouldbeenhancedinordertodevelopanewkindof‘integrativeresponsibility’thatissuitableforthecurrentspecializedsocieties.Thisrationalityseesscientistsasaspecialkindofcitizenthatpossessesspecializedknowledgethatcanbeusedtodevelopsocietyinbetterdirections.

Reulexivity:scienceshouldlearnfromsocietalproblems

andprovidesolutions

Contribution:scienceshouldberegulatedbysocietytoensurethatoutcomesaare

useful

Demarcation:scienceshouldcontinuouslyquestionitsown

motivesandmethods

Integration:societalactorsshouldbeincludedinthe

processandconductofscienceinorderto

inuluencethedirectionofresearch

20

TheRRIparadigminEuropealignswiththeIntegrationrationalityinthatitstressesthatresearchandinnovationactivitiesneedtobecomemoreresponsivetosocietalchallengesandconcerns.TheindicatorsthataresuggestedforthemonitoringofRRIinECcountriesassumesuchrationality,butweshouldbeawarethattheotherrationalitiesareequallypresent(cf.Randlesetal.,2016)andhencewearepursuingmeasurementofonlypartofthelifeworldofscience.Infact,theresultsfromtheMoRRIvisioningworkshopshowthattheIntegrationrationalityisstillinthevisionstage,andneedsfurthersupport.IfOpenScienceinthecontextofRRIisthemainaim,thesystemsworldneedstosupportscienceandsocietydevelopinginthedirectionofIntegration.Andinfact,theResponsibilityNavigator(Kuhlmannetal.,2016),developedintheRes-AGorAproject,5supportsdecision-makerstogovernsuchactivitiestowardsmoreconsciousresponsibility.Whatisconsidered“responsible”willalwaysbedefineddifferentlybydifferentactorgroupsinresearch,innovation,andsociety–theResponsibilityNavigatorisdesignedtofacilitaterelateddebate,negotiationandlearninginaconstructiveandproductiveway.TheResponsibilityNavigatorsupportstheidentification,developmentandimplementationofmeasuresandproceduresthatcantransformresearchandinnovationinsuchawaythatresponsibilitybecomesaninstitutionalisedambition.IntheMoRRIprojectwedevelopedthesetofindicatorstomonitortheprogressalongtheRRIkeys.SubsequentlywereflectedonthemainstreamingofRRIinthelifeworldofscienceandshowedthatthisisavisionratherthanareality.However,theactualgoalistogenerateinsightsintothebenefitsemergingfromRRI:whattypeofbenefitscanbeexpected?Obviously,itismuchmorechallengingtomonitorbenefitsbecause,whenemergingfromscientificdiscoveries,theyarealmostalwaysassociatedwithrisksanduncertaintiesthattaketimetomeasure/eliminate.IntheMoRRIprojectasetofcasestudieswascarriedouttoobtainanintegrativeandbroadviewofhoweachoftheindividualkeysrelatestoeachotherandtoidentify–untilnow–unknownbenefits.BeingabletoidentifysuchbenefitsgoesbeyondtherathersimplisticideaofmonitoringRRIthroughindicators.Identifiablebenefitswillfurtherhelpinshapingandrefiningindicators.Benefitsdonotariseinisolation:thepathwaysthatleadtobenefitareasdiverseasthenumberofcasestudies.Findingacommondenominatorinsuchdiversityhelpsinstructuringthebenefits.IntheMoRRIprojectwedescribepathwaystobenefitashavingacausalrelationshipwithobservable‘first’and‘secondorder’effects.Forinstance,genderpolicyinterventions(cause)focusontherightforequalrepresentation,agenderedcontentofscienceandtheaddressingofbarriersthatprohibitthesetwoelements.Theultimatefirstordereffectofsuchpolicyisaresearchandinnovationsystemfreeofgenderbias.Thesecondordereffectsofsuchpolicyaredemocratic(equalrepresentationatalllevels),economic(betterdesignedproducts),andsocial(moreinclusive)benefitstosocietybroadlyandtotheworldbeyond.Impactpathwaysleadtocognitivetransformations,whicharerelatedtothecontentofscience;proceduraltransformationswhicharerelatedtotheenactmentofthesciencesystem;andcompetenceandcapabilitytransformationswhicheffectallactorsintheR&Isystem(andbeyond).

5Moreinformationontheprojectavailableatwww.res-agora.eu

21

Pathwaystobenefitarealwayscontext-dependent,howeveritisincreasinglyapparentthatRRIdrivesanumberofunderlyingprocessesleadingtowardmore‘responsible’configurationsofscienceinandforsociety.Indeed,manycasestudiesdemonstrateprocessesofpluralization,legitimizationandinclusionwhichworkinthedirectionofproducingboththedirectbenefitofarecognizablymoreresponsibleR&Isystemandarangeofmorediffuseddemocratic,economicandsocietalbenefitsforcitizensandstakeholdersatlarge..Disclaimer:TheMoRRIconsortiumwillusetheSTI2016ConferenceinValenciaasaplatformforpresentinganddiscussingthecompilationofindicatorstappingintothenotionofRRI,addressingspecificallytheperipheralstatusofRRIcomparedtootheractivitiesinresearchandinnovation;bothinthesenseofinadequatemeasuresandinthesenseofthelowerstatusofthisdomainofactivity.Furthermore,thenextstepsandchallengesoftheproject,particularlyrelatedtotheidentificationof‘benefits’ofRRI,willbediscussed.TheMoRRIconsortiumisledbytheTechnopolisGroup(ViolaPeter,NikosMaroulis),andfurtherconsistsof:CenterforScienceandTechnologyStudies,UniversityLeiden(IngeborgMeijer);AarhusUniversity(NielsMejlgaard);INGENIOCSIC-UPV(RichardWoolley,IsmaelRafols);InstituteforAdvanceStudiesVienna(ErichGriessler,AngelaWroblewski);FraunhoferISI(RalfLindner,SusanneBuehrer);UniversityCollegeLondon(JackStilgoe);UniversityofAthens(LenaTsipouri).

ReferencesArchambault,E.,Amyot,D.,Deschamps,P.,Nicol,A.,Provencher,F.,Rebout,L.and

Roberge,G.(2014).ProportionofOpenAccessPapersPublishedinPeer-ReviewedJournalsattheEuropeanandWorldLevels—1996–2013.Rapport,CommissionEuropéenneDGRecherche&Innovation;RTD-B6-PP-2011-2:StudytoDevelopaSetofIndicatorstoMeasureOpenAccess.

Barben,D.;Fisher,E.;Selin,C.;Guston,D.(2008):AnticipatoryGovernanceofNanotechnology:Foresight,Engagement,andIntegration.In:Hackett,E.J.;Amsterdamska,O.;Lynch,M.;Wajcman,J.(Eds.):TheHandbookofScienceandTechnologyStudies,Cambridge,MITPress.979-1000

Bauer,M.W.,Allum.,NandMiller,S.(2007):Whatcanwelearnfrom25yearsofPUSsurveyresearch?Liberatingandexpandingtheagenda.PublicUnderstand.Sci.16,79–95. Björk,B-C.,Welling,P.,Laakso,M.,Majlender,P.,Hedlund,T.,andGuðnason,G.2010.“OpenAccesstotheScientificJournalLiterature:Situation2009.”PLoSONE5(6):e11273.doi:10.1371/journal.pone.0011273.

Borrás,S.andEdler,J.(2014):Introduction:ongovernance,systemsandchange.In:Borrás,S.andEdler,J.(eds.):TheGovernanceofSocio-TechnicalSystems.ExplainingChange,Cheltenham:EdwardElgarPublishing,1-22

Brom,F.W.A.;Chaturvedi,S.,Ladikas,M.,Zhang,W.(2015):InstitutionalizingEthicalDebatesinScience,Technology,andInnovationPolicy:AComparisonofEurope,IndiaandChina.In:Ladikas,M.;S.Chaturvedi;Zhao,Y.;Stemerding,D.(Eds.)ScienceandTechnologyGovernanceandEthics.Springer,Cham,HeidelbergNewYork,Dordrecht,London,9-23.DOI:10.1007/978-3-319-14693-5_12

Bucchi,M.(2008).Ofdeficits,deviationsanddialogues.InM.Bucchi&B.Trench(Eds.),HandbookofPublicCommunicationofScienceandTechnology(pp.57–

22

76).Routledge.Callon,M.(1999).Theroleoflaypeopleintheproductionanddisseminationof

scientificknowledge.Science,Technology&Society,4,81–94.Caprile,M.,Addis,E.,Castano,C.,Linge,I.,Larios,M.,Moulders,D.&Vazquez-

Cuperio,S.((2012):Meta-analysisofgenderandscienceresearch,Synthesisreport,Luxembourg,PublicationsOfficeoftheEuropeanUnion.Online:https://ec.europa.eu/research/science-society/document_library/pdf_06/metaanalysis-of-gender-and-science-research-synthesis-report.pdf

Catalyst(2004),ThebottomLine:ConnectingCorporatePerformanceandGenderDiversity,NewYork,SanJose,Toronto.

Costas,R.,Meijer,I.,Zahedi,Z.,andWouters,P.(2013).Thevalueofresearchdata-Metricsfordatasetsfromaculturalandtechnicalpointofview.AKnowledgeExchangeReport(pp.1–48).

Davis,K.E.,Fisher,A.,Kingsbury,B.,Merry,S.E.(2012).GovernancebyIndicators.GlobalPowerthroughQuantificationandRankings.OxfordUniversityPress,Oxford.DOI:10.1093/acprof:oso/9780199658244.001.0001

DenHertog,P.,Jager,C.J.,vanKan,A.,TeVelde,R.,Veldkamp,J.,Aksnes,D.W.,Sivertsen,G.,vanLeeuwen,T.,vanWijk,E.(2014).Science,Technology&InnovationIndicators.Publicationnumber:2010.056-1435.Thisreportiselectronicallyavailableatwww.sti2.nl

Dewey,J.(1934).TheSupremeIntellectualObligation.ScienceEducation,18,1–4.EuropeanCommission(EC2012).Towardsbetteraccesstoscientificinformation:Boostingthebenefitsofpublicinvestmentsinresearch.COM(2012)401final.

EuropeanCommission(EC2014).CommunicationfromtheCommission.Towardsathrivingdata-driveneconomy.COM(2014)442final

EuropeanCommission(EC2015);IndicatorsforpromotingandmonitoringResponsibleResearchandInnovation,reportoftheexpertgrouponpolicyindicatorsforresponsibleresearchandinnovationhttp://ec.europa.eu/research/swafs/pdf/pub_rri/rri_indicators_final_version.pdf

EuropeanUnion(2012):ResponsibleResearchandInnovation.Europe’sabilitytorespondtosocietalchallenges.Availableat:http://ec.europa.eu/research/sciencesociety/document_library/pdf_06/responsible-research-and-innovation-leaflet_en.pdf

Farhan,H.,Alonso,J.,Davies,T.,Tennison,J.,Heath,T.,&Berners-lee,T.(2013).OpenDataBarometer,1–45.

Felt,U.,Fochler,M.;Müller,A.,Strassnig,M.(2009):Unrulyethics:onthedifficultiesofabottom-upapproachtoethicsinthefieldofgenomics.PublicUnderstandingofScience18(3),354-371.

Fisher,E.,etal.,Mahajan,R.L.,Mitcham,C.(2006).MidstreamModulationofTechnology:GovernanceFromWithin.BulletinofScience,Technology&SocietyVol.26,No.6,December2006,485-496DOI:10.1177/0270467606295402

Friedman,B.(1996):Value-sensitivedesign.Interactions.Volume3Issue6,Nov./Dec.1996pp.16-23doi:10.1145/242485.242493

Glerup,C.andMajaHorst,M.(2014)Mapping‘socialresponsibility’inscience,JournalofResponsibleInnovation,1:1,31-50,DOI:10.1080/23299460.2014.882077

Grießler,E.,Littig,B.(2006):NeosokratischeDialogezuethischenFragenderXenotransplantation.EinBeitragzurBearbeitungethischerProblemeinpartizipativerTechnikfolgenabschätzung.Buchinger,E.;Felt,U.(Hrsg.):Technik-undWissenschaftssoziologieinÖsterreich.StandundPerspektiven.ÖZS,Sonderheft8/2006,131-157.

Grin,J.,Grunwald,A.(2000,eds.):VisionAssessment:ShapingTechnologyin21stCenturySociety.Heidelberg:Springer2000,193S.

Guston,D.H.andSarewitz,D.(2002)Real-timetechnologyassessment.TechnologyinSociety24(1–2):93–109.

Guston,D.H.(2014):Understanding‘anticipatorygovernance’.In:SocialStudiesofScience,4482:218-242

Hagendijk,R.,&Irwin,A.(2006).Publicdeliberationandgovernance:engaging

23

withscienceandtechnologyincontemporaryEurope.Minerva,44(2),167-184.Hellström,T.(2003):Systemicinnovationandrisk:technologyassessmentandthe

challengeofresponsibleinnovation.In:TechnologyinSociety25(3),p.369–384KarinenR.andGuston,D.H.(2010)Towardanticipatorygovernance:Theexperiencewith

nanotechnology.In:KaiserM,KurathM,MaasenS,Rehmann-SutterC(eds)GoverningFutureTechnologies:NanotechnologyandtheRiseofanAssessmentRegime.Dordrecht:Springer,pp.217–232.

Kuhlmann,S.,Edler,J.,Ordóñez-Matamoros,G.,Randles,S.,Walhout,B.,Gough,C.,Lindner,R.(2016):ResponsibilityNavigator.In:Lindner,R.;Kuhlmann,S.;Randles,S.;Bedsted,B.;Gorgoni,G.;Griessler,E.;Loconto,A.;Mejlgaard,N.(eds):NavigatingTowardsSharedResponsibilityinResearchandInnovation.Approach,ProcessandResultsoftheRes-AGorAProject,Karlsruhe,p.135-158https://indd.adobe.com/view/eaeb695e-a212-4a34-aeba-b3d8a7a58acc

Ladikas,M.;S.Chaturvedi;Zhao,Y.;Stemerding,D.(Eds.2015)ScienceandTechnologyGovernanceandEthics.Springer,Cham,HeidelbergNewYork,Dordrecht,London.

Lindner,R.;Warnke,P.;Kukk,P.(2015):Updateoftheliteraturereview&Visioningexercise.ProgressReportD5.1,DeliverableoftheProjectMonitoringtheEvolutionandBenefitsofResponsibleResearchandInnovation(MoRRI),Brussels.

Lipinsky,A.(2014),GenderEqualityPoliciesinPublicResearch,Luxembourg,PublicationsOfficeoftheEuropeanUnion.Online:http://ec.europa.eu/research/pdf/199627_2014%202971_rtd_report.pdf

McKinseyandCompany(2007).WomenMatter:Genderdiversity,acorporateperformancedriver,McKinsey&CompanyInc.Online:http://www.mckinsey.de/sites/mck_files/files/Women_Matter_1_brochure.pdf

Mejlgaard,N.andGriessler,E.(2016):MonitoringRRIinEurope:approachandkeyobservations.In:Lindner,R.;Kuhlmann,S.;Randles,S.;Bedsted,B.;Gorgoni,G.;Griessler,E.;Loconto,A.;Mejlgaard,N.(eds):NavigatingTowardsSharedResponsibilityinResearchandInnovation.Approach,ProcessandResultsoftheRes-AGorAProject,Karlsruhe,p.115-118.https://indd.adobe.com/view/eaeb695e-a212-4a34-aeba-b3d8a7a58acc

MoRRI(2015).AseriesofdocumentsproducedaspartoftheEuropeanCommission’sservicecontractRTD-B6-PP-00964-2013,”Monitoringtheevolutionandbenefitsofresponsibleresearchandinnovation”.Thefollowingdeliverableshavebeenproducedsofar:- AnalyticalReportontheDimensionofCitizenEngagementandParticipationofSocietal

ActorsinResearchandInnovation.Sub-task2.5,deliverableD.2.1.- AnalyticalReportontheGenderEqualityDimension.Sub-task2.5,deliverableD.2.3.- AnalyticalReportontheDimensionofScienceLiteracyandScientificEducation.Sub-task

2.5,deliverableD.2.2.- AnalyticalReportontheDimensionofOpenAccess.Sub-task2.5,deliverableD.2.4.- AnalyticalReportontheDimensionofResearchandInnovationEthics.Sub-task2.5,

deliverableD.2.4.1.- AnalyticalReportontheDimensionsofResearchandInnovationGovernance.Sub-task2.5,

deliverableD2.4.2- SynthesisreportonexistingindicatorsacrossRRIdimensions,Task3,Progress

report,DeliverableD3.1,May2015- MetricsandindicatorsofResponsibleResearchandInnovation,Task3,

Progressreport,DeliverableD3.2,September2015Nielsen,M.V.;Lindner,R.;Bryndum,N.;Burchardt,U.;Schofield,M.;Stilgoe,J.(2015):

NavigatingtowardsResponsibleResearchandInnovation.ChallengesforPolicyandGovernance.In:Michalek,T.;Scherz,C.;Hennen,L.;Hebáková,L.;Hahn,J.;Seitz,S.(Hg.):TheNextHorizonofTechnologyAssessment.ProceedingsfromthePACITA2015ConferenceinBerlin,Berlin,S.57-62

Owen,R.,MacnaghtenP.,StilgoeJ.(2012)Responsibleresearchandinnovation:Fromscienceinsocietytoscienceforsociety,withsociety.ScienceandPublicPolicy39;751–760.

24

Pandza,K.andEllwood,P.(2013).Strategicandethicalfoundationsforresponsibleinnovation.Res.Policy42(5);1112–1125doi:10.1016/j.respol.2013.02.007

Polanyi,M.(2000).Therepublicofscience:Itspoliticalandeconomictheory.Minerva,38(1),1-21.

Randles,S.;Laredo,P.;Loconto,A.;Walhout,B.;Lindner,R.(2016):Framingsandframeworks:Sixgrandnarrativesofdefactorri.In:Lindner,R.;Kuhlmann,S.;Randles,S.;Bedsted,B.;Gorgoni,G.;Griessler,E.;Loconto,A.;Mejlgaard,N.(eds):NavigatingTowardsSharedResponsibilityinResearchandInnovation.Approach,ProcessandResultsoftheRes-AGorAProject,Karlsruhe,p.31-36.https://indd.adobe.com/view/eaeb695e-a212-4a34-aeba-b3d8a7a58acc

Ravn,T.,Mejlgaard,N.andRask,A.(2014):InventoryofPEmechanismsandinitiativesD.1.1.Availableat:http://pe2020.eu/wp-content/uploads/sites/15/2014/02/PE2020-FINALD.1.1-report.pdf

Rip,A.(2014):ThepastandfutureofRRI.LifeSciences,SocietyandPolicy10:17http://www.lsspjournal.com/content/10/1/17

Rip,A.,Misa,T.J.,Schot,J.W.(1995).ManagingtechnologyinSociety.Theapproachofconstructivetechnologyassessment.PinterPublishers,London.

Rowe,G.andFrewer,L.J.(2005):ATypologyofPublicEngagementMechanisms.ScienceTechnology&HumanValues.30:251.

Schicktanz,S.;Schweda,M.,Wynne,B.(2012):Theethicsof‚publicunderstandingofethics‘–whyandhowbioethicsexpertiseshouldincludepublicandpatients‘voices.MedHealthCareandPhilos15:129-139.

Schiebinger,L.andSchraudner,M.(2011),InterdisciplinaryApproachestoAchievingGenderedInnovationsinScience,Medicine,andEngineering,InterdisciplinaryScienceReview,Vol.36,No.2,154–67.

Shapin,S.(2008):TheScientificLife.Amoralhistoryoflatemodernvocation.TheUniversityofChicagoPress.

Stilgoe,J.,Owen,R.,Macnaghten,P.(2013)Developingaframeworkforresponsibleinnovation.Res.Policy42;1568-1580,http://dx.doi.org/10.1016/j.respol.2013.05.008

TheRoyalSociety.(1985).ThePublicUnderstandingofScience.Retrievedfromhttps://royalsociety.org/~/media/Royal_Society_Content/policy/publications/1985/10700.pdf

VandenEynden,V.andBishop,L.(2014).SowingtheSeed:Incentivesandmotivationsforsharingresearchdata,aresearcher’sperspective.AKnowledge ExchangeReport.

VonSchomberg,R.(2013)."Avisionofresponsibleinnovation".In:R.Owen,M.HeintzandJBessant(eds.)ResponsibleInnovation.London:JohnWiley.

Wickson,F.,andCarew,A.(2014).Qualitycriteriaandindicatorsforresponsibleresearch&innovation.Learningfromtransdisciplinarity.JournalofResponsibleInnovation,DOI:10.1080/23299460.2014.963004

Wilsdon,J.andWillis,R.(2004)See-throughscience:whypublicengagementneedstomoveupstream.ProjectReport.Demos,London.

Wilsdon,J.,etal.(2015).TheMetricTide:ReportoftheIndependentReviewoftheRoleofMetricsinResearchAssessmentandManagement.DOI:10.13140/RG.2.1.4929.1363

Habermas,J.(1987).Thetheoryofcommunicativeaction.VolumeIILifeworldandsystem:acritiqueoffunctionalistreason.BostonBeaconPress.

Wynne,B.(1995).PublicUnderstandingofScience.InJasanoff,S.etal.(eds.).HandbookofScienceandTechnologyStudies.SagePublications,London.