Project ref.no. CIP-Pilot 325101 Project acronym OpenScienceLink Project full title Open Semantically-enabled, Social-aware Access to
Scientific Data Project site http://opensciencelink.eu Project start February 2013 Project duration 3 years EC Project Officer Martin Májek
Deliverable
Deliverable type Report Distribution level Public Deliverable Number D8.2.3 Deliverable title Stakeholders, Technical and Financial Evaluation Contractual date of delivery M36 (January 2016) Actual date of delivery February 2016 Relevant Task(s) WP8/Tasks 8.1, 8.2, 8.3 Partner Responsible NKUA Other contributors CNR, TUD, NTUA, LUHS, NKUA, KU Leuven, Procon,
TI Number of pages 87 Author(s) Costas Pantos, IordanisMourouzis, AdomasBunevicius,
InesaBirbilaite, TodorTagarev, PetyaTagareva, VassilikiAndronikou, EfstathiosKaranastasis, Giorgio Iervasi,Alessandro Pingitore,Freyja van den Boom, Laurens Naudts, Michael R. Alvers, George Tsatsaronis
Internal Reviewers Liliana Alvers, Michael Schroeder, EfthymiosChondrogiannis
Status & version Keywords Evaluation Framework, Project's Results, Methodologies,
The current deliverable presents a multi-facet evaluation of the 3rd year version of the OSLplatformcoveringdifferentperspectives.First,wehavecollectedandanalyzedfeedbackfromallstakeholders ofthe OpenScienceLink ecosystem, including researchers, scholars, universities,research organizations, researchsponsors, funding agencies, funding authorities, open-accesspublishersand the media. The evaluation was based on severalmodalities includingquestionnaires,interviews,satisfactionsurveysandothertoolsthatwerespecifiedaspartoftheproject’s evaluation framework. At a second level, we present evaluation of the project’splatformand services froma technical and technological perspective in terms of performance,scalability, expandability, robustness, novelty and technological longevity.The third part isdevoted to thebusiness and financial evaluationof theOpenScienceLink services, according tothe different businessmodels for open access thatwere studied inWP9. In general this taskprovides insights for the formulationof realisticbusinessandsustainabilityplans. Asa result,the current deliverable is structured following the aforementioned rationale, presentinganalytically the results of evaluation based on the tools and methodologies (questionnaires,interviews,KeyPerformance Indicators (KPIs),measuresetc) thatweredefinedatdeliverable8.1.
List of Figures Figure1:Summaryofnumericquestionnaireresponsevalues..................................................................10Figure2:Summaryof themost importantOpenScienceLinkprojectachievementsaccording toquestionnaireresponses...............................................................................................................................................11Figure 3: Summary of OpenScienceLink project aspects that can be improved according toquestionnaireresponses...............................................................................................................................................11Figure4:EvaluationoftheOpenScienceLinkplatformasawhole............................................................13Figure5:Summaryofplatformuser’ssatisfaction...........................................................................................14Figure6:Openaccesspublishingbetween1993and2009..........................................................................60Figure7:DifferentPublisherTypesforOpenAccessArticles......................................................................61Figure8:OpenAccessArticlesacrossdifferentdisciplines..........................................................................62
List of Tables Table1:MissionPerspectiveKPIsforOverallPlatform.................................................................................15Table2:ResourcePerspectiveKPIsfortheOverallPlatform......................................................................16Table3:InternalBusinessProcessesKPIsfortheOverallPlatform.........................................................17Table4:LearningandGrowthPerspectiveKPIsfortheOverallPlatform..............................................17Table5:MissionPerspectiveKPIsforPilot1......................................................................................................21Table6:ResourcePerspectiveKPIsforPilot1...................................................................................................21Table7:InternalBusinessProcessesKPIsforPilot1......................................................................................21Table8:LearningandGrowthPerspectiveKPIsforPilot1..........................................................................22Table9:MissionPerspectiveKPIsforPilot2......................................................................................................23Table10:ResourcePerspectiveKPIsforPilot2.................................................................................................23Table11:InternalBusinessProcessesKPIsforPilot2...................................................................................23Table12:LearningandGrowthPerspectiveKPIsforPilot2........................................................................24Table13:MissionPerspectiveKPIsforPilot3....................................................................................................25Table14:ResourcePerspectiveKPIsforPilot3.................................................................................................25Table15:InternalBusinessProcessesKPIsforPilot3...................................................................................25Table16:LearningandGrowthPerspectiveKPIsforPilot3.......................................................................25Table17:MissionPerspectiveKPIsforPilot4....................................................................................................26Table18:ResourcePerspectiveKPIsforPilot4.................................................................................................27Table19:InternalBusinessProcessesKPIsforPilot4...................................................................................27Table20:LearningandGrowthPerspectiveKPIsforPilot4........................................................................28Table21:MissionPerspectiveKPIsforPilot5....................................................................................................28Table22:ResourcePerspectiveKPIsforPilot5.................................................................................................29Table23:InternalBusinessProcessesKPIsforPilot5...................................................................................29Table24:ClassesofTestsbasedontheISO9646.............................................................................................30Table25:TestsBehaviorTypes.................................................................................................................................30Table26:SubcategoriesandBehaviorTests.......................................................................................................32Table27:TheSubfieldsoftheTestCaseIdentifier...........................................................................................33Table28:TestCaseStructure.....................................................................................................................................34Table29:SummaryofTechnicalEvaluationResults.......................................................................................59Table30:EuropeanOpen-AccessFunders...........................................................................................................64Table31:Pilot1Competitors.....................................................................................................................................67Table32:SpendingsforPilot1..................................................................................................................................68Table33:PossibleRevenuesforPilot1.................................................................................................................68Table34:ExamplesofauthorfeesforpublicationinOAjournals.............................................................70Table35:Pilot2Competitorsandbusinessmodels.........................................................................................73Table36:Pilot3Competitorsandbusinessmodels.........................................................................................75Table37:Pilot4Competitorsandbusinessmodels.........................................................................................78Table38:Pilot5Competitorsandbusinessmodels.........................................................................................80
1.1 Objectives of Evaluation This task gives emphasis on themulti-facet evaluation of the project’s results (i.e. coveringmultiple perspectives), as well as on the elicitation and documentation of best practicesassociated with open access to scientific information. The main objectives of the evaluationprocessareto:Establish an evaluation framework (including scientifically sound methodologies, KPIs andtools)forthedisciplinedevaluationoftheproject’sresults.To evaluate the project’s results with the involvement of all stakeholders of theOpenScienceLinkplatformi.e.takingintoaccountandanalyzingfeedbackfromallstakeholders.ToevaluatetheOpenScienceLinkfromatechnicalandtechnologicalperspective.Toevaluate theOpenScienceLinkresultsagainst theirbusinesspotentialand thepossibilityofrenderingfinancialreturns/benefitsforthevariousstakeholders.To elicit and document best practices and blueprints associated with the implementation,exploitationanduseofmodelsforopenaccesstoscientificinformation.OpenScienceLink will also study the business potential of open access paradigms, throughinvestigating and pursuing multiple business models including author fees, hard copy sales,advertisements,sponsorships,aswellassubscriptionbasedmodels.
1.2 Evaluation Methodology The evaluation methodology includes a set of methodological tools tools and KPIs forevaluating the project from a usability, business and technical perspective,while at the sametimeelicitingandanalyzingtheopinion/feedbackofallstakeholders.KeyPerformanceIndicatorsarearguablyanimportantinstrumentformonitoringtheproject’swork and evaluating the results. Taking into consideration four main perspectives (Missionperspective,resourceperspective,InternalBusinessprocesses,Learning&Growthperspective),we have composed an expansive list of KPIs for each of the pilots as well as the overallOpenScienceLinkplatform.InadditiontotheKPIs,thedifferentaspectsoftheOpenScienceLinkprojectareevaluatedusingquestionnaires,reviewformsandstakeholderinterviews.
2 Stakeholders Evaluation In order to get detailed feedback from stakeholders about their satisfaction with theperformanceoftheprojectinterviewswereconductedbyconsortiummembers.Inadditiontotheevaluationofindividualsubjectsmulti-purposemethodologiesincludedquestionnaires(forall subjects), as well as key performance indicators and relatedmeasures specifically for theOpenScienceLinkplatformanditspilots.Overall, there are three basic subjects of evaluation within OpenScienceLink. First, themanagementoftheprojectandtheoverallexecutionoftheworkplan, includingthetimelyachievementsofthepredefinedmilestones.Second,theOpenScienceLinkplatform,whichhasthemainroletoimplementthefiveprojectpilots.Third,theperformanceandtheoutcomeoftheactualOpenScienceLinkpilots.
2.1 Management and Overall Progress of the Project With regards to the evaluation of the OpenScienceLink project execution, and based on thedescription ofwork of the project (OpenScienceLink Consortium, 2013), themajor propertiesthatweremonitoredpertaintotheevaluationofthetimelycompletionandsubmissiontotheECofreportsanddeliverables,aswellastheachievementoftheproject'smilestones. Inaddition,theoverallmonitoringofthemanagementoftheproject,e.g.,financialmanagement,distributionandorganizationofwork,organizationofprojectmeetings,isalsoanimportantproperty.There are many ways to measure the performance of the OpenScienceLink services directlyusingdata that canbe collectedautomaticallywhile running theplatform (e.g., thenumberofusers).However,therearealso“softer”criteria,e.g.,relatingtothequalityoftheuserexperienceor the overall satisfactionwith the project’s progress, that can not bemeasured as easily. Inordertobeabletoalsomeasuresuchcriteria,wehavedesignedonlinequestionnairesthatcanbe submitted tousersof thedifferent functionalities inorder toprovide themwitha fast andeasyway toprovide feedbackabout theirexperiencewith theOpenScienceLinkplatform.Thisfeedback isusedby the technicalpartners forprioritizing requiredupdatesanddesigning thenextiterationoftheplatform.The firstquestionnairewasdesignedwith the intentionofmeasuring the internal satisfactionwith the project’s progress among the members of the consortium. Most questions can beansweredwithanumericalvalueonascalebetween1(“disappointing”)and5(“excellent”).Thefollowingquestionsfollowedthistemplate:
Theresultsofthequestionnaireareshowninthefollowingchart.Thevalues(x-axis)representthe weighted average of the responses by the questionnaire participants, where a value ofdesignates the worst score, and the value of 5 the top (best) score. Overall, the internalevaluation via the questionnaire shows that within the third year the progress was verysatisfactory.
Inaddition to thesequestions that requirednumerical responses, the following twoquestionsrequestedtextualanswersinordertoretrievein-depthfeedbackfromtheconsortiummembersaboutwhichaspectsoftheprojecttheyconsidermostandleastsuccessful:
• Please enter three aspects of the project that you thinkwere themajor achievementsduringthethirdyear
Theresultsofthesetwoquestionnairesaresummarizedinthefollowingfigures.Astheresultssuggest,amongthemost importantachievements in the thirdyearof theprojecthasbeentheoperation of the Biomedical Data Journal, and the updated features of the OpenScienceLinkplatform (publicly available at: http://opensciencelink.org). In parallel, the participants to thequestionnaire suggested that the exploitation of the projects’ resultsisthe main aspect whichshould attract focus in the future, given the very important results of the project and theirimpact.
2.2 Evaluation of the Overall Platform's Performance AsfarastheevaluationoftheOpenScienceLinkplatformisconcerned,themainpropertiesthatweremonitoredpertaintothetechnicalaspectsoftheimplementationandperformanceoftheplatform,theevaluationoftheoveralluserexperience,thedegreeoffulfilmentofthecollecteduserandtechnicalrequirements,and,theoverallimpactoftheplatform.A questionnaire was designed with the intention of evaluating the release of theOpenScienceLinkplatform(Year3).Thecollecteddatacoveredthefollowingevaluationaspects:platformasawhole,usersatisfaction,novelty,andrecommendationsforfuturefeatures.Inthefollowingwepresenttheresultsofthisevaluation.
2.2.1 Evaluation of the Platform as a Whole For thepurposesof the evaluationof theOpenScienceLinkplatformas awhole, the followingquestionsweredistributedtothepartners:PleasegiveanoverallratingfortheOpenScienceLinkplatformasawhole. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseratethefunctionalityofPilot1,i.e.,theuploadofdatasetsandtheoverviewofuploadeddatasets. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot1thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot2,i.e.,thecreationofreviewcalls. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot2thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot3,i.e.,theanalysisoftrendsforagivenquery. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot3thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot4,i.e.,thesuggestionofcollaborations. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot4thatcanbeimprovedAnswertype:textPleaseratethefunctionalityofPilot5,i.e.,evaluationofresearchentities. Answertype:scalebetween1(“disappointing”)and5(“excellent”)PleaseentertheaspectsofPilot5thatcanbeimprovedAnswertype:textTheresultsofthisevaluationaresummarizedinthefollowingfigure.Withregardstothemostpositive aspects of the platform in its current form, the threemajor pointswere: simple andclearinterface,fastresponsetime,and,up-to-dateresults.Regardingthemostimportantaspectsof the platform that may be improved, these include: explanation or user guidance of the
Figure 4: Evaluation of the OpenScienceLink platform as a whole.
2.2.2 User Satisfaction Forthepurposesoftheevaluationofuserssatisfaction,thefollowingquestionswereincludedinquestionnaires:Howintuitiveistheuserinterfaceoftheplatforminyouropinion?Answertype:scalebetween1(“disappointing”)and5(“excellent”)Pleaseratetheplatform'sresponsetimetoyourinputAnswertype:scalebetween1(“veryslow”)and5(“veryfast”)Willyouusetheplatformagaininthefuture?Answertype:scalebetween1(“definitelynot”)and5(“definitelyyes”)Pleaseenterthethreemostpositiveaspectsoftheplatforminitscurrentform. Answertype:textPleaseenterthethreeaspectsoftheplatformthatyoufeelcanbeimproved. Answertype:textPlease enter awish list of functionalities you believe are interesting to be included in futurereleasesAnswertype:textTheresultsofthisevaluationaresummarizedinthefollowingfigure.Theevaluationtookplaceconsidering and testing again the last version of the OpenScienceLink platform. Overall, theresponsetimeoftheplatformwasevaluatedverywell,withtherestofthequestionssuggestingthattheuserinterfacewassignificantlyimprovedcomparedtothealpharelease.
2.2.3Key Performance Indicators (KPI) ThissectiondescribestheKPIsthatwereusedforthispurposecomparingtheexpectedwiththeactual progress that has been achieved during the 3rdyear. Following the evaluationmethodology adopted by the OpenScienceLink project, the KPIs have been grouped based onfourperspectives:(1)Mission,(2)Resource,(3)InternalBusinessProcesses,and,(4)Learningand Growth. The KPIs for the four perspectives, the anticipated progress for year 3, and theactualprogressreported,arepresentedinthefollowingfourtablesrespectively.
KPI Measure Expected
progressYear3
ActualprogressYear3
Userparticipation Number of userregistrations in theOpenScienceLinkplatform
>=1100
73(butmorethan400unique/anonymousvisits
inthesearchtab)
Active userparticipation
Number of users usingthe system at least onceper month within a 3monthperiod
>=600 73
Stakeholderparticipation
Number of institutions(beyondtheconsortium)involved in theOpenScienceLink pilotoperations
>=30 9
Funderparticipation
Number of researchsponsorsand/or fundingauthorities (beyond theconsortium) involved inthe OpenScienceLinkpilotoperations
2.3 OpenScienceLink Pilots Evaluation The evaluation of the OpenScienceLink pilots, the degree of fulfilment of the respectiverequirementsandtheoverallimpactareveryimportant,inadditiontotheconsiderationofthedegree of satisfaction of the engaged stakeholders and interested parties. Besides that, themethodology that has been used to accomplish the pilots' results has to be evaluated bycomparingittoexistingstateoftheartmethodologieswhenapplicable.
2. Pilot 3 has great potential. The trend detectionmechanismwas characterized as veryuseful.
3. All Pilots combined provide an ecosystem that offers open access to scientificinformation,fromtheinitialstageofdatasetandpaperpreparation,toaccess,re-usageandexploitation.
4. TheOSLplatformprocessespersonal,end-userdata,suchastheend-users’fullname,e-mailaddress,general fieldsof interestandemploymentdetails, for theperformanceofitspilotservices.Thedataareforinstanceusedtopersonalizetheend-user’sexperience.In order to facilitate research collaboration and evaluation, end-user data is alsorequired.Although theOSLplatform’sprivacypolicy is inaccordancewith thecurrentEuropeandataprotectionframework,futuredataprocessingactivitieswillhavetotakeinto account the upcoming data protection regulation (once it has been adopted andimplementedby theEuropean legislator).Theaddendum toD3.2already includesdataprotection compliance guidelinesunder the general data protection regulation.1Inaddition,anonymizationremainskey.Personaldatasetsshouldnotbeuploadedontotheplatform.Asdatasetsaremadeavailablefordownloadaftertheyhavebeenreviewedbyusers of the platform, the guidelines for reviewers should also clearly state that theuploadingofpersonaldataisforbidden.Indeed,atthemoment,theplatform’sreviewers,aswellastheresearchersthemselves,areinthebestpositiontodetectwhetherornotanuploadeddatasetcontainspersonaldata. If theywouldhaveanydoubtsconcerningtheanonymizednatureofdatasets,thepublicationoftherevieweddatasetshouldnotbeallowed.Furthermore,compromiseddatasetsshouldbepulledoftheplatform.
5. With regard to data mining activities performed by the OSL platform, future legaldevelopments should also be taken into account. In a recent communication, theEuropean Commission has recognized that the lack of a clear text- and data miningprovisionharmstheEU’sscientificleadership.Legislativeproposalsarenowconsideredthatwillallowpublicinterestresearchorganizationstocarryouttextanddataminingofcontent they have lawful access to, with full legal certainty, for scientific researchpurposes.SuchanexceptioncouldbenefitthefuturedevelopmentoftheOSLplatform.2
Therefore, and where necessary, proper authorisation should be obtained from the 1OSL,LegalandIPRManagementFrameworkSpecification(January2016).2European Commission, Communication towards a modern, more European copyright framework’COM(2015) 626 final, p. 7-8; available at:http://ec.europa.eu/newsroom/dae/document.cfm?action=display&doc_id=12524
7. It should beexaminedhowuseful couldbe the collectionof amoredetailedpersonalprofile of the user beyond name, surname, e-mail and institutional information.Howeveritshouldbetakenintoaccountthatthemorepersonalinformationiscollected,themorelaborioustheplatformwillbefortheusers.Theadditionalinformationcouldbeoptionalfortheuseraftertheregistration.Thisadditionalinformationmayinclude:
2.3.1 Pilot 1: Open-access Data Journals Development Theevaluationannotatesthefollowing:
1. Inthefuture, itwillremainimportantto informtheplatformend-users,researchers inparticular, on the relevant legal framework concerning the open access to scientificinformation. In this regard, legal guidelines have been formulated as part of WP8 toinformplatformend-usersonrelevantmattersconcerningIPRanddataprotection.
2. Uploadedopen-accessdata-setsmightcontainanonymizeddata,i.e.datasetspertainingto previously identifiable individuals. License-holders should be prohibited from re-identifyinganyindividualandfromusingthedatatotakeanymeasureordecisionwithregardtothere-identifiedindividuals.Upondetectingthatadatasetiscompromised,i.e.has become re-identifiable, the license-holder (or any other party) should notify thelicensor.Uponnotification,theplatformshouldsuspendorterminatetheaccessibilitytothedata,forinstancebyremovingthefilefromtheplatform.Reasonableeffortsshouldalso be made to limit the negative effects of compromised datasets, for instance bydeletingallorpartsofthecompromiseddatasets.Thisshouldalsoincludeaprominentnoticeontheplatformandwebsiteaccessedbygroupsorindividualswhoarelikelytobere-usingthedata.
3. Thelastyearoftheprojectaveryimportantupdateofpilot1hasbeenaddedconcerningtheuploadingofnewfilesafterreviewingofdatapapersandtheabilityofeditorsandreviewers to access the new revised files but also the original ones and makecomparisons
4. Afuturedevelopmentshouldfocusonthecreationofmoredetailedmetadatainordertoachievebetter-structuredandsemantically linkeddatasets.Tothisend,thefollowingcharacteristicsofadatasetcouldbeimportant:• themethodologyused to collect data (e.g. echocardiographic evaluation, Cardiac
magnetic resonance, Exercise testing, Bloodmeasurements etc) and the variablesmeasured in each case (e.g. for echocardiographic evaluation, Variable 1: Leftventricularinternaldiameteratdiastole(LVIDd),Variable2:Leftventricularinternaldiameter at systole (LVIDs), Variable 3: Ejection fraction (EF%) etc) could beretrievedfromthesubmitteddatasets
Number of identifiednew trends andaddition of respectivemembers to theEditorialBoard
2 3
Expanding theapplication of the openaccess data journalparadigm
Numberof concepts fornew open access datajournalsdeveloped 1 1
Table 8: Learning and Growth Perspective KPIs for Pilot 1.
2.3.2 Pilot 2: A novel open, semantically-assisted peer review process Theevaluationannotatesthefollowing:
1. The platform contains 2 different functionalities for the selection of reviewers. Onefunctionality creates a list of suggested reviewers based on the metadata and theabstract of each submission. The 2nd functionality creates a list of potential reviewersbasedonthekeywordsthathavebeenselectedbytheeditor.Bothareveryuseful.
2. During the last year the platform has added the functionality of evaluating reviewers.This tool could be very helpful in order to avoid the high rate of decline of invitedreviewers.Evaluationisbasedatthemomentinascoreassignedbytheeditor.Itwouldbe very helpful if the platform also included some statistics about the performance ofreviewerssuchasnumberofpending reviews, rateof acceptanceof review invitations,meantimetocompletereviewetc.
3. Theideaofaopenreviewingprocesswouldbeofaddedvaluetotheplatformandshouldbeconsideredforfuturedevelopment.Forthisreason,asocialnetworkcouldbecreatedenabling reviewers to discuss and express opinions on the reviews and the reviewresults. Inside this network, a reviewerwill be able to see the comments of the otherreviewersandvoteinfavorornot.
Profilegrowth Number of newplatform profilescreated by invitedreviewers
120 ~15
Table 12: Learning and Growth Perspective KPIs for Pilot 2.
2.3.2 Pilot 3: Services for detection and analysis of research trends
Theevaluationannotatesthefollowing:
1. During the lastyear, theplatformhasupdated trendsdetection thathasbecomemorefinegrained.
2. For future development of trends detection, important information about researchinterestcouldbecollectedfromsocialmediaandblogsandnotonlyfromanalysisofthenumberofpublications.
3. The provision of services for the detection and analysis of research trends maynecessitate the re-use of third party content, e.g. through linking, harvesting, text-anddataminingtechniques.TheOSLplatformshouldcontinuetoascertainthelicensesthatrestuponthedatabasestheywishtomine.AsmentionedinD3.2.OpenAccesslicenseslackuniformityandtheymayincluderestrictionsontheuseofthecontenttowhichtheypertain. If the license is unclear or ambiguous, data mining activities should not beperformedwithouthavingobtainedproperauthorisation.Hence,itisadvisabletoseekalicense for the application of text and data mining tools to third party databases.Licensingconditionsshouldalsobecarefullyexamined:whichrestrictionsareimposedby the relevant right holders on the free exchange of information. If the contents ofresearchpapersaretobeanalysedaswell,licensesshouldnotonlybeobtainedforthere-utilization of the data base, but also for the contents of the database. Licensesobtained must also be compatible with the license OSL wishes to grant its users.
2.3.4 Pilot 4: Services for dynamic researchers’ collaboration Theevaluationannotatesthefollowing:
1. Futuredevelopmentofthispilotshouldfocusonexpandingthefunctionalitytobeableto suggest potential new collaborations. A crucial factor to consider in potentialcollaborations is the complementary methodological expertise that could be derivedfrompublicationsorfromuserprofiles.
2. Theprovisionofservicesfordynamicresearchers’collaborationrequirestheprocessingofpersonaldata.Inparticular,dataoftheOSLplatform’send-userswillbeprocessedinordertofacilitatethecollaborationamongresearchers.TheOSLserviceshavebeenbuiltwith respect of the current European data protection framework. Although at themomenttheOSLplatform’sprivacypolicyisstillvalid,withregardtofutureprocessingactivities, itwill be necessary to take into account the newData Protection regulationonceithasbeenadoptedbytheEuropeanlegislator.Thelegalevaluationrequirementsprovided in D8.1 have been altered and added as an addendum to D3.2, taking intoaccountthechangestothedataprotectionframework.
Table 17: Mission Perspective KPIs for Pilot 4.
KPI Measure
ExpectedprogressYear3
ActualprogressYear3
Number of fundedprojects (beyondOpenScienceLink)
Numberofprojectsinwhichthe pilot models, servicesand tools are used andfurtherdeveloped
>=2 2
Number of InterestedStakeholders
Number of Stakeholders towhom the pilot has beenpresented and who haveexpressedinterest
>=2 3
KPI Measure Expected
progressYear3
ActualprogressYear3
UserAttraction
Number of Researchers/Scholarshavingusedthepilot services at leastonce
>=1,000 ~70
UserEngagement Number of Researchers/Scholarshavingusedthepilot services more thanonce
3 Technical Evaluation Inthissection,thetechnicalevaluationofthedeployedOpenScienceLinkplatformisconducted.ThetechnicaltestingoftheplatformhasbeenconductedaccordingtotheISOTestingStandardISO9646. In the following sections, the three classesof tests aredefined, and the actual testswiththerespectiveresultsarepresented.
3.1 Technical Evaluation: The Test Cases TheISO9646definesthreeclassesoftestsasshowninthefollowingtable.
TestsClasses Description
BasicTests(BT)
A small number of tests (1 or 2) that are run to actuallydemonstratethattheplatformhasbeencorrectlyset-up.
As the name suggests, these tests verify the correctbehaviour of the system under test under specificconditions.
Unsuccessful/InvalidTests(I)
These tests verify that the system under test acts “as itshould” under circumstances such as unexpected events,parameters out of range, invalid or unsupported options,negotiation failures and invalid combinations ofparameters.
InthissectionwewillpresentthetestscasesthatwereappliedattheOpenScienceLinkplatform.Initially we will define some broader categories for the tests cases. Also we will present thestructureofeachtestcasedefiningwhatweshouldmonitorabouteachone.
3.1.1 Tests Subclasses Inthefollowingtabletherearesomesubclassesofthe“main”testclassesseenintheprevioussection.Inthefirstcolumnthereisthetestclassinwhicheachsubclassbelongsto,whileinthelast column we can see the types of test (Behaviour Type) that will be performed. Thesesubclassesdepictthefunctionalitythatwillbetested.Foreachoneafewtestswillbeperformed.Thesetestswillbeanalyzedinthefollowingsections.
3.1.3 Test Case Structure Eachtestcaseshouldcontainallthefieldsmentionedinthetablebelow:
Field DescriptionID Auniqueidentifierforthetestcase.Title Thetitleofthetestcase.Description Abriefdescriptionofthetestcase.Setup Theprerequisitesinordertohaveaccessinthistestcase.TestingProcedure The testing procedure that will be used in this test case. Here we
should answer in the question: “How we will examine that thesystemprovidesthefunctionalitydescribed”.
3.2 Basic Tests (BT) Two basic tests have been specified. Their purpose is to ensure that the platform is up andrunning and a connection can be established between the application components of theplatform.
3.2.1 Valid OpenScienceLink Platform Setup
TestCaseID OpenScienceLink_BT_OPS_V_01
Test CaseTitle
ValidOpenScienceLinkPlatformSetup
Test CaseDescription
Ensure that all OpenScienceLink components and services are up andrunning.
The administrators start their application servers and ensure that allnecessarycomponents (hardwareandservers)areupandrunning tostarttheirapplications.
Theuserentersavalidusernameandpasswordon the login form (eitherClientApplicationorWebarea)andpressesthe“Login”button
ExpectedResult
Upon successful login, the system shows the main window of the ClientApplication (for normal users), or the main page of the OpenScienceLinkwebarea(foradministratortesters).
Theuserreceivesaconfirmatione-mailthathisaccounthasbeencreated,and he should be able to use the credentials to login again, and see hissavedprofiledetails.
Verify the correct operation of the editing and saving of an existing userprofile.
SetupUsing an existing account, the ability to edit and save the changes to theexistingaccountistested.
TestingProcedure
The user logs in to the platform with his existing account credentials,browsestohisprofiletab,andalterstheinformation.Oncethechangearesaved, thenew informationshouldbepartofhisexistingprofile, includingallchangesmade.
ExpectedResult
The saved changes in theprofile are storedandvisible in any future loginattempt.Thestoredprofileisnowthenewprofileoftheexistinguser.
Theuserenters the“UploadDataset”page, fills inall thenecessary fields,provided a zip filewith the dataset or an external URL and presses “SaveChanges”.Thedatasetwithsuchname/descriptionorURLhasalreadybeenuploaded.
ExpectedResult
The submission process is not performed; the user gets themessage thatthedatasetalreadyexists.
Theuser enters the “UploadDataset” page, fills in all thenecessary fields,but does not provide a zip file with the dataset or an external URL andpresses“SaveChanges”.
Theusergoesto“EditorActivities”tabandpresses“AddIssue”.Then(s)heselects the volume, fills in the title and deadline fieldswith the values ofalreadyexistingissue,andpresses“Savechanges”.
ExpectedResult
The creation of a new issue is not completed, a user gets amessage thatsuchissuealreadyexists.
The user goes to “Editor Activities” tab and presses “Add Issue” andperforms the following operations on journal issues (by clicking onrespectivethumbnails):“deletetheissue”,“edittheissue”,“viewtheissue”,“viewdatasetssubmittedforthisissue”.
The user attempts to create a review call by filling the form that can beaccessed from the „editor activities“ tab of the platform, but leaves outrequiredinformation(e.g.,thevenueforthecall).
ExpectedResult
Presentationof anerrormessage to theuser indicating that requireddatahasnotbeenprovided.
The user clicks the „upload“ button on the „editor activities“ tab of theplatform,leavesoutrequiredinformationwhilefillingtheformandselectsafileforupload.
ExpectedResult
Presentationof anerrormessage to theuser indicating that requireddatahasnotbeenprovided.
The user adds a relatively large set of very diverse terms describing aresearchtopicandrequestsreviewersuggestions.
ExpectedResult
The system presents an error message to the user indicating that noreviewercanbe found for theentered setof terms. It recommends to theusertoremovesometermswithlowimportance.
For testing purposes, the user adds him- or herself as a reviewer andcompletestheinvitationprocess.
ExpectedResult
The system sends ane-mail invitation to theuser containing the text thatwasenteredwhile fillingthe invitationformaswellasa linktothereviewform.Theeditor'soverview table for the invited reviewers shows that theuserhasbeeninvitedasareviewer.
Theuser's“ReviewerActivities”overviewnowcontainsalinktothereviewform.Theeditor'soverview table for the invited reviewers shows that theuseracceptedtheinvitation.
The user fills out all required fields of the review form and submits thereview.
ExpectedResult
Presentation of a confirmation message to the user indicating that thereviewhasbeensubmittedsuccessfully.Theeditor'soverviewtablefortheinvited reviewers shows that the user submitted the review and nowcontainsalinktothereview.
SetupTheuser is logged inaseditorandwishesto invitereviewersforaspecificissue. The suggested reviewer is also a user of the OpenScienceLinkPlatform.
TestingProcedure
Theusergoesto“EditorActivities”tab,andselectstoinvitereviewersforaspecific article/dataset. After adding any desired keywords s(he) presses“FindReviewers”andispresentedwithalistofrelevantreviewersfromtheOSLPlatformsorPubMed/MEDLINEAuthors.
The user requests to be presented with the ranked lists of Country, City,Journals,Authorsbasedonthenumberofpublicationswithoutenteringanytermsdescribingatopic.
ExpectedResult
Presentation of an errormessage to the user indicating that no data hasbeenprovided.
The user requests to be presented with the ranked lists of Country, City,Journals, Authors based on the number of publications and has enteredinappropriatetermswhichdonotdescribeabiomedicaltopic.
ExpectedResult
Presentationofanerrormessage to theuser indicating that inappropriatedatahasbeenprovided.
To verify the correct operation of the request for evaluation score for anAuthor
SetupThe user needs to have logged in the platform and have performed arequestforpublicationvolumebasedonatopic
TestingProcedure
TheplatformautomaticallyrequestsforanevaluationscoreforeachoftheAuthors that are displayed in the ranked list of Authors as a result of arequestforpublicationvolumebasedonatopic.
ExpectedResult
Presentation of a numerical value ranging from 0 to 100 for each of theAuthorsintherankedlistofAuthors.
3.5 Technical Evaluation Results The following table summarizes the technical evaluation results.As the table shows, all of thecontactedtestsweresuccessful,but,in5cases,theerrormessagesshouldbeaddedoredited(5I tests), Overall, this evaluation and testing shows that the first release of the platform isworkinginatechnicallysatisfactorymanner.
4.1.1 The overall open access landscape Agrowth trendhasbeenobserved for theopenaccess (OA)publicationmarketover thepastyears.Accordingto(Laakso,Welling,Bukvova,Nyman,&Björk,2011)openaccesspublicationsproductionhas increasedmore than10 timeswithin thepastdecadeboth in termsofarticlesandjournals.Infact,theirstudyindicatesthatOAisalreadyinitsconsolidationperiod.Numbersare quite indicative. The directory of open access journals (DOAJ)already includesmore than9800journalsand1.5millionarticlespublishedfromover120countries.OpenDOAR(Directoryof OpenAccess Repositories)hasmore than 2500 repositories listed, ROAR (Registry of OpenAccess Repositories)presents information about more than 2,500 repositories, while theROARMAP (Registry ofOpenAccessRepositoriesMandatoryArchivingPolicies)includesmorethan440openaccessmandatepolicies.TheDirectoryofOpenAccessBooks (DOAB) includesmore than 1,645Academic peer-reviewed books from55publishers. Moreover, the BASE(BielefeldAcademicSearchEngine)searchengine,whichfocusesonacademicopenaccesswebresources has already indexedmore than 56million documents and 2700 sources. HighWirePress by Stanford Universitycomprises an archive of over 2.3 million free full-text providedarticles. In the biomedical domain, in particular, PMC (PubMed Central)has already indexedmore than 734,000 OA articles with more than 1,200 journals providing their content withimmediatefreeaccessandover970journalsofferingalloftheirarticlesopenlyaccessible.
Figure 6: Open access publishing between 1993 and 2009.
Astudyin(Laakso&Björk,2012)showsthatalthoughinitiallyopenaccesspublishinghasbeenflourishing in the world of universities and scientific societies, commercial publishers havebecomekeyactorsontheOAscene,withanalmosttenfoldincreaseinthenumberofOAarticlesbeingpublishedwithina6yearsperiod(2005-2011).
Figure 7: Different Publisher Types for Open Access Articles.
Giventhatprofitcomprisesafundamentalconceptinthecommercialworldandpublicationofresearch includes a series of costs, reconciliation of publishing fees for OA journals has beenattempted through research and subscription funding. From this perspective, there are twomainmodelsforOAprovision;greenandgoldOA(Harnad,etal.,2004).InGreenOA,scientistspublish their findings through institutional or central repositories, or self-archiving (i.e., byplacing their peer-reviewed post-prints on OA web pages). In fact, research institutions andfunders, including the European Commission and Harvard, mandate their scientists to maketheirresearchfindingspubliclyavailableinordertomaximisetheimpactofthescientificworkconducted. In Gold OA, publishing is done through OA journals, such as BioMed Central andPLoS,whichmay either allowpublications for free or set a publishing fee to authors or theirfunders.Graduallymorethoroughclassificationsareintroduced,suchasdiamondOAreferringtopublishinginjournalswhichdonotchargeanyfeestoauthorsorreaders.Currently,onethirdofOAjournalschargepublishingfees,whileothersreceiveinstitutional,governmental,orthird-partyfunding(Kurata,Morioka,Yokoi,&Matsubayashi,2013).
4.1.2 Open Access and Biomedical Research AspresentedinFigure8,thevolumeofopenaccesspublicationshastremendouslyincreasedforthemajorscientificdisciplines,withthebiomedicinedomainshowingagrowthofmorethan16timesinnumberofOAarticlespublishedbetween2000and2011.Infact,accordingtoastudyin2013, OA has shown tremendous growth in the biomedical field with more than 50% of OAarticlescomingforit(Kurata,Morioka,Yokoi,&Matsubayashi,2013).OneofthemaintopicsfordebateoverOApublishingfocusesonitsassumednegativeeffectonthequalityofscientificpublishing.However,recentfindings(Laakso&Björk,2012)showthatOAjournals, and in the medical and health domain in particular, which are indexed in Web ofScienceand/orScopusandarepublishedinthefourlargestpublishingcountriesshowthesamescientific impact andqualitywith subscription journals. Interestingly,OA journalswhichposearticleprocessingchargesinordertofundpublishingareonaveragecitedmorethanotherOAjournals.
Figure 8: Open Access Articles across different disciplines.
4.2 Pilot 1:Open-access Data journals development
4.2.1 Analysis of Different Business Models and Competitors
Tomeetthecostofpublishing,authorswhopublishopen-accessareaskedtopayanopenaccessfee or article-processing charge (APC) per published paper. This cost is usually covered byresearchfundsfromthedepartmentsthatcontributetothiswork.
To partially remove this burden from the individual authors, some publishers have createddifferent Membership programs. A Membership Program enables academic and researchinstitutions, societies, groups, funders and corporations to actively support open access inscholarlypublishing,andhelpensurethemostwidespreaddisseminationoftheworkpublishedby their researchersormembers.Dependingon the typeofMembership,Member institutionscoversomeorallofthepublicationcostfortheirindividualresearcherswhentheysubmittoacertainpublisher.
Starting May 2014, this data journal will allow for thepublishing,discoveryandreusingofresearchdata.http://www.nature.com/scientificdata/about/
Different scientificfocusforthedata
Data in Brief(Elsevier)
Thejournal“welcomessubmissionsthatdescribedatafromallresearch areas”. It is indexed by DOAJ, Science Direct, andPubMed Central. It provides indication of impact via“Altmetrics”, illustrating the engagement of social mediacommunitieswitharticlesinDatainBriefbasedontheamountof activity from Twitter, Facebook, science blogs, mainstreamnews, and other sources captured by Altmetric.com for eachpublicationinthelastsixmonths.Volume1hasbeenpublishedinDecember2014with18datapapers.BySeptember2015 ithas published four volumes (one volume per quarter: vol. 12with 12; vol. 3 with 43; and vol. 43 with 102 data papers).About¾ofalldatapapersarefromthefieldsofBiochemistry,GeneticsandMolecularBiology;Pharmacology,ToxicologyandPharmaceutical Science; Immunology and Microbiology. Thearticleprocessing fee is500USD,witha50%discountby theendof2015.http://www.journals.elsevier.com/data-in-brief/
Direct competitor,with substantialemphasis onbiomedicalandlifesciences
LODUMaimsatconnectingdifferentdatasourcesacrossthe15facultiesanddepartmentsattheUniversityofMünsterthroughthe implementation of Open Access and Linked Open Dataprinciplesacrosstheuniversity.http://data.uni-muenster.de/
Potential contentsource
OpenAire EUproject
The FP7 project OpenAIRE aimed at offering the means topromote and realize the widespread adoption of the OpenAccess Policy by aggregating research publications whichcomprise European funded research output. OpenAIREplus,being its successor, further aims at linking this data to the
accompanying research and project information, datasets andauthor information. Apart from the data, it also provides aseries of functionalities such as statistics and reporting toolswhichcanbeusefultoprojectmanagers.http://www.openaire.eu/
re3data
It is a global registry of research data repositories fromdifferent academic disciplineswhich is funded by the GermanResearchFoundation(DFG).http://www.re3data.org/
Potential Contentsource
myExperiment
Itcomprisesapublicrepositoryofscientificworkflowswhichiscurrently supported by three European Commission 7thFramework Programme (FP7) projects: BioVeL, SCAP and theWf4Ever Project and the e-Research SouthandmyGridEPSRCPlatformgrants.http://www.myexperiment.org/
Potential Contentsource
OpenPHACTS
The Open PHACTS Discovery Platform integratespharmacologicaldatafromavarietyofinformationsourcesandoffers tools and services for applying questions on this datawithanaimtofacilitatepharmacologicalresearch.http://www.openphacts.org
Potential Contentsource
BiodiversityDataJournal
The first issue of the Biodiversity Data Journal, featuring 27articles,appearedinitsfinalversioninSeptember2013.BDJispublishedbyPensoftLtd.,Bulgaria.http://biodiversitydatajournal.com Potential Content
source, if theOpenScienceLinkdomainbroadens.Potentialpublisherand editor whocould benefit fromthe data journalmanagement andreview servicesand additionallythe trend miningand scientometricsservices.
GeoscienceDataJournal
At the end of November 2013, five articles appear on thewebsite of the Geoscience Data Journal, including the leadeditorialarticle[20].ThejournalispublishedbyWiley.http://onlinelibrary.wiley.com/journal/10.1002/%28ISSN%292049-6060
JournalofOpenArchaeologyData
The JournalofOpenArchaeologyData ispublishedbyUbiquityPress.Thefirstvolumewithninearticlesappearedin2012.AsofendNovember2013,volume2featurestwodataarticles.http://openarchaeologydata.metajnl.com
JournalofOpenPsychologyData
The Journal of Open Psychology Data is also published byUbiquity Press. By the end of November 2013 the journalwebsite provides access to two articles, one of which is theeditorialarticle.http://openpsychologydata.metajnl.com
Earth SystemScienceData
Earth System Science Data (ESSD) is an international,interdisciplinary journal for the publication of articles onoriginal research data (sets), furthering the reuse of high-quality data of benefit to Earth system sciences. The editorsencourage submissions on original data or data collectionswhichareofsufficientqualityandhavepotentialtocontributetotheseaims.Thejournalmaintainssectionsforregular-lengtharticles, brief communications (e.g. on additions to data sets)andcommentaries,aswellasreviewarticlesandspecialissues.http://earth-system-science-data.net/
Journal of TheJournalofChemical&EngineeringDataisamonthlyjournal
devoted to the publication of experimental data and theevaluation and prediction of property values. It is the onlyAmerican Chemical Society journal primarily concerned witharticles containing experimental data on the physical,thermodynamic, and transport properties of welldefinedmaterials including complex mixtures of known compositionsandsystemsofenvironmentalandbiochemicalinterest.http://pubs.acs.org/journal/jceaax
Journal ofPhysical andChemicalResearchData
TheJournalofPhysicalandChemicalReferenceDataispublishedby the American Institute of Physics (AIP) for the NationalInstituteofStandardsandTechnology(NIST).Theobjectiveofthe Journal is to provide critically evaluated physical andchemical property data, fully documented as to the originalsources and the criteria used for evaluation, preferably withuncertainty analysis. Critical reviews of measurementtechniques may also be included if they shed light on theaccuracyofavailabledata inatechnicalarea.Papersreportingcorrelationsofdataorestimationmethodsareacceptableonlyiftheyarebasedoncriticaldataevaluationandiftheyproduce“reference data”—the best available values for the relevantproperties.http://jpcrd.aip.org/
InternationalJournal ofRoboticsResearch
A leadingpeer-reviewed journal in its field formore than twodecades, the International Journal of Robotics Research (IJRR)was the first scholarly publication on robotics research. IJRRoffers incisiveand thought-provokingoriginal researchpapersand articles, perceptive reviews, and lively editorials onground-breaking trends issues, technical developments, andtheories in robotics by the outstanding scholars andpractitioners in the field. The Journal covers more than justnarrowtechnicaladvances-itembracesawidevarietyoftopics.IJRR also publishes high quality, peer reviewed datasets andmultimedia extensions alongside articles. This journal is amemberoftheCommitteeonPublicationEthics(COPE).http://ijr.sagepub.com/
F1000Research
F1000Research isanOpenSciencepublishingplatformofferingimmediate publication of posters, slides and articles with noeditorialbias.Allarticlesbenefitfromtransparentpeerreviewandtheinclusionofallsourcedata.http://f1000research.com
EcologicalArchives - DataPapers
EcologicalArchivespublishesmaterialsthataresupplementaltoarticles that appear in the ESA journals (Ecology, EcologicalApplications, Ecological Monographs, Ecosphere, EcosystemHealthandSustainabilityandBulletinof theEcologicalSocietyof America), as well as peer-reviewed data papers withabstractspublishedintheprintedjournals.EcologicalArchivesispublishedindigital,Internet-accessibleform.http://esapubs.org/archive/default.htm
GigaScience
Itisajournalwhichpublishes‘big-data’studiesfromtheentirespectrum of life and biomedical sciences. It is with BioMedCentral and supported by BGI – a Chinese non-profitorganisation which claims to be the largest genomics
organisation. It is indexed in the Directory of Open AccessJournals(DOAJ),PubMedandPubMedCentral.
interestedprimarily in datajournalmanagement andreview servicesand additionally inthe trend miningand scientometricsservices.
GenomicsDataItisanopenaccessjournalpublishedbyElsevierwhichcoversall aspects of genome-scale analysis, including nucleic acidsanalysis, microarray and next-gen sequencing data and allorganisms.
JournalofOpenPublic HealthData
It is a data journal which is published by Ubiquity Press. Itsmainfocusisondatawithreusabilitypotentialorwhichcanbeusedforresearchvalidationpurposes.
ScientificData
ItisadatajournaltobepublishedbyNaturePublishingGroupwhich currently focuses on datasets from the life, biomedicaland environmental science communities, but is intended tocover a broad range of scientific disciplines. It has beenscheduled to launch inMay 2014. Datasets are not hosted bythepublisher. Insteadcommunity-recogniseddatarepositoriesare expected to store them, if available, or other repositoriessuchastheDryad.
Table 31: Pilot 1 Competitors.
4.2.2 Insights for the formulation of realistic business and sustainability plans
A.Spendingsperyear
ActivityRoughcostestimate
A0 Policies,businessprocess,legalissues OSL
A1 WebsiteandpresenceinsocialnetworksInitial development, software and functionality upgrades, expanding thepresenceinsocialnetworks
2,000
A2 Journalmanagement1/2 position, bachelor’s degree, working knowledge of English, someunderstanding of biomedical issues (2,500 Euro per month, includingsocialandotherinsurance,includingtheinsurancepaidbytheemployingcompany)=12monthsx1,250Europermonth(articleprocessing,e.g.pagesetting,isincludedhere)
15,000
A3 Printingandmailing4issuesx1,500Euro(4 colours cover, the usual issue is with black and white body; whennecessary, individualpages /galley/waybeprinted in colour;printout–between300and500copies;format–A4)
B4 Author’sfees(institutional membership may provide stability and increasepredictability,butisnotexpectedtoincreaserevenues,sinceitwillleadtowaivingauthors’fees)4issuesx15articlesperissueinaverage=60articlesFeeof400Europerarticle(oranequivalentamountforaninstitutionalmembership)Remark: Fees for publications by members of the Editorial Board and‘active’reviewersmaybewaived.Hence,thiscalculationisbasedonlyonthenumberofarticlesforwhichauthorsarerequestedtopaythefee.
24,000
Total 36,000
Table 33: Possible Revenues for Pilot 1.
Possibilitiestoreducecosts:
Increasetheefficiencyofjournalmanagementandjournalpreparation1. Assignapersonwithlowerqualificationtomanagethejournal(notdesirable).2. Seeksynergiesbyassigningthesamepersontomanagethreeorfourjournals.3. Request that authors use a dedicatedwriting tool (that provides the articles in print-
5. Requestthatarticlesbewritteninalanguagethatdoesnotrequirecopy-editing.6. Provide paid services for formatting submissions to journal requirements (in case a
contributionisnotformattedaccordingtojournalrequirements)7. Provide paid copy-editing services for cases when a contribution is not written with
EasyChair supports threedifferent kinds of licenses(free, professional andexecutive). There arerestrictions on the number offreelicensesthatcanbeissuedto a single conference.Professional conferenceorganisers and societies orcompanies organizing anumber of conferences arepotentialcustomers.Pricesforallnon-free licensesarebasedonthenumberofsubmissions
FluidReview has flexiblepricing options that can meettheneedsof anyorganization,bigorsmall.Afreetrialperiodis provided for testing. Aspecial team works with thecustomer to determine whichplan and schedule is bestsuited for his requirements.FluidReview requires anannualcommitment.Afteroneyear, thecustomermaycancelhis subscription or continuewith the service. Flexiblepayment schedules areprovided.
PublicKnowledgeProject - OpenJournalSystems
OJSisdescribedinDeliverable9.4.3.(Table3).
All of the events (discussionforums, lectures, seminars,workshops, and symposia ontopical and timely issues fromevery discipline) arefreeandopen to the public, students,faculty,staffandschools.OJSisbased on the support of ourprivate and academicsponsors.
Arnetminer follows the thecommercial open sourcebusiness model. Because theopen source product isavailable for free, potentialcustomers can download,install, and use the productwithout getting in touch withthe commercial firm behindtheproduct.At thesametime,the firm can track viadownload registration and
community forum activitieswho is actually using theproduct. A lead analysis canthendeterminewhichoftheseusers might be potentialcustomers. More often thannot, however, the firm willwait until a non-paying usersteps forward and asks for asales contact to purchase anyof the services outlined in therevenue generation section.Thus, leads emerge from theexisting user community,either voluntarily or byanalysis.
It is a peer-reviewmanagement system, used bythousands of events in over100 countries, that facilitatesthe management ofconferences, workshops, andsymposia, yet is flexibleenough that it also powersjournals, grants, books, andcompetitions.OpenConfisalsomultilingual, with translationsfor author and reviewerinterfaces included for over adozen languages. OpenConf isavailable in multiple editionsto meet various needs, andmayeitherbe licensed foruseonone’sownserverorfromamanagedhostingservice.
Peerage ofScience
It is described in Deliverable 9.4.3. (Table 3).https://www.peerageofscience.org
For Authors and Reviewers,the submission and peerreview process in Peerage ofScience is free of any charges.Subscribingjournals,however,have different publishingmodels. Open-access journalsusually require fees fromauthors upon acceptance;these fees must be stated toAuthorsinthepublishingoffersent via Peerage of Science,but Peerage of Science is notcollectinganyfeesitself.
AcademicKarma
Academic Karma is described in Deliverable 9.4.3.(Table3).http://academickarma.org/
Authors access faster, higherquality and more transparentpeer review outside thejournalsystem,forfree.Editors make access to peer-
review freely available to all,bringing down costs of open-accesspublishing.Reviewers maintain thequality and reproducibility ofthe open access scientificliterature whiledemonstratingexpertiseintheirfield.
Winnower is an open accessonline scholarly publishingplatform that employs openpost-publication peer reviewwhich is free for all membersandworks in fourbasic steps:Submission, Review, RevisionandArchival.
Table 35: Pilot 2 Competitors and business models
4.3.2 Targeted customers and stakeholders. Targetedcustomersandstakeholdersforpilot2havebeenanalysedinDeliverable9.4.3.(Table9and10).
4.4 Pilot 3: Services for detection and analysis of research trends
4.4.1 Analysis of different business models and competitors
TechCast is an academic researchproject that depends on membershipsubscriptionstosupportitsresearch.Itoffers professional subscriptions for 1,2 and 3 years, as well as student andacademic group subscriptions. Trialmembership is also available forlimited access. Subscriptions include 2options
1. Professional Subscription forGeneral Managers, TechnologyOfficers, CIOs, Strategic Planners,Technology Transfer, BusinessDevelopment, and others inbusiness and governments who
Arnetminerfollowsthethecommercialopen source business model. In thismodel,theproductisavailableforfree,potential customers can download,install, and use the product withoutgetting in touch with the commercialfirm behind the product. At the sametime, the firm can track via downloadregistration and community forumactivities who is actually using theproduct. A lead analysis can thendetermine which of these users mightbe potential customers. More oftenthan not, however, the firm will waituntil a non-paying user steps forwardandasksforasalescontacttopurchaseany of the services outlined in therevenuegenerationsection.Thus,leads
Competitor BriefDescription Businessmodelemerge from the existing usercommunity, either voluntarily or byanalysis.
TrendMinerProject
It is described in Deliverable 9.4.3. (Table 4).http://www.trendminer-project.eu/index.php/obj TrendMinerisanEU-fundedproject
GoogleTrends Google Trends is described in Deliverable 9.4.3.(Table4).http://www.google.com/trends/
Google Trends follows the thecommercial open source businessmodel. Someof the functionalities areavailable for free, potential customerscan use the productwithout getting intouchwiththecommercialfirmbehindtheproduct.Registereduserscanhaveaccesstomorefunctionalities.Analysiscan determine which of the usersmightbepotentialcustomers.
Institute ForTheFuture
IFTF is described in Deliverable 9.4.3. (Table 4). http://www.iftf.org/
IFTF is an independent, non-profitresearch institute IFTF is sustainablewiththeimportantfinancialsupportofacademic institutions and privatesponsors.
EssentialScienceIndicators
ESI is described in Deliverable 9.4.3. (Table 4). http://wokinfo.com/products_tools/analytical/essentialscienceindicators/
ESI depends on membershipsubscriptions offering a singlesubscription model or anacademic/research institutionsubscription. Registered users need tosign in to have access to a web-basedplatform.It includes 10 year rolling coverage,updatedeverytwomonths.
Table 36: Pilot 3 Competitors and business models
4.4.2 Insights for the formulation of realistic business and sustainability plans Trendanalysis is according tomanydifferent sources thebase fordecisionmaking todayandespecially in the future. The availability and –more – important the accessibility of big datavolumes ask for tools to interpret available data. The big financial potential lies behind theanalysisoftrendsinordertobeabletocreateproductswhentheyareneeded.Companiesbeingabletodelivercustomtailoredproductswhentheyareneededmostwillhaveauniquesellingpoint andwill therefore be better than competitors. As a good example a US based companyGenomeQuestdotrendanalysisintheareaofBioinformatics–specializedingenesequencesinpatents. The company’s revenue has doubled every year and is now at around 10million USdollars.Suchasimplebusinessmodelnotevenprovidingadvancedstatisticaland/orsemanticalgorithmsallowsforsuchatremendousgrowth.The technologies developed in Open Science Link will go far beyond what players likeGenomeQuestdo.We foreseeaCAGRof15 to30% in theareaof trendanalysis. In2016 theglobalmarkedwill have reached a statewhere especially Europe should be stable positionedbefore global M&A deals start. Our “product” will by then be is in fully developed state andbeyondstateoftheart.
4.5 Pilot 4: Services for Dynamic Researchers' Collaboration
4.5.1 Analysis of different business models and competitors
Competitor BriefDescription Businessmodel
ResearchGate
ResearchGate is described in Deliverable9.4.3. (Table 5).https://www.researchgate.net
ResearchGate is a for-profit enterprise.Theyhavehadtworoundsoffundingandhave investors thathavepreviouslybeeninvolved with Facebook, LinkedIn andPayPal Researchgate has raised anotherinvestment round of $35M with moneyfrom Microsoft and Tenaya Capital,Dragoneer Investment Group and ThriveCapital.. Over the longer haul, the firmhopes to charge companies anduniversities forusing it toadvertise jobs,and to operate a marketplace forlaboratory materials. It has no plans topost other advertising, though, nor tochargeitsusersdirectly.
ArnetMinerArnetMiner is described in Deliverable9.4.3.(Table5).http://arnetminer.org/
Arnetminer follows the the commercialopen source business model. In thismodel, the product is available for free,potentialcustomerscandownload,install,and use the product without getting intouch with the commercial firm behindthe product. At the same time, the firmcan track via download registration andcommunity forum activities who isactuallyusingtheproduct.Aleadanalysiscan thendeterminewhich of theseusersmightbepotentialcustomers.Moreoftenthannot,however,thefirmwillwaituntilanon-payinguserstepsforwardandasksforasalescontacttopurchaseanyoftheservices outlined in the revenuegeneration section. Thus, leads emergefrom theexistinguser community, eithervoluntarilyorbyanalysis.
The business model depends onsubscriptionstosupportitssustainability.Trial membership is also available forlimited access.Subscriptions areexamined on a case by case basisaccording to the size and needs of thecustomer
Academia.edu Academia.eduis described in Deliverable9.4.3.(Table5).https://www.academia.edu
Theservice is freeofchargeto itsover2million users and does not offer anypremium services or institutionalmemberships. However, CEO RichardPriceexpressedhis belief that thebusiness will generate revenue in thefuture by providing ‘trending researchdatatoR&Dinstitutionsthatcanimprovethequalityoftheirdecisionsby10-20%.’
MethodspaceMethodspace is described in Deliverable9.4.3. (Table 5).http://www.methodspace.com/
Methodspace is a non-profit free onlinenetwork of researchers, resources anddebates.
The platform serves two interrelatedcustomer segments with distinct offers.Theindividualresearcherisservedwithafreemiummodel,userscanupgradetheircloudspace foramonthly feestartingat4,99 USD. In addition, since May2012Mendeley partners withinformation management servicesprovider SWETS for the MendeleyInstitutional Edition which offers real-timeanalyticstolibrariesforafee.
MicrosoftAcademicSearch
It is described in Deliverable 9.4.3. (Table5).http://academic.research.microsoft.com
Microsoft Academic Search services areprovidedbyMicrosofttothepublicfreeofcharge. The user must complete theregistration process by providing withcurrent, complete and accurateinformation as prompted by theapplicable registration form. Forpurposes of maintaining and improvingMicrosoft Academic Search services,Microsoft may record activities atMicrosoftAcademicSearchsiteandmakeuseofcertaininformation.
Biowebspin It is described in Deliverable 9.4.3. (Table5).
Biowebspin identifies itself as a leadingprofessional network in Life Sciences,connecting academia and industry.Biowebspin is the platform to find andconnect with the right partners, and theplace to network, work, and look upinformation thanks to smart tools andboards.Inbetatestfromlate2012toearly2013, Biowebspin is now in the top 3 ofthe most visited websites worldwide inLifeSciences.BiowebspinSAwascreatedin 2013 in Switzerland (capital: CHF550,000).
LinkedIn It is described in Deliverable 9.4.3. (Table5).
LinkedIn is publicly held and has adiversifiedbusinessmodelwithrevenuescoming from member subscriptions,advertisingsales,andtalentsolutions
DIRECT2Experts It is described in Deliverable 9.4.3. (Table5).
TheDIRECT2Expertsnetworkisopentoall biomedical institutions, is a pilotproject facilitated by the ResearchNetworking Working Group of the NIH-supported Clinical & TranslationalScienceAward(CTSA)
BioMedUSA BioMedUSA is described in Deliverable9.4.3.(Table5).
Theopenaccessserviceisentirelyfreeofchargetoallscientistsandadministratorsatnon-profitresearchinstitutionsaswellas at commercial biomedicalorganizations.
Table 37: Pilot 4 Competitors and business models.
4.5.2 Insights for the formulation of realistic business and sustainability plans Theaforementionedplayersareavitalsignthatcollaborationcan’tbeestimatedhighenough.Asastrongtrendinsciences,subjectsgetmoreandmorecomplexandcannotbeinvestigatedby one genius brain alone. The average number of authors of high-end publications (Nature,Science,Celletc.)increasesconstantly.Itisofcrucialinteresttoprovidetoolsforcollaborationbeyond todays habits: basically sending unstructured emails to possible collaborators. Datamustbe integrated,automaticallyanalysedandamachineshouldbeable todrawconclusionsand suggest hypothesis to teams. InOpen Science Linkwepartly aim for suchnewways andgenerallyspeakingweseeahugemarket.OnechallengeablefactisthatResearchGateraised30million€fromTheGatesFoundation.Targetedcustomersandstakeholdersforpilot4havebeenanalysedinDeliverable9.4.3.(Table12).
4.6 Pilot 5:Research evaluation services
4.6.1 Analysis of different business models and competitors
Competitor BriefDescription Businessmodel
ThomsonReutersWeb ofScience
WebofScience®is described in Deliverable 9.4.3. (Table 6). http://thomsonreuters.com/web-of-science/
Thomson Reuters depends onmembership subscriptions offering asingle subscription model or anacademic/research institutionsubscription. Registered users need tosign in to have access to a web-basedplatform.
Thomson Reuters depends onmembership subscriptions offering asingle subscription model or anacademic/research institutionsubscription. Registered users need to
ResearchGate is a for-profit enterprise.Theyhavehadtworoundsoffundingandhaveinvestorsthathavepreviouslybeeninvolved with Facebook, LinkedIn andPayPal Researchgate has raised anotherinvestment round of $35M with moneyfrom Microsoft and Tenaya Capital,Dragoneer InvestmentGroupandThriveCapital.. Over the longer haul, the firmhopes to charge companies anduniversitiesforusingittoadvertisejobs,and to operate a marketplace forlaboratory materials. It has no plans topost other advertising, though, nor tochargeitsusersdirectly.
ArnetMiner ArnetMiner is described in Deliverable9.4.3.(Table6).http://arnetminer.org/
Arnetminer follows the the commercialopen source business model. In thismodel, the product is available for free,potential customers can download,install, and use the product withoutgettingintouchwiththecommercialfirmbehindtheproduct.Atthesametime,thefirmcan trackviadownload registrationand community forum activities who isactually using the product. A leadanalysis can then determine which oftheseusersmightbepotentialcustomers.More often than not, however, the firmwill wait until a non-paying user stepsforward and asks for a sales contact topurchase any of the services outlined inthe revenue generation section. Thus,leads emerge from the existing usercommunity, either voluntarily or byanalysis.
GoogleScholarMetrics
Google Scholar Metrics is described inDeliverable 9.4.3. (Table 6).http://scholar.google.lt
Itpresentsevaluationsof journalsbasedon popular evaluation metrics. TheOpenScienceLink models and evaluationservices could be feeding the platformenrichingtheprovidedevaluation.
Harzing’sPublish orPerish
Harzing’s Publish or Perish is describedin Deliverable 9.4.3. (Table 6).http://www.harzing.com/pop.htm
Microsoft Academic Search is describedinDeliverable9.4.3.(Table6).
Microsoft Academic Search services areprovided byMicrosoft to the public freeof charge. The user must complete theregistration process by providing withcurrent, complete and accurateinformation as prompted by theapplicable registration form. Forpurposes of maintaining and improvingMicrosoft Academic Search services,
Competitor BriefDescription BusinessmodelMicrosoft may record activities atMicrosoftAcademicSearchsiteandmakeuseofcertaininformation.
Scopus
Scopus is described in Deliverable 9.4.3.(Table6).http://www.elsevier.com/online-tools/scopus
Due to high demand, scopus can onlyprovide pricing information to therepresentative of the institution whowould decide on purchasing asubscription. Institutional access isrequiredtofullybenefitfromScopus.
It is a relatively small company withfocus on engineering and domainknowledge. Started by wining intoElsevier’s Apps for Science competition.TheytookonadditionalinvestmentfromDigitalScience,andarestillapartof thegroup today.Usersnow include someoftheworld’sleadingjournals,fundersandinstitutions.
Table 38: Pilot 5 Competitors and business models. Targetedcustomersandstakeholdersforpilot5havebeenanalysedinDeliverable9.4.3.(Table13).
4.6.2 Insights for the formulation of realistic business and sustainability plans Researchevaluationisasimportantastrendanalysisandcanpartlybebasedonorprofitfromit.Asstatedabove,trendanalysisandtrenddetectionarekeywhenitcomestobefirstmover.The demand for reliable, scientific correct measures is obvious. Doing trend analysis onmediocredesignedindicatorsisrathersenseless.Sothefirstneedistocreatereliableindicators,whichgobeyondtheabovementionedcompetitors.OpenScienceLinkdoesexactlythis.Havingthem at hand the next issue which can be addressed is flexibility which should range fromresearcher,toresearchgroup,todepartment,toUniversity,tocity,countyandfinallystate.Alsoourmultidimensionalapproachtofocusforexampleonbioinformaticsandbeingabletoprofilee.g.EuropeagainstNorthAmericasetsOSLapartfromexistingtechnologies.Allinallitcanbenamedrealistic,thatourtechnologiesbringaddedvaluetocustomers.Therebythebusinessmodelcanbehighjackedbutextendedtomoreflexibility,moreaccuracyandmoresemanticcrosslinksforaddressingmeaninginsteadof“words”.Thinkableissellingsingletopicsinayearlysubscription(e.g.Bioinformatics,Europe,2015).SingleindividualsmaygetitcheapwhereInstitutionsgetahigherpricedproductandsoon.SeeingtheroughnumberfromThomsonReutersthemarketseemstobehugeandfastgrowing.WeestimatealsohereaCAGRof10-20%.
5 Summary and Conclusions Thisreportservesasayear3evaluationoftheOpenScienceLinkproject fromtheperspectiveofinterestedstakeholders,technicalachievementsandbusinessplans.Basedonstakeholdersviews,amongthemostimportantachievementsinthisperiodhasbeenthepublicationofthefirstdatapapersinBiomedicalDataJournal,andthedevelopmentofnewversion of the OpenScienceLink platform.With regards to the issues that should attractmoreattention in the future, the three major points were: the exploitation, the attraction of moreusers of the platform and knowledge transfer.Stakeholders have also provided a number ofsuggestions/criticisms that are reported in this document and will be taken into account forimprovement of the platform in the future. Based on previously defined Key PerformanceIndicators(KPIs)theactualprogressreportedhereshowsthatmostofthegoalswereachievedeveniftheinitialplanofexpectedprogresscouldbecharacterizedasveryambitious.Concerning the technicalevaluation,almostallof thecontacted testsweresuccessful,and thismatureversionoftheplatformwasshowntoworktechnicallyinaverysatisfactorymanner.Finally,fromafinancialpointofview,weevaluatedtheoverallmarketintowhichtheplatformand its services are positioned, the competitors and the business models that they use. Acomparison of theOpenScienceLink pilot serviceswith these products has been conducted inordertounveiladvantagesoftheOSLplatformovercompetitors.AdetailedanalysisofbusinessplanshasbeenincludedinaseparatedeliverableD9.3.
Kurata,K.,Morioka,T.,Yokoi,K.,&Matsubayashi,M.(2013).Remarkablegrowthofopenaccessin the biomedical field: analysis of PubMed articles from 2006 to 2010. PLoS ONE,8:e60925.
Laakso,M., & Björk, B.-C. (2012). Anatomy of open access publishing: a study of longitudinaldevelopmentandinternalstructure.BMCMedicine10(1).
