ContaCt Standard Performance Evaluation Corporation (SPEC) 7001 Heritage Village Plaza, Suite 225 Gainesville, VA 20155, USA SPEC Research Group Chair: Samuel Kounev ([email protected]) Web: http://research.spec.org ® ContEntS of thiS iSSuE 2 SPEC Research Group Officers 2 SPEC Research Working Groups 3 Foreword from Chairs and Editor 3 SPEC Research Group Mission Statement 3 SPEC Announcements 5 Report: Cloud Working Group 6 Report: IDS Benchmarking Working Group 6 Report: Big Data Working Group 7 Report: DevOps Performance Working Group 8 Three Distinctions of Alexandru Iosup 9 Winner of the SPEC Distinguished Dissertation Award 2015 10 Welcome to ICPE 2016 in Delft 10 ICPE 2017 Preliminary Announcement 10 LIMBO: a New Peer-reviewed Tool in SPEC Repository 11 Graphalytics: Benchmarking Platforms for Large- Scale Graph Processing 12 SPEC Symposium’16 – Energy Efficiency 13 InspectIT: Open Source APM Solution 14 diagnoseIT: A new APM Research Project 15 SPECtacular Awards 2015 and 2016 fouR SPEC RESEaRCh woRkinG GRouPS REPoRt on thEiR PRoGRESS The SPEC Research Working Groups report on the progress, articles, benchmarks, and technical reports published in the year 2015. The Working Groups are always open for new members, feel invited to join us! Read more on pages 5-7 iCPE 2017 will bE hEld in l’aquila, italy Vittorio Cortellessa and Walter Binder, the General Chairs of the next ACM/SPEC International Conference on Performance Engineering (ICPE 2017), invite inter- esting high-quality submissions. The conference will take place in March 2017 in L’Aquila, Italy. Read more on page 10 yufEi REn winS thE SPEC diStinGuiShEd diSSERtation awaRd Among multiple high-quality dissertations that have been submitted to compete in the SPEC Distinguished Dissertation Award, the committee awarded Yufei Ren for his work on contemporary operating system virtualization techniques. Additionally, Cong Xu was distinguished with honorable mention. Read more on page 9 nEw toolS in thE SPEC REPoSitoRy of PEER-REviEwEd toolS LIMBO and DynamicSpotter were accepted for the SPEC peer-reviewed tools repository. More tools are currently under review and further submissions are invited. Read more on pages 3 and 10 1 SPEC Research SM Group newsletter Volume 2, Issue 1 March 2016 ℠ Research
Transcript
ContaCt
Standard Performance Evaluation Corporation (SPEC)7001 Heritage Village Plaza, Suite 225Gainesville, VA 20155, USA
SPEC Research GroupChair: Samuel Kounev ([email protected])Web: http://research.spec.org
®
ContEntS of thiS iSSuE 2 SPECResearchGroupOfficers2 SPECResearchWorkingGroups3 ForewordfromChairsandEditor3 SPECResearchGroupMissionStatement3 SPECAnnouncements5 Report:CloudWorkingGroup 6 Report:IDSBenchmarkingWorkingGroup6 Report:BigDataWorkingGroup7 Report:DevOpsPerformanceWorkingGroup8 ThreeDistinctionsofAlexandruIosup9 Winner of the SPEC Distinguished Dissertation
Award201510 WelcometoICPE2016inDelft10 ICPE2017PreliminaryAnnouncement10 LIMBO: a New Peer-reviewed Tool in SPEC
fouR SPEC RESEaRCh woRkinG GRouPS REPoRt on thEiR PRoGRESS TheSPECResearchWorkingGroupsreportonthe
progress, articles, benchmarks, and technical reportspublished in the year 2015.TheWorkingGroups arealwaysopenfornewmembers,feelinvitedtojoinus!
Read more on pages 5-7
iCPE 2017 will bE hEld in l’aquila, italy
VittorioCortellessaandWalterBinder, theGeneralChairsofthenextACM/SPECInternationalConferenceonPerformanceEngineering(ICPE2017),inviteinter-esting high-quality submissions. The conference willtakeplaceinMarch2017inL’Aquila,Italy.
Read more on page 10
yufEi REn winS thE SPEC diStinGuiShEd diSSERtation awaRd Amongmultiplehigh-qualitydissertationsthathave
beensubmittedtocompeteintheSPECDistinguishedDissertation Award, the committee awarded YufeiRen for his work on contemporary operating systemvirtualization techniques. Additionally, Cong Xu wasdistinguishedwithhonorablemention.
Read more on page 9
nEw toolS in thE SPEC REPoSitoRy of PEER-REviEwEd toolS
LIMBOandDynamicSpotterwereaccepted for theSPEC peer-reviewed tools repository. More tools arecurrently under review and further submissions areinvited.
Steering Committee: Samuel kounev, University of Würzburg, Germany qais noorshams, IBM, Germany tilmann Rabl, Bankmark UG, Germany Petr tůma, Charles University of Prague, Czech Republic J. nelson amaral, University of Alberta, Canada andré van hoorn, University of Stuttgart, Germany alexandru iosup, TU Delft, The Netherlands lizy John, University of Texas at Austin, USA klaus-dieter lange, HPE, USA Meikel Poess, Oracle Corporation, USA
Publicity officer: Christoph heger, NovaTec Consulting GmbH, Germany
newsletter Editor: Piotr Rygielski, University of Würzburg, Germany
SPEC RESEaRCh woRkinG GRouPS Cloud working Group
Chair: alexandru iosup, TU Delft, The Netherlands Secretary: aleksandar Milenkoski, University of Würzburg, Germany Release Manager: nikolas herbst, University of Würzburg, Germany http://research.spec.org/working-groups/rg-cloud-working-group.html
idS benchmarking working GroupChair: Marco vieira, University of Coimbra, Portugal vice-Chair: alberto avritzer, Siemens Corporate Research, USA Secretary: aleksandar Milenkoski, University of Würzburg, Germany Release Manager: aleksandar Milenkoski, University of Würzburg, Germany http://research.spec.org/working-groups/ids-benchmarking-working-group.html
big data working GroupChair: tilmann Rabl, Bankmark UG, Germanyvice-Chair: todor ivanov, Goethe University Frankfurt, Germany Secretary: Meikel Poess, Oracle Corporation, USA Release Manager: John Poelman, IBM, USA http://research.spec.org/working-groups/big-data-working-group.html
devops Performance working GroupChair: andré van hoorn, University of Stuttgart, Germany vice-Chair: felix willnecker, fortiss GmbH, Germany Secretary: Robert heinrich, KIT, Germany Release Manager: alexander wert, NovaTec Consulting GmbH, Germany
We are delighted to present to you the next is-sue of the SPEC Research Group Newsletter. Thisregular publication provides information on latestdevelopments, news, and announcements relevant tothe benchmarking and quantitative system evaluationcommunities.Our newsletter is part of ourmission tofostertheexchangeofknowledgeandexperiencesbe-tweenindustryandacademiainthefieldofquantitativesystemevaluationandanalysis.
The past year has been very intense and excitingfor the SPEC RG. To the major activities and mile-stones reached,we include: theorganizationof ICPE2016,acceptingtwonewtoolsfortheSPECrepositoryof peer-reviewed tools, successful year for the fourworkinggroups,releasingtwotechnicalreports:(“NewCloud Metrics”, and “Performance-oriented DevOps:ResearchAgenda”),selectingthewinnerof theSPECDistinguishedDissertationAward,andmanymore.
We have been actively working on preparation,planningandorganizationoftheICPE2016.Wehopethat the vividexchangeof ideasduring theupcomingICPE2016willbeagreatmotivationforthenextyearofscientificandengineeringwork.
Wehopethatyouwillenjoyreadingthenewsletter.We welcome and encourage your contributions forarticlesandsuggestionsforfuturecoverage.
Samuel Kounev (University of Würzburg), André van Hoorn (University of Stuttgart),
Piotr Rygielski (University of Würzburg)
SPEC RESEaRCh GRouP MiSSion StatEMEnt
The SPEC Research Group (RG) is one of thefour groups of the Standard Performance Evalua-tion Corporation (SPEC). Its mission is to promoteinnovative research in the area of quantitative sys-temevaluation and analysis by serving as a platformfor collaborative research efforts fostering the inter-action between industry and academia in the field.
The scopeof the group includes computer bench-marking, performance evaluation, and experimentalsystemanalysisconsideringbothclassicalperformancemetricssuchas response time, throughput,scalabilityandefficiency,aswell asothernon-functional systemproperties includedunder the termdependability,e.g.,availability, reliability, and security. The conducted re-search efforts span the design of metrics for system
SPEC announCEMEntS
Zenodo repository for persisting iCPE 2016 submissions artifacts
January 20, 2016
Pursuing itsmissiontoserveasaplatformforcol-laborativeresearchefforts,theSPECResearchGroup(RG) has opened a new community on the Zenodoopendigitalrepository.ThepurposeofthecommunityistoarchiveandshareresearchartifactsassociatedwithRGactivities,inparticulartheACM/SPECInternationalConference on Performance Engineering (ICPE).By providing a stable location for artifacts required toreproduceandreusetheresearchresults,RGaimstobroadenthepracticalimpactofICPEcontributionsandnurture an environment where detailed disclosure ofresearchresultsisanacceptedpractice.Atthetimeofwriting,overa thirdof theauthorspresentingat ICPE2016hasalreadyuploaded theassociatedartifacts tothe repository.Moreuploadsarebeingprepared.Therepositoryisavailableonlineundertheprovidedlink.
http://zenodo.org/collection/user-spec-rg
Petr Tůma (Charles University of Prague)
the SERt 1.1.1 January 18, 2016
The first quarter of 2016 sees the release of ver-sion1.1.1of theServerEfficiencyRatingTool (SERT)fortheUSEPAENERGYSTARforComputerServersprogram.ThisupdateaddssupportfortheIBMPower8processor family in little endian mode, validated withUbuntu14.04.03andIBMJ98.0SR1.
ItalsoincludesanewrevisionofPTDaemon,v1.8.0.Thisaddssupport for theYokogawaWT310E&333EandtheHiokiPW3335,3336&3337poweranalyzers.AllworkinsinglechannelACmode,andalsoinmulti-channelmodeforthosemodelsthatsupportmorethanoneinputchannel.
As with every SERT update, the user guide hasbeenthoroughlyrevised,basedonuserfeedbackandsupportquestionsreceivedsincethelastrelease.ThisupdateisavailableinbothEnglishandMandarin.
Mike Tricker (SPECpower Committee Secretary)
evaluation aswell as the development ofmethodolo-gies,techniquesandtoolsformeasurement,loadtest-ing, profiling, workload characterization, dependabil-ity and efficiency evaluation of computing systems.
article accepted for publication in the prestigious journal “aCM Computing Surveys”
July 28, 2015
A paper produced by the SPEC RG IDSBenchmarkingWorkingGroup has been accepted forpublication inACM Computing Surveys (CSUR). Thepapersurveysthestate-of-the-artintheareaofevalua-tionofintrusiondetectionsystems.
CSURpublishessurveyarticlesandtutorialsrelatedto computer scienceandcomputing.According to theISI Journal Citation Reports, CSUR has among thehighestimpactfactorsofallcomputersciencejournals.
Aleksandar Milenkoski, Marco Vieira, Samuel Kounev, Alberto Avrtizer, and Bryan D. Payne. Evaluating Computer Intrusion Detec-tion Systems: A Survey of Common Practices. ACM Computing Sur-
veys, 2015, ACM.
www.research.spec.org
the 13th iEEE international Conference on autonomic Computing (iCaC 2016)
September 29, 2015
The 13th IEEE International Conference onAutonomic Computing (ICAC 2016) will be held inWürzburg,GermanyonJuly19-22,2016.
ICACistheleadingconferenceonautonomiccom-puting,itsfoundations,principles,engineering,technol-ogies, and applications. Nowadays, complex systemsofalltypes,likelarge-scaledatacenters,cloudcomput-inginfrastructures,cyber-physicalsystems,theinternetof things, self-organizing systems, organic computingsystems, cognitive computing systems, or self-awarecomputing systems, are increasingly complex, involv-ingmanyactive, interconnectedcomponentsrequiringcareful coordination.Being impossible forahuman tomanagesuchsystems,theautonomiccomputingpara-digmwith itssupport forself-managementcapabilitiesbecomes increasingly indispensable for the compo-nentsofourITworld.
http://icac2016.uni-wuerzburg.de/
www.research.spec.org
SPEC SfS2014 benchmark beceived an update November 19, 2015
This update fixes a possible sensitivity to systemsthathaveinlinecompressionenabledandincorporatesallerratafixessincetheinitialrelease.Goingforward,
dynamicSpotter was accepted for SPEC tools Repository
January 21, 2016
AnewtoolwasacceptedforSPECRG’srepositoryof peer-reviewed tools for quantitative system evalu-ation and analysis: DynamicSpotter. DynamicSpotteris a framework for measurement-based, automaticdetection of software performance problems in Java-based enterprise software systems. DynamicSpottercombines the concepts of software performance anti-patternswithsystematicexperimentation.
More tools are currently under review and furthersubmissionsareencouraged.
http://research.spec.org/tools/
www.research.spec.org
SPECjbb2015 released November 18, 2015
SPECannouncesthereleaseofSPECjbb2015,anenhanced Java server benchmark that now supportsvirtualizedhosts,addsstricterperformancevalidation,andspecifiesfasterresponsetimesforamoreaccurateassessmentof total capacity inavarietyof real-worlddeployments.
https://spec.org/jbb2015/
www.spec.org
book: Server Efficiency—Metrics for Computer Servers and Storage
June 19, 2015
Klaus-Dieter Lange, SPEC board member andSPECpower chair, togetherwithKarlHuppler, a long-time supporting contributor to SPEC, are the authorsof the recently released “Server Efficiency—MetricsforComputerServers andStorage”, the 12th book intheASHRAEDatacomSeries.Hupplercalls thebook“avaluableguideforchoosingenergy-efficiencymeas-ures for computer servers andassociated storage.” ItcanbepurchasedontheASHRAEwebsite.
TheSPECRGCloudGrouphascontinued,overthecourseof2015,topursueitslong-termmissionoftakingabroadapproach, relevant forbothacademiaand in-dustry,tocloudbenchmarking,quantitativeevaluation,andexperimentalanalysis.Ourgroupfocusesonnovelcloudpropertiessuchaselasticity,performance isola-tion, dependability, and other non-functional systemproperties, inadditiontoclassicalperformance-relatedmetricssuchasresponsetime,throughput,scalability,andefficiency.Amongthekeywordsmostdiscussedinthegroup,over2015,arecloudmetrics,elasticity,auto-scaling, service definition and configuration, and dataasaservice.
Thescopeofthegroupis‘todevelopnewmethod-ologicalelementsforgainingdeeperunderstandingnotonlyofcloudperformance,butalsoofcloudoperationand behavior, through diverse quantitative evaluationtools, including benchmarks, metrics, and workloadgenerators’.In2015,throughbi-weeklyonlinemeetingsfacilitated by WebEx and SPEC, we have advancedworkon4maintopicsandanumberofothersub-topics:
(1)CloudUsagePatterns(CUPs):Thegoalof thisactivity is to define a formalism for expressing cloudusagepatternsandscenarios.The joint authors,whorepresentsevenorganizations,propose[1]asimpleyetexpressivetextualandvisual formalism,whichcanbeusedbybothgeneral users and cloudexperts.A keyfeatureofthetextualformalismisitsconciseness;thisgoesincontrasttootherformalismsthatalsofocusonthe executability of the specification. By expressingover ten patterns commonly seen in academic andindustrialpractice,theauthorsshowthatCUPispracti-cal.Anextendedarticle,withnewcoreconcepts,wascreatedduring2015andiscurrentlyundersubmission.
(2)CloudMetricsSurveyandDesign:Thisongoingactivity focuses on surveying existing cloud metricsandonthedesignofkeymissingmetricsthatallowthequantitativeassessmentandcharacterizationoftypicalcloudusagescenarios.Amongthekeynewmetrics,thejointauthorsfocusonvariousformsofelasticity[2]andrisk-quantifyingmetrics[3].Acontinuationofthiswork,andanextendedsetofmetrics,havebeendevelopedby the group over 2015 and are being prepared forpublication.
(3)BUNGEE:Thisongoingactivity focuseson thedevelopmentandvalidationofaJava-basedframeworkfocusing on cloud elasticity, especially for IaaS cloudplatforms and auto-scaling environments. The toolprovidesloadandstress-testingfunctionality,andauto-matestheanalysisofthequalityoftheelasticbehaviorof the system under test through several elasticity
metrics.Currently,BUNGEEsupportsCloudStackandAmazonAWS based deployments. The group is cur-rentlyextendingBUNGEEtosupportmorecloudenvi-ronmentsandmetrics.
(4) Benchmarking Auto-Scaling Techniques: Thisongoing activity, which various members of the RGCloudGroup are just starting, is aiming to conduct aquantitative analysis and comparison of auto-scalingtechniquesinvirtualizedenvironments.
CurrentparticipantsintheRGCloudGroupincludethe Delft University of Technology (Delft), the IBMT.J.WatsonResearchCenter (USA), LundUniversity(Sweden), MITRE (USA), Oracle (USA), Salesforce.com (USA),SAP (Germany),TataTCS (India),UmeåUniversity (Sweden), and the University of Würzburg(Germany).Severalofthemembersplayleadingrolesintheirinstitutions,themandothershavewonimportantdistinctionsover thecourseof2015.Forexample,Dr.SamuelKounevcurrentlyleadstheDescartesSoftwareEngineeringGroupof theUniversityofWürzburg,andDr.AlexandruIosuphaswontheNetherlandsTeacheroftheYear2015awardandhasbeenselectedtojointheYoungAcademyoftheRoyalNetherlandsAcademyofArtsandSciences.Besidesthefrequentparticipants,wehavewelcomedvariousinvitedspeakersandguestparticipants. For 2016, we are actively seeking newparticipantsandactivities.
In2015,theRGCloudGrouphasbeenrepresentedby its members at many international academic andindustrial venues, and has helped organize variousinternational events, such asHeteroPar and the flag-shipconferenceofSPEC,theACM/SPECInternationalConference on Performance Engineering (ICPE).Internally within SPEC, our group has contributedto the review and subsequent publication of severalbenchmarking tools.Concluding, 2015was a full andsuccessfulyearfortheRGCloudGroup.Wearelook-ingforwardtoanevenmoresuccessful2016!
Acknowledgments: part of this work has been submitted by the group as a poster to the ICPE2016 conference, to inform our commu-nity about the opportunity of collaborating with the RG Cloud Group.
[1] A. Milenkoski, A. Iosup, S. Kounev, K. Sachs, P. Rygielski, J. Ding, W. Cirne, and F. Rosenberg. Cloud usage patterns: A formal-ism for description of cloud usage scenarios. CoRR, abs/1410.1159,
2014. [2] N. R. Herbst, S. Kounev, and R. H. Reussner.Elasticity in cloud
computing: What it is, and what it is not. ICAC’13, 2013, pages 23-27, 2013.
[3] V. van Beek, J. Donkervliet, T. Hegeman, S. Hugtenburg, and A. Iosup. Self-expressive management of business-critical workloads
in virtualized datacenters. IEEE Computer, 48(7):46-54, 2015. [4] N. R. Herbst, S. Kounev, A. Weber, and H. Groenda. BUNGEE:
an elasticity benchmark for self-adaptive IaaS cloud environments. In SEAMS 2015, 2015, pages 46-56, 2015.
TheSPECRG IDSBenchmarkingWorkingGroupsuccessfullyconcluded itsagenda for2015and faces2016witharenewedcommitment.In2015,theWorkingGroup further established its research agendahavingseveralworkspublished.
TheSPECRG IDSBenchmarkingWorkingGrouppublished a paper focusing on evaluation of IDSesin virtualized environments at RAID 2015 (the 18thInternational Symposium on Research in Attacks,Intrusions and Defenses) [1]. This symposium bringstogether leading researchers and practitioners fromacademia, government, and industry to discuss novelsecurity problems, solutions and technologies relatedtointrusiondetection,attacksanddefenses.ThepaperpresentshInjector,atoolforgeneratingIDSevaluationworkloads by injecting such attacks during regularoperationofavirtualizedenvironment.Itdemonstratestheapplicationofthetoolandshowsitspracticaluse-fulness by evaluating a representative IDS designedtooperate in virtualizedenvironments.Thevirtualizedenvironment of the industry-standard benchmarkSPECvirt_sc2013 isusedasa testbed,whosedriversgenerateworkloads representative ofworkloads seeninproductionenvironments.Thisworkenables for thefirst timethe injectionofattacks invirtualizedenviron-mentsforthepurposeofgeneratingrepresentativeIDSevaluationworkloads.
Inaddition,apaperproducedbytheSPECRGIDSBenchmarkingWorkingGroupwaspublishedbyACMComputing Surveys (CSUR) [2]. The paper surveysthestate-of-the-artintheareaofevaluationofintrusiondetectionsystems.CSURpublishessurveyarticlesandtutorials related to computer science and computing.According to the ISI Journal Citation Reports, CSURhasamong thehighest impact factorsofall computersciencejournals.
[1] Aleksandar Milenkoski, Bryan D. Payne, Nuno Antunes, Marco Vieira, Samuel Kounev, Alberto Avritzer, and Matthias Luft. Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection. In The 18th International Symposium on Re-search in Attacks, Intrusions, and Defenses (RAID 2015), Kyoto, Ja-
pan, Springer. November 2015, Acceptance Rate: 23%.[2] Aleksandar Milenkoski, Marco Vieira, Samuel Kounev, Alberto
Avrtizer, and Bryan D. Payne. Evaluating Computer Intrusion Detec-tion Systems: A Survey of Common Practices. ACM Computing Sur-veys, 48(1):12:1-12:41, September 2015, ACM, New York, NY, USA.
Aleksandar Milenkoski (University of Würzburg), Marco Vieira (Uni-versity of Coimbra), Nuno Antunes (University of Coimbra), Alberto
Avritzer (Siemens Corporate Research, USA)
REPoRt: biG data woRkinG GRouPJanuary 14, 2016
Bigdataisarapidlyevolvingdisciplineincomputerscience utilizing a diverse spectrum of technical plat-forms and serving a wide range of applications. Thisis because, with the combinations of large volumesof data, heterogeneous data formats, and the rapidlyimprovingperformanceofbothhardwareandbigdatasystems, it is hard to generalize architectural aspectsthatbestsuitallapplication requirements,making theinvestigationandstandardizationofsuchsystemsverydifficult.
Asbigdatasystemsareevolving,thereisaninher-ent need to evaluate and quantify their performancewith the ultimate goal of comparing these systems.Comparisons are desirable in different dimensions,suchassoftwarestack,hardware,usecase,andtuningparameters.Thatis,onemightwanttocompareapar-ticularsoftwarestackondifferenthardwaresystems,aparticularhardwaresettingondifferentsoftwarestacks,oronesoftwarestackonaparticularhardwarewithdif-ferenttunings.
With the rapid increase in big data solutions, bothacademia and industry alike are developing newbenchmarks at a rapid pace. Driven by the “velocityof change”manyperformancebenchmark developers“cut corners” by customizing their benchmarks toocloselytothearchitecturalcharacteristicofthesystemtheywanttobenchmark,insteadofabstractingitscoreperformance attributes. These benchmarks become“islandsolutions”thatonlyfitthesystemstheytargetedin the firstplace.Thisapproachworkswell if thegoalistocomparetheperformanceofaparticularsoftwarestack on a particular hardware setting. However, thisdoesnotworkwelltocomparetheperformanceofdif-ferentsoftwarestacksonthesamehardwareplatformsorviceversa.
Employing traditional ways of specifying standard-izedbenchmarkswithpredefinedworkloadsisnottrivialfor bigdata systems,becauseof the combinationsoflargevolumesofdata,heterogeneousdataformats,andvelocityofchangesintheprocessingtechnologyusedinbigdatasolutions.Asaconsequence,manycompa-niesandresearchinstitutionsaredevelopingtheirown“islandsolutions”thatonlyfitsystemstheytarget.Itisachallengeforbothindustryandacademiatokeeptrackofthelargenumberofemergingbenchmarks.
Therefore, in 2015, the Big Data Working Groupfocused on surveying current big data benchmark-ing efforts and published a paper on this topic atthe Transaction Processing Performance Council’sTechnicalConference (TPCTC) [1]. Furthermore, thegroup co-organized the 6th Workshop on Big Data
The DevOps Performance Working Group [1] isconcerned with combining application performancemanagement (APM) and software performance engi-neering (SPE) activities for business-critical softwaresystems.TheneedforabetterSPE/APMintegrationisdrivenbyanincreasedinterrelationofdevelopmentandoperation teams in corporate environments, includingnewsoftwaredevelopmentandorganizationparadigmsknown as DevOps. The DevOps movement aims onmerging the classical development and operationssilos to end-to-end responsible teams.This paradigmchanges requires increased collaboration and a highdegreeofautomation in thedevelopmentanddeploy-mentprocesses.Thecommunityhasfailedtosystem-atically integrate performance management practicesinto theDevOpsworld,even thoughmodernsoftware
Benchmarking [2] inToronto and held its first face toface meeting there.As an outreach activity, a posterand lightning talkwaspresentedat the8thExtremelyLargeDatabaseConference[3].
The group holds weekly meetings, with internalcalls and open, public calls alternating. For the opencalls, presentations on big data benchmarking, bigdata systems, performance tuning, and related re-searchare invited.These calls arewell attendedandreceived. Topics in 2015 were BigBench, IoTAbench,big data in Finance, SparkBench, Oracle Big DataSQL, Graphalytics, HiBench, Benchmarking Facets,and benchmarking on modern clusters. All presenta-tionscanbefoundontheworkinggroup’swebsite[4].Intheinternalcalls,thegroupfurtherstheresearchonbig data benchmarking. In 2016, besides extendingtheongoingresearchactivities,thegroupplanstoco-organizethe8thWorkshoponBigDataBenchmarking,whichwill be held in Barcelona, Spain hosted by theBarcelonaSupercomputingCenter.
[1] Todor Ivanov, Tilmann Rabl, Meikel Poess, Anna Queralt, John Poelman, and Nicolas Poggi. Big Data Benchmark Compen-dium. Seventh TPC Technology Conference on Performance Evalua-
tion & Benchmarking (TPCTC 2015).[2] Sixth Workshop on Big Data Benchmarking, Toronto, Canada
(WBDB2015.ca). http://clds.sdsc.edu/wbdb2015.ca[3] John Poelman, Meikel Poess, Tilmann Rabl, Anna Queralt, To-
dor Ivanov, and Chaitan Baru. SPEC RG Big Data Working Group: An Introduction. 8th Extremely Large Databases Conference (XLDB ’15).
developmentparadigmsprovidegreatopportunities todo that.Only holistic performancemanagement, inte-grating Dev andOps throughout the whole life-cycle,leadstohigh-qualitysoftware.
The DevOps Performance Working Group fostersand facilitates research in combining APM and SPEactivities, e.g., by experience sharing, agreement ondefinitions, specificationofmetrics,anddisseminationofnovelmethods,techniques,andtoolsforquantitativeevaluation.Thegroup focuseson the challengesandopportunities of performance evaluations in DevOpsenvironments. We aim to identify cross-communitycollaboration, and to set the path for long-lasting col-laborationstowardsperformance-awareDevOps.
As the major joint activity in 2015, the DevOpsWorking Group created and published the technicalreport SPEC-RG-2015-01 “Performance-orientedDevOps:AResearchAgenda” [2].The report surveysperformanceevaluationmethods,techniques,andbest-practicesforalllifecyclephasesofenterprisesoftwareapplicationsand identifies researchchallenges for theupcomingDevOpsera.Thisreportactsasaresearchagendaforthenextyearsinthisnarrowdomainrelatedto different communities including performance andsoftwareengineering,cloudcomputing,andbigdata.
In February 2015, the groupmet for its first face-to-face meeting, co-located with the RELATE WinterSchool at the University of Würzburg, Germany. Theagenda of the one-day meeting comprised detaileddiscussions on the technical report, two invited talks,anddiscussionson futureworkinggroupactivities.13groupmembersparticipatedinthemeeting.
Together with two EU projects, the working groupinitiated the International Workshop on Performance-Aware DevOps (QUDOS). QUDOS 2015 [3],technically cosponsored by the SPEC RG, was col-located with the renowned 10th Joint Meeting of theEuropean Software Engineering Conference and theACM SIGSOFT Symposium on the Foundations ofSoftware Engineering (ESEC/FSE) in Bergamo, ItalyinSeptember2015.Theinitialworkshopcomprised10talks,8scientificpapers,andaround25registeredpar-ticipants.DuetothegreatinterestinthisnewworkshopasecondeditionofQUDOSisplannedforfall2016.
Several sub-projects among subsets of themembers have been conducted. A comparison andintegration of resource demand estimation (LibReDE)and measurement (PMWT) techniques resulted ina publication at the 12th European Workshop onPerformance Engineering (EPEW). Papers and toolson theWESSBASapproach forspecifying,extracting,and generating session-based workloads have beenpublished.AcollaborativeresearchprojectdiagnoseITon“Expert-GuidedAutomaticDiagnosisofPerformance
A current major joint activity is the developmentof a reference architecture and a process model forperformance-awareDevOps.The technical reportanda surveyon the industrialDevOpsperformanceprac-tices, concurrently conducted by the working group,will serve to identify the core design rationale. ThesurveyaimsondetectingcurrentDevOpspractices incontext of performance evaluation to align the refer-encearchitectureandprocesstoindustrypractice.Theoutcomeofthisprojectwillhelpcompaniestotransformfrom classical development and operations silos to aperformance-awareDevOpsprocessmodel.
In September 2016, the group will co-organize aGI-Dagstuhl seminar entitled “Software PerformanceEngineering in theDevOpsWorld” [4].Themaingoalof theGI-DagstuhlSeminar is tobringtogetheryoungresearchers intheareasofsoftwareengineering,per-formanceengineering, cloud computing, and big datatopresenttheircurrentresearchprojects,toexchangeexperience and expertise, to discuss research chal-lenges,andtodevelopideasforfuturecollaborations
The group meets on a biweekly basis to discusscurrent activities and new project ideas, includinginvitedpresentations.In2015,22meetingswereheld,including talks about performance problem detection,performance regression testing, and joint projects ofthe group. For more information about the DevOpsPerformance Working Group, (including mission, ac-tivities,meetings,presentations,projects)visitourwebpage [1]. If youare interested in following thediscus-sionsorcontributingactively,pleaseget in touchwiththeworkinggroupchairs.
[1] SPEC Research Group, DevOps Performance Working Group. Website
[2] A. Brunnert, A. van Hoorn, F. Willnecker, A. Danciu, W. Has-selbring, C. Heger, N. Herbst, P. Jamshidi, R. Jung, J. von Kistowski, A. Koziolek, J. Kroß, S. Spinner, C. Vögele, J. Walter, A. Wert: Per-formance-oriented DevOps: A Research Agenda. Technical Report SPEC-RG-2015-01, SPEC Research Group–DevOps Performance Working Group, Standard Performance Evaluation Corporation (SPEC), 7001 Heritage Village Plaza Suite 225, Gainesville, VA20155,
USA, August 2015.
https://research.spec.org/index.php?id=1345
[3] 1st International Workshop on Quality-aware DevOps (QUDOS 2015).
http://qudos2015.fortiss.org/
[4] GI-Dagstuhl Seminar 16394, Software Performance Engineering in the DevOps World. Website.
http://www.dagstuhl.de/16394
André van Hoorn (University of Stuttgart), Felix Willnecker (fortiss GmbH)
thREE diStinCtionS in 2015 foR alExandRu ioSuP, of thE Cloud
woRkinG GRouP January 18, 2016.
Alexandru Iosup, Associate Professor at the DelftUniversity of Technology, has received three impor-tant distinctions in 2015.Hewas namedNetherlandsTeacheroftheYear[1]onJanuary17,2016,hasbeenselectedasanewmemberof theYoungAcademyoftheRoyalNetherlandsAcademyofArtsandSciences(KNAW)[2]onNovember25,2015,andannouncedasthewinneroftheNetherlandsAwardforICTResearch2016[3]onDecember17,2015.
The title of Netherlands Teacher of the Year isawardedbytheInterstedelijkStudentenOverleg(Inter-CityStudentConsultation,orISO)toateacherwhoisknownforhisorherinvolvement,passion,andinnova-tiveteachingmethods.Thisisaccordingtothejurythecase for Iosup and his slogan ‘every student counts’.Hehasachievedhighpassingrates,highachievement,andhighinvolvementincoursesforwhichperformanceevaluation and benchmarking are important, such asthe1st-yearBachelors’courseComputerOrganization,andtheM.Sc.-levelcourseCloudComputing.
Dr.Iosupusesgamificationinhisteachinginorderto help students learn in ways that suit their person-alities. Students can follow their own routes to learn-ing, thus earning credits. Theymay choose amongstexploratory, social, or competitive elements. Somestudentsarefondoffreeassignments(e.g.,designingandprogramminggames),whileothersprefer toworkin small groups and yet others seek the challenge ofbreakingthroughtomoredifficultmaterial,likereachingahigherlevelinagame.Inordertokeepthestudentsfurtherinvolved(or‘onboard’),theymaycompletetheentry quiz once, and they may participate in discus-sionsonthenewestandmostimportantresearchandindustrialchallengesatthebeginningofeverylecture.This motivates students to go further and distinguishthemselves. By creating challenges, excitement, aquestforaparticulargoalandrewards,Iosupkeepshisstudentsexceptionally involved.Manystudentsaresomotivated that theywish to takeextraclassesanddoextrawork for thesolepurposeof learningsomethingnew,withoutanyotherreward;forthesestudentsIosuporganizesextra lecturesandhalf-dayworkshops.Thejury and his students were quite enthusiastic aboutIosup: ‘Heknows thateachstudent isunique,andheisable tousegamification tobringout thebest inallstudents.’
In November 2015, Alexandru Iosup has beenselectedwithnineotherscientistsfromotherdomainsto join the YoungAcademy, which is an independent
science platformof leading young researchers.Everyyear, the Young Academy selects ten talented newresearchers toadd to its ranks.Membersareelectedforfiveyears.
Dr. Iosup has been recognized for being an inter-nationalpioneerinthefieldofdistributedsystems.Hisresearchwas considered essential to emerging fieldssuch as cloud computing, to popular internet applica-tionssuchase-Scienceandonlinegaming,andtothefuture of large-scale and/or gamification-based teach-ing.Iosupisanadvocateofopenaccessanddevotesagreatdealofhis time tosupervising talentedyoungresearchers.Heisinvolvedinmasterclassesontech-nology, in particular in his fieldof research, forDutchengineersandstudents.
InDecember 2015,Dr.Alexandru Iosup has beenannounced as the winner of the Netherlands Awardfor ICT Research 2016. The prestigious NetherlandsAward for ICTResearch ispresentedannuallyby theRoyal Dutch Organization of Sciences (KoninklijkeHollandsche Maatschappij der Wetenschappen,KHMW, founded 1752) to a scientific researcher, notolderthan40yearsold,whohasconductedinnovativeresearchorisresponsibleforascientificbreakthroughinthefieldofICT.TheprizeisaninitiativeoftheRoyalNetherlands Academy of Arts and Sciences via ICTResearch Platform Nederland (IPN), in collabora-tion with the Netherlands Organisation for ScientificResearch (NWO). On 23 March 2016, Dr. Iosup willreceivehisawardataspecialceremonyduringtheICT.OPEN2016event.
Dr.Iosupplanstousethe€50,000prizemoneytostrengthenhis researchgroup inorder to launchnewresearch into graph-processing systems, startingwithbenchmarking and as a collaboration between theSPEC RG Cloud Group and the Big Data WorkingGroup. The jury praised Dr. Iosup for his academicachievements,hisactivecollaborationwithcompanies,andhiseffortstobringICTresearchtotheattentionofawideaudience.
The SPEC Research Group wishes to commendAlexandruforthesedistinctions.
yufEi REn fRoM Stony bRook univERSity, nEw yoRk winS thE 2015
SPEC diStinGuiShEd diSSERtation awaRd
January 12, 2016
The SPEC Distinguished Dissertation Award isan annual award that aims to recognize outstandingdoctoral dissertations within the scope of the SPECResearchGroupintermsofscientificoriginality,scien-tific significance,practical relevance, impact,andpre-sentation.This year, theawardcommitteeconsidered15excellentdissertationsubmissionsfromuniversitiesaround the world: Pennsylvania State University,Virginia Tech, Karlsruhe Institute of Technology,Australian National University, University CollegeDublin,UniversityofLisbon,ColoradoStateUniversity,UniversityofMalaya.
The winning dissertation from Yufei Ren of NewYork’sStonyBrookUniversity,nominatedbyProfessorDantongYu,isaboutscalableend-to-enddataI/Ooverenterprise and datacenter networks. The committeeparticularly appreciates the insights presented in thisdissertationforimprovingdata-intensiveapplicationsindatacentersfromaholisticperspective.Thepracticallyorientedworkacrossmultiple layersof thenetworkedstoragestackcoupledwiththesolidexperimentsdrewpraise from thecommitteemembers.Theaward is tobe handed over at the ICPE 2016Conference in theNetherlandsinMarch2016.
In addition to selecting the winner, the committeedecided to recognize one dissertation with honorablemention. This is the dissertation of Cong Xu, nomi-natedbyProfessorYuanXieofthePennsylvaniaStateUniversity, forcontributions tomodeling,circuitdesignandmicroarchitectureofemergingresistivememory.
The award selection committee for 2015 chairedbyProf.LizyKurianJohnoftheUniversityofTexasatAustin,consistedofthefollowingmembers:
• MauricioBreternitz,AMD,USA,• Yao-MinChen,Oracle,USA• RameshRadhakrishnan,Dell,USA• SeetharamiSeelam,IBM,USA• PetrTuma,CharlesUniversity,CzechRepublic• Zhibin Yu, China Shenzhen Institute of
liMbo: a tool foR ModEllinG, ExtRaCtinG and GEnERatinG tiME-vaRyinG load intEnSity
PRofilESJanuary 14, 2016
LIMBO isanEclipse-based tool for thecreationofloadprofileswithvariableintensityovertimebothfromscratchand fromexistingdata [1].Primarily, LIMBO’sintendeduseisthedescriptionofloadarrivalbehaviorsin open workloads. Specifically, LIMBO can be em-ployedforthecreationofcustomrequestoruserarrivaltime-stamps or for the re-parameterization of existingtraces.
LIMBObasesontheDescartesLoadIntensityModel(DLIM) for the formalizeddescriptionof its load inten-sity profiles. The DLIM formalism can be understoodas a structure for piece-wise defined and combinedmathematical functions that capture specific elementsof a time-series like seasonal patterns, trends, burstsandnoise.
wElCoME to iCPE 2016 in dElftJanuary 12, 2016
The 7th ACM/SPEC International Conference onPerformance Engineering (ICPE) 2016 which takesplaceinMarch12-16,2016inDelftinTheNetherlandswith all its different tracks attracted a number ofhigh–quality submissions selected in intensive reviewprocessesbydifferenttrackcommittees.
In the research trackwe had 57 submissionswith19beingaccepted forpresentationat theconference.In the Visions and Work-In-Progress track, 6 out of15 contributions were selected. In the industry trackadditional 7 articles out of 17 made it into the finalconference.
Inadditiontothosescientificarticlestheprogrammealso includes a poster and demo track, tutorials, adoctoral symposium, and several interesting work-shops, forexample, the2ndWorkshoponChallengesin Performance Methods for Software Development(WOSP-C2016);5thInternationalWorkshoponLarge-Scale Testing (LT 2016); 2nd International WorkshoponPerformanceAnalysisofBigDataSystems (PABS2016).
WearegladtoseeyouinDelftinMarch2016!Steffen Becker (TU Chemnitz), Alexandru Iosup (TU Delft)
iCPE 2017 in l’aquila– PREliMinaRy announCEMEnt
January 12, 2016
The ACM/SPEC International Conference onPerformanceEngineering (ICPE)providesa forumfortheintegrationoftheoryandpracticeinthefieldofper-formanceengineering.ICPEisanannualjointmeetingthathasgrownoutoftheACMWorkshoponSoftwarePerformance (WOSP) and the SPEC InternationalPerformanceEngineeringWorkshop(SIPEW).Itbringstogether researchers and industry practitioners toshare ideas, discuss challenges, and present resultsof bothwork-in-progressand state-of-the-art researchonperformanceengineeringofsoftwareandsystems.ICPE 2017will be held in L’Aquila (Italy), around thebeginningofApril.
L’Aquila is the capital city of theAbruzzi region inItaly, and it is located approximately 100 kilometerseastfromRome,withwhichit isconnectedbyahigh-way through themountains. Laid out withinmedievalwallsuponahillside in themiddleofanarrowvalley,tallsnow-cappedmountainsoftheGranSassomassifflank the town. A maze of narrow streets, lined with
Baroque and Renaissance buildings and churches,openontoelegantpiazzas.
HomeoftheUniversityofL’Aquila,itisalivelycol-lege town.TheDepartmentofComputerScienceandEngineering,andMathematics(DISIM[1])isveryactivein the international researchcontext, and inparticularthe Software Engineering andArchitecture Group in-cludesmembersthathavecontributedtothebirthandthegrowthofWOSPfirstandthenICPEconferences.
ThehostinstitutionwillbetheUniversityofL’Aquila,in collaboration with Gran Sasso Science Institute(GSSI,[2]).GSSIwasrecentlyborninL’AquilathankstoOSCE funding, and today it represents a key per-spectiveforthecityandtheterritory.
The contact person for ICPE 2017 is VittorioCortellessa [3], who will be General Co-Chair alongwithWalterBinder[4]fromUniversityofLugano.AnneKoziolek[5]fromKarlsruheInstituteofTechnology,andEvgenia Smirni [6] fromCollege ofWilliam andMarywillbeProgramCo-Chairs.
GRaPhalytiCS: bEnChMaRkinG PlatfoRMS foR laRGE-SCalE GRaPh
PRoCESSinG January 18, 2016
Graphsmodel social networks, humanknowledge,and other vital information for business, governance,and academic practice. Although both industry andacademia are developing and tuning many graph-processingalgorithmsandplatforms, theperformanceofgraph-processingplatformshasneverbeenexploredor compared in-depth. Moreover, graph processingexposes new bottlenecks in traditional HPC, distrib-uted, multi-/many-core, and multi-threaded systems(forexample,seethemajordifferencesinTop500andGraph500rankings).
Complementing other tools for benchmarkinggraph-processing systems, the Delft University ofTechnology,memberoftheRGCloudGroup,isleadinga jointbenchmarkingactivitythathasalreadyresultedin the creation of the Graphalytics benchmark andopen-source tools [1,2].Graphalytics focuses on newmethodsandtoolsforperformanceevaluationandopti-mizationforgraphprocessingplatforms,andinparticu-laronenabling itsusers toexploreand toexplain theperformance dependency Platform-Algorithm-Dataset.Ithasalreadyresultedinnewknowledgeandmethod-ology[3].Oneofthekeyresultsin2015hasbeenthedesign and development of theGranula performancecomponent for fine-grained performance data collec-tionandanalysis,whichoffers forbigdataprocessesunique features for the analysis of bottlenecks andresourceusage.
Inanextstep,thesegeneratedtime-stampscanbeused for both open workload-based benchmarking orsimulations.Forexample,TimestampTimerplug-in forJMeter already allows the precisely timed sending ofrequeststoasystem-under-test.
LIMBOoffersa rangeof features foreasy load in-tensitymodelingandanalysis,including,butnotlimitedto,avisualdecompositionofloadintensitytime-seriesintoseasonalandtrendparts,asimplifiedloadintensitymodelaspartofamodelcreationwizard,andanauto-matedmodelinstanceextractor.
LIMBO’sdevelopmenthasbeenstartedinlate2013at the Karlsruhe Institute of Technology and is nowmaintainedattheUniversityofWürzburg.
SinceLIMBOwasacceptedfortheSPECRG’stoolrepository in 2014, it has been adopted in researchprojectslikeEUFP7projectsCactosandCloudScale.In addition, LIMBO has become a foundation for aCloud Elasticity Benchmarking Framework namedBUNGEE[2].
[1] Jóakim von Kistowski, Nikolas Roman Herbst, Daniel Zoller, Samuel Kounev, and Andreas Hotho. Modeling and Extracting Load Intensity Profiles. In Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems
(SEAMS 2015), Firenze, Italy, May 18-19, 2015.[2] Nikolas Roman Herbst, Samuel Kounev, Andreas Weber,
and Henning Groenda. BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments. In Proceedings of the 10th Inter-national Symposium on Software Engineering for Adaptive and Self-
Managing Systems (SEAMS 2015), Firenze, Italy, May 18-19, 2015
http://descartes.tools/limbo
Jóakim v. Kistowski and Nikolas Herbst (University of Würzburg)
LIMBO shows the decomposition of a load intensity profile.
SPEC SyMPoSiuM’16–EnERGy EffiCiEnCy: fREMont, Ca JanuaRy
11th, 2016 January 18, 2016
The inaugural SPEC’s Symposium’16–EnergyEfficiency was held in Fremont, California in January2016. We were delighted to having brought togetherrepresentativesfromtheU.S.EnvironmentalProtectionAgency, The Green Grid, ISO, ITIC, academic re-searchers,andSPECbenchmarkdevelopers inordertoshareideas,discusschallenges,andpresentresultsof bothwork‐in‐progressand state‐of‐the‐art researchonserverefficiency.Itwaswithgreathonortowelcomethe keynote speaker JeremyArnold (IBM) and listen-ing to his speech “What is aGoodBenchmark?”.Hediscussed what makes a good benchmark, and howbenchmarksshouldbeselectedandusedbasedontheperformancemeasurementgoalsoftheuser.
Sanjay Sharma (Intel) described the history and
2015isavailabletothepublicasaGitHubopen-sourcesoftware [2]. It has been tested by the Graphalyticsdevelopment team,andbyamulti-institutiondevelop-ment team affiliated with LDBC [3], including Neo4jandSPECmembersOracle,Intel,andIBM(ongoing).Ithasalsobeenused invariousacademicand indus-try projects, such asMSc thesis projects at theDelftUniversityofTechnology.In2016,itaimstobecomeaSPEC-endorsedtool.
In 2015, theGraphalytics team has presented themethods, tools, and core results also as tutorials, atICPE 2015 in January (preliminary features), and atSC|15 in November (mature tools and optimizationprocesses). The tutorials focused on real-world ex-periences with commonly used systems, from graphdatabases such as Virtuoso and Neo4j, to parallelGPU-based systems such as Totem and Medusa, todistributedgraph-processingplatformssuchasGiraphand GraphX. The tutorials have included significanthands-on and Q&A components, but also in-depthanalysisofsystemperformanceandbottlenecks.
We invite the SPEC Newsletter audience to tryGraphalyticsandcontactourteam.
[3] Mihai Capota, Tim Hegeman, Alexandru Iosup, Arnau Prat-Pérez, Orri Erling, Peter A. Boncz: Graphalytics: A Big Data Bench-mark for Graph-Processing Platforms. GRADES@SIGMOD/PODS
2015: 7:1-7:6
Alexandru Iosup (TU Delft)
evolution of SPECpower_ssj2008, the industry’s firstserverenergyefficiencybenchmark.
Hansfried Block (Fujitsu) summarized the featuresof theServer EfficiencyRatingTool (SERT), togetherwiththehistoryofitsevolutionandthecurrentusewiththeEPAENERGYSTARprogramtoday.
John Clinger (ICF International/EPA) provided in-sightintothepossiblefutureevolutionoftheU.S.EPA’sENERGY STAR for Computer Servers program. HealsoprovidedideasforhowtheuseoftheSERTmightevolve,includingtheadditionofmetrics.
Jóakim von Kistoswki described in his session,Research and SPEC Chauffeur, how he and his re-search colleagues at the University of Würzburg areusing the Chauffeur WDK (derived from the SERTprogram) to rapidly develop new tools for measuringenergyefficiency.
Xiaomeng Lu shared her insight, connections be-tween industry and policymakers, into the challengeswesternorganizations facewhen interactingwithAsiaPacific governments and policy-makers on energy ef-ficiencyandbenchmarking.
Henry Wong (Intel, ISO & TGG) described in hispresentation ISO and Global Government Programs,how ISOcanhelp tominimize thepoliticalaspectsof
Thepanel“PanelDiscussion:TheFutureofEnergyEfficiency of Server“ with John Clinger, HenryWong,Xiaomeng Lu, and Klaus-Dieter Lange discussedthe premise: “The dawn of performance engineeringis over”. The discussion covered the tools and pro-grams togetherwith their ideas for future internationalcollaboration.
Klaus and Xiaomeng delivered a SERTDemonstrationonhow theSERT is used tomeasuretheenergyefficiencyofaserver,usingaserver,alap-top,apoweranalyzer,andatemperaturesensor.
ThesecondpresentationoftheSPECSymposium’16takes place in Delft in the Netherlands. This eventis co-located with the International Conference onPerformanceEngineering(ICPE)2016,attheTUDelftintheUniversityAula.Thissymposiumincludescontentderived from that presented in Fremont in January,alongwithEU-centricsessionsandapaneldiscussion.ItishopedthatindustryandacademicrepresentativeswhowereunabletoattendtheUSeventwillbepresent.Representatives of severalAsianGovernment bodiesand standards organizations have been invited. Theyarepresentingcontentspecific to theirnationalobjec-tivesfor internationalcooperationonenergyefficiencystandardsandtesting.
Mike Tricker (SPECpower Committee Secretary)
inspectit thE oPEn SouRCE aPM Solution
January 5, 2016
Today’s businesses have understood what impactthe performance and scalability of their softwareapplications has on revenue and total cost of owner-ship. These businesses use application performancemanagementsolutions tomaintain thesatisfying levelof service quality, to detect problems early and to di-agnose problems quickly to insure a stable revenuestreamand reasonable totalcostofownership.Somebusinessesremainwithanunsatisfiedneedtomanagethe performance of their applications. Becausemanybusinesses do not want to take the risk of a vendorlock-in that may dissolve or discontinue the productor cannot copewith theacquisitionandannualmain-tenance costs. The public sector on the other handmaynotbeabletomakeabenefitofusingtheproductduetobudgetrestrictionsorifthesourcecodecannotbeexamined forsafetyorsecurity reasons.Researchgroups are also often not able to make use of the
commercial solutions because of the high acquisitioncosts. In lightof theseobservations, thematured toolinspectITwasmadepublic toprovideanopen-sourceapplicationperformancemanagementsolution.
Following this mission, inspectIT is the first open-source tool of its kind. Software performance expertscanmonitor execution traces from applications underanalysis and drill down into traces to isolate the rootcauses of performance problems. The rich graphicaluserinterfacepaysattentiontodelivergreatuserexpe-riencedesignedtosupportthetasksofaperformanceexpert’sdailybusiness indiagnosingproblems. In thelastdecade, inspectIThasproven its reason forexis-tenceinmanycustomerprojectsinindustry.Therecentcollaborationwithresearchgroupsshowstheneedforan independent performance analysis tool. Providinga standardized data interface enables researchers tocollectdata,exportthisdata,sharethedatawithothergroupsandimportthedataofothersforanalysis.Thisobjective is already supported by the collaborationwith researchers of different universities to standard-ize a non-proprietary data format for traces (currentlyreferredtoastheCommonTraceAPI—CTA).Thedataformat is used to export traces from inspectIT and toimporttracesfromothertoolsofthecommunity.
The intended users of inspectIT are performanceexpertsanddevelopers.Bydesign,testingandtrouble-shooting scenarios (so called performance firefights)are the primary task which is intentionally supportedby inspectIT. For this particular purpose, inspectITprovidesasophisticatedgraphicaluserinterfacethatistailoredbasedontheexperienceofadecadetoalloweasy access to the collected performance data (e.g.,traces,timings,CPUandmemoryutilization,etc.)andquick isolationof aproblem’s root cause. inspectIT isdivided into three main components: (1) the agents,(2) the Central Measurement Repository, and (3) thegraphical user interface. The agents are responsibleforcollecting thedata. inspectITcurrentlyprovidesanagent for Java-based applications.The Java agent isstarted with the system under analysis (SUA) and isresponsiblefor instrumentingthesystemtocollectthedesireddata.Theagentiscurrentlycapableofcollect-ing HTTP requests, execution traces, timings (e.g.,response times,CPU times), thrownexceptions,SQLqueries and system measures (e.g., CPU utilization,Memoryutilization).Oneormoreagents,dependingonthe concrete environment, send the collected data totheCentralMeasurementRepositorywhichisresponsi-bleformanagingthereceiveddata.Thegraphicaluserinterfaceenablestheperformanceexperttoaccessthemanaged data and to execute performance analysessuchasdiggingintotracestoisolatetherootcauseofaproblem,browsethrownexceptions,andviewresourceutilization.Theuser interfacealsoshows theeffective
14
nEw aPM RESEaRCh PRoJECt: diagnoseit
January 5, 2016
In the last decade, application performancemanagement (APM) solutions have been developedsupporting enterprises with monitoring capabilitiesandearlydetectionofperformanceproblems.LeadingapplicationAPMsolutionsmostlysupportonlyalertingand visualization of performance-relevant measures.Theconfigurationof the software instrumentation, thediagnosis of performance problems, and the isolationoftheconcreterootcause(s)oftenremainerror-proneand frustratingmanual tasks.To thisday, these tasksareperformedbycostlyandrareperformanceexperts.Inorder to improve thissituation,NovaTecConsultingGmbHandtheUniversityofStuttgart(ReliableSoftwareSystems Group) launched the collaborative researchproject diagnoseIT on “Expert-guided AutomaticDiagnosis of Performance Problems in EnterpriseApplications”.Hereby,thecoreideaistoformalizeAPMexpert knowledge to automatically execute recurringAPM tasks suchas the configuration of ameaningfulsoftware instrumentation and the diagnosis of perfor-manceproblemstoisolatetheirrootcause.Bydelegat-ing thedescribed tasks todiagnoseIT, expertsdonothavetodealwithsimilarproblemsoverandoveragain.Instead,theexpertcanfocusonmorechallenging(andinteresting)tasks.
instrumentation of SUA and allows to configure theinstrumentationbyassociatingnamespaces,classesorinterfacesoftheSUAwiththeparticulartypeofrequireddata(e.g.,timings,executiontrace).
inspectIT is released under the GNUAGPL v3 li-cense.ThequalityofinspectITisamajorobjectivethatdrives the integrationofanychangecommitted to theproject’s codebase.Thesourcecode ismanagedonGithub[1]togetherwithreadytouseDockercontainers[2].While the current release is inspectIT 1.6, furtherreleases includingbug fixesandnew featuresareal-readyplannedintheroadmapfortheupcomingmajorreleases.
For more information about the inspectIT projectandhowtocontributetoitsdevelopment,visitourwebpageandgetintouchwithus.
[1] https://github.com/inspectIT
[2] https://github.com/inspectit-docker
Patrice Bouillet, Christoph Heger, Ivan Senic, Stefan Siegl, Alexander Wert (NovaTec Consulting GmbH)
The automated diagnosis that analyzes traces(individual or as a stream provided by monitoringcapabilities ofAPM solutions) is designed as follows:possiblesymptomsofperformanceproblemsarepro-vided as formalized expert knowledge—an extensiblesetofrules.Whenasymptomisdetectedinatrace,therootcausediagnosisisstartedwithouttheneedforhu-man interaction.Rules thatperform localizationof theproblemareappliedfirst,followedbytechnologyand/ordomain-specificrules,whichareusedtosemantifytheisolatedrootcause.Iftheavailabledataisnotsufficientto perform an exhaustive diagnosis, diagnoseIT willrequestmoredata fromtheunderlyingAPMtool,withrespecttooverallsystemperformance.
diagnoseITprovidestheanalysisresultsto itsuserintheformofcomprehensivereportsthatincludequali-tative (e.g., problem’s location, type, andanti-pattern)andquantitative information (impactof theproblem innumbers).Amajorgoalofthereportistodescribetheproblem to non-experts making it possible to providedifferentreporttypestoindividualroles.
The available APM solutions have a proprietarydataformattopersistandprocessmonitoringdatathatmakes it difficult and inconvenient, if not impossible,for third-party access. In order to enable communitycollaboration,dataexchange,andtool-independentap-proaches in research,ourgoal is tomakediagnoseITindependent of concreteAPM solutions by accessingthemonitoringdatathroughtheuseofanindependentAPI(currentlyreferredtoastheCommonTraceAPI—CTA) and adapters for particular tool-dependent dataformats.ThespecificationoftheCTAisanongoingtaskmotivatedbyservingasfoundationforcommunitycol-laborationonAPMinteroperability.
diagnoseIT started in March 2015 with a durationof two years. It is sponsored by theGerman FederalMinistryofEducationandResearch(BMBF),andsup-portedbytheSPECRGDevOpsPerformanceWorkingGroup. For more information about the diagnoseITproject,aswellastheCTAandhowtocontributetoitsdevelopment,visitourwebpageandgetintouchwithus.
http://diagnoseit.github.io/
Christoph Heger, Alexander Wert, Stefan Siegl (NovaTec Consult-ing GmbH), André van Hoorn, Dušan Okanović (Reliable Software
SPEC awaRdS in yEaR 2015 and 2016 January 18, 2016
During SPECRG’s 2015Yearly Meeting inAustin(Texas,USA),RGmembersJohnMurphy,Klaus-DieterLange,WalterBinderandAndrévanHoornwere rec-ognized by the SPEC President for their outstandingservicetoSPECin2014.
John Murphy (University College Dublin) wasawarded with the SPEC 2014 PresidentialAward forservingasGeneralChair of the ICPE2014 recogniz-ing his contributions to the success of the confer-ence. Klaus Dieter Lange (HPE) and Walter Binder(University of Lugano, Switzerland) were recognizedwith SPECtacular Award for serving as co-generalChair for ICPE 2014.André vanHoorn (University ofStuttgart) receivedSPECAward forproject leadershipintheDevOpsPerformanceWorkingGroup.
The year 2015 was also rich in events andactivities that required outstanding work and com-mitment of Research Group members. During the
SPECtacular awards await the awarding ceremony in Freemont, 2016. Photo: David Reiner (SPEC)
annualmeetinginFreemont,California,USA(January2016), two Research Group members were awardedwith SPECtacular Awards: Aleksandar Milenkoski(University ofWürzburg) for his excellence and com-mitmentinIDSBenchmarkingWorkingGroup,andKaiSachs(SAP)forservingasProgramCommitteeChairforICPE2015heldinAustin,Texas.
