Nanjing University
International Partnership in Information Technology He (Jason) Zhang
4 June 2018 @ NTNU
Nanjing · China
Beijing
Shanghai
Hong Kong
❖ A cultural city with 2,600 years history
❖ the capital of China for ten dynasties, including the Republic of China
❖ A city of excellent specialists for scientific education
❖ 53 general higher universities with over 820,000 college students
❖ 540 various scientific research organisations with 530,000 research staffs
❖ No.1 among all Chinese cities in terms of the portion of population receiving higher education
❖
Nanjing University
❖ Data Technologies❖ Data Mining
❖ mining software process from repository
❖ mining software metrics for process modeling
❖ Natural Language Processing
❖ NLP for grey literature review
❖ NLP in support of qualitative analysis & synthesis
❖ The first Chinese modern university with the combination of education & research established in 1902
❖ 29 schools, institutes, or departments, 3 of which are the Department of Computer Science & Technology, Software Institute, and Artificial Intelligence Institute
❖ The top ranked State Key Research Laboratory of Novel Software Technology in all 32 state key research laboratories in information sciences in past 15 years
Curricula @ Software Institute
Nanjing University · DevOps Research Centre
❖ Software Institute
❖ Faculty Members
❖ Prof. He (Jason) Zhang
❖ Assoc. Prof. Dong Shao
❖ Dr. Guoping Rong
❖ Dr. Zheng Li
❖ Students❖ 8 Ph.D Students
❖ 20+ Master & Thesis Students
Traditional Research Interests
❖ Software Engineering Process
❖ Empirical Software Engineering
❖ Quality Assurance
❖ Software Architecture
❖ Service-Oriented Computing
Why DevOps?❖ “Through 2016, ~80% of outages impacting mission-critical services will be
caused by people and process issues, and more than 50% of those outages will be caused by change / configuration / release integration and hand-off issues.” – Gartner Report
❖ “The Knight Capital Group was an American global financial services firm […].[…] Knight was the largest trader in U.S. equities, with a market share of 17.3% on NYSE and 16.9% on NASDAQ. The company agreed to be acquired by Getco LLC in Dec 2012 after a trading error lost $460 million” – Knight Capital from Wikipedia
❖ This took 45 minutes and was an upgrade error.
DevOps is about…❖ “DevOps is a set of practices intended to reduce the time between committing a
change to a system and the change being placed into normal production, while ensuring high quality.”
❖ DevOps is about…
❖ Bringing “agile” methods to operation
❖ Encouraging collaboration between development and operations staff, get them talking
❖ Shared goals and teams of Devs and Ops
❖
Requirements
•Treat Operations personnel as first class stakeholders
•Get their input when developing requirements
Development
•Small teams•Limited coordination
•Unit tests
Build
•Build tools•Supports continuous integration
Testing
•Automated testing•Integration testing
Deployment
•Deployment tools•Supports continuous deployment
Execution
•Monitoring•Responding to error conditions
DevOps Influences…❖ DevOps practices will influence industry
❖ the way you organise teams
❖ the way you build systems
❖ even the structure of the systems that you build
❖ DevOps practices will influence research
❖ Software Architecture
❖ split application into small, well-scoped microservices
❖ Software Process
❖ Ops as first-class citizens throughout the lifecycle
Grand Questions of Interest
❖ What are the impacts of DevOps on software industry in general, or software practitioners in particular?
❖ How to support software organisations to adopt or migrate to DevOps?
❖ How to reengineer software technologies and advance the state-of-the-art of DevOps?
Data Tech. Empirical Methodology
Archit. Quality Security Deploy
Process
DevOps
Acad
emic
Ach
ieve
men
t
Data Mining
NLP
Industrial Impact
SLR
MLR
QRS
Doc Engineering
Mic
rose
rvic
es
DD
D
Logg
ing
Con
tain
er
Dev
SecO
ps
API
Gat
eway
Secu
re A
rch.
AgilityCI Lean KanbanCD Automated Process
Toolchain
DL
Qualitative Methods
Log
Ana
lysi
s
MobileInternet
Big Date
Cyber-PhysicalSystem
Smart autonomous system
。。。
Application Domains
Software Engineering
Fundamental Competency
Cyber-Security
Nanjing University · DevOps Research Centre
DFD
Organisation & Culture
Fundamental Competency ❖ Empirical Methodologies
❖ Evidence-Based Software Engineering (Systematic Reviews)
❖ methodological issues: data synthesis, quality assessment, validity …
❖ Qualitative Research in Software Engineering
❖ ethnography, field study, grounded theory …
❖ qualitative data analysis and synthesis
❖ Grey Literature in Software Engineering
❖ evidence extraction & assessment of grey literature
❖ multivocal literature review
❖ Data Technologies
❖ Data Mining
❖ mining software process from repository
❖ mining software metrics for process modelling
❖ Natural Language Processing
❖ NLP in support of qualitative analysis & synthesis
Qualitative Evidence SynthesisResearch Synthesis
aggregative
< method >
interpretive
< extracted information > < study design >
booleanlogic
frequency structuredquestions
< data >
findings raw data
meta-summary contentanalysis
comparativeanalysis
case survey
diversedesign
meta-ethnography
comparabledesign
< purpose >
theorygeneration
understanding
groundedtheory
narrativesynthesis
thematicsynthesis
DevOps Oriented Software Engineering ❖ Toolchain
❖ “DevOps Tooling Technical Report”
❖ Microservices Architecture❖ Impacts on Quality Attributes
❖ Microservice Granularity
❖ Migration from Monolith to Microservice System
❖ Decomposition for Microservices
❖ constraints
❖ domain-specific design
❖ data-flow diagram
❖ Benchmark Example for Performance Evaluation
❖ pet-store project
DevOps Tooling
MSBuildMSBuild
AzureAzure
Academic Literature: 144Academic Literature: 144
: 178: 178
: 76
: 76
: 117: 117
DevOps Tooling
Data-Flow Driven Service Decomposition
DevOps Oriented Software Engineering ❖ Quality
❖ Logging Practices
❖ *Security
❖ Secure Architecture
❖ DevSecOps
❖ *Deployment & Operation
❖ Process & Agility
❖ Automated Software Process
❖ Mining Software Process
❖ Software Process Simulation Modelling
❖ hybrid simulation modeling
❖ model verification & validation
❖ Agile / Lean / Kanban Development
❖ Interplay between Process and Product
❖ Organisation & Culture
Metrics for Process ModellingProcess
…
……
Product
Legend
Resource
……
Requirement Design Implementation Review Test Delivery & Maintenance Phase IndependentTime(270)
-HNKLLDL
HGM L
Project Duration
-HNKLLDL
HGM L
Test Duration
-HNKLL
Fix Duration
LTask Duration
-HNKLL
Review Duration
-HNKL-HNKL D
Work Time-HNKL
L
Interval between Releases
LLead time
LAnswer time
LDelivery time
HGM L
Domain Discovery Delay
HGM L
Design Filling Delay
HGM L
Analysis Discover Delay
HGM L
New Requirements Feedback Delay
L %
Requirement Selection Time L
Coding Duration
-HNKLLDL
Rework Duration
L
Interval between Requirements
-HNKL
Issue Assessment
Duration
Requirement Design Implementation Review Test Delivery & Maintenance Phase IndependentDefect
(246)
ML
# Injected Requirement
Defects
Injection Rate during Design
ML
# Design DefectsPercentage of
Fixed Requirement
DefectsDesign Defect
Density
MLML -HNKML HGMML LDML GCM
Injection Rate during Coding
ML
# Injected Code Defects
ML HGMFix Rate
Percentage Fixed DefectsML
# Fixed Defects
MLML LM L
Detected Defects in Test
Defects Detected in Reviews % ML -HNK
Defect Detection Rate
Overlap of Detection of Reviewers
ML
# Detected Defects
Rate of Defect Delivered
ML
# Detected Defects after
Test
ML
Released Defect Density
CF GLCHGE LLDefect Severity
ML -HNKML HGM
Defects Injection Rate
MLML GCM
Defect Density
CF GLCHGE LLDefect Type
ML
# Remaining Defects
Requirement Implementation Review Test Delivery & Maintenance Phase IndependentTask Size
(112)
A L
Task Size per Inspection
.GLI MCHGL# Inspections
GCML
Amount of ReworkLDL
L K MHKC L
Amount of Implementation
Work
MNK L CF
Demand for New Features
NCK F GMLTest Task Size
LDLTotal Tasks
KLHG DL LD-HNKL LD
Task Size
LHNK L LD
Task Required Resource
LDL L K MHKRequired Tasks
N LML
# Maintenance Requests
CF GLCHGE LLDelivery Times
Requirement Design ImplementationChange(48)
GA L# Requirement Changes
GA L CFRate of Requirement Changes Change request acceptance Rate Change Request Time Span
GA L# Design Changes
GA L# Code Changes
Design Implementation Review Test Phase IndependentEffort(143)
KLHG -HNKLKLHG LKLHG DLKLHG HGM LL
Total Effort
KLHG DLEffort per Item
Effort Allocation Percentage
KLHG -HNKLCoding Effort KLHG L
Test Effort
-HNKL GCMTest Effort RateKLHG DL
Effort per Test Case
-HNKLDesign Effort Review Effort
KLHG LRework Effort
Meeting EffortPreparation
EffortKLHG DLKLHG L M
Fixing Effort
Requirement Design Implementation Review Test Delivery & Maintenance Phase IndependentProcess
Factor(106)
CF GLCHGE LLTime Pressure
CF GLCHGE LLCoding Method
CF GLCHGE LLTask Priority
CF GLCHGE LLPolicy
CF GLCHGE LLRisk Impact
CF GLCHGE LLQuality Strategy
…
CF GLCHGE LLTest Method
OC PL
# Periodical Reviews
CF GLCHGE LL
Review Adjustment
Policy
CF GLCHGE LL
Domain Discovery Factor
CF GLCHGE LL
Analysis Discovery Factor
CF GLCHGE LL
Design Discovery Acceptance Factor
CF GLCHGE LL
New Requirements Feedback Factor
CF GLCHGE LL
New Requirements Scaling Policy
CF GLCHGE LLRequirements Volatility
CF GLCHGE LLRelease PolicyCF GLCHGE LL
Maintenance Adjustment
Factor
Design Implementation Review Test Delivery & Maintenance Phase IndependentDevelopment
Rate(75)
.
LLDL .M K MCHG
Development Rate
Discover RateH NF GMLH NF GML HGM
Design Rate Implementation Rate GCML CF
Delivery RateReview Rate Test Rate-HNKL MLDL
Rework Rate
Phase IndependentProcess Static Attributes(13)
CF GLCHGE LL# Increments
CF GLCHGE LL# Sprints
CF GLCHGE LL
Stage of Development
CF GLCHGE LL
Stage of Development
CF GLCHGE LL
Multisite Development
KLHGLTeam Size
FL# Team
KLHGL# PersonsKLHGL# Testers
KLHGL
# Experienced Staffs
KLHGLDatabase Size
HHFL
# Available Inspection Rooms
CECMC L # -HNKL DTest Facility Availability
Team Personal EnvironmentResource Size(194)
Team PersonalProductivity(107)
-HNKLLM L L
I C C MCHGL -HNK
Team Productivity / /
Individual Productivity
TeamPersonnel Continuity(34)
Turnover Rate Hiring RateKLHGL
Assimilation Rate …
Team Personal EnvironmentResource Factor(19)
CF GLCHGE LLTeam Cohesion
CF GLCHGE LLSelf Esteem
CF GLCHGE LLTeam Utility CF GLCHGE LL
Culture
CF GLCHGE LLAchievement Need
CF GLCHGE LLNative Language
CF GLCHGE LLPenalty Factor
Personal EnvironmentCapability(82)
CF GLCHGE LLCommunication Overhead
CF GLCHGE LLTraining Overhead
CF GLCHGE LLTask Switch Overhead
TeamOverhead(20)
CF GLCHGE LLSkill Level
CF GLCHGE LLExpertise
CF GLCHGE LLCapability
CF GLCHGE LLKnowledge
CF GLCHGE LLUse of Tools
Team PersonalExperience(27)
CF GLCHGE LLTeam Experience
CF GLCHGE LLIndividual Experience
Requirement Design Implementation Test Phase Independent
Product Size(185)
A LFPsTasksUnits
Size of Design Document
A LFPsStoriesUnits
Size of Requirement
DocumentH NE LKLHG$ LHGMKC NMCHGLGCML
Size of Product/System/ Project
CE LGCML
Size of Component
.L
Size of CodeLM L L# Test Cases
Size of Test Code
Test Case Pass Rate
Design Implementation Test Phase IndependentDetail Measure(29)
C AK FLHGMKHE EHPL
Detail Measure of Design Document CLCHG HNGM
H NEH E O KC E LEH E O KC E L
Detail Measure of Code
HGMKHE EHPLL L L
Detail Measure of Test Case HK L A
K AK I LGM G L
/ PHK L
Detail Measure of Artefact
Phase IndependentProduct Feature(30)
HK L AK AK I LGM G L
/ PHK L
Detail Measure of ArtefactCF GLCHGE LLSystem Type
CF GLCHGE LLUser Type
MNK L# Features
CF HG L# Time Zones
…
Phase Independent
Complexity(40)
CF GLCHGE LLComplexity
CF GLCHGE LLMaintainability
CF GLCHGE LLReliability
CF GLCHGE LLReusability
Phase IndependentEffectiveness(34)
CF GLCHGE LLTest Effectiveness
CF GLCHGE LLReview Effectiveness
Test Review
CF GLCHGE LLEffectiveness of the Process Phase IndependentCost(48) Test Phase Independent
Sub-Category (# Metrics) EntityInternal Category:
Sub-Category (# Metrics) EntityExternal Category:
GCMR%
Metric
…Rarely used and we can not list all of them:
The specific metric with its unit:
Phase Independent
Maintainability(2)
Phase Independent
Reliability(8)
Phase Independent
Reusability(4)
GCMR%
MetricThe Metric belongs to this Sub-Category and falls in this Type
-HNKTester Cost
Quality(28)
-HNKL MDefect Cost
Individual Cost
Project Cost
KLHG LCost per Effort …
MLML
LD OCL
Quality
MRisk Start/End Date …
Performance(8) Phase Independent
CF GLCHGE LLCost Performance Index
CF GLCHGE LLSchedule Performance Index
* HK -HNKDevelopment Performance …
Legend
Process Intenal
Process External
Resource Intenal
Product Intenal
Resource External
Product SizeDetail Measure
Product Features Complexity
Maintainability
Reliability
Quality
Time
Defect
Task Size
Change
Effort
ProcessFactor
Development Rate
Effectiveness
Cost
Resource Size
Productivity
Personnel Continuity
Capability
Experience
Generates& Detects& Fixes(69)
Product External
Overhead
Measures(5)
Impacts (15)
Impacts(3)Impacts(9)
Impacts(17)
Extends
Extends
Extends
Extends
Extends
Extends
Extends
Extends
Extends
Measures(9)
Changes(6)
Releases(9)
Extends Extends
Extends
Extends
ExtendsExtends
Extends
Extends
Extends
Measures����
Measures(1)
Measures(1)
Measures(1)
Measures(5)
Measures(7)
Impacts(22)
Measures(10)
Extends
Extends
Impacts (7)
Measures���Impacts(3)
Requires(1)
Impacts(2)
Impacts���
Extends Impacts(4)
Impacts (5)
Measures(9)
Measures(35)
Process Static Attributes
Resource Factor
Impacts(11)
Measures(12)
Mesures(16)
Measures (12)
Extends
Impacts(1)
Requires(33)
Produces(33)Measures(16)
Requires(25)Produces(25)Requires(18)
Produces(18)
Category
Sub-Category
Label(# evidence)
Extends
Requires(9)
Measures (4)
Impacts(7)
Impacts(8)
Metrics for Process Modelling
Industrial Impacts❖ Synergy by combining research, industry, and education
❖ 《DevOps: Theory, Method, and Practice》(1st DevOps textbook in China)
❖ DevOps curricular and teaching/training platform
❖ Industrial Collaborations on DevOps: Huawei, ZTE, Transwarp, DaoCloud …
❖ 《DevOps · China 2016 Report》
❖ 《DevOps · China 2018 Report》(available soon)
❖ 《DevOps Tooling Technical Report》
❖ 《Software Architecture for Practitioners》@ GeekTime
❖ ArchSummit 2017 / 2018 (Shenzhen)
❖ Keynotes & invited talks at CNUTCon 2017 / NASAC 2017 / NJSD 2018 / DOIS 2018 …
❖ DevOps tracks at IAS 2017 & NJSD 2018
❖ DevOps Special Group of CCF · Software Engineering & System Software Committees