Date post: | 12-May-2015 |
Category: |
Technology |
Upload: | devnology |
View: | 1,469 times |
Download: | 0 times |
Devnology: Software Operation Knowledge in Practice
Dr. Slinger Jansen
Utrecht University
Introductie UU en Software~VOC
09-08-13 open dagen 3
Who am I?: Slinger Jansen
§ Scientist
§ Entrepreneur
§ Advisor
§ Dad
§ Driver of old cars
Software Intensive Systems Developers suffer from
§ Increasing quality requirements
§ Decreasing release times
§ Data overflow of feedback data
We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us. - Ray Ozzie
Solution: Generic Data Acquisition and Data Mining
§ Generic data acquisition through instrumentation
§ Data gathering without altering the source code (aspect-oriented development)
§ Microsoft found out through their crash reporting program that 50% of crashes are caused by 2% of their bugs
State of the practice
§ Release sub-process - Release at times that are convenient to customer: 57%
- Use of formal release planning: 45% - All stakeholders can access the planning: 85% - Use of formal release scenario: 56% - Use of versioned configuration management: 54% - Explicit management of in-house build tools: 70% - Use of COTS: 38%
§ Delivery sub-process - Informing the customer through a website: 84% - Other channels not used often - Delivering updates through the web: 85% - Respondents that use CD & DVD: 40% - Manual product pull: 92%
- Automatic product push: 12%
State of the practice
§ Deployment sub-process - Use of an update tool: 30% - Tool is able to update at runtime: 56%
- First time installation failure: 8% - Possibility of de-installing previous release: 89% - Undo an update: 59% - Check of customer configuration: 73%
- Data backup: 59% § Usage sub-process
- Use of license agreement: 73% - Knowledge about the customer platform: 73% - Use of error reports: 35%
- Use of usage reports: 40%
Vision: Continuous Customer Configuration Updating
§ Continuously release software after every change to a release repository
§ Continuously (be able to) release to customers
§ Continuously (be able to) deploy an update (run-time?)
§ Continuously change your software
§ (You should at least strive for continuous release!)
Possibly part of a Software Supply Network
Software Developer
Cust. 1
Cust. m
. . .
Software Developer
Cust 1
Cust n
. . .
Operational Environment 1
Operational Environment n
Operational Environment m
. . .
. . .
Slinger Jansen and Wilfried Rijsemus: Balancing Total Cost of Ownership and Cost of Maintenance Within a Software Supply Network , proceedings of the IEEE International Conference on Software Maintenance (ICSM2006, Industrial track), Philadelphia, PA, USA, September, 2006.
Or even an ecosystem!
Do you “own” one?
SOK Propagation in Software Ecosystems
§ Relationship management in software ecosystems – Keystones use various participation incentives to bind niche players to their
ecosystem(s)
– SOK resulting from software compatibility monitoring is input for relationship management
– Other incentives are used to strenghten relationship
• Developer keys > certification programs > private SDK access
SOK Propagation in Software Ecosystems
§ Software quality management – Vendors use SOK to manage various types of software quality
– Particularly useful in:
• Determining root cause of in-the-field software failures that do not occur at the software vendor site
• Isolation of Heisenbugs
– A computer bug that disappears or alters its characteristics when an attempt is made to study it
– EFfective SOK utilization quality management success
Continuous feedback mechanisms § Email
§ SMS
§ Ambient Orbs
§ Windows taskbar monitor
§ Sound
What does your office use?
Pragmatic Process Improvement through SOK
§ Introduction – Many software vendors that acquire SOK, … just acquire it.
• Limited data mining, limited ‘integration’ in software processes
• Unclear who is responsible
– No structural integration with processes and tools in-place
• No protocols or workflows to involve acquired data
• SOK is not utilized to support decisions and steering
– Release management
– Requirements prioritization
– etc.
§ What do you collect, use and report on?
§ What kind of actions are connected to these data?
SOK Lifecycle
37
Who acquires data without using it? And yes, support calls count!
Einde Theoretisch Deel
SOK Integration Template Method
§ The method is designed to guide vendors in: – Analysis of integration environment and target processes
– Identification of relevant and valuable operation information
– Integration of selected information in, and adaption of, target processes
– Presentation of integrated operation information
§ We assume that, before using the method:
– Software operation information
– One or more target processes
§ are accessible and available for integration
SOK Integration Template Method
§ What do you want to look at immediately when you enter your office tomorrow morning?
§ OK, but how should we use the method? – For Stabiplan, we built Errata Garden
• Crash report mining and -analysis tool
• Allows access to in-the-field operation information based on acquired crash reports
SOK Integration Template Method
§ OK, but how should we use the method? – Stabiplan had questions like
• How do we make people use this information?
• How do we prevent productivity to be descreased by EG monitoring?
• How frequently should Errata Garden reports be monitored?
• Who is responsible?
§ How can information provided by Errata Garden be effectively integrated with our product software processes?
SOK Integration Template Method
§ OK, but how should we use the method? – Template method was instantiated with three processes
• Software maintenance
• Software product management
• Customer support
– Action research
• Researchers present at the software vendor site
• Observing phenomena while adhering to CAR principles
– Researcher-Client Agreement, Learning through Reflection, etc.
Davison, R.M., Martinsons, M.G., Kock, N.: Principles of canonical action research. Information Systems Journal 14, 65–86 (January 2004)
Software maintenance
Software product management Customer support
Developers not used to involve reports during
maintenance
PMs rely on customer support and salesmen for release planning
Supporters rely on customer explanation for softw. operation
insight
Accelerated software maintenance
Directed software product (release)
management
Increased customer intimacy
Independent requests from developers Evaluation of EG tool Short talks
Error report meta information (status)
Aggregated error report properties, operation
trends
Operation information enabling particular
identification of customers
Activity and deliverable descriptions
Software maintenance
Software product management Customer support
Maintainers were selected by us and appointed by CEO
PMs were selected by us and appointed by
CEO
Customer supporters were selected by us
and appointed by CEO
Analysis of maintenance process: at least once a week
Analysis of SPM process: at least
every Scrum sprint
Analysis of customer support process:
possibly every call
EG list view with error report status
EG aggregated error report view
EG customer-spectific error report views
Creating and introducing error report labeling
Creating and introducing error
report aggregation; organizing meetings
Extending error report format and EG
analysis functionality
Activity and deliverable descriptions
Software maintenance
Software product management Customer support
EG extension
EG extension, harvest meeting introduction for
frequent analysis of error reports
EG extension
Evaluative talks Evaluative talks Evaluative talks
Activity and deliverable descriptions
SOK Integration Template Method
§ Main result: Increased software quality – Report submissions increased about linearly from February, 2009
until June, 2010
– Trend was broken in September, 2010, while a major new version was released
SOK Integration Template Method
§ Based on observations and interviews, lessons learned are identified – What should a vendor do to effectively involve such information in its
product software processes?
– What should it absolutely not do? What are considered pitfalls?
– Observing integration in retrospect, what are your lessons learned?
SOK Integration Template Method: Lessons Learned
§ 1. Integration Processes Should be Lean – Handle additional (administrative) tasks pragmatically:
additional administration may negate the time gained by integration of operation information
SOK Integration Template Method: Lessons Learned
§ 2. Integration Responsibilities and Results Should Be Evangelized – Shared responsibility is no responsibility
– Increase awareness and acceptance of new way of working
• Both among employees as at management level!
Who is your evangelist?
SOK Integration Template Method: Lessons Learned
§ 3. SOK Integration Opens Up Black Boxes – Yes, informed development, software failure fix prioritization, etc.
– But: improvement areas that were unnoticed before, are highlighted as such after SOK integration
• Increased developer awareness (software quality, end-user situation)
• More customer-central, pro-active development mentality
– `Build what the customer will use, before the customer asked for it'
• Clearer and quicker interdepartemental communication
• No more HiPPOs!
SOK Integration Template Method: Lessons Learned
§ 4. Continuous Refinement of SOK Integration Objectives and Requirements Leads to Optimization of Integration Results – Integration results depend on integration objectives and requirements
– Both should be evaluated and refined continuously to correspond to your product strategy, as well as the needs of your customers
Challenges
§ Visualization
§ Alignment business vs technical metrics
§ Big data (or just real time data) & data mining
§ Errors at different levels
Data Mining and Visualization
Business Metrics vs…
65
What’s your company’s mission statement?
Technical metrics
66
Performance Metrics
Memory
Nonpaged System Memory SizePaged Memory SizePaged System Memory SizePeak Paged Memory SizePeak Virtual Memory SizePeak Working SetPrivate Memory SizeVirtual Memory SizeWorking Set
Processor
Privileged Processor TimeTotal Processor TimeUser Processor Time
Network Interface
Bytes Received/secBytes Sent/secPackets Outbound ErrorsPackets Received ErrorsTotal Bytes ReceivedTotal Bytes Sent
Disk
% Disk Time% Free SpaceAvg. Disk Queue LengthAvg. Disk sec/TransferDisk Reads/secDisk Transfers/secDisk Writes/secFree Megabytes
Web Tier
Page Type CountQuery String CountTotal Request DurationAuthenticate Request DurationAuthorize Request DurationResolve Request Cache DurationAcquire Request State DurationRequest Handler Execute DurationRelease Request State DurationUpdate Request Cache DurationModule Type CountMethod Type Count
Usage Metrics
User Details
Customer IDIP/Hostname
Application Details
Web site vs web applicationHosting
User Actions
Actions PerformedPage/function/method accessTimeframeEntry/Exit pointSession IDErrors
Tata Steel produceert, bewerkt en distribueert hoogwaardig staal voor producten die het leven vergemakkelijken.
Belastingdienst: Geld innen voor de BV NL
Aviva Solutions: In eerste plaats biedt Aviva Solutions u goede ICT-oplossingen door het bouwen van applicaties. Maar buiten zorgen voor continuïteit en vernieuwen van uw applicaties en systemen, helpt Aviva Solutions u graag bij andere ICT-vragen. Als dé Microsoft Platform Specialist zijn we immers expert op het gebied van: Integratie, Business Intelligence, Custom Solutions & Ontwikkelstraten, Information Worker en Internet & Commerce.
AFAS Software automatiseert bedrijven en organisaties uit alle branches met complete en moderne (online) bedrijfssoftware.
So… How to get started?
§ Assess most important KPIs
§ Collect them (tons of tools available)
§ Analyse them (tons of tools available)
§ Report them (…)
§ ACT!
?