Debugging, Troubleshooting

Distributed Web & Cloud Applications

San Diego, July 27th, 2013

SolCalCode Camp

Theo Jungeblut• Engineering manager & lead by day

at AppDynamics in San Francisco

• Coder & software craftsman by night

• Architects decoupled solutions tailored to business needs & crafts maintainable code to last

• Worked in healthcare and factory automation, building mission critical applications, framework & platforms

• Degree in Software Engineeringand Network Communications

• Enjoys cycling, running and eatingtheo@designitright.net

www.designitright.net

Warning• Contains also proprietary

software

• This areas are pointed out

• Please walk away now,if that is not acceptable for you, it’s okay

Overview• Inquiry about the Status Quo • Some Facts -> Dev Process & Ops Reality• Monolithic Applications• Debugging & Logging• Distributed Applications• Side Effects you need to troubleshot• Performance Counter• Microsoft IntelliTrace• AppDynamics• Summary• Q & A

We need YOU!

How agile are You?

The Impact of Process

* from Wikipedia: http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg

How many releases do your company

have a year?

Typical Releases a Year*

58%

17%

21%3%

2+ monthsmonthlyweeklydaily

http://www.appdynamics.com/blog/2011/12/14/storm-clouds-in-2012-summary-of-appdynamics-apm-customer-survey/

Development + Operations = DevOps

* from Wikipedia: http://en.wikipedia.org/wiki/File:Agile-vs-iterative-flow.jpg

Difference between Dev & Ops

Dev• Getting / being agile: need for frequent updates

with new feature and bug fixes• Achieves stability trough running latest

Ops• Highest goal: stable execution flawless execution,

by knowing the environment and software• Achieves stability through careful rollouts

“The Past/Present”*

WebServer DBUser

* or maybe still your present?

“The Past/Present” - Architecture

Data Access Layer

Business Logic Layer

Presentation Layer

DB

How are you troubleshooting?

Traditional Debugging

Traditional Debugging

Advantages• Visibility for executed code path

Disadvantages• Performance impact• Requires software installation• Often not possible in production• Typical does not find easily distributes issues• Only real time results

Log Files

Advantages• Output adjustable without restart• Accepted in Production

Disadvantages• Manual, time intense creation of log statements• Manual, time intense process of result review• Limited scope & visibility • Potentially high performance impact based on IO

“The Present/Future”- Architecture

Microsoft MSDN: http://msdn.microsoft.com/es-es/architecture/gg189193

“The Present/Future”ASP.NET MVC 3/4

(IIS)User

WAS Tomcat

Azure Web Role

SQL Server MySQL

Active Directory

Azure Worker Role

WAS

3rd Party Web Services

JavaApplication

MongoDB

3rd Party Web Services

HTTP/JSON

LDAP

WCF

WCF HTTPJDBC

ADO.NET

.NET Service Bus

.Net Service Bus

HTTP

ADO.NET

HT

TP

The (Netflix) Distributed Reality *

* http://techblog.netflix.com/2012/06/netflix-operations-part-i-going.html

What has this to do with

Debugging and

Troubleshooting?

Who does really know how their application

is performing?

What impactsApplication

Performance?

Concurrency

http://en.wikipedia.org/wiki/File:Smithposter.jpg

• Deadlocks• Raise Conditions• Unexpected complexity• …

Data Volume

http://en.wikipedia.org/wiki/File:Smithposter.jpg

• constantly increasing• Volume hides details• Handling volume

impacts resources• …

Resources

http://blog.laptopmag.com/amd-cries-foul-but-tests-show-battery-life-inferior-to-intels/battery_morpheus

• are limited• Danger of Starvation• …

Who is Performance Testing

every release?

http://www.slideshare.net/appdynamics/what-can-devops-learn-from-formula-1

Dev + Ops = Success• DevOps has to be a combined approach• Mostly no visibility in real performance

& production state

IntelliTrace™

• “Flight Recorder” for your Process• Records configurable details• Allows offline analysis• Microsoft specific Tool in Visual Studio

Let’s take a look

http://msdn.microsoft.com/en-us/library/hh398365.aspx

Summary IntelliTrace

Advantages• Allows great visibility into single process/AppPool• Allows to play back test or production issue

Disadvantages• Limited to Single Process / AppPool Visibility• Should be only temporally enable in production• Requires configuration effort for balance between

details & performance

Performance Counter• Collect machine wide and process

specific data• Allows on- and offline analysis• Microsoft specific tool build into

Windows

http://www.codeproject.com/Articles/8590/An-Introduction-To-Performance-Counters

Summary Performance Counter

Advantages• Huge OOTB support in Windows/IIS/etc.• Low overhead• Extensible

Disadvantages• Collecting application specific data requires

source code changes

AppDynamics

• Application Performance Management• Optimized for Production• Monitoring instead of investigating• Providing the “Right-Data”

Let’s take a look

http://www.appdynamics.com/products-editions.php

Summary AppDynamics

Advantages• Works in production on high performance sides• Monitors, Notify, Troubleshoot• Allows instance handling in cloud environments• Free lite version

Disadvantages• Full version not free• Does not include every application detail

Build your own Opinion (for Free)

• Lite Version free but limited

• 30 days free Trial of Pro & Azure Version

http://www.appdynamics.com/products-free-download.php

Summary

• Understand Dev Impact to Ops Execution• Understand the business impact of

production issues• Monolithic is simpler to Troubleshoot• Distributed is more complex• The right tools can make a Big Difference

Downloads, Feedback & Comments:

Q & A

Graphic by Nathan Sawaya courtesy of brickartist.com

theo@designitright.netwww.designitright.net www.speakerrate.com/theoj

www. slideshare.net/theojungeblut

Blog, Rating, Slides

http://www.DesignItRight.net

www.speakerrate.com/theoj

www.slideshare.net/theojungeblut

… thanks for you attention!

And visit and support the

www.sandiegodotnet.com

Please fill out the feedback, and…

www.speakerrate.com/theoj