Why measure anything?

We want to continually improve, to further the MoFo mission.

Humans, and also organizations, can't know if they can or should do something differently

unless they get feedback or data to prompt the question:

"How can we/I do better at this?"

Absent of any data, everything looks fine

First Requirement of Improvement

Realize there is something that needs improvement

__________________

Knowing the truth is often scary and uncomfortable, but at least you know when to run!

Fast, Confident, and Awesome

If an org, team, or individual:

-Knows, definitively, how they are doing now

-Knows, definitively, how they are doing after a change

-Knows, definitively, what variables changed when

Then that org, team, or individual:

-will be much faster to determine success vs. failure, making it possible to resolve faster.

-will inevitably grow and learn, further and faster

-generates more trust...with users, contributors, org, team

and maybe most importantly...

-We'll be more confident to try more things, knowing we can quickly detect and resolve.

-Thus, as we will try more things, it will lead more often to success.

Had Good Metrics Indicating Dogs Were Tame...Confident.

What kind of stats are we talking?

Not just CPU and App Performance!

metrics << app / productmetrics << projects / teams

metrics << users / customersmetrics << contributorsmetrics << operations

metrics << performancemetrics << developers

metrics << support

Proposed Layers of Metrics

1. Organization Layer - "How is Mozilla Foundation doing at its mission?"

2. Team Layer - "How well is this team working together and accomplishing goals? "

3. Application/Service Layer - "How are things with X application or service?"

4. Operations Layer - "How are we doing, logistically?"

Organizational Metrics ExamplesHow many visits to all of our sites this year vs. last year?

How many people have donated to MoFo, and how much?

How many times were we or any of our sites mentioned on social media and the news?

How many new "webmakers" did we bring into our ecosystem across all applications and teams?

Team Metrics Examples

How many hours were worked/PTO'd on the team?

How many projects is that team involved in this week?

How many collaborator / volunteer / staff lines of code / features were added?

How many milestones did the team meet this month?

Application Metrics Examples

What was the average response time for popcorn.webmaker.org this week?

How many videos were edited this week?

How many new videos were created this week?

What was the # of page visits for popcorn on average by hour?

What was the ratio of errors to successes for popcorn's key transactions this week?

Operations Metrics ExamplesHow much disk is currently being used?

How much memory and CPU utilization do we have in production?

How many gigabytes of logs were generated this week?

What was the slowest loading page per application this week?

How much did our infrastructure cost?

What tools will we be using?

Primary Tools● Newrelic● StatsD/Graphite● Google Analytics● Geckoboard

Secondary Tools (Not covering today)● AWS, Pingdom, Opsview

New Relic (Server Monitoring)

New Relic

Real time data on your server and app performanceExamples:● what code is running slowly or erroring?● how long are end users waiting before they

can interact with our content?● How many database writes per minute?● End user performance monitoring ● Which deploy caused a problem?

Google Analytics

Google Analytics

Numbers on who is using your app, how and when they are using it

● what content is popular with who?● which parts of the site are causing people to

leave?● where are our users coming from?● what things have they done while on our

sites?● Can track campaigns, goals (ie, made a

popcorn video), conversion rates

statsd/graphite

Multiple stats, one stream

statsd/graphite

statsd - "Collects data by increment counters and timers, but does not show, sort, or average(etc....)"

graphite - Shows the data in a presentable way, allowing you to line up multiple metrics and perform calculations, such as sort, avg, std deviation

(Examples of a few things we might count/time)● someone made a project, remixed a project● someone has been issued a badge● someone has decided to delete their profile

StatsD examples (counting new vids in popcorn)(note the data structure, env.app.what)

Time something (PHP example)$start = microtime(true);postvideo();StatsD::timing("production.butter.postvideo.response.200", (microtime(true) - $start) * 1000);

Count something (PHP)StatsD::increment("production.butter.postvideo.response.200");

Geckoboard

Geckoboard

A friendly view into all of our high level stats.

● this is where Mark/Ryan will come in● this is where we highlight the individual

metrics we get judges against● one stop-shop over all MoFo/webmaker

properties● encouraged to make public

Metrics Flow

Things to Beware Of

● Presenting data out of context for audience● Instrumentation of anything to New Relic,

Google Analytics, Geckoboard should go over ssl

● Do not use New Relic auto instrumentation to capture any params in the database

● Gameable Metrics

Go fourth and metric!

See what you can do with these client libs:

● statsd:○ https://django-statsd.readthedocs.org/en/latest/○ https://github.com/sivy/node-statsd

● geckoboard:○ http://pypi.python.org/pypi/django-geckoboard○ https://github.com/danjenkins/node-geckoboard-

push ○ https://github.com/Hebo/node-geckoboard

● newrelic custom instrumentation:○ https://newrelic.com/docs/instrumentation/custom-

metric-collection

Links/ReferencesGood read with a further intro for statsd/graphiteHow we will setup Opsview to watch StatsD/GraphiteGoogle Analytics: Setting up goals and tracking conversions

Easy setup instructions for statd/graphite server(if you want to test locally)