+ All Categories
Home > Technology > FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Date post: 05-Dec-2014
Category:
Upload: new-relic
View: 354 times
Download: 1 times
Share this document with a friend
Description:
Your application stack is made up of a myriad assortment of technologies, from cloud services, to the database, caching, networking, queuing and more. In order to gain a comprehensive view of your application performance, you need performance visibility into all these technologies. And, you'd like that visibility within a single tool that you already know and love. The New Relic Platform gives you exactly that, an in-depth, real time insight into the performance of each component of your application stack. And best of all, it's free! Find out all about the New Relic Platform and learn what it has to offer you in its wide selection of Plugins, providing insight into AWS, MySQL, Redis, MongoDB, RabbitMQ, Varnish, and many many more. Its extensible architecture means that if there isn't a plugin for the technology you want, writing one is not only possible, but easy!
56
Monitor all the Things! KEVIN MCGUIRE - DIRECTOR OF ENGINEERING, PLATFORM OCTOBER 25/2013 Wednesday, November 6, 13
Transcript
Page 1: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Monitor all the Things!

KEVIN MCGUIRE - DIRECTOR OF ENGINEERING, PLATFORMOCTOBER 25/2013

Wednesday, November 6, 13

Page 2: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

The evolution of monitoring

Wednesday, November 6, 13

Page 3: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

The evolution of monitoring

Wednesday, November 6, 13

Page 4: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

The evolution of monitoring

Wednesday, November 6, 13

Page 5: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

The evolution of monitoring

Wednesday, November 6, 13

Page 6: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Your end user experience depends on more

Wednesday, November 6, 13

Page 7: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

A lot more

Wednesday, November 6, 13

Page 8: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

You can’t manage what you can’t monitor

Wednesday, November 6, 13

Page 9: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

You can’t manage what you can’t monitor

Wednesday, November 6, 13

Page 10: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

You can’t manage what you can’t monitor

Wednesday, November 6, 13

Page 11: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Solution?

Wednesday, November 6, 13

Page 12: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

MONITOR ALL THE THINGS!

Solution?

http://hyperboleandahalf.blogspot.com/

Wednesday, November 6, 13

Page 13: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Platform gives complete visibility

Wednesday, November 6, 13

Page 14: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Platform gives complete visibility

Wednesday, November 6, 13

Page 15: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Platform gives complete visibility

Wednesday, November 6, 13

Page 16: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Plugins for, well, everything!68 69 published plugins29 different publishersToo big to fit in presentation!

Wednesday, November 6, 13

Page 17: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

We got databases

Wednesday, November 6, 13

Page 18: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

We got yer NoSQL stores

Wednesday, November 6, 13

Page 19: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Yup, caching too

Wednesday, November 6, 13

Page 20: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Queuing, web server, load blance, ...

Wednesday, November 6, 13

Page 21: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

And my personal favorite:

Wednesday, November 6, 13

Page 22: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

And my personal favorite:

You go Yuri Yasiyarov, wherever you are!

Wednesday, November 6, 13

Page 23: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Integrated, 1st class experience

Wednesday, November 6, 13

Page 24: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Integrated, 1st class experience

Wednesday, November 6, 13

Page 25: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Integrated, 1st class experience

Wednesday, November 6, 13

Page 26: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Integrated, 1st class experience

Wednesday, November 6, 13

Page 27: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Integrated, 1st class experience

Wednesday, November 6, 13

Page 28: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Customized chartsMySQL plugin has MySQL’y stuff

Wednesday, November 6, 13

Page 29: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Install is a snapEasy as...

1

Wednesday, November 6, 13

Page 30: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Install is a snapEasy as...

1 2[ {

"name" : "Localhost","host" : "localhost","user" : "USER_NAME_HERE","passwd" : "USER_PASSWD_HERE"

},]

licenseKey=blahblahblah12345

Wednesday, November 6, 13

Page 31: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Install is a snapEasy as...

1 2 3[ {

"name" : "Localhost","host" : "localhost","user" : "USER_NAME_HERE","passwd" : "USER_PASSWD_HERE"

},]

java -jar newrelic_mysql_plugin.jar

licenseKey=blahblahblah12345

Wednesday, November 6, 13

Page 32: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Independent runtime

+ GUID +

RPM

MySQL UI Memcached UI

VarnishUI

Infrastructure

MySQLServer 2

MemcachedServer

MySQLServer 1

Platform Agents

VarnishAgent

Memcached Agent

MySQLAgent...

VarnishServer

...

Externally monitored

New Relic MetricStore

Wednesday, November 6, 13

Page 33: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

SaaS

+ GUID +

Customer BData

Customer BData

Customer BData

RPM

SaaS UI

Memcached UI VarnishUI

SaaS Service

Customer B Customer XCustomer A

...HTTP API

New Relic MetricStore

SaaS ‘agent code’

Customer Xlicense key

Wednesday, November 6, 13

Page 34: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Don’t see one you like?

Built 9 prior to launch,now at 14! Built the Redis Plugin in a

couple hours, and used it to solve performance issues the same day

Built 6! Built 4! Built 3!

Wednesday, November 6, 13

Page 35: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Simple APIs + SDKs = Easy to write

Wednesday, November 6, 13

Page 36: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Simple APIs + SDKs = Easy to write

•SDKs do all the “heavy lifting”•Java (primary), Ruby•Runs polling cycle•Connecting to New Relic •Aggregation

Wednesday, November 6, 13

Page 37: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Simple APIs + SDKs = Easy to write

•SDKs do all the “heavy lifting”•Java (primary), Ruby•Runs polling cycle•Connecting to New Relic •Aggregation

•HTTP API•SaaS•Non Java/Ruby

Wednesday, November 6, 13

Page 38: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Anatomy of a plugin agent(yes, it’s really that simple)public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }

   private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }

   public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();  

       reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    }}

Wednesday, November 6, 13

Page 39: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Anatomy of a plugin agent(yes, it’s really that simple)public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }

   private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }

   public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();  

       reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    }}

Unique identifier (GUID), agent version

Wednesday, November 6, 13

Page 40: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Anatomy of a plugin agent(yes, it’s really that simple)public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }

   private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }

   public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();  

       reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    }}

Query metrics from system

Unique identifier (GUID), agent version

Wednesday, November 6, 13

Page 41: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Anatomy of a plugin agent(yes, it’s really that simple)public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }

   private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }

   public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();  

       reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    }}

Query metrics from system

Unique identifier (GUID), agent version

Value

Wednesday, November 6, 13

Page 42: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Anatomy of a plugin agent(yes, it’s really that simple)public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }

   private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }

   public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();  

       reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    }}

Query metrics from system

Unique identifier (GUID), agent version

Identifier Value

Wednesday, November 6, 13

Page 43: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Anatomy of a plugin agent(yes, it’s really that simple)public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }

   private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }

   public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();  

       reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    }}

Query metrics from system

Unique identifier (GUID), agent version

Identifier Units Value

Wednesday, November 6, 13

Page 44: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Anatomy of a plugin agent(yes, it’s really that simple)public  class  MySQLAgent  extends  Agent  {    public  MySQLAgent()  {        super("com.mycompany.mysql",  "1.0.0");    }

   private  Map<String,  Number>  gatherMetrics()  {        Map<String,  Number>  metrics  =  MySQL.runSQL(getConnection(),  "SHOW  GLOBAL  STATUS");          return  metrics;    }

   public  void  pollCycle()  {        Map<String,  Number>  metrics  =  gatherMetrics();  

       reportMetric("network/bytes_reads",                            "bytes/second",        metrics.get("bytes_reads"));        reportMetric("network/bytes_writes",                          "bytes/second",        metrics.get("bytes_writes"));        reportMetric("queries/com_select",                              "selects/second",    metrics.get("com_select"));        reportMetric("queries/com_insert",                              "inserts/second",    metrics.get("com_insert"));        reportMetric("queries/slow_queries",                          "queries/second",    metrics.get("slow_queries"));        reportMetric("connections/connected",                        "connections",          metrics.get("connections_connected"));        reportMetric("innodb/buffer_pool_pages_flushed",  "pages",                      metrics.get("innodb_buffer_pool_pages_flushed"));        reportMetric("innodb/buffer_pool_pages_dirty",      "pages",                      metrics.get("innodb_buffer_pool_pages_dirty"));        //  ...    }}

Query metrics from system

Send to NR

Unique identifier (GUID), agent version

Identifier Units Value

Wednesday, November 6, 13

Page 45: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Easy to build UIs

Wednesday, November 6, 13

Page 46: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Custom DashboardsFast and easy UI creation

Wednesday, November 6, 13

Page 47: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Optionally publishLists in Plugin Central, makes available the UIs

Wednesday, November 6, 13

Page 48: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Install UI = run the agent

+ GUID +

RPM

MySQL UI Memcached UI

VarnishUI

Infrastructure

MySQLServer 2

MemcachedServer

MySQLServer 1

Platform Agents

VarnishAgent

Memcached Agent

MySQLAgent...

VarnishServer

...

Externally monitored

New Relic MetricStore

Wednesday, November 6, 13

Page 49: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Install UI = run the agent

+ GUID +

RPM

MySQL UI Memcached UI

VarnishUI

Infrastructure

MySQLServer 2

MemcachedServer

MySQLServer 1

Platform Agents

VarnishAgent

Memcached Agent

MySQLAgent...

VarnishServer

...

Externally monitored

New Relic MetricStore

Wednesday, November 6, 13

Page 50: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

“But Kevin, how much will all this cost?”

Wednesday, November 6, 13

Page 51: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

“But Kevin, how much will all this cost?”

Free!!Wednesday, November 6, 13

Page 52: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

New! Plugin community reviewsHelp improve plugins, give feedback

Wednesday, November 6, 13

Page 53: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

New! Private publishingDistribute within multiple accounts, closed Betas

Wednesday, November 6, 13

Page 54: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

In progress: Deployment recipesSimplified configuration and deployment across your data center

Wednesday, November 6, 13

Page 55: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Get involved!

✓ Review plugins

✓ Contribute to the plugins • Many are open source• Write one!

✓ Contribute to the SDKs

✓ Help us with Chef• Try out our recipes• Try authoring recipes

Wednesday, November 6, 13

Page 56: FUTURESTACK13: Monitor All the Things from Kevin McGuire, Director of Engineering at New Relic

Thank you!

Wednesday, November 6, 13


Recommended