© 2014 VMware Inc. All rights reserved.
Uncovering the Hidden Truth In Log Data with vCenter Insight™ April 2014
VMware vForum Istanbul 2014 Serdar Arıcan
VMware Strategy
2
Software-Defined Data Center Virtual Datacenter:
The ideal infrastructure for private,
public, and hybrid clouds.
Hybrid Cloud Computing Extend your data center into the cloud.
End User Computing Connectivity & Control:
Desktop, App, & Mobile Virtualization
& Provisioning
To help customers realize the promise of the Software-Defined Data Center, VMware introduced the Cloud Suite
VMware is “Going Bigger on Management!”
Business momentum
• 40% bookings growth YoY
• Accelerating Sphere installed base
penetration
• ~2.5x more customers in last year!
Partner commitment
• 3,000+ selling Mgmt (200% YoY growth)
• 2,500+ engaged in Mgmt competency
• Partners invested in Mgmt have grown
bookings 200% more than non-
competency partners
Market leadership
• Ranked #1 by IDC, 451 Research and
Info-Tech
Automation
Service Catalog
Governance
Release Automation
Operations
Service Health
Capacity Optimization
Configuration Standards
IT Business
Cost Transparency
Benchmarking
Service Quality
VMware Cloud Management
4 Compute
Physical
Hardware
Private
Clouds
Public
Clouds
Hybrid Cloud
VMware &
vCloud Datacenter Partners
Virtualized Infrastructure Abstract & Pool
Compute Abstraction
= Server Virtualization
Storage
Storage Abstraction =
Software-Defined
Storage
Network
Network Abstraction
= Virtual Networking
Applications Modern SaaS Traditional
The Control Plane for the Software-Defined Data Center and the Hybrid Cloud
Most complete, innovative management portfolio purpose-built for cloud
• Scalable solutions for dynamic, hybrid cloud environments
• Better automation driven by analytics, policy and vCloud / SDDC integration
• Converged disciplines of performance, capacity and configuration management
Fast time to value
• Easy. Fast deployment and administration
• Unencumbered by huge and complex legacy management frameworks
Best for vSphere, yet open and heterogeneous
• Co-developed with vSphere team with deep insight into platform
• Multi-hypervisor/platform management. Works across hundreds of public cloud
providers
From the trusted market leader
• Virtualization pioneer and Cloud Systems Management market leader*
• The only cloud era management vendor with the vision, scale, portfolio breadth and
track record to deliver comprehensively to the enterprise
1
2
3
* Per IDC based on 2012 Cloud Systems Management market share
4
Why VMware Cloud Management is Better
VMware Cloud Management Leadership
1st in four of six Cloud
Management and Automation
categories:
• Self-service Catalog
• Unified Cloud
Management Console
• Cloud Governance
• Metering and Billing
- 451 Research
1st in Cloud Systems
Management Software
Market Share
- IDC “Champion” in CMP
Vendor Landscape
- Info-Tech
Log Insight Overview
What Could You Do If You Had Insight?
8
Solve problems
faster—
from days to hours vCenter™
Log
Insight™ Find problems you
didn’t know you had
Get actionable
insight into what
it means
Our Vision
Create an intuitive and fast log management platform for cloud operations that delivers proactive
analytics through machine learning
9
vCenter Log Insight 2.0 Overview
Intelligent Operations
• Predictive analytics/Intelligent Grouping for faster problem
resolution
• Faster analytical queries than the leading solution1
• Improved analytical visualizations
Built with vSphere in Mind
• Powerful Log Management for VMware Products
• Support for Vmware products incl. NSX, vCAC, Horizon View
• Built in 2 way alerting with vC Ops
Unified Management
• Open and extensible platform/marketplace for content packs
What’s New
Overview
Delivers the best real-time log management
for VMware environments, across physical,
virtual, and cloud environments.
• Elastic Scale – up to 6 node clusters
• Improved Query Performance – Up to 6X faster than leading
solution
• Improved Data Collection – 30% improvement on ingestion
• Predictive Analytics –Machine Learning based Intelligent
Grouping
• Better visualization – New Analytic Visualizations
• Integration with vC Ops 2-way alert visualization
• Built by VMware Experts – vSphere analytics built in
• Predictable Pricing No surprises on storage costs
New!
Log Insight Technical Overview
Cloud / Data Center
Log
Management
OS
Logs
VC
Logs
App
Logs
System
Stats
Security
Logs
API Syslog
Analyze
• Can analyze any unstructured time-series data,
configuration etc.
• Automatically identifies structures in the data
• No need for ETL or databases
Scale
• Central, scale-out store (no-SQL) for all collected logs
• Configurable retention and archiving
• Maintenance free
Best for vSphere
• Queries, alerts, fields, charts
in the vSphere Content Pack
Log Insight UI - Interactive Log Analytics
Interactive Visualization of
Query Results, Plus Easy
Drop-Down Menu Options
Log Insight Features
• Proactive Analytics
– Significantly reduces manual parsing effort
– Automated data summarization
• Logs are clustered by event type, so rare messages can be easily spotted
– Intelligent schema detection adds structure to unstructured data
– Smart fields to aid in extraction
• Interactive Analytics
– Google-like search query capability
– Filter events on any criteria, on any field
– Flexible event views
– On-the-fly visualizations
13
vCenter Operations and Log Insight Leverage all your IT data for comprehensive visibility in one place
• Intelligent operations through predictive analytics across all machine data
• Policy-based automation enables proactive management and automated remediation
• Unified management for comprehensive visibility in one place, from vSphere to Hyper-V, AWS and physical infrastructure
Structured Data
Metrics Alerts Events
VMware vCenter
Operations
Capacity, Performance and
Configuration Management Events
Launch in Context
Unstructured Data
Logs Messages
VMware vCenter Log
Insight
Log analytics, aggregation,
and search
Public
Cloud
Scale-out Architecture With Ingestion High Availability
Load Balancer (UDP & TCP)
Log messages (Syslog:514 and HTTP:9000)
Log Insight Worker Nodes
Master Node
Web UI/Query (HTTP:80)
Syslog (TCP/UDP:514)
CFAPI (HTTP:9000)
Proactive Analytics
Log Insight Proactively Discovers Structure
Log Insight proactively learns:
from:
Through Machine Learning!
Then you can query it like a database!
Dashboard Enhancements Greatly Improve Effectiveness as Debugging Tools
• Dashboard Filters
– Can now apply a filter to all charts on a dashboard page
– Speeds up investigation of individual entities (hosts) or groups (clusters)
• Dashboard Linking
– Hyperlink from a value in one chart to a dashboard constraint in another
– Log Insight automatically discovers appropriate links
• Both Features Will Significantly Increase Content Pack Capabilities
– Richer workflows
Windows Agent
Collection Framework & API
• REST-based HTTP API
– For sending logs to Log Insight
– For clients to request configuration data
– Has all the benefits of an HTTP API (compression, proxies, easy to use, etc)
• Windows Agent
– VERY SMALL : 3.5MB RAM and 0.4% of 1 CPU core @ 100 events/s
– Deployable via standard Windows management (.msi)
– Monitors Windows Events
– Monitors text file logs (like “tail –f” on Unix/Linux)
Where to Go Next
• Communities
– https://communities.vmware.com/community/vmtn/vcenter/vcenter-log-insight
– Experts: http://loginsight.vmware.com
• Download Link:
– http://www.vmware.com/products/vcenter-log-insight
Questions?
Log Insight is the Right Solution For:
• Gathering, Analyzing, & Searching large amounts of unstructured data
• Real-time troubleshooting that requires quick answers
• VMware environments
– VMware experts have curated troubleshooting dashboards
– Native support for vCOps & vSphere
• Taming a large, complex infrastructure
Log Insight Features
• Deploys as a Virtual Appliance
– Scale-out architecture with high availability
– One deployment of 6 nodes can handle approx 45K events/second
• Microsoft Active Directory support
• Accepts logs over Syslog or RESTful API
– Any syslog agent can forward logs to Log Insight
– Lightweight Windows agent for forwarding logs is available
24
Primary Use Cases
• Troubleshooting and Root Cause Analysis
– Follow the trail from vCOps to logs to get to root cause to an observed problem
– I observed a problem (e.g. slowness), try to troubleshoot the problem and identify the
part of the stack that is responsible (e.g. network delay vs storage)
– Identify the needle in the haystack in real time when troubleshooting a problem
• Monitoring
– Monitor metrics and events (performance & change) that are visible only in logs
– Escalate alerts to on-call staff or via vCOps
– Identify problems proactively, ensure SLAs and comply to IT policies
• Unstructured Data Warehouse
– Collect all the data in one place without the need for custom parsing, transformation
of data
– Get full visibility across all your IT environment from a single place
25
Log Insight Features
• Archiving for long-term data storage
• Super-Powered Dashboards
– Dashboards called Content Packs are curated by experts for top applications & devices
– Add custom dashboards & share them with colleagues
– Beautiful visualizations
– Easily add new data filters on the fly
– Interact between dashboard widgets
• Native support for VMware products
– Simple to configure ESXi hosts
– vCenter Operations Manager interactive capabilities
26
Log Insight + vCenter Operations Manager
27
Automated correlation of performance and log data (Requires vCenter Operations Advanced or Enterprise)
Log Insight + vCenter Operations Manager
28
Launch Log Insight from vCenter Operations in-context
Escalate alerts & events directly to vCenter Operations
Much more on its way!