David  Andrzejewski  -­‐  @davidandrzej  Data  Sciences  Engineering,  Sumo  Logic  OC  Big  Data  Meetup,  September  17,  2014      

Mining  human-­‐scale  insights  from  log  data  with  machine  learning  

   

   

4  

Logs  

The  Problem  We  Solve  

Human Generated

Machine Generated

Orders, Blogs, Social Networks, HR, Inventory, Manufacturing

Networks, Servers, Hypervisors

Security Devices, Desktops

Web Servers, Email

Applications, Mobile

Clickstream

Machine Data is the largest, fastest growing, most complex segment of Big Data.

2003 2005 2007 2009 2011 2013 2015

“More Logs Are Created In A Single Day Now Than in All of FY 2003,” Gartner

Sumo  Logic  

6  

“Turning Machine Data Into IT and Business Insights”

Learn, classify, predict

Search, monitor, visualize

Use Cases

Availability & Performance

Customer Insights

Security and Compliance

7  

8  

Monitoring  and  reporOng  

TroubleshooOng  and  root  cause  analysis  

9  

Custom App Code

Server / OS

Virtualization

Databases

Network

Open Source Software

Middleware

12/20/2011 17:23:44 PST [user=234fsf] failed transaction, sessionid:2F0A232324, [host=pay002.sjc] amount=1725.00

12/20/11 17:23:34 AMQ7163: WebSphere MQ job number 18429 started FOR client_session=2F0A232324.

12202011 17:23:27 /usr/local/build/mysql/libexec/mysqld: Abnormal shutdown [18429]

20-12-2011 17:23:19 database-host login[3866]: DEAD_PROCESS: 18429 ttys000

Dec 20, 2011 17:22:14,,, message=Created virtual machine user-3 on esxi01.office.thedomain.com

<134>Dec 20 2011 17:22:12: %PIX-6-106100: access-list inside_access_out denied tcp inside/68.162.72.163(4326) -> outside/45.200.244.124(3127) hit-cnt 1(first hit)

66.249.67.24 - - [20/Dec/2011:17:23:40 -0700] ”POST /APP/Order.php HTTP/1.1" 304 146 "-" SESSION=2F0A232324

Customer ID Session ID

Job number

Process ID

Root cause!

Anatomy  of  a  log  message:  Five  W’s  

10  

Anatomy  of  a  log  message:  Five  W’s  

11  

! When?  Timestamp  with  Ome  zone  

Anatomy  of  a  log  message:  Five  W’s  

12  

! When?  Timestamp  with  Ome  zone  ! Where?  Host,  module,  code  locaOon  

Anatomy  of  a  log  message:  Five  W’s  

13  

! When?  Timestamp  with  Ome  zone  ! Where?  Host,  module,  code  locaOon  ! Who?  AuthenOcaOon  context  

Anatomy  of  a  log  message:  Five  W’s  

14  

! When?  Timestamp  with  Ome  zone  ! Where?  Host,  module,  code  locaOon  ! Who?  AuthenOcaOon  context  ! What?  Log  level  and  key-­‐value  pairs  

! Logs:  like  “computer  tweets”  ! TwiZer  2013*    

• Peak  @  ~144k  TPS    • Avg  ~6k  tweets  /  second  

! Log  data  • Example:  1  TB  /  day    • Avg  ~25k  logs  /  second  

Inhuman  scale  

15  

* https://blog.twitter.com/2013/new-tweets-per-second-record-and-how

“A  distributed  system  is  one  in  which  the  failure  of  a  computer  you  didn't  even  know  existed  can  render  your  own  computer  unusable.”  -­‐  Leslie  Lamport    

Inhuman  complexity  

16  

Transport for London December 2013

Key to symbols Explanation of zones

1

3

45

6

2

789

Station in both zones

Station in both zones

Station in both zones

Station in Zone 9

Station in Zone 6

Station in Zone 5

Station in Zone 3Station in Zone 2

Station in Zone 1

Station in Zone 4

Station in Zone 8

Station in Zone 7

National Rail

Riverboat services

Airport

Tramlink

Interchange stations

Step-free access from street to platform

Step-free access from street to train

Emirates Air Line

Check before you travel

Key to lines

Metropolitan

Victoria

Circle

Central

Bakerloo

DLR

London Overground

Piccadilly

Waterloo & City

Jubilee

Hammersmith & City

Northern

DistrictDistrict open weekends, public holidays and some Olympia events

Emirates Air Line

Bank Waterloo & City line open between Bank and Waterloo 0621-0030 Mondays to Fridays and 0802-0030 Saturdays. Between Waterloo and Bank 0615-0030 Mondays to Fridays and 0800-0030 Saturdays. Closed Sundays and Public Holidays.---------------------------------------------------------------------------------Camden Town Sunday 1300-1730 open for interchange and exit only.---------------------------------------------------------------------------------Canary Wharf Step-free interchange between Underground, Canary Wharf DLR and Heron Quays DLR stations at street level.---------------------------------------------------------------------------------Cannon Street Open until 2100 Mondays to Fridays and 0730-1930 Saturdays. Closed Sundays.---------------------------------------------------------------------------------Embankment Bakerloo and Northern line trains will not stop at this station from early January 2014 until early November 2014. ---------------------------------------------------------------------------------Emirates Greenwich Peninsula and Emirates Royal DocksSpecial fares apply. Open 0700-2000 Mondays to Fridays, 0800-2000 Saturdays, 0900-2000 Sundays and 0800-2000 Public Holidays. Opening hours are extended by one hour in the evening after 1 April 2014 and may be extended on certain events days. Please check close to the time of travel. ---------------------------------------------------------------------------------Heron Quays Step-free interchange between Heron Quays and Canary Wharf Underground station at street level.---------------------------------------------------------------------------------Hounslow WestStep-free access for manual wheelchairs only.---------------------------------------------------------------------------------Kilburn No step-free access from late January 2014 until mid May 2014.---------------------------------------------------------------------------------Stanmore Step-free access via a steep ramp. ---------------------------------------------------------------------------------Turnham Green Served by Piccadilly line trains until 0650 Mondays to Saturdays, 0745 Sundays and after 2230 every evening. At other times use District line.---------------------------------------------------------------------------------Waterloo Waterloo & City line open between Bank and Waterloo 0621-0030 Mondays to Fridays and 0802-0030 Saturdays. Between Waterloo and Bank 0615-0030 Mondays to Fridays and 0800-0030 Saturdays. Closed Sundays and Public Holidays.No step-free access from late January 2014 until late July 2014.---------------------------------------------------------------------------------West India QuayNot served by DLR trains from Bank towards Lewisham before 2100 on Mondays to Fridays.---------------------------------------------------------------------------------

River Thames

A

B

C

D

E

F

1 2 3 4 5 6 7 8 9

1 2 3 4 5 76 8 9

A

B

C

D

E

F

2 2

22

2

5

8 8 6

2

4

4

65

41

3

2

43

3

36 3 1

1

3

3

59 7 7Special fares apply

5

5

4

4

4

AmershamChorleywood

Mill Hill East

Rickmansworth

Perivale

KentishTown West

CamdenRoad

Dalston Kingsland

Wanstead Park

Vauxhall

Hanger Lane

Edgware

Burnt Oak

Colindale

Hendon Central

Brent Cross

Golders Green

WestSilvertown

EmiratesRoyal Docks

EmiratesGreenwichPeninsula Pontoon Dock

LondonCity Airport

WoolwichArsenal

King George V

Hampstead

Belsize Park

Chalk Farm

Chalfont &Latimer

Chesham

New CrossGate

Moor Park

NorthwoodNorthwoodHills

Pinner

North Harrow

Custom House for ExCeL

Prince Regent

Royal Albert

Beckton Park

Cyprus

GallionsReach

Beckton

Watford

Croxley

Fulham Broadway

LambethNorth

HeathrowTerminal 4

Harrow-on-the-Hill

KensalRise

BethnalGreen

Westferry

SevenSisters

Blackwall

BrondesburyPark

HampsteadHeath

HarringayGreen Lanes

LeytonstoneHigh Road

LeytonMidland Road

HackneyCentral

NorthwickPark

PrestonRoad

RoyalVictoria

WembleyPark

Rayners Lane

Watford High Street

RuislipGardens

South Ruislip

Greenford

Northolt

South Harrow

Sudbury Hill

Sudbury Town

Alperton

Pimlico

Park Royal

North Ealing

Acton Central

South Acton

Ealing Broadway

Watford Junction

West Ruislip

Bushey

Carpenders Park

Hatch End

North Wembley

West Brompton

Ealing Common

South Kenton

Kenton

Wembley Central

Kensal Green

Queen’s Park

Gunnersbury

Kew Gardens

Richmond

Stockwell

Bow Church

Stonebridge Park

Harlesden

Camden Town

Willesden Junction

Headstone Lane

Parsons Green

Putney Bridge

East Putney

Southfields

Wimbledon Park

Wimbledon

Island Gardens

Greenwich

Deptford Bridge

South Quay

Crossharbour

Mudchute

Heron Quays

West India Quay

Elverson Road

Oakwood

Cockfosters

Southgate

Arnos Grove

Bounds Green

Theydon Bois

Epping

Debden

Loughton

Buckhurst Hill

WalthamstowQueen’s Road

Woodgrange Park

Leytonstone

Leyton

Wood Green

Turnpike Lane

Manor House

Stanmore

Canons Park

Queensbury

Kingsbury

High Barnet

Totteridge & Whetstone

Woodside Park

West Finchley

Finchley CentralWoodford

South Woodford

Snaresbrook

Hainault

Fairlop

Barkingside

Newbury Park

East Finchley

Highgate

Archway

Devons Road

Langdon Park

All Saints

Tufnell Park

Kentish Town

Neasden

Dollis Hill

Willesden Green

South Tottenham

Swiss Cottage

ImperialWharf

Brixton

Kilburn

West Hampstead

Blackhorse Road

Acton Town

CanningTown

Finchley Road

Highbury &Islington

Canary Wharf

Stratford

StratfordInternational

FinsburyPark

Elephant & Castle

Stepney Green

Barking

East Ham

Plaistow

Upton Park

Poplar

West Ham

Upper Holloway

PuddingMill Lane

Kennington

Borough

Elm ParkDagenham

East

DagenhamHeathway

Becontree

Upney

Heathrow Terminal 5

Finchley Road& Frognal

CrouchHill

Northfields

Boston Manor

South Ealing

Osterley

Hounslow Central

Hounslow East

Clapham North

Clapham High Street

Oval

Clapham Common

Clapham South

Balham

Tooting Bec

Tooting Broadway

Colliers Wood

South Wimbledon

Arsenal

Holloway Road

Caledonian Road

Morden

West Croydon

HounslowWest

Hatton Cross

HeathrowTerminals 1, 2, 3

ClaphamJunction

WestHarrow

Brondesbury CaledonianRoad &

Barnsbury

TottenhamHale

WalthamstowCentral

HackneyWick

Homerton

WestActon

Limehouse EastIndia

Crystal Palace

ChiswickPark

RodingValley

GrangeHill

Chigwell

Redbridge

GantsHill

Wanstead

Ickenham

TurnhamGreen

Uxbridge

Hillingdon Ruislip

GospelOak

Mile End

Bow Road

Bromley-by-Bow

Upminster

Upminster Bridge

Hornchurch

Norwood Junction

Sydenham

Forest Hill

Anerley

Penge West

Honor Oak Park

Brockley

Harrow &Wealdstone

Cutty Sark for Maritime Greenwich

Ruislip Manor

Eastcote

Wapping

New Cross

Queens RoadPeckham

Peckham Rye

Denmark Hill

Surrey Quays

Whitechapel

Lewisham

Kilburn Park

Regent’s Park

KilburnHigh Road

EdgwareRoad

SouthHampstead

GoodgeStreet

Shepherd’s BushMarket

Goldhawk Road

Hammersmith

Bayswater

Warren Street

Aldgate

Euston

Farringdon

BarbicanRussellSquare

Kensington(Olympia)

MorningtonCrescent

High StreetKensington

Old Street

St. John’s Wood

Green Park

BakerStreet

NottingHill Gate

Victoria

AldgateEast

Blackfriars

Mansion House

Temple

Cannon Street

OxfordCircus

BondStreet

TowerHill

Westminster

PiccadillyCircus

CharingCross

Holborn

Tower Gateway

Monument

Moorgate

Leicester Square

London Bridge

St. Paul’s

Hyde Park Corner

Knightsbridge

StamfordBrook

RavenscourtPark

WestKensington

NorthActon

HollandPark

Marylebone

Angel

Queensway MarbleArch

SouthKensington

SloaneSquare

WandsworthRoad

Covent Garden

LiverpoolStreet

GreatPortland

Street

Bank

EastActon

ChanceryLane

LancasterGate

Warwick AvenueMaida Vale

Fenchurch Street

Paddington

BaronsCourt

GloucesterRoad St. James’s

Park

Latimer RoadLadbroke Grove

Royal Oak

Westbourne Park

Bermondsey

Rotherhithe

ShoreditchHigh Street

Dalston Junction

Haggerston

Hoxton

Wood Lane

Shepherd’sBush

WhiteCity

King’s CrossSt. Pancras

EustonSquareEdgware

Road

Southwark

Embankment

Stratford High Street

Abbey Road

Star Lane

Waterloo

TottenhamCourt Road

Canonbury

Shadwell

Earl’sCourt

NorthGreenwich

CanadaWater

All-­‐too-­‐human  messiness  and  variety  

17  

! (wildly)  varying  formats    • prind,  JSON,  XML,  Windows,  X-­‐delimited,  ...  

! Specialized  knowledge  

   

[2008-05-07 09:50:08.450 'App' 3560 verbose] [VpxdHeartbeat] Invalid heartbeat from 10.17.218.46

Q:  how  to  get  human-­‐scale  insights  from  log  data?  

18  

Q:  how  to  get  human-­‐scale  insights  from  log  data?  

19  

A: machine learning (and friends) !   Unsupervised pattern discovery !   Anomaly / outlier detection !   Supervised classification !   Time-series data modeling !   Graph analysis !   Probabilistic data structures

Too  many  logs!  “data  disorientaOon”  

   

~60k results: 30 minutes, one component

21  

Unsupervised clustering !   Given: set of items !   Do: group similar items

22  

Unsupervised clustering !   Given: set of items !   Do: group similar items

DisOll  logs  down  to  underlying  structure  

Results  "compressed”  ~1000x    

   

printf("Health status check: %s is %s”,

hostid, hoststatus)

Health status check: zim-5 is OK

Health status check: gir-3 is OK

Health status check: gir-2 is TIMED OUT

Health status check: dib-1 is OK

In  the  beginning,  there  was  the  prind()  

   

Log generation

printf("Health status check: %s is %s”,

hostid, hoststatus)

Health status check: zim-5 is OK

Health status check: gir-3 is OK

Health status check: gir-2 is TIMED OUT

Health status check: dib-1 is OK

Health status check: *** is ***

Reverse  engineering  prind()  

       

Log generation

“magic”

27  

1.  Define string distance function

2.  Do distance-based clustering

���������

��������

Unsupervised clustering !   Given: log messages !   Do: group by “signature”

Drill-­‐down  into  the  original  raw  logs  

   

29  

Partially supervised clustering !   Given: set of items + side info !   Do: group similar items

30  

Partially supervised clustering !   Given: set of items + side info !   Do: group similar items

Too  many  wildcards!  

31  

“Hint”  from  human  user  

32  

Not  enough  wildcards!  

33  

“Hint”  from  human  user  

34  

unknown  unknowns  

35  

36  

Outlier detection !   Given: data points !   Do: identify outliers

37  

Outlier detection !   Given: data points !   Do: identify outliers

38  

Health check OK

Request processed

Txn timeout, retry

Anomaly detection !   Given: log data !   Do: flag anomalies

39  

Health check OK

Request processed

Txn timeout, retry

Anomaly detection !   Given: log data !   Do: flag anomalies

InvesOgate  and  annotate  events  

40  

logs

signatures

RAW DATA

HUMAN

InvesOgate  and  annotate  events  

41  

logs

signatures

RAW DATA

HUMAN

InvesOgate  and  annotate  events  

42  

logs

signatures

event

RAW DATA

HUMAN

InvesOgate  and  annotate  events  

43  

logs

signatures

event

RAW DATA

HUMAN

timeline / alerts

44  

Supervised classification !   Given: labeled data points !   Do: predict future labels

45  

Supervised classification !   Given: labeled data points !   Do: predict future labels

46  

Supervised classification !   Given: log data,

annotated events !   Do: classify new

occurrences

event

timeline / alerts

User  action  webID=7F92  Initiating  requestID=082A  for  webID=7F92  …    …  orderID=34C8  received  for  requestID=082A  …  Retrieving  userID=11D2  for  requestID=082A  …  …  accountID=1234  access,  userID=11D2  …  ERROR  accountID=1234  not  found!    PROCESSING  FAILED:  webID=79F92  

Connected components !   Given: nodes/edges !   Do: identify component

User  action  webID=7F92  

User  action  webID=7F92  Initiating  requestID=082A  for  webID=7F92  …  

User  action  webID=7F92  Initiating  requestID=082A  for  webID=7F92  …    …  orderID=34C8  received  for  requestID=082A  …  

User  action  webID=7F92  Initiating  requestID=082A  for  webID=7F92  …    …  orderID=34C8  received  for  requestID=082A  …  Retrieving  userID=11D2  for  requestID=082A  …  

User  action  webID=7F92  Initiating  requestID=082A  for  webID=7F92  …    …  orderID=34C8  received  for  requestID=082A  …  Retrieving  userID=11D2  for  requestID=082A  …  …  accountID=1234  access,  userID=11D2  …  

User  action  webID=7F92  Initiating  requestID=082A  for  webID=7F92  …    …  orderID=34C8  received  for  requestID=082A  …  Retrieving  userID=11D2  for  requestID=082A  …  …  accountID=1234  access,  userID=11D2  …  ERROR  accountID=1234  not  found!    PROCESSING  FAILED:  webID=79F92  

Time-series detection !   Given: time-series metric data !   Do: identify unusual data pts

Time-series detection !   Given: time-series metric data !   Do: identify unusual data pts

Level change

Time-series detection !   Given: time-series metric data !   Do: identify unusual data pts

Level change Spikes

“Bollinger  bands”  –  rolling  window  approach  

µ± 3σ

58  

Top-K identification !   Given: stream of observations !   Do: identify k most frequent

(WITH FIXED MEMORY!)

59  

Top-K identification !   Given: stream of observations !   Do: identify k most frequent

(WITH FIXED MEMORY!)

...

60  

Top-K identification !   Given: stream of observations !   Do: identify k most frequent

(WITH FIXED MEMORY!)

... 4 3 2 2

61  

Top-K identification !   Given: stream of observations !   Do: identify k most frequent

(WITH FIXED MEMORY!)

... 4 3 2 2

TOP  2  

62  

Top-K identification !   Given: stream of observations !   Do: identify k most frequent

(WITH FIXED MEMORY!)

Count-Min Sketch (Cormode & Muthukrishnan, 2003)

63  

Top-K identification !   Given: stream of observations !   Do: identify k most frequent

(WITH FIXED MEMORY!)

Count-Min Sketch (Cormode & Muthukrishnan, 2003)

64  

Top-K identification !   Given: stream of observations !   Do: identify k most frequent

(WITH FIXED MEMORY!)

Count-Min Sketch (Cormode & Muthukrishnan, 2003)

65  

Cardinality estimation !   Given: stream of observations !   Do: identify number of distinct

items (WITH FIXED MEMORY!)

66  

Cardinality estimation !   Given: stream of observations !   Do: identify number of distinct

items (WITH FIXED MEMORY!)

...

67  

Cardinality estimation !   Given: stream of observations !   Do: identify number of distinct

items (WITH FIXED MEMORY!)

...

|{ , , , }| = 4

68  

Cardinality estimation !   Given: stream of observations !   Do: identify number of distinct

items (WITH FIXED MEMORY!)

HyperLogLog (Flajolet et al, 2007)

Hooray!  Monoid  homomorphism!  

69  

logs

logs

logs

f(s1 + s2) = f(s1)⊕ f(s2)

<  FINAL  OBLIGATORY  PLUG  >  

70  

freesumo.com