© Copyright 4/7/2015 BMC Software, Inc1

Robert Stinnett (@robertstinnett)

CARFAX Automation AnalystOctober 14, 2014

We Came, We Saw, We Processed

© Copyright 4/7/2015 BMC Software, Inc2

How #BigData and #Hadoop integrated into @BMCControlM at CARFAXCARFAX helps millions of people buy, sell and service their used cars better.

© Copyright 4/7/2015 BMC Software, Inc3

Agenda

1. Workload Automation at CARFAX

2. Big Data and Hadoop Initiative

3. Reduce, Reuse, Recycle

4. Batches, Unite!

© Copyright 4/7/2015 BMC Software, Inc4

Support traditional batch,

data transfer and SLA/SLE

management across

various datacenters.

Everything under one

roof. Integrate with other

software packages to

create an Enterprise wide

workload management

system.

Capacity on demand.

React to the hyper-

growing business.

Workload Automation at CARFAX

1

Manage Integrate Scale

© Copyright 4/7/2015 BMC Software, Inc5

• 129,000+ processes a day

• 350 batch nodes

• 5 Different Datacenters

• 1 Unified Workload Management Platform

© Copyright 4/7/2015 BMC Software, Inc6

Data can be loaded and

live less than 30 minutes

after receipt

1 record to process today,

1 million records to

process tomorrow

Data is what makes

CARFAX who we are. We

can’t afford to “hope we

got it right”.

It’s All About The Data

#Fast #Dynamic #Reliable

© Copyright 4/7/2015 BMC Software, Inc7

Where Does it Come From?

34,000 Data Sources

13 billion records in our VHDB

Data comes in many formats, even pictures and PDFs!

CARFAX receives data in “any format, any time, any method”. We process more data in an hour than many businesses do in an entire month.

© Copyright 4/7/2015 BMC Software, Inc8

Our business is all about

data. We’ve been doing

Big Data long before it was

“cool”.

It had to integrate with

our currently business

processes. It is a vital part

of our data services and

will provide and consume

data from many other

applications.

We see Hadoop and other

Big Data initiatives

replacing many of our

legacy data processing

systems.

Big Data & Hadoop

2

#BigData #Integration #Future

© Copyright 4/7/2015 BMC Software, Inc9

Hadoop is still evolving and maturing. Thousands of pilot projects out there, but very few production installations.

We learned we were one of the pioneers when it came to integration.

© Copyright 4/7/2015 BMC Software, Inc10

Nobody wanted to

reinvent the wheel. We

didn’t need another

scheduling system.

Hadoop team wanted to

hit he ground running.

Reuse what they were

already familiar with.

Integrate with our existing

DevOps practices across

CARFAX. No silos!

Reuse, Reduce, Recycle

3

#DoMoreWithLess #TimetoMarket #DevOps

© Copyright 4/7/2015 BMC Software, Inc11

Agile Development +

Agile Operations

Increased usage of data center automation and configuration management tools

#DevOps

© Copyright 4/7/2015 BMC Software, Inc12

We’ve managed to free up

our personnel to do

awesome things, and let

the automation handle

the routine stuff.

Integration, more agile

operations, compliance

and remediation. These

are what we see on the

horizon.

This wasn’t a one person,

or one team project. It

was a whole company

initiative. It has been an

amazing journey, yet

we’ve only just begun.

Batches, Unite!

4

#Today #Tomorrow #AmazingJourney

© Copyright 4/7/2015 BMC Software, Inc13

Overall Integration Strategy

Basic Batch

File Transfer Under Control-M Control

Database Processing,Web Services

Bladelogic,Java

Informatica

Start “Run this script”SUMMARY Today “Manage our workloads” Future “Make it all just happen”

20102008 20132004 2012 Beyond

Hadoop,SAS, ServiceNOW,“Write Our Own”

© Copyright 4/7/2015 BMC Software, Inc14

#KeyTakeaways

1. Hadoop isn’t a silo, it’s part of other IT processes.

2. Significantly reduce your learning curve by using what you already have.

3. Integrations reduce management headaches.

4. Evolution from batch to workload

© Copyright 4/7/2015 BMC Software, Inc15

@robertstinnett

robertstinnett@Carfax.com