DevOps or: How I Learned to Stop Worrying and Love the Cloud

transcript

o r :How IL e a r n e dto S to pWorrying

a n dLove the

DevOps@nxhack 2010/12/31

INTRODUCTION TO

http://www.flickr.com/photos/aereimilitariorg/3953794205/

@nxhack

Background:　25年以上情報処理業界　17年間イソターネット業界 Bias:　文系エソジニア　運用な人　*NIXな人

伝えたい事: DevOps 運動と何ぞや

対象

WebアプリケーションWebサービスの事業者

スタートアップで成功急成長し、組織作りが必要になった方...

技術系の組織運営にお困りの方...

最近、リリース遅延が常態化しつつある方...

スピード感がなくなってきて、活気が無くなってきたなと思っている方...

衝撃の事実

http://www.etsy.com/

http://twitter.com/allspaw/status/22742505774

11デプロイ/日

http://twitter.com/allspaw/status/29580411400

15 デプロイ/日

DevOpsのキーワード•WebOps•Agile•Cloud•Realistic-ITIL

Web Operations

http://www.flickr.com/photos/cote/54408562/

TraditionalSystems Operations

http://www.flickr.com/photos/arthur-caranta/2926332140

Web Operations

Web AppsWeb Services

Shrink WrapXhttp://www.flickr.com/photos/cogdog/116957071/

The infrastructure is the application.

The application is the infrastructure.

http://omniti.com/seeds/dissecting-todays-internet-traffic-spikes

WebOps

WebアプリケーションWebサービスの事業者によくある風景…

http://www.flickr.com/photos/johnjoh/368511463/

Startupほんの少しだけ起業論

ScalableGrow fast

ResearchDevelopment

DeploymentRelease

CustomerFeedback

ContinuallyPivot

(Iterate)

Hypotheses Testing

PDCA cycleX

http://www.flickr.com/photos/sdasmarchives/4727993474/

Colonel John (Richard) Boyd (January 23, 1927 – March 9, 1997) was a United States Air Force fighter pilot and Pentagon consultant of the late 20th century, whose theories have been highly influential in the military, sports, and business.

OODA Loop

Continuous DevelopmentContinuous Deployment

ContinuousDelivery

http://www.amazon.co.jp/dp/0321601912/

ContinuousDelivery

STARTUP CTO’s MISSION

Care about everything...

•Research...•Development...•Operation...(Server, Apps, Storage, Network...)•Facility...(DC, Server, Storage, Network...)•Customer Support...

•everything...

http://www.flickr.com/photos/power_on/5092140218/

COWBOY

http://www.flickr.com/photos/foolswisdom/43145184/

RockStar

A HARD DAY'S NIGHT

http://www.flickr.com/photos/ceonyc/3346786138

順調に成長

規模が大きくなる

仕事が回らない

組織作り

IPO準備

会社の転機

http://www.flickr.com/photos/power_on/5092140218/

COWBOYX

http://www.flickr.com/photos/foolswisdom/43145184/

RockStarX

Transition

What VC’s Don’t Tell You:The Transition ‒ Founders Leave

http://steveblank.com/

http://www.flickr.com/photos/krmnclara/4859716369/

IPO準備事業継続性リスク排除

属人化排除内部統制内部牽制機構

http://www.flickr.com/photos/bpt/321626320/

GOVERNANCE

MANAGEMENT

http://www.flickr.com/photos/deemonita/4043301909/

官僚機構は規模が大きくなれば必要だし合理的。内部統制は官僚機構の統治には必須。

しかし

目的と手段を取り違える

頭の悪い人が多いという罠

残念な人が多いという罠

Organizational Chart

Board CEO

COO CIOCTO CFO

Dev Ops

職務分掌規定

Devの役割Opsの役割

http://www.flickr.com/photos/aaronjacobs/64368770/

Developer

FeaturesEvolution

http://www.flickr.com/photos/aaronjacobs/64368770/

やったぜ!新機能ビルド成功♪

リリースにギリ間に合った!ふぅねむねむ～

http://www.flickr.com/photos/troyholden/4036820768/

Operator

Services Stability

安全・安定

ダメダメ!もっと早く仕様を渡さんかい!

運用さん

http://www.flickr.com/photos/greg-palmer/3490903215/

Ops focuson failure

hardware failure

http://www.flickr.com/photos/purplemattfish/3760706750/

hardware failure

ぐはっ!故障

hardware failure

Avoid SPOF

hardware failure

Avoid SPF

冗長設備の投資対効果を定量的に示してください。

bursting

ヤフーに貼られたテレビで紹介された

bursting

Capacity Planning

http://www.amazon.co.jp/dp/0596518579

bursting

Capacity Planning

http://www.amazon.co.jp/dp/0596518579

性能設計の根拠と投資対効果を定量的に示してく

ださい。

software failure

http://www.flickr.com/groups/webopsviz/pool/with/4819476138/

software failure

テスト環境では問題なかったぞ

software failure

本番環境と同じテスト環境本番負荷テスト

software failure

Testing????Testing?テスト環境構築の

投資対効果を定量的に示してください。

operation mistake

http://www.flickr.com/photos/proimos/4199675334/

operation mistake

指示書・手順書うがぁ失敗!

operation mistake

http://www.amazon.co.jp/dp/4122018331http://www.amazon.co.jp/dp/0691004129

指示書・手順書

operation mistake

http://www.amazon.co.jp/dp/4122018331http://www.amazon.co.jp/dp/0691004129

実はこれが一番多い

ContinuousDelivery

CHANGE CAUSES OUTAGES

http://www.flickr.com/photos/adjourned/308053363/

http://www.flickr.com/photos/sis/124916325/

http://www.flickr.com/photos/carve/2856365431/

Dev Ops

リリースするよ

無理!詳細仕様渡せ

Dev Ops

障害発生!情報くれー

おーーーーい

しーん...

Dev Ops

リリースじゃぽいっ! むむむむ!!

Dev Ops

Wall of Confusion

Dev Ops

Wall of Confusion

俺達が価値を守っているんだ

俺達が価値を作っているんだ

奴等はスピード感に欠ける・遅すぎ!

奴等はサービスが分かっていない

http://www.flickr.com/photos/rocketjim54/2955889085/

アンタのせいだ!

I'm afraidI can't do that.

I'm sorry, DEVs.

Dev vs. Ops

WebOps

AgileCloud

ちょっと待てよ

残念な人が多いという罠

我々の目的って

Dev & Ops

Goal:To Enable

Our Business

We make Our Servicestogether

To Enable

ContinuousDelivery

じゃぁどうする

Developer

Manifesto for Agile Software Development

We are uncovering better ways of developingsoftware by doing it and helping others do it.Through this work we have come to value:

Individuals and interactions over processes and toolsWorking software over comprehensive documentation

Customer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items onthe right, we value the items on the left more.

http://agilemanifesto.org/

agile and iterative

ディーン・レフィングウェル (著), 玉川憲 (監修, 翻訳)

DevelopersProduct Owners

Testers

DevelopersProduct Owners

Testersたいへんよくできました

OperatorsSystem AdminsDB Admins

Network EngineersDesigners

OperatorsSystem AdminsDB Admins

Network EngineersDesigners

がんばりましょう

どうすんのよ

Operator

Infrastructure:BootstrappingConfigurationMonitoring

agile operations

agile operations?

Infrastructure is HARD

http://www.flickr.com/photos/johnseb/3425464/

Infrastructure is HARD

http://www.flickr.com/photos/johnseb/3425464/

agile Infrastructure

IaaSInfrastructure as a Services

Server: EC2Storage: S3, EBSNetwork: ELB, CloudFrontMonitoring: CloudWatchDataBase: RDS, SimpleDB

Pooled IT Resources

On DemandSelf ServicePay as you goFair pricing

% ec2-run-instances ami-b232d0db -n 3 --availability-zone us-east-1a! ! ! !RESERVATION r-385c5950 012301230123 defaultINSTANCE i-5bca5a30 ami-b232d0db pending 0 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtualINSTANCE i-59ca5a32 ami-b232d0db pending 1 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtualINSTANCE i-5fca5a34 ami-b232d0db pending 2 m1.small 2010-04-07T12:25:47+0000 us-east-1a aki-94c527fd ari-96c527ff monitoring-disabled ebs paravirtual

サーバー 3台用意してね☆

Infrastructure as Code

API! API! API! API! API!Infrastructure as Code! Infrastructure as Code! Infrastructure as Code!

http://www.flickr.com/photos/programmerman/2315566040/

Give me an API, or give me DEATH!

http://www.flickr.com/photos/bitchcakes/3895840169/

Bootstrapping:Infrastructure as a ServicesInfrastructure as Code

Realistic-ITIL

agile Infrastructure

を使いこなす

http://www.flickr.com/photos/robenjoyce/4208638969/

Tools!

DevOpsTool Chainshttp://code.google.com/p/devops-toolchain/

Release

Generalized architecture

Control

Orchestration

Dispatcher

Provisioning

Deploy

Config management

OS boot/install

Artifact repository

CI Server Issue tracker

SCM Repository

Asset inventory

Host naming

Identity

CMDBMonitoring

Events

Trending Reporting

Trending

Workflows

Resources

Topology

Configuration

Sources

Scheduler

Wednesday, June 23, 2010

http://code.google.com/p/devops-toolchain/

Release management

Artifact

Trackerbugzillamantistrac

svncvsgithg

Repository

yum/yastartifactoryarchivaDFS/openEFSHTTTP

eggsgemsperlmod

tgz/ziprpm/deb/pkg/msijar/war/ear

makemavenant/ivy

sbtrakephing

Monitoring and control

Measurement Instrumentation

Charting, History, SPC

Alarm Mgmt

Op Console, Control

Runbook Automation

nagioszenossopennms

rrdtoolcacti

ControlTierJobschedulerOpenScheduler

Provisioning

Bcfg2cfengineChefPuppetSmart Frog

Application Service Orchestration

System Configuration

Cloud or VMImage

LaunchOS

Install

KickstartJumpstartCobblerOpenQRMxCAT

XenlxcopenVZEucalyptusKVM

CapistranoControlTierFabricFuncmCollective

Cloud: OS Install:

Command and Control

Configuration

Bootstrapping

自動化できうるところは全て自動化する

agile Infrastructure を活用しインフラ構築・構成を含めて

自動化する

Configuration:Configuration Management

agile Infrastructure を活用するよう System Architecture を見直す

System Architecture変えないといけない

従来の考え方から脱却新しい考え方が必要

障害対応

http://www.flickr.com/photos/greg-palmer/3490903215/

MTBFMean Time Between FailureX

Failure as a Service

Failure is a failurebut

embrace failure

Your sitewill fail!

http://www.flickr.com/photos/gaelx/2912634902/

MTTDMean Time To Detect

MTTRMean Time To Resolve

hardware failure

壊れたら捨てる瞬時に入れ替え

bursting

必要なだけ起動する不要になれば落とす

software failure

本番環境と同じテスト環境

operation failure

自動化

Monitoring:Measurement

VisualizeShare

http://blog.b3k.us/ooda.html

WebOps OODA LoopDiagram

ActDecideOrientObserve

ExecutionPlanningResourcing

AnalysisVisualizationCorrelation

MetricsMonitoringAlertingAlarming

Benjamin Black, b@b3k.us

Visible DevVisible Ops

http://www.amazon.co.jp/dp/443407296X/

THE VISIBLE OPS HANDBOOK - 見える運用

Kevin Behr (著), Gene Kim (著), George Spafford (著)

官野厚訳出版社: ブイツーソリューション (2006/1/6)

Web OperationsKeeping the Data On Time

John Allspaw, Jesse Robbins

Publisher:O'Reilly MediaReleased:June 2010

Opsの役割

Operator

Services Stability

次のステップへ

http://www.flickr.com/photos/alexthomp/3545005410/

Operator

System AvailabilityEfficiency

Dev ♥ Ops

To EnableOur Businesstogether

Culture

http://www.flickr.com/photos/24443965@N08/3285794855/

PeopleProcessTools

http://www.flickr.com/photos/pmtorrone/23362393/

DevOpsCultural and

Professional Movement

Question

http://www.flickr.com/photos/statelibraryqueensland/4903167275/

Appendix:John Allspaw (twitter) (blog)・SlideShare・10+ Deploys Per Day: Dev and Ops Cooperation at Flickr　　(どのプレゼンも素晴らしいですが、このスライドは必ず見てください)

Andrew Clay Shafer (twitter) (blog)・SlideShare・DevOps ‒ You’re Doing IT Wrong « fate = will && choice || circumstance

Adam Jacob (twitter)・SlideShare・Velocity 2010 ‒ Adam Jacob on DevOps・Choose Your Own Adventure OSCON 2010 Edition (PDF)

Appendix:John M Wills (twitter)・SlideShare・What Devops Means to Me

Damon Edwards (twitter) (blog)・DevOps is not a technology problem. DevOps is a business problem.

PodCast:・DevOps Cafe・IT Management Podcast

Conference / UnConference:・DevOps DAYS・OpsCamp・O’REILLY Velocity

Appendix:日本語での情報:・すべての道は DevOps へと : DevOps ‒ mizzy.org ‒ Trac・DevOps: OpenSource Provisioning Toolchain : DevOps ‒ mizzy.org ‒ Trac・DevOpsって何？

・クラウドと「DevOps」を理解する ‒ builder by ZDNet Japan

My Blog:・すでにここにある雲 DevOps

このスライド:・http://www.slideshare.net/nxhack

FGD 135Peace IS OUR PROFESSION

DevOps or: How I Learned to Stop Worrying and Love the Cloud

Technology