Assurer – A server testing/monitoring framework

Assurer – A server testing/monitoring framework

Gosuke Miyashitahttp://mizzy.org/

MyselfMyself

Known as Chocoboy Miyashita somewhere

Working in paperboy&co.Mainly developed public/private web servic

e APIs with perl and catalyt... but now also woking on servers and net

works and so on Today’s my son’s entrance ceremony of a

n elementary school

What’s Assurer ?What’s Assurer ?

A server testing/monitoring framework like Plagger Plugin mechanism YAML configurable assets Test by Test::Base

Originally developed for one time test of servers that are newly built

Monitoring seems like testing, so decided to cover monitoring

Where the name Assurer derived from ?Where the name Assurer derived from ?

Pronounce “A-shu-ra”Plagger like tools ended with “er”

Archer by tokuhirom Observer by Hatena (not in public) Dishuber by Yappo Precure by Kan Fushihara

Testing is also called “Quality Assurance”

MISS

Image of exec Assurer #0Image of exec Assurer #0

Image of exec Assurer #1Image of exec Assurer #1

Image of Asura BasterImage of Asura Baster

Image of the real AsuraImage of the real Asura

Image of fake AsuraImage of fake Asura

Execution phases of AssurerExecution phases of Assurer

Execution phases of Assurer Execution phases of Assurer

Test phase Execute test

Nofity phase Notify test results

Format phase Format test results

Publish phase Output formatted test results

Relation of each phasesRelation of each phases

Test

PublishFormat

Nofity

Details of each phasesDetails of each phases

Test phaseTest phase

Test phaseTest phase

Execute testPlugins on this phase:

Test::HTTP Test::SMTP Test::DBI Test::Ping etc

Config of test phaseConfig of test phase

test: - module: HTTP config: host: www.mizzy.org content: It works!

Notify phaseNotify phase

Notify フェーズNotify フェーズ

Notify test resultsPlugin on this phase:

Notify::IRC

Config of notify phaseConfig of notify phase

notify: - module: IRC config: daemon_port: 9991 nickname: assurerbot server_host: chat.freenode.net server_port: 6667 server_channels: - #assurer - #assurer-jp charset: iso-2022-jp announce: notice

Notify::IRC exampleNotify::IRC example

Format phaseFormat phase

Format phaseFormat phase

Format test resultsPlugins on this phase:

Format::Text Fomat::HTML

Config of format phaseConfig of format phase

format: - module: HTML config: css: /css/style.css

Publish phasePublish phase

Publish phasePublish phase

Output formatted resultsPlugins on this phase:

Publish::Term Publish::Mail Publish::File

Config of publish phaseConfig of publish phase

publish: - module: Mail config: subject: Test results from Assurer

to: someone@example.com

from: root@example.com

publish: - module: Mail config: subject: Test results from Assurer

to: someone@example.com

from: root@example.com

Format::Text + Publish::TermFormat::Text + Publish::Term

Format::HTML + Publish::FileFormat::HTML + Publish::File

Format::Text + Publish::MailFormat::Text + Publish::Mail

Summary for nowSummary for now

Assurer is a serverPlagger Test, Notify, Format and Publish phasesEasily change tests and outputs with plugi

ns

Extra functionsExtra functions

Hosts and RolesHosts and Roles

Hosts and rolesHosts and roles

Write target hosts and roles in one place on config.yaml

Config without hosts and rolesConfig without hosts and roles

test: - module: HTTP config: host: www0.mizzy.org

- module: HTTP config: host: www1.mizzy.org

Config with hosts and rolesConfig with hosts and roles

test: - module: HTTP role: web

hosts: web: - www0.mizzy.org - www1.mizzy.org

FilterFilter

FilterFilter

Filtering test results on format and notify phase

Filtering formatted results on publish phase

Filter on format phaseFilter on format phase

Only OK results are formatted as text format

format: - module: Text filter: module: Status status: ok

Filter on notify phaseFilter on notify phase

Only NOK results are notified to IRC

notify: - module: IRC filter: module: Status status: not ok

Filter on publish phaseFilter on publish phase

Text formatted results are displayed on a terminal,html formatted results are sent as a mail

format: - module: Text - module: HTMLpusblish: - module: Term filter: module: Type type: text/plain - module: Mail filter: module: Type type: text/html

Config validationConfig validation

Config validationConfig validation

Kwalify – YAML schema validator http://www.kuwata-lab.com/kwalify/

Validate common configValidate config of each pluginKwalify schemas for each plugin put under

assets/kwalify/plugins

Mistake on common configMistake on common config

tests: # test is right - module: Test::HTTP

$ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] - [/] Expected required key `test‘ - [/tests] Unexpected key `tests' at line 46

Mistake on plugin configMistake on plugin config

test: - module: HTTP config: contents: XXX # content is right

$ ./assurer.pl -c config.yaml Assurer::ConfigLoader [fatal] Config error in Test::HTTP - [/contents] Unexpected key `contents' at line 66

Shell modeShell mode

Shell modeShell mode

Execute shell command on hosts in config.yaml

Shell modeShell mode

$ ./assurer.pl --shellassurer> uptime[www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users,

load average: 0.24, 0.16, 0.17[svn.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users,

load average: 0.24, 0.16, 0.17[ftp.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users,

load average: 0.24, 0.16, 0.17

Exec on hosts belong to specified roleExec on hosts belong to specified role

$ ./assurer.pl --shell --role=webassurer> uptime[www.mizzy.org] 21:48:24 up 5 days, 23:08, 2 users, load average: 0.24, 0.16, 0.17

Special command !onSpecial command !on

assurer> !on app1.foo.com \ app2.foo.com do uptime# Exec on app1.foo.com and app2.foo.com only

assurer> !on /.*\.foo\.com/ do \ uptime# Exec on hosts matched .*\.foo\.com only

Special command !withSpecial command !with

assurer> !with web db do uptime# exec on hosts belong to web role and db role

assurer> !with /web|mail/ do uptime# exec on hosts belong to role match web or mail

Special command !testSpecial command !test

assurer> !test HTTPAssurer::Plugin::Test::HTTP [info] Testing HTTP on www0.mizzy.org ...

ok 1 - HTTP status code of http://mizzy.org:80 is 200

not ok 2 – Content of http://mizzy.org:80 matches 'It works!‘

assurer>

Special command !testSpecial command !test

assurer> !test HTTP on app1.foo.com # test http on app1.foo.comassurer> !test HTTP on /.*.foo.com/ # can use regexpassurer> !test HTTP with web # test http on hosts belong to web roleassurer> !test HTTP with /web|mail/# can use regexp

Parallel testingParallel testing

Pallarel testingPallarel testing

$ assurer.pl --para=20

Multiple test processes on test phaseDefault is 5Can change number processes working at

once

Distributed testingDistributed testing

Distributed testingDistributed testing

localhost

test0.mizzy.org www0.mizzy.org

www1.mizzy.org

www2.mizzy.org

test1.mizzy.org

test2.mizzy.org

1. ssh assurer_test.pl

3. Return result

2. Exec test

Config for distributed testingConfig for distributed testing

exec_on: - host: test0.mizzy.org priority: 3 - host: test1.mizzy.org priority: 2 - host: test2.mizzy.org priority: 1

Auto config generationAuto config generation

Auto config generationAuto config generation

# assurer.pl --discover -c template.yaml

Scan hosts in template.yaml by Nmap::Scanner

Detect services working on the hosts and generete config.yaml automatically

config templates for each plugin put under assets/discover (http.yaml, mysql.yaml, etc)

Integragte Test::WWW::MechanizeIntegragte Test::WWW::Mechanize

Integrate Test::WWW::MechanizeIntegrate Test::WWW::Mechanize

Can write tests using Test::WWW::Mechanize

Put .pl files under assets/plugins/Test-WWW-Mechanize

Pre-set $mech, $host on execution

asset file for Test::WWW::Mechanizeasset file for Test::WWW::Mechanize

# $host and $mech are set by Assure core

$mech->get_ok("http://$host", "got htttp://$host");

$mech->content_contains('It works!', "Content matches 'It works!'");

Inside AssurerInside Assurer

Assurer::Test moduleAssurer::Test module

Assurer::Test moduleAssurer::Test module

Export Test::More like methodsis,() like(), ok()Output test results formatted as TAP (Test

Analyze Protocol)

Inside test pluginsInside test plugins

Inside test pluginsInside test plugins

package Assurer::Plugin::Test::HTTP;use base qw( Assurer::Plugin::Test );use Assurer::Test;

sub register { my $self = shift; $self->register_tests( qw/ status content server / );}

sub status { my ( $self, $context, $args ) = @_; ... 中略 is( $res->code, '200', 'HTTP status code of $self->{url} is 200' );}

Internal mechanism in test phaseInternal mechanism in test phase

Internal mechanism in test phaseInternal mechanism in test phase

assurer.pl Assurer::bootstrap()

Assurer::Dispatch::run()

call

exec directly or by ssh

exec tests

return testresults

call

assurer_test.plassurer_test.pl

assurer_test.plassurer_test.pl

assurer_test.pl

Assurer::ResultAssurer::Result

Assurer::ResultAssurer::Result

Assurer::Result

generate resultobjects

next phase

Assurer::Dispatch::run()Assurer::Dispatch::run()

Exec assurer_test.pl for each test Pallarelize by POE::Wheel::RunNumber of processed controlled by POE::

Component::JobQueuePass config for plugin and context object s

erialized by YAML and encoded by Base64

assurer_test.plassurer_test.pl

assurer_test.pl -–config=LS0tCmNvbmZpZzoKICBjb250ZW50OiBJdCB3b3JrcyEKICBob3N0OiB3d3cubWl6enkub3JnCm1vZHVsZTogSFRUUApuYW1lOiBIVFRQIHRlc3QKcm9sZTogd2ViCg==--context=LS0tICEhcGVybC9oYXNoOkFzc3VyZXIKYmFzZV9kaXI6IC9ob21lL21peWEvc3ZrL0Fzc3VyZXIKY29uZmlnOgogIGZvcm1hdDoKICAgIC0gY29uZmlnOiAmMSB7fQogICAgICBtb2R1bGU6IFRleHQKICBnbG9iYWw6CiAgICBob3N0OiB+CiAgICBpbnRlcnZhbDogMwogICAgbG9nOgogICAgICBsZXZlbDogZGVidWcKICAgIG5vX2RpYWc6IDAKICAgIHJldHJ5OiAzCiAgaG9zdHM6CiAgICBmdHA6CiAgICAgIC0gZnRwLm1penp5Lm9yZwogICAgc3ZuOgogICAgICAtIHN2bi5taXp6eS5vcmcKICAgIHdlYjoKICAgICAgLSB3d3cubWl6enkub3JnCiAgICAgIC0gc3ZuLm1penp5Lm9yZwogICAgICAtIHRyYWMubWl6enkub3JnCiAgbm90aWZ5OgogICAgLSBjb25maWc6CiAgICAgICAgYW5ub3VuY2U6IG5vdGljZQogICAgICAgIGNoYXJzZXQ6IGlzby0yMDIyLWpwCiAgICAgICAgZGFlbW9uX3BvcnQ6IDk5OTEKICAgICAgICBuaWNrbmFtZTogYXNzdXJlcmJvdAog ... cont ...

Why POE::Wheel::Run ?Why POE::Wheel::Run ?

Would like to exec tests as other processes Need non-blocking I/O with single process an

d POE Can easily distribute to remote servers (exec

directly on local, exec by ssh on remote ）Communication with test processes

Can communicate easily with test processes by STDIN, STDOUT and STDERR

RoadmapRoadmap

RoadmapRoadmap

Integrate with Nagios/NRPE (Test::Nagios)

Enhance notify phaseStore Plugin (store test results in databas

e)Integrate with Archer, puppet (share host li

sts)Test of Assurer itsself

Web site and IRC channelsWeb site and IRC channels

Web sites and IRC channelsWeb sites and IRC channels

http://assurer.jp/trachttp://mizzy.org/#assurer@chat.freenode.net#assurer-ja@chat.freenode.net

ThanksThanks