Date post: | 10-Jan-2017 |
Category: |
Technology |
Upload: | bdup |
View: | 1,589 times |
Download: | 0 times |
● root DevOops, automation @
● Distributed systems & Data
● Go - Python - Cobol
● Machine Learning
Who’s this guy ?
m6web.kafkam6web.php
m6web.ntpdm6web.spark
[...]
playbooks - ops
ansible-vault
control-masterServers
ElasticSearch / Kibana
Global Architecture
Roles
«This is the Unix philosophy: Write programs that do one thing and do it well.
Write programs to work together […]»Doug McIlroy
Roles
● Ex: ELK should *NOT* be a single role○ java○ elasticsearch○ logstash○ kibana○ nginx○ open-ssl○ ssl-certificates
Roles
● One role = One public repository
● ~ 60 repos
● Code review / Pull Requests
● Continuous integration
- name: Install python-software-properties apt: name=python-software-properties state=installed update_cache=yes cache_valid_time=86400
Roles - Code conventions and readability
- name: Install dependencies
apt:
name: “ {{ item }} “
state: installed
update_cache: yes
cache_valid_time: “{{ global_cache_valid_time }}”
with_items: “ {{ kafka_dependencies }}”
Roles - Code conventions and readability
Roles - Code conventions and readability
[ANSIBLE0004] Git checkouts must contain explicit versionexample.yml:36Task/Handler: using git module
[ANSIBLE0003] mismatched bracesexample.yml:2debug oops a missing {{bracket}
[ANSIBLE0002] Trailing whitespaceexample.yml:19 action: do nothing
Roles - Code conventions and readability
λ ~ role-skeleton (master|✔) ls -1defaultsfileshandlersmetataskstemplatestestsvarsREADME.md
Roles - Code conventions and readability---# tasks file for m6web.kafka
- name: Check if broker_id is set fail: msg: "You must set kafka_broker_id variable !" when: kafka_broker_id is not defined
- include: users.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::users'
- include: packages.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::packages'
- include: config.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::config'
- include: service.yml tags: - 'role::m6web.kafka' - 'role::m6web.kafka::service'
Roles - Code conventions and readability
Roles - Tags
// Change ntpd config on *all* servers
$ ansible-playbook -i inventory playbooks/all.yml \
--tags ‘role::m6web.ntpd::config
// Change kafka config for dev
$ ansible-playbook -i inventory playbooks/dc1.yml \
--tags ‘role::m6web.kafka::config’ \
--limit lyon-dev
Roles - Tests
pre-commit (by Yelp)
Ansible-lint
YAML syntax
Python jinja2 syntax
Playbook syntax
Idempotence
ServerSpec
One shell script run in
Roles - Tests - ServerSpec
describe package('cassandra') do it { should be_installed }end
describe service('cassandra') do it { should be_running }end
describe command('nodetool status') do its(:stdout) { should contain('UN 127.0.0.1') }end
Roles - Tests - ServerSpec
ports = [7199, 9160, 7000, 9042, 61621]
ports.each do |p| describe host('127.0.0.1') do it { should be_reachable.with( :port => p, :proto => 'tcp' ) } endend
● Real orchestration
● Glue between roles, inventories, variables, and secrets
● Ops Playbook
● Packer / Vagrant Playbook
● Soon to come Docker Playbooks
Playbooks
● Groups of hosts (clusters, logical roles)
● Single Server (aptly, jenkins-master etc…)
● One shot (upgrade bash, upgrade openssl)
● Utilities (install_sysdig, upgrade_system)
Playbooks
● Ansible SSH private key
● SSL private key
● Nginx passwords
● Unix passwords
● AWS / GCE Access keys
Secrets - What ?
● Private git repository
● ansible-vault encrypt - ansible-vault decrypt
● pre-commit hook prevent clear text to be commited
● Control Master has vault unsealed (manually)
Secrets - How ?
$ ./get-package-version
--inventory ../inventory
--hosts all
--package "zsh" | jq "."
{
"down": [],
"detailled": {
"5.0.5-4": ["srv-x1"],
"5.0.2-3": ["srv-z1","srv-z2"],
"4.3.10-5": ["srv-y1","srv-y2"]
},
"summary": {
"5.0.5-4": 1,
"5.0.2-3": 2,
"4.3.10-5": 2
}
}
Scripting Ansible (not 2.0)
Scripting Ansible (not 2.0)
import ansible.runner
from collections import defaultdict
import json
import argparse
Scripting Ansible (not 2.0)runner = ansible.runner.Runner(
module_name='shell',
module_args=' '.join(["dpkg-query",
"-f=\'${Version}\' -W",
args['package']]),
pattern=args['hosts'],
remote_user=args['user'],
inventory=inventory,
forks=10
)
res = runner.run()
Scripting Ansible (not 2.0)
for host in sorted(res['contacted'].keys()):
version = res['contacted'][host]['stdout']
cnt['summary'][version] += 1
cnt['detailled'][version].append(host)
print json.dumps(cnt)
$ ./get-package-version
--inventory ../inventory
--hosts all
--package "zsh" | jq "."
{
"down": [],
"detailled": {
"5.0.5-4": ["srv-x1"],
"5.0.2-3": ["srv-z1","srv-z2"],
"4.3.10-5": ["srv-y1","srv-y2"]
},
"summary": {
"5.0.5-4": 1,
"5.0.2-3": 2,
"4.3.10-5": 2
}
}
Scripting Ansible (not 2.0)
Scripting Ansible (not 2.0)
● How many servers with kernel 3.13.0-43 ?
● How many VMs > 12G ram ? with 2 vCPUS ?
● Which server needs reboot ?
● Which server needs security updates ?
● List all servers with uptime less than X minutes
● Index all ohai facts in Elasticsearch
● Every night, run a --check --diff
○ Did we silently changed something today ?
Scripting Ansible (not 2.0)
+ + ++
● Simple but powerful orchestrator over SSH
● Agent / Server Less
● Dynamic inventories
● Unix Philosophy
● Python <3 <3 <3
● Community
What we <3
● No curated roles or playbook
○ simple != manageable
○ steep learning curve (best practices + team work)
● No emphasis on tests (roles / playbook)
● Conditionnals can be tricky
● Ansible-vault
What we <3 (less)
● v2 (core + API)
● Goss: ServerSpec in Go
● Pew or conda VS virtualenv
● Ansible runner + Celery = <3 (some caveats)
● HashiCorp Vault
Dig !