SYSTEMS ADMINISTRATION FOR

CODERSHints & tips to increase reliability & reduce

maintenance time.

WHAT IS A SYSTEM?An assemblage or combination of things or parts

forming a complex or unitary whole.

WHAT DOES A SYSTEMS ADMINISTRATOR DO?

• Introduces new technologies into an environment

• Analyses system logs and identifies potential issues with a system.

• Plans and performs routine maintenance

• Performs and maintains backups

• Installs and configures new software and hardware

WHAT DOES A SYSTEMS ADMINISTRATOR DO?

• Manages user accounts

• Responsibility for security

• Responsibility for documentation of the system

• Plans systems upgrades and outages to apply upgrades

• Troubleshooting reported problems

• Deals with, often frustrated, system users...  ... etc. etc.

A COMPUTER SYSTEM• Many components working together - software (application, web server, OS), hardware (disks, RAM, CPU) & others (networking equipment, switches, routers, load balancers)

• Provides stability and maintainability that underpins the entire application.

• Supports your software for its lifetime.

• Can provide parts of your application. Sometimes a problem has already been solved by some other software.

START AT BEGINNING

• Start sysadmin tasks at the beginning of the project.

• Write tools to aid deployment.

• Write tools to set up environments.

• Iterate over these tools and stabilise for production

ENVIRONMENTS

• Dev, QA, Live

• Dev, Test, QA, UAT, Live ~~ Dev, Test, QA, UAT, Staging, Live

• The nearer they get to live, the closer the should resemble live.

• Dev environment should at least be the same major versions, preferably OS version.

• Vagrant is a useful tool for this.

SSH

• Probably the most frequently used tool

• Forwarding SSH agent to allow key use remotely (e.g. git, hopping between servers)

• Tunnels for access to remote resources

• Reverse tunnels for remote access to local resource

• Easy to configure the client

SSH-AGENT

• Generate keys >2048 bits (e.g. ssh-keygen -b 4096)

• ssh-add to load default key (~/.ssh/id_rsa)

• ssh-copy-id <server> to copy to remote server

• ssh -A <server> to forward agent back to local instance.

• Agent runs at login for modern Linux desktop, Mac OS.

SSH-TUNNELS

• Local access to remote: ssh -L3307:localhost:3306 <server>

• Remote access to local: ssh -R:3307:localhost:3306 <server>

• SOCKS proxy: ssh -D5050 <server>

SSH CLIENT CONFIGURATION

• Per user configuration: ~/.ssh/config

• Config options can be set per host or via wild card, e.g. User, ForwardAgent, Hostname & many more

• manpage: ssh_config

UNIX/LINUX PRINCIPLES• Most things in Linux & UNIX are text.

• Each command line tools does one task and does it well.

• Command line tools process text with relative ease.

• Much of the text is separated into fields - especially logs, or as key = value pairs.

• There are standard locations for many types of file.

BASIC TOOLS

• cat - display text

• grep - find text

• awk - field processing (and more)

• sed - search and replace text

• wc - count

• cut - simple field processing

• head, tail - print first and last lines of text

• sort - sort text

LOCATION, LOCATION, LOCATION

• /etc - configuration

• /usr - read-only user data

• /var - variable length files (caches, logs, temporary files)

• /home - users' home directories

• /opt - optional applications

• /srv - served site specific data

• See the Filesystem Hierarchy Standard. Same across most distros

VARIABLE LENGTH FILES

• /var/log - Logs go here

• /var/cache - Cached files

• Watch your permissions

• During normal operation, /usr, /opt should be able to be mounted read only

SOFTWARE DEPLOYMENT

• Use vendor supplied packages whenever possible:

• Reduces risk of misconfigurations

• Easier to seek help

• Usually well tested

• Easier upgrades, timely security fixes

• Building from source will take a fair amount of time, CPU

• Ruby may be an exception. PHP isn't

CHOICE OF LINUX DISTRIBUTION

• Two main camps - Debian and RedHat

• Red Hat Enterprise Linux is rock solid but expensive & packages tend to be older. CentOS is Enterprise Linux recompiled from the same source RPMs.

• Debian stable is rock solid but packages tend to be old. Community/3rd party support only.

• Ubuntu LTS is pretty solid, packages are more recent than EL. Well supported in the Cloud - AWS, OpenStack especially.

SOURCE OF PACKAGES

• Use as stable, well testing packages as much possible

• Ubuntu main, Debian stable ideally

• For EL distros, EPEL augments core packages well

• For EL, IUS provide recent versions of MySQL, PHP but is less well tested.

• Avoid one person repos, PPAs if at all possible.

BUILDING FROM SOURCE

• Do not build on live servers. Deploy only compiled code.

• Ideally produce a package.

• Avoid if possible. Increased risk of problems - more moving parts.

DIAGNOSTICS

• Check disk space: df -h 100% full is bad.

• Check logs: /var/log, /var/log/syslog, /var/log/messages - get to know your logs.

• dmesg for hardware information.

• Check RAM (free -m) and CPU usage with top.

• Install sysstat package early on - sar will gather data. Also gives you iostat, vmstat, mpstat.

SECURITY

• Install denyhosts/fail2ban to help protect SSH.

• Disable SSH in as root, use SSH keys.

• Use host based firewalls, AWS security groups.

• Don’t run your servers as root. Try to split them over different users with clear paths between them. One user nginx, one. php-fpm

• Audit trials are useful.

BACKUPS

• Databases: Dump the DB, don’t take hot copies of the DB files,

• Make use of your hosting providers backup services.

• Make sure you can restore. Test regularly.

PROCESS• Repeat manual tasks often

• Try to use the same deployment system across stages

• Get live up early, treat it as UAT and deploy to it regularly. Avoid 'big bang' deployment

• Use what suits - don't blindly follow trends, assess risks as suits the type of project.

• Small steps, iterative improvement. Agile, Kanban, Lean etc.

AUTOMATION• CFEngine, Puppet, Chef can get you quick wins. They can quickly become hard to manage. Learning curves are steep.

• Ansible is simple to get going on. Can be hacked at and still get good results. Data driven. Pretty new, but growing fast.

• Nothing wrong with shell/Python/Ruby/Perl scripts. Configuration management tools are not essential.

• Packaging gets you out of a lot of automation tasks.

THAT’S A LOT OF STUFF!

• Not touched on DR, monitoring, OS provisioning, storage, networking...

• Hire a sys-admin :)

• A good sys-admin will work with you...

• ...to let you get on with the job you enjoy.

QUESTIONS?

matt@monki.org.uk

THANKS!

matt@monki.org.uk