15-Apr-19
1
Software Engineering in the Cloud
Diomidis SpinellisDepartment of Management Science and TechnologyAthens University of Economics and Business
[email protected]@CoolSWEng
1
2
15-Apr-19
2
3
4
15-Apr-19
3
A 1972 Problem
5
6
15-Apr-19
4
Solving the problem in 1990
perl –lane 'for(@F){push@E,"@F";push@F,shift@F}END{print for sort@E}'
Solving the problem in 2019
• The data set IS huge
• Users can be billions
• Hardware advances have not exactly kept pace with data increases
7
8
15-Apr-19
5
9
10
15-Apr-19
6
11
12
15-Apr-19
7
13
14
15-Apr-19
8
15
16
15-Apr-19
9
Cloud computing levels the playing field
17
18
15-Apr-19
10
Overview• Requirements• Design (Cloud components — AWS, Azure)• Software construction (Puppet)• Software testing (Nagios/Netdata)• Software maintenance• Software configuration management• Software engineering management• Software engineering process (Rundeck)• Software engineering methods• Software quality• Software economics
19
20
15-Apr-19
11
Design Strategies• Existing
– Structured (functional) design– OO design– Data structure oriented design– Component based design
• Cloud based design; decide on:– Build / use– Choose abstraction levels (IaaS, PaaS, SaaS)– Components– Requirements to components map
21
22
15-Apr-19
12
23
24
15-Apr-19
13
25
26
15-Apr-19
14
Design Issues• Concurrency must be designed from the beginning
– sharding– replication– …
• Event handling can be outsourced• Data persistence will be on the cloud• Distribution on a planet scale• Fault tolerance, availability zones• Confidentiality and integrity
– trust models, key people– identities and roles (across the stack)– 2FA– risks
• Cloud design patterns
27
28
15-Apr-19
15
Software Construction: System Configuration Management
• System as software• Anticipating change (including vendor)• Reuse• Management (planning, measurement)• Quality• Integration• Performance analysis• Tool support
29
30
15-Apr-19
16
Don’t configure systems by hand
• Modern cloud-based systems are complex
– Many hosts
– Many services
– Many components
• This requires automated configuration
Automate configuration of• Hosts• Available users and roles• Installed software• Running services• Configuration files• Scheduled tasks• Networking• Storage• Monitoring• Security
31
32
15-Apr-19
17
Benefits
• Deploy a new host
• Move to a different environment
• Put configuration under revision control
• Track issues
• Repair misconfigured systems
• Executable documentation
Notable tools
33
34
15-Apr-19
18
Rules• Covering
– Infrastructure• Networking• Storage
– Applications– Services– Scheduled tasks
• Each rule has– Preconditions– Postconditions
Package example
package { 'posfix':
ensure => present
}
35
36
15-Apr-19
19
The Puppet Trifecta
• Package
• File
• Service
File example
file {'/etc/postfix/main.cf':
owner => 'root',
group => 'root',
require => Package["postfix"],
source => 'puppet:///modules/postfix/main.cf'
}
37
38
15-Apr-19
20
Service example
service { 'postfix':
require => [
Package['postfix'],
File['/etc/postfix/main.cf'
],
],
enable => 'true',
ensure => 'running',
}
Building up
• Modules
• Nodes
• Inheritance
• Client-server configuration
39
40
15-Apr-19
21
Example modules• vcsrepo• firewall (iptables)• apache• mysql, postgresql, sqlserver• ntp• java• Tomcat• registry• aws
41
42
15-Apr-19
22
Example module useapache::vhost { 'wsgi.example.com':
port => '80',docroot => '/var/www/pythonapp',wsgi_application_group => '%{GLOBAL}',wsgi_daemon_process => 'wsgi',wsgi_daemon_process_options => {processes => '2',threads => '15',display-name => '%{GROUP}',
},wsgi_import_script => '/var/www/demo.wsgi',wsgi_import_script_options =>{ process-group => 'wsgi', application-group => '%{GLOBAL}' },
wsgi_process_group => 'wsgi',wsgi_script_aliases => { '/' => '/var/www/demo.wsgi' },
}
43
44
15-Apr-19
23
Test Techniques• Software
– Based on software engineer’s experience– Input domain– Code-based– Fault-based– Model-based
• Cloud System– Continuous monitoring– At multiple levels
• Infrastructure• Middleware,• Software
45
46
15-Apr-19
24
47
48
15-Apr-19
25
49
50
15-Apr-19
26
Nagios specifications• Objects
– services– commands
• notifications (email, issue, SMS)• host check• service check
– contacts– hostgroups– templates
• Servers
Server specificationdefine host {
use linux-serverhost_name ldapalias LDAPaddress ldap.example.comcontact_groups adminsicon_image ldap.pngstatusmap_image ldap.gd2
}
51
52
15-Apr-19
27
Service specificationdefine service {
hostgroup_name basichost_name githubservice_description sshcheck_command check_sshuse flappy-service
}
define service {hostgroup_name basicservice_description AMI updatescheck_command check_nrpe!check_amiuse generic-service
}
define service {hostgroup_name basicservice_description EC2 DNS servercheck_command check_nrpe!check_dns_serveruse generic-service
}
Command specificationdefine command{
command_name check_local_diskcommand_line $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$}
define command{command_name check_local_loadcommand_line $USER1$/check_load -w $ARG1$ -c $ARG2$}
define command{command_name check_httpcommand_line $USER1$/check_http -I $HOSTADDRESS$ $ARG1$}
53
54
15-Apr-19
28
NRPE script#!/bin/bash## Check if there are AWS Linux updates#
if grep 'security update' /etc/motd >/dev/nullthen
echo "AMI CRITICAL: `grep 'security update' /etc/motd`"exit 2
fi
if grep 'update' /etc/motd >/dev/nullthen
echo "AMI WARNING: `grep 'update' /etc/motd`"exit 1
fi
echo "AMI OK: Last update: `stat --printf=%z /var/cache/yum/x86_64/latest/amzn-main/packages/`"exit 0
55
56
15-Apr-19
29
Nagios Plugins• Anti-Virus• Backup and Recovery• Business Management and Logic• Cloud• Clustering and High-Availability• CMS and Blog Software• Databases• E-Commerce, Billing and Financial• Email and Groupware• Games• Grid Computing• Hardware• Helpdesk and Ticketing• Instant Messaging• Internet Domains and WHOIS• Inventory Management• Java Applications and Servers• License Management
• Log Files• Network and Systems Management• Network Connections, Stats and Bandwidth• Network Protocols• Notifications• Operating Systems• Others• Printing• Remote Access• Reporting• Security• Software• System Metrics• Telephony• Uncategorized• Videoconferencing• Web 2.0 Services• Web Servers• Websites, Forms and Transactions
57
58
15-Apr-19
30
Software Engineering Process
• Handling of Cloud resources
• Involvement of operations personnel
• Automate operation processes
• Manage through workflow automation
59
60
15-Apr-19
31
Rundeck
• Workflow automation tool
• Operations management
• Operations procedures become self-service jobs
• Control
• Visibility
Basics
• Projects
• Jobs– Groups
• Nodes
• Commands
• Activity
61
62
15-Apr-19
32
63
64
15-Apr-19
33
65
66
15-Apr-19
34
67
68
15-Apr-19
35
Examined Tools
• AWS/Azure/... for cloud components
• Puppet for system configuration management
• Nagios and Netdata for system monitoring
• Rundeck for operations workflow automation
69
70
15-Apr-19
36
Thank you!
www.spinellis.gr
@CoolSWEng
github.com/dspinellis
Image Credits• Guiding light (Node length)• Rajiv.Pant derivative work: Wylve (DevOps)• PiccoloNamek (Cirrus clouds)• Visitor7 (Data center)• Diglinks (Nagios)• DotNetTricks (Azure Services)• PP3 in the Slovnaft oil refinery Mikulova CC BY-SA 3.0• Funnel by Donovan Govan. CC BY-SA 3.0
71
72
