CIS 191: Linux and Unix

Class 8Oct 28th, 2015

Practical Networking and Cloud Solutions

Final Project

• Raspberry Pi!– Basically a mini computer

• Why?– RPi is fun!– You have to set up a Linux environment from scratch– You will be limited by available computing power– Many online tutorials and resources

Final Project

• Raspberry Pi!• Idea

– Anything you want

• Hardware– Pi 1 by default. If you need Pi 2, request it in advance with a

valid reason– SD card and Wifi adapter– LEDs, resistors, capacitors, breadboards, batteries, cobbler– Sensors (light, pressure, sound, accelerometer etc.) on demand

Final Project

• Proposal due next week• Pi set-up bootcamp

– Tentatively next Saturday + the Monday after that

• Demo in last 191 lecture• You can also demo in the shared 19x project demo

Outline

Cloud Computing Overview

Amazon AWS Services

A Very Brief HTTP Overview

REST APIs

JSON Format

What is Cloud Computing?

• Generally a system of programming on a computer that exists somewhere else

• Possibly (and often!) not even really a computer, but a virtual slot on an existing server

What is Cloud Computing?

• Examples:

What is Cloud Computing?

• Examples:– Amazon AWS Elastic Compute Cloud (EC2)– Google App Engine– Microsoft Azure– Heroku app hosting– Eniac!

Cloud Computing – An Analogy

• Power sources– Everybody used to have their own!– Inconvenient…

Cloud Computing – An Analogy

• Power sources– Everybody used to have their own!– Inconvenient…

• So we scaled and built centralized power plants– Metered usage!– Outsource the problem of building, running, maintaining

generators– Pay-as-you-go model

Why Put Computers in the Cloud?

• Cost effective• Quick prototyping• Little to zero maintenance• Redundancy• Device independence• Easier collaboration

Outline

Cloud Computing Overview

Amazon AWS Services

A Very Brief HTTP Overview

REST APIs

JSON Format

Amazon Web Services

• A number of services– Amazon Elactic Compute Cloud (EC2) – Virtual machines– Amazon Simple Storage Service (S3) – Key-value store– Amazon DynamoDB -- Fast and flexible NoSQL database– Amazon Relational Database System (RDS) – Full virtual

distributed database system of your choosing– Elastic Load Balancing – Balance incoming traffic across EC2

instances– And much much more!

• All of these are free for the 1st year, some are free forever

Amazon AWS Account

• To use Amazon web services, you have to first make an AWS account at aws.amazon.com

• When you do this, you have to input a credit card number they can charge to – don’t worry, you won’t be charged as long as you follow some basic guidelines– And even if you were charged it would probably be a dollar or

two at most

• Many AWS services offer a “free tier” for the first year– This tier tends to be super slow and limited, but good enough

for our purposes

Zooming in: Amazon EC2

• After making an account on Amazon AWS, you can set up your very own EC2 instance– You can even pick the size!

• To do this, you must select “EC2” from the AWS management console, and then select some preferences– I’d suggest selecting the free tier, which is a “micro” instance– Yes it’s as small and limited as it sounds – but that’s fine!– I’d also suggest selecting an ubuntu operating system, since

that’s what we’ve been working with in class

Zooming in: Amazon EC2

• You’ll be asked to download a .pem security key; put this in a safe place

• After downloading the key, you’ll be able to log in to your ec2 instance by running – ssh –i my_key.pem ubuntu@ec2_IP_address

• You can get the IP address by looking at the instance’s details

Setting up EC2

• sudo apt-get install ...• Exactly the same as working in the virtual machine

command line!

Security Groups

• By default, your EC2 will only be accessible by you, over ssh– By using the pem key file and ssh -i!

• If you want to host a website, you’ll need to explicitly allow this in security groups by allowing access by everyone to port 80– This is the default internet port

Aside: Ports

• A port on your computer is like a sub-IP address (sort of)• These allow requesters to target particular applications

or services running on a machine!• So, web server applications tend to be running on port

80 of a larger server

Pricing Issues

• Make sure to run the micro instance if you don’t want to be charged!

• Make sure to terminate your instance when you are finished with it!– Shutting down isn’t the same…– Access this by right clicking on the instance in the console!

• Double check you don’t have any more resources allocated when you are finished using EC2

Data Issues – Don’t Lose It!

• If you store data on the virtual hard disk of an instance and the instance stops running for some reason– You terminate it– It fails (rare but it happens!)

• You will LOSE YOUR DATA!!!• If you want more security, you can use Elastic Block Store

– This is basically data replication across EC2 instances, for an extra fee

– This option will ensure data safety

Where Exactly Is My Instance?

• Who knows?• All you know is the general region in which the instance

runs, which you can request• Generally, you should pick an instance near where your

target audience is (in this case, us-east)– Reduce network latency as much as possible

Outline

Cloud Computing Overview

Amazon AWS Services

A Very Brief HTTP Overview

REST APIs

JSON Format

What is HTTP, anyway?

• We use it every day, but it’s more than just the http:// at the beginning of a web address

• HTTP specifies a protocol for sending data• This is really just an agreement between two machines,

and it allows for a friendly discourse between them

HyperText Transfer Protocol (HTTP)

• First specified in 1990, and it runs on top of TCP/IP– This ensures that we are already assured that data will reach the

target machine, and the packets we send will arrive in-order

• Default machine port is 80 for unsecure HTTP and 443 for secure (with secure socket layer technology – “Secure” basically means RSA encrypted traffic

• The protocol is actually very simple!– Choice: 1.0 (old and dumb) or 1.1 (new and shiny)– Always go for 1.1

GET /index.html HTTP/1.1Host: google.com

Basic “Methods” for HTTP – To Server

• GET– Retrieve contents of a URI

• HEAD– Retrieve only metadata of a URI

• POST– Adds new information– Intended for newsgroup posts; today, used to implement dynamic

content through web forms

• PUT– Store information at a URI

• DELETE– Delete information at a URI

HTTP Parameters

• HTTP can carry key-value pairs to the server!– Very useful for communication between client and server

• This typically look like – key=value

• These pairs can either be in the request URI, or in the request body

• The request body follows all http headers and will generally carry key/value pairs– Sometimes carries more information than this, however! Up to

the server how to interpret it– Useful as a communication standard… More on this soon

Zooming In: GET

• This is what your browser performs when you visit a page!

• You can add GET parameters (which the server can read) right to your URI– http://mysite.com/?myname=Spencer&mypet=Mochi

Zooming in: POST

• This is what your browser sends when you submit a form• For example, when you place an order at Amazon.com• Information can be sent via GET requests as well, but

POST is generally more secure– GET variables are included directly in the URI!– POST information is not stored by the browser or cached by the

server – GET requests are, for faster loading and response time– POST also has no restrictions on data length!– In general, POST when you want to send information to be

stored at a website, especially sensitive data

Zooming in: POST

• Also used to add something new to a page• For example, one might add a blog entry using the POST

request format• Especially useful if you want to post a new blog entry,

and don’t care what the resulting identifier is– I want to add a NEW post

Zooming in: POST

POST /order.html HTTP/1.1Host: www.my-shopper.comContent-Length: 114Content-Type: text/html

item1=6173849&item2=7272384&item3=32904234&password=monkeyfarts&username=doodooface&creditcardno=5555555555555555

Zooming in: PUT

• Used to update information at a URI• Pretty much the same as POST, except that the user

names the resource to be saved– Since you are naming an existing resource, you are updating it!

• Can also do different things depending on context…– But this is what it was intended for!

Zooming in: PUT

PUT /blogentries?id=2012 HTTP/1.1Host: www.my-site.comContent-Length: 35Content-Type:Text.html

Today was a terrible day because…

HTTP Status Codes – From Server

• Reports how the request was processed• Some common codes (way too many to list here)

– 200 OK– 301 Moved Permanently– 401 Unauthorized– 403 Forbidden– 404 Not Found– 500 Internal Server Error

• This last one is just a catch-all

HTTP Headers

• Included underneath an HTTP request or response• Specify more information for your request• These can allow you to set cookies, specify your current

browser, the types of charsets you will accept…

GET /index.html HTTP/1.1Host: 158.130.53.72:80Accept: text/html, application/xhtmlConnection: keep-alive

HTTP Headers

• An important header to remember is the Content-Length header

• This is used when a request (typically post or put) has a body!

HTTP Responses• Sent back to you by the server after you request a page• Include the protocol version and the status code, along with

some headers (additional information) and the content you requested– If you sent a HEAD request, just the headers are returned

HTTP/1.1 200 OKDate: …Server: ……

<html><head><title>…

Media (MIME) Types

• Specify nature of data requested or transferred– Defined in RFC 2046– Included in ‘Content-Type’ header– Includes at least type and subtype

• Examples– text/html– application/xml– text/plain– image/png– image/jpeg– video/quicktime

Outline

Cloud Computing Overview

Amazon AWS Services

A Very Brief HTTP Overview

Networking Tools

JSON Format

Telnet

• You can connect to a service via telnet and play around

telnet www.google.com 80

GET /rotots.txt 80HOST: www.google.com

ssh

• Secure shell via RSA encryption• What if we don’t encrypt the communication?

ssh

• Secure shell via RSA encryption• What if we don’t encrypt the communication?

– Your password, credit card information can be exposed in plaintext on any part of the network!

sshfs: Mounting A Remote Filesystem

• sshfs allows us to treat a remote resource as if it were located locally!

$ mkdir html$ sshfs eniac:public_html/ html$ vim html/index.html…

sshfs: Unmounting a Remote Filesystem

• On Ubuntu– You’ll have to use fusermount –u my_mount_directory

• On OSX– You’ll have to user umount my_mount_directory– If the directory is being stubborn and won’t unmount, then run

umount –f my_mount_directory

• If things are being really stubborn, you can always break out the ban hammer with– killall sshfs

sshfs: Uses

• Why would you want to pretend a remote resource is actually a local resource?– You can use cp instead of the more cumbersome scp

• Everything will be copied over the invisible ssh connection maintained by sshfs!

– You can keep your local editor and bash settings– Graphical programs are run locally on the remote file, rather

than remotely with graphics forwarding (faster)– You can run programs installed on your machine, provided with

data located elsewhere (also faster)

rsync: A File Copying Tool

• Rather than just doing a straightforward copy, rsync is a more powerful and clever copying mechanism

• rsync connects via ssh, just as scp does• However, rsync is smarter about what it copies

– rsync copies file differences rather than files– When copying large folders over the net, this is much faster…

• rsync preserves permissions, symbolic links, and other operating system features

• See man rsync for more details (if you have rsync installed)

nmap: Network Exploration Made Easy!

• The “Network Mapper” nmap explores a network or a host, and looks for “interesting ports”– Based on how a host responds to edge cases in the IP

specification (among other related information), nmap can guess the operating system, which ports are open, and which services are offered

• nmap is useful for examining your own computer– And other people’s computers…

Outline

Cloud Computing Overview

Amazon AWS Services

A Very Brief HTTP Overview

Networking Tools

JSON Format

JavaScript Object Notation

• Used very commonly across the web because of its simplicity!

• Lots of REST APIs return JSON on GET requests– and expect it for POST and PUT requests…

Why JavaScript?

• This format is literally the same format used by JavaScript to represent objects

• It’s very compact and easily readable• Also has the benefit that it can be directly copied and

pasted into a javascript interpreter!

Return of the Dictionary

• JSON Objects are just dictionaries, really…• And they look like dictionaries, too!• Because they are just dictionaries at heart, they can be

interpreted easily by many languages and stored natively• Java, Python, and Ruby all have JSON interpreters, for

example• Some open-source bash projects as well!

– http://kmkeen.com/jshon/ parses JSON into native bash arrays

JSON - Example

GET /users?id=6504 HTTP/1.1Host: www.my-api.com

200 OK{ “name” : “Spencer Lee”, “id” : 6504, “interests” : { “programming”: [“Python”, “Bash”,”JavaScript”], “animals” : [“cats”,”dogs”,”meercats”], “Pet” : “Mochi” } …..}

Example of a useful REST-ful API

• PARSE! • A no-sql datastore that is easy to use, with a full REST API• Learn more at https://parse.com/docs/rest• May be useful for your projects!