Date post: | 30-May-2018 |
Category: |
Documents |
Upload: | ari-lerner |
View: | 217 times |
Download: | 0 times |
of 55
8/14/2019 Cloud.la PoolParty Presentation
1/55
Part 1: HadoopA demo about setting up ahadoop cluster with PoolParty
8/14/2019 Cloud.la PoolParty Presentation
2/55
Service oriented
deploymentI want to deploy a service
(http/mail/etc.)
I dont care about the platformI want it to work
I want to do it a million timesI want it now
8/14/2019 Cloud.la PoolParty Presentation
3/55
Self managing
8/14/2019 Cloud.la PoolParty Presentation
4/55
GruntworkWhy?
8/14/2019 Cloud.la PoolParty Presentation
5/55
8/14/2019 Cloud.la PoolParty Presentation
6/55
Lets be
Cutting-edge
IntuitiveTool-driven
Lazy
8/14/2019 Cloud.la PoolParty Presentation
7/55
8/14/2019 Cloud.la PoolParty Presentation
8/55
ToolsBecause we are human
8/14/2019 Cloud.la PoolParty Presentation
9/55
8/14/2019 Cloud.la PoolParty Presentation
10/55
8/14/2019 Cloud.la PoolParty Presentation
11/55
Big frontier
Shell scripts
CapistranoPackage managers (apt, yum,tpkg)
ChefPuppet
lots of settlements
8/14/2019 Cloud.la PoolParty Presentation
12/55
8/14/2019 Cloud.la PoolParty Presentation
13/55
PoolPartyEnjoyable cloud infrastructure
8/14/2019 Cloud.la PoolParty Presentation
14/55
8/14/2019 Cloud.la PoolParty Presentation
15/55
8/14/2019 Cloud.la PoolParty Presentation
16/55
8/14/2019 Cloud.la PoolParty Presentation
17/55
8/14/2019 Cloud.la PoolParty Presentation
18/55
8/14/2019 Cloud.la PoolParty Presentation
19/55
8/14/2019 Cloud.la PoolParty Presentation
20/55
8/14/2019 Cloud.la PoolParty Presentation
21/55
8/14/2019 Cloud.la PoolParty Presentation
22/55
Demo
8/14/2019 Cloud.la PoolParty Presentation
23/55
Part 2
Discussion of distributed algorithms, theHermes project and nosql
Distributed Algorithms
8/14/2019 Cloud.la PoolParty Presentation
24/55
What?
A distributed algorithm is analgorithm designed to run oncomputer hardware constructedfrom interconnected processors.
(Wikipedia)
8/14/2019 Cloud.la PoolParty Presentation
25/55
Why?
Because scale is becomingincreasingly important
Datacenters are becomingaccessible
Commodity hardware is cheapNetwork is cheaper
8/14/2019 Cloud.la PoolParty Presentation
26/55
8/14/2019 Cloud.la PoolParty Presentation
27/55
When?
Now
8/14/2019 Cloud.la PoolParty Presentation
28/55
8/14/2019 Cloud.la PoolParty Presentation
29/55
8/14/2019 Cloud.la PoolParty Presentation
30/55
Assorted types
MapReduce
Atomic CommitConsensusMutual exclusion (distributed
mutex)
Distributed search
8/14/2019 Cloud.la PoolParty Presentation
31/55
Why its easy
Math is fun
8/14/2019 Cloud.la PoolParty Presentation
32/55
Why its hard
Account for failure
Unsafe networksData sharding
Job distribution
8/14/2019 Cloud.la PoolParty Presentation
33/55
8/14/2019 Cloud.la PoolParty Presentation
34/55
8/14/2019 Cloud.la PoolParty Presentation
35/55
Decentralization
assumptionsNodes are prone to failure
Nodes are homogenous*The dataset is largeNodes are cheap (easy to
add/remove)
Network is unowned
8/14/2019 Cloud.la PoolParty Presentation
36/55
Scale
Greater utilization of hardware
InexpensiveCooperative application space
And its green
8/14/2019 Cloud.la PoolParty Presentation
37/55
8/14/2019 Cloud.la PoolParty Presentation
38/55
NoSql
Scaling relational databases is noteasy and seriously no fun at allKey/Value stores are easier to
scale
8/14/2019 Cloud.la PoolParty Presentation
39/55
NoSql
BigTable (column orienteddatabase)
CassandraVoldemortScalaris
8/14/2019 Cloud.la PoolParty Presentation
40/55
Paxos
An algorithm for decidingconsensus within a network of
unreliable nodes.
Transaction layer
Atomic commitsStrong data consistency
8/14/2019 Cloud.la PoolParty Presentation
41/55
Paxos
Devised by Leslie Lamport in 1990Published in 1998Based on viewsource replication
(published 2 years earlier)
8/14/2019 Cloud.la PoolParty Presentation
42/55
Big names
Googles Chubby (and BigTable)IBM San Volume Controller
Microsoft
8/14/2019 Cloud.la PoolParty Presentation
43/55
What?
8/14/2019 Cloud.la PoolParty Presentation
44/55
8/14/2019 Cloud.la PoolParty Presentation
45/55
What? (contd)
8/14/2019 Cloud.la PoolParty Presentation
46/55
What? (contd)
8/14/2019 Cloud.la PoolParty Presentation
47/55
What? (contd)
8/14/2019 Cloud.la PoolParty Presentation
48/55
What? (contd)
8/14/2019 Cloud.la PoolParty Presentation
49/55
What? (contd)
8/14/2019 Cloud.la PoolParty Presentation
50/55
8/14/2019 Cloud.la PoolParty Presentation
51/55
HermesOpen-source internode communicationproject
8/14/2019 Cloud.la PoolParty Presentation
52/55
What
Erlang-y
Consensus algorithmsDistributed mutex
Mapping/Reduction
8/14/2019 Cloud.la PoolParty Presentation
53/55
Where? (almost)http://github.com/auser/hermes/tree/master
https://github.com/auser/hermes/tree/masterhttps://github.com/auser/hermes/tree/master8/14/2019 Cloud.la PoolParty Presentation
54/55
mailto:[email protected]:[email protected]8/14/2019 Cloud.la PoolParty Presentation
55/55
Thanks
Ari Lerner
AT&T CloudTeam
And all the various funny imagesources
irc.freenode.net/#poolpartyrbYou