Date post: | 16-Apr-2017 |
Category: |
Technology |
Upload: | philipbalinov |
View: | 237 times |
Download: | 1 times |
Cloud-Based Architecture and Big Data
Philip Balinov,
DevOps Engineer
SOME DEFINITIONS
►def. CLOUD COMPUTING
l - distributed computing over a network
l
- the ability to run a program or application on many connected computers at
the same time.
►def. BIG DATA
l
data sets so large and complex that it becomes difficult to process using
traditional data processing applications
Retail
Finance
E-Commerce
Telecommunication
B2B
Publishing/Media
Government & NGO
Automotive
Travel
KOMFO'S CLIENTS
KOMFO PLATFORM WORKFLOW OVERVIEW
EXTERNAL PROVIDERS
MODERN COMPANY STRUCTURE
►Agile software development: Scrum, Kanban
► Short product feature lifecycle
► Continuous delivery
► Self-organizing teams
► Cross-functional teams
AGILE DEVELOPMENT, CONTD.
►Scrum
► Excellent for product development
► Short, iterative cycles (sprints), commit to deadlines
► Self-organizing teams
►Kanban
► Developed at Toyota to optimize supply-chain
► Good for ad-hoc tasks, e.g. support, bugfixing
AGILE DEVELOPMENT, CONTD.
►Agile development requires agile operations
► Underlying architecture must scale together with the product
► Continuous integration
► Quality assurance
► Deployment
►The solution: de-couple development and operations
► DevOps
USE CASE
►Fast and dependable communication with external providers
►On-demand resource scaling
►Flexibility
►Storage, indexing & analysis of huge amounts of data
►Dependability
CLOUD CONCEPTS
►Three main service models
► IaaS – EC2, Rackspace, Azure, HP, Oracle
► PaaS – Google App Engine, Heroku
► SaaS – Gmail, Wordpress.com, Salesforce, Office 365
CLOUD CONCEPTS, CONTD
►Four deployment models
► Public
► Private
► Community
► Hybrid
CLOUD INTERNALS
SERVERS
STORAGE
EXTERNAL SERVICES
(CDN, NETWORKING, SaaS)
VMs VMs
PROS AND CONS
Dedicated Cloud
+ simplicity
+ performance
+ predictability
+ tried and tested
+ agility
+ ease of use
+ scalability
+ stability
- inflexible
- ineffective
- nightmare in case of disaster
- vendor lock-in
- blackboxed
- fluctuations
- helpless in case of disaster
CLOUD ARCHITECTURE
►Application servers scale up and down based on load
►Application software written for parallelism
►Communication between nodes via messaging service
►Write for eventual consistency
OK, so we have an (endlessly) scalable cloud app
now.
But didn't we just create a bottleneck elsewhere?
MIX & MATCH
►Crunch numbers in the cloud
► Application servers
► Slow running tasks
► Temporary services
► Test servers
MIX & MATCH, CONTD.
►Traditional servers for:
► Incompatible apps (single-threaded, memory, disk intensive, specialized
hardware) do not work well in cloud environments
► Database servers are best kept on dedicated machines in our experience
DATABASES, NOSQL
►def. NoSQL
l
a mechanism for storage and retrieval of data that is modeled in means other
than the tabular relations used in relational databases.
DATABASES
• Use the best tool for the job depending on the task
• NoSQL Advantages
► Social networks generate a lot of data
► Complex interconnections, cyclical dependencies
► Aggregations must be performed on both new and old data
► Structure of foreign sources may change on short notice
DATABASES, NOSQL CONTD
►Riak, Hadoop, MongoDB, Cassandra, Redis
►In-memory dataset for faster operation
►No predefined structure
►Integrated sharding, load-balancing and failover
►Versatility - can be used for anything from data storage to real-time messaging to search
indexes
DATABASES, LONG TERM STRATEGY
►Data quickly becomes irrelevant
►Archive it, but still be accessible
►Online Data Warehouse solutions
► Amazon Redshift
► Keep Everything
► Terabytes for pennies
WE ARE LOOKING FOR…
Senior Web Developer
Junior Web Developer
Junior QA
DevOps Engineer
Questions?