Date post: | 16-Apr-2017 |
Category: |
Technology |
Upload: | amazon-web-services |
View: | 522 times |
Download: | 0 times |
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Adrian Cockcroft - AWS VP Cloud
Architecture Strategy
December 1, 2016
Open Source at AWSContributions, Support and Engagement
ARC213
What to Expect from the Session
• Examine how leading AWS customers use open source
• See the value of an open source program
• Understand the trends in open source adoption
• Baseline existing contributions from AWS
• Begin the discussion of next steps in engagement
What do customers want?
Capital One use of open source
Capital Onehttps://developer.capitalone.com/open-source/
“We’re an open source first organization —
actively using, contributing to and managing
open source software projects.”
Capital One Open Source Projectshttps://capitalone.github.io
Cloud Custodian – AWS Lambda based policy engine – ARC312
Hygieia – DevOps delivery pipeline dashboard
OAS-nodegen – Open API Specification support
Ops-Pipeline – scripts for creating environments
Hydrograph – User interface for ETL management
Giraffez – Client library for Teradata
React Native PathJS Charts
SWHttpTrafficRecorder – Swift mobile library
CQRS Manager for Distributed Reactive Services
Netflix open source program
From 2012 NetflixOSS Presentation by @adrianco
Current NetflixOSS Programhttp://www.slideshare.net/aspyker/netflix-open-source-building-a-distributed-and-automated-open-source-program
NetflixOSS in 2016http://netflix.github.io
Around 200 projects currently on Github
Deep integration into SpringCloud Java framework
Hystrix circuit-breaker ported and adopted widely
Curator migrated from NetflixOSS to Apache Curator
New: Titus team working with AWS ECS – See CON313
Startups inspired by NetflixOSS:• DynomiteDB supporting and extending Dynomite Redis clustering
• Armory providing Spinnaker deployment pipeline as a service
• Gremlin implementing Chaos Monkey testing as a product
Many Benefits of Open Source Programs
Give back to the communities that you depend on
Attract and curate better developers
Improved code quality, docs, modularity and re-use
Pooled development and testing resources
Validate architectural decisions in public
Enable fast-followers and remain centered in an ecosystem
Build trust and basis for collaboration
Open Source is the Default
Leading customers have their own open source programs
Many AWS products have an open source component
Blackduck Annual Future Of Open Source survey“The survey reports that 78 percent of its respondents are now running their businesses with open
source software, and two-thirds are building software for their customers that’s based on open source
software. More significant, the percentage of respondents actually participating in open source
projects has increased from 50 percent to 64 percent, and 88 percent say they expect to contribute
to projects within the next three years.”
Baseline current status
Memberships
Code contributions
Projects hosted on GitHub
Open source based services
Open Source Memberships
Linux Foundation Silver Member
• Founding advisory member of Xen @ Linux Foundation
• Founding member of the Core Infrastructure Initiative
• Founding member of the Open Container Initiative
ToDo Group - Companies committed to open source
Alliance for Open Media – Video format
Open-MPI - consortium for high performance computing
Amazon & AWS contributions
Bug Fix and Feature Contributions
Linux & Drivers
Xen
Apache Tomcat
PostgreSQL
Docker
Boto
Apache Hadoop
Apache Hive
Apache Bigtop
Apache Oozie
Apache Drill
Apache Zeppelin
Apache Pig
Cloudera HUE
Apache Lucene
Apache Solr
Kuromoji
ElasticSearch
CBMC
MXNet
Moses
Apache Joshua
Highlights for Bug/Feature Contributions
Linux Contributions
• ENA – network driver
• Annapurna – ARM64 chipset support
• AWS recently hired Linux committer David Woodhouse
Boto – Python SDK – AWS hired creator Mitch Garnaat
Hadoop ecosystem projects – over 100 contributions
MXNet – Deep Learning Frameworkhttp://www.allthingsdistributed.com/2016/11/mxnet-default-framework-deep-learning-aws.html
Created at CMU
Major investments by AWS
Scalable, high performance
Supported alongside many other deep learning frameworks
Contributions to developer experience, documentation
Tools for visualization, development, migration
Supporting academic contributions with AWS credits
See MAC401 - Scalable Deep Learning Using MXNet
Repositories Owned by AWS & Amazonhttps://github.com/aws https://github.com/awslabs https://github.com/amznlabs https://blox.github.io/
Blox – Container orchestration for ECS – CON316
s2n – Secure replacement for openssl – NET405
Chalice – Python serverless microframework for AWS – DEV308
AWS-CLI – Built by Mitch Garnaat – DEV402
AWS-shell – Autocomplete based user interface for AWS-CLI
AWS SDKs
AWS Mobile SDKs
Cloud9 Ace – Cloud based interactive development editor
Cfncluster – Build and manage HPC clusters
ION – Amazon data serialization libraries
Blox – Container Orchestration Frameworkhttps://blox.github.io/
New governance model for Blox – built in the open
All development and roadmap in public
Provides a venue for collaboration with AWS customers
Intent to build an ecosystem
Contribute/port schedulers
See more details in CON316
Contributing to Bloxhttps://blox.github.io/
• Blox is licensed under Apache 2.0
• Open an issue or pull request
• Watch our roadmap on GitHub
• Check out our Gitter channel
Open Source Made Easy by AWS Services
Amazon EMR – Elastic Hadoop
Amazon ElastiCache - Memcached
Amazon RDS – MySQL
Amazon RDS – PostgreSQL
Amazon RDS – Scalable Aurora back-end for MySQL & PostgreSQL
AWS OpsWorks – Chef Automate Server
Amazon ECS – Docker Orchestration
Amazon CloudSearch
Amazon Elasticsearch
Let AWS Do The Undifferentiated Heavy Lifting
Amazon EMR updates & patches are frequent and automatic
Engagement and next steps
Program Team
Represents all of Amazon
Manage consumption of projects
Manage engineering contributions
Represent AWS (foundations etc.)
Work with legal on licensing
Assist with standalone projects
Operate GitHub accounts
Did everything to this point
AWS Teams Managing Open Source
New – Outbound Programs Team
Engage with AWS customers
Support academic credit program
Create content: blogs, talks etc.
Represent AWS (foundations etc.)
Present at industry events
Manage example code projects
Measure sentiment/engagement
Starting now!
AWS Cloud Credits
for Research
AWS has been supporting global research projects through the
Cloud Credits for Research program since 2009.
AWS believes that researchers are key drivers of technological innovation
AWS Cloud Credits for Research is aimed at helping researchers move their
research endeavors to the cloud so they can innovate more quickly and at
lower cost
AWS has been supporting global research projects through the
Cloud Credits for Research program since 2009.
We support researchers who seek to:
Build reusable tools to facilitate their future research and the research of
their community
Perform pilot, proof of concept, or benchmark tests evaluating the efficacy
of moving research workloads or open data sets to the cloud
Train a broader community on the usage of cloud for research workloads via
workshops or tutorials
AWS Cloud Credits for Research Recipientshttps://aws.amazon.com/research-credits/2016-recipients/
Categories
Proof of Concept
Development of Open Source Software
Development of a Cloud Service
Open Dataset Development
Benchmark for Comparison
Workshop or Tutorial
AWS Credits for Open Source Projects
Awarded on a case-by-case basis:
Berkeley AMP Lab – Founding Sponsor
Apache Mesos, Spark and other open source tools
Open-MPI Consortium support
PostgreSQL project was given credits to support testing
How can we help?
Adrian Cockcroft
@adrianco
Experience:
Startup Open Source Ecosystems
NetflixOSS
New - Outbound TeamContact us via AWS account management, or in person at #reinvent
Zaheda Bhorat
@zahedab
Experience:
UK Gov Open Document Format
Salesforce Open Source Program
Google Summer of Code
Sun OpenOffice & Netbeans
First steps, we’re listening…
Thank you!
Remember to complete
your evaluations!
Related Sessions
DCS201 - Open-Source Resources and Certifications
BDM401 - Deep Dive: Amazon EMR Best Practices & Design Patterns
BAP305 - Zero to Google Chrome in 60 Minutes: Lightweight and Inexpensive Client Devices for Amazon WorkSpaces
NET405 - Amazon s2n: Cryptography and Open Source at AWS
MAC401 - Scalable Deep Learning Using MXNet
CON316 - State of the Union: Containers
CON313 - Netflix: Container Scheduling, Execution, and Integration with AWS
DAT305 - Deep Dive on Amazon Relational Database Service
DEV308 - Chalice: a Serverless Microframework for Python
DEV402 - The Effective AWS-CLI User