Date post: | 25-Dec-2014 |
Category: |
Technology |
Upload: | jose-luis-martinez |
View: | 382 times |
Download: | 3 times |
Building an AWS SDK
Granada Perl Workshop 2014
Jose Luis Martinez
JLMARTIN on CPAN
Twitter: @pplu_io
About me
CTO @CAPSiDE
Training Partner for AWS
Delivering AWS official training
All four AWS certifications
Real experience with real platforms
Deliver consulting and continuous operation
Writing Perl for +10 years
And still having fun with it!
About AWS
Mostly known for EC2 and S3
But lot's more services! (+30)
Everything is an API
“Programmable datacenter”
Software Defined Everything
About AWS
Mostly known for EC2 and S3 But lot's more services! (+30) Everything is an API “Programmable datacenter”
Software Defined Everything
It’s the your wet dream
Perl & AWS
Bad news
No official SDK
Official ones: Ruby, PHP, Python, JS, .Net, Java
Note: python wasn't official initially!
Perl & AWS
Good news
CPAN
And an active community
Lots of modules for different services (sometimes more than one)
Perl & AWS: The CPAN
Inconsistencies
In method naming
Arbitrary defaults ('eu-west-1', 'us-east-1')
Not up to date with latest APIs
Keeping up with AWS is hard!!!
“Not so popular” services not covered
IAM, CloudFormation, SWF, RedShift, CloudSearch, etc
Almost no Role support
AWS::CLIWrapper is a good generic solution. Not native
So lets write an entire SDK!
Hard enough
You can end up with…
or
Challenges: # of services
30+ services
Some with 100+ API calls
Each call with it’s parameters
Want to return objects
Evolution
Challenges: # of services
So lets start classifying
Challenges: Endpoints
Endpoint Type Services
Regional Services available in a región
the rest
Single Global services 6
Challenges: WebService Types
One URL REST
Params: QueryResponse: XML
17 3
Params: JSONResponse: JSON
10 1
Challenges: Signatures
Type Services
v2 2
v3https 1
cloudfront 1
v4 the rest
Challenges: API versions
More tan one API version is live
That’s why not-up-to-date callers still work
RDS
4 versions
CloudFront
3 versions
EC2
4 versions
AWS Pace of innovation
Every month there are changes
Changes mean APIs change (slightly)
Very hard to keep up!
AWS Pace of innovation
Every week there are announcements
Changes mean APIs change (slightly)
Very hard to keep up!
Lets start writing!
Hand write classes and methods, and parameters
Hand write classes and methods, and parameters
Parse docs
After all… you would have to read them to write the classes
Parse docs
After all… you would have to read them to write the classes
No published spec
Introspect an official SDK!
After all AWS already has read the docs for you
No published spec Introspect an official SDK!
That’s your spec!
After all AWS already has read the docs for you
Data driven SDKs!
JS SDK has datastructures that define calls
Proof of concept
Win++! Boto and PHP2 too!
Later migrate to using Boto
Data driven SDKs!
JS SDK has datastructures that define calls
Proof of concept
Win++! Boto and PHP2 too!
Later migrate to using Boto
The solutions
Modern Perl to the rescue
Try to be very lazy Parameter validation: Create Moose classes that validate
the parameters passed to the call. May change in the future.
Got very quick results!
Hide all possible implementation from the user Try not to leak implementation
That way you can change it at will
Meta lets you inspect the objects! Call objects get marshalled into datastructured via meta
Result datastructures get marshalled into objects via meta
Roles
Modern Perl: Roles
All functionality gets pulled in via Roles
Code is basically in different roles, Classes are “serialized configurations” of how to call APIs
Makes it easy to
Generate service classes from datastructures
Change behaviour for tests
Roles are used for
One global endpoint vs endpoints per region
Caller (Query, JSON)
Response (XML, JSON)
Signatures (V2, V3, V4)
IO
Aws is a class factory
Dynamic class construction
User never constructs service classes directly
Aws class method uses default config
my $ec2 = Aws->service(‘EC2’)->new( . . . )
Dynamic class construction
User wants custom functionality
my $aws = Aws->new( config => AWS::SDK::Config->new( caller => 'Net::AWS::TestCaller‘ ) );
my $ec2 = $aws->service(‘EC2’)->new( . . . )
service loads Aws::EC2, creates a new class with TestCaller
Win: Hello to the async world
Still playing around in examples
my $aws = Aws->new( config => AWS::SDK::Config->new( caller => 'Net::AWS::MojoCaller‘ ) );
my $ec2 = $aws->service(‘EC2’)->new( . . . )
my $asgs = $ec2->DescribeAutoScalingGroups;
See examples/asyncAutoScaling.pl
Win: Also going “fully async”
my $aws = Aws->new( config => AWS::SDK::Config->new(caller => 'Net::AWS::MojoAsyncCaller'));
my $delay = Mojo::IOLoop->delay(sub { my ($delay, @responses) = @_; Dumper($_) for @responses;});
foreach my $region ( "us-east-1", "ap-northeast-1", "sa-east-1", "ap-southeast-1", "ap-southeast-2", "us-west-2", "us-west-1", "eu-west-1", 'invented-region') { my $ctrail = $aws->service('CloudTrail')->new( region => $region, delay => $delay, ); print "Doing a call for $region\n"; my $list = $ctrail->DescribeTrails;}
$delay->wait;
Lessons Learned
Lessons learned
First shot with MooseX::Declare
Error messages are HORRIBLE!
Change to Moops didn't go well
Finally plain old Moose
Lessons learned
Namespace finding (still in the works)
AWS vs Aws
Thanks @clintongormley for the article!
Lessons learned
Enums (still in the works)
APIs sometimes return unexpected stuff
Lessons learned
Abstract $request object passed around
Finally marshalled into the UserAgents’ taste
Still open stuff
Load time for big services (EC2)
Not all classes will be used all the time
Generate classes on demand
Calling convention
Now CamelCased
Not so perlish
Calling from objects (not service objects)
$queue->DeleteQueue(); from a queue object
Attribute Traits
Access to Arrays and HashRefs via nicer methods
Lots more!!!
Use AWS? Use Perl? Want to help?
Fork your heart out:
https://github.com/pplu/aws-sdk-perl/