Application Secret Management w/ AWS

By Emmanuel Apau

@technogrouch

https://www.bonfire.com/black-code-collective/

Why do i care about secret management?

Insert Name Here

Secret management - Use cases● Infrastructure Engineering

○ SSH keys, SSL certificates, and configuration (e.g kubeconfig)

● Application Engineering○ API keys○ Database credentials

● Single usage○ https://onetimesecret.com/

● External service integration○ E.g. Kubernetes secrets syncing

Sooo… many… options

Key Vault

Parameter Store Demo

Parameter Store

● Ability to reuse iam policies & roles for access management● Change management auditing via with AWS CloudTrail.

○

● Managed service == No maintenance just configuration● Encrypted at rest with KMS

Parameter Store

Standard vs Advanced

# Allowed Max size # History Values Max throughput - transactions per second (TPS)

Standard 10,000 4 KB 100 40 API TPS ● Shared API limit:

○ GetParameter, GetParameters, GetParametersByPath

Advanced 100,000 8 KB 100 100 API TPS - GetParametersByPath1000 API TPS

● Shared API limit:○ GetParameter and GetParameters

Standard vs Advanced

# Allowed Max size # History Values Max throughput - transactions per second (TPS)

Standard 10,000 4 KB 100 40 API TPS ● Shared API limit:

○ GetParameter, GetParameters, GetParametersByPath

Advanced 100,000 8 KB 100 100 API TPS - GetParametersByPath1000 API TPS

● Shared API limit:○ GetParameter and GetParameters

Use Case Versioning/Auditing Price

SSM Parameter Store

1. API Keys2. DB

credentials3. Misc

Key/Pair values

VersioningHistoryCloudTrail Auditing

https://aws.amazon.com/systems-manager/pricing/Standard

● Free Storage● $0.05 per 10,000 Parameter Store API

interactionsAdvanced

● $0.05 per secret● $0.05 per 10,000 Parameter Store API

interactions

Secret Manager Password Rotations

(e.g. Databases)

CloudTrail Auditing $0.40 per secret per month$0.05 per 10,000 API calls

Parameter Store VS Secret Manager

Lets Math it Out

Assume you have 5,000 parameters, of which 500 are advanced parameters and interact with each parameter 24 times per day, equating to 3,600,000 interactions per 30-day month.

Assume you have enabled higher throughput, so your monthly bill will be the sum of the cost of the advanced parameters storage and the API interactions, as follows:

Cost of 500 advanced parameters = 500 * $0.05 per advanced parameter = $25

Cost of 3.6M API interactions = 3.6M * $0.05 per 10,000 interactions = (3.6m/10,000) * $0.05 = $18

Total monthly cost = $25 + $18 = $43

Secret Integration

Programmatically in the application using the AWS SDKa. Subject to rate limiting via API at scale

var params = { Name: 'STRING_VALUE', /* required */ WithDecryption: true || false};ssm.getParameter(params, function(err, data) { if (err) console.log(err, err.stack); // an error occurred else console.log(data); // successful response

});

Secret Integration

On EC2 host start-up, via user-data to store secrets as Environment Variables via aws cli

export DB_CONNECTION =$(aws --region=us-east-2 ssm get-parameter --name "db_connection" --query 'Value')

Secret Integration

On container service registration. Secrets can be merged into the definition, and registered as environment variables e.g. ECS Task Definitions

Secret Integration

K8s Service definition

Secret Tips

1. Diversity of secrets per environments

2. Fine tuned decrypt access roles for admins, developers, pms

3. Share secrets via secure channels, e.g lastpass NOT slack or pastebin

4. Use Temporary credentials where possible.a. E.g token based RDS authentication

5. Make sure everyone understands the secret management process

Enforcer-Reloaded CLI

Features

● CLI tool to create/list/update aws parameter store

secrets

● Allows chunking of large secrets greater than 4kb○ Breaks large secrets into 4kb chunks

● List functionality to easily audit secrets○ Versions○ Change dates○ Users modified

● Kubernetes synchronization helper function

Future Wants:

● Handle Advanced secrets

https://github.com/kave/enforcer-reloaded

Enforcer-Reloaded CLI Demo

We’re Hiring!

https://jobs.lever.co/socialtables/