Application Secret Management w/ AWS
By Emmanuel Apau
@technogrouch
https://www.bonfire.com/black-code-collective/
Why do i care about secret management?
Insert Name Here
Secret management - Use cases● Infrastructure Engineering
○ SSH keys, SSL certificates, and configuration (e.g kubeconfig)
● Application Engineering○ API keys○ Database credentials
● Single usage○ https://onetimesecret.com/
● External service integration○ E.g. Kubernetes secrets syncing
Sooo… many… options
Key Vault
Parameter Store Demo
Parameter Store
● Ability to reuse iam policies & roles for access management● Change management auditing via with AWS CloudTrail.
○
● Managed service == No maintenance just configuration● Encrypted at rest with KMS
Parameter Store
Standard vs Advanced
# Allowed Max size # History Values Max throughput - transactions per second (TPS)
Standard 10,000 4 KB 100 40 API TPS ● Shared API limit:
○ GetParameter, GetParameters, GetParametersByPath
Advanced 100,000 8 KB 100 100 API TPS - GetParametersByPath1000 API TPS
● Shared API limit:○ GetParameter and GetParameters
Standard vs Advanced
# Allowed Max size # History Values Max throughput - transactions per second (TPS)
Standard 10,000 4 KB 100 40 API TPS ● Shared API limit:
○ GetParameter, GetParameters, GetParametersByPath
Advanced 100,000 8 KB 100 100 API TPS - GetParametersByPath1000 API TPS
● Shared API limit:○ GetParameter and GetParameters
Use Case Versioning/Auditing Price
SSM Parameter Store
1. API Keys2. DB
credentials3. Misc
Key/Pair values
VersioningHistoryCloudTrail Auditing
https://aws.amazon.com/systems-manager/pricing/Standard
● Free Storage● $0.05 per 10,000 Parameter Store API
interactionsAdvanced
● $0.05 per secret● $0.05 per 10,000 Parameter Store API
interactions
Secret Manager Password Rotations
(e.g. Databases)
CloudTrail Auditing $0.40 per secret per month$0.05 per 10,000 API calls
Parameter Store VS Secret Manager
Lets Math it Out
Assume you have 5,000 parameters, of which 500 are advanced parameters and interact with each parameter 24 times per day, equating to 3,600,000 interactions per 30-day month.
Assume you have enabled higher throughput, so your monthly bill will be the sum of the cost of the advanced parameters storage and the API interactions, as follows:
Cost of 500 advanced parameters = 500 * $0.05 per advanced parameter = $25
Cost of 3.6M API interactions = 3.6M * $0.05 per 10,000 interactions = (3.6m/10,000) * $0.05 = $18
Total monthly cost = $25 + $18 = $43
Secret Integration
Programmatically in the application using the AWS SDKa. Subject to rate limiting via API at scale
var params = { Name: 'STRING_VALUE', /* required */ WithDecryption: true || false};ssm.getParameter(params, function(err, data) { if (err) console.log(err, err.stack); // an error occurred else console.log(data); // successful response
});
Secret Integration
On EC2 host start-up, via user-data to store secrets as Environment Variables via aws cli
export DB_CONNECTION =$(aws --region=us-east-2 ssm get-parameter --name "db_connection" --query 'Value')
Secret Integration
On container service registration. Secrets can be merged into the definition, and registered as environment variables e.g. ECS Task Definitions
Secret Integration
K8s Service definition
Secret Tips
1. Diversity of secrets per environments
2. Fine tuned decrypt access roles for admins, developers, pms
3. Share secrets via secure channels, e.g lastpass NOT slack or pastebin
4. Use Temporary credentials where possible.a. E.g token based RDS authentication
5. Make sure everyone understands the secret management process
Enforcer-Reloaded CLI
Features
● CLI tool to create/list/update aws parameter store
secrets
● Allows chunking of large secrets greater than 4kb○ Breaks large secrets into 4kb chunks
● List functionality to easily audit secrets○ Versions○ Change dates○ Users modified
● Kubernetes synchronization helper function
Future Wants:
● Handle Advanced secrets
https://github.com/kave/enforcer-reloaded
Enforcer-Reloaded CLI Demo
We’re Hiring!
https://jobs.lever.co/socialtables/