v0.2
ChefConf 2015Chef & Security Workshop
v0.2
Introductions
v 3.0.0
S C O T T F O R D
smford22
smford22
@sford422
Solutions Engineer
v0.2
Goals
• Participants will learn the key access points on the Chef server and know how to secure them
• Participants will learn how to utilize industry standards for security and how the Chef analytics platform can be leveraged to streamline auditing and compliance
6
Goals
• Participants will understand the value of Chef Vault as compared to encrypted data bags
• Participants will successfully use Chef Vault to generate, distribute and encrypt/decrypt secrets both via CLI and the Chef client run
• Participants will learn strategies for managing and rotating keys
7
Goals
v0.2
Setups
Login to HipChat• For the purpose of this class, and in an effort to work
together as a team, we have setup a public HipChat channel for all of us to use. You will need to create an account if you do not have one, but you can find that link here…
• http://larryebaum.hipchat.com/chat
HipChat Quick Tips• @all notifies everyone regardless if they are present or not
• @here notifies everyone currently in the room
• @username to address a specific person in the room
• /code <paste code> allows you to paste code snippets in the room
Get logged in and say hello!
Chef Servershttp://goo.gl/L6CPdz
Find your card…
Chef Server Chef Analytics Server
Record the hostnames…
CloudShare NodeRed Cards: https://use.cloudshare.com/Class/xfjh0 Blue Cards: https://use.cloudshare.com/Class/cegfd Login: <COLOR><CARD><SUIT>@mail.com Password: chef
CloudShare Node
The authenticity of host 'uvo1qrwls0jdgs3blvt.vm.cld.sr (69.195.232.110)' can't be
established.
RSA key fingerprint is d9:95:a3:b9:02:27:e9:cd:74:e4:a2:34:23:f5:a6:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'uvo1qrwls0jdgs3blvt.vm.cld.sr,69.195.232.110' (RSA) to
the list of known hosts.
[email protected]'s password:
Last login: Mon Jan 6 16:26:24 2014 from
host86-145-117-53.range86-145.btcentralplus.com
[chef@CentOS63 ~]$
Lab - Login$ ssh chef@<EXTERNAL_ADDRESS>
CheckpointAt this point you should have •One virtual machine (VM) or server that you’ll use for the lab exercises
•The IP address or public hostname •An application for establishing an ssh connection
• ‘sudo’ or ‘root’ permissions on the VM
Exercise: Set up a working directoryMake a working directory on your laptop under your home directory called ‘~/workshop’, i.e. • Windows:- C:\Users\you\workshop
• Mac/*nix:- /Users/you/workshop
Navigate to this working directory
Exercise: Download the Chef Fundamentals Repo
Download the Chef Fundamentals working repohttps://github.com/learnchef/chef-fundamentals-repo
Exercise: Download the Chef Fundamentals Repo
Locally shared atafp://10.35.1.104 or smb://10.35.1.104
Share among yourselves using AirDrop or USB stick (passing around)
Archive: chef-fundamentals-repo-master.zip
bb06ea2c0cabaa855e4cb1d1c43bbe4d75caf70d
creating: chef-fundamentals-repo-master/
inflating: chef-fundamentals-repo-master/Berksfile
inflating: chef-fundamentals-repo-master/README.md
inflating: chef-fundamentals-repo-master/Vagrantfile
creating: chef-fundamentals-repo-master/cookbooks/
creating: chef-fundamentals-repo-master/cookbooks/apache/
inflating: chef-fundamentals-repo-master/cookbooks/apache/CHANGELOG.md
inflating: chef-fundamentals-repo-master/cookbooks/apache/README.md
creating: chef-fundamentals-repo-master/cookbooks/apache/attributes/
creating: chef-fundamentals-repo-master
...
Exercise: Extract the repo$ cp ~/Downloads/chef-fundamentals-repo-master.zip . $ unzip chef-fundamentals-repo-master.zip
Exercise: Create a New Org•Connect to your assigned Chef Server •Create an account •Create a new organization
Exercise: Download knife.rb; create validator .pem
Download the knife.rb & copy validator .pem text Paste text into file: <ORGNAME-validator.pem>
Exercise: Download your client .pem
•You may reuse your Chef server setup from the morning Analytics workshop
•Only do this if you don’t already have the .pem file available on your laptop • Reset and copy/paste your key to <USER.pem>
$ cd ~/workshop/chef-fundamentals-repo-master $ mkdir .chef $ mv ../<yourname>.pem .chef/ $ mv ../<your-org>-validator.pem .chef/ $ cp ~/Downloads/knife.rb .chef/
Exercise: Create and populate a .chef dir
• knife.rb & .pem files reside in the .chef directory which can be in o <current-directory>/.chef o /etc/.chef o ~/.chef
WARNING: Certificates from ec2-52-11-224-17.us-west-2.compute.amazonaws.com will be fetched and placed in your trusted_cert
directory (/Users/larryeichenbaum/workshop/chef-fundamentals-repo-master/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
Adding certificate for ec2-52-11-224-17.us-west-2.compute.amazonaws.com in /Users/larryeichenbaum/workshop/chef-fundamentals-repo-master/.chef/trusted_certs/ec2-52-11-224-17_us-west-2_compute_amazonaws_com.crt
Exercise: Setup Trust with Chef Server$ knife ssl fetch
<your-org>-validator.pem
Exercise: Test your workstation$ knife client list
Uploading logrotate [1.4.0] Uploading cron [1.2.8] Uploading chef-client [3.4.0] Uploaded 3 cookbooks.
Exercise: Upload Cookbook$ knife cookbook upload logrotate cron chef-client
OPEN IN EDITOR:
SAVE FILE!
Exercise: Edit base role
name "base"
description "Base Server Role"
run_list "recipe[chef-client::delete_validation]", "recipe[chef-client]"
roles/base.rb
Updated Role base!
Exercise: Upload role$ knife role from file base.rb
uvo164727i3mvh1jup2.vm.cld.sr --2014-05-13 04:31:10-- https://www.opscode.com/chef/install.sh
uvo164727i3mvh1jup2.vm.cld.sr Resolving www.opscode.com... 184.106.28.90
uvo164727i3mvh1jup2.vm.cld.sr Connecting to www.opscode.com|184.106.28.90|:443... connected.
uvo164727i3mvh1jup2.vm.cld.sr HTTP request sent, awaiting response... 200 OK
uvo164727i3mvh1jup2.vm.cld.sr Length: 15934 (16K) [application/x-sh]
uvo164727i3mvh1jup2.vm.cld.sr Saving to: `STDOUT'
uvo164727i3mvh1jup2.vm.cld.sr
100%[======================================>] 15,934 --.-K/s in 0s
uvo164727i3mvh1jup2.vm.cld.sr
uvo164727i3mvh1jup2.vm.cld.sr 2014-05-13 04:31:10 (538 MB/s) - written to stdout [15934/15934]
uvo164727i3mvh1jup2.vm.cld.sr
uvo164727i3mvh1jup2.vm.cld.sr Downloading Chef 11.8.2 for el...
uvo164727i3mvh1jup2.vm.cld.sr downloading https://www.opscode.com/chef/metadata?v=11.8.2&prerelease=false&nightlies=false&p=el&pv=6&m=x86_64
uvo164727i3mvh1jup2.vm.cld.sr to file /tmp/install.sh.41533/metadata.txt
uvo164727i3mvh1jup2.vm.cld.sr trying wget...
uvo164727i3mvh1jup2.vm.cld.sr url https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-11.8.2-1.el6.x86_64.rpm
...
Exercise: Bootstrap the target node$ knife bootstrap <external address> \ --sudo –x chef –P chef –N "node1" -–bootstrap-version 12.1.1
node1:
run_list:
role[base]
Exercise: Set run list for managed node$ knife node run_list set node1 'role[base]'
[email protected]'s password:
Last login: Sat Mar 28 11:13:08 2015 from ool-43570b90.dyn.optonline.net
_______________
/ ___/ __/_ __/
/ (_ / _/ / /
\___/___/ /_/
_______ ___________
/ __/ _ \/ __/ __/ _ \
_\ \/ ___/ _// _// // /
/___/_/ /___/___/____/
___ ___ _____
/ __| __|_ _|
| (_ | _| | |
\___|___| |_|
___ _____ ___ ___ __ __ ___
/_\ \ / / __/ __|/ _ \| \/ | __|
/ _ \ \/\/ /| _|\__ \ (_) | |\/| | _|
/_/ \_\_/\_/ |___|___/\___/|_| |_|
Exercise: SSH to Node$ ssh chef@<external name>
Starting Chef Client, version 11.12.8
resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"]
Synchronizing Cookbooks:
- chef-client
Compiling Cookbooks...
...
Exercise: Run Chef Clientchef@node1$ sudo chef-client
root 8933 0.3 2.2 130400 37816 ? Sl 03:19 0:01 /opt/chef/embedded/bin/ruby /usr/bin/chef-client -d -c /etc/chef/client.rb -L /var/log/chef/client.log -P /var/run/chef/client.pid -i 1800 -s 300
Exercise: Verify chef-client is runningchef@node1$ ps awux | grep chef-client
v0.2
Securing the Chef Server
Securing Chef• Secure Chef
http://commons.wikimedia.org/wiki/File:Trakošćan_2007.JPG
Securing Chef• Secure Chef • Security is two fold –
• Inner => Chef • Outer =>OS, environment, other tools, etc.
http://cadw.wales.gov.uk/docs/cadw/publications/WHS_additional_maps_plans_EN.pdf
Chef-Specific SecurityInstall… Omnibus version on a secure internal network • Avoid using insecure public networks unless you have valid reason to do so.
Chef-Specific SecurityLock down… Only open needed ports
• iptables • EC2 security groups
http://www.geograph.org.uk/photo/1613103
Chef-Specific SecurityLock down… Only open required ports • Standalone Chef server 80 & 443
Chef-Specific SecurityLock down… Only open required ports • Chef Server HA Front end: 80 & 443 Back end: 4321, 80, 443, 9683, 9090, 8000, 5432, 5672, 16379
Chef-Specific SecurityLock down… Only open required ports • With Chef analytics 80, 443, 5672
UPDATE PIC
Chef’s Secure… What’s Next?
http://images.mocpages.com/user_images/26934/12878311242_DISPLAY.jpg
Keeping Chef SecureImportant Chef Updates
https://lh6.googleusercontent.com/-9GWoh0a_t7E/AAAAAAAAAAI/AAAAAAAAADg/K7JOQKVPRtA/
photo.jpg
Keeping Chef SecureImportant Chef updates • Chef blogs
• Installation guides
Keeping Chef SecureImportant Chef updates • Chef blogs • Twitter feeds, email distro
Keeping Chef SecurePerform updates
http://www.maclife.com/files/imagecache/futureus_imagegallery_fullsize/gallery/02_1.jpg
Keeping Chef SecurePerform updates •docs.chef.io • Step-by-step instruction
Keeping Chef SecureSecurity incident alerts
http://commons.wikimedia.org/wiki/File:Red_Emergency_Light.jpg
Keeping Chef SecureSecurity incident alerts • Chef blogs
• Important security updates • Common security solutions
Keeping Chef SecureSecurity incident alerts •Chef blog •Status feeds • status.chef.io • twitter feed • tumblr feed
There's More
But Wait…
Non-Chef-Specific Security
http://www.securitygeneration.com/wp-content/uploads/2011/02/chain-padlock-security-fail.jpg
Non-Chef-Specific SecurityRestrict… who can access the Chef server: • Chef server stores all! • If you have access, you have access to database, cookbooks, etc.
http://midstatelockandsafe.com/wp-content/uploads/2010/10/Keys-To-The-Castle.jpg
Non-Chef-Specific SecurityRestrict further… Two-factor authentication • Chef controls/manages all your infrastructure.
• Be protective… • Or explain later. http://vignette3.wikia.nocookie.net/koc/images/7/75/
Defend_victory.jpg/revision/latest?cb=20101126001303
Non-Chef-Specific SecurityRestrict further… Two-factor authentication • Google Authenticator
• iOS, Android, Blackberry • Supports multiple accounts • Backup codes and devices
https://itunes.apple.com/us/app/google-authenticator/id388497605?mt=8
Non-Chef-Specific SecurityRestrict further… two-factor authentication • Duo Security
• iOS, Android, Blackberry • Palm, Windows Phone 7, etc. • Supports multiple accounts • Backup codes and devices
https://itunes.apple.com/us/app/duo-mobile/id422663827?mt=8
Non-Chef-Specific SecurityRestrict further… admins only via SSH keys • Passwords not THAT secure • Avoid duplicate passwords on multiple hosts
• More convenient • Reduces likelihood of brute-force attacks
http://www.correderajorge.es/wp-content/uploads/2013/09/public_key_cryptography_sm.png
http://commons.wikimedia.org/wiki/File:Crypto-Nerd.png
Non-Chef-Specific SecuritySecure Chef cookbooks • Can be a point of entry 1000011001
0BACKDOOR10011100010010000101000100101000001000110
Non-Chef-Specific SecuritySecure Chef cookbooks • Can be a point of entry • Contains the code that defines your servers
https://www.flickr.com/photos/bradhigham/14760496352/in/photolist-ispVZE-mNbLn-6AXqjt-89yaQo-oukpjq-prBQef-jqtAUm-8fQsXR-cBahob-anqncf-8qtez7-639A8i-4x7zyB-oVXXQ4-8hUGrR-gYkwn3-nGjyXe-61PBPr-86n2rp-89uVET-Ci1ba-2616FK-7ZiXZa-ct1ddE-
cZtAH9-5n3EuN-6KDsVR-8Szby-q5U5FY-2cHYb4-fjHU2S-cCxp4E-nac2Mw-5oh1AN-uWzN6-3aRRbX-85ZcvN-6jPyC1-oudwgE-8xYTVr-6B13jx-a6RZTN-5sn8t4-7smmD-992dQL-7ZZHTn-oupy5F-dP8sHj-bE8t-7YRahJ
Non-Chef-Specific SecuritySecure Chef cookbooks • Can be a point of entry • Contains the code that defines your servers
• Contains the code that runs on your servers
http://cdn.arstechnica.net/wp-content/uploads/2014/01/falken.jpg
Yea,
I'm
old
whi
te g
uy…
but
hav
e yo
u re
ally
not
see
n W
ar G
ames
?
Non-Chef-Specific SecuritySecure code repo • Be cautious with public GitHub or equivalent
https://www.flickr.com/photos/gsimmonsonca/3285952133/in/photolist-61nn68-6W6HKX-9tkjXz-787xod-9Gxpiz-9iQjBN-qNyuAp-au8kjx-8ApgTa-49iUwy-8e9adZ-
gGdDuw-9uu261-7wDeBV-5TaCkm-B5xwM-59vc3E-72sPTp-ad5jTc-oXiowS-7Bu8hZ-JgGsp-3cQcoc-aggJUx-b3inBe-asbtAs-5bBJMD-6rch3G-6aDzQR-dX8M7i-24L3Yq-6XCAQH-3H5HE-Dcfrn-dAPegg-8vJGry-8dfP18-dcLWaq-7MyZPH-nPFmX9-h8YsAC-Brr18-cCczjL-898gbm-cXUrLq-nvyPib-3XiC5w-8v6sC-8cc475-6oKosU
Non-Chef-Specific SecuritySecure code repo • Be cautious with public GitHub or equivalent • grants insight to your infra code
• GitHub Enterprise or equivalent • internal network • secure • control access
https://connect.decknetwork.net/i/github_enterprise2.png
Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks
Premier Malt Products Co.; Printing Not Stated edition (1928)
Public Domain Cooking
How to Share Your Very Own Cook Book
Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks • Can be a point of entry
https://www.flickr.com/photos/innoxiuss/3306092527/in/photolist-639A8i-4x7zyB-oVXXQ4-8hUGrR-gYkwn3-nGjyXe-61PBPr-Ci1ba-2616FK-7ZiXZa-ct1ddE-7YRahJ-cZtAH9-5n3EuN-6KDsVR-8Szby-q5U5FY-2cHYb4-fjHU2S-cCxp4E-nac2Mw-5oh1AN-uWzN6-3aRRbX-85ZcvN-6jPyC1-oudwgE-8xYTVr-6B13jx-a6RZTN-5sn8t4-7smmD-992dQL-7ZZHTn-oupy5F-dP8sHj-bE8t-59G4t1-9fTpzb-9gKoEX-cvzKn5-qLC9TW-8s59Mp-2cHYtX-kKVeFB-rmWHuR-9xKv5y-8iAAWZ-guvBw9-9CuSNo
Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks • Can be a point of entry • Read and review everything
https://www.flickr.com/photos/bcgovphotos/8113334531/in/photolist-637KYD-6rCqSw-ayHV31-h4FzN1-QtXAi-bz31qy-bz31ud-bz2ZPE-bMWH56-bMWH2e-bMWFmc-bz2ZDb-bz2ZvG-bz2ZyC-bMWFyH-bMWGTi-bz312W-bMWF5B-bMWGGM-bz2ZSY-bMWFH4-
aMGyv-dmWXG8-2TzBBJ-58mAsb-6bTs8x-hCCAGd-dmYcfU-akUamc-qw7r3R-b4uCGP-9fhgkr-knVUPZ-mrMJKx-mcYPuV-bMWH6P-bMWF9B-bMWGRg-bMWGMx-
bMWGWa-bz31gE-bz2YrY-bz2YA9-bMWE2z-bMWFak-bz2YuE-bMWDUx-h4FodQ-h4FoUD-5B38Au
Non-Chef-Specific SecuritySupermarket content • Publicly shared cookbooks • Can be a point of entry • Read and review everything • Test run in isolation-- if your security requires it
https://www.flickr.com/photos/riacale/4230091725/in/photolist-jxpBYJ-6nssFe-fMcDHU-piQkus-dwnAtk-84xhpn-apLYmH-4H6hbz-4T59iu-q5c9ra-5BK2bU-qHjXax-5Rb5eJ-piHCcJ-6HeMaE-6HaJxV-4zQbEB-buLxSf-iWs9-GiDTU-7rNjTX-4Q2u8A-9gFVjE-bpeN7A-dJGb51-
dGokV5-98ukrU-6334S-9gF9xM-7EpEWX-7vAbgF-dy4bk7-5NjDk8-4CvBbo-8TftWn-68aLxw-6v7j4D-6cuhSL-3bm4U-3bm4S-3bm4T-fz4T94-95m3Ti-6ugcDf-6NqRS7-dMPypX-p9tcgy-3bm4V-5RAXGm-5NoTru
Non-Chef-Specific SecurityGeneral security practice • Follow accepted guidelines
https://www.cisecurity.org
Non-Chef-Specific SecurityGeneral security practice • Follow accepted guidelines • CIS security benchmarks
https://www.cisecurity.org
Non-Chef-Specific SecurityGeneral security practice • Follow accepted guidelines • CIS security benchmarks • Benchmark documents • Metrics definitions
https://www.cisecurity.org
v0.2
Configuring Chef ClientsSSL Verification and chef-client Management
Lesson Objectives•After completing the lesson, you will be able to:
• Configure nodes to use SSL certificate validation
The Problem and Success Criteria•The problem: Make sure the Chef client service is is validating all SSL connections
•Success criteria: Use the chef-client cookbook to accomplish these tasks
SSL Certificate ValidationChef 11 defaults to not verifying certificates when it makes HTTPS connections - hence the SSL error when running chef-client • Enable SSL Verification by setting ssl_verify_mode :verify_peer in your config file
• If enabled, affects all SSL endpoints (eg., remote_files, etc.)
• Two useful knife commands: • knife ssl check - makes an SSL connection to your Chef server or any
other HTTPS server and tells you if the server presents a valid certificate • knife ssl fetch - allows you to automatically fetch a server’s certificates
to your trusted certs directory
Using the chef-client CookbookThe chef-client cookbook manages the Chef client: • client.rb configuration • service (runit, systemd, init, SMF, etc.) • Validation key deletion
Examining the chef-client CookbookWe’re already using two recipes on the node from the chef-client: •chef-client::service (via chef-client::default)
•chef-client::delete_validation
Managing client.rb settings
default_attributes( "chef_client" => { "config" => { "ssl_verify_mode" => ":verify_peer", "log_level" => ":info" } } )
chef_server_url "https://api.opscode.com/organizations/MYORG"
validation_client_name "MYORG-validator"
ssl_verify_mode :verify_peer
node_name "config-ubuntu-1204"
log_level :info
• For example, the following attributes (in a role)
• Will render the following configuration (/etc/chef/client.rb)
OPEN IN EDITOR:
SAVE FILE!
Exercise: Edit base role; add chef-client::config
name "base"
description "Base Server Role"
run_list "recipe[chef-client::delete_validation]", "recipe[chef-client::config]", "recipe[chef-client]"
default_attributes(
"chef_client" => {
"config" => {
"ssl_verify_mode" => ":verify_peer",
"log_level" => ":info"
}
}
)
roles/base.rb
Updated Role base!
Exercise: Upload role$ knife role from file base.rb
Starting Chef Client, version 11.12.8
resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"]
Synchronizing Cookbooks:
- chef-client
Compiling Cookbooks...
...
Exercise: Run chef-clientchef@node1$ sudo chef-client
…
ssl_verify_mode :verify_peer
log_level :info
…
Exercise: Verify that Config is Updatedchef@node1$ sudo cat /etc/chef/client.rb
Special Handling for non-CA Signed Certs• Hosted Chef’s certificate is signed by a known CA • If you run your own CA, or your Chef server’s certificate is self-signed, you can use knife to import it
• If you do, it is also necessary to import to your Chef server, the certificate from the server used to generate your Chef certificate
Review Questions• How do you remove the validation .pem on the node? • What recipe configures chef-client? • How do you enable SSL verification on chef-client?
v0.2
Chef 12 SSL
Chef 12 SSLSSL certificate validation improvements: • New default settings for SSL certificate validation • Default in favor of validation • New knife ssl command
Chef 12 SSLInitial knife connections • You’ve just spun up a new Chef 12 server • You’ve just installed Chef 12 client on your local system
• You’ve created a user account and an organizationnode_name 'someNode' client_key 'clientKey.pem' validation_client_name 'org-validator' validation_key 'org-validator.pem' chef_server_url 'https://chef-server.some.domain/organization/org
ERROR: SSL Validation failure connecting to host: chef-server.example.com - SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
ERROR: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
Chef 12 SSL$ knife client list
Chef 12 SSLWhy the error? • SSL verification enabled by default for all requests • Certificate is self-signed • No signing CA that can be verified • Hence, knife command fails
Chef 12 SSLknife ssl check: • Verify the connection and certificate
• Follow steps indicated to retrieve the server SSL certificate and store it locally as a ‘trusted’ certificate
• Or…
Connecting to host chef-server.example.com:443 ERROR: The SSL certificate of chef-server.example.com could not be verified Certificate issuer data: /C=US/ST=WA/L=Seattle/O=YouCorp/OU=Operations/CN=chef-server.example.com/[email protected]
Configuration Info: OpenSSL Configuration: * Version: OpenSSL 1.0.1j 15 Oct 2014 * Certificate file: /opt/chefdk/embedded/ssl/cert.pem
* Certificate directory: /opt/chefdk/embedded/ssl/certs
Chef SSL Configuration: * ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/Users/jtimberman/Downloads/chef-repo/.chef/trusted_certs" TO FIX THIS ERROR: If the server you are connecting to uses a self-signed certificate, you must configure chef to trust that server's certificate. By default, the certificate is stored in the following location on the host where your chef-server runs: /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt
Copy that file to your trusted_certs_dir (currently: /Users/<userName>/Downloads/chef-repo/.chef/trusted_certs) using SSH/SCP or some other secure method, then re-run this command to confirm that the server's certificate is now trusted.
WARNING: Certificates from chef-server.example.com will be fetched and placed in your trusted_cert directory (/Users/newUser/Downloads/chef-repo/.chef/trusted_certs).
Knife has no means to verify these are the correct certificates. You should verify the authenticity of these certificates after downloading.
Adding certificate for chef-server.example.com in /Users/newUser/Downloads/chef-repo/.chef/trusted_certs/chef-server.example.com.crt
Chef 12 SSL$ knife ssl fetch
Chef 12 SSL•The certificate should be verified to ensure that it is in fact the certificate on the Chef server.
•Compare checksums •Knife commands will now work!!
Chef 12 SSLDo I need to do this on every node?
http://pixabay.com/static/uploads/photo/2014/04/03/10/52/circle-311551_640.png
Chef 12 SSLknife ssh command •Invokes SSH commands (in parallel) on a subset of nodes within an organization
•Based on the results of a search query made to the Chef server
•Syntax: knife ssh SEARCH_QUERY SSH_COMMAND (options)
node-output.example.com WARNING: Certificates from chef-server-example.com will be fetched and placed in your trusted_cert node-output.example.com directory (/etc/chef/trusted_certs). node-output.example.com node-output.example.com Knife has no means to verify these are the correct certificates. You should node-output.example.com verify the authenticity of these certificates after downloading. node-output.example.com node-output.example.com Adding certificate for chef-server.example.com in /etc/chef/trusted_certs/chef-server.example.com.crt
Chef 12 SSL$ knife ssh 'name:*' 'sudo knife ssl fetch –c /etc/chef/client.rb
Chef 12 SSLSSL verification also works in Chef 11
SAVE FILE!
OPEN IN EDITOR:
Chef 12 SSL
Execute$ sudo chef-server-ctl reconfigure
nginx['ssl_certificate'] = "/etc/pki/tls/certs/your-host.crt" nginx['ssl_certificate_key'] = "/etc/pki/tls/private/your-host.key"
/etc/opscode/chef-server.rb
Chef 12 SSLCertificates expire and grow moldy
http://postimg.org/image/4cvafd4v1/full/ http://www.saveyourstuff.com/blog/wp-content/uploads/2010/04/dsc_0370-300x200.jpg
Chef 12 SSLRegenerate SSL certificates before expiration • Important aspect of protecting Chef server from vulnerabilities
•Helps prevent information stored in Chef server from becoming compromised
Chef 12 SSL•Stop the Chef server $ chef-server-ctl stop •Verify the FQDN for the Chef server $ hostname -f
•Delete certificate files $ rm /var/opt/opscode/nginx/ca/$<FQDN>.crt$ rm /var/opt/opscode/nginx/ca/$<FQDN>.key
Chef 12 SSL•If you are using custom SSL certificates, refer to the file names/location as referenced by the values:
nginx['ssl_certificate'] nginx['ssl_certificate_key'] within file:
/etc/opscode/chef-server.rb
•Delete those files; regenerate using same CA
Chef 12 SSL•Reconfigure server to create new SSL certs
$ chef-server-ctl reconfigure •Start the Chef server
$ chef-server-ctl start
Review Questions•What is the knife command used to acquire the SSL keys from the Chef server?
•What is the key difference between the Chef 11 server and Chef 12 server with regard to SSL certificates?
•How frequently do Chef server SSL certifications auto regenerate?
v0.2
Chef-Vault
• Open source project published by Nordstrom
• Distributed as a RubyGem
• Bundled within chef-dk
• Gem needs to be installed on all workstations/servers that will either encrypt or decrypt data
105http://meta.wikimedia.org/wiki/
File:SLNSW_17216_Sliding_grille_at_the_entrance_to_the_Rural_Banks_vault_Martin_Place.jpg
http://flickr.com/photos/77947478@N00/2631203510
Not this
kind…
Chef Vault
*** LOCAL GEMS *** Chef-vault (2.4.0)
Check Chef Vault Version$ chef gem list chef-vault
$ Fetching: chef-vault-2.5.0.gem (100%) Successfully installed chef-vault-2.5.0 1 gem installed
Install Chef Vault (if necessary)…$ chef gem install chef-vault
• Command line structure
• Includes knife plugins
• Works just like normal data bags
• Secrets live in data bags on the Chef server knife vault create [VAULT] [ITEM] [VALUES] --mode MODE --search SEARCH --admins ADMINS --json FILE knife vault decrypt [VAULT] [ITEM] [VALUES] --mode MODE knife vault delete [VAULT] [ITEM] --mode MODE knife vault remove [VAULT] [ITEM] [VALUES] --mode MODE --search SEARCH --admins ADMINS knife vault update [VAULT] [ITEM] [VALUES] --mode MODE --search SEARCH --admins ADMINS --json FILE ...
108
Command Line Options…
We’ll use a new command here, knife vault create, to create the item that stores the secret info…
• We can directly pass the unencrypted secret without storing it
• Alternatively, we can store the unencrypted secret on disk in a JSON formatted file
109
Create the Item…
$ knife vault create secrets vaultuser \ '{"vaultuser":"<INSERT PASSWD>"}' \ --search 'role:base' \ --admins <YOUR USER> --mode client
110
Create Data Bag…
111
Alternate: Create JSON file…OPEN IN EDITOR: ./secrets_vaultuser.json
SAVE FILE!
{"vaultuser":"<SOME PASSWD>"}
112
Alternate: Create Data Bag…$ knife vault create secrets vaultuser \ --json secrets_vaultuser.json \ --search 'role:base' \ --admins <YOUR USER> --mode client
$ knife data bag show secrets
113
vaultuservaultuser_keys
Verify Content on Chef Server…
•The first item in the data bag, vaultuser, contains the actual secret.
•The second item indicates what nodes and user accounts have access to decrypt the password.
114
Examine the Data Bag Contents…
$ knife data bag show secrets vaultuser
115
id: vaultuservaultuser: cipher: aes-256-cbc encrypted_data: j+/fFM7ist6I7K360GNfzSgu6ix63HGyXN2ZAd99R6H4TAJ4pQKuFNpJXYnC SXA5n68xn9frxHAJNcLuDXCkEv+F/MnW9vMlTaiuwW/jO++vS5mIxWU170mR EgeB7gvPH7lfUdJFURNGQzdiTSSFua9E06kAu9dcrT83PpoQQzk= iv: cu2Ugw+RpTDVRu1QaaAfug== version: 1
Examine First Item in Data Bag…
$ knife vault show secrets vaultuser \ ’vaultuser’ --mode client
116
id: vaultuser vaultuser: <PASSWORD>
Decrypt First Item Using Knife…
Recall, the second item indicates what nodes and user accounts have access to decrypt the password.
117
Can Nodes Decrypt This?
$ knife data bag show secrets vaultuser_keys
118
admins: adminUserclients: os-945926465950316 os-2790002246935003id: vaultuser_keysadminUser: 0Q2bhw/kJl2aIVEwqY6wYhrrfdz9fdsf8tCiIrBih2ZORvV7EEIpzzKQggRX...XdmdfMKNYGaM6kzG8cwuKZXLAgGAgblVUB1HP8+8kQ==os-2790002246935003: kGQLsxsFmBe9uPuWxZpKiNBnqJq55hQZJLgaKdjG2Vvivv98RrFGz1y8Xbwe...rK3s5vD7v6jpkUW12Wj74Lz3Z6x3sKuIDzCtvEUnWw==os-945926465950316: XzTJrJ3TZZZ1u9L9p6DZledf3bo2ToH2yrLGZQKPV6/ANzElHXGcYrEdtP0q14Nz1NzsqEftzviAebUUnc6ke91ltD8s6hNQQrPJRqkUoDlM7lNEwiUiz/dD+sFI6CSzQptO3zPrUbAlUI1Zog5h7k/CCtiYtmFRD6wbAWnxmCqvLhO1jwqLVNJ1vfjlFsG77BDm2HFw7jgleuxRGYEgBfCCuBuW70FAdUTvNHIAwKQVkfU/Am75UYm7N4N0E+W76ZwojLoYtXXTV/iOGG1cw3C75SVAmCsBOuxUK/otub67zsNDsKToKa+laxzXGylrmkTricYXIqVpIQO8OL5nnw==
Examine Second Item in Data Bag…
NODE
NODE
Let’s use Chef Vault to manage a user resource with a password that is set to the value stored within our encrypted data bag. •Create a cookbook
•Add cookbook to base role
CD into your cookbooks directory
119
Manage a User Password
$ chef generate cookbook vault
120
Compiling Cookbooks…Recipe: code_generator::cookbook. . .
Create Cookbook
121
Create the Cookbook…OPEN IN EDITOR: /cookbooks/vault/recipes/default.rb
SAVE FILE!
. . .# All rights reserved - Do Not Redistribute#
chef_gem "chef-vault"require "chef-vault"
vault = ChefVault::Item.load("secrets", "vaultuser")hashed_password = vault['vaultuser'].crypt("$6$")
user "vaultuser" do password hashed_password home "/home/vaultuser" supports :manage_home => true shell "/bin/bash" comment "Chef Vault User"end
• This is where the decryption happens…
• Item.load takes two variables: the data bag (e.g. vault) and the item
• The value of the item is returned
122
Breaking It Down…vault = ChefVault::Item.load("secrets", "vaultuser")
• Setting a variable to equal the decrypted 'password' that has been re-encrypted with SHA-512 required by the operating system for user passwords.
• Using a case statement you could set the re-encryption to match that required by different operating systems.
123
Breaking It Down…hashed_password = vault['vaultuser'].crypt("$6$")
• Within the user resource the important attribute is password as we’re referencing the local variable vault and the vault user key
124
Breaking It Down…user "vaultuser" do password hashed_password home "/home/vaultuser" supports :manage_home => true shell "/bin/bash" comment "Chef Vault User"end
Uploading vault [0.1.0]
Uploaded 1 cookbook.
Upload Vault Cookbook$ knife cookbook upload vault
node1:
run_list:
role[base]
recipe[vault]
Exercise: Edit run list for managed node$ knife node run_list add node1 'recipe[vault]'
Starting Chef Client, version 11.12.8
resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"]
Synchronizing Cookbooks:
- chef-client
- vault
Compiling Cookbooks...
...
Exercise: Run chef-clientchef@node1$ sudo chef-client
Password: <PASSWORD>
vaultuser@os-279000224593500d:~$
Use Newly Created Account…chef@node1$ su - vaultuser
Password: chef-vault
vaultuser@os-2790002246935003:~$ id
uid=1001(vaultuser) gid=1001(vaultuser)groups=1001(vaultuser)
vaultuser@os-2790002246935003:~$ pwd
/home/vaultuser
Use Newly Created Account…chef@node1$ su - vaultuser
id
homedir
What if one of our admins leaves and we want to change their associated password/secret?
130
http://sd.keepcalm-o-matic.co.uk/i/keep-calm-and-change-your-password-16.png
Chef Vault Edits
$ knife vault create secrets \ vaultuser '{"vaultuser": "<NEW PASSWORD>"}' \ --search 'role:base' \ --admins adminUser --mode client
131
ERROR: ChefVault::Exceptions::ItemAlreadyExists: secrets/vaultuser already exists, use 'knife encrypt remove' and 'knife encrypt update' to make changes.
Create Data Bag…
$ knife vault update secrets vaultuser \ '{"vaultuser":"<NEW PASSWORD>"}' \ --search 'role:base' \ --admins adminUser --mode client
132
Update Data Bag Instead…
$ knife vault update secrets vaultuser \ '{"vaultuser":"<NEW PASSWORD>"}' \ --admins adminUser --mode client
133
Shortened Update Data Bag Instead…
Secrets/vaultuser
Vaultuser: <NEW PASSWORD>
Decrypt First Item Using Knife…$ knife vault show secrets vaultuser vaultuser --mode client
Starting Chef Client, version 11.12.8
resolving cookbooks for run list: ["chef-client::delete_validation", "chef-client"]
Synchronizing Cookbooks:
- chef-client
- vault
Compiling Cookbooks...
...
Exercise: Run chef-clientchef@node1$ sudo chef-client
Password: <NEW PASSWORD>
vaultuser@os-279000224593500d:~$
Reconverge and Validate…$ su - vaultuser
v0.2
Audit Controls and Rules
DependenciesChef Analytics Version 1.1.0 is current production release
Chef Server Version 12.0.3 or higher; Enterprise Chef 11.3 or higher
Chef Client Version 11 or 12 required; 12.1 required for audit-mode
Reporting Version 1.2.3 suggested for full feature/functionality
Analytics Components
Analytics FeaturesControls Automated tests, built into a cookbook and then used to test the state of the system for compliance
Audit Mode Execution mode of chef-client that evaluates control / control-groups that are defined within recipes •Can be run in combination with a standard run after resources converge
•As a separate chef-client run with no convergence
Analytics FeaturesChef Actions
A collection of data gathered and made available Keeps track of what happened during chef-client runs across your Chef managed systems
Reporting Reports can be generated for the entire organization and they can be generated for specific nodes.
Rules Track specific outcomes of audits and issue notifications to various endpoints and/or services
Writing ControlsMethods Two methods are built into the recipe DSL that can be used to configure tests for audit-mode: • control_group • control
Writing Controlscontrol_group •Defines a group of control methods that comprise a single audit
•Can contain several control methods
Writing Controlscontrol •Tests written within a recipe that are used to test the state of the system for compliance
•A single control block defines one or more validations
•Contains an it statement •Each it statement contains one or more expect statements
Writing Controlscontrol_group "audit name" do control "name" do it "should do something" do expect(something).to/.to_not be_something end endend
Writing Controlscontrol_group "audit name" do
control "name" do
it "should do something" do
expect(something).to/.to_not be_something
end
end
control "nameTwo" do
it "should do something new" do
expect(something).to/.to_not be_something
expect(somethingElse).to/.to_not be_something
end
end
end
Writing Controlsexpect statements can use matchers •Matchers are available for: o directory o file o package o port o service
Writing ControlsSome examples… directory: it "should be a directory" do expect(file("/var/directory")).to be_directory end
it "should be mounted with an ext4 partition" do expect(file("/")).to be_mounted.with( :type => 'ext4' ) end
Writing ControlsSome examples… file: it "should be executable by owner" do expect(file("/etc/file")).to be_executable.by("owner") end
it "should be mode 440" do expect(file("/etc/file")).to be_mode(440) end
Writing ControlsSome examples… package, port: it "should be installed" do expect(package("httpd")).to be_installed end
it "should not be listening" do expect(port(23)).to_not be_listening end
Writing ControlsSome examples… service: it "should be enabled" do expect(service("ntpd")).to be_enabled end
it "should be running" do expect(service("ntpd")).to be_running end
Writing ControlsPutting it together… control_group 'check sshd configuration' do
control 'sshd package' do it 'should be installed' do expect(package('openssh-‐server')).to be_installed end end
control 'sshd configuration' do let(:config_file) { file('/etc/ssh/sshd_config') } it 'should exist with the right permissions' do expect(config_file).to be_file expect(config_file).to be_mode(644) expect(config_file).to be_owned_by('root') expect(config_file).to be_grouped_into('root') end it 'should not permit RootLogin' do expect(config_file.content).to_not match(/^PermitRootLogin yes/) end it 'should explicitly not permit PasswordAuthentication' do expect(config_file.content).to match(/^PasswordAuthentication no/) end it 'should force privilege separation' do expect(config_file.content).to match(/^UsePrivilegeSeparation sandbox/) end end end
Let's Perform Audits…
http://2.bp.blogspot.com/---1IokhPScE/Ulb3PorYrJI/AAAAAAAAAik/MwNuvpkUmyU/s1600/AuditAllTheThings.png
. . .
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
. . .
Check Remote Node for Port 25 Listeninguser@centos$ netstat –plnt | grep :25
Compiling Cookbooks...
Recipe: code_generator::cookbook
* directory[/Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab] action create
- create new directory /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab
* template[/Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/metadata.rb] action create_if_missing
- create new file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/metadata.rb
- update content in file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/metadata.rb from none to f8e5d5
. . .
* execute[initialize-git] action run
- execute git init .
* cookbook_file[/Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/.gitignore] action create
- create new file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/.gitignore
- update content in file /Users/larryeichenbaum/chef-repo-larryebaum/cookbooks/auditlab/.gitignore from none to dd37b2
Generate a Cookbook$ chef generate cookbook auditlab
OPEN IN EDITOR:
SAVE FILE!
Create Recipe
#
# Cookbook Name:: auditlab
# Recipe:: default
#
# Copyright (c) 2015 The Authors, All Rights Reserved.
control_group 'check smtp configuration' do
control 'smtp port' do it 'should not be listening' do expect(port(25)).to_not be_listening end end
end
/cookbooks/auditlab/recipes/default.rb
Uploading auditlab [0.1.0] Uploaded 1 cookbook.
Upload Cookbook$ knife cookbook upload auditlab
node1: run_list: role[base] recipe[vault] recipe[auditlab]
Set Node Runlist$ knife node run_list add <node> 'recipe[auditlab]'
New chef-client Run Option--audit-mode <MODE> •Audit mode evaluates custom rules, or audits, defined within recipes
•Can be run as part of normal chef-run, at the end •Can be run separately, where no converge occurs o --audit-mode disabled (default) o --audit-mode enabled o --audit-mode audit-only
[2015-03-06T22:29:50+00:00] WARN: Chef-client has been configured to skip converge and run only audits. Audit mode is an experimental feature currently under development. API changes may occur. Use at your own risk. * To enable audit mode after converge, use command line option `--audit-mode enabled` or set `:audit_mode = :enabled` in your config file. * To disable audit mode, use command line option `--audit-mode disabled` or set `:audit_mode = :disabled` in your config file. * To only run audit mode, use command line option `--audit-mode audit-only` or set `:audit_mode = :audit_only` in your config file. Audit mode is disabled by default. Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase
Execute client-run Audit Modeuser@centos$ sudo chef-client –-audit-mode audit-only
. . . check smtp configuration smtp port should not be listening (FAILED - 1)
Failures:
1) check smtp configuration smtp port should not be listening Failure/Error: expect(port(25)).to_not be_listening expected Port "25" not to be listening . . . Finished in 0.05565 seconds (files took 0.27206 seconds to load) 1 example, 1 failure
Failed examples:
rspec # check smtp configuration smtp port should not be listening
Execute client-run Audit Modeuser@centos$ sudo chef-client –audit-mode audit-only
OPEN IN EDITOR:
SAVE FILE!
Modify Audit
#
# Cookbook Name:: auditlab
# Recipe:: default
#
# Copyright (c) 2015 The Authors, All Rights Reserved.
control_group 'check smtp configuration' do
control 'smtp port' do it 'should be listening' do expect(port(25)).to be_listening end end
end
/cookbooks/auditlab/recipes/default.rb
Uploading auditlab [0.1.0] Uploaded 1 cookbook.
Re-upload Cookbook$ knife cookbook upload auditlab
Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase
check smtp configuration smtp port should be listening
Finished in 0.05616 seconds (files took 0.2733 seconds to load) 1 example, 0 failures Auditing complete
Running handlers: Running handlers complete Chef Client finished, 0/0 resources updated in 2.365794244 seconds 1/1 Audits succeeded
Execute client-run Audit Modeuser@centos$ sudo chef-client –audit-mode audit-only
Who Tracks Audit Results?
http://s237.photobucket.com/user/dharma-putra/media/AuditReportandAuditorOpinion.png.html
Audit Rules…Audits are of limited value if the results are unknown or not communicated: •Rules track the events generated by audits •Rules track specific outcomes of the audit tests •Rules send notifications about these outcomes o http webhook o email o chat services, etc.
Audit Rule Details…Each rule must be associated with a message type •Messages sent to analytics as rules are triggered •Message types:
•action (msgs about actions that occur on Chef Server) •run_control (audit related) •run_control_group (audit related) •run_converge (msgs sent at end of chef-client run) •run_resource (msgs sent during chef-client run) •run_start (msgs sent at start of chef-client run)
Chef Analytics Login…
Chef Analytics Login…
Chef Analytics Login…
Let's Write Rules…
http://advisorspension.com/wp-content/uploads/2014/09/money-rules.jpg
rules 'pci check smtp rules'
rule on run_control
when
name = "should be listening" and
status = "success"
then
notify("Hipchat",<YOUR NAME>: "A server is running SMTP on port 25."
end
end
Create New Rule
Notifications and Reports…
https://openclipart.org/image/300px/svg_to_png/3743/zeimusu-Warning-notification.png
Data Flow
Event Processing and Rules Engine
User Interface
Notifications
IntegrationsChef Server
Event stream
Create Notification…
http://advisorspension.com/wp-content/uploads/2014/09/money-rules.jpg
Create Notification…
1345044
1345hEV944u5vEwPawKgUL8LOYaN2zqJGSwAUDdvHk00044
Create Notification…
1345044
hEV944u5vEwPawKgUL8LOYaN2zqJGSwAUDdvHk00
Create Notification…
hEV944u5vEwPawKgUL8LOYaN2zqJGSwAUDdvHk00
} ONE LINE, NO LINE BREAKS
Create Notification…
http://advisorspension.com/wp-content/uploads/2014/09/money-rules.jpg
[2015-03-06T22:29:50+00:00] WARN: Chef-client has been configured to skip converge and run only audits. Audit mode is an experimental feature currently under development. API changes may occur. Use at your own risk. * To enable audit mode after converge, use command line option `--audit-mode enabled` or set `:audit_mode = :enabled` in your config file. * To disable audit mode, use command line option `--audit-mode disabled` or set `:audit_mode = :disabled` in your config file. * To only run audit mode, use command line option `--audit-mode audit-only` or set `:audit_mode = :audit_only` in your config file. Audit mode is disabled by default. Starting Chef Client, version 12.1.0 resolving cookbooks for run list: ["auditlab"] Synchronizing Cookbooks: - auditlab Compiling Cookbooks... Starting audit phase
Execute client-run Audit Modeuser@centos$ sudo chef-client –audit-mode audit-only
Verify Notification Received
http://advisorspension.com/wp-content/uploads/2014/09/money-rules.jpg
Key Management / Rotation
http://upload.wikimedia.org/wikipedia/commons/c/c6/Llave_bronce.jpg
Key Management / Rotation•New in Chef server 12.0.3 (still in progress) •Multi-key authentication •Chef-server-ctl commands to build key rotation
Key Management / Rotation•Store and manage multiple keys for users and clients
•Server will check request against all valid keys for user or client
•Enables server to know if request is from a user or client
•Resolves user-client ambiguity •Optional expiration dates
Key Management / Rotation•Backwards compatible •API support underway •Still in progress, so don’t delete your ‘default’ keys just yet
Key Rotation CommandsSubcommands to chef-server-ctl •add-client-key •add-user-key •delete-client-key •delete-user-key • list-client-key • list-user-key
Key Rotation Syntax$ chef-‐server-‐ctl add-‐client-‐key ORG_NAME CLIENT_NAME \ PATH_TO_KEY [-‐-‐expiration-‐date DATE] [-‐-‐key-‐name NAME]
$ chef-‐server-‐ctl add-‐user-‐key USER_NAME PATH_TO_KEY [-‐-‐expiration-‐date DATE] [-‐-‐key-‐name NAME]
$ chef-‐server-‐ctl delete-‐client-‐key ORG_NAME CLIENT_NAME KEY_NAME
$ chef-‐server-‐ctl delete-‐client-‐key USER_NAME KEY_NAME
$ chef-‐server-‐ctl list-‐user-‐keys ORG_NAME CLIENT_NAME [-‐-‐hide-‐public-‐keys]
$ chef-‐server-‐ctl list-‐user-‐keys USER_NAME [-‐-‐hide-‐public-‐keys]
https://mastersreview.com/files/2014/06/summary.jpg
Summary•We’ve learned how native data bag encryption differs from the more complex data bag encryption we can accomplish via Chef Vault
• We’ve learned about how to lock down specific access points to Chef Server application and how to apply industry standards for security to further secure our Chef Server infrastructure
Summary•We’ve successfully learned how to use Chef Vault to generate, distribute and encrypt/decrypt secrets
•We’ve successfully performed a security audit of a Chef managed server and learned how we can extend this functionality in a broader capacity
Summary•We’ve learned about new Chef Server features for managing and rotating keys