Post on 18-Jan-2016
transcript
Octopus Deploy Customization and Automation
with .NET and PowerShellThere and back again
Who am I?
Ian Paullin Architect/DevOps/.NET Programmer 5 years with Avanade, 10 years in .NET New Dad Seahawks Fan
What are we talking about?
Our story with Octopus Intro to Octopus REST API How to get started Demos
Octo.exe PowerShell Demo Post message to Slack
Why are we talking about this?
Octopus is awesome Automation is awesome Octopus is extremely extensible Octopus extensibility + automation = more awesome
I thought it would be like..
In retrospect..
Our Story
Take Octopus Deploy to the masses. Ensure 24/7 uptime. Limit specific users to deploy to production.
February 2014 – May 2014 (Pilot approx. 10 teams) June 2014 – February 2015
Goal - 120 projects on board by Feb/Mar 2015 February 2015 upgrade: from 2.5.4 to 2.5.12 (eventually to 2.5.13)
Problems emerged with upgrade March 2015 – October 2015
Stabilization + growth November 2015 – January 2016
More stabilization and automation More users/projects/servers Testing Octopus 3.1.x – 3.2.x
Where are we today?
5 Octopus servers with 5 NuGet servers Total tentacles: 1564 Total # of project groups: 255 Total # of projects: 509 Busiest Day: Wednesday – avg. 105 deployments Weekly deployment avg: 300-450 NuGet packages: A lot (e.g. 200Gb on just one of our NuGet servers)
Scaling
Octopus can do it, but we didn’t realize all contributing factors NuGet issues – with free NuGet.Server
Was recommended to use NuGet.Lucene or NuGet Gallery MyGet.org/ProGet No feed maintenance means large feeds over time (3-10gb and thousands of packages) Solution
Reduce packages in feeds (manually )) IIS Fileshares circumvent issue with NuGet client DLL
Octopus Server File Handles Max capacity of listening tentacles (listening uses 1 thread per machine!) NuGet communication issues Solution
Determine upper limit Balance team tentacle size across number of servers Switch to polling mode for tentacles (no thread per each polling tentacle like listening mode) Deleting old/stale projects
Lessons Learned
Research was difficult (at the time) No real documentation on Octopus on massive scaling (everything is relative) My failure to reach out before hand to determine how to best scale Our use case maybe outside of scope of Octopus? Not quite – but 3.x changes everything
about Octopus Deploy Estimation
Pilot proved it could work but estimating scaling or load was very obfuscated by numerous aspects (NuGet, tentacle communications, server configuration, etc.)
Initial estimate of teams moving to Octopus was way too low. Monitoring
SumoLogic – we had for IIS for NuGet.Server but didn’t use it correctly at beginning AppDynamics – added late to really foresee issues but invaluable today Not constantly watching environments – too many machines added
Lessons Learned (con’t)
RavenDB limitations of scale Primarily licensing issue limiting to 3 CPU cores and 6GB (embedded/ISV)
NancyFX is nice but.. Shielded information about user sessions from us Now Octopus 3.x has logging feature for sessions
Understand Limits/Tolerance If you don’t know them, find them out (break your stuff if you can) If you can’t find them, constantly watch out for multiple factors (monitoring) Monitoring especially at enterprise scale is not optional
What really worked well
Provisioning Automate process to create a team within Octopus. By hand, could take 25 to 30 minutes –
sometimes a lot longer. Automating process meant 30 to 45 seconds of filling out fields and executing a deployment
Automatically create NuGet feed on NuGet server for each project (isolation) Integration with Active Directory Groups and TFS AD Groups makes life much easier Can build a customize new Octopus Deploy server with one provisioning script
Permissions Permissions model is complicated, but extremely effective for our needs. This is 100%
automated from the provisioning process Reporting
Reports on how many deployments; how frequent; to which environments, etc. Error reports and frequency tracking
What really worked well (con’t)
Synchronizing user accounts with internal service Get list of users that are approved for production deployments for specific security teams – checks
every hour across all our instances Monitoring
AppDynamics – late start but very helpful in keeping a watch on stability Server metrics to gauge stability and measure improvements
Maintenance Weekly reboots – fully automated NuGet cleanup – not automated (yet) Changing IIS feeds to filesharing
Constant improvements Weigh pros/cons, time available and payoff Target daily or common tasks/procedures Backlog one-offs, random issues, odds/ends
Odds and ends
User Education Some of our users still don’t use optimize Octopus. They can cause lots of
problems with an entire instance and can make server unstable at least for 2.5.13 (backups, canceling/starting deployments, long running processes, etc.)
Documentation + videos + guidance sessions still not enough for some Good users are always willing to learn and improve but it takes time to see
benefits Excellent users are fewer in number, but really good in maximizing octopus Teams change therefore users change; re-education is cyclical and therefore
problematic over time
Support directly from Octopus Deploy Great support from small company Weekly meetings, research and testing extremely helpful Helping us now with our plans to migrate to 3.1.x – 3.2.x Continuing guidance and support on architecture design and
migration to 3.x
Keys to winning
Perseverance Keep at it
Watch closely and at high-level Monitoring + reports help make sense of what’s going on and what may
happen Continually improve aspects in and around Octopus Deploy
Improvements or tweaks, no matter how small, add value over time Automate
With Octopus REST API, we can script repeatable tasks for everything Customize
With Octopus REST API, we can extend Octopus in almost any way we want
One key to winning: The Octopus REST API View the API at http://<your octopus server>/api Documentation:
http://docs.octopusdeploy.com/display/OD/Octopus+REST+API https://github.com/OctopusDeploy/OctopusDeploy-Api/wiki http://docs.octopusdeploy.com/display/OD/Octopus.Client
Octopus REST API
Everything flows through the REST API Internally Externally Octo.exe
Octopus REST API - Demos
Demos Look at the octo.exe Query Octopus Deploy server directly via PowerShell Simple test message to Slack during deployment
Octopus REST API – DiY with Octo.exe Octo.exe
Great for automating deployments after build Quick Demo with Octo.exe
Octopus REST API – DiY with PowerShell/C# You need 3 DLLs (found in tentacle/server installation folder)
Newtonsoft.Json.dll Octopus.Client.dll Octopus.Platform.dll REST API URL exposes what’s available
APIKey and appropriate permissions for the user account (or service account) dictate what’s allowed for interaction; no username/password!
Quick Demo for PowerShell Report example
Octopus REST API – Sample Report
Ideas Any task done more than two or
three times Windows Tasks that can be
scheduled indefinitely Write to log or database for
auditing history Integration with 3rd party
tools/platforms
Examples Synchronization of users – if a
small subset of users is allowed Scheduled reboots – if necessary Security checks – especially for
AD groups Integration with Slack for
messages of deployments
Automation/Customization Ideas
Integration with Slack
Slack is a 3rd party collaboration/messaging platform that allows users to chat as well as programmable bots or external parties
Idea: for every deployment in Octopus, send message to Slack Demo
Octopus REST API conclusion
Not enough to just automate deployments. There’s lots of things around deployments that need automation as well.
Without the REST API, it’s much harder if not impossible to customize and automate as we did.
Many creative solutions possible – imagination and time are only barriers.
Thanks to..
Octopus Deploy team Damian Brady Vanessa Love Paul Stovell
Thadd Parker (@tptechgeek) For stats and sample report and sample C# code
Gandalf the Pug https://www.reddit.com/r/pics/comments/3pgtyr/gandalf_the_pug/