About Me20+ years as an Oracle DBA, ConsultantRAC SIG Board MemberIOUG Experience CommitteePerformance, DR, HA, AutomationUltramarathons, Climbing, Canyoneering
Presenter
Presentation Notes
I’ve been working with Oracle as a DBA for over 20 years Ran my own consultancy for 12 years RAC SIG board member Member of the IOUG experience committee at collaborate 18 My interests are performance, disaster recovery, high availability and RAC, and automation I’m married with two daughters and a son, an almost 2yo grandson and a granddaughter on the way. I live in Idaho where there are only a few thousand people and you can still get cool personalized plates. I battle being a grandfather by shaving off the grey hairs, running ultramarathons and climbing, and talking about cool things like devops and docker and slack. Hello fellow kids! I’ve worked for BBcom for a little over five years. World’s largest supplier of online nutritional products. happy and excited to announce that I’m about to embark on a new adventure.
The Problem•Diverse mediums
•Alert fatigue
•Poor visibility and awareness
•Lack of accountability
•Multiple login vectors
•Limited access
•Escalation-heavy
Presenter
Presentation Notes
When I joined bodybuilding.com, the DBAs were being hit with all kinds of messaging and information, from different angles and sources. Email, SMS, chat. Informational messages, status updates, warning pages, legitimate alerts, tracking information. Some was being delivered via shell scripts, from agents, databases sending email directly, and from multiple 3rd party monitoring applications. Each app meant a separate place to config DND, etc. All this noise led to alert fatigue, making it difficult to sort through what was what. Adding to the confusion was not always knowing who was alerted when there was a problem. There was no good visibility, so it was easy for multiple people to be working on the same thing at once. Sometimes that would antagonize a situation as multiple people worked contradictory solutions simultaneously, or saw confusing data from a half-fixed system that would lead to bad choices. Lack of visibility extends to a lack of accountability. If I’m the only one getting a minor alert I might be tempted to ignore it. When it goes to Slack, anyone with membership permission in the channel can see it, so there’s no hiding a problem. With multiple systems come multiple logins, credentials, and limitations on access. All of these systems are behind a VPN, meaning that solving a problem required a laptop and internet access. Common database problems were also being escalated to the DBA, even when they were fairly routine.
The Objective•Consolidate messaging
•Minimize noise without compromising content
•Improve visibility and communication
•Automate and empower
•Categorization
•Security!
Presenter
Presentation Notes
Ideally I wanted something that would be a one-stop for every piece of information being emailed, chatted, or sent via page. I wanted to keep all of the informational content because sometimes it provides an informative source for identifying trends or answering questions. I didn’t want to have to manually run queries, or access the data on a separate log server. At the same time, I don’t want to be bothered every time some information is added to the library. When working a crisis I felt that an alert and the conversation it spawned should be in the same spot. Everyone should be aware of who was doing what and be able to coordinate easily in one place. And, the tool should preserve the history in a searchable form so that we wouldn’t have to rely on someone’s accurate documentation of the steps taken should it occur again. Furthermore I hoped for something that would allow a degree of automation to be included, accessible without a VPN connection, to reduce the number of escalations. I’d rather empower an end user to fix a problem than force them to go through a frustrating process and involve several people. There should be a means of categorizing or sorting messaging, by system, team, role, etc. Finally, it should be authenticated and secure, and give us the ability to manage access to information channels internally as well.
Presenter
Presentation Notes
If we could do all of this with one tool, that would be great!
What is Slack?•Messaging platform
•More than chat
•Popular
•Customizable
•Free or paid options
Presenter
Presentation Notes
Let’s digress for a moment and talk about Slack. First, let me say that I do not work for Slack, own stock or interest in the company, nor do I know anyone who works there or has any interest in them. This is based on my own personal experience with the tool. If you’re unfamiliar with it, Slack is a messaging platform with added features. It’s more than chat. It’s also very popular and it has a pretty robust API that lets you customize it pretty extensibly. It comes in free or paid options so you can experiment with it on your own if you’d like.
Presenter
Presentation Notes
I was first introduced to Slack in 2015 and at this tweet from the time describes it pretty well. Except at the time, I hated Slack. I hated it because it was being forced on me by some of our development teams, and it looked to me (at the time) to be another “oh cool here’s a shiny thing that <insert startup name> is using so we should, too”. All the problems I described before that we were having with the environment? Slack just added to them. It became another medium, a third chat client for me to run and monitor so that I could keep track of what people were doing and be available for requests. So at first I hated Slack and I resisted using it.
People Love to Hate Slack
Presenter
Presentation Notes
That put me in good company. People love to hate on Slack.
Presenter
Presentation Notes
People seriously love to hate on Slack. A lot. It took people a lot less time to start hating slack than it did email. But, there are actually some things about Slack that are worth your love.
Features• Alert on keywords
• Tailor alerts by channel
• Highlight channels
• Public and private channels
• Do not disturb
• Mobile, desktop apps
• AD authentication
• Invite people with @mentions
• Broadcast via @channel and @here
• Rich formatting options
• Attach files
Presenter
Presentation Notes
That said, there are things to like about Slack. When it came to integrating Slack into our monitoring and specifically our Oracle environment, some of the things I found that have been helpful include being able to alert on keywords. I work with a dev surprised that I always show up when there’s a conversation about Oracle or databases, calls me 5-9’s now. You can tailor alerting by channel. You can highlight certain channels in your channel list You can create private or public channels, though you should know that a private channel cannot be made public. You can authenticate via AD, and capture AD group memberships to authorize actions You can broadcast to @channel and @here There are some pretty robust formatting options available You can attach files
Slack API•Library at api.slack.com
•Build your own
•We mainly use three interactions
•Email to a Slack channel
•Incoming Webhooks
•Bots
Presenter
Presentation Notes
Almost everything you can do in Slack manually you can accomplish via the Slack API, too, plus more. “App store” style functionality The API seems to be fairly well documented. I’m not a super REST savvy developer and I was able to figure stuff out on my own and get things to work with my rudimentary python skills. There is a growing library of customizations available through Slack, plus a good community of developers working on neat things. Or, you can build your own. In our case, I built our own because there isn’t currently a ton of Oracle/Slack integration out there, compared to other platforms and databases. Almost everything I did was based on three main interactions between the database and Slack: The ability to email to a Slack channel, which doesn’t take any programming skill at all The incoming web hooks API The ability to automate things via bot
Emailing to Slack is the easiest way to integrate. Slack creates a unique email address for a channel and you send mail there. It’s not particularly pretty or slick, but if you have a black box app that you can’t work with or need a quick and dirty solution, this is for you. There’s not much to it, and you can get this set up in a few minutes
Incoming Webhooks• Add the Incoming Webhooks app to a channel
• Configure
• Integrate via OS script, PL/SQL, etc.
• Communicates via HTTPS
Presenter
Presentation Notes
Webhooks are a little more difficult to deal with but give you more power to send information to Slack and control its formatting. There’s a little more configuration to get into here, too, but you have flexibility. You can be basic or fancy, anything from a basic shell script that sends a curl request, to a PL/SQL package that needs a certificate in a wallet and ACLs. With each of these mechanisms, BTW, there are a lot of ways to do things. I’ve purposely stayed away from describing specific technical implementation b/c our env is a little awkward and we have some odd workarounds. But it’s not really rocket science. The api and documentation makes this all very accessible to you.
Presenter
Presentation Notes
Here’s an example of a web hook request sent out from a shell script that automates repair of our replication environment. This is an example of an alert I want to know about without getting an email. I want to be able to reference it when I need to but it’s not important enough to clutter my inbox.
Bots and Automation• Automate repetitive, boring tasks
• Simplify complex operations
• Provide an interactive interface with help features
The coolest thing we’ve done with Slack is introducing Larrybot. Larrybot automates repetitive and boring tasks for us. He simplifies complex operations He provides an interface that even a non-technical person can use to interact meaningfully with the Oracle databases. and, he provides a layer of separation between the database and an end user that allows someone we would never consider granting database access to perform database work. When a bot receives a command, it invokes a PL/SQL procedure that a Slack database user has been given execute permissions for. A bot can call a ton of things: A shell script, a Rundeck job, Bishop, Puppet, Chef, Docker, connect to EC2, you name it. It’s designed to be flexible so you can devise the solution that works best for your environment.
Bots and AutomationBefore LarryBot:
• End user discovers issue;
• Creates ticket;
• Ticket is assigned;
• Ticket is investigated;
• Known issue is identified;
• Ticket is escalated to dev/DBA;
• Dev/DBA runs a script
After LarryBot:
• End user discovers issue;
• Invokes Larrybot;
• Larrybot confirms and fixes!
Presenter
Presentation Notes
We run an old version of a monolithic application that has some known, shall we say, quirks. These things happen all the time. When they crop up, they can be confirmed programmatically and fixed, but it requires database access. We don’t want to give access to end users so when these conditions occur, and they happen a lot, it required a developer or DBA to log in and run a PL/SQL package with whatever values the end user had identified. The process was burdensome and time consuming. But giving Slack database access to run the PL/SQL packages that do that work means we can cut out a huge part of our database help desk work. The end user can invoke Larrybot directly by passing some parameters on the command line. We parse and validate them, confirm things programmatically, then just fix the problem and respond with the results. It makes for a happier help desk, happier on-call users and DBAs, and much happier users! What makes this really powerful is the ability to use a mobile app to invoke these tasks without having to access VPN or a laptop.
Slack is not Perfect• Reliability and downtime
• Lack of control
• Security
• Can be abused, some annoying apps
Presenter
Presentation Notes
Some downsides we’ve encountered with Slack center on reliability and downtime. It has not always been the most reliable platform, and their concept of “available” is sometimes a little curious. I’ve experienced outages that were never reported as actual outages. However, they have been getting better over the past year or two. Naturally you do not have control over a third party application. If Slack decides to discontinue a feature you’re using, you have no say. We were using XMPP gateways and they’re gone now, and we had to work around it somewhat unexpectedly. It’s a third party web application. You have no idea how good they are at security, so caveat emptor. Slack can be annoying and people can abuse some features. There’s a bot called GIFY that allows you to type something like “gify elephant” and it randomly posts a gif of an elephant it finds on the web. I’ve seen channels turn into a cascade of animated garbage.
Integrating Slack + Oracle• All notifications directed to
Slack
• Notifications directed to searchable, purposeful channels
• Bots to automate workflow
• End users are empowered
• Reduced alerting, quieter on-call
• Fewer escalations
• Secure enough
Presenter
Presentation Notes
Ultimately, we’ve been able to accomplish many of our goals. No more primary notifications via email or SMS. Everything goes to Slack and users can decide how it alerts them on their phone or desktop. Notifications are directed to purposeful channels, reducing clutter and noise. When something does happen, all involved can join the channel to see exactly what’s going on. There’s no longer confusion about who’s doing what. The chat becomes the documentation and it’s searchable down the road. We’ve automated workflows through bots, and empowered end users to fix known problems on their own without having to give them database access. That’s led to reduced on call stress and fewer escalations of trivial problems. We've weighed the risks of trusting a third party with this functionality and we’re careful about what’s included in messaging. We use private channels to limit the visibility of potentially sensitive information when appropriate.
Resources• Slack Documentation on Bot Users https://api.slack.com/bot-users
• Sean Scott: “Sending messages from an Oracle database to Slack”http://oraclesean.blogspot.com/2017/06/sending-messages-from-oracle-database.html
• Morten Braten: “Using the Slack Webhook API from PL/SQL”https://ora-00001.blogspot.com/2018/02/using-slack-webhook-api-from-plsql.html
• Morten Braten: Alexandria Slack Utility Package for APEXhttps://github.com/mortenbra/alexandria-plsql-utils/blob/master/ora/slack_util_pkg.pkb
