Enterprise Intelligence
Enterprise IntelligenceKinesis
to
Redshift
to
Qlik
Technology Partners:
PresenterPete Zybrick – Enterprise Solutions Architect
Over 30 years of experience designing and delivering
complex software solutions.
From Bell Labs to Big Data, Pete has architected,
managed, tested and implemented large scale mission
critical systems directly responsible for billions of
dollars in annual transactions.
As the leader of the Big Data technical programs within
IPC Global, Pete is responsible for building a
framework of collaboration between IPC Global and our
technology partners, Cloudera and AWS.
Kinesis to Redshift to Qlik
IPC Global’s AWS Certifications
AWS Certified Solutions Architect – Professional is a
certification that recognizes IT professionals that possess
unparalleled skills and the highest technical knowledge
necessary for designing, deploying, and managing
applications on the AWS platform.
AWS Certified SysOps Administrator – Associate Level is
a certification that validates IPC Global’s technical expertise
in deployment, management, and operations on the AWS
platform.
Kinesis to Redshift to Qlik
AWS Certified Developer – Associate Level is a
certification that validates IPC Global’s technical expertise in
developing and maintaining applications on the AWS
platform.
Objectives
• Provide QlikView Server Users with Near 100% Availability
• Provide QlikView Server Support Staff with Automation and Status Visibility
• Automated QlikView Service Restarts
• Capture QlikView Audit, Events, Performance and Session logs concurrently from multiple QlikView servers and store in central relational database
• Create a new Availability log, log message is written every 30 seconds describing the status of each QlikView service and Cluster Component (i.e. QMS, QDS, etc.)
• Process Rules against log messages in real time and take action(s)
• QlikView Dashboard to display key QlikView service availability metrics
• Scalable, Highly Available, Secure
Kinesis to Redshift to Qlik
AWS Technologies
• Virtual Private Cloud (VPC): Virtual Network to Manage Instances and Access
• Elastic Compute Cloud (EC2): Windows Server(s) and Linux instance(s)
• Kinesis/Kinesis Client Library (KCL): Distributed High Volume Message Delivery
• Simple Naming Service (SNS): Send Messages via Email, SMS and SQS
• Simple Queue Service (SQS): Point to Point Guaranteed Message Delivery
• Integrated Access Management (IAM): Security and Policy/Permission Management
• Redshift: Petabyte-Scale Columnar Relational Database
Kinesis to Redshift to Qlik
Scenarios
• Scenario 1: Basic Flow
• Scenario 2: Service Down, Restart Failure
• Scenario 3: Service Down, Restart Successful
Kinesis to Redshift to Qlik
Scenario 1: Basic Flow
MultiLogMgr
TailLog
QVS - First
Audit
Events
Sessions
Performance
QlikView
Server
ipcsvcmon Availability
QVS - Last
. . .
Audit
Events
Sessions
Performance MultiLogMgr
TailLog
QlikView
Server
ipcsvcmon Availability
Kinesis to Redshift to Qlik
Kinesis
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance 1
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance n
SNS
Redshift
SvcMon
Dashboard
mlmweb
Scenario 2: Service Down, Restart Failure
MultiLogMgr
TailLog
QVS - First
Audit
Events
Sessions
Performance
QlikView
Server
ipcsvcmon Availability
Kinesis to Redshift to Qlik
Kinesis
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance 1
SNS
Redshift
SQS
queue
service_name=QDS@uset1b-qlikview
service_status=CouldNotContact
QDS@uset1b-qlikview Down
Scenario 2: Service Down, Restart Failure
MultiLogMgr
TailLog
QVS - First
Audit
Events
Sessions
Performance
QlikView
Server
ipcsvcmon Availability
Kinesis to Redshift to Qlik
Kinesis
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance 1
SNS
Redshift
SQS
queue
service_name=QDS@uset1b-qlikview
service_status=RestartAttempt
Scenario 2: Service Down, Restart Failure
MultiLogMgr
TailLog
QVS - First
Audit
Events
Sessions
Performance
QlikView
Server
ipcsvcmon Availability
Kinesis to Redshift to Qlik
Kinesis
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance 1
SNS
Redshift
SQS
queue
service_name=QDS@uset1b-qlikview
service_status=Dead
QDS@uset1b-qlikview Dead
IPC Global 24x7 Support
Scenario 3: Service Down, Restart Successful
MultiLogMgr
TailLog
QVS - First
Audit
Events
Sessions
Performance
QlikView
Server
ipcsvcmon Availability
Kinesis to Redshift to Qlik
Kinesis
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance 1
SNS
Redshift
SQS
queue
service_name=QDS@uset1b-qlikview
service_status=CouldNotContact
QDS@uset1b-qlikview Down
Scenario 3: Service Down, Restart Successful
MultiLogMgr
TailLog
QVS - First
Audit
Events
Sessions
Performance
QlikView
Server
ipcsvcmon Availability
Kinesis to Redshift to Qlik
Kinesis
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance 1
SNS
Redshift
SQS
queue
service_name=QDS@uset1b-qlikview
service_status=RestartAttempt
Scenario 3: Service Down, Restart Successful
MultiLogMgr
TailLog
QVS - First
Audit
Events
Sessions
Performance
QlikView
Server
ipcsvcmon Availability
Kinesis to Redshift to Qlik
Kinesis
MultiLogMgr.KinesisApp
Kinesis Client Library
RuleHandler
EC2 Instance 1
SNS
Redshift
SQS
queue
service_name=QDS@uset1b-qlikview
service_status=RestartSuccessful
Dashboard
• Online Demo
Kinesis to Redshift to Qlik
Components
• On each QlikView Server• Dashboard
• ipcsvcmon: monitor QV services, write status to Availability log, process Restart requests (.Net, Windows Service)
• TailLog: tail the Audit, Availability, Events, Performance and Sessions logs, send to MultiLogMgr (Java, Windows Service)
• On Linux instance(s) • MultiLogMgr: process all inbound log messages from all source servers, write messages
to database, process rules, send emails/SMS messages, send ServiceStatus messages to ServiceMonitor (Java, linux daemon)
• mlmweb: Web application to maintain Rules and Source Systems (JSF2, JPA2, Tomcat7)
Kinesis to Redshift to Qlik
Summary
• Proactive
• Automation
• Visibility
• Near 100% Availability for IPC Global Hosted Clients
Kinesis to Redshift to Qlik
Enterprise Intelligence
Enterprise Intelligence
Technology Partners:
Kinesis
To
Redshift
To
Qlik