Post on 09-Jan-2017
transcript
Agenda1.Introduction
2.Environment Management
3.Metadata Synchronization and migrations
4.Monitoring
5.Resource Management
4
Introduction
Denodo deployments can go from a few developers and users
executing thousands of queries a day to hundreds of developers and
users executing hundreds of thousands of queries a day. In order to
manage both types of deployments Denodo provides multiple
features that simplify this task.
These features will help with the management of environments,
metadata synchronization, migrations, resource management and
monitoring.
5
Challenges in Large Deployments
Managing multiple environments with different dependent parameters such as
connectivity credentials
Migration process between different stages of the release lifecycle can be time
consuming and prone to errors
Resources can be over loaded based on the number of users or queries
Finding and diagnosing an issue in one of the servers of the deployment can
be difficult as there are multiple environments and/or servers that can be part
of a cluster
6
Solutions
Denodo fully integrates with a version control system
Dependent parameters are organized by environments
Simple migration process between different environments by providing both a GUI and
a set of scripts that can be fully automated and integrated with a current migration
protocol
Resource manager integrated in the administration tool to control the system
resources with a big level of granularity
Denodo monitoring and diagnosing web allows to configure a set of environments or
servers so all the information for each node, including historical, is available from a
single interface
7
Deployment Overview
Administration and Development Tool DEVELOPMENT
• Integrated Version Control for multipleenvironments
• Includes Import/Exportfunctionalities
• Resources are managed from the same tool
Load Balancer
STAGING PRODUCTION CLUSTER
• Integrated in the Denodo Server
• It provides a webinterface for easy access
Monitoring and diagnostic
Replicator & automated scripts
• GUI • Scripts that can be
integrated with other processes
Testing Tool
• Automated Testing• No programming
neededIntegrated
Version Control
MS TFSSubversion git
Local DevelopmentServer
9
Version Control
Built-in integration with version control systems: Subversion, Git and MS TFS
Support for team development and multiple environment migrations
Integrated dependency management and conflicts control
10
Version Control
Bob Alice
DevelopmentServer
StagingServer
Demo_svn_Bob Demo_svn_Alice
Demo_svn Demo_svn
VCS Repository
Demo_svn(consolidated Development
database)
Replicator & Scripts
11
Dependent parameters and Environments
Denodo includes the concept of environment for manage dependent metadata
such as the data sources connection properties (e.g. login/passwords)
Environments can represent, for example, different development servers for
different groups and/or different geographical locations
When loading metadata from a different environment, the process can be
configured to use the properties of the target environment instead of the origin
ones
12
Version Control Environments
By following this approach we can easily maintain all our dependent
metadata organized and ready to be used
13
Version Control
Bob Alice
DevelopmentServer in New York
StagingServer
Demo_svn_Bob Demo_svn_Alice
Demo_svn Demo_svn
VCS Repository
Demo_svn(consolidated Development
database)
Replicator & Scripts
Local Development Server in Palo Alto
PA EnvironmentProperties
NY EnvironmentProperties
Staging EnvironmentProperties
15
Metadata Synchronization and Migrations
Denodo provides different tools to help with metadata synchronization and migration process:
VDP Administration Tool GUI
Shell Scripts
Denodo Replicator
Denodo Testing Tool
16
VDP Administration Tool GUI
Denodo admin tool provides graphical wizards to export/import
metadata and environment dependent properties
17
Shell Scripts
Denodo provides scripts for performing periodical backup copies and import metadata
to a list of servers
These scripts offer the same options as the VDP admin interface for import and export
The scripts can be found in the Denodo_Home\bin folder but also as an standalone tool
in the DENODO_HOME\tools\db\denodo-db-tools folder
These scripts can be used to automate the promotion/migration process adding any
validation rules as needed
18
Denodo Testing Tool
18
The Denodo Testing Tool allows Denodo users to easily automate the testing of their
data virtualization scenarios.
Tests are specified in text files and organized in folders. No programming needed.
Test sets are executed in a completely automated manner.
19
Creating an automated script
We can combine the shell scripts with the Testing Tool and create a fully automated
process, for example our script can:
Obtain the consolidated version (VCS) from the development server
Load that version into the staging server but using the environment dependent
parameters file
Execute the Denodo Testing Tool group of tests
In case of error inform of tests that failed
In other case tag the version
20
Denodo Replicator
Denodo Replicator can be used to graphically manage an
environment and synchronize the metadata among its members
Metadata replication
22
■ See current sessions, queries, connections, cache load processes…
■ See resources usage in each server (CPU, memory, connections,…)
■ Inspect data sources and cache statistics (connection pools, response times, active requests…)
■ Inspect errors, warnings and system logs
■ Integrates Monitoring Functionality from VDP Admin Tool, Denodo Dashboard and Denodo Monitor
■ Go “back in time” to the moment where a
problem happened
■ Graphically inspect and browse all the
information provided by the Denodo Monitor
and server logs:
■ Active requests and sessions
■ Resources Usage
■ Data source statistics
■ Integrates diagnosing information currently
dispersed in logs:
■ Graphical Analysis of incidents
Monitoring and Diagnosing Tool (I)Graphical Monitoring of Servers and Clusters; Graphical Problem Diagnosing
24
Monitoring and Diagnosing Tool (III)
Environments and servers can be created
Servers can be added to an environment
Start monitoring a server/environment by double clicking in an item or by right clicking in the desired item and choosing the desired option
Diagnostic information about an specific server can be loaded by an option of the right context menu (new “child” of this server appears in the tree)
25
Monitoring and Diagnosing Tool (and IV)
Data of monitoring and diagnosing is organized in 8 categories:
State: Summary information of the state of the server/environment
Resources: Information about physical resources (memory, cpu,…)
Requests: Information about requests, including real-time execution trace
Session: Currently opened sessions, including client application
Cache: Information about cache load processes and cache contents
Datasources:Information about activity in the datasources
Threads: Information about threads (priorities, CPU usage)
Errors: Inspect logged errors and warnings
… and many others
Filter and sort information by any criteria: e.g. see data sources used by a query, see requests from a given session,…
Use Case Scenario: Slow queries (I)
This sample scenario shows a possible use case in diagnostic
mode
Users have reported performance problems with VDP server
‘server1’ during query execution times are bigger than expected
The issues were reported between 19:15 and 21:00 on March 3
The goal is to detect the root cause of the problem
26
Use Case Scenario: Slow queries (II)
The diagnostic logs saved during March 3 are loaded
A new main tab “Server1 March 3” is opened with data for the loaded diagnostic data
27
Use Case Scenario: Slow queries (III)
A time filter is created for the conflicting time period
Now you can inspect all the metrics of the system during that
period
28
Use Case Scenario: Slow queries (IV)
The ‘State’ tab shows a clear increment of the ‘waiting requests’
29
Use Case Scenario: Slow queries (V)
A filter on ‘Requests’ Tab shows that a significant number of queries failed or finished with timeout in that period
We explore the data sources used by some of the failed queries
30
Use Case Scenario: Slow queries (VI)
The connection pool of the data source is frequently full during
the period -> requests are being queued
Suggests pool is too small or more queries than usual
31
Use Case Scenario: Slow queries (VII)
We continue the analysis and order the requests by client
application. We notice the application ’single_customer’ view
executed many long-running requests during the period
32
Use Case Scenario: Slow queries (VIII)
We examine the query plan of some of the queries and notice that
they are complex, long-running reports
33
Use Case Scenario: Slow queries (and IX)
Possible solutions:
Use Resource Manager to limit the maximum number of concurrent
requests from the “Single Customer View” application, so they do not
monopolize the data source
Ask the administrators of the “Single Customer View” application to
distribute the execution of reports in a longer time period
Cache part of the data to distribute the workload
Increase the size of the pool if the data source can support the workload
34
Workload Management
Ensure fair distribution of resources among applications / users
Allocate available resources according to business priorities
Sessions classified into groups according to criteria such as user/role,
application, time,…
Sessions are assigned to resource groups, which establish restrictions:
Apply always or only under heavy CPU usage
Example restrictions:
Change execution priorities
Max. number of concurrent queries
Limit execution time
36
■ Rules Classify Sessions into Groups (e.g. by user, application,…)
■ E.g. Sessions from application ‘single customer view’ are assigned to plan called ‘limit_concurrent_queries_to_10’
Denodo Resource Manager (II)
Thanks!
www.denodo.com info@denodo.com
© Copyright Denodo Technologies. All rights reservedUnless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.