42
Some things you should know about Virtualized Exalogic Jacco H. Landlust, Sr. Managing Consultant Because 10 is just an arbitrary number Amsterdam • June 2016
Max. width
Max. height
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Some things you should know about Virtualized Exalogic
Jacco H. Landlust, Sr. Managing Consultant
Because 10 is just an arbitrary number
Amsterdam • June 2016
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Jacco H. Landlust Sr. Managing Consultant @ ING Group Services Responsible for all Engineered Systems and adjacent Infrastructure for Finance & Risk Group Services Lives in Dalen, the Netherlands Dad, spouse, adventurer, procrastinator, Oracle ACE and general tech-head.
Narrator of this Technology Tale
2
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Introduction
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Oracle Exalogic Elastic Cloud*: complexity across the board
4 * Snapshot from some Oracle roadmap slidedeck
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Compute Power • 2 socket, 12-core, 2.7 GHz Intel Xeon Ivy Bridge
processors • 256 GB of RAM upgradable to 512GB • 720 cores in a Full Rack; also available in half, quarter,
eight or multi-rack configurations • Redundant SSD, power, cooling, InfiniBand (IB) adapter Internal I/O Fabric and Data Center Connectivity • 40 Gb/sec internal InfiniBand I/O backplane • 10 Gigabit Ethernet connectivity to datacenter Integrated Storage • Sun ZS3-2 Network Attached Storage • ZFS Clustered for high availability • 80 TB SAS disks, 6.5 TB read cache, 292 GB write cache • Clones, snapshots, remote replication
Converged Infrastructure: Example X4-2
5
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Infiniband Gateways • act both as InfiniBand “leaf” switches and
InfiniBand to 10 Gb Ethernet Gateways • # of gateways depends on rack size • Uplinks via 4 QSFP+ allowing for
16 10Gb uplinks
Infiniband Leaf Switches • 36 ports • Compute nodes are directly connected
Infiniband Spine Switch • Technically the same as leaf switch • no direct hosts attached but does have
inter switch links (ISL) to provide alternate paths or for expanding the fabric
6
Infiniband specific components
Diagram taken from Exalogic Elastic Cloud Multirack Cabling Guide: https://docs.oracle.com/cd/E18476_01/doc.220/e18481/eled.htm
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• 4 types of cables • Pass-through, optical core, QSFP – QSFP:
10m.
• Pass-through, copper core, QSFP – QSFP:
1 m, 2 m, 3 m, 5 m
• Splitter, optical core, CXP - QSFP x3:
10 m, 20 m
• Splitter, copper core, CXP - QSFP x3:
1 m, 2 m, 3 m, 5 m
• Especially relevant in multi-rack
environment, create cable plan to track hardware to switchport configuration (track cabling)
7
Plan your network topology carefully
Physical limitations
Image taken from Exalogic Elastic Cloud Machine Owner's Guide : https://docs.oracle.com/cd/E18476_01/doc.220/e18478/network.htm#ELMOG75637
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Management Network (Ethernet Admin) • 1GbE network connects all devices
through built-in switch (Cisco) Private InfiniBand Network (IPoIB) • Primary path for all traffic inside rack • Inter-Compute Node communication • Compute Node or VM to ZFS Storage
Appliance • Compute Node or VM to Exadata DB
Server node External Management and Client Access (10GbE Datacenter Interconnect) (EoIB) • Bridged from InfiniBand fabric to
Ethernet • No 10GbE switching in the rack, IB
endpoints presented as NICs to datacenter
8
So how does this link together?
Image taken from Oracle Exalogic Elastic Cloud Datacenter Network Integration: http://www.oracle.com/us/products/middleware/exalogic/eec-datacenter-network-integration-1405367.pdf
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
InfiniBand adapters provide a couple of advanced features over normal network adapters • Data transfers can be initiated directly from userspace to the hardware, bypassing the
kernel and avoiding the overhead of a system call • The adapter can handle all of the network protocol of breaking a large message (even
many megabytes) into packets, generating/handling ACKs, retransmitting lost packets, etc. without using any CPU on either the sender or receiver
• Driver creates a network interface for each InfiniBand port on the system, which makes an HCA act like an ordinary NIC
• Oracle virtualized these NICs, allowing for private cloud deployments
Protocol defines how to send IP packets over IB: • IPoIB: IP over InfiniBand, for IB to IB network communication • EoIB: Ethernet over Infiniband, for IB to non IB network communication
Now what is this “over IB” stuff?
9
The secret sauce of ExaBus
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Infiniband Node is any component that is connected to ExaBus (physical or virtual)
• Partition • Group of InfiniBand nodes that are allowed to communicate with each another
• Partition Membership • A limited member can only communicate with full members • A full member can communicate with limited and full members • InfiniBand nodes in Exalogic can be configured as either limited or as full member
• Infiniband nodes on Exadata are always full member of default partition (and currently
do not support additional partitions when Exadata is connected to Exalogic) • No network security between Exalogic (VMs) and Exadata
• Infiniband nodes on Exalogic are full member of default partition after initial installation • This includes newly deployed VMs • Membership can be changed after deployment, however not via GUI
Infiniband Nodes and Partition Membership
10
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Network Description
IPoIB-default Default IB partition
IPoIB-admin CN interconnects
IPoIB-storage CN to ZFS
IPoIB-virt-admin EMOC to VM
IPoIB-ovm-mgmt OVM heartbeat
IPoIB-shared-storage VM to ZFS
Virtualized Networks on InfiniBand fabric - Control Stack Networks created during
deployment - IB Partitions for network isolation - Managed Partition Keys (PKeys)
11
Networks all over the place
Internal Networks One or more EoIB Client Access Network must be created EoIB networks connecting datacenter VLANs to Exalogic VMs - Multiple VLANs can be connected - Exalogic will manage
IP address allocation - IB Gateways do not
provide routing functionality
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• The subnet manager (SM) manages all operational characteristics of the InfiniBand network • Discovering the network topology • Assigning a local identifier (LID) to all ports connected to the network • Calculating and programming switch forwarding tables • Programming Partition Key (PKEY) tables at HCAs and switches • Monitoring changes in the fabric
• Not all IB firmware versions are compliant, be alert when patching
• Exalogic IB gateways must run the Subnet Manager • Only IB nodes created on IB gateways are aware of other IB partitions than default
• Failover to non-Exalogic leaf-switch (e.g. on Exadata) when both IB gateways are down • When no Subnet Manager is running at all, IB fabric goes down
Manage the Subnet Manager
12
Be cautious in Multi-Rack environment
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• A virtual NIC or VNIC maps an Ethernet connector on the InfiniBand Gateway Switch to a network interface within the compute node or virtual host.
• One or more VNICs can be assigned to a compute node or virtual host.
• If more than one VNIC is assigned, they can be different connectors on the same Sun Network QDR InfiniBand Gateway Switch or on different Sun Network QDR InfiniBand Gateway Switches.
• The default configuration is to map one connector on each of the two switches and to bond or balance the pair for increased availability
• On each InfiniBand Gateway there can be a maximum of 1K VNICs per logical gateway
So how is Network configured on (virtual) hosts?
13
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Exalogic Configuration Utility, for assigning IP addresses to hardware components and vServer networks
• Modifying IP address ranges for internal networks after install is very hard
• Prepare for multi-rack (connect racks) and expansions • Reserve IP addresses and do not
overlap allocated addresses of components that could possibly have to connect in the future
• Best is to prepare an IP plan before filling in ECU
• Cannot multi-rack more than two virtualized Exalogic on one IB-fabric
14
Which IP addresses are available?
How to define IP ranges used
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Internet Protocol Address (or IP Address) is an unique address that computing devices. IP address has 2 components: network address and host address
• Subnet: logical, visible subdivision of an IP network.
• Subnet mask is a 32-bit mask used to divide an range of IP address into subnets and specify the network's available host
• First and last bit in a subnet automatically assigned and cannot be used: • First bit of subnet for network • Last bit of subnet for broadcast
• Classless Inter-Domain Routing (CIDR) as identifier to more efficiently allocate IP
addresses.
• A default gateway in computer networking is the node that is assumed to know how to forward packets on to other networks
IP, (Sub)Net Mask and Subnets are core knowledge
15
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
IP, (Sub)Net Mask and Subnets are core knowledge
16
An Example based on 10.0.0.0
Netmask Network Number of Hosts Start IP End IP Broadcast
255.255.252.0 = /22 10.0.0.0/22 1,022 10.0.0.1 10.0.3.254 10.0.3.255
255.255.255.0 = /24 10.0.0.0/24 254 10.0.0.1 10.0.0.254 10.0.0.255
255.255.255.224 = /27 10.0.0.0/27 30 10.0.0.1 10.0.0.30 10.0.0.31
255.255.255.255 = /32 10.0.0.1/32 1 10.0.0.1 10.0.0.1 10.0.0.1
Online IP calculators help, e.g. http://jodies.de/ipcalc
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Requirements • IP addresses for the new network you
are creating. Has to be in CIDR format • VLAN ID for the EoIB fabric for which
you want to create a network • Gateway for the network
• Do not attempt to create more than one
EoIB network using the same VLAN ID.
• Specify the range of IP addresses that may be used by vServers based • Exclude IP addresses that are reserved
for VIPs, gateways, etc.
• Be VERY careful when removing IP addresses or modifying networks
17
Manage external EoIB networks on Exalogic
Images taken from Oracle Traffic Director on Exalogic: http://www.oracle.com/technetwork/middleware/otd/learnmore/otd-exalogic-topology-whitepaper-2196722.pdf
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Although latency is low, the network is 10Gbit on VMs
• EoIB traffic between VMs sometimes travels outside of the rack • Prepare for internal and external
endpoints
• Cannot add or remove networks from VMs after creation • Prepare for rebuild of VMs in design
• Using multicast is harder than suggested • Routes for multicast traffic should be
added
# iperf3 -c IPoIB-internal-network
Connecting to host XXXX, port 5201
[ 4] local 1XXXX port 28166 connected to XXXX port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 944 MBytes 7.92 Gbits/sec 0 1.04 MBytes
[ 4] 1.00-2.00 sec 979 MBytes 8.20 Gbits/sec 0 1.16 MBytes
[ 4] 2.00-3.00 sec 1.21 GBytes 10.4 Gbits/sec 0 1.34 MBytes
[ 4] 3.00-4.00 sec 1.17 GBytes 10.0 Gbits/sec 0 1.34 MBytes
[ 4] 4.00-5.00 sec 1.16 GBytes 9.96 Gbits/sec 0 1.46 MBytes
[ 4] 5.00-6.00 sec 1.16 GBytes 9.95 Gbits/sec 0 1.46 MBytes
[ 4] 6.00-7.00 sec 1.15 GBytes 9.91 Gbits/sec 0 1.52 MBytes
[ 4] 7.00-8.00 sec 1.16 GBytes 9.99 Gbits/sec 0 1.52 MBytes
[ 4] 8.00-9.00 sec 1.20 GBytes 10.3 Gbits/sec 0 1.52 MBytes
[ 4] 9.00-10.00 sec 1.18 GBytes 10.1 Gbits/sec 0 2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 11.3 GBytes 9.68 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 11.3 GBytes 9.68 Gbits/sec receiver
18
What Oracle doesn’t highlight about vServers networks
Every product has some caveats
# iperf3 -c EoIB-public
Connecting to host XXXX, port 5201
[ 4] local XXXX port 13719 connected to XXXX port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 985 MBytes 8.26 Gbits/sec 0 792 KBytes
[ 4] 1.00-2.00 sec 1.04 GBytes 8.92 Gbits/sec 0 826 KBytes
[ 4] 2.00-3.00 sec 1.04 GBytes 8.91 Gbits/sec 0 905 KBytes
[ 4] 3.00-4.00 sec 1.04 GBytes 8.90 Gbits/sec 0 905 KBytes
[ 4] 4.00-5.00 sec 1.01 GBytes 8.65 Gbits/sec 0 905 KBytes
[ 4] 5.00-6.00 sec 1.02 GBytes 8.72 Gbits/sec 0 905 KBytes
[ 4] 6.00-7.00 sec 1.01 GBytes 8.65 Gbits/sec 0 908 KBytes
[ 4] 7.00-8.00 sec 1.01 GBytes 8.65 Gbits/sec 0 908 KBytes
[ 4] 8.00-9.00 sec 1.01 GBytes 8.64 Gbits/sec 0 909 KBytes
[ 4] 9.00-10.00 sec 1.01 GBytes 8.65 Gbits/sec 0 909 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.1 GBytes 8.70 Gbits/sec 0 sender
[ 4] 0.00-10.00 sec 10.1 GBytes 8.69 Gbits/sec receiver
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
vrrp_instance VRRP1 {
state MASTER
# Specify the default network interface, used for cluster traffic
interface bond2
# The virtual router ID must be unique to each VRRP instance that you define
virtual_router_id 41
# Set the value of priority higher on the master server than on a backup server
priority 200
advert_int 1
authentication {
auth_type PASS
auth_pass 1066
}
virtual_ipaddress {
# add dev to route traffic via a non-default interface
10.XXX.XXX.XXX/22 dev bond0
}
}
Multicast & Oracle Traffic Director
19
Setup configuration for keepalived manually
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Cloud Management
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Exalogic Control is the built-in facility for • System management • vDC management & Cloud Administration • User and Account management • Monitoring • Has GUI and CLI (IaaS-api) Exalogic Control consists of: • Enterprise Manager Ops Center (EMOC) • Oracle VM Manager (OVMM) • DB repository for EMOC and OVMM • Runs on on 16 vCPU and 32 GB RAM
• 2 Enterprise Controllers a.k.a. Proxy Controllers (PC) • Runs on 2 vCPU and 4 GB RAM each
• Optional Oracle Virtual Assembly Builder (OVAB) Deployer • In isolated OVM repository from guest VMs
Exalogic Cloud Management and Monitoring
21
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Only one vDC per Exalogic • Overview of the server, storage, and network resources • Physical resources entitled to accounts as virtual
resources under quotas set by the cloud administrator • Accounts provide the required capabilities to manage
the allocated resources. • An account entitles designated cloud users the
privilege to use its virtual computing, network, and storage resources.
• A cloud user with access to different accounts can create virtual servers, known as vServers, and host or access application
• A cloud administrator is involved in setting up the infrastructure for the vDC, creating and managing vDCs, creating and managing accounts, and managing access to cloud users.
vServers have to run in a container
22
vDC: Virtual Datacenter
Image taken from Exalogic Elastic Cloud Administrator’s Guide https://docs.oracle.com/cd/E18476_01/doc.220/e25258/proc.htm
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Typical networks founds in VMs • IPoIB-Default • IPoIB-shared-storage • IPoIB-virt-admin Additional networks in VMs • Client access • Cluster traffic
Rack CN CPU RAM GW
Eight 4 144 1TB 2
Quarter 8 288 2TB 2
Half 16 576 4TB 2
Full 30 1080 7.5TB 4
23
Lets do some Math
Taking resource limits into account on X5-2
Other limits to take into account: • 80TB RAW Storage, leading to
effectively ~36 TB usable • 20 vCPU and 40 GB RAM for Exalogic Control • 1k VNICs per IB gateway
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• vCPU to CPU threads ratio impacts how many VMs you can deploy • vDC wide setting; this impacts all vServers, those previously created and those still to
be created. • Recommendation is not to make this ratio any larger than about 1:4 as a small vServer
(with just 1 vCPU) when under contention the lack of compute power can lead to instability and timeouts.
• CPU Cap is an absolute limit on the amount of CPU time that a vServer can be allocated,
it is defined as a % of a vCPU • Effectively this will reduce the power of your vServers but delay the time at which the
Xen scheduler is used to control vServer access to physical CPU • Stored per VM, so changing CPU Cap only impacts new vServers
• No CPU pinning on Exalogic • Trusted partitions can be managed via Oracle Enterprise Manager’s harvester to create
license report
CPU allocation on Exalogic
24
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
EMOC Users (Cloud User or Cloud Admin) • Each Local User ID must exist on EMOC VM with a valid password • Local User can be /etc/passwd or NIS if configured • Directory Servers support exists if needed Pre-requisites for Cloud User to create a vServer • Cloud User must be associated to an Account (EMOC “Account”) • An account must have adequate resources (e.g. vCPU’s, Memory, Networks, Storage) • There must be a supported vServer Template • Can create custom templates or download pre-configured templates from
MOS/edelivery • When BYO, mind the yum exclusion list
• The vDC must have enough capacity
EMOC & Exalogic Virtual Data Center (vDC)
25
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Start with the default Exalogic Base Guest Template for App deployment – can be modified too (WARNING: this has impact on backup via ExaBR)
• Add default vServer instance type – EXTRA_LARGE (16GB RAM, 4 vCPUs), LARGE (8GB RAM, 2 vCPUs), SMALL (4GB RAM, 1 vCPUs) or a user created type
• Resize vServers since 2.0.6 PSU Oct. 2015
• vNETs cannot be modified / added / removed after VM creation
• Distribution groups help with anti-affinity i.e. places vServer on separate physical nodes for redundancy
Template
Instance Type
Volumes
vNets
Distribution Group
26
vServer Lifecycle
Compute Node
Compute Node
Compute Node
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Starting, stopping and deleting a vServer • Must perform these tasks via Exalogic Control (BUI or CLI) • Do NOT use OVMM unless instructed by qualified Oracle engineers, this can impact
support • Do NOT use xm commands
• Can customize vServer to • Increase the root volume • Increase swap space • Create and Customize VM Templates:
http://docs.oracle.com/cd/E18476_01/doc.220/e25258/app_vm.htm#CHDJDJBB • Impact on backup tools
vServer Lifecycle
27
Start, Delete, Stop, Run, Customize
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• A vServer has many networks – system and user defined
• To access it from external network, it must be on a EoIB client access network
• If it does not have EoIB network, it must be accessed from another vServer that has access to both EoIB and IPoIB networks within the same account
• “Launch Virtual Console” from EMOC is particular about JDK and Security Exception • Use Java 1.7.45 or older (i.e. Java 6) • Add http://<ECVM>:58333 to the
Exception Site List in the client’s Java Control Panel
28
Logging into a User’s vServer
Know the vServer networks
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• vServer Volumes I/O traffic go through OVS • Under the covers NFS mounted by the
hypervisor
• vServer NFS traffic has direct communication via Virtual HCA • Allows for “clean” topology with all
data via NFS shares • Allowes for security on network access
(via exceptions) and ACL’s
29
vServer Storage
Volumes vs NFS
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Use ZFS’ IPoIB IP address along with recommended parameters for NFS mounts • Use the IP address of the proper partition
• Exalogic is a multihomed system where multiple IP addresses are configured on the InfiniBand network interface
Using NFS to mount ZFS shares
30
NFS v3 versus v4
NFS v3 is a stateless protocol NFS clients do not maintain the state between requests; the clients rely on the Network Lock Manager (NLM) protocol to support file locking. There are known limitations on the NLM based locking.
NFSv4 is a stateful protocol An NFS client obtains a client ID assigned during the initial negotiation phase and uses it for all subsequent requests. With the use of Client ID, the client IP address is not used in the file locking process. Therefore, NFSv4 addresses file locking problems on multihomed systems In addition, with locking built into the protocol, NFSv4 provides better lock performance than NFSv3 NFSv4 requires a naming service, such as Network Information Service (NIS) or LDAP, for ID mapping
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• NFSv4 requires a naming service for ID mapping • Network Information Service (NIS) or LDAP
• Can use existing naming service on the data center network
• The naming service is not related to the file locking problem
• If NIS or LDAP need to be configured: • Chapter 9 Exalogic Machine Owner’s Guide (E18478) • How To Configure NIS On Exalogic (Doc ID 1386387.1) • Oracle Exalogic Elastic Cloud - Setting Up LDAP Service for NFSv4 (Doc ID 1599868.1)
Naming Service for NFS v4
31
NIS, LDAP, etc.
Max. width
Max. height
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Tooling & Troubleshooting
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Designed to audit important configuration settings
within an Oracle Exalogic machine.
Exachk examines the following components:
• Compute nodes
• Storage servers
• InfiniBand fabric
• Ethernet network
• Control vServers
The tool audits the following configuration settings:
• Hardware and firmware
• Operating system kernel parameters
• Operating system packages
33
Oracle provided Tooling
1449226.1: Exachk
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• ExaPatch
• ExaPasswd • Warning: has been given multiple
issues in the past
• STIGfix • Warning: rollback before each patch
on Infra required
• ModifyLVMImg • Breaks with every OL upgrade • Mostly obsolete because of EKIT
• EMAgent PreSetup
• ExaLogs • Output can be very verbose, learn how
to limit output to relevant components
• ExaBR • Not so useful for backup of running
VMs because of snapshot method • Very useful for backup of firmware,
configuration and control-stack • Careful when using with custom VM
templates
34
Oracle provided Tooling
1912063.1: Exalogic Lifecycle Toolkit
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• CLI is maintained by product development
• Infrastructure As A Service (IaaS) command-line interface
• IaaS CLI offers the same functions and features as Cloud Management API
• CLI is installed on a machine outside of the Exalogic machine. This remote machine must have network access to Exalogic Control.
35
Exalogic vDC Management Using IaaS CLI
Manage the vDC from a remote machine
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• ListAccounts • ListDistributionGroups • ListNetworks • ListTemplates • ListVolumes • ListVServers • UploadTemplate • ListVServerTypes • ResizeVServers • StartVServers • StopVServers
Maintained by A-team. Functionality: • MountSystemImg • RenameRunningVServerLVM • CaptureAccount • CaptureVServer • CreateDistributionGroup • CreateNetwork • CreateVolume • CreateVServer • DeleteDistributionGroup • DeleteNetwork • DeleteTemplate • DeleteVolume • DeleteVServer
36
Oracle provided Tooling
1933252.1: Exalogic Kinetic Infrastructure Tools (EKIT)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• All compute nodes must be at the same software/firmware versions • Latest PSU, Infrastructure, ILOM, Operating System, Exalogic software, etc. • dom0 settings Exalogic Virtual dom0 Memory Recommendations (Doc ID 1582091.1)
• EoIB, IPoIB and eth-admin networks are plumbed and accessible • ping, ssh • All network services are valid: Gateway, DNS, NTP, NIS/LDAP, etc.
• Use the dcli command to execute on multiple compute nodes • Create a file containing all compute node host names or IP’s: e.g. cnodes.txt • Example: Verify that all nodes have the same expected release version/updates
/opt/exalogic.tools/tools/dcli –g cnodes.txt ‘imagehistory’ • Healthcheck • How to Perform a Healthcheck on Exalogic (Doc ID 1329262.1) • Best is to perform exachk
Component verification
37
Compute Node
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• Both storage nodes must be at the same software/firmware versions • EECS version, PSU • ILOM, ZFS software versions
• Storage nodes are in Active-Standby mode • configuration show cluster
• All necessary services are running • NFS (v3/v4), iSCSI, HTTP, akshd, RAID-1 • Do NOT turn off factory default settings
• All network interfaces and data links are up and running • Total storage utilization should be < 80% • Utilized powerful CLI for automation of management and monitoring
Component verification
38
Storage Node
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• All IB switches must be at the same software/firmware versions • version
• Only one NM2-GW switch is the Subnet Manager (SM) master • getmaster • smnodes list (should show only the IB-GW switches)
• When connected to Exadata the Subnet Manager (SM) master should reside on a Exalogic leaf switch
• All necessary VLANs, VNICs and partitions are configured • showvnics • showvlan • smpartition list active
• All ports are up? Fabric Monitor provides visual • Enough free disk space?
Component verification
39
Gateway Switch
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Questions?
40
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
• https://docs.oracle.com/cd/E18476_01/doc.220/e18478.pdf • http://www.oracle.com/us/products/middleware/exalogic/eec-datacenter-network-
integration-1405367.pdf • http://exablurb.blogspot.nl/2013/10/exalogic-and-cpu-oversubscription.html • Operations Runbook For Exalogic Elastic Cloud Software 2.0.6.x.x (Doc ID 1930293.1) • Exachk Health-Check Tool for Exalogic (Doc ID 1449226.1) • Exalogic Lifecycle Toolkit Release 14.2 (Doc ID 1912063.1) • Setting up the subnet manager in a multirack configuration containing Exalogic/BDA and
Exadata/SSC/Expansion Rack (Doc ID 1682501.1) • Oracle Exalogic Elastic Cloud - Credentials Management Guide (Doc ID 1594316.1)
References
41
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Follow us to stay a step ahead
ING.com
YouTube.com/ING
SlideShare.net/ING @ING_News LinkedIn.com/company/ING
Flickr.com/INGGroup Facebook.com/ING
