Download - Rock Solid Sametime for High Availability

© 2013 IBM Corporation

Rock Solid Sametime For High AvailabilityBP301: Rock Solid IBM Sametime For High AvailabilityGabriella DavisThe Turtle Partnership

Gabriella Davis

§Proud Nerd Girl–Mathmo / Problem Solver / System Designer / Optimist

§ccMail & Agenda then Lotus & WAS–I’m much older than I look

§Co-Author of Sametime 8.5.2 Admin Guide–Available at all good bookshops but mostly Amazon–Domino & Exchange, Sametime & Lync Server, Sharepoint

§Co-Author Connections101.net (being updated for Connections 4)

§ I present a lot globally & blog in fits and starts

§The Turtle Partnership–High Level Support of IBM Lotus products• 20% support, 40% system design and implementation, 40% development• 50% of our customers are in Europe and 50% in the US (nothing against Australasia mind

you)

2

There’s Nothing Sexy About High Availability

§Believe me I tried to get as many jokes in as I could

§There’s high availability and there’s disaster recovery and there’s load balancing and a slew of options in between

§Your Sametime install could be 3 servers or it could be 50 servers.–I’ve done both ends of the scale

§Build for now but design for the future–Adding new servers or services later on can be a simple process if you’ve designed correctly

from the start

3

How This Presentation Works

§There’s no single right way to do this, it depends on what services are critical to you and what demand you expect for each service

§My goal is to take you through each Sametime service and explain how you would design each for clustering, load balancing and failover

–I use lots of pictures so we can all visualise what I’m talking about–also hand puppets

§ I also want to talk about the cool things that you may not know happen under the hood and some things you need to be careful of

§There’s a lot of information here and I’m going to have to go fast so grab the presentation and feel free to ask questions afterwards !

4

Sametime Servers and Services

§ Instant Messaging

§Web clients, mobile clients, web based awareness

§Meetings

§Audio and Video services

§Audio / Video reflector services (for A/V across networks)

§Audio / Video traffic management

5

Sametime System Console

§The SSC is the management and administration environment for all Sametime servers and services

–You can build without a SSC but I don’t recommend it unless you love messing about with XML files

§The SSC is a WebSphere based application and cannot be clustered

§Servers within the same SSC are aware of each other–The Meeting servers know about the Media servers etc

§There can only be one cluster of each type in a SSC–One meeting cluster–One sametime proxy cluster–One proxy registrar cluster etc

–THIS IS GOING TO BE CRITICAL LATER ON WHEN WE TALK ABOUT PLANNING

§You can’t cluster the SSC (it was worth repeating)

6

Database Server§Several of the Sametime servers use a DB2 database for management

–Meetings–Gateway–System Console–Bandwidth Manager–Advanced

–even the Proxy server has a DB2 database it uses for iPhone traffic

§DB2 has several high availability models but the DB2 9.7 license supplied with your Sametime licensing entitles you to use DB2 in a HADR configuration with a single active and another passive server

–The switchover from active to passive is manual–The passive DB2 server cannot be used or accessed but will take updates from the active

server and can itself be made the active server at any time

§Without HADR you could lose your DB2 server–Instant Messaging, Audio and Video as well as Web / Mobile clients

will continue to work7

Edge Load Balancer

§Licensed for use with Sametime

§Very easy to deploy and configure on a wide range of platforms with a GUI interface for management

§Always use the new ULB IPV4/IPV6 load balancer and not the older IPV4 one which is being deprecated

§To set up a server for load balancing you need to –Assign a virtual ip address that can be used by each cluster of servers and that points to the

ULB machine• So if you’re load balancing Meetings, Sametime Proxy, Conference Manager, Proxy

Registrar, Packet Switcher and TURN server you need 6 separate unique virtual ips all pointing to the load balancer

–Create a Loopback adapter on your backend servers (ie your Meeting, Media etc servers) configured to use the virtual ip address

–Create routing rules in the ULB (Edge Load Balancer) wizard to forward traffic to the backend servers

8

Load Balancer - Tips

§You can use any load balancer to manage traffic to the Meeting, Sametime Proxy, Gateway, Advanced and Community Servers but the Audio / Video components must be configured to use MAC forwarding

§For many load balancer appliances that’s a network wide configuration setting that administrators don’t like to do (I have been told this many times :-))

–In those cases you could deploy the Edge LB to handle the A/V traffic

§To configure a loopback adapter on Windows 2008 you need to go to Device Manager and right mouse click on the server name then choose “Add Legacy Hardware” - you can then go ahead and add a loopback network adapter by choosing Microsoft as the provider

–There are specific network configuration settings that need to be run on a Windows 2008 server to set up the loopback adapter correctly • http://www-01.ibm.com/support/docview.wss?uid=swg21304795

9

Domino Clustering

§What does Domino clustering give us in a Sametime world?–vpuserinfo (contact list and privacy information) replicated between servers in seconds–directory information replicated between servers, if you set it up correctly• Beware of using a replica of Directory Assistance on multiple servers without proper

configuration–don’t cluster stconfig.nsf, its information is server specific

§ It can’t replicate Sametime specific information such as policies (which are held in stconfig.nsf and business card configuration which is held in the file userinfoconfig.xml)

10

Domino Clustering for Sametime - Tips

§ If you’re running the Sametime System Console, it controls Sametime configuration settings in stconfig.nsf including policies, trusted ips and LDAP configuration

§Changing the LDAP deployment in the SSC will cause it to overwrite the LDAP settings in stconfig.nsf when Domino next restarts

–but only once, if you then edit it the LDAP document in stconfig.nsf via Notes, it won’t be overwritten

–Make sure you edit the LDAP and Policies documents always in the same place - the SSC, don’t be tempted to quickly edit a Notes document in stconfig.nsf as the settings may not hold

11

Domino Clustering for Sametime - Weirdness

§ If you have multiple LDAP configurations in stconfig.nsf the priority is NOT the “Search Order” as defined on the LDAP document but is the order in which the LDAP documents appear in the view (by modified date).

–This priority determines which directory is searched first when finding contacts

§To ensure this doesn’t happen you’ll need to change the design of the “All - by form and date” view in stconfig.nsf to sort LDAP documents by Search Order field

The search order says that we want this directory searched 2nd

but because it was modified most recently, it will be searched first

Sametime Clustering

§What does Sametime Clustering give us?–The Community Server is central to everything Sametime does. –None of the other components function without a Community server to talk to–This makes clustering and availability of Community servers the most important component in

your design

–Users can log on to any Community server in the cluster and expect the same experience• Assuming you have your Domino clustering and Sametime customisation set up correctly

✦vpuserinfo.nsf replicating✦sametime.ini with same settings✦userinfoconfig.xml the same✦LDAP configuration in stconfig.nsf the same

–Other Sametime components such as the Sametime Proxy or Meeting servers will be able to recognise a cluster mate as a failover alternative

13

Sametime Clustering - Tips

§Verify Sametime.ini settings such as security level and allowable clients match on all cluster-mates

– VPS_ALLOWED_LOGIN_TYPES– VP_ONLY_SINGLE_LOGIN_ALLOWED– VP_SECURITY_LEVEL

§Don’t have one cluster server running a different version of Sametime than its other cluster-mates

§Users with no Sametime Server defined in their person document have no home Sametime server and can log into any server in the Community (ie in the Domino Directory)

–Users with a cluster name in the Sametime Server field in the person document can log into another server in the cluster

–Use an LDAP attribute to store the home Sametime server name or cluster name for a user so it’s accessible to other products such as Connections• The sametime server name should be in hierarchical format eg cn=st1,o=turtlehost or as a

cluster name cn=stcluster

14

Sametime Clustering - Weirdness

§Trusted ips in the stconfig.nsf determine what servers can connect to this Sametime server

–These might be other Sametime servers or Domino servers using iNotes or Sametime Proxy servers or Meeting or Media servers• The list can get pretty long

§The list of ips is updated in the Sametime System Console if you have one and this overwrites the configuration document in stconfig.nsf

–Don’t edit the list in stconfig.nsf if you are running a SSC as your edits will be overwritten

§One bug I continually see is when the trusted ips list in the SSC gets very long, it writes out to the Community Trusted IPs field in the Community Connectivity document as a long string instead of a list

–This breaks all trusted ip connections–You can open and save the Notes document to fix it, saving recalculates the field correctly as

a list–It will break again next time you update the SSC

15

Community Multiplexors - Multiple Multiplexors!

§The Community Multiplexor (MUX) is a service running as part of your Sametime Community Server

–It’s responsible for authenticating users and connecting them to the back end Sametime services

–In a standard install the MUX runs as a service on the Community Server–You can install a Community MUX separately from the Community server to offload the

authentication and client network connections to another machine• The MUX maintains a single network connection to the Sametime Community Server

increasing the capacity of the server to 100k concurrent users

§A MUX can be used to front-end multiple servers in a cluster, the assumption being that users can connect to any and all of the listed servers

§The sametime.ini file in a MUX contains a list of servers it can connect to[Config]VPMX_CAPACITY=80000[Connectivity]VPS_HOST=stchat1.turtlehost.net,stchat2.turtlehost.net, stchat3.turtlehost.net

16

Community Multiplexors - Tips

§You don’t have to have a separate multiplexor and if you’re going to only have one it itself becomes a single point of failure

§ It used to be that that MUX’s were ‘dumb’ in that they would round robin connections to their known servers, trying each one in turn, even if that server is down

–That’s no longer the case and the MUX’s are now service aware and will not route traffic to unavailable servers

–They have become load balancers for Sametime Community Server traffic

§Everything in Sametime world pivots around DNS and fully qualified hostnames, the hostname your Sametime clients ask for must eventually resolve to a Multiplexor somewhere

–If you have multiple multiplexors you will therefore need to use a load balancer to make sure traffic isn’t sent to a non responding multiplexor

17

So What Does It All Look Like?

18

Instant Messaging (Sametime Clients) Clustering

§Sametime servers clustered, users can access either server but the FQHN defined in their client connections will determine here they end up

§There is no failover in the Sametime client

19

Domino Cluster

Domino Server A Domino Server B

Sametime ClusterSametime Server

stchat1.connect13.comSametime Server

stchat2.connect13.com

Instant Messaging (Sametime Clients) Failover

§Load Balancer in front of Clustered Domino servers

§Users are directed to either server and get the same experience

20

Domino Cluster





Load Balancerstchat.connect13.com


§Use a Multiplexor instead of a Load Balancer in front of your Sametime cluster and take advantage of the failover logic built into it

§Define multiple servers in the sametime.ini file in the Multiplexor

21

Domino Cluster





Sametime Multiplexorstchat.connect13.com


§ If you don’t want your multiplexor to be a single point of failure then put multiple multiplexors behind a load balancer. The client configuration FQHN points to the load balancer and the multiplexors connect to either Community server

22

Domino Cluster





Sametime Multiplexorstmux1.connect13.com

Load Balancerstchat.connect13.com

Sametime Multiplexorstmux2.connect13.com

Instant Messaging (Web and Mobile) Clustering

§Multiple Community Servers

§Single Sametime Proxy Server

§Sametime Proxy Server can be directed to a single Community Server but can and will utilise any Community server it is trusted for in your Domino Directory

23

Sametime Community ServersNot Necessarily Clustered





Sametime Proxy ServerIBM WAS HTTP Proxy

stproxy.connect13.com (80/443)

Instant Messaging (Web and Mobile) Clustering

§Clustered Community Servers

§Single Sametime Proxy Servers

24

Sametime Community ServersCLUSTERED





stweb.connect13.com (80/443)

Instant Messaging (Web and Mobile) Failover

§Clustered Community Servers

§Multiple Sametime Proxy Servers (not clustered)

§Load Balancer

25

Sametime Community ServersCLUSTERED





stweb1.connect13.com (80/443)


stweb2.connect13.com (80/443)

Load Balancer

WebSphere Clustering

§Each Sametime server (other than the Community Server) is installed on a WebSphere node.

§Each node must have a single primary instance but can have multiple secondary instances installed as part of a cluster in either

–Horizontal (servers installed on different machines)–Vertical (servers installed on the same machine but uses their own dedicated resources)

§WebSphere has built in logic to share resources across Sametime servers in a cluster–You don’t have to configure it to do that

§BUT, the Sametime System Console can only manage 1 cluster of each server type

26

WebSphere - Tips

§Attempting to deploy multiple applications on a single server is a very bad idea

§Each application (Meetings, Media etc) must have its own FQHN and dedicated ip address

§To ensure WebSphere binds the correct ip address to the correct application (and avoids port conflicts with other applications already installed) you have to explicitly edit the underlying XML files to creating the hostname bindings

§Even then WebSphere’s will bind the bootstrap port to the first ip address on the box

§We’ve also discovered that all traffic sent from any of the servers has a source ip of the first ip on the box (Windows 2008 R2)

§ In general it’s to be avoided. It can be done but Virtual Machines is a far cleaner, easier and more reliable way to go

27

WAS Proxy Clustering

§The WebSphere WAS Proxy server can act as both a HTTP or SIP proxy to sit in front of your Sametime servers

§Each Sametime server installs on its own application port (default is 9081 / 9043 secure but that will increment by one for every other server installed on the same machine that conflicts)

§We don’t want to construct URLs that have port numbers in them so for HTTP services we use a WAS HTTP Proxy server in front of the Sametime application to manage the traffic on port 80 /443 secure

§They are cluster aware, able to identify when an application server is unavailable and redirect the traffic to that server’s cluster mate which provides built in failover

§The WAS Proxy is often installed on the same node and server as the application server it is managing but that doesn’t have to be the case, it can install standalone

28

WAS HTTP Proxy - Tip

§The WAS HTTP Proxy is much more than a reverse proxy–It is part of the Sametime System Console and will authenticate users before directing the

traffic onto a single destination - the application server it is supporting–A reverse proxy doesn’t have the same intelligence to validate that traffic should be allowed

through and should only be allowed through to a specific destination

§We primarily use WAS HTTP Proxies with Meeting and Sametime Proxy servers

§You can have multiple WAS HTTP Proxies providing service to the same application server

–You can cluster WAS HTTP Proxies and then they will also monitor each other

§Don’t create WAS HTTP Proxies with the SIP checkbox enabled, we use SIP for Audio / Video traffic and it can cause conflicts in the environment with other servers if you leave it enabled

29

Meeting Server Clustering

§Meeting Servers can be clustered in Sametime by installing a single server as a primary node and then a series of secondary nodes either on the same machine (vertical clustering) or, more usually, on multiple machines (horizontal clustering)

§All Meeting servers in a cluster share the same DB2 database

§ In a Meeting server cluster WebSphere treats all the servers as equal

§When a user creates a meeting WebSphere will decide which of the Meeting servers will host that meeting

–This is very important as all users attending the meeting will be directed to the same host meeting server

–it is outside of your control to determine which server will be selected

30

Meeting Server Clustering - Tips

§ If you build a Meeting Server cluster with servers in France, Singapore and Chicago - a meeting of users in Chicago could just as easily be directed to France

–WebSphere will choose which cluster-mate hosts each meeting when the meeting is first created

§The WAS HTTP Proxy will be able to detect if the host meeting server is down and redirect all user requests to an alternate server by rewriting the URL

§Don’t create a meeting server cluster with one server inside a firewall and one outside unless you are prepared to open ports to allow public access to your internal servers

§Consider your network architecture when designing your meeting servers, if you want multiple clusters you will need to have multiple System Consoles

§The Community Server a user logs in to determines their environment from SSC to Meeting, and Media Servers

–but users can log into different environments depending on where they are

31

Meetings Multiple Servers

§Three Meeting Servers, all independent, none clustered, each with their own DB2 database (and different DB2 servers if you want)

§Meetings will be created on a specific server as determined by the user creating the meeting “where do you want to create this meeting”

§The WAS HTTP Proxies are to ensure servers are accessible on 80 /443

32

stmeet1.connect13.com



Sametime Meeting Servers

IBM WAS HTTP Proxy

IBM WAS HTTP Proxy

IBM WAS HTTP Proxy

DB2 Active Server

STMeet Database(s)

DB2 Passive Server

STMeet Database(s)

Meetings Clustering

§Three Meeting Servers, this time in a cluster

§Meetings are created on any of the three servers, determined by WebSphere and outside of user control

§The WAS HTTP Proxies can proxy traffic for only one server each

§ It’s your decision if you want to separate servers and proxies onto their own machines, you don’t have to

33



Sametime Clustered Meeting Servers


IBM WAS HTTP Proxystmeet.connect13.com

DB2 Active Server

STMeet Database(s)

DB2 Passive Server

STMeet Database(s)

IBM WAS HTTP Proxystmeet.connect13.comIBM WAS HTTP Proxy

stmeet.connect13.com

Meetings Failover

§With the WAS HTTP Proxies themselves clustered together, each one can provide service for any of the Meeting servers that are clustered

34

Clustered Meeting Servers




Cluster WAS Proxy Servers


DB2 Active Server

STMeet Database(s)

DB2 Passive Server

STMeet Database(s)


Load Balancer

Media Manager

§Media Manager - contains all media components (other than Bandwidth Manager)–Conference Manager–Proxy Registrar–Packet Switcher

§Up to two Media Managers can be clustered, but only two and only one cluster in a SSC

35

Sametime Media ServersCLUSTERED

stmedia1.connect13.com stmedia2.connect13.com

WAS SIP Proxy

§When deploying a Proxy Registrar or Conference Manager as standalone components, each needs to have a dedicated WAS SIP Proxy server to handle traffic between the servers themselves and to / from the clients

§WAS SIP Proxies aren’t clustered and are deployed usually in a 1 - 1 relationship with the application server

–However any of the servers can provide Audio / Video service to users so it doesn’t matter which server a request is directed to

§The Packet Switcher component cannot be clustered and does not require a WAS SIP Proxy

§When creating a new WAS Proxy server for Audio / Video components you deselect HTTP and select SIP to tell the newly created server it will be handling SIP traffic only

§WAS SIP Proxies deploy on the standard ports of 5062 (SIP) and 5063 (SIPS). –These may be incremented by 1 if there is conflicting port activity on the same server

36

Audio Video Components - Clustered

§Some Individual components can be clustered–Proxy Registrar–Conference Manager

§Packet Switchers can’t be clustered but there can be multiple PS instances

§SIP Proxies must be installed in front of the Proxy Registrar and Conference Managers

37

Sametime Media ComponentsCLUSTERED

stcm1.connect13.com

stcm2.connect13.com

IBM WAS SIP Proxystcm.connect13.com

stpr1.connect13.com

stpr2.connect13.com

IBM WAS SIP Proxystpr.connect13.com

stps1.connect13.com

stps1.connect13.com

stps1.connect13.comProxy Registrar Cluster

Conference Manager Cluster

Multiple Packet Switchers (NON CLUSTERED)

Audio Video Components - Failover

§ In this design I have clustered the SIP Proxies so they provide failover for each other via a Load Balancer

§This isn’t strictly necessary if you have a Load Balancer in front as it should be able to detect an unavailable SIP Proxy and redirect the user requests accordingly

38


stcm1.connect13.com

stcm2.connect13.com

IBM WAS SIP Proxystcmsip1.connect13.com

stpr1.connect13.com

stpr2.connect13.com

IBM WAS SIP Proxystprsip1.connect13.com

stps1.connect13.com

stps1.connect13.com



Multiple Packet Switchers (NON CLUSTERED)IBM WAS SIP Proxy

stcmsip1.connect13.com


SIP Cluster

Load Balancer

SIP Cluster

Audio Video Failover - With Bandwidth Manager§Bandwidth Manager should be deployed in any environment where there is a significant

requirement for Audio / Video or where the user network is not consistent

§The Bandwidth Manager cannot be installed in the same cell as the Media components because of conflicts with their SIP configuration

39


stcm1.connect13.com

stcm2.connect13.com

IBM WAS SIP Proxystcmsip1.connect13.com

stpr1.connect13.com

stpr2.connect13.com


stps1.connect13.com

stps1.connect13.com



Multiple Packet Switchers (NON CLUSTERED)IBM WAS SIP Proxy

stcmsip1.connect13.com


SIP Cluster

Load Balancer

SIP Cluster

Bandwidth Manager Cluster

Separate Cell

stbwm1……

stbwm2……

WAS SIP Proxystbwmsip1

WAS SIP Proxystbwmsip2

BWM Cluster

SIP Cluster

Sametime Advanced

§Sametime Advanced can also be deployed as a cluster of multiple servers

§Each cluster-mate will use the same DB2 database

§We don’t deploy a WAS HTTP Proxy with Sametime Advanced because although many of the applications are on HTTP ports (9080 / 9443), the Broadcast Tools are on ports that cannot be proxied (1883, 8883)

§Since we can’t use a proxy in front of Sametime Advanced servers, if you want to access them from a public network, they must be located in a DMZ or somewhere you are confident exposing to a public interface

40

Clustered Advanced Servers

stadv1.connect13.com

stadv2.connect13.com

DB2 Active Server

STMeet Database(s)

Load Balancer

DMZPublic Internal

TURN Servers

§TURN Server handle audio and video traffic being redirected between clients on different networks

–Similar to the old reflector service on pre 8.5 Sametime

§The TURN Server is defined by its hostname in the configuration of the Media Components in the SSC

–A user is assigned media components based on which Community server and therefore SSC environment they log into

–Meetings don’t use the TURN server

§You can have as many TURN servers as you want fronted by a Load Balancer–users involved in a conference don’t need to be on the same TURN server

41

TURN Server - Tip

§ If you deploy TURN configuration on your Media servers then the FQHN you use for the TURN server has to be resolvable for all clients, both internal and external

§Multiple TURN servers on different networks can cause Audio and Video latency issues if the users involved in a conference are directed to different TURN servers such as one internal and one in a DMZ

§All users must be able to access the TURN server on port 3478 either UDP or TCP (whichever you’ve configured)

–UDP is more efficient than TCP for routing this type of traffic

42

Public Access, Firewalls and DMZ

§The biggest issue you may find with deploying high availability Sametime is in deciding how access will work

§ If you’re prepared to deploy a VPN to all users then you are basically building an internal only environment

§ If you want to deploy publicly then there are a few things to consider–In the DMZ we deploy the WAS proxy elements only ie the WAS HTTP Proxy that then

accesses the Meeting servers on the internal network–WebSphere has about 20 ports that need to be bi-drectionally open between the SSC and all

other server components, even those in the DMZ–In addition each server has to be able to connect to each other on discovery ports, there are

about 10 of those that need to be open and accessible between every server in the environment

–For public access both the Sametime Gateway and Sametime Advanced applications must be in the DMZ as their traffic can’t be proxied

§Think about how iPad, Android, Smartphone access will work - will have need to have WAS HTTP Proxies to the Sametime Proxy servers in the DMZ to support them?

–For notifications to iOS devices you’ll also need to open access to the apple gateway43

Thank You !

§Gabriella Davis–[email protected]–blog.turtleweb.com–www.twitter.com/gabturtle–bleedyellow.com (IM) lotuslive.com (IM) greenhouse.com (IM)–gabrielladavis (skype)–www.turtlepartnership.com

44

45

Legal Disclaimer

© IBM Corporation 2009. All Rights Reserved.

The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

IJava and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.