Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks Designed for organizations implementing Exchange on Lenovo servers and network Includes sizing recommendations for servers, storage, and networks Describes a low cost, highly available, site-resilient, simplified deployment Contains detailed Bill of Materials for servers, storage and networking Roland Mueller Last update: 16 October 2015
Transcript
Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Designed for organizations implementing Exchange on Lenovo servers and network
Includes sizing recommendations for servers, storage, and networks
Describes a low cost, highly available, site-resilient, simplified deployment
Contains detailed Bill of Materials for servers, storage and networking
8 Appendix A: Lenovo Bill of Materials .................................................... 27
8.1 BOM for the x3650 M5 ........................................................................................... 27
8.2 BOM for networking ............................................................................................... 28
9 Appendix B: Configuring the RAID1 boot array ................................... 29
10 Appendix C: Configuring the RAID0 arrays ......................................... 33
Trademarks and special notices ................................................................. 36
1 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
1 Introduction This document describes the reference architecture for a 4-node, low-cost implementation of Microsoft® Exchange Server 2013 that is based on the storage-dense, Lenovo® System x3650 M5 server that uses internal storage. The intended audience of this document is IT professionals, technical architects, sales engineers, and consultants to assist in planning, designing, and implementing Microsoft Exchange Server 2013.
Microsoft Exchange Server 2013 is the market leader in the enterprise messaging and collaboration market. Exchange Server 2013 builds upon the Exchange Server 2010 architecture and is redesigned for simplicity of scale, improved hardware utilization, and increased failure isolation. Exchange Server 2013 brings a rich set of technologies, features, and services to the Exchange Server product line. Its goal is to support people and organizations as their work habits evolve from a communication focus to a collaboration focus. At the same time, Exchange Server 2013 helps lower the total cost of ownership.
The architecture that is described in this document provides a low-cost solution for Exchange Server 2013 that meets the needs of an organization with 5,000 users (each with a 3.5 GB mailbox). By using the native high availability features of Exchange Server 2013 and storage-dense x3650 M5 servers, the solution that is described in this publication allows administrators to eliminate traditional backup methods, which frees critical enterprise resources. Also, the use of internal disks in a cache-enabled JBOD configuration drastically reduces the overall cost of the solution by eliminating storage area networks (SAN).
This document provides the planning, design considerations, and best practices for implementing the described architecture to support medium to large sized organizations. However, the principles and techniques that are described throughout this document can be expanded upon to support much larger user populations with the addition of storage and compute resources.
2 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
2 Business problem and business value This section describes the business challenges organizations face with email and how the reference architecture that is described in this document can help meet those challenges.
2.1 Business problem Today’s IT managers are faced with many obstacles when maintaining an enterprise-class email system. Proactively responding to obstacles, such as inefficiencies in communication that hinder productivity, changing regulations, and the threat of litigation, and the constant need to secure and protect valuable enterprise resources directly corresponds to the vitality of the business.
IT managers are also looking for efficient and cost-effective ways to manage and grow their IT infrastructure with confidence. Good IT practices recognize the need for high availability and maximum resource utilization. Responding quickly to changing business needs with simple, fast deployment and configuration while maintaining healthy systems and services is critical to meeting the dynamic needs of the organization. Natural disasters, malicious attacks, and even simple software upgrade patches can cripple services and applications. 2.2 Business value Exchange Server 2013 actively protects your communications with built-in defenses against email threats. Multi-layered anti-spam filtering comes with continuous updates to help guard against increasingly sophisticated spam and phishing threats, while multiple anti-malware engines work to protect your email data from viruses.
This Lenovo reference architecture for Microsoft Exchange Server 2013 provides businesses with an affordable, interoperable, and reliable industry-leading solution for their Exchange infrastructure. Built around the latest Lenovo servers and networking this offering takes the complexity out of the solution. This reference architecture combines Microsoft software, consolidated guidance, and validated configurations for compute, network, and storage. The design provides a high level of redundancy and fault tolerance across the compute servers, storage, networking, and application layer to ensure high availability of resources and Exchange Server databases.
3 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
3 Requirements The functional and non-functional requirements for an enterprise email system are desribed in this section.
3.1 Functional requirements An enterprise email system should fulfill the following user requirements:
• Receive incoming messages with or without attachments • Send outgoing messages with or without attachments • Interpret message content • Distinguish between mailboxes • Store incoming messages distinctly according to mailbox • Store sent messages • Delete messages • Forward messages • Allow sending messages to multiple recipients (cc) • Allow sending messages to multiple recipients by using a private send list (bcc) • Prevent incoming messages from being read by someone other than the intended recipient • Prevent outgoing messages from being read by someone other than the intended recipient • Prevent mailboxes from being accessed by someone other than the mailbox’s assigned owner • Prevent messages from being sent by someone other than the mailbox’s assigned owner • Allow users to categorize their messages • Allow users to sort their messages based on criteria (such as date, sender, subject, or size) • Allow users to search their inbox • Provide users with a 3.5 GB mailbox • Provide the performance to send/receive 100 messages per day • Provide the storage and throughput for an average message size of 75 KB
An enterprise email system should fulfill the following administrator requirements:
• Create a mailbox • Configure the default email address for new user accounts • Delete a mailbox • Move a mailbox • Move a mailbox database • Recover accidentally deleted messages • Monitor performance • Prevent spam messages from reaching recipients • Create public folders • Configure and change a mailbox quota • View the current mailbox size, message count, and last logon for a user • Prevent a mailbox from exceeding its quota size • Configure journaling
4 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
3.2 Non-functional requirements Table 1 lists the non-functional requirements for an enterprise email system.
Table 1. Non-functional requirements
Requirement Description Supported by
Scalability Solution components can scale for growth
Compute and storage can be scaled within a rack or across racks without service downtime. As described, the solution supports 5,000 users. However, the solution scales linearly. For example, if eight servers are deployed, the configuration supports 10,000 users.
Load balancing Workload is distributed evenly across compute servers
Network interfaces are teamed and load balanced.
High availability Single component failure does not lead to whole system unavailability
Hardware architecture ensures that computing, storage, and networking is automatically switched to remaining components; redundancy in hardware.
Physical footprint Compact solution Lenovo compute servers, network devices, and software are integrated into one rack with validated performance and reliability.
Support Available vendor support • Hardware warranty and software support are included with component products.
• Separately available commercial support from Microsoft.
Hardware and software components can be modified or customized to meet various unique customer requirements.
Robustness Solution continuously works without routine supervision
Integration tests on hardware and software components.
Security Solution provides means to secure customer infrastructure
• Security is integrated in the x3650 M5 server hardware with System x Trusted Platform Assurance, which is an exclusive set of industry-leading security features and practices.
• Networks are isolated by virtual LAN (VLAN).
High-performance Solution components are high-performance
Reference architecture provides information for capacity and performance planning of typical deployments.
5 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
4 Architectural overview The design consists of two x3650 M5 servers, a load balancer, and two RackSwitch™ G8124E networking switches that are deployed in each of the two data centers. Redundant paths connect the servers to the network preventing a network component failure from disrupting connectivity to the email environment. Together, these design features eliminate the need for traditional backups of the Exchange environment. Figure 1 shows the overall architecture of the solution.
Figure 1. Architectural overview
6 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
5 Component model This section describes the logical component view of the Exchange Server 2013 environment. Figure 2 shows a high-level component model.
Figure 2. Exchange Server 2013 logical component view
5.1.1 Key concepts and terminology The following basic concepts and terminology are used throughout this section:
Exchange Admin Center (EAC) – The EAC is the web-based management console in Microsoft Exchange Server 2013 that is optimized for on-premises, online, and hybrid Exchange deployments. The EAC replaces the Exchange Management Console (EMC) and the Exchange Control Panel (ECP), which were the two interfaces that were used to manage Exchange Server 2010.
Exchange Control Panel (ECP) – The ECP is a web application that runs on a Client Access Server (CAS) and provides services for the Exchange organization.
Exchange Web Services (EWS) – EWS provides the functionality to enable client applications to communicate with the Exchange server.
Internet Information Services (IIS) – IIS is an extensible web server that was created by Microsoft for use with Windows NT family.
Internet Message Access Protocol (IMAP) – IMAP is a communications protocol for email retrieval and storage developed as an alternative to POP.
7 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Microsoft Exchange ActiveSync (EAS) – EAS is a communications protocol that is designed for the synchronization of email, contacts, calendar, tasks, and notes from a messaging server to a smartphone or other mobile device.
Microsoft Outlook® Web App (OWA) – OWA (formerly Outlook Web Access) is a browser-based email client with which users can access their Microsoft Exchange Server mailbox from almost any web browser.
Offline Address Book (OAB) – The OAB is a copy of an address list collection that was downloaded so a Microsoft Outlook user can access the address book while disconnected from the server. Microsoft Exchange generates the new OAB files and then compresses the files and places them on a local share.
Outlook Anywhere – Outlook Anywhere is a service that provides RPC/MAPI connectivity for Outlook clients over HTTP or HTTPS by using the Windows RPC over HTTP component. In previous versions of Exchange Server, this function was used for remote or external access only. However, in Exchange Server 2013, all Outlook connectivity is via HTTP/HTTPS (even for internal clients).
Post Office Protocol (POP) – The POP is an application-layer Internet standard protocol that is used by local email clients to retrieve email from a remote server over a TCP/IP connection
Real-time Transport Protocol (RTP) – RTP is a network protocol for delivering audio and video over IP networks.
Remote PowerShell (RPS) – RPS allows you to use Windows PowerShell on your local computer to create a remote Shell session to an Exchange server if you do not have the Exchange management tools installed.
RPC Client Access (RPC) – In Microsoft Exchange Server 2007, the Client Access server role was introduced to handle incoming client connections to Exchange mailboxes. Although most types of client connections were made to the Client Access server, Microsoft Office Outlook still connected directly to the Mailbox server when it was running internally with the MAPI protocol.
A new service was introduced with Exchange Server 2010 to allow these MAPI connections to be handled by the Client Access server. The RPC Client Access service provides data access through a single, common path of the Client Access server, with the exception of public folder requests (which are still made directly to the Mailbox server). This change applies business logic to clients more consistently and provides a better client experience when failover occurs.
Remote Procedure Call over HTTP – The RPC over HTTP component wraps RPCs in an HTTP layer that allows traffic to traverse network firewalls without requiring RPC ports to be opened. In Exchange 2013, this feature is enabled by default because Exchange 2013 does not allow direct RPC connectivity.
Session Initiation Protocol (SIP) – SIP is a protocol that is used for starting, modifying, and ending an interactive user session that involves multimedia elements, such as video, voice, and instant messaging.
Simple Mail Transfer Protocol (SMTP) – SMTP is an Internet standard for email transmission.
Unified Messaging (UM) – UM allows an Exchange Server mailbox account that was enabled for UM to receive email, voice, and fax messages in the Inbox.
8 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
6 Operational model This section describes an operational model for deploying Microsoft Exchange Server 2013 that uses Lenovo x3650 M5 servers.
6.1 Hardware components This section describes the components in an Exchange Server 2013 deployment.
6.1.1 Lenovo System x3650 M5 At the core of this solution, the System x3650 M5 server delivers the performance and reliability that is required for business-critical applications, such as Exchange Server 2013. System x3650 M5 servers can be equipped with up to two 18-core Intel E5-2600 v3 processors, and up to of 3 TB of TruDDR4™ of memory. Up to seven PCIe 3.0 expansion slots, four 1 Gb network ports, and an optional embedded dual-port 10 Gb network adapter provide ports for your data and storage connections.
The x3650 M5 includes an on-board RAID controller and the choice of spinning hot swap SAS or SATA disks and SFF hot swap solid-state drives (SSDs). The large number of drive slots in the x3650 M5 makes it the perfect platform for running Microsoft Exchange Server mail environments from local disk.
The x3650 M5 supports the following components:
• Up to 8 2.5-inch Gen3 Simple Swap hard disk drives (HDDs) • Up to 8 3.5-inch Simple Swap HDDs • Up to 24+2+2 SAS/SATA 2.5-inch Gen3 Hot Swap HDDs • Up to 12+2 SAS/SATA 3.5-inch Hot Swap HDDs
The x3650 M5 also supports remote management via the Integrated Management Module (IMM), which enables continuous management capabilities. All of these key features, including many that are not listed here, help solidify the dependability that customers are accustomed to with System x servers.
Figure 3 shows the System x3650 M5.
Figure 3. Lenovo System x3650 M5
For more information, see the Lenovo Press product guide: lenovopress.com/tips1193-system-x3650-m5
9 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
6.1.2 ServeRAID M5210 RAID controller The ServeRAID™ M5210 SAS/SATA controller for System x is part of the ServeRAID M Series family that offers a complete server storage solution consisting of RAID controllers, cache and flash modules, energy packs, and software feature upgrades in an ultra-flexible offerings structure. M5210 comes as a small form factor PCIe adapter.
Two internal x4 HD Mini-SAS connectors provide connections to up to 32 internal drives (depending on the server model). To use cache, the optional 2 GB on-board data cache (DDR3 that is running at 1866 MHz) with battery backup upgrade is used.
Figure 4 shows the ServeRAID M5210 Controller with an optional cache installed.
Figure 4. ServeRAID 5210 SAS/SATA controller with optional cache installed
Note: When the ServeRAID M5210 RAID controller is used in a pure JBOD (just a bunch of disks) configuration, the controller cannot be installed with the optional on-board data cache (DDR3 that is running at 1866 MHz) with battery backup; the disk drives are passed directly through to the operating system. Pure JBOD deployments are significantly affected by the lack of battery-backed cache; therefore, to allow the use of cache in this reference architecture, RAID0 is used to create discreet, single-disk arrays. For more information about the ServeRAID M5210 SAS/SATA controller, visit the Lenovo Press product guide: lenovopress.com/tips1069-serveraid-m5210-sas-sata-controller
6.1.3 Lenovo RackSwitch G8124E The Lenovo RackSwitch G8124E is a 10 Gigabit Ethernet switch that is specifically designed for the data center and provides a virtual, cooler, and easier network solution. The G8124E offers 24 10 Gb ports in a high-density, 1U footprint.
Designed with ultra-low latency and top performance in mind, the RackSwitch G8124E provides line-rate, high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. The G8124E also supports Converged Enhanced Ethernet (CEE) and Data Center Bridging for support of Fiber Channel over Ethernet (FCoE) and can be used for network-attached storage (NAS) or iSCSI.
The G8124E is virtualized and supports VMready technology, which is an innovative, standards-based solution to manage virtual machines in small to large-scale data center and cloud environments. VMready works with all
10 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
leading virtualization providers. The G8124E also supports Virtual Fabric, which allows for the carving up of a physical NIC into 2 - 8 vNICs for improved performance, availability, and security while reducing cost and complexity.
The G8124E is cooler and implements a choice of directional cooling to maximize data center layout and provisioning. Its superior airflow design complements the hot-aisle and cold-aisle data center cooling model.
The G8124E is easier to configure with server-oriented provisioning via point-and-click management interfaces. Its industry-standard CLI and easy interoperability simplifies configuration for those familiar with Cisco environments.
The Lenovo RackSwitch G8124E is shown in Figure 5.
Figure 5. Lenovo RackSwitch G8124E
The RackSwitch G8124E includes the following benefits:
• A total of 24 SFP+ ports that operate at 10 Gb or 1 Gb Ethernet speeds • Optimal for high-performance computing and applications that require high bandwidth and low latency • All ports are non-blocking 10 Gb Ethernet with deterministic latency of 570 nanoseconds • VMready helps reduce configuration complexity and improves security levels in virtual environments • Variable-speed fans automatically adjust as needed, which helps to reduce energy consumption • Easy, standards-based integration into Cisco and other networks helps reduce downtime and learning
curve
For more information, see Lenovo Press product guide: lenovopress.com/tips0787
11 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
6.2 Logical mapping of Exchange components In this section, we describe how the components of an Exchange Server 2013 deployment are mapped to the logical design of the solution.
Figure 6 shows the components that are shown in Figure 2 as they map to the logical design of the solution.
Figure 6. Component mapping to the logical design
Lenovo recommends deploying the Client Access Server (CAS) role on the same compute server as the mailbox role (multi-role deployment). Therefore, the CAS role is deployed in a 1:1 ratio with the mailbox role.
6.3 High Availability design for Exchange 2013 This section describes the high availability functionality for an Exchange Server 2013 environment.
6.3.1 Key Exchange high availability concepts and terminology The following Exchange high availability concepts and terminology are used in this section:
12 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Mailbox database – A mailbox database is a unit of granularity in which mailboxes are created and stored. A mailbox database is stored as an Exchange database (.edb) file. In Microsoft Exchange Server 2013, each mailbox database has its own properties that can be configured.
Highly available database copies – Highly available database copies are configured with a replay lag time of zero. As their name implies, highly available database copies are kept up-to-date by the system, can be automatically activated by the system, and are used to provide high availability for mailbox service and data.
Lagged database copy – Lagged database copies are configured to delay transaction log replay for some time. Lagged database copies provide point-in-time protection, which can be used to recover from store logical corruptions, administrative errors (for example, deleting or purging a disconnected mailbox), and automation errors (for example, bulk purging of disconnected mailboxes).
6.3.2 Environment sizing When a new email system is implemented, it is important to correctly profile the user population to determine the average number of emails that are sent and received per day and the average message size.
Microsoft provides the Exchange 2013 Server Role Requirements Calculator tool to help accurately design an organization’s Exchange infrastructure. The calculator was used for the environment sizing that is used in this document. The calculator can be downloaded from this web page: blogs.technet.com/b/exchange/archive/2013/05/14/released-exchange-2013-server-role-requirements-calculator.aspx
The sizing can be determined by using the defined requirements and the profile information with the calculator that is provided by Microsoft.
The configuration that is described in this document supports a maximum of 5,000 users. It can be expanded to multiple servers and storage for more compute capacity or storage.
6.3.3 Database availability groups A database availability group (DAG) is the base component of the high availability and site resilience framework that is built into Microsoft Exchange Server 2013. A DAG is a group of up to 16 mailbox servers that hosts a set of mailbox databases and provides automatic database-level recovery from failures that affect individual servers or databases.
A DAG is a boundary for mailbox database replication, database and server switchovers, failovers, and an internal component called Active Manager. Active Manager, which runs on every server in a DAG, manages switchovers and failovers.
Any server in a DAG can host a copy of a mailbox database from any other server in the DAG. When a server is added to a DAG, it works with the other servers in the DAG to provide automatic recovery from failures that affect mailbox databases, such as a disk failure or server failure.
Figure 7 shows the DAG design that is described in this reference architecture. An active/passive, multi-site implementation with a user population of 5,000 requires a DAG that spans both data centers. Two mailbox servers (with the Mailbox and CAS roles installed) are required at each site to host the active copy, the two
13 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
passive copies, and the lagged copy of each mailbox database.
Figure 7. DAG design component diagram
The active database copies are hosted by the two mailbox servers in the primary data center because the primary data center is closest to most of the user population. Hosting the active databases close to the users prevents users from losing access to their email if there is a wide area network (WAN) outage.
The DAG is assigned a witness server (a file server) that is used as another vote to maintain quorum if there is a WAN outage. The DAG’s witness server should be placed in the same data center that hosts the active database copies during normal runtime, as shown in Figure 7. For example, the DAG’s witness server is located in the primary data center because the active databases are hosted there and most of the user population is located closest to that data center.
If there is a WAN failure, the primary data center has three quorum votes (the two mailbox servers and the witness server), but the secondary data center has only two quorum votes (the two mailbox servers). Therefore, the databases that are hosted in the primary datacenter remain active and the databases in the secondary data center are taken offline. This prevents the databases in the secondary data center from becoming out of sync with the database copies in the primary data center. Because the DAG’s user population is located near the primary data center the users do not lose access to their mailboxes because they do not have to traverse the WAN.
Figure 8 shows the environment during a WAN outage.
14 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 8. The Exchange environment during a WAN outage
6.3.4 Exchange database distribution The DAG consists of four mailbox servers (two in each data center). Sixteen mailbox databases are required to meet the needs of the 5,000 users. The two mailbox servers in the primary data center each host eight active mailbox databases. Also, each server hosts the passive copies of the active databases that are hosted by the other mailbox server (for example, if MBX1 hosts the active copy of a database, MBX2 hosts the passive copy of that database). At the secondary data center, two mailbox servers each host eight passive mailbox database copies and eight lagged mailbox database copies of the databases that are active in the primary data center.
Figure 9 shows a detailed view of the database layout for the DAG in the Exchange environment at normal runtime (all mailbox servers are operational).
15 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 9. Exchange environment when all mailbox servers are operational
If there is a server failure or a mailbox server is taken offline for maintenance, the active copies of the databases that are hosted on the affected server go offline and the passive copies that are hosted by the other mailbox server in the same data center become active, as shown in Figure 10.
16 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 10. Exchange environment with a single server failure
If there is a second server failure in the primary data center, the passive copies that are hosted by the mailbox server in the secondary data center become active. At this point, Lenovo recommends playing the logs forward on the lagged database copies to convert them to a highly-available passive copy of the database rather than a lagged copy. Doing so prevents a disruption of mail service if the environment sustains a third server failure, as shown in Figure 11.
17 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 11. Exchange environment with two server failures
Finally, if there is a third server failure, the passive database copies on the remaining mailbox server become active to support the entire user population of 5,000, as shown in Figure 12.
18 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 12. Exchange environment with a single mailbox server operational
6.3.5 CAS availability Although the DAG provides high availability for the mailbox databases, the CAS role requires separate consideration. In Exchange Server 2010, CAS server high availability was achieved by using an array of CAS servers that were load balanced by using a network load balancer. In Exchange Server 2013, the CAS array was replaced with the concept of a single namespace for Outlook connectivity.
In a default installation, each CAS server registers its fully qualified domain name (FQDN) as its internal host name in Outlook Anywhere. When an Outlook client makes a connection to a CAS server, it connects to the server’s registered internal host name. If the server fails, the connection times out and Outlook Anywhere automatically discovers an available CAS server and establishes a connection. However, this process is slow and can leave an Outlook client disconnected for some time. To reduce the time that is required to create a connection, each CAS server can be configured to use a single namespace as its internal host name in Outlook Anywhere. This configuration requires registering the single namespace as the internal host name for each CAS server and creating a dynamic name system (DNS) record on the DNS server that points to the single namespace. This technique ensures Outlook clients take less time to re-establish connectivity to one of the other IP addresses to which the shared namespace resolves.
19 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Note: DNS round-robin can be used for loud distribution, but a network load balancer is a better option because it provides faster switching capabilities. When a network load balancer is used, the single namespace resolves to the virtual IP (VIP) that is defined by the network load balancer rather than the IP address of a CAS server. When the network load balancer detects a server failure, it redirects incoming connections to CAS servers that remain online.
6.3.6 Backup strategy Exchange native data protection features, such are multiple highly available database copies and lagged database copies, are used in replace of traditional backups. This methodology is used to reduce the overall cost and complexity of the solution.
6.4 Compute server sizing Two Lenovo x3650 M5 servers are installed in each of the data centers. The recommended configuration for the compute servers includes the following components:
• Minimum of 128 GB RAM • Two Intel Xeon E5-2630 v3 (Haswell) 2.4 GHz 8-core processors • Two dual-port Emulex 10 Gb network adapters (only two ports are needed, but two adapters provides
extra redundancy) • 12x 4TB NL SATA LFF HDDs • ServeRAID M5210 SAS/SATA controller for System x • ServeRAID M5200 Series 2 GB Flash/RAID5 Upgrade
6.5 Storage To reduce the cost and complexity of the solution, the entire Exchange environment is hosted on local disks.
6.5.1 Storage Partitioning This section describes the storage partitioning that is required to support the configuration as described in this document.
In our Exchange testing, we found the storage controller cache to have a significant impact on performance. With cache enabled, the performance bottleneck shifts from the disk subsystem to the server’s processors significantly improving storage performance. Because cache cannot be enabled for a JBOD configuration when using the ServeRAID M5210 controller, the drives that are used in this Exchange solution are configured as discreet, single-disk RAID0 arrays. These arrays host the Exchange databases and log files, and provide capacity for a restore volume and an AutoReseed volume.
The volume for the operating system (OS) and the Exchange transport database is a two-disk, RAID1 array.
Table 2 lists the recommended storage requirements in this reference architecture.
20 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Table 2. Storage requirements
Volumes for Exchange Mailbox Databases (each volume is required on each VM)
Volume Description Volume Number and Size RAID Level
OS/Transport Database 1x 3,814 GB Disk pair (RAID1)
Mailbox database and logs 8x 3,814 GB Single disk (RAID0)
Restore volume 1x 3,814 GB Single disk (RAID0)
AutoReseed Volume 1x 3,814 GB Single disk (RAID0)
Figure 13 shows the volumes with the underlying RAID architecture and how they map to the drives in the x3650 M5 server.
Figure 13. Storage partitioning on the x3650 M5 local disks
Configuration of the ServeRAID M5210 is performed from within the UEFI shell of the server.
For more information about creating a RAID1 boot array for the Windows operating system, see “Appendix B: Configuring the RAID1 boot array” on page 29.
For more information about creating the RAID0, single-disk arrays for the Exchange databases and logs and the Restore and AutoReseed volumes, see “Appendix C: Configuring the RAID0 arrays” on page 33.
After the RAID0 arrays are created, create the volumes from within the Windows operating system.
6.5.2 Storage configuration for Exchange databases This section describes the Exchange Server 2013 mailbox database and log placement.
To adequately protect the Exchange databases without traditional forms of backups and when using cache-enabled JBOD rather than a standard RAID configuration, the organization requires at least two copies of each database at each site for a total of four copies of each database. A total of 16 active databases are required to support 5,000 users. Each of the 16 active databases has two more passive copies plus a lagged copy to provide high availability and point-in-time recovery. Therefore, 64 database copies are needed to support the entire user population. The databases are divided among four servers; therefore, each server hosts 16 databases.
Each volume on the server (except for the operating system volume, the Restore volume, and the AutoReseed volume) hosts two database copies. Figure 14 shows the database distribution among the servers. The logs follow the same pattern and are on the same drives as their respective databases (in separate folders).
21 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 14. Database copy distribution among the servers
There is no single storage configuration that is appropriate for every organization. Lenovo recommends gaining a thorough understanding of the capacity needs of your organization before implementing the storage design and then monitoring the solution for bottlenecks.
Microsoft provides comprehensive guidance on sizing and capacity planning for Exchange. For more information about Exchange sizing and capacity planning, see the TechNet article that is available at this web page: technet.microsoft.com/en-us/library/dn879075(v=exchg.150).aspx
6.6 Networking This section describes the networking topology and includes design guidance to correctly configure the network environment for redundancy and failover.
This reference architecture uses two ultra low-latency, high-performance, Lenovo RackSwitch G8124 10 Gb network switches to provide primary data communication services.
22 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
6.6.1 VLANs A combination of physical and virtual isolated networks are configured on the compute servers and the switches to satisfy isolation best practices.
VLANs are used to provide logical isolation between the various types of data traffic. Table 3 lists the VLANs that are required to support the Exchange Server 2013 workload as described in this reference architecture.
Table 3. VLAN definitions
Network VLAN Name Description
VLAN 70 Management Network A network that is used for host management, storage management, and out-of-band communication to IMM devices.
VLAN 40 Corporate Network / MAPI Network A network reserved for connecting to the domain controller and the corporate network. MAPI traffic uses this network.
VLAN 30 Exchange (Replication Network) A network for Exchange database replication.
Note: The VLANs that are described in this section are sharing the bandwidth of the single NIC team. Quality of service (QoS) is applied from within the Windows OS to ensure each VLAN has available bandwidth.
6.6.2 NIC teaming and virtual network adapter configuration All data traffic is over a single NIC team that is comprised of the 10 Gb network adapters in each of the compute servers. The NIC team is created by using the Windows Server 2012 R2 in-box NIC teaming feature that provides fault tolerance and load balancing for the networks.
After NIC teaming is configured, several virtual network adapters are created and assigned to appropriate VLANs.
6.6.3 Exchange mailbox server network adapter configuration Three virtual network adapters are created from within the Windows OS on each mailbox server and assigned VLANs by using Windows PowerShell. These adapters are used for accessing the management network (VLAN 70), accessing the corporate intranet and for MAPI traffic (VLAN 40), and for Exchange mailbox database replication (VLAN 30).
Figure 15 shows the virtual network adapters and their assigned VLANs.
23 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 15. Virtual network adapter configuration for the Hyper-V cluster servers
6.6.4 LACP and vLAG configuration To maintain VLAN information when multiple network switches are interconnected, an ISL is required. However, because a single ISL is limited in bandwidth to the 10 Gbps of a single connection and is not redundant, two ISL connections are recommended. Create two ISL connections by physically cabling switch 1 to switch 2 with two 10 Gb networking cables. For example, two external ports of switch 1 are cabled to two external ports of switch 2.
LACP is used to combine the two physical ISL connections into a single virtual link, called a trunk group. LACP teams provide for higher bandwidth connections and redundancy between LACP team members.
Important: Lenovo recommends enabling network health checking on networking switches that are configured as VLAG peers. Although the operational status of the VLAG peer is typically determined via the ISL connection, enabling network health checking provides an alternative means to check peer status if the ISL link fails. As a best practice, use an independent link between the two switches (for example, use the 1 Gb management port).
LACP teams are formed on the ISLs between the switches and on the host connections to the switches, that provides host connection redundancy. To maintain maximum bandwidth over the multiple connections, vLAGs are also configured on the LACP teams.
Note: Disabling Spanning Tree on the LACP teams helps avoid the wasted bandwidth that is associated with links blocked by Spanning Tree.
The vLAG/LACP configuration that is used for this reference architecture is shown in Figure 16.
24 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 16. LACP/vLAG design
6.6.5 Network load balancer For performance and resilience reasons, load balancing user connections to the CAS server role on the mailbox servers is required. Microsoft recommends the use of a layer 4 network load balancer, such as the Network Load Balancing feature in Windows Server 2012.
In Exchange Server 2013, the network load balancer no longer must be configured to ensure persistence across CAS servers. For a specific protocol session, CAS now maintains a 1:1 relationship with the Mailbox server that is hosting the user’s data. If the active database copy is moved to a different mailbox server, the CAS closes the active sessions to the previous server and re-establishes sessions to the new server.
For more information about network load balancing and configuration, see this web page: blogs.technet.com/b/exchange/archive/2014/03/05/load-balancing-in-exchange-2013.aspx
25 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
6.7 Deployment Example Figure 17 shows the Exchange Server 2013 environment (as described in this reference architecture) that is deployed in a 25U rack in both data centers. The rack contains a network load balancer, two Lenovo RackSwitch G8124E Network Switches, and two System x3650 M5 servers.
Figure 17. Deployment example: Hardware in each data center
26 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
7 Deployment considerations A successful deployment and operation of an Exchange Server 2013 enterprise solution can be significantly attributed to a set of test-proven planning and deployment techniques. Proper planning includes sizing the required compute resources (CPU and memory), storage (capacity and IOPS), and networking (bandwidth and VLAN assignment) that is needed to support the infrastructure. This information can then be implemented by using industry standard best practices to achieve optimal performance and growth headroom that is necessary for the life of the solution.
27 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
8 Appendix A: Lenovo Bill of Materials This appendix contains the Bill of Materials (BOMs) for the hardware for Exchange Server 2013 deployments on the x3650 M5. There are sections for the compute servers and for the networking switches that are orderable from Lenovo.
The BOM lists in this appendix are not meant to be exhaustive and must always be verified with the configuration tools. Any description of pricing, support, and maintenance options is outside the scope of this document.
The connector cables are configured with the device for connections between Top of Rack (TOR) switches and servers. The TOR switch configuration includes only transceivers or other cabling that is needed for failover or redundancy.
8.1 BOM for the x3650 M5 Table 4 lists the Bill of Materials for the x3650 M5 servers, as described in section 6.4 “Compute server sizing” on page 19. Table 4 lists quantities for one x3650 M5 server. Four servers are required to meet the configuration specifications as described in this reference architecture.
A5EK Intel Xeon Processor E5-2630 v3 8C 2.4GHz 20MB 1866MHz 85W 1 A5FF System x3650 M5 12x 3.5" Base without Power Supply 1 A5EA System x3650 M5 Planar 1 A5R5 System x3650 M5 PCIe Riser 2 (1 x16 FH/FL + 1 x8 FH/HL Slots) 1 ASQG System x 1300W High Efficiency Titanium AC Power Supply (200-240V) 2 6400 2.8m, 13A/125-10A/250V, C13 to IEC 320-C14 Rack Power Cable 2 6372 Line cord - 2.8m, 10A/250V, C13 to NEMA 6-15P (US) 1 A1ML Integrated Management Module Advanced Upgrade 1 A5FN System x3650 M5 PCIe Riser 1 (1 x16 FH/FL + 1 x8 FH/HL Slots) 1 A47G Super Cap Cable 425mm for ServRAID M5200 Series Flash 1 A5FX System x Enterprise 2U Cable Management Arm (CMA) 1 A5FV System x Enterprise Slides Kit 1 A5GE x3650 M5 12x 3.5" HS HDD Assembly Kit 1 A3YZ ServeRAID M5210 SAS/SATA Controller 1 A3Z2 ServeRAID M5200 Series 2GB Flash/RAID 5 Upgrade 1 A3W9 4TB 7.2K 6Gbps NL SATA 3.5" G2HS HDD 12 A5B8 8GB TruDDR4 Memory (2Rx8, 1.2V) PC4-17000 CL15 2133MHz LP RDIMM 16 A5UT Emulex VFA5 2x10 GbE SFP+ PCIe Adapter 2 A5G5 System x3650 M5 Riser Bracket 1 9297 2U Bracket for Emulex 10GbE Virtual Fabric Adapter 2
28 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
A5FT System x3650 M5 Power Paddle Card 1 A1PJ 3m Passive DAC SFP+ Cable 2 A5G1 System x3650 M5 EIA Plate 1 A5V5 System x3650 M5 Right EIA for Storage Dense Model 1
8.2 BOM for networking Table 5 lists the BOM for the network switches, as described in section 6.6 “Networking" on page 21. Table 5 lists quantities for one RackSwitch G8124E network switch. Four network switches are recommended to meet the configuration requirements as described in this reference architecture.
Table 5. Lenovo RackSwitch G8124E
Code Description Quantity
7159BR6 Lenovo System Networking RackSwitch G8124E (Rear to Front) 1 A1PJ 3m Passive DAC SFP+ Cable 2 00D6185 Adjustable 19" 4 Post Rail Kit 1
29 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
9 Appendix B: Configuring the RAID1 boot array Configuration of the ServeRAID M5210 is performed from within the UEFI shell of the server.
Complete the following steps to configure the internal storage:
1. Power on the server you want to configure and press F1 to enter UEFI Setup when the UEFI splash window opens.
2. From the UEFI System Configuration and Boot Management menu, select System Settings and press Enter, as shown in Figure 18.
Figure 18. UEFI System Configuration and Boot Management menu
3. From the System Settings menu, scroll down and select Storage and press Enter.
4. From the Storage menu, select the storage adapter and press Enter, as shown in Figure 19.
Figure 19. Available RAID Adapters
5. From the Main menu, select Configuration Management and press Enter, as shown in Figure 20.
30 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 20. ServeRAID M5210 Main Menu
6. From the Configuration Management menu, select Create Virtual Drive - Advanced and press Enter.
7. From the Create Virtual Drive - Advanced menu, ensure that RAID1 is selected as the RAID level and 256 KB is selected as the Stripe size. Select Drives and press Enter, as shown in Figure 21.
Figure 21. Create Virtual Drive menu showing correct RAID level and stripe size
8. From the Select Drives menu, highlight the HDDs in slots 0 and 1 and press the Enter to select them. Then, select Apply Changes and press Enter, as shown in Figure 22.
31 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 22. Select Drives menu
9. From the Success page, select OK and press Enter. You are returned to the Create Virtual Drive – Advanced menu.
10. From the Create Virtual Drive - Advanced menu, select Save Configuration and press Enter.
11. From the Warning page (as shown in Figure 23), confirm your choice and press Enter to select it. Next, select Yes and press Enter.
Figure 23. Warning page
12. From the Success page, select OK and press Enter.
13. Press Esc twice to return to the Main Menu. Select Controller Management and press Enter.
14. From the Controller Management page (as shown in Figure 24), scroll down and select Boot Device and press Enter.
32 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 24. Selecting the boot device
15. Select the RAID1 array and press Enter.
16. Press Esc to exit to the Main Menu.
17. After you configure the RAID1 boot array, exit UEFI and boot to your installation media to install the Windows Server 2012 R2 operating system.
33 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
10 Appendix C: Configuring the RAID0 arrays Configuration of the ServeRAID M5210 is performed from within the UEFI shell of the server.
Complete the following steps to configure the internal storage:
1. Power on the server that you want to configure and press F1 to enter UEFI Setup when the UEFI splash window opens.
2. From the UEFI System Configuration and Boot Management menu, select System Settings (as shown in Figure 25) and press Enter.
Figure 25. UEFI System Configuration and Boot Management menu
3. From the System Settings menu, scroll down and select Storage and press Enter.
4. From the Storage menu, select the storage adapter (see Figure 26) and press Enter.
Figure 26. Available RAID adapters
5. From the Main menu, select Configuration Management (as shown in Figure 27) and press Enter.
34 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 27. ServeRAID M5210 Main Menu
6. From the Configuration Management menu, select Create Virtual Drive - Advanced and press Enter.
7. From the Create Virtual Drive - Advanced menu, ensure that RAID0 is selected as the RAID level and 256 KB is selected as the Stripe size and then select Select Drives and press Enter (see Figure 28).
Figure 28. Create Virtual Drive menu showing the correct RAID level and stripe size
8. From the Select Drives menu, highlight the first HDD and press the Enter to select it, as shown in Figure 20. Then, select Apply Changes and press Enter.
35 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
Figure 29. Select Drives menu
9. From the Success page, select OK and press Enter. You are returned to the Create Virtual Drive – Advanced menu.
10. From the Create Virtual Drive - Advanced menu, select Save Configuration and press Enter.
11. From the Warning page (see Figure 30), highlight Confirm and press Enter to select it. Next, select Yes and press Enter.
Figure 30. Warning page
12. From the Success page, select OK and press Enter.
13. Repeat steps 7 - 12 to build RAID0 arrays on each individual remaining drive.
14. After you configure the drives, exit UEFI and boot to Windows normally.
36 Reference Architecture: System x3650 M5 Scalable Solution for Microsoft Exchange Server 2013 Using Internal Disks
