7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
1/27
Virtualizing MS Exchange
2010 at General Mills -Architecture Details andLessons Learned
Aaron Kopel, General Mills
Curtis Witt, VMware, Inc.
APP-BCA2053
#vmworldapps
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
2/27
2
Disclaimer
This session may contain product features that are
currently under development.
This session/overview of the new technology represents
no commitment from VMware to deliver these features in
any generally available product.
Features are subject to change, and must not be included in
contracts, purchase orders, or sales agreements of any kind.
Technical feasibi lity and market demand will affect final delivery.
Pricing and packaging for any new technologies or features
discussed or presented have not been determined.
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
3/27
3
Session Objectives & Agenda
General Mills Background
Exchange 2007 Architecture
Exchange 2010 Architecture
Business Drivers and Benefits
VMware Environment and Configuration
Key Takeaways
Q & A
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
4/27
4
General Mills Background
Global Company
Headquartered in
Minneapolis, MN 35,000 employees
across 80+ countries
$16.6B annual revenue
Virtualization Highlights
Virtualizing since 2007
94% virtual at HQ,74% virtual globally
VMware in 80+ sitesacross 20+ countries
vSphere ESXi 4.1 and 5.0
2200+ server VMs on 250+ hosts
1200+ View 5.1 users
Lab Manager
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
5/27
5
General Mills Exchange Landscape
32,000+ Mailboxes
Highly mobi le user base 10,000+ Mobile devices
Utilize Microsoft Unified Messaging
Regional centralization Exchange infrastructure in 8 sites
US: 25,735
UK: 3,030
Australia: 593
China: 824
Hong Kong: 838
India: 561S. Africa: 103
Argentina: 958
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
6/27
6
Exchange 2007 Architecture
Started physical and migrated to virtual
Virtualized everything except the Cluster Continuous Replication
(CCR) clusters
Server roles in Minneapolis
4 Client Access (CAS) servers
2 Hub Transport (HUB) servers
2 Unified Messaging (UM) servers
3 pairs of CCR clusters (Physical Servers with DAS)
Server roles at remote sites
9 all-in-one servers
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
7/27
7
Exchange 2010 Architecture
Started 100% virtual never ran physical
Server roles in Minneapolis
6 CAS servers
4 Hub servers
3 UM servers
One 7-node Database Availability Group (DAG)
Server roles at remote sites
7 all-in-one servers
Performance and sizing
Microsoft Exchange 2010 Mailbox Server Role Requirements Calculator
VMware Exchange 2010 on VMware Best Practices Guide
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
8/27
8
Exchange 2010 DAG Architecture
Single 6-node DAG (split across 2 sites), each host ing 8 active and
8 passive databases
Approximately 4,200 mailboxes per DAG node (25,000+ Total)
2GB Mailbox Quota
All mail groomed with 1-year max retention
Each nodes active databases replicate 2nd copy to other DAG node
VMs, never on the same host or in the same datacenter
7th DAG node houses 3rd copy of all 48 passive databases for
backup processing
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
9/27
9
Exchange 2010 DAG Database Layout
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
10/27
10
Virtualization Business Drivers and Benefits
Agility
Right size VMs initially, and hot-add vRAM and vCPU as needed
Easily expand shared storage to dynamically grow
Ability to quickly add additional servers
Operational Simplicity and Efficiency
No need to create one-off architecture
Core Compute, Network & Storage is the same as the rest of the environment
Easier planned maintenance
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
11/27
11
VMware Environment
Started on ESXi 4.1, now on ESXi 5.0
8 ESXi blade hosts dedicated for Exchange
Blade enclosures are shared with other general purpose ESXi hosts
No Exchange-purposed blade hosts share the same enclosure
No two DAG nodes share the same blade host
Everything located at 2 sites, interconnected with private fiber
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
12/27
12
Compute / Memory Config
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
13/27
13
HA / DRS / vMotion
HA Admission Control enabled with 25% CPU / Memory reserved
for failover capacity (2 hosts)
DRS Set to Fully Automated priority 1 & 2 recommendations
DAG Nodes set to Manual (no DRS)
Using DRS Groups / Rules to pin VMs to Primary or DR Datacenter
and to separate like Exchange server roles.
Succeeding with vMotion and DAG nodes
SameSubnetDelay increased from 1000ms to 2000ms
SameSubnetThreshold left at default of 5
vSphere 5 improved vMotion performance for DAG nodes
Work very closely with Exchange team to test / qualify this!!
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
14/27
14
Compute / Memory Lessons Learned
VMs created with hot-add CPU and Memory enabled
Reserved all memory for DAG nodes
Not using CPU reservations
Most server roles required increases from original CPU/memory
configurations
DAG: 46 vCPUs and 24GB48GB
CAS: 816GB
UM: 24 vCPUs and added another server
Make sure to size CPU / Memory for failover / maintenance capacity
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
15/27
15
Datacenter Network Config
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
16/27
16
Host Network Config
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
17/27
17
Networking Results and Lessons Learned
Using VMXNET3 (default settings)
Mailbox VMs have 2 vNICs connected to Exchange and Backup
Networks
Overall daytime bandwidth averages ~2Gb/s
Separated vMotion / Backup network to relieve production network
bandwidth
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
18/27
18
Datacenter FC SAN Config
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
19/27
19
Disk / LUN / VMFS Layout
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
20/27
20
DAG Virtual Machine Storage Config
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
21/27
21
Storage Results and Lessons Learned
You can succeed with shared storage!
Make sure your SAN has the required capacity and throughput.
(Run Jetstress!)
Plan for Background Database Maintenance
(runs 24x7 @ 5MB/s/Database)
Total Exchange IOPS (all roles) is around 8000IOPS + 3000 for
7th
DAG node Needed to dedicate 4 FA ports per site for Exchange on Tier-1 SAN
Each FA port could handle about 4000IOPS
Increase VMFS heap size to 256 MB
Allows for increase from 20 TB to 64 TB of open VMDK capacity per ESXi host Needed for the gigantic 7th DAG node
Details at: http://kb.vmware.com/kb/1004424
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
22/27
22
Uber Diagram
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
23/27
23
Key Takeaways
1. Be on the good side of your Exchange and Storage Teams!
2. Enable hot-add vCPU and vRAM for all VMs easily adjustsizing for all server roles.
3. Design storage so it is quick and easy to grow!
4. Compared with Exchange 2007, disk I/O is definitely reduced,but at the expense of CPU and Memory
5. PowerCLI is handy for configuration of complex VMs(lots of controllers and VMDKs).
6. Review the VMware E2K10 Best Practices Guide(s)
http://www.vmware.com/files/pdf/Exchange_2010_on_VMware_-_Best_Practices_Guide.pdf
http://www.vmware.com/files/pdf/using-vmware-HA-DRS-and-vmotion-with-exchange-2010-dags.pdf
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
24/27
24
After The Conference
APP-BCA1684 - Virtualizing Exchange Best Practices
By Alex Fontana, VMware
(Monday, 1:00 PM 2:00 PM and Tuesday, 3:30 PM 4:30 PM)
Review content for this session and Alexs session
after the conference, at http://www.vmworld.com
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
25/27
25
THANK YOU
Questions?
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
26/27
FILL OUT
A SURVEY
EVERY COMPLETE SURVEY
IS ENTERED INTO
DRAWING FOR A
$25 VMWARE COMPANY
STORE GIFT CERTIFICATE
7/27/2019 BCA2053-Virtualizing MS Exchange 2010 at General Mills - Architecture Details and Lessons Learned_Final_US.pdf
27/27
Virtualizing MS Exchange
2010 at General Mills -Architecture Details andLessons Learned
Aaron Kopel, General Mills
Curtis Witt, VMware, Inc.
APP-BCA2053
#vmworldapps