Deploying and Managing Microsoft Exchange Server 2010 Transport Servers

Charlie ChungLead Program ManagerMicrosoftSession Code: UNC311

Session Objectives And Takeaways

Session Objective(s): Describe new High Availability and Service Level Reporting features of the Exchange Server 2010 transport platformExplain how to deploy Exchange Server 2010 transport server including coexistence with Exchange Server 2007 and Exchange Server 2003

Deploy highly available transport designs that deliver messages with low latencyUnderstand key coexistence scenarios

Agenda

Exchange Server 2010 Transport OverviewNew Transport High Availability Features Managing and Reporting Transport SLAExchange 2010 Routing overviewInteroperability and coexistence with Exchange Server 2003 and 2007Exchange 2010 EdgeSync Enhancements

Exchange Server 2010 System Architecture Enterprise Network

ExternalSMTP

servers

MailboxStorage of mailbox

items

Edge TransportRouting & AV/AS

Unified MessagingVoice mail & voice access

Phone system (PBX or VOIP)

Client AccessClient connectivity

Web services

Hub TransportRouting & Policy

Web browser

Outlook (remote user)

Mobile phone

Outlook (local user)Line of business application

Active Directory

Exchange Server 2010 Hub Transport Role Architecture

From:To:

1. User composes message in Outlook and it is stored in users Outbox

2. Mailbox submission service listens for store event notification of new message and notifies an in-site Hub Transport

RPC

3. Hub Transport retrieves message from sender’s mailbox and submits to queue

4. Hub Transport categorizes message and applies message policies

5. Hub Transport delivers message to Hub Transport server in target AD site

6. Hub Transport delivers message to mailbox server in same AD site

Mailbox

Mailbox

Hub Transport

Hub Transport

Message Delivery Flow

Transport High Availability

Transport High Availability ArchitectureResiliency Issues in Exchange Server 2007

Transport database is statefulLoss of service results in loss of mailHardware redundancy for high availability

Transport dumpster impacts the environmentIn extreme cases, up to 200% increase in IOPS/message due to many SGs and inefficient cache usage when compared to similar scenarios without dumpsterRedelivery after MDB failover results in entire quota being redelivered and store removing duplicates

Transport database corruption causes downtimeMail storms due to rogue user/program

Transport High Availability ArchitectureExchange 2010 Resiliency Improvements

Shadow Redundancy is a new feature of Edge and Hub transport roles

Provides redundancy for messages in transitTransport becomes near-statelessEliminates need for RAID1/10 storage for queue database 50% write I/O is eliminatedEnabled by default

Transport resilient to database corruptionWill move/delete old database and restart service

Throttling of MAPI and SMTP client submissionsPrevent mail storms due to accidental misuse, misbehaving software and malware

How does Shadow Redundancy Work?

1

2

1. Hub (shadow) delivers message to Edge1 (primary)Detects that Edge1 supports Transportredundancy through XSHADOW verbHub moves message to shadow queue and stamps Edge1 as current, primary owner

2. Edge1 (primary) receives message (becomes “primary owner”)Edge1 delivers message to next hop Edge1 updates discard status of the message indicating delivery complete to foreign MTA

Hub

Edge1 Edge2

Foreign MTA

How does Shadow Redundancy Work?

1

2

3. Success: Hub (shadow) queries Edge1 (primary) for expiry statusHub issues XQDISCARD command (next SMTP Session),Edge1 checks local discard status and responds with list of messages considered delivered Hub deletes messages from its shadow queue

4. Failure: Hub (shadow) queries Edge1 (primary) discard status and resubmits

Hub opens SMTP session, issues XQDISCARD command (heartbeat)—if Hub can’t contact Edge1 within 15 minutes (3X timeout interval), resubmits messages in shadow queue—resubmitted messages are delivered to Edge2 (go to #1)

43

Hub

Edge1 Edge2

Foreign MTA

Shadow Redundancy Primary Server State Tracking

Shadow server needs to determine Identity of Primary ServerIf identity change detected, shadow messages for primary are resubmitted

“Heartbeat” needed to determine when shadow server should resubmit shadow messages for delivery over alternate route

Failure to complete successful heartbeat results in resubmission of shadow messages (default 3 attempts at 5 min interval)

“Discard Status” needed to determine when shadow server can delete shadow message after delivery completed

At end of each SMTP session, shadow server issues XQDISCARD command which returns list of unique ID’s that can be removed from shadow queue

Hub

Hub

Edge

Mailbox

1

2

4

3

4

5

Mailbox

2

Client

0

SMTPClient

4

Internet

Shadow Redundancy Supported Scenarios

6

6

Ex2007 Hub

2) E2010 Intra-Org SMTP1) Mailbox Submission to Hub Role

3) Delivery to Mailbox Role4) Inbound SMTP from Interop MTA

6) Outbound delivery (without redundancy)

0) Client Submission (without redundancy)

3

5) Side Effects (NDR, Journal Report)

5

5

Shadow Redundancy1) Mail Submission Service

MSExchangeMailSubmission saves shadow message copy in sender’s “Sent Items” folder, critical properties of message are hashed to ensure it is valid for resubmission

“Implicit” heartbeat piggybacks on RPC (Remote Procedure Call) notification used for store driver submission“Explicit” heartbeat invokes extra RPC in absence of store driver submissionsShadow message discard status also piggybacks on MSRPC used for store driver submission

Remaining shadow message(s) resubmitted from “Sent Items” after 3 explicit heartbeat failures

Shadow Redundancy2) SMTP Service Extensions

New SMTP service extensionsXSHADOWXQDISCARD

Used to provide redundancy between Exchange 2010 transport servers over SMTP

Intra-Forest message transfer using Exchange Servers authentication (Hub-Hub, Hub-Edge)Cross-Forest message transfer using externally secured send and receive connections

Saves copy of message on previous hop until next hop fully delivers all recipients

Shadow RedundancyXSHADOW Configuration

Organization Configuration (*-TransportConfig)ShadowRedundancyEnabled : True ShadowHeartbeatRetryCount : 3ShadowHeartbeatTimeoutInterval : 00:05:00ShadowMessageAutoDiscardInterval : 2.00:00:00

Receive Connector ConfigurationAuthentication Mechanisms enable advertisement of SMTP service extensions

Exchange ServersExternally Secured

Permissions enables client to use commandsms-Exch-SMTP-Accept-Xshadow

Send Connector ConfigurationPermissions enable use of commands

ms-Exch-SMTP-Send-XShadow

Shadow RedundancySMTP Session with “Implicit Heartbeat”

< 220 PRIMARY.TEST.COM Microsoft ESMTP MAIL Service ready at Tue, 4 Sep 2007 10:07:15 -0700> EHLO SHADOW.TEST.COM< 250-PRIMARY.TEST.COM Hello [10.197.93.136]< 250 XSHADOW> XSHADOW FzHkA/yKi0GHWQnBHzdbOg==< 250 VUjDMdghpkm4OwsLyqZcag==> MAIL FROM:<sender@test.com> SIZE=1005 XSHADOW=e21e97f4-f911-47d5-99aa-6b3c8757f73b> RCPT TO:<recipient@test.com>< 250 2.1.0 Sender OK< 250 2.1.5 Recipient OK> BDAT 1336 LAST< 250 2.6.0 <cc7c2203-cfc8-4cd2-b589-eddca8513b14@SHADOW.TEST.COM> Queued mail for delivery> XQDISCARD 50< 251 OK, no discard events> QUIT< 221 2.0.0 Service closing transmission channel

Shadow RedundancySMTP Session with “Explicit Heartbeat”

< 220 PRIMARY.TEST.COM Microsoft ESMTP MAIL Service ready at Tue, 4 Sep 2007 10:12:27 -0700> EHLO SHADOW.TEST.COM< 250-PRIMARY.TEST.COM Hello [10.197.93.136]< 250 XSHADOW> XSHADOW FzHkA/yKi0GHWQnBHzdbOg==< 250 VUjDMdghpkm4OwsLyqZcag==> XQDISCARD 50< 250 e21e97f4-f911-47d5-99aa-6b3c8757f73b> QUIT< 221 2.0.0 Service closing transmission channel

Queue ViewerShadow Queue

Queue ViewerShadow Message

Shadow Redundancy3) Mailbox Delivery

Transport Dumpster continues to provides redundancy for final delivery to mailboxActiveManager provides MDB replication feedback to transport , used to control which messages are retained in the Transport Dumpster

When log containing delivered message has been replicated to all MDB copies, message is truncated from Transport Dumpster

Dumpster size is now a function of MDB log replication latency and frequency of feedback, maximum size limited by quota when one or more MDB copies not healthyMailbox Role requests re-delivery from all hub servers in all AD sites hosting copy of MDB after cross-site failover

Shadow Redundancy4) Delayed Acknowledgement

“Best Effort” shadow redundancy for any SMTP implementation that doesn’t support XSHADOW and XQDISCARD

No shadow redundancy for outgoing messages to these systemsDelayed Acknowledgement after end of data sequence

250 response delayed up to 30 sec (default) while categorization and delivery are attemptedIf transport server fails before acknowledgement, client resubmits

Message will “skip” the delayed ack when DelayedAckSkippingEnabled is true and any of the following conditions exist:

Submission queue in suspended stateMessage is deferred due to transient errorDelivery queue in retry or suspended stateDelivery queue size exceeds DelayedAckSkippingQueueLength value defined in EdgeTransport.exe.config (default 100)Message routed to unreachable queue

Shadow RedundancyDelayed Acknowledgement Configuration

Organization Configuration (*-TransportConfig)ShadowRedundancyEnabled

Receive Connector ConfigurationMaxAcknowledgementDelay

Default 30 secondsDisable by setting to 0 secondsDo not exceed 60 seconds for client connectorDo not exceed 10 minutes for default connector

EdgeTransport.exe.configDelayedAckSkippingEnabledDelayedAckSkippingQueueLength

Shadow Redundancy5) Side Effect Messages

System generated messages (Journal Report, NDR) are considered “side effects” of original message submission

Resubmission of shadow message copy will occur if “primary” and any associated “side effect” messages are not delivered before server failure

Resubmission of shadow message copy will result in the same “side effect” messages as the original message

Shadow RedundancyDiagnostics

Message Tracking Log RESUBMIT events indicate when messages are resubmitted due to shadow redundancy heartbeat failure or transport dumpster redeliverySMTP Receive Protocol log provides info events for delayed acknowledgement including reason for DelayAck skippingMSExchangeTransport Shadow Redundancy Perfmon object

“Current Messages Acknowledged Before Relay Completed” provides count of messages accepted without redundancy

Events indicate when transport receives redelivery requests from mailbox role for each MDB after failover, when resubmission job is completed and how many messages were resubmitted by transport from transport dumpster

Queue Database ResiliencyAutomated Recovery

Transport detects fatal ESE exceptions associated with Queue databaseMoves or Deletes database

Default to move (requires manual action before subsequent recoveries are attempted)Optionally enable delete action in app.config (no manual operation necessary unless failure occurs)

Service process restarts worker processNew Queue database created

Method not always successfulHardware failures (drive, controller, etc) require manual recovery actions

Throttling Message SubmissionsManage using *-ThrottlingPolicy cmdlets

Throttling policies are applied per-userTransport settings in Default Throttling policy are disabled by defaultDefault Policy can be overridden with custom policy applied to individual users

MessageRateLimit throttles rate of message submission from authenticated user or anonymous IP address

Evaluated per-server over 1 minute period SMTP returns transient errors when rate exceededMail Submission Service defers messages in outbox once rate has been exceeded, retries submission periodically

RecipientRateLimit throttles number of messages submittedEvaluated over 24 hour periodCentral accounting on mailbox role using MSExchangeThrottling serviceError returned to client for all submission attempts once quota exceeded

Transport Service Level Management and Reporting

Self-HealingStandardized Recovery Process

Recovery

Diagnosis

Awareness

Transport Service Level ManagementMonitoring, Incident Management and Reporting

Key Heath Indicators: Message Latency, AvailabilityService Level Metrics

NoiseGaps

Scope/Impact/Expertise

HA is mitigationAlert the right person

Root Cause Analysis (% identified)

Mean Time to Recovery (MTTR)

Repo

rting

Alert when Service Level Threatened

Instrumentation and Analysis Tools

Capa

city

Pla

nnin

gEn

d U

ser E

xper

ienc

e

Processes that impact ability

to meet SLA objectives

Perf

orm

ance

aga

inst

SLA

ob

jecti

ves

Transport Service Level ManagementAwareness through Proactive Monitoring

Key Health Indicators (KHI) used to determine when user experience impacted

Delivery Latency to determine if delivered messages are meeting SLA objectivesSubmission Availability to determine if server is available to accept new messagesDSN Generation to determine if server is failing to deliver messagesDelivery Completion to determine if server is unable to complete delivery

Transport Service Level Management Measuring Delivery Latency

Exchange Server 2010 measures latency of every component involved with delivering message end-to-endPrevious Hop latency using Received Headers timestamps for measuring delivery latency on legacy transport servers

Define IP ranges using InternalSmtpServers parameter on transport configuration (*-TransportConfig)Recommend NTP for accurate measurements

get-message cmdlet has new IncludeLatencyComponent parameter to determine latency of message in queue“MSExchangeTransport Component Latency” Perfmon object counters for local server percentile latency measurements over moving 5 minute windowEnd-to-End latency of “delivered” messages can be determined from message tracking logs on final hub

Measuring Delivery LatencyMessage Tracking Log Details

[PS] C:\>get-messagetrackinglog –server:df-mlt-01 -messageid: <E26375F9F42D49F3BE8C142DB50E1517@redmond.corp.microsoft.com>" | ConvertTo-MessageLatency.ps1 | FT -a ComponentServerFqdn,ComponentCode,ComponentName,ComponentLatency

ComponentServerFqdn ComponentCode ComponentName ComponentLatency------------------- ------------- ------------- ----------------msw-sfw-r03.redmond.corp.microsoft.com TOTAL Total Server Latency 00:00:03tk5-exsmh-c102.redmond.corp.microsoft.com TOTAL Total Server Latency 00:00:23tk5-exhub-c103.redmond.corp.microsoft.com TOTAL Total Server Latency 00:00:08TK5EX14MLTC101.redmond.corp.microsoft.com TOTAL Total Server Latency 00:00:00df-h14-01.exchange.corp.microsoft.com TOTAL Total Server Latency 00:00:00DF-MLT-01.exchange.corp.microsoft.com TOTAL Total Server Latency 00:00:00

Hop 1: 3rd Party Application MTA (Previous Hop Latency)

Hops 2,3: Exchange Server 2007 (Previous Hop Latency)

Hops 4,5,6: Exchange Server 2010 (Latency Tracker)End-to-End Delivery

Latency of ~34 seconds

Measuring Transport Service LevelsSystem Center Aggregation and Reporting

Hub1 Hub2

SQL ServerDatabase

Edge

System CenterRoot Management Server

Primary Datacenter Location

Remote Locations

SQL ReportingServices

Server and User Statistics Logs periodically generated locally on each server

System Center agents aggregate perfmon measurements and data from

logs via SCOM RMS to SQL Server Database

SQL Stored Procedures aggregate raw data into hourly and daily tables

SQL Reporting Services used to display data

Measuring Transport Service LevelsStatistics Log Generation

Server statistics log generated hourly (00:00-23:00) containing traffic summaryServerStatisticsLogMaxAge : 30.00:00:00ServerStatisticsLogMaxDirectorySize : 250 MB (262,144,000 bytes)ServerStatisticsLogMaxFileSize : 10 MB (10,485,760 bytes)ServerStatisticsLogPath : C:\Program Files\Microsoft\

Exchange Server\V14\TransportRoles\Logs\ServerStats

Active user statistics log generated every 8 hours (00:00, 08:00, 16:00) containing summary of user usageActiveUserStatisticsLogMaxAge : 30.00:00:00ActiveUserStatisticsLogMaxDirectorySize : 250 MB (262,144,000 bytes)ActiveUserStatisticsLogMaxFileSize : 10 MB (10,485,760 bytes)ActiveUserStatisticsLogPath : C:\Program Files\Microsoft\

Exchange Server\V14\TransportRoles\Logs\ActiveUsersStats

Exchange Server 2010Routing Overview

Exchange Server 2010 RoutingFew changes from Exchange 2007 routing architectureDirect connections (point-to-point routing)

Prefer direct IP connection between source and destinationBased on AD site topology and site link costsQueue mail as close to destination as possible

Deterministic routingSimplify design to follow a consistent pattern make planning and troubleshooting easierNo longer relies on Exchange Link State informationOptimize bytes over the wire by bifurcating based on route

Simplify deploymentAutomatic configurationConsolidated topology concepts

Active Directory Sites Are The Routing Boundary

Automatic load balancing and fault toleranceMailbox will load balance submissions across all Hubs in local AD site When mailbox and Hub roles coexist on same server, local Hub preferredHub will load balance connections across all Hubs in remote AD SiteHub will deliver to any mailbox in local AD site

Uses the AD site topology to calculate back-offDirect connect FIRST, unless forced through Hub SitesProvides for queuing at the point of failureAvailability information is not cachedAlways try all Hub servers within remote AD site before back-offEach new connection uses same algorithmWhen bifurcation (delayed fan-out) is required Equal cost path arbitrationHop countAlphabetic based upon site name

Cos

t = 1

00

Cost = 100

Cost = 100

Cost = 100

Cost = 100 Site 1 Site 2

Site 3

Site 11 Site 21Direct Connect

Backoff Route #1Backoff Route #2

Final Backoff

OriginatorRecipient #1

“Best” Route Between AD Sites

Interoperability andCoexistence with Exchange Server 2003 andExchange Server 2007

Coexistence with Exchange Server 2003All Exchange 2007/2010 servers are within a single routing group Introduction of first Exchange 2007/2010 Hub role results in creation of routing group connectors (single source/target bridgehead on each)

Add source and target bridgehead servers for fault tolerance and load balancing between these two connected routing groupsExchange 2003 RGC bridgehead cannot be a cluster

Coexistence with Exchange Server 2003Exchange 2007/2010 Routing to Exchange 2000/2003 recipient

Chooses least cost RGC route to Exchange 2003 recipient based on routing group connector costs (AD cost not included)Chooses least cost route within the Exchange 2007/2010 routing group to the AD site containing RGC “bridgehead” based upon AD site link cost

Exchange 2000/2003 routing to Exchange 2007 recipientServer picks least cost route to the Exchange 2007/2010 Routing Group regardless of AD site where recipient mailbox locatedExchange 2007/2010 “bridgehead” routes within Exchange 2007/2010 Routing Group to the AD site containing recipient mailbox based upon AD site link cost

Exchange 2010 Transition Topology

Cost = 100

Cost = 100

Cost = 100

Cost = 100

Site 1 Site 2

Site 11

Site 13 Site 23

Exchange Routing Group (DWBGZMFD01QNBJR)

Exchange 2003 Routing Group 13

Exchange 2003 Routing Group 1

Exchange 2003 Routing Group 2

Exchange 2003 Routing Group 23

RGCCost=10

RGCCost=10

RGCCost=10

E2010

Routing Group Connector

(RGC)Cost=10

RGCCost=10

RGCCost=10

RGCCost=10

Recipient #2

Recipient #1Disable Link State on all E2K/E2K3 Servers!!!

E2010

Originator

E2010

Bifurcate

E2010

E2010E2010

E2010

E2010

Disabling Link StateSuppresses communication of minor link state changes (link up or down)Used when you have multiple routes to/from the Exchange 2010/2007 Routing GroupMust be done to every Exchange 2003 server in the organization to prevent loopsAll versions only use least cost routeControlled via registryHKLM\System\CurrentControlSet\Services\RESvc\ParametersDWORD: SuppressStateChangesValue: 1

From:To:

1. User composes message in Outlook and it is stored in users Outbox

2. Exchange 2007 Mailbox submission service listens for store event notification of new message and notifies an in-site Exchange 2007 Hub Transport server

RPC

3. Exchange 2007 Hub Transport retrieves message from sender’s mailbox and submits to queue, categorizes message, applies Exchange 2007 policy and drops in “Version 14” delivery queue4. Exchange 2007 Hub Transport delivers message to Exchange 2010 Hub Transport server in same AD site using SMTP

6. Exchange 2010 Hub Transport delivers message to Exchange 2010 mailbox server in same AD site

Mailbox

Mailbox

Hub Transport

Hub Transport

Message Delivery FlowExchange Server 2007 Exchange Server 2010

5. Exchange 2010 Hub Transport receives message via SMTP, categorizes message, applies Exchange 2010 policy, queues to Exchange 2010 mailbox server

Coexistence with Exchange Server 2007Routing version boundary change:

Exchange 2010 Mailbox servers can only submit to Exchange 2010 Hub Transport serversExchange 2010 Hub Transport servers can only deliver to Exchange 2010 Mailbox servers

Exchange 2007 Mailbox servers can only submit to Exchange 2007 Hub Transport serversExchange 2007 Hub Transport servers can only deliver to Exchange 2007 Mailbox servers

Exchange 2010 Hub Transport servers can communicate with Exchange 2007 Hub Transport servers via SMTP (and vice versa)Inter-site routing has no version preference

Hub role will load-balance inter-site traffic to all hubs in target site

Subscribed Edge servers:Have no version preference when routing inbound/outbound trafficExchange 2010 Hub Transport will become authoritative for Edgesync

Edge Transport RoleEdgeSync Improvements

Better Performance for EdgeSync via Deltasync ModeUnder this mode, each time EdgeSync service only reads the delta change since last sync and updates the target accordingly

Support for safe senders and blocked senders Configurable Safe List quotasAdministrator defined blocked sendersAutomatic update of Safe Sender list propagation into Active Directory

Key LearningsUnderstand how New Transport High Availability and Service Level Reporting features of the Exchange Server 2010 can lower the capex and opex costs for Hub ServersUnderstand how Exchange Server 2010 mail routing coexistence works with Exchange Server 2007 and Exchange Server 2003 so you can plan your upgradeAware of the new instrumentation, tools, and reports for you to measure the SLA of mail flow in your environment.

UNC Track Call to Action!Learn More!

Related Content at TechEd on “Related Content” SlideAttend in-person or consume post-event at TechEd Online

Check out learning/training resources at Microsoft TechNetExchange Server and Office Communications Server

Check out Exchange Server 2010 atVirtual Launch Experience (VLE) at thenewefficiency.com

Try It Out!Download the Exchange Server 2010 TrialTake a simple Web-based test drive of UC solutions through the 60-Day Virtual Experience

question & answer

www.microsoft.com/teched

Sessions On-Demand & Community

http://microsoft.com/technet

Resources for IT Professionals

http://microsoft.com/msdn

Resources for Developers

www.microsoft.com/learning

Microsoft Certification & Training Resources

Resources

Complete an evaluation on CommNet and enter to win an Xbox 360 Elite!

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,

IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.