How We Do It: Building & Managing SharePoint Online

Doron Bar-Caspi & Roberto TaboadaSharePoint Online

SPC121

ObjectivesShow how we designed & built SharePoint OnlineProvide insights into building & managing a service of the scale of SharePoint OnlineUseful to you regardless of where you are with SharePoint OnlineYou are evaluating SharePoint OnlineYou use or manage a tenancy on SharePoint OnlineYou use or manage SharePoint on premises

AgendaOverview of Office 365 & SharePoint OnlineService topology & automation at cloud scaleNew customer signup & tenant configurationUpgrade & patchingQ & A

Related presentationsOverview of Office 365

Overview of the New SharePoint Online

What's New in Managing Your SharePoint Online Environment

SharePoint Online and Office 365 Security, Trust & Privacy

Deep Dive: Capabilities of SharePoint Online's Public Website

Getting the Most out of SharePoint Online for Small Business

Andy O'Donald Mark Kashman Philip Newman Mike KostersitzJosh Stickler & Kevin Gjerstad Jeremy Mazner

Migrating to SharePoint Online in Office 365-Strategy & Best Practices

Overview of SharePoint Licensing

SharePoint 2013 Upgrade Overview & Deep Dive

Designing Your SharePoint Server 2013 Enterprise Deployment

How We Do It: Operating SharePoint Online

Phil Cohen Sajan Parihar Sean Livingston Luca Bandinelli & Steve Walker

Ben CanningToday @ 5:00

Overview of Office 365 & SharePoint Online

Office 365 for all customer segments

EnterpriseMidsize Business

EducationSmall Business

Consumer Government

Designed for generalist IT

Targeted up to 250 users

Hard cap of 300 users

No IT required

Targeted up to 10 users

Hard cap of 25 users

Data isolation (US only)

Advanced admin tools

Range of offers by user type

Free email and collaboration

Enterprise admin tools

Range of offers by user type

Advanced admin tools

Enterprise IT capabilities

Office client

Office 365 Midsize Business

Office 365 Small Business

Office 365 Education

Office 365 Enterprise & Government

Office 365 Enterprise

Office 365 Home Premium

Layers of Office 365

Office 365 Portals

Sign Up Experience

Tenant Administration

Landing Pages

Office 365 Platform Services

Commerce & Billing

Identity Platform

Authentication

Internet DNS

Office 365 Services

SharePoint Online

Exchange Online

Lync Online Office ProWeb AppsOther

• Group service plans into packages• Offered on specific terms: trial, paid,

volume• Examples: Office 365 Small Biz (P1),

Office 365 Enterprise (E3)

Office 365

Plans• Define capabilities that a

tenant or user is entitled to• Examples: SPO for Enterprise

Service Plans

• SharePoint, Exchange, Lync & others

• Translate Plans into experiences & features

Services

SharePoint Online componentsSharePoint – actual bits & featuresSame bits used in on-premises deploymentsAll features must conform to service fabric horizontals—”cloud ready”

Service Fabric – components needed to run serviceDeployment & Environments – TopologyIdentity & Sign InProvisioning Tenants & Users Tenant AdminUpgradeHigh Availability & Disaster RecoveryTelemetry, Incident Management, Debugging & Patching Code in the Service

Zoom in on topology, provisioning & upgradeDeep dive into system topology & deployment, customers onboarding & upgrades

SharePoint Online ethosSharePoint Online promise to customersService is something you do not something you ship — service is a verb

Big AffordableScale to wide range of customers, billions of sites, petabytes of data

Affordable for customers by making service quick, efficient & reliable

Easy Always availableDeployment, management & monitoring is trivial for us & customers, from small businesses to huge IT shops

99.9% guaranteed uptime, safe & secure—customers trust us with mission-critical data

It takes humans doing ongoing work — operational work, incident management & problem management

How does it look behind the scenes?

Layers of SharePoint Online

Services1+ services run within VM role Hundreds of services interacting

Virtual Machine RolesVMs performing different roles Units of scalability called

“Networks”

PhysicalDatacenters Machines Physical network

Physical machines are used purely as virtual machine hosts for compute-centric tasks:• Greatly improves manageability.

Deployment becomes entirely VHD-based.• Greatly improves predictability. The VHD

we test is the VHD we run in production.• Negligible performance impact from

virtualizing.

Virtual Machine Layer: Hyper-V

SharePoint Online Topology

WFE

App Server

Crawl WFE

CA

Timer Jobs

Sandbox

Content:

Fed AppFed

Query

Fed CA

Fed Idx

Federated Services:

SQL SQL

SQL:

SQL SQL AD AD

Directory:

Stamp 1:

WFE

App Server

Crawl WFE

CA

Timer Jobs

Sandbox

Content:

Fed AppFed

Query

Fed CA

Fed Idx

Federated Services:

SQL SQL

SQL:

SQL SQL AD AD

Directory:

Stamp 2..N:

Network 1..N:

AD Sync

Prov.

SCOM

ULS

SPDiag

WER

DNS

SMTP

Admin

Backup

NLB

NLB

Datacenter 1..N:

WFE

App Server

Crawl WFE

CA

Timer Jobs

Sandbox

Content:

Fed AppFed

Query

Fed CA

Fed Idx

Federated Services:

SQL SQL

SQL:

SQL SQL AD AD

Directory:

Stamp 1:

WFE

App Server

Crawl WFE

CA

Timer Jobs

Sandbox

Content:

Fed AppFed

Query

Fed CA

Fed Idx

Federated Services:

SQL SQL

SQL:

SQL SQL AD AD

Directory:

Stamp 2..N:

Network 1..N:

AD Sync

Prov.

SCOM

ULS

SPDiag

WER

DNS

SMTP

Admin

Backup

NLB

NLB

Disaster Recovery Datacenter 1..N:Grid

Manager

Global Directory

Tenant Admin (UI)

Commerce backend

DNS (multiple)

OrgID Auth, Svc.

Incident Management

Azure (Windows/SQL)

CDN Services

Grid ManagerGrid manager is made up of:Stateless front-ends

Contain sets of binariesAPIs/web servicesHandle of the Business Logic

Stateful back-ends SQL ServersStore information in databases

Remote orchestration scripts (“Jobs”)GM controls all remote components using remote PowerShell scripts

Grid Manager

StatefulDB’s

StatelessFront ends

GM DR Farm

API/web service

Grid Manager resides in a Network

Grid Manager operational principles• Grid Manager stores the state and info on all managed

objects (every box in the Topology slide)• All code (jobs, scripts) is idempotent; Grid Manager has about

200 Job types (e.g. DeployVM, ConfigureSQL, etc.) + remote scripts

• Each object’s state change request is logged as a new “desired state”

• Jobs are queued in the Job database, and executed according to the objects’ pre-defined state machine

• Grid manager runs code remotely to Central Admin boxes via PowerShell

• Note: the jobs that run are version/build sensitive

Job logic sample: DeployVM# FILENAME: DeployHyperVVirtualServer.ps1## Deploys a VM (logical server) in SharePoint Online

param(....) # ----------------------------------------------------------------------------- # Step 1 # Provisions/creates the VM in Hyper-V # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 2 # Misc. permissions are applied # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 3 # Execute Configure-$Role script # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 4 # Install SCOM Agent, configure Logging # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 5 # Misc configurations # -----------------------------------------------------------------------------

Job logic sample: DeployVM cont. # ----------------------------------------------------------------------------- # Step 6 # Configure outgoing network settings # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 7 # For SQL machines only, install specific H/W drivers # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 8 # Run verification tests # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 9 # Change Management bookkeeping # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 10 # Additional Driver updates # -----------------------------------------------------------------------------{…}

Job logic sample: DeployVM cont.# ----------------------------------------------------------------------------- # Step 11 # Test machine connectivity (only for SQL roles) # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 12 # Schedule patching job # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 13 # Wait for patch job to complete # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 14 # Connect the VM to the Global logging system # -----------------------------------------------------------------------------{...} # ----------------------------------------------------------------------------- # Step 15 # Adds the machine to the load balancer pools # NOTE: THIS MUST REMAIN AS THE LAST STEP # -----------------------------------------------------------------------------{...}

Deployment jobs sequence logic

Configure SQL Farm

Configure ID Mgmt Farms

Configure Fed Svcs Farm

Deployment jobs sequence logic cont.

Configure Content Farm

Post deployment

binary patching

Deployment jobs sequence logic cont.

Post deployment

binary patching

Prepare Content DBs

Zones, Networks, Farms, VMs…

New customer signup & tenant configuration

Layers of SharePoint provisioningOffice 365 Portals

Sign Up Experience

Office 365 Directory

Admin Experience

Sign up & select services Store all tenant info

Manage users & licenses, add services

SharePoint Online Prov.

Routing & Sync SharePoint Directory

Provisioning Service

Assign tenant to network & farm

Cache key tenant & user props

Get tenant ready & publish links

SharePoint Tenant Manager

Web Services Job Manager Job Queue

Schedule jobsManage all jobs, incl. prov All work to be done

SharePoint Provisioning Jobs Post-provisioning Runtime behaviors

Create site collections & config features

Additional sites & features

Light up features for tenant or user via licensing

We do this at high scale every day

30K new tenants a week, 4K new tenants a day

Handle tenant lifecycle changes for many tenants

Includes different offerings,2010 trials & 2013 preview

User onboarding, license assignments, Changing plans, de-provisioning

Spikes in demand, such as 2013 preview launch

Coordinate with other activities

Preview doubled our weekly load Upgrades, patching, etc.

We set up different tenant typesEnterprise Tenant team collaboration site

We set up different tenant typesDeveloper Tenant site has developer features turned on

Routing tenants to networks & farmsAssign tenants to network based on several factorsAvailable capacity: avoid routing to full farmsOperational activity: for example, upgradesGeographical locationTenant version: during betas support multiple versions of SharePointService plan: have the ability to route different types of tenants to different farms (e.g. Gov)

Network map can be adjusted at any timeWithin network, farms can also be opened or closedSame factors as for network management

Network map snapshotGlobal Map SharePoint/SPO-NW-05 8 Region: SA SharePoint/SPO-NW-01 4 SharePoint/SPO-NW-02 8 SharePoint/SPO-NW-03 8 Country: BR SharePoint/SPO-NW-04 1 Region: AS SharePoint/SPO-NW-09 4 SharePoint/SPO-NW-10 8

Region: OC SharePoint/SPO-NW-09 4 SharePoint/SPO-NW-10 8Region: EU;AF SharePoint/SPO-NW-05 4 SharePoint/SPO-NW-06 8 SharePoint/SPO-NW-07 8 SharePoint/SPO-NW-08 8Region: NA;AN SharePoint/SPO-NW-01 4 SharePoint/SPO-NW-02 8 SharePoint/SPO-NW-03 8

SharePoint tenants routed to multiple networks by region

Map also includes country-specific & plan-specific routing

Network map snapshotTag: o365.microsoft.com/version=15 Global Map SharePoint/SPO-NW-12 1 Region: AN;NA;SA SharePoint/SPO-NW-11 1 Region: AF SharePoint/SPO-NW-12 1 Region: EU SharePoint/SPO-NW-12 1 Region: AS SharePoint/SPO-NW-13 1 Region: OC SharePoint/SPO-NW-13 1

SharePoint 2013 preview tenants routed to specific networks per region

• Tenants are tagged with their experience version

• This map will expand as more of our capacity is opened to 2013 traffic

• Transition from all 2010, to separate 2010 & 2013 NWs, to all 2013.

Syncing changes for a tenantReceive a new tenant or detect changes to existing tenantChanges include: new users, license assignments, service plan changes, version changes

Sync down new information & store it in SharePoint Online directoryTrigger provisioning actions

Provisioning serviceWindows Service in charge of setting up tenantCreate SharePoint site collections for the tenantJobs run PowerShell scripts on SharePointIdempotent operations for resiliencyDifferent topology for different tenant types (for example, different site collections)

Set up DNS entries & domains via Internet DNS serviceCreate user groups & ensure data can be used by SharePoint (e.g. people picker)

Layers of provisioningPre-provisioningCreate tenants in advance

ProvisioningRename & assign pre-created tenants, or create tenants from scratchPublish cross-service links

Post provisioningAdditional configuration steps for all tenants (e.g. workflow) or scenario specific (e.g. Project)

Runtime behaviorsUse tenant & user license to control access to featuresProvisioning passes the licensing information into SharePoint

Provisioning role running in a specific network

Runs a specific version of the service fabric

Preprovision Job pre-creating tenants in different languages

Can handle tenant types & versions

Deploy Site job completed

Created site collection for a tenant

Several scenarios: Preprovision, create from scratch, delete/recreate

ReassignSite job executing to take over a site collection for a tenant

Main action is to rename the site

Enterprise tenant with Project Online

SharePoint 2013 experience

Calculate & set storage quota

Not a synthetic tenant

Tenant has Enterprise, Web Apps & Project service plans

Supporting SharePoint 2010 & 2013 tenants

Farm upgrade starts prior to General Availability

Upgrade farm to 2013 bits

Tenant version & UX remains 2010 for all existing tenants

Continue to route 2010 tenants to it—effectively a 2010 farm

Provisioning can create 2010 or 2013 tenants on upgraded farm

Create or modify 2010 or 2013 tenants

Tenants are tagged as a specific version when they sign up

Provision 2010 or 2013 site topology & features

For example: Small Business service plan

Open farm to 2013 tenants

Support both side by side

Phase out 2010 tenant provisioning

SharePoint tenant upgrade prep

Office 365 upgrade orchestration

Tenant upgrade execution

Monitor completion & inform Office 365 that tenant ready

Trigger update of tenant when all services ready

Notify tenant that upgrade is complete

Collect data from services on Tenant readiness to upgrade

Notify O365 upgrade layer that tenant is ready for upgrade

Monitor upgrade completion from workloads

Start upgrade of tenant: set version to 15

Prepare report on tenants ready to upgrade

Upgrading tenants to SharePoint 2013

Reprovision Tenant as 2013

Tenant Admin is in 2013 modeCreate new Site Collections in 2013 modeRun feature configPublish new cross service links

Upgrade in H/A Fashion

Upgrade should have minimal impact on customers.

Tenants should always be able to read their data during upgrade.

SharePoint Online Upgrade == Service Fabric Upgrade + SharePoint Core Product UpgradeRollback for failed Upgrades is a key scenario that we make sure is always robust

i.e. we guarantee an upgrade or rollback within the maintenance window.

Upgrade highlights

SharePoint Online Upgrade Flow

… …

Meta MSSUPA

Load Balancer

Federated farm

Content farm

BDC APSearch

SharePoint Stamp Upgrade

… …

Meta MSSUPA

Load Balancer

Old Federated farm

BDC APSearch

… …

Old Content farm

New Federated farm

1. New Fed. Svc. Farm

… …

Meta MSSUPA

Load Balancer

Old Federated farm

BDC

ROAPSearch

… …

Old Content farm

New Federated farm

AP

2. Upgrading Svc DBs

… …

ROMeta MSS

ROUPA

Load Balancer

Old Federated farm

ROBDC

ROAP

ROSearch

… …

Old Content farm

New Federated farm

Meta MSSUPA BDC APSearch

3. Finalizing the Upgrade

… …

Meta MSSUPA

Load Balancer

New Federated farm

Old Content farm

BDC APSearch

4. Removing old farm

… …

Meta MSSUPA

Load Balancer

New Federated farm

Old Content farm

BDC APSearch

… …

New Content farm

5. New Content Farm

… …

Meta MSSUPA

Load Balancer

New Federated farm

Old Content farm

BDC APSearch

… …

New Content farm

RO

6. Upgrading Content DBs

… …

Meta MSSUPA

Load Balancer

New Federated farm

BDC APSearch

… …

New Content farm

RO

RO

RO

… …

Old Content farm

7. Diverting Traffic

Q & A

Evaluate this session now on MySPC using your laptop or mobile device: http://myspc.sharepointconference.com

MySPC

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.