Search Topology and OptimizationMike MaadaraniCanada, -UTC 5:00 EST

April 16th /17th, 2014

App Dev and Architecture for over 18 years (15 Years Microsoft, 3 Years with the “Other Guys”)

Business focused on Enterprise Content Management, Publishing Sites, & Search

Technology focused on SharePoint, SQL Server and SharePoint Integration

Architect, trainer, and presenter

Mike MaadaraniSharePoint ArchitectNavigatusOttawaCanada

Contact

@mikemaadaranimike@maadarani.comwww.maadarani.com

SP24 Search SessionsSession

Time (GMT)

Time (EST) Title Level Speaker Twitter Handle

#SP24S090 5:00 1:00Using the Content Enrichment Web Service with SharePoint Server 2013 Search

Technical - Advanced

Sezai Komur - NEC Australia

@sezai

#SP24S021 6:00 2:002013 Search, Display Templates, Query Rules, Result Types

Technical - Beginner

Omer Zubair - CSC

@Omer_Zubair

#SP24S047 8:00 4:00Real-life experience building search-driven applications for product-centric sites

Technical - Advanced

Marius Constantinescu - blue-infinity SA

@c_marius

#SP24S032 10:00 6:00Custom Indexing Connectors - How to integrate external system into your SharePoint Enterprise Search

Technical - Advanced

Heinrich Ulbricht - Communardo Software GmbH

@h_ulbricht

#SP24S083 11:00 7:00Search First Migration with SharePoint 2013 – Benefits / Disadvantages

Technical - Intermediate

Max Melcher - Alegri International Service GmbH

@maxmelcher

#SP24S034 14:00 10:00The Search Immaturity Cycle, and How to Create a Search Strategy

Business - Intermediate

Jeff Fried - BA Insight

@jefffried

#SP24S009 16:00 12:0010 Things to Know about Search in SP2013 and Office 365

Business - Beginner

Agnes Molnar - Search Explained

@molnaragnes

#SP24S019 18:00 14:00 Search Topology and OptimizationTechnical - Advanced

Mike Maadarani - MCM Consulting

@mikemaadarani

#SP24S050 22:00 18:00Search Driven Application Development in SharePoint 2013

Technical - Intermediate

Matt Youngstrom - Magenic

@spguru

Configuring SSA and PS

Topology ScenariosAgenda

Relevancy, Query Builder, &Optimization

SharePoint 2013 Search Overview

Architecture and Resource Utilization

Search in SharePoint 2010

Crawl Component

Query Component

SharePoint 2010 Search Service Application

Crawl Indexing Engine

Query Engine

Search Admin

Property Store (SQL)

Content

UserWFE

FAST Search for SharePoint 2010

FAST Content SSA

FAST Query SSA

FAST back-end components(managed separately)

Extensibility:• Sandbox• Entity

Extraction

Crawl Indexing Engine

Query Engine

Content Pipeline

Analysis Engine

Query Pipeline

Search AdminContent

UserWFE

… In SharePoint 2013

SharePoint 2013 Search Service Application

Index Component

Query Engine

Content Pipeline

Content ProcessingComponent

CrawlComponent

Query ProcessingComponent

AnalyticsProcessingComponent

Query Pipeline

Search Admin

Admin Component

Entire index on local disk

Property Store (SQL)

Content

UserWFE

Analysis Engine

Crawl Indexing Engine

Link/query analysis & recommendations

Separate crawl and indexing

Extensibility:• Web

callout• Entity

Extraction

SharePoint 2013 Search Architecture

SearchAdmin

Content UserCrawlContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

Crawl

Search Admin

Link

Analytics Reporting

FAST Search Index

SharePointSP AppsDevicesNon-SP UX

HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS

Public API

Search topology components

Content Query

Why Search is so important?

I just uploaded a document. Make it searchable, quick!

FAST

Why Search is so important?EASY

Why Search is so important?EASY

Why Search is so important?Search Driven Applications

noderunner.exe noderunner.exe noderunner.exe noderunner.exe

Where is Search?

Windows servicesSharePoint Search Host Controller service

Runtime/lifecycle control of search components (except crawler) hostcontrollerservice.exe

SharePoint Server Search service

Crawl Componentmssearch.exemssdmn.exe

ProcessesNoderunner.exe

Runtime environment for search components (except crawler)

msseearch.exemssdmn.exe

CrawlComponentnoderunner.exe

Search Runtime Environment

hostcontrollerservice.exe

Host Controller

Sh

are

Poin

t A

pp

Serv

er

Admin entitiesSearch Service Instance: Provisioning of the search service on each boxSearch Service Application: SharePoint Configuration entity

Still there, but only Crawl Component

AdminComponent

Query ProcessingComponent

Content ProcessingComponent

IndexComponent

Analytics ProcessingComponent

Where do I host my components?

CPU loadDriving factors

QPS

Query transformations

Network loadDriving factors

Number of index partitions

Size of queries and results

Example:

20 index partitions @ 20 qps => 200/100 Mbit/s in/outbound

Query processing component (QPC)

Item count

DPS QPS

Load impact (relative)

CPU NetworkDisk

CPU loadDriving factors

QPS and item count

Guidelines per index component @ 2 GHz CPU

1M items: 5 QPS per CPU core

5M items: 2 QPS per CPU core

10M items: 1 QPS per CPU core

Disk loadDriving factors

QPS and item count

New content invalidates caches

Disk size: 500GB @ 10M items per index component

Index component

Item count

DPS QPS

Load impact (relative)

CPU NetworkDisk

Crawl component

CPU loadDriving factors

Documents per second

Link discovery

Crawl management

Network loadDriving factors

Downloading items from content sources

Passing items on to CPC

Disk loadAll documents are temporarily stored in data folder

Item count

DPS QPS

Load impact (relative)

CPU NetworkDisk

Content processing component

CPU loadDriving factors

Documents per second

Document size and complexity

Feature extractionEstimate: 5-10 DPS per CPU core

Network loadDriving factors

Documents per second

Document size

Item count

DPS QPS

Load impact (relative)

CPU NetworkDisk

Analytics processing component

CPU loadDriving factors

Number of items

Site activity

Disk loadLocal disk used for temporary storage

Bulk load, primacy concern is load isolation

Network loadSame as for CPU load

PLUS: Network traffic increases when distributing APC across multiple machines

Item count

DPS QPS

Load impact (relative)

CPU NetworkDisk

Search administration component

Low CPU and network load

Load increase with more components in the search topology Item

countDPS QPS

Load impact (relative)

CPU NetworkDisk

Create your SSA$SSADB = "SharePoint_Demo_Search"$SSAName = "Search Service Application SP24"$SVCAcct = "mcm\sp_search"$SSI = get-spenterprisesearchserviceinstance -local

#1. Start the search services for SSIStart-SPEnterpriseSearchServiceInstance -Identity $SSI

#2. Create the Application Pool$AppPool = new-SPServiceApplicationPool -name $SSAName"-AppPool" -account $SVCAcct

#3. Create the search application and set it to a variable$SearchApp = New-SPEnterpriseSearchServiceApplication -Name $SSAName -applicationpool $AppPool -databaseserver SQL2012 -databasename $SSADB

#4. Create search service application proxy$SSAProxy = new-SPEnterpriseSearchServiceApplicationProxy -name $SSAName" Application Proxy" -Uri $SearchApp.Uri.AbsoluteURI

#5. Provision Search Admin ComponentSet-SPEnterpriseSearchAdministrationComponent -searchapplication $SearchApp -searchserviceinstance $SSI

#6. Create the topology$Topology = New-SPEnterpriseSearchTopology -SearchApplication $SearchApp

#7. Assign server(s) to the topology$hostApp1 = Get-SPEnterpriseSearchServiceInstance -Identity "SP2013"New-SPEnterpriseSearchAdminComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchCrawlComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchContentProcessingComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchAnalyticsProcessingComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchQueryProcessingComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchIndexComponent -SearchTopology $Topology -SearchServiceInstance $hostApp1 –IndexPartition 0

#8. Create the topology$Topology | Set-SPEnterpriseSearchTopology

Small Search Topology

Fault tolerant small search topology

Host

VM

Index QPC

VM

Admin

Crawl

CPC

APC

Host

VM

Index QPC

VM

Admin

Crawl

CPC

APC

Other SharePoint applications

Web front end

Admin

Crawl

CPC

APC

Index

QPC

Small search farm (up to 10M items)

Resources @ 10M items8x CPU cores24 GB RAM800 GB disk

Sized independently

Separate disk

for index

Scaling to medium search topology

Adm

Crawl

Index Index IndexIndex QPCCPC

APC

Adm

Crawl

Index Index Index IndexQPCCPC CPC

APC

QPC

Extend your SSA#2. Extend the Search Topology:$hostApp1 = Get-SPEnterpriseSearchServiceInstance -Identity "SP2013"$hostApp2 = Get-SPEnterpriseSearchServiceInstance -Identity "SPSearch"Start-SPEnterpriseSearchServiceInstance -Identity $hostApp1Start-SPEnterpriseSearchServiceInstance -Identity $hostApp2

#3. Keep running this command until the Status is Online:Get-SPEnterpriseSearchServiceInstance -Identity $hostApp1 Get-SPEnterpriseSearchServiceInstance -Identity $hostApp2

#4. Once the status is online, you can proceed with the following commands:$ssa = Get-SPEnterpriseSearchServiceApplication$active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa –Active$newTopology = New-SPEnterpriseSearchTopology -SearchApplication $ssaNew-SPEnterpriseSearchAdminComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchCrawlComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchContentProcessingComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1

New-SPEnterpriseSearchAnalyticsProcessingComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchQueryProcessingComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1New-SPEnterpriseSearchIndexComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp1 –IndexPartition 0New-SPEnterpriseSearchAdminComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2New-SPEnterpriseSearchCrawlComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2New-SPEnterpriseSearchContentProcessingComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2New-SPEnterpriseSearchAnalyticsProcessingComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2New-SPEnterpriseSearchQueryProcessingComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2New-SPEnterpriseSearchIndexComponent -SearchTopology $newTopology -SearchServiceInstance $hostApp2 –IndexPartition 1

#5. Activate the topology:Set-SPEnterpriseSearchTopology -Identity $newTopology

Medium Search Topology

Tweaking Your Results

Challenges: IntentWhere is my talk

Project Plan?

Are Documents held at the same place?

I wonder if there are references from

previous projects?Different people have different intentsQuery Rules help you handle intents

There is rarely a single right answer

Infrastructure Project

Conceptual Relevance Flow

For all queries:

Authorities: Level 1: http://employment

Ranking model: {incorporate user ratings}

Query:HR Employmentquarterly report

Search Web Part

Query Processing Engine

Document Collection

Thesaurus: HR Human ResourcesBest bets: HR Employment /HR/employment

(WORDS HR, Human Resources) AND(WORDS employees, employed) AND (WORDS quarterly, quarterlies) AND(WORDS report, reports, reported)

Mixed Results for:• HR Employment best bet• HR Employment quarterly

report• HR Employment

ContentType=reports

Dynamic Reordering Rules: Quarterly Report {prefer docs from http://reports}

Query Rule: {Terms} Quarterly Report {Terms} ContentType=“reports”

Authorities: SSA-level configuration

Sites that are important

Sites with low intrinsic relevance

Takes ~24hrs to propagate

Authorities: Connected

Authorities: Connected

1

0

1

1

2

4

3

2

∞

4

Setting an authority affects all sites connected through hyperlinks

Sites are weighted by distance to the authority

Tune Search Results

Created at the SSA, Tenant, Site Collection or SiteSSA

Site Collection

Site

Query Rules

ConditionWhen Do I apply the rule?

ActionWhat to do when the rule is matched?

PublishingWhen should the rule be active?

Query Rules

Exact match, beginning or end Ad-hoc or term store dictionary Match a regex (advanced) Is this query more likely aimed at

the following source…? Do people mostly click on result of

the following type…?

Conditions Show a promoted result Show a block of results Replace the core results

with a different query

Actions

Query Rules

Dynamically Ranking Change

Part of the query

Results Ranking

Query Builder

Query Builder

High Availability and Performance

Better Search Quality

Better management

Friendly results and tools

Session Objective and Takeaways

...the anchor will pick some of them up!

Questions?Post your questions in the chat...

SP24 Search SessionsSession

Time (GMT)

Time (EST) Title Level Speaker Twitter Handle

#SP24S090 5:00 1:00Using the Content Enrichment Web Service with SharePoint Server 2013 Search

Technical - Advanced

Sezai Komur - NEC Australia

@sezai

#SP24S021 6:00 2:002013 Search, Display Templates, Query Rules, Result Types

Technical - Beginner

Omer Zubair - CSC

@Omer_Zubair

#SP24S047 8:00 4:00Real-life experience building search-driven applications for product-centric sites

Technical - Advanced

Marius Constantinescu - blue-infinity SA

@c_marius

#SP24S032 10:00 6:00Custom Indexing Connectors - How to integrate external system into your SharePoint Enterprise Search

Technical - Advanced

Heinrich Ulbricht - Communardo Software GmbH

@h_ulbricht

#SP24S083 11:00 7:00Search First Migration with SharePoint 2013 – Benefits / Disadvantages

Technical - Intermediate

Max Melcher - Alegri International Service GmbH

@maxmelcher

#SP24S034 14:00 10:00The Search Immaturity Cycle, and How to Create a Search Strategy

Business - Intermediate

Jeff Fried - BA Insight

@jefffried

#SP24S009 16:00 12:0010 Things to Know about Search in SP2013 and Office 365

Business - Beginner

Agnes Molnar - Search Explained

@molnaragnes

#SP24S019 18:00 14:00 Search Topology and OptimizationTechnical - Advanced

Mike Maadarani - MCM Consulting

@mikemaadarani

#SP24S050 22:00 18:00Search Driven Application Development in SharePoint 2013

Technical - Intermediate

Matt Youngstrom - Magenic

@spguru

Thank you!