Presentedby:

Introduction

RussHoubergSharePointMCSMKnowledgeLakeProfessionalServices

russ@knowledgelake.comlinkedin/rhoubergtwitter/rhouberg

Up for discussion

Abriefhistory Concepts:SP2010 SP2013 Concepts:NewFunctionality FASTItsinthere ComponentTopologyOverhaul SchemaManagement PowerShellItsUnavoidable! ScalableArchitectureExamples ECMImpact Questions?

SharePoint2007 FASTESP

CustomConnector

SharePoint2010 FASTSearchforSharePoint2010

SearchSSAisFASTAware TheBypass:EnterpriseSearch FS4SP EnterpriseSearch

SharePoint2013 EnterpriseSearchw/FASTDNA

Hybridvigorbringsthebestofbothworlds

EnterpriseSearch(andECM)arestillgoingtobegenerallyonpremise(notcloud)

Hybridvigor

A brief history

Concepts: SP2010 SP2013Translating(orreplacing)thesyntax

SharePoint2010 SharePoint2013

ManagedProperty (Multiple) SearchSchemas

BestBets PromotedResults(QueryRule)

Scope andFederatedLocation ResultSource

ContentByQuery Content BySearch

Incremental Crawl ContinuousCrawl

Mirrored IndexPartition IndexReplica

MCM MCSM

Concepts: New FunctionalityNowwithbetterstuff

Concepts: New Functionality

CrawlingProcessingChanges

WenowhaveFull,IncrementalandContinuousCrawls

NomoreHostDistributionRules

NomoreFASTBypass

Nowwithbetterstuff

Concepts: New Functionality

ContinuousCrawlBenefits

Nomorewaitingforindexmerge

Doesnotwaitforothercrawlstocomplete

Canhavemultiplecontinuouscrawlsrunningsimultaneously

Continuouscrawlsignoreserrors

Continus CrawlFacts

Runsevery15minutesbydefault

DefaultintervalcanbechangedwithPowerShell

ShouldbeusedinsteadofincrementalcrawlsforSharePointcontentsources

Nowwithbetterstuff

Concepts: New Functionality

OtherBackendChanges

Schemamanagementatsitecollectionlevel

Crawldatabaserebalancing

ResultSources&RemoteSPIndex

Nowwithbetterstuff

Concepts: New Functionality

QueryRules QueryRuletoimplement

PromotedResults(bestbets) QueryRuletoaugmentsearch

resultswithentitymetadata(fromSPorBCS)

QueryRuletomanipulateresultsranking

Nowwithbetterstuff

Concepts: New Functionality

DEMOContentBySearch WebPart

Nowwithbetterstuff

FAST Its in thereFASTbitsarenowblendedin

FAST Its in there

ComponentsoftheFASTcrawlprocessing,indexstructure,relevanceandrankingmodels,entityextraction,andpipelineextensibilityoptions(throughwebservices)areallinfluencesofFASTtechnology.

FASTbitsarenowblendedin

Component Topology OverhaulBreakingoutoftheshell

Component Topology Overhaul

WhatitDoes

Crawlscontentsourcestopopulateindex

Deliverscrawlitems(binary)andmetadatatocontentprocessor

Invokesconnectorsorprotocolhandlerstointeractwithcontentsourcestoretrievedata

Usesoneormorecrawldatabasestostoreinfoaboutcrawlitemsandcrawlhistory

ImportantFacts

Wecanhavemultiplecrawlcomponents

MSRecommends:2CrawlComponentsperSearchServiceApplication

MSRecommends:8(4vm)CPU/8GBRAMperCrawlComponent

CrawlComponent

Component Topology Overhaul

WhatitDoes

Processescrawlitemsandfeedstoindexcomponent

Transformscrawlitemsintoartifactsthatcanbeincludedinsearchindex(Performsdocumentparsingandpropertymapping)

Writesinformationaboutlinksandurls inlinkdatabase(whichareanalyzedbyanalyticstocalculaterelevanceandcurrency Resultswrittenbacktosearchindexbycontentprocessingcomponent

Generatesphoneticnamevariationstoimprovepeoplesearch

ImportantFacts

Wemustonlyhaveone(1)crawlprocessingcomponentperserver morewillhurt,nothelpcrawlperformance

Maxof2persearchserviceapplication FeedingSessionsarescaledbasedonCPUcoresusingadefaultcoefficientof3

8(cores)*3=24feedingsessions4(cores)*3=12feedingsessions

MSRecommends:8(4vm)CPU/8GBRAMperContentProcessingComponent

FeedingsessionsrequireRAM MoreRAMisnecessarywhenmorecoresarepresent monitoringrequired

ContentProcessingComponent

Component Topology Overhaul

WhatitDoes Runsanalyticsjobsthatanalyzecrawlitemsanduserinteractionwithsearchresultstoperformbothsearchanalyticsandusageanalytics

AnalyzesLink&Anchortextanalysis,Cleardistance,SearchClicks,DeepLinks,SocialTags,SocialDistance,SearchReports,Recommendations,UsageCounts,ActivityRanking

Improvessearchrelevanceandcreatesearchresults

Outputincludedinsearchindexbycontentprocessor

ImportantFacts Maximumof6persearchserviceapplication AddmoreAnalyticsProcessingComponentstoimproveanalyticsperformance

MSRecommends:8(4vm)CPU/8GBRAM/300GBdiskspaceperAnalyticsProcessingComponent.

InteractswithAnalyticsReportingtostorestatisticalinformation

InteractswithLinkdatabasetostoreinformationaboutsearchesandcrawleddocuments

AnalyticsProcessingComponent

Component Topology Overhaul

WhatitDoes

Receivesprocesseditemsfromcontentprocessingcomponentandwritestheitemstotheindexfile

Receivesqueriesfromthequeryprocessingcomponentandreturnsresultsets

RedistributescontentamongindexpartitionswhenindexarchitectureischangedbySearchAdministrationComponent

ImportantFacts

Maximumof60indexpartitions(20indexpartitionsX3indexreplicas)persearchserviceapplication

MustprovisiononeIndexComponentforeachindexreplica.

MSRecommends:8(4vm)CPU/16GBRAM/500GBdiskspaceperIndexComponent.

IndexComponent

Component Topology Overhaul

Indexpartitionislogicalportionofentiresearchindex(sameasbefore)

Indexpartitionisservedbyoneormoreindexcomponents Indexcomponentscanbeprimary"replica"orsecondary"replica" PrimaryReplicaiscontactedbycontentprocessingcomponenttowritenewdataintheindex

SecondaryReplicaisreadonlycopythatgetupdatedwiththedata. Addingreplicasimprovesqueryperformanceunderload Addpartitionstohandleincreasedcontentcorpus Can'tremovepartitionafterithasbeenadded.

IndexArchitecture

IndexPartitionsandReplicas

Component Topology Overhaul

WhatitDoes

Analyzesandprocessesqueriesandresults

Afterreceivingaquery,itanalyzesandprocessesthequerytooptimizeprecision,recallandrelevance

Submitsprocessedqueriestotheindexcomponent

Processestheresultsetreturnedbytheindexcomponentbeforereturningtothequeryingentity.

ImportantFacts

Maximumof1perserver MSRecommends:8(4vm)CPU/8GBRAMperQueryProcessingComponent.

QueryProcessingComponent

Schema Management Schemachanges Granularconfiguration Canbeextendedatthesite

collectionlevelormanagedgloballybyfarmadmin

Fullcrawlneededtopickupchanges

ItsallaboutSchemanow

PowerShell Its unavoidableYournewbestfriend

MINIDEMOWhatyoucantdowiththeUI

PowerShell Its unavoidableGetSPEnterpriseSearchStatusNewSPEnterpriseSearchAdminComponentGetSPEnterpriseSearchCrawlContentSourceGetSPEnterpriseSearchCrawlCustomConnectorGetSPEnterpriseSearchCrawlDatabaseGetSPEnterpriseSearchCrawlExtensionGetSPEnterpriseSearchCrawlMappingGetSPEnterpriseSearchCrawlRuleNewSPEnterpriseSearchCrawlComponentNewSPEnterpriseSearchCrawlContentSourceNewSPEnterpriseSearchCrawlCustomConnectorNewSPEnterpriseSearchCrawlDatabaseNewSPEnterpriseSearchCrawlExtensionNewSPEnterpriseSearchCrawlMappingNewSPEnterpriseSearchCrawlRuleRemoveSPEnterpriseSearchCrawlContentSourceRemoveSPEnterpriseSearchCrawlCustomConnectorRemoveSPEnterpriseSearchCrawlDatabaseRemoveSPEnterpriseSearchCrawlExtensionRemoveSPEnterpriseSearchCrawlMappingRemoveSPEnterpriseSearchCrawlRuleSetSPEnterpriseSearchCrawlContentSourceSetSPEnterpriseSearchCrawlDatabaseSetSPEnterpriseSearchCrawlRuleSetSPEnterpriseSearchCrawlLogReadPermissionRemoveSPEnterpriseSearchCrawlLogReadPermissionRemoveSPEnterpriseSearchCrawlLogReadPermissionImportSPEnterpriseSearchTopologyExportSPEnterpriseSearchTopologySetSPEnterpriseSearchTopologyRemoveSPEnterpriseSearchTopologyRemoveSPEnterpriseSearchComponentNewSPEnterpriseSearchTopologyNewSPEnterpriseSearchQueryProcessingComponentNewSPEnterpriseSearchIndexComponent

NewSPEnterpriseSearchContentProcessingComponentGetSPEnterpriseSearchTopologyGetSPEnterpriseSearchServiceGetSPEnterpriseSearchServiceApplicationGetSPEnterpriseSearchServiceApplicationProxyGetSPEnterpriseSearchServiceInstanceNewSPEnterpriseSearchServiceApplicationNewSPEnterpriseSearchServiceApplicationProxyRemoveSPEnterpriseSearchServiceApplicationRemoveSPEnterpriseSearchServiceApplicationProxyRestoreSPEnterpriseSearchServiceApplicationResumeSPEnterpriseSearchServiceApplicationSetSPEnterpriseSearchServiceSetSPEnterpriseSearchServiceApplicationSetSPEnterpriseSearchServiceApplicationProxyStartSPEnterpriseSearchServiceInstanceStopSPEnterpriseSearchServiceInstanceSuspendSPEnterpriseSearchServiceApplicationUpgradeSPEnterpriseSearchServiceApplicationBackupSPEnterpriseSearchServiceApplicationIndexUpgradeSPEnterpriseSearchServiceApplicationSiteSettingsRestoreSPEnterpriseSearchServiceApplicationIndexRemoveSPEnterpriseSearchServiceApplicationSiteSettingsGetSPEnterpriseSearchOwnerSuspendSPEnterpriseSearchServiceApplicationSetSPEnterpriseSearchServiceInstanceGetSPEnterpriseSearchMetadataCategoryGetSPEnterpriseSearchMetadataCrawledPropertyGetSPEnterpriseSearchMetadataManagedPropertyGetSPEnterpriseSearchMetadataMappingNewSPEnterpriseSearchMetadataCategoryNewSPEnterpriseSearchMetadataCrawledPropertyNewSPEnterpriseSearchMetadataManagedPropertyNewSPEnterpriseSearchMetadataMappingRemoveSPEnterpriseSearchMetadataCategoryRemoveSPEnterpriseSearchMetadataManagedPropertyRemoveSPEnterpriseSearchMetadataMappingSetSPEnterpriseSearchMetadataCategory

SetSPEnterpriseSearchMetadataCrawledPropertySetSPEnterpriseSearchMetadataManagedPropertySetSPEnterpriseSearchMetadataMappingGetSPEnterpriseSearchPropertyRuleCollectionGetSPEnterpriseSearchPropertyRuleGetSPEnterpriseSearchQueryAndSiteSettingsServiceGetSPEnterpriseSearchQueryAndSiteSettingsServiceInstanceGetSPEnterpriseSearchQueryAndSiteSettingsServiceProxyGetSPEnterpriseSearchQueryAuthorityGetSPEnterpriseSearchQueryDemotedGetSPEnterpriseSearchQueryKeywordGetSPEnterpriseSearchQueryScopeGetSPEnterpriseSearchQueryScopeRuleGetSPEnterpriseSearchQuerySuggestionCandidatesGetSPEnterpriseSearchRankingModelGetSPEnterpriseSearchSecurityTrimmerNewSPEnterpriseSearchQueryAuthorityNewSPEnterpriseSearchQueryDemotedNewSPEnterpriseSearchQueryKeywordNewSPEnterpriseSearchQueryScopeNewSPEnterpriseSearchQueryScopeRuleNewSPEnterpriseSearchRankingModelNewSPEnterpriseSearchSecurityTrimmerRemoveSPEnterpriseSearchQueryAuthorityRemoveSPEnterpriseSearchQueryDemotedRemoveSPEnterpriseSearchQueryKeywordRemoveSPEnterpriseSearchQueryScopeRemoveSPEnterpriseSearchQueryScopeRuleRemoveSPEnterpriseSearchRankingModelRemoveSPEnterpriseSearchSecurityTrimmerSetSPEnterpriseSearchQueryAuthoritySetSPEnterpriseSearchQueryKeywordSetSPEnterpriseSearchQueryScopeSetSPEnterpriseSearchQueryScopeRuleSetSPEnterpriseSearchRankingModelStartSPEnterpriseSearchQueryAndSiteSettingsServiceInstanceStopSPEnterpriseSearchQueryAndSiteSettingsServiceInstanceImportSPEnterpriseSearchPopularQueries

SetSPEnterpriseSearchResultItemTypeSetSPEnterpriseSearchQuerySpellingCorrectionRemoveSPEnterpriseSearchResultItemTypeNewSPEnterpriseSearchResultItemTypeImportSPEnterpriseSearchThesaurusGetSPEnterpriseSearchResultItemTypeGetSPEnterpriseSearchQuerySpellingCorrectionGetSPEnterpriseSearchLanguageResourcePhraseGetSPEnterpriseSearchSiteHitRuleNewSPEnterpriseSearchLanguageResourcePhraseNewSPEnterpriseSearchSiteHitRuleRemoveSPEnterpriseSearchLanguageResourcePhraseRemoveSPEnterpriseSearchSiteHitRuleGetSPEnterpriseSearchVssDataPathGetSPEnterpriseSearchContentEnrichmentConfigurationSetSPEnterpriseSearchPrimaryHostControllerSetSPEnterpriseSearchLinguisticComponentsStatusSetSPEnterpriseSearchContentEnrichmentConfigurationRemoveSPEnterpriseSearchContentEnrichmentConfigurationNewSPEnterpriseSearchContentEnrichmentConfigurationGetSPEnterpriseSearchLinguisticComponentsStatusGetSPEnterpriseSearchHostControllerSetSPEnterpriseSearchLinksDatabaseRepartitionSPEnterpriseSearchLinksDatabasesMoveSPEnterpriseSearchLinksDatabasesRemoveSPEnterpriseSearchTenantSchemaRemoveSPEnterpriseSearchTenantConfigurationRemoveSPEnterpriseSearchLinksDatabaseRemoveSPEnterpriseSearchFileFormatNewSPEnterpriseSearchLinksDatabaseNewSPEnterpriseSearchFileFormatNewSPEnterpriseSearchAnalyticsProcessingComponentImportSPEnterpriseSearchCustomExtractionDictionaryGetSPEnterpriseSearchLinksDatabaseGetSPEnterpriseSearchFileFormatSetSPEnterpriseSearchFileFormatStateGetSPEnterpriseSearchComponentGetSPEnterpriseSearchServiceApplicationBackupStore

Yournewbestfriend

Scalable Architecture Examples

SmallSearchFarm(~10MillionItems)|4PhysicalServers|6VirtualServers

Scalable Architecture Examples

MediumSearchFarm(~40MillionItems)|8PhysicalServers|14VirtualServers

Scalable Architecture Examples

LargeSearchFarm(~100MillionItems)|18PhysicalServers|28VirtualServers

Scalable Architecture Examples

???

ExtremeSearchFarm(>100MillionItems)

ECM Impact

Schemacanbemanagedbysiteadmins,reducingtheloadonsearchadministrator

Schemaisconfiguredtoallowmoregranularity(query,retrieve,refine,sort,etc) Affectscontentindexsize

Remoteresultsourcescanbecrawledlocallyandthenqueriedbyremotefarms.HugeimpactongeodistributedsearchThirdpartysolutionsmaybeabletofacilitateblendedsearchresults!

Individualitemscanberecrawledeasily

AutomaticURLbalancingincrawldatabasesminimizeshostnamerestrictionsforlargearchiverepositories

Scalabilitylimitchangeswillhaveabigimpactonfarmdesignforlargearchivecontentrepositoriesinthenearfuture.

Whatdoesitallmean?

Questions?

StumptheChump(youprobablycan)

Thankyou!

The end

Search ArchitectureIntroductionUp for discussionA brief historyConcepts: SP2010 SP2013Concepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityConcepts: New FunctionalityFAST Its in thereFAST Its in thereComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulComponent Topology OverhaulSchema ManagementPowerShell Its unavoidablePowerShell Its unavoidableScalable Architecture ExamplesScalable Architecture ExamplesScalable Architecture ExamplesScalable Architecture ExamplesECM ImpactQuestions?The end