BA Insight SharePoint 2013 Enterprise Search Guide

The Essential Guide to Enterprise Search in SharePoint 2013Everything You Need to Know to Get the Most Out of Search and Search-based Applications

SharePoint 2013 The essenTial guide To enTerprise search 2

Jeff Fried, cTo, Ba insightJeff is a long-standing search nerd. he was the Vp of products for semantic search company lingoMotors, Vp of advanced solutions for FasT search, and technical product manager for all Microsoft enterprise search products. he is also a frequent writer, who has authored 50 technical papers and co-authored two new books on sharepoint and search. he holds over 15 patents, and routinely speaks at industry events.

agnes Molnar, MVpagnes is a Microsoft sharepoint MVp and a senior solutions consultant for Ba insight. she has also co-authored and contributed to several sharepoint books. she is a regular speaker at technical conferences and symposiums around the world.

Michael himelstein, vTspMichael has more than 20 years of practical experience developing, deploying, and architecting search-based applications. in this role he has advised hundreds of the largest companies around the world around unified information access. He was previously a Technology solutions Manager in the enterprise search group at Microsoft.

Tony Malandain Tony Malandain is a co-founder of BA-Insight. Tony architected and built the first version of the product which gained significant momentum on the Microsoft Office sharepoint server (Moss) and positioned Ba insight as the leading enhanced search vendor for sharepoint. Tony was awarded a patent for the core aptivrank technology, which monitors usage behavior of search users to influence relevancy automatically.

eric Moore eric Moore is the lead for Ba insight’s search interactions and content enrichment teams. he is accustomed to living at the leading edge of search, and has deep experience with multimedia search, XMl search, and content enrichment. prior to Ba Insight, Eric worked for five years at FAST and on the Microsoft Search Platform team. eric has developed state of the art products, algorithms, and platforms for specialized information workers.

about the authorS


IntroductIon


There’s a lot to say about sharepoint 2013, and about search in sharepoint 2013. This e-book is focused only on search, and is meant to give you a working understanding of the new features so that you can get oriented with them and think about how you will deploy and use them. it does not try to cover everything, nor is it meant to be a hands-on guide.

In this book we will be covering five key areas as they relate to search. These key areas are color coded, and represented by the blocks below. each section contains short chapters that can be read independently or continuously. The goal is to enable readers to focus on the information they need to learn about at the moment.

user experience

Working with Queries

& results

Working with

Content

architecture, Deployment &

operations

applications & Development

not every area of search has changed in sharepoint 2013, and those that are currently familiar with search won’t be lost at sea. For example, the deployment model, services architecture, and crawling and connector subsystems are pretty much the same as with sharepoint 2010. end users will see a dramatically different search ui, but they will be able to use it with no training (it’s quite intuitive). if you have built up a competency in search, you’ll be able to take it further in many ways — which we highlight throughout this e-book.

deeper dives:Technet — What’s new in SharePoint 2013 search

Blog article from Microsoft Search Group

TechNet landing page refreshed weekly with articles on SharePoint 2013

Highlights of Search in SharePoint 2013

What’S in thiS e-book?

http://technet.microsoft.com/en-us/library/ee667266%28v=office.15%29.aspx

http://sharepoint.microsoft.com/blog/Pages/BlogPost.aspx?pID=1030

http://technet.microsoft.com/en-us/library/cc262043%28v=office.15%29.aspx

http://www.bainsight.com/resources-for-sharepoint-fast-search/Pages/SharePoint-Search-2013-eBook.aspx

highlights and Key Take-aways

What’S neW? There’s a new development model for sharepoint 2013

generally, and for Search specifically.

There’s a new content extensibility Web service (ceWs) that opens up content processing for extension.

search is used pervasively throughout the sharepoint 2013 platform, and powers the new web content management (WcM) and e-discovery capabilities, topic pages, the content-by-search web part, myTasks, mysiteView, and more — along with great enterprise search, people search, and site search.

benefitS This makes extending search much more accessible, and

will foster a lot of exciting search-based applications.

a lot of great possibilities are now open to developers.

Your users will get more done and enjoy a variety of applications, both built in and tailored — all powered by search.



What’S in thiS e-book?

What’S neW? under the hood, there is a new architecture, a new search

core, and many new modules that are the culmination of the FasT acquisition — not just combining the best of FasT and SharePoint search, but some significant innovations from a continued investment in search.

benefitS search deployment and management is different, and largely

better. Making search hum for o365 — fully multi-tenant, smoothly scalable and fault-tolerant, and manageable at multiple levels — was a key goal for this release and there are big benefits for on-premise deployments too.

architecture, Deployment & operations

user experienceWhat’S neW? The face of search is totally revamped — not just in

keeping with the new sharepoint uX overall, but with deep refinements, better display for results using Result Blocks, a hover panel with previews, and more.

benefitS The search experience is easy, clean, and fast.

Working with Queries & resultsWhat’S neW? in sharepoint 2013 search scopes, federated locations, and best

bets are now deprecated in favor of result sources, query rules, and result templates.

benefitS sharepoint 2013 is light-years ahead of other search platforms

in this area. result sources, query rules, and result templates off remarkable control over search presentation. These are brand-new concepts, well worth learning — they arm site administrators and site collection administrators with the tools to field powerful, effective search.

Working with ContentWhat’S neW? crawling is an area that has changed least with sharepoint

2013, but there are still some great enhancements, including continuous crawling.

Business connectivity services has continued to evolve and now supports claims tokens through the Bdc.

The content processing and linguistics capabilities in sharepoint 2013 search are very strong and extensible. There’s lots of new capabilities including a completely new file parsing mechanism.

benefitS With continuous crawling, users get fresher content faster.

complex security scenarios are more tractable (though

still hard).

This platform offers a lot of power to developers, as well as providing some key capabilities end users will notice.

6 introduction SharePoint 2013 Search is here

7 Chapter 1 user experience — the new face of Search in SharePoint 2013 8 raising the Bar: The sharepoint 2013 user experience 10 First class search interactions: More to love 12 The sharepoint 2013 search center overview 14 Refiners and Faceted Navigation 16 search center setup

18 Chapter 2 Working with Queries and results — new Mechanisms in SharePoint 201319 Query processing: the search engine’s automatic Transmission22 Query rules and Query suggestions 26 result Types and result Templates

28 Chapter 3 Working with Content — Crawling, Connectors, and Content Processing29 content capture 33 content processing 36 linguistics processing

40 Chapter 4 architecture, Deployment, and operations — Getting under the hood41 new architecture, single search engine core 45 indexing and partitions 47 analytics 49 Federation and result sources 52 search in exchange54 search administration 58 upgrade and Migration

63 Chapter 5 applications and Development — new Models for Search-based applications64 The new development Model in sharepoint 2013 69 The content enrichment Web service (ceWs) 71 search-Based applications in sharepoint 2013

77 Conclusion

table of ContentS


sharepoint 2013 search is here There’s a new search in Townsharepoint 2013 has arrived, and it is chock full of new capabilities and features. This is a release with major architectural changes, built “for the next 15 years”, and it is very different from sharepoint 2010.

With sharepoint 2013, the enterprise search capabilities are dramatically different and very exciting. search has a new face, a new development model, and some remarkable built-in features. For search Jedis this new platform has a lot to love, it is:

• clean, fast, and easy to use.

• straightforward to install, administer, and scale.

• provides very powerful high-end search features.

• Makes creating search-based applications simpler than ever.

For search Jedi apprentices, this release will change your world. search is the “Force” used pervasively throughout sharepoint 2013 and has the power to transform the way your business uses sharepoint.

What is intriguing about this release is that it’s very clear that Microsoft’s investment and innovation around search hasn’t stopped — it has accelerated. They’ve hit a key design target (easy, powerful search that runs on premise or in the cloud) right on the money. since this release is a key architectural change for

sharepoint and a huge architectural change for search specifically, there are also many new features to build on. peeking under the hood, there is evidence that there’s more innovation to come in future releases — powerful new mechanisms which aren’t fully used yet.

This isn’t a perfect release — there are some things that take getting used to, some areas that still need sanding, and some situations where you need to write code or turn to partners to boost the power of your search capabilities. We’ll point out some of these areas where you can turbocharge your search in this e-book.

search technology (and basically all software that does sophisticated things around human language) is extremely hard in general. high-end search is very powerful, and can be applied in a myriad of situations, so covering everything is at odds with making search easy. The approach of providing hooks for extensibility and encouraging partners and customers to use them works — and Microsoft has a great set of partners to pull this off.

search is still hard — don’t let the easy, simple user experience fool you into thinking otherwise. But Microsoft has done a remarkable job making this high-end technology accessible and easy for the mainstream. You will get enormous benefit from this release, so get to know it.

introDuCtion

7

ChaPter 1 user experience – The new Face of search in sharepoint 2013


Chapter 1 The New Face oF Search iN SharepoiNT 2013

raising the Bar: The sharepoint 2013 user experience user experience broadly characterizes the way that people, users, work through user interfaces and information and product-specific concepts to get work done. sharepoint’s users, broadly, can be pegged to two groups:

Business End Users — regular, line of business users who utilize SharePoint for specific tasks and projects.

IT Users — iT professionals who manage, configure, and customize SharePoint for business users.

For any new generation of a product, user experience goals are straightforward: make it easier for the user to get work done faster, cheaper, and better. a simple, intuitive, attractive design also helps. consumers expect ease-of-use and a certain amount of slickness when it comes to interacting with products; the bar is high when it comes to how they can get work done.

With sharepoint 2013, there are several developments surrounding user experience

that business users can look forward to:

• Modern UI/Windows 8 Look and Feel — The new look and feel confronts users with the most “radical” update in 20 years (ui news link below) to prepare for a multi-device world. This look and feel for the Windows operating system supports mobile and has the ability to boost productivity for an increasingly mobile work force.

• Mobile and Tablet Deployment — Support for fluid layouts, touch, and voice interaction mean that using sharepoint on Microsoft’s surface tablet and the apple ipad is much easier and smoother. This means that users can access information anywhere at anytime, with the same ease-of-use they’re familiar with from their desktop.

• SharePoint 2013 and Applications — The bar is also going up when it comes to ease of access to information. sharepoint 2013 is able to field experiences that are mobile and search driven, as well as for customer and employee only facing sites. There are a variety of full-fledged applications that run on your desktop, in your browser, and on leading mobile devices and present new ways to access and interact with sharepoint information, further enhancing the user experience and productivity.



Open for Designersif you’re familiar with sharepoint you know that you can customize your interface to make it look nearly any way you want to — but you also know that the vast majority of business users leave the look and feel as the default and never change it. With sharepoint 2013, you no longer use powerpoint to create themes in a proprietary format. it’s easy to theme sites using hTMl (including support for hTMl5) — as shown below. This opens up sharepoint design to a much wider range of customization by designers, and will result in a lot of very attractive sharepoint sites.

The sharepoint 2013 user experience is a platform-wide update, ready for a new generation of interaction. changes in the underlying presentation tier, service architecture, Object Model (OM), and Office apps all further the goal of making it easier to configure and deploy valuable applications in this new delivery environment.

Mobile Challenges and Opportunities: Windows 8 and Metro Windows 8 devices have a new interaction flow. The desktop, charms, apps, and tiles are distinctly different from the familiar Windows 7 desktop. This represents an opportunity for application developers to create truly engaging user experiences that work across many devices. however, it also poses a challenge for developers to learn the Windows 8 stack, and a learning curve for users. Touch is highly intuitive and highly engaging; that said, the question will be how and when do users gain their first experience and confidence with idiomatic Windows 8. Will learning be amortized in context of your project, or someone else’s?

Metro, as seen so far in sharepoint 2013, is a sparer, less dense way of presenting information, which is good from a user experience perspective. it also means there is less information displayed per page of results, and that decrease may trouble users who rely on “recall” over “precision” in their browsing and scanning. The solution to this problem may be to present information more effectively, to make the less more. in order to provide richer results, the design and consequent development of processing and enrichment processes will require new skills from the sharepoint application developer.

deeper divesTechNet on mobile devices and SharePoint 2013 »

Blog with highlights of Design Features in SharePoint 2013 »

Article on Windows 8 UI »

SharePoint 2013 UI blog »

http://technet.microsoft.com/en-us/library/fp161351(v=office.15).aspx

http://technet.microsoft.com/en-us/library/fp161351(v=office.15).aspx

http://www.brightstarr.com/sharepoint-technology-and-application-insights/design-features-in-sharepoint-2013

http://www.brightstarr.com/sharepoint-technology-and-application-insights/design-features-in-sharepoint-2013

http://www.pcworld.com/article/2012024/the-windows-8-ui-how-do-interface-and-usability-experts-rate-all-the-changes.html

http://www.dshift.com/blog/sharepoint-2013-user-interface-highlights/



First class search interactions — More to love sharepoint 2013 has revamped the user experience overall (not just for search), and offers nice user experience improvements for everyone. highlights of the previous release, sharepoint 2010, included the roll out of the “ribbon” across all of Office and SharePoint, and the first roll out of Office Web Applications.

Search specific developments for the sharepoint 2013 platform for the end user include a flatter, cleaner, and more responsive interface. The “flatness” comes from a top down design that makes the transition of views in sharepoint Views (sites and document libraries), search Views (search sites) and detail Views (snippet and document) invisible. This improved responsiveness comes from the new architecture of the sharepoint 2013 presentation tier, which extensively uses modern hTMl, Javascript, and aJaX style interactions with responsive sharepoint search and metadata services.

But that’s not all folks, there’s a lot more to appreciate about the new search user experience with sharepoint 2013 search:

• Document Previews — Office documents are rendered in the page for easy viewing, so there’s less interruption going from one view to the next.

• Interactive Elements — Fly outs or hover card patterns are implemented quickly and cleanly. Search results fly in and additional information about what you are looking for is available with a flick of the mouse.

• Transitions Across SharePoint Tasks — The disjunction between “contextual search” and “search sites” is gone in sharepoint 2013. There are fewer obvious differences between apps; this version of sharepoint does not feel stitched together like previous versions. new developments include the seamless flow between functions such as people search and search verticals.

• Productivity — search helps users quickly return to important sites and documents by remembering what they have previously searched and clicked. The results of previously searched and clicked items are displayed as query suggestions at the top of the results page.

• Search Mechanisms Under the Hood — Queries, interpreting queries, returning relevant results, and the presentation of those results are pervasive across sharepoint 2013. it’s not always obvious that search is “there”, but



search technologies are used across the sharepoint 2013 platform, and key new interfaces lower the complexity of customization iT professionals and application developers need to do in order to support business users.

search powers a number of areas which may or may not be obvious as search:

• Upgrades to People Search and Social Features — making it easy to explore and find people, expertise, and conversations that are important to the task at hand.

• New Social Features — My sites, communities, Teams, and conversations create dynamic content that are quickly indexed via constant incremental crawls and returned through sharepoint 2013 search.

• Personalization Features — search suggestions are personalized, and include visited documents, as described in the chapter on query rules and query suggestions. These show up “as if by magic”, and many users enjoy them without thinking about search at all.

overall, the search interfaces are clearer and brighter, and all the different parts of sharepoint apps seem to work better together. it is also much easier to customize search-driven experiences in sharepoint 2013 than with any other enterprise search platform.

The New Face of Searchsearch in sharepoint 2013 has a completely different look and feel from previous versions; the ui has been largely rewritten. The new face of search in sharepoint 2013 is easy-to-use, clean, and intuitive — it offers easy exploration and navigation of information while presenting information in an actionable format. This is a far cry from the ten blue links concept that the industry has been living with for nearly 20 years. There are also a number of changes that have been made to enhance ease-of-access to information supporting both productivity and mobility.

While the out of the box interface is clean, and we view this as a positive enhancement, it is not as information-dense as heavy search users demand. There are a number of search-based applications that can bridge user requirements surrounding information access and analysis and we will provide several next steps and options for review at the end of this e-book.

deeper divesSearch User Interfaces book by Marti Hearst »

http://searchuserinterfaces.com/book/

http://searchuserinterfaces.com/book/



The sharepoint 2013 search center — overviewThe sharepoint 2013 search center has inherited the new look of sharepoint 2013 — overall it is clean, modern, and dynamic. as you can see from the screenshot below, it is quite different than what you are used to seeing. The familiar tabbed interface is apparent, but it has a more streamlined look and feel and includes some new out of the box tabs such as videos. There are also more actions that can be done directly from the search interface, including a hover panel.

some of the capabilities from FasT show up in this release as well — deep refiners and document previews in particular. These have been taken to the next level with additional features such as the ability to show histograms for dates, and allow for a search inside the refiners. While both capabilities are welcome, they are somewhat limited — whetting your appetite for more.

*Note: The refiner counts are turned off by default, but they appear with one click in the web part configuration panel.

Document Previews and the Hover Panelone of the most exciting new features added to sharepoint 2013 is the integration of document previews right within the search results. This feature leverages a new standalone server that hosts Office Web Applications. With Office Web Apps users can now open a document in a web client environment with reasonably high fidelity while preserving format, fonts, sizing, etc.

a key component within the document preview display is the “take a look inside” functionality. This provides the ability to jump specifically to a relevant section of the document, based on extraction of sections for several document types. For example, because it is likely that the slide titles in a powerpoint presentation were designed by the presenter to summarize the content of each slide, these titles are extracted and shown as links. This feature is also available for Word documents and excel documents (focused on graphs and named tables) as well as sharepoint sites (top sub sites and document libraries).

There are some limitations to the document preview features with sharepoint 2013. it is relatively slow and missing functionality that other preview products take for granted. This includes search term hit highlighting, the ability to immediately jump to the most relevant page of the document, as well as copy and paste functionality from within the preview. Breadth of content types is another area where sharepoint 2013 previews falls short — they are only available for content hosted in SharePoint, and only for a limited set of file



formats (for example, Word and powerpoint, but not pdF). This preview technology was not designed for documents to be consumed via this interface, but rather to determine if this is the particular document that you have been looking for. notwithstanding these limitations, though, document previews are a boon to the user and a great addition to search.

The hover panel paradigm works well in the search center. This can be customized and may vary based on content type or tab. default actions with document preview include the edit, send, and View library features, as well as Follow, a social feature. They also allow some actions directly from the search page, including editing content directly in Office Web Apps.

People Searchpeople search is another strong part of the search center. as with sharepoint 2010, people search lights up with actions when used together with lync, and phonetic search

is available by default. The new hover panel provides a great way to show profiles and content, in addition to social connections.

For many applications, people will want to customize the search center, because it is not as information-dense as heavy search users or search-based applications demand. This type of customization is easy to do, and we’ll cover it later in the chapters about query rules, result sources, and development model

overall, the sharepoint 2013 search center interface is better than any other search ui we’ve seen on the market. it appears to

be very robust, and holds true to Microsoft’s ‘works anywhere’ commitment. it functions smoothly both in the cloud with Office 365 and on premise, as well as in all of major browsers (internet explorer, Mozilla, chrome), and the



experience on tablets like the ipad is pretty good. a word to the wise: just don’t let a sexy demo or quick test drive lull you into thinking that ‘it just magically works’. as with all search products, the navigation depends on having decent metadata.

overall the out of the box interface is clean, fast, and provides relevant results — so the basic ‘must have’ elements of great search are covered. There are also a lot of exciting capabilities that make exploration easier, give users insight, and enable action directly from search.

of course, everything works better with search when all

of the products that are part of the search machine are Microsoft. — for example, people search lights up with actions when used with lync; myTasks work with project server ; and previews work only documents stored in SharePoint with recent Office formats, and require a separate oWa server. if you don’t have servers that run these other products, the additional features associated with them simply don’t show up. however, search still works very well even without them. When you have all these parts in place, though, they work extremely well together — a big accomplishment for Microsoft with strong productivity benefits to the end user.

Refiners and Faceted navigationless than ten years ago, the idea of using faceted metadata for flexible search and navigation was just being hatched in an academic research project called the Flamenco project. now it is de rigueur, it has proven to be effective and enterprise search without it is subpar. Microsoft added search refinement in SharePoint 2010, with the refiners populated by whatever content is in the associated managed properties. SharePoint 2010 created refiners out of the top n results (called “shallow refiners” — top 50 results was the default),

and FasT search for sharepoint created “deep refiners” out of the entire result set, even if it was millions of items.

With sharepoint 2013, there are now two different modes for the refiner web part: standard search results, and faceted navigation. For standard search results, refiners are generated as they were with FasT search for SharePoint. You can now define display templates to use for rendering different kinds of refinements, which is a big win over SharePoint 2010. All refiners are now deep refiners.

Faceted navigation is more dynamic. it is used in conjunction with term sets (served from the

deeper divesTechNet — creating a search center in SharePoint 2013 »

Intro to the hover panel »

Longitude Search Overview »

http://technet.microsoft.com/en-us/library/hh582314(v=office.15).aspx


http://www.bainsight.com/Pages/sharepoint-fast-search-document-preview.aspx



term store), which are also used for navigation in document libraries. With faceted navigation a term from the term store filters what kind of data should display. if the managed property is ‘refinable’, the refiners that show can depend on the term. This is handy in many search scenarios, including the online store scenario which inspired it. For example, users can use faceted navigation in an online store to find products more easily. The scenario below uses the term store terms camera and laptop and managed properties Megapixel count, color, and Manufacturer. so, with faceted navigation your terms would look like this:

• For the term Camera, add refiners for Megapixel count and Manufacturer

• For the term Laptop add refiners for color and Manufacturer

The refiners that show up now are based on that term, which can be set based upon a page or catalog hierarchy, so that you get the following whether you navigate or search to laptops:

Configuring these refiners via the term store is convenient, and there are built-in tools that make is easy to create a hierarchy, customize the refiners within the hierarchy, and set up a very dynamic experience, as shown below.

Navigation and Search Unified hierarchy is also used to create results pages, as part of the WcM part of sharepoint 2013. navigation settings are based on the same hierarchy, so that users can search, navigate, or refine their way to their result. Navigation controls also have built-in customization, as shown below.



as you can see, Faceted navigation is quite a powerful capability. Refiners are available everywhere, they adjust dynamically and can be configured to an exact design — all controlled

by metadata. All refiners used in Faceted Navigation are deep refiners, so there are no gaps caused by a missed item in the deeper result set.

search center setup For the iT professional, sharepoint 2013 offers more control over the logic of search applications, and it exposes that control in a clear, consistent, and logical model. We’ve outlined key concepts as they relate to search center setup and how they are used to deliver search results.

• Query Configuration — Query rules are used to control ranking, query intent classifications, synonyms, and query rewriting in sharepoint 2013.

• Presentation Configuration — Query rules and display Templates determine what result snippet gets shown for what class of query, what type of document, and for what category of user. The integration of search query processing across the platform means that display templates and query rules are applicable throughout the application.

• Faceted Navigation — Metadata used for top down navigation (“Faceted navigation”) and metadata exposed as search results for bottom-up refinement are now both managed through the term store.

On the Premise or In the Cloud? Get Going Fastersetting up a new search center is pretty straightforward. as illustrated below, site administrators can easily set up a sharepoint 2013 search center to run on premise or in the cloud.

deeper divesTechNet Managed metadata overview »

Technet — configure facted navigation in SharePoint 2013 »

http://technet.microsoft.com/en-us/library/ee424402.aspx

http://technet.microsoft.com/en-us/library/jj679902(v=office.15).aspx



The search center itself is a site template, and the good news is that with this latest release some of the rough edges from sharepoint 2010 have been removed. For example, this template now inherits design elements from a master page, so you don’t need to jump through hoops to make it match your design. This does not mean that you don’t still need to think about how to manage the ‘universal search center’ — which may serve many site collections with different themes and designs — but you now have easier control.

Changes to Sites and Site TemplatesThere are a number of changes to sites and site templates overall in sharepoint 2013. The facilities for sharing (requesting and granting site permissions) are completely revamped and considerably improved, as shown in the screenshot below.

The document Workspace site template has been removed in sharepoint 2013, simplifying the list of templates available when a new site collection is created. This will be a big change to users since this template was a workhorse

in sharepoint 2010. Most Meeting Workspace site templates from in 2010 have also been discontinued in sharepoint 2013 — including the basic, blank, decision, and social meeting workspace templates and the multipage meeting template. They have been replaced by features from other parts of sharepoint and from onenote and lync, which all support collaborative work, live conferences, smaller meetings, note-taking, and storage of notes and other conference-generated commentary. The benefit is that projects with multiple contributors and collaboration across geographically distributed teams is streamlined.

The facilities for web content management (covered in the applications section) are remarkably improved — and totally driven by search. This makes creating externally-facing sites and applications much more effective. if you have responsibility for explaining and exposing a service or product to a market inside your company, the business-focused features that are new in sharepoint 2013 are a strong proposition for inside the firm audiences. For example, if you provide consulting services internally for a legal practice area, recommendations, customization of search experience based on queries and personalized interaction, etc. enable users to find relevant information more quickly.

deeper divesTechNet — creating a search center in SharePoint 2013 »

Blog on using the Content by Search Web Part »

http://technet.microsoft.com/en-us/library/hh582314(v=office.15).aspx

http://www.sharepointnutsandbolts.com/2012/10/using-content-search-web-part-and.html

18

ChaPter 2 Working with Queries and results – new Mechanisms in sharepoint 2013


Chapter 2 New MechaNisMs iN sharepoiNt 2013

Query processing: The search engine’s automatic TransmissionThe search experience involves many different processes, so creating a great search experience requires covering everything — from the moment information is pulled from the source systems to the moment it is presented to the user in search results. sharepoint historically had strong coverage on the crawling side via its Business connectivity service and protocol handler framework, and strong coverage on the presentation side via its XslT driven core results web parts. FasT search for sharepoint on the 2010 platform then brought coverage of the content processing area via its pipeline extensibility framework as well as its built-in entity extractors. sharepoint 2013 completes the coverage by providing a strong query processing framework, shown below.

But what does “Query processing” mean exactly? if you’re familiar with sharepoint 2010, think of query processing as the evolution of search scopes, federated locations, and best bets. With intranet search indexes now frequently reaching tens of millions of items, formulating the right query is more and more critical to finding relevant information. Fortunately, there are a number of techniques you can use to reformulate the query by

understanding the intent behind the query. You can leverage information such as:

• Where the query originated from. For example, if you run a search from your company’s helpdesk intranet site, you are likely to be looking for FaQs, how tos, or iT specialists. The search engine can now capture that intent to provide more targeted results.

• Who launched the query. if you are based in the united states, and searching for employee benefits, you are more than likely looking for U.S. employee benefits than for canada or united Kingdom.

• What concepts or entities can be recognized in the query. For example, if you were searching for an expense report form, the search engine will return the excel spreadsheet, infopath form, or web page which enables you to file your expense report.

Query Processing in Action an example of query processing techniques combined would be a search for a weather forecast on Bing. The very first result you’ll get at the

top is the weather forecast for your location. Bing automatically understands the concept behind the query, and then correlates it with information about you, the user (in this case, your location) to provide you with the forecast. it is also worth noting that this answer is not displayed like the other results on screen. it is instead carefully rendered in a visual format to enable you to quickly make a decision based on that information.



Query processing in sharepoint 2013 is intended exactly for these scenarios; to enable a smart, targeted search experience which understands what the user is searching for and to provide the optimal result straight from the search page. This is a very exciting new capability in sharepoint 2013, as it will open up many opportunities to rapidly build new applications driven by search which will look nothing like the standard list of ten blue links.

Getting in Gear: Result Sources, Query Rules, and Result Typesso let’s dive now into the details of what sharepoint 2013 offers for query processing. We referred to query processing earlier as the evolution of search scopes and best bets. We meant it — literally! in sharepoint 2013, search scopes, federated locations and best bets are now deprecated in favor of result sources, query rules, and result blocks.

Result Sources enable you to focus searches and subset of the total information accessible in your organization by applying extra conditions to the search queries on behalf of the end-user. stated as such, they sound very much like 2010 search

scopes. The key difference here is that the extra conditions enabled in 2013 go far and beyond what 2010 could do. sharepoint 2013 comes with a strong query builder to apply conditions based on the user, the search page url (or any parameter found in it), the site, or the current date. result sources can also be used to return results from remote content, much like federated locations in sharepoint 2010. (The result sources construct is covered in greater detail in the

Federation chapter of this e-book).

Query Rules allow conditional transformation of queries and results based on custom logic. imagine you want to simplify searching for budget spreadsheets in your organization. using query rules, you can type simple search queries such as: budget spreadsheet project X and behind the scenes the request can be transformed into something much more elaborate. The query rule could recognize the terms budget and spreadsheet in the

search query and rewrite the query so that the document content type must be ‘budget’, the file type Excel, and the file content match the project name you specified in the search keywords. additionally, the results would be sorted from the most recently modified file so that the freshest information is returned first. it is worth noting that the same Query builder functionality used for result sources is also available here as a means to define conditions on query rules or transform user queries



The last major new feature introduced for query processing on sharepoint 2013 is the Result Type construct. a result type supports the presentation of results in a tailored way, and the result block contains a small subset of results that are related in a specified manner. For instance, you can create several result blocks for sales collateral, knowledge base articles, documentation, etc. so that when a user searches for a specific product you can make sure to always return the top two or three pieces of sales collateral or knowledge base articles matching this query.

in spite of the enhanced capabilities these tools provide, you may run into scenarios where they are not suitable or flexible enough for a particular search scenario. For example, geo-searches (ranking or search results filtering based distance), personalized queries (complex query changes based on who executes the query), synonyms expansion, etc. are not supported. in these scenarios you can still rely on the search api to build your own web part or search application that implements the appropriate logic. The api, is for the most part comparable with the version seen in sharepoint 2010 with a few exceptions. The main exceptions include the removal of the FulltextsqlQuery class and syntax which have been deprecated, and the appearance of the searchexecutor class which allows you to execute multiple related queries in one shot.

No Speed LimitsMicrosoft has made it very easy to create search pages using this new functionality. in fact, you don’t need programing experience

to create pages as all the functionality is user friendly and has point and click interfaces. Microsoft made it even easier by pushing this functionality not only to site collection administrators, but to administrators as well. That’s right, farm level privileges are not required — as long as you own a site (such as your personal site) you can use these capabilities to build your own search center.

Two examples of applications you can build using these new features:

• A manufacturing dashboard that displays all about a specific part based on its part number. information could include the inventory level, the last orders for that part, the instructions on how to use that part, and forum discussions from your customers about that part.

• A knowledge portal, that enables you to share FaQs, knowledge base articles, documentation, or tutorials to empower your support or helpdesk team.

powering your applications via search has never been easier. The chapter on search-Based applications has many more examples, and we encourage you to explore what’s possible, and even to try building some of your own.

deeper divesTechnet on query processing »

Blog — overview of search in SharePoint 2013 »

List of terms for query builder »

New KQL syntax in SharePoint 2013 »

http://technet.microsoft.com/en-us/library/jj219620%28v=office.15%29


http://technet.microsoft.com/en-us/library/jj683123%28v=office.15%29.aspx

http://msdn.microsoft.com/en-us/library/ee558911%28v=office.15%29.aspx



Query rules and Query suggestionsin the last chapter we introduced Query processing and several new key concepts including Query rules and Query suggestions. now, let’s go into greater detail on these and some other query features like spell check and rank management. Working with these features, you can customize search to a great degree, without writing code.

Query RulesQuery rules are a brand new feature in sharepoint 2013, and they are designed to enable you to act upon the intent of a query and provide a remarkable amount of control and configurability. The Query Rules framework is composed of three top level components: Query conditions, Query actions, and publishing options. These are all configurable via PowerShell, or via the UI shown to the right.

Query Conditions are rule sets that are meant to determine the intent of the query (does the query meet a rule?) options for this include:

• Query contains a specific word or words

• Query contains a word in a specific dictionary

• Query contains an action word that matches a specific phrase or term set

• Query is common in a different source (like Videos result source)

• results include a common result type (like file type)

• advanced rules which can match across a set of terms, dictionary, regular expression, etc.

if the query is against a particular result source (see the result source chapter in this book) or category, result source conditions can also be applied. if the Query condition is met, Query actions are then triggered.



Query Actions specify a series of actions that take place once a query condition is met (what to do if the rule is met). These actions include:

• Assign a promoted result — This replaces the “Best Bet” and a former FasT search for sharepoint 2010 feature known as “Visual Best Bets”. The configuration of the promoted results allow you to specify if the returned action should be treated as a best bet (hyperlink) or as a fully formatted hTMl block (Visual Best Bet)

• Create and assign a results block — When a condition is met, one or more results blocks can be triggered. result blocks specify an additional query to run and how to display results. This feature includes a full query designer so you can build and test queries before finalizing them. You can also include the results above those returned by core results, or interleaved by ranking. additionally you can choose custom display templates instead of the default for the result or results block.

• Change the ranked results by changing the query — This allows you to assign additional parameters and weighting (XranK Boosts) values to the query (Query Transforms for those familiar with FasT). For example, if the condition of the rule is met, apply XranK constant

boost of x number of points. XranK is a FasT capability that allows you to override the default relevancy ranking by boosting the relevancy score for particular results at query time.

• Publishing Options — publishing options determine when a query rule is active (When to do this?) a rule may be active in a specific time interval (start date, end date) or always active (by default). You can also configure a review date (triggers an e-mail reminder to review this rule).

The power of query rules is not only in the flexibility they provide, but also the richness and complexity that can be derived from them. imagine a single Query condition being met, which then triggers a visual best bet, a results block from a remote sharepoint site, a results block from a cloud source, and a query transform that will boost results coming from the cloud. in addition, rules would determine that these actions are only taken between november 25th and december 26th. an example of how this would work in an intranet scenario, would be if you had a query rule that was active only during insurance open enrollment windows.

Query SuggestionsQuery suggestions enable users to ask better questions, and make it simpler to search for information. This feature was sorely lacking in sharepoint 2010. in sharepoint 2013, Query suggestions are supercharged, thanks in part to the addition of the analytics processing component and the analytics reporting



database. These components provide for analytics aggregation and persistent storage of these analytics.

some key new features include:

• My Queries — personal Query log (in analytics database), which factors your personal sharepoint activity into the query suggestions.

• My Sites — This capability tracks sites you have visited, and factors them into the query suggestions.

• Our Terms — This feature uses information related to the most frequent queries across all users that “match” the search terms.

Query suggestions now take two forms: pre-Query suggestions and post-Query suggestions. Both of these help the user ask better questions by showing you what others have asked before; they differ in when they are displayed and how people use them.

Pre Query Suggestions include both a list of queries from other users, and a list of items you have clicked on before, as shown in the screenshot below.

pre-Query suggestions occur prior to a query being executed. The goal of pre-query suggestions is to aid users in selecting a

query to help them find information, and to assist them in writing better queries. These suggestions are provided in two forms:

1 a list of items that others are typing for their queries.

2 a list of items you have clicked on before from your personal query log.

a key aspect of this feature is that it will never provide a suggestion to a search that did not yield a click-through (someone clicking on the document), and it will never provide a suggestion if the results would lead to a dead end (zero-result query).

Post Query Suggestions are provided after a query is executed and when results are displayed. These suggestions are based upon the results that you have clicked on at least twice. They provide a quick means to go back to a document that you regularly review or select. They are similar to the “related Queries” provided with sharepoint 2010. suggestions can also be tuned (inclusions and exclusions) within



the service application admin pages. it is also important to note that these are not tuned at the site collection level, but only at the ssa level.

Query Spell Correctionspell correction is a familiar and very useful feature, since humans are prone to misspelling and of course fat-fingering. SharePoint 2013 provides spell correction by default, as shown below:

in sharepoint 2010, spell correction was implemented as a series of XML files that defined inclusion and exclusion items for the dictionary. in sharepoint 2013, Query spell correction is managed from within the term store of the Managed Metadata service. Within the term store, Query spellcheck exclusions and inclusions are nodes within the term store, as shown below. dynamic dictionary creation is still supported, but is now managed from within the term store.

Within the user interface for search, Query Spell Corrections can be configured to use “did You Mean” type functionality for query transforms.

Working Wonders with QueriesThe mechanisms for query rules, query suggestion, and query spell checking are new with sharepoint 2013, and they may take some getting used to. previously, there were some capabilities in sharepoint 2010 that processed queries such as the keyword features that applied to synonyms, best bets, and promotions/demotions that are now replaced by query rules. once you become familiar with these new features, you will find you can work wonders.

in spite of all of the obvious pluses, there are some limitations with query rules. You can’t call a program from a query rule, which blocks a variety of use cases. For example, synonym expansion is done on full queries, and on pre-built synonyms. This makes expansion easy to understand but has been a big annoyance to many search administrators in sharepoint 2010. This limitation can be addressed, but only through applications available through the Microsoft partner ecosystem — not via query rules. calling applications based on query patterns (for example, pulling up an aTM location app when users search for ‘bank branch near me’) is feasible in sharepoint 2013, but not directly from query rules. however, these limitations are important only for a specialized set of search applications. The power of query rules, and query processing generally, in sharepoint 2013 is light-years ahead of other search platforms.

learn to use these mechanisms, and you will be in a great position to dazzle your business users with the power of search.



result Types and result TemplatesThere’s another new concept in sharepoint 2013 search, called result types. result types let you control how search results will be displayed, and let you display different content in different formats. For example, if you have e-mails, documents, and database records in the same result set, you may want to use different formats for each and display different managed properties for each. With sharepoint 2010, this meant creating complex xslt, and there was no easy way to group similar results together for presentation. With sharepoint 2013, wizards ease configuration of displayed results, and hTMl and Javascript enable you to add finishing touches if needed.

The screenshot below has multiple result types, presented in result blocks. Videos, documents, personal recommendations, and a “visual best bet” (though it’s no longer called that) all have their own presentation and their own result template.

Results Framework ReduxThe results framework is composed of three parts (as shown below):

• Rules Engine — a list of rules to determine if the result type should be triggered.

• Property List — associates the rule to document type, content type, or other managed property within sharepoint search.

• Rendering Template — Defines how that particular result will be displayed.

deeper divesGood blog post on query rules »

TechNet on query processing »

List of terms for query builder »

New KQL syntax in SharePoint 2013 »

http://sharepointfieldnotes.blogspot.com/2012/10/whats-new-in-sharepoint-2013-search.html

http://technet.microsoft.com/en-us/library/jj219620%28v=office.15%29





Result Types UnleashedThe power of result Types really becomes evident when looking at a real-world scenario. in the scenario below you have multiple documents that have been assigned content types (i.e. specification documents, data sheets, etc.)

Within Result Types you can:

1 Specify a rule based upon specific criteria. The rules can contain fairly advanced features, such as Boolean logic (i.e. and or noT), equality (i.e. = or !=), or comparison ( < or > ). These rules can also be applied to managed properties. For example the rule might be contentType= “spec documents”).

2 specify which managed properties you would like to have returned once rule conditions have been met. You must specify at least one

managed property before it can be used in a rendering template.

3 specify where you would like the requested property list items to be displayed using a tagging convention as follows (-#= contenttype =#-) by using a rendering template. The rendering template consists of a template that is composed of hTMl and might contain Javascript. Within this simple to edit template (not like editing XslT in SharePoint 2010) you can call specific graphics (icons, etc.) and be stylize it in any way that you would normally stylize hTMl.

result types may seem complex to master, but once you become familiar with them you will appreciate how powerful they are. There are impressive tools in sharepoint 2013 that facilitate ease of use, and formatting is done using any tool you are familiar with. (sharepoint designer has dropped the ability to do this kind of formatting, which will be annoying to some, but there are lots of great tools available to work with hTMl and Javascript.)

With sharepoint 2010, very few people actually did the kind of formatting and result templating that was possible — it was too complex and arcane to use. With sharepoint 2013, you will quickly find that result types and result templates are enjoyable to work with, and you’ll discover that you use them naturally to make search results look great and work well for users.

deeper divesCustomizing search results via Result Types and Display Templates »

Technet — query variables »



28

ChaPter 3 Working with content — crawling, connectors, and content processing

28

Chapter 3 Crawling, ConneCtors, and Content ProCessing


content capturecapturing content is fundamental to search — if it’s not crawled and indexed, you can’t find it! The process of connecting to content sources, crawling them to get content, and making that content searchable is far more complex than most people realize. it was also one of the most frustrating areas to manage with sharepoint 2010.

as a quick orientation, the basic function of a crawler is shown in the figure below. The concept is simple enough: the crawler connects securely to a given content source, maps the content from the source system to the crawled properties of the search engine, and feeds the engine in either a full crawl or an incremental crawl (which finds any changes). What makes content capture different from one search engine to the next is the breadth of connectors, coverage of different security models, and data types, the performance (both throughput and latency), the robustness, and the ease of administration. sharepoint 2013 does well on all counts — although most connectors are supplied by Microsoft’s partners, not Microsoft.

sharepoint 2013 supports multiple crawl components, crawl databases, and content sources as shown below. There are a number of connectors included out of the box:

• sharepoint

• hTTp (web crawler)

• File share

• Business data connectivity (Bdc) Framework — also includes these connectors that are built on the Bdc framework:

– exchange public Folders

– lotus notes

– documentum connector

– Taxonomy connector (connects to MMs)

• People Profile Connector



Connector and Crawling ChangesFor the most part, these connectors are essentially the same as the connectors in sharepoint 2010. The connector and crawler infrastructure are the part of sharepoint 2013 taken most directly from sharepoint search, so they have the fewest changes. While few, there are still some notable changes.

The web crawler has some nice updates that address previous headaches. These changes include:

Anonymous Crawl for HTTP anonymous authentication allows any user on a web site to access any public content without providing a user name and password challenge. sharepoint 2013 allows you to get at these web sites without associating crawl to a user account. This is handy for general web crawling and makes the setup of web crawls simpler. sharepoint 2010 used the spsearch account to log into sites, which stymied many people trying to crawl sharepoint sites with anonymous access, public web sites, and the like. previously there were work arounds, but they were painful. The updated functionality now offers a pain free way to perform this task.

Asynchronous Web Part Crawl a common way to improve performance of sharepoint sites has been to load web parts asynchronously, which dramatically speeds up the first display of the page. However, crawling these pages for search also delivered incomplete information. in sharepoint 2013 search, the crawler now gets a full rendering of the page in order to index them. This doesn’t work for all asynchronous pages, just for most out of the box web part content. But it takes care of the vast majority of problems in this area.

overall, the most noticeable change in content capture is continuous crawling. This is a new method of insuring you have the most current data in your search index, and is available only for sharepoint content. rather than living with a latency of several minutes and with full crawls that might take many minutes to start populating content in the index, you’ll see content within seconds!

When you enable continuous crawls (using the ui shown below), a crawl schedule no longer applies — you are running crawls in parallel and the crawler gets changes from sharepoint sites every n minutes (set to 15 minutes by default but this parameter is changeable). continuous crawls do not stop for errors, but rather note the error and continue to crawl content. continuous crawls can occur while other crawls (full or incremental) are active or starting, where incremental crawls need to wait for other incremental crawls to complete prior to starting to crawl. With this capability you can now keep content fresh, and won’t experience mysterious delays when additional content sources are added.



The Taxonomy connector is new in sharepoint 2013, and you will see it at work even when you don’t use it explicitly, since the term store is much more integrated with search. as you will read in other chapters of this book, you can now create entity extractors directly from term sets, set up WcM page hierarchies using the term store, define faceted navigation using taxonomies, and much more.

Building New ConnectorsWhen you start getting into search, you quickly find that you want to get at more and more kinds of content from more and more places. data silos are everywhere, and search lets you bridge these silos easily and securely. in order to do this, you need a connector for each content source — and many organizations have dozens of systems that require connectors well beyond what comes out-of-the box w. luckily there are two options: leverage a rich set of partner-built connectors or (if you are a developer), create new ones yourself.

sharepoint 2013 will still support existing protocol handlers (which are custom interfaces often written in unmanaged c++ code), using an interface used since Moss 2003 and deprecated since sharepoint 2010. These can

still be good for high performance or particular tasks. But the primary way to create connectors is through the Bdc Framework, which was introduced in sharepoint 2010 as part of Business connectivity services (Bcs). Bcs is an umbrella term for a set of technologies that brings data from external systems into SharePoint Server 2013 and Office 2013 (shown in the figure below).

as with sharepoint 2010, you can make new connectors pretty simply. For systems with static schemas, straightforward security, and moderate performance needs, this is not a huge job. There are some great improvements in Business connectivity services as a whole — for example, there’s tooling specifically to create external content Types against odata sources, there are representational state Transfer (resT) and client side object Model (csoM) interfaces, and external content Types that can be scoped to a single sharepoint app. unfortunately, none of these apply to search — creating an indexing connector for search is not the same as creating an external content Type.

The Business data connectivity (Bdc) framework is largely the same in sharepoint 2013 as it was in sharepoint 2010, when it comes to search. There is one notable change though — claims tokens are supported through the Bdc. previously, only active directory (ad)-format access control lists (acls) were supported, which made it nearly impossible to cover some complex security scenarios. With claims support, many of these scenarios are tractable — though still very much the domain of experts.



one warning — you shouldn’t underestimate the effort involved in connector development, deployment, and maintenance. don’t fear connector development, but watch out for the classic “quicksand” trap. Too often a development project gets to basic connectivity quickly but then struggles to get security right and to get high performance and scale. if and when this is successful, the project is then dragged further down in troubleshooting and maintenance, since things change every time the source system changes. plan your development carefully to avoid this trap. The best way to avoid it is to consider pre-built connectors for any complex system — that way you don’t have to build your own from scratch, and you don’t have to maintain it.

Changes from FAST Search for SharePointif you are used to FasT, there are a number of changes you will notice. These changes are all a byproduct of moving to a common, single search engine. First, there is no way to ‘push’ content to index in sharepoint 2013. (With FasT, there was a mechanism called the content api).

There are also three connectors that you will notice are gone:

• Lotus Notes which had performance, security, and flexibility features beyond the notes connector included with sharepoint 2010 and 2013.

• The Enterprise Web Crawler which rendered dynamic sites, had high performance, and several high-end features.

• Java Database Connectivity (JdBc) connector which supported direct sQl access to databases.

Though these may seem like big gaps, there are ways to cover this functionality in sharepoint 2013, either with different mechanisms (many cases covered by the JdBc connector can be done via the Bdc), or with pre-built connectors from Microsoft partners.

From Crawl to IndexMany of the most significant content capture changes you’ll see with sharepoint 2013 search don’t actually result from the connector and crawling components. For example, the content processing component adds some remarkable capabilities that show up to the end user looking like better content. The indexer has lower latency and is much more robust, which is one key to continuous crawling and also alleviates many of the weird issues people encountered with crawling after outage events with sharepoint 2010 (which could cause the crawler and index to be out of sync.) additionally, improvements in schema management make mapping content much simpler with sharepoint 2013.

all of these areas are covered in other chapters of this book — but they contribute to the improvements discussed above to provide robust, scalable, and high performance content capture. This is a great foundation to build on for any search deployment or search-based application.

deeper dives:TechNet on managing continuous crawls »

MSDN on searching new content with SharePoint 2013 »

Longitude Connectors Overview »


http://msdn.microsoft.com/en-us/library/jj163206(v=office.15).aspx

http://www.bainsight.com/sharepoint-fast-search-connectors/Pages/default.aspx



content processingcontent processing is an essential pillar of search quality, but it is typically invisible to the end user. The development of content processing in sharepoint 2013 is focused on implementing platform-wide capabilities, and integrating and supporting built-in search-based applications such as WcM and e-discovery. in order to support the wide range of scenarios that depend on search, Microsoft provided extensibility, so that customers and partners can leverage the new search platform and hook into content processing.

The content processing component is brand new with sharepoint 2013. it takes content from the crawler and prepares it for indexing, as shown below. With sharepoint 2013, there is also a new analytics processing component that feeds information into content processing.

New Content Processing Subsystem — With a Heritage

To understand the changes in the content processing structure within sharepoint 2013, it is useful to look at the heritage of this release, especially for the content processing component. Those familiar with the final version of a “stand-alone” search engine offering from Microsoft, Fast search for internet sites 2010 (aka Fsis), understand that Fsis was composed of three main structural components, as shown below. They were:

• Core FAST search Engine (FasT esp 5.3) — in red in the figure below — which was a complete search engine, to which new components were added on the content side and query side.

• Content Transformation Services (cTs) which was responsible for content processing and ingestion and introduced the concept of processing flows. Flows are much more dynamic and expressive than the straight linear pipeline architecture found in Fast search for sharepoint 2010.

• Interaction Management Services (iMs) which managed all query and result processing, using processing flows.



In SharePoint 2013, the underlying dataflow engine for content processing, which was first introduced as cTs, has been extended and enriched to host the content processing tasks for the entire sharepoint platform. successful integration of a new content processing flow for search and enrichment for the whole SharePoint platform is a significant investment and engineering achievement. The benefits are potential scale out, improved management, “cloud ready” system architecture, and an improvement to Microsoft’s ability to integrate new content enrichment features inside the sharepoint platform.

New Capabilities for the IT ProFor the iT professional concerned with how content is processed, enriched, and made ready for search, these sharepoint 2013 content processing feature areas stand out (we cover these in more depth in the chapter on linguistics):

• Linguistics features, in particular around phonetic search for person names, continue to improve in scope depth. cross-lingual name search (via people search), for example, is a remarkable feature that makes it easy to find people (since human names are notoriously hard to spell right).

• Entity Extraction management, which was previously done via a set of separate files and ad hoc PowerShell scripting, is now moved into the Term store — a big win because there is now a good ui and a robust set of tools with it.

• New format handlers implement document parsing. They replace iFilters for ooB document metadata.

• Higher throughput for Office document types and for pdF.

• Automatic content-based file format detection removes dependencies on file extensions.

• Content processing throughput and error reporting (this is tied to crawl reporting) is comprehensive and far simpler to understand.

search analytics processing (which we cover in more depth in the chapter on analytics) is an important new platform capability. The analytics module feeds information back into content processing for a variety of purposes- for example, to improve search relevance based on user behavior. usage and search action events — document exposures and document click-throughs — are recorded into a new sharepoint 2013 analytics store. They are then processed in a form that enables search relevance to account for, for example, popular content, relevant query terms, or, in the context of recommendations, boosts for related user/related item results. This also supports search history boosts.

Hooks for the Savvy DeveloperFor developers familiar with the extensibility of FasT search for sharepoint, sharepoint 2013 offers similar mechanisms. however, the content processing flow and search index are not as open as with previous FasT platforms — they are more of a streamlined and closed utility. You



will enjoy how easy it is to set up and operate these capabilities, and how little head-scratching you do in development — but you will be frustrated at how little you can get at. This is a sensible tradeoff in the context of a major platform upgrade and in accommodation of a hosted multi-tenant deployment model (o365). The capabilities and ability to extend them is still there, but it feels limited. There are times that it takes sophistication and inventiveness to do what you want with the hooks provided.

The extension point for content processing is the content enrichment Web service (ceWs). This is a new mechanism to enable content processing, called from a content processing flow at a single point, as shown below. We will cover ceWs in more depth in its own chapter, and touch on its applications in the chapter on linguistics.

* Note the CEWS call-out is not part of O365 and is only available for the Enterprise Edition of SharePoint 2013.

sharepoint’s management of content processing is highly scalable and streamlined. sp2013 content processing straddles the on-premise deployment of sharepoint and the deployment of sharepoint in hosted form via o365. if content enrichment beyond what is provided in sharepoint 2013 is important for your application, especially for content you already have, prepare to look for custom solutions that leverage the content enrichment Web service.

deeper divesMSDN Section on Content Enrichment Web Service (CEWS) »

TechNet content processing description »


http://technet.microsoft.com/en-us/library/jj219738(v=office.15).aspx#ContentProcComponent



linguistics processing linguistic processing, which aims to leverage the meaning of documents or words, is the ‘special sauce’ of search — and one of the most mysterious and difficult to understand areas. human language is a tricky thing, and algorithms aimed at understanding it are complex and imperfect — yet this is what makes it seems like ‘search just works’ for end users. linguistic tools, such as spellchecking of queries or grammatical normalizing of content or queries, can greatly simplify users’ search experience. covering the wide variety of languages (sharepoint 2013 search covers 85 languages, from afrikaans to Zulu) also means that you can find content that is generated by users from across geographic boundaries.

linguistic processing is applied to both content and queries (as shown above), using a similar framework under the hood. as mentioned in other chapters, the content processing and query processing components have a heritage from modules called cTs and iMs, and they share an underlying framework for processing flows.

in preparing content for indexing, linguistics are applied in stages, each one building on the previous one. The figure below gives an overview of these steps in what is often called the ‘pipeline’. (The steps in gray are not ooB, but illustrate some of what is possible by adding third-party components.)

First, files must be parsed, teasing the indexable text out of powerpoint, onenote, pdF, etc.

during this process the language is detected, since processing english is different from processing Japanese. Words and patterns (dates, times, urls, etc) are found, based on the text and language. next, the ‘magic’ begins — a variety of types of Text analytics technology is then applied. stemming or lemmatization (which allows forms of the same base word to be matched, for example “sing”, “singing”, “sung”, or “incorporate”

and “incorporating”), synonyms (matching, for example, “car” and “auto”), and concept detection of various forms deal with the wide variety of ways humans say essentially the same thing. entity extraction, which is a key linguistic capability for sharepoint 2013 search, and techniques like categorization, relationship extraction, and sentiment analysis add metadata



that greatly improves the ability to find and explore information.

Microsoft has the deepest natural language processing development capability on earth, because it has labs around the planet. This was strengthened with the FasT acquisition, since one of FasT’s specialties was linguistics applied to search. strong language processing features show up in sharepoint 2013 search, which has continued a tradition of steady improvement in this area and has some extremely strong linguistic technology, including many improvements from sharepoint 2010. some of the changes will be directly apparent to the end user, but many of them show up in subtle ways, and some are only relevant to specialists handling unusual situations. For those coming from sharepoint 2010 search, there’s some remarkable new capabilities and improvements. For those coming from a FasT based platform, the capabilities are familiar, but are now much easier to work with. There are some capabilities you are used to from FasT which are no longer there — we mention the major ones as we cover each area.

There are some changes in sharepoint 2013 that will be noticeable to nearly all search deployments: document parsing is foremost, but also synonym management and custom entity extractors. some changes will only be apparent or available to those extending search, and some will be visible only to a specialized group of deployments.

Changes in Document Parsingsharepoint 2013 introduces a completely new document parsing facility, with some big improvements. These changes include:

• Automatic file format detection no longer relies on file extensions, eliminating the kind of errors that happened when users or applications do creative things like making .memo files.

• “Deep link extraction” works like a table of contents generator and allows you to click into previews for Word and powerpoint formats.

• Metadata extraction for titles, authors, and dates provides better metadata and is much easier to understand than the techniques used in sharepoint 2010 (where “optimistic Title extraction” was one of the top sources of user confusion).

• High-performance format handlers for hTMl, docX, ppTX, TXT, image, XMl and pdF formats mean faster crawls and indexing.

The new parsing facility is enabled by default and supports 55 of the most common file formats, including things like Montage, Visio and onenote. By comparison, the 2010 Microsoft Filter pack supported 15 formats, and the advanced Filter pack (available for FasT only) supported 422. For most deployments, this means you will no longer have to seek out third party iFilters — though the iFilter api is still supported and there is a rich assortment of IFilters on the market that cover file types beyond the ooB 55.

Other Changes You’ll NoticeLanguage detection has changed with sharepoint 2013. in sharepoint 2010, language



detection was done ‘chunk wise’ on document parts like paragraphs. now a much larger part of the document is used. The advantage of this is that language detection is generally better — the more language you can look at the more reliably you can tell what language it’s in. There is a downside to this approach, however — documents that have mixed languages — partly in english and partly in French, for example, aren’t handled as well.

The Term store (MMs) is well integrated with search now, which provides a number of big benefits. Customizations to Query Spelling correction are now managed in the term store — both inclusions and exclusions (shown below).

Property Extraction (previously a FasT-only feature) is also manageable in the term store (shown below). however, only company names are available — if you were using property extraction for people names or place names, you’ll need to find a third-party alternative. some things are still managed

outside of the term store — synonyms via a ui or powershell, custom extractors via powershell, and spell correction via a dynamic dictionary based on content in the index or a static ooB dictionary.

Offensive Content Filtering was a feature that could be enabled in FasT search for sharepoint. This feature, made it easy to shield users from obscenities and profane language that is found in content (even business content) remarkably often. however, it is no longer supported with sharepoint 2013, so you’ll need to find a third-party alternative if this is important to you. substring search, another FasT-only feature, was also removed. This provided n-gram matching without taking into consideration word boundaries, which was good for applications like part numbers.

Changes in ExtensibilityThere are notable changes in how you can extend linguistics processing with sharepoint 2013. These include:

Custom Extractors (previously FasT only) are more powerful, and you can have more of them (12 rather than the five allowed with FasT search for sharepoint). These allow you to provide a list of terms (via powershell) and match them in the content, populating managed properties with consistent metadata which is the lifeblood of information discovery.

Custom Word-Breaking now requires only one language-independent dictionary, rather than the one-dictionary-per-language facility in sharepoint 2010.



Customize stemming (done via registry settings in sharepoint 2010) is no longer supported. Third party specialists will find ways to customize this level of linguistics and handle specialized cases.

The biggest change is the availability of the content enrichment Web service (ceWs). This provides a way to add linguistic processing of any type, such as the examples in gray in pipeline figure above (concept extraction, relationship extraction, geo-tagging, summarization, etc). With FasT search for sharepoint, it was possible to extend the content processing pipeline through a sandboxed application, but this was both slow and limited in the information it could access. sharepoint 2013 introduces a much more open api which makes it possible to add specialized linguistics at lower levels as well as sophisticated text analytics. ceWs is covered in more depth in a separate chapter.

Putting Linguistics to workall of these cool capabilities come into their own when developing more specialized search based applications. This has become much more powerful with the application development hooks and tooling available, and you should expect to see some amazing applications built on sharepoint 2013 using these capabilities.

deeper divesTechnet article on linguistic search features in SP 2013 »

MSDN Section on Custom Word Breakers »

Longitude AutoClassifier Overview »

http://technet.microsoft.com/en-us/library/jj219499(v=office.15)


http://www.bainsight.com/Pages/sharepoint-metadata-auto-classification.aspx

40

ChaPter 4 architecture, deployment, and operations — getting under the hood

40

Chapter 4 GettinG under the hood


new architecture, single search engine coreThe first and foremost change to search within sharepoint 2013 is there is only one search engine core. The idea that you would use the FasT engine for content and the sharepoint engine for people is completely eliminated in this release. There is now only one search engine within the sharepoint 2013 platform — which you can think of as bringing FasT to all search tiers. powerful indexing, linguistics, extraction, and query expressiveness that are the heritage of FasT are now evident throughout the platform.

To appreciate the evolution from sharepoint 2010, it’s worth looking at the history in this area. The acquisition of FasT search and Transfer in 2008 was regarded by the industry as a major step forward in taking the lead in the enterprise search marketplace. The incorporation of FasT within the overall sharepoint 2010 architecture allowed organizations to leverage enterprise class search capabilities in a platform that was within the cost and budget requirements of today’s enterprises. unfortunately, the acquisition occurred midway between release cycles. This forced Microsoft to determine which features would be available in the wave 14 release (sharepoint 2010), and which features would need to be included in the next release.

FasT search for sharepoint is a very powerful product but there are numerous rough edges due primarily to the lack of time in the previous development cycle. The timeline also resulted in a hybrid architecture, with separate sharepoint and FasT farms, as shown below. This could be awkward and confusing to work with.

With the release of sharepoint 2013, the full realization of Microsoft’s investment in FasT search and Transfer is now evident. The capabilities now available take enterprise search to a whole new level. They are the result of a new search architecture. The architecture, shown below, is relatively simple, though much of it is new.



There is a good walkthrough of the components on Technet, which we won’t repeat here. each of the components shown are covered in at least one chapter of this book as well. however before we move forward, there are a few essential things to understand:

• search is fully integrated into sharepoint, and there is no longer a separate search server. certainly, a sharepoint 2013 server or services farm can be used only for search. To do this, you do want to have the MMs (term store) and user Profile service, at minimum — much as you did in sharepoint 2010.

• There are four different databases, each independent from the other. all of them can be partitioned, mirrored, and managed. The crawl database scales with the amount of content crawled, so this is typically the database that has multiple instances in a large search deployment.

• every component can be scaled out for capacity and for fault tolerance. previously, there could be only one search administration component, which meant you had to do creative workarounds to create truly fault-tolerant configurations.

• search is now multitenant — except for a few things, such as the ceWs api. Much more administration can be done at the site collection (or tenant) level.

Not ‘just’ a Merger of FAST and SharePointYou can think of this architecture as bring FasT to every tier of sharepoint, but it is much more than that. This is not a mere merging of FasT and sharepoint — nearly every component in this architecture is new. Just as sharepoint 2013 is a major architectural release overall, search is in many ways a radical re-architecture. The computational platform underlying the search based interaction for sharepoint 2013 is a powerful distributed dataflow engine (called noderunner).

an illustration that underscores this is shown below. This is the same architecture, though not using the official technology. The Crawl and ooB connectors (aka crawl component), are the least changed part of search in sharepoint 2013, and they retain the mssearch.exe name under the hood. The content processing Framework and interaction Management Framework (aka Query processing Component) are running flows, similar to CTS and iMs in the FasT search for internet sites 2010 product (see the content processing chapter). These are running under noderunner. so is the search core — which is neither the FasT esp core nor the sharepoint search core.



it’s a new, next-gen search core that was the result of a decade of research and development at FasT, hardened through the Microsoft development process.

also new in this architecture is the analyzer (aka analytics processing component), which we cover in the chapter on analytics. The content processing component writes information about links and urls to the link database. in turn, the analytics processing component writes information related to the relevance of these links and urls to the search index via the content processing component. This enables some powerful capabilities like recommendations and usage-based relevance enhancement.

if you look inside the search service, you will find several search processes. This includes Mssearch.exe (for the crawl component), noderunner.exe (which hosts search components), and a host controller (a Windows service that supervises noderunner processes. The host controller monitors noderunner processes, detects failures, and restarts processes if they do fail. There can be multiple noderunner instances on the same

server, each hosting one search component. on a default single server install there will be 5 instances of the noderunner.exe process, as shown to the left.

Although there is a fascinating dataflow engine and a next-gen search core, those are not exposed for developers — the only points of configuration for interaction are ResultSources, Queryrules, and ceWs. in sharepoint 2013, configuration alternatives are circumscribed to assure that no configuration would result in excessive resource consumption for that instance relative to other instances that may be running through the same service. so, Queryrules run effectively in a sandbox that restricts calls to non-sharepoint services.

Full Range of Search TopologiesWhen people think of architecture, some think of deployment topologies — machines, nodes, and processes. There is lots of good material on this physical architecture, which we will not repeat here. But we’ll give you a flavor.

as with sharepoint 2010, the minimum configuration is just one node, and the minimum configuration with fault tolerance is two nodes (FasT search for sharepoint required two and four nodes respectively). scaling from there to ultra-scale search (including the scale of o365) is possible, and you can grow incrementally.

The medium farm topology, shown below based on the Technet recommendation at www.microsoft.com/en-us/download/details.aspx?id=30383, is capable of supporting approximately 40 million items in the index. note

http://www.microsoft.com/en-us/download/details.aspx?id=30383

http://www.microsoft.com/en-us/download/details.aspx?id=30383



that we expect the density (items per node) of sharepoint 2013 search to go up dramatically over time, just as FasT search for sharepoint density did. The initial focus has been on scale-out in order to support o365, not on density.

What Matters is it Worksas a developer or user you don’t really need to know about the underlying algorithms or dataflow engine used in search. In fact, the search algorithms used by almost all search cores are a complex combination of linguistics and statistics, tuned heuristically. You can enjoy the result, and learn how to operate it well,

apply it to different applications, and develop on top of it.

What will you notice about this architecture? There are many things beyond the capabilities that meet the eye. For example:

• The core engine is different, so relevance is different. since Microsoft has a lot of data with which to tune relevance, you’ll notice first that the relevance is better ooB. But if you had customized relevance or spent time focused on it, you may have some work to do — or you may have a pleasant surprise.

• Indexing is atomic in the new search core. That has some very interesting implications, but mostly you’ll notice that it’s more robust and that you can do ‘normal’ backup and restore. For nearly all search engines it’s a dirty secret that data can occasionally get lost in indexing (so one in a million items may go missing), and an outage can result in needing a full re-index — but this core will be different.

• Scale-out is possible on a huge scale — big enough to run o365, and big enough for any challenge you can throw at it. FasT was always great at large scale, but this is



a different level; there should be less black art to building out big or high-throughput systems.

ultimately, what matters is that it works. other than the dogfooding done at Microsoft (which is pretty big), there isn’t much production experience with sharepoint 2013 yet, but every indication is that this is an architecture that is extremely solid — for both sharepoint generally and search specifically.

indexing and partitions in sharepoint 2013, there is a brand new search indexing core that is optimized for high volume throughout and overall scalability. The index component is the core of search; it accepts and administers both content and queries. content data is indexed and stored in index partitions while the index component simultaneously handles queries and generates results.

like many other features of sharepoint 2013, the index component and related architecture resembles FasT, with the ability to separate indexes into partitions for query loads and data volumes alike. This is a significant improvement over sharepoint 2010. The index is completely contained in these partitions and stored in the file system, without requiring a separate dip into sQl for metadata or for security entitlements — another huge improvement over sharepoint 2010, where the merge of results and security prevented deep refinement and also could bring performance to a snail’s pace.

index partitions are separate, which provide a lot of flexibility. They can be stored individually on disk in a file set. Alternately, they can be further divided into discrete sections containing a unique index component.

deeper divesSharePoint 2013 — Search Logical Architecture »

Technet Search technical diagrams »

TechNet on Planning for SharePoint 2013 »

http://www.nothingbutsharepoint.com/sites/itpro/Pages/SharePoint-2013-Search-Logical-Architecture.aspx

http://technet.microsoft.com/library/bcbae7bd-656b-4003-969c-8411b81fcd77%28Office.15%29.aspx#search in Technical diagrams for SharePoint 2013 http://technet.microsoft.com/library/bcbae7bd-656b-4003-969c-8411b81fcd77%28Office.15%29.aspx

http://technet.microsoft.com/en-us/library/cc261834.aspx



Microsoft has also developed a new nomenclature to describe the structure of the index. in FasT search for sharepoint 2010, the structure of the index and configuration was described in terms of rows and columns. adding columns increased the amount of content you can index and adding rows increased query volume throughput and redundancy.

in sharepoint 2013 they have now adopted a Partition/Replica model to define functions within the overall search index, as shown below. partitions are logical divisions of the overall search index. The entire index is composed of the aggregation of all the primary replicas across the logical partitions. When content is sent to the indexing component, a transaction is generated to acknowledge receipt of the content. each partition then indexes the content from this transaction log. secondary replicas are created as read only copies of the primary replica for scaling query volume of adding redundancy to the overall architecture.

Within a partition, there is only one primary replica that is responsible for writing data in the partition. each partition can be served by

one or more replicas of the index. The indexing component is responsible for managing and distributing the index across partitions. if an additional partition is added, the indexing component is responsible for the re-distribution of data across all the partitions. it is important to note that you can add additional partitions without re-indexing the data, but removal of a partition will force a complete re-indexing of all content.

A Simpler, More Robust ApproachThis new structure of the search index in sharepoint 2013 allows for a fully redundant, scalable means of indexing content. The fact that you are not copying index files from server to server and row to row means there is considerably lower latency to making search indexes replicated and available. This also significantly reduces the server to server chatter that existed in previous versions. each partition operates independently thereby increasing throughput and performance of the overall search sub-system.

In a nutshell, the benefits of this approach are:

1 Better indexing throughput

2 less network chatter

3 Faster availability of the search index.

as previously mentioned, the indexer is now atomic, which is a major breakthrough in search technology. Though the change is invisible to you, so you’ll notice that it’s more robust and that you can do ‘normal’ backup and restore. indexing and partitioning are deep stuff, and this is a new core capability done well.



analytics analytics are an often overlooked area, but have a crucial role in search — both in providing insight into user behavior and system operations, and in improving the user experience. sharepoint 2013 has a new analytics architecture, which merges web analytics (where people click and navigate) and search analytics (what people search for and what results they get). This is a great improvement over sharepoint 2010, where the web analytics service application was quite limited in both capability and scale. The result is called the Web analytics platform, which has been completely redesigned and integrated into the search service application of sharepoint 2013.

The analytics architecture consists of the analytics processing component, analytics reporting database and link database (as shown below). The analytics processing component analyzes crawled items (search analytics) and how users interact with search results (usage analytics). it uses the information to improve search relevance, and to create search reports, recommendations, and deep links.

The analytics processing component extracts two kinds of information:

• search analytics information such as links, anchor text, information related to people, metadata, etc. from items that it receives via the content processing component and stores the information in the link database.

• usage analytics information such as the number of times an item is viewed, from the front-end via the event store.

The analytics processing component analyzes both types of information. The results are then returned to the content processing component to be included in the search index. results from usage analytics are also stored in the analytics reporting database for reporting purposes. The analytics component updates the sharepoint search index at time intervals set via a timer job, so it is independent of the crawl schedule. This can be confusing if you are trying to understand why search relevance changed. There is an extension point for custom events, but the analytics processing and search index update data flows are sealed from enrichment updates outside the sharepoint 2013 crawl.

The results are most visible to the user as reports and recommendations. But there are several other ways that analytics shows up:

• Search relevance is enhanced based on user behavior (views, click thru, etc.)

• Popularity of content and of topics in discussion threads — which is driven from number of views as well as number of unique users to view — and can be viewed directly



• Popularity can also be used to create views through the content by search (cBs) Web part

usage analytics in WcM are particularly important, since they provide essential insight into the effectiveness of your web site. These analytics are search driven, built to scale (scaling was a weakness in sharepoint 2010), and open for extension. a “Top pages” web part is included by default. some data like view counts are also pushed into the index so it can be included in search results, sorted on (i.e. what’s most viewed), etc.

personalized search queries and personal query suggestions in sharepoint 2013 are based on analytics data and usage information for each user. recommendations (both item-to-item and popularity based) are available through this approach, as shown below. The “recommended for you” list is simply a preconfigured Content by search web part — it looks like a static list but it’s generated dynamically by search.

The addition of both the link database and the analytics reporting database provide for a great deal more personalization, analysis, and relevancy within the engine. The analytics reporting database has been added to keep track of all forms of analytics. search analytics analyze crawled items and how users interact with search results. These actions are stored

in the event store within the Web Front end (WFe) server and are regularly pushed to the analytics processing component where the actions are analyzed and reconciled. They are then pushed into the analytics reporting database and made available to the query and processing components. This allows for search to keep track of user actions, queries, and trends to provide the user with better search results and suggestions. This database now powers features such as personal and engine-wide query suggestions, favorites, and other search personalization components not found in any other enterprise search platform today.

Within the analytics system, there are five parts:

• Event: each item comes into the system as an event with certain parameters

• Filtering and Normalization: each event is looked at for special handling, normalization, and filtering; some are filtered out

• Custom Events: You can configure up to 12 custom events in addition to what comes ooB

• Calculation: sum or average across events

• Reports: a number of default reports are available, including top queries, most popular documents in a library or site, and historic usage of an item (view counts)



The figure below shows an overview of the data flow for usage analytics, usage events, and recommendations.

note that 2010 web analytics aren’t supported running 14 mode, so running in 14 mode means running without any analytics.

Better Analytics Mean Better SearchThe quality of search results has direct correlation to the quality of the query and the volume of information that you provide to the search engine. in sharepoint 2013, the addition of the analytics reporting database significantly increases the quality and quantity of information that is provided to the search engine. Knowledge about the person asking the question and the community asking the question greatly improves the quality of results

provided by the engine, as well as improving the quality of queries the user issues.

Federation and result sourcesFederation has been present in sharepoint since Microsoft released search server 2008 and service pack 2 for Moss 2007. in a nutshell, this is the ability to query multiple search indexes on behalf of the user and to return all of these results together in a single view. Thanks to federation, users no longer have to use multiple search centers in order to search all content accessible within their organization. instead they can go to a single search page and get all results available in one place.

deeper divesTechNet overview of Analytics in SharePoint 2013 »




Federating or Indexing?Whenever someone is newly introduced to federation the immediate next questions that come up is: how does federation relate to indexing? Why should i continue to index remote systems if i can federate these?

The truth is that indexing, if possible, is always better. if you index the content you can control relevancy, freshness, performance, faceted navigation, and filtering for the end-users (among other things). When you federate across search indices, you essentially relinquish control of these and become dependent on what the other system is capable of. With federation, your page will also be as slow as the slowest search engine queries and as relevant as the weakest sear engine queried. so federating results must be done carefully.

Federation has proven very useful for scenarios where indexing may not be desirable or even feasible. For instance, your content is spread across multiple offices with low bandwidth connection, making any remote crawling last for days. in such conditions, you would not be able to keep your index fresh enough for your end-users. another scenario is when you have so much content to index that it may not fit within a farm. Imagine, for instance, a 50,000-employee company wanting to search across sharepoint and e-mails. even at a low estimate of 10,000 items per mailbox (that’s roughly six months for an information

worker), this would represent over half a billion items to index! Finally, the remote source may not allow for crawling, technically or through license restrictions (imagine a secured deep-web content provider). in these cases federation is pretty much the only way to go.

Result Sources for Federation in SharePoint 2013sharepoint 2013 offers improved federation capabilities via a functionality called result source. on top of the open search protocol already supported in Moss 2007 and sharepoint 2010, you can now federate results from remote sharepoint farms via result sources. This allows sharepoint 2013 to better cover distributed organizations. a result source is quite easy to configure, as shown below.

While the options on sharepoint 2010 to provide organization-wide search were limited to a multi-search center or a published centralized search service, sharepoint 2013 let you federate across farms. You can now have one farm per region or office location and federate results across farms using results sources. You can do the same between your intranet and extranet farms.

While simple on the surface, this functionality fills a serious gap that existed in the overall



scalability of sharepoint 2010. in the marketplace, FasT and sharepoint were being criticized for not having a global systems architecture. The approach was to tell users to centrally index all content in a large central farm, if the latency allowed. For global organizations, this was often not feasible.

There are limitations to the remote result source construct. it is limited to sharepoint 2013 and requires that all federated farms be upgraded to sharepoint 2013. results are not interleaved, which is what users typically expect; rather, they are provided in result blocks. Refiners are also not combined in any way. overcoming these limitations is an exercise left to partners. But despite these limitations, remote result sources are a major step forward and a great feature to use.

result sources also take over the function of scopes in sharepoint 2010. They are a more powerful tool than both scopes and federation, and are worth getting to know.

Exchange 2013 Result Sourcesharepoint 2013 also allows administrators to federate results between sharepoint and Exchange, providing a unified search experience where users can search both sharepoint content and their mailboxes through a single search center without having to index. exchange remains in control of indexing the mailboxes and users can search across systems using federation with no additional hardware requirement. This is available because exchange 2013 has the same underlying search core (see the exchange search chapter)

Security via oAuthsharepoint 2013 can also provide security-trimmed results in a much more streamlined way. The Kerberos protocol is no longer a pre-requisite to providing security-trimmed results. instead sharepoint 2013 offers strong security support through federation by leveraging the claim-based authentication mechanism built into sharepoint 2013 or by using the single-sign-on/secure store service. a trust must be established between the farms using a new method called oauth which allows the passing of the current user’s claims to the remote farm when making the search request. This is similar in concept to establishing a trust between servers to consume service applications. oauth is a new methodology replacing Kerberos shared authentication.

When combining result sources and result blocks, administrators can offer their users a single list of results comprised of both local and remote results. The remote results are shown as result blocks (one per source) either above all results, or merged within the local results returned. note however that faceted navigation and property filtering are still driven by local content only and do not reflect any filters or facets available from the remote indexes.

Office 365 and SharePoint OnlineOffice 365 has rendered organizations more agile by enabling them to consume sharepoint as a service without having to worry about capacity, backup, or maintenance. however it also created a new challenge as organizations migrating to the cloud were now facing siloed data with some content available online and



some content available within the organization network only. There was no single place to search both sets of content from. sharepoint 2013 solves this scenario by enabling remote sharepoint result sources to also support sharepoint online, therefore enabling scenarios where sharepoint online can federate with the on-premise search engine or vice versa. result sources represent a key piece of technology to help organization migrate to sharepoint online.

search in exchangesearch in exchange 2013 has been given a facelift. pull back the curtain, and it is the same new search core used with sharepoint 2013, optimized for large volumes of e-mail.

To provide some comparison, Microsoft exchange server 2010 search allows users to perform full-text searches across documents and attachments in messages that are stored in their mailboxes. exchange search (also known as full-text indexing) creates the initial index by crawling all messages in mailboxes within an exchange 2010 database. as new messages arrive, exchange 2010 search updates the index based on notifications from the Microsoft exchange information store service.

The figure below shows an overview of the exchange search architecture in exchange 2010. Full-text indexes are not stored in your exchange databases. The search index data for a particular mailbox database is stored in a directory that resides in the same location as the database files.

in exchange 2013, the exsearch capability is replaced with a new search engine and index.

This provides a much more powerful, more effective search for exchange users — available through outlook and outlook web access alike.

Another significant outcome of this change is that exchange 2013 can appear as a result source to sharepoint 2013, as introduced in the chapter on Federation. This opens up a number of scenarios combining e-mail and other documents. in previous versions of sharepoint, you had the ability to connect to, and index exchange public folders but not personal inboxes. That remains the same with sharepoint 2013 (unless third party connectors are used), but now there is an ability to federate to exchange.

deeper divesMicrosoft’s comparison of indexing vs. federating »

TechNet — configuring result sources »

Federation Use Cases »

Federation vs. Indexing »



http://www.bainsight.com/Pages/federated-search-for-sharepoint-use-cases.aspx

http://www.domorewithsearch.com/federating-versus-indexing/



The key concept to understand in regard to this functionality is that each system handles the data resident within its silo (e-mail, tasks, contacts in exchange 2013 and documents and lists in sharepoint 2013). as discussed in the Federation chapter, there is some downside to this approach — federation does not provide the same content processing, relevance, or performance as indexing. But this level of integration between sharepoint and exchange is a wonderful feature that will help many users. You can get a single view across exchange and sharepoint, as shown below.

one of the new key features in sharepoint 2013 that relies heavily upon this tight integration between sharepoint 2013 and exchange 2013 is the new enterprise content Management (ecM) stack and the associated e-discovery components. From the e-discovery perspective, the integration of sharepoint and exchange allow for in place preservation of information within sharepoint and exchange. The e-discovery console allows for dashboard view of integrated, enterprise-wide case management.

A Unified View is a Better ViewSince the first release of SharePoint, there has always been a desire to be able to support searching your personal inbox to provide a more holistic view of your information. in

previous versions of sharepoint, there was support for indexing content from Microsoft exchange, but only in public folders. With the release of sharepoint 2013 and the fact that exchange 2013 is using the same search infrastructure it is now possible to provide federated access to personal inbox results within sharepoint 2013.

The primary benefits of this approach are:

1 exchange 2013 and sharepoint 2013 leverage the same core search sub-system

2 possible to include federated personal inbox results from exchange 2013

3 eliminates the need to re-index all inbox data within sharepoint 2013



search administrationThere tends to be a preconception that search requires no administration. This is due in part to the simplicity of the search interface and the general lack of awareness of how search works. But it is also due to people’s experience of web search, where they don’t have to do any upkeep. little do they realize that google.com has over 4,000 people administering search full time!

administering enterprise search doesn’t take that much work, but it does need to be someone’s job (even if not a full time job). There are two main levels of administration: system administration (installation, configuration, topology management), and search administration (rules, best bets, looking for no-results searches).

Simpler Architecture Means Simpler Administrationsharepoint 2013 search is simpler to administer on many levels than sharepoint 2010 was. part of this is that there is only one search engine core, and no hybrid architecture (see the one search core chapter). For FasT search for sharepoint, you had to install two farms (a FasT farm and a sharepoint farm) and make them work together, including creating multiple search service applications. There was extra work in installation, extra work in configuration, and extra work in reconfiguring sites. There was also more troubleshooting because the architecture was more complex.

There is now only one search core, only one installation, and only one search service application. There is a much simpler architecture, as shown below. as a result sharepoint 2013 is much simpler to install, configure, and troubleshoot as a result.

deeper divesTechNet — What’s new in Exchange 2013 »

Overview of eDiscovery and In-Place Holds (SharePoint 2013) »

http://technet.microsoft.com/en-us/library/jj150540(v=exchg.150).aspx

http://technet.microsoft.com/library/d6a85e37-4a3c-4c89-913c-563bb9c8a013(Office.15).aspx




Multiple Administration Componentsas mentioned in a previous chapter, the search administration component is now fault tolerant, a big advantage for sharepoint 2013. The administration database now contains only configuration and log information (it also held security entitlements in sharepoint 2010). There are new tools to export and import configuration information, including PowerShell commands, so there are some very cool things you can do in configuration management.

Administration at Multiple Levelscentral administration is still your friend with sharepoint 2013, and still the place where you create search service applications. You will find some new services applications on sharepoint 2013 (such as the Machine Translation service) and improvements on existing ones. But many of the operations will be familiar. The screenshot below shows an example list of service applications.

as a new thing in sharepoint 2013, you now have site collection level search administration too. it’s pretty similar to central administration, naturally with a few limitations. site collection administrators can set up and manage app catalogs, do term store management and User Profile Management, as shown in the screenshot below. site collection administrators also have the power to manage some search settings in their site collections — a huge step forward.

search administration at this level is pretty comprehensive, as you can see just by looking at the search administration screen below (note that this is from Office 365, where it’s called tenant administration)


it is natural that this level of administration was introduced in sharepoint 2013 because of the emphasis on running multitenant in the cloud. site collection administrators can start crawls; create result sources, and much more. This includes creating managed properties, which could only be done via central administration in sharepoint 2010, despite the fact that site collection or site administrators typically understand their content and crawled properties much better than central iT.

site administrators also have much more power with sharepoint 2013. They cannot create managed properties, but they have significant control over search — which applies to their sites. The table below shows some examples of what site collection and site administrators can do.

Administering the New Mechanismsin other chapters we described new mechanisms like Query rules, result Types, and result sources. These are very powerful for the administrator. a search service application administrator can create result sources, and the site collection administrator’s site owner and site designers can also create and configure result

sources in order to give powerful search options to their end users. Query rules and result Types can be managed down to the site level. These have a wizard for configuration (for example, the query builder interface) with a built-in preview of what the results look like. result sources are easy to manage, as shown below.

There are very significant improvements in analytics, resulting from the new analytics module. There are also better crawl reports, and process reports (see below). since the host controller (described in the one search core chapter) is monitoring all noderunner processes, it can give the administrator a lot of insight into the system operations.



PowerShell is Your Friendpowershell support was added to sharepoint 2010 and many administrators fell in love with it — for good reason. There are even more powershell options in sharepoint 2013. This includes more powershell commands for search: general search administration, crawling, search service application, querying, metadata, and topology. in sharepoint 2013, powershell can now manage content sources and crawlers, not just report status. There are new options for creating a new search topology based on an XML configuration file, along with export and import commands. This means you will be able to create the same search topology in your staging environment, in your test environment, development environment and production environment. This can be very useful for performance testing, custom development, creating standardized configurations, etc.

Ranking models are still configured via

powershell like in sharepoint 2010, but in 2013 site collection administrators now have the ability to call a specific ranking model defined by the SSA admin from within query components at the site level. This means that site collection administrators can do much more with relevance control and ranking, choosing from a library created by the central administrator.

powershell is available at all levels: central, site collection, and site administration, which gives

you much more power. For example, we can create a PowerShell script for configuring all our search settings from the very beginning, from creating a search service application, modifying its settings, creating the content sources, etc. powershell can also retrieve, create, or modify query results. in addition powershell can get keywords, modify ranking models, and more. if you haven’t learned powershell already, you will definitely want to learn it now.

Big Advances in Search Administrationsearch administration is still a complex task in sharepoint 2013, but Microsoft made the job much easier in this new version. The new single search core provides the power of FasT with a much simpler configuration than FAST Search for sharepoint. search topology administration is still complex, but the topology can be much bigger and much more powerful. There are



improvements on administration from all sides: crawling, content processing, query processing, analytics, and user experience. This is a search that administrators can learn to love.

upgrade and MigrationYou will love the capabilities of sharepoint 2013, and you probably own them already. But how much pain is it to move to them? Many organizations endured a very painful move from sharepoint 2003 to sharepoint 2007 and are still wary, despite a generally smooth move from sharepoint 2007 to sharepoint 2010.

The good news is that this release has put a lot of focus on upgrades and there is a lot of good material. in order to move customers on o365 to the new release, Microsoft had to develop techniques for doing this more smoothly than ever before.

The bad news is that upgrades are still tricky, especially for large and highly customized sharepoint farms. even though the upgrade itself is fairly straightforward, there are usually lots of factors besides the software itself — the hardware necessary to handle an upgrade (there are no in-place upgrades to the new version), the user awareness and education, and the work needed to take advantage of new features.

There are techniques that can reduce the risk and pain of upgrades, especially for search. These include things like use of cross-version federation and ‘search-first migration’. But let’s start with a look at the standard basic upgrade.

Database Attach UpgradeThe only upgrade method for going from sharepoint 2010 to sharepoint 2013 is a database attach upgrade. (in-place upgrades are now only for build-to-build changes). This works for both content databases and services databases.

deeper divesTechNet — Index of Windows PowerShell cmdlets for SharePoint 2013 »

Technet — search topology in SharePoint Server 2013 »

SharePoint 2013 Developer Dashboard »

TechNet — Manage the search schema in SharePoint 2013 »

TechNet —View search diagnostics in SharePoint Server 2013 »

http://technet.microsoft.com/library/ff678226%28office.15%29.aspx

http://technet.microsoft.com/library/ff678226%28office.15%29.aspx



http://blah.winsmarts.com/2012-7-SharePoint_2013_Developer_dashboard.aspx

http://technet.microsoft.com/library/81890ff0-e2f9-4752-8e8e-2e8502c76311(Office.15).aspx

http://technet.microsoft.com/library/81890ff0-e2f9-4752-8e8e-2e8502c76311(Office.15).aspx

http://technet.microsoft.com/library/5cf2a498-d879-4673-b4d4-2eaa36695ff9(Office.15).aspx

http://technet.microsoft.com/library/5cf2a498-d879-4673-b4d4-2eaa36695ff9(Office.15).aspx



The search databases have changed significantly with sharepoint 2013. The search administration database supports a database attach upgrade, but the search index databases do not. as with essentially all search engines, to do an upgrade you will need to recrawl your content. one very nice advantage with sharepoint 2013 is that you can use powershell to make this happen with much less effort.

The database attach method does help a lot with search. content sources, server mappings, scheme, federated locations, scopes, best bets, and the like are all preserved and upgraded. as mentioned in the search administration chapter, there are tools for configuration import and export as well as powershell commands that can do very interesting things, including automate and tailor the upgrade process.

Deferred Site Collection UpgradeThe visual upgrade available in sharepoint server 2010 has been replaced by a deferred site collection upgrade in sharepoint 2013. This allows existing 2010 site collections to work unchanged in sharepoint 2013. no sharepoint 2010 installation is required; sharepoint 2013 has all of the required SharePoint 2010 files included.

This process is much safer, because it is deeply backwards compatible. it is the default for all site collections upon a database upgrade, which then automatically are running in “14 mode” on sharepoint 2013 servers. There is a new facility for health checks along with the upgrade, and a cool capability to create upgrade evaluation sites. essentially, this makes a side by side copy of an existing site collection, and allows

users to preview an existing site in “15 mode”. deferred site collection upgrade permits use of sharepoint 2010’s ui with fewer operational hassles, while retaining the master page, Jscript, spF, and css applications of sharepoint 2010.

This is an expensive operation, so you probably don’t want to use it everywhere, but it is a great facility to allow for safe, well managed upgrades — both from the software perspective and the user perspective.

With search, an upgrade of search centers generates result templates that include the hover panel, and which have previews (when a separate Office Web Apps server or set of servers is available). scopes are upgraded but can’t be changed — they are replaced by the new result Templates, but the corresponding result templates aren’t generated automatically.

Working Across Versions: Search-First Migrationsince search is greatly improved in sharepoint 2013, it may be worth considering a “search first” upgrade. This lets you get the benefit of the new features and capabilities, without needing to do everything at once. You can upgrade your content farms at any pace that works for you, while serving everything from sharepoint 2013 search.

This pattern uses something called sharepoint 2013 Federated services. only a few federated services support this: Search, Profile, Social, secure store, Managed Metadata, and Bcs. But this is everything you need to do a search first migration.



There are several steps to a search-first migration, as shown below.

sequentially the steps are as follows:

1 Deploy and configure a new SharePoint 2013 services farm, including a search center. Migrate the search settings from the SharePoint 2010 farm. When the search-first migration is complete, this farm provides search functionality to end-users who are still working in the sharepoint server 2010 farm.

2 crawl all content in the sharepoint server 2010 farm by using a crawler (or multiple crawlers) in the sharepoint 2013 services farm. continue to crawl this content regularly.

3 Configure the SharePoint Server 2010 farm to consume search from the 2013 services farm, using federated services. some things will be best consumed by doing redirects (for example using a new search center with the new functionality can’t be done via federated services).

The search-first migration pattern opens the door for a much wider set of possibilities — hybrid solutions.

Hybrid SolutionsWhen you talk about the upgrade of search from sharepoint 2010 to sharepoint 2013 — there is the potential for some hybrid solutions using different versions of sharepoint or using cloud and on-premise sharepoint instances in the same company.

generally, hybrid means a combination of on-prem and cloud content in a single view. There are several ways to accomplish this, including indexing and federation — as mentioned in the Federation chapter. The figure below illustrates the various permutations of hybrid configurations.



crawling and indexing content from the cloud (such as from o365) is a very solid way to create a unified view, and has the benefit that indexing generally has: unified content processing, solid and consistent relevance and navigation, and consistent fast performance. although this scenario is not supported by ooB connectors with sharepoint 2013, there are partner-built connectors that accommodate it.

With sharepoint 2013, the remote result source construct means that a view can be created using federation, specifically between o365 and on-prem sharepoint. There are limitations to the remote result source construct. it is limited to sharepoint 2013 and requires that all federated farms to be upgraded to sharepoint 2013. results are not interleaved, which is what users typically expect; rather, they are provided in result blocks. And refiners are not combined in any way. overcoming these limitations is an exercise left to partners. But despite these limitations, remote result sources are a major step forward and a great feature to use.

Cross-version Configurationshow do these scenarios help with upgrade and migration? if you extend them to cross-version configurations, it becomes clear. Search-first migration is an example of crawling on-prem content from on-prem search (the upper left scenario in the figure above), but across versions. By crawling sharepoint 2010 content from sharepoint 2013 search, you can provide an upgrade path that can be done a step at a time, maximizing the benefit to users while minimizing initial effort.

The same idea applies to more general scenarios. When you have more than one sharepoint farm, you can handle cross-version scenarios. You can have a search on sharepoint 2013, while you have content and other applications on sharepoint 2010 or even in sharepoint 2007. You can have sharepoint 2013 in the cloud with sharepoint 2010 on-prem. You can include other content in the cloud that should be crawled, such as Microsoft crM online or salesForce.com. With these techniques, it’s possible to field a very broad with different versions and different options. This helps with many things, including migration.

Federation applies well to cross-version scenarios. although sharepoint 2013 only supports same-version remote result sources, it is feasible for partners to create federation across multiple versions, which appear as a result source. A configuration like the one shown below provides many benefits. With respect to upgrades and migration, it means that legacy search systems can be left in place and federated into a sharepoint 2013 search center. While this is not as good as combining all content into a common index, it is a very useful technique that allows you to upgrade or migrate complex systems a piece at a time.



Cross-version Hybrid Configurationsa variant of this is support of cross-version hybrid configurations. Specifically, you may wish to adopt sharepoint 2013 online via o365 to have different versions. You may not actually have a choice in the matter, since o365 will shift to sharepoint 2013 fairly quickly, faster than you may be ready to upgrade your on prem sharepoint farms. But the remote result source mechanism in sharepoint 2013 is very powerful, and has solved many of the toughest aspects of managing hybrid configurations with O365 — such as security and single sign-on. it is feasible (though not ooB) to apply this to a cross-version hybrid configuration as shown below.

Migration From Previous Search VersionsThe changes in sharepoint 2013 search are powerful and far reaching. Fielding a single new search core resolves a historical challenge with Microsoft search (a complex product lineup with many different versions). one tricky aspect of this change is that migrating from previous search versions depends on the flavor of search you are migrating from.

The case of migrating from sharepoint 2010 search to sharepoint 2013 search is the best supported one. There are some gotchas in this migration as mentioned throughout this e-book, but the process is generally smooth and well covered by Microsoft. going from sharepoint 2010 search to sharepoint 2013 is a step up in nearly every way, so there aren’t that many rough spots to consider. if you are migrating from FasT search for sharepoint, many of the same tools and techniques apply, but there are more corner cases to consider and more feature changes to consider.

if you are moving from FasT esp or FasT Search for Internet Sites, there are significantly more considerations. The migration patterns and techniques still apply, but you are more likely to have a heavily customized search deployment that uses special FasT features which have been supplanted by other mechanisms. There is help available however. Microsoft has a big ecosystem of partners and there are some that have specific focus, tools, and techniques for this kind of migration. You may not get direct support from Microsoft, but you can tap into this ecosystem for help.

Summary — Options for Upgrade and Migration upgrading to sharepoint 2013 can be seamless, and there are valuable tools and processes provided ooB. The only supported approach is a database attach upgrade, so you should expect to provide extra hardware resources for your upgrade. But the deferred site collection upgrade facility provides a safe approach to upgrades and lets you delegate the work for



each site collection to the appropriate owner if you like.

upgrading search is part of upgrading sharepoint, and the standard upgrade process from sharepoint 2010 to sharepoint 2013 covers search well. But search poses special challenges — the more complex and customized your search configuration is, the more challenging the upgrade will be.

search also offers solutions to many upgrade challenges for sharepoint as a whole. since search bridges information silos, it can bridge across different farms, across different versions, and across on-prem and in the cloud instances. Techniques such as search-first migration, crawling remote farms from sharepoint 2013, and use of federation are available — not ooB but through Microsoft’s ecosystem. A unified view across these different dimensions provides users a great experience while allowing you to upgrade or migrate one piece at a time.

deeper divesServices upgrade overview for SharePoint Server 2013 »

SharePoint Online administration »

BA Insight resources for Integrating O365 content »

TechNet — SharePoint Server 2010 deprecated search features »

TechNet — FAST Search Server 2010 for SharePoint deprecated features »

http://technet.microsoft.com/en-us/library/ee731990%28v=office.15%29

http://technet.microsoft.com/en-us/library/ee731990%28v=office.15%29

http://technet.microsoft.com/en-us/sharepoint/gg144571.aspx

http://www.bainsight.com/resources-for-sharepoint-fast-search/Pages/Integrating-Office365.aspx

http://technet.microsoft.com/en-us/library/ff607742(v=office.15).aspx#section3




64

ChaPter 5 applications and development — new Models for search-Based applications

64


Chapter 5 New Models for search-based applicatioNs

The new development Model in sharepoint 2013 With a new development model for sharepoint 2013 and for search, the capability to extend search is much more accessible. We think this development will foster a lot of exciting search-based applications.

development with sharepoint 2013 emphasizes standard web technologies such as Javascript and hTMl, client side programming and remote calls. There’s a focus on running applications in the cloud, and there are several options for extending the out-of-the-box capabilities of the product. There is also the option to build business solutions with no or minimal use of server-side code.

Javascript and modern hTMl and css know-how are important for the ui designer and developer on sharepoint 2013. it should be easier for designers to use tools they are familiar with. Visual studio 2012 offers strong tooling for both Office 2013 apps and sharepoint 2013 applications and solutions. a key goal the sharepoint 2013 for customization scenarios was to make developing applications for sharepoint much more like developing Facebook apps.

A New Programming ModelThe figure below gives a birds’ eye view of the changes between the sharepoint 2010 and sharepoint 2013 programming models. in sharepoint 2010, your custom code ran either server-side in sharepoint (as fully trusted code or in a sandboxed solution), or via a client side object Model (csoM). The sharepoint

2010 csoM is a Windows communication Foundation (WcF) service with three different proxies to enable silverlight, Javascript, and .net managed clients to call into sharepoint remotely. With sharepoint 2013 the server side code runs off the sharepoint server farm via declarative hooks like apps, declarative workflow and remote events which then communicate back to sharepoint using csoM or resT.

There are lots of advantages to this model. Traditional sharepoint development was heavy lifting and had a steep learning curve; the new sharepoint 2013 model is much more manageable which will open up sharepoint to a much wider audience of developers. server-side code can impact the performance of sharepoint, be complex to install and upgrade, and can’t be run on public cloud services.

The csoM in sharepoint 2013 is much more powerful — you can do almost everything the server side apis did in sharepoint 2010. in addition, it supports odata — now the



leading industry protocol for performing crud (create, read, update and delete) operations against data, as shown below. depending on your deployment scenario, you can still use sandbox and farm solutions to push server side code to sharepoint 2013, however, Microsoft recommends that developers follow the new app model as the preferred way of building their custom applications for sharepoint 2013. The message is “don’t make any new sandbox solutions” and “build new farm solutions only if you absolutely have to”.

Apps for SharePointThere’s a new way of packaging and deploying code in sharepoint 2013 which is aimed at development of lightweight apps. apps for sharepoint don’t live in sharepoint. They execute in the browser client or on a remote Web server ; they’re granted permission into sharepoint sites via oauth (a standard for providing delegated authorization to apps); they communicate over the new sharepoint 2013 csoM apis. There are three types of apps you can build for sharepoint 2013:

1 SharePoint-Hosted App (which runs within the browser)

2 Provider-Hosted App (which runs on another web server in the datacenter or cloud)

3 Azure Auto-Hosted App (which runs in an azure instance which is invisibly provisioned by Office 365)

apps are simple and powerful, but they have a number of limitations, and there are still many cases where sharepoint solutions are called for instead. anything that uses server-side code, does farm-level work, has a high level of complexity, or has installation coupling or dependencies calls for a sharepoint solution rather than a sharepoint app.

What’s Special for SearchThe sharepoint 2013 search csoM opens most (but not all) of the Query object model functionality for online, on-premises, and mobile development;

the search results data is in Javascript object notation (Json). Queries support two language syntaxes: KQl (Keyword Query language) and FQl (Fast Query language); sQl is no longer supported.

in addition to the csoM, there is a resT (representational state Transfer) service, so you can remotely execute queries against the sharepoint 2013 search service from client applications by using any technology that supports resT web requests. The search resT service exposes two endpoints, query and suggest, and will support both geT and posT operations. results are returned in either XMl or Json format.

at one level, a search app is just another sharepoint app, and a search solution is



just another sharepoint solution. This is revolutionary enough: it means you can use search via a resT interface, include it in an Office App, and use it easily in combination with other parts of sharepoint. But customizing search also means creating or customizing connectors using Bcs or a protocol handler (see the content capture chapter), customizing linguistics using the content enrichment Web service (ceWs) (see the content processing, linguistics, and ceWs chapters), working with other service applications, and more. There are numerous search-specific web parts, including the new content by search web part (shown below), which is a powerful “swiss-army knife” tool.

search combines well with other parts of sharepoint — with content management, with workflows, with BI, and with sites. It also can be used with several of the new service applications in sharepoint 2013. These include the Machine Translation service that supports

automated language translation of files (think multilingual search), and the Work Management service that provides task aggregation functionality.

if you are doing query-side-only work, you might be able to use an app model. But for the most part, developing sophisticated search-based applications will remain the domain of sharepoint solutions with sharepoint 2013. There are several things (connectors and pipeline extensibility) which are still per ssa and not per tenant.

Building Search-based Applicationssharepoint 2013 is a great platform for building search-based applications. These run a wide gamut, from configuring departmental centers using query rules and result blocks, through extending content processing to add domain-specific entity extraction, to creating brand new user experiences. They are often specific to role, industry, and topic — and they usually have a strong and measurable business value because the end users use them for specific purposes. Wherever there is an identifiable group with a need to work with unstructured content (or a mix of structured and unstructured content) there’s a need for a search-based application. We discuss this more in the chapter on search-Based applications.

nothing is perfect, and there are still challenges with the search development model. some of the limits of sharepoint 2013 include:

• on the content side, there is no ‘push’ api for content, nor an ability to do partial updates. continuous crawls



are limited to sharepoint content only. There’s no mechanism for getting external data indexed into o365. developers that want to approximate these from the outside have to live with limited performance and build a very complex structures or use third party frameworks.

• Many of the mechanisms inside search are sealed and can’t be extended. Update groups, query flows, analytics processing, web crawling are examples. it’s completely understandable that these be kept intact from meddling developers, and there are some of these that can be influenced safely using partner products. But it’s frustrating to see these mechanisms and not be able to touch them.

• The sharepoint app model and sharepoint Marketplace are aimed at lightweight, simple apps and not something you would use for a full business application today.

developing with search is still hard. intrinsically, areas like content processing and relevance are imperfect, since we’re dealing with human language and subjective opinions of the ‘right’ answer. There are no joins or aggregation internal to search so there are limits to combining structured and unstructured content. But sharepoint 2013 is far ahead of any search platform in terms of available capabilities, performance, ease, and safety of development. and there is a strong ecosystem with available building blocks and complementary capabilities to use in creating great applications with search.

deeper divesBook chapter on developing with search from Wrox “Professional SharePoint 2010 Development” »

MSDN overview on developing with SharePoint 2013 »

MSDN section on building search queries with SharePoint 2013 »

SharePoint 2013 Developer Dashboard »

http://msdn.microsoft.com/en-us/library/hh144966.aspx





http://blah.winsmarts.com/2012-7-SharePoint_2013_Developer_dashboard.aspx



The content enrichment Web service (ceWs) as mentioned in the content processing chapter, one of the biggest changes in sharepoint 2013 is the availability of the Content Enrichment Web Service (CEWS). This provides a way to add linguistic processing of any type, such as concept extraction, relationship extraction, geo-tagging, summarization, etc. With FasT search for sharepoint, it was possible to extend the content processing pipeline through a sandboxed application, but this was both slow and limited in the information it could access. sharepoint 2013 introduces a much more open api which makes it possible to add specialized linguistics at lower levels as well as sophisticated text analytics.

ceWs is the key extension point for content processing — in fact, the only extension point outside of changing the content or modifying it in a custom connector. There is no content api in sharepoint 2013 for updating metadata into search index independent of a crawl.

ceWs calls an external web service using soap via a proxy, as shown below.

Big Changes in Pipeline ExtensibilityceWs replaces FasT search for sharepoint’s pipeline extensibility stage, which had a number of shortcomings. With FasT search for sharepoint, an executable was run within a sandbox near the end of the content processing pipeline — this was a major performance bottleneck and deployment headache. some crawled properties were available, but derived properties were not. no properties could be modified — you could return things in new properties but only to a limited extent. and the executable was called for all content, so filtering logic was needed outside the pipeline, and the performance penalty of calling it was incurred for all content.

With sharepoint 2013, things have changed dramatically. using a web service callout opens up many options and removes some of the difficulties in writing pipeline extension stages. The processing pipeline passes designated managed properties (including document text) to the remote service. There are hidden and read-only properties, but some managed properties (like Title) can be modified.

The mechanism for ceWs is fairly simple:

• The content processing component sends a SOAP RPC call to a configurable endpoint over hTTp.

• The payload contains an array of property objects.

• The web service performs some custom logic on the array of property objects, and returns an array of modified or new property objects.



• The web service must send a response to the web service client within a given timeout.

• No specific authentication or encryption mechanisms are supported as part of the contract. You can, however, apply your own security on the transport mechanism.

a trigger condition is registered in the ContentEnrichmentConfiguration object which allows control of when the content flow calls out to an external web service. A set of powershell commandlets are provided to control the configuration, and there are robust error handling mechanisms built in.

What to Look Out ForFor those familiar with pipeline extensibility, you will find CEWS easy to use. However, there are a variety of limitations and gotchas to look out for. one key difference in ceWs from FasT search for sharepoint is where it is called. Specifically, you get content and managed properties after document parsing but before word breaking, as shown below.

The advantage of this is that you can provide custom linguistics even at a fairly low level, and influence other aspects of the pipeline. The control afforded by this is wonderful and will be exciting to those wanting to address specific linguistic processing at a low level.

The disadvantage is that you can’t leverage the work done in the pipeline when you are doing external processing, as shown below. This not only means extra work as a developer, but introduces the potential that linguistic processing could get out of sync.

The extensibility call outs are invoked synchronously, in line with the processing flow, so long-running enrichment tasks or batch-oriented processing tasks will require enrichment data flow management independent of and outside sharepoint 2013. not all managed properties (or any crawled properties) are visible to the ceWs and less state (potentially useful for supplemental linguistics processing) is exposed than in Fs4sp.

Finally, the ceWs is visible as a single logical endpoint to the potentially many content processing flow instances in sharepoint 2013. There is only one ContentEnrichmentConfiguration object active, and only one trigger, etc. This means



that throughput management, and support for multiple enrichment stages (more than once instance of taxonomy classifiers or custom entity extractors) need to be managed externally, which will pose some interoperability challenges if you are interested in doing multiple types of content enrichment.

* Note: The CEWS call out is not part of O365 and is only available for the Enterprise Edition of SharePoint 2013.

ceWs is a new mechanism in sharepoint 2013. it has many nice aspects — it is a more standard, higher performance mechanism than that available in the past. it also provides the ability to modify some managed properties, making it possible to address use cases that were nearly impossible with FasT search for sharepoint.

ceWs also has limitations, and using it will require special attention by developers. But all mechanisms have limitations. overall, Microsoft has provided a strong and essential extensibility mechanism that lets you do magic things with content processing and linguistics.

search-Based applications with sharepoint 2013 sharepoint 2013 is designed to support applications. Many parts of sharepoint operate out-of-the-box as applications (formerly called ‘workloads’, although this term doesn’t seem to be used much with the new release). in addition to a new development model (covered in the previous chapter), a new app model and app marketplace, and an emphasis on running applications in the cloud, there are many capabilities to leverage in building new applications. Mobile applications, which played poorly with sharepoint 2010, are fully supported now. sharepoint is, more than ever, an application platform with a set of prebuilt applications and apps included.

search-based applications are applications like any other, except that they take advantage of search technology in addition to other elements of SharePoint to create flexible and powerful user experiences. Because search is essential for dealing with diverse content, especially unstructured content, applications using search are found everywhere, and their importance is growing rapidly — in step with the explosion of content volume. Yet search is generally not well understood or fully used by developers. even though search is simple on the outside, it is complicated on the inside. Many people aren’t comfortable with the notion of a search-driven application until they see one.

deeper divesMSDN Section on Content Enrichment Web Service (CEWS) »





A Platform for Search-based Applicationssharepoint 2013 is explicitly meant to support search-based applications. As the figure below shows, search is built as an extensible platform. There are both general-purpose search and some pre-built search-based applications included — and search is also used pervasively throughout sharepoint, especially in WcM and Mysites. Most importantly there are great facilities for deploying apps and applications using search, with tooling and hooks specifically for application developers. so partners and customers can create search-Based applications and deploy them on the same platform.

Out-of-the-box Applications There are three ‘general-purpose’ search applications included out-of-the-box with sharepoint 2013. intranet search -typically used for all employees to find content throughout the enterprise, benefits from personalized search results based on search history and rich contextual previews. people search (which includes the advances from sharepoint 2010 such as phonetic name search) is integrated

with lync — which provides presence information and makes it easy to connect with people directly from search results. site search (aimed at making public web sites easily navigable) is a big step up with this release as well. There are also search facilities built into each site — for example, every document library now has a search box at the top that enables users to search across metadata and the full text of its documents, and the result list is presented as a standard sharepoint view rather than as a results page.

a video search sBa is provided out-of-the-box, including a pre-built presentation format that makes it easy to recognize the video content you’re looking for. There are significant enhancements in video support for sharepoint 2013 generally, including a built-in hTMlhTMl 5 video player. The use of video including enterprise podcasts will be on the rise, so video search is now an important facility.

Search Driven Web Content Management Web content management makes extensive use of search in sharepoint 2013. search makes it possible to create compelling user experiences, and drives several key features.

content by search — The new content by search web part displays indexed content, letting you show content dynamically across multiple site collections. users don’t know this is search powered, it just looks like well-presented content, as illustrated in the screenshot below. For a case like online catalogs, this is an essential mechanism and one that works very well.



There’s an hTMl-based presentation template model that makes it easy to fine tune the look and feel, and built-in web part editors to set up the query driving the content presentation, as shown below. This doesn’t require writing any code and is well within the reach of a business analyst. You see immediate previews of what the results will look like.

Metadata Navigation — as described in the chapter on refinement and faceted navigation, facets are available for users to drill into content. In addition to refiners (which are driven from the values in the content),

metadata navigation defined from values in the term store is available.

Page hierarchies, URLs, and Topic Pages — Pages and page hierarchies are easily defined from the term store. You can also generate topic pages, which makes seo straightforward. The figure below illustrates how this works; sharepoint now generates ‘friendly urls’ which makes this process work like any ‘normal’ site.

Recommendations — a new recommendation facility is included which can surface suggestions based either on popularity or on correlations between items (see chapter on analytics)

There are other exciting things about WcM with sharepoint 2013. standard web design tools and workflows are supported; there are great facilities for content variations including a built-in language translation service; you can publish easily across sites, and video and images are easily embedded and beautifully rendered across multiple devices and resolutions. The urls generated are clean, and search-engine optimization is directly supported. You can use catalog-enabled sites for scenarios such as a content repository, knowledge base, or product catalog. But the heart of WcM in this release is search, which makes dynamic page generation and remarkable site experiences possible.



MySites — Driven by SearchThe social features in sharepoint, including Mysites are dramatically enhanced, building on the capabilities introduced in sharepoint 2010. sharepoint 2013 adds new features that improve and facilitate the enterprise social activities within the organization: you can follow people as well as content, share personal documents easily and keep track of access, keep up-to-date with activities of interest. under the hood, there are two lists for providing social features: the Microfeed list and the social list.

search drives several key social features in sharepoint 2013, even ones where it’s not apparent that search is used under the hood. clicking on a hash tag in a post or discussion shows a list of all conversations about that topic enterprise-wide. in Mysites, users can access a list of all sharepoint tasks assigned to them, regardless of which sites the assignments are stored in. They can also see the documents they are following, as illustrated below. another example is in “My docs: shared with me”, which shows you all the documents shared with you from everyone’s My documents. it looks like a form view but, in reality, it uses search underneath to aggregate content from all Mysites across site collections. Behind the curtain, there’s a query against a “ShareWith” field for your name, which also filters out docs shared with everyone. all security trimmed, naturally.

e-Discovery — Driven by searchsharepoint 2013 has gone one giant step further toward fielding a full e-Discovery

application. There is now unified discovery across exchange, sharepoint and lync, as shown below. exchange now has the same search infrastructure as sharepoint, which makes unifying the search much easier (lync archives via exchange). The discovery center in SharePoint uses this to provide a unified console, with in-place holds that don’t impact the end user’s ongoing work. There’s more to e-discovery than search, of course — preservation, holds, policy management, and export. But search is the cornerstone and is what makes it possible to recall all the information needed to react to legal actions, without getting irrelevant information that you have to sift through.

The e-discovery functionality in sharepoint server 2013 provides is a big step up from



SharePoint 2010, and is probably the first time you could consider this to be a full applications. There are several parts to e-discovery:

• a site collection where you perform e-discovery queries across multiple sharepoint farms and exchange servers and preserve the items that are discovered.

• in-place preservation of exchange mailboxes and sharepoint sites — including sharepoint list items and sharepoint pages — while still allowing users to work with site content.

• support for searching and exporting content from file shares.

• The ability to export discovered content from exchange server 2013 and sharepoint server 2013.

The eDiscovery Center site template creates a portal for discovery cases and lets you conduct searches, place content on hold, and export content. For each case, you create a new collaboration site that uses the eDiscovery Case site template. You can export the results of an ediscovery search for later import into a review tool.

sharepoint 2013 provides in-place holds — content that is put on hold is preserved, but users can still change it. The state of the content at the time of preservation is recorded. if a user changes the content or even deletes it, the original, preserved version is still available.

To implement ediscovery across an enterprise, you configure SharePoint 2013

Search to crawl all file shares and websites that contain discoverable content, and configure the central search service application to include results from exchange server 2013. any content from sharepoint 2013, exchange 2013, or a file share or website that is indexed by search or by exchange server 2013 can be discovered from the ediscovery center.

Customize, Extend, and Create New Search-Based Applicationssearch-based applications are found over a very wide range of roles, industries, and levels of sophistication. There are common patterns to these applications; the table below shows just a few of these application patterns. sharepoint 2013 provides models that span a spectrum from simple configuration, through extension of capabilities, to creation of new sophisticated search based applications. The new mechanisms in sharepoint 2013 for customizing user experience (query rules, result blocks, and result sources) and the ability to theme sharepoint easily provide a lot of power



for customizing search experiences without any code at all. Many areas can be extended — connectors, content processing, relevance, query processing, and ui — with moderate effort and standard tools. Fully custom code is supported as well.

We find that the use of modular building blocks speeds the construction of search based apps dramatically. since these applications follow common patterns, a relatively small number of sophisticated modules can cover a large number of applications. if you undertake a sophisticated search-based application, consider what’s available on the market as well as what you might build yourself — since pre-built building blocks can save substantial time and reduce risk.

Acceleration of Search With SharePoint 2013Microsoft has taken a big step forward in helping people do more with search:

• it is far easier to own and use high-end search capabilities

• search is used pervasively

• some search-based applications are built-in

• it is easier to create and operate tailored search-based applications

We expect many more interesting applications to emerge as a result.

deeper divesBook chapter on developing with search from Wrox “Professional SharePoint 2010 Development” »

Overview of eDiscovery and In-Place Holds (SharePoint 2013) »

Blog on using the Content by Search Web Part »

BA Insight »

Search as a Development Platform »

TotalView Search-Based Applications »





http://www.sharepointnutsandbolts.com/2012/10/

http://www.bainsight.com/technology/Pages/search-integration.aspx

http://www.bainsight.com/technology/Pages/search-app-development.aspx

http://www.bainsight.com/enterprise-search-applications/Pages/default.aspx

77

conclusion

77


This e-book has covered a lot of ground, since sharepoint 2013 has so many underlying changes, new capabilities, and new features. We’ve tried to cover everything in concise, readable chapters, across five major sections: User Experience; Working with Queries; Working with content; architecture, deployment, and operations; and development and applications.

This new platform has a lot to love; it is:

• clean, fast, and easy to use

• straightforward to install, administer, and scale

• provides very powerful high-end search features

• Makes creating search-based applications simpler than ever

Microsoft has done a remarkable job making this high-end technology accessible and easy for the mainstream. however, it is not a perfect platform, and there are still challenges with search. search is, after all, a journey.

Ba insight is entirely focused on the road that lies ahead for search and sharepoint 2013, and we stand ready to help you on your journey. as you learn more about sharepoint 2013 and search, here are some things to consider and some steps we’d suggest:

ConCluSion

user experience

Working with Queries

& results

Working with

Content

architecture, Deployment &

operations



ConCluSion

thinGS to ConSiDer

sharepoint 2013 includes a very powerful new search engine.

There are new mechanisms in sharepoint 2013 (result sources, query rules, and result types) that replace familiar ones, and take some getting used to. These are now in the hands of site collection administrators and site administrators, so there is much more control at that level.

crawling and Bcs have evolved further in sharepoint 2013, including a new continuous crawl feature, however connectors are still largely left to partners.

The new search core in sharepoint 2013 is different from either FasT or sharepoint 2010, and you will notice improvements in relevance, performance, and robustness.

Hybrid configurations across on-prem SharePoint 2013 and o365 are supported ooB using result sources. cross-version configurations are not supported OOB but there are techniques and partner products for these cases.

Though sharepoint 2013 search is great, there are still limitations and cases where the mechanisms don’t cover what you wish to accomplish.

The term store is now an administrative center for entity extraction, query suggestions, faceted refinement, WCM page hierarchies, and more.

if you are coming from FasT, you will recognize a lot of concepts and powerful features. But you will also notice a number of things ‘missing’.

sharepoint 2013 has a new development model that is lightweight and available to a much wider range of developers. search in sharepoint 2013 is a powerful platform designed to support search-based applications.

SuGGeSteD next StePS

get to know the new release asap — download the bits, read about it, and confer with folks that know it.

Try to develop a champion amongst your site administrators, who learns the new tools. set up a playpen system where people can get used to the new mechanisms.

Take stock of your current and future content sources and

think about extending search to more content. look at learning how to make simple connectors yourself, and at Microsoft connector partners for more complex systems.

consider how quickly you can migrate to the new platform. Factor in techniques which allow you to upgrade a step at a time, such as search-first migration and cross-version federation.

consider adopting o365 quickly, in ways that you don’t need to do it all at once. Talk to Microsoft partners about federation, cross-version configurations, and migration.

look to the Microsoft partner ecosystem for training,

components, and innovative solutions.

get familiar with the term store. Find out where there are key lists in your organization (product names, project names, industries, etc) — you will be able to import these into the term store and use them for entity extraction.

Focus on the problem, not the specific mechanism — there’s a way to get it solved with this platform. Turn to Microsoft partners for products that round out all the possibilities.

consider applying Javascript developers to building sharepoint apps. look around your organization for opportunities to apply search-based applications.

BA Insight is a leader in agile information integration, enabling business to drive innovation by leveraging

all knowledge and data across the enterprise. Offering new generation, cost effective alternative to

expensive systems integration, the company‘s award-winning technology provides a scalable foundation

for liberating enterprise data, both structured and unstructured. Microsoft’s go-to partner for advanced

search technologies, BA Insight enables customers to leverage their investments in SharePoint, FAST and

other enterprise systems, and extend them with an overlay of easy-to-assemble, highly targeted business

applications. Since 2004, more than three million users around the world have relied on BA Insight for

low-cost, on-demand access to the information they need.

To learn more about BA Insight, visit www.bainsight.com.

Ba insight is social!read our blog: www.DoMoreWithSearch.com Follow us on Twitter : @bainsight linkedin group: Microsoft Enterprise Search Or find us on Facebook

http://www.bainsight.com

http://www.domorewithsearch.com

https://twitter.com/bainsight

http://www.linkedin.com/company/ba-insight

https://www.facebook.com/pages/BA-Insight/107664795939626

Date post:	31-Dec-2015
Category:	Documents
Upload:	hye-rim-kim
View:	197 times
Download:	8 times

BA Insight SharePoint 2013 Enterprise Search Guide

Documents