Monday, 7 August 2017

SharePoint 2013 Search - Remove an index component

Make sure that the current active topology is healthy and that the search component that you are about to remove is Active.

Note: If you have more than one active index replica for an index partition, you can remove an index replica by performing the procedure Remove a search component in the article Manage search components in SharePoint Server 2013.

You cannot remove the last index replica of an index partition using this procedure. If you have to remove all index replicas from the search topology, you must remove and re-create the Search service application and create a completely new search topology that has the reduced number of index partitions.

Step #1: Get the Search Service Instance and Start
$ssi4 = Get-SPEnterpriseSearchServiceInstance -Identity "CA1VMSPUATAPP04"

Step #2: Wait for the Search Service to Come Online
Start-SPEnterpriseSearchServiceInstance -Identity $ssi4

Step #3: Clone the Active Search Topology
$ssa = Get-SPEnterpriseSearchServiceApplication
$active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa -Active
$clone = New-SPEnterpriseSearchTopology -SearchApplication $ssa -Clone SearchTopology $active

#To see which components you have, use the following command.
Get-SPEnterpriseSearchStatus -SearchApplication $ssa

Step #4: Get the ID of the Index Component to Remove
$indexComponentID = (Get-SPEnterpriseSearchComponent -SearchTopology $clone -Identity IndexComponent3).componentID

Step #5: Remove the original Index Component
Remove-SPEnterpriseSearchComponent -Identity $indexComponentID.GUID -SearchTopology $clone -confirm:$false

#Repeat step #4 and #5 for any additional servers you want to remove the index or any other component(like Admin,Query,Contentprocessing..etc) for.

Step #6: Activate Search Topology Again
Set-SPEnterpriseSearchTopology -Identity $clone

Step #7: Monitor the Distribution of the Index 
Get-SPEnterpriseSearchStatus -SearchApplication $ssa

SharePoint 2013 Search - Add Index partition and Index replica

For each 10 million items in the search index, you have to add a new index partition. When you add a new index partition, the search index has to be re-partitioned. Depending on the size of the search index, this re-partitioning can take several hours to complete.

You add an index replica to the search topology to achieve fault tolerance for an existing index partition. You place the index replicas on separate failure domains on separate servers. When you add an index replica, you add a new index component to the search topology and associate it with the index partition that you want to make a replica of.

Step #1:Get Search Service Instance and start
$ssi3 = Get-SPEnterpriseSearchServiceInstance -Identity "CA1VMSPUATAPP03"
$ssi4 = Get-SPEnterpriseSearchServiceInstance -Identity "CA1VMSPUATAPP04"
Start-SPEnterpriseSearchServiceInstance -Identity $ssi3
Start-SPEnterpriseSearchServiceInstance -Identity $ssi4

Step #2:Wait for Search Service Instance to come online
Get-SPEnterpriseSearchServiceInstance -Identity $ssi

Step #3:clone the Active search topology
$ssa = Get-SPEnterpriseSearchServiceApplication
$active = Get-SPEnterpriseSearchTopology -SearchApplication $ssa -Active
$clone = New-SPEnterpriseSearchTopology -SearchApplication $ssa -Clone SearchTopology $active

Step #4: Create new index component
For example: If you have an existing index partition 0 with index replicas on App01 and App02 B, and you want to add a new index partition with index replicas on App03 and App04:

New-SPEnterpriseSearchIndexComponent SearchTopology $clone -SearchServiceInstance $ssi3 -IndexPartition 1 -RootDirectory "D:\SPIndex"
New-SPEnterpriseSearchIndexComponent SearchTopology $clone -SearchServiceInstance $ssi4 -IndexPartition 1 -RootDirectory "D:\SPIndex"

Step #5:Activate the cloned search topology
Set-SPEnterpriseSearchTopology -Identity $clone

Step #6:Verify that your new topology is active and that the index component representing the new index replica is added.
Get-SPEnterpriseSearchTopology -Active -SearchApplication $ssa

Step #7:Monitor the Distribution of the Index
Get-SPEnterpriseSearchStatus -SearchApplication $ssa -Text

Repeat this command until all search components, including the new index component, output the state Active. For a large search index, this could take several hours.

Thursday, 3 August 2017

Create and Configure SharePoint 2013 Search service application

When you first create the Search Service Application, the search topology by default has all search components assigned to the server which is running the Central Administration. In multi servers farm scenario you need to change this topology using PowerShell.

Search architecture that you choose depends on how much content has to be searchable:
You can refer the below diagram to have better understanding for the topology that we are about to configure for 0-20 million items:
In this environment, we have five servers: two Front-Ends, two Application servers and one Index server.

Server Names:
Below are the steps that we are going to perform:
  • Create Search Service Application and Proxy
  • Add a New Index Component
  • Add a New Crawl Component
  • Add a New Content Processing Component
  • Add a New Analytics Component
  • Add a New Admin Component
We have assigned the following rules to each:
  • App01, App02 server are responsible for Query Processing and Index partitions.
  • App03, App04 has the Admin, Crawler, Content Processing and Analytics Processing roles.
#Load the SharePoint PowerShell snap-in or run these commands at the SharePoint Management Shell
Add-PSSnapin Microsoft.SharePoint.PowerShell -ErrorAction SilentlyContinue

#How to create a search service application.
$SearchAppPoolName = "Search_Service_AppPool"
#You’ll need to change the default content crawl account. By default Search uses the Farm account, which is a really bad idea. 
$SearchAppPoolAccountName = "corp\ca1_uat_searchpool"
$SearchServiceName = "Search Service Application"
$SearchServiceProxyName = "Search Service application proxy"
$DatabaseName = "SearchService_DB"

#Create a Search Service Application Pool
$spAppPool = New-SPServiceApplicationPool -Name $SearchAppPoolName -Account $SearchAppPoolAccountName -Verbose

#Start Search Service Instance on all Servers
Start-SPEnterpriseSearchServiceInstance $ssiAPP01 -ErrorAction SilentlyContinue
Start-SPEnterpriseSearchServiceInstance $ssiAPP02 -ErrorAction SilentlyContinue
Start-SPEnterpriseSearchServiceInstance $ssiAPP03 -ErrorAction SilentlyContinue
Start-SPEnterpriseSearchServiceInstance $ssiAPP04 -ErrorAction SilentlyContinue

#Create Search Service Application
$ServiceApplication = New-SPEnterpriseSearchServiceApplication -Partitioned -Name $SearchServiceName -ApplicationPool $spAppPool.Name -DatabaseName $DatabaseName 

#Create Search Service Proxy
New-SPEnterpriseSearchServiceApplicationProxy -Partitioned -Name $SearchServiceProxyName -SearchApplication $ServiceApplication

Search service application has been created. Now, we need to configure different search components as described above and then finalize the search topology.

#Clone the active search topology
The command creates a clone search topology that can be referenced with $clone if you continue to use the same SharePoint 2013 Management Shell to add or remove search components and to activate the search topology.

$clone = $ServiceApplication.ActiveTopology.Clone()

#Get Search Service Instance
$ssiapp01 = Get-SPEnterpriseSearchServiceInstance -Identity "CA1VMSPUATAPP01"
$ssiapp02 = Get-SPEnterpriseSearchServiceInstance -Identity "CA1VMSPUATAPP02"
$ssiapp03 = Get-SPEnterpriseSearchServiceInstance -Identity "CA1VMSPUATAPP03"
$ssiapp04 = Get-SPEnterpriseSearchServiceInstance -Identity "CA1VMSPUATAPP04"

#We need two admin components
New-SPEnterpriseSearchAdminComponent SearchTopology $clone -SearchServiceInstance $ssiapp03
New-SPEnterpriseSearchAdminComponent SearchTopology $clone -SearchServiceInstance $ssiapp04
#We need two content processing components for High Availability
New-SPEnterpriseSearchContentProcessingComponent SearchTopology $clone -SearchServiceInstance $ssiApp03
New-SPEnterpriseSearchContentProcessingComponent SearchTopology $clone -SearchServiceInstance $ssiApp04
#We need two analytics processing components for High Availability
New-SPEnterpriseSearchAnalyticsProcessingComponent SearchTopology $clone -SearchServiceInstance $ssiApp03
New-SPEnterpriseSearchAnalyticsProcessingComponent SearchTopology $clone -SearchServiceInstance $ssiApp04
#We need two crawl components for High Availability
New-SPEnterpriseSearchCrawlComponent SearchTopology $clone -SearchServiceInstance $ssiApp03
New-SPEnterpriseSearchCrawlComponent SearchTopology $clone -SearchServiceInstance $ssiApp04
#We need two query processing components for High Availability
New-SPEnterpriseSearchQueryProcessingComponent SearchTopology $clone -SearchServiceInstance $ssiApp01
New-SPEnterpriseSearchQueryProcessingComponent SearchTopology $clone -SearchServiceInstance $ssiApp02

As per the plan we have created all the search components.

Assuming that you do not have have index partition, and that partition has no replicas.  So let’s look first at adding a new search partition.

Note: To scale out the search index, you add a new index partition for each 10 million items.

To achieve fault tolerance for the search index, you add an index replica of an existing index partition to the search topology. Each index replica contains the same information.

#We need two index partitions 0,1 (0-20 million items)
#If you have 0-10 million items then 
New-SPEnterpriseSearchIndexComponent SearchTopology $clone -SearchServiceInstance $ssiApp01 -IndexPartition 0 -RootDirectory "E:\SPIndex"
New-SPEnterpriseSearchIndexComponent SearchTopology $clone -SearchServiceInstance $ssiApp02 -IndexPartition 1 -RootDirectory "E:\SPIndex"

#Activate the Topology

# Verify the Topology 
$ssa = Get-SPEnterpriseSearchServiceApplication
Get-SPEnterpriseSearchTopology -Active -SearchApplication $ssa

# Verify the diagnostic information for all search components in the active topology of a Search Service Application.
Get-SPEnterpriseSearchStatus -Text -SearchApplication $ssa

That's it. Our search service application configuration completed and you need to run the full crawl after adding the content sources.


Overview of search in SharePoint Server 2013

In this post we are going to discuss about SharePoint search architecture, SharePoint search components, SharePoint search databases, and the SharePoint search topology.

The search architecture contains:
  1. Search components
  2. Search databases
Overview of search components and search databases:
There are some major changes in SharePoint server 2013 when compared to SharePoint server 2010. Below are the 6 different search service components in SharePoint 2013.
  • Crawl Component
  • Content Processing Component
  • Indexing Component
  • Query Processing Component
  • Analytics Processing Component
  • Search Administration Component

Crawl Component:
  1. This component takes care of crawling the content sources and collects the crawled properties and metadata and then passes crawled items to the content processing component.
  2. The crawl component uses one or more crawl databases to temporarily store information about crawled items and to track crawl history.
Crawl database
  1. The crawl database contains detailed tracking and historical information about crawled items.
  2. This database holds information such as the last crawl time, the last crawl ID and the type of update during the last crawl.
  3. Manages crawl operations.
  4. Each crawl database can have one or more crawl components associated with it.

Content processing component:
  1. This component receives the information (crawled items) from the crawl component and then processes and sends it to the indexing component. It also interacts with the analytics processing component and is responsible for mapping crawled properties to the managed properties.
  2. The content processing component writes information about links and URLs to the link database.

Indexing Component:
  1. This component receives the processed items from the content processing component and writes those items to the search index. Each Index files are stored on a disk in the server that hosts the index component.
  2. It also receives the queries from Query processing component and sends back the results.
    Index partition:
  • Dividing the search index into separate portions, called index partition.
  • Each index partition holds one or more index replicas(mirror/copy) that contain the same information.
 Index partition vs index replicas:
An index replica is a copy of the index. This is commonly used for availability. For example, create a replica of the index on more than one server so that your queries can be served by more than one server.
A partition is a chunk of the index. Create a new partition for scale, the recommendation is 10M items. So if you are indexing a lot of content you may create multiple partitions.

Query Processing Component:
This component handles incoming query requests and sends them to the indexing component for results. It also takes care of query optimization.
  1. Query Processing Component analyzes and processes search queries and results.
  2. When the query processing component receives a query from the search front-end, it performs linguistic processing first (like word breaking and stemming), then analyzes and further processes the query to optimize precision, recall and relevance. In the end, the processed query will be submitted to the index component.
  3. The index component returns a result set based on the processed query back to the query processing component.

Analytics Processing Component:
  1. The analytics processing component performs two types of analyses: search analytics and usage analytics.
  2. This component uses information from these analyses to improve search relevance, create search reports, and generate recommendations and deep links.
  3. The results from the analyses are added to the items in the search index. In addition, results from usage analytics are stored in the analytics reporting database.

Search analytics VS usage analytics
Search analytics is about extracting information such as -- links, the number of times an item is clicked, anchor text, data related to people, and metadata – from the link database. This information is important to relevance.
Usage analytics is about analyzing usage log information received from the front-end via the event store. Usage analytics generates usage and statistics reports.

About the link database
The link database stores information extracted by the content processing component. In addition, it stores information about search clicks; the number of times people click on a search result from the search result page. This information is stored unprocessed, to be analyzed by the analytics processing component.

About the Analytics reporting database
The analytics reporting database stores the results of usage analytics. In addition, the analytics reporting database also stores statistics information from the analyses. SharePoint uses this information to create Excel reports that show different statistics.
About the event store:
  1. The event store holds usage events that are captured on the front-end, such as the number of times an item is viewed.
  2. These usage events are stored as log files on the application server that hosts the analytics processing component.

Search administration Component:
  1. The search administration component is responsible for running a number of system processes that are essential to search.
  2. This component manages administrative processes as well as changes to the search topology, such as adding or removing search components and servers.
Search administration database
Stores search configuration data, such as the topology, crawl rules, query rules, and the mappings between crawled and managed properties. Only one search administration database per Search service application.

Friday, 19 August 2016

Install .Net Framework 3.5 on Windows Server 2012 R2

Windows Server 2012 R2 includes the .NET Framework 4.5.1 as an OS component, and it is installed by default. It also includes the .NET Framework 3.5 SP1 as an OS component that is not installed by default.

If you run Setup(Microsoft SQL Server 2014 R2) and you do not have .NET 3.5 SP1, SQL Server Setup requires you to download and install .NET 3.5 SP1 before you continue with the SQL Server installation.

If you are trying to install .NET Frame 3.5 from the Server Manager GUI, you will see the below alert when installing the feature.

"Do you want to specify an alternate source path? One or more installation selections are missing source files on the destinatino server.…”

If your computer does not have access to windows update, then specify an alternate source path by clicking on "Specify an alternate source path" and enter the path like "C:\sources\sxs"
After you have specified the alternate source, close the warning (by clicking the x), then click Install.

Thursday, 18 August 2016

New and improved features in SharePoint Server 2016

The following information provides a summary of the new features in SharePoint Server 2016 release.


In 2013, server type(application server/search server/distributed cahce server) will be determined by what services you ran in your server.

In 2016, new feaure "MinRoles" get added. MinRoles will apply a set of services to a server based on the role you select it during installation.

Zero Downtime Patching:

The biggest improvement in patching for SharePoint 2016 is In-place, online, installation of patches. You can install the patches in the middle of the day without affecting users accessing the farm at all.

Microsoft removed the downtime which is required to update the SharePoint servers.

In the past with SharePoint, for general updates you got about 37 individual files to process for each patch, plus 10 for each language pack installed. In SharePoint 2016, you get 2 files to process, plus 1 file per language pack.

Compliance features:

New compliance features for SharePoint Server 2016 include the document deletion and in-place hold policies.
The document deletion policy allows you to delete documents in users' OneDrive for Business sites after specific periods of time.

Integrated Project Server:

Project Server 2016 is installed with SharePoint Server 2016 Enterprise, though is licensed separately.

Max File Size for Uploads:

The default maximum file size in SharePoint 2013 is 250 MB, which can be increased up to 2GB. SharePoint 2016 mas file upload size is 10GB.
Note: End users will likely get a time out while uploading large files.

File Name Support for Special Characters:

SharePoint has blocked file names that included the &, ~, {, and } characters, file names that contained a GUID, file names with leading dots, and file names longer than 128 characters were blocked in previous versions. SharePoint 2016 will allow these to be used and these restrictions are 
removed in SharePoint Server 2016.

Content Database Size:

In SharePoint 2013 maximum content database size is 200GB. Microsoft posted that maximum content database size is expected to be 1TB. And each content database will be able to hold approximately 100,000 site collections.

Site Collections Per Content Database:

The recommended amount of site collections for SharePoint 2013 is 2,000 with a maximum of 5,000 collections. In SharePoint 2016 you will be able to maintain 100,000 site collections per content database.

Fast Site Collection/Site Creation:

In SharePoint 2013 site creatino takes around 40seconds. Using a template, in SharePiont 2016 users are able to create the site collection very quickly(1second). This will require a configuration with powershell.
SPSiteMaster Windows PowerShell cmdlets help to create sites and site collections quickly.

Override 5,000 View Threshold:

SharePoint has a List View Threshold (LVT) of 5,000 items. Many end users don't know that they had to index their columns to manage the threshold.
In SharePoint 2010 or 2013 indexing the columns is available but end user had to manually index their columns to break the view throshold.
In SharePoint 2016 Microsoft has automated the creation of indexed columns.

Hybrid App Launcher:

Hybrid app launcher helps users to have a more seamless experience when navigating between SharePoint Server and Office 365.

The extensible hybrid app launcher is designed to help you get to your Office 365 apps and services from SharePoint Server. Once you enable this feature, you’ll see the Office 365 Delve and Video apps, along with your custom Office 365 tiles, appear in your SharePoint Server app launcher.
In SharePoint Server 2016, this feature is enabled as part of Hybrid Sites Features.

SharePoint Search Service Application:

With SharePoint 2013, 100 million indexed items per search service application and 10 million per index partition. SharePoint 2016 search supports indexing up to 500 million items per Search Service application.

Open Document Format (ODF):

SharePoint 2016 added new support for Open Document Format (ODF) files to use in documen library templates. It enables users to create new files in a document library and save as ODF files so that users can edit the new file with a program of their choise.

Durable Links:

Durable Link is not the same as Document ID. A Durable Link is a URL of Hyperlink to a Document stored in SharePoint 2016 Document Library.
Durable Links proivde a resource ID to each document. This resource ID will be stored in the content database.

If Durable Links work in you Farm, you will see that links to Documents look like this:

When the user opens the document using Durable Links, SharePoint 2016 looks up the file by the Resource ID and opens it in Office Online Server Preview. If you move or rename this document, the link you see here will continue to work and redirect you to the same document.

Search for sensitive content:

Search for sensitive content across SharePoint Server 2016, SharePoint Online, and OneDrive for Business.

SharePonit 2016 controls 51 built-in sensitive information types (like credit cards, passport numbers, Social Security numbers, and more).

The sensitive data is exposed through an eDiscovery site(You have to choose eDiscovery template to create the site in Central Admin). And then use DLP Queries from the eDiscovery site collection to discover sensitive content.

SMTP Connection Encryption:

SharePoint 2016 brings a lot of improvement in the field of out-going & Incoming emails. We can send emails using encryption and we can configure the SMTP to non-default ports (default Port is 25).

Sites page pinning:

This is a new feature with SharePoint 2016 which helps users see and follow sites. Users can pin sites that are seen on the sites page. A pinned site shows at the top of the list of followed sites.

MinRole overview in SharePoint 2016

SharePoint 2016 is taking a very different approach on roles and services. In SharePoint 2013, whether your server was a "search server" or a "distributed cache server" was all determined by what services you ran on it.

In 2016, that get a new wrinkle added to it through the use of MinRoles. MinRoles apply a set of services to a server based on the role you assign it during installation.

SharePoint automatically configures the services on each server based on the server's role. Any services running on those servers that are not in that predefined role, or stopped services that should be running, will cause health monitors to report that the server is out of compliance.

Note: You cannot choose to have the servers be multiple roles. If you want a server to perform multiple role functions, you’ll want to select "Custom" during the installation, which will let you pick and choose the services on the server in the same way as you do in SharePoint 2013.

SharePoint Server 2016 has six Server roles to choose in server installations:

Front-end: Service Application, services,and components that serve user requests. These servers are optimized for fast performance.

Application: Service Application, services,and components that serve back-end requests. Like search crawl, timerjobs. These servers are optimized for high throughput.

Distributed Cache: Service Application, services,and components that are required for a distributed cache. Optionally, you configure these server to load balance farm traffic.

Search: Service Application, services,and components that are required for search.

Custom: Custom service applications, services, and components that are outside of MinRole topology. The farm administrator has full control over which service instances can run on servers.

Single server farm: 
Service applications, services, and components required for a single-machine farm. A Single-Server Farm is meant for development, testing, and very limited production use. A SharePoint farm with the Single-Server Farm role cannot have more than one SharePoint server in the farm.

The Standalone Install mode is no longer available in SharePoint Server 2016 IT Preview. The Single-Server Farm role replaces the Standalone Install mode available in previous SharePoint Server releases. With 2016 you don't get SQL  Express included in the installation. The SharePoint administrator must separately install and prepare Microsoft SQL Server for SharePoint.

MinRole Topologies:      

Server role
* Content Farm ?
*Shared Services Farm ?
*Search Farm ?
Front-end web
Distributed cache
Yes, if hosting Search
Yes, if hosting Search