+ All Categories
Home > Technology > Kerberos: The Four Letter Word

Kerberos: The Four Letter Word

Date post: 12-May-2015
Category:
Upload: kenneth-maglio
View: 1,018 times
Download: 0 times
Share this document with a friend
Popular Tags:
61
Kerberos Kerberos The Four Letter Word 1 It’s a real pain in the as
Transcript
Page 1: Kerberos: The Four Letter Word

Kerberos

Kerberos

The Four Letter Word

1

It’s a real pain in the as

Page 2: Kerberos: The Four Letter Word

Kerberos

#GMSQL

Page 3: Kerberos: The Four Letter Word

KerberosBio

Ken MaglioMicrosoft Solution ArchitectWorld Wide Technology, Inc.

@kenmaglio/in/[email protected]

3

Introduct

Page 4: Kerberos: The Four Letter Word

KerberosIntroduction

Today:•Walk through the configuration of Kerberos •Prep for Business Intelligence (BI) solutions •SharePoint 2010 • SSRS Integrated Mode

•SQL Server 2012

No Demos – Sorry! ( like I want to setup more Kerberos environments – rly? )

Benefits

Page 5: Kerberos: The Four Letter Word

KerberosBenefits

Delegation of client credentials •pass that identity to other network services on the client's behalf•NTLM does not allow this delegation – “double-hop”•Claims authentication, like Kerberos authentication, can be used to delegate client credentials but requires the back-end application to be claims-aware

Security •AES encryption, mutual authentication, support for data integrity and data privacy

Potentially better performance•Less traffic to the domain controllers compared with NTLM

Assump

Page 6: Kerberos: The Four Letter Word

KerberosAssumptions

You know how to:

•install SQL Server 2012•work with Windows Server 2008 R2•work with IIS 7•work with SharePoint 2010 (central admin mainly)

Kick T

Page 7: Kerberos: The Four Letter Word

KerberosKick The Tires

Getting started

Environment:Windows Server 2008 R2 – Active Directory – blah blah blah

SharePoint 2010 with Two Web ApplicationsIntranetPortalReportingPortal

SQL Server 2012 RDBM for SharePoint Databases

SQL Server 2012 Analysis Services

ShareP

Page 8: Kerberos: The Four Letter Word

KerberosActive Directory

DNS Records

Register a DNS A Record for the web application – just don’t use CNames

Active

Page 9: Kerberos: The Four Letter Word

KerberosActive Directory

Service Accounts

Create a service accounts for the web applications’ IIS application pool

Active

Page 10: Kerberos: The Four Letter Word

KerberosActive Directory

SPN Configuration

Register Service Principal Names (SPN) for the web applications on the service account created for the web application’s IIS application pool

Identify Service Accounts used for Web Application IIS Application Pool : {Domain Name}\{App Pool Acct}

Register SPN the Service Account:SetSPN -S HTTP/{Server Host Name} {Domain Name}\{App Pool Acct}SetSPN -S HTTP/{Server Host Name}.{FQDN} {Domain Name}\{App Pool Acct}

ExampleSetSPN -S HTTP/IntranetPortal myDom12\sp10_PortalIntranetSetSPN -S HTTP/IntranetPortal.myDom12.local myDom12\sp10_PortalIntranet SetSPN -S HTTP/ReportingPortal myDom12\sp10_PortalReportingSetSPN -S HTTP/ReportingPortal.myDom12.local myDom12\sp10_PortalReporting

Share

Page 11: Kerberos: The Four Letter Word

KerberosSharePoint Configuration

Configure Managed Accounts

SharePoint

Enter in the Name and Password and click OK for both of the Accounts

Page 12: Kerberos: The Four Letter Word

KerberosSharePoint Configuration

Portal Creation

SharePoint

Page 13: Kerberos: The Four Letter Word

KerberosSharePoint Configuration

Portal Creation

SharePoint

Page 14: Kerberos: The Four Letter Word

KerberosSharePoint Configuration

RSS Test Page Setup

RSS Feeds make a good Kerberos test of SharePoint, since SharePoint generally requires authentication to access its information, even when accessing RSS.

SharePoint

Add 2 RSS Web Parts to the new TestRSS pages in the Reporting and the Intranet Portals.

Page 15: Kerberos: The Four Letter Word

KerberosSharePoint Configuration

RSS Test Page Setup

The RSS Feeds can be enabled from most lists or libraries. Under the List/Library Tab a button can be seen for RSS Feed. This will launch a new page containing the RSS Information. Copy the URL for a page on each site to be used in the next step.

Each of the Web parts can be edited to change the name and the RSS properties.

Results:

SharePoint

Page 16: Kerberos: The Four Letter Word

KerberosSharePoint Configuration

Web Application Configuration – Kerberos On

IIS Con

Click on the Web Application to select it and then from the ribbon click Authentication Providers

Click the Default Zone to setup our authentication

Once done click Save and Close the Authentication Provider window.

Repeat the other Web Application

Page 17: Kerberos: The Four Letter Word

KerberosIIS Configuration

IIS Site Authentication

Since SharePoint sits on top of IIS the settings for the IIS Authentication also need to be changes.

IIS Con

Page 18: Kerberos: The Four Letter Word

KerberosIIS Configuration

Kernel-Mode Authentication

18

IIS Conf

In the Right Panel click on Advanced Settings…Verify that in Advanced Settings the Enable Kernel-mode authentication is NOT checkedVerify that Kernel mode authentication is disabled

Kernel mode authentication is not supported in SharePoint Server 2010. By default, all SharePoint Server Web Applications should have Kernel Mode Authentication disabled by default on their corresponding IIS web sites.

Page 19: Kerberos: The Four Letter Word

KerberosIIS Configuration

Providers

19

Verify

Under Providers Add Negotiate from Available Providers and move it to the first of the Enabled Providers.

Page 20: Kerberos: The Four Letter Word

KerberosVerify

Checking RSS with Kerberos

Once Kerberos is in place in AD, SharePoint, and IIS a refresh of the RSS Page will show the results we expect.

Active

One final task is needed to restrict this access. Delegation

Page 21: Kerberos: The Four Letter Word

Kerberos

It may seem redundant to configure delegation from a service to itself, such as the portal service account delegating to the portal service application, but this is required in scenarios where you have multiple servers running the service. This is to address the scenario where one server may need to delegate to another server running the same service; for instance a WFE processing a request with a RSS viewer which uses the local web application as the data source

Active Directory

Delegation

SQL C

To configure delegation you can use the Active Directory Users and Computer snap-in. Right-click each service account and open the properties dialog.

Shortcut?NO!!!Note that when you return to the delegation

dialog you do not actually see all the SPNs selected. To see all SPNs, check the Expanded check box in the lower left hand corner. This restriction will allow SharePoint to only delegate it’s credentials to the other User or Computer.

Perform these steps for each service account in your environment that requires delegation.

Page 22: Kerberos: The Four Letter Word

KerberosSQL CONFIGURATION

Configure DNS

Configure DNS for the SQL Server in your environment.

In this example we have one SQL Server, dcSQL12.myDom12.local, running on port 1433 at IP 10.0.0.4. The SQL Server database engine is running on the default instance.

SQL C

Page 23: Kerberos: The Four Letter Word

KerberosSQL CONFIGURATION

SPN for SQL

For SQL Server to authenticate clients using Kerberos authentication, you have to register a service principal name (SPN) on the service account that is running SQL Server. Service principal names for the SQL Server database engine use the following format for configurations that are using the default instance and not a SQL Server named instance.MSSQLSvc/<FQDN>:port

Default InstanceSetSPN -S MSSQLSVC/{Host Server Name} {Domain Name}\{Sql Svc Acct}SetSPN -S MSSQLSVC/{Host Server Name}.{FQDN} {Domain Name}\{Sql Svc Acct}SetSPN -S MSSQLSVC/{Host Server Name}:1433 {Domain Name}\{Sql Svc Acct}SetSPN -S MSSQLSVC/{Host Server Name}.{FQDN}:1433 {Domain Name}\{Sql Svc Acct}

Named Instance SetSPN -S MSSQLSVC/{Host Server Name}:{Instance Name} {Domain Name}\{Sql Svc Acct}SetSPN -S MSSQLSVC/{Host Server Name}.{FQDN}:{Instance Name} {Domain Name}\{Sql Svc Acct}

In our example, we configured the SQL Server SPN on the SQL Server database engine service account (myDom12\SQL12_Engine) with the following SetSPN command:SetSPN -S MSSQLSVC/dcSQL12 myDom12\SQL12_EngineSetSPN -S MSSQLSVC/dcSQL12.myDom12.local myDom12\SQL12_EngineSetSPN -S MSSQLSVC/dcSQL12:1433 myDom12\SQL12_EngineSetSPN -S MSSQLSVC/dcSQL12.myDom12.local:1433 myDom12\SQL12_Engine

SQL C

Page 24: Kerberos: The Four Letter Word

KerberosSQL CONFIGURATION

SQL Server named instancesIf you use SQL Server named instances instead of the default instance, you have to register SPNs specific to the SQL Server instance and for the SQL Server browser service. See the following articles for more information about configuring Kerberos authentication for names instances:

Registering a Service Principal Namehttp://go.microsoft.com/fwlink/?LinkID=196796

An SPN for the SQL Server Browser service is required when you establish a connection to a named instance of SQL Server 2005 Analysis Services or of SQL Server 2005http://go.microsoft.com/fwlink/?LinkId=196799

Verify

Page 25: Kerberos: The Four Letter Word

KerberosVerify

Verify SQL Server Kerberos configuration

Reboot the computers that are running SharePoint ServerThis action restarts all services and forces them to re-connect and re-authenticate by using Kerberos authentication.

Open SQL Server Management Studio and run the following queries from a server other than the SQL server, since it would not need Kerberos to validate itself on the same server.

SELECT auth_scheme FROM sys.dm_exec_connections WHERE session_id = @@spid ;

Verify

Page 26: Kerberos: The Four Letter Word

KerberosVerify

Verify SQL Server Kerberos configuration

Additionally you can get more information:

SQL C

If Kerberos authentication is configured correctly, you see Kerberos in the auth_scheme column of the query results

Page 27: Kerberos: The Four Letter Word

KerberosSQL CONFIGURATION

Create a test SQL Server DB and test table

To test delegation across the various SharePoint Server service applications covered in the scenarios, you have to configure a test data source for those services to access. In the final step of this scenario, you configure a test database called "KerbTest" and a test table called "Sales" to be used later.

In SQL Server Management Studio, create a new database called "KerbTest". Keep the default settings when creating this database.

CREATE TABLE [dbo].[Sales]([RowID] [int] IDENTITY(1,1) NOT NULL,[Region] [nvarchar](10) NOT NULL,[Year] [nvarchar](40) NOT NULL,[Amount] [money] NOT NULL

) ON [PRIMARY]GO

AnalysisSave the table with the name "Sales".

Populate with data

Page 28: Kerberos: The Four Letter Word

Kerberos

Analysis Services Configuration

Setup Analysis Services

Just like standard RDBM setup, we will need to configure DNS for Analysis services, and of course install Analysis services.

I’ll spare the additional screen shots and walkthroughs – hoping you know how to install Analysis services, and setup DNS to point to your instance.

The first step we’ll need to ensure is done is Configuring Active Directory for the SPNs used by the Analysis Services instance.

Analysis

Page 29: Kerberos: The Four Letter Word

Kerberos

Analysis Services Configuration

SSAS SPNsFor SQL Server Analysis Services to authenticate clients by using Kerberos authentication, you have to register a service principal name (SPN) on the service account that is running SQL Server. The SPN for a default Analysis Services instance uses the following format: MSOLAPSvc.3/{FQDN}

So for a single Analysis Services Data Source the format would beSetSPN -S MSOLAPSvc.3/{Server Host Name} {Domain Name}\{SQL Svc Acct}SetSPN -S MSOLAPSvc.3/{Server Host Name}.{FQDN} {Domain Name}\{SQL Svc Acct }

We will configure the Analysis Services using the default SQL instance so the SPN on the Analysis Services service account (myDom12\SQL12_SSAS) will require the following SetSPN commands:SetSPN -S MSOLAPSvc.3/dcSQL12 myDom12\SQL12_SSASSetSPN -S MSOLAPSvc.3/dcSQL12.myDom12.local myDom12\SQL12_SSAS

To Confirm thisSetSPN-L myDom12\SQL12_SSAS

Analysis

Page 30: Kerberos: The Four Letter Word

Kerberos

Analysis Services Configuration

SSAS Named InstancesIf the data source uses a named instance of Analysis Services, you cannot specify a port after the colon. If you do, it is interpreted as part of the hostname or domain name. Instead, you must use the actual instance name for all functionality to work correctly. MSOLAPSvc.3/{FQDN}:{Instance Name}

When we configure the Analysis Services using the default SQL instance so the SPN on the Analysis Services service account for that Instance (myDom12\ SQL12_SSAS_AnlSvc) will require the following SetSPN commands:SetSPN -S MSOLAPSvc.3/dcSQL12:SSAS myDom12\SQL12_SSAS_AnlSvcSetSPN -S MSOLAPSvc.3/dcSQL12.myDom12.local:SSAS myDom12\ SQL12_SSAS_AnlSvc

Analysis

Page 31: Kerberos: The Four Letter Word

Kerberos

Analysis Services Configuration

Verify SSAS Kerberos configurationOnce the SPN is configured, verify the Kerberos connection to the cluster by using Excel 2010.Open Excel 2010 on the client computer using a domain account that has access to at least one database in the Analysis Services instance and open a data connection to your Analysis Services instance by selecting the Data tab, clicking From Other Sources, and then clicking From Analysis Services. Open Excel and click on the Data Tab

Analysis

From the From Other Source drop-down select From Analysis Services

Page 32: Kerberos: The Four Letter Word

Kerberos

Analysis Services Configuration

Verify SSAS Kerberos configuration

Analysis

In the Data Connection Wizard, type dcSQL12 in the Server name box, then click Next.

Page 33: Kerberos: The Four Letter Word

Kerberos

Analysis Services Configuration

Verify SSAS Kerberos configuration

C2WTS

From the SQL Server, dcSQL12, Check the Windows Security Log to see an entry that indicates the access was made using Kerberos.

Page 34: Kerberos: The Four Letter Word

KerberosC2WTS

Claims to Windows Token Service (C2WTS)

The Claims to Windows Token Service (C2WTS) is a component of the Windows Identity Foundation (WIF) which is responsible for converting user claim tokens to windows tokens.

As a best practice you should run the C2WTS using a dedicated service account and not as Local System (the default configuration). The C2WTS service account requires special local permissions on each server the service runs on so be sure to configure these permissions each time the service is started on a server. Optimally, you should configure the service account’s permissions on the local server before starting the C2WTS, but if done after the fact you can restart the C2WTS from the Windows services management console (services.msc).

C2WTS

Page 35: Kerberos: The Four Letter Word

KerberosC2WTS

DNS

C2WTSPermission for the AccountNext, configure the required local server permissions that the C2WTS

requires. You will need to configure these permissions on each server the C2WTS runs on.

Create a service account in Active Directory to run the service under. In this example we created myDom12\SP10_svcC2WTS.

Page 36: Kerberos: The Four Letter Word

KerberosC2WTS

Local Security Policy for the AccountIn Local Security Policy (secpol.msc) under Local Policies | User Rights Assignment give the service account the following permissions:

C2WTS

Page 37: Kerberos: The Four Letter Word

KerberosC2WTS

Central AdministrationFrom Central Administration click on the link to SecurityUnder Security | Configure Managed Service Accounts click on Configure managed Accounts

Register managed account for C2WTS service account =>Go back to Security | Configure Service Accounts

Change the managed account for the Claims to Windows Token Service to use the newly created C2WTS Managed Account.

C2WTS

Page 38: Kerberos: The Four Letter Word

KerberosC2WTS

Central AdministrationUnder services, select Application Management | Service Applications click on Manage services on server.

Verify that you are on the correct server by making any needed change to the server selection box in the upper right hand corner select the server(s) running excel services

Find the Claims to Windows Token Service start it. If it is already running it will need to be restarted, and the corresponding Windows Service will need to be restarted

C2WTS

Page 39: Kerberos: The Four Letter Word

KerberosC2WTS

Windows Service for C2WTS There is a known issue with the C2WTS where it may not automatically startup successfully on system reboot. A workaround to the issue is to configure a service dependency on the Cryptographic Services service.

Open the Command Prompt window and entersc config "c2wts" depend= CryptSvc

Find the Claims to Windows Token Service in the services console.

C2WTS

Open the properties for the service and click on the Dependencies tab. Make sure Cryptographic Services is listed.

Page 40: Kerberos: The Four Letter Word

KerberosC2WTS

Windows Service for C2WTS

Restart the C2WTS from the services console.

In addition, if you experience issues with the C2WTS after restarting the service it may also be required to reset the IIS application pools that communicate with the C2WTS.

This will complete the transition of the C2WTS from using a local account to a domain account. And once it is using a domain account an SPN can be assigned.

C2WTS

Page 41: Kerberos: The Four Letter Word

KerberosC2WTS

SPN for C2WTS

Add an arbitrary Service Principal Name (SPN) to the service account to expose the delegation options for this account in Active Directory Users and Computers. The SPN can be any format because we do not authenticate to the C2WTS using Kerberos authentication. It is recommended to not use an HTTP SPN to avoid potentially creating duplicate SPNs in your environment.SetSPN -S {Arbitrary Protocol}/{Arbitrary Name} {Domain Name}\{C2WTS Svc Acct}

In our example we registered SP10C2WTS/C2WTSsvc to the myDom12\SP10_svcC2WTS using the following command:SetSPN -S SP10C2WTS/C2WTSsvc myDom12\SP10_svcC2WTS 

SSRS

Page 42: Kerberos: The Four Letter Word

KerberosSSRS

REPORTING SERVICES

Authentication in this scenario begins with the client authenticating with Kerberos authentication at the web front end. SharePoint Server 2010 will convert the Windows authentication token into a claims token using the local Security Token Service (STS). The SQL Reporting service application will accept the claims token and convert it into a windows token (Kerberos) using the local Claims to Windows Token Service (C2WTS) that is a part of Windows Identity Foundation (WIF). The SQL Reporting Services service application will then use the client’s Kerberos ticket to authenticate with the backend data source.

42

SSRS

Page 43: Kerberos: The Four Letter Word

Kerberos

SQL Reporting Services service accountAs a best practice, SQL Reporting Services should run under its own domain identity. To configure the SQL Reporting Service Application, an Active Directory account must be created. In this example, the following accounts were created:

43

Page 44: Kerberos: The Four Letter Word

KerberosSSRS

SPNs

SPN FormatSetSPN -S {Arbitrary Protocol}/{Host Server Name} {Domain Name}\{Service Account}

SQL Reporting Services SPN ConfigurationSetSPN -S spSSRSSvc/ReportingPortal myDom12\sp10_svcSSRS12SetSPN -S spSSRSSvc/ReportingPortal.myDom12.local myDom12\sp10_svcSSRS12

44

SSRS

Page 45: Kerberos: The Four Letter Word

KerberosSSRS

VERITY SPNS

Verification of SPNsVerify the SPN for data source service account exists run the following SetSPN command. Format: SetSPN -L {Domain Name}\{Service Account}

SQL Reporting Service AccountSetSPN -L myDom12\SP10_SvcSSRS12

---- we did these prior to now ----Data Source Account SetSPN -L myDom12\SQL12_Engine

C2WTS Account SetSPN -L myDom12\SP10_SvcC2WTS

45

SSRS

Page 46: Kerberos: The Four Letter Word

KerberosSSRS

Delegation

To allow SQL Reporting Services to delegate the client’s identity Kerberos constrained delegation must be configured. It is required to configure constrained delegation with protocol transition for the conversion of claims token to windows token via the WIF C2WTS.Each server running SQL Reporting services must be trusted to delegate credentials to each back-end service SQL Reporting will authenticate with. In additional, the SQL Reporting services service account must also be configured to allow delegation to the same back-end services.

46

SSRS

Principal Type Principal Name Delegates To ServiceUser myDom12\SP10_SvcSSRS12 MSSQLSVC/dcSQL12.myDom12.local:1433

User myDom12\SP10_SvcC2WTS MSSQLSVC/ dcSQL12.myDom12.local:1433

Page 47: Kerberos: The Four Letter Word

Kerberos

1. Open the Active Directory Object’s properties in Active Directory Users and Computers.

2. Navigate to the Delegation tab.3. Select Trust this user for delegation to specified services only.4. Select Use any authentication protocol. This enables protocol transition

and is required for the service account to use the C2WTS.5. Click the add button to select the service principal allowed to delegate to.6. Select User and Computers. 7. Enter the service account running the service you wish to delegate to. In

this example it is the service account for the SQL Server service: myDom12\SQL12_Engine

8. Click OK. 9. Select the services for the SQL Server data source10.Click OK.11.You should now see the selected SPNS in the services to which this

account can presented delegated credentials list. 12.Clicking Expanded will show both the short and long form of the SPNs

entered for the data source.13.Click OK

SSRS Constrained DelegationTo configure constrained delegation from SQL Reporting Services to the Data Source follow these steps.

SSRS47

SSRS

Page 48: Kerberos: The Four Letter Word

KerberosSSRS

C2WTS Constrained DelegationTo configure constrained delegation from C2WTS to the Data Source follow the same procedure you just did for SSRS Constrained Delegation – resulting in the following when done:

.

48

SSRS

In this example it is the service account for the SQL Server service. myDom12\SQL12_Engine

Page 49: Kerberos: The Four Letter Word

KerberosSSRS

SharePoint Create Managed Account

49

SSRS

Page 50: Kerberos: The Four Letter Word

KerberosSSRS

Reporting Services service Start the Reporting Services service

50

SSRS

Note: Be sure that the service is NOT running on Servers it should not be as this can lead to issues with C2WTS.

Page 51: Kerberos: The Four Letter Word

KerberosSSRS

SSRS 12 Service Application

51

SSRS

Once it has finished it will present you with a completion message and then a link to some further configuration, which will present a message letting you know if the SQL Server Agent service is running.

Page 52: Kerberos: The Four Letter Word

KerberosSSRS

SSRS 12 Service Application

52

SSRS

In order for the service application work as expected certain permissions need to be assigned to the application pool account. Click the "Download Script" command to get a dynamically generated script that you must then run in the SQL

SQL Reporting Services needs to access the SQL Agent through an account. Enter the SQL Agent account for the SharePoint SQL Instance

When complete the SQL Reporting Services Service Application will be created

Page 53: Kerberos: The Four Letter Word

KerberosSSRS

SSRS Service Account Permissions A required step in configuring SharePoint Server 2010 Office Web Applications is allowing the web application’s service account access to the content databases for a given web application. In this example, we will grant the SQL Reporting Service account access to the portal web application’s content database by using Windows PowerShell.

53

SSRS

Run the following command from the SharePoint 2010 Management Shell:$w = Get-SPWebApplication -Identity http://ReportingPortal$w.GrantAccessToProcessIdentity("myDom12\SP10_svcSSRS12")

The change to the SQL can be seen in the SQL Instance used for the SharePoint Farm by viewing the SQL Reporting Services Application Pool account Security Login Properties

Page 54: Kerberos: The Four Letter Word

KerberosSSRS

Testing Create a document library for reports

Validate site collection settings for Reporting Services

54

SSRS

Page 55: Kerberos: The Four Letter Word

KerberosSSRS

Testing Create and publish a test report in SQL Server Business Intelligence Development Studio

55

SSRS

Page 56: Kerberos: The Four Letter Word

KerberosSSRS

Testing Create and publish a test report in SQL Server Business Intelligence Development Studio

56

SSRS

Page 57: Kerberos: The Four Letter Word

KerberosSSRS

Testing Create and publish a test report in SQL Server Business Intelligence Development Studio

57

SSRS

Page 58: Kerberos: The Four Letter Word

KerberosSSRS

Testing Create and publish a test report in SQL Server Business Intelligence Development Studio

Validate in IE

58

Gotchas

Page 59: Kerberos: The Four Letter Word

KerberosGotchas

 Things to note:

Mixed Mode Active Directory (2k3/2k8)“The Given Key Was Not Present in the Dictionary”

Delegation – No Shortcuts

Rushing – Don’t

Summary

Page 60: Kerberos: The Four Letter Word

Kerberos

Summary

 If you follow these steps – hopefully you’ll avoid undo pain

When in doubt call Microsoft Support – they do have a Kerberos Troubleshooter they’ll have you run.Possible to run the tool in an offline mode – hopefully you read between the lines here.

Don’t skip steps, don’t take shortcuts, don’t do things out of order.

When all else fails, find a hard wall, pound your head against wall, call in sick and have someone else do it.

… You can always call Oakwood too … I guess

Setting up Kerberos – Slow – Painful – Time Consuming

Page 61: Kerberos: The Four Letter Word

Kerberos

Please fill out the evaluation and turn it in to this session’s host.

#GMSQL


Recommended