Issue Definition:
We are getting unauthorized error in the SharePoint logs and the credentials are passed by secure store service and we are unable to find the user name for the service. From the event logs we could see that the account is locked out.
Temporary Resolution:
The account "searchserviceaccount" was locked and we involved the AD team to unlock the account and then everything started working as expected.
Logs:
06/04/2018 14:43:44.66 w3wp.exe (0x3878) 0x2054 SharePoint Foundation Topology e5mc Medium WcfSendRequest: RemoteAddress: 'https://app01:32122/7cbc7f8f4e3d42ebae9cfbc72a582cd9/SecureStoreService.svc/https' Channel: 'Microsoft.Office.SecureStoreService.Server.ISecureStoreServiceApplication' Action: 'http://schemas.microsoft.com/sharepoint/2009/06/securestoreservice/ISecureStoreServiceApplication/GetRestrictedCredentials' MessageId: 'urn:uuid:9c94020e-ca09-4972-a63f-2f8b775a668a' 99e46d9e-6a17-a08a-7953-a69226556ff7
06/04/2018 14:43:44.70 w3wp.exe (0x3878) 0x2054 Business Connectivity Services Business Data ahe2m High Web Exception : System.Net.WebException: The remote server returned an error: (401) Unauthorized. at System.Net.HttpWebRequest.GetResponse() at Microsoft.SharePoint.BusinessData.SystemSpecific.OData.ODataHttpClientRequestMessage.GetResponse() at Microsoft.SharePoint.BusinessData.SystemSpecific.OData.ODataConnection.ExecuteRequest(IBCSODataRequest requestMsg) 99e46d9e-6a17-a08a-7953-a69226556ff7
In order to probe more on whats causing the account lockout , extensive set of logging is required. Hence we followed the below action plan:
On the client machine perform the below:
# Account Lockout and Management Tools
http://www.microsoft.com/downloads/details.aspx?familyid=7AF2E69C-91F3-4E63-8629-B999ADDE0B9E&displaylang=en
# Please also perform the clean boot on the problematic server (please monitor the issue after performing below)
- Open command prompt: type msconfig go to services Hide all Microsoft services disable all third party services.
- It will ask you for a reboot.
- Uninstall the antivirus.
If the client doesn’t lock out after performing the above mentioned steps, we will conclude that the third party service is causing the issue or the antivirus is the cause.
# Please enable auditing on the primary domain controller in domain controllers OU:
Computer Configuration\Windows Settings\Security Settings\Local Policies\Audit Policy\ Audit account logon events - Failure Computer Configuration\Windows Settings\Security Settings\Local Policies\Audit Policy\ Audit account management- Success & Failure. Computer Configuration\Windows Settings\Security Settings\Local Policies\Audit Policy\ Audit logon events- Failure.
Below are the events that we need to look:
4740- account has been locked out
4771- bad password attempts, we will look for caller computer ID.
4625 –account failed to logon.
# With the originating machine identified, we can further check on it for the following:
a. Please check if there is any service running under a domain account that is locked. If yes, it could be old password cached on the client machine. Please then re-configure the service starting account with a new password. ( This lock out happening on Search as well as app server, since it is a service account we never used to change the password)
b. Please check if there is any network drive mapping is configured on the client, and is using the account that is locked. If yes, it could be old password cached on the client machine. Please re-configure the drive map with a new password. ( This is not happening on the client machine, it is happening on the SharePoint server)
c. Please check if there is any cache in “stored username and password” in control panel. If yes, please delete them.
d. Please check if there is any scheduled task on the client, either to use the domain account to launch, or the task is about authentication. If yes, please remove the task temporarily. ( We don’t see any task configured with this since it is our crawler account)
e. Please check if there is any manually created script running at background on the client. If yes, please stop the process temporarily. ( I don’t think so we have any script running in the background)
#Enable logs:
1) Enable netlogon logging on all the DCs.
2) Enable auditing on all DCs -
Nltest /DBFlag:2080FFFF
https://support.microsoft.com/en-in/kb/109626
- Account logon - Success and failure
- Logon events - success and failure
- Account management – success and failure
3) Use the lockout status tool to see which DC the bad passwords are being sent to.
4) Once the bad passwords count increases, review the logs of that DC to check from where the bad password is coming from.
5) Once the source has been identified, then enable auditing on that machine accordingly -
If the source machine is Windows 7 / Server 2008 R2 or upwards, then – Execute in all the servers
auditpol /set /subcategory:"Kerberos Authentication Service" /failure:enable auditpol /set /subcategory:"logon" /failure:enable auditpol /set /subcategory:"Account Lockout" /success:enable /failure:enable auditpol /set /subcategory:"User Account Management" /success:enable /failure:enable auditpol /set /subcategory:"Credential Validation" /failure:enable auditpol /set /subcategory:"Process Creation" /success:enable
Once you collect the logs disable auditing:
auditpol /set /subcategory:"Kerberos Authentication Service" /failure:disable auditpol /set /subcategory:"logon" /failure:disable auditpol /set /subcategory:"Account Lockout" /success:enable /failure:disable auditpol /set /subcategory:"User Account Management" /success:enable /failure:disable auditpol /set /subcategory:"Credential Validation" /failure:disable auditpol /set /subcategory:"Process Creation" /success:disable
Root Cause:
The issue has been identified due to mismatch on the lmcompatabilitylevel at DC end, the action plan is to match the settings across the domain and forest. In case if the issue still persists after implementing it from their end then we need to have the same compatibility at SharePoint server level, but there are some challenges where the application might get break which still relays on this.
So we see that the account lockouts are coming from a mismatch in lmcompatabilitylevel mismatch.
Action plan and Fix:
Implement the setting on the dc to be consistent to match across the board. This will help avoid inconsistent results when connecting to the domain controllers. The dc setting does not take a reboot but may be a short time and it will pick up the setting. It would be the best practice setting the setting at a gpo level to enforce the setting across all dc.
Here is a screen shot where you would find the policy. This one sets the dc at 5
This is the screen shot of the policy set at 3
Action plan for the SharePoint servers:
If this does not work you will need to set either a registry key or group policy on the SharePoint servers. THIS WILL REQUIRE A REBOOT ON CLIENTS TO BE EFFECTIVE.
MS not recommended that you match the settings on the servers as you did the dc.
Here’s a few links on lmcompatabilitylevel.
Lmcompatabilitylevel most misunderstood setting.
Security guidance for NTLMv1 and LM network authentication
simple breakdown of how ntlm works
Common sharepoint issues with NTLM authentication
We should not be running into this but when you do a lot of ntlm auth you may need to watch out for maxconcurrentapi setting.
Here is a few articles on maxconcurrenatpi.
Optimizing NTLM auth through multi-domain environment.
Performance tuning for ntlm auth using maxconcurrentapi
management pack
script to check for maxconcurrentapi issues
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters\
Reg dword MaxConcurrentApi
Value in decimal 2 – 150
Dc need to be all at the same level and good idea to have trust at the same level as well.
Member servers may need to be bumped up as well. but not as high as domain controllers.
No comments:
Post a Comment