SPSearchServiceSettings: change Windows Service Account cause issue on the Search Host Controller service

[julmsy]

Hello SharePointDsc team,

Details of the scenario you tried and the problem that is occurring

When you call the SPSearchServiceSettings resource and WindowsServiceAccount parameter to change the service account, the following occurs:

• Service is shutting down
• The new service account is set on the service
• Service is starting... and get stuck in starting position (in Central Admin). On the Windows services.msc panel, you can see that service trying to start every about each minute.

Verbose logs showing the problem

Nothing interesting on the verbose logs, however in SharePoint ULS logs, you can see the following error:

09.10.2021 19:15:50.39	hostcontrollerservice.exe (0x1160)	0x0E50	Search	Search HostController	ad5eq	Unexpected	WcfService: Got CommunicationException out when trying to open the HostController servicehost. Shutting down process.  - threadId: 15 - exception: System.ServiceModel.CommunicationException: The service endpoint failed to listen on the URI 'net.tcp://localhost/ceres/hostcontroller/' because access was denied.  Verify that the current user is granted access in the appropriate allowAccounts section of SMSvcHost.exe.config. ---> System.ComponentModel.Win32Exception: Access is denied     at System.ServiceModel.Activation.SharedMemory.Read(String name, String& content)     at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.ReadEndpoint(String sharedMemoryName, String& listenerEndpoint)     --- End of inner exception stack trace ---     at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.ReadEndpoint(String sharedMemoryName, String& listenerEndpoint)     at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.HandleServiceStart(Boolean isReconnecting)     at System.ServiceModel.Channels.SharedConnectionListener.SharedListenerProxy.Open(Boolean isReconnecting)     at System.ServiceModel.Channels.SharedConnectionListener.StartListen(Boolean isReconnecting)     at System.ServiceModel.Channels.SharedTcpTransportManager.OnOpenInternal(Int32 queueId, Guid token)     at System.ServiceModel.Channels.SharedTcpTransportManager.OnOpen()     at System.ServiceModel.Channels.TransportManager.Open(TransportChannelListener channelListener)     at System.ServiceModel.Channels.TransportManagerContainer.Open(SelectTransportManagersCallback selectTransportManagerCallback)     at System.ServiceModel.Channels.TransportChannelListener.OnOpen(TimeSpan timeout)     at System.ServiceModel.Channels.ConnectionOrientedTransportChannelListener.OnOpen(TimeSpan timeout)     at System.ServiceModel.Channels.TcpChannelListener`2.OnOpen(TimeSpan timeout)     at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)     at System.ServiceModel.Dispatcher.ChannelDispatcher.OnOpen(TimeSpan timeout)     at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)     at System.ServiceModel.ServiceHostBase.OnOpen(TimeSpan timeout)     at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)     at Microsoft.Ceres.HostController.WcfServer.WcfService.StartServiceEndpoint()	

Suggested solution to the issue

After search and test, I found the missing permission that cause this issue: the new service account that set on the SharePoint Search Host Controller must be added in the IIS_IUSRS local group. Currently the newly service account is only added in WSS_WPG local group automatically.

The DSC configuration that is used to reproduce the issue (as detailed as possible)

        SPSearchServiceSettings SearchServiceSettings
        {
            IsSingleInstance = 'Yes'
            ContactEmail = $ConfigurationData.NonNodeData.SPFarm.SearchServiceSettings.ContactEmail
            WindowsServiceAccount = New-Object System.Management.Automation.PSCredential ($ConfigurationData.NonNodeData.SPFarm.Accounts.SearchAccount.UserName, ($ConfigurationData.NonNodeData.SPFarm.Accounts.SearchAccount.Password | ConvertTo-SecureString -Key (1..16)))
            DependsOn = "[SPSearchServiceApp]SearchServiceApp"
            PsDscRunAsCredential = $setupAccount
        }

The Build/Cumulative Update of SharePoint is running

SharePoint 2016
BuildVersion : 16.0.5134.1001 (March 2021 CU)

The operating system the target node is running

OsName               : Microsoft Windows Server 2019 Standard
OsOperatingSystemSKU : StandardServerEdition
OsArchitecture       : 64-bit
WindowsVersion       : 1809
WindowsBuildLabEx    : 17763.1.amd64fre.rs5_release.180914-1434
OsLanguage           : en-US
OsMuiLanguages       : {en-US}

Version and build of PowerShell the target node is running

Name                           Value
----                           -----
PSVersion                      5.1.17763.1
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.17763.1
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version of the DSC module that was used

Name          Version Path
----          ------- ----
SharePointDSC 4.7.0   C:\Program Files\WindowsPowerShell\Modules\SharePointDSC\4.7.0\SharePointDSC.psd1

Let me know if you would like more information.

Edit -- Added information/testing:

• If I reboot the node after SPSearchServiceSettings/WindowsServiceAccount, and having the SharePoint Search Host Controller trying to start (so the account was not added in IIS_IUSRS group), then after the reboot the situation is the following: the windows service is running, but the SharePoint Service Instance is still on Provisioning status.
• After a Stop/Start-SPServiceInstance on Search Host Controller Service, the status is... Online.

At that point, it really confuse if you need to have the service account added in IIS_IUSRS group. However, if you add this account into that group when the problem occurs, the Windows service start automatically in the next minute.

[ykuijs]

Hi @julmsy, I am trying to reproduce this issue but unfortunately I am unable to. Can you share a little more about your topology? Do you have dedicated Search servers? Is this occurring on all servers? Can you reproduce the issue consistently?

[julmsy]

Hi @ykuijs,

For now the topology is simple: Single-server farm, as I'm working to provision DEVs environments.
And yes, I can reproduce the issue consistently when I run DSC scripts to install/build/configure the new environment.

[ykuijs]

I have just tested to update the search account three times (twice on SP2016 and once on SP2019) and all work without issues.

Are these farms part of an Active Directory? If so, since you are using a single server farm: Does this server also host the AD (100% stand alone environment)?

Have you also tried the most recent patch levels (16.0.5134 is March 2021)?

[ykuijs]

And is your environment hardened in any way? Like with group policies?

[julmsy]

Yes, these farm are part of an Active Directory, but AD (as well as SQL) are on separate servers. The SharePoint Server only have SharePoint, no other service.
No GPO or other security stuff, as I'm developing & testing on my own VMs for now.

Here is what I done in my script in sequence:

1. Add Install service account in local Administrator group
2. Install Windows features
3. SPInstallPrereqs
4. SPInstall
5. SPInstallLanguagePack
6. SPProductUpdate (March 2021)
7. SPFarm
8. [...] (provision some SA)
9. SPSearchServiceApp
10. SPSearchServiceSettings
11. SPSearchTopology

I can try with a most recent CU to see if there is a different behavior. Give me some days.

[ykuijs]

Currently my test SP2016 environment is running the Dec 2020 CU. Am updating to the most recent CU as well.

[ykuijs]

FYI: I have just updated my environment to the same patch level you were using (March 2021). Changing the service account keeps working fine with that CU. Curious to see what effect updating to the most recent CU has on your environment.