KB Article | Forcepoint Support

Notes & Warnings

Verification and Troubleshooting:

Helpful Commands:

SPN:

  • When adding SPN to the service account, it must be uppercase. The WCG code looks only for an uppercase "HTTP" entry. If not found, it does not get properly added to the keytab.
  • SPNs associated to the account used when joining WCG to AD domain: setspn -L <account_name>
User-added image
  • Look up the SPN itself: setspn -Q HTTP/<fqdn_of​_lb>. This will verify if the SPN exists and to which account it is associated.
User-added image
  • Multiple SPNs per service account: Although active directory allows this, the WCG code will only look for the first one returned by Active Directory. So it is recommended at this time to use one SPN per service account.

 

Kerberos Tickets

  • To clear kerberos tickets: klist purge

User-added image

  • See kerberos client tickets (only on client machines): klist
  • See what DCs the client machine is binding to: klist query_bind

Configuration and Verification

To verify that the client is authenticating to the proxy with the load-balancer’s Kerberos ticket, the client’s Kerberos ticket cache can be checked.

  1. Log onto the client, verify the proxy settings, and attempt to browse to any site. By the time a response comes back from the server, the client will have attempted Kerberos authentication.
  • On Windows 7 clients, open a command prompt and run "klist".
  • On older Windows systems with no klist utility, download "kerbtray" from Microsoft. Start it, browse to a site, and then double-click on the kerbtray icon in the system tray to see the current tickets.
  • On Linux, run "klist".
  1. Verify that the HTTP service ticket matches the FQDN to which the client is pointing.
  • If there are no HTTP service ticket matches, the AD has been misconfigured.
  • Ensure that the explicit proxy is setup to use the FQDN of the load balancer and not the IP address. Kerberos tickets are only granted by FQDN,
  • If the ticket matches the FQDN but authentication is falling back to NTLM, indicating that the Content Gateway may be misconfigured.

Here is an example of klist output with the load balancer's FQDN as the authenticated service. This illustrates what Kerberos should look like from a client when authentication is working correctly.

User-added image

  1. You can also verify that Kerberos is working correctly, or troubleshoot a problem, from within the Content Gateway manager.
1. Verify that the instance of Content Gateway has a good connection to the Domain controller.
2. Log on to the WCG manager and navigate to Configure > Access Control > Integrated Windows Authentication.

User-added image

  1. Locate the Troubleshoot Authentication button at the bottom of the page.
  2. On the Monitor > Security > Integrated Windows Authentication page, check to see if there are authentication failures or errors, or run a test to see if the proxy can authenticate properly.

User-added image

 

Administrative vs. Service Accounts

It is important to distinguish the Administrative Account and Service Account terms:

  • Administrative Account is usually a member of Domain Administrators group with elevated rights and is used to log in interactively and manage information systems. Due to elevated privileges, the Administrative Account should have strict password policy with expiration and complexity requirements
  • Service Account is used as a permanent logon credentials for running services and should have minimal (non-admin) privileges to serve particular task only. Service Account should not have password expiration policy and, in many cases, password change option should be blocked.


Password Policy

For the purpose of proper Kerberos keys validation it is mandatory that the Service Account password will never be changed after joining the domain. If a password change is required, the domain should be un-joined, password changed, and then domain joined back again with the new password. Otherwise, the Kerberos keys will mismatch between Active Directory and Content Gateway, so the load balanced Kerberos authentication will fail and fall back to NTLM.

Note: Machine (Content Gateway computer) accounts that have permanent Kerberos keys are not affected.

Changing from Administrative to Service account

If domain is unjoined and then joined back with a different account (not the one used for initial join), it is recommended to remove all the Content Gateway computer (machine) accounts from Active Directory between unjoin and second join operations, otherwise the domain join attempt might fail.

It is typical for the customers to join the domain with Administrative Account (a member of Administrators group). However during migration from legacy to load balancing configuration customers are strongly recommended to use a different Service Account without password expiration (see definition above). This means that initial domain join (before load balancing was introduced) may create Content Gateway computer (machine) accounts in Active Directory with ACLs allowing only Administrative Account to modify it. If a different Service Account is used for the second domain join operation, existing Content Gateway computer (machine) account will have the older ACLs which do not permit Service Account to modify the Content Gateway computer account. Simply removing the older Content Gateway computer accounts and re-trying the join should solve this issue.

Non-admin users cannot join domain (issue 1)

By default, non-administrative (e.g. Domain User) accounts have a limit of 10 (ten) domain join operations per account. After using 10 attempts, no more joins are allowed for that account.

In this case it is recommended to follow the Microsoft KB251335. If no acceptable solution is found, it is suggested to create a new Service Account and re-assign the SPN properly:

  1. Remove the SPN form old Service Accounts using setspn - D syntax.
  2. Add the SPN to the new Service Account using setspn -A syntax. (See instructions below.)

Non-admin users cannot join domain (issue 2)

In some environments it is enough to grant write permissions on the CN=Computers container in Active Directory to the service account used to join the domain.
 

Disjoining the domain and clearing up the service account

  1. Disjoin the domain in Content Gateway manager.
  2. Remove the SPNs from Active Directory using setspn -D command: C:\Users\Websense>setspn -D HTTP/LB.domain.org forcepoint-svc
  3. Re-join the domain in Content Gateway manager.
  4. Run IWA test (Monitor > Integrated Windows Authentication > Run test) and verify there are no longer any additional entries in keytab.

Problem Description

Kerberos authentication requires that clients send Content Gateway's Kerberos ticket to Content Gateway. Because the clients are pointing to a load-balancer's VIP via the FQDN, the client instead sends the load-balancer's ticket to Content Gateway, which causes Kerberos authentication to fail.

It's not possible to configure clients to request Content Gateway's Kerberos ticket, because the client's OS handles the ticket request based on the FQDN of the proxy to which the clients are explicitly pointed. As a result, Integrated Windows Authentication (IWA) using Kerberos fails client authentication in a load-balanced environment when Content Gateway is deployed as an explicit proxy.

Note: The following applies to explicit proxy deployments only. Transparent proxy deployments do not require this configuration to authenticate successfully using Kerberos in a load balanced environment.

Resolution

Note: This functionality described below is part of the v8.2 release of TRITON AP-WEB and above (including v8.5.3). Support for this is also available for versions v7.8.4 - v8.1, but a hotfix is required. Please contact Technical Support for assistance.

Because it’s not possible to stop clients from sending a load-balancer’s Kerberos ticket to Content Gateway, the proxies must be configured to accept the load-balancer’s ticket, making the proxies appear as the load-balancer within the scope of Kerberos.

To resolve this, customers must create a new Active Directory “Service Account” and use that account to create SPNs for the Load Balancer's FQDN.

This new SPN will be added to all of the appliances' keytabs when it is joined to the Active Directory domain and will thus help with Kerberos authentication.

Note: This document does not cover the configuration of the load balancer. Refer to the documentation for the specific load balancer in your environment for details on configuration.

Enabling the Functionality:

1. Set the Service Principal Name (SPN) to the account that will be used to join WCG's to the Active Directory domain.
  1. In Active Directory Administrative Center (Windows Server 2012R2), select your domain you wish to manage and create a new user account:

User-added image
 

You may name your account anything. In this example we will call it "forcepoint_svc".

 

User-added image

Note: Unless explicitly enforced by security settings, this account does not have to be Administrator. The account used should have privileges sufficient to join the domain, as defined by company policy. For information on required permissions, see Permissions Required for Accounts Used to Join Proxies to a Domain for Integrated Windows Authentication.

Standard users can join up to 10 machines to the domain. (This is a default Microsoft limit.)

  1. It is recommended to disable the password expiration/ change to this service account unless your IT policy prohibits this.

 

User-added image

You may also want to remove it from any sensitive groups in the AD.

Note: For Rules Based Authentication with Multiple Realms, you will need to create different service accounts for the different domains.

In multi-realm configuration, the user used for joining the machine to all but the first domain must be a domain admin because a computer in the AD that uses a hostname outside the domain must be added. (See the "Samba Client is not in the same domain as the AD domain" section of this site.)

  1. Ensure your new account is enabled. When searching for the account, if there is a "Disabled" message in the status bar when you select the account, it must be enabled first.

User-added image
 

To enable the account, right click and choose "Enable":


User-added image
 

  1. Add the SPN for the Load Balancer to the newly created domain account.
Add the service principal to the Active Directory account using the safe mode of the setspn command.

For example, if the Load Balancer FQDN is "LB.domain.org" and the domain account is "forcepoint-svc", use the following command:

setspn -S HTTP/lb.domain.org forcepoint_svc

User-added image

Note: It is important that the "HTTP" text in the SPN is uppercase as shown in the example, otherwise the WCG code responsible for reading the SPN will not locate it and authentication will fail. 

  1. Join WCG to the Active Directory Domain.
a. Open the Content Gateway console and join the domain(s) using the account created that contains the SPN for the load balancer handling traffic for our proxies. In this example, we are using "forcepoint_svc".
b. Repeat for the remaining Content Gateway appliances that are handled by the load balancer.

Each machine must be joined as their unique hostnames. If the computer objects do not already exist in AD, they will be created upon joining.

  1. Confirm your changes are effective by verifying the new keytab on each proxy.
a. After joining the domain, run IWA "Run Test" in the UI. (Monitor > Integrated Windows Authentication > Run test)
User-added image

b. Look for the output of the "net ads keytab list" command in the diagnostic output above. It should list additional keytabs containing the LB hostname.
 User-added image


Permissions required for accounts used to join proxies to a Domain for Integrated Windows Authentication


Required permissions

The following 10 permissions are required for the account toward the "Computers" OU in a domain.

In the scope of "This object and all descendant objects", two permissions are required:

  1. Create Computer Objects
  2. Delete Computer Objects

In the scope of "Descendant Computer Objects", eight permissions are required:

  1. Read All Properties
  2. Write All Properties
  3. Read Permissions
  4. Modify Permissions
  5. Change Password
  6. Reset Password
  7. Validated write to DNS host name
  8. Validated write to service principle name

Configuration

  1. From the View menu, select Advanced Features to enable it.

User-added image

  1. In the Navigation Pane, right click Computers OU and select Properties.

User-added image

  1. In the Properties window, select the Security tab.

User-added image
 

  1. Click Advanced to open the Advanced Security Settings for Computers window.
  2. Click Add to open the Permission Entry for Computers window.

User-added image

  1. Click Select a principal and select the account you are configuring permissions for. Click OK.

User-added image

User-added image

  1. From the Applies to drop down menu, select This object and all descendant objects.

User-added image

  1. Check the Create Computer objects and Delete Computer objects boxes. Click OK.

User-added image

  1. From the Advanced Security Settings for Computers window, again click Add.
  2. Click Select a principal and select the account you are configuring permissions for again. Click OK.
  3. From the Applies to drop down menu, select Descendant Computer objects.

User-added image

  1. Check the boxes for the following permissions:
  • Read All Properties
  • Write All Properties
  • Read Permissions
  • Modify Permissions
  • Change Password
  • Reset Password
  • Validated write to DNS host name
  • Validated write to service principle name
User-added image
  1. Click OK.

Note that there is no 10 machine limit for accounts that gain permissions by the above method.





Definitions:


Service Principal Name (SPN): Within the context of Kerberos, an SPN is the name that identifies an instance of a service. For the purposes of this article, an SPN can be thought of as an FQDN for a specific service (e.g., Content Gateway's HTTP proxy service). An example of an SPN for the HTTP service of a Content Gateway that is joined to the domain “exampleDomain.com”, with the hostname, “exampleWCG”, is:

HTTP/exampleWCG@EXAMPLEDOMAIN.COM
 

Kerberos Key Distribution Center (KDC): The KDC is the server that authenticates clients and services, manages SPNs, and distributes Kerberos tickets. Most often, the KDC operates within, and is synonymous with, Windows Active Directory (AD).
The KDC must contain the SPNs of the load-balancers so that when a client’s ticket request comes in, the KDC can actually return a ticket. With load-balancers that do not natively support Kerberos, the KDC will not have the SPNs of the load-balancers, so they must be created.
If clients are pointing to load-balancers by IP address instead of the FQDN, the SPNs can be created in the KDC without affecting production. If clients are pointing to load-balancers by FQDN, then adding the SPN breaks transparent NTLM authentication because the clients begin to attempt Kerberos authentication. As a result, clients will have to manually enter proxy authentication credentials. So the timing of adding the SPN is important, and should only be done during a change window and after the workaround has been applied to all proxies within the load-balancer pool.






Keywords:
Kerberos; Integrated Windows Authentication; load balanced; spn; alias; shared host name; service principle name; content gateway; authentication; iwa

Article Feedback



Thank you for the feedback and comments.