KB Article | Forcepoint Support

Problem Description

I have received a Tech Alert stating maintenance is going to happen to a Cloud Cluster. What do I need to check or do to ensure my users are not impacted?

Resolution

In most situations, maintenance for a cloud data center (cluster) will happen for a short duration outside of production hours. In the event that maintenance is extended, is longer than one day, is during normal business hours, or an emergency happens, the following FAQ may help. Questions include:  

"Will I be affected?"

Only connections going through the cluster undergoing maintenance should be affected.
  • If using Forcepoint's GeoDNS service, no downtime is expected as customers will be failed-over to the nearest available cluster.
  • If using IPsec or GRE, secondary tunnels may be in use.
  • If using Email Security Cloud, secondary clusters may be in use. Emails in quarantine from the affected cluster may not be able to be released until maintenance is over. 
  • If using the Reporting service in the Security Portal, no impact is expected as secondary data stores will be queried.
  • See the Tech Alert for specific information released regarding the maintenance. 

 

"How will we know when the maintenance is done?" 

Monitor the progress of the cluster by visiting the Forcepoint Trust page. 


"How do I know which cluster I'm going to?" 

  • For Web Security Cloud and Hybrid:
  1. Have a test computer connected to the Cloud. 
  2. Visit: http://query.webdefence.global.blackspider.com/?with=all
  3. Under Hostname near the top, an entry appears as aaa##a.srv.mailcontrol.com. The letter directly before .srv is the cluster being used.
  4. For more information regarding which cluster this is, see Cloud service cluster (cluster) IP addresses and port numbers.
  • For Email Security Cloud:
  1. Open the Forcepoint Security Portal
  2. Navigate to Email > Settings > DNS Records & Service IPs.
  3. The top section titled MX Record DNS Entries will show the hosts, in cust#####-1.in.mailcontrol.com and  cust#####-2.in.mailcontrol.com format. The first one is the primary, the second one is the failover.
  4. Open Command Prompt
  5. Type nslookup cust#####-1.in.mailcontrol.com where the #'s match what was found in the manager.
  6. Note the IP address showing.
  7. Type nslookup cust#####-2.in.mailcontrol.com where the #'s match what was found in the manager.
  8. Note the IP address showing.
  9. Verify the both IP addresses against Cloud service cluster (cluster) IP addresses and port numbers to confirm what cluster is in use.
  1. Open Forcepoint Security Manager.
  2. Navigate to Settings > Hybrid Service > Hybrid Configuration.
  3. Locate the cust#####-1.in.mailcontrol.com and cust#####-2.in.mailcontrol.com entries. The #'s will be specific to your account.
  4. Open Command Prompt
  5. Type nslookup cust#####-1.in.mailcontrol.com where the #'s match what was found in the manager.
  6. Note the IP address showing.
  7. Type nslookup cust#####-2.in.mailcontrol.com where the #'s match what was found in the manager.
  8. Note the IP address showing.
  9. Verify the both IP addresses against Cloud service cluster (cluster) IP addresses and port numbers to confirm what cluster is in use.


"I'm using a local PoP (vPoP) connection, will I be affected?"

If the location for processing traffic is one of the clusters mentioned to be undergoing maintenance, your customers may be affected. To verify:
  1. Using a computer connected to the cloud service, visit: http://query.webdefence.global.blackspider.com/?with=all
  2. Under Hostname near the top, an entry appears as aaa##a.srv.mailcontrol.com. The letter directly before .srv is the cluster being used.
  3. For more information regarding which cluster this is, see:


"How do I test redundancy or fail over?"

Check with your network administrator for ways to halt connection to the cluster undergoing maintenance to test if still able to reach the internet through a different cluster. 
  • Web Security Cloud or Hybrid cluster failover testing via firewall for endpoint, PAC file, IPsec and GRE:
  1. Block traffic on a test machine/environment to the cloud cluster IP range for the affected cluster in the access list, then navigate to any website.
  2. If websites are accessible, visit: http://query.webdefence.global.blackspider.com/?with=all.
  3. Under Hostname near the top, an entry appears as aaa##a.srv.mailcontrol.com. The letter directly before .srv is the cluster being used.
  4. For more information regarding which cluster this is, see Cloud service cluster (cluster) IP addresses and port numbers.
  5. If the location is a different cluster than when the traffic is not blocked, redundancy is working as expected. 
  • Email Security Cloud and Hybrid testing:
  1. Visit Wormly's Test your SMTP Mail Server (MX) website.
  2. Type the second cluster record hostname ( cust#####-2.in.mailcontrol.com ) in the SMTP server field.
  3. Type your working email address in the Recipient email field. 
  4. Click Test this SMTP Server.
  5. Beneath the Test button, review the results box. The first line stating  SERVER -> CLIENT: 220 cluster-X.mailcontrol.com indicates which Cloud cluster received the email. 
  6. Check that you received the email. If so, mail flow is working as expected from the secondary cluster.
In most cases, using the cluster hostname will have the least impact to end customers as the conversion to another cluster should happen seamlessly. However, if an IP is specifically assigned, issues can arise and websites will not be accessible.

"Redundancy testing failed, or customers are still connecting to the cluster undergoing maintenance. What do I do?"

This scenario is usually due to having used an IP address rather than the cluster hostname to connect to the cloud. This is not a suggested configuration by Forcepoint due to this sort of scenario. Other cases may be due to the secondary cluster not being allowed through the firewall.  For more specific to product information: 
  • Hosts File
Ensure there are no hosts file entries that may resolve hostnames without DNS for the Cloud Datacenters.
Important Forcepoint Technical Support does not support custom PAC files.
Raise a case with Technical Support for instructions on submitting the edited PAC file to the Cloud. 
  • Hybrid On-Premises Failover for Explicit Proxies: See Configuring failover to the hybrid service
    1. Open Forcepoint Security Manager.
    2. Navigate to Settings > Hybrid Configuration > Filtered Locations.
    3. Click on a Filtered Location IP address.
    4. Click Advanced.
    5. If Enable failover to hybrid service is marked, check the specific cluster assigned. 
    6. If this cluster is the cluster which is undergoing maintenance, change the cluster. 
    7. Click Save
    8. Click Save and Deploy
    9. Repeat for each Filtered Location. 
    10. If any changes were made, navigate to Settings > Hybrid Configuration > Scheduling.
    11. Click Send User Data Now as well as Send Policy Data Now
  • Email Cloud and Hybrid:
    1. Locate the MX record hostnames in the cust####-#.in.mailcontrol.com from the earlier step How do I know which cluster I am going to?.
    2.  Verify the Nslookup for both mailcontrol.com entries from the earlier step "How do I know which cluster I'm going to?".
    • If both entries are going to the same cluster, raise a case with Technical Support.
    • If the 2.in.mailcontrol.com does not resolve on the local DNS, try Google DNS:
      1. Open Command Prompt.
      2. Type nslookup cust#####-2.in.mailcontrol.com 8.8.8.8 where the #'s match what was found in the manager.
        • If this works, there is a DNS issue in the local DNS for cust#####-2.in.mailcontrol.com that needs to be resolved by the local DNS Server administrator.
        • If you receive the error "No response from server" or "DNS  Request Timed out" , contact your local DNS Server Administrator. 
        • If you receive the error "Non-existent domain" , raise a case with Technical Support.


"I am seeing Web Cloud or Hybrid extreme latency or pages not loading at the temporary location. What do I do?"

A small decrease in performance is expected during a fail over scenario when connecting to a cluster that is not the preferred cluster for your location. In cases where there is extreme latency resulting in web pages not loading, receiving time-outs, or all websites are taking a long time to load (such a few seconds) follow the information below. 
  1. Collect logs showing the latency issue: 
  2. Raise a case with Technical Support.



Keywords:
maintenance; cloud; cluster; hybrid; ipsec; tunnel; downtime; failover;  Extended Maintenance Procedures; tech alert; service; saas; uptime; email blocking; emails stuck; quarantine

Article Feedback



Thank you for the feedback and comments.