KB Article | Forcepoint Support

Notes & Warnings

Note For NGFW specific sg commands, see Forcepoint NGFW Engine commands section from NGFW Online Help or NGFW Product Guide available at Forcepoint Support Portal.

Problem Description

Next Generation Firewall allows access to various Linux utilities that can be used for troubleshooting issues and problems. Many such utilities are listed and explained below in the resolution section.
 

Resolution

The Linux utilities that can be used for troubleshooting are listed below:

ip [ link | addr | route | neigh ]
 
The iproute2 suite includes various networking utilities, for listing interface IP addresses, link statuses and counters, routing tables and ARP table. You can also append "show dev ethX" to these commands to only show relevant results for a particular interface.

If netlinks are used then a separate routing table is created for each netlink, as a netlink route is in essence a policy route. The command ip route only shows the main routing table, so to also see netlink routes you must use the syntax ip route list table all. The all in this command may also be substituted by the netlink routing table ID. To see the routing table IDs and routing rules that direct traffic to particular netlinks routing tables, use the command ip rule.

Note that in a cluster, if a CVI address exists on an interface it is shown on all nodes and ip addr shows it with a /32 netmask. NDI addresses are listed with the actual netmask of the interface. Also note that in a cluster both ip addr and ip link show the CVI MAC address if the node is dispatcher for that interface; if the node is not the dispatcher for a particular interface then the physical MAC is assigned on that interface.

root@node:~# ip -s link
4: eth0_0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:2c:51:66 brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    15003203929 77579247 0       0       0       77572163
    TX: bytes  packets  errors  dropped carrier collsns
    23812448271 87585152 0       0       0       0
...
root@node:~# ip addr
4: eth0_0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:25:90:2c:51:66 brd ff:ff:ff:ff:ff:ff
    inet 10.199.1.11/24 brd 10.199.1.255 scope global eth0_0
       valid_lft forever preferred_lft forever
...
root@node:~# ip route
default via 10.20.1.100 dev eth1_0.103  proto static  src 10.20.1.11
10.199.1.0/24 dev eth0_0  proto static  scope link  src 10.199.1.11
10.1.1.0/24 via 192.168.10.150 dev eth1_1.102  proto static  src 192.168.10.11
...
root@node:~# ip neigh
192.168.10.10 dev eth1_1.102 lladdr 00:0c:29:a7:1f:23 DELAY
10.199.1.12 dev eth0_0 lladdr 00:25:90:d0:f6:86 STALE
192.168.10.81 dev eth1_1.102 lladdr 00:0c:29:2f:c8:0d REACHABLE
10.160.1.81 dev eth1_1.104  FAILED
 
netstat

Short for network statistics. Displays information about connections that originate or terminate on the engine itself. This is useful for investigating problems where the engine needs to open connections to other components or vice versa. For example, Log server connections, Management server connections, Web filtering (connection to Brightcloud servers) or User Agent/MLC connections.
   
root@node:~# netstat -anvp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
...
tcp        0      0 192.168.10.12:28105     192.168.10.10:3020      ESTABLISHED 3548/sendlogd
tcp        0      0 160.160.1.12:17477      160.160.1.32:16661      ESTABLISHED 3561/uiad
tcp        0      0 192.168.10.12:4987      192.168.10.10:51012     ESTABLISHED 2419/mgmtd
...
 
ps

Short for process status. The ps command lists the currently-running processes, current memory and CPU usage, which is useful for analyzing engine load or incorrectly behaving processes.

root@node:~# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   1712   400 ?        S    Feb18   0:22 /sbin/init
root         2  0.0  0.0      0     0 ?        S    Feb18   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    Feb18   0:03 [ksoftirqd/0]
...
root      3261  0.0  0.0      0     0 ?        S    Feb18   0:00 [sg_name_cache]
root      3277  0.2 42.2 974652 838360 ?       Ssl  Feb18  24:22 /usr/sbin/sg-inspection id=106
 
free
 
The free command displays memory statistics. Linux will always buffer and cache data, and the most important is the free memory within the +/- buffers/cache line. In the following example, it is 1044268 (50% of available RAM is still free on demand). This lists the usable free memory within the engine. A very low number is an indication of memory running out. Another indication of problems would be a high level of Swap usage. In engine version 5.10 and newer the kernel includes also MemAvailable value in /proc/meminfo which is an even more accurate approximation of actually available memory, as it will subtract for example non-reclaimable cache from the total.
 
root@node:~# free
             total       used       free     shared    buffers     cached
Mem:       2001284    1615864     385420          0     155572     503276
-/+ buffers/cache:     957016    1044268
Swap:       499704          0     499704

df
 
The df command lists disk utilization. This is useful for checking free space on the /data and /spool partitions, when you see an error or warning in firewall Status tab in SMC and there is an indication that either of these partitions are filling up. Note the two partitions mounted on / are always listed as 100% used as the active/inactive software images are stored there.

root@node:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                177M  177M     0 100% /
tmpfs                 970M  476K  969M   1% /dev
/dev/discs/disc0/part5
                      177M  177M     0 100% /
tmpfs                 970M  228K  969M   1% /tmp
/dev/sda8             485M  103M  382M  22% /data
/dev/sda9             2.5G  167M  2.3G   7% /spool
none                  4.0G  345M  3.7G   9% /spool/ipsfp

mount

Used if certain file systems are experiencing errors and have been re-mounted as read-only filesystems (ro).
 
root@node:~# mount
rootfs on / type rootfs (rw)
tmpfs on /dev type tmpfs (rw,relatime)
/dev/discs/disc0/part6 on / type romfs (ro,relatime)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
none on /dev/pts type devpts (rw,relatime,mode=600)
/dev/sda8 on /data type ext3 (rw,relatime,errors=remount-ro,barrier=1,data=ordered)
/dev/sda9 on /spool type ext3 (rw,relatime,errors=remount-ro,barrier=1,data=ordered)
none on /spool/ipsfp type tmpfs (rw,relatime,size=4194304k)

In this example, no issues can be seen.
 
Note The / partition (root) is always mounted as read-only.

busybox

Common linux utilities packaged into one executable. Below are examples from nslookup, arping and du to test DNS functionality, perform duplicate IP address detection and check folder sizes.

busybox nslookup    

Performs name system lookup. Useful for troubleshooting web filtering issues or any other problem related to functionality where the NGFW engine needs to do DNS queries.

root@node:~# busybox nslookup forcepoint.com
Server:    10.10.10.10
Address 1: 10.10.10.10 dns.example.com

Name:      forcepoint.com
Address 1: 54.191.140.180 ec2-54-191-140-180.us-west-2.compute.amazonaws.com

busybox arping

With arping, you can determine if there are duplicate MAC addresses in the network segment.

root@node:~# busybox arping -b -I eth3.102 192.168.10.11
ARPING to 192.168.10.11 from 192.168.10.12 via eth3.102
Unicast reply from 192.168.10.11 [0:10:f3:18:19:19] 0.208ms
Unicast reply from 192.168.10.11 [0:c:29:2e:d6:b7] 0.268ms
Unicast reply from 192.168.10.11 [0:10:f3:18:19:19] 0.208ms
Unicast reply from 192.168.10.11 [0:c:29:2e:d6:b7] 0.246ms
Unicast reply from 192.168.10.11 [0:10:f3:18:19:19] 0.203ms
Unicast reply from 192.168.10.11 [0:c:29:2e:d6:b7] 0.239ms
Unicast reply from 192.168.10.11 [0:10:f3:18:19:19] 0.200ms
Unicast reply from 192.168.10.11 [0:c:29:2e:d6:b7] 0.256ms

You can see above there is a duplicate IP address in the network since the reply comes from two different MACs.

busybox du -s * | sort -n

With a combination of du and sort, you can determine more specifically where in a partition the space is being reserved (refer to df command for information on checking partitions). Here the focus is on /spool partition. By default du reports file and folder sizes in kilobytes.

root@node:/# cd /spool
root@node:/spool# busybox du -s * | sort -n
...
8       urlcatd
12      eth0cap.cap
16      lost+found
20      blacklist
20      insp_recordings
24      dhcp-server
84      logserver_policy
224     monitoring
344     log
3296    dump
24900   clamav
72152   av
root@node:/spool# cd av
root@node:/spool/av# busybox du -s * | sort -n
4       scan
72144   data
root@node:/spool/av# cd data
root@node:/spool/av/data# busybox du -s * | sort -n
4       av_db_version.mon
4       dbfetch.mon
72132   manupdate
root@node:/spool/av/data# cd manupdate/
root@node:/spool/av/data/manupdate# busybox du -s * | sort -n
4       oem.ini
8       legal.txt
400     avvnames.dat
764     avvclean.dat
70952   avvscan.dat
root@node:/spool/av/data/manupdate#

No issue is listed seen in the above output, /spool utilization is within normal range. It shows that /spool/av/data/manupdate is the largest directory under /spool and that avvscan.dat is the largest file there.
Alternatively all file and folder sizes in /spool could be listed in one go with command busybox du -am /spool | sort -n, sorted by size in megabytes in ascending order.

top

Shows statistics in real-time from engine CPU and memory utilization. Can be used to quickly see what process is using the most resources on engine. To see CPU statistics per CPU, press 1 when top is running.
 
scp

Used to copy files securely from/to the engine. Syntax example below:
   
scp SOURCE DESTINATION
scp filename.pcap root@10.10.10.10:/home/root/

 
If you need to check your NGFW health status, Please refer the KB Health check on NGFW




Keywords: ngfw; troubleshooting utilities; troubleshooting commands; health check

Article Feedback



Thank you for the feedback and comments.