Troubleshooting Overview¶

In general you can monitor the probe's behavior by watching the log file it generates as well as the standard output it generates to the console. Here we document common errors with the probe.

tail -f /mnt/probe/log/XXXX.log

Troubleshooting Overview

Launcher Hang¶

In rare cases the probe will complete its successfully but the python launcher script will hang. If this happens, simply control-C the launcher if you are still attached to the terminal. Or use ps -ef to find the probe process and kill it.

CPU Compatibility¶

Occasionally, the probe will launch and finish without scanning any files and may produce an error related to log files. When viewing logs you may see:

probe terminated by signal: SIGILL

Check the CPU Compatibility:

cat /sys/devices/cpu/caps/pmu_name

Review the CPU Requirements.

CGroup Error¶

You may get the following error on older Linux builds:

Loading docker image
Starting probe docker container
783735ffbf1a722ebbcd43622476ea2364a8873dc2a6f95a4d006778636ed513
/usr/bin/docker-current: Error response from daemon: oci runtime error: container_linux.go:235: starting container process caused "process_linux.go:258: applying cgroup configuration for process caused \"Cannot set property TasksAccounting, or unknown property.\"".
Failed starting the probe

If you do you need to update the systemd related packages. Here are the versions we use:

rpm -qa |grep -i systemd
systemd-libs-219-67.el7_7.2.x86_64
systemd-sysv-219-67.el7_7.2.x86_64
systemd-219-67.el7_7.2.x86_64
oci-systemd-hook-0.2.0-1.git05e6923.el7_6.x86_64

Privilege Error¶

When the container is launched it may fail with this error message:

Starting probe docker container

docker: Error response from daemon: privileged mode is incompatible with user namespaces.  You must run the container in the host namespace when running privileged mode.

This message is a warning that your docker environment will not allow docker to run with heightened permissions. This probably caused by a more secure docker configuration, such as placing the following text into the /etc/docker/daemon.json file which basically prevents a container from running as root with privileges:

{
 "userns-remap": "dockremap:dockremap"
}

The easiest way to address that is to hand edit the VAST provided probe_launcher.py and look for the 'docker run' line. It will look something like this:

cmd = f'docker run --privileged -v {args.metadata_dir}:/probe_mnt/{args.metadata_dir} -v {args.output_dir}:/probe_mnt/{args.output_dir} '

Notice the --privileged. Remove it, save the file, and try again. If that doesn't fix the issue, we've found that removing the userns-remap line and restarting the docker service can be more effective.

Illegal Instruction¶

If the probe fails with a core dump and it shows an illegal instruction, you may be using an old CPU type which is not compatible with our compiled code. We require Intel Broadwell or newer compatible CPUs.

If you use GDB to debug the core you can confirm by looking for something like this:

GDB output from core:
$ gdb -c core.97296 


For help, type "help".
Type "apropos word" to search for commands related to "word".
Core was generated by `/vast/install/probe/sim_estimator --similarity-function fast_hash_8 --split-win'.
Program terminated with signal SIGILL, Illegal instruction.
#0  0x00007fffec9deea7 in ?? ()
(gdb)