We have written some scripts that you can use to get your ThreatSTOP block lists. Download the scripts here.
The script queries the ThreatSTOP DNS server and stores the results in a file that PF can use to build a table. The resulting file is a list of IP addresses in CIDR format. You can then create a rule that will block access to and from the table.
If this is a new device, please allow up to 15 minutes for our systems to be updated.
The only prerequisite for OpenBSD systems is one Perl module and its dependencies. These are available as binary packages for OpenBSD. To install the modules run the following as root:
pkg install p5-libwww
This will install the LWP Perl module and any dependencies it defines.
- threatstop-pf.sh: The main script. It downloads the block lists and creates the files PF uses to populate a table.
- loguploadclient.pl: Perl script that uploads the log file
- sendlog.sh: Converts the PF log file to plain text and calls loguploadclient.pl to upload the script.
- install.sh: Installation script.
- test.sh: Script to run a quick test to make sure the block lists are ready to be downloaded.
- apl.sed: Supporting sed script to parse DNS APL query results.
- prt.sed: Supporting sed script to parse DNS PTR query results.
- threatstop.conf.example: Example configuration file.
The IP address provided in the configuration sample (18.104.22.168) while accurate, is not currently supported by BSD Packet Filtering. We apologize for the inconvenience. In the interim please provide the IP address 22.214.171.124, this is our legacy DNS server but will work with the current script.
After downloading and extracting the script, copy the configuration from the ThreatSTOP website into a file named “threatstop.conf” in the same directory where you extracted the downloaded file. There are some settings in the configuration that you will need to verify:
- OUT_DIR: Directory where the files will be saved that PF will use to build the tables.
- BLOCK_FILE: Location and name of the file with IP addresses to block.
- ALLOW_FILE: Location and name of the file with IP addresses to allow.
- BLOCK_TABLE: The name of the PF table that will hold the IP address to block.
- ALLOW_TABLE: The name of the PF table that will hold the IP addresses to allow.
- RELOAD_PF: The script can automatically flush and reload the ThreatSTOP tables.
- logfile: Temporary file used by the sendlog.sh script.
- url: URL to upload the plain text output of the log file.
Copy the configuration below to a file named “threatstop.conf” in the same directory where the download was extracted.
ThreatSTOP Configuration file
# Final location of the files PF will use for the tables OUT_DIR="/var/db" # ThreatSTOP DNS Servers # You can add or remove entries. Each server must be separated by a space DNS_SERVERS="126.96.36.199" # The block and/or allow list BLOCK_LIST=<block list name>.<ThreatSTOP account ID>.threatstop.local ALLOW_LIST=<allow list name>.<ThreatSTOP account ID>.threatstop.local # The name of the files pf will use for the tables BLOCK_FILE="$OUT_DIR/tsblock.txt" ALLOW_FILE="$OUT_DIR/tsallow.txt" # The name of the table for PF BLOCK_TABLE="ThreatSTOP" ALLOW_TABLE="ThreatSTOP-Allow" # Automatically reload PF after the files are created RELOAD_PF="YES" # # Options for the sendlog.sh script # # Location and name of the PF log file PFLOG="/var/log/pflog.0" # Name of the log file to upload. File is created # and deleted by the sendlog.sh script logfile="/tmp/pflog.txt" # URL to upload the logs to url=https://www.threatstop.com/cgi-bin/logupload.pl
Before running the script it needs to be installed. There is a install.sh script that needs to be run. The script copies the threatstop-pf.sh, sendlog.sh, loguploadclient.pl, ptr.sed, and apl.sed to /usr/local/sbin. It copies the threatstop.conf file to /usr/local/etc.
In order to have the block lists updated at regular intervals, it creates a cron job that will run the script every two hours. It also creates a job to run the sendlog.sh script to upload the log file every night at 1:00 AM.
To have the logs uploaded on a daily basis, the script modifies the /etc/newssyslog.conf file so that the /var/log/pflog is rotated every night at 12:00 AM. In the newsyslog.conf file, it changes:
/var/log/pflog 600 3 250 * ZB "pkill -HUP -u root -U root -t - -x pflogd"
/var/log/pflog 600 3 * $D0 ZB "pkill -HUP -u root -U root -t - -x pflogd"
As root, run the the “install.sh” script:
./install.sh /usr/local/sbin does not exist. Creating. Copying scripts to "/usr/local/sbin" Copying configuration file to "/usr/local/etc" Adding crontab entry to run the "threatstop-pf.sh" script every 2 hours and upload the log file with the "sendlog.sh" script every day at 1:00 AM Backing up original crontab to "crontab.original" Changing the newsyslog.conf file to rotate pflog every day at 12:00 AM Creating backup of newsyslog.conf
Before running the main script to get your block lists and create the files, we need to make sure that your block lists are ready. We have included the “test.sh” script that you can run to make sure everything is ready. If the test does not work within an hour of creating your device, please contact support for assistance.
./test.sh DNS Server 188.8.131.52 is ready. The test was successful. You are now ready to run the threatstop-pf.sh script.
After making sure the test script works, you can now run the main script, threatstop-pf.sh. The script gets your block lists and creates the files that PF can use to populate a table. The files are saved to /var/db. In order for PF to load any updates to the ThreatSTOP lists, PF needs to flush and reload the table. In the configuration file, there is an option to reload PF after the block lists have been downloaded. The commands that are run to do this are:
# threatstop-pf.sh Starting ThreatSTOP v1.3 update on Fri Aug 12 07:42:05 PDT 2016 Using DNS Server 184.108.40.206 Processing allow list <allow list name>.<ThreatSTOP account ID>.threatstop.local Processing block list <block list name>.<ThreatSTOP account ID>.threatstop.local Completed getting all the lists Adding 26424 blocked addresses Adding 3 allowed addresses Reloading pf...done Finished ThreatSTOP update at Fri Aug 12 07:42:15 PDT 2016 Run Length: 0 hour(s) 0 minute(s) 10 second(s)
This will create the files “tsblock.txt” and tsallow.txt” in the /var/db directory.
Once the files containing the addresses to block are created, you will need to configure PF to use the file as a table. In the /etc/pf.conf file, add the table definition:
table <ThreatSTOP> persist file "/var/db/tsblock.txt" table <ThreatSTOP-Allow> persist file "/var/db/tsallow.txt"
With the table defined, you can create the rules to block traffic to and from the addresses in the table. We recommend that the rules to block the ThreatSTOP table be placed before any rules that allow incoming or outgoing traffic. The “quick” directive tells PF to treat the rule as the last matching rule. Any rules after it are not evaluated.
pass in quick from <ThreatSTOP-Allow> pass out quick to <ThreatSTOP-Allow> block drop in log quick from <ThreatSTOP> block drop out log quick to <ThreatSTOP>
After you make the changes to the /etc/pf.conf file, PF will need to be reloaded to read the updated configuration:
/sbin/pfctl -f /etc/pf.conf
To view the addresses in the table, run the command:
# pfctl -T show -t ThreatSTOP # pfctl -T show -t ThreatSTOP-Allow
To view the updated rules, run the command:
/sbin/pfctl -s rules
Sending Your Logs
We have a log parsing feature where we can take your firewall log, parse it and compare the source and destination IP addresses to what is in our database. You can then login to our website and see the results. This allows you, and us, to see how effective we are in protecting your network infrastructure.
The log file written by PF is in binary format and must be converted before it is sent to ThreatSTOP. We have included a script that converts the log to plain text and uploads the file to the secure ThreatSTOP website.
The installation script has already configured your system to run the sendlog.sh script every night at 1:00 AM.
If you would like to upload a log now, run the following command as root:
/usr/local/sbin/sendlog.sh Log file uploaded successfully
Restore to Previous State
If you decide to return to your pre-ThreatSTOP configuration, you will need to perform the following actions to disable and remove ThreatSTOP from your system:
- Stop the VM from updating the firewall by deleting the user crontab:
- Remove the ThreatSTOP address groups from the policies using them (or delete the policies completely).
- Delete the ThreatSTOP address groups (TSBlock-(number) and TSAllow-(number)).