This document provides documentation for managing user-defined lists using the ThreatSTOP UDL Utility.

Overview

The ThreatSTOP UDL Utility helps you manage (add, replace, view) your User-Defined Block / Allow lists without having to form API requests. The utility takes parameterized input and interacts with the API directly.

User Defined Lists (UDL) overview

  • User-Defined Lists allow you to add your own records to your Policies.
  • Records can be IP addresses (for use in IP and DNS Policies), or Domains (for use in DNS Policies).
  • User-Defined Lists can be used for blocking or whitelisting purposes.
  • After creating a User-Defined list, attach it to one or several of your custom policies.
  • User-defined Lists are restricted to a maximum of 10,000 records each. Multiple user-defined lists can be attached to a policy.

Downloading the utility

Operating System Platform SHA256
Windows TBD 547d693f7a549b8a65d3e6c35747cfa04dc38d55700189a4035c2a92ad1a1001
MacOS TBD d7d4cd0fbb46d6a0ce1fe47dc7885ffa2c81d35837d23fa2e53277fe2c2443a7

Getting UDL Utility parameter usage

  • You can retrieve the usage information by running ThreatSTOP-UDL-Util --help
usage: ThreatSTOP-UDL-Util [-h] [--source SOURCE] --type {ip,domain} --key KEY
                           [--list LIST] --action {view,add,replace,list}
                           [--dry-run] [--verbose] [--silent]

ThreatSTOP User-Defined List Utility

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       File name or URL (IPs/domains to load)
  --type {ip,domain}    'ip' or 'domain'
  --key KEY             ThreatSTOP REST API Key
  --list LIST           Custom List Name
  --action {view,add,replace,list}
  --dry-run             Dry-run (no changes)
  --verbose             Show HTTP request details
  --silent              Silent mode (only output data - no info messages)

We will go over each parameter below, except “-h, –help” which is described here.

SOURCE - Input source file / URL format

The source parameter can read from a local file or a URL. The contents of the file/URL should be as follows:

  • One entry per line
  • One comma per line
  • Comma delimited fields
  • Newline terminated lines
  • IP or Domain (depending on list type)
  • IP addresses may include a CIDR netmask (a.b.c.d/n)
  • Comment is an optional field (string up to 1024 characters)

Example:

1.2.3.4, Bad Actor
1.2.3.0/30, Malicious Network

Creating a new local IP source file

  • lines beginning with a ‘#’ will be ignored
  • source file contents:
    cat > deny_ips.csv << EOF
    64.87.3.133,bad.threatstop.com
    # this comment will be ignored
    116.31.116.41/32, Malicious Behavior
    EOF
    

Creating a new Domain local source file

  • lines beginning with a ‘#’ will be treated as comments
  • source file contents
    cat > deny_domains.csv << EOF
    # most policies contain the following domain to test the policy is working
    bad.threatstop.com, test deny rule
    EOF
    

TYPE - UDL type of list

  • There are two types of user defined lists - one type for IP addresses and one for Domains. They are managed independently.
  • Domain lists can only be used in DNS Defense (RPZ) policies, while IP lists can be used in both IP Defense and DNS Defense policies.
  • The typical workflow consists of:
    1. Creating the user list, a one time operation, and retrieving its Name
    2. Assigning it to a policy (not covered in this example, can be done via API or Portal)
  • This utility allows you to add entries to a UDL, view entries from a UDL, or Replacing entire contents of a UDL.

KEY - UDL API key

  • A ThreatSTOP REST API key is assigned Read/Write permissions to various portal objects
  • Keys are generated using the ThreatSTOP Admin Portal User interface (API Keys page)
  • The API key is tied to an account, and can manage the lists from the associated account. To manage lists in multiple accounts, a key from each account must be used.
  • API Endpoint: https://rest.threatstop.com/v4.0/
  • Authentication error codes
    • The API returns HTTP/401 if the key is missing
    • The API returns HTTP/403 if the key is invalid or doesn’t have access to the requested resource

LIST - Custom List Name

  • The utility references the UDL by its list name.
  • See Listing UDLs below.

ACTION - to be performed on the UDL.

View - a user defined list contents

  • You can retrieve the contents of an existing User-defined list with the ‘view’ action.
  • You will need an API key with read access to User Defined Lists.
$ ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action view --list DenyUDL --type ip
2018-11-08 16:24:05,321 Retrieve UDL List (ip)
2018-11-08 16:24:05,494 REST request successful
2018-11-08 16:24:05,494 Found 1 UDLs
DenyUDL,xx5x58x7-xxx5-424x-x9xx-892x76x8xx66,7
2018-11-08 16:24:05,494 Retrieve UDL Content (xx5x58x7-xxx5-424x-x9xx-892x76x8xx66)
2018-11-08 16:24:05,671 REST request successful
#entry,comments
116.31.116.41,
132.232.185.49,

Creating a source file from output of an existing UDL

  • You can redirect the output of the utility to re-create a local source file.
    ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action view --list DenyUDL --type ip --silent > deny.csv
    
  • before attemping to use the source file make sure the first line starts with “#entry,comments”.

Add - entries to an existing UDL

  • The following example demonstrates adding IPs/Domains to an existing UDL.
  • You will need an API key with write access to User Defined Lists.
  • You will need an IP/Domain source file to read rules from.
./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action add --list DenyUDL --type ip --source deny.csv

Replace - entire contents of a UDL

  • Replacing the entire contents of an existing UDL is done with the “replace” action.
  • It is recommended to backup your UDL before beginning. See Creating a source file from and existing UDL.
  • You will need an API key with write access to User Defined Lists.
./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action replace --list DenyUDL --type ip --source deny.csv
2018-11-09 13:38:14,460 Retrieve UDL List (ip)
2018-11-09 13:38:14,630 REST request successful
2018-11-09 13:38:14,630 Found 1 UDLs
DenyUDL,xx5x58x7-xxx5-424x-x9xx-892x76x8xx66,9
2018-11-09 13:38:14,631 Update UDL Content (xx5x58x7-xxx5-424x-x9xx-892x76x8xx66)
2018-11-09 13:38:14,851 REST request successful

Deleting entries in a UDL

While the utility doesn’t allow deleting directly, you can save the current UDL to a local file. Remove entries you wish to delete, then replace the UDL.

List - IP / Domain UDL names

  • Listing UDLs requires a valid API key.
  • Type of UDL (ip or domain)
$ ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action list --type ip
2018-11-08 16:07:28,213 Retrieve UDL List (ip)
2018-11-08 16:07:28,566 REST request successful
2018-11-08 16:07:28,567 Found 1 UDLs
#Name,List_id,Records
denyUDL,<UDL UUID>,<# of Records>

The UDL Name would be “denyUDL” in the example above.

DRY-RUN

This parameter allows you to test a command without actually making changes.

Below is an example of an error due to an invalid source file format. The dry-run flag is useful for testing.

$ ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action replace --list denyUDL --type ip --source deny.csv --dry-run
2018-11-09 13:43:22,955 Retrieve UDL List (ip)
2018-11-09 13:43:23,099 REST request successful
2018-11-09 13:43:23,099 Found 1 UDLs
denyUDL,<UDL UUID>,<# of Records>
ValueError: too many values to unpack
[31488] Failed to execute script

VERBOSE

Outputs additional HTTP debugging data.

...
REQUEST: GET /v4.0//user_ip_lists HTTP/1.1
REQUEST: Connection: keep-alive
REQUEST: Accept-Encoding: gzip, deflate
REQUEST: Accept: */*
RESPONSE: Content-Type: application/json
RESPONSE: Content-Length: 910
RESPONSE: Connection: keep-alive
RESPONSE: Cache-Control: no-store, no-cache, must-revalidate, max-age=0
RESPONSE: Pragma: no-cache
...

SILENT

  • only outputs data without informational messages / logging data.

Troubleshooting

  • If the command encounters an error, the request will not be completed. In other words if anything fails, no changes will be committed.

Error response codes

  • Response
    • HTTP/201 (CREATED) on success.
    • HTTP/400 if the request could not be processed
    • HTTP/500 for REST server errors
➜  ~ ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action add --list DenyUDL --type ip --source deny.csv