User-Defined List Utility | ThreatSTOP Dochub

This document provides documentation for managing user-defined lists using the ThreatSTOP UDL Utility.

Overview

The ThreatSTOP UDL Utility helps you manage (add, replace, view) your User-Defined Block / Allow lists without having to form API requests. The utility takes parameterized input and interacts with the API directly.

User Defined Lists (UDL) overview

User-Defined Lists allow you to add your own records to your Policies.
Records can be IP addresses (for use in IP and DNS Policies), or Domains (for use in DNS Policies).
User-Defined Lists can be used for blocking or whitelisting purposes.
After creating a User-Defined list, attach it to one or several of your custom policies.
User-defined Lists are restricted to a maximum of 32,000 records each. Multiple user-defined lists can be attached to a policy.

Supported Operating Systems

Windows 7 / 8 / 10 (x86 / x64)
MacOS 10.13 High Sierra / 10.14 Mojave

Downloading the utility

Operating System	SHA256
Windows / MacOS	bfcc835001b1d437d76e6d26a06fad1cce7730fa44ab8fbb98b5efa823dd3764

Getting UDL Utility parameter usage

You can retrieve the usage information by running ThreatSTOP-UDL-Util --help

usage: ThreatSTOP-UDL-Util [-h] [--source SOURCE] --type {ip,domain} --key KEY
                           [--list LIST] --action {view,add,replace,list}
                           [--verbose] [--silent]

ThreatSTOP User-Defined List Utility

optional arguments:
  -h, --help            show this help message and exit
  --source SOURCE       File name or URL (IPs/domains to load)
  --type {ip,domain}    'ip' or 'domain'
  --key KEY             ThreatSTOP REST API Key
  --list LIST           Custom List Name
  --action {view,add,replace,list}
  --verbose             Show HTTP request details
  --silent              Silent mode (only output data - no info messages)

We will go over each parameter below, except “-h, –help” which is described here.

SOURCE - Input source file / URL format

The source parameter can read from a local file or a URL. The contents of the file/URL should be as follows:

One entry per line
One comma per line
Comma delimited fields
Newline terminated lines
IP or Domain (depending on list type)
IP addresses may include a CIDR netmask (a.b.c.d/n)
Comment is an optional field (string up to 1024 characters)

Example:

1.2.3.4, Bad Actor
1.2.3.0/30, Malicious Network

Creating a new local IP source file

lines beginning with a ‘#’ will be ignored

source file contents:

cat > deny_ips.csv << EOF
64.87.3.133,bad.threatstop.com
# this comment will be ignored
116.31.116.41/32, Malicious Behavior
EOF

Creating a new Domain local source file

lines beginning with a ‘#’ will be treated as comments

source file contents

cat > deny_domains.csv << EOF
# most policies contain the following domain to test the policy is working
bad.threatstop.com, test deny rule
EOF

TYPE - UDL type of list

There are two types of user defined lists - one type for IP addresses and one for Domains. They are managed independently.
Domain lists can only be used in DNS Defense (RPZ) policies, while IP lists can be used in both IP Defense and DNS Defense policies.
The typical workflow consists of:
1. Creating the user list, a one time operation, and retrieving its Name
2. Assigning it to a policy (not covered in this example, can be done via API or Portal)
This utility allows you to add entries to a UDL, view entries from a UDL, or Replacing entire contents of a UDL.

KEY - UDL API key

A ThreatSTOP REST API key is assigned Read/Write permissions to various portal objects
Keys are generated using the ThreatSTOP Admin Portal User interface (API Keys page)
The API key is tied to an account, and can manage the lists from the associated account. To manage lists in multiple accounts, a key from each account must be used.
API Endpoint: https://rest.threatstop.com/v4.0/
Authentication error codes
- The API returns HTTP/401 if the key is missing
- The API returns HTTP/403 if the key is invalid or doesn’t have access to the requested resource

LIST - Custom List Name

The utility references the UDL by its list name.
See Listing UDLs below.

ACTION - to be performed on the UDL.

View - a user defined list contents

You can retrieve the contents of an existing User-defined list with the ‘view’ action.
You will need an API key with read access to User Defined Lists.

$ ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action view --list DenyUDL --type ip
2018-11-08 16:24:05,321 Retrieve UDL List (ip)
2018-11-08 16:24:05,494 REST request successful
2018-11-08 16:24:05,494 Found 1 UDLs
DenyUDL,xx5x58x7-xxx5-424x-x9xx-892x76x8xx66,7
2018-11-08 16:24:05,494 Retrieve UDL Content (xx5x58x7-xxx5-424x-x9xx-892x76x8xx66)
2018-11-08 16:24:05,671 REST request successful
#entry,comments
116.31.116.41,
132.232.185.49,

Creating a source file from output of an existing UDL

You can redirect the output of the utility to re-create a local source file.

./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action view --list DenyUDL --type ip --silent > deny.csv

before attemping to use the source file make sure the first line starts with “#entry,comments”.

Add - entries to an existing UDL

The following example demonstrates adding IPs/Domains to an existing UDL.
You will need an API key with write access to User Defined Lists.
You will need an IP/Domain source file to read rules from.

./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action add --list DenyUDL --type ip --source deny.csv

Replace - entire contents of a UDL

Replacing the entire contents of an existing UDL is done with the “replace” action.
It is recommended to backup your UDL before beginning. See Creating a source file from and existing UDL.
You will need an API key with write access to User Defined Lists.

./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action replace --list DenyUDL --type ip --source deny.csv
2018-11-09 13:38:14,460 Retrieve UDL List (ip)
2018-11-09 13:38:14,630 REST request successful
2018-11-09 13:38:14,630 Found 1 UDLs
DenyUDL,xx5x58x7-xxx5-424x-x9xx-892x76x8xx66,9
2018-11-09 13:38:14,631 Update UDL Content (xx5x58x7-xxx5-424x-x9xx-892x76x8xx66)
2018-11-09 13:38:14,851 REST request successful

Deleting entries in a UDL

While the utility doesn’t allow deleting directly, you can save the current UDL to a local file. Remove entries you wish to delete, then replace the UDL.

List - IP / Domain UDL names

Listing UDLs requires a valid API key.
Type of UDL (ip or domain)

$ ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action list --type ip
2018-11-08 16:07:28,213 Retrieve UDL List (ip)
2018-11-08 16:07:28,566 REST request successful
2018-11-08 16:07:28,567 Found 1 UDLs
#Name,List_id,Records
denyUDL,<UDL UUID>,<# of Records>

The UDL Name would be “denyUDL” in the example above.

VERBOSE

Outputs additional HTTP debugging data.

...
REQUEST: GET /v4.0//user_ip_lists HTTP/1.1
REQUEST: Connection: keep-alive
REQUEST: Accept-Encoding: gzip, deflate
REQUEST: Accept: */*
RESPONSE: Content-Type: application/json
RESPONSE: Content-Length: 910
RESPONSE: Connection: keep-alive
RESPONSE: Cache-Control: no-store, no-cache, must-revalidate, max-age=0
RESPONSE: Pragma: no-cache
...

SILENT

only outputs data without informational messages / logging data.

Troubleshooting

If the command encounters an error, the request will not be completed. In other words if anything fails, no changes will be committed.

Error response codes

Response
- HTTP/201 (CREATED) on success.
- HTTP/400 if the request could not be processed
- HTTP/500 for REST server errors

➜  ~ ./ThreatSTOP-UDL-Util --key x12x3x151x4x8107x4x2x286x97x48xxxx97820x895391x1x12x57x535x7xx8x --action add --list DenyUDL --type ip --source deny.csv