This page documents the backup and restore functions introduced in the ThreatSTOP Centralized Manager (TSCM) as of TSCM-Core v1.47.
To check to see what version you are on run tsadmin version --all
. If you’re on a version older than v1.47, you can simply run sudo apt-get update && sudo apt-get dist-upgrade -y
to download all the newest ThreatSTOP software packages.
threatstop@tsclient /opt/threatstop/backup » tsadmin version --all
[INFO ] : ThreatSTOP Versions:
Module Version
-------------------- ----------
TSCM-Core 1.50-07
ts-keyring
Network considerations
- The TSCM works under the assumption its network configuration will not change, either by DHCP reservation or static IP address configuration. If the device leased its IP Address via DHCP reservation, ensure the current operating IP address has been re-assigned to the new TSCM.
- We’ve bundled a network configuration utility along with the TSCM. Simply run
tsadmin network
to start the configuration.
- We’ve bundled a network configuration utility along with the TSCM. Simply run
- The restore process attempts to restore the static network configuration detected by the backup.
- If you’ve customized the network configuration to use anything other than
eth0
, please pass the--interface
argument to the backup & restore commands.
- If you’ve customized the network configuration to use anything other than
- If you change IP Addresses and over-ride the restore behavior to bypass the network configuration, keep in mind you’ll have to update the devices logging settings and any other artifacts that may still be pointing to the previous TSCMs IP Address.
TSCM Migration
If you intend migrating from one TSCM to another of the same base operating system (OS), the following steps are a best practices guide.
- Create snapshots of your TSCM(s) if your hypervisor supports them.
- Download, boot and fully update the latest TSCM available (http://downloads.threatstop.com/TSCM.ova).
- Update the current TSCM to the latest ThreatSTOP packages available.
sudo apt-get update && sudo apt-get dist-upgrade
- On the current TSCM, disable the cron entry and wait until there are no
ts-update
processes running on the system. See example below. Do not proceed whilets-update
processes are running. Check running processes by callingsudo ps -augx | grep ts-update
. - Either reconfigure the new TSCM with the network configuration from the current TSCM or proceed with the restore which attempts to do it for you.
- After restore is complete, update each device to ensure communication and credentials are working as expected.
Disable cron and safety check
# Ensure no ts-update processes are running
ps -ef|grep ts-update|grep -v grep
# disable cron from running any further updates/log-uploads etc..
sudo mv /etc/cron.d/multidevice-core ~/multidevice-core.cron.d
# to restore sudo mv ~/multidevice-core.cron.d /etc/cron.d/multidevice-core
Backup
The backup feature creates an archive file containing configuration files, log files & some system information such as detected network settings.
To run the backup process SSH into the TSCM as the threatstop
user and run:
tsadmin backup
On your first time invoking the backup, you will be asked if you would like to encrypt the backups. If you agree you’ll be prompted to enter in an encryption passphrase. This is recommended as the backups contain sensitive information.
Backup Considerations
A backup is just an archive of configuration files at the current date & time it was taken. It is advisable to make regular backups as settings are changed.
- Consider the size of the backups when scheduling backup automation
ls -lh /opt/threatstop/backup
- Consider storing the backup configuration files off the TSCM.
- Consider encryption if the backups are stored off the TSCM
- If you’ve configured a network setup not involving the standard network interface
eth0
, make sure you pass the interface in via the--interface=ethX
argument where ethX corresponds to the customized interface.
Setting up a cron task
The following shell script snippet will create a scheduled task to backup & purge old backups.
cat | sudo tee /etc/cron.d/tscm_backup << EOF
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │ 7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * <user to run task as> <command to execute>
# periodic TSCM backups on the 1st and 15th of every month
55 * 1,15 * * threatstop /usr/local/bin/ts-backup > /var/log/threatstop/ts-backup.log 2>&1
# purge backups older than 120 days old
59 * 1 * * threatstop find /opt/threatstop/backup -type f -name "*.tsba*" -mtime +120 -exec rm '{}' \;
EOF
Adjust the backup retention and backup intervals as you see fit, but remember to be mindful of disk space allocation.
Creating off-device backups
/opt/threatstop/backup/threatstop_latest.tsbak.gpg
. This will not contain any backup information as it only points to the latest backup. Backup archives will contain a date like this: /opt/threatstop/backup/threatstop_backup_2024-04-19_20_04_19.tsbak.gpg
.The backup archives are stored in /opt/threatstop/backup
. The backup files are named threatstop_backup_YYYY-MM-DD_hh_mm_ss.tsbak[.gpg]
where YYYY is the four digit year, MM the two digit month, DD the two digit day, hh_mm_ss the time in UTC offset.
We automatically create a symlink to the latest backup file which can be located at threatstop_latest.tsbak.gpg
for encrypted backups or threatstop_latest.tsbak
for non-encrypted backups. This makes it easier to script an external task to fetch the backup via a pre-determined backup filename.
To get the path for the latest backup archive run readlink /opt/threatstop/backup/threatstop_latest.tsbak*
as shown below:
threatstop@tsclient:$ readlink /opt/threatstop/backup/threatstop_latest.tsbak*
/opt/threatstop/backup/threatstop_backup_2024-04-19_20_04_19.tsbak.gpg
Once you have the path you can transfer the backup several ways. Below we’ll demonstrate several methods. Choose one of the methods below to get the backup files from the old TSCM to the new TSCM.
tsadmin backup
command.METHOD 1. Sending backups from current TSCM to new TSCM via RSYNC
The following method requires the rsync utility. It is installed by default on the TSCM Ubuntu based appliances but is optional on the RHEL appliances. To install the rsync utility run the following:
RHEL7 - sudo yum install rsync
RHEL9+ - sudo dnf install rsync
# sending the backups from the current TSCM to the new TSCM
export CURRENT_TSCM_IP_ADDRESS=10.1.1.43
export NEW_TSCM_IP_ADDRESS=10.1.1.50
rsync -av threatstop@$CURRENT_TSCM_IP_ADDRESS:/opt/threatstop/backup threatstop@$NEW_TSCM_IP_ADDRESS:/opt/threatstop/
METHOD 2. Copying all backups from current TSCM to new TSCM using TAR and SSH
The following commands are meant to be run while logged in via SSH to the new TSCM.
# replace 10.1.1.50 with the real internal IP address of the new TSCM
# replace 10.1.1.43 with the real internal IP address of the current TSCM
export CURRENT_TSCM_IP_ADDRESS=10.1.1.43
export NEW_TSCM_IP_ADDRESS=10.1.1.50
if [ ! -d /opt/threatstop/backup ];then
echo "creating /opt/threatstop/backup"
mkdir -p /opt/threatstop/backup
chown -R threatstop:threatstop /opt/threatstop/backup
fi
cd /opt/threatstop && ssh threatstop@$NEW_TSCM_IP_ADDRESS "cd /opt/threatstop && tar czvf - backup" | tar xzvf -
METHOD 3. Copying your backup from your current TSCM to the new TSCM via SCP
- log into your current TSCM
- run the commands below
export NEW_TSCM_IP_ADDRESS=10.1.1.50 scp $(readlink /opt/threatstop/backup/threatstop_latest.tsbak*) threatstop@$NEW_TSCM_IP_ADDRESS
METHOD 4. Copying an individual backup
The following commands should be run on your workstation. Note, you’ll need to have scp
utility installed on your machine to follow along. Windows users you can install winscp.
- The following command will pull a specific backup from the TSCM
- replace
threatstop_backup_2024-04-19_20_04_19.tsbak
with your actual backup filename found by running thereadlink
command above. ```bash export CURRENT_TSCM_IP_ADDRESS=10.1.1.43 scp threatstop@$CURRENT_TSCM_IP_ADDRESS:/opt/threatstop/backup/threatstop_backup_2024-04-19_20_04_19.tsbak .
Now you'll need to reverse that command to send it to the new TSCM.
```bash
export NEW_TSCM_IP_ADDRESS=10.1.1.50
scp *.tsbak* threatstop@$NEW_TSCM_IP_ADDRESS:/opt/threatstop/backup/
Restore
- Before beginning the restore process, it’s advisable to fully update the distribution packages.
sudo apt-get update && sudo apt-get dist-upgrade
- Next, copy the backup archive to either the
threatstop
users home directory or the/opt/threatstop/backup/
directory so it can be indexed by the restore process. The Posix permissions should be set tothreatstop:threatstop
and mode 0664. - Now you’re ready to run the restore process by typing
tsadmin restore
as the threatstop user. - The restore proceedure starts by checking the network configuration on the TSCM. If the IP Address detected on the system is not the IP address stored in the backup, you will be prompted to update the network configuration, reboot and start the restore process again.
- The first prompt will show you a list of backups it was able to index. Type in the number corresponding to the backup you wish to restore.
- Next the restore will prompt if you’d like to restore the SSH configuration. This would contain the SSH host keys of the system the backup took place on.
- You will be prompted if you’d like to restore the Cron configuration files. You should only do this if you’ve made modifications to the standard Cron schedule/settings. If in doubt, choose
no
. - The restore process attempts to restore all devices. If there are any issues during restore you will get a message outlining what they were.
- You should run
tsadmin update <device name>
on each device that was restored to ensure the restore worked as expected.
Restore Considerations
- Ensure the TSCM the backup came from is offline before beginning the restore process, otherwise we may have contention over the IP address.
- It is advisable to create a snapshot if your VM hypervisor supports it, before attempting to restore.
- The TSCM attempting to run the restore should be assigned the same IP Address as the original TSCM. The restore process attempts to do this as part of the restore.
- The restore will be as up-to-date as the last time you backed it up.
- The restore process will automatically update certain devices that pull threat intelligence from the TSCM. However most devices such as (Cisco ASA/ISR, Fortigate) will have to be updated manually since the update could take considerable amount of time depending on the policy size.
- The restore process has the ability to restore uniquely generated SSH host keys in the event you have existing infrastructure expecting the same SSH fingerprint.
- You can run the restore process more than once, but note that it will warn you about proceeding to restore on a TSCM if it already has devices setup.
- Device configurations will be overwritten without confirmation. E.g. If you have a device already installed named
test-asa
and the restore has a device configuration fortest-asa
, it will overwrite local configuration files if you proceed with the restore after the initial warning & confirmation. - While it is possible to restore without applying the network configuration, it is generally a good idea to attempt to use the same IP Address since it will likely be used for logging and/or possibly auth setup.
Restore without applying network configuration
To bypass applying the network configuration add --bypass_netconf=1
to the tsadmin restore
command. While not advisable, this may be necessary for certain edge cases. Keep in mind you’ll have to update Firewall device configuration to ensure logging is pointing to the new updated IP address, and communication between the TSCM and the device is working ok.
Utility options
Option | Description |
---|---|
--interface |
network interface Default: eth0 |
--action |
[backup, restore] Default: backup |
--gpg_passphrase |
encryption / decryption passphrase |
--gpg_passphrase_file |
encryption / decryption passphrase file |
--debug |
[1..7] Default: 5 |
--bypass_netconf=1 |
allows restore without applying network config |
--leave_tmp_files=1 |
leaves tmp logs for debugging |
--force_apply_netconf=1 |
attempts to write network configuration even if IP already detected on system |