TSCM Backup & Restore

This page documents the backup and restore functions introduced in the ThreatSTOP Centralized Manager (TSCM) as of TSCM-Core v1.47.

To check to see what version you are on run tsadmin version --all. If you’re on a version older than v1.47, you can simply run sudo apt-get update && sudo apt-get dist-upgrade -y to download all the newest ThreatSTOP software packages.

threatstop@tsclient /opt/threatstop/backup » tsadmin version --all
[INFO ] : ThreatSTOP Versions:
Module               Version
-------------------- ----------
  TSCM-Core          1.50-01
  ts-keyring         

Network considerations

• The TSCM works under the assumption its network configuration will not change, either by DHCP reservation or static IP address configuration. If the device leased its IP Address via DHCP reservation, ensure the current operating IP address has been re-assigned to the new TSCM.
  • We’ve bundled a network configuration utility along with the TSCM. Simply run tsadmin network to start the configuration.
• The restore process attempts to restore the static network configuration detected by the backup.
  • If you’ve customized the network configuration to use anything other than eth0, please pass the --interface argument to the backup & restore commands.
• If you change IP Addresses and over-ride the restore behavior to bypass the network configuration, keep in mind you’ll have to update the devices logging settings and any other artifacts that may still be pointing to the previous TSCMs IP Address.

TSCM Migration

If you intend migrating from one TSCM to another of the same base operating system (OS), the following steps are a best practices guide.

1. Create snapshots of your TSCM(s) if your hypervisor supports them.
2. Download, boot and fully update the latest TSCM available (http://downloads.threatstop.com/TSCM.ova).
3. Update the current TSCM to the latest ThreatSTOP packages available. sudo apt-get update && sudo apt-get dist-upgrade
4. On the current TSCM, disable the cron entry and wait until there are no ts-update processes running on the system. See example below. Do not proceed while ts-update processes are running. Check running processes by calling sudo ps -augx | grep ts-update.
5. Either reconfigure the new TSCM with the network configuration from the current TSCM or proceed with the restore which attempts to do it for you.
6. After restore is complete, update each device to ensure communication and credentials are working as expected.

Disable cron and safety check

# Ensure no ts-update processes are running
ps -ef|grep ts-update|grep -v grep
# disable cron from running any further updates/log-uploads etc..
sudo mv /etc/cron.d/multidevice-core ~/multidevice-core.cron.d
# to restore sudo mv ~/multidevice-core.cron.d /etc/cron.d/multidevice-core

Backup

The backup feature creates an archive file containing configuration files, log files & some system information such as detected network settings.

To run the backup process SSH into the TSCM as the threatstop user and run:

tsadmin backup

On your first time invoking the backup, you will be asked if you would like to encrypt the backups. If you agree you’ll be prompted to enter in an encryption passphrase. This is recommended as the backups contain sensitive information.

Backup Considerations

A backup is just an archive of configuration files at the current date & time it was taken. It is advisable to make regular backups as settings are changed.

• Consider the size of the backups when scheduling backup automation ls -lh /opt/threatstop/backup
• Consider storing the backup configuration files off the TSCM.
  • Consider encryption if the backups are stored off the TSCM
• If you’ve configured a network setup not involving the standard network interface eth0, make sure you pass the interface in via the --interface=ethX argument where ethX corresponds to the customized interface.

Setting up a cron task

The following shell script snippet will create a scheduled task to backup & purge old backups.

cat | sudo tee /etc/cron.d/tscm_backup << EOF
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * <user to run task as> <command to execute>
# periodic TSCM backups on the 1st and 15th of every month
55 * 1,15 * *  threatstop  /usr/local/bin/ts-backup > /var/log/threatstop/ts-backup.log 2>&1
# purge backups older than 120 days old
59 * 1 * *  threatstop  find /opt/threatstop/backup -type f -name "*.tsba*" -mtime +120 -exec rm '{}' \;
EOF

Adjust the backup retention and backup intervals as you see fit, but remember to be mindful of disk space allocation.

Creating off-device backups

The backup archives are stored in /opt/threatstop/backup. The backup files are named threatstop_backup_YYYY-MM-DD_hh_mm_ss.tsbak[.gpg] where YYYY is the four digit year, MM the two digit month, DD the two digit day, hh_mm_ss the time in UTC offset.

We automatically create a symlink to the latest backup file which can be located at threatstop_latest.tsbak.gpg for encrypted backups or threatstop_latest.tsbak for non-encrypted backups. This makes it easier to script an external task to fetch the backup via a pre-determined backup filename.

To get the path for the latest backup archive run readlink /opt/threatstop/backup/threatstop_latest.tsbak* as shown below:

threatstop@tsclient:$ readlink /opt/threatstop/backup/threatstop_latest.tsbak*
/opt/threatstop/backup/threatstop_backup_2024-04-19_20_04_19.tsbak.gpg

Once you have the path you can transfer the backup several ways. Below we’ll demonstrate several methods. Choose one of the methods below to get the backup files from the old TSCM to the new TSCM.

METHOD 1. Sending backups from current TSCM to new TSCM via RSYNC

The following method requires the rsync utility. It is installed by default on the TSCM Ubuntu based appliances but is optional on the RHEL appliances. To install the rsync utility run the following:

RHEL7 - sudo yum install rsync RHEL9+ - sudo dnf install rsync

# sending the backups from the current TSCM to the new TSCM 
export CURRENT_TSCM_IP_ADDRESS=10.1.1.43
export NEW_TSCM_IP_ADDRESS=10.1.1.50
rsync -av threatstop@$CURRENT_TSCM_IP_ADDRESS:/opt/threatstop/backup threatstop@$NEW_TSCM_IP_ADDRESS:/opt/threatstop/

METHOD 2. Copying all backups from current TSCM to new TSCM using TAR and SSH

The following commands are meant to be run while logged in via SSH to the new TSCM.

# replace 10.1.1.50 with the real internal IP address of the new TSCM
# replace 10.1.1.43 with the real internal IP address of the current TSCM
export CURRENT_TSCM_IP_ADDRESS=10.1.1.43
export NEW_TSCM_IP_ADDRESS=10.1.1.50
if [ ! -d /opt/threatstop/backup ];then
    echo "creating /opt/threatstop/backup"
    mkdir -p /opt/threatstop/backup
    chown -R threatstop:threatstop /opt/threatstop/backup
fi
cd /opt/threatstop && ssh threatstop@$NEW_TSCM_IP_ADDRESS "cd /opt/threatstop && tar czvf - backup" | tar xzvf - 

METHOD 3. Copying your backup from your current TSCM to the new TSCM via SCP

1. log into your current TSCM
2. run the commands below

   export NEW_TSCM_IP_ADDRESS=10.1.1.50
   scp $(readlink /opt/threatstop/backup/threatstop_latest.tsbak*) threatstop@$NEW_TSCM_IP_ADDRESS
   

METHOD 4. Copying an individual backup

The following commands should be run on your workstation. Note, you’ll need to have scp utility installed on your machine to follow along. Windows users you can install winscp.

• The following command will pull a specific backup from the TSCM
• replace threatstop_backup_2024-04-19_20_04_19.tsbak with your actual backup filename found by running the readlink command above. ```bash export CURRENT_TSCM_IP_ADDRESS=10.1.1.43 scp threatstop@$CURRENT_TSCM_IP_ADDRESS:/opt/threatstop/backup/threatstop_backup_2024-04-19_20_04_19.tsbak .

Now you'll need to reverse that command to send it to the new TSCM.
```bash
export NEW_TSCM_IP_ADDRESS=10.1.1.50
scp *.tsbak* threatstop@$NEW_TSCM_IP_ADDRESS:/opt/threatstop/backup/

Restore

• Before beginning the restore process, it’s advisable to fully update the distribution packages. sudo apt-get update && sudo apt-get dist-upgrade
• Next, copy the backup archive to either the threatstop users home directory or the /opt/threatstop/backup/ directory so it can be indexed by the restore process. The Posix permissions should be set to threatstop:threatstop and mode 0664.
• Now you’re ready to run the restore process by typing tsadmin restore as the threatstop user.
• The restore proceedure starts by checking the network configuration on the TSCM. If the IP Address detected on the system is not the IP address stored in the backup, you will be prompted to update the network configuration, reboot and start the restore process again.
• The first prompt will show you a list of backups it was able to index. Type in the number corresponding to the backup you wish to restore.
• Next the restore will prompt if you’d like to restore the SSH configuration. This would contain the SSH host keys of the system the backup took place on.
• You will be prompted if you’d like to restore the Cron configuration files. You should only do this if you’ve made modifications to the standard Cron schedule/settings. If in doubt, choose no.
• The restore process attempts to restore all devices. If there are any issues during restore you will get a message outlining what they were.
• You should run tsadmin update <device name> on each device that was restored to ensure the restore worked as expected.

Restore Considerations

• Ensure the TSCM the backup came from is offline before beginning the restore process, otherwise we may have contention over the IP address.
• It is advisable to create a snapshot if your VM hypervisor supports it, before attempting to restore.
• The TSCM attempting to run the restore should be assigned the same IP Address as the original TSCM. The restore process attempts to do this as part of the restore.
• The restore will be as up-to-date as the last time you backed it up.
• The restore process will automatically update certain devices that pull threat intelligence from the TSCM. However most devices such as (Cisco ASA/ISR, Fortigate) will have to be updated manually since the update could take considerable amount of time depending on the policy size.
• The restore process has the ability to restore uniquely generated SSH host keys in the event you have existing infrastructure expecting the same SSH fingerprint.
• You can run the restore process more than once, but note that it will warn you about proceeding to restore on a TSCM if it already has devices setup.
• Device configurations will be overwritten without confirmation. E.g. If you have a device already installed named test-asa and the restore has a device configuration for test-asa, it will overwrite local configuration files if you proceed with the restore after the initial warning & confirmation.
• While it is possible to restore without applying the network configuration, it is generally a good idea to attempt to use the same IP Address since it will likely be used for logging and/or possibly auth setup.

Restore without applying network configuration

To bypass applying the network configuration add --bypass_netconf=1 to the tsadmin restore command. While not advisable, this may be necessary for certain edge cases. Keep in mind you’ll have to update Firewall device configuration to ensure logging is pointing to the new updated IP address, and communication between the TSCM and the device is working ok.

Utility options