TSCM Backup & Restore

This page documents the backup and restore functions introduced in the ThreatSTOP Centralized Manager (TSCM) as of TSCM-Core v1.47.

To check to see what version you are on run tsadmin version --all. If you’re on a version older than v1.47, you can simply run sudo apt-get update && sudo apt-get dist-upgrade -y to download all the newest ThreatSTOP software packages.

threatstop@tsclient /opt/threatstop/backup » tsadmin version --all
[INFO ] : ThreatSTOP Versions:
Module               Version
-------------------- ----------
  TSCM-Core          1.50-01
  ts-keyring         

Network considerations

• The TSCM works under the assumption its network configuration will not change, either by DHCP reservation or static IP address configuration. If the device leased its IP Address via DHCP reservation, ensure the current operating IP address has been re-assigned to the new TSCM.
  • We’ve bundled a network configuration utility along with the TSCM. Simply run tsadmin network to start the configuration.
• The restore process attempts to restore the static network configuration detected by the backup.
  • If you’ve customized the network configuration to use anything other than eth0, please pass the --interface argument to the backup & restore commands.
• If you change IP Addresses and over-ride the restore behavior to bypass the network configuration, keep in mind you’ll have to update the devices logging settings and any other artifacts that may still be pointing to the previous TSCMs IP Address.

TSCM Migration

If you intend migrating from one TSCM to another, the following steps are a best practices guide.

1. Create snapshots of your TSCM(s) if your hypervisor supports them.
2. Download, boot and fully update the latest TSCM available (http://downloads.threatstop.com/TSCM.ova).
3. Update the current TSCM to the latest ThreatSTOP packages available. sudo apt-get update && sudo apt-get dist-upgrade
4. On the current TSCM, disable the cron entry and wait until there are no ts-update processes running on the system. See example below. Do not proceed while ts-update processes are running.
5. Either reconfigure the new TSCM with the network configuration from the current TSCM or proceed with the restore which attempts to do it for you.
6. After restore is complete, update each device to ensure communication and credentials are working as expected.

Disable cron and safety check

# Ensure no ts-update processes are running
ps -ef|grep ts-update|grep -v grep
# disable cron from running any further updates/log-uploads etc..
sudo mv /etc/cron.d/multidevice-core ~/multidevice-core.cron.d
# to restore sudo mv ~/multidevice-core.cron.d /etc/cron.d/multidevice-core

Backup

The backup feature creates an archive file containing configuration files, log files & some system information such as detected network settings.

To run the backup process SSH into the TSCM as the threatstop user and run:

tsadmin backup

On your first time invoking the backup, you will be asked if you would like to encrypt the backups. If you agree you’ll be prompted to enter in an encryption passphrase. This is recommended as the backups contain sensitive information.

Backup Considerations

A backup is just an archive of configuration files at the current date & time it was taken. It is advisable to make regular backups as settings are changed.

• Consider the size of the backups when scheduling backup automation ls -lh /opt/threatstop/backup
• Consider storing the backup configuration files off the TSCM.
  • Consider encryption if the backups are stored off the TSCM
• If you’ve configured a network setup not involving the standard network interface eth0, make sure you pass the interface in via the --interface=ethX argument where ethX corresponds to the customized interface.

Setting up a cron task

The following shell script snippet will create a scheduled task to backup & purge old backups.

cat | sudo tee /etc/cron.d/tscm_backup << EOF
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │
# │ │ │ │ │
# * * * * * <user to run task as> <command to execute>
# periodic TSCM backups on the 1st and 15th of every month
55 * 1,15 * *  threatstop  /usr/local/bin/ts-backup > /var/log/threatstop/ts-backup.log 2>&1
# purge backups older than 120 days old
59 * 1 * *  threatstop  find /opt/threatstop/backup -type f -name "*.tsba*" -mtime +120 -exec rm '{}' \;
EOF

Adjust the backup retention and backup intervals as you see fit, but remember to be mindful of disk space allocation.

Creating off-device backups

The backup archives are stored in /opt/threatstop/backup. The backup files are named threatstop_backup_YYYY-MM-DD_hh_mm_ss.tsbak[.gpg] where YYYY is the four digit year, MM the two digit month, DD the two digit day, hh_mm_ss the time in UTC offset.

We automatically create a symlink to the latest backup file which can be located at threatstop_latest.tsbak.gpg for encrypted backups or threatstop_latest.tsbak for non-encrypted backups. This makes it easier to script an external task to fetch the backup via a pre-determined backup filename.

Restore

• Before beginning the restore process, it’s advisable to fully update the distribution packages. sudo apt-get update && sudo apt-get dist-upgrade
• Next, copy the backup archive to either the threatstop users home directory or the /opt/threatstop/backup/ directory so it can be indexed by the restore process. The Posix permissions should be set to threatstop:threatstop and mode 0664.
• Now you’re ready to run the restore process by typing tsadmin restore as the threatstop user.
• The restore proceedure starts by checking the network configuration on the TSCM. If the IP Address detected on the system is not the IP address stored in the backup, you will be prompted to update the network configuration, reboot and start the restore process again.
• The first prompt will show you a list of backups it was able to index. Type in the number corresponding to the backup you wish to restore.
• Next the restore will prompt if you’d like to restore the SSH configuration. This would contain the SSH host keys of the system the backup took place on.
• You will be prompted if you’d like to restore the Cron configuration files. You should only do this if you’ve made modifications to the standard Cron schedule/settings. If in doubt, choose no.
• The restore process attempts to restore all devices. If there are any issues during restore you will get a message outlining what they were.
• You should run tsadmin update <device name> on each device that was restored to ensure the restore worked as expected.

Restore Considerations

• Ensure the TSCM the backup came from is offline before beginning the restore process, otherwise we may have contention over the IP address.
• It is advisable to create a snapshot if your VM hypervisor supports it, before attempting to restore.
• The TSCM attempting to run the restore should be assigned the same IP Address as the original TSCM. The restore process attempts to do this as part of the restore.
• The restore will be as up-to-date as the last time you backed it up.
• The restore process will automatically update certain devices that pull threat intelligence from the TSCM. However most devices such as (Cisco ASA/ISR, Fortigate) will have to be updated manually since the update could take considerable amount of time depending on the policy size.
• The restore process has the ability to restore uniquely generated SSH host keys in the event you have existing infrastructure expecting the same SSH fingerprint.
• You can run the restore process more than once, but note that it will warn you about proceeding to restore on a TSCM if it already has devices setup.
• Device configurations will be overwritten without confirmation. E.g. If you have a device already installed named test-asa and the restore has a device configuration for test-asa, it will overwrite local configuration files if you proceed with the restore after the initial warning & confirmation.
• While it is possible to restore without applying the network configuration, it is generally a good idea to attempt to use the same IP Address since it will likely be used for logging and/or possibly auth setup.

Restore without applying network configuration

To bypass applying the network configuration add --bypass_netconf=1 to the tsadmin restore command. While not advisable, this may be necessary for certain edge cases. Keep in mind you’ll have to update Firewall device configuration to ensure logging is pointing to the new updated IP address, and communication between the TSCM and the device is working ok.

Utility options