Overview
I’ve installed and integrated Oxidized on CTWUG services. It does automated configaration backup and tracking or all routers that are on NMS which are essentially all OSPF routers.
This is a replacement for our old script based backup process as discussed here.
How it works
- NMS has a script running adding any OSPF router ID (with a valid DNS entry) onto NMS. So all OSPF RBs with SNMP configured should appear on NMS.
- Oxidized pulls a list of all routers from NMS and tries to login to it over SSH. It exports he config as a text file.
- Oxidized removes any sensitive information from this text file.
- It then compares with the existing config from that router for differences and then updates if these are different.
- It rechecks every router at least once every 24h. This is not at a scheduled time though.
- It also monitors logs submitted to nms and rescans when a config might have changed.
Logging in
The software logs in using the ctwug login, but I plan to make a specific wms login for all WMS funcationality going forward. This is also to enable login to other router types.
Mikrotik
For Mikrotik this is done via the following command:
/export hide-sensitive
This will export the whole config as a RSC script.
It excludes any passwords or secrets and also excludes user info.
The script also enables gametime (on the extracted export only, not on the router). So all backups will appear with gametime scripts enabled. This is to avoid causing new backups everytime gametime switches on or off (I did not start with this so there is probably one or two gametime flips on each router’s history).
EdgeOS
For EdgeOS it runs:
show configuration
It scrubs some secret stuff after pulling this.
Other
It supports many other router models including other UBNT OS.
Viewing Information
The information can be viewed in 3 places:
Oxidized Web
Available at https://oxidized.ctwug.za.net/. This is only accessible on the WUG. This lists all routers by hostname (as per NMS).
You can search for your router and then click on the icons on the right.
gives you the last backed up config.
takes you to a screen where you can see versions and DIFFs between different versions.
queues up the router for the config to checked and updated ASAP.
NMS
LibreNMS offers integration with Oxidized. All routers now have an additional menu item called config where both the current config and previous versions are available.
Gitlab ctwug/oxidized repository
As Oxidized processes config changes essentially in the same way as git it’s quite easy to setup a repository that follows this. I have done so at the above location.
Restoring
Generally speaking this is done by finding the relevant config and loading or running it on the relevant router. Generally one needs to pay attention to the OS version to make sure the script is run on exactly the same OS version as from which it was backed up.
Mikrotik
- Find the config on one of the three sources above and save it.
- On Oxidized click on the
and save the file as something.rsc. You can select older versions also if you want. - On Gitlab find the file in the repository and click on it. There is then a “Open Raw” button (
). Click on it and save as something.rsc. - On NMS find the relevant router and go to the config screen. Select the text in the grey box and copy and paste into a text edit. Save it as
something.rsc. (Note that routers that have been down for some time are deleted from NMS, so it may longer appear here, but will still be available in gitlab.)
- Make sure the router you are restoring to has the same ROS version as the script. Otherwise it may cause errors. Small differences might not be a problem, but watch out for errors when running the script.
- Reset the router you are trying to restore to default configuration. The script assumes a default configuration as a base.
- Copy the
something.rscscript to the router using your method of choice (e.g. drag into WinBox,scpor FTP). - Open a terminal on the router and type
/import something.rsc. If it produces no output it has worked. - Reset the things that the script does not backup:
- Create users you need as the script does not backup users
- Add OSPF authentication-key under
/routing ospf interfacesas this key is dropped from script. - Run WMS join script again to add
ctwuguser and add radius secrets etc.
Other Routers
We can add specific instructions as we need but generally:
- Obtain the config as per above saving as a file to your PC. You may need to rename it with a different extension.
- Load the configuration file on the router.
- Run / load the configuration file using the relevant command on the router.
Passwords / Sensitive Information
If you store passwords and sensitive information in scripts or comments please be aware that this will be included and accessible to other wuggers. Please keep this in mind. The scripts only remove sensitive information in the obvious places (like secret/key/password fields).
This would be accessible anyway via ctwug login on routers. So you should anyway not store sensitive information there.
Plans
- Please test this and provide feedback here.
- Currently running both systems but I plan to deactivate the backup scripts if this process proves itself.
- Create a standard wms user on all routers which is only to be used by WMS.
- Add other supported routers to NMS (e.g. UBNT). I’ve been testing things on erx.puck. Thanks / sorry @JellyBean.
- Add wms logins to these supported routers.
- Add log monitoring to rescan configuration when log messages such as
device changed by spinappear.
Wiki
This post is editable by anyone so please update if I’ve missed stuff.
