What is it ?¶
NethServer HotSync aims to reduce downtime in case of failure. Normally, when an hardware damage occurs, the time needed to restore service is:
- fix or buy another server: from 4h to 2 days
- install the OS: 30 minutes
- restore backup: from 10 minutes to 8 hours
In summary, users are able to start working again with data from the night before failure after a few hours or days. Using HotSync, time 1 and 3 become 0 and 2 becomes 5 minutes (time to activate the slave server). Users are able to start working again in few minutes, using data from a few minutes before the crash.
Two server machines are needed:
- MASTER: active server on production
- SLAVE: warm backup server
How it works¶
MASTER machine is the currently production server and it makes a backup in the SLAVE machine every “n” minutes. When a crash occurs on the master, slave becomes active taking the same IP address of the master. Doing so the downtime is minimal, only the time to launch some commands on the SLAVE.
On both MASTER and SLAVE, install nethserver-hotsync:
# yum install nethserver-hotsync
[root@master]# config setprop rsyncd password <PASSWORD> [root@master]# config setprop hotsync role master [root@master]# config setprop hotsync SlaveHost <SLAVE_IP> [root@master]# signal-event nethserver-hotsync-save
[root@slave]# config setprop rsyncd password <PASSWORD> [root@slave]# config setprop hotsync role slave [root@slave]# config setprop hotsync MasterHost <MASTER_IP> [root@slave]# signal-event nethserver-hotsync-save
The <PASSWORD> must be the same on both master and slave.
If MySQL or PostgreSQL are installed, they will be synchronized by default. To disable databases synchronization:
[root@master]# config setprop hotsync databases disabled [root@master]# signal-event nethserver-hotsync-save
How to restore¶
The following procedure puts the SLAVE in production when the master has crashed.
switch off MASTER
if the SLAVE machine must run as network gateway, connect it to the router/modem with a network cable
following command changes the IP address and cuts off the network connection. If you are connected through an ssh console, launch the
on SLAVE launch command, and read carefully its output
go to Server Manager page
Networkand reassign roles to network interfaces if required
[root@slave]# /sbin/e-smith/signal-event post-restore-data
if an USB backup is configured on MASTER, connect the backup HD to SLAVE
How to restore original server¶
To put again in production original crashed server:
configure it as SLAVE
switch off current MASTER
from bash execute:
restore configuration backup
How to synchronize custom paths¶
It is possible to customize HotSync adding all kind of data through the use of plugins. The directory of plugins is /etc/hotsync.d. Executable files in this directory, are executed before the synchronization.
INCLUDE_FILE and EXCLUDE_FILE are the files that contain the list of paths to include and to exclude to/from HotSync. Those two variables are passed as arguments to the scripts of this directory when they are executed.
If you want to add files to the synchronization, append them to the INCLUDE_FILE. Append them to EXCLUDE_FILE to remove from the synchronization.
How to force synchronization from MASTER to SLAVE¶
From MASTER bash launch the command:
How to force packages installation on SLAVE¶
“hotsync-slave” script extracts from MASTER configuration backup the list of packages to install and install them. You can force the operation executing bash command:
is a shell script launched by cron every 15 minutes
uses a lockfile to ensure that only one instance at a time is executed
uses secure communication with rsync over stunnel
creates a list of files to be included and another one to be excluded from rsync (using backup-data configuration and hotsync own logic)
launch backup on MASTER
launch an rsync that copy listed files from MASTER to SLAVE in a secure manner using stunnel
if something fails, root is notified with an email
you can check files that will be copied on the next synchronization using the command:
Automatically executed on SLAVE every 60 minutes, extracts from MASTER configuration backup the list of packages to install and install them.
- nethserver-freepbx > 14.0.3
- nethserver-webtop5 (z-push state is not synchronized)
Not supported packages¶
HotSync management using Cockpit Graphical Interface¶
It can be possible to administrate HotSync from cockpit web graphical interface installing nethserver-cockpit-hotsync.
Configuration using Cockpit Web Gui¶
- On both MASTER and SLAVE browse to cockpit web gui -> “Applications” -> “NethServer Hotsync” -> “Settings”
- select the “role”, then insert the requested data and click “Save” button
Restore using Cockpit Web Gui¶
- From SLAVE browse to cockpit web gui -> “Applications” -> “NethServer Hotsync” -> “Settings”
- click on “Promote to Master” button