HotSync

Nota

Questo pacchetto non è supportato in NethServer Enterprise

Avvertimento

Il moduflo HotSync è da considerarsi una release beta. Pertanto è opportuno provarla in ambienti di test prima di procedere in produzione.

Avvertimento

For a correct restore, it’s suggested to configure HotSync on two identical servers or two servers with same network cards number, name and position. If the master and slave servers differ, the restore procedure may behave unexpectedly (see Troubleshooting).

HotSync mira a ridurre i tempi di inattività in caso di problemi, sincronizzando il NethServer con un altro sistema gemello, che verrà attivato manualmente in caso di guasto del server master.

Normalmente, quando si verifica un problema hardware, il tempo necessario per ripristinare il servizio è:

  1. riparazione/approvvigionamento nuovo server: da 4h a 2 giorni
  2. installazione sistema operativo: 30 minuti
  3. ripristino backup: da 10 minuti a 8 ore

In summary, users are able to start working again with data from the night before failure after a few hours/days. Using HotSync, time 1 and 3 are 0, 2 is 5 minutes (time to activate spare server). Users are able to start working again in few minutes, using data from a few minutes before the crash.

Per impostazione predefinita, tutti i dati inclusi nel backup vengono sincronizzati ogni 15 minuti. Anche i database MariaDB sono sincronizzati, a meno che la sincronizzazione dei database non sia disabilitata. Le applicazioni che utilizzano PostgreSQL sono sincronizzate (Mattermost, Webtop5) a meno che la sincronizzazione dei database non sia disabilitata.

Terminologia

  • MASTER è il sistema di produzione SLAVE è il server di riserva
  • SLAVE è acceso, con un indirizzo IP diverso da MASTER
  • Ogni 15 minuti, MASTER esegue un backup su SLAVE
  • If an error occurs, an email is sent to root (admin if mail server is installed)
  • SLAVE check updates and makes some system operations every 60 minutes

Installazione

Install nethserver-hotsync on both MASTER and SLAVE from Software Center or execute from command line:

yum install -y nethserver-hotsync --enablerepo=nethforge

Configurazione

You can configure HotSync from Cockpit interface: access it from Master and Slave, select role and fill required fields with password and IP. The <PASSWORD> must be the same on master and slave.

You can also configure HotSync from command line using these commands:

Master

[[email protected]]# config setprop rsyncd password <PASSWORD>
[[email protected]]# config setprop hotsync role master
[[email protected]]# config setprop hotsync SlaveHost <SLAVE_IP>
[[email protected]]# signal-event nethserver-hotsync-save

Slave

[[email protected]]# config setprop rsyncd password <PASSWORD>
[[email protected]]# config setprop hotsync role slave
[[email protected]]# config setprop hotsync MasterHost <MASTER_IP>
[[email protected]]# signal-event nethserver-hotsync-save

If mysql or postgresql are installed, they will be synchronized by default. You can disable databases sync from Master Cockpit interface or from command line on master machine with this command:

[[email protected]]# config setprop hotsync databases disabled
[[email protected]]# signal-event nethserver-hotsync-save

Nota

If you are using HotSync to restore FreePBX leave databases enabled, otherwise FreePBX database will not be restored properly.

Abilitazione/Disabilitazione

HotSync is enabled by default. To disable it uncheck the checkbox into HotSync Cockpit GUI or use this command:

[[email protected]]# config setprop hotsync status disabled
[[email protected]]# signal-event nethserver-hotsync-save

and to re-enable it re-check the checkbox on interface or use CLI:

[[email protected]]# config setprop hotsync status enabled
[[email protected]]# signal-event nethserver-hotsync-save

Nota

After HotSync is configured, it’s a good practice to launch hotsync command on master host. After master has properly syncronized, access the slave and execute hotsync-slave. You can force these commands also from Cockpit GUI and check /var/log/messages logs. As best practice, the first syncrhonization should be done via command line to better check if everything is properly configured.

Avvertimento

After HotSync is configured and hotsync command executed properly, note that hotsync-slave command must be executed at least one time before proceed with hotsync-promote. You can launch it manually or wait 60 minutes for automatic execution.

Ripristino: promuovere lo SLAVE in produzione

La seguente procedura mette in produzione lo SLAVE in caso di arrewsto anomalo del master.

  1. Switch off MASTER.

  2. If the SLAVE machine must run as network gateway, connect it to the router/modem with a network cable.

  3. On SLAVE, if you are connected through an SSH console, launch the screen command, to make your session survive to network outages:

    As best practice, execute following procedure using a local console and not via SSH connection.

  4. lanciare sullo SLAVE il seguente comando e leggere attentamente l’output generato

    [[email protected]]# hotsync-promote
    

    If no Internet connection is detected (e.g. you are restoring a firewall on a machine that was passing through crashed master for Internet connection), the scripts will purpose you some options

    1. Restore master network configuration (IMPORTANT: use this option only if two servers are identical - NIC number, names and positions must be identical)
    2. Fix network configuration from Cockpit GUI (when restoring on different hardware)
    3. Continue without internet: assign correct roles before proceed with this option. Some events could fails (not recommended)
    

    else restore will start automatically. If you are restore on different hardware you could encounter DC errors.

Avvertimento

When restoring on identical hardware choose option 1 and network configuration will be overwritten, else choose option 2. It’s not recommended to start the promote procedure without Internet access. When restoring on a different hardware and you’ve choosed option 2, you can encounter DC errors. Please see Troubleshooting.

  1. If necessary go to Server Manager or Cockpit GUI, in page Network and reassign roles to network interfaces as master one. Remember also to recreate bridge if you have configured DC. In case of DC errors consult troubleshooting section before proceed with network restore.

  2. After everything has been restored, launch the command

    [[email protected]]# /sbin/e-smith/signal-event post-restore-data
    
  3. Update the system to the latest packages version

    [[email protected]]# yum clean all && yum -y update
    
  4. If an USB backup is configured on MASTER, connect the backup HD to SLAVE

Troubleshooting

After restore on different hardware DC is not working

Console could report some errors like these

[ERROR] /usr/libexec/nethserver/sambads: failed to add service primaries to system keytab
Action: /etc/e-smith/events/nethserver-mail-server-update/S50nethserver-sssd-initkeytabs FAILED

To solve this, restore network configuration as master (including bridges) and then launch

/sbin/e-smith/signal-event nethserver-dc-save
/sbin/e-smith/signal-event nethserver-sssd-save

After restore permissions on ibays are not correct

Restore permissions from Cockpit GUI, under File Server, open shared folder menu and click on Restore permissions.

After network restore server is unreachable

If you cannot reach server after a network reconfiguration, check configuration and, if it’s correct, try launching this commands

/sbin/e-smith/signal-event interface-update
/sbin/e-smith/signal-event nethserver-firewall-base-update

If you cannot reach the server yet, use network-recovery tool.

Suggested check after restore

When all issues have been solved, please make that: - configuration is restored properly - all enabled services are working - applications interfaces (e.g. freepbx, webtop) are working - file server is working and users can log into shared folders - email server is working and users can send and receive emails - asterisk is working and users can make calls

Finally, reboot the system and check all services are working after boot.

Pacchetti supportati

All nethserver packages are supported. Here is a list of major NethServer packages:

  • nethserver-antivirus
  • nethserver-backup-config
  • nethserver-backup-data
  • nethserver-base
  • nethserver-c-icap
  • nethserver-cockpit
  • nethserver-collectd
  • nethserver-cups
  • nethserver-dante
  • nethserver-dc
  • nethserver-dedalo
  • nethserver-directory
  • nethserver-dnsmasq
  • nethserver-duc
  • nethserver-ejabberd
  • nethserver-evebox
  • nethserver-fail2ban
  • nethserver-firewall-base
  • nethserver-freepbx > 14.0.3
  • nethserver-httpd
  • nethserver-hylafax
  • nethserver-iaxmodem
  • nethserver-ipsec-tunnels
  • nethserver-janus
  • nethserver-letsencrypt
  • nethserver-lightsquid
  • nethserver-mail
  • nethserver-mattermost
  • nethserver-mysql
  • nethserver-ndpi
  • nethserver-netdata
  • nethserver-nextcloud
  • nethserver-ntopng
  • nethserver-nut
  • nethserver-openssh
  • nethserver-openvpn
  • nethserver-pulledpork
  • nethserver-restore-data
  • nethserver-roundcubemail
  • nethserver-samba
  • nethserver-samba-audit
  • nethserver-squid
  • nethserver-squidclamav
  • nethserver-squidguard
  • nethserver-sssd
  • nethserver-subscription
  • nethserver-suricata
  • nethserver-vpn-ui
  • nethserver-vsftpd
  • nethserver-webtop5 (lo stato di z-push non è sincronizzato)

Packages nethserver-ntopng and nethserver-evebox are reinstalled without migrating history.

Avvertimento

To avoid errors on the slave host, do not make any changes to the modules from the Cockpit GUI except the HotSync module.