Developer Manual

See also

• Web site
• Community
• Wiki
• Administrator manual

Introduction

NethServer is an Open Source operating system for the Linux enthusiast, designed for small offices and medium enterprises. It’s simple, secure and flexible.

NethServer is ready to deliver your messages, to protect your network with the built-in firewall, share your files and much more, everything on the same system.

Target audience

This manual is intended for developers who are interested about NethServer internals. Reading this manual you should be able to learn how to extend, enpower and debug the NethServer platform.

Get involved

The NethServer project welcomes anyone who would like to become involved in the project. This is a brief list of things to do (in a sparse order):

• Writing documentation
• Bug reporting, bug triaging, QA testing
• Translating the web interface
• Developing / coding in Perl, PHP, Python, Bash

Note

This manual is a work in progress. If you can’t find some information please check also our wiki: http://wiki.nethserver.org and always feel free to ask on http://community.nethserver.org

Development process

Issues

An issue is a formal description of a known problem, or desired feature, inside a tracker. There are two kind of issues:

Bug

describes a defect of the software; it must lead to a resolution of the problem. For example, a process crashing under certain conditions.

Feature/Enhancement

describes an improvement of the current code or an entirely new feature. For example, remove a harmless warning of a running process or designing a new UI panel.

Bugs and enhancements will always produce some code changes inside one or more git repositories.

Each repository is associated with one or more RPM packages. Changes to the code produce new releases of RPM packages.

Do I need to open a new issue?

Yes, if what you’re talking about will produce some code. By the way, it’s perfectly reasonable not to fill issues for occasional small fixes, like typos in translations.

When implementing small fixes, always avoid commits to the master branch. Open a Pull requests and carefully describe the problem. Creation of issues can be avoided only for trivial fixes which require no QA effort.

Issues are not a TODO list. Issues track the status changes of a job, the output of the job will be a new RPM resolving the issue itself. If you are exploring some esoteric paths for new feature or hunting something like a heisenbug , please write a draft wiki page with your thoughts, then create a new issue only when you’re ready to write a formal description and produce some output object.

A process for a new feature should be something like this:

• Open a new topic on http://community.nethserver.org and discuss it.
• If the feature is complex, a dedicated wiki page could be written on https://github.com/NethServer/dev/wiki (or http://wiki.nethserver.org/).
  • Create a wiki page with notes and thoughts (team contributions are welcome!).
  • If the wiki page is a feature design document, the feature can simply point to the wiki page.
  • The wiki page should become a changelog for a new release.
  • When the wiki page is pretty “stable” and the whole thing is well outlined, a team member will create one or more new issues.
• Open the issue on GitHub https://github.com/NethServer/dev/issues/new.

Writing issues

How to write a bug report:

• Apply the bug label
• Point to the right software component with the associated version
• Describe the error, and how to reproduce it
• Describe the expected behavior
• If possible, suggest a fix or workaround
• If possible, add a piece of system output (log, command, etc)
• Text presentation matters: it makes the whole report more readable and understandable

How to write a feature or enhancement:

• Everybody should understand what you’re talking about: describe the feature with simple words using examples
• If possible, add links to external documentation
• Text presentation and images matter: they make the whole report more readable and understandable

More information:

• https://developer.mozilla.org/en-US/docs/Mozilla/QA/Bug_writing_guidelines
• http://fedoraproject.org/wiki/Bugs_and_feature_requests
• http://fedoraproject.org/wiki/How_to_file_a_bug_report

Issue priority

Bugs should always have priority over features.

The priority of a bug depends on:

• security: if it’s a security bug, it should have maximum priority
• number of affected users: more affected users means more priority

Issue tracker

The NethServer project is hosted on GitHub and is constituted by many git repositories. We set one of them to be the issue tracker:

https://github.com/NethServer/dev

Issues created on dev help coordinating the development process, determining who is in charge of what.

Issue labels

Issues can be tagged using a set of well-known labels:

• bug: a defect of the software
• testing: packages are available from testing repositories (see QA team member (testing))
• verified: all test cases were verified successfully (see QA team member (testing))
• invalid: invalid issue, not a bug, duplicate or wontfix. Add a detailed description and link to other related issue when using this tag.

An issue without a label is considered a new feature or an enhancement.

Until June 2024, both NethServer 7 (NS7) and NethServer 8 (NS8) are actively supported. You can use the ns7 label label to mark the issues for NethServer 7.

Before introducing new labels, please discuss them with the development team and make sure to describe carefully the new label inside the label page.

Developer

The Developer.

• Sets the Assignee to himself.
• Bundle his commits as one or more GitHub Pull requests
• For enhancements, writes the test case (for bugs the procedure to reproduce the problem should be already set).
• Writes and updates the Documentation associated with the code.
• Finally, clears the Assignee.

If the issue is not valid, it must be closed using the invalid label. A comment must convey the reason why it is invalid, like “duplicate of (URL of issue), wontfix because …”.

QA team member (testing)

The QA team member.

• Takes an unassigned issue with label testing and sets the Assignee field to him/herself.
• Tests the package, following the test case documentation written by the Developer.
• When test finishes he/she removes the testing label and clears the Assignee field. If the test is successful, he/she sets the verified label, otherwise he/she alerts the Developer and the Packager to plan a new process iteration.

Packager

The Packager coordinates the Developer and QA member work. After the Developer opens one or more pull requests:

• Selects issues with open pull requests
• Reviews the pull request code and merges it
• Builds and uploads the RPMs to the testing repository and sets the testing label (see Building RPMs)

After the QA member has completed the testing phase:

• Takes an unassigned issue with label verified
• Commits a release tag (see Building RPMs).
• Re-builds the tagged RPM.
• Uploads the RPM to updates (or base) repository.
• Pushes the release tag and commits to GitHub
• Closes the issue, specifying the list of released RPMs

When the package is CLOSED, all related documentation must be in place.

Pull requests

A Pull Request (PR) is the main method of submitting code contributions to NethServer.

You can find an overview of the whole workflow here: https://guides.github.com/introduction/flow/

Submitting a pull request

When submitting a PR, check that:

1. PR is submitted against master (for current stable release)
2. PR title contains a brief explanation of the feature, fix or enhancement
3. PR comment contains a link to the related issue, in the form NethServer/dev#<number>, eg: NethServer/dev#1122
4. PR comment describes the changes and how the feature is supposed to work
5. Multiple dependent PRs in multiple repositories must include the dependencies between them in the description
6. Select at least one PR reviewer (GitHub suggestions are a usually good)
7. Select yourself as the initial PR assignee

Managing an open pull request

After submitting a PR, before it is merged:

1. If enabled, automated build process must pass
   • If the build fails, check the error and try to narrow down the reason
   • If the failure is due to an infrastructure problem, please contact a developer who will help you
2. Another developer must review the pull request to make sure it:
   • Works as expected
   • Doesn’t break existing stuff
   • The code is reasonably readable by others developers
   • The commit history is clean and adheres to Commit message style guide
3. The PR must be approved by a developer with commit access to NethServer on GitHub:
   • Any comment raised by a developer has been addressed before the pull request is ready to merge

Merging a pull request

When merging a PR, make sure to copy the issue reference inside the merge commit comment body, this step will be used by automation tools:

• to write notification about published RPMs inside the referenced issue
• to automatically create RPMs changelog

If the commit history is not clear enough, or you want to easily revert the whole work, it’s acceptable to squash before merge. Please make sure the issue reference is present inside the comment of the squashed commit.

Also, avoid adding the issue references directly inside non-merge commit messages to have a clean GitHub reference graph.

Example of a good merge commit:

commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Merge: xxxxxxx yyyyyyy
Author: Mighty Developer <mighty.developer@netheserver.org>
Date:   Thu Dec 14 17:12:19 2017 +0100

    Merge pull request #87 from OtherDev/branchXY

    Add new excellent feature

    NethServer/dev#1122

Example of a merged PR with squash:

commit xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Author: Mighty Developer <mighty.developer@netheserver.org>
Date:   Thu Dec 14 17:12:19 2017 +0100

  Another feature (#89)

  NethServer/dev#1133

Draft pull requests

The use of draft pull requests is recommended to share an on-going development. Draft pull requests can be used to test possible implementations of features that do not have an issue yet. If the draft pull request does not reference an issue it must have an assignee.

RPM Version numbering rules

NethServer releases carry the version number of the underlying CentOS. For example NethServer 7 beta1 is based on CentOS 7.

Packages have a version number in the form X.Y.Z-N (Eg. nethserver-myservice-1.0.3-1.ns7.rpm):

• X: major release, breaks retro-compatibility
• Y: minor release, new features - big enhancements
• Z: bug fixes - small enhancements
• N: spec modifications inside the current release - hotfixes

Commit message style guide

Individual commits should contain a cohesive set of changes to the code. These seven rules summarize how a good commit message should be composed.

1. Separate subject from body with a blank line
2. Limit the subject line to 50 characters
3. Capitalize the subject line
4. Do not end the subject line with a period
5. Use the imperative mood in the subject line
6. Wrap the body at 72 characters
7. Use the body to explain what and why vs. how

For merge commits, and commits pushed directly to master branch (avoid whenever possible!), also add the issue reference inside the commit body.

Documentation

The developer must take care to write all documentation on:

• wiki page during development
• Developer Manual and/or README.rst before release
• Administrator Manual before release
• Inline help before release

Packages should be inside the testing or nethforge-testing repositories until all documentation is completed.

New packages

Before creating a new package, make sure it’s a good idea. Often a simple documentation page is enough, and it requires much less effort. When trying new things, just take care to write down on a public temporary document (maybe a wiki page) all steps and comments. If the feature collects many requests, it’s time to think about a new package. Otherwise, the temporary document can be moved to a manual page.

When creating a new package, make sure the following requirements are met:

• Announce it on http://community.nethserver.org
• Create an issue describing the package
• Create a personal repository on GitHub
• Add a GPL license and copyright notice in the COPYING file
• Add a README.rst file, with developer documentation
• If needed, create a pull request for the NethServer/comps or NethServer/nethforge-comps repository, to list the package in the Software center page.
• Build the package and push it to testing or nethforge-testing repository

See also Building RPMs.

Package updates

Updates to RPM packages must obey the following rules:

• New features/enhancements and bug fixes must not alter the behavior of existing systems
• New behaviors must be enabled by an explicit and documented sysadmin operation
• RPM packages must support updates from any previous release of the same branch

Minor release from upstream

On every upstream (CentOS) minor release, the QA team should check the following hot points before clearing the way to the new release:

• Samba: authenticated access to file shares
• Mail server
  • sending and receiving mail
  • antivirus filter on received mail
• Groupware
  • basic mail features
  • access to calendars and contacts
• Web proxy
  • web access with transparent proxy
  • antivirus filter on using EICAR test
• Web applications (eg. NextCloud)
• Asterisk with dahdi kernel modules and FreePBX

ISO releases

Usually, the NethServer project releases a new ISO image in the following cases:

• when the upstream project releases a new ISO image. The NethServer ISO is rebased on it.
• when packages bundled in the ISO receive new features that affect the installation procedure and/or the initial system configuration.

The NethServer ISO is almost equivalent to the upstream one, except for the following points:

• Additional boot menu options and graphics
• Additional Anaconda kickstart scripts and graphics
• Additional RPMs from the NethServer project

See also Building ISO.

Pre-releases

Before any final ISO release, the software development process goes through some test versions, usually called alpha, beta and release candidate (RC). These releases are an excellent way to experiment with new features, but may require some experience using a Linux system and/or the command line.

Alpha releases are not ready to be used in production because some features are not finished, furthermore upgrade to the final release will not be supported (but may be possible).

Beta releases could be used in production, especially if new features are not used on mission-critical systems. Upgrades to the final release are supported.

Release candidates (RC) can be run in production, all features are supposed to be complete and bug-free. The upgrade to the final release will be minor or less.

RPM package rules

Naming and events conventions

Each package name MUST be composed of

• a prefix, corresponding to the product name: nethserver-, neth-, ….
• the feature/function/daemon/software: base, directory, httpd-admin …

Each package MUST contain a <packagename>-update event, raised each time the package is installed/updated and when the system is re-configured (for instance, after another package has been uninstalled). The update event should

• configure the package on first install
• take care of upgrading current installation in case of package update

Note

You should not add code in %post and %pre sections of the spec file. All the logic must be inside the -update event.

Each package MAY contain a <packagename>-save event, raised by the console or the web interface to adjust the package configuration after some DB value has changed.

For example, given a package named nethserver-dnsmasq:

• update event: nethserver-dnmasq-update
• save event: nethserver-dnmasq-save

Install/Update process

Just after a package transaction (install/update), the NethServer yum plugin will:

• execute all nethserver-update-<package>: events of installed/updated packages in the current transaction
• execute runlevel-adjust event to start/stop all configured services
• execute firewall-adjust event to open/close firewall ports

In case of manual installation , the update, firewall-adjust and runlevel-adjust events must be fired manually using the signal-event command.

Uninstall process

After a package is removed, the NethServer yum plugin will:

• execute all nethserver-update-package event of installed packages
• execute runlevel-adjust event to start/stop all configured services
• execute firewall-adjust event to open/close firewall ports

Service packages

A service package is an RPM which is responsible for a system service configuration and management. The package must follow all rules listed above plus some more conventions.

Configuration DB default

Mandatory:

• type: service
• status: the current service status, can be enabled or disabled

Optional:

• TCPPort: a tcp listening port
• UDPPort: a udp listening port
• TCPPorts: a list of tcp listen ports. Eg: 123,678
• UDPPorts: a list of udp listen ports. Eg: 123,678

For example, the package nethserver-puppet managing the service puppet will contain:

• /etc/e-smith/db/configuration/defaults/puppet/type
• /etc/e-smith/db/configuration/defaults/puppet/status

Beside this, each packge MUST always declare its own set of keys and properties inside default databases.

Events and actions

nethserver-base package provides two generic events:

• runlevel-adjust: enable/disable the service using chkconfig command and start/stop the service
• firewall-adjust: read TCPPort(s) and UDPPort(s) props and open the specified ports in the firewall configuration

Both events are handled by the system, so there is no need to link these events into the package.

Further documentation: perldoc /usr/share/perl5/vendor_perl/NethServer/Service.pm

Orphan services

During runlevel-adjust event, the system will stop any orphan service. A orphan service is a running service not controlled by any nethserver-package. A service is an orphan if there is a service record in configuration db, and there is no db defaults (in /etc/e-smith/db/configuration/defaults).

Internationalization

These are the coding conventions for NethServer i18n. Each package repository should respect them.

• The developer prepares the translation source strings when the code has been written.

• Each translation catalog must be mapped to a resource on Transifex.

• Whenever new strings are added or existing ones are changed, source catalogs must be pushed into Transifex with Transifex client:

  tx push -s
  

• Once translators complete their work a new release of nethserver-lang can occur.

Currently both gettext and Server Manager specific formats are supported for language catalogs.

Server Manager (Cockpit)

Packages providing an application for the Cockpit Server Manager must adhere to the guidelines documented here:

https://nethserver.github.io/nethserver-cockpit/i18n/#translations

Server Manager (Nethgui)

To configure the Transifex client and find the string catalog files execute the txinit.sh script on the repository root. The script can run multiple times, when new catalogs are added to the repository.

PHP files

All source language catalog files are placed in root/usr/share/nethesis/NethServer/Language/en. We assume en is en-US.

The catalog format is a common PHP file where the array variable $L is assigned some keys:

<?php

$L['string_id_1'] = 'This is an example';
$L['string_id_2'] = 'What is a PHP array catalog?';

1. Omit ending sequence ?>.
2. Use UTF-8 encoding.

Given a module with name “Test”, the source language file will be root/usr/share/nethesis/NethServer/Language/en/NethServer_Module_Test.php.

When the Server Manager is running in debug mode, missing translation labels can be found in /var/log/messages. To enable the debug,

• Unlock index_dev.php controller:

  touch /usr/share/nethesis/nethserver-manager/debug
  

• Prepend index_dev.php on URLs path, eg: https://<ipaddress>/index_dev.php/en/<module>.

gettext files

The gettext format is used by the TODO scripts in the old Server Manager.

The xgettext command can be used to extract strings from the source code. The resulting .pot file must be named locale/<rpmname>.pot.

For example we will translate a new TODO message for the web interface.

Steps to setup translation for a new todo:

1. Be sure gettext is installed in your system

2. Add the todo script to the git repository

3. Create locale directory inside the git repository root:

   mkdir locale/
   

4. Extract the strings (from a Python source code):

   PACKAGE=$(basename `pwd`)
   xgettext --msgid-bugs-address "translations@nethserver.org" --package-version "1.0.0"  \
   --package-name "$PACKAGE" --foreign-user -d "$PACKAGE" -o "locale/${PACKAGE}.pot" \
   -L Python root/etc/nethserver/todos.d/*
   

5. Run the txinit.sh script and commit newly created files into git.

6. Upload the new .pot resource to Transifex (special permissions are needed):

   tx push -s
   

Databases

Overview

All user-editable configuration parameters on NethServer are stored in plain text database.

These values are used to generate the system configuration files, such as those found in the /etc/ directory. The configuration databases may be modified by various programs on the system, including the web interface or scripts run from the command line by a system administrator.

Each entry in the database is either a simple key/value pair or a key and a collection of related property/value pairs.

Simple entries

Simple configuration database entries take the form of a key/value pair:

[root@nsrv -]# config show SystemName
SystemName=myserver

Complex entries

More complex entries consist of a key, a type, and a collection of property/value pairs:

[root@nsrv -]# config show sysconfig
sysconfig=configuration
    Copyright=
    ProductName=NethServer
    Registration=none
    Release=4
    Version=6.4

Use complex entries whenever possible

Access from the command line

You can access database entries from the command line using the config command, as shown above, or the db command. The config command provides a shorthand for accessing the configuration database. The following commands are equivalent:

[root@nsrv -]# config show SystemName
SystemName=nsrv

[root@nsrv -]# db configuration show SystemName
SystemName=nsrv

The db command allows you to access all of the databases. For example to show the details of the test entry from accounts db:

[root@nsrv -]# db accounts show test
test=user
    City=
    Company=
    Department=
    FirstName=test
    LastName=test
    PhoneNumber=
    Street=
    Uid=5000
    VPNClientAccess=yes
    VPNRemoteNetmask=255.255.255.0
    VPNRemoteNetwork=192.168.1.0

For more options see help of db command:

db -h

Access via the Perl API

You can also access configuration database entries programmatically using the esmith::ConfigDB and related Perl modules, which are abstractions for the esmith::DB module. For example, we can retrieve and show the admin account details like this:

use esmith::AccountsDB;
my $db = esmith::AccountsDB->open or die "Couldn't open AccountsDB\n";
my $admin = $db->get("admin") or die "admin account missing from AccountsDB\n";
print $admin->show();

For documentation on Perl API use the perldoc command. Eg:

perldoc esmith::ConfigDB

Database initialization

The configuration databases are initialized from files in the /etc/e-smith/db/ hierarchy. These files can perform one of three actions:

• Create a database entry and set it to a default value, if the entry does not already exist.
• Migrate an entry from a previous value to a new value.
• Force a database entry to a specific value, regardless of its current setting (use with care!)

This design allows each package to provide part of the system configuration, or migrate the system configuration values as required. Note that a single database property can only be owned by one package. Database initialization is run during system install, system upgrade and after new software has been installed.

If you examine the /etc/e-smith/db/configuration/ directory you will see three subdirectories: defaults/, force/ and migrate/ to match the three options above. A similar structure exists for each of the other databases. A new database can be created by populating a new directory tree under the /etc/e-smith/db/ directory.

Configuration databases can also be initialized using a special /usr/libexec/nethserver/initialize-<dbname>-database script, where dbname is the database name. For example: /usr/libexec/nethserver/initialize-mycustomdb-database.

Defaults files

Defaults files are simple text files. If the corresponding database key/property already exists, it is skipped. Otherwise, the key/property is created and the value loaded. For example, this file:

[root@nsrv -]# cat /etc/e-smith/db/configuration/defaults/sshd/status
enabled

It would create the sshd database entry if it doesn’t already exist, create the status property for that entry, again if it doesn’t already exist, and finally set the status property to enabled.

Forcing database initialization

Simply call the action: /etc/e-smith/events/actions/initialize-default-databases

Force files

Force files are just like defaults files, except they overwrite the existing value. So, this file:

[root@nsrv -]# cat /etc/e-smith/db/configuration/force/sysconfig/Version
6

It would create the Version property of the sysconfig entry and unconditionally set its value to 6.

Warning

Do not use force fragments if not really necessary!

Migrate fragments

Migrate fragments are small pieces of Perl text which can be used to perform more complex migrations than is possible with defaults and force files. They would normally be used to replace database keys or properties with new names, or to adjust policy settings during an upgrade.

Each fragment is passed a reference to the current database in the $DB variable. This variable is an instance of the appropriate esmith::DB subclass, e.g. esmith::AccountsDB when the accounts database migrate fragments are being executed. This means that you can use the methods of that subclass, for example esmith::AccountsDB->users().

Here is an example of a migrate fragment, which replaces the outdated popd entry with the new name pop3:

{
    my $popd = $DB->get("popd") or return;
    my $pop3 = $DB->get("pop3") ||   $DB->new_record("pop3", { type => "service" });
    $pop3->merge_props($popd->props);
    $popd->delete;
}

This fragment checks whether the database (the configuration database in this case) has a popd entry. If that entry does not exist, the migrate fragment returns immediately. If the popd entry exists, we need to convert it, so we retrieve the pop3 entry (or create it if it doesn’t already exist). We then merge the properties from the popd entry into the pop3 entry and finally delete the popd entry.

If this migrate fragment is run again, it will return immediately as the popd entry has already been deleted.

Important notes about migrate fragments

• Please be careful with migrate fragments. Although they should only modify entries within the current database, there are no restrictions placed on what they can do. The ability to open and even modify other databases may be required to perform a migration.
• Migrate fragments must be safe to run multiple times. They should migrate the value when required and do nothing in other cases.
• Migrate fragments should never call croak or die. This will cause the database migration to stop. If an error is detected, call carp or warn to note the error in the logs.
• Migrate fragments should call good termination with return(0) rather than exit(0).
• Migrate fragments should be owned by the package requiring the migration so that the migration only occurs when that package is installed.
• Migrate fragments should be self-contained and ideally perform only one migration per fragment.
• DO NOT USE to initialize default database values.

Evaluation order

When a database is loaded:

• migrate scripts are run first
• then defaults are loaded
• and finally any force files are loaded.

This order allows migration of old format entries to occur prior to loading of new default values. Remember, defaults will not change an existing database property.

Best practices

• The configuration databases should only be modified using the tools and APIs provided.
• The order of the entries and the order of properties is undefined.
• The keys and property names are currently treated in a case-sensitive manner, though this may change in the future. Do not create keys or property names which differ only by their case.
• Underscores and hyphens are valid in key and property names, but should normally be avoided.
• Do not “overload” an existing property with a new value. If the existing values do not meet your requirements, discuss your implementation with the developers. Values which are not known by the base may cause serious issues on upgrade. If the existing panels have three choices, do not invent new choices without enhancing the panel to support them.
• The type pseudo-property is used internally and is reserved .
• By convention, database keys are lower case, and property names are stored in mixed case. The type, status and access properties are exceptions to this convention.
• The storage location and internals of the databases is subject to change.
• The configuration databases are currently stored as pipe-delimited flat text files in the /var/lib/nethserver/db/ directory.

Namespace issues

All entries in a single database share the same namespace. Users, groups, information bays, printers, and other entries in the accounts database currently all share one namespace. This means that you cannot have a user with the same name as an information bay, group or other entry in the accounts database.

However, it would be possible to have a host named fredfrog as well as a user named fredfrog as they are stored in separate databases and thus different namespaces.

List of available database

Table of databases

The following table summarizes

• the database name
• the perl module that manages it and
• the package that provides it

Databases provides by the base system:

Name	Perl module	Package	Description
configuration	esmith::ConfigDB	nethserver-base
hosts	esmith::HostsDB	nethserver-hosts
networks	esmith::NetworksDB	nethserver-base
domains	esmith::DomainsDB	nethserver-mail-common

Each modules can define its own new databases. Some relevant databases are:

Name	Perl module	Package	Description
accounts	esmith::AccountsDB	nethserver-directory
domains	esmith::DomainsDB	nethserver-mail-common

Templates

Design of the template system

Every piece of software has its own configuration format, and writing parsers for each one is a complex, time-consuming and error-prone process. The template system software avoids the whole issue by using templates which generate the correct configuration.

Configuration files are over-written when templates are expanded. In a few specific cases, the existing configuration file is parsed and rewritten in-place. This is done where the configuration file is also automatically updated by some other process.

Templates are stored under /etc/e-smith/templates/ in a directory hierarchy which matches the standard filesystem. For example, the template for /etc/inittab is stored in the /etc/e-smith/templates/etc/inittab/ directory. Each template is stored as a directory of template fragments and processed by the Perl Text::Template module.

The template fragments are concatenated together in ASCIIbetical order (US-ASCII sort order) and the complete file is parsed to generate the appropriate configuration files for the service. The use of fragments is part of NethServer modular and extensible architecture; it allows third-party modules to add fragments to the configuration where necessary.

The Text::Template module

The Text::Template module allows arbitary Perl code to be embedded in a template file by surrounding it in curly braces ({ and }). The code inside the braces is interpreted and its return value replaces the section between, and including, the braces. For instance:

The answer is { 40 + 2 }

becomes:

The answer is 42

Variables can be passed in from the program which is expanding the template, hence:

{
    $OUT = ';'
    for my $item ( qw(bread milk bananas) )
    {
         $OUT .= "\* $item\n";
    }
}

would expand to:

Shopping list
* bread
* milk
* bananas

The template system uses this mechanism to automatically pass in global configuration variables from the configuration database which can then be used to fill out the configuration files.

For example, the /etc/hosts template could be fairly simple and composed of two fragments:

[root@test hosts]$ ls /etc/e-smith/templates/etc/hosts
10localhost  20hostname

Fragments can have static content. For example, the first fragment:

127.0.0.1       localhost

The second is more complex and relies on values from the configuration database:

{
    $OUT .= "$LocalIP\t";
    $OUT .= " ${SystemName}.${DomainName}";
    $OUT .= " ${SystemName}";
}

Note that the whole fragment is enclosed in braces. Within those braces is a section of Perl code.

When this template is expanded, it results in the following configuration file:

# ================= DO NOT MODIFY THIS FILE =================
#
# Manual changes will be lost when this file is regenerated.
#
# Please read the developer's guide, which is available
# at https://dev.nethesis.it/projects/nethserver/wiki/NethServer
# original work from http://www.contribs.org/development/
#
# Copyright (C) 2015 Nethesis S.r.l.
# http://www.nethesis.it - support@nethesis.it
#
127.0.0.1       localhost
192.168.10.1    nsrv.nethesis.it nsrv

The header block comes “for free” as part of the template system, courtesy of an optional file template-begin, which is always processed as the first fragment. If it isn’t provided, the text shown with # comments is included. If target configuration file do not support line comment beginning with #, please provide a custom or empty template-begin.

The other lines are provided by the two fragments shown above. Note the use of the configuration database variables: $LocalIP, $SystemName and $DomainName. All simple entries in the configuration database are provided as global variables to the templates.

Note that all of the template fragments are concatenated together before evaluation, so it is possible to set values in fragments which are used in later fragments. This is a very useful model for reducing the code in individual template fragments.

The complex entries in the configuration database are also provided as global variables to the templates. However, they are provided as Perl hashes instead of simple scalars. For example, here is how you might configure the Network Time Protocol (NTP) server /etc/ntp.conf file:

server { $ntpd{NTPServer} }
driftfile /etc/ntp/drift
authenticate no

The NTPServer setting is stored in the ntpd configuration database record, and so can be accessed via the hash accessor $ntpd{NTPServer}.

template-begin and template-end

Each template directory can contain two optional files template-begin and template-end . The template-begin file is always processed as the first file of the template, and the template-end file is always processed as the last file.

If the directory does not contain a template-begin file, the contents of /etc/e-smith/templates-default/template-begin is used automatically.

If the directory does not contain a template-end , nothing is appended to the template output. It is mostly used to provide the closing block for configuration files written in languages such as HTML and PHP, through a link to an entry in the templates-default/ directory.

/etc/e-smith/templates-default

The /etc/e-smith/templates-default directory contains a set of template-begin and template-end files for various languages. For example, if your template generates a perl script, you would link template-begin to /etc/e-smith/templates-default/template-begin-perl and automatically get the #!/usr/bin/perl -w line and a comment containing the contents of the default template-begin file.

Note

You may also need a templates.metadata configuration file if your generated file needs to be executable.

Template fragment ordering

Template fragments are assembled in ASCII-betical order, with two exceptions: template-begin always comes first, and template-end always comes last. Template fragments are often named to start with a two digit number to make the ordering obvious, but this is not required.

Templates for user home directories: templates-user

Most of the templates on the system map to single, fixed output files, such as /etc/hosts. However, templates are also used to generate configuration files such as mail delivery instructions for users. These templates are stored in the /etc/e-smith/template-user/ tree.

As these templates have a variable output filename, they are expanded using small pieces of Perl code in action scripts.

Local site overrides: templates-custom and templates-user-custom

It is possible that the standard templates are not correct for a particular installation, and so the local system administrator can override the existing templates by placing files in the templates-custom tree. This is a parallel tree to the normal templates hierarchy, and is normally empty. There is also a template-user-custom tree for overriding entries in the templates-user tree. Be aware of overwriting all settings of a template if you copy the whole template to custom templates. This means, if there is an update at the original template it is overritten with your old version at custom template. If you only want to add a line to the config create an empty custom template and do it there.

If a templates-custom entry exists for a template, it is merged with the standard templates directory during template expansion, using the following rules:

• If a fragment of the same name exists in both templates and templates-custom, the one from templates-custom is used, and the one from the standard templates tree is ignored.
• If the fragments in templates-custom have different names from those in templates, they are merged into the template as if they were in the templates directory.
• If the templates-custom entry is a file, rather than a directory, it completely overrides the standard template.

To make this concrete, let’s assume we have the following template structure in /etc/e-smith/templates/etc/book.conf:

10intro
30chapter3
40chapter4
80synopsis

and in /etc/e-smith/templates-custom/etc/book.conf:

30chapter3
50chapter5

The resulting template would be processed in this order:

• template-begin from /etc/e-smith/templates-default
• 10intro from /etc/e-smith/templates/etc/book.conf
• 30chapter3 from /etc/e-smith/templates-custom/etc/book.conf
• 40chapter4 from /etc/e-smith/templates/etc/book.conf
• 50chapter5 from /etc/e-smith/templates-custom/etc/book.conf
• 80synopsis from /etc/e-smith/templates/etc/book.conf
• template-end (empty), nominally from /etc/e-smith/templates-default

How to resolve conflicts with standard templates

It is possible that the standard templates may specify behaviour which is not appropriate for your application. In many cases the templates will be driven by configuration database settings which allow their behaviour to be customized, which should be the first thing to check.

In many cases, your application only needs to extend the behaviour of the template by adding one or more fragments. This should be your second option and can be achieved by simply adding your fragment in the correct place in the list of fragments.

In rare cases the standard template specifies a behaviour which conflicts with your application. In these cases, you should do all of the following:

• Create a templates-custom directory to match the existing one in the templates hierachy.
• Copy the conflicting fragment, and only that fragment, to the templates-custom directory. The fragment should have the same name in both directories. At this point you have not changed the behaviour of the system as the templates-custom entry will be preferred, but will behave identically.
• Modify the copy in templates-custom to suit your required behaviour.
• Inform the NethServer team about the problem. Please attach your modified template (or even better, a patch file) and provide details of why you think that the standard template should be changed.

The expansion of templates

To expand your custom templates to their destination you have to use the following command:

expand-template <template destination>

where template destination has to be changed with the true path to the configuration file.

For Example you want to add something to the samba configuration, then you have to build a custom template fragment under /etc/e-smith/template-custom/etc/samba/smb.conf/ directory and execute the command:

expand-template /etc/samba/smb.conf

Subdirectory templates

It is also possible to split templates into further subdirectories. This can be very useful for evaluating the same fragments in a loop, for example for each virtual domain in httpd.conf or each ibay in smb.conf.

Two examples of this can be found in /etc/e-smith/templates/etc/httpd/conf/httpd.conf/80VirtualHosts which loops over the /etc/e-smith/templates/etc/httpd/conf/httpd.conf/VirtualHosts/ directory, and /etc/e-smith/templates/etc/smb.conf/90ibays which performs a similar loop over the /etc/e-smith/templates/etc/smb.conf/ibays/ directory.

Template expansion

The system is designed to ensure consistent and reliable operation, without requiring command-line access. Whenever an event is signalled, the relevant templates for that event are expanded and the services are notified of the configuration changes.

Requesting expansion of a template in an event is a simple matter of creating an empty file under the templates2expand hierarchy for that event.

See events manual chapter for further information.

Template permissions and ownership: templates.metadata

Templates are normally expanded to be owned by root and are not executable, which is a reasonable default for most configuration files. However, templates may need to generate configuration files which are owned by a different user, or which need to be executable or have other special permissions. This can be done by creating a templates.metadata file which defines the additional attributes for the expansion.

Note

Configuration files should generally not be writable by any user other than root. In particular, configuration files should not normally be writable by the www user as this poses a significant security risk. Installation advice which says chmod 777 is almost invariably wrong.

For example, here is the metadata file /etc/e-smith/templates.metadata/etc/ppp/ip-up.local:

UID="root"
GID="daemon"
PERMS=0755

which sets the group to daemon and makes the script executable. Note that the file is readable by members of the daemon group, but it is not writable by anyone but root. It is also possible to use the same template to generate multiple output files, such as in this example:

TEMPLATE_PATH="/etc/sysconfig/network-scripts/route-ethX"
OUTPUT_FILENAME="/etc/sysconfig/network-scripts/route-eth1"
MORE_DATA={ THIS_DEVICE => "eth1" }
FILTER=sub { $_[0] =~ /^#/ ? '' : $_[0] } # Remove comments

The templates.metadata file for route-eth0 just uses eth0 instead of eth1 on the second and third lines. Note also the FILTER setting which allows post-processing of the generated template.

There are many examples under /etc/e-smith/templates.metadata/ and the full list of options can be seen with:

perldoc esmith::templates

Template deletion: templates.metadata

A template once expanded in the file system cannot be deleted automatically, you can only change its content. The template will still exist until you delete it by a templates.metadata file.

For example to delete /etc/myExample, create the file /etc/e-smith/templates.metadata/etc/myExample with just the following line

DELETE=1

Then

expand-template /etc/myExample

Perl API: processTemplate

In rare circumstances you may need to call processTemplate directly. Explicit calls to processTemplate are typically only used when the output filename is variable:

use esmith::templates;
foreach my $name (@names)
{
    [...]
    processTemplate(
    {
      TEMPLATE_PATH => "/etc/myservice/user.conf",
      OUTPUT_FILENAME => "/etc/myservice/$name.conf"
    );
    [...]
}

bq. Content is available under GNU Free Documentation License 1.3 unless otherwise noted. Original document from: http://wiki.contribs.org/

Actions and events

Actions

An action is a program, frequently written in a scripting language, which performs a single task. It is typically an encapsulation of a task usually done by a system administrator, such as editing a configuration file or reconfiguring a service. Actions are not called directly; they are always called by signalling an event.

The actions are stored in the /etc/e-smith/events/actions/ directory. These actions are then linked into the relevant events as the same action may need to be performed in more than one event. To create a new action called myaction you simply create a program to perform the action myaction and save it as /etc/e-smith/events/actions/myaction . Actions can be written in any programming language, although additional platform support is provided for Perl code.

An example action script is:

#!/bin/bash
/usr/sbin/lokkit --update

Action script parameters

Action scripts are always called with at least one parameter; the name of the current event. Many action scripts can be called with a single additional parameter. This parameter is usually a configuration database key, for example the username being modified. Action scripts rarely require more than two parameters. The details should be stored in the configuration database(s) and only the key should be passed to the action scripts. All configuration details must be stored in the configuration databases and the database key passed as the parameter to the action. This allows other scripts to be added to the event.

Since the system passes the name of the current event as the first parameter, it is often beneficial to write action scripts which are polymorphic based on the event name. For example, the code to create a user and the code to modify an existing user may be only slightly different and may well benefit from being in a single script. Example:

use strict;
my $event = $ARGV[0];
my $myarg = $ARGV[1];

exit 0;

Note

Whenever possible, avoid to call events from within action scripts.

Action code libraries

To promote code reusability and components abstraction some Perl modules are available under /usr/share/perl5/vendor_perl/NethServer/ and /usr/share/perl5/vendor_perl/esmith/. For instance,

NethServer::Password

Secret generation and persistent storage, under /var/lib/nethserver/secrets/.

NethServer::Service

Service manager agnostic API. No matter if a service is managed by systemd, Upstart or SysV init script: use this API to gain control over it.

NethServer::Directory

Access to LDAP, service accounts and ACL management, low-level user and group management.

NethServer::MailServer

Obtain accounts and mail addresses relations. Manage IMAP ACLs.

esmith::templates

Template processing and expansion.

esmith::events

Event execution and tracking.

For more informations about a specific module, refer its perldoc documentation.

Events

Events are a mechanism which allows the system to trigger a set of actions in response to actual events that happen on the system. When one of the users interfaces modifies the configuration databases, it must signal an event to regenerate the various server application configuration files according to the new configuration.

Note: The user interface must never modify configuration files directly. Neither should to the administrator from command line.

Each event is associated with a list of actions which should be performed when the event occurs and is defined as a subdirectory of /etc/e-smith/events/ containing symbolic links to the appropriate actions, loosely modelled after the ‘’System V init’‘mechanism for starting servers. For example, if you examine the /etc/e-smith/events/interface-update directory:

[root@nsrv actions]# ll /etc/e-smith/events/interface-update/
total 8
lrwxrwxrwx. 1 root root   34 Feb  6 11:19 S04interface-config-adjust -> ../actions/interface-config-adjust
lrwxrwxrwx. 1 root root   33 Feb  6 11:19 S25interface-config-reset -> ../actions/interface-config-reset
lrwxrwxrwx. 1 root root   33 Feb  6 11:19 S30interface-config-write -> ../actions/interface-config-write
lrwxrwxrwx. 1 root root   35 Feb  6 11:19 S70interface-config-restart -> ../actions/interface-config-restart
lrwxrwxrwx. 1 root root   36 Feb  6 11:19 S75interface-config-hostname -> ../actions/interface-config-hostname
drwxr-xr-x. 2 root root 4096 Feb  6 11:20 services2adjust
drwxr-xr-x. 3 root root 4096 Dec 18 11:17 templates2expand

The symbolic links are given prefixes such as S15, S85, etc. to specify the order in which the actions should be executed in a similar manner to the System V init mechanism. You can change the actions performed by an event by changing the links in the event directory. You can also create a new event by creating another subdirectory of /etc/e-smith/events/.

Implicit actions

Most events contain two common tasks: expanding various templates and adjusting (e.g. restarting) the relevant services. For this reason, two implicit actions are included in all events. These implicit actions mean that additional code does not need to be written to perform these common tasks. The implicit actions are represented by entries in the services2adjust/ and templates2expand/ subdirectories.

services2adjust

The services2adjust/ directory contains links mapping a specific service to the action to perform on that service. For example, if signalling the event in question requires that the ntpd service is restarted, you simply include the link ntpd -> restart in the services2adjust directory. The implicit action services2adjust would then restart the ntpd service. As an example, the services2adjust/ directory for the nethserver-httpd-update event is shown below:

# ls> l /etc/e-smith/events/nethserver-httpd-update/services2adjust/
total 0
lrwxrwxrwx. 1 root root 7 Oct 2 09:05 httpd -> restart

templates2expand

The templates2expand/ directory contains a list of the configuration files which need to be regenerated from their templates. This list consists of a collection of empty files with the same file name as the configuration file to be expanded and in a heirarchy mirroring their location on the system. For example, to expand templates for the /etc/samba/smb.conf configuration file, simply include the empty file etc/samba/smb.conf in the templates2expand/ directory of the relevant event.

Order of implicit actions

The implicit actions are implemented by inserting the action script generic_template_expand early in the list of actions to be run in an event and the adjust-services action near the end of the list. You should normally link your action scripts in the range S10 to S80 so that they occur after templates2expand and before services2adjust.

Note

The generic_template_expand action is currently run at S05 and adjust-services is run at S90.

Signalling events

The signal-event program takes an event name as an argument, and executes all of the actions in that event, providing the event name as the first parameter and directing all output to the system log. It works by listing the entries in the event directory and executing them in sequence. So for example, the command:

signal-event interface-update

will perform all the actions associated with the interface-update event, which is defined by the contents of the /etc/e-smith/events/interface-update/ directory.

Events with arguments

So far we have described the following general principle throughout the system; changes are made by altering the database values, then signalling events. The actions triggered by each event typically regenerate entire configuration files, taking into account the latest configuration information.

However, some changes are best made incrementally. For example, consider the user-create event. One of its actions updates the LDAP directory, which it could do by deleting all of the users and recreating them based on the updated accounts database. However, this is inefficient and would lose any additional LDAP attributes which may have been stored. It would be better to simply add the new user incrementally, using the default LDAP schema.

But how is the action code to know which user was just added? The new username is passed as an argument to the user-create event. This way the action programs triggered by the user-create event have a choice. They can either ignore the username argument and regenerate their output based on the updated list of accounts, or they can pay attention to the username argument, retrieve the rest of the information about the new user from the accounts database, and perform the incremental work to add the user.

Note

Action scripts should normally take at most two arguments. The first is always the event name. The second optional argument is a key into one of the databases.

Events are not currently serialized. In most cases overlapping events will not cause issues, but caution should be exercised when events are signalled from programs.

Standard events and their arguments

The table below summarises the key NethServer events and their argument if required. Remember, each action script is always called with the event name as the first argument. The arguments listed in this table are provided as the second argument.

Event	Arguments	Description
certificate-update		The server public key certificate has been updated
group-create	Group key	Called when a group is created
group-delete	Group key	Called when a group is deleted
group-modify	Group key	Called when a group is modified
group-create-pseudonyms		Signalled when the automatic creation of group email address is required
host-create	Host key	Called when a host is created
host-delete	Host key	Called when a host is deleted
host-modify	Host key	Called when a host is modified
hostname-modify		Called when the SystemName or DomainName keys have been modified
ibay-create	Shared folder key	Called when a shared folder is created
ibay-delete	Shared folder key	Called when a shared folder is deleted
ibay-modify	Shared folder key	Called when a shared folder is modified
interface-update		Called when a network interface configuration is updated in networks db
logrotate-update		Change default log retention and rotation policies
trusted-networks-update		The set of trusted networks is changed
migration-import	Path to migration directory	Import migration data from the given directory
notifications-save		Set notification configuration (root forward, mail sender address)
password-expired	Username, expire date	The given username password will expire on expiredate
password-modify	User key	Called when a user password is modified
password-policy-update	User key	Called when the system password policy has been changed
post-backup-config		Called after configuration backup end
post-backup-data		Called after data backup end
post-restore-config		Called after restore of configuration
post-restore-data		Called after restore of data
pre-backup-config		The pre-backup-config event creates consistent system state for the backup
pre-backup-data		The pre-backup-data event creates consistent system state for the backup
pre-restore-config		Called before restore of configuration
pre-restore-data		Called before restore of data
pseudonym-create	Pseudonym key	Called when a pseudonym is created
pseudonym-delete	Pseudonym key	Called when a pseudonym is deleted
pseudonym-modify	Pseudonym key	Called when a pseudonym is modified
user-create	User key	Called when a user is created
user-cleanup	User key	Remove all user data
user-delete	User key	Called when a user is deleted
user-modify	User key	Called when a user is modified
user-create-pseudonyms	User key	Called when the automatic creation of user’s email address(es) is required
user-lock	User key	Called when a user account is locked
user-unlock	User key	Called when a user account is unlocked
system-initialization		Initialize all system after installation
software-repos-save		Configure software repositories

Handling deletions

When adding a user, the user is created in the accounts database, and various actions, such as creating the Linux account, are performed in the user-create event. However, when deleting a user, we want to maintain the accounts database entry for as long as possible, in case there is information which the actions in the user-delete event might need in order to cleanly delete the users. The system convention for handling deletions is:

• Change the type of the entry to mark it as being in the process of being deleted e.g. a’‘user’‘entry becomes a’‘user-deleted’‘entry.
• Signal the relevant deletion event - e.g.’‘user-delete’‘
• Remove the entry from the database, but only if the event succeeds. With this approach, the action scripts can decide whether to ignore the’‘user-deleted’’ entries when performing their tasks.

Event logs

Warning

Output of event logs will be soon refactored!

All events, and all actions run by the event, are logged to the messages system log. Here is an example action log, which has been formatted onto multiple lines to enhance readability:

Feb 2 13:22:33 gsxdev1 esmith::event[4525]:
 S65sshd-conf=action|
 Event|remoteaccess-update|
 Action|S65sshd-conf|
 Start|1138846952 730480|
 End|1138846953 66768|
 Elapsed|0.336288

From this single log, we can see the action script name, which event it was called in, when it started, ended and how long it took (0.34 seconds). Now, let’s add an action script which always fails and signal the event again:

Feb 2 16:11:54 gsxdev1 esmith::event[4787]:
 S99false=action|
 Event|remoteaccess-update|
 Action|S99false|
 Start|1138857114 58910|
 End|1138857114 81920|
 Elapsed|0.02301|
 Status|256

Note that this log has a new field Status, which is added if the action script returns a false (non-zero) exit status. Suppressing the Status field when it is zero (success) makes it much easier to find failed actions in the logs.

Warning

If an action script fails, the entire event fails. The other actions scripts in the event are run, but the whole event is marked as having failed.

System validators

System validators provide an extensible UI-independent data validation layer.

On one hand UI implements fast grammar and/or syntax checks on input data. On the other, the system validators performs in-depth system consistency checks.

Design

Validators have a behaviour very similar to events.

• A validator is a directory inside /etc/e-smith/validators.
• Each validator directory has a descriptive name, eg. user-name for a validator which validate a new user name.
• A validator is composed by an arbitrary number of actions saved inside /etc/e-smith/validators/actions directory and linked inside validator directory.
• A success validation occurs when all scripts return 0 (success validation) or at least one script returns 2 (sufficient valid condition).

A validator action are always called with a single parameter which is the value to be validated. Actions must return one of these exit values:

• 0: successful validation
• 1: validation failed
• 2: sufficient validation
• other value: specific error state

When a script returns 2 (sufficient validation) no further script will be processed.

Inside nethserver-devtools package there is validator_actions() function which help creating links to actions just like event_actions function. See perldoc esmith::Build::CreateLinks for details.

Invoking a validator:

validate <validator-name> <value-to-validate>

Eg:

validate user-name goofy

Services

A service is a software which usually runs in background. The system will ensure service status accordingly to its configuration. A service in configuration database is something like this:

httpd=service
    status=enabled
    access=public
    TCPPorts=80,443

Where httpd is the service name and status tells the system if the service should be enabled or disabled.

When the status property is switched between enabled/disabled state, the change will be reflected into runlevel configuration. This is what runlevel-adjust event and action do for all configured services. There is also another action called adjust-services which does the same thing for services registered on a single event.

A service without a record in the configuration database is ignored and can be manually manged using systemctl. See Add a new service.

Control a service

Enable a service:

config setprop myservice status enabled
signal-event runlevel-adjust

Disable a service:

config setprop myservice status disabled
signal-event runlevel-adjust

Where myservice is the service name to be enabled or disabled.

Access network service

A network service is a service running on the server which expose UDP or TCP ports. Ports can be listed in following properties:

• TCPPort: a single TCP port
• TCPPorts: a comma separated list of TCP ports
• UDPPort: a single UDP port
• UDPPorts: a comma separated list of UDP ports

If both TCPPort and TCPPorts properties are set, TCPPorts has the precedence. If both UDPPort and UDPPorts properties are set, UDPPorts has the precedence.

A service can be accessible from public or private LAN. This configuration is saved on access property. The property is a comma separated list of zones (green, red, blue, orange).

Example of a service with UDP port 1122 open to the Internet:

config setprop myservice status enabled UDPPort 1122 access green,red

Example of a service with TCP ports 1122 an 2233 open to local network:

config setprop myservice status enabled TCPPorts 1122,2233 access green

The ports are opened only if the status property is set to enabled.

Add a new service

Any software can configure the init system using the standard systemctl command. This approach always work for third-party software.

On the other hand, if the service must be controlled by NethServer, create a new record inside configuration database:

config set myservice service status enabled

Where myservice is the name of the new service.

Make sure also there are defaults values inside the directory /etc/e-smith/db/configuration/defaults: if the key is present inside the configuration database, but not inside defaults, the service will be stopped. Given the above example, create these files:

mkdir -p /etc/e-smith/db/configuration/defaults/myservice
echo "service" > /etc/e-smith/db/configuration/defaults/myservice/type
echo "enabled" > /etc/e-smith/db/configuration/defaults/myservice/status

Signal the new service to the system:

signal-event runlevel-adjust

Add a new network service

If a service not controlled by NethServer needs one or more open ports, use the TCPPort(s) or UDPPort(s) prop to declare the port(s) and signal the firewall to open it:

config set fw_myservice service status enabled TCPPort 12345 access green
signal-event firewall-adjust

Otherwise, if the service is controlled by NethServer, you can add the properties directly to the service key. For the service myservice on above example:

config set myservice service status enabled TCPPort 12345 access green
signal-event firewall-adjust

See nethserver-firewall-base.

Auto-generated random URL

Sometimes you need to install web applications which don’t have built-in authentication. A good solution can be an auto-generated random URL known only to some special users. It’s also a best practice to restrict access to those applications using Apache allow and deny rules.

This feature is implemented in nethserver-lib using genRandomHash function. The function will generate a SHA1 random hash

Example from [[nethserver-collectd-web]]:

my $alias = $collectd->prop('alias') || "";

# initialize alias if needed
if ($alias eq "") {
    $alias = esmith::util::genRandomHash();
    $confdb->set_prop('collectd-web','alias',$alias);
}

Random alias should be generated inside an action, like <package_name>-conf. The action must be executed before template-expand in a position before 05. Example from createlinks:

my $event = "nethserver-samba-audit-update";

event_actions($event, qw(
      initialize-default-databases 00
      nethserver-samba-audit-conf 02
));

Migration from NethService/SME Server

Migration is the process to convert a SME Server (or NethService) machine into a NethServer.

1. In the old host, create a full backup archive and move it to the new NethServer host.

2. In the new server, install all packages that cover the same features of the old one.

3. Explode the full backup archive on some directory (for instance /var/lib/migration)

4. Signal the event:

   signal-event migration-import /var/lib/migration
   

   This step will require some time.

5. Search for any ERROR string in /var/log/messages

Coding conventions

Most modules have already a migration action which handles the step automatically.

A migration action:

• must be named as <packagename>-migrate
• must be linked into migration-import event
• must migrate old properties values to new ones
• can copy original data files to the new location
• must take care to apply the imported configuration, possibly using the <packagename>-update event

During migration some properties will not be imported:

• UDPPort, TCPPort, UDPPorts, TCPPorts: all network ports will be reset to new defaults
• DNS forwarder, green IP address, default gateway: these properties are filled up in bootstrap-console

All e-smith databases are moved in /var/lib/nethserver/db directory.

Code snippets

A simple migrate action in perl.

#!/usr/bin/perl
use esmith::DB::db;
use esmith::event;
use strict;
my $event = shift;
my $sourceDir = shift;
my $esmithDbDir = 'home/e-smith/db';
my $errors = 0;
if {
  die;
}
my $srcConfigDb = esmith::DB::db]>open\_ro(join('', $sourceDir, $esmithDbDir,'configuration')) || die("Could not open source configuration database in $sourceDir");
my $dstConfigDb = esmith::DB::db->open || die;
my $service = ‘ejabberd’;
my $old = $srcConfigDb->get($service);
my $new = $dstConfigDb->get || $dstConfigDb->new_record($service);
$new->merge_props($old->props);
# Apply configuration
if( ! esmith::event::event_signal('nethserver-ejabberd-update')) {
 exit(1);
}
exit 0;

Remember to change the service name and add a license header.

Add the migrate action to createlinks:

#-----------------------------------
# actions for migration-import event
#-----------------------------------
$event ="migration-import";
event_actions($event, '<packagename>-migrate' => 60);

Packages

Each packages with special migration notes is listed below.

nethserver-base

Properties not migrated:

• PasswordSet
• UnsavedChanges
• bootstrap-console
• dns
• sysconfig
• SystemMode
• green network configuration
• ActiveAccounts (moved to nethserver-directory, calculated on-the-fly)

nethserver-backup

No migration is possible. The backup must be reconfigured.

nethserver-directory

Home directories: user’s home directoriy migrates into /var/lib/nethserver/home, admin’s home directory migrates into /var/lib/nethserver/migration/admin, and a symlink is created in /root/admin-migration-<TIMESTAMP>.

nethserver-hylafax

After migration check the configuration of incoming fax notification.

nethserver-httpd

The ibay-virtualhost relation has been designed differently from SME/NethService. An automatic migration is not always possible; the resulting configuration must be checked manually.

The global-pw-remote’ case is currently not implemented in NethServer and is mapped as global-pw. The reason is we do not want make distinctions between internal/external connections.

nethserver-mail-server

During pseudonyms migration,

• pseudonyms pointing to admin and shared accounts are mapped to postmaster, as any other account not existing in destination AccountsDB. Thus the resulting configuration requires post-migration supervision.
• recursive pseudonyms (pointing to another pseudonym) are flattened and a relation with a user or group account record is established.

Index of shared mailboxes is not migrated. Each user must re-share its own mail directory. To workaround this problem copy the original index file (/etc/dovecot/sharedmailbox/dict.db) to the new location (/var/lib/nethserver/vmail/shared-mailboxes.db) and restart dovecot. See http://wiki2.dovecot.org/SharedMailboxes/Shared for more information.

Forbidden “" in folder names

The dovecot plugin listplugin (http://wiki2.dovecot.org/Plugins/Listescape) is enabled, and uses backslash “" as escape character. If original folder names contains “", run the following command after post-migration mail synchronization, to rename them:

find /var/lib/nethserver/vmail/ -type d -regex '.*\\.*' -prune | (while read -r SRC; do echo mv -iv "$SRC" "${SRC//\\/\\5c}"; done )

nethserver-mail-filter

No wildcards expansions are supported by nethserver-mail-filter UI interface; only full mail addresses or domain names. The migration action must map email addresses in the form *domain.tld to domain names, and log a warning whenever another form of wildcard expansion is used.

Also recipient blacklists are not implemented and bayes DB is not migrated

Certificate Management

nethserver-base provides a set of templates that output PEM-formatted certificate parts:

• certificate/key RSA private key
• certificate/crt public certificate
• certificate/pem both key+crt parts

Configuration is inside the configuration database. Example:

pki=configuration
  KeyFile=
  CrtFile=
  ChainFile=
  CertificateDuration=365
  CommonName=

A certificate consumer daemon should expand those templates to its own certificate paths, by installing the proper configuration under /etc/e-smith/templates.metadata.

For instance nethserver-httpd adds the following template configuration:

• /etc/e-smith/templates.metadata/etc/pki/tls/private/localhost.key

TEMPLATE_PATH="certificate/key"
OUTPUT_FILENAME="/etc/pki/tls/private/localhost.key"
PERMS=0600
UID="root"
GID="root"

• /etc/e-smith/templates.metadata/etc/pki/tls/certs/localhost.crt

TEMPLATE_PATH="certificate/crt"
OUTPUT_FILENAME="/etc/pki/tls/certs/localhost.crt"
PERMS=0600
UID="root"
GID="root"

Set OUTPUT_FILENAME, PERMS, UID and GID values according to daemon configuration.

Default behavior

By default, CrtFile and KeyFile properties have empty values. In this case, nethserver-base generates a self-signed certificate during nethserver-base-update event.

Default SELinux-aware certificate locations are:

• /etc/pki/tls/private/NSRV.key: private key
• /etc/pki/tls/certs/NSRV.crt: CA certificate

A daily cron job checks certificate validity. If expired, the self-signed certificate is re-generated and certificate-update event is signaled.

Default certificate duration is set to 365 days. To change it:

db configuration setprop pki CertificateDuration 3650

The certificate Common Name is set to system FQDN. To override this value type:

db configuration setprop pki CommonName custom.cn

Let’s Encrypt

NethServer can request and renews Let’s Encrypt (LE) certificates. The main helper /usr/libexec/nethserver/letsencrypt-certs can be executed also from command line.

For more info, see:

/usr/libexec/nethserver/letsencrypt-certs -h

Database properties under pki key inside configuration database:

• LetsEncryptMail: (optional) registration mail for LE notifications
• LetsEncryptDomains: comma-separated list of domains added to certificate SAN field
• LetsEncryptChallenge: challenge to use for validating the certificate, default is http. It accepts also values like dns-<provider>. Where <provider> is the name of the DNS provider. See the full list of available DNS provider plugins by executing certbot -h certonly. More info at https://certbot.eff.org/docs/using.html?highlight=dns#dns-plugins.

DNS challenge

To use the DNS challenge, follow these steps:

• install the required certbot plugin plugin using yum; to see the list of available package use yum search certbot-dns
• set LetsEncryptChallenge property to correct DNS plugin
• configure all required properties accordingly to plugin documentation

When using the dns challenge, make sure to set extra properties accordingly to certbot configuration. All properties for the dns challenge should be in the form LetsEncrypt_<certbot_option>, where <certbot_option> is the option specific to certbot DNS plugin.

Digitial Ocean example

1. Install the plugin:

   yum install python2-certbot-dns-digitalocean
   

2. Configure the challenge type:

   config setprop pki LetsEncryptChallenge dns-digitalocean
   

3. Configure required props accordingly to https://certbot-dns-digitalocean.readthedocs.io/en/stable/:

   config setprop pki LetsEncrypt_dns_digitalocean_token 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
   

4. Request certificate for domain myserver.nethserver.org:

   /usr/libexec/nethserver/letsencrypt-certs -v -d myserver.nethserver.org
   

Yum plugin

The nethserver-yum package contains the nethserver_events which extends the post-RPM transaction hook. It executes the *-update event of each nethserver-* package involved in the transaction, preserving the order of RPM dependencies.

Signalling update events after RPM cleaning up assures that any old event handler and template fragments have been removed.

The configuration file is /etc/yum/pluginconf.d/nethserver.conf. Available options are:

• enabled: 1 (default) or 0, enable or disable the plugin
• verbose: 1 or 0 (default), enable or disable debugging messages

Caching

When NethServer is behind a proxy server you can force to bypass an intermediate caching by adding this option to the yum.conf file, under the main section:

http_caching=none

Be aware that yum.conf is a template and not using the cache can raise load on remote repository servers, use with care!

Web interface

The web interface is the main configuration method for NethServer. The main goal is to hide the system configuration complexity behind a simple and clear interface.

It runs on a special httpd instance and is accessible at: https://your_server:980 or https://your_server/server-manager (if nethserver-httpd module is installed).

Nethgui framework

Nethgui framework is provided with a set of components and basic classes to quickly build a modern web user interface. Its main goals are:

• provide a simplified API for coding, localizing and realizing the web user interface and behaviour for project:NethServer based products.
• perform user authentication and authorization;
• offer test facilities to help module testing and debugging.

Key features:

• Object-oriented and embeddable
• Plugin support
• REST-aimed architecture
• Almost no external dependencies
• User roles
• AJAX driven HTML5 Widget library
• Unobtrusive JavaScript code
• Compatible with console-based and mobile browsers

NethServer Web User Interface is built upon Nethgui framework.

Creating web UI module

In this page you can find a working example of a simple UI for a new module, but, first a little of theory.

Web UI structure

All code is organized in three main directories:

• /usr/share/nethesis/Nethgui: framework libraries (can be used for other projects)
• /usr/share/nethesis/NethServer: actual implementation of modules UI
• /usr/share/nethesis/nethserver-manager: web root directory

Nethgui is a MVC framework that aims to abstract the developer from the backend and frontend. This means that you don’t have to deal with reading and writing properties from DB, neither care about HTML, CSS and JavaScript on the client side. You just write the controller. Of course, if you want, you can play with all three layers but it shouldn’t be necessary in the most of cases.

Each module is composed by 4 parts:

• Controller: /usr/share/nethesis/NethServer/Module/.php
• View: /usr/share/nethesis/NethServer/Template/.php
• Translation: /usr/share/nethesis/NethServer/Language//NethServer\_Module*.php
• Inline help : /usr/share/nethesis/NethServer/Help/NethServer_Module*\ .html

If needed, a module can add extra resources:

• Specific authorization (JSON format): /usr/share/nethesis/NethServer/Authorization/.json - See Nethgui:Authorization
• Custom CSS: /usr/share/nethesis/NethServer/Css/.css - See Including CSS
• Custom JavaScript: /usr/share/nethesis/NethServer/Js/.js - See Including JS
• Unit tests: /usr/share/nethesis/NethServer/Test/Unit/NethServer/Module/.php - See Nethgui unit test
• Utility libraries: /usr/share/nethesis/NethServer/Tool/.php

Writing the code

We add a UI to the package nethserver-ejabberd module. The UI will expose 2 properties of the ejabberd db key.

Properties:

• status: start and stop the ejabberd daemon on boot, can be enabled or disabled
• WelcomeText: welcome text, can be anything

Controller

First, we create the controller which has 3 main functions:

• initializeAttributes: handle module position in menu
• initialize: bind the properties to the database and set the validator
• onParametersSaved: apply the configuration

Here is the controller (/usr/share/nethesis/NethServer/Module/Ejabber.php):

class Ejabber extends \Nethgui\Controller\AbstractController
{
     // Add the module under the 'Configuration' section,
     protected function initializeAttributes(\Nethgui\Module\ModuleAttributesInterface $base){
        return \Nethgui\Module\SimpleModuleAttributesProvider::extendModuleAttributes($base, 'Configuration', 30);
    }
         // Declare all parameters
    public function initialize(){
             parent::initialize();
            // Bind 'WelcomeText' view parameter to 'WelcomeText' prop in ejabberd key of configuration db
            $this->declareParameter('WelcomeText', Validate::ANYTHING, array('configuration', 'ejabberd', 'WelcomeText'));
    }

        // Execute actions when saving parameters
    protected function onParametersSaved($changes) {
        // Signal nethserver-ejabberd-save event after saving props to db
        $this->getPlatform()->signalEvent('nethserver-ejabberd-save@post-process');
    }
 }

View

Show all fields using built-in functions. If needed, you can add extra HTML markup but remember that the output must be functional on any device (desktop, mobile, text browser, etc).

Template (/usr/share/nethesis/NethServer/Template/Ejabber.php):

echo $view->header()->setAttribute('template', $T('Ejabber_Title'));

// add simple panel
echo $view->panel()
    //add 'status' parameter checkbox with value when checked and unchecked
    ->insert($view->checkbox('status', 'enabled')->setAttribute('uncheckedValue', 'disabled'));
    //add 'WelcomeText' text input field
echo $view->panel()->insert($view->textInput('WelcomeText'));
;

// show submit and help buttons
echo $view->buttonList($view::BUTTON_SUBMIT | $view::BUTTON_HELP);

Translation

Translation files, are simple PHP files containing an associative array. All module language files are placed in /usr/share/nethesis/NethServer/Language/<lang>. Given a module with name “Test”, the english language file will be /usr/share/nethesis/NethServer/Language/en/NethServer_Module_Test.php.

Warning messages about missing translations can be found in /var/log/messages after Nethgui debug is enabled. To enable the debug, use index_dev.php on urls, eg: https://<ipaddress>/index_dev.php/en/<module>.

English translation (/usr/share/nethesis/NethServer/Language/en/NethServer_Module_Ejabber.php):

<?php

$L['Ejabber_Title'] = 'Chat server';
$L['status_label'] = 'Enable Ejabber chat server';
$L['WelcomeText'] = 'Welcome!';

Inline help

Help pages are RST documents compiled into xHTML pages at package build time.

===========
Chat server
===========

Ejabber is a chat server that implements the Jabber/XMPP protocol Jabber / XMPP, it support TLS on standard XMPP ports (5222 or 5223).

The chat server uses system users to login.

More examples

More examples can be found here or browsing the existing modules.

Dashboard

The dashboard module is the landing page of NethServer Web UI. It aims to give an overview of system status. Dashboard is fully pluggable and extensible: each NethServer module can add it’s own tab or even a widget inside the System status or Applications tab.

There are three default tabs:

• System status
• Services
• Applications

The dashboard module heavily uses Javascript but works correctly even on smartphones.

SystemStatus tab

It’s the main tab composed by three main submodules:

• System release: show the system release
• Resource usage:
  • system load
  • cpu number
  • uptime
  • physical memory usage statistics
  • virtual memory (swap) usage statistics
  • root partition usage statistics
• Network
  • configuration of each assigned netwok interface (IP address, netmask, etc.)
  • real-time statistics of received and transmitted data
  • host name, domain name, default gateway

This tab can be extended using plugins.

Services tab

Service tab lists all system services, including current status (running or stopped), configured status (enabled or disabled) and associated TCP and UDP ports. This tab cannot be customized.

Applications tab

Applications tab lists all installed application with extra information. Each package can create a special PHP class which describes the installed application.

Extending the dashboard

The dashboard heavily uses Nethgui framework to render all information, so you need a little bit of understanding on how project:Nethgui PHP framework actually works. Don’t panic!

Nethgui is a MVC framework specially designed for NethServer web UI tasks. All files are inside /usr/share/nethesis/NethServer path, with the following sub-paths:

• Module: controller files
• Template: template files
• Language: language files

Application custom widget

This is the simplest widget inside the Applications tab. This kind of widget should be used by all web applications. Let see the nethserver-collectd-web example. The package must include a PHP file inside Module/Dashboard/Applications/ directory. The script will describe a new class which extends the AbstractModule class and implements the ApplicationInterface interface. A little scary, eh? There is the full example:

<?php

class Collectd extends \Nethgui\Module\AbstractModule implements \NethServer\Module\Dashboard\Interfaces\ApplicationInterface
{
   public function getName()
   {
       return "Collectd Web";
   }

   public function getInfo()
   {
        $cweb = $this->getPlatform()->getDatabase('configuration')->getKey('collectd-web');
        $hostname = $this->getPlatform()->getDatabase('configuration')->getType('SystemName');
        $domain = $this->getPlatform()->getDatabase('configuration')->getType('DomainName');
        return array(
           'url' => "http://$hostname.$domain/".$cweb['alias']
        );
   }
}

Not so scary :)

Basically, the ApplicationInterface requires to implement two simple methods:

• getName(): return the name of the application. The name is used to alphabetically sort all the applications inside the Application tab.
• getInfo(): return an arbitrary associative array. All <key,value> pairs will be printed inside a little auto-generated widget. If the array contains a key starting with ‘url’ string, the value of the key will be wrapped inside an <a href=''> HTML tag.

We can now analyze the above getInfo function implementation. First we read the collectd-web key from the configuration db. All properties are then accessible as an associative array. Then we read the value of the special keys SystemName and DomainName. Finally we fill the return array with the url where the web application will be accessible.

System status custom widget

Let’s take a SystemRelease class which add a widget inside the SystemStatus tab:

• Controller: Module/Dashboard/SystemStatus/SystemRelease.php
• Template: Template/Dashboard/SystemStatus/SystemRelease.php
• English translation: Language/en/NethServer_Module_Dashboard_SystemStatus_SystemRelease.php
• Italian translation: Language/it/NethServer_Module_Dashboard_SystemStatus_SystemRelease.php

Translation files are simple PHP associative arrays. The language files in the examples are self-explanatory.

You can find all examples inside the nethserver-ui-examples repository.

Controller

First of all, let’s introduce few keys concepts about HTTP request handling in Nethgui. When the browser open an URL, for example /SystemRelease, the system will search for a module named SystemRelease. If the request contains some query parameters, the server side module will invoke two functions process() and prepareView(), if no query is specified only the function prepareView will be invoked.

The process will handle the query and prepare all data for the prepareView function. For example, here is the process implementation of a simple SystemRelease module:

<?php
...
public function process()
{
    $this->release = $this->readRelease();
}

Simple. The process will invoke a private method readRelease which reads a file, and save the result on a private attribute. All data are now available and we have to send them back using the prepareView function:

<?php
...
public function prepareView(\Nethgui\View\ViewInterface $view)
{
   //if no query is specified, make sure to initialize the data
   if (!$this->release) {
       $this->release = $this->readRelease();
   }
   $view['release'] = $this->release;
}

The release attribute is mapped inside the $view array, ready to be sent to the client. When the client requests the page for the first time with no query, we need to fill the release attribute because the process is not previously called. Then all the $view data will be used to fill the HTML template. If the client send a JSON request with a timestamp as a parameter (which is the standard behavior for ajax calls) the module will invoke process and prepareView, then all data will be formatted in JSON format.

Request examples:

• HTML rendering: /Dashboard/SystemStatus/SystemRelease
• JSON response: /Dashboard/SystemStatus/SystemRelease.json

Finally, if you want to order the widget inside the System Status tab, you should define a variable $sortId, like:

public $sortId = 40;

View

It’s time to create a simple template to show the data from the controller. Below there’s an example using a built-in CSS class.

<?php
echo "<div class='dashboard-item'>";
echo "<dl>";
echo $view->header()->setAttribute('template',$T('release_title'));
echo "<dt>".$T('release_label')."</dt><dd>"; echo $view->textLabel('release'); echo "</dd>";
echo "</dl>";
echo "</div>";

This is a static template without any use of Javascript. Inside the template file, you always have access to the $view variable where all data are stored by the previous explained prepareView function. There also a very useful function called $T(…) used for translations. The most important part of this examples is the call $view->textLabel(‘release’). This line is an helper which extract the release variable from the view an wrap it a span HTML tag identified by an auto-generated class (useful for Javascript processing).

A full example can be found in the nethserver-ui-examples repository under the dashboard directory.

Custom Tab

Any NethServer package can add a custom tab inside the dashboard. To create a new tab you have to write a class extending an existing controller inside the /usr/share/nethesis/NethServer/Module/Dashboard directory. See “API”:http://dev.nethserver.org/nethgui/Documentation/Api/ for a list of available controllers.

See mytab example in nethserver-ui-examples repository for more information.

Creating inline help pages

The online documentation is written in RST format. Each document is converted to XHTML at package build time. After installing a package, the server-manager web application serves the corresponding XHTML version.

To get a list of help pages visit http://<your_server>:980/<lang>/Help. The page will show a list of available documents on the left column and a list of templates to generate a new help page on the right column. You need to create an help page for each supported language using the template: just copy and paste the code to the destination file.

All help pages must be saved to the following absolute path: /usr/share/nethesis/NethServer/Help/NethServer_Module_<name>.rst

Editing rules

To write an online help document, respect the following rules.

• The document must start with a title such as

==============
Document title
==============

• If a page is divided in tab, each tab title is in the form:

Tab title
=========

• Each field must be described as a definition list. The indentation of

My field
     This is my description

• A field description can contain a bullet list:<pre>

My field
    This is my description

    * First element
    * Second element

• If any words needs emphasis, use the asterisk character:

This is my *important* word.

# Actions, usually identified by buttons like Create or Configure should be given a separate section. Where two actions have similar forms, like Create and Modify, they can be merged together.

Create / Modify
---------------

• The Delete action should be documented only to clarify its side

  effects.

Creating help for plugin modules

Some NethServer modules have a plugin behavior which means the module will load all help documents inside a well known directory. Loaded files will be rendered as parts of the parent document.

Inside the parent document, add an include directive like this:

.. raw:: html

   {{{INCLUDE NethServer_Module_User_Plugin_*.html}}}

When creating inline help for web ui plugin, the following rules apply:

# do not repeat the name of parent module # use the ^ character for headers # add this comment at the top of the file

.. --initial-header-level=3

The actual level value depends on the type of plugin. Usually 2 or 3 applies. For instance, see NethServer_Module_SharedFolder_Plugin_Samba.rst inside nethserver-samba package.

Building docs inside RPMS

RST documentation needs to be compiled in HTML. To include HTML files inside the RPM, remember to add this macro in the spec file, under the %build section:

%{makedocs}

Example:

%build
...
%{makedocs}
...
perl createlinks

RST editors

Here some RST editor with syntax hilighting.

Windows platform:

• http://notepad-plus-plus.org/
• www.geany.org
• http://www.sublimetext.com/ (non-free)

Linux platform:

• vim (of course!)
• gedit

Web:

• http://rst.ninjs.org/
• https://notex.ch/

TODO API

The TODO API is specific for the Server Manager web application. It is designed to execute a list of checks and possibly report the outcome to the admin user.

An RPM can install one or more executable scripts under /etc/nethserver/todos.d/.

• The script must print the results formatted according to JSON [1] and the following schema:

  {
      "text": "free text",
      "icon": "info-circle",
      "action": {
          "url": "/User",
          "label": "Link label"
      }
  }
  

  icon and action keys are optional. The only required key is text. The url value is actually an absolute path to the Server Manager module. Future versions may support real URLs.

• If the script exit code is non-zero, or if the output is not correctly JSON-encoded, an error message is sent to the system log.

• The script must be aware of locale settings, as its output is displayed on the user’s browser [2].

The executable helper /usr/libexec/nethserver/admin-todos is responsible for the invocation of scripts, validation of output and error reporting. It is executed by the AdminTodo UI module.

The AdminTodo known callers are:

• Dashboard
• Software center
• Network
• Backup (configuration)

Translations

A TODO script must be locale-aware. Use the gettext library for code internationalization (i18n) and follow guidelines from Internationalization.

References

[1]	JSON (JavaScript Object Notation) is a lightweight data-interchange format. http://json.org/

[2]	GNU gettext utilities http://www.gnu.org/software/gettext/

Building RPMs

To build RPMs for NethServer the following methods are provided:

• nethserver-makerpms for local builds with Podman [1] and GitHub Actions automated builds
• nethserver-mock local build with the Mock [2] configuration files pointing to NethServer YUM repositories

Both methods allow to build also RPMs from other projects. Check out the documentation carefully.

Development vs Release builds

A common convention about .spec files says that the Release tag has to append the RPM dist macro as suffix in a similar way:

Release: 1%{?dist}

In NethServer,

• when the package is built for a release, use the dist value .ns7 (or .ns6…);
• when the package is compiled for QA/testing, the dist macro should reference the source code git commit hash. For instance, dist can be set to .2.g8f7ddad.ns7.

This convention ensures that each RPM is unique and it is always possible to reproduce the build and its history.

In particular, when building for QA/testing, the build methods linked above prepend the number of commits in the git log history since the last tagged release commit, followed by .g, followed by a short git commit hash reference. If the last commit in the repo has a release tag, the dist macro is set to .ns7 instead.

A tagged release commit is a git commit with a tag, starting with a digit and not containing any “-” (minus) symbol. For instance 0.1.2r1 and 0ok are valid release tags, whilst v0.1.2 and 0.1.2-1 are not.

The releasetag command helps to create a properly tagged release commit in the RPM source code repository.

Publishing RPMs

Once RPMs are built, they can be uploaded to packages.nethserver.org with sftp. Ask on community.nethserver.org to obtain the SFTP account required for this purpose.

Use the uploadrpms command to publish the RPMs in the current working directory. The command looks like the following

$ uploadrpms username@packages.nethserver.org:nscom/7.8.2003/nethforge-testing *.rpm

Replace 7.8.2003 with the correct NS version number. Also replace nethforge-testing with the target repository name.

As alternative to the above manual procedure, it is possible to run the build on GitHub Actions: it automatically publishes the RPMs to packages.nethserver.org, too. Refer to nethserver-makerpms documentation.

References

[1]	Podman is a daemonless Linux container engine. https://podman.io/

[2]	Mock is a tool for building packages. http://fedoraproject.org/wiki/Projects/Mock

Building ISO

To create a NethServer ISO on a NethServer system, follow these steps:

1. Install nethserver-createiso package

2. Make sure mock cache is clean, execute as root user:

   rm -rf /var/cache/mock/nethserver-iso*
   

3. Log in as a non-privileged user, member of the mock group

4. Download CentOS minimal ISO

5. Run createiso command

   createiso -n nethserver -v 7.9.2009 -i CentOS-7-x86_64-Minimal-2009.iso
   

Package groups

The Software center page in the Server Manager displays the YUM groups (comps) metadata. This is an XML file added to a YUM repository metadata which describes the packages groups and categories composition.

Submit a change proposal

To modify how groups and categories are displayed in the Software center page a GitHub account is required.

1. edit (on the GitHub web interface) an .xml.in file

• https://github.com/NethServer/comps/blob/master/nethforge-groups.xml.in
• https://github.com/NethServer/comps/blob/master/nethserver-groups.xml.in

2. open a pull request

How to edit .xml.in files

Information about *.xml.in files is available here:

• about the workflow, see the repository README
• about the file format and the comps XML structure, see the Fedora Project Wiki

nethserver-antivirus

Manage default options for all clamav instances.

The configuration is saved inside the configuration database in the clamd key.

Properties:

• OfficialSignatures: can be enabled or disabled. If set to disabled, all default signatures will be deleted and freshclam will be disabled.
• UnofficialSignaturesRating: can be low, medium or high. Select the rating risk of all sources. An higher rating means more virus detected but also an increased probability of false positives.

Database example:

clamd=configuration
   OfficialSignatures=disabled
   UnofficialSignaturesRating=low

Disable official signatures

Official signatures find only very few virus but have a big memory usage. To disable the official signatures:

config setprop clamd OfficialSignatures disabled
signal-event nethserver-antivirus-update

Select signature rating

Unofficial signatures can be selected using their rating. To change the rating use:

config setprop clamd UnofficialSignaturesRating medium
signal-event nethserver-antivirus-update

Cockpit API

dashboard/read

This api returns the information to show in dashboard page. The following data is retrieved for every ClamAV instance installed:

• current status of the instance
• number of malware found
• number of signatures loaded
• memory used by the ClamAV instance
• number of occurrences of malware by type

Input

• no input

Output example

{
  "rspamd": {
    "installed": true,
    "active": true,
    "malwareFound": 0,
    "signatures": 172557,
    "malwareStats": [],
    "memoryUsedKb": 250808,
    "sinceIso": "2019-09-23T12:19:01+0200"
  },
  "squidclamav": {
    "installed": true,
    "active": false,
    "malwareFound": 0,
    "signatures": 172557,
    "malwareStats": [],
    "memoryUsedKb": 0,
    "sinceIso": "2019-09-23T12:18:04+0200"
  }
}

logs/read

This api returns the list of installed ClamAV instances

Input

• no input

Output example

{
  "installedInstances": [
    "clamd@rspamd",
    "clamd@squidclamav"
  ]
}

settings/read

This api retrieves clamd configuration and the memory used by every ClamAV instance installed.

Input

• appInfo: clamdConfig or totalMemory

Input example (clamdConfig)

{
  "appInfo": "clamdConfig"
}

Output example (clamdConfig)

{
  "configuration": {
    "type": "configuration",
    "name": "clamd",
    "props": {
      "OfficialSignatures": "enabled",
      "UnofficialSignaturesRating": "low"
    }
  }
}

Input example (totalMemory)

{
  "appInfo": "totalMemory"
}

Output example (totalMemory)

{
  "totalMemory": "991"
}

settings/validate

This api validates input for clamd configuration.

Input

• OfficialSignatures: enabled if ClamAV official signatures should be enabled, else disabled
• UnofficialSignaturesRating: Third-party signatures rating, valid values are low, medium or high. Higher rating means more blocked threats but also a more possibility of false positive. Recommended ratings are low and medium

Input example

{
  "OfficialSignatures": "disabled",
  "UnofficialSignaturesRating": "low"
}

Output example

{
  "state": "success"
}

settings/update

This api updates clamd configuration.

Input

• same as settings/validate

nethserver-avahi

Configure avahi-daemon to start at boot. The Avahi mDNS/DNS-SD daemon implements Apple’s Zeroconf architecture.

It’s used by CUPS for printers auto-discovery. See cups-section.

Database:

avahi-daemon=service
   UDPPort=5353
   access=private
   status=enabled

nethserver-backup-config

Configuration backup contains only system configuration files (passwd, config databases, etc). It’s scheduled to be executed every night and will create a new archive only if any file is changed in the last 24 hours.

Backup libraries use conf.d directory behavior (see perldoc NethServer::Backup). When a backup is started, the system will search for all files in /etc/backup-config.d directory. This directory can contain .include and .exclude files. Each file contain a list of file to include/exclude into/from the backup.

Example file /etc/backup-config.d/nethserver-base.include

/etc/e-smith/templates-custom
/etc/e-smith/templates-user-custom
/etc/ssh
/etc/sudoers
/etc/passwd
/etc/shadow
/etc/group
/etc/gshadow

Exclusions are evaluated after all inclusions.

All libraries are inside the nethserver-backup-config package.

Configuration backup

The nethserver-backup-config package implements the backup of configuration and relies on the backup-config key inside the configuration database.

Properties:

• status : enable or disable the automatic backup, can be enabled or disabled. Default is enabled.

Backup

The main command is /sbin/e-smith/backup-config which starts the backup process (if enabled). The backup process has 3 steps:

• pre-backup-config event: used to prepare data, for example a LDAP dump of users
• backup-config-execute action: actually execute the backup if any file is changed in the last 24 hours. The backup file is saved in /var/lib/nethserver/backup/backup-config.tar.xz (see perldoc NethServer::BackupConfig)
• post-backup-config event: used to post-process the backup file, for example to copy the backup to a remote server or encrypting the archive

The configuration backup runs every night and it creates a new backup only if:

• destination file does not exist
• or new files are added or removed to/from the backup set
• or content of any file inside the set is changed

This package does not provide any default action in the pre-backup-config and post-backup-config events. But you can create a script inside the post-backup-config event to copy the configuration backup to a remote machine using, for example, the SSH protocol.

The configuration backup is included inside the data backup.

Restore

The main command is /sbin/e-smith/restore-config which starts the restore process:

• pre-restore-config event: used to prepare the system, for example stop a running service
• restore-config-execute action: search for a backup file in the well-known directory (see above) and restore it
• post-restore-config event: used to apply restored configuration, for example reinstall packages and load the LDAP dump

This package does not provide any action in the pre-restore-config event.

Please note that if no /var/lib/nethserver/backup/backup-config.tar.xz file is found, the restore-config command will try to access the data backup to retrieve the configuration backup.

The restore-config options:

• --no-reinstall : disable package reinstall during restore
• --mask-unit=<name> : use systemd to mask the specified unit (example: disable httpd-admin restart during restore)
• --skip-network: do not restore the network configuration

Customization

Add custom include/exclude inside following files:

• /etc/backup-config.d/custom.include
• /etc/backup-config.d/custom.exclude

nethserver-backup-data

This module implements data backup for NethServer using different engines.

Available engines:

• duplicity: execute a full backup once a week, an incremental snapshot all other days of the week. Compression is enabled by default, encryption is not currently supported. It supports only storage backend which can be mounted on a local directory. Used also for the single backup.
• restic: always execute incremental backups using file deduplication. Encryption is always enabled, compression is not implemented. It supports local and remote backends.
• rsync: Time Machine-style backup using rsync. Very fast and reliable, the destination contains just regular files which can be easily accessed.

The nethserver-backup-data package requires nethserver-backup-config.

Configuration

Backups can be scheduled in different hours to multiple storage backends.

Global properties

This configuration is applied to all backups. It uses the key backup-data inside configuration database.

Properties:

• IncludeLogs: if enabled, add /var/log directory to backup, can be enabled or disabled. Default is disabled.

Local properties

This configuration is applied only to the selected backups. Every backup record is saved inside the backups database. Each record can have 3 different types:

• duplicity
• restic

• rsync

The key of the record is referred as the backup name.

Common properties:

• status : enable or disable the automatic backup, can be enabled or disabled. Default is enabled. Regardless of this property, the backup is always executed if started manually
• BackupTime: time of the scheduled backup. Must be in the cron-style syntax: Es. 15 7 * * *. Runs on 7:15.
• VFSType : set the backup medium, can be usb, cifs, nfs or webdav.
• SMBShare: contains the Samba share name
• SMBHost : host name of the SMB server
• SMBLogin : login user for the SMB server
• SMBPassword : password for the SMB server
• USBLabel : contains the filesystem label
• NFSHost : host name of the NFS server
• NFShare : contains the NFS share name
• Notify: if set to always, always send a notification with backup status; if set to error, send a notification only on error; if set to never, never send a notification
• NotifyTo: send the notification to given mail address, default is root@localhost
• WebDAVUrl : contains the WebDAV URL address
• WebDAVLogin : login user for the WebDAV server
• WebDAVPassword : password for the WebDAV server
• CleanupOlderThan : time to retain backups, accept duplicity syntax (eg. 7D, 1M). Default is: never (keep all backups)

Supported VFSType:

• cifs : save the backup on a remote SMB server. Authentication is mandatory.
• nfs : save the backup on a remote NFS server. No authentication supported.
• usb : save the backup on a USB device. The device must have a writable filesystem with a custom label. When the backup is started, the system will search for an USB device with the filesystem label saved in USBLabel.
• webdav : save the backup on a WebDAV server. When using a secure connection make sure the target WebDAV server has a valid SSL certificate, otherwise the system will fail mounting the filesystem.

Backward compatibility

To retain the backward compatibility with the old “single backup” feature, a backup named backup-data has the following special features:

• can have a NotifyFrom prop to specify the sender address of notification mail
• the backup can be modified from the old Server Manger and the status is reported inside the dashboard
• can be selectively restored using nethserver-restore-data package

Backup

The main command is /sbin/e-smith/backup-data -b <name> which starts the backup process. The backup is composed of three parts:

• pre-backup-data event: prepare the system (eg. dump of mysql tables)
• /etc/e-smith/events/actions/backup-data-<program> action: execute the backup This actions must implement full/incremental logic and should also take care to mount and umount the destination
• post-backup-data: cleanup. Actions in this event can also implement retention policies

Logs and wrapper

Everything is logged to standard output and standard error.

If the backup is executed using backup-data-wrapper, a new log will be created inside /var/log/backup.

After backup execution, the wrapper will also call all executable scripts inside /etc/backup-data.hooks/ directory.

Each script is invoked with the following parameters:

• backup name
• log file
• backup exit code

Default hooks

• backup-dashboard-status: save the status of the backup /var/spool/backup/status-<backup_name>
• backup-notify: send backup notifications mails

Adding a backup

1. Create a backup record with all required options. Example: create a restic backup named t1 using sFTP backend:

   db backups set t1 restic VFSType sftp SftpHost 192.168.1.123 SftpUser root SftpPort 22 SftpDirectory /mnt/t1 status enabled BackupTime 3:00 CleanupOlderThan 30D Notify error NotifyTo root@localhost
   

2. Enable the configuration:

echo -e "Nethesis,1234" > /tmp/t1-password; signal-event nethserver-backup-data-save t1  /tmp/t1-password

Start a backup

Start the backup, by passing the name of the backup to backup-data command. Example:

backup-data -b t1

Disk usage

Each backup script creates statistics about disk utilization on the backup destination. Statistics are available only for: cifs, nfs and usb.

Data are saved inside /var/spool/backup/disk_usage-<backup_name>.

Indexing

In the pre-backup-data event the disk analyzer (Duc) make an indexing of filesystem, useful to create the graphical tree.

The name of the actions is /etc/e-smith/events/actions/nethserver-restore-data-duc-index and it compose the JSON file to create the navigable graphic tree.

The indexing feature is limited to the backup named backup-data.

Customization

Global

Add custom include/exclude inside following files:

• /etc/backup-data.d/custom.include
• /etc/backup-data.d/custom.exclude

This configuration is applied to all backups.

Local

Each backup can override the global list of include/exclude by creating two special files:

• /etc/backup-data/<name>.include
• /etc/backup-data/<name>.exclude

Where name is the name of the backup.

Retention policy

All backups can be deleted after a certain amount of time. Cleanup process takes place in post-backup-data event. See CleanupOlderThan property.

Restore

Restore from command line

The main command is /sbin/e-smith/restore-data -b <name> which starts the restore process:

• pre-restore-data event: used to prepare the system (Eg. mysql stop)
• restore-data-<program> action: search for a backup in the configuration database and restore it
• post-restore-data event: used to inform programs about new available data (eg. mysql restart)

To restore all data into the original location, use:

restore-data -b <name>

To restore a file or directory, use:

restore-file -b <name> <position> <path>

List backup contents

The list of file inside each backup can be obtained using:

/sbin/e-smith/backup-data-list -b <name>

Duplicity

The default program used for backup is duplicity using the globbing file list feature. Encryption is disabled and duplicity cache is stored in /var/lib/nethserver/backup/duplicity/ directory. We plan to support all duplicity features in the near future.

See http://duplicity.nongnu.org/ for more information.

Extra options

Properties valid only for duplicity engine, see “Single backup” section for an explanation of each property:

• Type : can be full or incremental. If full, a full backup will be executed every time. If incremental, a full backup will be executed once a week at FullDay, all other backups will be incremental
• FullDay : number of day of the week when a full backup will be executed. Can be a number from 0 (Sunday) to 6 (Saturday). Defaults is 0.
• VolSize : size of chunks in MB, if supported by Program. Default is 250

Storage backends

Supported VFSType :

• usb
• cifs
• nfs
• webdav

Listing backup sets

To list current backup sets:

1. Mount the backup directory
2. Query duplicity status
3. Umount the backup directory

Just execute:

/etc/e-smith/events/actions/mount-`config getprop backup-data VFSType`
duplicity collection-status --no-encryption --archive-dir /var/lib/nethserver/backup/duplicity/ file:///mnt/backup/`config get SystemName`
/etc/e-smith/events/actions/umount-`config getprop backup-data VFSType`

Restic

Implement backup engine using restic (https://restic.net/), it can be used as duplicity replacement for standard backup or as multiple backup.

In restic, cleanup operations are composed by two commands: forget, to remove a snapshot, and prune, to actually remove the data that was referenced by the deleted snapshot. The prune operation is quite slow and should be executed at least once a week.

Extra options

• Prune: execute the pruning on the specified time. Valid values are:
  • always: run the prune every time at the end of backup
  • a number between 0 and 6: run the prune on the selected week day (0 is Sunday, 1 is Monday)

Storage backends

Supported VFSType :

• usb
• cifs
• nfs
• webdav
• s3: Amazon S3 (or compatible server like Minio)
• sftp: FTP over SSH
• b2: BackBlaze B2
• rest: Restic REST server

sftp

SFTP

Connection to remote host uses a specific public key. A password is needed only once to copy the public key to the remote host. SSH client configuration is added to /etc/ssh/sshd_config.

Properties:

• SftpHost: SSH host name or IP address
• SftpUser: SSH user
• SftpPort: SSH port
• SftpDirectory: destination directory, must be writable by SSH user

Example:

db backups set t1 restic status enabled BackupTime '15 7 * * *' CleanupOlderThan 30D Notify error NotifyTo root@localhost Prune 1 \
VFSType sftp SftpHost 192.168.1.2 SftpUser root SftpPort 22 SftpDirectory /mnt/t1
echo -e "Nethesis,1234" > /tmp/t1-password; signal-event nethserver-backup-data-save t1  /tmp/t1-password

The temporary file containing the password will be deleted at the end of nethserver-backup-data-save event.

s3

Amazon S3 (https://aws.amazon.com/s3/) compatible (like https://www.minio.io/).

Properties

• S3AccessKey: user access key
• S3Bucket: bucket (directory) name
• S3Host: S3 host, use s3.amazonaws.com for Amazon
• S3SecretKey: secret access key

Example:

db backups set t1 restic VFSType s3 BackupTime '15 7 * * *' CleanupOlderThan never Notify error NotifyTo root@localhost status enabled Prune always\
S3AccessKey XXXXXXXXXXXXXXXXXXXX S3Bucket restic-demo S3Host s3.amazonaws.com S3SecretKey xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx status enabled
signal-event nethserver-backup-data-save t1

How to setup Amazon S3 access keys: https://restic.readthedocs.io/en/stable/080_examples.html

b2

BackBlaze B2 (https://www.backblaze.com/b2/cloud-storage.html)

Properties:

• B2AccountId: B2 account name
• B2AccountKey: B2 account secret key
• B2Bucket: B2 bucket (directory)

Example:

db backups set t1 restic VFSType b2 BackupTime '15 7 * * *' CleanupOlderThan never Notify error NotifyTo root@localhost status enabled \
B2AccountId B2AccountId xxxxxxxxxxxx B2AccountKey xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx B2Bucket restic-demo
signal-event nethserver-backup-data-save t1

Rest

Restic REST server (https://github.com/restic/rest-server)

Properties:

• RestDirectory: destination directory
• RestHost: REST server host name or IP address
• RestPort: REST server port (default for server is 8000)
• RestProtocol: REST protocol, can be http or https
• RestUser: user for authentication (optional)
• RestPassword: password for authentication (optional)

Example:

db backups set t1 restic VFSType rest BackupTime '15 7 * * *' CleanupOlderThan never Notify error NotifyTo root@localhost status enabled \
RestDirectory t1 RestHost 192.168.1.2 RestPassword mypass RestPort 8000 RestProtocol http RestUser myuser
signal-event nethserver-backup-data-save t1

Database example:

t2=restic
   BackupTime=1 7 * * *
   CleanupOlderThan=never
   Notify=error
   NotifyTo=root@localhost
   SMBHost=192.168.1.234
   SMBLogin=test
   SMBPassword=test
   SMBShare=test
   VFSType=cifs
   status=enabled
t3=restic
   BackupTime=15 7 * * *
   CleanupOlderThan=never
   NFSHost=192.168.1.234
   NFSShare=/test
   Notify=error
   NotifyTo=root@localhost
   VFSType=nfs
   status=enabled

REST server

To manually install the REST server, download it from https://github.com/restic/rest-server/releases and save it under /usr/local/bin/rest-server, example Linux 64bit:

R=0.9.7; wget https://github.com/restic/rest-server/releases/download/v$R/rest-server-$R-linux-amd64.gz -O - | zcat > /usr/local/bin/rest-server
chmod a+x /usr/local/bin/rest-server

Then configure it for NethServer:

wget https://raw.githubusercontent.com/restic/rest-server/master/examples/systemd/rest-server.service -O - | sed 's/www\-data/apache/g' > /etc/systemd/system/rest-server.service
systemctl daemon-reload
systemctl start rest-server
systemctl enable rest-server
config set rest-server service TCPPort 8000 access green status enabled
signal-event firewall-adjust

rsync

Implement Time machine-style backup engine using rsync_tmbackup.sh (https://github.com/laurent22/rsync-time-backup), based on rsync (https://rsync.samba.org/). It can be used as duplicity replacement for standard backup or as multiple backup.

Retention policy

Backup sets are automatically deleted when the disk is getting full.

More info on expiration strategy: https://github.com/laurent22/rsync-time-backup#backup-expiration-logic

Storage backends

Supported VFSType :

• usb
• cifs
• nfs
• webdav
• sftp: FTP over SSH

sftp

SFTP

Connection to remote host uses a specific public key. A password is needed only once to copy the public key to the remote host. SSH client configuration is added to /etc/ssh/sshd_config.

Properties:

• SftpHost: SSH host name or IP address
• SftpUser: SSH user
• SftpPort: SSH port
• SftpDirectory: destination directory, must be writable by SSH user

Example:

db backups set t1 rsync status enabled BackupTime '15 7 * * *' Notify error NotifyTo root@localhost \
VFSType sftp SftpHost 192.168.1.2 SftpUser root SftpPort 22 SftpDirectory /mnt/t1
echo -e "Nethesis,1234" > /tmp/t1-password; signal-event nethserver-backup-data-save t1  /tmp/t1-password

The temporary file containing the password will be deleted at the end of nethserver-backup-data-save event.

Database example:

t2=rsync
   BackupTime=1 7 * * *
   Notify=error
   NotifyTo=root@localhost
   SMBHost=192.168.1.234
   SMBLogin=test
   SMBPassword=test
   SMBShare=test
   VFSType=cifs
   status=enabled
t3=rsync
   BackupTime=15 7 * * *
   NFSHost=192.168.1.234
   NFSShare=/test
   Notify=error
   NotifyTo=root@localhost
   VFSType=nfs
   status=enabled

nethserver-base

This package implements all core configuration.

Network

Network configuration is saved inside the NetworksDB (/var/lib/nethserver/db/networks).

Example of a database containing an interface:

eth0=ethernet
   bootproto=none
   device=eth0
   gateway=192.168.1.254
   ipaddr=192.168.1.1
   netmask=255.255.255.0
   onboot=yes
   role=green

Each entry describes a network interface according to CentOS/RHEL specification for network-scripts files:

<device_name> = type
       role = <role>
       <param> = <value>

The type variable is the type of interface. Valid values are:

• ethernet
• bond
• bridge
• alias
• ipsec
• xdsl

The <device_name> variable is the name for the device.

The role property is a mandatory parameter which describes the interface role. Valid values are:

• green
• orange
• blue
• red

If the role property is empty, the interface is not used by the system.

There are also 3 special roles:

• bridged: interface is part of a bridge
• slave: interface is part of a bond
• alias: interface is an alias of another interface
• xdsl-disabled: xdsl disabled interface

See also Roles and zones for the meaning of each color.

All <param>/<value> are all valid CentOS network parameter for the specified interface. All parameters must be lowercase. Example:

• ippaddr
• dhcp_hostname
• netmask
• slave
• …

All parameters will be mapped 1-to-1 to the configuration file

Example

One green ethernet:

db networks set eth0 ethernet role green ipaddr 192.168.1.4 netmask 255.255.255.0 network 192.168.1.0 onboot yes bootproto static

File content:

green=ethernet|bootproto|static|device|green|ipaddr|192.168.1.4|netmask|255.255.255.0|network|192.168.1.0|onboot|yes|role|green

Bond options

Any property starting with BondOpt prefix is used as bonding options.

Example:

bond0=bond
   BondOptMode=0
   BondOptMiimon=80
   bootproto=none
   gateway=192.168.1.100
   ipaddr=192.168.1.2
   netmask=255.255.255.0
   role=green

Templates

The network database can be manipulated using the esmith::NetworksDB perl module. For more information use:

perldoc esmith::NetworksDB

If you need to access the local IP address within a template, use this code snippet:

use esmith::NetworksDB;
my $ndb = esmith::NetworksDB->open_ro() || return;;
my $LocalIP = $ndb->green()->prop('ipaddr') || '';

Note

Old templates used a variable called LocalIP to access the green IP address. This variable is no more available.

Events

All network configurations are applied by interface-update event.

Database initialization

All interfaces are imported from configuration files to database using the script: /usr/libexec/nethserver/update-networks-db .

The networks database is updated Whenever an interface is plugged into the system.

DHCP on red interfaces

When configuring a red interface in DHCP mode, enable also the above options:

• peer_dns to avoid resolv.conf overwriting from dhclient
• persistent_dhclient to enforce dhclient to retry in case of lease request errors

Remember also to remove all gateway IP address from green devices. This configuration will create the correct routes and correctly set DHCP options on dnsmasq.

Bridge

Create a bridge interface from command line. The new interface will have green role (eth0 was the previous green interface):

db networks delprop eth0 ipaddr netmask bootproto
db networks setprop eth0 role bridged bridge br0
db networks set br0 bridge bootproto static device br0 ipaddr 192.168.1.254 netmask 255.255.255.0 onboot yes role green
signal-event interface-update

Reset network configuration

In case of misconfiguration, it’s possible to reset network configuration by following these steps.

1. Delete all logical and physical interfaces from the db

   Display current configuration:

   db networks show
   

   Delete all interfaces:

   db networks delete eth0
   

   Repeat the operation for all interfaces including bridges, bonds and vlans.

2. Disable interfaces

   Physical interfaces:

   ifconfig eth0 down
   

   In case of a bridge:

   ifconfig br0 down
   brctl delbr br0
   

   In case of a bond (eth0 is enslaved to bond0):

   ifenslave -d bond0 eth0
   rmmod bonding
   

3. Remove configuration files

   Network configuration files are inside the /etc/sysconfig/network-scripts/ directory in the form: /etc/sysconfig/network-scripts/ifcfg-<devicename>. Where devicename is the name of the interface like eth0, br0, bond0.

   Delete the files:

   rm -f /etc/sysconfig/network-scripts/ifcfg-eth0
   

   Repeat the operation for all interfaces including bridges, bonds and vlans.

4. Restart the network

   After restarting the network you should see only the loopback interface:

   service network restart
   

   Use ifconfig command to check the network status.

5. Manually reconfigure the network

   Choose an IP to assign to an interface, for example 192.168.1.100:

   ifconfig eth0 192.168.1.100
   

   Then reconfigure the system:

   signal-event system-init
   

   The interface will have the chosen IP address.

6. Open the web interface and reconfigure accordingly to your needs

Zeroconf network

Zeroconf network (http://www.zeroconf.org/) shouldn’t be usefull on a server. It can be safely disabled using these commands:

config setprop sysconfig ZeroConf disabled
signal-event interface-update

Log retention and rotation

By default logs are rotated weekly and kept for 4 weeks. Some packages come with different defaults, but the majority do not specify a custom rotate value.

Logrotate db property:

• Rotate: rotation frequency, can be daily, weekly, monthly. Default is weekly
• Times: rotate log files Times number of times (days, weeks or months) before being removes, default is 4
• Compression: can be enabled or disabled. Defaults is disabled

Example:

logrotate=configuration
  Compression=disabled
  Rotate=weekly
  Times=4

Keep logs for 6 months, rotate once a week:

config setprop logrotate Rotate weekly
config setprop logrotate Times 24
signal-event nethserver-base-update

Transport Layer Security

The TLS policy page controls how individual services configure the Transport Layer Security (TLS) protocol, by selecting a policy identifier.

Each module implementation decides how to implement a specific policy identifier, providing a trade off between security and client compatibility. Newer policies are biased towards security, whilst older ones provide better compatibility with old clients.

You can enforce the TLS policy (20180330), or choose the legacy one (empty policy property) if your clients are not supported/maintained anymore (Windows XP for example).

TLS db property in configuration database: ::

tls=configuration

policy=

The event to expand the templates of all rpm which use TLS is tls-policy-save

Repositories

The default YUM repository set of NethServer is composed of

• nethserver-base: it contains packages and dependencies from core modules. It is updated when a new milestone is released. Enabled by default.
• nethserver-updates: it contains updated packages. If needed, these updates can be applied without requiring manual intervention. Enabled by default.
• nethforge: communty provided modules for NethServer. Enabled by default.
• nethserver-testing: contains packages under QA process. Disabled by default.
• ce-base: (ce- stands for CentOS) base packages from CentOS. Enabled by default.
• ce-updates: updated packages from CentOS. Enabled by default.
• ce-sclo-rh and ce-sclo-sclo: SCL repositories. Both enabled by default.
• ce-extras: extra RPMs. Enabled by default.
• epel: Extra Packages for Enterprise Linux. Enabled by default.

Packages published in above repositories should always allow a non-disruptive automatic update.

YUM repositories configuration

Two special .conf files control how NethServer configures and invokes YUM:

• /etc/nethserver/pkginfo.conf: list of YUM repositories that have their groups listed

  on the Software Center

• /etc/nethserver/eorepo.conf: list of YUM repositories enabled by software-repos-save event, every non-listed repository will be disabled

This is the list of .repo files providing the default repositories configuration:

• /etc/yum.repos.d/NethServer.repo

  • ce-base
  • ce-updates
  • ce-extras
  • ce-sclo-sclo
  • ce-sclo-rh
  • nethserver-base
  • nethserver-updates

• /etc/yum.repos.d/epel.repo

  • epel

• /etc/yum.repos.d/NethForge.repo

  • nethforge

ce-*, nethserver-* and nethforge repositories are accessed with a full release number (e.g. 7.6.1810), preventing unwanted upgrades to the next minor release version. See the software-repos-upgrade event for details.

The EPEL repository does not support accessing RPMs using a minor release like 7.5.1804 but only using a major release like 7.

Warning

When a subscription is enabled the default repositories are disabled. See /etc/yum.repos.d/subscription.repo

Third party repositories

It’s possible to install third party repositories, using standard CentOS methods.

If such repositories support access using minor release, they can be safely added to eorepo.conf and pkginfo.conf using a template-custom.

yum-cron

Since NethServer 7.5.1804, nethserver-yum-cron has been merged into nethserver-base. The cron job runs each night with a random time before to start of 6 hours.. You can decide who receive the notifications (default is root), which updates to do, if you just check, download, or install automatically the updates.

Original author: Stephane de Labrusse (@stephdl)

Database

Properties:

• applyUpdate: can be yes or no. If set to yes, downloaded updates will be installed
• customMail: comma-separated list of extra mail recipients, as default a mail will be sent to root
• download: can be yes or no. If set to yes, download new package updates
• messages: can be yes or no. Whether a message should be emitted when updates are available
• randomWait: random number of minutes to wait before executing the download procedure - NS6: 1 to 60 minutes - NS7: 1 to 360 minutes, negative and the job start immediately
• status: can be enabled or disabled. When enabled, a cron script will search for package updates

Database example:

yum-cron=service
   applyUpdate=yes
   customMail=
   download=no
   messages=no
   randomWait=360
   status=enabled

Notifications

Mail for root user can be forwarded to external addresses.

Properties:

• EmailAddress: comma-separated list of mail addresses; messages sent to root user will be forwarded to listed addresses
• KeepMessageCopy: can be yes or no; if set to yes, messages will be always delivered also to local root mail folder
• SenderAddress: a valid mail address; if not empty, messages sent by root (like cron notifications) will be sent using the specified address. A good value could be: no-reply@<domain> (where <domain> is the domain of the server). If not set, messages will be sent using root@<fqdn> as sender address.

Database example:

root=configuration
   EmailAddress=myuser@nethserver.org
   KeepMessageCopy=yes
   SenderAddress=no-reply@nethserver.org

Usage:

config setprop root EmailAddress myuser@nethserver.org SenderAddress no-reply@nethserver.org
signal-event notifications-save

Let’s Encrypt certificates

This package also requests and automatically renews Let’s Encrypt (LE) certificates. It adds httpd ACME-related configuration for all defined virtual hosts.

The main helper /usr/libexec/nethserver/letsencrypt-certs can be executed also from command line.

For more info, see:

/usr/libexec/nethserver/letsencrypt-certs -h

Database properties under pki key inside configuration database:

• LetsEncryptRenewDays: days to the expiration, the certificate will be renewd when the condition is met
• LetsEncryptMail: (optional) registration mail for LE notifications
• LetsEncryptDomains: comma-separated list of domains added to certificate SAN field
• LetsEncryptChallenge: challenge to use for validating the certificate, default is http. It accepts also values like dns-<provider>. Where <provider> is the name of the DNS provider. See the full list of available DNS provider plugins by executing certbot -h certonly. More info at https://certbot.eff.org/docs/using.html?highlight=dns#dns-plugins.
• LetsEncryptShortChain: can be enabled or disabled. If enabled, the certificate chain will not contain the expired DST Root CA X3

DNS challenge

To use the DNS challenge, follow these steps:

• install the required certbot plugin plugin using yum; to see the list of available package use yum search certbot-dns
• set LetsEncryptChallenge property to correct DNS plugin
• configure all required properties accordingly to plugin documentation

When using the dns challenge, make sure to set extra properties accordingly to certbot configuration. All properties for the dns challenge should be in the form LetsEncrypt_<certbot_option>, where <certbot_option> is the option specific to certbot DNS plugin.

Digitial Ocean example

1. Install the plugin:

   yum install python2-certbot-dns-digitalocean
   

2. Configure the challenge type:

   config setprop pki LetsEncryptChallenge dns-digitalocean
   

2. Configure required props accordingly to https://certbot-dns-digitalocean.readthedocs.io/en/stable/:

   config setprop pki LetsEncrypt_dns_digitalocean_token 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
   

3. Request certificate for domain myserver.nethserver.org:

   /usr/libexec/nethserver/letsencrypt-certs -v -d myserver.nethserver.org
   

nethserver-blacklist

nethserver-blacklist provides two kinds of blacklists: IP blacklist and DNS blacklist. The former blocks connections from/towards a set if IP addresses, while the latter blocks DNS requests to a set of domains.

IP blacklist

Configuration

IP blacklist is managed by properties of blacklist key inside configuration database:

• status: can be enabled or disabled, if enabled selected Categories will be blocked using ipsets
• Categories: a comma-separeted list of blacklist categories, a valid category must have a corresponding file inside /usr/share/nethserver-blacklist/ipsets. Example: given the category test, a file named /usr/share/nethserver-blacklist/ipsets/test.netset or /usr/share/nethserver-blacklist/ipsets/test.ipset must exist
• Url: the GIT URL from where blacklists will be downloaded
• Whitelist: a comma-separated list of hosts excluded from the blacklists. The host can be an IP, a CIDR, an host object or a CIDR object
• MaxElem : It does define the maximal number of elements which can be stored in the set, default 131072. You have to stop shorewall before to destroy the set, the shorewall service start will created it again with the new value.

config setprop blacklist MaxElem 262144
systemctl stop shorewall
for S in $(ipset -L -name | grep '^bl-'); do : ; ipset destroy  $S; done
systemctl start shorewall

Example:

blacklist=configuration
   Categories=category1
   Url=https://my.nethserver.org/git/myrepo
   Whitelist=
   status=disabled

Blacklists

Blacklist implementation is based on ipset:

• download ipset-based blacklist from a remote GIT repository
• block access from/to black listed IPs
• support whitelisting
• lists are updated every 20 minute, ipsets are reloaded on change
• all blocked IPs are logged inside /var/log/firewall.log

GIT repository

The remote GIT repository may contain one or more files with .netset or .ipset using Firehol iplists format, each file represent a category.

In addition to Firhol file format, the files can contain a Confidence tag inside the header. Example:

# ...
# Category        : attacks
# Version         : 18
# Confidence      : 8
# ...

The confidence is a number between 0 and 10, a higher number means less false positives. Lists with a confidence greater than 8, should be safe for production.

The repository can also contain a special file named whitelist.global which is a special netset that can be used to quickly remove false positive from a blacklist.

Setup a blacklist server

You can build your own server to distribute blacklists based on Firehol iplists. First install a clean NethServer, then follow the below steps.

Install required packages:

yum install https://github.com/firehol/packages/releases/download/2021-01-01-1948/firehol-3.1.7-11.el7.noarch.rpm https://github.com/firehol/packages/releases/download/2021-01-01-1948/iprange-1.0.4-2.el7.x86_64.rpm

yum install https://repo.ius.io/ius-release-el7.rpm

yum install -y git224-core --enablerepo=ius

Create the git repository and serve it using Apache:

cat << EOF >> /etc/httpd/conf.d/git.conf
SetEnv GIT_PROJECT_ROOT /var/www/html/git/
SetEnv GIT_HTTP_EXPORT_ALL
ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/

<Directory "/usr/libexec/git-core/">
    <Files "git-http-backend">
        Options +ExecCGI
        Require all granted
    </Files>
</Directory>
EOF

systemctl restart httpd

Configure and enable update-ipsets:

cat << EOF > /etc/firehol/update-ipsets.conf
BASE_DIR=/var/www/html/git/ipsets
HISTORY_DIR=/var/www/html/git/ipsets/history
ERRORS_DIR=/var/www/html/git/ipsets/errors
GIT_CMD=/usr/bin/git
EOF

mkdir -p /var/www/html/git/ipsets
git -C /var/www/html/git/ipsets init
/usr/sbin/update-ipsets enable dshield feodo fullbogons spamhaus_drop spamhaus_edrop sslbl zeus_badips ransomware_rw firehol_level1
/usr/sbin/update-ipsets run dshield feodo fullbogons spamhaus_drop spamhaus_edrop sslbl zeus_badips ransomware_rw firehol_level1

Set up a cron to regularly update the ipsets:

cat << EOF >> /etc/cron.d/update-ipsets
*/19 * * * * root /usr/sbin/update-ipsets
EOF

See also https://github.com/firehol/blocklist-ipsets/wiki/downloading-ip-lists

DNS blacklist

DNS blacklist uses Pi-Hole FTLDNS under the hood.

Configuration

DNS blacklist is managed by properties of ftl key inside configuration database:

• status: can be enabled or disabled, if enabled selected Categories will be blocked using FTLDNS
• Categories: a comma-separeted list of blacklist categories. Valid categories have a corresponding file inside /usr/share/nethserver-blacklist/dnss Example: given the category test, a file named /usr/share/nethserver-blacklist/dnss/test.dns must exist
• Url: the GIT URL from where blacklists will be downloaded
• Bypass: a comma-separated list of hosts whose DNS requests are always allowed. A host can be an IP, a CIDR, an host object or a CIDR object
• Roles: a comma-separated list of firewall zones where DNS blacklist is enabled
• UDPPorts, TCPPorts: the ports FTLDNS is listening to
• access: the zones ftl systemd service has access to

Example:

ftl=service
    Bypass=
    Categories=category1,category2
    Roles=green
    TCPPorts=1153
    UDPPorts=1153
    Url=https://my.nethserver.org/git/myrepo
    access=green
    status=disabled

Blacklists

Blacklist implementation is based on Pi-Hole gravity database:

• download DNS blacklists from a remote GIT repository and insert them into gravity database
• block DNS requests for listed domains
• support bypass
• lists are updated every 20 minute, ipsets are reloaded on change
• configuration process is logged inside /var/log/pihole-FTL.log
• blocked requests and other statistics are available through FTLDNS telnet API

GIT repository

The remote GIT repository may contain one or more files with .dns extension listing domains to block. Each file represents a category. Category files can contain Maintainer, Category and Confidence tag inside the header. Confidence is a number between 0 and 10, a higher number means less false positives. Lists with a confidence greater than 8 should be safe for production.

Example content:

#
# Maintainer      : John Doe
# Category        : Malware
# Confidence      : 6
#

unwanted.domain.com
dangerousdomain.net
malwaresite.net
...

GEOIP blacklist

Configuration

GEOIP blacklist is managed by properties of geoip key inside configuration database:

• status: can be enabled or disabled, if enabled selected Categories will be blocked using ipsets
• Categories: a comma-separeted list of geoip blacklist categories, a valid category must have a corresponding file inside /usr/share/nethserver-blacklist/geoips without the extensions netset. Example: given the category fr.netset, a file named ``/usr/share/nethserver-blacklist/geoips/fr.netset``must exist
• Url: the GIT URL from where blacklists will be downloaded (default ipdeny.com)
• Whitelist: a comma-separated list of hosts excluded from the blacklists. The host can be an IP, a CIDR, an host object or a CIDR object
• MaxElem : It does define the maximal number of elements which can be stored in the set, default 131072. You have to stop shorewall before to destroy the set, the shorewall service start will created it again with the new value.

config setprop geoip MaxElem 262144
systemctl stop shorewall
for S in $(ipset -L -name | grep '^geo-'); do : ; ipset destroy  $S; done
systemctl start shorewall

Example:

geoip=configuration
   Categories=fr,es
   Url=https://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz
   Whitelist=
   status=disabled

blacklist

Geoip blacklist implementation is based on ipset:

• download geoip blacklist from the website ipdeny.com
• block access from/to black listed IPs
• support whitelisting
• lists are updated each night, ipsets are reloaded on change
• all blocked IPs are logged inside /var/log/firewall.log

Example

The geoip list must be enabled manually:

config setprop geoip status enabled
signal-event nethserver-blacklist-save geoips

Once downloaded you can enable the geo banning by the command line:

config setprop geoip status enabled Categories es,fr Whitelist '195.234.41.0/24,195.234.42.3'
signal-event nethserver-blacklist-save geoips

The command above takes care to whitelist a CIDR network or a specific IP

nethserver-cgp

Expose collectd metrics using Collectd Graph Panel (CGP): https://github.com/pommi/CGP

Database example:

cgp=configuration
   alias=02bf3f8364beea0d5f23044bf14d31d93f63e98d

The alias is a random path generated once on first install and it will be used to compose CGP URL.

So in this case will be acessibile from https://<fqdn>:980/02bf3f8364beea0d5f23044bf14d31d93f63e98d.

Where <fqdn> is the fully qualified domain name of the server or its IP address.

nethserver-cockpit

This module implement new Server Manager UI using Cockpit project.

API and NethServer Cockpit developer guide: https://nethserver.github.io/nethserver-cockpit/

nethserver-collectd

This package automatically configure basic collectd plugins and it’s part of nethserver-statistics yum package group.

This package will install a base collectd configuration. Default enabled modules:

• cpu
• load
• processes
• memory
• swap
• uptime
• df
• disk
• interface
• network
• rrdtool
• ping

No configuration is needed, collectd is enabled by default when installed.

Ping plugin

The ping plugin sends an ICMP packet every 5 seconds to: * upstream DNS * every checkip of enabled provider (see Multi WAN)

Additional hosts must be added to the PingHosts property:

config setprop collectd PingHosts www.nethesis.it,www.nethserver.org
signal-event nethserver-collectd-update

Web interface

To view graphs of collected data, install and nethserver-cgp.

Cleanup

Every day a cronjob (/etc/cron.daily/collectd_cleanup) takes care to clean up all RRD files not updated during the last day.

Interesting plugins

The following can be manually installed:

• collectd-nut
• collectd-sensors

Database

Configuration is saved inside the configuration database. Example:

collectd=service
   PingHosts=
   status=enabled

nethserver-cups

CUPS is the printer server. It converts the page descriptions produced by your application (put a paragraph here, draw a line there, and so forth) into something your printer can understand and then sends the information to the printer for printing.

Configuration

Use the web configuration tool: http://<server>:631

You’ll need to create 2 printers:

• Raw printer for use with Samba and Windows hosts
• A PPD defined printer to user via IPP and Linux hosts

You already have a PDF printer configured and ready to use. All files printed via PDF Printer are saved in the user home directory, inside the directory called “Desktop”. You can download the files using samba.

Printer discovery

CUPS uses avahi to discover network-attached printer. If you wish to use this features, just install the optional package nethserver-avahi. See avahi-section.

Configuration DB

Reference:

cups=service
   TCPPort=631
   UDPPort=631
   access=private
   status=enabled

Links

• Offical CUPS web site: http://www.cups.org/
• Offical AVAHI web site: http://avahi.org/

nethserver-dc

The nethserver-dc package runs a systemd-nspawn container (nsdc) with a vanilla Samba 4 inside of it. It downloads, installs, configures and provision an Active Directory domain controller based on Samba.

The nsdc container needs an IP address in a green network, different from the host machine one. It enslaves its network interface to a host bridge, with green role. If needed, this bridge is created automatically.

This is a typical configuration:

# config show nsdc
nsdc=service
   ProvisionType=newdomain
   IpAddress=192.168.122.50
   bridge=br0
   status=enabled

nethserver-dc-save event

• it creates and configures systemd-nspawn machine (nethserver-dc-install action). The Samba domain is provisioned by the samba-provision.service unit, according to the ProvisionType prop value. Supported values are:
  • newdomain (default): domain and realm are taken from local system and won’t be possible to change them anymore. For instance if system domain is nethserver.org domain will be NETHSERVER and realm nethserver.org.
  • ns6upgrade: connect the LDAP service running on the host machine and migrate the WS/PDC domain from ns6 backup to an Active Directory domain. The realm and domain name are set as described in the newdomain provision type.
  • sme8migration: this provision procedure is driven by nethserver-dc-migrate action. It requires a running NSDC, provisioned as newdomain. It copies Samba 3 files from the migration source directory to the NSDC chroot, then restarts the samba-provision and samba services in the running NSDC container.
• it creates a network bridge if needed, or select an existing one and save it in nsdc bridge db prop (nethserver-dc-create-bridge action)
• it waits for the machine to come up (nethserver-dc-waitstart)
• it joins the domain of new machine using default credentials (nethserver-dc-join).
• it sets the password policy (nethserver-dc-password-policy)

Realmd writes a lot of information on the system journal. See journalctl command.

Running commands in nsdc

To get a shell inside the nsdc container, you can run

# systemd-run -M nsdc -t /bin/bash

As alternative, run a specific command in the container with the nsdc-run helper:

# nsdc-run cat /etc/hostname

The -- string helps to separate the argument list:

# nsdc-run -- ls --help

For more information:

# nsdc-run --help

Always make sure to close interactive sessions to avoid errors like:

Failed to get machine PTY: No such file or directory

To get a list of sessions run

loginctl

Manual Join

nethserver-dc-join action joins automatically to domain. If for any reason the join is invalid you can attempt a manual join following this procedure

Check nsdc is running:

systemctl status nsdc

Check the DNS is responding:

# host -t SRV _ldap._tcp.$(config getprop sssd Realm)
_ldap._tcp.nethsever.org has SRV record 0 100 389 nsdc-vm8.nethsever.org.

Clean up any previous join state:

config setprop sssd Provider none
signal-event nethserver-sssd-leave

Join the domain:

realm join -v -U admin $(config getprop sssd Realm)

You can replace admin with any other administrative account name. The command above prompts for a password. When join is successful:

config setprop sssd Provider ad
signal-event nethserver-sssd-save

If everything goes well

getent passwd administrator@$(hostname -d)
# output: administrator@nethserver.org:*:261600500:261600513:Administrator:/home/administrator@nethserver.org:/bin/bash

/usr/libexec/nethserver/list-users -s administrator
# output: {"administrator": ...

Once domain is joined, you can manage users from interface. From command line, you can use net command

# net ads info

Factory reset

The “Start DC” procedure from the “Accounts provider” page is designed for a single run. If it fails, re-installing the whole server can be avoided by running the following command

signal-event nethserver-dc-factory-reset

The command cleans up the DC state and prepare it for new provisioning run. Any existing user and group account is erased.

If a full DC re-install is desired, after factory reset event, run also

rm -rf /var/lib/machines/nsdc

Uninstall nethserver-dc

Execute:

signal-event nethserver-sssd-remove-provider

Upgrade the container

The upgrade procedure will:

• stop the container
• upgrade the chroot base system
• upgrade samba
• restart the container

To upgrade, execute:

signal-event nethserver-dc-upgrade

Changing the IP address of DC

Warning

Before applying this procedure, read carefully the official Samba wiki page.

The IP address of nsdc container must be in the same network of the bridged green interface. If needed, first change the address of the green interface, then proceed with the following.

Example, change the network address:

• current host IP: 192.168.101.7
• current nsdc container IP: 192.168.122.77
• new nsdc container IP: 192.168.101.77

Execute the nethserver-dc-change-ip with the new ip address:

signal-event nethserver-dc-change-ip <new_ip_address>

Example:

signal-event nethserver-dc-change-ip 192.168.101.77

Note that the event will fail if the new nsdc ip address is not in the same network of the green interface.

Alternate UPN suffix

The default UPN (User Principal Name) suffix for a user account is the SSSD realm, but the nsdc container is configured to use also an extra UPN suffix set to the FQDN of the host machine.

Example:

• Host FQDN: nethserver.org
• SSSD realm: ad.nethserver.org
• Default UPN: ad.nethserver.org
• Extra UPN: nethserver.org

If required, the administrator can use RSAT tools to select the extra UPN for a specific user.

References:

• https://technet.microsoft.com/en-us/library/cc772007%28v=ws.11%29.aspx
• https://msdn.microsoft.com/en-us/library/ms680537%28v=vs.85%29.aspx

Enable check password script

This manual fix is only needed for old installations.

Edit /var/lib/machines/nsdc/etc/samba/smb.conf and add

check password script = /usr/local/sbin/checkpassword.pl

to the global section. Restart samba on the NSDC container:

systemctl -M nsdc restart samba

Apply the current system password policy to the Samba DC:

signal-event password-policy-update

ns-samba binary build

0. Clone the git repo at https://github.com/NethServer/ns-samba, enter its directory

1. Download pristine sources from the Samba project

   wget https://download.samba.org/pub/samba/stable/samba-4.8.6.tar.gz https://download.samba.org/pub/samba/stable/samba-4.8.6.tar.asc
   

2. Check the sources signature

   gunzip -c samba-4.8.6.tar.gz | gpg --verify samba-4.8.6.tar.asc -
   

3. Bump the source hash sum

   sha1sum samba-4.8.6.tar.gz > SHA1SUM
   

4. Edit ns-samba.spec to fix version references and %changelog section

6. Commit changes and add (signed) tag (i.e. 4.8.6)
7. Push commit and tag to remote git repository to start the automated x86_64 build
8. Once the automated build has finished successfully, fill the new relase page here: https://github.com/NethServer/ns-samba/releases
9. Copy the RPM URL to the local nethserver-dc.spec file
10. Update SHA1SUM with the new RPM hash sum

nethserver-directory

The nethserver-directory implements user authentication and authorization. All accounts are saved inside OpenLDAP.

Features:

• RFC2307 schema, user and group account management
• PAM LDAP password storage
• Password strength control
• Service accounts

Schema and base DN

Following schema are always loaded by OpenLDAP: core, cosine, nis, inetorgperson.

The LDAP tree is always accessible with the following base DN: dc=directory,dc=nh. But there is also an overlay which maps the domain name to the base DN. For example, given the domain mydomain.com, the corresponding DN will be dc=mydomain,dc=com.

Accounts are saved inside following branches:

• Users: ou=People,dc=directory,dc=nh
• Groups: ou=Groups,dc=directory,dc=nh

All users are in the primary group named locals.

Examples

List all entries, with root access and automatic bind (unix domain socket):

ldapsearch -Y EXTERNAL

List all entries with libuser bind:

ldapsearch  -D cn=libuser,dc=directory,dc=nh -w `cat /var/lib/nethserver/secrets/libuser` -h 127.0.0.1

User account states

Account management is done via libuser which exports some tools (luseradd, luserdel, etc) to create/modify/delete users, groups and set passwords.

The process of user/group creations is:

• invoke the create event which uses libuser to add the record inside LDAP
• invoke the group event if the user must be member of additional groups
• invoke the password policy event to change user password expiration
• invoke the password change event to set a password

Logging

OpenLDAP doesn’t output any log with standard configuration. When logging is enabled, all logs are saved inside /var/log/slapd. But its verbosity can be changed at run time by issuing this command:

# ldapmodify -Y EXTERNAL <<EOF
dn: cn=config
changetype: modify
replace: olcLogLevel
olcLogLevel: 256
EOF

The command above changes the OpenLDAP config DB and sets the log verbosity to trace “connections/operations/results” (256). Check the debugging levels table from OpenLDAP site for more details: http://www.openldap.org/doc/admin24/slapdconf2.html#olcLogLevel%3A%20%3Clevel%3E.

Service accounts

The following service accounts are configured by the nethserver-directory-dit-setup action:

• cn=libuser,dc=directory,dc=nh. Granted read and write access from 127.0.0.1.
• cn=ldapservice,dc=directory,dc=nh. Granted read only access to non-sensitive attributes, from localhost, or from any other IP address with TLS.

The developer can use the NethServe::Directory Perl module to handle additional service accounts with ad-hoc permissions, if the existing ldapservice, libuser accounts and anonymous binds do not fit his requirements.

A service account is composed by three parts:

• a LDAP user
• a password
• an ACL to access LDAP fields

Perl code snippet to create a service account with read access:

use NethServer::Directory;
...
NethServer::Directory->new()->configServiceAccount('myservice', NethServer::Directory::FIELDS_READ) || die("Failed to register myservice account")

Perl code snippet to use created password:

use NethServer::Password;
my $pwd = NethServer::Password::store('myservice');

User accounts

Authenticated binds are granted to TLS protected connections, or connections from 127.0.0.1. User DN are in the form:

uid=<username>,ou=People,dc=directory,dc=nh

Anonymous access

Some LDAP clients and/or legacy environments may require anonymous bind to the LDAP accounts database. Currently anonymous access is granted to non-sensitive fields.

Configuration for client:

• Host: ip address of the server
• Port: 389
• Base DN: dc=directory,dc=nh

Administrative access

An existing DN (i.e. administrator) can be granted full administrative privileges on the whole dc=directory,dc=nh tree. By default, the designated user is defined in config DB, under the admins key.

` ldapmodify -Y EXTERNAL <<EOF dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootDN olcRootDN: uid=administrator,ou=People,dc=directory,dc=nh EOF `

Inspect OpenLDAP ACLs

Service accounts require OpenLDAP ACLs tuning. To inspect the current ACLs type:

ldapsearch -LLL -Y EXTERNAL -b cn=config -s one 'objectClass=olcDatabaseConfig' olcAccess 2>/dev/null

If output appears to be base64-encoded type:

ldapsearch -LLL -Y EXTERNAL -b cn=config -s one 'objectClass=olcDatabaseConfig' olcAccess 2>/dev/null | perl -MMIME::Base64 -MEncode=decode -n -00 -e 's/\n +//g;s/(?<=:: )(\S+)/decode("UTF-8",decode_base64($1))/eg;print'

Upgrade to Active Directory

If the LDAP database has been restored from a ns6 backup set, it is possible to upgrade it to a local Active Directory accounts provider.

A ns7 LDAP database cannot be upgraded to Active Directory. It lacks the Samba LDAP schema extensions required by the Samba classic upgrade procedure.

The nethserver-directory-ns6upgrade event

• removes the nethserver-directory RPM
• installs and configures nethserver-dc

Before running the event, assign a free IP address to the nsdc Linux container, installed by nethserver-dc RPM. Ensure it is a free IP address of a green network.

config set nsdc service IpAddress A.B.C.D
signal-event nethserver-directory-ns6upgrade

nethserver-dnsmasq

DNS

The system will resolve host and domain names using DNS queries to external DNS servers. The configuration is saved inside the dns key from nethserver-base package.

Properties:

• NameServers: comma separated IP list of external DNS
• role: can be set to none or resolver. If role is set to none the server will always use external DNS. For resolver role see DNS server.

Example:

dns=configuration
   NameServers=8.8.8.8,208.67.222.222
   role=none

Hosts

The system can handle local DNS records. When the server performs a DNS lookup, first it will search inside local DNS records. If no local record is found, an external DNS query will be done.

Note

Local DNS record will always override records from external DNS servers.

DNS records are called hosts and are saved inside the hosts database. Each entry is saved inside the /etc/hosts file.

There are three types of records:

• local: hosts inside the internal network
• remote: hosts outside the internal network
• self: alias for the server itself

Records of type local and remote can have following properties:

• IpAddress: address of the host
• Description: optional description
• MacAddress: mac address of the host. Used only for DHCP reservation. See IP reservation.

For hosts inside local network, the record key doesn’t have the domain part. Example:

host1=local
    Description=Internal network host #1
    IpAddress=192.168.1.23

For hosts outside local network, the record key must have the domain part. Example:

external.otherdomain.tld=remote
    Description=Other domain host
    IpAddress=8.9.10.11

Records of type self can have following properties:

• Description: optional description

Example:

vhost1.domain.tld=self
    Description=Virtual Host #1

DNS server

The system uses dnsmasq as DNS and DHCP server and it directly resolves all hosts inside its domain. All other names will be queried to external DNS servers.

The server will forward reverse lookups to upstream DNS servers, only if upstream DNS servers are inside a private network (eg. network address is 192.168.x.x).

The option bind-interfaces is always enabled, as consequence (from dnsmasq man):

This option has been patched to always use SO_BINDTODEVICE socket option when binding to interfaces. As consequence, dnsmasq WILL NOT ANSWER to any DNS Queries that come to the socket with the correct destination IP address, but originally on different interface. This behavior differs from the original dnsmasq upstream version and is used for security reasons.

Properties:

• CacheSize: entry to be cached by server, default is 4000
• dhcp-boot: directly pass parameters to dhcp-boot option
• except-interface: comma-separated list of interfaces. Do not listen to listed interfaces, useful to avoid conflicts with libvirt
• tftp-status: can be enabled or disabled. If enabled, enable the TFTP server for BOOTP (port 67)
• access: default is private, do NOT set to public
• DomainRedirection: specify a dns server for a particular domain (comma separated). The domain.org:192.168.1.1 will send all queries *.domain.org for internal machines to 192.168.1.1. The special server address # means, “use the standard servers”, so sub.domain.org:# will send all queries for *sub.domain.org to the default DNS server of the domain name.

Database example:

dnsmasq=service
  AllowHosts=
  CacheSize=4000
  DenyHosts=
  DomainRedirection=domain.org:192.168.1.1,sub.domain.org:#
  TCPPort=53
  UDPPorts=53,67
  access=private
  dhcp-boot=pxelinux.0,myserver.mydomain.com,192.168.1.1
  except-interface=virbr0,tunspot
  status=enabled
  tftp-status

DHCP

The system can act as DHCP server for the local network. Machines which are configured by DHCP have their names automatically included in the DNS server.

The DHCP can be enabled only on green and blue interfaces (see Roles and zones). Configuration is saved inside the dhcp database.

Each record of range type is associated to an ethernet interface and can have following properties:

• status: can be enabled or disabled
• DhcpRangeStart: first IP address of DHCP range
• DhcpRangeEnd: last IP address of DHCP range
• DhcpLeaseTime: seconds of lease time. Default is 86400
• DhcpGatewayIp: (optional) set a custom gateway ip. If not set, the gateway is the ip address of associated interface (record key)

The key of the record is the name of the associated interface. Example:

eth0=range
  DhcpGatewayIp=
  DhcpLeaseTime=86400
  DhcpRangeEnd=192.168.1.100
  DhcpRangeStart=192.168.5.200
  status=enabled

Hosts inside the blue network can always access the local DNS server.

The gateway for clients will be:

• if set, the value of property DhcpGatewayIp
• otherwise if the server has a red interface, the gateway is the IP address of the interface where the DHCP is enabled (eg. IP of the blue interface for clients in the guest’s network)
• otherwise if the server has only a green interface, the gateway of the green interface will be used

IP reservation

It’s possible to reserve IPs for specific devices associating the MAC address of the device with the reserved IP. The reservation is saved inside the hosts database.

Example:

host1=local
    Description=Internal network host #1
    IpAddress=192.168.1.23
    MacAddress=08:00:27:48:BF:F3

TFTP server

TFTP module contains configuration fragments that enables dnsmasq built-in TFTP server.

TFTP server has no authentication or encryption support.

When installed tftp is disabled by default and need to be enabled with:

config setprop dnsmasq tftp-status enabled
signal-event nethserver-dnsmasq-save

The package also add directory /var/lib/tftpboot that is the root of tftp server.

Enabling TFTP adds 5 new configuration options to /etc/dnsmasq.conf. Here variables explanation according with dnsmasq documentation

• enable-tftp: enable tftp server
• tftp-secure: allow only files owned by the user dnsmasq is running as will be send over the net
• dhcp-boot= ...: Set the boot filename for netboot/PXE. You will only need this is you want to boot machines over the network and you will need a TFTP server; driven by db prop
• tftp-root=/var/lib/tftpboot: Set the root directory for files available via FTP.
• dhcp-option=66, LOCAL_IP: set local ip as default tftp server for machines that receive dhcp from this server

Properties

• status: can be enabled or disabled. If enabled, TFTP server is configured and port 69 UDP is opened.
• UDPPort: UDP port used. Only 69 is allowed.
• access: define if access is public, private or none.
• dhcp-boot: Set the boot filename for PXE. Ths is needed for booting machines over the network. Empty by default.
• type: only service is allowed.

Test TFTP

Testing is very simple:

Enable TFTP server:

config setprop dnsmasq tftp-status enabled
signal-event nethserver-dnsmasq-save

Create a file to share, owned by nobody user:

echo "test"  > /var/lib/tftpboot/foobar
chown nobody:nobody /var/lib/tftpboot/foobar

From another machine, install tftp and get file (on Fedora):

yum install tftp

Always from the other machine, allow incoming UDP connection from our TFTP server. Loading TFTP conntrack module should be enough:

modprobe nf_conntrack_tftp

Connect to TFTP server:

tftp TFTP_SERVER_HOST

…and get the file:

tftp> get foobar

Configure a PXE server

Those instructions set up a PXE server for CentOS Install and configure syslinux and nethserver-tftp:

yum install syslinux
cp /usr/share/syslinux/{pxelinux.0,menu.c32,memdisk,mboot.c32,chain.c32} /var/lib/tftpboot/
config setprop tftp dhcp-boot pxelinux.0
signal-event nethserver-tftp-save
mkdir /var/lib/tftpboot/pxelinux.cfg

Create the file /var/lib/tftpboot/pxelinux.cfg/default with the following content:

default menu.c32
prompt 0
timeout 300

MENU TITLE PXE Menu

LABEL CentOS
kernel CentOS/vmlinuz
append initrd=CentOS/initrd.img

Create a CentOS directory:

Create a CentOS directory:

mkdir -p /var/lib/tftpboot/CentOS

Copy inside the directory vmlinuz and initrd.img files. These files can be found inside the ISO or browsing the yum os mirror.

Change files owner to nobody:

chown -R nobody /var/lib/tftpboot/*

nethserver-duc

Official site: https://github.com/zevv/duc

This tool is used to visualize disk usage in a simply and nice graph.

Features:

• Indexing filesystem very fast
• Interact with graph with click and double click to navigate in the directories tree.

Overview

This tool is composed by two parts: duc indexer and duc viewer. The first one is launched every day and it indexes the / directory and it creates a file that is loaded by duc viewer to show graph.

In the graph are showed only directories over certain size, in particularly this size is 10MB as you can see in file /etc/e-smith/events/actions/nethserver-duc-index.

Configuration

By default, duc will skip directories on different file systems. To change this behavior and include all filesystems execute:

config setprop duc OneFileSystem disabled

nethserver-ejabberd

The chat function is implemented using ejabberd XMPP server. Enabled features are:

• LDAP based roster
• SSL/TLS
• Admin group

If you want to give admin permissions to an existing user, just add the user to the special group jabberadmins. The jabberadmins must be created manually.

If the system has a remote account provider also reconfigure the ejabberd service:

sss_cache -g jabberadmins
signal-event nethserver-ejabberd-save

When used with AD backend, following limitations apply:

• The shared roster doesn’t support groups
• The shared roster displays the list of user names (not full names)

See also the Server Manager UI API documentation: https://github.com/NethServer/nethserver-ejabberd/blob/master/UI-API.md

Configuration

Properties:

• WebAdmin: Enable ejabberd built-in web interface. Can be enabled or disabled, default is disabled
• S2S: Enable the server-to-server (S2S) for XMPP federation (Port number: 5269). Can be enabled, default is disabled
• ModMamStatus: The XEP-0313: Message Archive Management (mod_mam). Can be enabled, default is disabled
• ModMamPurgeDBStatus: Purge the Mnesia database of old messages of mod_mam
• ModMamPurgeDBInterval: Remove messages older than X days, default is 30
• ShaperFast: Download speed limit in bytes/second for admin users, default is 1000000
• ShaperNormal: Download speed limit in bytes/second for users, default is 500000
• LdapTlsPortOverride: The user roster LDAP client does not support STARTTLS. If the system account provider is configured with STARTTLS, configure the user roster LDAP client to use this TCP port number during connections instead (default is port 636).

When enabled, web-based administration interface listens on 5280 port. You need a user inside jabberadmins group to login.

Default access to server ports is set to public on following ports: 5280, 5222, 5223.

The XMPP server can be accessed using BOSH protocol (https://xmpp.org/extensions/xep-0206.html) at URL /http-bind.

Example:

• server FQDN: mail.nethserver.org
• BOSH URL: https://mail.nethserver.org/http-bind

nethserver-evebox

EveBox configuration for NethServer:

• it requires nethserver-suricata
• it runs as suricata user, only if /var/log/suricata/eve.json file exists
• Suricata must be running and eve log must be enabled
• it uses SQLite
• no ElasticSearch support
• listens on port 5636, accessible only from localhost
• web interface served via reverse proxy: https://<host>:980/<alias> (see alias prop below)
• once a week internal GeoIP database is updated by a cron job

Properties:

• Retention: positive integer, if greater than 0, events will be discarded after the given number of days

Database:

evebox=service
   TCPPort=5636
   access=
   alias=be771675da40887ccf9eb98d05b88dc80559b6a6
   Retention=30
   status=enabled

Links

• Official site: https://evebox.org/

nethserver-firewall-base

NethServer can work into two basic modes:

• server mode: the system will be a standard host inside the network offering services like e-mail or file server.
• gateway mode: the system is the gateway and firewall of the local network

The system has an abstraction layer for firewall base functions, like opening ports for running services. Actually two implementations are available:

• server mode: standard lokkit (default on CentOS)
• gateway mode: advanced Shorewall configuration

The gateway functionality is built around three modules:

• nethserver-base: high-level abstraction of the firewall
• nethserver-firewall-base: Shorewall-based implementation
• nethserver-lsm: link status monitor for multi-wan configurations

Roles and zones

Each network interface has a role which maps to a firewall zone. The firewall has the following built-in zones, ordered from the most to the least privileged:

• green: local network, it’s considered almost trusted. Hosts in this network can access any other zone. Hosts connected via VPN can be considered in green zone.
• blue: guest network. Hosts in this network can access orange and red zones, can’t access green zone
• orange: DMZ network. Hosts in this network can access red zone, can’t access green and blue zones
• red: external/internet networks. Hosts in this network can access only firewall zone

There is also a special firewall zone which represents the firewall itself. The firewall can access any other zone.

Each network interface with a configured role is a firewall zone. Roles are mapped to Shorewall zones as:

• green -> loc
• red -> net
• blue -> blue
• orange -> orang (in Shorewall, a zone name can’t be longer than 5 chars)
• firewall -> FW

Custom zone names are directly mapped to Shorewall respecting the limit of 5 chars.

Red interfaces can be configured with static IP address or using DHCP. All other interfaces can be configured only with static IP addresses.

General configuration

Properties of firewall key inside configuration db:

• event: event to call when firewall-adjust event is fired
• ExternalPing: if enabled, allow ping responses on external interface
• WanMode: multi-wan mode. Default is balance, can be:
  • balance: traffic is balanced among red interfaces in weighted mode
  • backup: traffic is routed via wan interface with maximum weight, all other interfaces are used as fallback
• nfq: if enabled, traffic from external networks will be passed to NFQ and scanned with Snort. See ips.
• Policy: can be permissive or strict. See above.
• MACValidation: if enabled, the firewall will check the traffic against a list of known MAC addresses (see: IP/MAC binding)
• MACValidationPolicy: can be accept or drop. Default is drop. See man shorewall.conf for all valid values
• InterfaceRoleList: list of network interface roles configurable from web interface. Default is: green,red,blue,orange
• CheckIP: comma-separeted list of IP monitored by LSM, to check if a provider is up or down
• MaxNumberPacketLoss: number of maximum consecutive packets lost, over this threshold the provider is considered down
• MaxPercentPacketLoss: percentage of maximum packet lost, over this threshold the provider is considered down
• PingInterval: seconds between ICMP packets sent by LSM, default is 5
• NotifyWan: can be enabled or disabled, if enabled a mail is sent every time a provider changes its own state
• NotifyWanFrom: sender address for mails sent if NotifyWAN is set to enabled
• NotifyWanTo: recipient address for mails sent if NotifyWAN is set to enabled
• VpnPolicy: can be permissive or strict. If set to permissive traffic between all VPNs (Rodwarrior OpenVPN, OpenVPN tunnels, IPSec tunnels) will be permitted
• SipAlg can be enabled (default) or disabled. It enables/disables the application level gateway (ALG) for SIP|H323 protocol (the netfilter conntrack kernel modules for these protocols)

Example

firewall=configuration
   CheckIP=8.8.8.8,208.67.222.222
   Docker=disabled
   ExternalPing=enabled
   HairpinNat=disabled
   MACValidation=disabled
   MACValidationPolicy=drop
   MaxNumberPacketLoss=10
   MaxPercentPacketLoss=50
   NotifyWan=disabled
   NotifyWanFrom=root@localhost
   NotifyWanTo=root@localhost
   PingInterval=5
   Policy=permissive
   SipAlg=enabled
   TCLinklayer=
   VpnPolicy=strict
   WanMode=balance

Events

The main event for firewall configuration is firewall-adjust. The event contains a single action which fires the event named in the property event inside the firewall key into the configuration database.

Other events:

• lokkit-save: base firewall implementation using lokkit
• nethserver-firewall-base-save: firewall implementation using Shorewall
• wan-uplink-update: fired when the status of an external interface changes
• conntrack-adjust: fired when we want to reset the conntrack and remove kernel modules used by shorewall

The wan-uplink-event event takes at least two parameters:

• provider name: name of the provider involved
• action: can be up or down, describing the new provider status

Example:

signal-event wan-uplink-update down myisp

Policy

For every network packet traveling between firewall zones, the system will evaluate a list of rules to allow/block the specific traffic. Policies are default firewall rules which will be applied only if no other rule matches the ongoing traffic.

Firewall implements two standard policies:

• Permissive: will enable all traffic from green (loc) zone to red (net) zone.
• Strict: will block all traffic from green (loc) zone to red (net) zone. Permitted traffic should be explicitly allowed.

The firewall configures 4 default zones with built-in policies (see above). In the schema below, traffic is permitted from left to right and blocked from right to left:

GREEN -> BLUE -> ORANGE -> RED

To override a policy, you should create a firewall rule between zones.

IP/MAC binding

When MACValidation option is enabled, the firewall analyzes all the traffic based on a well-known list of IPs associated to MAC addresses. If the host generating the traffic is not inside the list, MACValidationPolicy will be applied. The list of IP/MAC association is created from DHCP reservations.

Thus, enabling MACValidation and leaving MACValidationPolicy set to drop, will block all traffic from hosts without a DHCP reservation.

Rules

Firewall rules can allow or deny traffic matching certain conditions. Rules are saved inside the fwrules database as records of type rule.

Each rule record has the following fields:

• key: a unique key identifier
• Position: integer sorting key
• Src, Dst: {literal*|*reference} where
  • literal is an IP, a CIDR, any (any source/destination) or fw (the firewall itself)
  • reference has the form prefix;value, where prefix can be a DB type (host, host-group, zone, iprange, cidr, mac) or the string role, value is a DB key or an interface role name (green, red…). mac objects are not supported inside the Dst field.
• Action: can be ACCEPT, DROP or REJECT
  • ACCEPT allows the traffic
  • REJECT denies the traffic, an ICMP port unreachable packet is sent to the source address
  • DROP discards the traffic without informing the source address (the connection will timeout)
• Service: (optional) can be a service object, a port number or a ndpi object. If a port number is used, both TCP and UDP protocols are matched.
• Time: (optional) can be a time object, the rule will be enabled only if the time conditions is matched
• Log: can be none or info. If value is info, all matched packets will be logged in /var/log/firewall.log. Defaults to none
• status: can be enabled or disabled. Default is enabled
• State: (optional) select on which type of connection the rule will be applied to:
  • new or empty: default, the rule will be applied only to new connections
  • all: the rule will be applied to new and established/related connections
• Description: (optional)

Example of a rule accepting traffic:

1=rule
    Src=host;myhost
    Dst=192.168.1.2
    Service=service;ssh
    Action=accept
    Position=32

Accept all traffic from myhost to myserver for ssh service (port 22):

db fwrules set 1 rule Src "host;myhost" Dst "host;myserver" Service ssh Action ACCEPT Log none status enabled Position 8765

Drop all traffic from 192.168.1.0/24 to 192.168.4.1 on TCP and UDP port 25:

db fwrules set 2 rule Src  192.168.1.0/24 Dst 192.168.4.1 Service 22 Action DROP Log none status enabled Position 5469

Template Fragment

Rules in the firewall can be added manually by a template fragment in the folder /etc/e-smith/templates/etc/shorewall/rules

For example drop a file 40YourSpecificRule

## 40nethvoice

{
    my $iax = $nethvoice{'AllowExternalIAX'} || 'disabled';

    my $webrtc = $nethvoice{'AllowExternalWebRTC'} || 'disabled';

    if ($iax eq 'enabled') {

        $OUT .= "# Enable IAX from red interfaces\n";

        $OUT .= "?COMMENT Enable IAX from red interfaces\n";

        $OUT .= "ACCEPT\tnet\t\$FW\tudp\t4569,5036\n";
    }

    if ($webrtc eq 'enabled') {

        $OUT .= "# Enable WebRTC from red interfaces\n";

        $OUT .= "?COMMENT Enable WebRTC from red interfaces\n";

        $OUT .= "ACCEPT\tnet\t\$FW\ttcp\t8089\n";
    }

    $OUT .= "?COMMENT\n";
}

You can use all the settings below but you might be interested by the shorewall documentation also at http://shorewall.net/manpages/shorewall-rules.html)

• \t -> write a tab space (can be also written : $OUT .= "ACCEPT  net  $FW  tcp  8089\n";)
• ACCEPT -> allows the traffic
• REJECT -> denies the traffic, an ICMP port unreachable packet is sent to the source address
• DROP -> discards the traffic without informing the source address (the connection will timeout)
• REDIRECT -> redirect the traffic to another firewall zone

The target may optionally be followed by “:” and a syslog log level (e.g, REJECT:info or Web(ACCEPT):debug).

• loc -> green (Local network)
• net -> red (Internet network)
• blue -> blue (Guest network)
• orang -> orange (DMZ network)
• $FW -> firewall
• tcp -> tcp port (comma separated list of ports)
• udp -> udp port (comma separated list of ports)

then you must expand your templates and restart your firewall by : signal-event firewall-adjust

Firewall objects

Firewall module uses objects to simplify rules creation. The use of objects is not mandatory but it’s strongly encouraged.

Supported objects are:

• Host
• Group of host
• Service
• CIDR
• Ip range
• Zone
• Time
• MAC address

A host is an already defined entry inside the hosts db, or a new key of type host:

name=host
    IpAddress=IP
    Description=

A host-group is a group of hosts inside the hosts db. Hosts in a host-group should always be reachable using the same interface. For example: a group of host inside the LAN or on the Internet. A host-group db entry can be something like:

name=host-group
    Members=host1,host2

A CIDR is a group of hosts defined as a CIDR network. It’s saved inside the hosts db:

mycidr=cidr
    Address=192.168.100.0/24
    Description=

A IP range is a group of hosts defined as a range of IP. It’s saved inside the hosts db:

myrange=iprange
    Description=
    End=192.168.100.20
    Start=192.168.100.10

A zone represents a network zone which can be associated to an interface or a set of IP address. A zone entry in networks database can be something like:

name=zone
   Network=CIDR
   Interface=eth0

A configured network interface is automatically a zone.

A service can have a protocol and one or more ports. A service entry in fwservices database can be something like:

name=fwservice
   Protocol=tcp/udp/tcpudp/icmp
   Ports=port/port range

A service can also be a refence in the format ndpi;<protocol> where protocol is a supported protocol from nDPI kernel module. To see a list of supported protocols:

db NethServer::Database::Ndpi keys

A time condition is a time record entry in fwtimes database. All times are saved in local time and converted to UTC on template expansion.

Database example:

db fwtimes setprop officehours WeekDays 'Mon,Tue,Wed,Thu' TimeStart '09:00' TimeStop '18:00'

A MAC address is a mac record entry inside macs database. The MAC must always have a Zone property which specifies the network segment where the device is connected. It’s something like:

mac1=mac
   Address=52:54:00:05:2d:c3
   Description=My mac test
   Zone=green

Rules based on mac address

It’s possible to create rules based on MAC address only using a template-custom. For example to block internet access to a host on local network using its MAC address:

mkdir -p /etc/e-smith/templates-custom/etc/shorewall/rules
echo "DROP      loc:~xx-xx-xx-xx-xx-xx          net" > /etc/e-smith/templates-custom/etc/shorewall/rules/90mymac

Where xx-xx-xx-xx-xx-xx is the MAC address to block.

See man shorewall-rules for more information.

Port forwarding

All port-forwards are saved inside the portforward db.

Each record has:

• key: auto-increment id
• type: pf
• protocol: tcp/udp
• src: can be a port number or a range in the form xxxx:yyyy
• dst: can be a port number, if empty the value of src is used
• dstHost: destination host, can be an IP address or a hos firewall object
• allow: allowed ip address or network, see SOURCE at http://www.shorewall.net/4.2/manpages/shorewall-rules.html
• status: enabled/disabled
• oriDst: original destination ip, for example alias for a wan interface. If empty, the port forward is valid for all red interface
• description: optional description

Source NAT (sNAT)

All NAT one-to-one configurations are stored in networks db.

Each value is a new attribute for an existing alias key and the name of attribute is FwObjectNat that contains the reference of an associated host:

eth1:0=alias
    FwObjectNat=host;host_name
    ipaddr=11.11.11.11
    netmask=255.255.255.0
    role=alias

During template-expanding phase, the associated host is mapping with referenced IP and added in shorewall nat configuration. The file is /etc/shorewall/nat.

More information are available here: http://shorewall.net/NAT.htm

Traffic shaping

Traffic shaping is implemented using Shorewall mangle and FireQOS: each mangle rule sets a well-known marker, markers are used to match traffic inside FireQOS tc classes.

The firewall needs to know how much inbound and outbound bandwidth has a red interface. The bandwidth value (expressed in kbit) is stored inside FwInBandwidth and FwOutBandwidth properties, wich are parts of the network interface record inside the networks db. Each red interface can have also the TCLinklayer property, see FireQoS documentation supported values <https://firehol.org/fireqos-manual/fireqos-params-class/#linklayer-linklayer-name-ethernet-atm>.

FireQOS tutorial suggests to use 90% of the declared bandwidth to shape the inbound traffic faster.

On red interfaces with FwInBandwidth and FwOutBandwidth set, ethernet offloading is automatically disabled.

Example:

enp0s20f2=ethernet
   FwInBandwidth=30000
   FwOutBandwidth=24000
   TCLinklayer=ethernet
   bootproto=none
   gateway=1.2.3.4
   ipaddr=1.2.3.5
   netmask=255.255.255.0
   role=red

All traffic shaping rules are saved inside the fwrules database with the same format. Valid actions for traffic shaping rules are:

• class;<name>: set associated tc class.
• provider;<name>: force the traffic to the provider specified by name

tc classess

tc classes are saved inside the tc database with type class.

Each tc class has the following properties:

• BindTo: empty (default) or comma-separated list of red interfaces. If one ore more interface is listed, the class is applied only to given interface
• Description: optional class description (used only in the UI)
• Mark: integer value which identify the marker used for this class. Maximum is 63
• MaxInputRate: maximum download rate, expressed in percentage of the total download bandwidth
• MaxOutputRate: maximum upload rate, expressed in percentage of the total upload bandwidth
• MinInputRate: reserved download rate, expressed in percentage of the total download bandwidth
• MinOutputRate: reserved upload rate, expressed in percentage of the total upload bandwidth
• Unit: bandwidth unit of measure for TC classes, default to %, supported values are from FireQoS doc

Example:

high=class
   BindTo=
   Description=
   Mark=2
   MaxInputRate=
   MaxOutputRate=
   MinInputRate=10
   MinOutputRate=10
   Unit=%

low=class
   BindTo=ens1
   Description=
   Mark=2
   MaxInputRate=
   MaxOutputRate=
   MinInputRate=10
   MinOutputRate=10
   Unit=kbps

Assumptions and limitations

1. All nDPI traffic is marked in forward chain. When a nDPI protocol is found, the whole connection is marked.
2. Priority rules are in post chain and can use nDPI markers. If a priority rule uses a role (interface) as source, the rule can’t be added to postrouting chain since the packet is already natted: Shorewall will move the rule on top of forwarding chain.
3. nDPI rules can’t block the http/https traffic if web proxy is enabled in transparent mode.
4. All nDPI markers are read from /proc/net/xt_ndpi/proto and shifted by 8 bits.
5. Divert rules can’t be used with nDPI, because an established TCP connection can’t be moved between providers.
6. Prerouting table is reserved by Shorewall for handlind the multi wan scenario.

See also:

• https://github.com/firehol/firehol/wiki/FireQOS
• https://github.com/firehol/firehol/wiki/FireQOS-Tutorial
• http://shorewall.net/manpages/shorewall.conf.html

Divert rules

A divert rule is used to force traffic to a specific provider.

For example, this rules will route all traffic to port 22 via the provider named myadsl:

1=rule
    Src=192.168.1.0/24
    Dst=0.0.0.0/0
    Service=fwservice;ssh
    Action=provider;myadsl
    status=enabled
    Position=2
    Description=

Properties:

• key: numeric id
• Src: can be a ‘any’, role (execpt red), zone (not interface), host object, ip address, ip range or CIDR
• Dst: can be a zone (not interface), host object, ip address, ip range or CIDR
• Action: provider object, in the form of “provider;<name>”
• Service: (optional) can be a service object
• status: can be enabled or disabled. Default is enabled
• Position: integer sorting key
• Description: (optional)

A rule is ignored during template expansion if:

• the source is red role
• the destination is a role which is not red
• source, destination and service are all set to any
• the provider doesn’t exists
• destination is set to any

Multi WAN

NethServer firewall can handle 15 red (WAN) interfaces. Implementation uses Shorewall with LSM (Link Status Monitor). The LSM daemon takes care of monitoring WAN connections (interface) using ICMP traffic and it informs Shorewall about interface up/down events. Each interface can be checked using multiple IPs (see checkip property below). At least one IP must be reachable to mark the WAN connection as usable. If no IP is specified (recommended option), the system will uses well-known default IPs (Google DNS and OpenDNS).

For each configured provider, LSM will send ping to a configured IP (checkip). When a provider status changes, the system will signal a wan-uplink-update event.

Inside the event, the action nethserver-shorewall-wan-update invokes:

• shorewall enable <interface> when a red interface is usable
• shorewall disable <interface> then a red interface is not usable

When an interface is disabled, all associated routes will be deleted.

When a new TCP connection is started, a route is selected and all successive packets will always be routed via same interface. If the used interfaces goes down, the connection is closed.

Actually two behaviors are implemented: balanced and active-backup.

Balanced

All red interfaces are simultaneously used accordingly to the configured weight (see below).

Example:

Given a connection A with weight 2, and connection B with weight 1, the firewall will route a double number of connections via A over B.

Active-backup

Red interfaces are ordered using the configured weight: higher the weight, higher the route priority. The interface with maximum weight will be the active connection, all other interfaces will be used if the active one goes down.

Example

Given 3 wan connections:

• A with weight 3
• B with weight 2
• C with weight 1

All traffic is routed via A. On failure of A, all traffic is routed via B. When B goes down, C is used. Whenever A comes backup, all traffic is again routed through it.

Providers

Providers are an abstraction over red interfaces (see man shorewall-providers). All providers must have a weight which is used to select the route for packets.

A provider record inside the networks database has following properties:

• key: name of provider
• interface: associated red interface, it’s mandatory
• weight: weight of connection expressed with an integer number, it’s mandatory
• Description: (optional) custom description

Example:

myisp=provider
  interface=eth1
  weight=5
  Description=my fast provider

Multi WAN example

1. Configure two interfaces as red, for example eth1 and eth2

db networks setprop eth1 role red
db networks setprop eth2 role red
signal-event interface-update

2. Create two providers:

db networks set firstisp provider interface eth1 weight 2
db networks set secondisp provider interface eth2 weight 1

3. Re-configure the firewall:

signal-event firewall-adjust

See /var/log/firewall.log to check for up/down events.

Routes can be checked using:

shorewall show routing

Static routes

Static routes are saved inside the routes database with a record of type static. Example:

8.8.4.4=static
    Description=My route
    Mask=255.255.255.255
    Router=89.97.220.225

Each record has the following properties:

• key: network address
• Mask: network mask
• Router: gateway for the network
• Description: a custom description (optional)

There is also a special type of static route called provider-static. These routes have the same properties as described above and are used to correctly route traffic for link monitor. This type of rules should never be manually edited.

nethserver-firewall-migration

Migrate firewall configuration to NethSecurity: https://github.com/NethServer/nethsecurity

Migrate to another machine

Execute on NS7:

firewall-export

Generated file can be found at /var/lib/nethserver/firewall-migration/export.tar.gz.

Access the NethSecurity installation and upload the export.tar.gz archive using SSH, then execute:

ns-import export.tar.gz

In-place migration

The firewall-migrate procedure will requires about 400MB of free disk space. The script will:

• prepare a NethSecurity image containing the firewall export archive
• write the image to the target disk
• reboot the system with the newly installed NethSecurity

At first boot, NethSecurity will automatically import the configuration.

Test the image build process:

• execute firewall-migrate
• if no error occurs, verify the /usr/share/nethserver-firewall-migration-builder/nethsecurity.img.gz file exists

To start the migration process, pass the target device as first argument:

firewall-migrate /dev/vda

NOTE: The script will wipe the NethServer installation. The script does not ask for confirmation!

Cleanup

The downloaded image is preserved to speed up multiple firewall-migrate runs. To cleanup existing images and download a new one on next run, execute:

rm -rf /usr/share/nethserver-firewall-migration-builder/

Not migrated

The following features will not be exported:

• Web proxy (Squid)
• Suricata
• UPS (NUT)

nethserver-freepbx

This package configures FreePBX and Asterisk for NethServer MariaDB, Asterisk 13 and FreePBX 14 will be installed and configured.

FreePBX Web UI Access

FreePBX is now reachable at https://IP_ADDRESS/freepbx from green interfaces. To add an IP address or a network from red allowed to access interface, configure it from NethServer Web UI under “PBX Access” page

Backup

Since FreePBX configuration is stored in MariaDB, database dump are split inside data and configuration backup:

• configuration backup: contains the asterisk db with all configurations (extensions, trunks, etc.)
• data backup: contanins the asteriskcdrdb db containg all PBX events

Open services from external networks

IAX and WebRTC access can be opened to external networks from server manager web gui -> PBX Access

SIP and other ports, from command line (Dangerous! Do it only if you know what you’re doing)

config setprop asterisk access public
signal-event firewall-adjust

Users

User are synced with NethServer users. You can manually force Userman module sync with

/usr/bin/scl enable rh-php56 "/usr/sbin/fwconsole userman sync"

WebRTC

You can create WebRTC extensions following this official guide from freepbx:

• http://wiki.freepbx.org/display/FPG/WebRTC+Phone-UCP#WebRTCPhone-UCP-EnablingWebRTCPhoneforauser

After the installation of Certificate Manager and WebRTC modules there is a valid self-signed certificate that can be used for WebRTC.

How to test it

1. Install “Certificate management” module

2. From “Advanced settings” page:

   • Enable the mini-HTTP Server
   • Set “SIP Channel Driver” to chan_pjsip

3. From “SIP settings”, inside the PJSIP tab, set the following values:

   • Certificate Manager: default
   • SSL Method: default
   • Verify Client: No
   • Verify Server: No
   • udp: yes
   • tcp: no
   • tls: no
   • ws: yes
   • wss: yes
   • Port listen: 5160

4. Configure a new PJSIP extension and set the following values:

   • Enable AVPF: yes
   • Enable ICE Support: yes
   • Media use received transport: no
   • RTP symmetric: yes
   • Media encryption: DTLS-SRTP
   • Require RTP (media) encryption: yes
   • Enable DTLS: yes
   • Use certificate: default
   • DTLS verify: no
   • DLTS setup: Act/Pass
   • DTLS Rekey Interval: 0

5. Restart asterisk and temporarly disable the firewall:

   systemctl restart asterisk shorewall clear

6. Open https://<server_ip>:8089/ws URL with the browser and accept the certificate

7. Try with one of the below WebRTC cients:

• http://jssip.net/
• https://www.doubango.org/sipml5/

Known bugs

• WebRTC under Chrome is not allowed in HTTP
• WebRTC in UCP module on FreepBX may not work as expected. (http://community.freepbx.org/t/webrtc-phone-with-https/26698/9)
• WebRTC may not work on Firefox

Custom User Management configuration

The nethserver-freepbx-conf-users action configures users using NethServer SSSD configuration. This creates an entry in userman FreePBX module called NethServer [AD|LDAP].

If you need to edit this entry and you don’t want it to be modified when nethserver-freepbx-conf-users is launched again, change it’s name adding “Custom” (or any other string) at the end. Example: ‘NethServer AD’ -> ‘NethServer AD Custom’

If you remove NethServer [AD|LDAP] string, another entry will be created by nethserver-freepbx-conf-users action.

To check user synchronization, use this command:

/usr/bin/scl enable rh-php56 -- /usr/sbin/fwconsole userman --syncall --force --verbose

Syncronization need a secure connection, use SSL or enable STARTTLS in Account Provider configuration in NethServer Web GUI

Update from legacy OpenLDAP driver to OpenLDAP2 driver

Since nethserver-freepbx-14.0.5, if NethServer users are configured using OpenLDAP, FreePBX users are configured using FreePBX OpenLDAP 2 driver instead of legacy one. If you have installed nethserver-freepbx before 14.0.5, and your user provider is configured using LDAP, you’re using legacy driver. You can check in FreePBX User Manager module interface if NethServer LDAP driver is “OpenLDAP Directory (Legacy)”

Updating from legacy driver to the new one, allows to permit access to FreePBX interface and UCP to LDAP users, but migration isn’t automatical because users would lose default extension associated and other custom options. The openldap_migration_from_legacy script, does the driver migration and restore users default_extensions. Other custom users options could be lost anyway. To execute migration, launch:

/usr/src/freepbx/openldap_migration_from_legacy

Cockpit API

settings/read

This api returns asterisk and httpd-fpbx configuration.

Input

• config: asterisk or httpd-fpbx

Input example (asterisk)

{
  "config": "asterisk"
}

Output example (asterisk)

{
  "configuration": {
    "type": "service",
    "name": "asterisk",
    "props": {
      "status": "enabled",
      "TCPPorts": "5060,5061,5038,8088,8089",
      "AllowExternalIAX": "disabled",
      "access": "green",
      "AllowExternalSIPS": "enabled",
      "AllowExternalWebRTC": "disabled",
      "UDPPorts": "4569,5036,5060,5160,10000:20000"
    }
  }
}

Input example (httpd-fpbx)

{
  "config": "httpd-fpbx"
}

Output example (httpd-fpbx)

{
  "configuration": {
    "type": "service",
    "name": "httpd-fpbx",
    "props": {
      "status": "enabled",
      "access": "green",
      "ValidFrom": "10.20.30.40/255.255.255.0,10.0.0.5/255.255.255.0"
    }
  }
}

settings/validate

This api validates an IP address allowed to access PBX web interface

Input

• ipAddress: an IP address
• netmask: a network netmask

Input example

{
  "ipAddress": "10.20.30.40",
  "netmask": "255.255.255.0"
}

Output example

{
  "state": "success"
}

settings/update

This api updates asterisk and httpd-fpbx configuration.

Input

• config: externalAccess or webInterfaceAccess
• allowExternalIAX: boolean flag to enable or disable external telephony access through IAX protocol (only if config: externalAccess)
• allowExternalSIPS: boolean flag to enable or disable external telephony access through secure SIP TLS protocol (only if config: externalAccess)
• webInterfaceAccess: list of IP addresses and netmasks allowed to access PBX web interface (only if config: webInterfaceAccess)

Input example (externalAccess)

{
  "config": "externalAccess",
  "allowExternalIAX": "enabled",
  "allowExternalSIPS": "enabled"
}

Input example (webInterfaceAccess)

{
  "config": "webInterfaceAccess",
  "webInterfaceAccess": [
    {
      "ipAddress": "5.6.7.8",
      "netmask": "255.255.255.0"
    },
    {
      "ipAddress": "10.20.30.40",
      "netmask": "255.255.255.0"
    }
  ]
}

nethserver-hotsync

What is it ?

HotSync aims to reduce downtime in case of failure, syncing your NethServer with another one, that will be manually activated in case of master server failure.

Normally, when a hardware damage occurs, the time needed to restore service is:

• fix or buy another server: from 4h to 2 days
• install the OS: 30 minutes
• restore backup: from 10 minutes to 8 hours

In summary, users are able to start working again with data from the night before failure after a few hours/days. Using HotSync, time 1 and 3 are 0, 2 is 5 minutes (time to activate spare server). Users are able to start working again in few minutes, using data from a few minutes before the crash.

By default all data included in backup are synchronized every 15 minutes. MariaDB databases are synchronized too, unless databases synchronization isn’t disabled. Applications that use PostgreSQL are synchronized (Mattermost, Webtop5) unless databases synchronization isn’t disabled.

Requirements

Two server machines are needed:

• MASTER: active server on production
• SLAVE: warm backup server

For a correct restore, it’s suggested to configure HotSync on two identical servers or two servers with same network cards number, name and position. If the master and slave servers differ, the restore procedure may behave unexpectedly (see Troubleshooting section on NethServer docs).

How it works

MASTER machine is the currently production server and it makes a backup in the SLAVE machine every “n” minutes. When a crash occurs on the master, slave becomes active taking the same IP address of the master. Doing so the downtime is minimal, only the time to launch some commands on the SLAVE.

Installation

On both MASTER and SLAVE, install nethserver-hotsync:

# yum install nethserver-hotsync --enablerepo=nethforge

Configuration

MASTER configuration:

[root@master]# config setprop rsyncd password <PASSWORD>
[root@master]# config setprop hotsync role master
[root@master]# config setprop hotsync SlaveHost <SLAVE_IP>
[root@master]# signal-event nethserver-hotsync-save

SLAVE configuration:

[root@slave]# config setprop rsyncd password <PASSWORD>
[root@slave]# config setprop hotsync role slave
[root@slave]# config setprop hotsync MasterHost <MASTER_IP>
[root@slave]# signal-event nethserver-hotsync-save

The <PASSWORD> must be the same on both master and slave.

If MySQL or PostgreSQL are installed, they will be synchronized by default. To disable databases synchronization:

[root@master]# config setprop hotsync databases disabled
[root@master]# signal-event nethserver-hotsync-save

You can update these settings also from Cockpit interface.

How to restore

The following procedure puts the SLAVE in production when the master has crashed.

1. switch off MASTER

2. if the SLAVE machine must run as network gateway, connect it to the router/modem with a network cable

3. following command changes the IP address and cuts off the network connection. If you are connected through an ssh console, launch the screen command

   [root@slave]# screen
   

4. on SLAVE launch command, and read carefully its output

   [root@slave]# hotsync-promote
   

   If no Internet connection is detected (e.g. you are restoring a firewall on a machine that was passing through crashed master for Internet connection), the scripts will purpose you some options:

   1. Restore master network configuration (IMPORTANT: use this option only if two servers are identical - NIC number, names and positions must be identical)
   2. Fix network configuration from Cockpit GUI (when restoring on different hardware)
   3. Continue without internet: assign correct roles before proceed with this option. Some events could fails (not recommended)

5. If necessary go to Server Manager or Cockpit GUI, in page Network and reassign roles to network interfaces as master one. Remember also to recreate bridge if you have configured DC. In case of DC errors consult troubleshooting section before proceed with network restore.

6. launch command

   [root@slave]# /sbin/e-smith/signal-event post-restore-data
   

7. if an USB backup is configured on MASTER, connect the backup HD to SLAVE

How to restore original server

To put again in production original crashed server:

1. configure it as SLAVE

2. synchronize it

3. switch off current MASTER

4. from bash execute:

   # hotsync-slave
   

5. restore configuration backup following above instructions

How to synchronize custom paths

It is possible to customize HotSync adding all kind of data through the use of plugins. The directory of plugins is /etc/hotsync.d. Executable files in this directory, are executed before the synchronization.

INCLUDE_FILE and EXCLUDE_FILE are the files that contain the list of paths to include and to exclude to/from HotSync. Those two variables are passed as arguments to the scripts of this directory when they are executed.

If you want to add files to the synchronization, append them to the INCLUDE_FILE. Append them to EXCLUDE_FILE to remove from the synchronization.

How to force synchronization from MASTER to SLAVE

From MASTER bash launch the command:

# hotsync

How to force packages installation on SLAVE

“hotsync-slave” script extracts from MASTER configuration backup the list of packages to install and install them. You can force the operation executing bash command:

# hotsync-slave

Components

hotsync

• is a shell script launched by cron every 15 minutes

• uses a lockfile to ensure that only one instance at a time is executed

• uses secure communication with rsync over stunnel

• creates a list of files to be included and another one to be excluded from rsync (using backup-data configuration and hotsync own logic)

• launch backup on MASTER

• launch an rsync that copy listed files from MASTER to SLAVE in a secure manner using stunnel

• if something fails, root is notified with an email

• you can check files that will be copied on the next synchronization using the command:

  hotsync --dry-run
  

hotsync-slave

Automatically executed on SLAVE every 60 minutes, extracts from MASTER configuration backup the list of packages to install and install them.

Supported packages

• nethserver-antivirus
• nethserver-backup-config
• nethserver-backup-data
• nethserver-base
• nethserver-c-icap
• nethserver-cockpit
• nethserver-collectd
• nethserver-cups
• nethserver-dante
• nethserver-dc
• nethserver-dedalo
• nethserver-directory
• nethserver-dnsmasq
• nethserver-duc
• nethserver-ejabberd
• nethserver-evebox
• nethserver-fail2ban
• nethserver-firewall-base
• nethserver-freepbx > 14.0.3
• nethserver-httpd
• nethserver-hylafax
• nethserver-iaxmodem
• nethserver-ipsec-tunnels
• nethserver-janus
• nethserver-letsencrypt
• nethserver-lightsquid
• nethserver-mail
• nethserver-mattermost
• nethserver-mysql
• nethserver-ndpi
• nethserver-netdata
• nethserver-nextcloud
• nethserver-ntopng
• nethserver-nut
• nethserver-openssh
• nethserver-openvpn
• nethserver-pulledpork
• nethserver-restore-data
• nethserver-roundcubemail
• nethserver-samba
• nethserver-samba-audit
• nethserver-squid
• nethserver-squidclamav
• nethserver-squidguard
• nethserver-sssd
• nethserver-subscription
• nethserver-suricata
• nethserver-vpn-ui
• nethserver-vsftpd
• nethserver-webtop5 (z-push state is not synchronized)

Packages nethserver-ntopng and nethserver-evebox are reinstalled without migrating history.

Please read the docs before proceed with restore.

nethserver-httpd

NethServer tries to remain compliant with the upstream configuration for httpd.

In Apache 2.4 every global option is inherit by all virtual hosts, except for the Rewrite directives.

In a clean installation the only defined virtual host is the one on port 443. When nethserver-httpd-virtualhosts is installed, the package adds a new default virtual host. This virtual host includes the /etc/httpd/conf.d/default-virtualhost.inc file to mimic the upstream behavior.

See also “Certificate management” and “Virtualhosts” section for further information.

Templates

Available templates under /etc/e-smith/templates/ directory:

• etc/httpd/conf.d/default-virtualhost.inc: common configuration included inside default virtual host on port 80
• etc/httpd/conf.d/nethserver.conf: default httpd configuration, includes conf.d/default-virtualhost.inc. Normally a package shouldn’t add a fragment to it.
• etc/httpd/conf.d/welcome.conf: configures the “Welcome” page if there is no default index page for the root URL
• httpd/vhost: everything inside will be included in default-virtualhost.inc. DO NOT USE, it’s here only for backward compatibility

Web applications

Every package which needs to change the Apache configuration:

• should add a static (or template-generated) file inside /etc/httpd/conf.d/ directory
• must include all Rewrite options inside a fragment for default-virtualhost.inc

Example

Package named nethserver-mywebapp which requires a rewrite rule from http to https. The web application must be accessible under http://<server_address>/mywebapp.

Static file /etc/httpd/conf.d/mywebapp.conf:

Alias /mywebapp /usr/share/mywebapp
<Directory "/usr/share/mywebapp">
    AllowOverride None
    Options None
    Require all granted
</Directory>

Rewrite rule fragment /etc/e-smith/templates/etc/httpd/conf.d/default-virtualhost.inc/30mywebapp:

#
# 30mywebapp
#

RewriteEngine on
RewriteRule ^/mywebapp(/.*)?$ https://%\{SERVER_NAME\}/mywebapp$1 [L,R=301]

Inside the createlinks file, the configuration should be applied during the update event:

my $event = "nethserver-mywebapp-update";
event_templates($event,
                   '/etc/httpd/conf.d/default-virtualhost.inc',
);

event_services($event,
                   'httpd' => 'reload',
);

configuration database

httpd=service
  SSLCipherSuite=DEFAULT:!EXP:!SSLv2:!DES:!IDEA:!SEED:+3DES
  TCPPorts=80,443
  access=green,red
  status=enabled
  XForwarded=disabled

• XForwarded: this setting forwards to all the proxypass vhosts the full https scheme and not only host header (in X-Forwarded-Host). It is a limitation of mod_proxy, the virtualhost on the proxy apache must transmit the original information of the request.

vhost database

A new vhost database is defined by this module. It contains records of type vhost, similar to:

vh1=vhost
    Access=private
    Description=description_text
    ForceSslStatus=enabled
    FtpPassword=ftp_password_value
    FtpStatus=enabled
    PasswordStatus=enabled
    PasswordValue=http_password_value
    ServerNames=www.nethserver.org,www.example.com
    SslCertificate=/etc/pki/tls/certs/NSRV.crt
    status=enabled
    PhpRhVersion=default
    PhpCustomSettings=disabled
    AllowUrlfOpen=enabled
    MemoryLimit=512
    UploadMaxFilesize=4
    PostMaxSize=8
    MaxExecutionTime=0
    MaxFileUploads=20

The database contains a special default record which represents the default virtual host:

default=vhost
  Description=Default virtual host
  FtpPassword=
  FtpStatus=enabled

This virtual host is always enabled and can’t be deleted. If FTP access is enabled, the user will be chrooted inside /var/www/html directory.

proxypass database

The proxypass database contains records of type VhostReverse, similar to:

ns7loc7.org=VhostReverse
    AllowEncodedSlashes=disabled
    CertVerification=no
    Description=
    HTTP=yes
    HTTPS=yes
    PreserveHost=yes
    SslCertificate=
    Target=http://localhost:8000
    ValidFrom=
    WebSockets=disabled
    WebSocketsPath=

• AllowEncodedSlashes (enabled/disabled) : Some web applications may expect encoded path separators (%2F for /) in the request URL. Apache does not accept the path separator in the encoded form. Set this prop to enabled if the encoded form can be safely forwarded to the backend (see Target) URL.
• CertVerification (yes/no) : If the Target URL has the https scheme, accept its certificate even if it is not valid.
• PreserveHost (yes/no): When enabled, this option will pass the HTTP Host header line from the incoming request to the proxied host, instead of the hostname specified in the Target URL field.
• WebSockets (enabled/disabled) : The WebSocket allows to open a two-way interactive communication session between the user’s browser and the server. The WebSocketsPath is optional.
• Target: The URL where the original request is forwarded. An URL has the form <scheme>://<hostname>:<port>/<path>.
• HTTPS and HTTP (yes/no): If enabled, the URL path or virtual host name can be accessed only with an SSL/TLS connection.
• SslCertificate: Select a certificate that is compatible with the virtual host name. Leave empty to use the default SSL certificate.
• ValidFrom : Restrict the access from the given list of CIDR networks. Elements must be separated with a , (comma).

ProxyPass, similar to:

folder=ProxyPass
    Description=
    HTTP=yes
    HTTPS=yes
    Target=http://localhost:8000
    ValidFrom=

• HTTPS and HTTP (yes/no): If enabled, the URL path or virtual host name can be accessed only with an SSL/TLS connection.
• ValidFrom : Restrict the access from the given list of CIDR networks. Elements must be separated with a , (comma).
• Target: The URL where the original request is forwarded. An URL has the form <scheme>://<hostname>:<port>/<path>.

rh PHP software collection

With the new cockpit server-manager the PHP version can be modified inside the virtualhost panel. It installs a pool of PHP-FPM, this new php version is only relevant to the current apache virtualhost. The prop PhpRhVersion is used to set the PHP version (default is the PHP 5.4,

php71, php72, php73).

The documentation page of the project are: - https://www.softwarecollections.org/en/scls/rhscl/rh-php71/ - https://www.softwarecollections.org/en/scls/rhscl/rh-php72/ - https://www.softwarecollections.org/en/scls/rhscl/rh-php73/

If the prop PhpCustomSettings is set to disabled the PHP setting values are inherited from the default values of PHP (from esmith configuration database), if enabled each vhost gets its own PHP values from its vhost props.

Events

signal-event nethserver-httpd-update
signal-event nethserver-httpd-save

NethServer 6 upgrade

Shared folders from NethServer 6 with property HttpStatus set to enabled can be migrated to virtual hosts using the vhost-migrate-ibay event:

signal-event vhost-migrate-ibay <ibay-name>

If the original ibay was availble to any virtual hosts (HttpVirtualHost = __ANY__), the ibay will be migrated inside the default virtual host. Otherwise a new virtual host record will be created.

The migration process is also available from the web interface.

UI plugins

The Modify action can be extended to display additional tabs, by adding a controller and the respective template under ModifyPlugin/ directories.

See the Samba User plugin on NethServer 6.x as an example

Welcome page

If there is no index page for the root URL, a default welcome page is shown accessing the HTTP and HTTPS ports of the server. If Cockpit UI is installed, the welcome page displays a customizable app launcher; users can choose which apps to show in the launcher by accessing Cockpit Applications page and clicking the Add to home page command in the kebab menu of any app.

It is possible to display an alternative welcome page that replaces the app launcher:

• create a subdirectory inside /usr/share/httpd/noindex/, e.g. mywebsite

• put a custom index page (e.g. index.html) inside /usr/share/httpd/noindex/mywebsite/

• create a subdirectory res inside /usr/share/httpd/noindex/mywebsite/

• put all the assets used by your page (images, scripts, …) inside /usr/share/httpd/noindex/mywebsite/res/

• the assets can be accessed from the index page using the prefix /res_mywebsite/, e.g. <link rel="stylesheet" type="text/css" href="/res_mywebsite/style.css">

• make sure the primary page is called index.html

• execute the following commands:

  config setprop httpd HomePage mywebsite
  signal-event nethserver-httpd-update
  

• your custom welcome page is now accessible the the HTTP and HTTPS ports

If you want to switch back to the default app launcher page execute:

config setprop httpd HomePage nethserver
signal-event nethserver-httpd-update

nethserver-httpd-admin

Apache configuration for the web administration console This Apache instance listens SSL connections on a non-standard port (by default 980) and execute system commands through sudo.

The main PHP scripts (controllers) are located in /usr/share/nethesis/nethserver-manager/ directory.

Apache mod_rewrite configuration hides index.php in URLs. A generic production URL has the following form: https://<hostname>:980/<lang>/<ModuleId>

An additional controller index_dev.php is available for development. It produces more verbose logging, uses uncompressed JS and CSS libraries, and shows clear text names of widget target names. The URL form for development is: https://<hostname>:980/index_dev.php/<lang>/<ModuleId>

To enable index_dev.php controller, you need to execute the following command:

# touch /usr/share/nethesis/nethserver-manager/debug

smwingsd daemon

To speed up database read operations, avoiding sudo calls, the smwingsd Perl daemon is contacted. As httpd-admin, the daemon is controlled by systemd, and on start reads configuration from /etc/smwingsd.conf. It listens on the local Unix socket /var/run/smwingsd.sock. You can contact it with a command like this:

# php -r '$payload=json_encode(array_slice($argv, 1), TRUE); print pack("CN", 0x10, strlen($payload)) . $payload;' configuration getjson sysconfig \
  | nc -U /var/run/smwingsd.sock  \
  | cut -b 6-   \
  | python -mjson.tool
{
   "name": "sysconfig",
   "props": {
       "Copyright": "",
       "ProductName": "NethServer",
       "Registration": "none",
       "Release": "rc1",
       "Version": "6.5"
   },
   "type": "configuration"
}

Customization

You can customize some parts of the NethServer web interface:

• logo
• favicon
• colors
• top bar background image
• menu bar background image

Top bar and menu bar

The page ships two different themes:

• light-polygons (default): includes background only for top bar
• color-polygons: includes backgrounds for top and menu bars

Enable color-polygons:

config setprop httpd-admin headerBackground color-polygons/toolbar.png
config setprop httpd-admin menuBackground color-polygons/sidebar.png

Enable light-poligons:

config setprop httpd-admin headerBackground light-polygons/toolbar.png
config setprop httpd-admin menuBackground ''

Custom themes

• Choose a name for the theme, like mytheme

• Create a directory named mytheme inside the /usr/share/nethesis/nethserver-manager/images directory

  mkdir -p /usr/share/nethesis/nethserver-manager/images/mytheme
  

• Copy the top bar background inside the new directory and name it toolbar.png

• Copy the menu bar background inside the new directory and name it sidebar.png

• Set the new backgrounds

  config setprop httpd-admin headerBackground mytheme/toolbar.png
  config setprop httpd-admin menuBackground mytheme/sidebar.png
  

Colors

Simple configuration database entries for a theme green and red:

config setprop httpd-admin colors "green,red,#1d247c"

Reload the interface, colors should be applied respectively on: top header, left menu, text headers (also in tables and links on left menu)

Logo and favicon

Copy a custom logo (best size: 185 x 30) inside /usr/share/nethesis/nethserver-manager/images/ directory. Change the default logo with this command:

config setprop httpd-admin logo mylogo.png

Copy a custom favicon inside /usr/share/nethesis/nethserver-manager/images/ directory. The favicon can be in PNG or ICO format, recommended sizes are 16x16 or 32x32 pixels. Change the default logo with this command:

config setprop httpd-admin favicon myfavicon.png

Reload the interface and check the new logo and favicon are loaded.

Note

Favicons are often cached by browsers: be patient and reload the page after a little while.

Database example

httpd-admin=service
   ForcedLoginModule=
   MaxSessionLifeTime=
   MaxSessionIdleTime=
   SSL=enabled
   TCPPort=980
   access=green,red
   colors=
   favicon=favicon.png
   headerBackground=
   logo=
   menuBackground=
   status=enabled

Session timeouts

The Server Manager session validity is subjected to forced logoff under the following conditions

• after an idle/inactivity period. It occurs when no user activity is detected in a certain amount of time
• after the maximum session lifetime has elapsed. It occurs if the user does not log out voluntarily

To configure the timeouts (both expressed as seconds) run the following command:

config setprop httpd-admin MaxSessionIdleTime 900 MaxSessionLifeTime 86400

To disable the session timeouts, set the values to 0 or the empty string:

config setprop httpd-admin MaxSessionIdleTime '' MaxSessionLifeTime ''

The new timeout values will affect new sessions. They does not change any active session.

nethserver-hylafax

Hylafax is a fax server for receiving and sending faxes.

Key features:

• Modem automatic configuration
• Virtual modem
• Ingoing/outgoing fax configuration
• Extensible accounting system
• Mail2Fax
• Mail server integration

Modem configuration

Hylafax+ uses a tool called faxaddmodem which guides the user throw modem configuration. We developed a simple non-interactive wrapper called neth-addmodem.

The script reads basic information, like the fax device name, from the SME db and use these data to setup the modem.

Accounting (FaxAccounting)

Every time a fax is received or sent, Hylafax will execute the /var/spool/hylafax/etc/FaxAccounting script. FaxAccounting runs all executable scripts inside the directory /var/spool/hylafax/etc/accounting in alphabetical order.

The default configuration will execute the script 00savefiles which saves all submitted and received faxes to /var/lib/nethserver/fax/docs

Incoming fax (FaxDispatch)

Hylafax requires a mail address where incoming faxes will be sent. If SendTo prop is not set, all faxes will be forwarded to root mailbox.

When a fax is received, Hylafax will execute the /var/spool/hylafax/etc/FaxDispatch script. FaxDispatch runs all executable scripts inside the directory /var/spool/hylafax/etc/dispatch in alphabetical order.

The default configuration will execute the script 90print. If hylafax prop PrintReceived is enabled, all faxes are printed into the PrinterName printer.

Outgoing fax (FaxNotify)

When a fax is sent, Hylafax will execute the /var/spool/hylafax/etc/FaxNotify script. FaxNotify runs all executable scripts inside the directory /var/spool/hylafax/etc/notify in alphabetical order.

Custom scripts

When adding a custom script not handled by an RPM, please add the script in a directory inside /var/lib/nethserver/fax/ and link it to the final destination. In fact the /var/spool/hylafax/etc/ directory is not (actually) in the backup.

So, for example, if you wish to add a 90test script to accounting, save the files in the /var/spool/hylafax/etc/accounting directory and link to the Hylafax configuration directory:

ln -s /var/lib/nethserver/fax/accounting/90test /var/spool/hylafax/etc/accounting/90test

Authentication

Clients are authenticated using reverse dns queries: all ip resolved with the same domain as server itself are automatically authenticated. If reverse dns query, client can authenticate using user name and password of a user inside faxmaster group.

Clients should always use username, despite of authentication method used.

Internationalization

Notification on incoming and outgoing faxes can be localized using the Lang property. Default value is en_US.

Client configuration

The only tested client is YajHFC (see http://www.yajhfc.de/). Current configuration works only with LAN clients.

Configuration:

• Insert server ip (or hostname) and username
• Leave blank the password field
• Disable passive mode

Configuration DB

Example:

hylafax=configuration
   AreaNumber=
   ClientShowReceived=disabled
   CountryCode=39
   DateFormat=DMY
   Debug=disabled
   DispatchFileType=ps
   FaxDevice=ttyS0
   FaxName=fax
   FaxNumber=
   InternationalPrefix=00
   Lang=it_IT
   LongDistance=0
   Mail2Fax=disabled
   Mode=both
   NotifyFileType=tif
   NotifyMaster=always
   PBXPrefix=
   PaperSize=A4
   PrintReceived=disabled
   PrinterDriver=ljet4
   PrinterName=
   Resolution=196
   RingsBeforeAnswer=1
   SambaFax=disabled
   SambaFaxName=SambaFax
   SendReport=disabled
   SendTo=aa``test.tld
   SummaryReport=disabled
   WaitDialTone=enabled

hylafax-hfaxd=service
   TCPPort=4559
   status=enabled
   access=private

hylafax-faxq=service
   status=enabled

nethserver-iaxmodem

The module uses modems database to store record describing a iax modem. After a new record is created, the system will generate all configuration files in /etc/iaxmodem using the action nethserver-iaxmodem-modemsetup.

When a modem is selected under the nethserver-hylafax web ui, the action nethserver-iaxmodem-hylasetup will create a sutiable modem configuration file under /var/spool/hylafax/etc directory.

Multiple IAX modems

Current nethserver-hylafax implementation doesn’t handle more than one modem. To use multiple IAX modems:

1. Create a IAX from web UI

2. Copy an existing config.IAXxx file and change number and identification strings Eg:

   cp /var/spool/hylafax/etc/config.ttyIAX351 /var/spool/hylafax/etc/config.ttyIAX350
   

3. Copy an existing faxgetty configuration, and change the device name inside the new file: Eg:

   cp /etc/init/faxgetty-ttyIAX350.conf /etc/init/faxgetty-ttyIAX351.conf
   sed -i 's/ttyIAX350/ttyIAX351/g' /etc/init/faxgetty-ttyIAX350.conf
   

4. Start services

   start faxgetty-ttyIAX351
   service hylafax restart
   

Configuration database:

mymodem=iax
   cidName=NethServer
   cidNumber=351
   extension=351
   password=351
   server=192.168.1.1

nethserver-ipsec-tunnels

This pacakge provides the configuration of VPN tunnels based using IPSec. The implementation can support custom properties to override the configuration from web interface.

All records of type ipsec-tunnel are saved insided the vpn database.

Every property in the form Custom_<name> will override any existing property. The same syntax can also be used to set any IPsec options supported by OpenSwan.

Example: override left prop

Given the following record:

nethesis-test=ipsec-tunnel
   compress=no
   dpdaction=hold
   esp=auto
   ike=auto
   left=192.168.2.246
   leftid=@nethesis
   leftsubnets=192.168.1.0/24
   pfs=yes
   psk=Nethesis,12345678911
   right=1.2.3.4.5
   rightid=@test
   rightsubnets=192.168.6.0/24
   status=enabled

The admin can override the left property:

db vpn setprop nethesis-test Custom_left %any
signal-event nethserver-ipsec-save

Example: set new option

Set aggressive mode:

db vpn setprop nethesis-test Custom_aggrmode yes
signal-event nethserver-ipsec-save

Logs

Logs can be inspected using this command:

journalctl -u ipsec.service

nethserver-janus

This package configure Janus Gateway https://github.com/meetecho/janus-gateway

Start Janus Gateway at boot

To start Janus Gateway and enable it at boot, execute:

config setprop janus-gateway status enabled
signal-event nethserver-janus-update

NAT modes

Janus-gateway can operate in three different NAT mode:

• STUN (default)
• ICE
• 1:1 NAT

There are 4 prop under janus-gateway key to configure NAT mode and options:

• NatMode <stun|ice|1:1>
• StunServer is the STUN server to use. Default is stun1.l.google.com. Ignored if mode isn’t stun
• StunPort is the STUN server port. Default is 19302. Ignored if mode isn’t stun
• PublicIP is the public IP address of the server that runs janus-gateway . Ignored if mode isn’t 1:1

For configuring TURN, create a template custom

STUN and ICE enforced interfaces

You can choose which interface should be used for the ICE candidates gathering. In default configuration Janus try to use all interfaces, except ‘vmnet*’, that is VMware interface which is known to cause problems, ‘tun*’ and ‘tap*’ interfaces for VPNs, ‘virb*’ for KVM and ‘vb-*’ for NethServer AD container. You can modify this behavior by explicit list interfaces to use or by listing interfaces to exclude. You can configure one of those two props:

• ICEEnforceList comma separated list of interfaces to use for ICE gathering. For instances “e0,e1”.
• ICEIgnoreList comma separated list of interfaces to exclude from ICE gathering. Janus will use all interfaces except those. For instance ‘e3,vmnet,10.0.0’.

Disable plugins

All plugins are enabled by default. To disable some of them execute:

config setprop janus-gateway DisabledPlugins "<PLUGINS_NAMES_SEPARATED_BY_COMMAS>"
signal-event nethserver-janus-update

Example:

config setprop janus-gateway DisabledPlugins "libjanus_voicemail.so,libjanus_recordplay.so"
signal-event nethserver-janus-update

To re-enable them:

config setprop janus-gateway DisabledPlugins ""
signal-event nethserver-janus-update

Debug problems with Admin Api

The Janus Admin Api are used to debug problems (e.g. ssl, ice, …).

nethserver-janus automatically enables Admin Api on HTTP localhost on port 7088.

To use it:

1. make an ssh tunnel:

   ssh IP -L 7088:localhost:7088
   

4. extract the Admin Api secret:

   grep admin_secret /opt/janus/etc/janus/janus.jcfg | cut -d ' ' -f 3
   

5. Follow the official documentation on how to use admin api: https://janus.conf.meetecho.com/docs/admin.html

Note: here is an example on how to use it with Node.js: https://gist.github.com/alepolidori/720e47448e82b99ef3495a1849773ecc

nethserver-lang

The localization process turns around the Transifex online tool.

The Transifex client, available as /usr/bin/tx command, is required. Some Linux distributions have already packaged it. For NethServer, it is available from EPEL.

yum --enablerepo=epel install transifex-client
yum install git

Edit ~/.transifexrc:

[https://www.transifex.com]
hostname = https://www.transifex.com
password = <YOURPASSWORD>
token =
username = <YOURUSERNAME>

Refer also to Internationalization in the Developer’s Manual.

Building a release RPM

Periodic builds of the RPM are started automatcally at Travis CI. See nethserver-makerpms in the Developer’s Manual for more information about the automated builds on Travis CI.

According to the .travis.yml and build.sh scripts, the build procedure pulls translations of existing catalogs from Transifex, creates, adds a release tag and pushes a git commit and tag. Then starts the build of a new RPM that is published in the updates repository.

As alternative, a release commit can be manually prepared with the :releasetag-section: command, as for other nethserver-* packages.

$ releasetag -T X.Y.Z
...edit the changelog entry
$ git push --follow-tags

The manual procedure is still useful when new resources are added to .tx/config. See the Adding new resources section below.

Building RPMs for testing

To manually build the RPMs for testing prepare the build environment:

• Install a build environment as described in Building RPMs.

• (Fork and) Clone the git repository into your machine:

  git clone https://github.com/NethServer/nethserver-lang.git
  

Then run this workflow with nethserver-makerpms:

1. Pull updates from Transifex:

   tx pull -a
   

2. (optional) clean up partially translated languages:

   git add <files for new configurations just added to .tx/config>
   git clean -f -d
   

3. Commit changes to git repository:

   git commit -a -m 'Pulled translation strings from Transifex'
   

4. Build RPMs:

   makerpms nethserver-lang.spec
   

5. If everything is ok, do not forget to push the commit back to GitHub, and open a Pull Request:

   git push
   

Adding new resources

After the developer has pushed sources to Transifex, the new resources must be added here, inside the .tx/config file.

For Cockpit resources, refer to https://nethserver.github.io/nethserver-cockpit/i18n/#translations-packages

For for PHP resources add to .tx/config a section like this

[nethserver.MyModule]
file_filter = locale/<lang>/server-manager/NethServer_Module_MyModule.php
source_file = locale/en/server-manager/NethServer_Module_MyModule.php
source_lang = en
type = PHP_ALT_ARRAY

Then retrieve all files:

tx pull -r nethserver.MyModule -a

Finally, add all files to the repository:

git add *NethServer_Module_MyModule*

And push the commit and open a Pull Request on GitHub

git push

nethserver-lightsquid

LightSquid, Web Proxy report

LightSquid is a lite and fast log analizer for squid proxy. It parses /var/log/squid/access.log log file on daily basis and generate an HTML report.

LightSquid is enabled by default when the nethserver-lightsquid package is installed. Web interface can be accessed at http://<server>/<hash>.

The hash is auto-generated (see below).

Database

The configuration is saved in the lightsquid key inside the configuration database.

Example:

lightsquid=configuration
   alias=44da17a293c2aae17815bb95af98883355520aef

nethserver-mail

Mail system implementation based on Postfix, Dovecot, Rspamd, OpenDKIM. The mail system configuration is splitted into many RPMs, described in the following sections.

nethserver-mail-common

• Common infrastructure for nethserver-mail-server and nethserver-mail-filter, Postfix-based.
• Relay
• Queue parameters: age + message size
• MX record configuration

nethserver-mail-smarthost

• Send mail through the given MTA (smarthost), with SMTP/AUTH
• StartTLS encryption
• Set sender address for mail sent from root user (see Notifications section under nethserver-base package README)

nethserver-mail-disclaimer

• Attach disclaimer/legal notice to outbound messages for certain domains
• Runs altermime with Postfix content_filter option

nethserver-mail-filter

• Based on Rspamd
• Anti-spam with DNSBL (see: nethserver-unbound)
• Anti-virus
• Attachment block
• Real-time Blackhole List (RBL)
• Sender Policy Framework (SPF)
• Customized spam threshold
• Sender WBL, Recipient whitelist

nethserver-mail-server

• IMAP/POP3 mailbox access protocols
• STARTTLS enabled by default
• Mail quota
• Sieve filters
• Postfix/dovecot-lda integration
• Multi-domain
• Domain-specific configuration
• Pseudonyms
• SMTP authentication
• Active Directory integration
• SpamAssassin’s Bayesian classifier training (spamtrainers group)
• Spam retention time
• Sender address restriction based on login name
• Dynamic group members address list expansion

nethserver-mail-ipaccess

IMAP access for a specific group of users. See IP-based IMAP access restriction.

nethserver-mail-getmail

The package configures getmail using cron.

For each enabled account the system:

• generates a .cfg file inside the /var/lib/getmail directory from the template /etc/e-smith/templates/getmailrc
• adds a line inside the /etc/cron.d/getmail, all getmail instances use a non-blocking flock
• delivers the messages using dovecot-lda

All operations are logged in /var/log/maillog.

If a virus is found inside a received mail, the message is dropped.

The evidence of log in /var/log/maillog:

Feb 14 18:19:10 vm5 clamd[1791]: instream(local): Eicar-Test-Signature FOUND
Feb 14 18:19:11 test getmail: msg  4/12 (702 bytes) msgid 000008bb5785fb1d from <root@test.neth.eu> dropped by filter Filter_classifier clamdscan (allow_root_commands="True", arguments="('-c', '/etc/clamd.d/amavisd.conf', '--stdout', '--no-summary', '--infected', '-')", command="clamdscan", exitcodes_drop="('1',)", exitcodes_keep="('0',)", group="None", ignore_stderr="False", path="/usr/bin/clamdscan", unixfrom="False", user="None")

nethserver-mail-p3scan

This package configures p3scan, full-transparent POP3 proxy-server for email clients.

• POP3 and POP3s proxy
• Anti-virus and anti-spam checks

mail-quarantine

This package makes a quarantine for spam. They are sent to a mailbox (you need to manually created it), waiting a review of the sysadmin. If enabled a mail notification is sent to the postmaster (root alias) for each quarantined email.

Database format

configuration

Postfix example:

postfix=service
   ...
   AccessPolicies=
   AlwaysBccStatus=disabled
   AlwaysBccAddress=
   MessageQueueLifetime=4
   MessageSizeMax=20000000
   ConnectionsLimit=
   ConnectionsLimitPerIp=
   SystemUserRecipientStatus=disabled
   ...
   SenderValidation=disabled
   DynamicGroupAlias=disabled
   HeloHost=
   SmartHostAuth=disabled
   SmartHostAuthStatus=disabled
   SmartHostName=192.168.5.252
   SmartHostPassword=password
   SmartHostPort=25
   SmartHostStatus=disabled
   SmartHostTlsStatus=enabled
   SmartHostUsername=ns1

• AccessPolicies: A comma separated list of values. Obsoletes SubmissionPolicyType prop. Currently defined values are smtpauth and trustednetworks.
• smtpauth enable SMTP/AUTH on port 25, otherwise it is enabled only on 587 and 465 submission ports
• trustednetworks allow relay from any host in trusted networks (green network included).
• AlwaysBccStatus {enabled,disabled}: if enabled any message entering Postifx mail system is copied to the given AlwaysBccAddress.
• AlwaysBccAddress: an email address that always receives a message copy (controlled by AlwaysBccStatus).
• SystemUserRecipientStatus {enabled,disabled} enabled, accept from any network the recipient addresses formed by user account names and domain part localhost, localhost.<domainname> and FQDN hostname.
• SenderValidation {enabled,disabled}, default disabled, checks the SMTP sender is consistent with /etc/login_maps and /etc/login_maps.pcre contents.

• DynamicGroupAlias {enabled,disabled}, default disabled,

  if enabled, create distribution lists based on system groups. See also the “Dynamic group aliases” section below.

• HeloHost. FQDN hostname used by Postfix when connecting to other MTAs

Dovecot example:

dovecot=service
    ...
    AdminIsMaster=disabled
    DeletedToTrash=disabled
    FtsLuceneStatus=enabled
    ImapStatus=enabled
    LmtpInetListenerStatus=disabled
    LogActions=disabled
    MaxProcesses=400
    MaxUserConnectionsPerIp=12
    PopStatus=enabled
    QuotaDefaultSize=20
    QuotaStatus=disabled
    SharedMailboxesStatus=enabled
    SharedSeen=disabled
    SpamFolder=Junk
    SpamRetentionTime=15d
    TlsSecurity=required
    RestrictedAccessGroup=

Properties:

• AdminIsMaster {enabled,disabled} allow root user to impersonate other users
• DeletedToTrash {enabled,disabled} deletedtotrash plugin
• FtsLuceneStatus {enabled,disabled} lucene indexed search plugin
• ImapStatus {enabled,disabled} IMAP protocol switch
• LmtpInetListenerStatus {enabled,disabled} open a TCP socket for LMTP protocol
• LogActions {enabled,disabled} IMAP actions logging plugin
• MaxProcesses N maximum number of worker processes spawned by dovecot. A single user session usually requires multiple processes.
• MaxUserConnectionsPerIp N maximum TCP connections for one user behind the same IP. This number will be multplied by 5 for connections coming from localhost.
• PopStatus {enabled,disabled} POP3 protocol switch
• QuotaDefaultSize N Default user quota size (1 unit is 10MB)
• QuotaStatus {enabled,disabled} General user mail quota switch
• SharedMailboxesStatus {disabled,enabled} Control the “Shared” IMAP namespace for per-user folder sharing
• SharedSeen {disabled,enabled} Control the Seen IMAP flag (enabled means all users will see an email as read as soon as the first user reads it)
• SpamFolder FolderName Deliver spam tagged messages to the given folder (applied to all users)
• SpamRetentionTime Nd Expunge messages in SpamFolder if older than the given time span. “d” is for days.
• TlsSecurity {optional,required} controls dovecot disable_plaintext_auth parameter: if set to required clear-text authentication methods are disabled, while optional enables them.
• RestrictedAccessGroup The value is a long group name, like domain admins@mydomain.tld. Members of the given group can login to dovecot services only from trusted networks. Install the
• VszLimit set the default_vsz_limit dovecot option, values are expressed in M nethserver-mail-server-ipaccess package to enable this feature.

p3scan example:

p3scan=service
  SSLScan=enabled
  SpamScan=enabled
  TCPPort=8110
  Template=/etc/p3scan/p3scan-en.mail
  VirusScan=enabled
  access=
  status=enabled

rspamd example:

rspamd=service
    BlockAttachmentClassList=Exec
    BlockAttachmentCustomList=doc,odt
    BlockAttachmentCustomStatus=disabled
    BlockAttachmentStatus=enabled
    OletoolsStatus=enabled
    Password=uO9QjlnRCDsT0ZCD
    RecipientWhiteList=
    SenderBlackList=
    SenderWhiteList=
    SpamCheckStatus=enabled
    SpamDsnLevel=20
    SpamGreyLevel=4
    SpamKillLevel=15
    SpamSubjectPrefixStatus=disabled
    SpamSubjectPrefixString=***SPAM***
    SpamTag2Level=6
    SpamTagLevel=2
    VirusAction=reject
    VirusCheckStatus=enabled
    VirusScanOnlyAttachment=false
    VirusScanSize=20000000
    VirusScanTimeout=90
    VirusScanSoftReject=enabled
    status=enabled

Properties:

• BlockAttachmentClassList {Exec,Arch} Reject the attachements matching the extension list
• BlockAttachmentCustomList List Reject the attachements matching the custom extension list
• BlockAttachmentCustomStatus {enabled,disabled} Enable the custom list of rejected extensions
• OletoolsStatus {enabled,disabled} Enable Oletools to reject suspicious microsoft office document macro
• Password Password to authenticate the user rspamd for the Rspamd UI
• RecipientWhiteList Do not perform checks for the recipient list, always accept
• SenderBlackList Do not perform checks for the sender list, always reject
• SenderWhiteList Do not perform checks for the sender list, always accept
• SpamCheckStatus {enabled,disabled} Enable the SPAM filter
• SpamSubjectPrefixStatus {enabled,disabled} Enable to rewrite the subject when a possible spam is detected
• SpamSubjectPrefixString string Rewrite the subject with the string when a possible spam is detected
• VirusAction Possible action when a virus is detected (reject is default, ‘rewrite_subject’ to tag as spam)
• VirusCheckStatus {enabled,disabled} Enable the virus check with Clamav
• VirusScanOnlyAttachment {true,false} If true only messages with non-image attachments will be checked
• VirusScanSize The messages > n bytes in size are not scanned (valuable for Clamav and Oletools)
• VirusScanTimeout Time in seconds that clamav is allowed to scan the email before to hit a timeout (default 90)
• VirusScanSoftReject {enabled,disabled} when clamav hit a timeout to scan an email, if enabled the email is soft rejected (try again)

domains

Record of type domain:

internal.tld=domain
  ...
  TransportType=none

mycompany.com=domain
  ...
  TransportType=Relay
  RelayHost=10.1.1.4
  RelayPort=25
  DisclaimerStatus=disabled

test.tld=domain
  ...
  TransportType=SmtpSink

example.com=domain
  ...
  TransportType=LocalDelivery
  UnknownRecipientsActionType=deliver
  UnknownRecipientsActionDeliverMailbox=jdoe
  AlwaysBccStatus=enabled
  AlwaysBccAddress=admin``there.org

other.net=domain
  ...
  TransportType=Relay
  RelayHost=mail.other.net
  RelayPort=25

accounts

Groups:

employees@domain.com=group
   ...
   MailStatus=enabled
   MailAccess=private

administrators@domain.com=group
   ...
   MailStatus=enabled
   MailAccess=public

info@domain.com=group
   ...
   MailStatus=enabled
   MailAccess=public

User:

jdoe=user
   FirstName=John
   LastName=Doe
   ...
   MailStatus=enabled
   MailQuotaType=custom
   MailQuotaCustom=15
   MailForwardStatus=disabled
   MailForwardAddress=
   MailForwardKeepMessageCopy=no

and his pseudonyms: ::

 john.doe``example.com=pseudonym
   Account=jdoe
   ControlledBy=system
   Access=public

 doe``=pseudonym
   Account=jdoe
   ControlledBy=operators
   Access=private

getmail

All records of type getmail are saved inside the getmail database.

Properties:

• The key is the mail account to be downloaded
• status: can be enabled or disabled, default is enabled
• Account: local user where messages will be delivered. Should be in the form user@domain
• Server: server of the mail account
• Username: user name of the mail account
• Password: password of the mail account
• Delete: numbers of days after downloaded messages will be deleted, -1 means never, 0 means immediately
• Time: integer number rappresenting the minutes between each check, valid valued are between 1 and 60
• FilterCheck: if enabled, check downloaded messages for viruses and spam using rspamc classifier

• Retriever: can be any getmail retriever, see Getmail official doc

  Retrievers available in the web interface:

  • SimplePOP3Retriever
  • SimplePOP3SSLRetriever
  • SimpleIMAPRetriever
  • SimpleIMAPSSLRetriever

Example:

db getmail set test@neth.eu getmail Account pippo@neth.eu status enabled Password Nethesis,1234 Server localhost Username test@neth.eu Retriever SimplePOP3Retriever Delete enabled Time 30 VirusCheck enabled SpamCheck enabled

quarantine

The properties are under the rspamd key (configuration database):

   rspamd=service
   ...
   QuarantineAccount=vmail+quarantine
   QuarantineSelector=is_reject
   QuarantineStatus=enabled
   SpamNotificationStatus=disabled


* ``QuarantineAccount``: The local email box where to send all spams (spam check is automatically disabled on this account). You must create it manually. You could send it to an external mailbox  but then you must disable the spam check on this server
* ``QuarantineSelector``: It is possible to move to quarantine all spams (add_header, rewrite_subject, reject), allowed values are ``is_reject`` (default) or ``is_spam``
* ``QuarantineStatus``: Enable the quarantine, spam are no more rejected: enabled/disabled (default)
* ``SpamNotificationStatus``: Enable the email notification when email are quarantined: enabled/disabled (default)

For example, the following commands enable the quarantine:

config setprop rspamd QuarantineAccount spam@domain.org QuarantineStatus enabled SpamNotificationStatus enabled
signal-event nethserver-mail-quarantine-save

Mail quota

The default mail quota is configured in dovecot.conf. Custom user mail quota is set by the dovecot-postlogin script, by reading /etc/dovecot/user-quota (which is a template). If a custom mail quota is set the UI interface does not show the updated value until the user performs an IMAP login.

Disabled users

By default all system users are also Dovecot users. To disable a user we configure a blacklist in dovecot.conf: /etc/dovecot/deny.passwd.

As Dovecot is configured as authentication backend for Postfix, a disabled user loses also SMTP AUTH access.

Testing Dovecot with Mutt

Read admin’s mail with Mutt IMAP client. Quickstart:

yum install mutt
cat - <<EOF > ~/.muttrc
set spoolfile="imaps://root@localhost/"
set folder=""
EOF
mutt

See: http://dev.mutt.org/doc/manual.html

When mutt starts always asks for the root password. To avoid typing the password again and again write it in .muttrc:

set spoolfile="imaps://root:PASSWORD@localhost/"
set folder=""

PASSWORD must be URL-encoded. For instance the slash character / is encoded as %2f.

Set special ACL on mailboxes

The nethserver-mail-shrmbx-modify action applies some predefined ACL settings to shared mailboxes (type the mailbox name twice: the action performs also rename):

/etc/e-smith/events/actions/nethserver-mail-shrmbx-modify EVENT OLDNAME NEWNAME ID PERM [ID PERM ...]

For instance, let’s grant full “admin” permissions to group “administrators”:

/etc/e-smith/events/actions/nethserver-mail-shrmbx-modify ev 'Public folder1' 'Public Folder One' group=administrators@$(hostname -d) ADMIN

You can also use doveadm to set special ACL on a shared mailbox:

doveadm acl set -u <user> <shared_mailbox> <subject> <flags>

Example: allow insert and expunge to user goofy on public mailbox testshare (domain of the machine is local.nethserver.org):

doveadm acl set -u goofy@local.nethserver.org Public/testshare "user=goofy@local.nethserver.org" insert expunge

IP-based IMAP access restriction

This feature allows to restrict IMAP access for a specific group. Members of the given group have IMAP access restricted to trusted networks.

1. Install nethserver-mail-ipaccess package

   yum install nethserver-mail-ipaccess
   

2. Set the limited group, remember to use the full group name: <group>@<domain>

   config setprop dovecot RestrictedAccessGroup <group>@<domain>
   

   Example for group collaborators@nethserver.org:

   config setprop dovecot RestrictedAccessGroup collaborators@nethserver.org
   

3. Apply the configuration

   signal-event nethserver-mail-server-save
   

Syntax of /etc/dovecot/ipaccess.conf

The dovecot-postlogin script enforces an IP-based access policy to dovecot services when the file :file:/etc/dovecot/ipaccess.conf exists and is readable.

The file is composed by comments and records. Comments are line starting with #, whilst records have the following syntax:

<long group name> = <cidr list>

A long group name is the group name with domain suffix, like domain admins@mydomain.tld.

The cidr list is a comma-separated list of IP and network addresses in CIDR format, like 127.0.0.1, 192.168.1.0/24, 10.1.1.2. The binary conversion is implemented by the NetAddr::IP Perl module. See perldoc NetAddr::IP for details.

Enable dovecot IMAP rawlog

This section describes how to record the list of IMAP commands sent by the client and the server during an IMAP session. For more information see Dovecot rawlog.

Create the file /etc/e-smith/templates-custom/etc/dovecot/dovecot.conf/90rawlog with the following contents:

#
# 90rawlog (custom)
#
import_environment = $import_environment DEBUG=1

service imap-postlogin \{
  executable = script-login -d rawlog -t /usr/libexec/nethserver/dovecot-postlogin
\}

Apply the new configuration

signal-event nethserver-mail-server-save

To enable IMAP rawlog for a specific user, identify the user (vmail) home directory with the following command:

# doveadm user -u first.user@dpnet.nethesis.it
userdb: first.user@dpnet.nethesis.it
system_groups_user: first.user@dpnet.nethesis.it
uid       : 987
gid       : 990
home      : /var/lib/nethserver/vmail/first.user@dpnet.nethesis.it

Warning

Always use the user long name form, which includes the @domain suffix. In our example first.user would not work

Create the dovecot.rawlog directory and change permissions:

mkdir "/var/lib/nethserver/vmail/first.user@dpnet.nethesis.it/dovecot.rawlog"
chown vmail:vmail "/var/lib/nethserver/vmail/first.user@dpnet.nethesis.it/dovecot.rawlog"

Detailed IMAP rawlogs are now created under the user’s (vmail) home directory. Each session is split into two files: .in file for client requests, .out file for server responses. For instance,

/var/lib/nethserver/vmail/first.user@dpnet.nethesis.it/dovecot.rawlog/20180913-143301-6293.in
/var/lib/nethserver/vmail/first.user@dpnet.nethesis.it/dovecot.rawlog/20180913-143301-6293.out

The initial timestamp helps to mix them together and obtain the complete IMAP session trace:

sort -n /var/lib/nethserver/vmail/first.user@dpnet.nethesis.it/dovecot.rawlog/20180913-143301-6293.*

Access the rspamd UI

The rspamd UI is available from the same httpd instance of Server Manager:

https://<IP>:980/rspamd

Access is granted to any account defined in /etc/httpd/admin-conf/rspamd.secret. By default a rspamd login is created automatically. Its password is available with the following command:

config getprop rspamd Password

Additional accounts can be created with the following command:

/usr/bin/htpasswd -b -m /etc/httpd/admin-conf/rspamd.secret username S3cr3t

If an account provider is configured, the default access policy to rspamd UI is granting access also to admin user and members of the domain admins group. Type config show admins for details.

Bayesian rules upgrade to rspamd

Each Junk (or junkmail) folder from users’ accounts, if present, can be used to train the Rspamd Bayesian filter database, by running the attached script:

bash /usr/share/doc/nethserver-mail-server-*/bayes_training.sh

Sender address validation

If the postfix/SenderValidation prop is set to enabled the SMTP server restricts the Mail from command usage. The sender address must be associated with the SMTP login name. The login/sender match is specified in the following Postfix tables, both implemented with an e-smith template:

• /etc/postfix/login_maps
• /etc/postfix/login_maps.pcre

To enable the SenderValidation:

config setprop postfix SenderValidation enabled
signal-event nethserver-mail-server-update

Postfix SMTP listening ports

The Postfix SMTP server listens on the following TCP ports

• 25, standard SMTP port; used by other MTAs
• 587, standard SMTP submission port; STARTTLS required by default to protect login passwords; used by MUAs
• 465, standard SMTPS submission port; TLS always required at socket level; used by MUAs which not support STARTTLS
• 10587, additional SMTP submission port for localhost only; no TLS required; used by local mail applications

Dynamic group aliases

If the postfix/DynamicGroupAlias prop is enabled an additional virtual_alias_maps TCP table is available. It expands a long group name to the group members list with a getent group call. The table is implemented in :file:/usr/libexec/nethserver/postfix-get-group. Note that group members lists are returned by SSSD: as such they obey to its caching rules.

When the DynamicGroupAlias general switch is enabled, individual groups can be disabled and marked private. If MailStatus prop is disabled the group long name is not considered a valid email address anymore. The MailAccess prop works like the Access prop for user records: if set to private only authenticated SMTP clients are allowed to use it as recipient.

Accounts DB group props example:

employees@domain.com=group
   ...
   MailStatus=enabled
   MailAccess=private

nethserver-makerpms

RPM builds by Linux containers

This is a simple RPM build environment based on the official CentOS Docker image.

It can build RPMs in the github actions environment, or on your local Fedora 31+/CentOS 8 machine.

Installation/upgrade

On Fedora 31+ and CentOS 8, run as a non-root user

$ curl https://raw.githubusercontent.com/NethServer/nethserver-makerpms/master/install.sh | bash

There must be podman [1] already installed though.

Build requirements

Requirements for building RPMs, implemented by the buildimage/makesrpm command:

• The .spec file is in the current working directory
• Source tarballs defined in the .spec file can be downloaded via http:// or ftp://.
• If a SHA1SUM file exists in the current working directory, the integrity of source tarballs is checked against it. Do not put any path in the SHA1SUM contents, only the tarball/source file name.
• If the current working directory contains a .git directory (thus it is a git repository workdir) the specfile source0 can be set to Source: %{name}-%{version}.tar.gz. Then git-archive generates a tarball of the repository itself at the HEAD commit.
• Other tarballs are downloaded automatically with spectool during the build.

Building RPMs locally

If the previous build requirements are met, change directory to the repository root then run

$ makerpms *.spec

Run without any argument to get a brief help

$ makerpms

To build a package for another distribution pass the DIST environment variable

$ DIST=.el7 makerpms *.spec

If you have a custom or development builder image to test, set the IMAGE environment variable, e.g.:

$ IMAGE=me/myimage:test makerpms *.spec

It is possible to override the builder command, with the COMMAND environment variable

$ COMMAND="whoami" makerpms *.spec
makerpms

Additional arguments can be passed to YUM before starting the binary build, to fetch dependencies not tracked by BuildRequires, enable additional repositories and so on…

$ YUM_ARGS="--enablerepo=nethserver-testing" makerpms *.spec

To pass additional arguments to Podman, use PODMAN_ARGS instead

$ PODMAN_ARGS="--volume=$PWD/myrepo.conf:/etc/yum.repos.d/myrepo.conf"

Optimizations

To speed up the build process, the YUM cache directory contents are preserved. Container instances share the named Podman volume makerpms-yum-cache.

To clear the YUM cache run

$ podman volume rm makerpms-yum-cache

Builder container images

The builder container images are updated periodically and are available at https://hub.docker.com/r/nethserver/makerpms.

• nethserver/makerpms:7 is the default image, for noarch builds

• nethserver/makerpms:buildsys7 is the image for x86_64 builds (GCC 4)

• nethserver/makerpms:devtoolset7 is the image for x86_64 builds with GCC 9 (devtoolset-9 from SCLo), then run makerpms in a SCLo environment, e.g. :

  $ COMMAND="scl enable devtoolset-9 -- makerpms" makerpms *.spec
  

For more info about how to build the images locally look at travis/build-container.sh.

The following command locally builds the images with Podman (instead of Docker):

$ cat travis/build-container.sh  | sed s/docker/podman/  | \
grep -v -E '\b(login|logout|push)\b'  | \
NSVER=7 IMAGE_REPO=me/myimage bash

To test the local image run makerpms as follow:

$ IMAGE=me/myimage:7 makerpms *.spec

Images for NethServer 6 are available as well: just replace 7 with 6.

Other commands

The install.sh scripts installs also the following commands:

• uploadrpms is a RPM publishing helper, specific for the NethServer community RPMs publishing policies
• releasetag is a release workflow helper, specific for the NethServer community release guidelines
• makesrpm builds just the .src.rpm package.

uploadrpms

The first time, before running uploadrpms ensure the following command works

$ sftp username@packages.nethserver.org

Accept the server SSH key fingerprint when asked.

The following command uploads all the RPMs in the current working directory to the nethforge testing repository for NethServer version 7.8.2003.

$ uploadrpms username@packages.nethserver.org:nscom/7.8.2003/nethforge-testing *.rpm

Replace 7.8.2003 with the correct NS version number. Also replace nethforge-testing with the target repository name.

The command output might complain about some SFTP disabled commands. Ignore those messages.

releasetag

The releasetag command executes a workflow that suits only those NethServer packages that expect a Version tag in the form X.Y.Z.

Some RPMs, (notably nethserver-release) require a different version schema and releasetag does not suit their release workflow. Refer to their README files for more information.

When releasetag is invoked:

• reads the git log history and fetches related issues from the issue tracker web site.
• updates the Version tag and the %changelog section in the .spec file.
• commits changes to the .spec file.
• tags the commit (with optional GPG signature).

To fetch issues from private GitHub repositories create a private GitHub access token. Select the repo scope only.

Copy it to ~/.release_tag_token and keep its content secret:

chmod 600  ~/.release_tag_token

Tip

The private access token is useful also for public repositories because authenticated requests have an higher API rate limit

The releasetag command is now ready for use. This is the help output:

releasetag -h
Usage: releasetag [-h] [-k KEYID] [-T <x.y.z>] [<file>.spec]

Sample invocation:

releasetag -k ABCDABCD -T 1.8.5 nethserver-mail-server.spec

To force a local GPG password prompt (tested on Fedora) prepend some additional environment variables:

GPG_TTY=$(tty) GPG_AGENT_INFO="" releasetag  -k ABCDABCD -T 1.8.5 nethserver-mail-server.spec

Replace ABCDABCD with your signing GPG key. The $EDITOR program (or git core.editor) is opened automatically to adjust the commit message. The same text is used as tag annotation. Usage of -k option is optional.

To push the tagged release to GitHub (and possibly trigger an automated build) ensure to add the --follow-tags option to git push invocation. For instance:

git push --follow-tags

To make --follow-tags permanent run this command:

git config --global push.followTags true

Building RPMs on github.com

github.com automatically builds RPMs and uploads them to packages.nethserver.org, if configured with enough environment variables and upload secrets.

Configuration

To automate the RPM build process using GitHub

• create a .github/workflows/make-rpms.yml file inside the source code repository hosted on GitHub.
• the repository must have actions enabled and upload secrets properly set up, please contact the organization maintainer on community.nethserver.org for help.

This is an example of .github/workflows/make-rpms.yml contents:

name: Make RPMs
on:
  push:
    branches:
      - master
      - main
  pull_request:
jobs:
  make-rpm:
    runs-on: ubuntu-22.04
    env:
      dest_id: core
      docker_image: ghcr.io/nethserver/makerpms:7
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
          ref: ${{ github.head_ref }}
      - name: Generate .env file
        run: |
          cat > .env <<EOF
            DEST_ID=${{ env.dest_id }}
            NSVER=7
            DOCKER_IMAGE=${{ env.docker_image }}
            GITHUB_ACTIONS=1
            GITHUB_HEAD_REF=${{ github.head_ref }}
            GITHUB_REF=${{ github.ref }}
            GITHUB_REPOSITORY=${{ github.repository }}
            GITHUB_RUN_ID=${{ github.run_id }}
            ENDPOINTS_PACK=${{ secrets.endpoints_pack }}
            SECRET=${{ secrets.secret }}
            SECRET_URL=${{ secrets.secret_url }}
            AUTOBUILD_SECRET=${{ secrets.autobuild_secret }}
            AUTOBUILD_SECRET_URL=${{ secrets.autobuild_secret_url }}
          EOF
      - name: Run prep-sources if present.
        run: if test -f "prep-sources"; then ./prep-sources; fi
      - name: Build RPM and publish
        run: |
          echo "Starting build..."
          docker run --name makerpms \
            --env-file .env \
            --hostname $GITHUB_RUN_ID-$GITHUB_RUN_NUMBER.nethserver.org \
            --volume $PWD:/srv/makerpms/src:ro \
            ${{ env.docker_image }} \
            makerpms-github -s *.spec
          echo "Build succesful."
          if [[ "${{ secrets.endpoints_pack }}" && "${{ secrets.secret }}" ]]; then
            echo "Publish configuration exists, pushing package to repo."
            docker commit makerpms nethserver/build
            docker run \
              --env-file .env \
              nethserver/build \
              uploadrpms-github
            echo "Publish complete."
          fi
          rm .env

Usage

GitHub Actions builds are triggered automatically when:

• one or more commits are pushed to the master or main branch of the NethServer repository, as stated in the .github/workflows/make-rpms.yml file inside the on.push.branches key
• A pull request is opened from a NethServer repository fork or it is updated by submitting new commits

After a successful build, (if secrets are available) the RPM is uploaded to packages.nethserver.org, according to the DEST_ID variable value. Supported values are core for NethServer core packages, and forge for NethForge packages.

Pull requests are commented automatically by nethbot [4] with the links to available RPMs.

Also issues are commented by nethbot if the following rules are respected in git commit messages:

1. The issue reference (e.g. NethServer/dev#1234) is present in the merge commit of pull requests
2. The issue reference is added to standalone commits (should be rarely used)

The build environment supports the following variables:

• DOCKER_IMAGE
• DEST_ID

DOCKER_IMAGE

The Docker build image can contain different RPMs depending on the tag:

• latest (or 7) contains only dependencies to build nethserver-* RPMS, like nethserver-base. It actually installs only nethserver-devtools and a basic RPM build environment without gcc compiler.

• buildsys7 is based on the previous environment. It also pulls in the dependencies for arch-dependant packages (like asterisk13 or ns-samba). It actually installs the buildsys-build package group, which provides the gcc compiler (version 4) among other packages.

• devtoolset7: it extends the buildsys7 with the devtoolset-9 SCLo packages set. It is possible to compile with gcc version 9, by prefixing the container entry point in the following way:

  docker run -ti [OPTIONS] scl enable devtoolset-9 -- makerpms-travis package.spec
  

  For instance, see the sofia-sip package.

DEST_ID

If DEST_ID=core:

• Builds triggered by pull requests are uploaded to the autobuild [2] repository
• Builds triggered by commits pushed to master are uploaded to the testing [3] repository. If a git tag is on the last available commit, the upload destination is the updates repository.

If DEST_ID=forge:

• Pull requests are uploaded to nethforge-autobuild
• Branch builds are uploaded to nethforge-testing, whilst tagged builds are uploaded to nethforge

Warning

In any case, the git tag must begin with a digit and not containing any “-” minus symbol. For instance the tag 0.1.12 is considered as a tagged build whilst v0.1.12-1 is not

References

[1]	Podman is a daemonless Linux container engine. https://podman.io/

[2]	Is a particular kind of repository in packages.nethserver.org that hosts the rpms builded automatically from github.com/features/actions. http://packages.nethserver.org/nethserver/7.9.2009/autobuild/x86_64/Packages/

[3]	Is a repository in packages.nethserver.org that hosts the rpms builded automatically from github.com/features/actions started form official nethserver github repository. http://packages.nethserver.org/nethserver/7.9.2009/testing/x86_64/Packages/

[4]	Is our bot that comments the issues and pull request with the list of automated RPMs builds. https://github.com/nethbot

nethserver-mattermost

Stack:

• Mattermost
• PostgreSQL 12 listening on non-standard port 55434
• Apache as proxy server

Apache configuration derived from: https://github.com/mattermost/docs/issues/1114

First configuration

Mattermost requires a dedicated virtualhost and it’s accessibile only from HTTPS.

To start Mattermost, execute:

config setprop mattermost VirtualHost mattermost.yourdomain.com status enabled
signal-event nethserver-mattermost-update

Then, access https://mattermost.yourdomain.com and perform the first configuration.

Database

Properties:

• TCPPort: Mattermost listen port, change only for development purpose
• VirtualHost: dedicated FQDN for virtual host
• status: can be enabled or disabled, default to disabled
• access: firewall access, leave blank or at least set to none

Example:

mattermost=service
   TCPPort=5432
   VirtualHost=mattermost.local.neth.eu
   access=
   status=enabled

Account synchronization

The mattermost-bulk-user-create command will:

• create a default team named as the Company from OrganizationContact
• read all users from local or remote Account Providers and create them inside Mattermost

Please note that:

• users disabled in the Server Manager or already existing in Mattermost will be skipped
• a random password will be generated for each user

Forcing a common default password

It’s possible to set a default password for each new Mattermost user, just append the default password to command invocation.

Example:

mattermost-bulk-user-create Password,1234

Log files

The mattermost service logs are created under the /opt/mattermost/logs/ directory and are rotated by Mattermost itself.

Cockpit API

read

Return mattermost status and configuration. No input required.

Output

Example:

{
 "status": {
   "users": "2",
   "teams": "1",
   "channels": "2",
   "posts": "2"
 },
 "configuration": {
   "props": {
     "status": "enabled",
     "access": "",
     "VirtualHost": "mattermost.nethserver.org",
     "TCPPort": "5432"
   },
   "name": "mattermost",
   "type": "service"
 }
}

validate

Constraints:

• VirtualHost: must be a valid FQDN

Input

Example:

{
 "props": {
   "status": "enabled",
   "VirtualHost": "mattermost.nethserver.org",
 },
 "name": "mattermost",
}

update

Same input as validate.

nethserver-mock

The nethserver-mock package provides some scripts to ease the process of building and releasing RPMs.

Configuring the environment

On NethServer, install nethserver-mock package, by typing:

yum install nethserver-mock

On Fedora, and other RPM-based distros run the command:

yum install <URL>

Or

dnf install <URL>

where <URL> is http://packages.nethserver.org/nethserver/7.3.1611/base/x86_64/Packages/nethserver-mock-1.3.2-1.ns7.noarch.rpm at the time of writing. The build process uses Mock and must be run as a non privileged user, member of the mock system group. Add your user to the mock group:

usermod -a -G mock <username>

Running the scripts

The make-rpms command eases building of the NethServer RPMs by hiding the complexity of other commands. It is designed to work inside the git repository directory of NethServer packages, but should fit other environments, too.

Start by cloning the git repository and move inside it. For instance

git clone https://github.com/nethesis/nethserver-mail-server.git
cd nethserver-mail-server

To build the RPM just type

make-rpms nethserver-mail-server.spec

The command writes the results into the current directory, assuming every change to the source code has been commited. If everything goes well they consist of:

• source RPM
• binary/noarch RPMs
• mock log files

To clean up the git repository directory, git clean may help:

git clean -x -n

Substitute -n with -f to actually remove the files!

Note

The make-rpms command is sensible to dist and mockcfg environment variables. If they are missing the default values are shown by invoking it without arguments.

For example:

dist=ns7 mockcfg=nethserver-7-x86_64 make-rpms *.spec

The make-rpms command in turn relies on other scripts

make-srpm

Builds the .src.rpm file.

prep-sources

Extracts and/or fetches the source tarballs.

The first Source tag in the .spec file is assumed refer to the local git repository. If an absolute URL is specified, only the last part is considered. Other SourceN tags must conform to the Fedora RPM guidelines [1]. The external sources are actually fetched by the spectool command.

If the file SHA1SUM exists in the same directory of the .spec file the tarballs are checked against it.

Development and Release builds

During the development, a package can be rebuilt frequently: increasing build numbers and unique release identifiers are useful during this stage to help the whole process.

When make-rpms is invoked, it checks the git log history and tags to decide what kind of build is required: development or release.

Release builds produce a traditional RPM file name, i.e.:

nethserver-mail-server-1.8.4-1.ns6.noarch.rpm

Development builds produces a marked RPM, i.e:

nethserver-mail-server-1.8.3-1.6gite86697e.ns6.noarch.rpm

Other differences in development from release are

• the %changelog section in .spec is replaced by the git log history since the last tag
• the number of commits since the last tag, and the latest git commit hash are extracted from git describe and prepended to the %dist macro.

Signing RPMs

The command sign-rpms is a wrapper around rpm --resign command. Its advantage is it can read a password for the GPG signature from the filesystem. Sample invocation:

sign-rpms -f ~/.secret -k ABCDABCD

The signature is added automatically by packages.nethserver.org.

Running on EL8

It’s possible to use nethserver-mock also on EL8 (CentOS 8). If the build fails, please remove dnf-plugin-spacewalk RPM from the system.

References

[1]	Referencing Source http://fedoraproject.org/wiki/Packaging:SourceURL

nethserver-mysql

Manage and configure MySQL server.

When installed, the module will trigger the mysql_secure_installation script generating default configuration:

• Auto-generated root password saved in /var/lib/nethserver/secrets/mysql
• No anonymous user
• No remote root login allowed
• No test database

Differences from SME Server:

• InnoDB tables enabled, removed InnoDB property
• TCP connections enabled: can be disabled setting LocalNetworkingOnly property to yes
• Removed OpenFilesLimit property
• max_allowed_packet option is 1MB (as MySQL default). To change the value set the MaxAllowedPacket property

The root password is also saved in /root/.my.cnf, so the root user can use mysql client without typing any password.

Database example:

mysqld=service
   LocalNetworkingOnly=no
   MaxAllowedPacket=16MB
   TCPPort=3306
   status=enabled

Loading data

There is a special action /etc/e-smith/events/actions/nethserver-mysql-init which can be used to load data inside the database.

OBSOLETE: For backward compatibility with SME Server, the mysql.init service can also be used instead of the new action. The use of the service is HIGHLY discouraged.

When launched, the script will search for links in /etc/e-smith/sql/init directory. If a link point to a file with .sql extension, the content of the file will be loaded using the mysql command. Otherwise if the file is executable it will be simply executed. If no error occurs, the link will be removed.

Given ad simple /tmp/test.sql script, you can load it using this commands:

ln -s /tmp/test.sql /etc/e-smith/sql/init/temp.sql
service mysql.init start

nethserver-net-snmp

Simple SNMP server.

Package: nethserver-net-snmp.

Database

Configuration is saved in snmpd key inside configuration database.

Available properties:

• Community: name of the SNMP community, default is public
• SysLocation: name of the server location, default is Unknown
• SysContact: name and mail address of the system administrator, default is Root <root@localhost>.

Example:

snmpd=service
   Community=public
   SysLocation=Unknown
   SysContact=Root <root@localhost>
   status=enabled
   access=private
   UDPPort=61

Query example:

snmpwalk -mALL -v2c -cpublic localhost

Configuration can be applied using the nethserver-net-snmp-save event.

nethserver-netdata

Basic configuration for netdata. nethserver-cockpit package access netadata on http://localhost:19999 and uses the data to graph system statistics.

Do not change listening port otherwise Cockpit will not be able to access netdata.

Configuration DB

Properties:

• Bind: bind IP addess, default to localhost
• Alarms: can be enabled or disabled. Default to disabled to avoid unwanted mail to root user
• Plugins: a comma-separated list of key and value for standard plugins. Example: tc:yes,charts.d:yes
• PythonPlugins: a comma-separated list of key and value for python plugins. Example: apache:yes,sensors:no

Please be careful when changing Plugins and PythonPlugins properties, because some plugins are used by Cockpit to create charts inside multiple pages.

Database example:

netdata=service
   Alarms=disabled
   Bind=localhost
   TCPPort=19999
   access=
   Plugins=proc:yes,tc:yes,charts.d:yes,python.d:yes,idlejitter:no,cgroups:no,checks:no,apps:no,node.d:no
   PythonPlugins=apache:yes,sensors:yes
   status=enabled

Access from LAN

To enable the access from LAN, execute the following:

config setprop netdata Bind 0.0.0.0 access green
signal-event nethserver-netdata-update
signal-event firewall-adjust

Access netdata from http://<server_green_ip>:19999.

Links

• Offical netdata site: https://my-netdata.io/

nethserver-nextcloud

This package can be installed before or after any user provider like nethserver-dc and nethserver-directory.

If nethserver-dc or nethserver-directory are installed, the nethserver-nextcloud-save event will automatically enable all local users.

The package does the following:

• create nextcloud mysql database
• create default database credentials: user nextcloud and password stored in /var/lib/nethserver/secrets/nextcloud
• add trusted domains to use with web access
• create default credentials for web login: user “admin” and password “Nethesis,1234”
• set english as the default language
• set the user data directory as /var/lib/nethserver/nextcloud

The configuration is stored inside the configuration db, under the nextcloud key. To show it:

config show nextcloud

Properties:

• TrustedDomains: list of trusted domains added to Nextcloud config file
• VirtualHost: set custom virtual host, e.g. mycloud.mydomain.it
• Wellknown: can be enabled or disabled. If enabled, add redirects for calDAV and cardDAV. This property has effect only if VirtualHost is empty.
• HonorAdStartTls: can be enabled or disabled. The StartTLS option from SSSD configuration was historyically ignored. If set to enabled, the sssd/StartTls prop value is honored.

Admin user

After installation the application is accesible using the following credentials:

• User: admin
• Password: Nethesis,1234

Please, remember to change the default password after the first login!

Customize virtual host

Set custom virtual host and add it to trusted domains as follow:

config setprop nextcloud VirtualHost <VHOST>.<DOMAIN_NAME>
config setprop nextcloud TrustedDomains <VHOST>.<DOMAIN_NAME>
signal-event nethserver-nextcloud-update

Backup

The Nextcloud backup includes the configuration file and all data of the users:

/var/lib/nethserver/nextcloud
/usr/share/nextcloud/config/config.php

The database is automatically saved by nethserver-mysql.

OCC

When using occ command, PHP 7.3 should be enabled inside the environment.

Invocation example:

occ ldap:show-config"

The occ command is just a wrapper around:

su - apache -s /bin/bash -c "source /opt/remi/php80/enable; cd /usr/share/nextcloud/; php occ ldap:show-config"

Log of php80-fpm

The log of php80-fpm can be found at /opt/remi/php80/log/php-fpm/error-nextcloud.log

Cockpit API

read

Return mattermost status and configuration.

Input

• app-info or configuration

Output

Example (app-info):

{
 "url": "https://your.host.domain"
}

Example (configuration):

{
 "props": {
   "Wellknown": "enabled",
   "VirtualHost": "eccoci.rva.org",
   "TrustedDomains": "a.b,b.c"
 },
 "stats": {
   "version": "16.0.2.1",
   "admin_pass_warn": true,
   "users": "3"
 }
}

validate

Constraints:

• VirtualHost: must be a valid FQDN or EMPTY
• TrustedDomains: must be an array of valid FQDN

Input

Example:

{
 "props": {
   "Wellknown": "enabled",
   "VirtualHost": "eccoci.rva.org",
   "TrustedDomains": "a.b,b.c"
 }
}

update

Same input as validate.

Full reinstall

As with many other applications in NethServer, un-installing the Nextcloud application does not remove the settings, stored files, or the database. Here are the suggested steps to do a full un-install and re-install with a fresh configuration:

1. Uninstall Nextcloud using the admin page
2. Drop the MySQL database: /opt/rh/rh-mariadb105/root/bin/mysql --socket="/var/run/rh-mariadb105-mariadb/nextcloud-mysql.sock" -e "drop database nextcloud;"
3. Remove the packages: yum remove nethserver-nextcloud
4. Remove the whole Nextcloud directory: rm -rf /usr/share/nextcloud/
5. Remove the e-smith DB configuration: config delete nextcloud
6. Remove the NethServer config directory (WARNING: will remove user data): rm -rf /var/lib/nethserver/nextcloud
7. Install Nextcloud from the Software Center

nethserver-ns8-migration

This package will migrate services from NethServer 7 to NethServer 8 (NS8).

On install, the nethserver-ns8-migration-update event creates a WireGuard private key. This key is later used to join the NS8 cluster VPN.

The ns8-join command adds NS7 to the cluster VPN and retrieves the its configuration:

ns8-join [--no-tlsverify] <ns8_host> <admin_user> <admin_pass>

During ns8-join execution, the nethserver-ns8-migration-save event establishes the VPN connection.

Once the VPN is established:

• subsequent NS8 API calls are routed through it
• a temporary NS8 external user domain is created

Warning

The NS8 cluster must be new, with no user domains and installed modules

Example:

ns8-join ns8.nethserver.org admin Nethesis,1234

After the join, it is possible to execute actions:

ns8-action <agent> <action> <json_data>

Example:

ns8-action cluster get-cluster-status '{}'

When NS7 modules are successfully migrated to NS8, run the ns8-leave command.

ns8-leave

That command:

• ensures the temporary external user domain is removed
• removes the NS7 node from the VPN
• stops the Wireguard VPN and cleans up its settings

Commands

This is the full list of commands provided by the tool. They are used by applications and APIs:

• ns8-join
• ns8-leave
• ns8-action
• ns8-bind-app
• ns8-abort

Applications

The migration tool can migrate a pre-defined set of NS7 applications (apps):

• nethserver-mail (with nethserver-webtop5, nethserver-roundcubemail)
• nethserver-nextcloud
• nethserver-mattermost
• nethserver-ejabberd
• account-provider (both local AD and LDAP)

Each application has two directories

1. sources (or install) directory with the migration implementation, under /usr/share/nethesis/nethserver-ns8-migration/apps/.
2. state directory, with current migration state information, under /var/lib/nethserver/nethserver-ns8-migration/

The app sources directory must provide the following files, and implement the described behavior:

1. bind.env, basic environment variables for the bind command.
   • MODULE_IMAGE_URL the Podman image URL to install and start on the remote module
   • MODULE_VOLUMES a list of name of Podman volumes that are created by the bind command on the remote side. These volumes can be filled by rsync uploads, together with the remote module state/ directory.
2. bind command. This command creates a remote NS8 module instance and saves the Rsync configuration in the app state directory.
3. migrate command. This command synchronizes the local application configuration and data with the remote Rsync endpoint. If it is passed the MIGRATE_ACTION=finish environment variable, it also finalizes the migration: it stops the local services and starts the remote ones.

Optionally the app can provide jq input translation filters for the migration APIs:

• input-start.jq
• input-finish.jq

The filter output is Bash-eval’ed and must define additional environment variables for the app migrate command.

Applications can read additional environment files, provided by the RPM package or created at runtime by the tool commands:

/etc/nethserver/agent.env
/var/lib/nethserver/nethserver-ns8-migration/agent.env
/var/lib/nethserver/nethserver-ns8-migration/environment

Nextcloud migration

Bind command example

MODULE_NODE_ID=1 ./bind

Sync command example

./migrate

Finish command example

MIGRATE_ACTION=finish NEXTCLOUD_VHOST=nc.example.com ./migrate

Ejabberd migration

Bind command example

MODULE_NODE_ID=1 ./bind

Sync command example

./migrate

Finish command example

MIGRATE_ACTION=finish ./migrate

Mattermost migration

Bind command example

MODULE_NODE_ID=1 ./bind

Sync command example

./migrate

Finish command example

MIGRATE_ACTION=finish MATTERMOST_VHOST=mattermost.example.com ./migrate

Email, Webtop, Roundcube migration

As both Webtop and Roundcube depends on the Email application, the migration of the three modules must occur at the same time and is controlled by the nethserver-mail app.

Bind command example

MODULE_NODE_ID=1 WEBTOP_NODE_ID=1 ROUNDCUBE_NODE_ID=2 ./bind

Sync command example

./migrate

Finish command example

MIGRATE_ACTION=finish WEBTOP_VHOST=webtop.example.com ROUNDCUBE_VHOST=rc.example.com ./migrate

Just for environment var reference, to finalize nethserver-webtop5 alone

MIGRATE_ACTION=finish MAIL_INSTANCE_ID=mail1 WEBTOP_VHOST=webtop.example.com ./migrate

Finally, to finalize nethserver-roundcubemail alone

MIGRATE_ACTION=finish MAIL_INSTANCE_ID=mail1 ROUNDCUBE_VHOST=rc.example.com ./migrate

Account provider

This application migrates the local account provider. Both AD and LDAP are handled. External account provider is not migrated: it must be manually configured in NS8 to reach the same LDAP server used by NS7.

File server

The Samba file server migration is part of Samba Account provider migration. It occurs unless the skip flag is set for the nethserver-samba application.

Migration APIs

The API responsible for apps migration is api/migration/update. Its basic input payload format is

{
  "app": "nethserver-testapp",
  "action": "start",
  "migrationConfig": {
    "appNode": 3
  }
}

It accepts the following action values for each NS7 module: start, sync, finish, abort.

1. start. Creates one module instance in the NS8 cluster. The local NS7 app bind script is called. Multiple destination modules are allowed too: for instance the nethserver-mail app controls the migration of nethserver-webtop5 and nethserver-roundcubemail, if they are installed.
2. sync. Synchronizes local app configuration and data with the remote module instance, by calling its migrate script.
3. finish. Completes the migration by calling the app migrate script with the special environment variable MIGRATE_ACTION=finish.
4. abort. Abort module migration. Remove module from NS8 cluster and cleanup local stace.

After the execution of the finish action the app is stopped and disabled in NS7.

The API api/migration/read returns the current migration status, for each known app:

echo '{"action":"listApps"}' | /usr/libexec/nethserver/api/nethserver-ns8-migration/migration/read

Package uninstallation

To remove the tool and its dependencies (if they are not required by other packages):

yum remove nethserver-ns8-migration kmod-wireguard wireguard-tools

Clean up configuration database:

config delete wg-quick@wg0
config delete agent
config delete ns8

Post-migration step back

Once a service has been migrated to the remote NS8 host it should not run any more on NS7. When the migrate command completes the application services are stopped and disabled.

Please note that some migrated applications may also add some custom templates. To list such template fragments use:

grep -lR ns8migration /etc/e-smith/templates-custom/

It is possible to manually re-enable the services with the following commands.

# Mail
config setprop dovecot status enabled
config setprop postfix status enabled
config setprop rspamd status enabled
config setprop opendkim status enabled
config setprop olefy status enabled

# Webtop
config setprop tomcat8@webtop status enabled

# Roundcube
config delprop roundcubemail migration

# Mattermost
config setprop mattermost status enabled
rm -rf /etc/e-smith/templates-custom/etc/httpd/conf.d/zz_mattermost.conf
signal-event nethserver-mattermost-update

# Nextcloud
rm -rf /etc/e-smith/templates-custom/etc/httpd/conf.d/zz_nextcloud.conf
rm -f /etc/e-smith/templates-custom/etc/httpd/conf.d/default-virtualhost.inc/40nextcloud
signal-event nethserver-nextcloud-update

# Account provider
config setprop slapd status enabled
config setprop nsdc status enabled
config setprop sssd status enabled

# File server
config setprop smb status enabled
config setprop smb nmb enabled
config setprop smb winbind enabled

# All modules
signal-event nethserver-ns8-migration-update
signal-event runlevel-adjust
signal-event firewall-adjust

Migration notes

Warning

Read carefully the sections below before finishing the migration of any application.

Webtop

If you purchased a Webtop license for additional custom fields or other components/integrations the following additional and manual steps are needed:

1. Before finishing the Mail app migration, access the Webtop administrative page and disable any subscribed license.
2. Finish the Mail app migration.
3. In the NS8 module, access the administrative page and enable the licenses again.

File server

File server migration (shared folders) is an optional step of the Samba account provider migration. It can be performed if the NS8-based DC IP address is in a private network and is routable from the NS7-based DC.

The NS8 DC cannot be assigned the cluster VPN IP address.

Warning

Exposing SMB and other AD services to public networks is dangerous.

Account provider

When the migration finishes, the local account provider (both AD and LDAP) is stopped and disabled. In this state, SSSD allows logging on the system with its local cache. If some services are left on the system it is necessary to remove the local account provider and configure NS8 as the remote account provider.

nethserver-ntopng

After installation, ntopng is disabled by default.

Ntopng web interface is accessible:

• from specified TCP port (default is 3000 with access only from green interfaces)
• from Server Manager using a proxypass: https://<server>:980/<alias> (see alias prop)

The software is configured to use a minimal redis instance named redis-ntopng.

Database Reference

Properties:

• Authentication: can be enabled or disabled. Default is disabled
• Interfaces: comma-separeted list of monitored interfaces
• alias: auto-generated alias to be used as proxypass URL on httpd-admin

ntopng=service
   Authentication=disabled
   Interfaces=br0
   TCPPort=3000
   access=green
   alias=4de50f46b8d3e5ec829aba759431b48ad8173768
   status=enabled

Download lates upstream release

To download directly from stable repository:

wget  -r --no-parent -A n2n*rpm -A ndpi*rpm -A ntopng*rpm -A pfring*rpm http://packages.ntop.org/centos/7/x86_64/Packages/

To access the stable repository:

cd /etc/yum.repos.d/
wget http://packages.ntop.org/centos-stable/ntop.repo -O ntop.repo

See also http://packages.ntop.org/centos-stable/

Links

• Official site: http://www.ntop.org/products/ntop/

nethserver-ntp

Manage server date and time, ntp server for lan clients.

How it works

When the time and date are modified, some services must be restarted. Each package must subscribe service2adjust interface. Note that there’s no need to restart rsyslog on date and time changing. Rsyslog daemon is smart enough to read new time setting on the fly.

Date and time can be set manually or automatically via a ntp server.

The current time zone is accessible from the configuration database, but the value is read and written using systemd commands.

Events

• nethserver-ntp-update, standard event triggered on package installation/update.
• nethserver-ntp-save, standard event triggered on UI module SAVE action

Actions

• nethserver-ntp-localtime, create a lint to the zoneinfo file from /usr/share/zoneinfo to /etc/localtime according to TimeZone key in _configuration_ database.
• nethserver-ntp-clock-adjust, adjust the system date and hardware clock when NTP is disabled. Requires the event name and timestamp arguments. When NTP is enabled, restarts the ntp daemon.

Database

Example:

chronyd=service
   NTPServer=pool.ntp.org
   access=private

Daemon status

Run following command command:

chronyc sourcestats

Sample output:

210 Number of sources = 1
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
yourntp.server.com    41  20   43m     +0.001      0.172    +58ns   227us

nethserver-nut

The nethserver-nut package provides: * Network UPS Tool (NUT) configuration * web UI for simple configuration * optional collectd support

Configuration

The package provides two configuration modes:

• Master: a master node is a machine directly connected to the UPS port
• Slave: a slave node is a machine connected to the master via tcp

If you need stand-alone mode, just configure the machine as master.

Master node

A master node receives events from UPS, sends events to slaves and takes decisions (eg. power off the machine).

DB properties:

• Mode: master
• Device: UPS port, can be a device like /dev/ttyS0 or auto if the UPS is USB
• Model: UPS driver name selected from /usr/share/driver.list
• Description: UPS model description like “APC - Smart-UPS USB (**)” (used only on web UI)
• Password: chosen password to communicate between master and slaves
• Ups: ups name, default is UPS
• User: upsmon user name, default is upsmon
• Notify: if enabled notifications about UPS events are sent to the admin user, default is disabled

The password is generated during first installation from the nethserver-nut-conf action.

Slave node

A slave node receives events from a master and takes decisions (eg. power off the machine).

DB properties:

• Mode: slave
• Master: host name or ip address of master node
• Password: chosen password to communicate between master and slaves
• User: upsmon user name, default is upsmon

Collectd-nut

If server is in master mode, collectd is configured to monitor the local connected UPS.

Optional package required:

• collectd-nut
• nethserver-collectd

nethserver-openssh

OpenSSH is the premier connectivity tool for remote login with the SSH protocol. It encrypts all traffic to eliminate eavesdropping, connection hijacking, and other attacks. In addition, OpenSSH provides a large suite of secure tunneling capabilities, several authentication methods, and sophisticated configuration options.

The OpenSSH suite consists of the following tools:

• Remote operations are done using ssh, scp, and sftp.
• Key management with ssh-add, ssh-keysign, ssh-keyscan, and ssh-keygen.
• The service side consists of sshd, sftp-server, and ssh-agent.

Properties

the props of the service to configure openssh:

sshd=service
   AllowGroups=group1,group2:sftp,group3
   AllowLocalGroups=wheel,adm
   LoginGraceTime=2m
   MaxAuthTries=6
   PasswordAuthentication=yes
   PermitRootLogin=yes
   Protocol=2
   SubsystemSftp=yes
   TCPPort=22
   UsePAM=yes
   access=green,red
   status=enabled
   StrongEncryption=enabled

• AllowGroups: a comma separated list of groups from the accounts provider, allowed to connect with SSH. If the option :sftp is specified then the group is restricted to SFTP. This prop is considered if the property $sssd{'ShellOverrideStatus'} is enabled.
• AllowLocalGroups: a comma separated list of group names in /etc/groups that are granted full SSH and SFTP access.
• SubsystemSftp: (yes|no) enable the sftp service
• LoginGraceTime: The time after which the server disconnects if the user has not successfully logged in.
• MaxAuthTries: Specifies the maximum number of authentication attempts permitted per connection. Once the number of failures reaches half this value, additional failures are logged.
• PasswordAuthentication: (yes,no) Specifies whether password authentication is allowed.
• PermitRootLogin: (yes,no) Specifies whether root can log in using ssh.
• TCPPort: Use this TCP port to use sshd
• UsePAM: (yes,no) Pam can be used to authentify user
• access: Allow ssh connection to your firewall following different zones.
• status: (enabled,disabled) Enable or disable the sshd service
• StrongEncryption: (enabled,disabled) Enable or disable the strong encryption by requiring strong ciphers, HostKeyAlgorithms, KexAlgorithms, MACs. Old clients might not be able to connect to the server once enabled.

Events

Two events takes care to expand and restart the service

nethserver-openssh-update (sshd is restarted) nethserver-openssh-save (sshd is reloaded)

One time password (2FA)

2FA is available only for password authentication. Authentication with public keys will not be affected by 2FA.

The Openssh server can be protected by a Two-factor authentication (2FA). It adds an extra layer of security required to access the new Server Manager. First, users will enter user name and password, then they will be required to provide a temporary verification code generated by an application running on their smartphone.

2FA is disabled by default. Each user can enable it by accessing the Two-factor authentication section under Settings page.

nethserver-openvpn

Supported VPN are:

• Client to server (roadwarrior)
• Network to network (net2net)

Events

• openvpn-tunnel-create: fired when a new tunnel is created, takes the tunnel name as argument
• openvpn-tunnel-delete: fired when a new tunnel is deleted, takes the tunnel name as argument
• openvpn-tunnel-modify: fired when a new tunnel is modified, takes the tunnel name as argument
• nethserver-openvpn-save: fired when roadwarrior account or server is changed
• openvpn-tunnel-upload: used to transform a given file into a ready-to-use tunnel client

Roadwarrior accounts

Accounts are used to identify clients connecting to the server itself. There are two types of accounts:

• user account: system user with VPN access using user name and password
• vpn-only account: simple account with VPN access only

Each account can be used in a roadwarrior connection (host to net). If a net to net tunnel is needed, VPNRemoteNetwork and VPNRemoteNetmask properties must be set to inform the server about remote routes. When a new account is created, a certificate with same name is generated inside /var/lib/nethserver/certs directory.

Properties:

• VPNRemoteNetwork: remote network
• VPNRemoteNetmask: remote netmask
• OpenVpnIp: reserved IP for the client

Database reference

Database: vpn

<name>=vpn
   VPNRemoteNetwork=
   VPNRemoteNetmask=
   OpenVpnIp=

Roadwarrior server

Certificates

All certificates are signed using NethServer default RSA key (/etc/pki/tls/private/NSRV.key).

CA environment

CA configuration is stored inside /var/lib/nethserver/ directory, all certificates are stored inside /var/lib/nethserver/certs. The nethserver-openvpn-conf action creates:

• serial, certindex.attr and /certindex: database of valid and revoked certificates
• crlnumber and /etc/openvpn/certs/crl.pem: certificate revocation list
• dh1024.pem: key for TLS negotation

Certificate creation

Certificates in PEM format can be created using the command:

/usr/libexec/nethserver/pki-vpn-gencert <commonName>

The commonName parameter is an unique name stored inside the certificate. The command will generate key, crt and csr file. Each generated certificate is referred with a numeric id and saved inside certindex database. OpenSSL will also create a certificate named using the generated id (eg. 02.pem).

Certificate revocation

Certificate revocation is done via the command:

/usr/libexec/nethserver/pki-vpn-revoke [-d] <commonName>

The commonName parameter is an unique name stored inside the certificate. If ‘-d’ option is enabled, also delete crt, csr, pem and key files

Certificate renewal

All certificates will expire after X days, where X is the value of CertificateDuration property inside pki key. Renew is done via the command:

/usr/libexec/nethserver/pki-vpn-renew <commonName>

The commonName parameter is an unique name stored inside the certificate.

Client configuration is generated using /usr/libexec/nethserver/openvpn-local-client command. The file will contain the CA certificate inside the <ca>.

Example:

/usr/libexec/nethserver/openvpn-local-client myuser

The OpenVPN server listens on a management socket: /var/spool/openvpn/host-to-net. It’s possible to retrieve server status and execute commands using the socket.

Available scripts:

• /usr/libexec/nethserver/openvpn-status: retrieve status of connected clients and return result in JSON format
• /usr/libexec/nethserver/openvpn-kill: kill a connected client, exits 0 on success, 1 otherwise

Example with netcat:

>INFO:OpenVPN Management Interface Version 1 -- type 'help' for more info
status
OpenVPN CLIENT LIST
Updated,Thu Jan 23 09:22:24 2014
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
GLOBAL STATS
Max bcast/mcast queue length,0
END

See more on management options: http://openvpn.net/index.php/open-source/documentation/miscellaneous/79-management-interface.html

Configuration database

Properties:

• status: enable or disable the OpenVPN server, can be enabled or disabled, default is disabled
• AuthMode: authentication mode, can be password, certificate or password-certificate
• UDPPort: server listen port, default is 1194
• Mode: network mode, can be routed or bridged. Default is routed.
• ClientToClient: can be enabled or disabled, default is disabled. When enabled, traffic between VPN clients is allowed
• Compression: can be enabled, disabled or a valid compression alghoritm (see man openvpn). If set to enabled, adaptive LZO compression is used
• Remote: comma-separated list of IPs or host names, it’s used as multiple remote option inside client configuration generation script
• PushDns: if set, push the specified DNS as DHCP option
• PushDomain: if set, push the specified domain as DHCP option
• PushExtraRoutes: if disabled, only routes for green interface are pushed, if enabled also all static routes will be pushed. Default is enabled
• PushNbdd: if set, push the specified NBDD as DHCP option
• PushWins: if set, push the specified WINS as DHCP option
• CustomRoutes: a comma separated listed of CIDR to be pushed as extra routes to VPN clients
• Topology: set roadwarrior server topology, for valid values see man openvpn
• NotifyStatus: can be enabled or disabled, default is disabled. If enabled, send a mail notification to NotifyAddresses upon roadwarrior user connect or disconnect
• NotifyAddresses: comma-separated list of mail addresses, destinations for NotifyStatus property

If mode is bridged:

• BridgeEndIP: first client IP pool, must be inside the LAN range and outside DHCP range
• BridgeStartIP: last client IP pool, must be inside the LAN range and outside DHCP range
• BridgeName: name of the bridge, default is br0
• TapInterface: name of bridged tap interface, default is tap0

If mode is routed:

• Network: network of VPN clients, eg. 192.168.6.0
• Netmask: netmask of VPN clients, eg. 255.255.255.0
• RouteToVPN: can be enabled or disabled, default is disabled. When enabled, all traffic from client will be routed via VPN tunnel

Reference

Example:

openvpn@host-to-net=service
   AuthMode=password
   BridgeEndIP=192.168.1.122
   BridgeName=br0
   BridgeStartIP=192.168.1.121
   ClientToClient=disabled
   Mode=routed
   Netmask=255.255.255.0
   Network=192.168.6.0
   NotifyAddresses=root@localhost
   NotifyStatus=enabled
   PushDns=
   PushDomain=
   PushExtraRoutes=enabled
   PushNbdd=
   PushWins=
   Remote=
   RouteToVPN=disabled
   TapInterfaces=tap0
   Topology=subnet
   UDPPort=1194
   access=green,red
   status=enabled

Tunnel topology

Available topologies are subnet and p2p

If topology is p2p:

• only psk authentication mode is available
• options LocalPeerIP and RemotePeerIP must be set for both server and client
• options RemoteNetworks must be set if the client should be able to reach networks behind the tunnel server

If topology is subnet:

• only password, certificate or password-certificate authentication mode are available
• options Network must be set for the server

Tunnel clients

OpenVPN clients are used to connect to a tunnel server obtaining a net2net VPN.

Common properties:

• AuthMode: default value is certificate. Possible values:
  • certificate: use x509 certificate. Certificates, including CA and private key, are saved in /var/lib/nethserver/certs/clients directory in a PEM file named key.pem
  • password: use user name and password
  • password-certificate: use user name, password and a valid x509 certificate
  • psk: use a pre-shared key
• Cipher: a valid OpenVPN cipher among openvpn --show-ciphers
• Compression: can be enabled, disabled or a valid compression alghoritm (see man openvpn). If set to enabled, adaptive LZO compression is used
• LocalPeerIP and RemotePeerIP: IP for the Point to Point hosts
• Password: password used for authentication, if AuthMode is password or password-certificate
• Protocol: can be udp or tcp, default is udp
• RemoteHost: a list of remote server hostnames or ip addresses
• RemoteNetworks: list of networks in CIDR format, for each network will be created a route. This networks will also be used by the firewall library to calculate the zone of VPN hosts used inside the firewall rules.
• RemotePort: remote host port
• User: username used for authentication, if AuthMode is password or password-certificate
• WanPriorities: an ordered list of red interfaces which will be used to connect to the server, can be used to prefer a faster WAN over a slower one
• Topology: can be subnet (default) or p2p
• status: enable or disable the OpenVPN server, can be enabled or disabled, default is enabled

Files:

• pre-shared key: /var/lib/nethserver/certs/clients/<name>.key
• certificates: /var/lib/nethserver/certs/<name>.{pem,key}

Database reference

Database: vpn

c1=tunnel
   AuthMode=psk
   Cipher=
   Compression=disabled
   Crt=
   Mode=routed
   Protocol=udp
   Psk=
   RemoteHost=1.2.3.4,8.8.6.7
   RemotePort=1122
   Topology=subnet
   WanPriorities=eth2,eth1
   status=enabled

clntnh2d=tunnel
   AuthMode=certificate
   Cipher=
   Compression=enabled
   Mode=routed
   Protocol=udp
   RemoteHost=1.2.3.4,8.8.6.7
   RemotePort=1244
   Topology=subnet
   status=enabled

Tunnel servers

Servers are instance of OpenVPN listening for incoming connections. Each server runs on its own port and can handle many clients.

When a server is created the following files will be generated:

• a certificate and a private key: /var/lib/nethserver/openvpn-tunnels/<name>.crt and /var/lib/nethserver/openvpn-tunnels/keys/<name>.key
• a pre-shared key /var/lib/nethserver/openvpn-tunnels/<name>.key

Certificate generation uses a custom version of easy-rsa, see /usr/share/nethserver-openvpn directory content.

Properties:

• Cipher: a valid OpenVPN cipher among openvpn --show-ciphers
• Compression: can be enabled, disabled or a valid compression alghoritm (see man openvpn). If set to enabled, adaptive LZO compression is used
• LocalNetworks: list of networks in CIDR format, each network will be pushed as route to the client
• LocalPeerIP and RemotePeerIP: IP for the Point to Point hosts
• Network: network address of the VPN tunnel
• Port: listen port
• Protocol: can be udp or tcp, default is udp
• PublicAddresses: list of public IPs or host names used by clients to connect to the server
• RemoteNetworks: list of networks in CIDR format, for each network will be created a local route
• Topology: can be subnet (default) or p2p
• status: enable or disable the OpenVPN server, can be enabled or disabled, default is disabled

Database reference

Database: vpn

server1=openvpn-tunnel-server
   Cipher=
   Compression=enabled
   LocalNetworks=192.168.1.0/24
   RemotelNetworks=192.168.6.0/24
   Network=10.10.11.0/24
   Port=1282
   Protocol=udp
   PublicAddresses=1.2.3.4,test.local.neth.eu
   Topology=subnet
   status=enabled

psk1=openvpn-tunnel-server
   Cipher=
   Compression=enabled
   LocalNetworks=10.132.0.0/16
   LocalPeer=10.68.218.1
   Port=1248
   Protocol=udp
   PublicAddresses=1.2.3.4,myfw.host.org
   RemotePeer=10.68.218.2
   Topology=p2p
   status=disabled

Templates

The main templates for the configuration are:

• /etc/e-smith/templates/etc/openvpn/host-to-net.conf: allows the customization of roadwarrior server
• /etc/e-smith/templates/openvpn-tunnel-server/: allows the customization of tunnel server, it’s expanded in /etc/openvpn/<tunnel_server_name>.conf
• /etc/e-smith/templates/openvpn-tunnel-client/: allows the customization of tunnel client, it’s expanded in /etc/openvpn/<tunnel_client_name>.conf

Log files and systemd instances

All OpenVPN services are handled using systemd instances of the openvpn service.

Every instance has the following form: openvpn@<instance_name> where instance_name is name of the key inside the vpn e-smith db. The roadwarrior server has a fixed name: host-to-net.

Instances can be inspected using systemctl command:

systemctl status openvpn@host-to-net
systemctl status openvpn@server1

The roadwarrior logs can be found here:

• /var/log/openvpn/host-to-net-status.log
• /var/log/openvpn/openvpn.log

The log of each OpenVPN instance can be seen using journalctl command. Example:

journalctl -u openvpn@client1
journalctl -u openvpn@host-to-net

nethserver-phonenehome

This tool is used to tracks all NethServer’s installations around the world.

• Use API to get and set installation
• Visualize the installations in Google Maps with nice markers

Overview

This tool is composed by two parts: server and client and the data are stored and read through REST APIs. The REST APIs can be found in index.php file. To send data to REST APIs check out the phone-home file in root/etc/cron.weekly.

Configuration

After the repository clonig, you must set the correct placeholders in certain files.

Server

config.php

Line 2 change __dbhost__, __dbuser__, __dbpass__, __dbname__ with your own credentials:

$GLOBALS['$dbhost'] = "__dbhost__";
$GLOBALS['$dbuser'] = "__dbuser__";
$GLOBALS['$dbpass'] = "__dbpass__";
$GLOBALS['$dbname'] = "__dbname__";

widget_map.html

Line 53 change __serverip__ with the server ip used to host REST APIs:

// ip server with api
var server_ip = '__serverip__';

Create table in your database

After setting up your own credentials, simply run (MySQL syntax):

CREATE DATABASE phonehome;
GRANT ALL ON phonehome.* TO user IDENTIFIED BY 'password';

CREATE TABLE IF NOT EXISTS phone_home_tb (
 uuid                  VARCHAR(40) PRIMARY KEY,
 release_tag           VARCHAR(10) NOT NULL,
 ip                    VARCHAR(16) NOT NULL,
 country_code          VARCHAR(5),
 country_name          VARCHAR(40),
 country_location_lat  VARCHAR(40),
 country_location_lng  VARCHAR(40),
 reg_date              TIMESTAMP
)

Client

phone-home

Create a file named phone-home in /etc/sysconfig and set the correct infos:

SERVER_IP=__serverip__
PROXY_SERVER=__proxyserver__
PROXY_USER=__proxyuser__
PROXY_PASS=__proxypass__
PROXY_PORT=__proxyport__

the variables PROXY_SERVER, PROXY_USER, PROXY_PASS, PROXY_PORT are not mandatory.

nethserver-postgres

PostgreSQL is a database server and it listens on standard port 5432.

Access policy

From localhost:

• the postgres user can access without password
• all other users must use password authentication

From any other network:

• all users must use password authentication

Example for accessing with postgres user:

su - postgres
psql

Backup

Backup and restore is not implemented system-wide: every application should provide its own scripts.

Backup actions must be linked inside the pre-backup-data event. Example of backup action:

#!/bin/bash

su - postgres -c "pg_dump myapp > /var/lib/nethserver/myapp/myapp.sql"

Restore actions must be linked inside the post-restore-data event. Example of restore action:

#!/bin/bash

if [ -f /var/lib/nethserver/myapp/myapp.sql ]; then
  drop_sql=`mktemp`
  chown postgres:postgres $drop_sql
  # drop all existing connections to the db and block new ones
  echo "UPDATE pg_database SET datallowconn = 'false' WHERE datname = 'myapp';" >> $drop_sql
  echo "SELECT pg_terminate_backend(procpid) FROM pg_stat_activity WHERE datname = 'myapp';" >> $drop_sql
  # drop the db, then recreate it
  echo "DROP DATABASE myapp;" >> $drop_sql
  password=`perl -e "use NethServer::Password; print NethServer::Password::store('myapp');"`
  echo "CREATE database myapp; CREATE USER sonicle WITH PASSWORD '$password'; GRANT ALL PRIVILEGES ON DATABASE myapp to myuser;" >> $drop_sql
  # allow new connections to db
  echo "UPDATE pg_database SET datallowconn = 'true' WHERE datname = 'myapp';" >> $drop_sql
  su - postgres -c "psql < $drop_sql"
  su - postgres -c "psql myapp < /var/lib/nethserver/myapp/myapp.sql"
  rm -f $drop_sql
fi

nethserver-pulledpork

Pulledpork it’s a rule management tool for Suricata (and Snort). Actual configuration downloads only Emerging Threats.net Open rulesets.

After the download, multiple files are created inside /etc/suricata/rules directory.

Official site: https://github.com/shirkdog/pulledpork

nethserver-release

This RPM can be installed on a plain CentOS 7 minimal. It provides the nethserver-install command that downloads, installs and configures additional YUM repositories and RPMs for the NethServer system.

Building a release RPM

1. The Version tag is fixed to “7”. Increment the .spec file Release tag (e.g.: “12%{?dist}”)
2. Write the %changelog entry in the .spec file
3. Commit the above changes
4. Create a git tag like “7r12”. Do not use any “-” (minus) sign as separator!
5. Push the tag and the commit to start the automated build on Travis CI

Builds started from a tagged commit are published to “updates”!

More information: https://docs.nethserver.org/projects/nethserver-devel/en/v7/building_rpms.html

nethserver-rh-php56-php-fpm

Configure default installation of PHP 5.6 running on FPM. The default configuration of FPM is from upstream and can freely change (no template): /etc/opt/rh/rh-php56/php-fpm.d/www.conf.

Adding new configuration

If you need a new configuration, simply drop a file inside /etc/opt/rh/rh-php56/php-fpm.d/ directory and execute:

signal-event nethserver-rh-php56-php-fpm-update

nethserver-rh-php71-php-fpm

Configure default installation of PHP 7.1 running on FPM. The default configuration of FPM is from upstream and has been customized to listen on port 9001 (template: /etc/opt/rh/rh-php71/php-fpm.d/z_nethserver.conf).

Adding new configuration

If you need a new configuration, simply drop a file inside /etc/opt/rh/rh-php71/php-fpm.d/ directory and execute:

signal-event nethserver-rh-php71-php-fpm-update

nethserver-roundcubemail

Roundcube is a fast webmail client written in PHP.

Database

Configuration is saved in roundcubemail key inside configuration database.

Available properites:

• Server: server address of the mail server, default is localhost
• access: can be public or private, default is public
  • public: webmail can be accessed from any networks
  • private: webmail can be accessed only from green interfaces and trusted networks
• PluginsList: comma separated list of enabled plugins, default is managesieve,markasjunk. Before adding an option to this property, please be sure the plugin is already installed. A list of bundled plugins can be found inside file:/usr/share/roundcubemail/plugins directory.
• skin: default is larry other skins can be found in /usr/share/roundcubemail/skins

Example:

roundcubemail=configuration
   PluginsList=managesieve,markasjunk
   Server=localhost
   access=private
   skin=larry

Configuration can be applied using the nethserver-roundcubemail-update event.

nethserver-samba

File and print server for a MS-Windows network based on Samba.

Configuration database

Example:

smb=service
   ...
   NetbiosAliasList=
   DeadTime=10080
   WinsServerStatus=disabled
   WinsServerIP=
   UseCups=enabled
   UseClientDriver=yes
   HomeAdmStatus=disabled
   ShareAdmStatus=disabled
   InheritOwner=yes

• NetbiosAliasList See netbios aliases parameter in smb.conf(5) manpage.
• DeadTime (days) See deadtime parameter in smb.conf(5) manpage.
• WinsServerStatus if enabled act as a WINS server.
• WinsServerIP ipaddress if WinsServerStatus is disabled, nmbd will register with the given WINS server. See wins server, remote announce, remote browse sync parameters in smb.conf(5) manpage.
• UseCups {enabled,disabled} Use cups as printing server.
• UseClientDriver {yes,no} See use client driver parameter in smb.conf(5) manpage.
• HomeAdmStatus {enabled,disabled} Enable home$ hidden, administrative share
• ShareAdmStatus {enabled,disabled} Set admin users = "@domain admins" on every shared folder
• InheritOwner {yes,no} Mapped to inherit owner property of smb.conf

Accounts database

Only records with type ibay.

Properties:

• SmbStatus if enabled, activates ibay sharing through SMB protocol

• SmbProfileType select the profile template to apply to the share (optional).

  The template path must be placed into /etc/e-smith/templates/etc/smb.conf/ and prefixed by ibay-. Eg: default profile is located at /etc/e-smith/templates/etc/smb.conf/ibay-default.

• SmbRecycleBinStatus: enable or disable the recycle bin; when a file is deleted it is moved inside the recycle bin.
• SmbShareBrowseable: controls the visibility of the shared folder, default is enabled.

Example:

iba1=ibay
   AclRead=domadmins,admin
   AclWrite=domadmins,admin
   Description=test
   GroupAccess=rw
   OtherAccess=r
   OwningGroup=locals
   SmbGuestAccessType=none
   SmbRecycleBinStatus=disabled
   SmbShareBrowseable=enabled
   SmbStatus=enabled

Shared folder profile

Note

Shared folder profile is not related to “Roaming profiles”!

Ibays serve different purposes and smb.conf provides a lot of parameters to configure a Samba share. It’s difficult to find a combination of parameters that can fit all the possible requirements. Thus an ibay configuration adheres to a profile.

An ibay profile is a smb.conf sub-template that expands a cohesive set of share parameters. Each ibay has SmbProfileType prop that selects the template to apply to the ibay. The template path must be placed into /etc/e-smith/templates/etc/smb.conf/ and prefixed by ibay-. Eg: default profile is located at /etc/e-smith/templates/etc/smb.conf/ibay-default.

The default profile is applied if the given custom profile is not found.

nethserver-samba-audit

Samba audit is a simple audit module for samba which uses vfs_full_audit module. All operations are logged in a file and a logrotate job parses all entries and store it into a MySQL db. Logs are browseable using web interface.

SambaAudit is based on a Samb Audit project. See: http://sourceforge.net/projects/smbdaudit/ Current implementation uses standard full_audit vfs module instead of mysql_audit.

Configuration

The package configures Samba standard vfs audit to log write and read actions inside the /var/log/smbaudit.log file. Every night, a script called by logrotate parses the log file and puts all audit actions inside the smbaudit MySQL database. The database can be explored using Cockpit or the legacy PHP UI.

The packages adds the following properties to the smb key:

• AuditAlias: auto-generate alias to access the legacy UI
• AuditLogRead: can be enabled or disabled. If enabled read actions are stored in the database during the parsing, if disabled only write actions will be written to the database.

Example:

smb=service
  AuditAlias=43d5xxxxxxxxxxxxxxxxf023e46a11a4b7cb233a
  AuditLogRead=disabled
  DeadTime=10080
  HomeAdmStatus=disabled
  InheritOwner=no
  NetbiosAliasList=
  ShareAdmStatus=disabled
  TCPPorts=139,445
  UseClientDriver=yes
  UseCups=enabled
  WinsServerIP=
  access=green
  status=enabled

To enable the audit for a shared folder, use the SmbAuditStatus property to the ibay record. Example:

test=ibay
   ...
   SmbAuditStatus=enabled
   ...

nethserver-smartd

Configure smartd daemon to start at boot. smartd is a daemon that monitors the Self-Monitoring, Analysis and Reporting Technology (SMART)

Database:

smartd=service
  status=enabled

nethserver-squid

This package configure the well-known Squid web proxy.

Squid rpms are from upstream.

Configuration

All properties are saved in the squid key under the configuration database.

Properties:

• BlueMode: change Squid operation mode on blue networks. It has same values and defaults of GreenMode
• BypassDomains: comma separeted list of domains bypassed when the proxy is set in transparent mode
• DiskCache: disabled by default, if enabled it actives the disk caching system for squid
• DiskCacheSize: maximum value of squid cache on disk
• GreenMode: change Squid operation mode on green networks. Can be: manual, authenticated, transparent, transparent_ssl. Default is: manual
• KrbPrimaryList: name for Kerberos keytab (used for Active Directory integration)
• KrbStatus: if set to enabled a ticket credential cache file is kept valid by the hourly cron job (used for Active Directory integration)
• MaxObjSize: objects larger than this setting will not be saved on disk. If speed is more desirable than saving bandwidth, this should be set to a low value
• MemCacheSize: value of squid cache on memory
• MinObjSize: can be left at 0 to cache everything, but may be raised if small objects are not desired in the cache.
• NoCache: comma separated list of domains which will be not cached
• ParentProxy: in the form host:port, if omitted port is default to 3128. Default is empty
• PortBlock: if enabled, block port 80 and 443. Default is: disabled
• SafePorts: comma separated list of ports thath can be accessed through the proxy. Listed ports will be added to the default list of safe and ssl ports

Database example

Example:

squid=service
   BlueMode=manual
   DiskCache=disabled
   DiskCacheSize=100
   GreenMode=transparent
   KrbPrimaryList=HTTP
   KrbStatus=enabled
   MaxObjSize=4096
   MemCacheSize=256
   MinObjSize=1
   NoCache=www.nethserver.org
   ParentProxy=
   PortBlock=disabled
   TCPPorts=3128,3129,3130
   access=private
   status=enabled

Transparent mode

When the proxy is enabled in transparent mode, all traffic destined to the port 80 is redirect to the Squid (port 3129). This configuration requires Shorewall.

SSL peek and splice

If the proxy is enabled in transparent SSL mode, also all traffic destined to port 443 is redirected to Squid (port 3130). The daemon does not inspect SSL traffic, but visited sites can be processed using the web filter.

Known bugs

You could find this kind of errors inside /var/log/squid/cache.log

2016/12/09 09:44:18 kid1| SECURITY ALERT: on URL: avatars0.githubusercontent.com:443
2016/12/09 09:44:18 kid1| SECURITY ALERT: Host header forgery detected on local=151.101.60.133:443 remote=192.168.5.22:40950 FD 166 flags=33 (local IP does not match any domain IP)

In this case, when accessing github, the avatars won’t be displayed by the browser, and you can find a “Timeout error” for the not loaded images.

This kind of errors can’t be fixed. See official documentation for workarounds:

• http://wiki.squid-cache.org/KnowledgeBase/HostHeaderForgery2
• http://lists.squid-cache.org/pipermail/squid-users/2016-September/012344.html

Authenticated mode

Authentication schema depends on system configuration:

• standard authentication for system users is done over LDAP
• if Samba AD is installed, clients can use Kerberos (SPNEGO/GSSAPI)

Bypasses

The implementation supports 3 kind of bypass:

• source bypass
• destination bypass
• domains bypass

Source and destination bypass

Bypass rules are saved inside the fwrules databases. A bypass can be of two types:

• bypass-src: listed origin host are bypassed
• bypass-dst: listed target host are bypassed

Properties:

• Host: a host object, like a remote or local host
• status: can be enabled or disabled
• Description: optional description

Bypass example:

boss=bypass-src
   Description=Boss without proxy
   Host=host;bosspc
   status=enabled

Domains bypass

All requests to domains listed inside the BypassDomains property will not be redirect to the transparent proxy.

The implementation uses the ipset feature of Dnsmasq. Each time a listed domain is accessed from the client, Dnsmasq resolves the IP and add it to squid-bypass ipset. The squid-bypass ipset is then used as exception inside Shorewall REDIRECT rule.

Notes:

• all clients must use the server as DNS
• Dnsmasq name resolution works for the listed domains and all sub-domains

Cache

There is an event called nethserver-squid-clear-cache that empties the cache.

Priority and divert rules

The squid database contains multiple records of type rule.

Each rule has following properties:

• key: it’s a numeric uniq id
• Action:
  • priority;low: add low priority packet marker using tcp_outgoing_mark
  • priority;high: add high priority packet marker using tcp_outgoing_mark
  • provider;<name>: add packet marker for provider <name> using tcp_outgoing_mark
  • force;<name>: force output traffic to <name> provider using tcp_outgoing_address
• Description: optional description
• Dst: comma-separeted list of domains, this is converted to a dstdomain ACL
• Src: firewall object, supported objects are: role, host, zone, ip range and cidr . This is converted to src ACL
• status: can be enabled or disabled

Example:

1=rule
   Action=priority;low
   Description=
   Dst=yahoo.com
   Src=host;giacomo
   status=enabled
2=rule
   Action=provider;fast
   Description=
   Dst=nethserver.org,nethesis.it
   Src=host;birro
   status=enabled
3=rule
   Action=force;slow
   Description=
   Dst=
   Src=cidr;cidr1
   status=enabled

WPAD

WPAD is located at /var/www/html/wpad.dat. The web server is configured to allow the download only from trusted and blue networks, but be aware that you need to manually open the httpd port for blue networks (see Add a new service).

The WPAD returns:

• DIRECT, if squid is disabled or the requesting client is inside a network where the proxy is configured in transparent mode
• IP of corresponding network interface, if the requesting client is inside a network where the proxy is configured in manual or authenticated mode
• proxy.<domain>, if the server is joined to Active Directory and the requesting client is inside a network where the proxy is configured in manual or authenticated mode

Also WPAD file includes all source and destination bypasses.

Miscellaneous options

The following options are always enabled:

• buffered logs
• SNMP support on port 3401

nethserver-squidguard

The package configures ufdbGuard, a URL filter for squid. The configuration is based on profiles. Each profile is composed by:

• a user, group of users, host or group of hosts accessing a web page
• a filter which includes allowed and denied sites
• a time frame in which the filter is active

The system comes with a default profile which is applied to any host/user who doesn’t fit on a specific profile.

Features of ufdbGuard:

• 3-4 times faster than squidGuard
• a multithreaded daemon with one copy of the database in memory
• detects HTTPS proxy tunnels
• detects SSH-based tunnels
• blocks HTTPS for URLs without FQDN
• blocks HTTPS for sites without a properly signed SSL certificate
• uses in-memory databases
• enforce the SafeSearch feature on Google and other search engines
• a test mode (-T option) allows you to test a URL filter database without actually blocking sites

For the block page on th server, see https://github.com/NethServer/blocked.nethserver.org/.

Inner workings

For each request, squid sends the URL to one ufdbgclient redirector (which runs as user squid), which in turn asks to the ufdb daemon (which runs as user ufdb).

Known limitations

• Transparent URL filtering on HTTPS websites can only block whole domains, because ufdbGuard can only receive

the domain name, not the full URL

• Redirected HTTPS show an error instead of the block page

Blacklists

Blacklists are updated every night using the script: /etc/cron.daily/update-squidguard-blacklists The script will download and merge all blacklists listed in /etc/squid/blacklists. This actions can take several minutes.

Databases

The package uses the squidguard key inside the configuration database, also it creates a new contentfilter database for profiles and filters configuration.

configuration

The key squidguard contains all settings. Properties are:

• BlockedFileTypes: comma separated list of blocked file extensions
• CustomListURL: URL to download a custom blacklist. The blacklist must follow SquidGuard standard
• DomainBlacklist: comma separated domain list, this domains are always blocked
• DomainWhitelist: comma separated domain list, this domains are always allowed
• Expressions: if enabled, allow regular expression on blacklists categories. Can be enabled or disabled, default is disabled
• IdleChildren: minimum number of idle processes. Default is 5
• Lists: comma separated list of blacklist names. Possible values are: toulouse and custom. If set to custom, make sure CustomListURL is set.
• MaxChildren: maximum number of processes. Default is 20
• RedirectUrl: custom URL for block page. See Block page
• StartupChildren: minimum number of process children on startup. Default is 5

Note

Modifying following parameters can greatly affect memory usage: IdleChildren, MaxChildren, StartupChildren

Example:

squidguard=configuration
   BlockedFileTypes=
   CustomListURL=
   DomainBlacklist=microsoft.com
   DomainWhitelist=nethserver.org,nethesis.it
   Expressions=disabled
   IdleChildren=5
   Lists=toulouse
   MaxChildren=20
   RedirectUrl=
   StartupChildren=5
   UrlBlacklist=
   UrlWhitelist=
   status=enabled

The service key ufdbGuard has enabled/disabled state according to the status prop of the squidguard (see nethserver-squidguard-ufdb-status action).

contentfilter

The contentfilter database can contain three kind of records:

• category: a custom categorized list of domains blocked or allowed. Custom categories can be added to a filter
• filter: an object describing which categories must be blocked or allowed
• time: when the filter must be applied, it contains week days and time
• profile: a link between the above objects describing WHO (host or user), WHAT (filter) and WHEN (time)

Categories

Properties:

• Domains: comma separated list of domains
• Description: optional description

Category example:

mycategory=category
   Description=My Category
   Domains=nethesis.it,nethserver.org

Filters

Properties:

• BlackList: enable or disable the global blacklist (DomainBlacklist). Can be enabled or disabled
• BlockAll: can be enabled or disabled. If disabled, all categories listed in Categories are blocked and all other sites are allowed. If enabled, all listed categories in Categories are allowed and all other sites are blocked
• BlockFileTypes: enable or disable the global file extension list (BlockedFileTypes). Can be enabled or disabled
• BlockIpAccess: if enabled, sites can be accessed only using a domain name (not an IP address). Can be enabled or disabled
• Categories: comma separated list of categories blocked or allowed. If a category is not present inside the SquidGuard db (/var/squidGuard/Blacklists), the category will be excluded from configuration file to avoid ufdbGuard panic-mode (no filter)
• Description: optional description
• WhiteList: enable or disable the global whitelist (DomainWhitelist. Can be enabled or disabled
• Removable: can be yes or no. If set to no the record can’t be removed from web interface

Filter example:

myfilter=filter
   BlackList=enabled
   BlockAll=disabled
   BlockFileTypes=disabled
   BlockIpAccess=disabled
   Categories=aggressive,alcohol,weapons,warez
   Description=Default filter
   WhiteList=enabled

Times

Properties:

• Days: comma separated list of week days. Valid values are:
  • m: Monday
  • t: Tuesday
  • w: Wednesday
  • h: Thursday
  • f: Friday
  • a: Saturday
  • s: Sunday
• Description: optional description
• EndTime: hour of the day in 24h format or empty
• StartTime: our of the day in 24h format or empty

Time example:

worktime=time
   Days=m,t,w,h,f
   Description=Work time
   EndTime=18:30
   StartTime=08:30

Profiles

Properties:

• Filter: a filter object
• *FilterElse: an optional filter object, applied when none of the referenced Time rules apply
• Src: it can be an object of type user, user group, host, host group, zone or role. Otherwise, if it is a string, the system will assume the profile is associated with an user from Active Directory; the system must be joined to a domain
• Time: a CSV list time object references (optional)
• Description: optional description
• Removable: can be yes or no. If set to no the record can’t be removed from web interface

Profile example:

myprofile=profile
   Description=My profile
   Filter=filter;badboys
   FilterElse=filter;enjoy
   Src=host;demo
   Time=time;worktime-am,time;worktime-pm

Block page

The block page is a CGI used to inform the user about the block reason. It’s a single page which can handle requests from ufdbGuard and SquidClamav (squidclamav-section).

The page is localized depending on browser language.

This configuration can be overwritten using RedirectUrl property.

Troubleshooting

Useful test commands:

echo "http://bit.ly 10.10.0.1/ - - GET" | /usr/sbin/ufdbgclient -d
echo "http://bit.ly 10.10.0.1/ user@mydomain.com - GET" | /usr/sbin/ufdbgclient -d

Logfiles:

/var/log/ufdbguard/ufdbguardd.log
/var/log/ufdbguard/ufdbgclient.log

nethserver-sssd

This package implements authentication and user management layers. Supported authentication providers are:

• OpenLDAP with POSIX attributes
• Samba or Windows Active Directory

It includes the following parts:

• SSSD configuration
• events for users and groups management
• web interface for user management
• password policy management
• system validators for users and groups
• SSSD perl library to ease the implementation of e-smith templates

The implementation can work in two modes:

• read-and-write: if nethserver-dc or nethserver-directory are installed, the system will provide all user management features like creation, modification and deletion of users and groups
• read-only: if users and groups are read from a remote source, the system will be able to consume them only using passwd database

Configuration DB format

sssd=service
    AdDns=192.168.1.12
    BindDN=ldapservice@AD.EXAMPLE.COM
    BindPassword=cjnsdkuyf8934tjhvsdkljcvsdv
    LdapURI=
    Provider=ad
    Realm=AD.EXAMPLE.COM
    Workgroup=EXAMPLE
    DiscoverDcType=dns
    ShellOverrideStatus=disabled
    status=enabled

• AdDns: IP address of the Active Directory DNS server
• LdapURI: LDAP server URI (use ldap:// or ldaps:// scheme)
• DiscoverDcType {dns,ldapuri}: dns query SRV records in AD DNS to find the DC name; ldapuri retrieve the DC name from the LdapURI prop value

• ShellOverrideStatus (default disabled) If enabled, the value of the passwd user’s shell field

  is set to /bin/bash and is no more read from the accounts provider. The event nethserver-sssd-save will expand and restart all services which use this property

Realm and workgroup

When the system is configured to use an Active Directory provider (Provider=ad), make sure to correctly set both Realm and Workgroup properties:

• Realm: this is the Kerberos realm and it’s case sensitive, but it’s usually configured in upper case as best practice. When the realm is used for DNS queries, it’s automatically forced to lower case.
• Workgroup: Samba NetBIOS name, maximum length is 15 characters. It’s usually the first part of the Realm in upper case

Events

Defined events are:

user-create

The event creates the user record inside the account provider database.

Parameters:

• username: must be unique
• name: full name of the user
• shell: default to /usr/libexec/openssh/sftp-server, if set to /bin/bash the user will be able to access the server using SSH

user-modify

The event changes the full name inside the account provider databases

Parameters:

• username
• name: full name of the user
• shell: default to /usr/libexec/openssh/sftp-server, if set to /bin/bash the user will be able to access the server using SSH

Note: shell option can’t be changed for AD users

user-delete

The event deletes the user and remove it from all groups.

Parameters:

• username

user-cleanup

This event deletes all data (home directory) and the emails of the user (given as parameter) if the email server is installed. This event is bind to the event user-delete. Every application which needs to remove the data of a user, must use this event to trigger a specific action, made to remove the data.

Parameters:

• username

user-lock

The event locks the user preventing the access. All new users are in locked state.

Parameters:

• username

user-unlock

The event unlocks the user preventing the access. This event should be called after the invoking password-modify event for the user.

Parameters:

• username

group-create

The event creates the group record inside the account provider database.

Parameters:

• groupname: must be unique
• members: a list of users member of this group

group-modify

The event changes the members of a group inside the account provider database.

Parameters:

• groupname: must be unique
• members: a list of users member of this group

group-delete

This event deletes a group record from the the account provider database.

Parameters:

• groupname

password-policy-update

This event configures password expiration of a single user or of all users.

Parameters

• username (optional)

• passexpires: it can be yes or no. If user is set and value is yes, the user password will expires after a predefined number of days (see passwordstrength{MaxPassAge})

  The duration of a password can be passwordstrength{MaxPassAge}

nethserver-sssd-remove-provider

This event removes any installed local account provider and also wipes the configuration of any remote account provider.

Before resetting the configuration, all accounts are dumped inside the following files in tsv format:

• /var/lib/nethserver/backup/users.tsv
• /var/lib/nethserver/backup/accounts.tsv

System users and groups

SSSD can access all users and groups from an account provider, but the Server Manager hides system users and groups.

The following users will not be accessible from the Server Manager:

• all users listed inside /etc/nethserver/system-users
• all users in /etc/passwd

The following groups will not be accessible from the web interface:

• all groups listed inside /etc/nethserver/system-groups
• all groups in /etc/group

The users and groups lists are retrieved by the following UI helpers:

• /usr/libexec/nethserver/list-users
• /usr/libexec/nethserver/list-groups

The number of entries returned by the server is limited. For instance, AD has a 1000 entries search results cap.

To retrieve the members of a group and the membership of a specific user:

• /usr/libexec/nethserver/list-group-members
• /usr/libexec/nethserver/list-user-membership

The Dashboard account counters are provided by:

• /usr/libexec/nethserver/count-accounts

All those helpers support the -A flag, to include hidden entries, and the -s flag to return entries without @domain suffix.

AD LDAP search

The Samba net ads search -k command can run an LDAP search against the AD LDAP servers. The command requires a valid Kerberos ticket and a configured environment variable, KRB5CCNAME, pointing to it.

The krb5exec command can set up the Kerberos ticket by authenticating with the machine credentials, providing the same environment where the UI helpers run, as explained in the previous section.

Putting the two commands together, the following command retrieves the admin account record from AD LDAP.

krb5exec net ads search -k sAMAccountName=admin

The same command with ldapsearch

krb5exec ldapsearch -Y GSSAPI -b <BIND_PATH> -h <LDAP_SERVER_NAME> sAMAccountName=admin

Replace <BIND_PATH> and <LDAP_SERVER_NAME> with values provided by

net ads info

NethServer::SSSD

NethServer::SSSD is the Perl library module to retrieve current LDAP configuration. It supports both Active Directory and OpenLDAP providers.

Template example:

{
    use NethServer::SSSD;
    my $sssd = NethServer::SSSD->new();

    $OUT .= "{ldap_uri, [".$sssd->ldapURI()."]}\n";

    if ($sssd->isAD()) {
        $OUT .= "{ldap_uids, [\"sAMAccountName\"]}.\n";
    }

}

All functions are documented using perldoc

perldoc NethServer::SSSD

This command prints out the current settings, by querying NethServer::SSSD methods. It requires the package openldap-clients

/usr/sbin/account-provider-test dump

Check the bind credentials are OK

/usr/sbin/account-provider-test

Join Active Directory

The Active Directory join operation is run by realmd. After the AD has been joined sucessfully the system keytab file is initialized as long as individual service keytabs, as defined on the respective service record (see `Service configuration hooks`_).

Leave and Re-Join Active Directory

To leave a remote AD go to the Accounts provider page.

For local AD provider, this is the manual leave procedure

realm leave
realm leave # two times

If the machine password or system keytab get corrupted, joining again the DC can fix them:

realm join -U admin $(config getprop sssd Realm)

…at prompt, type the admin’s password, then:

signal-event nethserver-sssd-save

If you leave and do not want to re-join, disable the sssd service permanently:

config setprop sssd status disabled Provider none
signal-event nethserver-sssd-save
signal-event nethserver-sssd-leave
signal-event nethserver-dnsmasq-save

Change the FQDN

Once we are bound to an account provider the FQDN cannot be changed any more. However, this procedure can be useful in early server configuration to fix a wrong FQDN. Please note that any existing account setting must be fixed manually. The procedure to do it is currently undefined.

For local account providers:

1. Execute the leave procedure explained above
2. Go to page System name and change the domain suffix in the FQDN field.
3. Re-join as explained above

For remote account providers the procedure is similar. Use the Accounts provider page to leave/join the domain.

Service configuration hooks for Kerberos/GSSAPI authentication

A service (i.e. dovecot, squid) record in the configuration DB can be extended with the following special props, that are read by smbads to create a Kerberos keytab file for the service

dovecot=service
   ...
   KrbStatus=enabled
   KrbKeytabPath=/var/lib/dovecot/krb5.keytab
   KrbPrimaryList=smtp,imap,pop
   KrbKeytabOwner=
   KrbKeytabPerms=

• KrbStatus {enabled,disabled} This is the main switch. If not set to enabled, smbads ignores the record when the service keytab is written.
• KrbKeytabPath Keytab file path. If empty, /var/lib/misc/nsrv-<service>.keytab is assumed
• KrbPrimaryList <comma separated words list> Defines the keytab contents. In Kerberos jargon a “primary” is the first part of the principal string, before the slash (/) character. Any primary in this list is exported to the keytab.
• KrbKeytabOwner The unix file owner. Default is the service name. This is applied to both the credentials cache file and the keytab file.
• KrbKeytabPerms The unix bit permissions in octal form. Default is 0400. This is applied to both the credentials cache file and the keytab file.

The implementation is provided by /usr/libexec/nethserver/smbads.

Individual services can link themselves to nethserver-sssd-initkeytabs action in the respective -update event.

It is up to the KDC administrator to configure the SPN in the KDC. For instance, with Active Directory run the following commands to add the SPNs for Dovecot:

kinit admin@$(config getprop sssd Realm)
net ads setspn list
net ads setspn add $(hostname -s) imap/$(hostname)
net ads setspn add $(hostname -s) pop/$(hostname)
net ads setspn add $(hostname -s) smtp/$(hostname)
kdestroy
signal-event nethserver-sssd-save

Account import scripts

There are some perl scripts under the documentation scripts/ directory.

rpm -qd nethserver-sssd

import_users

It is possible to create user accounts from a TSV (Tab Separated Values) file with the following format:

username <TAB> fullName <TAB> password <NEWLINE>

Sample invocation:

import_users users.tsv

Alternative separator character:

import_users users.csv ','

import_groups

It is possible to create groups from a TSV (Tab Separated Values) file with the following format:

groupname <TAB> member1 <TAB> ... <TAB> memberN <NEWLINE>

Sample invocation:

import_users groups.tsv

Alternative separator character:

import_groups groups.csv ','

import_emails

It is possible to create mail aliases from a TSV (Tab Separated Values) file with the following format:

username <TAB> emailaddress <NEWLINE>

See import_users section for a sample script invocation.

Password policy

The system can handle global or per-user policies. All policies are enforced by PAM and saved under passwordstrength inside the configuration database.

Available properties are:

• Users: change strength password for all users, can be:

  • strong: (default) strong passwords must conform to cracklib checks
  • none: no strength check

• PassExpires: can be yes (default) or no. If set to no password will not expire, if set to yes,

  following properties apply:

  • MaxPassAge: minimum number of days for which the user is forced to keep the same password (default 0)
  • MinPassAge: maximum number of days for which the user can keep the same password (default: 180)
  • PassWarning: a shell warning is displayed to the user X days before password expiration

Configuration can be applied using the password-policy-update event.

DB example:

passwordstrength=configuration
   MaxPassAge=180
   MinPassAge=0
   PassExpires=no
   PassWarning=7
   Users=none

nethserver-subscription

Manage NethServer Subscription clients

• Monitor the machine using Collectd and route notifications to a remote server
• Collect and send server informations to a remote server
• Download and install RPM updates automatically by configuring YUM repositories and nethserver-yum-cron

Database

Configuration is stored inside the configuration database under the subscription key.

Properties:

• AlertsUrl: URL used to send alerts and heartbeat
• AlertsAutoUpdates: if set to enabled, custom alerts will be downloaded each night from AlertsAutoUpdatesUrl.
• AlertsAutoUpdatesUrl: URL used to fetch custom alerts database
• ApiUrl: URL called from the UI to retrieve subscription plan informations
• InventoryUrl: API endpoint where the data are sent
• NsRelease: exact NethServer release request to mirrorlist
• Secret: sysem secret key generated by Dartagnan (https://github.com/nethesis/dartagnan)
• PricingUrl: URL used from the UI to access the subscription plans
• SystemId: system unique id
• SiteUrl: Product website URL
• DocsUrl: Product documentation URL
• HelpdeskUrl: Product helpdesk URL
• ExtraRepositories: Enable extra repositories in the subscription (comma separated : nethforge,repoA,repoB).

Example:

subscription=configuration
   AlertsAutoUpdates=enabled
   AlertsUrl=https://my.nethserver.com/api/
   ApiUrl=http://my.nethserver.com/api/
   InventoryUrl=http://my.nethserver.com/api/inventories/store/
   NsRelease=7.4.1708
   Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
   PricingUrl=https://my.nethserver.com/?action=newServer
   SystemId=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
   SiteUrl=https://www.nethserver.org/
   DocsUrl=http://docs.nethserver.org/
   HelpdeskUrl=https://community.nethserver.org/

Configuration

• NethServer Subscriptions (by Nethesis) – run

  config setprop subscription AlertsUrl https://my.nethserver.com/api/machine/
  config delprop subscription AlertsAutoUpdatesUrl
  config setprop subscription InventoryUrl https://my.nethserver.com/api/machine/inventories/store/
  signal-event nethserver-subscription-update
  

• NethServer Enterprise partner program (by Nethesis) – run

  config setprop subscription AlertsUrl https://my.nethesis.it/isa/
  config setprop subscription AlertsAutoUpdatesUrl https://my.nethesis.it/api/
  config setprop subscription InventoryUrl https://my.nethesis.it/isa/inventory/store/
  signal-event nethserver-subscription-update
  

YUM repositories

Once NethServer Subscriptions is activated, YUM configuration is changed to contact the subscribed repositories. Every night the list of enabled repositores will be reset to meet the subscription requirements.

The /etc/e-smith/events/actions/nethserver-subscription-eorepo action configures YUM repositories based on subscription. The behavior of the script can be changed using /etc/nethserver/eorepo.conf file which may contain the list of repositories to be enabled.

Restore original NethServer repositories

To revert the system configuration to its initial state and use the original NethServer repositories run

config setprop subscription SystemId '' Secret ''
signal-event software-repos-save

Enable extra repositories in subscription

If you need to enable an extra repository like NethForge, you can do it by the command line

config setprop subscription ExtraRepositories nethforge
signal-event software-repos-save

The list of repositories (comma separated : nethforge,repoA,repoB) must match a valid repository definition list in /etc/yum.repos.d. Beware it is case sensitive.

Events

• nethserver-subscription-update standard update event; it runs on new installation and subsequent updates
• nethserver-subscription-save runs when the registration token is set from the UI, altering the YUM repository configuration
• restore-yumrepos runs during pre-restore-config event. Actions and templates bound to it run in a temporary environment where the configuration DB is extracted from the configuration backup and temporarily applied to the system. It occurs before automatic installation of packages from the backup set. It is the point where the YUM configuration is restored to make subscribed repositories immediately available.

Alerts

Collectd listens for notification on a well known socket: /var/run/collectd.sock.

Notifications can be generated by Collectd thresholds, cron jobs or even external daemons like NMS.

Collectd thresholds

You can trigger an alarm named testalarm from command line:

echo -e "PUTNOTIF host=$(hostname) type=custom type_instance=testalarm severity=failure time=$(date +%s) message=\"$1 FAILURE\"" | nc -U /var/run/collectd.sock &>/dev/null

You can also resolve the above alarm:

echo -e "PUTNOTIF host=$(hostname) type=custom type_instance=testalarm severity=okay time=$(date +%s) message=\"$1 OK\"" | nc -U /var/run/collectd.sock &>/dev/null

Adding a threshold configuration

Create a /etc/collectd.d/threshold.conf template fragment using collectd threshold syntax. Enjoy collectd docs here https://collectd.org/documentation/manpages/collectd-threshold.5.shtml

Collectd python plugin

The nethserver_alerts.py file implements a Collectd plugin. The plugin dispatches each notification to remote Dartagnan instance. If the remote Dartagnan instance is not available, the plugin will save the notifications inside a local queue and will retry to resend them later.

Plugin options (inside /etc/collectd.d/threshold.conf):

• api_url URL where alerts will be dispatched
• lk: System ID
• secret: (optional) if set, the secret will be used as authorization token
• debug: enable the debug, can be yes or no. Default to no

NethServer Monitor Services (NMS)

NMS is a python daemon which monitor the status of all services registered inside the configuration db. Each service is checked every 300 seconds, if the status of a service changes, NMS writes a notification to Collectd socket.

Inventory

The inventory uses the facter software, from puppet (https://docs.puppet.com/facter/). Facter collects a standard set of informations (like CPU model, runing kernel, etc.) but the system also gather some custom facts.

Custom facts

Scripts for custom facts are inside /opt/puppetlabs/puppet/lib/ruby/2.1.0/facter/ directory. Each package can register a new custom fact, but this packages always provides the following:

• rpms, list of installed RPMs
• raid, RAID configuration
• templates_custom, list of configured template custom wihout content
• arp_macs, number of network devices
• users, list of configured users
• backup, info about backup status
• esmithdb, all esmith databases, password values are replaced with ***

Sending the inventory

Make sure that SystemId property under subscription key is already set. If the Secret property is set, it will be used as authorization token when sending the inventory.

The inventory will be sent every day using cron. If you want to manually send the inventory, use the following command:

/etc/cron.daily/nethserver-inventory

nethserver-suricata

The IPS (Intrusion Prevention System) module configures Suricata using the netfilter queue (NFQUEUE). NFQUEUE is an iptables and ip6tables target which delegate the decision on packets to a userspace software.

All traffic will be analyzed by Suricata itself and events are logged inside /var/log/suricata/eve.json. See EveBox for a report of blocking and alerting rules.

Suricata rules are managed by Pulledpork.

Manually enable/disable Suricata

Enabling:

config setprop suricata status enabled
signal-event firewall-adjust
signal-event nethserver-suricata-save

Disabling:

config setprop suricata status disabled
signal-event firewall-adjust
signal-event nethserver-suricata-save

Bypass

All bypasses are saved inside the ips database.

Each record with bypass type has the following properties:

• Host: it can be a firewall object or a raw IP/CIDR address
• status: it can be enabled or disabled
• Description: optional description

Example:

bypass1=bypass
  Description=
  Host=host;test1
  status=enabled
bypass2=bypass
  Description=
  Host=192.168.0.1
  status=disabled

Custom rules

If a file named /etc/suricata/rules/custom.rules, it will be included inside Suricata configuration. After creating the file, execute: signal-event nethserver-suricata-update.

Also remember to add it to the configuration backup:

echo /etc/suricata/rules/custom.rules >> /etc/backup-config.d/custom.include

Troubleshooting

When troubleshooting network traffic, just remember that Suricata will intercept all the traffic.

nethserver-unbonund

Unbound is a validating, recursive, and caching DNS resolver. Currently it’s only used only as DNS server for anti-spam DNSBL implementation.

When a new mail arrives, the anti-spam filter checks if sender IP address is listed in well-known blacklists. Some blacklists doesn’t allow multiple query from public DNS servers, thus the system needs a DNS server which can directly query the blacklist DNS.

If the mail filter is installed, dnsmasq configuration is changed, and specific domain query are redirect trough unbond. See: /etc/e-smith/templates/etc/dnsmasq.conf/26unbound_rbl.

Package: nethserver-unbound.

Database

Configuration is saved in unbound key inside configuration database.

Available properties:

• UDDPort: server address of the mail server, default is localhost
• access: default is none since unbound should not be used as a normal DNS server
• status: can be enabled or disabled, default is enabled

Example:

unbound=service
   UDPPort=10053
   access=none
   status=enabled

Configuration can be applied using the nethserver-unbound-update event.

nethserver-unixODBC

This package contains templates that generate ODBC configuration.

/etc/odbc.ini

The template that generates this file scan all configuration db searching for a key that has type=odbc (or ODBC for backward compatibility with SMEserver) and generates a section in /etc/odbc.ini. For example, this is ODBC object is used for asterisk cdr database:

# config show odbc-asteriskcdr
odbc-asteriskcdr=odbc
   Database=asteriskcdrdb
   Description=ODBC on asteriskcdrdb
   Driver=MySQL
   Port=3306
   Server=localhost

this object generates the /etc/odbc.ini section:

[odbc-asteriskcdr]
Server = localhost
Database = asteriskcdrdb
Driver = MySQL
Port = 3306
Description = ODBC on asteriskcdrdb

/etc/odbcinst.ini

This templates contains configuration for ODBC drivers. By default there are only MySQL and PostgreSQL driver configurated. The nethserver-unixODBC package requires those to be installed to:

# rpm -q --requires nethserver-unixODBC
...
mysql-connector-odbc
postgresql-odbc
...

Usage

Creation of a new ODBC driver is as simply as launching:

config set <new ODBC object name> odbc Description <description> Driver <MySQL|PostgreSQL> Server <server hostname> Database <database name> Port <database port>
signal-event nethserver-unixODBC-update

nethserver-vsftpd

The FTP module uses the vsftpd daemon. It is accessible only from local network and it is disabled by default. It supports both virtual and system users, but not at the same time. All virtual home directories are created inside /var/lib/nethserver/ftp.

The daemon will run as ftp user with passive mode enabled.

Virtual users

Virtual users are enabled by default.

Add a new user

Create a record inside the accounts db and activate changes:

db accounts set <user> ftp status enabled Password <pass> Chroot enabled
signal-event nethserver-vsftpd-save

The event will generate /etc/vsftpd/ftpusers.db (Berkeley db) containing all user credentials. The db is used for pam authentication (see: /etc/pam.d/vsftpd-virtual).

Properties:

• status: can be enabled or disabled. If enabled, the user can access the ftp server
• Password: the user password in clear text, can not be blank
• Chroot: can be enabled or disabled. If enabled, the user will be chrooted to /var/lib/nethserver/ftp/<user> directory. If disabled the user is not chrooted. ATTENTION: non-chrooted users can explore the entire file system
• ChrootDir: set a custom ChrootDir, it may be used to chroot a user inside a shared folder

Chroot a user inside a shared folder

Use these commands:

db accounts set <user> ftp status enabled Password <pass> Chroot enabled ChrootDir /var/lib/nethserver/ibay/<share_name>
signal-event nethserver-vsftpd-save

Where share_name is the name of the share, user the username and pass the password of the user.

System users

Warning

This configuration is highly discouraged, because user’s password is transmitted in clear text

After enabling system users, all virtual users will be disabled.

Enable system users:

config setprop vsftpd UserType system
signal-event nethserver-vsftpd-save

Enable an existing system user to access FTP server:

db accounts setprop myuser FTPAccess enabled
db accounts setprop myuser Shell /bin/bash
signal-event user-modify myuser

Apply configuration:

signal-event nethserver-vsftpd-save

To disable an already enabled user:

db accounts setprop myuser FTPAccess disabled
signal-event nethserver-vsftpd-save

If not explicitly disabled, all system users are chrooted inside their home directories. To disable a chroot for a system user:

db accounts setprop myuser FTPChroot disabled
signal-event nethserver-vsftpd-save

Custom chroot

When the FTP server uses system users, custom chroot is not supported: all users are chrooted inside their own home directory. Although it’s possible to change a system user home directory. Use the command below if the system user will used only for FTP access:

lusermod -d <your_custom_home> <user>

nethserver-webtop4

WebTop 4 is a full-featured groupware written in Java.

It’s composed by three parts:

• Java web application running on Tomcat 7
• PHP implementation of Active Sync protocol
• PostgreSQL database

Access to web application is forced in SSL mode.

Database

Configuration is saved in webtop key inside configuration database.

Available properties:

• PublicUrl: public URL used to publish resources for the cloud. If not set, default is http://<FQDN>/webtop
• ActiveSync: if set to enabled, it enables /Microsoft-Server-ActiveSync url. Default is enabled

Example:

webtop=configuration
    ActiveSync=enabled
    PublicUrl=

Configuration can be applied using the nethserver-webtop4-update event.

Troubleshooting

In case of errors, see following logs:

• Tomcat: /var/log/tomcat/catalina.out
• Active Sync: /var/log/z-push/z-push-error.log

To inspect z-push status use:

php /usr/share/webtop/z-push/z-push-admin.php

Known problems

When PostgreSQL is restarted, WebTop can loss the database connection. If a blank page is displayed, restart Tomcat with the following command:

systemctl restart tomcat

nethserver-webtop5

WebTop 5 is a full-featured groupware written in Java.

It’s composed by three parts:

• Java web application running on Tomcat 8
• PHP implementation of Active Sync protocol
• PHP implementation of CardDAV and CalDAV protocol
• PostgreSQL database

Access to web application is forced in SSL mode.

WebTop 5 has been split in 4 different RPMs:

• webtop5: Tomcat webapp and all third-party jars, derived from a WAR. It contains all jars developed by Sonicle. This package will be updated at each WebTop release
• webtop5-zpush: ActiveSync implementation for WebTop, it contains PHP code from z-push project (http://z-push.org/)
• webtop5-webdav: CardDAV and CalDAV implementation for WebTop, it contains PHP code from sabre/dav project (http://sabre.io/dav/)
• nethserver-webtop5: NethServer auto-configuration for WebTop

Database

Configuration is saved in webtop key inside configuration database.

Available properties:

• ActiveSyncLog: log level of z-push implementation. As default z-push will log only relevant errors.
• ActiveSyncLegactIds: can be enabled or disabled. If set to enabled, use backward compatibile z-push ids to avoid device full rsync on update. See “Active Sync” section for more info
• Debug: if set to true, enable debug for the web application. Default is false
• DefaultLocale: default locale for WebTop users. To list available locales execute: /etc/e-smith/events/actions/nethserver-webtop5-locale-tz
• DefaultTimezone: default timezone for WebTop users. To list available timezones: JAVA_HOME=/usr/share/webtop/ java ListTimeZones
• MinMemory and MaxMemory: minimun and maximum memory of Tomcat instance. Values are expressed in MB.
• PublicUrl: public URL used to publish resources for the cloud. If not set, default is http://<FQDN>/webtop
• DavServerUrl: Dav server URL for CalDAV and CardDAV clients configuration. If not set, default is https://<FQDN>/webtop-dav/server.php
• DavServerLog: log level of webtop-dav implementation. As default webtop-dav will log only relevant errors.
• PbxProvider: PBX provider name
• PbxProviderNethvoiceWebrestUrl: NethVoice base url for API calls, used when PbxProvider is set to nethvoice
• DefaultToolBarIconsSize: Default dimension of the toolbar icons, available values: small, medium, large. Default is medium
• RemoteCalendarAutosync: if set to enabled, it enables remote calendars auto-sync functionality. The sincronization interval can be set by user on remote calendar creation. Default is enabled
• RemoteCalendarAutosyncOnlywhenonline: if set to enabled the remote calendars is auto-sync only when calendar’s owner is online during the sincronization time. Default is disabled
• RemoteCategoryAutosync: if set to enabled , it enables remote categories auto-sync functionality. The sincronization interval can be set by user on remote category creation. Default is enabled
• RemoteCategoryAutosyncOnlywhenonline: if set to enabled the remote categoties is auto-sync only when category’s owner is online during the sincronization time. Default is disabled
• VirtualHost: set custom virtual host, e.g. mygroupware.mydomain.it. Virtual host can be used to access WebTop and configure ActiveSync/CalDAV/CardDAV clients
• KnownDeviceVerification: if set to enabled, a notification will be sent for any new authentication attempt from an unknown-device, default disabled
• KnownDeviceVerificationRecipients: a comma-separated list of email addresses to use as CCN recipients that will receive a copy of the unknown-device notice
• KnownDeviceVerificationNetWhitelist: a comma-separated list of networks, specified in CIDR format, from where all devices will be considered trusted, and no notifications will be sent

Example:

webtop=configuration
    ActiveSyncLog=ERROR
    Debug=false
    DefaultLocale=en_US
    DefaultTimezone=Etc/UTC
    MaxMemory=1024
    MinMemory=512
    PublicUrl=

Configuration can be applied using the nethserver-webtop5-update event.

Reset admin password

1. Access the database

   su - postgres -c 'psql webtop5'
   

2. Copy & paste the following query:

   UPDATE "core"."local_vault" SET "password_type"='PLAIN', "password"='admin' WHERE ("domain_id"='*') AND ("user_id"='admin');
   

3. Access the web interface using admin user with password admin.

Troubleshooting

Please note that nethserver-webtop5 is composed by many parts. Each part has its own logs and troubleshooting best practices.

Web application

The web application logs are inside /var/log/webtop/webtop.log.

Tomcat

Tomcat instance is managed by systemd unit called tomcat8@webtop. All logs are saved inside /var/lib/tomcats/webtop/logs/ directory. The logs are rotated daily and deletes after 2 days.

Active Sync

Active Sync is implemented using a PHP application called z-push. All logs are inside /var/log/z-push/ directory.

To inspect z-push status use:

sudo -u apache scl enable rh-php73 'php /usr/share/webtop/z-push/z-push-admin.php'

It is also possibile to enable z-push debug using these commands:

config setprop webtop ActiveSyncLog DEBUG
signal-event nethserver-webtop5-update

Instead of DEBUG you can use any constant supported by z-push implementation, but remove the LOGLEVEL_ prefi. See /usr/share/webtop/z-push/inc/zpush.config.php.

You can test Active Sync using this command (please set user, password and server_name):

curl -k -u goofy@local.neth.eu:password https://server_name/Microsoft-Server-ActiveSync

You should see an HTML output containing the string:

GET not supported

Legacy ids

When the ActiveSyncLegacyIds is set to enabled, the z-push implementation is affected by the following limitations:

• a user can’t have more than one calendar with the same name
• resources with very long names (eg. calendars) can cause synchronization problems

If such problems occur, please switch to new id implementation:

config setprop webtop ActiveSyncLegacyIds disabled
rm -rf /var/log/z-push/state/*
signal-event nethserver-webtop5-update

Please note that after switching to new implementation, all devices will require a full synchronization.

CardDAV and CalDAV

CardDAV and CalDAV are implemented using a PHP application called webtop-dav. All logs are inside /var/log/webtop-dav/ directory.

It is also possibile to enable webtop-dav debug using these commands:

config setprop webtop DavServerLog DEBUG
signal-event nethserver-webtop5-update

Instead of DEBUG you can use any constant supported by webtop-dav implementation. See /usr/share/webtop/webtop-dav/lib/webtop/Log.php.

To enable browser-plugin for directory indexes of the Dav server:

config setprop webtop Debug true
signal-event nethserver-webtop5-update

Tomcat instance

WebTop uses its own Tomcat instance running on port 58080.

The instance is launched with some special Java options, see content of /etc/sysconfig/tomcat8@webtop.

License Headers

Always add a COPYING in the documentation directory of your package, containing the full GPLv3 license.

Change the Copyright line according to your needs.

PHP, Javascript, CSS

/*
 * Copyright (C) 2020 Nethesis S.r.l.
 * http://www.nethesis.it - nethserver@nethesis.it
 *
 * This script is part of NethServer.
 *
 * NethServer is free software: you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License,
 * or any later version.
 *
 * NethServer is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with NethServer.  If not, see COPYING.
 */

Perl, Python and BASH

#
# Copyright (C) 2020 Nethesis S.r.l.
# http://www.nethesis.it - nethserver@nethesis.it
#
# This script is part of NethServer.
#
# NethServer is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License,
# or any later version.
#
# NethServer is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with NethServer.  If not, see COPYING.
#

Rebranding Administrator Manual

It’s possible to create a custom version of the Administrator Manual.

Environment

1. Clone the documentation repository

2. Create a directory inside the main administrator-manual/<lang> directory. Example for a new rebranding called NethService and available only in Italian:

   mkdir administrator-manual/it/nethservice
   

3. Enter the directory, and create the structure:

   cd nethservice
   mkdir _templates _static _build _themes
   

4. Copy the makefile and configuration from parent directory:

   cp ../Makefile ../conf.py .
   

Contents

First, create a custom index.rst with required chapters. Example:

My section
-------------

.. toctree::
      :maxdepth: 2

   installation
   newchapter

To add a chapter, create new rst file inside the current directory. Example for newchapter.rst:

===========
New chapter
===========

This is a new chapter.

If you wish to reuse existing chapters, create links to the parent directory. Example:

ln -s ../installation.rst installation.rst

Product name and version

Edit the conf.py by setting product name and version. Feel free to customize anything you need but make sure to edit at least following variables:

• project
• release

Create a rst_prolog file with the macro for product name and download site. Content of rst_prolog file:

.. |product| replace:: NethService
.. |download_site| replace:: http://www.nethesis.it

Theme

Choose and existing Sphinx theme or copy a new theme inside the _themes directory. If you want to use a custom theme, remember to set following variables inside conf.py:

html_theme_path = ['_themes']
html_theme = 'mynewshinytheme'

Artworks

If you wish to add custom artworks like logo and favicon, edit these variables inside conf.py:

html_static_path = ['_static']
html_logo = '_static/logo.png'
html_favicon = '_static/favicon.ico

License

This documentation is distributed under the terms of Creative Commons - Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.

You are free to:

• Share — copy and redistribute the material in any medium or format
• Adapt — remix, transform, and build upon the material

The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

• Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
• NonCommercial — You may not use the material for commercial purposes.
• ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

This is a human-readable summary of (and not a substitute for) the full license available at: http://creativecommons.org/licenses/by-nc-sa/4.0/

Architecture documentation is from SME Server project and is licensed under GNU Free Documentation License 1.3 (http://www.gnu.org/copyleft/fdl.html). See http://wiki.contribs.org/ for original documentation.

Indices and tables

• Index
• Search Page