System logs

Log records generated by any cluster node are collected and stored in the leader node. See Logs persistence for additional information.

Accessing logs

The System Logs page allows users to efficiently search logs based on date, text query, or context. The available contexts include:

• Cluster: Displays logs from all cluster nodes.

• Node: Shows logs from a specific node.

• Application: Presents logs from a particular application instance.

Users can select between two modes of operation:

• Dump mode Retrieves a limited number of log records within a specified time period. The maximum number of lines can be adjusted using the Max lines field.

• Follow mode Displays logs in real-time, providing live updates.

If the Search query field is set, only log records matching the given string are returned. Only exact substring matches are allowed.

For comparative analysis of logs from two nodes or applications, follow these steps:

• Click on Add search button.

• Optionally, switch to Vertical layout from the three-dots menu for a side-by-side comparison.

For example, comparing logs can be useful to see the Traefik log from one side, to see incoming HTTP requests, and Nextcloud logs on the other side to see that application activity.

Note

By default, log searches are directed to the active Loki instance. If there are inactive Loki instances within the cluster, it is possible to select them to search past log entries.

Command line interface

In addition to accessing logs via the System Logs web page, users can utilize the api-server-logs command for log searching. Below are examples illustrating its usage:

1. Basic invocation: Enters “follow mode” for the entire cluster:

   api-server-logs logs
   

2. Follow mode for application instance: Enables follow mode for the specified application (module) instance, such as traefik1. The --entity flag selects the context:

   api-server-logs logs --entity module --name traefik1
   

3. Dump mode for specific instance in a time period: Initiates dump mode for the same instance within a specific time period. Dates must adhere to the ISO8601 format:

   api-server-logs logs --mode dump --entity module --name traefik1 --from 2024-04-09T16:43:22Z --to 2024-04-09T16:55:31Z
   

4. Changing output timezone: Modifies the output timezone while maintaining the same query. Refer to timedatectl list-timezones for a full list of options:

   api-server-logs logs --timezone America/New_York --mode dump --entity module --name traefik1 --from 2024-04-09T16:43:22Z --to 2024-04-09T16:55:31Z
   

Logs persistence

Upon cluster creation, a Loki [1] core module instance is installed on the leader node and designated as the active instance. The leader node, like any other worker node, continuously streams its log data to this active Loki instance [2].

Adjusting Settings

Navigate to the Settings page, click on the System logs card and then on the three-dots menu to modify log retention (select Edit retention) or assign a user-friendly name to the active Loki instance (select Edit label).

Understanding log retention

Log retention refers to the maximum age of stored log records. Records older than the retention period are automatically purged. By default, System logs have a retention period of 365 days, but this can be customized to any desired duration. For compliance with common regulations and best practices, a recommended retention period is 200 days or longer.

Inactive Loki instances

When a worker node is promoted to leader, a new Loki instance is installed on it and becomes the active instance, while the old instance is marked as inactive.

• The new active instance inherits the retention setting from the old one.

• An inactive instance retains logs based on its last retention setting.

• Log searches can still be performed within an inactive instance.

• Restoring a Loki instance from backup renders it inactive.

• To remove an inactive instance, select the three-dots menu and choose the Uninstall action.

Logs forwarding

On the leader node, you have the capability to forward log streams managed by the active Loki instance to either an external syslog server or to the Cloud Log Manager. This enables centralized log aggregation, making it easier to monitor, analyze, and troubleshoot your NethServer environment.

To enable forwarders, navigate to Settings -> System Logs page, where all Loki instances, including the active one, are listed. Here, you can configure services by opening the Loki card menu.

If the forwarder is enabled, its status can be viewed in two places:

• On the Cluster Status page under the System Logs card.

• On the Settings -> System Logs page under the active Loki instance card.

The displayed states are:

• Enabled: A green circle with a description indicates the forwarder is active.

• Failed: A red circle with a description indicates the forwarder has encountered an issue.

Syslog

A syslog server receives, stores, and manages log messages from network devices and applications, facilitating centralized log monitoring and analysis.

Before setting up the forwarder, ensure your syslog server is functioning properly. You will need the following information to enable the syslog forwarder:

• Host name or IP address: The IP address of the syslog server.

• Port: The port number on which the syslog server listens.

• Protocol: The protocol used for server communication (UDP or TCP).

• Filter: Choose whether to forward all log records or only those related to security events.

• Format: Log format (RFC 3164 or RFC 5424).

• Export start date: Specify from when logs should be forwarded. You can choose the last timestamp [3] or manually enter a date and time.

Cloud Log Manager

Note

This service is available only with subscription.

The Nethesis Cloud Log Manager is a centralized solution for collecting, storing, and managing logs from various devices within an organization. It allows for real-time event collection from systems such as Linux, Windows, firewalls, switches, and hypervisors, centralizing all logs into a single interface.

To enable the forwarder for Cloud Log Manager, navigate to the Settings page and select System Logs. Open the three-dots menu of the active Loki instance, then click on Configure Cloud Log Manager action. Fill the form with the following information:

• Cloud Log Manager URL: The URL of the Cloud Log Manager (usually https://nar.nethesis.it/).

• Company unique key: This key, also known as ‘tenant’, identifies and associates the cluster logs within a company in Cloud Log Manager. You can find it in Cloud Log Manager web application, under Users and Companies > Companies.

• Export start date: Specify from when logs should be forwarded. You can choose the last timestamp [3] or manually enter a date and time.

Take note of the cluster identifier, displayed in the notice at the top of the form. The cluster identifier value is a string like cluster-02834ab3 and will appear in the Host column of the Cloud Log Manager web application.

Once the form is saved, the export procedure begins. After a few minutes, the cluster security logs will be available at https://naradmin.nethesis.it/.

Only security logs are forwarded to the Cloud Log Manager. To forward the full log stream, which may generate considerable data, refer to Syslog.

Footnotes

[1]

Grafana Loki is a special database designed to store, index and search system logs. For more information, see nethserver/ns8-loki

[2]

The promtail.service core service operates on every node, reading system journals, forwarding new records to the active Loki instance, and preserving the last sent journal cursor position to ensure seamless restarts without log loss.

[3] (1,2)

Last timestamp indicates the last time that the forwarder successfully sent logs. This allows the forwarder to resume forwarding from where it was interrupted, ensuring continuity in log management.