System logs#

Almost everything is logged inside journalctl and sent to the local log server.

By default, Grafana Loki is installed on the leader node, it collects the logs from all cluster nodes. A rootful Promtail container runs on all nodes, including the leader one. It sends all logs to the Loki server.

From the leader node, it is possible to query the logs of all nodes.

Logs are accessible from the System logs page. You can filter logs by date, a text query or context. Available contexts are:

  • cluster: all logs from any source

  • node: all logs from a given node

  • app: logs from a given application instance, regardless where it’s currently running

You can see only a subset of the lines or follow the log to see what’s happening in real time.

Sometimes is useful to compare the logs of two applications side-by-side. You can do it by following these steps:

  • setup the filter for a refined search

  • click on Add search button

  • setup the new filter

  • select Vertical layout from the three-dots menu

Logs are now shown side-by-side to easily correlate events.

Command line#

If you’re familiar with the command line, recent logs are visible using the journalctl command and services can be inspected using the systemctl command. As root use journalctl to see messages from agents, rootful and rootless modules.

You can also use the api-server-logs to query directly the Loki server. Example to inspect the log of the traefik1 module:

api-server-logs logs -e module -n traefik1

You can also enable the automatic completion of the above command. First, install the bash-completion package.

On RHEL-like distributions:

dnf install bash-completion -y

On Debian distribution:

apt-get install bash-completion -y

Then, generate the completion script:

api-server-logs completion bash > /etc/bash_completion.d/api-server-logs.sh

Logout and login from the shell to enable the completion.