System logs#
Almost everything is logged inside journalctl and sent to the local log server.
By default, Grafana Loki is installed on the leader node, it collects the logs from all cluster nodes. A rootful Promtail container runs on all nodes, including the leader one. It sends all logs to the Loki server.
From the leader node, it is possible to query the logs of all nodes.
Logs are accessible from the System logs
page.
You can filter logs by date, a text query or context. Available contexts are:
cluster
: all logs from any sourcenode
: all logs from a given nodeapp
: logs from a given application instance, regardless where it’s currently running
You can see only a subset of the lines or follow the log to see what’s happening in real time.
Sometimes is useful to compare the logs of two applications side-by-side. You can do it by following these steps:
setup the filter for a refined search
click on Add search button
setup the new filter
select
Vertical layout
from the three-dots menu
Logs are now shown side-by-side to easily correlate events.
Command line#
If you’re familiar with the command line, recent logs are visible using the journalctl
command
and services can be inspected using the systemctl
command.
As root use journalctl
to see messages from agents, rootful and rootless modules.
You can also use the api-server-logs
to query directly the Loki server.
Example to inspect the log of the traefik1 module:
api-server-logs logs -e module -n traefik1
You can also enable the automatic completion of the above command.
First, install the bash-completion
package.
On RHEL-like distributions:
dnf install bash-completion -y
On Debian distribution:
apt-get install bash-completion -y
Then, generate the completion script:
api-server-logs completion bash > /etc/bash_completion.d/api-server-logs.sh
Logout and login from the shell to enable the completion.