Almost everything is logged inside journalctl and sent to the local log server.
By default, Grafana Loki is installed on the leader node, it collects the logs from all cluster nodes. A rootful Promtail container runs on all nodes, including the leader one. It sends all logs to the Loki server.
From the leader node, it is possible to query the logs of all nodes.
Logs are accessible from the
System logs page.
You can filter logs by date, a text query or context. Available contexts are:
cluster: all logs from any source
node: all logs from a given node
app: logs from a given application instance, regardless where it’s currently running
You can see only a subset of the lines or follow the log to see what’s happening in real time.
Sometimes is useful to compare the logs of two applications side-by-side. You can do it by following these steps:
setup the filter for a refined search
click on Add search button
setup the new filter
Vertical layoutfrom the three-dots menu
Logs are now shown side-by-side to easily correlate events.
If you’re familiar with the command line, recent logs are visible using the
and services can be inspected using the
As root use
journalctl to see messages from agents, rootful and rootless modules.
You can also use the
api-server-logs to query directly the Loki server.
Example to inspect the log of the traefik1 module:
api-server-logs logs -e module -n traefik1
You can also enable the automatic completion of the above command.
First, install the
On RHEL-like distributions:
dnf install bash-completion -y
On Debian distribution:
apt-get install bash-completion -y
Then, generate the completion script:
api-server-logs completion bash > /etc/bash_completion.d/api-server-logs.sh
Logout and login from the shell to enable the completion.