Author: Aravindh Puthiyaparambil (Red Hat)
Kubernetes 1.27 introduced a new feature called Node log query that allows viewing logs of services running on the node.
Cluster administrators face issues when debugging malfunctioning services running on the node. They usually have to SSH or RDP into the node to view the logs of the service to debug the issue. The Node log query feature helps with this scenario by allowing the cluster administrator to view the logs using_kubectl_. This is especially useful with Windows nodes where you run into the issue of the node going to the ready state but containers not coming up due to CNI misconfigurations and other issues that are not easily identifiable by looking at the Pod status.
The kubelet already has a /var/log/ viewer that is accessible via the node proxy endpoint. The feature supplements this endpoint with a shim that shells out to
journalctl, on Linux nodes, and the
Get-WinEvent cmdlet on Windows nodes. It then uses the existing filters provided by the commands to allow filtering the logs. The kubelet also uses heuristics to retrieve the logs. If the user is not aware if a given system services logs to a file or to the native system logger, the heuristics first checks the native operating system logger and if that is not available it attempts to retrieve the first logs from
On Linux we assume that service logs are available via journald, and that
journalctl is installed. On Windows we assume that service logs are available in the application log provider. Also note that fetching node logs is only available if you are authorized to do so (in RBAC, that's get and create access to
nodes/proxy). The privileges that you need to fetch node logs also allow elevation-of-privilege attacks, so be careful about how you manage them.
To use the feature, ensure that the
NodeLogQueryfeature gate is enabled for that node, and that the kubelet configuration options
enableSystemLogQuery are both set to true. You can then query the logs from all your nodes or just a subset. Here is an example to retrieve the kubelet service logs from a node:
# Fetch kubelet logs from a node named node-1.example kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"
You can further filter the query to narrow down the results:
# Fetch kubelet logs from a node named node-1.example that have the word "error" kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet&pattern=error"
You can also fetch files from
/var/log/ on a Linux node:
kubectl get --raw "/api/v1/nodes/<insert-node-name-here>/proxy/logs/?query=/<insert-log-file-name-here>"
You can read thedocumentationfor all the available options.