kubelet_token_requests.md 2.2 KB
Understand the alert
This alert is related to Kubernetes Kubelet token requests. It monitors the number of failed Token() requests to an alternate token source. If you receive this alert, it means that your system is experiencing an increased rate of token request failures.
What does a token request in Kubernetes mean?
In Kubernetes, tokens are used for authentication purposes when making requests to the API server. The Kubelet uses tokens to authenticate itself when it needs to access cluster information or manage resources on the API server.
Troubleshoot the alert
- Investigate the reason behind the failed token requests
Check the Kubelet logs for any error messages or warnings related to the token requests. You can use the following command to view the logs:
journalctl -u kubelet
Look for any entries related to Token() request failures or authentication issues.
- Verify the alternate token source configuration
Review the Kubelet configuration file, usually located at /etc/kubernetes/kubelet/config.yaml. Check the authentication and authorization sections to ensure all the required settings have been correctly configured.
Make sure that the specified alternate token source is available and working correctly.
- Check the API server logs
Inspect the logs of the API server to identify any issues that may prevent the Kubelet from successfully requesting tokens. Use the following command to view the logs:
kubectl logs -n kube-system kube-apiserver-<YOUR_NODE_NAME>
Look for any entries related to authentication, especially if they are connected to the alternate token source.
- Monitor kubelet_token_requests metric
Keep an eye on the kubelet_token_requests metric using the Netdata dashboard or a monitoring system of your choice. If the number of failed requests continues to increase, this might indicate an underlying issue that requires further investigation.