Analytiks Logo
Back to Tech Corner
DevOps

(July 2023 – DevOps) Minikube fails to start under Linux - It seems the kubelet isn't running or is healthy

Minikube is a lightweight Kubernetes implementation that creates a VM on your local machine and deploys a simple cluster containing only one node. Minikube is available for Linux, macOS, and Windows systems and is predominantly used while learning to work with Kubernetes; among other uses.

I have a minikube installed on a Linux environment and ran into an issue. The troubleshooting process and resolution is documented below.

Server Information1

  • AWS Ubuntu medium for Master 2 vCPU + 4gb ram
  • Linux ip-172-31-27-156 4.4.0-1128-aws /#142-Ubuntu SMP Fri Apr 16 12:42:33 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • Docker version 18.09.7, build 2d0083d

Issue

During kubeadm init

Minikube error message

Looking at the above problem, it seems that the kubelet failed to start. Execute the following command to see details of the issue:

tail /var/log/syslog

Showed:

Jan 27 15:45:24 ip-172-31-27-156 kubelet/[10517/]: E0127 15:45:24.337666 10517 server.go:302/] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: //systemd// is different from docker cgroup driver: //cgroupfs//"

Jan 27 15:45:24 ip-172-31-27-156 systemd/[1/]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Jan 27 15:45:24 ip-172-31-27-156 systemd/[1/]: kubelet.service: Unit entered failed state.
Jan 27 15:45:24 ip-172-31-27-156 systemd/[1/]: kubelet.service: Failed with result 'exit-code'.

The above log shows that the cgroup driver of kubelet is cgroupfs, and the cgroup driver of docker is systemd. The inconsistency between the two causes kubelet to fail to start.

Solve the problem

  1. I have tried to modify the cgroup driver of kubelet (file location: /etc/systemd/system/kubelet.service.d/10-kubeadm.conf), but it will be overwritten every time minikube is started, so I can only give up this method and instead modify the cgroup driver settings of docker.
  2. Open the file /usr/lib/systemd/system/docker.service, as shown below.
  3. Change the systemd in the red box to cgroupfs:
    • Note for Ubuntu path2

docker.service - Docker Application Container Engine

Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)

Active: active (running) since Thu 2022-01-27 15:21:12 UTC; 2h 22min ago

Docker service configuration file
  1. Reload the configuration information and restart the service:
Systemctl commands

After completing the above modifications, restart minikube and it will be successful, as follows:

Successful Minikube startup

I hope this article can provide some references when you encounter similar problems.

Note

For those who do not want to make changes in docker.service (or do not have permission) alternate method is to change the driver in kublet.service.d to match the value in Docker. Ubuntu docker distribution by default uses the native control group driver cgroupfs while kublet assumes systemd. As an alternate you may change the kublet config in (/etc/systemd/system/kubelet.service.d.

Suggested change:

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --cgroup-driver=cgroupfs"

1 Server is no longer available and was part of a training lab.

2 Docs: https://docs.docker.com

View All Articles