How I Learned to Loathe the Internet

One of the most popular things on the internet is the marriage of cheap computer hardware and Kubernetes.  Take some commodity Raspberry Pi's, network them together, toss on some microk8s, and the world is your oyster in its entirety.

Unfortunately, this is a lie.  The documents lie, they want you to trust them then completely lead you astray; they'll take your hope and feed it to you via missing commands and a bit of hand-waving.

Am I angry? No, I conquered that mountain. This post it to make sure other people can also conquer that mountain.

The Installation

So, you've got a fancy Raspberry Pi 4 you want to use? Then let's go! I'm using Ubuntu Server 21.10 for this procedure because (1) I'm very familiar with Ubuntu, and (2) Ubuntu Server 20.04 LTS won't boot off of an NVMe drive connected via USB without sacrificing children to the forgotten gods of yore.

Take your SD Card/USB thumbdrive/NVMe drive and get your storage media configured.  If you want to use PXE boot to do your business, this is definitely not the guide for you.  I recommend the Raspberry Pi Imager software to get this done as it's reliable and I've used it more times than I care to admit over the last week.

Once this is done, use the following options to figure out how to proceed:

  • SD card: insert in the back of the Raspberry Pi.
  • USB thumbdrive: hook up to a USB2 or 3 port depending on the drive
  • NVMe drive: update the X873 NVMe hat that you bought with the recommended firmware by fighting with the ASMedia firmware update software, then see that you're still having issues so then track down some shady firmware that came out of Crucial.  Swap out the NVMe drive with others until you realize that the HP EX920 that you've been trying to use to get things going refuses to work for longer than 3 minutes at a time.  Once you figure that out, use a different drive like a WD SN550 and cry as the whole ordeal has taken years off your life.

So, at this point, I'll let you configure the basics on your Raspberry Pi. You can use the default ubuntu user, but I prefer making another user, setting a pretty strong password, then disabling/deleting the ubuntu user.

Then, run all of your updates:

$ sudo apt-get update && sudo apt-get -y dist-upgrade && sudo reboot

For the amount of time it will take, consult the list below:

  • SD Card: an eternity
  • USB thumbdrive: a bit if it's a good USB3 drive, an eternity if it's a USB2 thumbdrive you picked up at Microcenter in 2007 that just happened to be laying around your computer and you had no other options
  • NVME USB monstrosity: best case scenario, decently fast but you still have to wait for all those kernel files

The Missing Commands

So, you could just run sudo snap install microk8s --classic and try to spin up something like nginx, but you'd probably miss the fact that your Rasperry Pi is screaming under ~40% load assuming the snap install is successful and the microk8s start command went off without a hitch.

What you should do:

  1. Install MicroK8s (sudo snap install microk8s --classic)
  2. Go ahead and add yourself to the microk8s group: sudo usermod -a -G microk8s put_your_username_here.  This will make it easier to run commands and will keep MicroK8s from complaining at you.
  3. Install you a nice container backend to make stuff more awesome: sudo apt-get install containerd.  You'll definitely need it.
  4. Next, the first missing command: sudo apt-get install linux-modules-extra-raspi.  We need that package otherwise networking is literally going to eat itself.  Nothing you try to deploy will even think about working as flanneld literally can't do anything.
  5. So, with that out of the way: go ahead and reboot.
  6. Log back in, and let's continue.
  7. Since everything is looking good, let's fire up us a Kubernetes cluster! Run the microk8s start command and wait.
  8. Once that is done, we have another issue: your cluster is in high availability mode and is consuming the aforementioned 40% of your Raspberry Pi's anemic CPU time.  Let's fix that!
  9. Disable high availability mode: microk8s disable ha-cluster.  Very simple command, and it's going to reset stuff.  Once it's finished, go ahead and bounce the system again.

This should get you very, very close to a fully functional cluster.  I'm still annoyed at some of the things that have to happen that seem to lay just outside of the online documents, but I hope this helps someone else out!