Single-Node Kunernetes on Baremetal: Part One
A working Kubernetes installation in a single lunch break!
This is a write-up of my project to get Kubernetes working as a single-node, general purpose install on a baremetal server provided by OVH.
It took me a long time to find the right information on how to do this, as many of the components are new with little documentation provided.
DISCLAIMER
A single-node instance like the one I’m about to describe is incredibly useful for a personal system or learning tool, but should not be confused with something that’s acceptable for production workloads.
Background
Why are you doing this?
I’ve had a few low-end virtual servers for a while in Sydney and Warsaw that I’ve been using for various tasks, but I’d been straddling the limit of their (tiny) resources for quite some time.
To compound the frustration, I’ve been using OVH’s OpenStack object storage containers as large virtual disks (thank you S3QL). Imagine using a server based in Sydney while its main data disk is over http on the other side of the world.
Regardless, it was time for a change, and the recent introduction of some temptingly cheap local baremetal servers has swayed me.
My Needs
I had to find a solution that would tick some boxes:
- It should have at least 200GiB of space
- It should be easily backed up (especially my Nextcloud data)
- It should be easy to partition different workloads and control their resource usage
- It should be secure
- It should be easy to manage and automate (I’m sick of the traditional special-snowflake server method)
- It should allow multiple domains and TCP services on the same static IP
Kubernetes in an Hour
Software I Used
- RancherOS: a ridiculously small distro where all the things are Dockerised
- RKE: the Rancher Kubernetes Engine
- Rancher: lubricant for Kubernetes
- MetalLB: Google’s solution to software load-balancing
- Helm and Tiller: a sort-of package manager for Kubernetes
Part 1: Preparation
I decided to use RancherOS for the server, as everything running on the system is dockerised and it’s designed for Kubernetes. Unfortunatelyt this is not one of the standard OS offerings by OVH.
OS Installation
Luckily, OVH do give you a Java Web Start KVM for your (Supermicro) server with the ability to connect an ISO to the virtual disc drive. The installation ISO for RancherOS is only about 100MiB, so this is absolutely perfect.
RancherOS uses the standard cloud-config YAML referencing to ingest its initial settings during installation. All you need is something similar to this:
#cloud-config
hostname: your.public.dns.address
rancher:
network:
interfaces:
eth0:
dhcp: true
dns:
nameservers:
# Using Google's DNS
- 8.8.8.8
- 8.8.4.4
ssh_authorized_keys:
- ssh-ed25519 AAAAblahblahblahtypicalkey email@add.ress
Once RancherOS is up and running, you can use some of the tricks in this helpful Gist to get a software RAID running in RancherOS:
# Create empty partition tables on each disk with G
sudo fdisk /dev/sda
sudo fdisk /dev/sdb
# Install once on each disk
sudo ros install -i rancher/os:v1.5.0 -t gptsyslinux -c cloud-config.yml -a "rancher.state.mdadm_scan" -d /dev/sda --no-reboot
sudo ros install -i rancher/os:v1.5.0 -t gptsyslinux -c cloud-config.yml -a "rancher.state.mdadm_scan" -d /dev/sdb --no-reboot
# Configure RAID
sudo mdadm --create /dev/md0 --level=1 -- metadata=1.0 --raid-devices=2 /dev/sda1 /dev/sdb1
# Final sanity checks
sudo fsck /dev/md0
sudo resize2fs /dev/md0
sudo fsck /dev/md0
sudo reboot
Kubernetes Initialisation
The Rancher Kubernetes Engine is painless to set up and get going. I’ll step through performing a single-node install using Let’s Encrypt as the certificate manager, but veering off the path a little bit to make things work seamlessly on the node.
First off you need to tell RKE to install everything it needs, using some YAML spec. There are a number of things you can tag and configure, though for my deployment I left everything at its default. Create your rke.yaml
file for deployment:
ignore_docker_version: true
ssh_agent_auth: true
nodes:
- address: public.dns.add.ress
hostname_override: public.dns.add.ress
user: rancher
role: [controlplane,worker,etcd]
network:
plugin: canal
options:
canal_iface: eth0
services:
etcd:
snapshot: true
creation: 6h
retention: 24h
As long as you’re using ssh-agent
for your SSH key, you don’t have to specify the actual key you’re using.
Next up, install Kubes and do some basic configuration:
rke up --config rke.yaml
export KUBECONFIG=$(pwd)/kube_config_rke.yaml
kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm install stable/cert-manager --name cert-manager --namespace kube-system
helm install rancher-stable/rancher --name rancher --namespace cattle-system --set hostname=your.public.dns.name
You might need to space the above commands out just to ensure that everything is deploying properly, or use kubectl
to check the rollout status of each.
You’re done, and should have a ready-to-go basic Kubernetes node. To make it truly useful, you’ll need to get down and dirty with some additional software.
Part 2: MetalLB
As this installation is only single-node, and I wanted to make sure that I could run all the services I needed to, I had to ensure that I had a way to share ports on the public IP address without resorting to hand-crafted iptables rules.
Why Use a Load Balancer?
It might seem counter-intuitive to run a load balancer service on a single node (and single IP) installation, but you need to know about how Kubernetes networking works for non-HTTP services. For the sake of a single node, you essentially have three options:
ClusterIP
as part of a service for internal-only access (just like linking Docker containers)NodePort
to directly expose the service on your Kubernetes node, on a random port from 30000 to 32767- As part of a
DaemonSet
… which I will not go into
As a DaemonSet
is not a recommended way to go about simple deployments, none of the options are especially useful for exposing a typical port (say, IMAP).
Enter MetalLB, Google’s software implementation of load-balancing for Kubernetes. Exposing a service to the world is as simple as:
- Installing MetalLB
- Telling MetalLB which IPs it can use
- Telling a service to expose a port on a particular IP
Luckily, the load balancer implementation supports sharing multiple services with unique ports on a single IP address. This is especially useful when you only have a single IP.
Setting it Up
Create a YAML spec containing the IP address of your service to instantiate the default pool:
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- your.public.ip.addr
Then install and apply:
kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml
kubectl apply -n metallb-system lb-pool.yaml
Exposing Workloads
Following the official documentation, getting it to work is as easy as assigning your service to the specific IP and putting in an IP sharing annotation.
Take a look at these two sample services: a typical bastion host and a generic game server with a voice port. MetalLB will expose multiple ports on the same IP address when you set the services up:
apiVersion: v1
kind: Service
metadata:
annotations:
metallb.universe.tf/allow-shared-ip: ip1
name: bastion-exposer
namespace: bastion
spec:
loadBalancerIP: your.public.ip.addr
ports:
- name: ssh
port: 1234
protocol: TCP
targetPort: 22
selector:
service: disposable
type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
annotations:
metallb.universe.tf/allow-shared-ip: ip1
labels:
game.service: game1-exposer
name: game1-service
namespace: gamehost
spec:
ports:
- name: game1
port: 5678
targetPort: 5678
- name: voice
port: 2468
targetPort: 2468
selector:
game.service: game1
type: LoadBalancer
loadBalancerIP: your.public.ip.addr
The important part of the service spec is the annotation:
metadata:
annotations:
metallb.universe.tf/allow-shared-ip: ip1
As long as the metadata.annotations.allow-shared-ip
key is matched for everything using the same spec.loadBalancerIP
, every port will happily be opened up on that IP. It really is that easy.
Part 3: Automatic Certificates
This last part is a rough guide on how to use cert-manager to generate Let’s Encrypt SSL HTTPS certificates on-demand for any web services you’re using.
I’m doing this using the default ingress-nginx
that ships with the Rancher Kubernetes Engine. The one prerequisite is that you have working DNS to point to your web services’ addresses, since we’ll be using the normal HTTP challenge with Let’s Encrypt’s ACME service.
Creating a ClusterIssuer
Cert-manager lets you configure two kinds of issuers: the standard Issuer
resource that’s bound to a single namespace, or the ClusterIssuer
resource that’s available across all namespaces. As part of a normal cluster, you’d want to use namespaced issuers as part of good security, but for a single node where you’re the only tenant, it’s much easier to go with the latter type.
Cert-manager comes with a sample cluster issuer, but it’s better to create your own so you know what you’re doing. Simply throw together a YAML file to deploy one, following the official doco:
apiVersion: v1
items:
- apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
email: your@email.address
http01: {}
privateKeySecretRef:
key: ""
name: letsencrypt-cluster
server: https://acme-v02.api.letsencrypt.org/directory
When it’s done and working, you can use kubectl get clusterissuer -o yaml
to make sure it’s worked. It should output something resembling the following:
status:
acme:
uri: https://acme-v02.api.letsencrypt.org/acme/acct/47017662
conditions:
- lastTransitionTime: "2018-12-04T05:31:27Z"
message: The ACME account was registered with the ACME server
reason: ACMEAccountRegistered
status: "True"
type: Ready
Creating Virtual SSL Servers
This part took me a while to figure out—every time I deployed an ingress for a service, it never generated a certificate. The official doco for this seems to be missing (at least for me) one little piece of YAML that made it all work.
For example, getting this Ghost instance exposed externally required the following service and ingress spec:
---
apiVersion: v1
kind: Service
metadata:
labels:
ghost.service: ghost-web-service
name: ghost-web-service
namespace: ghost
spec:
ports:
- name: http
port: 2368
targetPort: 2368
selector:
ghost.service: ghost-web
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ghost-ingress
annotations:
kubernetes.io/ingress.class: nginx
certmanager.k8s.io/cluster-issuer: letsencrypt
certmanager.k8s.io/acme-challenge-type: http01
spec:
tls:
- hosts:
- kelsey.id.au
secretName: ghost-cert
rules:
- host: kelsey.id.au
http:
paths:
- path: /
backend:
serviceName: ghost-web-service
servicePort: 2368
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: ghost-cert
spec:
secretName: ghost-cert
dnsNames:
- kelsey.id.au
acme:
config:
- http01:
ingressClass: nginx
domains:
- kelsey.id.au
issuerRef:
name: letsencrypt
kind: ClusterIssuer
By creating an empty certificate
object that references the domain and ClusterIssuer
, and putting the ClusterIssuer
and ACME challenge type in the Ingress
, the issuer puts it all together and generates a certificate for you.
Closing Thoughts
While this guide gives you a pretty neat Kubernetes instance, the resulting ‘cluster’ has its shortcomings, some of which I’m still trying to find a good solution to. You need to ask questions like:
How do I provision persistent storage for my deployments?
Take a look at the built-in storage classes for Kubernetes. There’s not a lot on offer if you want to have flexible storage using the host’s baremetal disks.
How do I back-up and restore my data?
If you lose the node, you lose everything, but in this case it’s not as easy to set up a simple cron job to back up your data, let alone for things like etcd
. This is a topic that I’ll cover in a future article.
Otherwise, I hope this was useful for you. Feel free to reach out if there’s a topic you’d like me to cover as I continue learning.