In this edition: longer lasting data!

If you're coming here from part one, you should have a single-node Rancher Kubernetes node just waiting to be explored. This part will go into more detail about some important components and how to use them to your advantage.

Storage in a Nutshell

Kubernetes, just like Docker, is stateless. As soon as you stop a pod, the data it contained is gone forever. This is only really a bad thing if you're running something that needs to have persistent storage, like a database or wiki.

Keep in mind that most micro-services are designed to be configured through config-maps, secrets and environment variables without needing to store data, but there are easy ways to provide more permanent storage.

Persistent Volumes

Storage in Kubernetes is centered around the idea of Persistent Volumes being able to fulfil claims for storage. There are many types of built-in PVs that can be useful depending on the architecture and cloud environment of your system, such as NFS or vSphere. Take a look at the official docs to see a list of all supported PV types.

As this is a standalone server that's not a part of the cluster (and more on this in another part), you probably won't have access to many of these offerings. Luckily, you can specify that a PV should simply use the local server's storage for the files. For instance:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: testing-pv
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/testing-pv"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: testing-pvc
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

This will create a 5GB volume under /mnt of the server, and a PVC that will be bound to that. You can then use testing-pvc as a PVC for a container.

Storage Provisioners

The above example isn't ideal in the real world. Every time you need to deploy a workload, you'll have to manually configure additional storage for each PVC that needs to be fulfilled.

Kubernetes has the concept of storage classes, which are a way to provide different kinds of storage to your cluster.

When you have a class set up, you can use dynamic volume provisioning to automatically provision just the right amount of storage that any PVCs require. By setting the reclaimPolicy in the storage class, you can also ensure that PVCs that are destroyed will clean up after themselves by removing the underlying PV and storage on-disk.

Example: NFS Provisioner

As this describes a single-node cluster, it makes sense to use the local disk as a storage source for containers running on it. There is a dead-simple NFS provisioner available that does the job well.

Installing via Rancher

As the previous section dealt with getting Rancher up and running, you should be able to follow along with deploying the server via a Helm chart in the GUI to make the process extremely easy:

1. Log into the Rancher UI and ensure you're at the Global scope:

2. Click on Catalogs, and make sure Helm Stable is set to Enabled
3. Drop into the Cluster: local scope and click on Storage ▶️ Persistent Volumes
4. Add a new volume
5. Choose Local Node Path and a sane path (e.g. /mnt/ganesha), ensuring the directory is created
6. Drop into the System scope and click Catalog Apps at the top
7. Launch, and select nfs-provisioner from Library
8. Enable persistent volume, and ensure the volume size matches the above PV

Watch the deployment and ensure everything sets itself up properly (it should!)

Dynamic Provisioning in Action

Now that a dynamic provisioner and default storage class exist, the above example can be simplified to this:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: testing-pvc
spec:
  storageClassName: nfs-provisioner
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Since the storage class is specified (or even if it isn't when it's set as the default), Kubernetes will automatically create a PV on the NFS "server" to match the PVC created. Pretty slick!

Backups

So, you have containers being able to tap into the local storage on your node. Great! But what happens if that node, or its disk, fails?

Stash and restic

Luckily, AppsCode have open-sourced a product called Stash: a backup system for Kubernetes that's built on top of the excellent restic binary (which you should check out for your other projects).

In a nutshell, Restic is a system that provides a very, very simple way to back up to a wide range of destinations; from simple rsync locations to S3 or OpenStack buckets. Stash builds on top of this by creating sidecars in your deployments that access the persistent storage of your containers while they're running, which get snapshotted, deduplicated, encrypted and sent offsite.

Installing Stash

Just like the above nfs-provisioner, Stash provides a Helm chart to make it easy to install in Kubernetes. In this example I'll step through doing it via the command line instead.

There's not much to the process: these steps are largely taken from the official guide:

helm repo add appscode https://charts.appscode.com/stable/
helm repo update
helm install appscode/stash --name stash-operator --version 0.8.3 --namespace kube-system

It's very important that this is installed into the kube-system namespace. If you've followed the last part, you will most certainly have Role-based Access Control in effect, so Stash needs to be able to create its own role bindings to operate.

Configuring Backups

Prerequisites

First and foremost, you are going to need a location to back up to. The restic docs contain a list of all supported back-ends.

Secondly, you'll need your API credentials (e.g. AWS_SECRET_ACCESS_KEY) for your chosen service. I'm using BackBlaze/B2 in this example, as I've configured it that way for my services.

Choosing a Target

Let's take a look at an example workload to find out what's needed to get Stash running. You can pull out the deployment's YAML with kubectl get -n your-namespace deployment foobar -o yaml if you don't have it handy.

For instance, my Gitea deployment looks something like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    git.service: gitea-web
  name: gitea-web
  namespace: git
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        git.service: gitea-web
    spec:
      containers:
      - image: gitea/gitea:1.6
        name: gitea-web
        ports:
        - containerPort: 3000
        - containerPort: 22
        resources: {}
        volumeMounts:
        - mountPath: /data
          name: gitea-web-claim0
      restartPolicy: Always
      volumes:
      - name: gitea-web-claim0
        persistentVolumeClaim:
          claimName: gitea-web-claim0

As shown in the YAML, the deployment is mounting gitea-web-claim0 to /data, and is running in the git namespace. The workload can be found using the label git.service: gitea-web.

Telling Stash What to Do

Next, you should make a secret containing your API keys and a password used to encrypt your repository:

pwgen -B 16 1 | tr -d '\n' > RESTIC_PASSWORD
echo -n 'your_b2_account_id' > B2_ACCOUNT_ID
echo -n 'your_b2_account_key' > B2_ACCOUNT_KEY
kubectl create secret generic -n git gitea-restic-secret \
    --from-file=./RESTIC_PASSWORD \
    --from-file=./B2_ACCOUNT_ID \
    --from-file=./B2_ACCOUNT_KEY

It's very important that all trailing whitespace is removed from these files.

You should then create the spec for the restic that Stash will look after:

apiVersion: stash.appscode.com/v1alpha1
kind: Restic
metadata:
  name: gitea-backups
  namespace: git
spec:
  selector:
    matchLabels:
      git.service: gitea-web
  fileGroups:
  - path: /data
    retentionPolicyName: 'keep-last-14'
  backend:
    b2:
      bucket: git-data
    storageSecretName: gitea-restic-secret
  schedule: '30 1 * * *'
  volumeMounts:
  - mountPath: /data
    name: gitea-web-claim0
  retentionPolicies:
  - name: 'keep-last-14'
    keepLast: 14
    prune: true

It's in the restic specification that you can dictate the schedule and number of backups to keep. You can be creative here, and have multiple schedules running multiple retention policies. This is also where you specify that the destination is B2, and that the bucket name is git-data.

Be warned: applying the Stash spec will cause the running container to restart as the sidecar is created!

To finish, apply your restic spec with kubectl apply -n git -f git-restic.yaml.

Confirming Your Backups

Simply run the following:

kubectl get -n git repo          
NAME                   BACKUP-COUNT   LAST-SUCCESSFUL-BACKUP   AGE
deployment.gitea-db    1349           51m                      56d
deployment.gitea-web   1342           21m                      56d

Conclusion

By adding persistent storage into your Kubernetes node, and ensuring that the data is properly backed up, you can get a lot more mileage out of just a single node. While I would certainly recommend against running any kind of production workloads on just one server, for personal services this is a great compromise and ensures your data is at least kept safe.

In future posts I'll be exploring more ways to add security and redundancy to Kubernetes, but for now I hope you find this information helpful.

A working Kubernetes installation in a single lunch break!

This is a write-up of my project to get Kubernetes working as a single-node, general purpose install on a baremetal server provided by OVH.

It took me a long time to find the right information on how to do this, as many of the components are new with little documentation provided.

DISCLAIMER
A single-node instance like the one I'm about to describe is incredibly useful for a personal system or learning tool, but should not be confused with something that's acceptable for production workloads.

Background

Why are you doing this?

I've had a few low-end virtual servers for a while in Sydney and Warsaw that I've been using for various tasks, but I'd been straddling the limit of their (tiny) resources for quite some time.

To compound the frustration, I've been using OVH's OpenStack object storage containers as large virtual disks (thank you S3QL). Imagine using a server based in Sydney while its main data disk is over http on the other side of the world.

Regardless, it was time for a change, and the recent introduction of some temptingly cheap local baremetal servers has swayed me.

My Needs

I had to find a solution that would tick some boxes:

• It should have at least 200GiB of space
• It should be easily backed up (especially my Nextcloud data)
• It should be easy to partition different workloads and control their resource usage
• It should be secure
• It should be easy to manage and automate (I'm sick of the traditional special-snowflake server method)
• It should allow multiple domains and TCP services on the same static IP

Kubernetes in an Hour

Software I Used

• RancherOS: a ridiculously small distro where all the things are Dockerised
• RKE: the Rancher Kubernetes Engine
• Rancher: lubricant for Kubernetes
• MetalLB: Google's solution to software load-balancing
• Helm and Tiller: a sort-of package manager for Kubernetes

Part 1: Preparation

I decided to use RancherOS for the server, as everything running on the system is dockerised and it's designed for Kubernetes. Unfortunatelyt this is not one of the standard OS offerings by OVH.

OS Installation

Luckily, OVH do give you a Java Web Start KVM for your (Supermicro) server with the ability to connect an ISO to the virtual disc drive. The installation ISO for RancherOS is only about 100MiB, so this is absolutely perfect.

RancherOS uses the standard cloud-config YAML referencing to ingest its initial settings during installation. All you need is something similar to this:

#cloud-config
hostname: your.public.dns.address

rancher:
  network:
    interfaces:
      eth0:
        dhcp: true
    dns:
     nameservers:
     # Using Google's DNS
     - 8.8.8.8
     - 8.8.4.4

ssh_authorized_keys:
  - ssh-ed25519 AAAAblahblahblahtypicalkey email@add.ress

Once RancherOS is up and running, you can use some of the tricks in this helpful Gist to get a software RAID running in RancherOS:

# Create empty partition tables on each disk with G
sudo fdisk /dev/sda
sudo fdisk /dev/sdb

# Install once on each disk
sudo ros install -i rancher/os:v1.5.0 -t gptsyslinux -c cloud-config.yml -a "rancher.state.mdadm_scan" -d /dev/sda --no-reboot
sudo ros install -i rancher/os:v1.5.0 -t gptsyslinux -c cloud-config.yml -a "rancher.state.mdadm_scan" -d /dev/sdb --no-reboot

# Configure RAID
sudo mdadm --create /dev/md0 --level=1 -- metadata=1.0 --raid-devices=2 /dev/sda1 /dev/sdb1

# Final sanity checks
sudo fsck /dev/md0
sudo resize2fs /dev/md0
sudo fsck /dev/md0

sudo reboot

Kubernetes Initialisation

The Rancher Kubernetes Engine is painless to set up and get going. I'll step through performing a single-node install using Let's Encrypt as the certificate manager, but veering off the path a little bit to make things work seamlessly on the node.

First off you need to tell RKE to install everything it needs, using some YAML spec. There are a number of things you can tag and configure, though for my deployment I left everything at its default. Create your rke.yaml file for deployment:

ignore_docker_version: true
ssh_agent_auth: true
nodes:
  - address: public.dns.add.ress
    hostname_override: public.dns.add.ress
    user: rancher
    role: [controlplane,worker,etcd]

network:
    plugin: canal
    options:
        canal_iface: eth0

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 24h

As long as you're using ssh-agent for your SSH key, you don't have to specify the actual key you're using.

Next up, install Kubes and do some basic configuration:

rke up --config rke.yaml
export KUBECONFIG=$(pwd)/kube_config_rke.yaml
kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
helm init --service-account tiller
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm install stable/cert-manager --name cert-manager --namespace kube-system
helm install rancher-stable/rancher --name rancher --namespace cattle-system --set hostname=your.public.dns.name

You might need to space the above commands out just to ensure that everything is deploying properly, or use kubectl to check the rollout status of each.

You're done, and should have a ready-to-go basic Kubernetes node. To make it truly useful, you'll need to get down and dirty with some additional software.

Part 2: MetalLB

As this installation is only single-node, and I wanted to make sure that I could run all the services I needed to, I had to ensure that I had a way to share ports on the public IP address without resorting to hand-crafted iptables rules.

Why Use a Load Balancer?

It might seem counter-intuitive to run a load balancer service on a single node (and single IP) installation, but you need to know about how Kubernetes networking works for non-HTTP services. For the sake of a single node, you essentially have three options:

• ClusterIP as part of a service for internal-only access (just like linking Docker containers)
• NodePort to directly expose the service on your Kubernetes node, on a random port from 30000 to 32767
• As part of a DaemonSet... which I will not go into

As a DaemonSet is not a recommended way to go about simple deployments, none of the options are especially useful for exposing a typical port (say, IMAP).

Enter MetalLB, Google's software implementation of load-balancing for Kubernetes. Exposing a service to the world is as simple as:

1. Installing MetalLB
2. Telling MetalLB which IPs it can use
3. Telling a service to expose a port on a particular IP

Luckily, the load balancer implementation supports sharing multiple services with unique ports on a single IP address. This is especially useful when you only have a single IP.

Setting it Up

Create a YAML spec containing the IP address of your service to instantiate the default pool:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:  
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - your.public.ip.addr

Then install and apply:

kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.7.3/manifests/metallb.yaml
kubectl apply -n metallb-system lb-pool.yaml

Exposing Workloads

Following the official documentation, getting it to work is as easy as assigning your service to the specific IP and putting in an IP sharing annotation.

Take a look at these two sample services: a typical bastion host and a generic game server with a voice port. MetalLB will expose multiple ports on the same IP address when you set the services up:

apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/allow-shared-ip: ip1
  name: bastion-exposer
  namespace: bastion
spec:
  loadBalancerIP: your.public.ip.addr
  ports:
  - name: ssh
    port: 1234
    protocol: TCP
    targetPort: 22
  selector:
    service: disposable
  type: LoadBalancer
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    metallb.universe.tf/allow-shared-ip: ip1
  labels:
    game.service: game1-exposer
  name: game1-service
  namespace: gamehost
spec:
  ports:
  - name: game1
    port: 5678
    targetPort: 5678
  - name: voice
    port: 2468
    targetPort: 2468
  selector:
    game.service: game1
  type: LoadBalancer
  loadBalancerIP: your.public.ip.addr

The important part of the service spec is the annotation:

metadata:
  annotations:
    metallb.universe.tf/allow-shared-ip: ip1

As long as the metadata.annotations.allow-shared-ip key is matched for everything using the same spec.loadBalancerIP, every port will happily be opened up on that IP. It really is that easy.

Part 3: Automatic Certificates

This last part is a rough guide on how to use cert-manager to generate Let's Encrypt SSL HTTPS certificates on-demand for any web services you're using.

I'm doing this using the default ingress-nginx that ships with the Rancher Kubernetes Engine. The one prerequisite is that you have working DNS to point to your web services' addresses, since we'll be using the normal HTTP challenge with Let's Encrypt's ACME service.

Creating a ClusterIssuer

Cert-manager lets you configure two kinds of issuers: the standard Issuer resource that's bound to a single namespace, or the ClusterIssuer resource that's available across all namespaces. As part of a normal cluster, you'd want to use namespaced issuers as part of good security, but for a single node where you're the only tenant, it's much easier to go with the latter type.

Cert-manager comes with a sample cluster issuer, but it's better to create your own so you know what you're doing. Simply throw together a YAML file to deploy one, following the official doco:

apiVersion: v1
items:
- apiVersion: certmanager.k8s.io/v1alpha1
  kind: ClusterIssuer
  metadata:
    name: letsencrypt
  spec:
    acme:
      email: your@email.address
      http01: {}
      privateKeySecretRef:
        key: ""
        name: letsencrypt-cluster
      server: https://acme-v02.api.letsencrypt.org/directory

When it's done and working, you can use kubectl get clusterissuer -o yaml to make sure it's worked. It should output something resembling the following:

status:
acme:
    uri: https://acme-v02.api.letsencrypt.org/acme/acct/47017662
conditions:
- lastTransitionTime: "2018-12-04T05:31:27Z"
    message: The ACME account was registered with the ACME server
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

Creating Virtual SSL Servers

This part took me a while to figure out—every time I deployed an ingress for a service, it never generated a certificate. The official doco for this seems to be missing (at least for me) one little piece of YAML that made it all work.

For example, getting this Ghost instance exposed externally required the following service and ingress spec:

---
apiVersion: v1
kind: Service
metadata:
  labels:
    ghost.service: ghost-web-service
  name: ghost-web-service
  namespace: ghost
spec:
  ports:
  - name: http
    port: 2368
    targetPort: 2368
  selector:
    ghost.service: ghost-web
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ghost-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt
    certmanager.k8s.io/acme-challenge-type: http01
spec:
  tls:
  - hosts:
    - kelsey.id.au
    secretName: ghost-cert
  rules:
  - host: kelsey.id.au
    http:
      paths:
      - path: /
        backend:
          serviceName: ghost-web-service
          servicePort: 2368
---
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: ghost-cert
spec:
  secretName: ghost-cert
  dnsNames:
  - kelsey.id.au
  acme:
    config:
    - http01:
        ingressClass: nginx
      domains:
      - kelsey.id.au
  issuerRef:
    name: letsencrypt
    kind: ClusterIssuer

By creating an empty certificate object that references the domain and ClusterIssuer, and putting the ClusterIssuer and ACME challenge type in the Ingress, the issuer puts it all together and generates a certificate for you.

Closing Thoughts

While this guide gives you a pretty neat Kubernetes instance, the resulting 'cluster' has its shortcomings, some of which I'm still trying to find a good solution to. You need to ask questions like:

How do I provision persistent storage for my deployments?

Take a look at the built-in storage classes for Kubernetes. There's not a lot on offer if you want to have flexible storage using the host's baremetal disks.

How do I back-up and restore my data?

If you lose the node, you lose everything, but in this case it's not as easy to set up a simple cron job to back up your data, let alone for things like etcd. This is a topic that I'll cover in a future article.

Otherwise, I hope this was useful for you. Feel free to reach out if there's a topic you'd like me to cover as I continue learning.