Skip to content
Snippets Groups Projects
Sheogorath's avatar
Sheogorath authored
This patch adds the initial structure for the documentation of the
Shivering-Isles GitOps Infrastructure repository. It uses mdbook to
create an easy to read documentation.

It uses mdbook, because it's an easy single-binary tool to generate
documentation. This fits the tooling schema of koolbox quite well and
can be easily integrated into the automatic updates toolchain already
used. Further it uses markdown, which allows easy writing of further
documentation and re-use of existing documentation.

The documentation structure re-uses various existing READMEs from
components. These might be replaced in the long run.

It also lays out future plans for documentation, which is currently
still on the TODO list. This should help to keep the overall structure
in mind and provide the general direction.

References:
https://rust-lang.github.io/mdBook/index.html
bfdafa63
History

Shivering-Isles GitOps Infrastructure

This repository contains the Kubernetes objects that are synced and managed by flux in order to be deployed.

Usage

Finally in order to boostrap fluxcd in your cluster. For SI-GitLab this would look like this:

export GITLAB_TOKEN=<project access token able to write the API and repository>
flux bootstrap gitlab \
  --hostname=git.shivering-isles.com \
  --ssh-hostname=git.shivering-isles.com:2222 \
  --ssh-key-algorithm ed25519 \
  --owner=<your user / team> \
  --repository=<your repository name> \
  --path=clusters/<your cluster name>

Ideas & ToDo's

This toolchain is still under development. Before it will be used in production there are still some things left to do:

  • Buy hardware for the project.
  • Provide CLI container that contains all tools.
  • Automate overlay network deployment (calico)
  • Use encrypted overlay network (calico+wireguard)
  • Automate cluster monitoring deployment (kube-prometheus)
  • Automate ingress-controller deployment (ingress-nginx)
  • Automate policy enforcement (kyverno) deployment
  • Encrypt root filesystems for all nodes (LUKS + clevis)
  • Enforce SELinux on the deployed machines
  • Automate system upgrades using Kubernetes (system-upgrade-controller)
  • Automate system configuration using Kubernetes (system-upgrade-controller)
  • Provide an fully encrypted (handled on host level) storage class (longhorn)
  • Deploy cert-manager
  • Deploy credentials for cert-manager
  • Automate ingress-controller default certificate deployment
  • Add encrypted deployment instructions (SOPS + fluxcd)
  • Integrate Renovatebot with this repository to manage updates.
  • Automate Kubernetes upgrades
  • Deploy kubelet with proper certificates
  • Automate ingress-controller configuration for proxy-protocol
  • Document usage and thoughts in repository and blog posts
  • Automate flux OpenPGP bootstrap
  • Migrate apps to gitops and Kubernetes
  • Move to immutable base-system

Assumption

Building smaller, more-minimalistic, plain Kubernetes clusters will be cheaper than OpenShift with OKD and more stable since etcd doesn't have to write a ton of data to disk and there aren't two API server running that take up to 3GB of RAM per master node.

The goal is still to manage everything GitOps style, but more iterative and slowly grinding the way forward before clusters will become productive.

Original assumptions / Lessons Learned

This repository is focused on a setup based on OpenShift, OKD to be specific. Therefore some installations and settings might be based on the expectation of OKD's default setup instead of going the plain Kubernetes way of inventing everything ourselves.

Sadly this previous assumption didn't hold up. OpenShift on Hetzner Cloud resulted in quite annoying downtimes during upgrades. While the origin of the problem was not fully determined, it was proven that severe spikes in etcd writing fsyncs of up to 600ms did play a major role in it.

Tools

To handle things properly, try to get the following tools (all included in koolbox):

  • kubectl
  • flux
  • sops (for secret handling)
  • helm (just for sake of completeness and validation)
  • terraform
  • make
  • git