feat(kOS): Add initial work for kOS

This patch provides the first release of kOS, a new Fedora CoreOS remix,
that is focused on the current setup of the SI infrastructure.

It basically implements all the little hacks and adjustments that are
currently active on the (existing) Fedora Server nodes running the
infrastructure and will provide a better replacement as soon as they are
put in production.

.gitlab-ci.yml

 ---
 include:
   - local: 'images/.utils/gitlab-ci.yaml'
+  - local: 'images/kos/.gitlab-ci.yaml'
   - local: 'charts/.utils/gitlab-ci.yaml'
 stages:
   - lint

docs/src/operating_system.md

-# Operating System
-
-For this setup Fedora is the Operating System of choice for multiple reasons. It provides both image-based installation methods (e.g. Fedora CoreOS) as well as package-based installation methods (e.g. Fedora Server) for many architectures and provides a modern and stable set of packages. Further it matches the developer machine OS, which helps with debugging and testing things locally before pushing them onto the deployments.
-
-## OS requirements
-
-The OS requirements to run the current setup are:
-
-- modern software versions
-- Kubeadm support
-- cri-o support
-- TPM-based LUKS encryption
-- SELinux support
-- (optional) cockpit integration
-- (optional) SSH access
-- automated updates
-
-## Setup script
-
-Currently the following script is used for set up:
-
-```
-#!/bin/bash
-
-# System upgrade
-dnf upgrade -y
-
-# Prepare Kubernetes
-dnf copr enable -y "sheogorath/kubernetes-1.22"
-dnf install -y kubernetes kubernetes-kubeadm
-
-# Install Crio as container engine
-dnf module enable -y cri-o:1.22
-dnf install -y cri-o cri-tools
-systemctl enable --now crio
-
-# Load kernel modules for Kubernetes and Calico
-modprobe br_netfilter
-modprobe wireguard
-cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
-br_netfilter
-wireguard
-EOF
-
-# Prepare sysctls for Kubernetes
-cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
-net.bridge.bridge-nf-call-ip6tables = 1
-net.bridge.bridge-nf-call-iptables = 1
-net.ipv4.ip_forward = 1
-EOF
-sysctl --system
-
-dnf install -y iptables
-
-# Disable systemd-resolved for CoreDNS
-rm -f /etc/resolv.conf
-cp /run/systemd/resolve/resolv.conf /etc/resolv.conf
-systemctl disable --now systemd-resolved
-
-# Prepare NetworkManager for Calico
-cat <<EOF | sudo tee /etc/NetworkManager/conf.d/calico.conf
-[keyfile]
-unmanaged-devices=interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:wireguard.cali
-EOF
-systemctl restart NetworkManager
-
-systemctl mask firewalld
-
-# Disable zram swap
-dnf remove -y zram-generator-defaults
-
-# Setup TPM encryption
-dnf install -y clevis-dracut
-clevis luks bind -d /dev/nvme0n1p3 tpm2 '{}'
-dracut -f
-
-# Install management interface
-dnf install -y cockpit cockpit-pcp
-
-reboot
-```
-
-*Be aware that this interactive due to TPM set up*
-
-## Filesystem Layout
-
-|Path|Filesystem|Size |Description|
-|----|----------|-----|-----------|
-|`/` |`xfs`     |15GiB|Root filesystem set up by Fedora Server layout.|
-|`/boot/efi`|`vfat`|600MiB|Filesystem for EFI, set up by Fedora Server layout.|
-|`/var/lib/containers` |`xfs` |50GiB|Filesystem for container images.|
-|`/var/lib/kubelet` |`xfs` |20GiB|Filesystem for kubelet related storage, such as `emptyDir`|
-|`/var/lib/longhorn`|`xfs`|varies|Filesystem for longhorn storage, this is used by longhorn to provide high-available storage across the clusters.|
-|`/var/lib/storage`|`xfs`|varies|Additional filesystem for longhorn storage, this is used by longhorn to provide high-available storage across the clusters.|
-
-## Setup addition SSD
-
-```bash
-# Setup LUKS recovery key
-cryptsetup luksFormat /dev/sda
-cryptsetup isLuks /dev/sda
-cryptsetup luksDump /dev/sda
-cryptsetup luksUUID /dev/sda
-cryptsetup luksOpen /dev/sda storage
-# Encrypt with local TPM
-clevis luks bind -d /dev/sda tpm2 '{}'
-mkfs.xfs /dev/mapper/storage
-echo "storage UUID=$(cryptsetup luksUUID /dev/sda) none discard,timeout=15" >> /etc/crypttab
-echo "/dev/mapper/storage   /var/lib/storage           xfs     defaults,x-systemd.device-timeout=0 0 0" >> /etc/fstab
-mkdir -p /var/lib/storage
-chcon -t container_file_t /var/lib/storage/
-mount -a
-df -h /var/lib/storage/
-# Make sure decryption on reboot works
-systemctl enable clevis-luks-askpass.path
-```

docs/src/operating_system.md

+../../images/kos/README.md
\ No newline at end of file

images/kos/.gitlab-ci.yaml

+### This separate build pipeline is required, because rpm-ostree and kaniko don't work together as expected.
+build-kos:
+  stage: build
+  image: quay.io/containers/podman:v4.3.1
+  variables:
+    CI_REGISTRY_IMAGE: quay.io/shivering-isles/kos
+  before_script:
+    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
+  script:
+    - podman build --pull
+      --label "org.opencontainers.image.source=$CI_PROJECT_URL/-/tree/$CI_COMMIT_SHA/images/kos"
+      --label "org.opencontainers.image.revision=$CI_COMMIT_SHA"
+      --label "org.opencontainers.image.title=kOS"
+      --label "quay.expires-after=12w"
+      -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA"
+      --format docker
+      images/kos
+    - podman push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA"
+    - podman push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA" "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
+  rules:
+    - if: '$CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"'
+      changes:
+        paths:
+          - images/kos/**/*
+        compare_to: main
+  tags:
+    - hetzner
+
+### This separate build pipeline is required, because rpm-ostree and kaniko don't work together as expected.
+build-kos-release:
+  stage: build
+  image: quay.io/containers/podman:v4.3.1
+  variables:
+    CI_REGISTRY_IMAGE: quay.io/shivering-isles/kos
+  before_script:
+    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
+  script:
+    - . images/kos/.release
+      export release
+    - podman build --pull
+      --label "org.opencontainers.image.source=$CI_PROJECT_URL/-/tree/$CI_COMMIT_SHA/images/kos"
+      --label "org.opencontainers.image.revision=$CI_COMMIT_SHA"
+      --label "org.opencontainers.image.title=kOS"
+      -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA"
+      --format docker
+      images/kos
+    - podman push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA"
+    - podman push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA" "$CI_REGISTRY_IMAGE:${release}"
+  rules:
+    - if: '$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE == "push"'
+      changes:
+        paths:
+          - images/kos/**/*
+  tags:
+    - hetzner    
\ No newline at end of file

images/kos/.release

+release=stable
\ No newline at end of file

images/kos/Dockerfile

+FROM quay.io/fedora/fedora-coreos:stable as coreos
+
+###
+## Repository modification
+###
+FROM quay.io/fedora/fedora:37 as repositories
+
+RUN dnf install -y 'dnf-command(config-manager)'
+
+RUN rm -rf /etc/yum.repos.d/
+
+COPY --from=coreos /etc/yum.repos.d/ /etc/yum.repos.d/
+
+RUN dnf config-manager  --save --setopt=fedora*.exclude=kubernetes*,cri-o*,cri-tools && \
+    dnf config-manager  --save --setopt=updates*.exclude=kubernetes*,cri-o*,cri-tools
+
+
+###
+## Build release metadata
+###
+FROM coreos as release-metadata
+
+COPY branding/release.sh /release.sh
+
+RUN ./release.sh
+
+###
+## OS build
+###
+FROM coreos
+
+# Add Branding
+COPY branding/fs/ /
+COPY --from=release-metadata /usr/lib/os-release /usr/lib/os-release
+
+# Install CPU boost config
+COPY cpu-boost/fs/ /
+
+# Kubernetes specific changes
+COPY kubernetes/fs/ /
+
+# Update repository configs
+COPY --from=repositories /etc/yum.repos.d/ /etc/yum.repos.d/
+
+# Remove unneeded libraries and tools
+RUN rpm-ostree override remove \
+      moby-engine \
+      zincati \
+      containerd && \
+    rpm-ostree cleanup -m && \
+    ostree container commit
+
+# Add custom Kubernetes repositories
+COPY repositories/kubernetes.repo /etc/yum.repos.d/
+
+# Fix kubernetes mountpoints
+RUN ln -s /opt/libexec/kubernetes/ /usr/libexec/kubernetes && \
+    ostree container commit
+
+# Install packages / manage services / commit
+RUN rpm-ostree install \
+      kubernetes-client \
+      kubernetes-node \
+      kubernetes-kubeadm \
+      cri-o \
+      cri-tools \
+      lm_sensors \
+      iscsi-initiator-utils \
+      cockpit \
+      clevis-dracut && \
+    rpm-ostree cleanup -m && \
+    systemctl mask rsyslog.service && \
+    systemctl mask systemd-resolved.service && \
+    rm /etc/resolv.conf && \
+    systemctl enable crio && \
+    systemctl enable clevis-luks-askpass.path && \
+    systemctl enable cpu-boost-disable.timer && \
+    systemctl enable iscsid && \
+    systemctl enable chronyd && \
+    ostree container commit
\ No newline at end of file

images/kos/Makefile

+include ../.utils/container-build.mk
+include ../../utils/help.mk
+
+
+ICON_SIZES := 16 22 24 32 36 48 96 256
+
+pre-build:
+	@echo do some stuff before the docker build
+
+post-build:
+	@echo do some stuff after the docker build
+
+
+logo-sizes: ## Generate all logo icon sizes
+	$(foreach var,$(ICON_SIZES),mkdir -p "logo/fs/usr/share/icons/hicolor/$(var)x$(var)/apps";)
+	$(foreach var,$(ICON_SIZES),ffmpeg -i logo/logo-square.png -vf scale="$(var):$(var)"  "logo/fs/usr/share/icons/hicolor/$(var)x$(var)/apps/kos-logo-icon.png";)
+	

images/kos/README.md

+kOS
+===
+
+kOS (read 'chaos') is a simple OS based on CoreOS with the applied modifications for this repositories needs. It'll provide an easy way to reliably update the base OS of the cluster.
+
+Concept
+---
+
+The base setup is simple: A general CoreOS setup on a host, which is then rebased to this container image using [the new, experimental coreos layering](https://github.com/coreos/enhancements/blob/main/os/coreos-layering.md). This should provide a solid base for the OS, and allows to use the regular container build pipeline to update tools and the OS itself.
+
+OS requirements
+---
+
+The OS requirements to run the current setup are:
+
+- [x] modern software versions
+- [x] Kubeadm support
+- [x] cri-o support
+- [x] TPM-based LUKS encryption
+- [x] SELinux support
+- [x] cockpit integration
+- [x] SSH access
+- [x] automated updates
+
+Why kOS?
+---
+
+kOS provides the needed flexibility without layering new packages into an image, which allows proper image validation in CI while reducing the risk of potential issues due to package dependency conflicts and alike, it also allows proper adjustment of RPM package sources.
+
+Other than upstream Fedora CoreOS, it is opinionated towards Kubernetes as orchestrator and prepares the OS accordingly with `cri-o` as container engine being prepared for specific Kubernetes versions along with `kubeadm` and the `kubelet` that come preinstalled.
+
+What does the k in kOS stand for? Kubernetes?
+---
+
+No. 🤷 I'm not sure yet, what it actually stands for, but it sounds nice, doesn't it?
\ No newline at end of file

images/kos/branding/release.sh

+#!/bin/sh
+
+. /etc/os-release
+
+KOS_VERSION="${KOS_VERSION:-${VERSION_ID}.$(date +%Y%m%d).$(date +%H%M)}"
+KOS_BUILD_ID="${KOS_BUILD_ID:-$(date +%Y%m%d).$(date +%H%M)}"
+
+
+cat > /usr/lib/os-release <<EOF
+NAME="kOS"
+VERSION="${KOS_VERSION}"
+ID=kos
+ID_LIKE="fedora"
+VERSION_ID=${VERSION_ID}
+VERSION_CODENAME=""
+LOGO=kos-logo-icon
+PLATFORM_ID="platform:f37"
+PRETTY_NAME="kOS ${KOS_VERSION} (CoreOS)"
+ANSI_COLOR="0;38;2;60;110;180"
+DEFAULT_HOSTNAME="kOS-node"
+HOME_URL="https://git.shivering-isles.com/shivering-isles/infrastructure-gitops"
+DOCUMENTATION_URL="https://git.shivering-isles.com/shivering-isles/infrastructure-gitops"
+SUPPORT_URL="https://shivering-isles.com/matrix"
+BUG_REPORT_URL="https://git.shivering-isles.com/shivering-isles/infrastructure-gitops/-/issues"
+OSTREE_VERSION="${KOS_VERSION}"
+BUILD_ID=${KOS_BUILD_ID}
+EOF
\ No newline at end of file

images/kos/cpu-boost/fs/etc/systemd/system/cpu-boost-disable.service

+[Unit]
+Description=Disable overclocking on CPU
+
+[Service]
+Type=oneshot
+ExecStart=/bin/sh -c "echo 0 > /sys/devices/system/cpu/cpufreq/boost"
\ No newline at end of file