This patch enables explicit pod security standards on all infrastructure
namespaces.

This patch prepares the switch to Kubernetes 1.24.x which switches to
PSS instead of PSP. Therefore it's a good start to prepare our most
important namespaces with the relevant labels to allow Pods to use
privileged runtime features.

References:
https://kubernetes.io/docs/concepts/security/pod-security-standards/
https://v1-23.docs.kubernetes.io/docs/concepts/security/pod-security-policy/
https://v1-23.docs.kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/

This patch Upgrades calico to version 3.23.0, which is a complicated
endeavour since it switches the helm release namespaces from default to
tigera-operator.

Besides the regular upgrade tasks, this reqires some explicit adjusting
of helm annotations and flux labels, in order to convince the cluster,
that's how it always has been.

The following tasks need to be done:

Before you start
---

Disable flux:

```
kubectl scale deployment -n flux-system source-controller --replicas 0
kubectl scale deployment -n flux-system helm-controller --replicas 0
kubectl scale deployment -n flux-system kustomize-controller --replicas 0
```

The upgrade
---

Push/merge this patch. (!!!)

Update helm release annotations:
```
kubectl patch installation default --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
kubectl patch apiserver default --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
kubectl patch podsecuritypolicy tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
kubectl patch -n tigera-operator deployment tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
kubectl patch -n tigera-operator serviceaccount tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
kubectl patch clusterrole tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
kubectl patch clusterrolebinding tigera-operator tigera-operator --type=merge -p '{"metadata": {"annotations": {"meta.helm.sh/release-namespace": "tigera-operator"}}}'
```

Patch flux labels:
```
kubectl patch installation default --type=merge -p '{"metadata": {"labels": {"helm.toolkit.fluxcd.io/namespace": "tigera-operator"}}}'
kubectl patch apiserver default --type=merge -p '{"metadata": {"labels": {"helm.toolkit.fluxcd.io/namespace": "tigera-operator"}}}'
kubectl patch podsecuritypolicy tigera-operator --type=merge -p '{"metadata": {"labels": {"helm.toolkit.fluxcd.io/namespace": "tigera-operator"}}}'
kubectl patch -n tigera-operator deployment tigera-operator --type=merge -p '{"metadata": {"labels": {"helm.toolkit.fluxcd.io/namespace": "tigera-operator"}}}'
kubectl patch -n tigera-operator serviceaccount tigera-operator --type=merge -p '{"metadata": {"labels": {"helm.toolkit.fluxcd.io/namespace": "tigera-operator"}}}'
kubectl patch clusterrole tigera-operator --type=merge -p '{"metadata": {"labels": {"helm.toolkit.fluxcd.io/namespace": "tigera-operator"}}}'
kubectl patch clusterrolebinding tigera-operator tigera-operator --type=merge -p '{"metadata": {"labels": {"helm.toolkit.fluxcd.io/namespace": "tigera-operator"}}}'
```

Remove flux labels from namespace:

```
kubectl label namespace tigera-operator helm.toolkit.fluxcd.io/namespace-
```

Get values:

```
helm get values -n default calico > values.yaml
```

Install calico:

```
helm repo add projectcalico https://projectcalico.docs.tigera.io/charts
helm install calico projectcalico/tigera-operator --version v3.23.0 --namespace tigera-operator --values values.yaml
```

Migrate flux helmrelease:

```
kubectl apply -n tigera-operator -f bootstrap/calico/release.yaml
kubectl patch helmrelease calico --type=json -p="[{'op': 'remove', 'path': '/metadata/finalizers'}]" -n default
kubectl delete helmrelease -n default calico
```

Delete old helm install:

```
kubectl delete secret -n default -l name=calico -l owner=helm
```

Starting flux again
---

```
kubectl scale deployment -n flux-system source-controller --replicas 1
kubectl scale deployment -n flux-system helm-controller --replicas 1
kubectl scale deployment -n flux-system kustomize-controller --replicas 1
```

References:
https://projectcalico.docs.tigera.io/archive/v3.23/release-notes/

Turns out the problem was based in the wiretrustee usage, that confused
calico, resulting in double-tunneling.

This reverts commit 8a519732.

With wireguard it's suggested to set the MTU to 60 bytes less than the
general MTU of the interfaces. the general interfaces have a standard
MTU of 1500.

References:
https://projectcalico.docs.tigera.io/networking/mtu#determine-mtu-size

This patch adds a broader networkpolicy for system-upgrades namespace,
which should ensure network access within the namespace.

It's the 3rd update of kyverno and each time, things break in minor
version. This is no modi operandi for this setup. Things are supposed to
be stable and solid to work with. Kyverno is too unstable for this
use-case.

This time the installation of the pods failed due to wrong deployment
names. This is nothing we change or adjust.

Further the removal doesn't have any major impact on the platform, since
network policies are already deployed via gitops from the `shared/`
directory.

BREAKING CHANGE: Removing kyverno and related CRDs/APIs.

Turns out, doesn't work as expected.

This reverts commit c11ca652.

Turns out kyverno doesn't like its new version, since the deployments
have the wrong name. Needs some further investigation.

This reverts commit fca00b09.

This patch replaces the registry used to pull calico images with quay.io
since docker hub complaints about rate limits

Currently kyverno gets stuck in 'terminating' state when draining a
node, this is due unavailability for the addmission webhook that it
provides itself.

This patch pushes the replica count up and adds a PDB, which should help
to make sure it doesn't happen again.

To (temporarily) fix the situation you can delete the the
validationwebhookconfigurations for kyverno until the pod is terminated
and rollout kyverno with a higher replica count again.