diff --git a/rfcs/0003-multi-tenancy-mode/README.md b/rfcs/0003-multi-tenancy-mode/README.md new file mode 100644 index 0000000000000000000000000000000000000000..20e22f75470dd3853de39baec842b41b8782992c --- /dev/null +++ b/rfcs/0003-multi-tenancy-mode/README.md @@ -0,0 +1,197 @@ +# RFC-0003 Flux Multi-Tenancy Mode + +## Summary + +For multi-tenant environments, we want to offer an easy way of configuring Flux to enforce tenant isolation +(as defined by the Soft Multi-Tenancy model from RFC-0001). + +When running in the multi-tenant mode, Flux will lock down access to sources (as defined by RFC-0002), +and will use the tenant service account instead of defaulting to `cluster-admin`. + +From an end-user perspective, the multi-tenancy mode means that: + +- Platform admins have to create a Kubernetes service account and RBAC in each namespace where + Flux performs source-to-cluster reconciliation on behalf of tenants. + By default, Flux will have no permissions to reconcile the tenants sources onto clusters. +- Source owners have to specify with which tenants they wish to share their sources. + By default, nothing is shared between tenants. + +## Motivation + +As of Flux v0.23.0, configuring Flux for soft multi-tenancy requires additional tooling such as Kyverno or OPA Gatekeeper +to overcome caveats such as: +- Flux does not require for a service account name to be specified on Flux custom resources that perform + source-to-cluster reconciliation. When a service account is not specified, Flux defaults to cluster-admin. +- Flux does not prevent tenants from accessing known sources outside of their namespaces. +- Flux does not prevent tenants from subscribing to other tenant's events. + +Flux users have been asking for a way to enforce multi-tenancy +without having to use 3rd party validation webhooks e.g. +[fluxcd/kustomize-controller#422](https://github.com/fluxcd/kustomize-controller/issues/422). + +### Goals + +- Enforce service account impersonation for source-to-cluster reconciliation. +- Enforce ACLs for cross-namespace access to sources. + +### Non-Goals + +- Enforce tenant's workload isolation with network policies and pod security standards as described + [here](https://kubernetes.io/blog/2021/04/15/three-tenancy-models-for-kubernetes/#security-considerations). + +## Proposal + +### User Stories + +#### Story 1 + +> As a platform admin, I want to install Flux with lowest privilege/permission level possible. + +#### Story 2 + +> As a platform admin, I want to give tenants full control over their assigned namespaces. +> So that tenants could use their own repositories and manager the app delivery with Flux. + +#### Story 3 + +> As a platform admin, I want to prevent tenants from changing the cluster-wide configuration. +> If a tenant adds to their repository a cluster-scoped resource such as a namespace or cluster role, +> Flux should reject the change and notify the tenant that this operation is not allowed. + +### Multi-tenant Bootstrap + +When bootstrapping Flux, platform admins should have the option to lock down Flux for multi-tenant environments e.g.: + +```shell +flux bootstrap --security-profile=multi-tenant +``` + +The security profile flag accepts two values: `single-tenant` and `multi-tenant`. +Platform admins may switch between the two modes at any time, either by rerunning bootstrap +or by patching the Flux manifests in Git. + +The `multi-tenant` profile is just a shortcut to setting the following container args in the Flux deployment manifests: + +```yaml + containers: + - name: manager + args: + - --default-service-account=flux + - --enable-source-acl=true +``` + +And for disabling cross-namespace references when using the notification API: + +```yaml +kind: Deployment +metadata: + name: notification-controller +spec: + template: + spec: + containers: + - name: manager + args: + - --same-namespace-refs=true +``` + +When running in the `multi-tenant` mode, Flux behaves differently: + +- The source-to-cluster reconciliation no longer runs under the service account of + the Flux controllers. The controller service account, is only used to impersonate + the service account specified in the Flux custom resources (`Kustomizations`, `HelmReleases`). +- When no service account name is specified in a Flux custom resource, + a default will be used e.g. `system:serviceaccount:<tenant-namespace>:flux`. +- When a Flux custom resource (`Kustomizations`, `HelmReleases`, `ImagePolicies`, `ImageUpdateAutomations`) + refers to a source in a different namespace, access is granted based the source access control list. + If no ACL is defined for a source, cross-namespace access is denied. +- When a Flux notification (`Alerts`, `Receivers`) + refers to a resource in a different namespace, access is denied. + +### Tenants Onboarding + +When onboarding tenants, platform admins should have the option to assign namespaces, set +permissions and register the tenants repositories onto clusters in a declarative manner. + +The Flux CLI offers an easy way of generating all the Kubernetes manifests needed to onboard tenants: + +- `flux create tenant` command generates namespaces, service accounts and Kubernetes RBAC + with restricted access to the cluster resources, given tenants access only to their namespaces. +- `flux create secret git` command generates SSH keys used by Flux to clone the tenants repositories. +- `flux create source git` command generates the configuration that tells Flux which repositories belong to tenants. +- `flux create kustomization` command generates the configuration that tells Flux how to reconcile the manifests found in the tenants repositories. + +All the above commands have an `--export` flag for generating the Kubernetes resources in YAML format. +The platform admins should place the generated manifests in the repository that defines the cluster(s) desired state. + +Here is an example of the generated manifests: + +```yaml +--- +apiVersion: v1 +kind: Namespace +metadata: + name: tenant1 +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: flux + namespace: tenant1 +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: flux + namespace: tenant1 +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin +subjects: + - kind: ServiceAccount + name: flux + namespace: tenant1 +--- +apiVersion: source.toolkit.fluxcd.io/v1beta1 +kind: GitRepository +metadata: + name: tenant1 + namespace: tenant1 +spec: + interval: 5m0s + ref: + branch: main + secretRef: + name: tenant1-git-auth + url: ssh://git@github.com/org/tenant1 +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1beta2 +kind: Kustomization +metadata: + name: tenant1 + namespace: tenant1 +spec: + interval: 10m0s + path: ./ + prune: true + serviceAccountName: flux + sourceRef: + kind: GitRepository + name: tenant1 +``` + +Note that the [cluster-admin](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) +role is used in a `RoleBinding`, this only gives full control over every resource in the role binding's namespace. + +Once the tenants repositories are registered on the cluster(s), the tenants can configure their app delivery +in Git using Kubernetes namespace-scoped resources such as `Deployments`, `Services`, Flagger `Canaries`, +Flux `Kustomizations`, `HelmReleases`, `ImageUpdateAutomations`, `Alerts`, `Receivers`, etc. + +## Alternatives + +Instead of introducing the security profile flag to `flux bootstrap`, +we could document how to patch each controller deployment with Kustomize. + +Having an easy way of locking down Flux with a single flag, make users aware of the security implications +and improves the user experience.