Ignition support
- Feature status: Experimental
- Feature gate: BootstrapFormatIgnition=true
The default configuration engine for bootstrapping workload cluster machines is cloud-init. Ignition is an alternative engine used by Linux distributions such as Flatcar Container Linux and Fedora CoreOS and therefore should be used when choosing an Ignition-based distribution as the underlying OS for workload clusters.
This document explains how Ignition support works.
For more generic information, see Cluster API documentation on Ignition Bootstrap configuration.
Overview
By default machine controller stores EC2 instance user data using SSM to store it encrypted, which underneath use multi part mime types, which are unlikely to be supported by Ignition.
EC2 user data is also limited to 64 KB, which is often not enough to provision Kubernetes controlplane because of the size of required certificates and configuration files.
To address those limitations CAPA can create and use S3 Bucket to store encrypted user data, which will be then pulled by the instances during provisioning.
IAM Permissions
To manage S3 Buckets and objects inside them, CAPA controllers require additional IAM permissions.
If you use clusterawsadm
for managing the IAM roles, you can use the configuration below to create S3-related
IAM permissions.
apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSIAMConfiguration
spec:
s3Buckets:
enable: true
See Using clusterawsadm to fulfill prerequisites for more details.
Enabling EXP_BOOTSTRAP_FORMAT_IGNITION feature gate
When deploying CAPA using clusterctl
, make sure you set BOOTSTRAP_FORMAT_IGNITION=true
and
EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true
environment variables to enable experimental Ignition bootstrap
support.
# Enable the feature gates controlling Ignition bootstrap.
export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true # Used by the kubeadm bootstrap provider.
export EXP_BOOTSTRAP_FORMAT_IGNITION=true # Used by the AWS provider.
# Initialize the management cluster.
clusterctl init --infrastructure aws
Bucket and object management
When you want to use Ignition user data format for you machines, you need to configure your cluster to specify which S3 bucket to use. Controller will then make sure that the bucket exists and that required policies are in place.
See the configuration snippet below to learn how to configure AWSCluster
to manage S3 bucket.
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSCluster
spec:
s3Bucket:
controlPlaneIAMInstanceProfile: control-plane.cluster-api-provider-aws.sigs.k8s.io
name: cluster-api-provider-aws-unique-suffix
nodesIAMInstanceProfiles:
- nodes.cluster-api-provider-aws.sigs.k8s.io
Buckets are safe to be reused between clusters.
After successful machine provisioning, bootstrap data is removed from the bucket.
During cluster removal, if S3 bucket is empty, it will be removed as well.
Bucket naming
Bucket naming must follow S3 Bucket naming rules.
In addition, by default clusterawsadm
creates IAM roles to only allow interacting with buckets with
cluster-api-provider-aws-
prefix to reduce the permissions of CAPA controller, so all bucket names should
use this prefix.
To change it, use spec.s3Buckets.namePrefix
field in AWSIAMConfiguration
.
apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1beta1
kind: AWSIAMConfiguration
spec:
s3Buckets:
namePrefix: my-custom-secure-bucket-prefix-
Supported bootstrap providers
At the moment only CABPK is known to support producing bootstrap data in Ignition format.
Trying it out
If you want to test Ignition support, use flatcar
cluster flavor.
Other bootstrap providers
If you want to use Ignition support with custom bootstrap provider which supports producing Ignition bootstrap
data, ensure that bootstrap provider sets the format
field in machine bootstrap secret to ignition
. This
information is used by the machine controller to determine which user data format to use for the instances.