- Feature status: Experimental
- Feature gate: BootstrapFormatIgnition=true
The default configuration engine for bootstrapping workload cluster machines is cloud-init. Ignition is an alternative engine used by Linux distributions such as Flatcar Container Linux and Fedora CoreOS and therefore should be used when choosing an Ignition-based distribution as the underlying OS for workload clusters.
This document explains how Ignition support works.
For more generic information, see Cluster API documentation on Ignition Bootstrap configuration.
By default machine controller stores EC2 instance user data using SSM to store it encrypted, which underneath use multi part mime types, which are unlikely to be supported by Ignition.
EC2 user data is also limited to 64 KB, which is often not enough to provision Kubernetes controlplane because of the size of required certificates and configuration files.
To address those limitations CAPA can create and use S3 Bucket to store encrypted user data, which will be then pulled by the instances during provisioning.
To manage S3 Buckets and objects inside them, CAPA controllers require additional IAM permissions.
If you use
clusterawsadm for managing the IAM roles, you can use the configuration below to create S3-related
apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSIAMConfiguration spec: s3Buckets: enable: true
See Using clusterawsadm to fulfill prerequisites for more details.
Enabling EXP_BOOTSTRAP_FORMAT_IGNITION feature gate
When deploying CAPA using
clusterctl, make sure you set
EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true environment variables to enable experimental Ignition bootstrap
# Enable the feature gates controlling Ignition bootstrap. export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true # Used by the kubeadm bootstrap provider. export EXP_BOOTSTRAP_FORMAT_IGNITION=true # Used by the AWS provider. # Initialize the management cluster. clusterctl init --infrastructure aws
Bucket and object management
When you want to use Ignition user data format for you machines, you need to configure your cluster to specify which S3 bucket to use. Controller will then make sure that the bucket exists and that required policies are in place.
See the configuration snippet below to learn how to configure
AWSCluster to manage S3 bucket.
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSCluster spec: s3Bucket: controlPlaneIAMInstanceProfile: control-plane.cluster-api-provider-aws.sigs.k8s.io name: cluster-api-provider-aws-unique-suffix nodesIAMInstanceProfiles: - nodes.cluster-api-provider-aws.sigs.k8s.io
Buckets are safe to be reused between clusters.
After successful machine provisioning, bootstrap data is removed from the bucket.
During cluster removal, if S3 bucket is empty, it will be removed as well.
Bucket naming must follow S3 Bucket naming rules.
In addition, by default
clusterawsadm creates IAM roles to only allow interacting with buckets with
cluster-api-provider-aws- prefix to reduce the permissions of CAPA controller, so all bucket names should
use this prefix.
To change it, use
spec.s3Buckets.namePrefix field in
apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1beta1 kind: AWSIAMConfiguration spec: s3Buckets: namePrefix: my-custom-secure-bucket-prefix-
Supported bootstrap providers
At the moment only CABPK is known to support producing bootstrap data in Ignition format.
Trying it out
If you want to test Ignition support, use
flatcar cluster flavor.
Other bootstrap providers
If you want to use Ignition support with custom bootstrap provider which supports producing Ignition bootstrap
data, ensure that bootstrap provider sets the
format field in machine bootstrap secret to
information is used by the machine controller to determine which user data format to use for the instances.