Powered by AWS Cloud Computing


Kubernetes-native declarative infrastructure for AWS.

What is the Cluster API Provider AWS

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.

The API itself is shared across multiple cloud providers allowing for true AWS hybrid deployments of Kubernetes. It is built atop the lessons learned from previous cluster managers such as kops and kubicorn.

Documentation

Please see our book for in-depth documentation.

Launching a Kubernetes cluster on AWS

Check out the Cluster API Quick Start for launching a cluster on AWS.

Features

  • Native Kubernetes manifests and API
  • Manages the bootstrapping of VPCs, gateways, security groups and instances.
  • Choice of Linux distribution between Amazon Linux 2, CentOS 7 and Ubuntu 18.04, using pre-baked AMIs.
  • Deploys Kubernetes control planes into private subnets with a separate bastion server.
  • Doesn’t use SSH for bootstrapping nodes.
  • Installs only the minimal components to bootstrap a control plane and workers.
  • Supports control planes on EC2 instances.
  • Experimental EKS support

Compatibility with Cluster API and Kubernetes Versions

This provider’s versions are compatible with the following versions of Cluster API:

Cluster API v1alpha3 (v0.3)Cluster API v1alpha4 (v0.4)
AWS Provider v1alpha3 (v0.5)
AWS Provider v1alpha3 (v0.6)
AWS Provider v1alpha4 (v0.7)

This provider’s versions are able to install and manage the following versions of Kubernetes:

Kubernetes 1.16Kubernetes 1.17Kubernetes 1.18Kubernetes 1.19Kubernetes 1.20Kubernetes 1.21
AWS Provider v1alpha3 (v0.5)
AWS Provider v1alpha3 (v0.6)
AWS Provider v1alpha4 (v0.7)

Each version of Cluster API for AWS will attempt to support two Kubernetes versions; e.g., Cluster API for AWS v0.3 supports Kubernetes 1.16, 1.17, 1.18 etc.

NOTE: As the versioning for this project is tied to the versioning of Cluster API, future modifications to this policy may be made to more closely align with other providers in the Cluster API ecosystem.


Kubernetes versions with published AMIs

See amis for the list of most recently published AMIs.


clusterawsadm

clusterawsadm CLI tool provides bootstrapping, AMI, EKS, and controller related helpers.

clusterawsadm binaries are released with each release, can be found under assets section.


Getting involved and contributing

Are you interested in contributing to cluster-api-provider-aws? We, the maintainers and community, would love your suggestions, contributions, and help! Also, the maintainers can be contacted at any time to learn more about how to get involved.

In the interest of getting more new people involved we tag issues with good first issue. These are typically issues that have smaller scope but are good ways to start to get acquainted with the codebase.

We also encourage ALL active community participants to act as if they are maintainers, even if you don’t have “official” write permissions. This is a community effort, we are here to serve the Kubernetes community. If you have an active interest and you want to get involved, you have real power! Don’t assume that the only people who can get things done around here are the “maintainers”.

We also would love to add more “official” maintainers, so show us what you can do!

This repository uses the Kubernetes bots. See a full list of the commands here.

Build the images locally

If you want to just build the CAPA containers locally, run

	REGISTRY=docker.io/my-reg make docker-build

Tilt-based development environment

See development section for details

Implementer office hours

Maintainers hold office hours every two weeks, with sessions open to all developers working on this project.

Office hours are hosted on a zoom video chat every other Monday at 10:00 (Pacific) / 13:00 (Eastern) / 18:00 (Europe/London), and are published on the Kubernetes community meetings calendar.

Other ways to communicate with the contributors

Please check in with us in the #cluster-api-aws channel on Slack.

Github issues

Bugs

If you think you have found a bug please follow the instructions below.

  • Please spend a small amount of time giving due diligence to the issue tracker. Your issue might be a duplicate.
  • Get the logs from the cluster controllers. Please paste this into your issue.
  • Open a new issue.
  • Remember that users might be searching for your issue in the future, so please give it a meaningful title to help others.
  • Feel free to reach out to the cluster-api community on the kubernetes slack.

Tracking new features

We also use the issue tracker to track features. If you have an idea for a feature, or think you can help kops become even more awesome follow the steps below.

  • Open a new issue.
  • Remember that users might be searching for your issue in the future, so please give it a meaningful title to help others.
  • Clearly define the use case, using concrete examples. EG: I type this and cluster-api-provider-aws does that.
  • Some of our larger features will require some design. If you would like to include a technical design for your feature please include it in the issue.
  • After the new feature is well understood, and the design agreed upon, we can start coding the feature. We would love for you to code it. So please open up a WIP (work in progress) pull request, and happy coding.

“Amazon Web Services, AWS, and the “Powered by AWS” logo materials are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.”

Getting Started

Quick Start

In this tutorial we’ll cover the basics of how to use Cluster API to create one or more Kubernetes clusters.

Installation

Common Prerequisites

Install and/or configure a Kubernetes cluster

Cluster API requires an existing Kubernetes cluster accessible via kubectl. During the installation process the Kubernetes cluster will be transformed into a management cluster by installing the Cluster API provider components, so it is recommended to keep it separated from any application workload.

It is a common practice to create a temporary, local bootstrap cluster which is then used to provision a target management cluster on the selected infrastructure provider.

Choose one of the options below:

  1. Existing Management Cluster

    For production use-cases a “real” Kubernetes cluster should be used with appropriate backup and DR policies and procedures in place. The Kubernetes cluster must be at least v1.19.1.

    export KUBECONFIG=<...>
    
  2. Kind

    kind can be used for creating a local Kubernetes cluster for development environments or for the creation of a temporary bootstrap cluster used to provision a target management cluster on the selected infrastructure provider.

    The installation procedure depends on the version of kind; if you are planning to use the Docker infrastructure provider, please follow the additional instructions in the dedicated tab:

    Create the kind cluster:

    kind create cluster
    

    Test to ensure the local kind cluster is ready:

    kubectl cluster-info
    

    Run the following command to create a kind config file for allowing the Docker provider to access Docker on the host:

    cat > kind-cluster-with-extramounts.yaml <<EOF
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    nodes:
    - role: control-plane
      extraMounts:
        - hostPath: /var/run/docker.sock
          containerPath: /var/run/docker.sock
    EOF
    

    Then follow the instruction for your kind version using kind create cluster --config kind-cluster-with-extramounts.yaml to create the management cluster using the above file.

Install clusterctl

The clusterctl CLI tool handles the lifecycle of a Cluster API management cluster.

Install clusterctl binary with curl on linux

Download the latest release; on linux, type:

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.4.3/clusterctl-linux-amd64 -o clusterctl

Make the clusterctl binary executable.

chmod +x ./clusterctl

Move the binary in to your PATH.

sudo mv ./clusterctl /usr/local/bin/clusterctl

Test to ensure the version you installed is up-to-date:

clusterctl version

Install clusterctl binary with curl on macOS

Download the latest release; on macOS, type:

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.4.3/clusterctl-darwin-amd64 -o clusterctl

Or if your Mac has an M1 CPU (”Apple Silicon”):

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v0.4.3/clusterctl-darwin-arm64 -o clusterctl

Make the clusterctl binary executable.

chmod +x ./clusterctl

Move the binary in to your PATH.

sudo mv ./clusterctl /usr/local/bin/clusterctl

Test to ensure the version you installed is up-to-date:

clusterctl version

Install clusterctl with homebrew on macOS and linux

Install the latest release using homebrew:

brew install clusterctl

Test to ensure the version you installed is up-to-date:

clusterctl version

Initialize the management cluster

Now that we’ve got clusterctl installed and all the prerequisites in place, let’s transform the Kubernetes cluster into a management cluster by using clusterctl init.

The command accepts as input a list of providers to install; when executed for the first time, clusterctl init automatically adds to the list the cluster-api core provider, and if unspecified, it also adds the kubeadm bootstrap and kubeadm control-plane providers.

Initialization for common providers

Depending on the infrastructure provider you are planning to use, some additional prerequisites should be satisfied before getting started with Cluster API. See below for the expected settings for common providers.

Download the latest binary of clusterawsadm from the AWS provider releases and make sure to place it in your path.

The clusterawsadm command line utility assists with identity and access management (IAM) for Cluster API Provider AWS.

export AWS_REGION=us-east-1 # This is used to help encode your environment variables
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<session-token> # If you are using Multi-Factor Auth.

# The clusterawsadm utility takes the credentials that you set as environment
# variables and uses them to create a CloudFormation stack in your AWS account
# with the correct IAM resources.
clusterawsadm bootstrap iam create-cloudformation-stack

# Create the base64 encoded credentials using clusterawsadm.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

# Finally, initialize the management cluster
clusterctl init --infrastructure aws

See the AWS provider prerequisites document for more details.

For more information about authorization, AAD, or requirements for Azure, visit the Azure provider prerequisites document.

export AZURE_SUBSCRIPTION_ID="<SubscriptionId>"

# Create an Azure Service Principal and paste the output here
export AZURE_TENANT_ID="<Tenant>"
export AZURE_CLIENT_ID="<AppId>"
export AZURE_CLIENT_SECRET="<Password>"

# Base64 encode the variables
export AZURE_SUBSCRIPTION_ID_B64="$(echo -n "$AZURE_SUBSCRIPTION_ID" | base64 | tr -d '\n')"
export AZURE_TENANT_ID_B64="$(echo -n "$AZURE_TENANT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_ID_B64="$(echo -n "$AZURE_CLIENT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_SECRET_B64="$(echo -n "$AZURE_CLIENT_SECRET" | base64 | tr -d '\n')"

# Settings needed for AzureClusterIdentity used by the AzureCluster
export AZURE_CLUSTER_IDENTITY_SECRET_NAME="cluster-identity-secret"
export CLUSTER_IDENTITY_NAME="cluster-identity"
export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default"

# Create a secret to include the password of the Service Principal identity created in Azure
# This secret will be referenced by the AzureClusterIdentity used by the AzureCluster
kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}"

# Finally, initialize the management cluster
clusterctl init --infrastructure azure
export DIGITALOCEAN_ACCESS_TOKEN=<your-access-token>
export DO_B64ENCODED_CREDENTIALS="$(echo -n "${DIGITALOCEAN_ACCESS_TOKEN}" | base64 | tr -d '\n')"

# Initialize the management cluster
clusterctl init --infrastructure digitalocean

The Docker provider does not require additional prerequisites. You can run:

clusterctl init --infrastructure docker
# Create the base64 encoded credentials by catting your credentials json.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export GCP_B64ENCODED_CREDENTIALS=$( cat /path/to/gcp-credentials.json | base64 | tr -d '\n' )

# Finally, initialize the management cluster
clusterctl init --infrastructure gcp
# The username used to access the remote vSphere endpoint
export VSPHERE_USERNAME="vi-admin@vsphere.local"
# The password used to access the remote vSphere endpoint
# You may want to set this in ~/.cluster-api/clusterctl.yaml so your password is not in
# bash history
export VSPHERE_PASSWORD="admin!23"

# Finally, initialize the management cluster
clusterctl init --infrastructure vsphere

For more information about prerequisites, credentials management, or permissions for vSphere, see the vSphere project.

# Initialize the management cluster
clusterctl init --infrastructure openstack

Please visit the Metal3 project.

In order to initialize the Packet Provider you have to expose the environment variable PACKET_API_KEY. This variable is used to authorize the infrastructure provider manager against the Packet API. You can retrieve your token directly from the Packet Portal.

export PACKET_API_KEY="34ts3g4s5g45gd45dhdh"

clusterctl init --infrastructure packet

The output of clusterctl init is similar to this:

Fetching providers
Installing cert-manager Version="v1.5.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.4.0" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.4.0" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.4.0" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v0.4.0" TargetNamespace="capd-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

Create your first workload cluster

Once the management cluster is ready, you can create your first workload cluster.

Preparing the workload cluster configuration

The clusterctl generate cluster command returns a YAML template for creating a workload cluster.

Required configuration for common providers

Depending on the infrastructure provider you are planning to use, some additional prerequisites should be satisfied before configuring a cluster with Cluster API. Instructions are provided for common providers below.

Otherwise, you can look at the clusterctl generate cluster command documentation for details about how to discover the list of variables required by a cluster templates.

export AWS_REGION=us-east-1
export AWS_SSH_KEY_NAME=default
# Select instance types
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large

See the AWS provider prerequisites document for more details.

# Name of the Azure datacenter location. Change this value to your desired location.
export AZURE_LOCATION="centralus"

# Select VM types.
export AZURE_CONTROL_PLANE_MACHINE_TYPE="Standard_D2s_v3"
export AZURE_NODE_MACHINE_TYPE="Standard_D2s_v3"

A ClusterAPI compatible image must be available in your DigitalOcean account. For instructions on how to build a compatible image see image-builder.

export DO_REGION=nyc1
export DO_SSH_KEY_FINGERPRINT=<your-ssh-key-fingerprint>
export DO_CONTROL_PLANE_MACHINE_TYPE=s-2vcpu-2gb
export DO_CONTROL_PLANE_MACHINE_IMAGE=<your-capi-image-id>
export DO_NODE_MACHINE_TYPE=s-2vcpu-2gb
export DO_NODE_MACHINE_IMAGE==<your-capi-image-id>

The Docker provider does not require additional configurations for cluster templates.

However, if you require special network settings you can set the following environment variables:

# The list of service CIDR, default ["10.128.0.0/12"]
export SERVICE_CIDR=["10.96.0.0/12"]

# The list of pod CIDR, default ["192.168.0.0/16"]
export POD_CIDR=["192.168.0.0/16"]

# The service domain, default "cluster.local"
export SERVICE_DOMAIN="k8s.test"
# Name of the GCP datacenter location. Change this value to your desired location
export GCP_REGION="<GCP_REGION>"
export GCP_PROJECT="<GCP_PROJECT>"
# Make sure to use same kubernetes version here as building the GCE image
export KUBERNETES_VERSION=1.20.9
export GCP_CONTROL_PLANE_MACHINE_TYPE=n1-standard-2
export GCP_NODE_MACHINE_TYPE=n1-standard-2
export GCP_NETWORK_NAME=<GCP_NETWORK_NAME or default>
export CLUSTER_NAME="<CLUSTER_NAME>"

See the GCP provider for more information.

It is required to use an official CAPV machine images for your vSphere VM templates. See uploading CAPV machine images for instructions on how to do this.

# The vCenter server IP or FQDN
export VSPHERE_SERVER="10.0.0.1"
# The vSphere datacenter to deploy the management cluster on
export VSPHERE_DATACENTER="SDDC-Datacenter"
# The vSphere datastore to deploy the management cluster on
export VSPHERE_DATASTORE="vsanDatastore"
# The VM network to deploy the management cluster on
export VSPHERE_NETWORK="VM Network"
# The vSphere resource pool for your VMs
export VSPHERE_RESOURCE_POOL="*/Resources"
# The VM folder for your VMs. Set to "" to use the root vSphere folder
export VSPHERE_FOLDER="vm"
# The VM template to use for your VMs
export VSPHERE_TEMPLATE="ubuntu-1804-kube-v1.17.3"
# The VM template to use for the HAProxy load balancer of the management cluster
export VSPHERE_HAPROXY_TEMPLATE="capv-haproxy-v0.6.0-rc.2"
# The public ssh authorized key on all machines
export VSPHERE_SSH_AUTHORIZED_KEY="ssh-rsa AAAAB3N..."

clusterctl init --infrastructure vsphere

For more information about prerequisites, credentials management, or permissions for vSphere, see the vSphere getting started guide.

A ClusterAPI compatible image must be available in your OpenStack. For instructions on how to build a compatible image see image-builder. Depending on your OpenStack and underlying hypervisor the following options might be of interest:

To see all required OpenStack environment variables execute:

clusterctl generate cluster --infrastructure openstack --list-variables capi-quickstart

The following script can be used to export some of them:

wget https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-openstack/master/templates/env.rc -O /tmp/env.rc
source /tmp/env.rc <path/to/clouds.yaml> <cloud>

Apart from the script, the following OpenStack environment variables are required.

# The list of nameservers for OpenStack Subnet being created.
# Set this value when you need create a new network/subnet while the access through DNS is required.
export OPENSTACK_DNS_NAMESERVERS=<dns nameserver>
# FailureDomain is the failure domain the machine will be created in.
export OPENSTACK_FAILURE_DOMAIN=<availability zone name>
# The flavor reference for the flavor for your server instance.
export OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR=<flavor>
# The flavor reference for the flavor for your server instance.
export OPENSTACK_NODE_MACHINE_FLAVOR=<flavor>
# The name of the image to use for your server instance. If the RootVolume is specified, this will be ignored and use rootVolume directly.
export OPENSTACK_IMAGE_NAME=<image name>
# The SSH key pair name
export OPENSTACK_SSH_KEY_NAME=<ssh key pair name>

A full configuration reference can be found in configuration.md.

# The URL of the kernel to deploy.
export DEPLOY_KERNEL_URL="http://172.22.0.1:6180/images/ironic-python-agent.kernel"
# The URL of the ramdisk to deploy.
export DEPLOY_RAMDISK_URL="http://172.22.0.1:6180/images/ironic-python-agent.initramfs"
# The URL of the Ironic endpoint.
export IRONIC_URL="http://172.22.0.1:6385/v1/"
# The URL of the Ironic inspector endpoint.
export IRONIC_INSPECTOR_URL="http://172.22.0.1:5050/v1/"
# Do not use a dedicated CA certificate for Ironic API. Any value provided in this variable disables additional CA certificate validation.
# To provide a CA certificate, leave this variable unset. If unset, then IRONIC_CA_CERT_B64 must be set.
export IRONIC_NO_CA_CERT=true
# Disables basic authentication for Ironic API. Any value provided in this variable disables authentication.
# To enable authentication, leave this variable unset. If unset, then IRONIC_USERNAME and IRONIC_PASSWORD must be set.
export IRONIC_NO_BASIC_AUTH=true
# Disables basic authentication for Ironic inspector API. Any value provided in this variable disables authentication.
# To enable authentication, leave this variable unset. If unset, then IRONIC_INSPECTOR_USERNAME and IRONIC_INSPECTOR_PASSWORD must be set.
export IRONIC_INSPECTOR_NO_BASIC_AUTH=true

Please visit the Metal3 getting started guide for more details.

There are a couple of required environment variables that you have to expose in order to get a well tuned and function workload, they are all listed here:

# The project where your cluster will be placed to.
# You have to get out from Packet Portal if you do not have one already.
export PROJECT_ID="5yd4thd-5h35-5hwk-1111-125gjej40930"
# The facility where you want your cluster to be provisioned
export FACILITY="ewr1"
# The operatin system used to provision the device
export NODE_OS="ubuntu_18_04"
# The ssh key name you loaded in Packet Portal
export SSH_KEY="my-ssh"
export POD_CIDR="192.168.0.0/16"
export SERVICE_CIDR="172.26.0.0/16"
export CONTROLPLANE_NODE_TYPE="t1.small"
export WORKER_NODE_TYPE="t1.small"

Generating the cluster configuration

For the purpose of this tutorial, we’ll name our cluster capi-quickstart.

clusterctl generate cluster capi-quickstart \
  --kubernetes-version v1.22.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml
clusterctl generate cluster capi-quickstart --flavor development \
  --kubernetes-version v1.22.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

This creates a YAML file named capi-quickstart.yaml with a predefined list of Cluster API objects; Cluster, Machines, Machine Deployments, etc.

The file can be eventually modified using your editor of choice.

See clusterctl generate cluster for more details.

Apply the workload cluster

When ready, run the following command to apply the cluster manifest.

kubectl apply -f capi-quickstart.yaml

The output is similar to this:

cluster.cluster.x-k8s.io/capi-quickstart created
awscluster.infrastructure.cluster.x-k8s.io/capi-quickstart created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-quickstart-control-plane created
awsmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-control-plane created
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0 created
awsmachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/capi-quickstart-md-0 created

Accessing the workload cluster

The cluster will now start provisioning. You can check status with:

kubectl get cluster

You can also get an “at glance” view of the cluster and its resources by running:

clusterctl describe cluster capi-quickstart

To verify the first control plane is up:

kubectl get kubeadmcontrolplane

You should see an output is similar to this:

NAME                            INITIALIZED   API SERVER AVAILABLE   VERSION   REPLICAS   READY   UPDATED   UNAVAILABLE
capi-quickstart-control-plane   true                                 v1.21.2   3                  3         3

After the first control plane node is up and running, we can retrieve the workload cluster Kubeconfig:

clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

Deploy a CNI solution

Calico is used here as an example.

kubectl --kubeconfig=./capi-quickstart.kubeconfig \
  apply -f https://docs.projectcalico.org/v3.20/manifests/calico.yaml

After a short while, our nodes should be running and in Ready state, let’s check the status using kubectl get nodes:

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

Azure does not currently support Calico networking. As a workaround, it is recommended that Azure clusters use the Calico spec below that uses VXLAN.

kubectl --kubeconfig=./capi-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico.yaml

After a short while, our nodes should be running and in Ready state, let’s check the status using kubectl get nodes:

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

Clean Up

Delete workload cluster.

kubectl delete cluster capi-quickstart

Delete management cluster

kind delete cluster

Next steps

See the clusterctl documentation for more detail about clusterctl supported actions.

Pre-built Kubernetes AMIs

New AMIs are built whenever a new Kubernetes version is released for each supported OS distribution and then published to supported regions.

clusterawsadm ami list command lists pre-built AMIs by Kubernetes version, OS, or AWS region. See clusterawsadm ami list for details.

Supported OS Distributions

  • Amazon Linux 2 (amazon-2)
  • Ubuntu (ubuntu-20.04, ubuntu-18.04)
  • Centos (centos-7)

Supported AWS Regions

  • ap-northeast-1
  • ap-northeast-2
  • ap-south-1
  • ap-southeast-1
  • ap-northeast-2
  • ca-central-1
  • eu-central-1
  • eu-west-1
  • eu-west-2
  • eu-west-3
  • sa-east-1
  • us-east-1
  • us-east-2
  • us-west-1
  • us-west-2

Most recent AMIs

Topics

Using clusterawsadm to fulfill prerequisites

Requirements

  • Linux or MacOS (Windows isn’t supported at the moment).
  • AWS credentials.
  • AWS CLI
  • jq

IAM resources

With clusterawsadm

Get the latest clusterawsadm and place it in your path.

Cluster API Provider AWS ships with clusterawsadm, a utility to help you manage IAM objects for this project.

In order to use clusterawsadm you must have an administrative user in an AWS account. Once you have that administrator user you need to set your environment variables:

  • AWS_REGION
  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_SESSION_TOKEN (if you are using Multi-factor authentication)

After these are set run this command to get you up and running:

clusterawsadm bootstrap iam create-cloudformation-stack

Additional policies can be added by creating a configuration file

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  controlPlane:
    ExtraPolicyAttachments:
      - arn:aws:iam::<AWS_ACCOUNT>:policy/my-policy
      - arn:aws:iam::aws:policy/AmazonEC2FullAccess
  nodes:
    ExtraPolicyAttachments:
      - arn:aws:iam::<AWS_ACCOUNT>:policy/my-other-policy

and passing it to clusterawsadm as follows

clusterawsadm bootstrap iam create-stack --config bootstrap-config.yaml

These will be added to the control plane and node roles respectively when they are created.

Note: If you used the now deprecated clusterawsadm alpha bootstrap 0.5.4 or earlier to create IAM objects for the Cluster API Provider for AWS, using clusterawsadm bootstrap iam 0.5.5 or later will, by default, remove the bootstrap user and group. Anything using those credentials to authenticate will start experiencing authentication failures. If you rely on the bootstrap user and group credentials, specify bootstrapUser.enable = true in the configuration file, like this:

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  bootstrapUser:
    enable: true

With EKS Support

The pre-requisities for EKS are enabled by default. However, if you want to use some of the optional features of EKS (see here for more information on what these are) then you will need to enable these features via the configuration file. For example:

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  eks:
    iamRoleCreation: false # Set to true if you plan to use the EKSEnableIAM feature flag to enable automatic creation of IAM roles
    managedMachinePool:
      disable: false # Set to false to enable creation of the default node role for managed machine pools
    fargate:
      disable: false # Set to false to enable creation of the default role for the fargate profiles

and then use that configuration file:

clusterawsadm bootstrap iam create-cloudformation-stack --config bootstrap-config.yaml

Enabling EventBridge Events

To enable EventBridge instance state events, additional permissions must be granted along with enabling the feature-flag. Additional permissions for events and queue management can be enabled through the configuration file as follows:

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  ...
  eventBridge:
    enable: true
  ...

Without clusterawsadm

This is not a recommended route as the policies are very specific and will change with new features.

If you do not wish to use the clusteradwsadm tool then you will need to understand exactly which IAM policies and groups we are expecting. There are several policies, roles and users that need to be created. Please see our controller policy file to understand the permissions that are necessary.

You can use clusteradwadm to print out the needed IAM policies, e.g.

clusterawsadm bootstrap iam print-policy --document AWSIAMManagedPolicyControllers --config bootstrap-config.yaml

SSH Key pair

If you plan to use SSH to access the instances created by Cluster API Provider AWS then you will need to specify the name of an existing SSH key pair within the region you plan on using. If you don’t have one yet, a new one needs to be created.

Create a new key pair

# Save the output to a secure location
aws ec2 create-key-pair --key-name default | jq .KeyMaterial -r
-----BEGIN RSA PRIVATE KEY-----
[... contents omitted ...]
-----END RSA PRIVATE KEY-----

If you want to save the private key directly into AWS Systems Manager Parameter Store with KMS encryption for security, you can use the following command:

aws ssm put-parameter --name "/sigs.k8s.io/cluster-api-provider-aws/ssh-key" \
  --type SecureString \
  --value "$(aws ec2 create-key-pair --key-name default | jq .KeyMaterial -r)"

Adding an existing public key to AWS

# Replace with your own public key
aws ec2 import-key-pair \
  --key-name default \
  --public-key-material "$(cat ~/.ssh/id_rsa.pub)"

NB: Only RSA keys are supported by AWS.

Setting up the environment

The current iteration of the Cluster API Provider AWS relies on credentials being present in your environment. These then get written into the cluster manifests for use by the controllers.

E.g.

export AWS_REGION=us-east-1 # This is used to help encode your environment variables
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<session-token> # If you are using Multi-Factor Auth.

Note: The credentials used must have the appropriate permissions for use by the controllers. You can get the required policy statement by using the following command:

clusterawsadm bootstrap iam print-policy --document AWSIAMManagedPolicyControllers --config bootstrap-config.yaml

To save credentials securely in your environment, aws-vault uses the OS keystore as permanent storage, and offers shell features to securely expose and setup local AWS environments.

Accessing cluster instances

Overview

After running clusterctl generate cluster to generate the configuration for a new workload cluster (and then redirecting that output to a file for use with kubectl apply, or piping it directly to kubectl apply), the new workload cluster will be deployed. This document explains how to access the new workload cluster’s nodes.

Prerequisites

  1. clusterctl generate cluster was successfully executed to generate the configuration for a new workload cluster
  2. The configuration for the new workload cluster was applied to the management cluster using kubectl apply and the cluster is up and running in an AWS environment.
  3. The SSH key referenced by clusterctl in step 1 exists in AWS and is stored in the correct location locally for use by SSH (on macOS/Linux systems, this is typically $HOME/.ssh). This document will refer to this key as cluster-api-provider-aws.sigs.k8s.io.
  4. (If using AWS Session Manager) The AWS CLI and the Session Manager plugin have been installed and configured.

Methods for accessing nodes

There are two ways to access cluster nodes once the workload cluster is up and running:

  • via SSH
  • via AWS Session Manager

Accessing nodes via SSH

By default, workload clusters created in AWS will not support access via SSH apart from AWS Session Manager (see the section titled “Accessing nodes via AWS Session Manager”). However, the manifest for a workload cluster can be modified to include an SSH bastion host, created and managed by the management cluster, to enable SSH access to cluster nodes. The bastion node is created in a public subnet and provides SSH access from the world. It runs the official Ubuntu Linux image.

Enabling the bastion host

To configure the Cluster API Provider for AWS to create an SSH bastion host, add this line to the AWSCluster spec:

spec:
  bastion:
    enabled: true

Obtain public IP address of the bastion node

Once the workload cluster is up and running after being configured for an SSH bastion host, you can use the kubectl get awscluster command to look up the public IP address of the bastion host (make sure the kubectl context is set to the management cluster). The output will look something like this:

NAME   CLUSTER   READY   VPC                     BASTION IP
test   test      true    vpc-1739285ed052be7ad   1.2.3.4

Setting up the SSH key path

Assumming that the cluster-api-provider-aws.sigs.k8s.io SSH key is stored in $HOME/.ssh/cluster-api-provider-aws, use this command to set up an environment variable for use in a later command:

export CLUSTER_SSH_KEY=$HOME/.ssh/cluster-api-provider-aws

Get private IP addresses of nodes in the cluster

To get the private IP addresses of nodes in the cluster (nodes may be control plane nodes or worker nodes), use this kubectl command with the context set to the management cluster:

kubectl get nodes -o custom-columns=NAME:.metadata.name,\
IP:"{.status.addresses[?(@.type=='InternalIP')].address}"

This will produce output that looks like this:

NAME                                         IP
ip-10-0-0-16.us-west-2.compute.internal   10.0.0.16
ip-10-0-0-68.us-west-2.compute.internal   10.0.0.68

The above command returns IP addresses of the nodes in the cluster. In this case, the values returned are 10.0.0.16 and 10.0.0.68.

Connecting to the nodes via SSH

To access one of the nodes (either a control plane node or a worker node) via the SSH bastion host, use this command if you are using a non-EKS cluster:

ssh -i ${CLUSTER_SSH_KEY} ubuntu@<NODE_IP> \
	-o "ProxyCommand ssh -W %h:%p -i ${CLUSTER_SSH_KEY} ubuntu@${BASTION_HOST}"

And use this command if you are using a EKS based cluster:

ssh -i ${CLUSTER_SSH_KEY} ec2-user@<NODE_IP> \
	-o "ProxyCommand ssh -W %h:%p -i ${CLUSTER_SSH_KEY} ubuntu@${BASTION_HOST}"

If the whole document is followed, the value of <NODE_IP> will be either 10.0.0.16 or 10.0.0.68.

Alternately, users can add a configuration stanza to their SSH configuration file (typically found on macOS/Linux systems as $HOME/.ssh/config):

Host 10.0.*
  User ubuntu
  IdentityFile <CLUSTER_SSH_KEY>
  ProxyCommand ssh -W %h:%p ubuntu@<BASTION_HOST>

Accessing nodes via AWS Session Manager

All CAPA-published AMIs based on Ubuntu have the AWS SSM Agent pre-installed (as a Snap package; this was added in June 2018 to the base Ubuntu Server image for all 16.04 and later AMIs). This allows users to access cluster nodes directly, without the need for an SSH bastion host, using the AWS CLI and the Session Manager plugin.

To access a cluster node (control plane node or worker node), you’ll need the instance ID. You can retrieve the instance ID using this kubectl command with the context set to the management cluster:

kubectl get awsmachines -o custom-columns=NAME:.metadata.name,INSTANCEID:.spec.providerID

This will produce output similar to this:

NAME                      INSTANCEID
test-controlplane-52fhh   aws:////i-112bac41a19da1819
test-controlplane-lc5xz   aws:////i-99aaef2381ada9228

Users can then use the instance ID (everything after the aws://// prefix) to connect to the cluster node with this command:

aws ssm start-session --target <INSTANCE_ID>

This will log you into the cluster node as the ssm-user user ID.

Additional Notes

Using the AWS CLI instead of kubectl

It is also possible to use AWS CLI commands instead of kubectl to gather information about the cluster nodes.

For example, to use the AWS CLI to get the public IP address of the SSH bastion host, use this AWS CLI command:

export BASTION_HOST=$(aws ec2 describe-instances --filter='Name=tag:Name,Values=<CLUSTER_NAME>-bastion' \
	| jq '.Reservations[].Instances[].PublicIpAddress' -r)

You should substitute the correct cluster name for <CLUSTER_NAME> in the above command. (NOTE: If make manifests was used to generate manifests, by default the <CLUSTER_NAME> is set to test1.)

Similarly, to obtain the list of private IP addresses of the cluster nodes, use this AWS CLI command:

for type in control-plane node
do
	aws ec2 describe-instances \
    --filter="Name=tag:sigs.k8s.io/cluster-api-provider-aws/role,\
    Values=${type}" \
		| jq '.Reservations[].Instances[].PrivateIpAddress' -r
done
10.0.0.16
10.0.0.68

Finally, to obtain AWS instance IDs for cluster nodes, you can use this AWS CLI command:

for type in control-plane node
do
	aws ec2 describe-instances \
    --filter="Name=tag:sigs.k8s.io/cluster-api-provider-aws/role,\
    Values=${type}" \
		| jq '.Reservations[].Instances[].InstanceId' -r
done
i-112bac41a19da1819
i-99aaef2381ada9228

Note that your AWS CLI must be configured with credentials that enable you to query the AWS EC2 API.

MachinePools

  • Feature status: Experimental
  • Feature gate: MachinePool=true

MachinePool allows users to manage many machines as a single entity. Infrastructure providers implement a separate CRD that handles infrastructure side of the feature.

AWSMachinePool

Cluster API Provider AWS (CAPA) has experimental support for MachinePool though the infrastructure type AWSMachinePool. An AWSMachinePool corresponds to an AWS AutoScaling Groups, which provides the cloud provider specific resource for orchestrating a group of EC2 machines.

The AWSMachinePool controller creates and manages an AWS AutoScaling Group using launch templates so users don’t have to manage individual machines. You can use Autoscaling health checks for replacing instances and it will maintain the number of instances specified.

Using clusterctl to deploy

To deploy a MachinePool / AWSMachinePool via clusterctl generate there’s a flavor for that.

Make sure to set up your AWS environment as described here.

export EXP_MACHINE_POOL=true
clusterctl init --infrastructure aws
clusterctl generate cluster my-cluster --kubernetes-version v1.16.8 --flavor machinepool > my-cluster.yaml

The template used for this flavor is located here.

AWSManagedMachinePool

Cluster API Provider AWS (CAPA) has experimental support for EKS Managed Node Groups using MachinePool through the infrastructure type AWSManagedMachinePool. An AWSManagedMachinePool corresponds to an AWS AutoScaling Groups that is used for an EKS managed node group. .

The AWSManagedMachinePool controller creates and manages an EKS managed node group with in turn manages an AWS AutoScaling Group of managed EC2 instance types.

To use the managed machine pools certain IAM permissions are needed. The easiest way to ensure the required IAM permissions are in place is to use clusterawsadm to create them. To do this, follow the EKS instructions in using clusterawsadm to fulfill prerequisites.

Using clusterctl to deploy

To deploy an EKS managed node group using AWSManagedMachinePool via clusterctl generate you can use a flavor.

Make sure to set up your AWS environment as described here.

export EXP_MACHINE_POOL=true
clusterctl init --infrastructure aws
clusterctl generate cluster my-cluster --kubernetes-version v1.16.8 --flavor eks-managedmachinepool > my-cluster.yaml

The template used for this flavor is located here.

Examples

Example MachinePool, AWSMachinePool and KubeadmConfig Resources

Below is an example of the resources needed to create a pool of EC2 machines orchestrated with an AWS Auto Scaling Group.

---
apiVersion: cluster.x-k8s.io/v1alpha3
kind: MachinePool
metadata:
  name: capa-mp-0
spec:
  clusterName: capa
  replicas: 2
  template:
    spec:
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
          kind: KubeadmConfig
          name: capa-mp-0
      clusterName: capa
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
        kind: AWSMachinePool
        name: capa-mp-0
      version: v1.16.8
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSMachinePool
metadata:
  name: capa-mp-0
spec:
  minSize: 1
  maxSize: 10
  availabilityZones:
    - us-east-1
  awsLaunchTemplate:
    instanceType: "${AWS_CONTROL_PLANE_MACHINE_TYPE}"
    sshKeyName: "${AWS_SSH_KEY_NAME}"
  subnets:
  - ${AWS_SUBNET}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
kind: KubeadmConfig
metadata:
  name: capa-mp-0
  namespace: default
spec:
  joinConfiguration:
    nodeRegistration:
      name: '{{ ds.meta_data.local_hostname }}'
      kubeletExtraArgs:
        cloud-provider: aws

Multi-tenancy

Starting from v0.6.5, single controller multi-tenancy is supported that allows using a different AWS Identity for each workload cluster. For details, see the multi-tenancy proposal.

For multi-tenancy support, a reference field (identityRef) is added to AWSCluster, which describes the identity to be used when reconciling the cluster.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSCluster
metadata:
  name: "test"
  namespace: "test"
spec:
  region: "eu-west-1"
  identityRef:
    kind: <IdentityType>
    name: <IdentityName>

Identity resources are used to describe IAM identities that will be used during reconciliation. There are three identity types: AWSClusterControllerIdentity, AWSClusterStaticIdentity, and AWSClusterRoleIdentity. Once an IAM identity is created in AWS, the corresponding values should be used to create a identity resource.

AWSClusterControllerIdentity

Before multi-tenancy support, all AWSClusters were being reconciled using the credentials that are used by Cluster API Provider AWS Controllers. AWSClusterControllerIdentity is used to restrict the usage of controller credentials only to AWSClusters that are in allowedNamespaces. Since CAPA controllers use a single set of credentials, AWSClusterControllerIdentity is a singleton, and can only be created with name: default.

For backward compatibility, AutoControllerIdentityCreator experimental feature is added, which is responsible to create the AWSClusterControllerIdentity singleton if it does not exist.

  • Feature status: Experimental
  • Feature gate: AutoControllerIdentityCreator=true AutoControllerIdentityCreator creates AWSClusterControllerIdentity singleton with empty allowedNamespaces (allowedNamespaces: {}) to grant access to the AWSClusterControllerIdentity from all namespaces.

Example:

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSCluster
metadata:
  name: "test"
  namespace: "test"
spec:
  region: "eu-west-1"
  identityRef:
    kind: AWSClusterControllerIdentity
    name: default
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterControllerIdentity
metadata:
  name: "default"
spec:
  allowedNamespaces:{}  # matches all namespaces

AWSClusterControllerIdentity is immutable to avoid any unwanted overrides to the allowed namespaces, especially during upgrading clusters.

AWSClusterStaticIdentity

AWSClusterStaticIdentity represents static AWS credentials, which are stored in a Secret.

Example: Below, an AWSClusterStaticIdentity is created that allows access to the AWSClusters that are in “test” namespace. The identity credentials that will be used by “test” AWSCluster are stored in “test-account-creds” secret.

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSCluster
metadata:
  name: "test"
  namespace: "test"
spec:
  region: "eu-west-1"
  identityRef:
    kind: AWSClusterStaticIdentity
    name: test-account
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterStaticIdentity
metadata:
  name: "test-account"
spec:
  secretRef:
    name: test-account-creds
    namespace: capa-system
  allowedNamespaces:
    selector:
      matchLabels:
        cluster.x-k8s.io/ns: "testlabel"
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    cluster.x-k8s.io/ns: "testlabel"
  name: "test"
---
apiVersion: v1
kind: Secret
metadata:
  name: "test-account-creds"
  namespace: capa-system
stringData:
 AccessKeyID: AKIAIOSFODNN7EXAMPLE
 SecretAccessKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

AWSClusterRoleIdentity

AWSClusterRoleIdentity allows CAPA to assume a role either in the same or another AWS account, using the STS::AssumeRole API. The assumed role could be used by the AWSClusters that is in the allowedNamespaces.

Example: Below, an AWSClusterRoleIdentity instance, which will be used by AWSCluster “test”, is created. This role will be assumed by the source identity at runtime. Source identity can be of any identity type. Role is assumed in the beginning once and after, whenever the assumed role’s credentials are expired.

---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSCluster
metadata:
  name: "test"
  namespace: "test"
spec:
  region: "eu-west-1"
  identityRef:
    kind: AWSClusterRoleIdentity
    name: test-account-role
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterRoleIdentity
metadata:
  name: "test-account-role"
spec:
  allowedNamespaces:
    list: # allows only "test" namespace to use this identity
      "test"
  roleARN: "arn:aws:iam::123456789:role/CAPARole"
  sourceIdentityRef:
    kind: AWSClusterStaticIdentity
    name: test-account-creds

Nested role assumption is also supported. Example: Below, “multi-tenancy-nested-role” will be assumed by “multi-tenancy-role”, which will be assumed by the “default” AWSClusterControllerIdentity

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterRoleIdentity
metadata:
  name: multi-tenancy-role
spec:
  allowedNamespaces:
    list: []
  durationSeconds: 900 # default and min value is 900 seconds
  roleARN: arn:aws:iam::11122233344:role/multi-tenancy-role
  sessionName: multi-tenancy-role-session
  sourceidentityRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: AWSClusterControllerIdentity
    name: default
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterRoleIdentity
metadata:
  name: multi-tenancy-nested-role
spec:
  allowedNamespaces:
    list: []
  roleARN: arn:aws:iam::11122233355:role/multi-tenancy-nested-role
  sessionName: multi-tenancy-nested-role-session
  sourceidentityRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
    kind: AWSClusterRoleIdentity
    name: multi-tenancy-role

Secure Access to Identitys

allowedNamespaces field is used to grant access to the namespaces to use Identitys. Only AWSClusters that are created in one of the Identity’s allowed namespaces can use that Identity. allowedNamespaces are defined by providing either a list of namespaces or label selector to select namespaces.

Note that the capa-eks-control-plane-system namespace will need to be included in the allow namespace list and/or have labels added to allow access to identities used by AWSClusters.

Examples

An empty allowedNamespaces indicates that the Identity can be used by all namespaces.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterControllerIdentity
spec:
  allowedNamespaces:{}  # matches all namespaces

Having a nil list and a nil selector is the same with having an empty allowedNamespaces (Identity can be used by all namespaces).

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterControllerIdentity
spec:
  allowedNamespaces:
    list: nil
    selector: nil

A nil allowedNamespaces indicates that the Identity cannot be used from any namespace.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterControllerIdentity
spec:
  allowedNamespaces:  # this is same with not providing the field at all or allowedNamespaces: null

The union of namespaces that are matched by selector and the namespaces that are in the list is granted access to the identity. The namespaces that are not in the list and not matching the selector will not have access.

Nil or empty list matches no namespaces. Nil or empty selector matches no namespaces. If list is nil and selector is empty OR list is empty and selector is nil, Identity cannot be used from any namespace. Because in this case, allowedNamespaces is not empty or nil, and neither list nor selector allows any namespaces, so the union is empty.

# Matches no namespaces
allowedNamespaces:
  list: []
# Matches no namespaces
allowedNamespaces:
  selector: {}
# Matches no namespaces
allowedNamespaces:
  list: null
  selector: {}
# Matches no namespaces
allowedNamespaces:
  list: []
  selector: {}

Important The default behaviour of an empty label selector is to match all objects, however here we do not follow that behavior to avoid unintended access to the identitys. This is consistent with core cluster API selectors, e.g., Machine and ClusterResourceSet selectors. The result of matchLabels and matchExpressions are ANDed.

In Kubernetes selectors, matchLabels and matchExpressions are ANDed. In the example below, list is empty/nil, so does not allow any namespaces and selector matches with only default namespace. Since list and selector results are ORed, default namespace can use this identity.

kind: namespace
metadata:
  name: default
  labels:
    environment: dev
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterControllerIdentity
spec:
  allowedNamespaces:
    list: null # or []
    selector:
      matchLabels:
        namespace: default
      matchExpressions:
        - {key: environment, operator: In, values: [dev]}

EKS Support in the AWS Provider

  • Feature status: Stable
  • Feature gate (required): EKS=true
  • Feature gate (optional): EKSEnableIAM=true,EKSAllowAddRoles=true

Overview

The AWS provider supports creating EKS based cluster. Currently the following features are supported:

  • Provisioning/managing an Amazon EKS Cluster
  • Upgrading the Kubernetes version of the EKS Cluster
  • Attaching a self-managed machines as nodes to the EKS cluster
  • Creating a machine pool and attaching it to the EKS cluster. See machine pool docs for details.
  • Creating a managed machine pool and attaching it to the EKS cluster. See machine pool docs for details
  • Managing “EKS Addons”. See addons for further details
  • Creating an EKS fargate profile
  • Managing aws-iam-authenticator configuration

Note: machine pools and fargate profiles are still classed as experimental.

The implementation introduces the following CRD kinds:

  • AWSManagedControlPlane - specifies the EKS Cluster in AWS and used by the Cluster API AWS Managed Control plane (MACP)
  • AWSManagedMachinePool - defines the managed node pool for the cluster
  • EKSConfig - used by Cluster API bootstrap provider EKS (CABPE)

And a number of new templates are available in the templates folder for creating a managed workload cluster.

SEE ALSO

Prerequisites

To use EKS you must give the controller the required permissions. The easiest way to do this is by using clusterawasadm. For instructions on how to do this see the prerequisites.

When using clusterawsadm and enabling EKS support a new IAM role will be created for you called eks-controlplane.cluster-api-provider-aws.sigs.k8s.io. This role is the IAM role that will be used for the EKS control plane if you don’t specify your own role and if EKSEnableIAM isn’t enabled (see the enabling docs for further information).

Additionally using clusterawsadm will add permissions to the controllers.cluster-api-provider-aws.sigs.k8s.io policy for EKS to function properly.

Enabling EKS Support

Support for EKS is enabled by default when you use the AWS infrastructure provider. For example:

clusterctl init --infrastructure aws

Enabling optional EKS features

There are additional EKS experimental features that are disabled by default. The sections below cover how to enable these features.

Machine Pools

To enable support for machine pools the MachinePool feature flag must be set to to true. This can be done using the EXP_MACHINE_POOL environment variable:

export EXP_MACHINE_POOL=true
clusterctl init --infrastructure aws

See the machine pool documentation for further information.

NOTE: you will need to enable the creation of the default IAM role. The easiest way is using clusterawsadm, for instructions see the prerequisites.

IAM Roles Per Cluster

By default EKS clusters will use the same IAM roles (i.e. control plane, node group roles). There is a feature that allows each cluster to have its own IAM roles. This is done by enabling the EKSEnableIAM feature flag. This can be done before running clusterctl init by using the the CAPA_EKS_IAM environment variable:

export CAPA_EKS_IAM=true
clusterctl init --infrastructure aws

NOTE: you will need the correct prerequisities for this. The easiest way is using clusterawsadm and setting iamRoleCreation to true, for instructions see the prerequisites.

Additional Control Plane Roles

You can add additional roles to the control plane role that is created for an EKS cluster. To use this you must enable the EKSAllowAddRoles feature flag. This can be done before running clusterctl init by using the CAPA_EKS_ADD_ROLES environment variable:

export CAPA_EKS_IAM=true
export CAPA_EKS_ADD_ROLES=true
clusterctl init --infrastructure aws

NOTE: to use this feature you must also enable the CAPA_EKS_IAM feature.

EKS Fargate Profiles

You can use Fargate Profiles with EKS. To use this you must enable the EKSFargate feature flag. This can be done before running clusterctl init by using the EXP_EKS_FARGATE environmnet variable:

export EXP_EKS_FARGATE=true
clusterctl init --infrastructure aws

NOTE: you will need to enable the creation of the default Fargate IAM role. The easiest way is using clusterawsadm and using the fargate configuration option, for instructions see the prerequisites.

Pod Networking

When creating a EKS cluster the Amazon VPC CNI will be used by default for Pod Networking.

When using the AWS Console to create an EKS cluster with a Kubernetes version of v1.18 or greater you are required to select a specific version of the VPC CNI to use.

Using the VPC CNI Addon

You can use an explicit version of the Amazon VPC CNI by using the vpc-cni EKS addon. See the addons documentation for further details of how to use addons.

Using an alternative CNI

There may be scenarios where you do not want to use the Amazon VPC CNI. EKS supports a number of alternative CNIs such as Calico and Weave Net (see docs for full list).

There are a number of ways to install an alternative CNI into the cluster. One option is to use a ClusterResourceSet to apply the required artifacts to a newly provisioned cluster.

When using an alternative CNI you will want to delete the Amazon VPC CNI, especially for a cluster using v1.17 or less. This can be done via the disableVPCCNI property of the AWSManagedControlPlane:

kind: AWSManagedControlPlane
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
metadata:
  name: "capi-managed-test-control-plane"
spec:
  region: "eu-west-2"
  sshKeyName: "capi-management"
  version: "v1.18.0"
  disableVPCCNI: true

You cannot set disableVPCCNI to true if you are using the VPC CNI addon or if you have specified a secondary CIDR block.

Additional Information

See the AWS documentation for further details of EKS pod networking.

Creating a EKS cluster

New “eks” cluster templates have been created that you can use with clusterctl to create a EKS cluster. To create a EKS cluster with self-managed nodes (a.k.a machines):

clusterctl generate cluster capi-eks-quickstart --flavor eks --kubernetes-version v1.17.3 --worker-machine-count=3 > capi-eks-quickstart.yaml

To create a EKS cluster with a managed node group (a.k.a managed machine pool):

clusterctl generate cluster capi-eks-quickstart --flavor eks-managedmachinepool --kubernetes-version v1.17.3 --worker-machine-count=3 > capi-eks-quickstart.yaml

NOTE: When creating an EKS cluster only the MAJOR.MINOR of the -kubernetes-version is taken into consideration.

Kubeconfig

When creating an EKS cluster 2 kubeconfigs are generated and stored as secrets in the managmenet cluster. This is different to when you create a non-managed cluster using the AWS provider.

User kubeconfig

This should be used by users that want to connect to the newly created EKS cluster. The name of the secret that contains the kubeconfig will be [cluster-name]-user-kubeconfig where you need to replace [cluster-name] with the name of your cluster. The -user-kubeconfig in the name indicates that the kubeconfig is for the user use.

To get the user kubeconfig for a cluster named managed-test you can run a command similar to:

kubectl --namespace=default get secret managed-test-user-kubeconfig \
   -o jsonpath={.data.value} | base64 --decode \
   > managed-test.kubeconfig

Cluster API (CAPI) kubeconfig

This kubeconfig is used internally by CAPI and shouldn’t be used outside of the management server. It is used by CAPI to perform operations, such as draining a node. The name of the secret that contains the kubeconfig will be [cluster-name]-kubeconfig where you need to replace [cluster-name] with the name of your cluster. Note that there is NO -user in the name.

The kubeconfig is regenerated every sync-period as the token that is embedded in the kubeconfig is only valid for a short period of time. When EKS support is enabled the maximum sync period is 10 minutes. If you try to set --sync-period to greater than 10 minutes then an error will be raised.

EKS Console

To use the Amazon EKS Console to view workloads running in an EKS cluster created using the AWS provider (CAPA) you can do the following:

  1. Create a new policy with the required IAM permissions for the console. This example can be used. For example, a policy called EKSViewNodesAndWorkloads.

  2. Assign the policy created in step 1) to a IAM user or role for the users of your EKS cluster

  3. Map the IAM user or role from step 2) to a Kubernetes user that has the RBAC permissions to view the Kubernetes resources. This needs to be done via the aws-auth configmap (used by aws-iam-authenticator) which is generated by the AWS provider. This mapping can be specified using in the AWSManagedControlPlane, for example:

kind: AWSManagedControlPlane
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
metadata:
  name: "capi-managed-test-control-plane"
spec:
  region: "eu-west-2"
  sshKeyName: "capi-management"
  version: "v1.18.0"
  iamAuthenticatorConfig:
    mapRoles:
    - username: "kubernetes-admin"
      rolearn: "arn:aws:iam::1234567890:role/AdministratorAccess"
      groups:
      - "system:masters"

In the sample above the arn:aws:iam::1234567890:role/AdministratorAccess IAM role has the EKSViewNodesAndWorkloads policy attached (created in step 1.)

EKS Addons

EKS Addons can be used with EKS clusters created using Cluster API Provider AWS.

Addons are supported in EKS clusters using Kubernetes v1.18 or greater.

Installing addons

To install an addon you need to declare them by specifying the name, version and optionally how conflicts should be resolved in the AWSManagedControlPlane. For example:

kind: AWSManagedControlPlane
apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
metadata:
  name: "capi-managed-test-control-plane"
spec:
  region: "eu-west-2"
  sshKeyName: "capi-management"
  version: "v1.18.0"
  addons:
    - name: "vpc-cni"
      version: "v1.6.3-eksbuild.1"
      conflictResolution: "overwrite"

Additionally, there is a cluster flavor called eks-managedmachinepool-vpccni that you can use with clusterctl:

clusterctl generate cluster my-cluster --kubernetes-version v1.18.0 --flavor eks-managedmachinepool-vpccni > my-cluster.yaml

Updating Addons

To update the version of an addon you need to edit the AWSManagedControlPlane instance and update the version of the addon you want to update. Using the example from the previous section we would do:

...
  addons:
    - name: "vpc-cni"
      version: "v1.7.5-eksbuild.1"
      conflictResolution: "overwrite"
...

Deleting Addons

To delete an addon from a cluster you need to edit the AWSManagedControlPlane instance and remove the entry for the addon you want to delete.

Viewing installed addons

You can see what addons are installed on your EKS cluster by looking in the Status of the AWSManagedControlPlane instance.

Additionally you can run the following command:

clusterawsadm eks addons list-installed -n <<eksclustername>>

Viewing available addons

You can see what addons are available to your EKS cluster by running the following command:

clusterawsadm eks addons list-available -n <<eksclustername>>

Enabling Encryption

To enable encryption when creating a cluster you need to create a new KMS key that has an alias name starting with cluster-api-provider-aws-.

For example, arn:aws:kms:eu-north-1:12345678901:alias/cluster-api-provider-aws-key1.

You then need to specify this alias in the encryptionConfig of the AWSManagedControlPlane:

kind: AWSManagedControlPlane
apiVersion: controlplane.cluster.x-k8s.io/v1alpha4
metadata:
  name: "capi-managed-test-control-plane"
spec:
  ...
  encryptionConfig:
    provider: "arn:aws:kms:eu-north-1:12345678901:alias/cluster-api-provider-aws-key1"
    resources:
    - "secrets"

Custom KMS Alias Prefix

If you would like to use a different alias prefix then you can use the kmsAliasPrefix in the optional configuration file for clusterawsadm:

clusterawsadm bootstrap iam create-stack --config custom-prefix.yaml

And the contents of the configuration file:

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  eks:
    enable: true
    kmsAliasPrefix: "my-prefix-*

EKS Cluster Upgrades

Control Plane Upgrade

Upgrading the Kubernetes version of the control plane is supported by the provider. To perform an upgrade you need to update the version in the spec of the AWSManagedControlPlane. Once the version has changed the provider will handle the upgrade for you.

You can only upgrade a EKS cluster by 1 minor version at a time. If you attempt to upgrade the version by more then 1 minor version the provider will ensure the upgrade is done in multiple steps of 1 minor version. For example upgrading from v1.15 to v1.17 would result in your cluster being upgraded v1.15 -> v1.16 first and then v1.16 to v1.17.

Consuming Existing AWS Infrastructure

Normally, Cluster API will create infrastructure on AWS when standing up a new workload cluster. However, it is possible to have Cluster API re-use existing AWS infrastructure instead of creating its own infrastructure. Follow the instructions below to configure Cluster API to consume existing AWS infrastructure.

Prerequisites

In order to have Cluster API consume existing AWS infrastructure, you will need to have already created the following resources:

  • A VPC
  • One or more private subnets (subnets that do not have a route to an Internet gateway)
  • A NAT gateway for each private subnet, along with associated Elastic IP addresses (only needed if the nodes require access to the Internet, i.e. pulling public images)
    • A public subnet in the same Availability Zone (AZ) for each private subnet (this is required for NAT gateways to function properly)
  • An Internet gateway for all public subnets (only required if the workload cluster is set to use an Internet facing load balancer or one or more NAT gateways exist in the VPC)
  • Route table associations that provide connectivity to the Internet through a NAT gateway (for private subnets) or the Internet gateway (for public subnets)
  • VPC endpoints for ec2, elasticloadbalancing, secretsmanager an autoscaling (if using MachinePools) when the private Subnets do not have a NAT gateway

You will need the ID of the VPC and subnet IDs that Cluster API should use. This information is available via the AWS Management Console or the AWS CLI.

Note that there is no need to create an Elastic Load Balancer (ELB), security groups, or EC2 instances; Cluster API will take care of these items.

If you want to use existing security groups, these can be specified and new ones will not be created.

Tagging AWS Resources

Cluster API itself does tag AWS resources it creates. The sigs.k8s.io/cluster-api-provider-aws/cluster/<cluster-name> (where <cluster-name> matches the metadata.name field of the Cluster object) tag, with a value of owned, tells Cluster API that it has ownership of the resource. In this case, Cluster API will modify and manage the lifecycle of the resource.

When consuming existing AWS infrastructure, the Cluster API AWS provider does not require any tags to be present. The absence of the tags on an AWS resource indicates to Cluster API that it should not modify the resource or attempt to manage the lifecycle of the resource.

However, the built-in Kubernetes AWS cloud provider does require certain tags in order to function properly. Specifically, all subnets where Kubernetes nodes reside should have the kubernetes.io/cluster/<cluster-name> tag present. Private subnets should also have the kubernetes.io/role/internal-elb tag with a value of 1, and public subnets should have the kubernetes.io/role/elb tag with a value of 1. These latter two tags help the cloud provider understand which subnets to use when creating load balancers.

Finally, if the controller manager isn’t started with the --configure-cloud-routes: "false" parameter, the route table(s) will also need the kubernetes.io/cluster/<cluster-name> tag. (This parameter can be added by customizing the KubeadmConfigSpec object of the KubeadmControlPlane object.)

Configuring the AWSCluster Specification

Specifying existing infrastructure for Cluster API to use takes place in the specification for the AWSCluster object. Specifically, you will need to add an entry with the VPC ID and the IDs of all applicable subnets into the network field. Here is an example:

For EC2

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: AWSCluster

For EKS

apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
kind: AWSManagedControlPlane
spec:
  network:
    vpc:
      id: vpc-0425c335226437144
    subnets:
    - id: subnet-0261219d564bb0dc5
    - id: subnet-0fdcccba78668e013

When you use kubectl apply to apply the Cluster and AWSCluster specifications to the management cluster, Cluster API will use the specified VPC ID and subnet IDs, and will not create a new VPC, new subnets, or other associated resources. It will, however, create a new ELB and new security groups.

Placing EC2 Instances in Specific AZs

To distribute EC2 instances across multiple AZs, you can add information to the Machine specification. This is optional and only necessary if control over AZ placement is desired.

To tell Cluster API that an EC2 instance should be placed in a particular AZ but allow Cluster API to select which subnet in that AZ can be used, add this to the Machine specification:

spec:
  failureDomain: "us-west-2a"

If using a MachineDeployment, specify AZ placement like so:

spec:
  template:
    spec:
      failureDomain: "us-west-2b"

Note that all replicas within a MachineDeployment will reside in the same AZ.

Placing EC2 Instances in Specific Subnets

To specify that an EC2 instance should be placed in a specific subnet, add this to the AWSMachine specification:

spec:
  subnet:
    id: subnet-0a3507a5ad2c5c8c3

When using MachineDeployments, users can control subnet selection by adding information to the AWSMachineTemplate associated with that MachineDeployment, like this:

spec:
  template:
    spec:
      subnet:
        id: subnet-0a3507a5ad2c5c8c3

Users may either specify failureDomain on the Machine or MachineDeployment objects, or users may explicitly specify subnet IDs on the AWSMachine or AWSMachineTemplate objects. If both are specified, the subnet ID is used and the failureDomain is ignored.

Security Groups

To use existing security groups for instances for a cluster, add this to the AWSCluster specification:

spec:
  network:
    securityGroupOverrides:
      bastion: sg-0350a3507a5ad2c5c8c3
      controlplane: sg-0350a3507a5ad2c5c8c3
      apiserver-lb: sg-0200a3507a5ad2c5c8c3
      node: sg-04e870a3507a5ad2c5c8c3
      lb: sg-00a3507a5ad2c5c8c3

Any additional security groups specified in an AWSMachineTemplate will be applied in addition to these overriden security groups.

To specify additional security groups for the control plane load balancer for a cluster, add this to the AWSCluster specification:

spec:
  controlPlaneLoadBalancer:
    AdditionalsecurityGroups:
    - sg-0200a3507a5ad2c5c8c3
    - ...   

Caveats/Notes

  • When both public and private subnets are available in an AZ, CAPI will choose the private subnet in the AZ over the public subnet for placing EC2 instances.
  • If you configure CAPI to use existing infrastructure as outlined above, CAPI will not create an SSH bastion host. Combined with the previous bullet, this means you must make sure you have established some form of connectivity to the instances that CAPI will create.

Specifying the IAM Role to use for Management Components

Prerequisites

To be able to specify the IAM role that the management components should run as your cluster must be set up with the ability to assume IAM roles using one of the following solutions:

Setting IAM Role

Set the AWS_CONTROLLER_IAM_ROLE environment variable to the ARN of the IAM role to use when performing the clustercrl init command.

For example:

export AWS_CONTROLLER_IAM_ROLE=arn:aws:iam::1234567890:role/capa-management-components
clusterctl init --infrastructure=aws

IAM Role Trust Policy

IAM Roles for Service Accounts

When creating the IAM role the following trust policy will need to be used with the AWS_ACCOUNT_ID, AWS_REGION and OIDC_PROVIDER_ID environment variables replaced.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.eks.${AWS_REGION}.amazonaws.com/id/${OIDC_PROVIDER_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "ForAnyValue:StringEquals": {
          "oidc.eks.${AWS_REGION}.amazonaws.com/id/${OIDC_PROVIDER_ID}:sub": [
            "system:serviceaccount:capa-system:capa-controller-manager",
            "system:serviceaccount:capi-system:capi-controller-manager",
            "system:serviceaccount:capa-eks-control-plane-system:capa-eks-control-plane-controller-manager",
            "system:serviceaccount:capa-eks-bootstrap-system:capa-eks-bootstrap-controller-manager",
          ]
        }
      }
    }
  ]
}

If you plan to use the controllers.cluster-api-provider-aws.sigs.k8s.io role created by clusterawsadm then you’ll need to add the following to your AWSIAMConfiguration:

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  clusterAPIControllers:
    disabled: false
    trustStatements:
    - Action:
      - "sts:AssumeRoleWithWebIdentity"
      Effect: "Allow"
      Principal:
        Federated:
        - "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.eks.${AWS_REGION}.amazonaws.com/id/${OIDC_PROVIDER_ID}"
      Condition:
        "ForAnyValue:StringEquals":
          "oidc.eks.${AWS_REGION}.amazonaws.com/id/${OIDC_PROVIDER_ID}:sub":
            - system:serviceaccount:capa-system:capa-controller-manager
            - system:serviceaccount:capa-eks-control-plane-system:capa-eks-control-plane-controller-manager # Include if also using EKS

With this you can then set AWS_CONTROLLER_IAM_ROLE to arn:aws:iam::${AWS_ACCOUNT_ID}:role/controllers.cluster-api-provider-aws.sigs.k8s.io

Kiam / kube2iam

When creating the IAM role the you will need to give apply the kubernetes.io/cluster/${CLUSTER_NAME}/role": "enabled" tag to the role and use the following trust policy with the AWS_ACCOUNT_ID and CLUSTER_NAME environment variables correctly replaced.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}.worker-node-role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

If you plan to use the controllers.cluster-api-provider-aws.sigs.k8s.io role created by clusterawsadm then you’ll need to add the following to your AWSIAMConfiguration:

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  clusterAPIControllers:
    disabled: false
    trustStatements:
      - Action:
        - "sts:AssumeRole"
        Effect: "Allow"
        Principal:
          Service:
          - "ec2.amazonaws.com"
      - Action:
        - "sts:AssumeRole"
        Effect: "Allow"
        Principal:
          AWS:
          - "arn:aws:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}.worker-node-role"

With this you can then set AWS_CONTROLLER_IAM_ROLE to arn:aws:iam::${AWS_ACCOUNT_ID}:role/controllers.cluster-api-provider-aws.sigs.k8s.io

Multi-AZ Control Plane

Overview

By default, the control plane of a workload cluster created by CAPA will span multiple availability zones (AZs) (also referred to as “failure domains”) when using multiple control plane nodes. This is because CAPA will, by default, create public and private subnets in all the AZs of a region (up to a maximum of 3 AZs by default). If a region has more than 3 AZs then CAPA will pick 3 AZs to use.

Configuring CAPA to Use Specific AZs

To explicitly instruct CAPA to create resources in specific AZs (and not by random), users can add a network object to the AWSCluster specification. Here is an example network that creates resources across three AZs in the “us-west-2” region:

spec:
  network:
    vpc:
      cidrBlock: 10.50.0.0/16
    subnets:
    - availabilityZone: us-west-2a
      cidrBlock: 10.50.0.0/20
      isPublic: true
    - availabilityZone: us-west-2a
      cidrBlock: 10.50.16.0/20
    - availabilityZone: us-west-2b
      cidrBlock: 10.50.32.0/20
      isPublic: true
    - availabilityZone: us-west-2b
      cidrBlock: 10.50.48.0/20
    - availabilityZone: us-west-2c
      cidrBlock: 10.50.64.0/20
      isPublic: true
    - availabilityZone: us-west-2c
      cidrBlock: 10.50.80.0/20

Specifying the CIDR block alone for the VPC is not enough; users must also supply a list of subnets that provides the desired AZ, the CIDR for the subnet, and whether the subnet is public (has a route to an Internet gateway) or is private (does not have a route to an Internet gateway).

Note that CAPA insists that there must be a public subnet (and associated Internet gateway), even if no public load balancer is requested for the control plane. Therefore, for every AZ where a control plane node should be placed, the network object must define both a public and private subnet.

Once CAPA is provided with a network that spans multiple AZs, the KubeadmControlPlane controller will automatically distribute control plane nodes across multiple AZs. No further configuration from the user is required.

Note: this method can also be used if you do not want to split your EC2 instance across multiple AZs.

Changing AZ defaults

When creating default subnets by default a maximum of 3 AZs will be used. If you are creating a cluster in a region that has more than 3 AZs then 3 AZs will be picked based on alphabetical from that region.

If this default behavior for maximum number of AZs and ordered selection method doesn’t suit your requirements you can use the following to change the behaviour:

  • availabilityZoneUsageLimit - specifies the maximum number of availability zones (AZ) that should be used in a region when automatically creating subnets.
  • availabilityZoneSelection - specifies how AZs should be selected if there are more AZs in a region than specified by availabilityZoneUsageLimit. There are 2 selection schemes:
    • Ordered - selects based on alphabetical order
    • Random - selects AZs randomly in a region

For example if you wanted have a maximum of 2 AZs using a random selection scheme:

spec:
  network:
    vpc:
      availabilityZoneUsageLimit: 2
      availabilityZoneSelection: Random

Caveats

Deploying control plane nodes across multiple AZs is not a panacea to cure all availability concerns. The sizing and overall utilization of the cluster will greatly affect the behavior of the cluster and the workloads hosted there in the event of an AZ failure. Careful planning is needed to maximize the availability of the cluster even in the face of an AZ failure. There are also other considerations, like cross-AZ traffic charges, that should be taken into account.

External AWS cloud provider with AWS EBS CSI driver

Overview

From Kubernetes 1.21 onwards, the support for its in-tree AWS cloud provider and the EBS CSI driver is removed, hence there is a need to use out-of-tree cloud provider (Cloud Controller Manager - CCM) and a CSI driver in CAPA. For details, see Status of project and documentation of Cloud provider AWS

Using external cloud provider and EBS CSI driver in AWS workloads

Once Management cluster is ready, install external CCM and EBS CSI driver onto the CAPA workload cluster either manually or using ClusterResourceSets (CRS). To install them with CRS, create a CRS resource on the management cluster with labels, for example csi: external and ccm: external labels; and AWSCluster should be created with matching labels csi: external and ccm: external for CSI and CCM, respectively. Below are the example CRS manifests:

Verifying dynamically provisioned volumes with CSI driver

Once you have the cluster with external CCM and CSI controller running successfully, you can test the CSI driver functioning with following steps after switching to workload cluster:

  1. Create a service (say,nginx)
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
  namespace: default
spec:
  clusterIP: None
  ports:
    - name: nginx-web
      port: 80
  selector:
    app: nginx
  1. Create a storageclass and statefulset for the service created above with the persistent volume assigned to the storageclass:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: aws-ebs-volumes
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  csi.storage.k8s.io/fstype: xfs
  type: io1
  iopsPerGB: "100"
allowedTopologies:
  - matchLabelExpressions:
      - key: topology.ebs.csi.aws.com/zone
        values:
          - us-east-1a
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx-statefulset
spec:
  serviceName: "nginx-svc"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: k8s.gcr.io/nginx-slim:0.8
          ports:
            - name: nginx-web
              containerPort: 80
          volumeMounts:
            - name: nginx-volumes
              mountPath: /usr/share/nginx/html
      volumes:
        - name: nginx-volumes
          persistentVolumeClaim:
            claimName: nginx-volumes
  volumeClaimTemplates:
    - metadata:
        name: nginx-volumes
      spec:
        storageClassName: "aws-ebs-volumes"
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 4Gi
  1. Once you apply the above manifest, the EBS volumes will be created and attached to the worker nodes.

IMPORTANT WARNING: The CRDs from the AWS EBS CSI driver and AWS out-of-tree cloud provider gives issue while installing the respective controllers on the AWS Cluster, it doesn’t allow statefulsets to create the volume on existing EC2 instance. We need the CSI controller deployment and CCM pinned to the control plane which has right permissions to create, attach and mount the volumes to EC2 instances. To achieve this, you should add the node affinity rules to the CSI driver controller deployment and CCM DaemonSet manifests.

tolerations:
- key: node-role.kubernetes.io/master
  effect: NoSchedule
- effect: NoSchedule
  key: node-role.kubernetes.io/control-plane 
affinity:
  nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
      - matchExpressions:
          - key: node-role.kubernetes.io/control-plane
            operator: Exists
      - matchExpressions:
          - key: node-role.kubernetes.io/master
            operator: Exists

Restricting Cluster API to certain namespaces

Cluster-api-provider-aws controllers by default, reconcile cluster-api objects across all namespaces in the cluster. However, it is possible to restrict reconciliation to a single namespace and this document tells you how.

Contents

Use cases

  • Grouping clusters into a namespace based on the AWS account will allow managing clusters across multiple AWS accounts. This will require each cluster-api-provider-aws controller to have credentials to their respective AWS accounts. These credentials can be created as kubernetes secret and be mounted in the pod at /home/.aws or as environment variables.
  • Grouping clusters into a namespace based on their environment, (test, qualification, canary, production) will allow a phased rolling out of cluster-api-provider-aws releases.
  • Grouping clusters into a namespace based on the infrastructure provider will allow running multiple cluster-api provider implementations side-by-side and manage clusters across infrastructure providers.

Configuring cluster-api-provider-aws controllers

  • Create the namespace that cluster-api-provider-aws controller will watch for cluster-api objects
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: my-pet-clusters #edit if necessary
EOF
  • Deploy/edit aws-provider-controller-manager controller statefulset

Specifically, edit the container spec for cluster-api-aws-controller, in the aws-provider-controller-manager statefulset, to pass a value to the namespace CLI flag.

        - -namespace=my-pet-clusters # edit this if necessary

Once the aws-provider-controller-manager-0 pod restarts, cluster-api-provider-aws controllers will only reconcile the cluster-api objects in the my-pet-clusters namespace.

For v1alpha1 please refer to https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/release-0.3/docs/roleassumption.md

Creating clusters using cross account role assumption using KIAM

This document outlines the list of steps to create the target cluster via cross account role assumption using KIAM. KIAM lets the controller pod(s) to assume an AWS role that enables them create AWS resources necessary to create an operational cluster. This way we wouldn’t have to mount any AWS credentials or load environment variables to supply AWS credentials to the CAPA controller. This is automatically taken care by the KIAM components. Note: If you dont want to use KIAM and rather want to mount the credentials as secrets, you may still achieve cross account role assumption by using multiple profiles.

Glossary

  • Management cluster - The cluster that runs in AWS and is used to create target clusters in different AWS accounts
  • Target account - The account where the target cluster is created
  • Source account - The AWS account where the CAPA controllers for management cluster runs.

Goals

  1. The CAPA controllers are running in an AWS account and you want to create the target cluster in another AWS account.
  2. This assumes that you start with no existing clusters.

High level steps

  1. Creating a management cluster in AWS - This can be done by running the phases in clusterctl
    • Uses the existing provider components yaml
  2. Setting up cross account IAM roles
  3. Deploying the KIAM server/agent
  4. Create the target cluster (through KIAM)
    • Uses different provider components with no secrets and annotation to indicate the IAM Role to assume.

1. Creating the management cluster in AWS

Using clusterctl command we can create a new cluster in AWS which in turn will act as the management cluster to create the target cluster(in a different AWS account. This can be achieved by using the phases in clusterctl to perform all the steps except the pivoting. This will provide us with a bare-bones functioning cluster that we can use as a management cluster. To begin with follow the steps in this getting started guide to setup the environment except for creating the actual cluster. Instead follow the steps below to create the cluster.

create a new cluster using kind for bootstrapping purpose by running:

kind create cluster --name <CLUSTER_NAME>

and get its kube config path by running

export KIND_KUBECONFIG=`kind get kubeconfig-path`

Use the following commands to create new management cluster in AWS.

clusterctl alpha phases apply-cluster-api-components --provider-components examples/out/provider-components.yaml \
--kubeconfig $KIND_KUBECONFIG

clusterctl alpha phases apply-cluster --cluster examples/out/cluster.yaml --kubeconfig $KIND_KUBECONFIG

We only need to create the control plane on the cluster running in AWS source account. Since the example includes definition for a worker node, you may delete it.

clusterctl alpha phases apply-machines --machines examples/out/machines.yaml --kubeconfig $KIND_KUBECONFIG

clusterctl alpha phases get-kubeconfig --provider aws --cluster-name <CLUSTER_NAME> --kubeconfig $KIND_KUBECONFIG

export AWS_KUBECONFIG=$PWD/kubeconfig

kubectl apply -f examples/out/addons.yaml --kubeconfig $AWS_KUBECONFIG

Verify that all the pods in the kube-system namespace are running smoothly. Also you may remove the additional node in the machines example yaml since we are only interested in running the controllers that runs in control plane node (although its not required to make any changes there). You can destroy your local kind cluster by running

kind delete cluster --name <CLUSTER_NAME>

2. Setting up cross account roles

In this step we will create new roles/policy in across 2 different AWS accounts. Let us start by creating the roles in the account where the AWS controller runs. Following the directions posted in KIAM repo create a “kiam_server” role in AWS that only has a single managed policy with a single permission “sts:AssumeRole”. Also add a trust policy on the “kiam_server” role to include the role attached to the Control plane instance as a trusted entity. This looks something like this:

In “kiam_server” role (Source AWS account):

{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
       "AWS": "arn:aws:iam::<AWS_ACCOUNT_NUMBER>:role/control-plane.cluster-api-provider-aws.sigs.k8s.io"
     },
     "Action": "sts:AssumeRole"
   }
 ]
}

Next we must establish a link between this “kiam_server” role on source AWS account and the role on target AWS account that has the permissions to create new cluster. Begin by running the clusterawsadm cli to create a new stack on the target account where the target cluster is created. Make sure you use the credentials for target AWS account before creating the stack.

clusterawsadm alpha bootstrap create-stack

Then sign-in to the target AWS account to establish the link as mentioned above. Create a new Role with the permission policy set to “controllers.cluster-api-provider-aws.sigs.k8s.io”. Lets name this role “cluster-api” for future reference. Add a new trust relationship to include the “kiam_server” role from the source account as trusted entity. This is shown below:

In “controllers.cluster-api-provider-aws.sigs.k8s.io” role(target AWS account)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<SOURCE_AWS_ACCOUNT_NUMBER>:role/kserver"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

3. Deploying the KIAM server & agent

By now, your target cluster must be up & running. Make sure your KUBECONFIG pointing to the cluster in the target account.

Create new secrets using the steps outlined here Apply the manifest shown below: Make sure you update the argument to include your source AWS account “--assume-role-arn=arn:aws:iam::<SOURCE_AWS_ACCOUNT>:role/kiam_server” server.yaml

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: kiam-server
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9620"
      labels:
        app: kiam
        role: server
    spec:
      serviceAccountName: kiam-server
      nodeSelector:
        node-role.kubernetes.io/master: ""
      tolerations:
      - operator: "Exists"
      volumes:
        - name: ssl-certs
          hostPath:
            # for AWS linux or RHEL distros
            # path: /etc/pki/ca-trust/extracted/pem/
            path: /etc/ssl/certs/
        - name: tls
          secret:
            secretName: kiam-server-tls
      hostNetwork: true
      containers:
        - name: kiam
          image: quay.io/uswitch/kiam:v3.2
          imagePullPolicy: Always
          command:
            - /kiam
          args:
            - server
            - --json-log
            - --level=warn
            - --bind=0.0.0.0:443
            - --cert=/etc/kiam/tls/server.pem
            - --key=/etc/kiam/tls/server-key.pem
            - --ca=/etc/kiam/tls/ca.pem
            - --role-base-arn-autodetect
            - --assume-role-arn=arn:aws:iam::<SOURCE_AWS_ACCOUNT>:role/kiam_server
            - --sync=1m
            - --prometheus-listen-addr=0.0.0.0:9620
            - --prometheus-sync-interval=5s
          volumeMounts:
            - mountPath: /etc/ssl/certs
              name: ssl-certs
            - mountPath: /etc/kiam/tls
              name: tls
          livenessProbe:
            exec:
              command:
              - /kiam
              - health
              - --cert=/etc/kiam/tls/server.pem
              - --key=/etc/kiam/tls/server-key.pem
              - --ca=/etc/kiam/tls/ca.pem
              - --server-address=127.0.0.1:443
              - --gateway-timeout-creation=1s
              - --timeout=5s
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
              - /kiam
              - health
              - --cert=/etc/kiam/tls/server.pem
              - --key=/etc/kiam/tls/server-key.pem
              - --ca=/etc/kiam/tls/ca.pem
              - --server-address=127.0.0.1:443
              - --gateway-timeout-creation=1s
              - --timeout=5s
            initialDelaySeconds: 3
            periodSeconds: 10
            timeoutSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: kiam-server
  namespace: kube-system
spec:
  clusterIP: None
  selector:
    app: kiam
    role: server
  ports:
  - name: grpclb
    port: 443
    targetPort: 443
    protocol: TCP

agent.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: kube-system
  name: kiam-agent
spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9620"
      labels:
        app: kiam
        role: agent
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
        - operator: "Exists"
      volumes:
        - name: ssl-certs
          hostPath:
            # for AWS linux or RHEL distros
            #path: /etc/pki/ca-trust/extracted/pem/
            path: /etc/ssl/certs/
        - name: tls
          secret:
            secretName: kiam-agent-tls
        - name: xtables
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
      containers:
        - name: kiam
          securityContext:
            capabilities:
              add: ["NET_ADMIN"]
          image: quay.io/uswitch/kiam:v3.2
          imagePullPolicy: Always
          command:
            - /kiam
          args:
            - agent
            - --iptables
            - --host-interface=cali+
            - --json-log
            - --port=8181
            - --cert=/etc/kiam/tls/agent.pem
            - --key=/etc/kiam/tls/agent-key.pem
            - --ca=/etc/kiam/tls/ca.pem
            - --server-address=kiam-server:443
            - --prometheus-listen-addr=0.0.0.0:9620
            - --prometheus-sync-interval=5s
            - --gateway-timeout-creation=1s
          env:
            - name: HOST_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - mountPath: /etc/ssl/certs
              name: ssl-certs
            - mountPath: /etc/kiam/tls
              name: tls
            - mountPath: /var/run/xtables.lock
              name: xtables
          livenessProbe:
            httpGet:
              path: /ping
              port: 8181
            initialDelaySeconds: 3
            periodSeconds: 3

server-rbac.yaml

---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: kiam-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kiam-read
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - watch
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kiam-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kiam-read
subjects:
- kind: ServiceAccount
  name: kiam-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kiam-write
rules:
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kiam-write
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kiam-write
subjects:
- kind: ServiceAccount
  name: kiam-server
  namespace: kube-system

After deploying the above components make sure that the kiam_server and kiam_agent pods are up and running.

4. Create the target cluster

Make sure you create copy of the “aws/out” directory called “out2”. To create the target cluster we must update the provider_components.yaml generated in the out2 directory as shown below (to be run from the repository root directory):

cp examples/out examples/out2
vi examples/out2/provider-components.yaml
  1. Remove the credentials secret added at the bottom of the provider_components.yaml and do not mount the secret
  2. Add the following annotation to the template of aws-provider-controller-manager stateful set to specify the new role that was created in target account.
      annotations:
        iam.amazonaws.com/role: arn:aws:iam::<TARGET_AWS_ACCOUNT>:role/cluster-api
  1. Also add this below annotation to the “aws-provider-system” namespace
  annotations:
    iam.amazonaws.com/permitted: ".*"

Create a new cluster using the steps similar to the one used to create the source cluster. They are as follows:

export SOURCE_KUBECONFIG=<PATH_TO_SOURCE_CLUSTER_KUBECONFIG>

clusterctl alpha phases apply-cluster-api-components --provider-components examples/out2/provider-components.yaml \
--kubeconfig $SOURCE_KUBECONFIG

kubectl -f apply examples/out2/cluster.yaml --kubeconfig $SOURCE_KUBECONFIG

kubectl apply -f examples/out2/machines.yaml --kubeconfig $SOURCE_KUBECONFIG

clusterctl alpha phases get-kubeconfig --provider aws --cluster-name <TARGET_CLUSTER_NAME> --kubeconfig $SOURCE_KUBECONFIG
export KUBECONFIG=$PWD/kubeconfig

kubectl apply -f examples/out2/addons.yaml

This creates the new cluster in the target AWS account.

Userdata Privacy

Cluster API Provider AWS bootstraps EC2 instances to create and join Kubernetes clusters using instance user data. Because Kubernetes clusters are secured using TLS using multiple Certificate Authorities, these are generated by Cluster API and injected into the user data. It is important to note that without the configuring of host firewalls, processes can retrieve instance userdata from http://169.254.169.254/latest/api/token

Requirements

  • An AMI that includes the AWS CLI
  • AMIs using CloudInit
  • A working /bin/bash shell
  • LFS directory layout (i.e. /etc exists and is readable by CloudInit)

Listed AMIs on 1.16 and up should include the AWS CLI.

How Cluster API secures TLS secrets

In 0.5.x/v1alpha3, by default, Cluster API Provider AWS will use AWS Secrets Manager as a limited-time secret store, storing the userdata using KMS encryption at rest in AWS. The EC2 IMDS userdata will contain a boot script to download the encrypted userdata secret using instance profile permissions, then immediately delete it from AWS Secrets Manager, and then execute it.

To avoid guessing keys in the AWS Secrets Manager key-value store and to prevent collisions, the key is an encoding the Kubernetes namespace, cluster name and instance name, with a random string appended, providing ~256-bits of entropy.

Cluster API Provider AWS also stores the secret ARN in the AWSMachine spec, and will delete the secret if it isn’t already deleted and the machine has registered successfully against the workload cluster API server as a node. Cluster API Provider AWS will also attempt deletion of the secret if the AWSMachine is otherwise deleted or the EC2 instance is terminated or failed.

This method is only compatible with operating systems and distributions using cloud-init. If you are using a different bootstrap process, you will need to co-ordinate this externally and set the following in the specification of the AWSMachine types to disable the use of a cloud-init boothook:

cloudInit:
  insecureSkipSecretsManager: true

Troubleshooting

Script errors

cloud-init does not print boothook script errors to the systemd journal. Logs for the script, if it errored can be found in /var/log/cloud-init-output.log

Warning messages

Because cloud-init will attempt to read the final file at start, cloud-init will always print a /etc/secret-userdata.txt cannot be found message. This can be safely ignored.

Secrets manager console

The AWS secrets manager console should show secrets being created and deleted, with a lifetime of around a minute. No plaintext secret data will appear in the console as Cluster API Provider AWS stores the userdata as fragments of a gzipped data stream.

Troubleshooting

Resources aren’t being created

TODO

Target cluster’s control plane machine is up but target cluster’s apiserver not working as expected

If aws-provider-controller-manager-0 logs did not help, you might want to look into cloud-init logs, /var/log/cloud-init-output.log, on the controller host. Verifying kubelet status and logs may also provide hints:

journalctl -u kubelet.service
systemctl status kubelet

For reaching controller host from your local machine:

 ssh -i <private-key> -o "ProxyCommand ssh -W %h:%p -i <private-key> ubuntu@<bastion-IP>" ubuntu@<controller-host-IP>

private-key is the private key from the key-pair discussed in the ssh key pair section above.

kubelet on the control plane host failing with error: NoCredentialProviders

failed to run Kubelet: could not init cloud provider "aws": error finding instance i-0c276f2a1f1c617b2: "error listing AWS instances: \"NoCredentialProviders: no valid providers in chain. Deprecated.\\n\\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\""

This error can occur if CloudFormation stack is not created properly and IAM instance profile is missing appropriate roles. Run following command to inspect IAM instance profile:

$ aws iam get-instance-profile --instance-profile-name control-plane.cluster-api-provider-aws.sigs.k8s.io --output json
{
    "InstanceProfile": {
        "InstanceProfileId": "AIPAJQABLZS4A3QDU576Q",
        "Roles": [
            {
                "AssumeRolePolicyDocument": {
                    "Version": "2012-10-17",
                    "Statement": [
                        {
                            "Action": "sts:AssumeRole",
                            "Effect": "Allow",
                            "Principal": {
                                "Service": "ec2.amazonaws.com"
                            }
                        }
                    ]
                },
                "RoleId": "AROAJQABLZS4A3QDU576Q",
                "CreateDate": "2019-05-13T16:45:12Z",
                "RoleName": "control-plane.cluster-api-provider-aws.sigs.k8s.io",
                "Path": "/",
                "Arn": "arn:aws:iam::123456789012:role/control-plane.cluster-api-provider-aws.sigs.k8s.io"
            }
        ],
        "CreateDate": "2019-05-13T16:45:28Z",
        "InstanceProfileName": "control-plane.cluster-api-provider-aws.sigs.k8s.io",
        "Path": "/",
        "Arn": "arn:aws:iam::123456789012:instance-profile/control-plane.cluster-api-provider-aws.sigs.k8s.io"
    }
}

If instance profile does not look as expected, you may try recreating the CloudFormation stack using clusterawsadm as explained in the above sections.

IAM Permissions

Required to use clusterawasadm to provision IAM roles via CloudFormation

If using clusterawsadm to automate deployment of IAM roles via CloudFormation, you must have IAM administrative access as clusterawsadm will provision IAM roles and policies.

Required by Cluster API Provider AWS controllers

The Cluster API Provider AWS controller requires permissions to use EC2, ELB Autoscaling and optionally EKS. If provisioning IAM roles using clusterawsadm, these will be set up as the controllers.cluster-api-provider-aws.sigs.k8s.io IAM Policy, and attached to the controllers.cluster-api-provider-aws.sigs.k8s.io and control-plane.cluster-api-provider-aws.sigs.k8s.io IAM roles.

EC2 Provisioned Kubernetes Clusters

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AllocateAddress",
        "ec2:AssociateRouteTable",
        "ec2:AttachInternetGateway",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateInternetGateway",
        "ec2:CreateNatGateway",
        "ec2:CreateRoute",
        "ec2:CreateRouteTable",
        "ec2:CreateSecurityGroup",
        "ec2:CreateSubnet",
        "ec2:CreateTags",
        "ec2:CreateVpc",
        "ec2:ModifyVpcAttribute",
        "ec2:DeleteInternetGateway",
        "ec2:DeleteNatGateway",
        "ec2:DeleteRouteTable",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSubnet",
        "ec2:DeleteTags",
        "ec2:DeleteVpc",
        "ec2:DescribeAccountAttributes",
        "ec2:DescribeAddresses",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeImages",
        "ec2:DescribeNatGateways",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeNetworkInterfaceAttribute",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:DescribeVpcAttribute",
        "ec2:DescribeVolumes",
        "ec2:DetachInternetGateway",
        "ec2:DisassociateRouteTable",
        "ec2:DisassociateAddress",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyNetworkInterfaceAttribute",
        "ec2:ModifySubnetAttribute",
        "ec2:ReleaseAddress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:RunInstances",
        "ec2:TerminateInstances",
        "tag:GetResources",
        "elasticloadbalancing:AddTags",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:ConfigureHealthCheck",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
        "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
        "elasticloadbalancing:RemoveTags",
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeInstanceRefreshes",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateLaunchTemplateVersion",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DeleteLaunchTemplate",
        "ec2:DeleteLaunchTemplateVersions",
        "ec2:DescribeKeyPairs"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:CreateAutoScalingGroup",
        "autoscaling:UpdateAutoScalingGroup",
        "autoscaling:CreateOrUpdateTags",
        "autoscaling:StartInstanceRefresh",
        "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:DeleteTags"
      ],
      "Resource": [
        "arn:*:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "autoscaling.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "spot.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/*.cluster-api-provider-aws.sigs.k8s.io"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:DeleteSecret",
        "secretsmanager:TagResource"
      ],
      "Resource": [
        "arn:*:secretsmanager:*:*:secret:aws.cluster.x-k8s.io/*"
      ]
    }
  ]
}

With EKS Support

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AllocateAddress",
        "ec2:AssociateRouteTable",
        "ec2:AttachInternetGateway",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateInternetGateway",
        "ec2:CreateNatGateway",
        "ec2:CreateRoute",
        "ec2:CreateRouteTable",
        "ec2:CreateSecurityGroup",
        "ec2:CreateSubnet",
        "ec2:CreateTags",
        "ec2:CreateVpc",
        "ec2:ModifyVpcAttribute",
        "ec2:DeleteInternetGateway",
        "ec2:DeleteNatGateway",
        "ec2:DeleteRouteTable",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSubnet",
        "ec2:DeleteTags",
        "ec2:DeleteVpc",
        "ec2:DescribeAccountAttributes",
        "ec2:DescribeAddresses",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeInstances",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeImages",
        "ec2:DescribeNatGateways",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeNetworkInterfaceAttribute",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:DescribeVpcAttribute",
        "ec2:DescribeVolumes",
        "ec2:DetachInternetGateway",
        "ec2:DisassociateRouteTable",
        "ec2:DisassociateAddress",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyNetworkInterfaceAttribute",
        "ec2:ModifySubnetAttribute",
        "ec2:ReleaseAddress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:RunInstances",
        "ec2:TerminateInstances",
        "tag:GetResources",
        "elasticloadbalancing:AddTags",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:ConfigureHealthCheck",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
        "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
        "elasticloadbalancing:RemoveTags",
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeInstanceRefreshes",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateLaunchTemplateVersion",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DeleteLaunchTemplate",
        "ec2:DeleteLaunchTemplateVersions",
        "ec2:DescribeKeyPairs"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:CreateAutoScalingGroup",
        "autoscaling:UpdateAutoScalingGroup",
        "autoscaling:CreateOrUpdateTags",
        "autoscaling:StartInstanceRefresh",
        "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:DeleteTags"
      ],
      "Resource": [
        "arn:*:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "autoscaling.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/aws-service-role/elasticloadbalancing.amazonaws.com/AWSServiceRoleForElasticLoadBalancing"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:CreateServiceLinkedRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/aws-service-role/spot.amazonaws.com/AWSServiceRoleForEC2Spot"
      ],
      "Condition": {
        "StringLike": {
          "iam:AWSServiceName": "spot.amazonaws.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": [
        "arn:*:iam::*:role/*.cluster-api-provider-aws.sigs.k8s.io"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:DeleteSecret",
        "secretsmanager:TagResource"
      ],
      "Resource": [
        "arn:*:secretsmanager:*:*:secret:aws.cluster.x-k8s.io/*"
      ]
    }
  ]
}

Required by the Kubernetes AWS Cloud Provider

These permissions are used by the Kubernetes AWS Cloud Provider. If you are running with the in-tree cloud provider, this will typically be used by the controller-manager pod in the kube-system namespace.

If provisioning IAM roles using clusterawsadm, these will be set up as the control-plane.cluster-api-provider-aws.sigs.k8s.io IAM Policy, and attached to the control-plane.cluster-api-provider-aws.sigs.k8s.io IAM role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeTags",
        "ec2:DescribeInstances",
        "ec2:DescribeImages",
        "ec2:DescribeRegions",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVolumes",
        "ec2:CreateSecurityGroup",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyVolume",
        "ec2:AttachVolume",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateRoute",
        "ec2:DeleteRoute",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteVolume",
        "ec2:DetachVolume",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:DescribeVpcs",
        "elasticloadbalancing:AddTags",
        "elasticloadbalancing:AttachLoadBalancerToSubnets",
        "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:CreateLoadBalancerPolicy",
        "elasticloadbalancing:CreateLoadBalancerListeners",
        "elasticloadbalancing:ConfigureHealthCheck",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DeleteLoadBalancerListeners",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:DetachLoadBalancerFromSubnets",
        "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
        "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
        "elasticloadbalancing:CreateListener",
        "elasticloadbalancing:CreateTargetGroup",
        "elasticloadbalancing:DeleteListener",
        "elasticloadbalancing:DeleteTargetGroup",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeLoadBalancerPolicies",
        "elasticloadbalancing:DescribeTargetGroups",
        "elasticloadbalancing:DescribeTargetHealth",
        "elasticloadbalancing:ModifyListener",
        "elasticloadbalancing:ModifyTargetGroup",
        "elasticloadbalancing:RegisterTargets",
        "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
        "iam:CreateServiceLinkedRole",
        "kms:DescribeKey"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Required by all nodes

All nodes require these permissions in order to run, and are used by the AWS cloud provider run by kubelet.

If provisioning IAM roles using clusterawsadm, these will be set up as the nodes.cluster-api-provider-aws.sigs.k8s.io IAM Policy, and attached to the nodes.cluster-api-provider-aws.sigs.k8s.io IAM role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeRegions",
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:GetRepositoryPolicy",
        "ecr:DescribeRepositories",
        "ecr:ListImages",
        "ecr:BatchGetImage"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

When using EKS, the AmazonEKSWorkerNodePolicy and AmazonEKS_CNI_Policy AWS managed policies will also be attached to nodes.cluster-api-provider-aws.sigs.k8s.io IAM role.

clusterawsadm

Kubernetes Cluster API Provider AWS Management Utility

Synopsis

clusterawsadm provides helpers for bootstrapping Kubernetes Cluster API Provider AWS. Use clusterawsadm to view required AWS Identity and Access Management (IAM) policies as JSON docs, or create IAM roles and instance profiles automatically using AWS CloudFormation.

clusterawsadm additionally helps provide credentials for use with clusterctl.

clusterawsadm [flags]

Examples

  # Create AWS Identity and Access Management (IAM) roles for use with
  # Kubernetes Cluster API Provider AWS.
  clusterawsadm bootstrap iam create-cloudformation-stack
  
  # Encode credentials for use with clusterctl init
  export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
  clusterctl init --infrastructure aws

Options

  -h, --help    help for clusterawsadm
  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm eks

Commands related to EKS

clusterawsadm eks [flags]

Options

  -h, --help   help for eks

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm eks addons

Commands related to EKS addons

clusterawsadm eks addons [flags]

Options

  -h, --help   help for addons

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm eks addons list-available

List available EKS addons

Synopsis

Lists the addons that are available for use with an EKS cluster

clusterawsadm eks addons list-available [flags]

Options

  -n, --cluster-name string   The name of the cluster to get the list of available addons for
  -h, --help                  help for list-available
  -o, --output string         The output format of the results. Possible values: table,json,yaml (default "table")
  -r, --region string         The AWS region containing the EKS cluster

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm eks addons list-installed

List installed EKS addons

Synopsis

Lists the addons that are installed for an EKS cluster

clusterawsadm eks addons list-installed [flags]

Options

  -n, --cluster-name string   The name of the cluster to get the list of installed addons for
  -h, --help                  help for list-installed
  -o, --output string         The output format of the results. Possible values: table,json,yaml (default "table")
  -r, --region string         The AWS region containing the EKS cluster

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm resource

Commands related to AWS resources

Synopsis

All AWS resources related actions such as:

List of AWS resources created by CAPA

clusterawsadm resource [command] [flags]

Options

  -h, --help   help for resource

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm resource list

List all AWS resources created by CAPA

Synopsis

List AWS resources directly created by CAPA based on region and cluster-name. There are some indirect resources like Cloudwatch alarms, rules, etc which are not directly created by CAPA, so those resources are not listed here. If region and cluster-name are not set, then it will throw an error.

clusterawsadm resource list [flags]

Examples

  # List AWS resources directly created by CAPA in given region and clustername
  clusterawsadm resource list --region=us-east-1 --cluster-name=test-cluster

Options

  -n, --cluster-name string   The name of the cluster where AWS resources created by CAPA
  -h, --help                  help for list
  -o, --output string         The output format of the results. Possible values: table, json, yaml (default "table")
  -r, --region string         The AWS region where resources are created by CAPA

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm version

Print version of clusterawsadm

clusterawsadm version [flags]

Options

  -h, --help            help for version
  -o, --output string   Output format; available options are 'yaml', 'json' and 'short'

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

  • clusterawsadm - Kubernetes Cluster API Provider AWS Management Utility
Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm ami

AMI commands

Synopsis

All AMI related actions such as:

Copy AMIs based on Kubernetes version, OS etc from an AWS account where AMIs are stored

     to the current AWS account (use case: air-gapped deployments)

(to be implemented) List available AMIs

clusterawsadm ami [command] [flags]

Options

  -h, --help   help for ami

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm ami copy

Copy AMIs from an AWS account to the AWS account which credentials are provided

Synopsis

Copy AMIs based on Kubernetes version, OS, region from an AWS account where AMIs are stored to the current AWS account (use case: air-gapped deployments)

clusterawsadm ami copy [flags]

Examples

  # Copy AMI from the default AWS account where AMIs are stored.
  # Available os options: centos-7, ubuntu-18.04, ubuntu-20.04, amazon-2
  clusterawsadm ami copy --kubernetes-version=v1.18.12 --os=ubuntu-20.04  --region=us-west-2
  
  # owner-id and dry-run flags are optional. region can be set via flag or env
  clusterawsadm ami copy --os centos-7 --kubernetes-version=v1.19.4 --owner-id=111111111111 --dry-run
  
  # copy from us-east-1 to us-east-2
  clusterawsadm ami copy --os centos-7 --kubernetes-version=v1.19.4 --region us-east-2 --source-region us-east-1

Options

      --dry-run                     Check if AMI exists and can be copied
  -h, --help                        help for copy
      --kubernetes-version string   Kubernetes version of the AMI to be copied
      --os string                   Operating system of the AMI to be copied
      --owner-id string             The source AWS owner ID, where the AMI will be copied from (default "258751437250")
      --region string               The AWS region in which to provision
      --source-region string        Set if wanting to copy an AMI from a different region

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm ami encrypted-copy

Encrypt and copy AMI snapshot, then create an AMI with that snapshot

Synopsis

Find the AMI based on Kubernetes version, OS, region in the AWS account where AMIs are stored. Encrypt and copy the snapshot of the AMI to the current AWS account. Create an AMI with that snapshot.

clusterawsadm ami encrypted-copy [flags]

Examples

  # Create an encrypted AMI:
  # Available os options: centos-7, ubuntu-18.04, ubuntu-20.04, amazon-2
  clusterawsadm ami encrypted-copy --kubernetes-version=v1.18.12 --os=ubuntu-20.04  --region=us-west-2
  
  # owner-id and dry-run flags are optional. region can be set via flag or env
  clusterawsadm ami encrypted-copy --os centos-7 --kubernetes-version=v1.19.4 --owner-id=111111111111 --dry-run
  
  # copy from us-east-1 to us-east-2
  clusterawsadm ami encrypted-copy --os centos-7 --kubernetes-version=v1.19.4 --owner-id=111111111111 --region us-east-2 --source-region us-east-1
  
  # Encrypt using a non-default KmsKeyId specified using Key ID:
  clusterawsadm ami encrypted-copy --os centos-7 --kubernetes-version=v1.19.4 --kms-key-id=key/1234abcd-12ab-34cd-56ef-1234567890ab
  
  # Encrypt using a non-default KmsKeyId specified using Key alias:
  clusterawsadm ami encrypted-copy --os centos-7 --kubernetes-version=v1.19.4 --kms-key-id=alias/ExampleAlias
  
  # Encrypt using a non-default KmsKeyId specified using Key ARN:
  clusterawsadm ami encrypted-copy --os centos-7 --kubernetes-version=v1.19.4 --kms-key-id=arn:aws:kms:us-east-1:012345678910:key/abcd1234-a123-456a-a12b-a123b4cd56ef
  
  # Encrypt using a non-default KmsKeyId specified using Alias ARN:
  clusterawsadm ami encrypted-copy --os centos-7 --kubernetes-version=v1.19.4 --kms-key-id=arn:aws:kms:us-east-1:012345678910:alias/ExampleAlias

Options

      --dry-run                     Check if AMI exists and can be copied
  -h, --help                        help for encrypted-copy
      --kms-key-id string           The ID of the KMS key for Amazon EBS encryption
      --kubernetes-version string   Kubernetes version of the AMI to be copied
      --os string                   Operating system of the AMI to be copied
      --owner-id string             The source AWS owner ID, where the AMI will be copied from (default "258751437250")
      --region string               The AWS region in which to provision
      --source-region string        Set if wanting to copy an AMI from a different region

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm ami list

List AMIs from the default AWS account where AMIs are stored

Synopsis

List AMIs based on Kubernetes version, OS, region. If no arguments are provided, it will print all AMIs in all regions, OS types for the supported Kubernetes versions. Supported Kubernetes versions start from the latest stable version and goes 2 release back: if the latest stable release is v1.20.4- v1.19.x and v1.18.x are supported. Note: First release of each version will be skipped, e.g., v1.21.0 To list AMIs of unsupported Kubernetes versions, --kubernetes-version flag needs to be provided.

clusterawsadm ami list [flags]

Examples

  # List AMIs from the default AWS account where AMIs are stored.
  # Available os options: centos-7, ubuntu-18.04, ubuntu-20.04, amazon-2
  clusterawsadm ami list --kubernetes-version=v1.18.12 --os=ubuntu-20.04  --region=us-west-2
  # To list all supported AMIs in all supported Kubernetes versions, regions, and linux distributions:
  clusterawsadm ami list

Options

  -h, --help                        help for list
      --kubernetes-version string   Kubernetes version of the AMI to be copied
      --os string                   Operating system of the AMI to be copied
  -o, --output string               The output format of the results. Possible values: table,json,yaml (default "table")
      --region string               The AWS region in which to provision

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap

bootstrap commands

Synopsis

In order to use Kubernetes Cluster API Provider AWS, an AWS account needs to be prepared with AWS Identity and Access Management (IAM) roles to be used by clusters as well as provide Kubernetes Cluster API Provider AWS with credentials to use to provision infrastructure.

clusterawsadm bootstrap [command] [flags]

Options

  -h, --help   help for bootstrap

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap credentials

Encode credentials to use with Kubernetes Cluster API Provider AWS

Synopsis

Encode credentials to use with Kubernetes Cluster API Provider AWS.

The utility will attempt to find credentials in the following order:

  1. Check for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
  2. Read the default credentials from the shared configuration files ~/.aws/credentials or the default profile in ~/.aws/config.
  3. Check for the presence of an EC2 IAM instance profile if it’s running on AWS.
  4. Check for ECS credentials.

IAM role assumption can be performed by using any valid configuration for the AWS CLI at: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html. For role assumption to be used, a region is required for the utility to use the AWS Security Token Service (STS). The utility resolves the region in the following order:

  1. Check for the --region flag.
  2. Check for the AWS_REGION environment variable.
  3. Check for the DEFAULT_AWS_REGION environment variable.
  4. Check that a region is specified in the shared configuration file.

The utility will then generate an ini-file with a default profile corresponding to the resolved credentials.

If a region cannot be found, for the purposes of using AWS Security Token Service, this utility will fall back to us-east-1. This does not affect the region in which clusters will be created.

In the case of an instance profile or role assumption, note that encoded credentials are time-limited.

clusterawsadm bootstrap credentials [flags]

Examples

  # Encode credentials from the environment for use with clusterctl
  export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
  clusterctl init --infrastructure aws

Options

  -h, --help   help for credentials

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap credentials encode-as-profile

Generate an AWS profile from the current environment

Synopsis

Generate an AWS profile from the current environment for the ephemeral bootstrap cluster.

The utility will attempt to find credentials in the following order:

  1. Check for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
  2. Read the default credentials from the shared configuration files ~/.aws/credentials or the default profile in ~/.aws/config.
  3. Check for the presence of an EC2 IAM instance profile if it’s running on AWS.
  4. Check for ECS credentials.

IAM role assumption can be performed by using any valid configuration for the AWS CLI at: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html. For role assumption to be used, a region is required for the utility to use the AWS Security Token Service (STS). The utility resolves the region in the following order:

  1. Check for the --region flag.
  2. Check for the AWS_REGION environment variable.
  3. Check for the DEFAULT_AWS_REGION environment variable.
  4. Check that a region is specified in the shared configuration file.

The utility will then generate an ini-file with a default profile corresponding to the resolved credentials.

If a region cannot be found, for the purposes of using AWS Security Token Service, this utility will fall back to us-east-1. This does not affect the region in which clusters will be created.

In the case of an instance profile or role assumption, note that encoded credentials are time-limited.

clusterawsadm bootstrap credentials encode-as-profile [flags]

Examples

  # Encode credentials from the environment for use with clusterctl
  export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
  clusterctl init --infrastructure aws

Options

  -h, --help            help for encode-as-profile
      --output string   Output for credential configuration (rawSharedConfig, base64SharedConfig) (default "base64SharedConfig")
      --region string   The AWS region in which to provision

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap iam

View required AWS IAM policies and create/update IAM roles using AWS CloudFormation

Synopsis

View/output AWS Identity and Access Management (IAM) policy documents required for configuring Kubernetes Cluster API Provider AWS as well as create/update AWS IAM resources using AWS CloudFormation.

clusterawsadm bootstrap iam [command] [flags]

Options

  -h, --help   help for iam

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap iam delete-cloudformation-stack

Delete an AWS CloudFormation stack

Synopsis

Delete the AWS CloudFormation stack that created AWS Identity and Access Management (IAM) resources for use with Kubernetes Cluster API Provider AWS.

clusterawsadm bootstrap iam delete-cloudformation-stack [flags]

Options

      --config string   clusterawsadm will load a bootstrap configuration from this file. The path may be
                        absolute or relative; relative paths start at the current working directory.
                        
                        The configuration file is a Kubernetes YAML using the
                        bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1/AWSIAMConfiguration
                        kind.
                        
                        Documentation for this kind can be found at:
                        https://pkg.go.dev/sigs.k8s.io/cluster-api-provider-aws/cmd/clusterawsadm/api/bootstrap/v1alpha1
                        
                        To see the default configuration, run 'clusterawsadm bootstrap iam print-config'.
  -h, --help            help for delete-cloudformation-stack
      --region string   The AWS region in which to provision

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap iam print-cloudformation-template

Print cloudformation template

Synopsis

Generate and print out a CloudFormation template that can be used to provision AWS Identity and Access Management (IAM) policies and roles for use with Kubernetes Cluster API Provider AWS.

clusterawsadm bootstrap iam print-cloudformation-template [flags]

Examples

  # Print out the default CloudFormation template.
  clusterawsadm bootstrap iam print-cloudformation-template
  
  # Print out a CloudFormation template using a custom configuration.
  clusterawsadm bootstrap iam print-cloudformation-template --config bootstrap_config.yaml

Options

      --config string   clusterawsadm will load a bootstrap configuration from this file. The path may be
                        absolute or relative; relative paths start at the current working directory.
                        
                        The configuration file is a Kubernetes YAML using the
                        bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1/AWSIAMConfiguration
                        kind.
                        
                        Documentation for this kind can be found at:
                        https://pkg.go.dev/sigs.k8s.io/cluster-api-provider-aws/cmd/clusterawsadm/api/bootstrap/v1alpha1
                        
                        To see the default configuration, run 'clusterawsadm bootstrap iam print-config'.
  -h, --help            help for print-cloudformation-template

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap iam print-config

Print configuration

Synopsis

Print configuration

clusterawsadm bootstrap iam print-config [flags]

Examples

  # Print the default configuration.
  clusterawsadm bootstrap iam print-config
  
  # Apply defaults to a configuration file and print the result
  clusterawsadm bootstrap iam print-config --config bootstrap_config.yaml

Options

      --config string   clusterawsadm will load a bootstrap configuration from this file. The path may be
                        absolute or relative; relative paths start at the current working directory.
                        
                        The configuration file is a Kubernetes YAML using the
                        bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1/AWSIAMConfiguration
                        kind.
                        
                        Documentation for this kind can be found at:
                        https://pkg.go.dev/sigs.k8s.io/cluster-api-provider-aws/cmd/clusterawsadm/api/bootstrap/v1alpha1
                        
                        To see the default configuration, run 'clusterawsadm bootstrap iam print-config'.
  -h, --help            help for print-config

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap iam print-policy

Generate and show an IAM policy

Synopsis

Generate and show an AWS Identity and Access Management (IAM) policy for Kubernetes Cluster API Provider AWS.

clusterawsadm bootstrap iam print-policy [flags]

Examples

  # Print out the IAM policy for the Kubernetes Cluster API Provider AWS Controller.
  clusterawsadm bootstrap iam print-policy --document AWSIAMManagedPolicyControllers
  
  # Print out the IAM policy for the Kubernetes Cluster API Provider AWS Controller using a given configuration file.
  clusterawsadm bootstrap iam print-policy --document AWSIAMManagedPolicyControllers --config bootstrap_config.yaml
  
  # Print out the IAM policy for the Kubernetes AWS Cloud Provider for the control plane.
  clusterawsadm bootstrap iam print-policy --document AWSIAMManagedPolicyCloudProviderControlPlane
  
  # Print out the IAM policy for the Kubernetes AWS Cloud Provider for all nodes.
  clusterawsadm bootstrap iam print-policy --document AWSIAMManagedPolicyCloudProviderNodes
  
  # Print out the IAM policy for the Kubernetes AWS EBS CSI Driver Controller.
  clusterawsadm bootstrap iam print-policy --document AWSEBSCSIPolicyController

Options

      --config string     clusterawsadm will load a bootstrap configuration from this file. The path may be
                          absolute or relative; relative paths start at the current working directory.
                          
                          The configuration file is a Kubernetes YAML using the
                          bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1/AWSIAMConfiguration
                          kind.
                          
                          Documentation for this kind can be found at:
                          https://pkg.go.dev/sigs.k8s.io/cluster-api-provider-aws/cmd/clusterawsadm/api/bootstrap/v1alpha1
                          
                          To see the default configuration, run 'clusterawsadm bootstrap iam print-config'.
      --document string   which document to show: [AWSIAMManagedPolicyControllers AWSIAMManagedPolicyControllersEKS AWSIAMManagedPolicyCloudProviderControlPlane AWSIAMManagedPolicyCloudProviderNodes AWSEBSCSIPolicyController]
  -h, --help              help for print-policy

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm bootstrap iam create-cloudformation-stack

Create or update an AWS CloudFormation stack

Synopsis

Create or update an AWS CloudFormation stack for bootstrapping Kubernetes Cluster API and Kubernetes AWS Identity and Access Management (IAM) permissions. To use this command, there must be AWS credentials loaded in this environment.

The utility will attempt to find credentials in the following order:

  1. Check for the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
  2. Read the default credentials from the shared configuration files ~/.aws/credentials or the default profile in ~/.aws/config.
  3. Check for the presence of an EC2 IAM instance profile if it’s running on AWS.
  4. Check for ECS credentials.

IAM role assumption can be performed by using any valid configuration for the AWS CLI at: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html. For role assumption to be used, a region is required for the utility to use the AWS Security Token Service (STS). The utility resolves the region in the following order:

  1. Check for the --region flag.
  2. Check for the AWS_REGION environment variable.
  3. Check for the DEFAULT_AWS_REGION environment variable.
  4. Check that a region is specified in the shared configuration file.
clusterawsadm bootstrap iam create-cloudformation-stack [flags]

Examples

  # Create or update IAM roles and policies for Kubernetes using a AWS CloudFormation stack.
  clusterawsadm bootstrap iam create-cloudformation-stack
  
  # Create or update IAM roles and policies for Kubernetes using a AWS CloudFormation stack with a custom configuration.
  clusterawsadm bootstrap iam create-cloudformation-stack --config bootstrap_config.yaml

Options

      --config string   clusterawsadm will load a bootstrap configuration from this file. The path may be
                        absolute or relative; relative paths start at the current working directory.
                        
                        The configuration file is a Kubernetes YAML using the
                        bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1/AWSIAMConfiguration
                        kind.
                        
                        Documentation for this kind can be found at:
                        https://pkg.go.dev/sigs.k8s.io/cluster-api-provider-aws/cmd/clusterawsadm/api/bootstrap/v1alpha1
                        
                        To see the default configuration, run 'clusterawsadm bootstrap iam print-config'.
  -h, --help            help for create-cloudformation-stack
      --region string   The AWS region in which to provision

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm controller

controller commands

Synopsis

All controller related actions such as:

Zero controller credentials and rollout controllers

clusterawsadm controller [command] [flags]

Options

  -h, --help   help for controller

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm controller print-credentials

print credentials the controller is using

Synopsis

print credentials the controller is using

clusterawsadm controller print-credentials [flags]

Examples

  # print credentials
  clusterawsadm controller print-credentials --kubeconfig=kubeconfig --namespace=capa-system

Options

  -h, --help                        help for print-credentials
      --kubeconfig string           Path to the kubeconfig file to use for the management cluster. If empty, default discovery rules apply.
      --kubeconfig-context string   Context to be used within the kubeconfig file. If empty, current context will be used.
      --namespace string            Namespace the controllers are in. If empty, default value (capa-system) is used (default "capa-system")

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm controller rollout-controller

initiates rollout and restart on capa-controller-manager deployment

Synopsis

initiates rollout and restart on capa-controller-manager deployment

clusterawsadm controller rollout-controller [flags]

Examples

  # rollout controller deployment
  clusterawsadm controller rollout-controller --kubeconfig=kubeconfig --namespace=capa-system

Options

  -h, --help                        help for rollout-controller
      --kubeconfig string           Path to the kubeconfig file to use for the management cluster. If empty, default discovery rules apply.
      --kubeconfig-context string   Context to be used within the kubeconfig file. If empty, current context will be used.
      --namespace string            Namespace the controllers are in. If empty, default value (capa-system) is used (default "capa-system")

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm controller update-credentials

update credentials the controller is using (i.e., update controller bootstrap secret)

Synopsis

Update credentials the controller is started with

clusterawsadm controller update-credentials [flags]

Examples

  # update credentials: AWS_B64ENCODED_CREDENTIALS environment variable must be set and be used to update the bootstrap secret
  # Kubeconfig file will be searched in default locations
  clusterawsadm controller update-credentials --namespace=capa-system
  # Provided kubeconfig file will be used
  clusterawsadm controller update-credentials --kubeconfig=kubeconfig  --namespace=capa-system
  # Kubeconfig in the default location will be retrieved and the provided context will be used
  clusterawsadm controller update-credentials --kubeconfig-context=mgmt-cluster  --namespace=capa-system

Options

  -h, --help                        help for update-credentials
      --kubeconfig string           Path to the kubeconfig file to use for the management cluster. If empty, default discovery rules apply.
      --kubeconfig-context string   Context to be used within the kubeconfig file. If empty, current context will be used.
      --namespace string            Namespace the controllers are in. If empty, default value (capa-system) is used (default "capa-system")

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

clusterawsadm controller zero-credentials

zero credentials the controller is started with

Synopsis

Zero credentials the controller is started with

clusterawsadm controller zero-credentials [flags]

Examples

  # zero credentials
  # Kubeconfig file will be searched in default locations
  clusterawsadm controller zero-credentials --namespace=capa-system
  # Provided kubeconfig file will be used
  clusterawsadm controller zero-credentials --kubeconfig=kubeconfig  --namespace=capa-system
  # Kubeconfig in the default location will be retrieved and the provided context will be used
  clusterawsadm controller zero-credentials --kubeconfig-context=mgmt-cluster  --namespace=capa-system

Options

  -h, --help                        help for zero-credentials
      --kubeconfig string           Path to the kubeconfig file to use for the management cluster. If empty, default discovery rules apply.
      --kubeconfig-context string   Context to be used within the kubeconfig file. If empty, current context will be used.
      --namespace string            Namespace the controllers are in. If empty, default value (capa-system) is used (default "capa-system")

Options inherited from parent commands

  -v, --v int   Set the log level verbosity. (default 2)

SEE ALSO

Auto generated by spf13/cobra on 24-Sep-2021

Developer Guide

Initial setup for development environment

Install prerequisites

  1. Install go
    • Get the latest patch version for go v1.16.
  2. Install jq
    • brew install jq on macOS.
    • chocolatey install jq on Windows.
    • sudo apt install jq on Ubuntu Linux.
  3. Install KIND
    • GO111MODULE="on" go get sigs.k8s.io/kind@v0.11.0.
  4. Install Kustomize
    • brew install kustomize on macOS.
    • choco install kustomize on Windows.
    • install instructions on Linux
  5. Install envsubst
  6. Install make.
  7. Install direnv
    • brew install direnv on macOS.

Get the source

Fork the cluster-api-provider-aws repo:

cd "$(go env GOPATH)"
mkdir sigs.k8s.io
cd sigs.k8s.io/
git clone git@github.com:<GITHUB USERNAME>/cluster-api-provider-aws.git
cd cluster-api-provider-aws
git remote add upstream git@github.com:kubernetes-sigs/cluster-api-provider-aws.git
git fetch upstream

Build clusterawsadm

Build clusterawsadm in cluster-api-provider-aws:

cd "$(go env GOPATH)"/src/sigs.k8s.io/cluster-api-provider-aws
make clusterawsadm
mv ./bin/clusterawsadm /usr/local/bin/clusterawsadm

Setup AWS Environment

Create bootstrap file and bootstrap IAM roles and policies using clusterawsadm

$ cat config-bootstrap.yaml

apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  bootstrapUser:
    enable: true

$ clusterawsadm bootstrap iam create-cloudformation-stack
Attempting to create AWS CloudFormation stack cluster-api-provider-aws-sigs-k8s-io

Customizing the bootstrap permission

The IAM permissions can be customized by using a configuration file with clusterawsadm. For example, to create the default IAM role for use with managed machine pools:

$ cat config-bootstrap.yaml
apiVersion: bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1
kind: AWSIAMConfiguration
spec:
  bootstrapUser:
    enable: true
  eks:
    iamRoleCreation: false # Set to true if you plan to use the EKSEnableIAM feature flag to enable automatic creation of IAM roles
    managedMachinePool:
      disable: false # Set to false to enable creation of the default node role for managed machine pools

Use the configuration file to create the additional IAM role:

$ ./bin/clusterawsadm bootstrap iam create-cloudformation-stack --config=config-bootstrap.yaml
Attempting to create AWS CloudFormation stack cluster-api-provider-aws-sigs-k8s-io

If you don’t plan on using EKS then see the documentation on disabling EKS support.

Sample Output

When creating the CloudFormation stack using clusterawsadm you will see output similar to this:

Following resources are in the stack:

Resource                  |Type                                                                                |Status
AWS::IAM::Group           |cluster-api-provider-aws-s-AWSIAMGroupBootstrapper-ME9XZVCO2491                     |CREATE_COMPLETE
AWS::IAM::InstanceProfile |control-plane.cluster-api-provider-aws.sigs.k8s.io                                  |CREATE_COMPLETE
AWS::IAM::InstanceProfile |controllers.cluster-api-provider-aws.sigs.k8s.io                                    |CREATE_COMPLETE
AWS::IAM::InstanceProfile |nodes.cluster-api-provider-aws.sigs.k8s.io                                          |CREATE_COMPLETE
AWS::IAM::ManagedPolicy   |arn:aws:iam::xxx:policy/control-plane.cluster-api-provider-aws.sigs.k8s.io |CREATE_COMPLETE
AWS::IAM::ManagedPolicy   |arn:aws:iam::xxx:policy/nodes.cluster-api-provider-aws.sigs.k8s.io         |CREATE_COMPLETE
AWS::IAM::ManagedPolicy   |arn:aws:iam::xxx:policy/controllers.cluster-api-provider-aws.sigs.k8s.io   |CREATE_COMPLETE
AWS::IAM::Role            |control-plane.cluster-api-provider-aws.sigs.k8s.io                                  |CREATE_COMPLETE
AWS::IAM::Role            |controllers.cluster-api-provider-aws.sigs.k8s.io                                    |CREATE_COMPLETE
AWS::IAM::Role            |eks-controlplane.cluster-api-provider-aws.sigs.k8s.io                               |CREATE_COMPLETE
AWS::IAM::Role            |eks-nodegroup.cluster-api-provider-aws.sigs.k8s.io                                  |CREATE_COMPLETE
AWS::IAM::Role            |nodes.cluster-api-provider-aws.sigs.k8s.io                                          |CREATE_COMPLETE
AWS::IAM::User            |bootstrapper.cluster-api-provider-aws.sigs.k8s.io                                   |CREATE_COMPLETE

Set Environment Variables

  • Create a security credentials in the bootstrapper.cluster-api-provider-aws.sigs.k8s.io IAM user that is created by cloud-formation stack and copy the AWS_ACCESS_KEY_ID and AWS_SECRETS_ACCESS_KEY. (Or use admin user credentials instead)

  • Set AWS_B64ENCODED_CREDENTIALS environment variable

    export AWS_ACCESS_KEY_ID=AKIATEST 
    export AWS_SECRET_ACCESS_KEY=TESTTEST
    export AWS_REGION=eu-west-1
    export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
    

Running local management cluster for development

Before the next steps, make sure initial setup for development environment steps are complete.

There are two ways to build aws manager from local cluster-api-provider-aws source and run it in local kind cluster:

Option 1: Setting up Development Environment with Tilt

Tilt is a tool for quickly building, pushing, and reloading Docker containers as part of a Kubernetes deployment. Many of the Cluster API engineers use it for quick iteration. Please see our Tilt instructions to get started.

Option 2: The Old-fashioned way

Running cluster-api and cluster-api-provider-aws controllers in a kind cluster:

  1. Create a local kind cluster
    • kind create cluster
  2. Install core cluster-api controllers (the version must match the cluster-api version in go.mod)
    • clusterctl init --core cluster-api:v0.3.16 --bootstrap kubeadm:v0.3.16 --control-plane kubeadm:v0.3.16
  3. Build cluster-api-provider-aws docker images
    • make e2e-image
  4. Release manifests under ./out directory
    • RELEASE_TAG="e2e" make release-manifests
  5. Apply the manifests
    • kubectl apply -f ./out/infrastructure.yaml

Developing Cluster API Provider AWS with Tilt

This document describes how to use kind and Tilt for a simplified workflow that offers easy deployments and rapid iterative builds. Before the next steps, make sure initial setup for development environment steps are complete.

Also, visit the Cluster API documentation on Tilt for more information on how to set up your development environment.

Create a kind cluster

First, make sure you have a kind cluster and that your KUBECONFIG is set up correctly:

kind create cluster

This local cluster will be running all the cluster api controllers and become the management cluster which then can be used to spin up workload clusters on AWS.

Get the source

Get the source for core cluster-api for development with Tilt along with cluster-api-provider-aws.

$ cd "$(go env GOPATH)"
$ mkdir sigs.k8s.io
$ cd sigs.k8s.io/
$ git clone git@github.com:kubernetes-sigs/cluster-api.git
$ cd cluster-api
$ git fetch upstream

Create a tilt-settings.json file

Next, create a tilt-settings.json file and place it in your local copy of cluster-api. Here is an example:

Example tilt-settings.json for CAPA managed clusters:


{
  "enable_providers": [
    "kubeadm-bootstrap",
    "kubeadm-control-plane",
    "aws"
  ],
  "allowed_contexts": [
    "kind-kind"
  ],
  "default_registry": "gcr.io/your-project-name-here",
  "provider_repos": [
    "/Users/username/go/src/sigs.k8s.io/cluster-api-provider-aws"
  ],
  "kustomize_substitutions": {
    "EXP_CLUSTER_RESOURCE_SET": "true",
    "EXP_MACHINE_POOL": "true",
    "EVENT_BRIDGE_INSTANCE_STATE": "true",
    "AWS_B64ENCODED_CREDENTIALS": "W2RlZmFZSZnRg=="
  }
}

Example tilt-settings.json for EKS managed clusters:

{
"default_registry": "gcr.io/your-project-name-here",
    "provider_repos": ["../cluster-api-provider-aws"],
    "enable_providers": ["eks-bootstrap", "eks-controlplane", "kubeadm-bootstrap", "kubeadm-control-plane", "aws"],
    "kustomize_substitutions": {
        "AWS_B64ENCODED_CREDENTIALS": "W2RlZmFZSZnRg==",
        "EXP_EKS": "true",
        "EXP_EKS_IAM": "true",
        "EXP_MACHINE_POOL": "true"
    },
    "extra_args": {
        "aws": ["--v=2"],
        "eks-bootstrap": ["--v=2"],
        "eks-controlplane": ["--v=2"]
    }
  }

Run Tilt!

To launch your development environment, run:

tilt up

kind cluster becomes a management cluster after this point, check the pods running on the kind cluster kubectl get pods -A.

Create workload clusters

Set the following variables for both CAPA and EKS managed clusters:

export AWS_SSH_KEY_NAME=<sshkeypair>
export KUBERNETES_VERSION=v1.20.2
export CLUSTER_NAME=capi-<test-clustename>
export CONTROL_PLANE_MACHINE_COUNT=1
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export WORKER_MACHINE_COUNT=1
export AWS_NODE_MACHINE_TYPE=t3.large

Set the following variables for only EKS managed clusters:

export AWS_EKS_ROLE_ARN=arn:aws:iam::<accountid>:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS
export EKS_KUBERNETES_VERSION=v1.15

Create CAPA managed workload cluster:

cat templates/cluster-template.yaml 
cat templates/cluster-template.yaml | $HOME/go/bin/envsubst > test-cluster.yaml
kubectl apply -f test-cluster.yaml

Create EKS workload cluster:

cat templates/cluster-template-eks.yaml 
cat templates/cluster-template-eks.yaml | $HOME/go/bin/envsubst > test-cluster.yaml
kubectl apply -f test-cluster.yaml

Check the tilt logs and wait for the clusters to be created.

Clean up

Before deleting the kind cluster, make sure you delete all the workload clusters.

kubectl delete cluster <clustername>
tilt up (ctrl-c)
kind delete cluster

Troubleshooting

  • Make sure you have at least three available spaces EIP and NAT Gateways to be created
  • If your git starts throwing this error
flag provided but not defined: -variables
Usage: envsubst [options...] <input>

you might need to reinstall the system envsubst

brew install gettetxt
# or
brew reinstall gettext

Make sure you specify which envsubst you are using

Coding Conventions

Below is a collection of conventions, guidlines and general tips for writing code for this project.

API Definitions

Don’t Expose 3rd Party Package Types

When adding new or modifying API types don’t expose 3rd party package types/enums via the CAPA API definitions. Instead create our own versions and where provide mapping functions.

For example:

- AWS SDK [InstaneState](https://docs.aws.amazon.com/sdk-for-go/api/service/ec2/)
- CAPA [InstanceState](https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/api/v1alpha4/types.go#L560:L581)

Don’t use struct pointer slices

When adding new fields to an API type don’t use a slice of struct pointers. This can cause issues with the code generator for the conversion functions. Instead use struct slices.

For example:

Instead of this

	// Configuration options for the non root storage volumes.
	// +optional
	NonRootVolumes []*Volume `json:"nonRootVolumes,omitempty"`

use

	// Configuration options for the non root storage volumes.
	// +optional
	NonRootVolumes []Volume `json:"nonRootVolumes,omitempty"`

And then within the code you can check the length or range over the slice.

Packages:

ami.aws.infrastructure.cluster.x-k8s.io/v1alpha1

Package v1alpha1 contains API Schema definitions for the ami v1alpha1 API group

Resource Types:

    AWSAMI

    AWSAMI defines an AMI.

    Field Description
    metadata
    Kubernetes meta/v1.ObjectMeta
    Refer to the Kubernetes API documentation for the fields of the metadata field.
    spec
    AWSAMISpec


    os
    string
    region
    string
    imageID
    string
    kubernetesVersion
    string

    AWSAMISpec

    (Appears on:AWSAMI)

    AWSAMISpec defines an AMI.

    Field Description
    os
    string
    region
    string
    imageID
    string
    kubernetesVersion
    string

    bootstrap.aws.infrastructure.cluster.x-k8s.io/v1alpha1

    Package v1alpha1 contains API Schema definitions for the bootstrap v1alpha1 API group

    Resource Types:

      AWSIAMConfiguration

      AWSIAMConfiguration controls the creation of AWS Identity and Access Management (IAM) resources for use by Kubernetes clusters and Kubernetes Cluster API Provider AWS.

      Field Description
      spec
      AWSIAMConfigurationSpec


      namePrefix
      string

      NamePrefix will be prepended to every AWS IAM role, user and policy created by clusterawsadm. Defaults to “”.

      nameSuffix
      string

      NameSuffix will be appended to every AWS IAM role, user and policy created by clusterawsadm. Defaults to “.cluster-api-provider-aws.sigs.k8s.io”.

      controlPlane
      ControlPlane

      ControlPlane controls the configuration of the AWS IAM role for a Kubernetes cluster’s control plane nodes.

      clusterAPIControllers
      ClusterAPIControllers

      ClusterAPIControllers controls the configuration of an IAM role and policy specifically for Kubernetes Cluster API Provider AWS.

      nodes
      Nodes

      Nodes controls the configuration of the AWS IAM role for all nodes in a Kubernetes cluster.

      bootstrapUser
      BootstrapUser

      BootstrapUser contains a list of elements that is specific to the configuration and enablement of an IAM user.

      stackName
      string

      StackName defines the name of the AWS CloudFormation stack.

      region
      string

      Region controls which region the control-plane is created in if not specified on the command line or via environment variables.

      eks
      EKSConfig

      EKS controls the configuration related to EKS. Settings in here affect the control plane and nodes roles

      eventBridge
      EventBridgeConfig

      EventBridge controls configuration for consuming EventBridge events

      partition
      string

      Partition is the AWS security partition being used. Defaults to “aws”

      secureSecretBackends
      []Cluster API AWS api/v1alpha4.SecretBackend

      SecureSecretsBackend, when set to parameter-store will create AWS Systems Manager Parameter Storage policies. By default or with the value of secrets-manager, will generate AWS Secrets Manager policies instead.

      AWSIAMConfigurationSpec

      (Appears on:AWSIAMConfiguration)

      AWSIAMConfigurationSpec defines the specification of the AWSIAMConfiguration.

      Field Description
      namePrefix
      string

      NamePrefix will be prepended to every AWS IAM role, user and policy created by clusterawsadm. Defaults to “”.

      nameSuffix
      string

      NameSuffix will be appended to every AWS IAM role, user and policy created by clusterawsadm. Defaults to “.cluster-api-provider-aws.sigs.k8s.io”.

      controlPlane
      ControlPlane

      ControlPlane controls the configuration of the AWS IAM role for a Kubernetes cluster’s control plane nodes.

      clusterAPIControllers
      ClusterAPIControllers

      ClusterAPIControllers controls the configuration of an IAM role and policy specifically for Kubernetes Cluster API Provider AWS.

      nodes
      Nodes

      Nodes controls the configuration of the AWS IAM role for all nodes in a Kubernetes cluster.

      bootstrapUser
      BootstrapUser

      BootstrapUser contains a list of elements that is specific to the configuration and enablement of an IAM user.

      stackName
      string

      StackName defines the name of the AWS CloudFormation stack.

      region
      string

      Region controls which region the control-plane is created in if not specified on the command line or via environment variables.

      eks
      EKSConfig

      EKS controls the configuration related to EKS. Settings in here affect the control plane and nodes roles

      eventBridge
      EventBridgeConfig

      EventBridge controls configuration for consuming EventBridge events

      partition
      string

      Partition is the AWS security partition being used. Defaults to “aws”

      secureSecretBackends
      []Cluster API AWS api/v1alpha4.SecretBackend

      SecureSecretsBackend, when set to parameter-store will create AWS Systems Manager Parameter Storage policies. By default or with the value of secrets-manager, will generate AWS Secrets Manager policies instead.

      AWSIAMRoleSpec

      (Appears on:ClusterAPIControllers, ControlPlane, EKSConfig, Nodes)

      AWSIAMRoleSpec defines common configuration for AWS IAM roles created by Kubernetes Cluster API Provider AWS.

      Field Description
      disable
      bool

      Disable if set to true will not create the AWS IAM role. Defaults to false.

      extraPolicyAttachments
      []string

      ExtraPolicyAttachments is a list of additional policies to be attached to the IAM role.

      extraStatements
      []Cluster API AWS api/v1alpha4.StatementEntry

      ExtraStatements are additional IAM statements to be included inline for the role.

      trustStatements
      []Cluster API AWS api/v1alpha4.StatementEntry

      TrustStatements is an IAM PolicyDocument defining what identities are allowed to assume this role. See “sigs.k8s.io/cluster-api-provider-aws/cmd/clusterawsadm/api/iam/v1alpha1” for more documentation.

      tags
      Cluster API AWS api/v1alpha4.Tags

      Tags is a map of tags to be applied to the AWS IAM role.

      BootstrapUser

      (Appears on:AWSIAMConfigurationSpec)

      BootstrapUser contains a list of elements that is specific to the configuration and enablement of an IAM user.

      Field Description
      enable
      bool

      Enable controls whether or not a bootstrap AWS IAM user will be created. This can be used to scope down the initial credentials used to bootstrap the cluster. Defaults to false.

      userName
      string

      UserName controls the username of the bootstrap user. Defaults to “bootstrapper.cluster-api-provider-aws.sigs.k8s.io”

      groupName
      string

      GroupName controls the group the user will belong to. Defaults to “bootstrapper.cluster-api-provider-aws.sigs.k8s.io”

      extraPolicyAttachments
      []string

      ExtraPolicyAttachments is a list of additional policies to be attached to the IAM user.

      extraGroups
      []string

      ExtraGroups is a list of groups to add this user to.

      extraStatements
      []Cluster API AWS api/v1alpha4.StatementEntry

      ExtraStatements are additional AWS IAM policy document statements to be included inline for the user.

      tags
      Cluster API AWS api/v1alpha4.Tags

      Tags is a map of tags to be applied to the AWS IAM user.

      ClusterAPIControllers

      (Appears on:AWSIAMConfigurationSpec)

      ClusterAPIControllers controls the configuration of the AWS IAM role for the Kubernetes Cluster API Provider AWS controller.

      Field Description
      AWSIAMRoleSpec
      AWSIAMRoleSpec

      (Members of AWSIAMRoleSpec are embedded into this type.)

      allowedEC2InstanceProfiles
      []string

      AllowedEC2InstanceProfiles controls which EC2 roles are allowed to be consumed by Cluster API when creating an ec2 instance. Defaults to *., where suffix is defaulted to .cluster-api-provider-aws.sigs.k8s.io

      ControlPlane

      (Appears on:AWSIAMConfigurationSpec)

      ControlPlane controls the configuration of the AWS IAM role for the control plane of provisioned Kubernetes clusters.

      Field Description
      AWSIAMRoleSpec
      AWSIAMRoleSpec

      (Members of AWSIAMRoleSpec are embedded into this type.)

      disableClusterAPIControllerPolicyAttachment
      bool

      DisableClusterAPIControllerPolicyAttachment, if set to true, will not attach the AWS IAM policy for Cluster API Provider AWS to the control plane role. Defaults to false.

      disableCloudProviderPolicy
      bool

      DisableCloudProviderPolicy if set to true, will not generate and attach the AWS IAM policy for the AWS Cloud Provider.

      enableCSIPolicy
      bool

      EnableCSIPolicy if set to true, will generate and attach the AWS IAM policy for the EBS CSI Driver.

      EKSConfig

      (Appears on:AWSIAMConfigurationSpec)

      EKSConfig represents the EKS related configuration config.

      Field Description
      disable
      bool

      Disable controls whether EKS-related permissions are granted

      iamRoleCreation
      bool

      AllowIAMRoleCreation controls whether the EKS controllers have permissions for creating IAM roles per cluster

      defaultControlPlaneRole
      AWSIAMRoleSpec

      DefaultControlPlaneRole controls the configuration of the AWS IAM role for the EKS control plane. This is the default role that will be used if no role is included in the spec and automatic creation of the role isn’t enabled

      managedMachinePool
      AWSIAMRoleSpec

      ManagedMachinePool controls the configuration of the AWS IAM role for used by EKS managed machine pools.

      fargate
      AWSIAMRoleSpec

      Fargate controls the configuration of the AWS IAM role for used by EKS managed machine pools.

      kmsAliasPrefix
      string

      KMSAliasPrefix is prefix to use to restrict permission to KMS keys to only those that have an alias name that is prefixed by this. Defaults to cluster-api-provider-aws-*

      EventBridgeConfig

      (Appears on:AWSIAMConfigurationSpec)

      EventBridgeConfig represents configuration for enabling experimental feature to consume EventBridge EC2 events.

      Field Description
      enable
      bool

      Enable controls whether permissions are granted to consume EC2 events

      Nodes

      (Appears on:AWSIAMConfigurationSpec)

      Nodes controls the configuration of the AWS IAM role for worker nodes in a cluster created by Kubernetes Cluster API Provider AWS.

      Field Description
      AWSIAMRoleSpec
      AWSIAMRoleSpec

      (Members of AWSIAMRoleSpec are embedded into this type.)

      disableCloudProviderPolicy
      bool

      DisableCloudProviderPolicy if set to true, will not generate and attach the policy for the AWS Cloud Provider. Defaults to false.

      ec2ContainerRegistryReadOnly
      bool

      EC2ContainerRegistryReadOnly controls whether the node has read-only access to the EC2 container registry


      bootstrap.cluster.x-k8s.io/v1alpha4

      Resource Types:

        EKSConfig

        EKSConfig is the Schema for the eksconfigs API

        Field Description
        metadata
        Kubernetes meta/v1.ObjectMeta
        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        EKSConfigSpec


        kubeletExtraArgs
        map[string]string
        (Optional)

        Passes the kubelet args into the EKS bootstrap script

        status
        EKSConfigStatus

        EKSConfigSpec

        (Appears on:EKSConfig, EKSConfigTemplateResource)

        EKSConfigSpec defines the desired state of EKSConfig

        Field Description
        kubeletExtraArgs
        map[string]string
        (Optional)

        Passes the kubelet args into the EKS bootstrap script

        EKSConfigStatus

        (Appears on:EKSConfig)

        EKSConfigStatus defines the observed state of EKSConfig

        Field Description
        ready
        bool

        Ready indicates the BootstrapData secret is ready to be consumed

        dataSecretName
        string
        (Optional)

        DataSecretName is the name of the secret that stores the bootstrap data script.

        failureReason
        string
        (Optional)

        FailureReason will be set on non-retryable errors

        failureMessage
        string
        (Optional)

        FailureMessage will be set on non-retryable errors

        observedGeneration
        int64
        (Optional)

        ObservedGeneration is the latest generation observed by the controller.

        conditions
        Cluster API api/v1alpha4.Conditions
        (Optional)

        Conditions defines current service state of the EKSConfig.

        EKSConfigTemplate

        EKSConfigTemplate is the Schema for the eksconfigtemplates API

        Field Description
        metadata
        Kubernetes meta/v1.ObjectMeta
        Refer to the Kubernetes API documentation for the fields of the metadata field.
        spec
        EKSConfigTemplateSpec


        template
        EKSConfigTemplateResource

        EKSConfigTemplateResource

        (Appears on:EKSConfigTemplateSpec)

        EKSConfigTemplateResource defines the Template structure

        Field Description
        spec
        EKSConfigSpec


        kubeletExtraArgs
        map[string]string
        (Optional)

        Passes the kubelet args into the EKS bootstrap script

        EKSConfigTemplateSpec

        (Appears on:EKSConfigTemplate)

        EKSConfigTemplateSpec defines the desired state of EKSConfigTemplate

        Field Description
        template
        EKSConfigTemplateResource

        controlplane.cluster.x-k8s.io/v1alpha4

        Package v1alpha4 contains API Schema definitions for the controlplane v1alpha4 API group

        Resource Types:

          AWSManagedControlPlane

          AWSManagedControlPlane is the Schema for the awsmanagedcontrolplanes API

          Field Description
          metadata
          Kubernetes meta/v1.ObjectMeta
          Refer to the Kubernetes API documentation for the fields of the metadata field.
          spec
          AWSManagedControlPlaneSpec


          eksClusterName
          string
          (Optional)

          EKSClusterName allows you to specify the name of the EKS cluster in AWS. If you don’t specify a name then a default name will be created based on the namespace and name of the managed control plane.

          identityRef
          Cluster API AWS api/v1alpha4.AWSIdentityReference
          (Optional)

          IdentityRef is a reference to a identity to be used when reconciling the managed control plane.

          network
          Cluster API AWS api/v1alpha4.NetworkSpec

          NetworkSpec encapsulates all things related to AWS network.

          secondaryCidrBlock
          string
          (Optional)

          SecondaryCidrBlock is the additional CIDR range to use for pod IPs. Must be within the 100.64.0.0/10 or 198.19.0.0/16 range.

          region
          string

          The AWS Region the cluster lives in.

          sshKeyName
          string
          (Optional)

          SSHKeyName is the name of the ssh key to attach to the bastion host. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

          version
          string
          (Optional)

          Version defines the desired Kubernetes version. If no version number is supplied then the latest version of Kubernetes that EKS supports will be used.

          roleName
          string
          (Optional)

          RoleName specifies the name of IAM role that gives EKS permission to make API calls. If the role is pre-existing we will treat it as unmanaged and not delete it on deletion. If the EKSEnableIAM feature flag is true and no name is supplied then a role is created.

          roleAdditionalPolicies
          []string
          (Optional)

          RoleAdditionalPolicies allows you to attach additional polices to the control plane role. You must enable the EKSAllowAddRoles feature flag to incorporate these into the created role.

          logging
          ControlPlaneLoggingSpec
          (Optional)

          Logging specifies which EKS Cluster logs should be enabled. Entries for each of the enabled logs will be sent to CloudWatch

          encryptionConfig
          EncryptionConfig
          (Optional)

          EncryptionConfig specifies the encryption configuration for the cluster

          additionalTags
          Cluster API AWS api/v1alpha4.Tags
          (Optional)

          AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

          iamAuthenticatorConfig
          IAMAuthenticatorConfig
          (Optional)

          IAMAuthenticatorConfig allows the specification of any additional user or role mappings for use when generating the aws-iam-authenticator configuration. If this is nil the default configuration is still generated for the cluster.

          endpointAccess
          EndpointAccess
          (Optional)

          Endpoints specifies access to this cluster’s control plane endpoints

          controlPlaneEndpoint
          Cluster API api/v1alpha4.APIEndpoint
          (Optional)

          ControlPlaneEndpoint represents the endpoint used to communicate with the control plane.

          imageLookupFormat
          string
          (Optional)

          ImageLookupFormat is the AMI naming format to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

          imageLookupOrg
          string
          (Optional)

          ImageLookupOrg is the AWS Organization ID to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg.

          imageLookupBaseOS
          string

          ImageLookupBaseOS is the name of the base operating system used to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupBaseOS.

          bastion
          Cluster API AWS api/v1alpha4.Bastion
          (Optional)

          Bastion contains options to configure the bastion host.

          tokenMethod
          EKSTokenMethod

          TokenMethod is used to specify the method for obtaining a client token for communicating with EKS iam-authenticator - obtains a client token using iam-authentictor aws-cli - obtains a client token using the AWS CLI Defaults to iam-authenticator

          associateOIDCProvider
          bool

          AssociateOIDCProvider can be enabled to automatically create an identity provider for the controller for use with IAM roles for service accounts

          addons
          []../../controlplane/eks/api/v1alpha4.Addon
          (Optional)

          Addons defines the EKS addons to enable with the EKS cluster.

          oidcIdentityProviderConfig
          OIDCIdentityProviderConfig
          (Optional)

          IdentityProviderconfig is used to specify the oidc provider config to be attached with this eks cluster

          disableVPCCNI
          bool

          DisableVPCCNI indcates the the Amazon VPC CNI should be disabled. With EKS clusters that the Amazon VPC CNI is automatically installed into the cluster. For clusters where you want to use an alternate CNI this option provides a way to specify that the Amazon VPC CNI should be deleted. You cannot set this to true if you are using the Amazon VPC CNI addon or if you have specified a secondary CIDR block.

          status
          AWSManagedControlPlaneStatus

          AWSManagedControlPlaneSpec

          (Appears on:AWSManagedControlPlane)

          AWSManagedControlPlaneSpec defines the desired state of AWSManagedControlPlane

          Field Description
          eksClusterName
          string
          (Optional)

          EKSClusterName allows you to specify the name of the EKS cluster in AWS. If you don’t specify a name then a default name will be created based on the namespace and name of the managed control plane.

          identityRef
          Cluster API AWS api/v1alpha4.AWSIdentityReference
          (Optional)

          IdentityRef is a reference to a identity to be used when reconciling the managed control plane.

          network
          Cluster API AWS api/v1alpha4.NetworkSpec

          NetworkSpec encapsulates all things related to AWS network.

          secondaryCidrBlock
          string
          (Optional)

          SecondaryCidrBlock is the additional CIDR range to use for pod IPs. Must be within the 100.64.0.0/10 or 198.19.0.0/16 range.

          region
          string

          The AWS Region the cluster lives in.

          sshKeyName
          string
          (Optional)

          SSHKeyName is the name of the ssh key to attach to the bastion host. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

          version
          string
          (Optional)

          Version defines the desired Kubernetes version. If no version number is supplied then the latest version of Kubernetes that EKS supports will be used.

          roleName
          string
          (Optional)

          RoleName specifies the name of IAM role that gives EKS permission to make API calls. If the role is pre-existing we will treat it as unmanaged and not delete it on deletion. If the EKSEnableIAM feature flag is true and no name is supplied then a role is created.

          roleAdditionalPolicies
          []string
          (Optional)

          RoleAdditionalPolicies allows you to attach additional polices to the control plane role. You must enable the EKSAllowAddRoles feature flag to incorporate these into the created role.

          logging
          ControlPlaneLoggingSpec
          (Optional)

          Logging specifies which EKS Cluster logs should be enabled. Entries for each of the enabled logs will be sent to CloudWatch

          encryptionConfig
          EncryptionConfig
          (Optional)

          EncryptionConfig specifies the encryption configuration for the cluster

          additionalTags
          Cluster API AWS api/v1alpha4.Tags
          (Optional)

          AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

          iamAuthenticatorConfig
          IAMAuthenticatorConfig
          (Optional)

          IAMAuthenticatorConfig allows the specification of any additional user or role mappings for use when generating the aws-iam-authenticator configuration. If this is nil the default configuration is still generated for the cluster.

          endpointAccess
          EndpointAccess
          (Optional)

          Endpoints specifies access to this cluster’s control plane endpoints

          controlPlaneEndpoint
          Cluster API api/v1alpha4.APIEndpoint
          (Optional)

          ControlPlaneEndpoint represents the endpoint used to communicate with the control plane.

          imageLookupFormat
          string
          (Optional)

          ImageLookupFormat is the AMI naming format to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

          imageLookupOrg
          string
          (Optional)

          ImageLookupOrg is the AWS Organization ID to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg.

          imageLookupBaseOS
          string

          ImageLookupBaseOS is the name of the base operating system used to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupBaseOS.

          bastion
          Cluster API AWS api/v1alpha4.Bastion
          (Optional)

          Bastion contains options to configure the bastion host.

          tokenMethod
          EKSTokenMethod

          TokenMethod is used to specify the method for obtaining a client token for communicating with EKS iam-authenticator - obtains a client token using iam-authentictor aws-cli - obtains a client token using the AWS CLI Defaults to iam-authenticator

          associateOIDCProvider
          bool

          AssociateOIDCProvider can be enabled to automatically create an identity provider for the controller for use with IAM roles for service accounts

          addons
          []../../controlplane/eks/api/v1alpha4.Addon
          (Optional)

          Addons defines the EKS addons to enable with the EKS cluster.

          oidcIdentityProviderConfig
          OIDCIdentityProviderConfig
          (Optional)

          IdentityProviderconfig is used to specify the oidc provider config to be attached with this eks cluster

          disableVPCCNI
          bool

          DisableVPCCNI indcates the the Amazon VPC CNI should be disabled. With EKS clusters that the Amazon VPC CNI is automatically installed into the cluster. For clusters where you want to use an alternate CNI this option provides a way to specify that the Amazon VPC CNI should be deleted. You cannot set this to true if you are using the Amazon VPC CNI addon or if you have specified a secondary CIDR block.

          AWSManagedControlPlaneStatus

          (Appears on:AWSManagedControlPlane)

          AWSManagedControlPlaneStatus defines the observed state of AWSManagedControlPlane

          Field Description
          networkStatus
          Cluster API AWS api/v1alpha4.NetworkStatus
          (Optional)

          Networks holds details about the AWS networking resources used by the control plane

          failureDomains
          Cluster API api/v1alpha4.FailureDomains
          (Optional)

          FailureDomains specifies a list fo available availability zones that can be used

          bastion
          Cluster API AWS api/v1alpha4.Instance
          (Optional)

          Bastion holds details of the instance that is used as a bastion jump box

          oidcProvider
          OIDCProviderStatus
          (Optional)

          OIDCProvider holds the status of the identity provider for this cluster

          externalManagedControlPlane
          bool

          ExternalManagedControlPlane indicates to cluster-api that the control plane is managed by an external service such as AKS, EKS, GKE, etc.

          initialized
          bool
          (Optional)

          Initialized denotes whether or not the control plane has the uploaded kubernetes config-map.

          ready
          bool

          Ready denotes that the AWSManagedControlPlane API Server is ready to receive requests and that the VPC infra is ready.

          failureMessage
          string
          (Optional)

          ErrorMessage indicates that there is a terminal problem reconciling the state, and will be set to a descriptive error message.

          conditions
          Cluster API api/v1alpha4.Conditions

          Conditions specifies the cpnditions for the managed control plane

          addons
          []AddonState
          (Optional)

          Addons holds the current status of the EKS addons

          identityProviderStatus
          IdentityProviderStatus
          (Optional)

          IdentityProviderStatus holds the status for associated identity provider

          Addon

          Addon represents a EKS addon

          Field Description
          name
          string

          Name is the name of the addon

          version
          string

          Version is the version of the addon to use

          conflictResolution
          AddonResolution

          ConflictResolution is used to declare what should happen if there are parameter conflicts. Defaults to none

          serviceAccountRoleARN
          string
          (Optional)

          ServiceAccountRoleArn is the ARN of an IAM role to bind to the addons service account

          AddonIssue

          (Appears on:AddonState)

          AddonIssue represents an issue with an addon

          Field Description
          code
          string

          Code is the issue code

          message
          string

          Message is the textual description of the issue

          resourceIds
          []string

          ResourceIDs is a list of resource ids for the issue

          AddonResolution (string alias)

          (Appears on:Addon)

          AddonResolution defines the method for resolving parameter conflicts.

          AddonState

          (Appears on:AWSManagedControlPlaneStatus)

          AddonState represents the state of an addon

          Field Description
          name
          string

          Name is the name of the addon

          version
          string

          Version is the version of the addon to use

          arn
          string

          ARN is the AWS ARN of the addon

          serviceAccountRoleARN
          string

          ServiceAccountRoleArn is the ARN of the IAM role used for the service account

          createdAt
          Kubernetes meta/v1.Time

          CreatedAt is the date and time the addon was created at

          modifiedAt
          Kubernetes meta/v1.Time

          ModifiedAt is the date and time the addon was last modified

          status
          string

          Status is the status of the addon

          issues
          []AddonIssue

          Issues is a list of issue associated with the addon

          AddonStatus (string alias)

          AddonStatus defines the status for an addon.

          ControlPlaneLoggingSpec

          (Appears on:AWSManagedControlPlaneSpec)

          ControlPlaneLoggingSpec defines what EKS control plane logs that should be enabled.

          Field Description
          apiServer
          bool

          APIServer indicates if the Kubernetes API Server log (kube-apiserver) shoulkd be enabled

          audit
          bool

          Audit indicates if the Kubernetes API audit log should be enabled

          authenticator
          bool

          Authenticator indicates if the iam authenticator log should be enabled

          controllerManager
          bool

          ControllerManager indicates if the controller manager (kube-controller-manager) log should be enabled

          scheduler
          bool

          Scheduler indicates if the Kubernetes scheduler (kube-scheduler) log should be enabled

          EKSTokenMethod (string alias)

          (Appears on:AWSManagedControlPlaneSpec)

          EKSTokenMethod defines the method for obtaining a client token to use when connecting to EKS.

          EncryptionConfig

          (Appears on:AWSManagedControlPlaneSpec)

          EncryptionConfig specifies the encryption configuration for the EKS clsuter.

          Field Description
          provider
          string

          Provider specifies the ARN or alias of the CMK (in AWS KMS)

          resources
          []*string

          Resources specifies the resources to be encrypted

          EndpointAccess

          (Appears on:AWSManagedControlPlaneSpec)

          EndpointAccess specifies how control plane endpoints are accessible.

          Field Description
          public
          bool
          (Optional)

          Public controls whether control plane endpoints are publicly accessible

          publicCIDRs
          []*string
          (Optional)

          PublicCIDRs specifies which blocks can access the public endpoint

          private
          bool
          (Optional)

          Private points VPC-internal control plane access to the private endpoint

          IAMAuthenticatorConfig

          (Appears on:AWSManagedControlPlaneSpec)

          IAMAuthenticatorConfig represents an aws-iam-authenticator configuration.

          Field Description
          mapRoles
          []RoleMapping
          (Optional)

          RoleMappings is a list of role mappings

          mapUsers
          []UserMapping
          (Optional)

          UserMappings is a list of user mappings

          IdentityProviderStatus

          (Appears on:AWSManagedControlPlaneStatus)

          Field Description
          arn
          string

          ARN holds the ARN of associated identity provider

          status
          string

          Status holds current status of associated identity provider

          KubernetesMapping

          (Appears on:RoleMapping, UserMapping)

          KubernetesMapping represents the kubernetes RBAC mapping.

          Field Description
          username
          string

          UserName is a kubernetes RBAC user subject

          groups
          []string

          Groups is a list of kubernetes RBAC groups

          OIDCIdentityProviderConfig

          (Appears on:AWSManagedControlPlaneSpec)

          Field Description
          clientId
          string

          This is also known as audience. The ID for the client application that makes authentication requests to the OpenID identity provider.

          groupsClaim
          string
          (Optional)

          The JWT claim that the provider uses to return your groups.

          groupsPrefix
          string
          (Optional)

          The prefix that is prepended to group claims to prevent clashes with existing names (such as system: groups). For example, the valueoidc: will create group names like oidc:engineering and oidc:infra.

          identityProviderConfigName
          string

          The name of the OIDC provider configuration.

          IdentityProviderConfigName is a required field

          issuerUrl
          string

          The URL of the OpenID identity provider that allows the API server to discover public signing keys for verifying tokens. The URL must begin with https:// and should correspond to the iss claim in the provider’s OIDC ID tokens. Per the OIDC standard, path components are allowed but query parameters are not. Typically the URL consists of only a hostname, like https://server.example.org or https://example.com. This URL should point to the level below .well-known/openid-configuration and must be publicly accessible over the internet.

          requiredClaims
          map[string]string
          (Optional)

          The key value pairs that describe required claims in the identity token. If set, each claim is verified to be present in the token with a matching value. For the maximum number of claims that you can require, see Amazon EKS service quotas (https://docs.aws.amazon.com/eks/latest/userguide/service-quotas.html) in the Amazon EKS User Guide.

          usernameClaim
          string
          (Optional)

          The JSON Web Token (JWT) claim to use as the username. The default is sub, which is expected to be a unique identifier of the end user. You can choose other claims, such as email or name, depending on the OpenID identity provider. Claims other than email are prefixed with the issuer URL to prevent naming clashes with other plug-ins.

          usernamePrefix
          string
          (Optional)

          The prefix that is prepended to username claims to prevent clashes with existing names. If you do not provide this field, and username is a value other than email, the prefix defaults to issuerurl#. You can use the value - to disable all prefixing.

          tags
          Cluster API AWS api/v1alpha4.Tags
          (Optional)

          tags to apply to oidc identity provider association

          OIDCProviderStatus

          (Appears on:AWSManagedControlPlaneStatus)

          OIDCProviderStatus holds the status of the AWS OIDC identity provider.

          Field Description
          arn
          string

          ARN holds the ARN of the provider

          trustPolicy
          string

          TrustPolicy contains the boilerplate IAM trust policy to use for IRSA

          RoleMapping

          (Appears on:IAMAuthenticatorConfig)

          RoleMapping represents a mapping from a IAM role to Kubernetes users and groups

          Field Description
          rolearn
          string

          RoleARN is the AWS ARN for the role to map

          KubernetesMapping
          KubernetesMapping

          (Members of KubernetesMapping are embedded into this type.)

          KubernetesMapping holds the RBAC details for the mapping

          UserMapping

          (Appears on:IAMAuthenticatorConfig)

          UserMapping represents a mapping from an IAM user to Kubernetes users and groups

          Field Description
          userarn
          string

          UserARN is the AWS ARN for the user to map

          KubernetesMapping
          KubernetesMapping

          (Members of KubernetesMapping are embedded into this type.)

          KubernetesMapping holds the RBAC details for the mapping


          infrastructure.cluster.x-k8s.io/v1alpha4

          Resource Types:

            AMIReference

            (Appears on:AWSMachineSpec)

            AMIReference is a reference to a specific AWS resource by ID, ARN, or filters. Only one of ID, ARN or Filters may be specified. Specifying more than one will result in a validation error.

            Field Description
            id
            string
            (Optional)

            ID of resource

            eksLookupType
            EKSAMILookupType
            (Optional)

            EKSOptimizedLookupType If specified, will look up an EKS Optimized image in SSM Parameter store

            AWSCluster

            AWSCluster is the Schema for the awsclusters API.

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSClusterSpec


            network
            NetworkSpec

            NetworkSpec encapsulates all things related to AWS network.

            region
            string

            The AWS Region the cluster lives in.

            sshKeyName
            string
            (Optional)

            SSHKeyName is the name of the ssh key to attach to the bastion host. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

            controlPlaneEndpoint
            Cluster API api/v1alpha4.APIEndpoint
            (Optional)

            ControlPlaneEndpoint represents the endpoint used to communicate with the control plane.

            additionalTags
            Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

            controlPlaneLoadBalancer
            AWSLoadBalancerSpec
            (Optional)

            ControlPlaneLoadBalancer is optional configuration for customizing control plane behavior.

            imageLookupFormat
            string
            (Optional)

            ImageLookupFormat is the AMI naming format to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

            imageLookupOrg
            string
            (Optional)

            ImageLookupOrg is the AWS Organization ID to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg.

            imageLookupBaseOS
            string

            ImageLookupBaseOS is the name of the base operating system used to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupBaseOS.

            bastion
            Bastion
            (Optional)

            Bastion contains options to configure the bastion host.

            identityRef
            AWSIdentityReference
            (Optional)

            IdentityRef is a reference to a identity to be used when reconciling this cluster

            status
            AWSClusterStatus

            AWSClusterControllerIdentity

            AWSClusterControllerIdentity is the Schema for the awsclustercontrolleridentities API It is used to grant access to use Cluster API Provider AWS Controller credentials.

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSClusterControllerIdentitySpec

            Spec for this AWSClusterControllerIdentity.



            AWSClusterIdentitySpec
            AWSClusterIdentitySpec

            (Members of AWSClusterIdentitySpec are embedded into this type.)

            AWSClusterControllerIdentitySpec

            (Appears on:AWSClusterControllerIdentity)

            AWSClusterControllerIdentitySpec defines the specifications for AWSClusterControllerIdentity.

            Field Description
            AWSClusterIdentitySpec
            AWSClusterIdentitySpec

            (Members of AWSClusterIdentitySpec are embedded into this type.)

            AWSClusterIdentitySpec

            (Appears on:AWSClusterControllerIdentitySpec, AWSClusterRoleIdentitySpec, AWSClusterStaticIdentitySpec)

            AWSClusterIdentitySpec defines the Spec struct for AWSClusterIdentity types.

            Field Description
            allowedNamespaces
            AllowedNamespaces
            (Optional)

            AllowedNamespaces is used to identify which namespaces are allowed to use the identity from. Namespaces can be selected either using an array of namespaces or with label selector. An empty allowedNamespaces object indicates that AWSClusters can use this identity from any namespace. If this object is nil, no namespaces will be allowed (default behaviour, if this field is not provided) A namespace should be either in the NamespaceList or match with Selector to use the identity.

            AWSClusterRoleIdentity

            AWSClusterRoleIdentity is the Schema for the awsclusterroleidentities API It is used to assume a role using the provided sourceRef.

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSClusterRoleIdentitySpec

            Spec for this AWSClusterRoleIdentity.



            AWSClusterIdentitySpec
            AWSClusterIdentitySpec

            (Members of AWSClusterIdentitySpec are embedded into this type.)

            AWSRoleSpec
            AWSRoleSpec

            (Members of AWSRoleSpec are embedded into this type.)

            externalID
            string
            (Optional)

            A unique identifier that might be required when you assume a role in another account. If the administrator of the account to which the role belongs provided you with an external ID, then provide that value in the ExternalId parameter. This value can be any string, such as a passphrase or account number. A cross-account role is usually set up to trust everyone in an account. Therefore, the administrator of the trusting account might send an external ID to the administrator of the trusted account. That way, only someone with the ID can assume the role, rather than everyone in the account. For more information about the external ID, see How to Use an External ID When Granting Access to Your AWS Resources to a Third Party in the IAM User Guide.

            sourceIdentityRef
            AWSIdentityReference

            SourceIdentityRef is a reference to another identity which will be chained to do role assumption. All identity types are accepted.

            AWSClusterRoleIdentitySpec

            (Appears on:AWSClusterRoleIdentity)

            AWSClusterRoleIdentitySpec defines the specifications for AWSClusterRoleIdentity.

            Field Description
            AWSClusterIdentitySpec
            AWSClusterIdentitySpec

            (Members of AWSClusterIdentitySpec are embedded into this type.)

            AWSRoleSpec
            AWSRoleSpec

            (Members of AWSRoleSpec are embedded into this type.)

            externalID
            string
            (Optional)

            A unique identifier that might be required when you assume a role in another account. If the administrator of the account to which the role belongs provided you with an external ID, then provide that value in the ExternalId parameter. This value can be any string, such as a passphrase or account number. A cross-account role is usually set up to trust everyone in an account. Therefore, the administrator of the trusting account might send an external ID to the administrator of the trusted account. That way, only someone with the ID can assume the role, rather than everyone in the account. For more information about the external ID, see How to Use an External ID When Granting Access to Your AWS Resources to a Third Party in the IAM User Guide.

            sourceIdentityRef
            AWSIdentityReference

            SourceIdentityRef is a reference to another identity which will be chained to do role assumption. All identity types are accepted.

            AWSClusterSpec

            (Appears on:AWSCluster, AWSClusterTemplateResource)

            AWSClusterSpec defines the desired state of AWSCluster

            Field Description
            network
            NetworkSpec

            NetworkSpec encapsulates all things related to AWS network.

            region
            string

            The AWS Region the cluster lives in.

            sshKeyName
            string
            (Optional)

            SSHKeyName is the name of the ssh key to attach to the bastion host. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

            controlPlaneEndpoint
            Cluster API api/v1alpha4.APIEndpoint
            (Optional)

            ControlPlaneEndpoint represents the endpoint used to communicate with the control plane.

            additionalTags
            Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

            controlPlaneLoadBalancer
            AWSLoadBalancerSpec
            (Optional)

            ControlPlaneLoadBalancer is optional configuration for customizing control plane behavior.

            imageLookupFormat
            string
            (Optional)

            ImageLookupFormat is the AMI naming format to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

            imageLookupOrg
            string
            (Optional)

            ImageLookupOrg is the AWS Organization ID to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg.

            imageLookupBaseOS
            string

            ImageLookupBaseOS is the name of the base operating system used to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupBaseOS.

            bastion
            Bastion
            (Optional)

            Bastion contains options to configure the bastion host.

            identityRef
            AWSIdentityReference
            (Optional)

            IdentityRef is a reference to a identity to be used when reconciling this cluster

            AWSClusterStaticIdentity

            AWSClusterStaticIdentity is the Schema for the awsclusterstaticidentities API It represents a reference to an AWS access key ID and secret access key, stored in a secret.

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSClusterStaticIdentitySpec

            Spec for this AWSClusterStaticIdentity



            AWSClusterIdentitySpec
            AWSClusterIdentitySpec

            (Members of AWSClusterIdentitySpec are embedded into this type.)

            secretRef
            string

            Reference to a secret containing the credentials. The secret should contain the following data keys: AccessKeyID: AKIAIOSFODNN7EXAMPLE SecretAccessKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY SessionToken: Optional

            AWSClusterStaticIdentitySpec

            (Appears on:AWSClusterStaticIdentity)

            AWSClusterStaticIdentitySpec defines the specifications for AWSClusterStaticIdentity.

            Field Description
            AWSClusterIdentitySpec
            AWSClusterIdentitySpec

            (Members of AWSClusterIdentitySpec are embedded into this type.)

            secretRef
            string

            Reference to a secret containing the credentials. The secret should contain the following data keys: AccessKeyID: AKIAIOSFODNN7EXAMPLE SecretAccessKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY SessionToken: Optional

            AWSClusterStatus

            (Appears on:AWSCluster)

            AWSClusterStatus defines the observed state of AWSCluster

            Field Description
            ready
            bool
            networkStatus
            NetworkStatus
            failureDomains
            Cluster API api/v1alpha4.FailureDomains
            bastion
            Instance
            conditions
            Cluster API api/v1alpha4.Conditions

            AWSClusterTemplate

            AWSClusterTemplate is the Schema for the awsclustertemplates API.

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSClusterTemplateSpec


            template
            AWSClusterTemplateResource

            AWSClusterTemplateResource

            (Appears on:AWSClusterTemplateSpec)

            Field Description
            spec
            AWSClusterSpec


            network
            NetworkSpec

            NetworkSpec encapsulates all things related to AWS network.

            region
            string

            The AWS Region the cluster lives in.

            sshKeyName
            string
            (Optional)

            SSHKeyName is the name of the ssh key to attach to the bastion host. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

            controlPlaneEndpoint
            Cluster API api/v1alpha4.APIEndpoint
            (Optional)

            ControlPlaneEndpoint represents the endpoint used to communicate with the control plane.

            additionalTags
            Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

            controlPlaneLoadBalancer
            AWSLoadBalancerSpec
            (Optional)

            ControlPlaneLoadBalancer is optional configuration for customizing control plane behavior.

            imageLookupFormat
            string
            (Optional)

            ImageLookupFormat is the AMI naming format to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

            imageLookupOrg
            string
            (Optional)

            ImageLookupOrg is the AWS Organization ID to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupOrg.

            imageLookupBaseOS
            string

            ImageLookupBaseOS is the name of the base operating system used to look up machine images when a machine does not specify an AMI. When set, this will be used for all cluster machines unless a machine specifies a different ImageLookupBaseOS.

            bastion
            Bastion
            (Optional)

            Bastion contains options to configure the bastion host.

            identityRef
            AWSIdentityReference
            (Optional)

            IdentityRef is a reference to a identity to be used when reconciling this cluster

            AWSClusterTemplateSpec

            (Appears on:AWSClusterTemplate)

            AWSClusterTemplateSpec defines the desired state of AWSClusterTemplate.

            Field Description
            template
            AWSClusterTemplateResource

            AWSIdentityKind (string alias)

            (Appears on:AWSIdentityReference)

            AWSIdentityKind defines allowed AWS identity types.

            AWSIdentityReference

            (Appears on:AWSClusterRoleIdentitySpec, AWSClusterSpec)

            AWSIdentityReference specifies a identity.

            Field Description
            name
            string

            Name of the identity.

            kind
            AWSIdentityKind

            Kind of the identity.

            AWSLoadBalancerSpec

            (Appears on:AWSClusterSpec)

            AWSLoadBalancerSpec defines the desired state of an AWS load balancer.

            Field Description
            scheme
            ClassicELBScheme
            (Optional)

            Scheme sets the scheme of the load balancer (defaults to internet-facing)

            crossZoneLoadBalancing
            bool
            (Optional)

            CrossZoneLoadBalancing enables the classic ELB cross availability zone balancing.

            With cross-zone load balancing, each load balancer node for your Classic Load Balancer distributes requests evenly across the registered instances in all enabled Availability Zones. If cross-zone load balancing is disabled, each load balancer node distributes requests evenly across the registered instances in its Availability Zone only.

            Defaults to false.

            subnets
            []string
            (Optional)

            Subnets sets the subnets that should be applied to the control plane load balancer (defaults to discovered subnets for managed VPCs or an empty set for unmanaged VPCs)

            additionalSecurityGroups
            []string
            (Optional)

            AdditionalSecurityGroups sets the security groups used by the load balancer. Expected to be security group IDs This is optional - if not provided new security groups will be created for the load balancer

            AWSMachine

            AWSMachine is the Schema for the awsmachines API

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSMachineSpec


            providerID
            string

            ProviderID is the unique identifier as specified by the cloud provider.

            instanceID
            string

            InstanceID is the EC2 instance ID for this machine.

            ami
            AMIReference

            AMI is the reference to the AMI from which to create the machine instance.

            imageLookupFormat
            string
            (Optional)

            ImageLookupFormat is the AMI naming format to look up the image for this machine It will be ignored if an explicit AMI is set. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

            imageLookupOrg
            string

            ImageLookupOrg is the AWS Organization ID to use for image lookup if AMI is not set.

            imageLookupBaseOS
            string

            ImageLookupBaseOS is the name of the base operating system to use for image lookup the AMI is not set.

            instanceType
            string

            InstanceType is the type of instance to create. Example: m4.xlarge

            additionalTags
            Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to an instance, in addition to the ones added by default by the AWS provider. If both the AWSCluster and the AWSMachine specify the same tag name with different values, the AWSMachine’s value takes precedence.

            iamInstanceProfile
            string
            (Optional)

            IAMInstanceProfile is a name of an IAM instance profile to assign to the instance

            publicIP
            bool
            (Optional)

            PublicIP specifies whether the instance should get a public IP. Precedence for this setting is as follows: 1. This field if set 2. Cluster/flavor setting 3. Subnet default

            additionalSecurityGroups
            []AWSResourceReference
            (Optional)

            AdditionalSecurityGroups is an array of references to security groups that should be applied to the instance. These security groups would be set in addition to any security groups defined at the cluster level or in the actuator. It is possible to specify either IDs of Filters. Using Filters will cause additional requests to AWS API and if tags change the attached security groups might change too.

            failureDomain
            string

            FailureDomain is the failure domain unique identifier this Machine should be attached to, as defined in Cluster API. For this infrastructure provider, the ID is equivalent to an AWS Availability Zone. If multiple subnets are matched for the availability zone, the first one returned is picked.

            subnet
            AWSResourceReference
            (Optional)

            Subnet is a reference to the subnet to use for this instance. If not specified, the cluster subnet will be used.

            sshKeyName
            string
            (Optional)

            SSHKeyName is the name of the ssh key to attach to the instance. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

            rootVolume
            Volume
            (Optional)

            RootVolume encapsulates the configuration options for the root volume

            nonRootVolumes
            []Volume
            (Optional)

            Configuration options for the non root storage volumes.

            networkInterfaces
            []string
            (Optional)

            NetworkInterfaces is a list of ENIs to associate with the instance. A maximum of 2 may be specified.

            uncompressedUserData
            bool
            (Optional)

            UncompressedUserData specify whether the user data is gzip-compressed before it is sent to ec2 instance. cloud-init has built-in support for gzip-compressed user data user data stored in aws secret manager is always gzip-compressed.

            cloudInit
            CloudInit
            (Optional)

            CloudInit defines options related to the bootstrapping systems where CloudInit is used.

            spotMarketOptions
            SpotMarketOptions
            (Optional)

            SpotMarketOptions allows users to configure instances to be run using AWS Spot instances.

            tenancy
            string
            (Optional)

            Tenancy indicates if instance should run on shared or single-tenant hardware.

            status
            AWSMachineStatus

            AWSMachineProviderConditionType (string alias)

            AWSMachineProviderConditionType is a valid value for AWSMachineProviderCondition.Type.

            AWSMachineSpec

            (Appears on:AWSMachine, AWSMachineTemplateResource)

            AWSMachineSpec defines the desired state of AWSMachine

            Field Description
            providerID
            string

            ProviderID is the unique identifier as specified by the cloud provider.

            instanceID
            string

            InstanceID is the EC2 instance ID for this machine.

            ami
            AMIReference

            AMI is the reference to the AMI from which to create the machine instance.

            imageLookupFormat
            string
            (Optional)

            ImageLookupFormat is the AMI naming format to look up the image for this machine It will be ignored if an explicit AMI is set. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

            imageLookupOrg
            string

            ImageLookupOrg is the AWS Organization ID to use for image lookup if AMI is not set.

            imageLookupBaseOS
            string

            ImageLookupBaseOS is the name of the base operating system to use for image lookup the AMI is not set.

            instanceType
            string

            InstanceType is the type of instance to create. Example: m4.xlarge

            additionalTags
            Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to an instance, in addition to the ones added by default by the AWS provider. If both the AWSCluster and the AWSMachine specify the same tag name with different values, the AWSMachine’s value takes precedence.

            iamInstanceProfile
            string
            (Optional)

            IAMInstanceProfile is a name of an IAM instance profile to assign to the instance

            publicIP
            bool
            (Optional)

            PublicIP specifies whether the instance should get a public IP. Precedence for this setting is as follows: 1. This field if set 2. Cluster/flavor setting 3. Subnet default

            additionalSecurityGroups
            []AWSResourceReference
            (Optional)

            AdditionalSecurityGroups is an array of references to security groups that should be applied to the instance. These security groups would be set in addition to any security groups defined at the cluster level or in the actuator. It is possible to specify either IDs of Filters. Using Filters will cause additional requests to AWS API and if tags change the attached security groups might change too.

            failureDomain
            string

            FailureDomain is the failure domain unique identifier this Machine should be attached to, as defined in Cluster API. For this infrastructure provider, the ID is equivalent to an AWS Availability Zone. If multiple subnets are matched for the availability zone, the first one returned is picked.

            subnet
            AWSResourceReference
            (Optional)

            Subnet is a reference to the subnet to use for this instance. If not specified, the cluster subnet will be used.

            sshKeyName
            string
            (Optional)

            SSHKeyName is the name of the ssh key to attach to the instance. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

            rootVolume
            Volume
            (Optional)

            RootVolume encapsulates the configuration options for the root volume

            nonRootVolumes
            []Volume
            (Optional)

            Configuration options for the non root storage volumes.

            networkInterfaces
            []string
            (Optional)

            NetworkInterfaces is a list of ENIs to associate with the instance. A maximum of 2 may be specified.

            uncompressedUserData
            bool
            (Optional)

            UncompressedUserData specify whether the user data is gzip-compressed before it is sent to ec2 instance. cloud-init has built-in support for gzip-compressed user data user data stored in aws secret manager is always gzip-compressed.

            cloudInit
            CloudInit
            (Optional)

            CloudInit defines options related to the bootstrapping systems where CloudInit is used.

            spotMarketOptions
            SpotMarketOptions
            (Optional)

            SpotMarketOptions allows users to configure instances to be run using AWS Spot instances.

            tenancy
            string
            (Optional)

            Tenancy indicates if instance should run on shared or single-tenant hardware.

            AWSMachineStatus

            (Appears on:AWSMachine)

            AWSMachineStatus defines the observed state of AWSMachine

            Field Description
            ready
            bool
            (Optional)

            Ready is true when the provider resource is ready.

            interruptible
            bool
            (Optional)

            Interruptible reports that this machine is using spot instances and can therefore be interrupted by CAPI when it receives a notice that the spot instance is to be terminated by AWS. This will be set to true when SpotMarketOptions is not nil (i.e. this machine is using a spot instance).

            addresses
            []Cluster API api/v1alpha4.MachineAddress

            Addresses contains the AWS instance associated addresses.

            instanceState
            InstanceState
            (Optional)

            InstanceState is the state of the AWS instance for this machine.

            failureReason
            Cluster API errors.MachineStatusError
            (Optional)

            FailureReason will be set in the event that there is a terminal problem reconciling the Machine and will contain a succinct value suitable for machine interpretation.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the Machine’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

            failureMessage
            string
            (Optional)

            FailureMessage will be set in the event that there is a terminal problem reconciling the Machine and will contain a more verbose string suitable for logging and human consumption.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the Machine’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

            conditions
            Cluster API api/v1alpha4.Conditions
            (Optional)

            Conditions defines current service state of the AWSMachine.

            AWSMachineTemplate

            AWSMachineTemplate is the Schema for the awsmachinetemplates API

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSMachineTemplateSpec


            template
            AWSMachineTemplateResource

            AWSMachineTemplateResource

            (Appears on:AWSMachineTemplateSpec)

            AWSMachineTemplateResource describes the data needed to create am AWSMachine from a template

            Field Description
            spec
            AWSMachineSpec

            Spec is the specification of the desired behavior of the machine.



            providerID
            string

            ProviderID is the unique identifier as specified by the cloud provider.

            instanceID
            string

            InstanceID is the EC2 instance ID for this machine.

            ami
            AMIReference

            AMI is the reference to the AMI from which to create the machine instance.

            imageLookupFormat
            string
            (Optional)

            ImageLookupFormat is the AMI naming format to look up the image for this machine It will be ignored if an explicit AMI is set. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

            imageLookupOrg
            string

            ImageLookupOrg is the AWS Organization ID to use for image lookup if AMI is not set.

            imageLookupBaseOS
            string

            ImageLookupBaseOS is the name of the base operating system to use for image lookup the AMI is not set.

            instanceType
            string

            InstanceType is the type of instance to create. Example: m4.xlarge

            additionalTags
            Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to an instance, in addition to the ones added by default by the AWS provider. If both the AWSCluster and the AWSMachine specify the same tag name with different values, the AWSMachine’s value takes precedence.

            iamInstanceProfile
            string
            (Optional)

            IAMInstanceProfile is a name of an IAM instance profile to assign to the instance

            publicIP
            bool
            (Optional)

            PublicIP specifies whether the instance should get a public IP. Precedence for this setting is as follows: 1. This field if set 2. Cluster/flavor setting 3. Subnet default

            additionalSecurityGroups
            []AWSResourceReference
            (Optional)

            AdditionalSecurityGroups is an array of references to security groups that should be applied to the instance. These security groups would be set in addition to any security groups defined at the cluster level or in the actuator. It is possible to specify either IDs of Filters. Using Filters will cause additional requests to AWS API and if tags change the attached security groups might change too.

            failureDomain
            string

            FailureDomain is the failure domain unique identifier this Machine should be attached to, as defined in Cluster API. For this infrastructure provider, the ID is equivalent to an AWS Availability Zone. If multiple subnets are matched for the availability zone, the first one returned is picked.

            subnet
            AWSResourceReference
            (Optional)

            Subnet is a reference to the subnet to use for this instance. If not specified, the cluster subnet will be used.

            sshKeyName
            string
            (Optional)

            SSHKeyName is the name of the ssh key to attach to the instance. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

            rootVolume
            Volume
            (Optional)

            RootVolume encapsulates the configuration options for the root volume

            nonRootVolumes
            []Volume
            (Optional)

            Configuration options for the non root storage volumes.

            networkInterfaces
            []string
            (Optional)

            NetworkInterfaces is a list of ENIs to associate with the instance. A maximum of 2 may be specified.

            uncompressedUserData
            bool
            (Optional)

            UncompressedUserData specify whether the user data is gzip-compressed before it is sent to ec2 instance. cloud-init has built-in support for gzip-compressed user data user data stored in aws secret manager is always gzip-compressed.

            cloudInit
            CloudInit
            (Optional)

            CloudInit defines options related to the bootstrapping systems where CloudInit is used.

            spotMarketOptions
            SpotMarketOptions
            (Optional)

            SpotMarketOptions allows users to configure instances to be run using AWS Spot instances.

            tenancy
            string
            (Optional)

            Tenancy indicates if instance should run on shared or single-tenant hardware.

            AWSMachineTemplateSpec

            (Appears on:AWSMachineTemplate)

            AWSMachineTemplateSpec defines the desired state of AWSMachineTemplate

            Field Description
            template
            AWSMachineTemplateResource

            AWSResourceReference

            (Appears on:AWSMachineSpec)

            AWSResourceReference is a reference to a specific AWS resource by ID, ARN, or filters. Only one of ID, ARN or Filters may be specified. Specifying more than one will result in a validation error.

            Field Description
            id
            string
            (Optional)

            ID of resource

            arn
            string
            (Optional)

            ARN of resource

            filters
            []Filter
            (Optional)

            Filters is a set of key/value pairs used to identify a resource They are applied according to the rules defined by the AWS API: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Filtering.html

            AWSRoleSpec

            (Appears on:AWSClusterRoleIdentitySpec)

            AWSRoleSpec defines the specifications for all identities based around AWS roles.

            Field Description
            roleARN
            string

            The Amazon Resource Name (ARN) of the role to assume.

            sessionName
            string

            An identifier for the assumed role session

            durationSeconds
            int32

            The duration, in seconds, of the role session before it is renewed.

            inlinePolicy
            string

            An IAM policy as a JSON-encoded string that you want to use as an inline session policy.

            policyARNs
            []string

            The Amazon Resource Names (ARNs) of the IAM managed policies that you want to use as managed session policies. The policies must exist in the same account as the role.

            AZSelectionScheme (string alias)

            (Appears on:VPCSpec)

            AZSelectionScheme defines the scheme of selecting AZs.

            Actions ([]string alias)

            (Appears on:StatementEntry)

            Actions is the list of actions.

            AllowedNamespaces

            (Appears on:AWSClusterIdentitySpec)

            AllowedNamespaces is a selector of namespaces that AWSClusters can use this ClusterPrincipal from. This is a standard Kubernetes LabelSelector, a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed.

            Field Description
            list
            []string
            (Optional)

            An nil or empty list indicates that AWSClusters cannot use the identity from any namespace.

            selector
            Kubernetes meta/v1.LabelSelector
            (Optional)

            An empty selector indicates that AWSClusters cannot use this AWSClusterIdentity from any namespace.

            Bastion

            (Appears on:AWSClusterSpec)

            Bastion defines a bastion host.

            Field Description
            enabled
            bool
            (Optional)

            Enabled allows this provider to create a bastion host instance with a public ip to access the VPC private network.

            disableIngressRules
            bool
            (Optional)

            DisableIngressRules will ensure there are no Ingress rules in the bastion host’s security group. Requires AllowedCIDRBlocks to be empty.

            allowedCIDRBlocks
            []string
            (Optional)

            AllowedCIDRBlocks is a list of CIDR blocks allowed to access the bastion host. They are set as ingress rules for the Bastion host’s Security Group (defaults to 0.0.0.0/0).

            instanceType
            string

            InstanceType will use the specified instance type for the bastion. If not specified, Cluster API Provider AWS will use t3.micro for all regions except us-east-1, where t2.micro will be the default.

            ami
            string
            (Optional)

            AMI will use the specified AMI to boot the bastion. If not specified, the AMI will default to one picked out in public space.

            BuildParams

            BuildParams is used to build tags around an aws resource.

            Field Description
            Lifecycle
            ResourceLifecycle

            Lifecycle determines the resource lifecycle.

            ClusterName
            string

            ClusterName is the cluster associated with the resource.

            ResourceID
            string

            ResourceID is the unique identifier of the resource to be tagged.

            Name
            string
            (Optional)

            Name is the name of the resource, it’s applied as the tag “Name” on AWS.

            Role
            string
            (Optional)

            Role is the role associated to the resource.

            Additional
            Tags
            (Optional)

            Any additional tags to be added to the resource.

            CNIIngressRule

            CNIIngressRule defines an AWS ingress rule for CNI requirements.

            Field Description
            description
            string
            protocol
            SecurityGroupProtocol
            fromPort
            int64
            toPort
            int64

            CNIIngressRules ([]../../api/v1alpha4.CNIIngressRule alias)

            (Appears on:CNISpec)

            CNIIngressRules is a slice of CNIIngressRule

            CNISpec

            (Appears on:NetworkSpec)

            CNISpec defines configuration for CNI.

            Field Description
            cniIngressRules
            CNIIngressRules

            CNIIngressRules specify rules to apply to control plane and worker node security groups. The source for the rule will be set to control plane and worker security group IDs.

            ClassicELB

            (Appears on:NetworkStatus)

            ClassicELB defines an AWS classic load balancer.

            Field Description
            name
            string

            The name of the load balancer. It must be unique within the set of load balancers defined in the region. It also serves as identifier.

            dnsName
            string

            DNSName is the dns name of the load balancer.

            scheme
            ClassicELBScheme

            Scheme is the load balancer scheme, either internet-facing or private.

            availabilityZones
            []string

            AvailabilityZones is an array of availability zones in the VPC attached to the load balancer.

            subnetIds
            []string

            SubnetIDs is an array of subnets in the VPC attached to the load balancer.

            securityGroupIds
            []string

            SecurityGroupIDs is an array of security groups assigned to the load balancer.

            listeners
            []ClassicELBListener

            Listeners is an array of classic elb listeners associated with the load balancer. There must be at least one.

            healthChecks
            ClassicELBHealthCheck

            HealthCheck is the classic elb health check associated with the load balancer.

            attributes
            ClassicELBAttributes

            Attributes defines extra attributes associated with the load balancer.

            tags
            map[string]string

            Tags is a map of tags associated with the load balancer.

            ClassicELBAttributes

            (Appears on:ClassicELB)

            ClassicELBAttributes defines extra attributes associated with a classic load balancer.

            Field Description
            idleTimeout
            time.Duration

            IdleTimeout is time that the connection is allowed to be idle (no data has been sent over the connection) before it is closed by the load balancer.

            crossZoneLoadBalancing
            bool
            (Optional)

            CrossZoneLoadBalancing enables the classic load balancer load balancing.

            ClassicELBHealthCheck

            (Appears on:ClassicELB)

            ClassicELBHealthCheck defines an AWS classic load balancer health check.

            Field Description
            target
            string
            interval
            time.Duration
            timeout
            time.Duration
            healthyThreshold
            int64
            unhealthyThreshold
            int64

            ClassicELBListener

            (Appears on:ClassicELB)

            ClassicELBListener defines an AWS classic load balancer listener.

            Field Description
            protocol
            ClassicELBProtocol
            port
            int64
            instanceProtocol
            ClassicELBProtocol
            instancePort
            int64

            ClassicELBProtocol (string alias)

            (Appears on:ClassicELBListener)

            ClassicELBProtocol defines listener protocols for a classic load balancer.

            ClassicELBScheme (string alias)

            (Appears on:AWSLoadBalancerSpec, ClassicELB)

            ClassicELBScheme defines the scheme of a classic load balancer.

            CloudInit

            (Appears on:AWSMachineSpec)

            CloudInit defines options related to the bootstrapping systems where CloudInit is used.

            Field Description
            insecureSkipSecretsManager
            bool

            InsecureSkipSecretsManager, when set to true will not use AWS Secrets Manager or AWS Systems Manager Parameter Store to ensure privacy of userdata. By default, a cloud-init boothook shell script is prepended to download the userdata from Secrets Manager and additionally delete the secret.

            secretCount
            int32
            (Optional)

            SecretCount is the number of secrets used to form the complete secret

            secretPrefix
            string
            (Optional)

            SecretPrefix is the prefix for the secret name. This is stored temporarily, and deleted when the machine registers as a node against the workload cluster.

            secureSecretsBackend
            SecretBackend
            (Optional)

            SecureSecretsBackend, when set to parameter-store will utilize the AWS Systems Manager Parameter Storage to distribute secrets. By default or with the value of secrets-manager, will use AWS Secrets Manager instead.

            ConditionOperator (string alias)

            ConditionOperator defines an AWS condition operator.

            Conditions (map[../../api/v1alpha4.ConditionOperator]interface{} alias)

            (Appears on:StatementEntry)

            Conditions is the map of all conditions in the statement entry.

            EKSAMILookupType (string alias)

            (Appears on:AMIReference)

            EKSAMILookupType specifies which AWS AMI to use for a AWSMachine and AWSMachinePool.

            Effect (string alias)

            (Appears on:StatementEntry)

            Effect defines an AWS IAM effect.

            Filter

            (Appears on:AWSResourceReference)

            Filter is a filter used to identify an AWS resource

            Field Description
            name
            string

            Name of the filter. Filter names are case-sensitive.

            values
            []string

            Values includes one or more filter values. Filter values are case-sensitive.

            IngressRule

            IngressRule defines an AWS ingress rule for security groups.

            Field Description
            description
            string
            protocol
            SecurityGroupProtocol
            fromPort
            int64
            toPort
            int64
            cidrBlocks
            []string
            (Optional)

            List of CIDR blocks to allow access from. Cannot be specified with SourceSecurityGroupID.

            sourceSecurityGroupIds
            []string
            (Optional)

            The security group id to allow access from. Cannot be specified with CidrBlocks.

            IngressRules ([]../../api/v1alpha4.IngressRule alias)

            (Appears on:SecurityGroup)

            IngressRules is a slice of AWS ingress rules for security groups.

            Instance

            (Appears on:AWSClusterStatus)

            Instance describes an AWS instance.

            Field Description
            id
            string
            instanceState
            InstanceState

            The current state of the instance.

            type
            string

            The instance type.

            subnetId
            string

            The ID of the subnet of the instance.

            imageId
            string

            The ID of the AMI used to launch the instance.

            sshKeyName
            string

            The name of the SSH key pair.

            securityGroupIds
            []string

            SecurityGroupIDs are one or more security group IDs this instance belongs to.

            userData
            string

            UserData is the raw data script passed to the instance which is run upon bootstrap. This field must not be base64 encoded and should only be used when running a new instance.

            iamProfile
            string

            The name of the IAM instance profile associated with the instance, if applicable.

            addresses
            []Cluster API api/v1alpha4.MachineAddress

            Addresses contains the AWS instance associated addresses.

            privateIp
            string

            The private IPv4 address assigned to the instance.

            publicIp
            string

            The public IPv4 address assigned to the instance, if applicable.

            enaSupport
            bool

            Specifies whether enhanced networking with ENA is enabled.

            ebsOptimized
            bool

            Indicates whether the instance is optimized for Amazon EBS I/O.

            rootVolume
            Volume
            (Optional)

            Configuration options for the root storage volume.

            nonRootVolumes
            []Volume
            (Optional)

            Configuration options for the non root storage volumes.

            networkInterfaces
            []string

            Specifies ENIs attached to instance

            tags
            map[string]string

            The tags associated with the instance.

            availabilityZone
            string

            Availability zone of instance

            spotMarketOptions
            SpotMarketOptions

            SpotMarketOptions option for configuring instances to be run using AWS Spot instances.

            tenancy
            string
            (Optional)

            Tenancy indicates if instance should run on shared or single-tenant hardware.

            volumeIDs
            []string
            (Optional)

            IDs of the instance’s volumes

            InstanceState (string alias)

            (Appears on:AWSMachineStatus, Instance)

            InstanceState describes the state of an AWS instance.

            NetworkSpec

            (Appears on:AWSClusterSpec)

            NetworkSpec encapsulates all things related to AWS network.

            Field Description
            vpc
            VPCSpec
            (Optional)

            VPC configuration.

            subnets
            Subnets
            (Optional)

            Subnets configuration.

            cni
            CNISpec
            (Optional)

            CNI configuration

            securityGroupOverrides
            map[../../api/v1alpha4.SecurityGroupRole]string
            (Optional)

            SecurityGroupOverrides is an optional set of security groups to use for cluster instances This is optional - if not provided new security groups will be created for the cluster

            NetworkStatus

            (Appears on:AWSClusterStatus)

            NetworkStatus encapsulates AWS networking resources.

            Field Description
            securityGroups
            map[../../api/v1alpha4.SecurityGroupRole]../../api/v1alpha4.SecurityGroup

            SecurityGroups is a map from the role/kind of the security group to its unique name, if any.

            apiServerElb
            ClassicELB

            APIServerELB is the Kubernetes api server classic load balancer.

            PolicyDocument

            PolicyDocument represents an AWS IAM policy document, and can be converted into JSON using “sigs.k8s.io/cluster-api-provider-aws/cmd/clusterawsadm/converters”.

            Field Description
            Version
            string
            Statement
            Statements
            Id
            string

            PrincipalID ([]string alias)

            PrincipalID represents the list of all identities, such as ARNs.

            PrincipalType (string alias)

            PrincipalType defines an AWS principle type.

            Principals (map[../../api/v1alpha4.PrincipalType]../../api/v1alpha4.PrincipalID alias)

            (Appears on:StatementEntry)

            Principals is the map of all identities a statement entry refers to.

            ResourceLifecycle (string alias)

            (Appears on:BuildParams)

            ResourceLifecycle configures the lifecycle of a resource.

            Resources ([]string alias)

            (Appears on:StatementEntry)

            Resources is the list of resources.

            RouteTable

            RouteTable defines an AWS routing table.

            Field Description
            id
            string

            SecretBackend (string alias)

            (Appears on:CloudInit)

            SecretBackend defines variants for backend secret storage.

            SecurityGroup

            (Appears on:NetworkStatus)

            SecurityGroup defines an AWS security group.

            Field Description
            id
            string

            ID is a unique identifier.

            name
            string

            Name is the security group name.

            ingressRule
            IngressRules
            (Optional)

            IngressRules is the inbound rules associated with the security group.

            tags
            Tags

            Tags is a map of tags associated with the security group.

            SecurityGroupProtocol (string alias)

            (Appears on:CNIIngressRule, IngressRule)

            SecurityGroupProtocol defines the protocol type for a security group rule.

            SecurityGroupRole (string alias)

            SecurityGroupRole defines the unique role of a security group.

            SpotMarketOptions

            (Appears on:AWSMachineSpec, Instance)

            SpotMarketOptions defines the options available to a user when configuring Machines to run on Spot instances. Most users should provide an empty struct.

            Field Description
            maxPrice
            string
            (Optional)

            MaxPrice defines the maximum price the user is willing to pay for Spot VM instances

            StatementEntry

            StatementEntry represents each “statement” block in an AWS IAM policy document.

            Field Description
            Sid
            string
            Principal
            Principals
            NotPrincipal
            Principals
            Effect
            Effect
            Action
            Actions
            Resource
            Resources
            Condition
            Conditions

            Statements ([]../../api/v1alpha4.StatementEntry alias)

            (Appears on:PolicyDocument)

            Statements is the list of StatementEntries.

            SubnetSpec

            SubnetSpec configures an AWS Subnet.

            Field Description
            id
            string

            ID defines a unique identifier to reference this resource.

            cidrBlock
            string

            CidrBlock is the CIDR block to be used when the provider creates a managed VPC.

            availabilityZone
            string

            AvailabilityZone defines the availability zone to use for this subnet in the cluster’s region.

            isPublic
            bool
            (Optional)

            IsPublic defines the subnet as a public subnet. A subnet is public when it is associated with a route table that has a route to an internet gateway.

            routeTableId
            string
            (Optional)

            RouteTableID is the routing table id associated with the subnet.

            natGatewayId
            string
            (Optional)

            NatGatewayID is the NAT gateway id associated with the subnet. Ignored unless the subnet is managed by the provider, in which case this is set on the public subnet where the NAT gateway resides. It is then used to determine routes for private subnets in the same AZ as the public subnet.

            tags
            Tags

            Tags is a collection of tags describing the resource.

            Subnets ([]../../api/v1alpha4.SubnetSpec alias)

            (Appears on:NetworkSpec)

            Subnets is a slice of Subnet.

            Tags (map[string]string alias)

            (Appears on:AWSClusterSpec, AWSMachineSpec, BuildParams, SecurityGroup, SubnetSpec, VPCSpec)

            Tags defines a map of tags.

            VPCSpec

            (Appears on:NetworkSpec)

            VPCSpec configures an AWS VPC.

            Field Description
            id
            string

            ID is the vpc-id of the VPC this provider should use to create resources.

            cidrBlock
            string

            CidrBlock is the CIDR block to be used when the provider creates a managed VPC. Defaults to 10.0.0.0/16.

            internetGatewayId
            string
            (Optional)

            InternetGatewayID is the id of the internet gateway associated with the VPC.

            tags
            Tags

            Tags is a collection of tags describing the resource.

            availabilityZoneUsageLimit
            int

            AvailabilityZoneUsageLimit specifies the maximum number of availability zones (AZ) that should be used in a region when automatically creating subnets. If a region has more than this number of AZs then this number of AZs will be picked randomly when creating default subnets. Defaults to 3

            availabilityZoneSelection
            AZSelectionScheme

            AvailabilityZoneSelection specifies how AZs should be selected if there are more AZs in a region than specified by AvailabilityZoneUsageLimit. There are 2 selection schemes: Ordered - selects based on alphabetical order Random - selects AZs randomly in a region Defaults to Ordered

            Volume

            (Appears on:AWSMachineSpec, Instance)

            Volume encapsulates the configuration options for the storage device

            Field Description
            deviceName
            string
            (Optional)

            Device name

            size
            int64

            Size specifies size (in Gi) of the storage device. Must be greater than the image snapshot size or 8 (whichever is greater).

            type
            VolumeType
            (Optional)

            Type is the type of the volume (e.g. gp2, io1, etc…).

            iops
            int64
            (Optional)

            IOPS is the number of IOPS requested for the disk. Not applicable to all types.

            throughput
            int64
            (Optional)

            Throughput to provision in MiB/s supported for the volume type. Not applicable to all types.

            encrypted
            bool
            (Optional)

            Encrypted is whether the volume should be encrypted or not.

            encryptionKey
            string
            (Optional)

            EncryptionKey is the KMS key to use to encrypt the volume. Can be either a KMS key ID or ARN. If Encrypted is set and this is omitted, the default AWS key will be used. The key must already exist and be accessible by the controller.

            VolumeType (string alias)

            (Appears on:Volume)

            VolumeType describes the EBS volume type. See: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html

            ASGStatus (string alias)

            (Appears on:AWSMachinePoolStatus, AutoScalingGroup)

            ASGStatus is a status string returned by the autoscaling API

            AWSFargateProfile

            AWSFargateProfile is the Schema for the awsfargateprofiles API

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            FargateProfileSpec


            clusterName
            string

            ClusterName is the name of the Cluster this object belongs to.

            profileName
            string

            ProfileName specifies the profile name.

            subnetIDs
            []string
            (Optional)

            SubnetIDs specifies which subnets are used for the auto scaling group of this nodegroup.

            additionalTags
            Cluster API AWS api/v1alpha4.Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

            roleName
            string
            (Optional)

            RoleName specifies the name of IAM role for this fargate pool If the role is pre-existing we will treat it as unmanaged and not delete it on deletion. If the EKSEnableIAM feature flag is true and no name is supplied then a role is created.

            selectors
            []FargateSelector

            Selectors specify fargate pod selectors.

            status
            FargateProfileStatus

            AWSLaunchTemplate

            (Appears on:AWSMachinePoolSpec)

            AWSLaunchTemplate defines the desired state of AWSLaunchTemplate

            Field Description
            name
            string

            The name of the launch template.

            iamInstanceProfile
            string

            The name or the Amazon Resource Name (ARN) of the instance profile associated with the IAM role for the instance. The instance profile contains the IAM role.

            ami
            Cluster API AWS api/v1alpha4.AMIReference
            (Optional)

            AMI is the reference to the AMI from which to create the machine instance.

            imageLookupFormat
            string
            (Optional)

            ImageLookupFormat is the AMI naming format to look up the image for this machine It will be ignored if an explicit AMI is set. Supports substitutions for {{.BaseOS}} and {{.K8sVersion}} with the base OS and kubernetes version, respectively. The BaseOS will be the value in ImageLookupBaseOS or ubuntu (the default), and the kubernetes version as defined by the packages produced by kubernetes/release without v as a prefix: 1.13.0, 1.12.5-mybuild.1, or 1.17.3. For example, the default image format of capa-ami-{{.BaseOS}}-?{{.K8sVersion}}-* will end up searching for AMIs that match the pattern capa-ami-ubuntu-?1.18.0-* for a Machine that is targeting kubernetes v1.18.0 and the ubuntu base OS. See also: https://golang.org/pkg/text/template/

            imageLookupOrg
            string

            ImageLookupOrg is the AWS Organization ID to use for image lookup if AMI is not set.

            imageLookupBaseOS
            string

            ImageLookupBaseOS is the name of the base operating system to use for image lookup the AMI is not set.

            instanceType
            string

            InstanceType is the type of instance to create. Example: m4.xlarge

            rootVolume
            Cluster API AWS api/v1alpha4.Volume
            (Optional)

            RootVolume encapsulates the configuration options for the root volume

            sshKeyName
            string
            (Optional)

            SSHKeyName is the name of the ssh key to attach to the instance. Valid values are empty string (do not use SSH keys), a valid SSH key name, or omitted (use the default SSH key name)

            versionNumber
            int64

            VersionNumber is the version of the launch template that is applied. Typically a new version is created when at least one of the following happens: 1) A new launch template spec is applied. 2) One or more parameters in an existing template is changed. 3) A new AMI is discovered.

            additionalSecurityGroups
            []Cluster API AWS api/v1alpha4.AWSResourceReference
            (Optional)

            AdditionalSecurityGroups is an array of references to security groups that should be applied to the instances. These security groups would be set in addition to any security groups defined at the cluster level or in the actuator.

            AWSMachinePool

            AWSMachinePool is the Schema for the awsmachinepools API

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSMachinePoolSpec


            providerID
            string
            (Optional)

            ProviderID is the ARN of the associated ASG

            minSize
            int32

            MinSize defines the minimum size of the group.

            maxSize
            int32

            MaxSize defines the maximum size of the group.

            availabilityZones
            []string

            AvailabilityZones is an array of availability zones instances can run in

            subnets
            []Cluster API AWS api/v1alpha4.AWSResourceReference
            (Optional)

            Subnets is an array of subnet configurations

            additionalTags
            Cluster API AWS api/v1alpha4.Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to an instance, in addition to the ones added by default by the AWS provider.

            awsLaunchTemplate
            AWSLaunchTemplate

            AWSLaunchTemplate specifies the launch template and version to use when an instance is launched.

            mixedInstancesPolicy
            MixedInstancesPolicy

            MixedInstancesPolicy describes how multiple instance types will be used by the ASG.

            providerIDList
            []string
            (Optional)

            ProviderIDList are the identification IDs of machine instances provided by the provider. This field must match the provider IDs as seen on the node objects corresponding to a machine pool’s machine instances.

            defaultCoolDown
            Kubernetes meta/v1.Duration
            (Optional)

            The amount of time, in seconds, after a scaling activity completes before another scaling activity can start. If no value is supplied by user a default value of 300 seconds is set

            refreshPreferences
            RefreshPreferences
            (Optional)

            RefreshPreferences describes set of preferences associated with the instance refresh request.

            capacityRebalance
            bool
            (Optional)

            Enable or disable the capacity rebalance autoscaling group feature

            status
            AWSMachinePoolStatus

            AWSMachinePoolInstanceStatus

            (Appears on:AWSMachinePoolStatus)

            AWSMachinePoolInstanceStatus defines the status of the AWSMachinePoolInstance.

            Field Description
            instanceID
            string
            (Optional)

            InstanceID is the identification of the Machine Instance within ASG

            version
            string
            (Optional)

            Version defines the Kubernetes version for the Machine Instance

            AWSMachinePoolSpec

            (Appears on:AWSMachinePool)

            AWSMachinePoolSpec defines the desired state of AWSMachinePool

            Field Description
            providerID
            string
            (Optional)

            ProviderID is the ARN of the associated ASG

            minSize
            int32

            MinSize defines the minimum size of the group.

            maxSize
            int32

            MaxSize defines the maximum size of the group.

            availabilityZones
            []string

            AvailabilityZones is an array of availability zones instances can run in

            subnets
            []Cluster API AWS api/v1alpha4.AWSResourceReference
            (Optional)

            Subnets is an array of subnet configurations

            additionalTags
            Cluster API AWS api/v1alpha4.Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to an instance, in addition to the ones added by default by the AWS provider.

            awsLaunchTemplate
            AWSLaunchTemplate

            AWSLaunchTemplate specifies the launch template and version to use when an instance is launched.

            mixedInstancesPolicy
            MixedInstancesPolicy

            MixedInstancesPolicy describes how multiple instance types will be used by the ASG.

            providerIDList
            []string
            (Optional)

            ProviderIDList are the identification IDs of machine instances provided by the provider. This field must match the provider IDs as seen on the node objects corresponding to a machine pool’s machine instances.

            defaultCoolDown
            Kubernetes meta/v1.Duration
            (Optional)

            The amount of time, in seconds, after a scaling activity completes before another scaling activity can start. If no value is supplied by user a default value of 300 seconds is set

            refreshPreferences
            RefreshPreferences
            (Optional)

            RefreshPreferences describes set of preferences associated with the instance refresh request.

            capacityRebalance
            bool
            (Optional)

            Enable or disable the capacity rebalance autoscaling group feature

            AWSMachinePoolStatus

            (Appears on:AWSMachinePool)

            AWSMachinePoolStatus defines the observed state of AWSMachinePool

            Field Description
            ready
            bool
            (Optional)

            Ready is true when the provider resource is ready.

            replicas
            int32
            (Optional)

            Replicas is the most recently observed number of replicas

            conditions
            Cluster API api/v1alpha4.Conditions
            (Optional)

            Conditions defines current service state of the AWSMachinePool.

            instances
            []AWSMachinePoolInstanceStatus
            (Optional)

            Instances contains the status for each instance in the pool

            launchTemplateID
            string

            The ID of the launch template

            failureReason
            Cluster API errors.MachineStatusError
            (Optional)

            FailureReason will be set in the event that there is a terminal problem reconciling the Machine and will contain a succinct value suitable for machine interpretation.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the Machine’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

            failureMessage
            string
            (Optional)

            FailureMessage will be set in the event that there is a terminal problem reconciling the Machine and will contain a more verbose string suitable for logging and human consumption.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the Machine’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of Machines can be added as events to the Machine object and/or logged in the controller’s output.

            asgStatus
            ASGStatus

            AWSManagedMachinePool

            AWSManagedMachinePool is the Schema for the awsmanagedmachinepools API

            Field Description
            metadata
            Kubernetes meta/v1.ObjectMeta
            Refer to the Kubernetes API documentation for the fields of the metadata field.
            spec
            AWSManagedMachinePoolSpec


            eksNodegroupName
            string
            (Optional)

            EKSNodegroupName specifies the name of the nodegroup in AWS corresponding to this MachinePool. If you don’t specify a name then a default name will be created based on the namespace and name of the managed machine pool.

            availabilityZones
            []string

            AvailabilityZones is an array of availability zones instances can run in

            subnetIDs
            []string
            (Optional)

            SubnetIDs specifies which subnets are used for the auto scaling group of this nodegroup

            additionalTags
            Cluster API AWS api/v1alpha4.Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

            roleName
            string
            (Optional)

            RoleName specifies the name of IAM role for the node group. If the role is pre-existing we will treat it as unmanaged and not delete it on deletion. If the EKSEnableIAM feature flag is true and no name is supplied then a role is created.

            amiVersion
            string
            (Optional)

            AMIVersion defines the desired AMI release version. If no version number is supplied then the latest version for the Kubernetes version will be used

            amiType
            ManagedMachineAMIType
            (Optional)

            AMIType defines the AMI type

            labels
            map[string]string
            (Optional)

            Labels specifies labels for the Kubernetes node objects

            taints
            Taints
            (Optional)

            Taints specifies the taints to apply to the nodes of the machine pool

            diskSize
            int32
            (Optional)

            DiskSize specifies the root disk size

            instanceType
            string
            (Optional)

            InstanceType specifies the AWS instance type

            scaling
            ManagedMachinePoolScaling
            (Optional)

            Scaling specifies scaling for the ASG behind this pool

            remoteAccess
            ManagedRemoteAccess
            (Optional)

            RemoteAccess specifies how machines can be accessed remotely

            providerIDList
            []string
            (Optional)

            ProviderIDList are the provider IDs of instances in the autoscaling group corresponding to the nodegroup represented by this machine pool

            capacityType
            ManagedMachinePoolCapacityType
            (Optional)

            CapacityType specifies the capacity type for the ASG behind this pool

            status
            AWSManagedMachinePoolStatus

            AWSManagedMachinePoolSpec

            (Appears on:AWSManagedMachinePool)

            AWSManagedMachinePoolSpec defines the desired state of AWSManagedMachinePool

            Field Description
            eksNodegroupName
            string
            (Optional)

            EKSNodegroupName specifies the name of the nodegroup in AWS corresponding to this MachinePool. If you don’t specify a name then a default name will be created based on the namespace and name of the managed machine pool.

            availabilityZones
            []string

            AvailabilityZones is an array of availability zones instances can run in

            subnetIDs
            []string
            (Optional)

            SubnetIDs specifies which subnets are used for the auto scaling group of this nodegroup

            additionalTags
            Cluster API AWS api/v1alpha4.Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

            roleName
            string
            (Optional)

            RoleName specifies the name of IAM role for the node group. If the role is pre-existing we will treat it as unmanaged and not delete it on deletion. If the EKSEnableIAM feature flag is true and no name is supplied then a role is created.

            amiVersion
            string
            (Optional)

            AMIVersion defines the desired AMI release version. If no version number is supplied then the latest version for the Kubernetes version will be used

            amiType
            ManagedMachineAMIType
            (Optional)

            AMIType defines the AMI type

            labels
            map[string]string
            (Optional)

            Labels specifies labels for the Kubernetes node objects

            taints
            Taints
            (Optional)

            Taints specifies the taints to apply to the nodes of the machine pool

            diskSize
            int32
            (Optional)

            DiskSize specifies the root disk size

            instanceType
            string
            (Optional)

            InstanceType specifies the AWS instance type

            scaling
            ManagedMachinePoolScaling
            (Optional)

            Scaling specifies scaling for the ASG behind this pool

            remoteAccess
            ManagedRemoteAccess
            (Optional)

            RemoteAccess specifies how machines can be accessed remotely

            providerIDList
            []string
            (Optional)

            ProviderIDList are the provider IDs of instances in the autoscaling group corresponding to the nodegroup represented by this machine pool

            capacityType
            ManagedMachinePoolCapacityType
            (Optional)

            CapacityType specifies the capacity type for the ASG behind this pool

            AWSManagedMachinePoolStatus

            (Appears on:AWSManagedMachinePool)

            AWSManagedMachinePoolStatus defines the observed state of AWSManagedMachinePool

            Field Description
            ready
            bool

            Ready denotes that the AWSManagedMachinePool nodegroup has joined the cluster

            replicas
            int32
            (Optional)

            Replicas is the most recently observed number of replicas.

            failureReason
            Cluster API errors.MachineStatusError
            (Optional)

            FailureReason will be set in the event that there is a terminal problem reconciling the MachinePool and will contain a succinct value suitable for machine interpretation.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the Machine’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of MachinePools can be added as events to the MachinePool object and/or logged in the controller’s output.

            failureMessage
            string
            (Optional)

            FailureMessage will be set in the event that there is a terminal problem reconciling the MachinePool and will contain a more verbose string suitable for logging and human consumption.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the MachinePool’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of MachinePools can be added as events to the MachinePool object and/or logged in the controller’s output.

            conditions
            Cluster API api/v1alpha4.Conditions
            (Optional)

            Conditions defines current service state of the managed machine pool

            AutoScalingGroup

            AutoScalingGroup describes an AWS autoscaling group.

            Field Description
            id
            string

            The tags associated with the instance.

            tags
            Cluster API AWS api/v1alpha4.Tags
            name
            string
            desiredCapacity
            int32
            maxSize
            int32
            minSize
            int32
            placementGroup
            string
            subnets
            []string
            defaultCoolDown
            Kubernetes meta/v1.Duration
            capacityRebalance
            bool
            mixedInstancesPolicy
            MixedInstancesPolicy
            Status
            ASGStatus
            instances
            []Cluster API AWS api/v1alpha4.Instance

            BlockDeviceMapping

            BlockDeviceMapping specifies the block devices for the instance. You can specify virtual devices and EBS volumes.

            Field Description
            deviceName
            string

            The device name exposed to the EC2 instance (for example, /dev/sdh or xvdh).

            ebs
            EBS
            (Optional)

            You can specify either VirtualName or Ebs, but not both.

            EBS

            (Appears on:BlockDeviceMapping)

            EBS can be used to automatically set up EBS volumes when an instance is launched.

            Field Description
            encrypted
            bool
            (Optional)

            Encrypted is whether the volume should be encrypted or not.

            volumeSize
            int64
            (Optional)

            The size of the volume, in GiB. This can be a number from 1-1,024 for standard, 4-16,384 for io1, 1-16,384 for gp2, and 500-16,384 for st1 and sc1. If you specify a snapshot, the volume size must be equal to or larger than the snapshot size.

            volumeType
            string
            (Optional)

            The volume type For more information, see Amazon EBS Volume Types (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html)

            FargateProfileSpec

            (Appears on:AWSFargateProfile)

            FargateProfileSpec defines the desired state of FargateProfile

            Field Description
            clusterName
            string

            ClusterName is the name of the Cluster this object belongs to.

            profileName
            string

            ProfileName specifies the profile name.

            subnetIDs
            []string
            (Optional)

            SubnetIDs specifies which subnets are used for the auto scaling group of this nodegroup.

            additionalTags
            Cluster API AWS api/v1alpha4.Tags
            (Optional)

            AdditionalTags is an optional set of tags to add to AWS resources managed by the AWS provider, in addition to the ones added by default.

            roleName
            string
            (Optional)

            RoleName specifies the name of IAM role for this fargate pool If the role is pre-existing we will treat it as unmanaged and not delete it on deletion. If the EKSEnableIAM feature flag is true and no name is supplied then a role is created.

            selectors
            []FargateSelector

            Selectors specify fargate pod selectors.

            FargateProfileStatus

            (Appears on:AWSFargateProfile)

            FargateProfileStatus defines the observed state of FargateProfile

            Field Description
            ready
            bool

            Ready denotes that the FargateProfile is available.

            failureReason
            Cluster API errors.MachineStatusError
            (Optional)

            FailureReason will be set in the event that there is a terminal problem reconciling the FargateProfile and will contain a succinct value suitable for machine interpretation.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the FargateProfile’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of FargateProfiles can be added as events to the FargateProfile object and/or logged in the controller’s output.

            failureMessage
            string
            (Optional)

            FailureMessage will be set in the event that there is a terminal problem reconciling the FargateProfile and will contain a more verbose string suitable for logging and human consumption.

            This field should not be set for transitive errors that a controller faces that are expected to be fixed automatically over time (like service outages), but instead indicate that something is fundamentally wrong with the FargateProfile’s spec or the configuration of the controller, and that manual intervention is required. Examples of terminal errors would be invalid combinations of settings in the spec, values that are unsupported by the controller, or the responsible controller itself being critically misconfigured.

            Any transient errors that occur during the reconciliation of FargateProfiles can be added as events to the FargateProfile object and/or logged in the controller’s output.

            conditions
            Cluster API api/v1alpha4.Conditions
            (Optional)

            Conditions defines current state of the Fargate profile.

            FargateSelector

            (Appears on:FargateProfileSpec)

            FargateSelector specifies a selector for pods that should run on this fargate pool

            Field Description
            labels
            map[string]string

            Labels specifies which pod labels this selector should match.

            namespace
            string

            Namespace specifies which namespace this selector should match.

            InstancesDistribution

            (Appears on:MixedInstancesPolicy)

            InstancesDistribution to configure distribution of On-Demand Instances and Spot Instances.

            Field Description
            onDemandAllocationStrategy
            OnDemandAllocationStrategy
            spotAllocationStrategy
            SpotAllocationStrategy
            onDemandBaseCapacity
            int64
            onDemandPercentageAboveBaseCapacity
            int64

            ManagedMachineAMIType (string alias)

            (Appears on:AWSManagedMachinePoolSpec)

            ManagedMachineAMIType specifies which AWS AMI to use for a managed MachinePool.

            Value Description

            "AL2_ARM_64"

            Al2Arm64 is the Arm AMI type.

            "AL2_x86_64"

            Al2x86_64 is the default AMI type.

            "AL2_x86_64_GPU"

            Al2x86_64GPU is the x86-64 GPU AMI type.

            ManagedMachinePoolCapacityType (string alias)

            (Appears on:AWSManagedMachinePoolSpec)

            ManagedMachinePoolCapacityType specifies the capacity type to be used for the managed MachinePool.

            Value Description

            "onDemand"

            ManagedMachinePoolCapacityTypeOnDemand is the default capacity type, to launch on-demand instances.

            "spot"

            ManagedMachinePoolCapacityTypeSpot is the spot instance capacity type to launch spot instances.

            ManagedMachinePoolScaling

            (Appears on:AWSManagedMachinePoolSpec)

            ManagedMachinePoolScaling specifies scaling options.

            Field Description
            minSize
            int32
            maxSize
            int32

            ManagedRemoteAccess

            (Appears on:AWSManagedMachinePoolSpec)

            ManagedRemoteAccess specifies remote access settings for EC2 instances.

            Field Description
            sshKeyName
            string

            SSHKeyName specifies which EC2 SSH key can be used to access machines. If left empty, the key from the control plane is used.

            sourceSecurityGroups
            []string

            SourceSecurityGroups specifies which security groups are allowed access

            public
            bool

            Public specifies whether to open port 22 to the public internet

            MixedInstancesPolicy

            (Appears on:AWSMachinePoolSpec, AutoScalingGroup)

            MixedInstancesPolicy for an Auto Scaling group.

            Field Description
            instancesDistribution
            InstancesDistribution
            overrides
            []Overrides

            OnDemandAllocationStrategy (string alias)

            (Appears on:InstancesDistribution)

            OnDemandAllocationStrategy indicates how to allocate instance types to fulfill On-Demand capacity.

            Overrides

            (Appears on:MixedInstancesPolicy)

            Overrides are used to override the instance type specified by the launch template with multiple instance types that can be used to launch On-Demand Instances and Spot Instances.

            Field Description
            instanceType
            string

            RefreshPreferences

            (Appears on:AWSMachinePoolSpec)

            RefreshPreferences defines the specs for instance refreshing.

            Field Description
            strategy
            string
            (Optional)

            The strategy to use for the instance refresh. The only valid value is Rolling. A rolling update is an update that is applied to all instances in an Auto Scaling group until all instances have been updated.

            instanceWarmup
            int64
            (Optional)

            The number of seconds until a newly launched instance is configured and ready to use. During this time, the next replacement will not be initiated. The default is to use the value for the health check grace period defined for the group.

            minHealthyPercentage
            int64
            (Optional)

            The amount of capacity as a percentage in ASG that must remain healthy during an instance refresh. The default is 90.

            SpotAllocationStrategy (string alias)

            (Appears on:InstancesDistribution)

            SpotAllocationStrategy indicates how to allocate instances across Spot Instance pools.

            Tags (map[string]string alias)

            Tags is a mapping for tags.

            Taint

            Taint defines the specs for a Kubernetes taint.

            Field Description
            effect
            TaintEffect

            Effect specifies the effect for the taint

            key
            string

            Key is the key of the taint

            value
            string

            Value is the value of the taint

            TaintEffect (string alias)

            (Appears on:Taint)

            TaintEffect is the effect for a Kubernetes taint.

            Taints ([]../../exp/api/v1alpha4.Taint alias)

            (Appears on:AWSManagedMachinePoolSpec)

            Taints is an array of Taints.


            Generated with gen-crd-api-reference-docs on git commit 06e02b2d.

            Cluster API Provider AWS Roadmap

            This roadmap is a constant work in progress, subject to frequent revision. Dates are approximations.

            Ongoing

            • Documentation improvements

            v0.5.x (v1alpha3+) - June/July 2020

            v0.6 (v1alpha4) ~ Q4 2020

            TBD

            • Implement MachinePools - Autoscaling groups and instances
            • Spot instance support for MachinePool ASG/Instance implementation
            • MachinePool implementation backed by Spot Fleet
            • Dual stack IPv4/IPv6 support
            • Windows nodes
            • Support for GPU instances and Elastic GPU
            • FIPS/NIST/STIG compliance