Kubernetes-native declarative infrastructure for OpenStack.

What is the Cluster API Provider OpenStack

The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management.

The API itself is shared across multiple cloud providers allowing for true OpenStack hybrid deployments of Kubernetes. It is built atop the lessons learned from previous cluster managers such as kops and kubicorn.

Launching a Kubernetes cluster on OpenStack

• Check out the Cluster API Quick Start to create your first Kubernetes cluster on OpenStack using Cluster API. If you wish to use the external cloud provider, check out the External Cloud Provider as well.

Features

• Native Kubernetes manifests and API
• Choice of Linux distribution (as long as a current cloud-init is available)
• Support for single and multi-node control plane clusters
• Deploy clusters with and without LBaaS available (only cluster with LBaaS can be upgraded)
• Support for security groups
• cloud-init based nodes bootstrapping

Compatibility with Cluster API and Kubernetes Versions

This provider’s versions are compatible with the following versions of Cluster API:

This provider’s versions are able to install and manage the following versions of Kubernetes:

This provider’s versions are able to install Kubernetes to the following versions of OpenStack:

Test status:

• ✓ tested
• + should work, but we weren’t able to test it

Each version of Cluster API for OpenStack will attempt to support two Kubernetes versions.

NOTE: As the versioning for this project is tied to the versioning of Cluster API, future modifications to this policy may be made to more closely aligned with other providers in the Cluster API ecosystem.

NOTE: The minimum microversion of CAPI using nova is 2.53 now due to server tags support, see code for additional information.

NOTE: We require Keystone v3 for authentication.

Development versions

ClusterAPI provider OpenStack images and manifests are published after every PR merge and once every day:

• With a Google Cloud account you can get a quick overview here
• The manifests are available under:
  • master/infrastructure-components.yaml: latest build from the main branch, overwritten after every merge
  • e.g. nightly_master_20210407/infrastructure-components.yaml: build of the main branch from 7th April

These artifacts are published via Prow and Google Cloud Build. The corresponding job definitions can be found here.

Operating system images

Note: Cluster API Provider OpenStack relies on a few prerequisites which have to be already installed in the used operating system images, e.g. a container runtime, kubelet, kubeadm,.. . Reference images can be found in kubernetes-sigs/image-builder. If it isn’t possible to pre-install those prerequisites in the image, you can always deploy and execute some custom scripts through the KubeadmConfig.

Documentation

Please see our book for in-depth documentation.

Getting involved and contributing

Are you interested in contributing to cluster-api-provider-openstack? We, the maintainers and community, would love your suggestions, contributions, and help! Also, the maintainers can be contacted at any time to learn more about how to get involved:

• via the cluster-api-openstack channel on Kubernetes Slack
• via the SIG-Cluster-Lifecycle Mailing List.
• during our Office Hours
  • bi-weekly on Wednesdays @ 14:00 UTC on Zoom (link in meeting notes)
  • Previous meetings: [ notes | recordings ]

In the interest of getting more new people involved we try to tag issues with good first issue. These are typically issues that have smaller scope but are good ways to start to get acquainted with the codebase.

We also encourage ALL active community participants to act as if they are maintainers, even if you don’t have “official” write permissions. This is a community effort, we are here to serve the Kubernetes community. If you have an active interest and you want to get involved, you have real power! Don’t assume that the only people who can get things done around here are the “maintainers”.

We also would love to add more “official” maintainers, so show us what you can do!

This repository uses the Kubernetes bots. See a full list of the commands here. Please also refer to the Contribution Guide and the Development Guide for this project.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

Github issues

Bugs

If you think you have found a bug please follow the instructions below.

• Please spend a small amount of time giving due diligence to the issue tracker. Your issue might be a duplicate.
• Get the logs from the cluster controllers. Please paste this into your issue.
• Open a new issue.
• Remember that users might be searching for your issue in the future, so please give it a meaningful title to help others.
• Feel free to reach out to the Cluster API community on the Kubernetes Slack.

Tracking new features

We also use the issue tracker to track features. If you have an idea for a feature, or think you can help Cluster API Provider OpenStack become even more awesome follow the steps below.

• Open a new issue.
• Remember that users might be searching for your issue in the future, so please give it a meaningful title to help others.
• Clearly define the use case, using concrete examples.
• Some of our larger features will require some design. If you would like to include a technical design for your feature, please include it in the issue.
• After the new feature is well understood, and the design agreed upon, we can start coding the feature. We would love for you to code it. So please open up a WIP (work in progress) pull request, and happy coding.

Table of Contents generated with DocToc

• Getting Started

Getting Started

Quick Start

In this tutorial we’ll cover the basics of how to use Cluster API to create one or more Kubernetes clusters.

Installation

Common Prerequisites

• Install and setup kubectl in your local environment
• Install kind and Docker

Install and/or configure a Kubernetes cluster

Cluster API requires an existing Kubernetes cluster accessible via kubectl. During the installation process the Kubernetes cluster will be transformed into a management cluster by installing the Cluster API provider components, so it is recommended to keep it separated from any application workload.

It is a common practice to create a temporary, local bootstrap cluster which is then used to provision a target management cluster on the selected infrastructure provider.

Choose one of the options below:

1. Existing Management Cluster

   For production use-cases a “real” Kubernetes cluster should be used with appropriate backup and disaster recovery policies and procedures in place. The Kubernetes cluster must be at least v1.20.0.

   export KUBECONFIG=<...>
   

OR

2. Kind

   kind can be used for creating a local Kubernetes cluster for development environments or for the creation of a temporary bootstrap cluster used to provision a target management cluster on the selected infrastructure provider.

   The installation procedure depends on the version of kind; if you are planning to use the Docker infrastructure provider, please follow the additional instructions in the dedicated tab:

   DefaultDockerKubeVirt

   Create the kind cluster:

   kind create cluster
   

   Test to ensure the local kind cluster is ready:

   kubectl cluster-info
   

   Run the following command to create a kind config file for allowing the Docker provider to access Docker on the host:

   cat > kind-cluster-with-extramounts.yaml <<EOF
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   nodes:
   - role: control-plane
     extraMounts:
       - hostPath: /var/run/docker.sock
         containerPath: /var/run/docker.sock
   EOF
   

   Then follow the instruction for your kind version using kind create cluster --config kind-cluster-with-extramounts.yaml to create the management cluster using the above file.

   Create the Kind Cluster

   KubeVirt is a cloud native virtualization solution. The virtual machines we’re going to create and use for the workload cluster’s nodes, are actually running within pods in the management cluster. In order to communicate with the workload cluster’s API server, we’ll need to expose it. We are using Kind which is a limited environment. The easiest way to expose the workload cluster’s API server (a pod within a node running in a VM that is itself running within a pod in the management cluster, that is running inside a docker container), is to use a LoadBalancer service.

   To allow using a LoadBalancer service, we can’t use the kind’s default CNI (kindnet), but we’ll need to install another CNI, like Calico. In order to do that, we’ll need first to initiate the kind cluster with two modifications:

   1. Disable the default CNI
   2. Add the docker credentials to the cluster, to avoid the docker hub pull rate limit of the calico images; read more about it in the docker documentation, and in the kind documentation.

   Create a configuration file for kind. Please notice the docker config file path, and adjust it to your local setting:

   cat <<EOF > kind-config.yaml
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   networking:
   # the default CNI will not be installed
     disableDefaultCNI: true
   nodes:
   - role: control-plane
     extraMounts:
      - containerPath: /var/lib/kubelet/config.json
        hostPath: <YOUR DOCKER CONFIG FILE PATH>
   EOF
   

   Now, create the kind cluster with the configuration file:

   kind create cluster --config=kind-config.yaml
   

   Test to ensure the local kind cluster is ready:

   kubectl cluster-info
   

   Install the Calico CNI

   Now we’ll need to install a CNI. In this example, we’re using calico, but other CNIs should work as well. Please see calico installation guide for more details (use the “Manifest” tab). Below is an example of how to install calico version v3.24.4.

   Use the Calico manifest to create the required resources; e.g.:

   kubectl create -f  https://raw.githubusercontent.com/projectcalico/calico/v3.24.4/manifests/calico.yaml
   

Install clusterctl

The clusterctl CLI tool handles the lifecycle of a Cluster API management cluster.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.6/clusterctl-linux-amd64 -o clusterctl

sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl

clusterctl version

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.6/clusterctl-darwin-amd64 -o clusterctl

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.6/clusterctl-darwin-arm64 -o clusterctl

chmod +x ./clusterctl

sudo mv ./clusterctl /usr/local/bin/clusterctl

clusterctl version

brew install clusterctl

clusterctl version

curl.exe -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.2.6/clusterctl-windows-amd64.exe -o clusterctl.exe

clusterctl.exe version

Initialize the management cluster

Now that we’ve got clusterctl installed and all the prerequisites in place, let’s transform the Kubernetes cluster into a management cluster by using clusterctl init.

The command accepts as input a list of providers to install; when executed for the first time, clusterctl init automatically adds to the list the cluster-api core provider, and if unspecified, it also adds the kubeadm bootstrap and kubeadm control-plane providers.

Enabling Feature Gates

Feature gates can be enabled by exporting environment variables before executing clusterctl init. For example, the ClusterTopology feature, which is required to enable support for managed topologies and ClusterClass, can be enabled via:

export CLUSTER_TOPOLOGY=true

Additional documentation about experimental features can be found in Experimental Features.

Initialization for common providers

Depending on the infrastructure provider you are planning to use, some additional prerequisites should be satisfied before getting started with Cluster API. See below for the expected settings for common providers.

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.5.2/clusterawsadm-linux-amd64 -o clusterawsadm

chmod +x clusterawsadm

sudo mv clusterawsadm /usr/local/bin

clusterawsadm version

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.5.2/clusterawsadm-darwin-amd64 -o clusterawsadm

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v1.5.2/clusterawsadm-darwin-arm64 -o clusterawsadm

chmod +x clusterawsadm

sudo mv clusterawsadm /usr/local/bin

clusterawsadm version

brew install clusterawsadm

clusterawsadm version

export AWS_REGION=us-east-1 # This is used to help encode your environment variables
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<session-token> # If you are using Multi-Factor Auth.

# The clusterawsadm utility takes the credentials that you set as environment
# variables and uses them to create a CloudFormation stack in your AWS account
# with the correct IAM resources.
clusterawsadm bootstrap iam create-cloudformation-stack

# Create the base64 encoded credentials using clusterawsadm.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

# Finally, initialize the management cluster
clusterctl init --infrastructure aws

export AZURE_SUBSCRIPTION_ID="<SubscriptionId>"

# Create an Azure Service Principal and paste the output here
export AZURE_TENANT_ID="<Tenant>"
export AZURE_CLIENT_ID="<AppId>"
export AZURE_CLIENT_SECRET="<Password>"

# Base64 encode the variables
export AZURE_SUBSCRIPTION_ID_B64="$(echo -n "$AZURE_SUBSCRIPTION_ID" | base64 | tr -d '\n')"
export AZURE_TENANT_ID_B64="$(echo -n "$AZURE_TENANT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_ID_B64="$(echo -n "$AZURE_CLIENT_ID" | base64 | tr -d '\n')"
export AZURE_CLIENT_SECRET_B64="$(echo -n "$AZURE_CLIENT_SECRET" | base64 | tr -d '\n')"

# Settings needed for AzureClusterIdentity used by the AzureCluster
export AZURE_CLUSTER_IDENTITY_SECRET_NAME="cluster-identity-secret"
export CLUSTER_IDENTITY_NAME="cluster-identity"
export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default"

# Create a secret to include the password of the Service Principal identity created in Azure
# This secret will be referenced by the AzureClusterIdentity used by the AzureCluster
kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" --namespace "${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}"

# Finally, initialize the management cluster
clusterctl init --infrastructure azure

[Global]
api-url = <cloudstackApiUrl>
api-key = <cloudstackApiKey>
secret-key = <cloudstackSecretKey>

export CLOUDSTACK_B64ENCODED_SECRET=`cat cloud-config | base64 | tr -d '\n'`

clusterctl init --infrastructure cloudstack

export DIGITALOCEAN_ACCESS_TOKEN=<your-access-token>
export DO_B64ENCODED_CREDENTIALS="$(echo -n "${DIGITALOCEAN_ACCESS_TOKEN}" | base64 | tr -d '\n')"

# Initialize the management cluster
clusterctl init --infrastructure digitalocean

# Enable the experimental Cluster topology feature.
export CLUSTER_TOPOLOGY=true

# Initialize the management cluster
clusterctl init --infrastructure docker

export PACKET_API_KEY="34ts3g4s5g45gd45dhdh"

clusterctl init --infrastructure packet

# Create the base64 encoded credentials by catting your credentials json.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export GCP_B64ENCODED_CREDENTIALS=$( cat /path/to/gcp-credentials.json | base64 | tr -d '\n' )

# Finally, initialize the management cluster
clusterctl init --infrastructure gcp

export IBMCLOUD_API_KEY=<you_api_key>

# Finally, initialize the management cluster
clusterctl init --infrastructure ibmcloud

# Initialize the management cluster
clusterctl init --infrastructure kubekey

METALLB_VER=$(curl "https://api.github.com/repos/metallb/metallb/releases/latest" | jq -r ".tag_name")
kubectl apply -f "https://raw.githubusercontent.com/metallb/metallb/${METALLB_VER}/config/manifests/metallb-native.yaml"
kubectl wait pods -n metallb-system -l app=metallb,component=controller --for=condition=Ready --timeout=10m
kubectl wait pods -n metallb-system -l app=metallb,component=speaker --for=condition=Ready --timeout=2m

GW_IP=$(docker network inspect -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}' kind)
NET_IP=$(echo ${GW_IP} | sed -E 's|^([0-9]+\.[0-9]+)\..*$|\1|g')
cat <<EOF | sed -E "s|172.19|${NET_IP}|g" | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: capi-ip-pool
  namespace: metallb-system
spec:
  addresses:
  - 172.19.255.200-172.19.255.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: empty
  namespace: metallb-system
EOF

# get KubeVirt version
KV_VER=$(curl "https://api.github.com/repos/kubevirt/kubevirt/releases/latest" | jq -r ".tag_name")
# deploy required CRDs
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${KV_VER}/kubevirt-operator.yaml"
# deploy the KubeVirt custom resource
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${KV_VER}/kubevirt-cr.yaml"
kubectl wait -n kubevirt kv kubevirt --for=condition=Available --timeout=10m

clusterctl init --infrastructure kubevirt

# Initialize the management cluster
clusterctl init --infrastructure openstack

export OSC_SECRET_KEY=<your-secret-key>
export OSC_ACCESS_KEY=<your-access-key>
export OSC_REGION=<you-region>
# Create namespace
kubectl create namespace cluster-api-provider-outscale-system
# Create secret
kubectl create secret generic cluster-api-provider-outscale --from-literal=access_key=${OSC_ACCESS_KEY} --from-literal=secret_key=${OSC_SECRET_KEY} --from-literal=region=${OSC_REGION}  -n cluster-api-provider-outscale-system 
# Initialize the management cluster
clusterctl init --infrastructure outscale

# Initialize the management cluster
clusterctl init --infrastructure vcd

clusterctl init --infrastructure vcluster

# Initialize the management cluster
clusterctl init --infrastructure virtink

# The username used to access the remote vSphere endpoint
export VSPHERE_USERNAME="vi-admin@vsphere.local"
# The password used to access the remote vSphere endpoint
# You may want to set this in ~/.cluster-api/clusterctl.yaml so your password is not in
# bash history
export VSPHERE_PASSWORD="admin!23"

# Finally, initialize the management cluster
clusterctl init --infrastructure vsphere

The output of clusterctl init is similar to this:

Fetching providers
Installing cert-manager Version="v1.10.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.0.0" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.0.0" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.0.0" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v1.0.0" TargetNamespace="capd-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

Create your first workload cluster

Once the management cluster is ready, you can create your first workload cluster.

Preparing the workload cluster configuration

The clusterctl generate cluster command returns a YAML template for creating a workload cluster.

Required configuration for common providers

Depending on the infrastructure provider you are planning to use, some additional prerequisites should be satisfied before configuring a cluster with Cluster API. Instructions are provided for common providers below.

Otherwise, you can look at the clusterctl generate cluster command documentation for details about how to discover the list of variables required by a cluster templates.

export AWS_REGION=us-east-1
export AWS_SSH_KEY_NAME=default
# Select instance types
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large

# Name of the Azure datacenter location. Change this value to your desired location.
export AZURE_LOCATION="centralus"

# Select VM types.
export AZURE_CONTROL_PLANE_MACHINE_TYPE="Standard_D2s_v3"
export AZURE_NODE_MACHINE_TYPE="Standard_D2s_v3"

# [Optional] Select resource group. The default value is ${CLUSTER_NAME}.
export AZURE_RESOURCE_GROUP="<ResourceGroupName>"

clusterctl generate cluster --infrastructure cloudstack --list-variables capi-quickstart

# Set this to the name of the zone in which to deploy the cluster
export CLOUDSTACK_ZONE_NAME=<zone name>
# The name of the network on which the VMs will reside
export CLOUDSTACK_NETWORK_NAME=<network name>
# The endpoint of the workload cluster
export CLUSTER_ENDPOINT_IP=<cluster endpoint address>
export CLUSTER_ENDPOINT_PORT=<cluster endpoint port>
# The service offering of the control plane nodes
export CLOUDSTACK_CONTROL_PLANE_MACHINE_OFFERING=<control plane service offering name>
# The service offering of the worker nodes
export CLOUDSTACK_WORKER_MACHINE_OFFERING=<worker node service offering name>
# The capi compatible template to use
export CLOUDSTACK_TEMPLATE_NAME=<template name>
# The ssh key to use to log into the nodes
export CLOUDSTACK_SSH_KEY_NAME=<ssh key name>

export DO_REGION=nyc1
export DO_SSH_KEY_FINGERPRINT=<your-ssh-key-fingerprint>
export DO_CONTROL_PLANE_MACHINE_TYPE=s-2vcpu-2gb
export DO_CONTROL_PLANE_MACHINE_IMAGE=<your-capi-image-id>
export DO_NODE_MACHINE_TYPE=s-2vcpu-2gb
export DO_NODE_MACHINE_IMAGE==<your-capi-image-id>

# The list of service CIDR, default ["10.128.0.0/12"]
export SERVICE_CIDR=["10.96.0.0/12"]

# The list of pod CIDR, default ["192.168.0.0/16"]
export POD_CIDR=["192.168.0.0/16"]

# The service domain, default "cluster.local"
export SERVICE_DOMAIN="k8s.test"

export ENABLE_POD_SECURITY_STANDARD="false"

# Required (made up examples shown)
# The project where your cluster will be placed to.
# You have to get one from the Equinix Metal Console if you don't have one already.
export PROJECT_ID="2b59569f-10d1-49a6-a000-c2fb95a959a1"
# The facility where you want your cluster to be provisioned
export FACILITY="da11"
# What plan to use for your control plane nodes
export CONTROLPLANE_NODE_TYPE="m3.small.x86"
# What plan to use for your worker nodes
export WORKER_NODE_TYPE="m3.small.x86"
# The ssh key you would like to have access to the nodes
export SSH_KEY="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDvMgVEubPLztrvVKgNPnRe9sZSjAqaYj9nmCkgr4PdK username@computer"
export CLUSTER_NAME="my-cluster"

# Optional (defaults shown)
export NODE_OS="ubuntu_18_04"
export POD_CIDR="192.168.0.0/16"
export SERVICE_CIDR="172.26.0.0/16"
# Only relevant if using the kube-vip flavor
export KUBE_VIP_VERSION="v0.5.0"

# Name of the GCP datacenter location. Change this value to your desired location
export GCP_REGION="<GCP_REGION>"
export GCP_PROJECT="<GCP_PROJECT>"
# Make sure to use same kubernetes version here as building the GCE image
export KUBERNETES_VERSION=1.23.3
# This is the image you built. See https://github.com/kubernetes-sigs/image-builder
export IMAGE_ID=projects/$GCP_PROJECT/global/images/<built image>
export GCP_CONTROL_PLANE_MACHINE_TYPE=n1-standard-2
export GCP_NODE_MACHINE_TYPE=n1-standard-2
export GCP_NETWORK_NAME=<GCP_NETWORK_NAME or default>
export CLUSTER_NAME="<CLUSTER_NAME>"

# Required environment variables for VPC
# VPC region
export IBMVPC_REGION=us-south
# VPC zone within the region
export IBMVPC_ZONE=us-south-1
# ID of the resource group in which the VPC will be created
export IBMVPC_RESOURCEGROUP=<your-resource-group-id>
# Name of the VPC
export IBMVPC_NAME=ibm-vpc-0
export IBMVPC_IMAGE_ID=<you-image-id>
# Profile for the virtual server instances
export IBMVPC_PROFILE=bx2-4x16
export IBMVPC_SSHKEY_ID=<your-sshkey-id>

# Required environment variables for PowerVS
export IBMPOWERVS_SSHKEY_NAME=<your-ssh-key>
# Internal and external IP of the network
export IBMPOWERVS_VIP=<internal-ip>
export IBMPOWERVS_VIP_EXTERNAL=<external-ip>
export IBMPOWERVS_VIP_CIDR=29
export IBMPOWERVS_IMAGE_NAME=<your-capi-image-name>
# ID of the PowerVS service instance
export IBMPOWERVS_SERVICE_INSTANCE_ID=<service-instance-id>
export IBMPOWERVS_NETWORK_NAME=<your-capi-network-name>

# Required environment variables
# The KKZONE is used to specify where to download the binaries. (e.g. "", "cn")
export KKZONE=""
# The ssh name of the all instance Linux user. (e.g. root, ubuntu)
export USER_NAME=<your-linux-user>
# The ssh password of the all instance Linux user. 
export PASSWORD=<your-linux-user-password>
# The ssh IP address of the all instance. (e.g. "[{address: 192.168.100.3}, {address: 192.168.100.4}]")
export INSTANCES=<your-linux-ip-address>
# The cluster control plane VIP. (e.g. "192.168.100.100")
export CONTROL_PLANE_ENDPOINT_IP=<your-control-plane-virtual-ip>

export CAPK_GUEST_K8S_VERSION="v1.23.10"
export CRI_PATH="/var/run/containerd/containerd.sock"
export NODE_VM_IMAGE_TEMPLATE="quay.io/capk/ubuntu-2004-container-disk:${CAPK_GUEST_K8S_VERSION}"

# The URL of the kernel to deploy.
export DEPLOY_KERNEL_URL="http://172.22.0.1:6180/images/ironic-python-agent.kernel"
# The URL of the ramdisk to deploy.
export DEPLOY_RAMDISK_URL="http://172.22.0.1:6180/images/ironic-python-agent.initramfs"
# The URL of the Ironic endpoint.
export IRONIC_URL="http://172.22.0.1:6385/v1/"
# The URL of the Ironic inspector endpoint.
export IRONIC_INSPECTOR_URL="http://172.22.0.1:5050/v1/"
# Do not use a dedicated CA certificate for Ironic API. Any value provided in this variable disables additional CA certificate validation.
# To provide a CA certificate, leave this variable unset. If unset, then IRONIC_CA_CERT_B64 must be set.
export IRONIC_NO_CA_CERT=true
# Disables basic authentication for Ironic API. Any value provided in this variable disables authentication.
# To enable authentication, leave this variable unset. If unset, then IRONIC_USERNAME and IRONIC_PASSWORD must be set.
export IRONIC_NO_BASIC_AUTH=true
# Disables basic authentication for Ironic inspector API. Any value provided in this variable disables authentication.
# To enable authentication, leave this variable unset. If unset, then IRONIC_INSPECTOR_USERNAME and IRONIC_INSPECTOR_PASSWORD must be set.
export IRONIC_INSPECTOR_NO_BASIC_AUTH=true

clusterctl generate cluster --infrastructure nutanix --list-variables capi-quickstart

clusterctl generate cluster --infrastructure openstack --list-variables capi-quickstart

wget https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-openstack/master/templates/env.rc -O /tmp/env.rc
source /tmp/env.rc <path/to/clouds.yaml> <cloud>

# The list of nameservers for OpenStack Subnet being created.
# Set this value when you need create a new network/subnet while the access through DNS is required.
export OPENSTACK_DNS_NAMESERVERS=<dns nameserver>
# FailureDomain is the failure domain the machine will be created in.
export OPENSTACK_FAILURE_DOMAIN=<availability zone name>
# The flavor reference for the flavor for your server instance.
export OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR=<flavor>
# The flavor reference for the flavor for your server instance.
export OPENSTACK_NODE_MACHINE_FLAVOR=<flavor>
# The name of the image to use for your server instance. If the RootVolume is specified, this will be ignored and use rootVolume directly.
export OPENSTACK_IMAGE_NAME=<image name>
# The SSH key pair name
export OPENSTACK_SSH_KEY_NAME=<ssh key pair name>
# The external network
export OPENSTACK_EXTERNAL_NETWORK_ID=<external network ID>

# The outscale root disk iops
export OSC_IOPS="<IOPS>"
# The outscale root disk size
export OSC_VOLUME_SIZE="<VOLUME_SIZE>"
# The outscale root disk volumeType
export OSC_VOLUME_TYPE="<VOLUME_TYPE>"
# The outscale key pair
export OSC_KEYPAIR_NAME="<KEYPAIR_NAME>"
# The outscale subregion name
export OSC_SUBREGION_NAME="<SUBREGION_NAME>"
# The outscale vm type
export OSC_VM_TYPE="<VM_TYPE>"
# The outscale image name
export OSC_IMAGE_NAME="<IMAGE_NAME>"

clusterctl generate cluster --infrastructure vcd --list-variables capi-quickstart

export CLUSTER_NAME=kind
export CLUSTER_NAMESPACE=vcluster
export KUBERNETES_VERSION=1.23.4
export HELM_VALUES="service:\n  type: NodePort"

clusterctl generate cluster --infrastructure virtink --list-variables capi-quickstart

# The vCenter server IP or FQDN
export VSPHERE_SERVER="10.0.0.1"
# The vSphere datacenter to deploy the management cluster on
export VSPHERE_DATACENTER="SDDC-Datacenter"
# The vSphere datastore to deploy the management cluster on
export VSPHERE_DATASTORE="vsanDatastore"
# The VM network to deploy the management cluster on
export VSPHERE_NETWORK="VM Network"
# The vSphere resource pool for your VMs
export VSPHERE_RESOURCE_POOL="*/Resources"
# The VM folder for your VMs. Set to "" to use the root vSphere folder
export VSPHERE_FOLDER="vm"
# The VM template to use for your VMs
export VSPHERE_TEMPLATE="ubuntu-1804-kube-v1.17.3"
# The public ssh authorized key on all machines
export VSPHERE_SSH_AUTHORIZED_KEY="ssh-rsa AAAAB3N..."
# The certificate thumbprint for the vCenter server
export VSPHERE_TLS_THUMBPRINT="97:48:03:8D:78:A9..."
# The storage policy to be used (optional). Set to "" if not required
export VSPHERE_STORAGE_POLICY="policy-one"
# The IP address used for the control plane endpoint
export CONTROL_PLANE_ENDPOINT_IP="1.2.3.4"

Generating the cluster configuration

For the purpose of this tutorial, we’ll name our cluster capi-quickstart.

clusterctl generate cluster capi-quickstart --flavor development \
  --kubernetes-version v1.25.3 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

export CLUSTER_NAME=kind
export CLUSTER_NAMESPACE=vcluster
export KUBERNETES_VERSION=1.25.0
export HELM_VALUES="service:\n  type: NodePort"

kubectl create namespace ${CLUSTER_NAMESPACE}
clusterctl generate cluster ${CLUSTER_NAME} \
    --infrastructure vcluster \
    --kubernetes-version ${KUBERNETES_VERSION} \
    --target-namespace ${CLUSTER_NAMESPACE} | kubectl apply -f -

clusterctl generate cluster capi-quickstart \
  --infrastructure="kubevirt" \
  --flavor lb \
  --kubernetes-version ${CAPK_GUEST_K8S_VERSION} \
  --control-plane-machine-count=1 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

clusterctl generate cluster capi-quickstart \
  --kubernetes-version v1.25.3 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

This creates a YAML file named capi-quickstart.yaml with a predefined list of Cluster API objects; Cluster, Machines, Machine Deployments, etc.

The file can be eventually modified using your editor of choice.

See clusterctl generate cluster for more details.

Apply the workload cluster

When ready, run the following command to apply the cluster manifest.

kubectl apply -f capi-quickstart.yaml

The output is similar to this:

cluster.cluster.x-k8s.io/capi-quickstart created
dockercluster.infrastructure.cluster.x-k8s.io/capi-quickstart created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-quickstart-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-control-plane created
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0 created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/capi-quickstart-md-0 created

Accessing the workload cluster

The cluster will now start provisioning. You can check status with:

kubectl get cluster

You can also get an “at glance” view of the cluster and its resources by running:

clusterctl describe cluster capi-quickstart

To verify the first control plane is up:

kubectl get kubeadmcontrolplane

You should see an output is similar to this:

NAME                    CLUSTER           INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE    VERSION
capi-quickstart-g2trk   capi-quickstart   true                                 3                  3         3             4m7s   v1.25.3

After the first control plane node is up and running, we can retrieve the workload cluster Kubeconfig.

clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

kind get kubeconfig --name capi-quickstart > capi-quickstart.kubeconfig

Deploy a CNI solution

Calico is used here as an example.

kubectl --kubeconfig=./capi-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico.yaml

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

kubectl get vm

NAME                                  AGE    STATUS    READY
capi-quickstart-control-plane-7s945   167m   Running   True
capi-quickstart-md-0-zht5j            164m   Running   True

kubectl get vmi

NAME                                  AGE    PHASE     IP             NODENAME             READY
capi-quickstart-control-plane-7s945   167m   Running   10.244.82.16   kind-control-plane   True
capi-quickstart-md-0-zht5j            164m   Running   10.244.82.17   kind-control-plane   True

curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.4/manifests/calico.yaml -o calico-workload.yaml

sed -i -E 's|^( +)# (- name: CALICO_IPV4POOL_CIDR)$|\1\2|g;'\
's|^( +)# (  value: )"192.168.0.0/16"|\1\2"10.243.0.0/16"|g;'\
'/- name: CLUSTER_TYPE/{ n; s/( +value: ").+/\1k8s"/g };'\
'/- name: CALICO_IPV4POOL_IPIP/{ n; s/value: "Always"/value: "Never"/ };'\
'/- name: CALICO_IPV4POOL_VXLAN/{ n; s/value: "Never"/value: "Always"/};'\
'/# Set Felix endpoint to host default action to ACCEPT./a\            - name: FELIX_VXLANPORT\n              value: "6789"' \
calico-workload.yaml

kubectl --kubeconfig=./capi-quickstart.kubeconfig create -f calico-workload.yaml

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

kubectl --kubeconfig=./capi-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

NAME                                          STATUS   ROLES           AGE   VERSION
capi-quickstart-g2trk-9xrjv                   Ready    control-plane   12m   v1.25.3
capi-quickstart-g2trk-bmm9v                   Ready    control-plane   11m   v1.25.3
capi-quickstart-g2trk-hvs9q                   Ready    control-plane   13m   v1.25.3
capi-quickstart-md-0-55x6t-5649968bd7-8tq9v   Ready    <none>          12m   v1.25.3
capi-quickstart-md-0-55x6t-5649968bd7-glnjd   Ready    <none>          12m   v1.25.3
capi-quickstart-md-0-55x6t-5649968bd7-sfzp6   Ready    <none>          12m   v1.25.3

Clean Up

Delete workload cluster.

kubectl delete cluster capi-quickstart

Delete management cluster

kind delete cluster

Next steps

See the clusterctl documentation for more detail about clusterctl supported actions.

Table of Contents generated with DocToc

• Required configuration
  • OpenStack version
  • Operating system image
  • SSH key pair
  • OpenStack credential
    • Generate credentials
  • Availability zone
  • DNS server
  • Machine flavor
• Optional Configuration
  • Log level
  • External network
  • API server floating IP
    • Disabling the API server floating IP
    • Restrict Access to the API server
  • Network Filters
  • Multiple Networks
  • Subnet Filters
  • Ports
  • Security groups
  • Tagging
  • Metadata
  • Boot From Volume
  • Timeout settings
  • Custom pod network CIDR
  • Accessing nodes through the bastion host via SSH
    • Enabling the bastion host
    • Obtain floating IP address of the bastion node

Required configuration

The cluster configuration file can be generated by using clusterctl generate cluster command. This command actually uses the template file and replace the values surrounded by ${} with environment variables. You have to set all required environment variables in advance. The following sections explain some more details about what should be configured.

Note: You can use the template file by manually replacing values.

Note: By default the command creates highly available control plane with internal OpenStack cloud provider. If you wish to create highly available control plane with external OpenStack cloud provider or single control plane without load balancer, use external-cloud-provider or without-lb flavor respectively. For example,

# Using 'external-cloud-provider' flavor
clusterctl generate cluster capi-quickstart \
  --flavor external-cloud-provider \
  --kubernetes-version v1.24.2 \
  --control-plane-machine-count=3 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

# Using 'without-lb' flavor
clusterctl generate cluster capi-quickstart \
  --flavor without-lb \
  --kubernetes-version v1.24.2 \
  --control-plane-machine-count=1 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

OpenStack version

We currently require at least OpenStack Pike.

Operating system image

We currently depend on an up-to-date version of cloud-init otherwise the operating system choice is yours. The kubeadm bootstrap provider we’re using also depends on some pre-installed software like a container runtime, kubelet, kubeadm, etc.. . For an examples how to build such an image take a look at image-builder (openstack).

The image can be referenced by exposing it as an environment variable OPENSTACK_IMAGE_NAME.

SSH key pair

The SSH key pair is required. You can create one using,

openstack keypair create [--public-key <file> | --private-key <file>] <name>

The key pair name must be exposed as an environment variable OPENSTACK_SSH_KEY_NAME.

In order to access cluster nodes via SSH, you must either access nodes through the bastion host or configure custom security groups with rules allowing ingress for port 22.

OpenStack credential

Generate credentials

The env.rc script sets the environment variables related to credentials. It’s highly recommend to avoid using admin credential.

source env.rc <path/to/clouds.yaml> <cloud>

The following variables are set.

Note: Only the external cloud provider supports Application Credentials.

Note: you need to set clusterctl.cluster.x-k8s.io/move label for the secret created from OPENSTACK_CLOUD_YAML_B64 in order to successfully move objects from bootstrap cluster to target cluster. See bug 626 for further information.

Availability zone

The availability zone names must be exposed as an environment variable OPENSTACK_FAILURE_DOMAIN.

By default, if Availability zone is not given, all Availability zone that defined in openstack will be a candidate to provision from, If administrator credential is used then internal Availability zone which is internal only Availability zone inside nova will be returned and can cause potential problem, see PR 1165 for further information. So we highly recommend to set Availability zone explicitly.

DNS server

The DNS servers must be exposed as an environment variable OPENSTACK_DNS_NAMESERVERS.

Machine flavor

The flavors for control plane and worker node machines must be exposed as environment variables OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR and OPENSTACK_NODE_MACHINE_FLAVOR respectively.

Optional Configuration

Log level

When running CAPO with --v=6 the gophercloud client logs its requests to the OpenStack API. This can be helpful during debugging.

External network

If there is only a single external network it will be detected automatically. If there is more than one external network you can specify which one the cluster should use by setting the environment variable OPENSTACK_EXTERNAL_NETWORK_ID.

The public network id can be obtained by using command,

openstack network list --external

Note: If your openstack cluster does not already have a public network, you should contact your cloud service provider. We will not review how to troubleshoot this here.

API server floating IP

Unless explicitly disabled, a floating IP is automatically created and associated with the load balancer or controller node. If required, you can specify the floating IP explicitly by spec.apiServerFloatingIP of OpenStackCluster.

You have to be able to create a floating IP in your OpenStack in advance. You can create one using,

openstack floating ip create <public network>

Note: Only user with admin role can create a floating IP with specific IP.

Note: When associating a floating IP to a cluster with more than 1 controller node, the floatingIP will be associated to the first controller node and the other controller nodes have no floating IP assigned. When the controller node has the floating IP status down CAPO will NOT auto assign the floating IP address to any other controller node. So we recommend to only set one controller node when floating IP is needed, or please consider using load balancer instead, see issue #1265 for further information.

Disabling the API server floating IP

It is possible to provision a cluster without a floating IP for the API server by setting OpenStackCluster.spec.disableAPIServerFloatingIP: true (the default is false). This will prevent a floating IP from being allocated.

WARNING

If the API server does not have a floating IP, workload clusters will only deploy successfully when the management cluster and workload cluster control plane nodes are on the same network. This can be a project-specific network, if the management cluster lives in the same project as the workload cluster, or a network that is shared across multiple projects.

In particular, this means that the cluster cannot use OpenStackCluster.spec.nodeCidr to provision a new network for the cluster. Instead, use OpenStackCluster.spec.network to explicitly specify the same network as the management cluster is on.

When the API server floating IP is disabled, it is not possible to provision a cluster without a load balancer without additional configuration (an advanced use-case that is not documented here). This is because the API server must still have a virtual IP that is not associated with a particular control plane node in order to allow the nodes to change underneath, e.g. during an upgrade. When the API server has a floating IP, this role is fulfilled by the floating IP even if there is no load balancer. When the API server does not have a floating IP, the load balancer virtual IP on the cluster network is used.

Restrict Access to the API server

NOTE

This requires “amphora” as load balancer provider at in version >= v2.12

It is possible to restrict access to the Kubernetes API server on a network level. If required, you can specify the allowed CIDRs by spec.APIServerLoadBalancer.AllowedCIDRs of OpenStackCluster.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackCluster
metadata:
  name: <cluster-name>
  namespace: <cluster-namespace>
spec:
  allowAllInClusterTraffic: true
  apiServerLoadBalancer:
    allowedCidrs:
    - 192.168.10/24
    - 10.10.0.0/16

All known IPs of the target cluster will be discovered dynamically (e.g. you don’t have to take care of target Cluster own Router IP, internal CIDRs or any Bastion Host IP). Note: Please ensure, that at least the outgoing IP of your management Cluster is added to the list of allowed CIDRs. Otherwise CAPO can’t reconcile the target Cluster correctly.

All applied CIDRs (user defined + dynamically discovered) are written back into status.network.apiServerLoadBalancer.allowedCIDRs

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackCluster
metadata:
  name: <cluster-name>
  namespace: <cluster-namespace>
status:
  network:
    apiServerLoadBalancer:
      allowedCIDRs:
        - 10.6.0.0/24 # openStackCluster.Status.Network.Subnet.CIDR
        - 10.6.0.90/32 # bastion Host internal IP
        - 10.10.0.0/16 # user defined
        - 192.168.10/24 # user defined
        - 172.16.111.100/32 # bastion host floating IP
        - 172.16.111.85/32 # router IP
      internalIP: 10.6.0.144
      ip: 172.16.111.159
      name: k8s-clusterapi-cluster-<cluster-namespace>-<cluster-name>

If you locked out yourself or the CAPO management cluster, you can easily clear the allowed_cidrs field on OpenStack via

openstack loadbalancer listener unset --allowed-cidrs <listener ID>

Network Filters

If you have a complex query that you want to use to lookup a network, then you can do this by using a network filter. More details about the filter can be found in NetworkParam

By using filters to look up a network, please note that it is possible to get multiple networks as a result. This should not be a problem, however please test your filters with openstack network list to be certain that it returns the networks you want. Please refer to the following usage example:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
  networks:
  - filter:
      name: <network-name>

Multiple Networks

You can specify multiple networks (or subnets) to connect your server to. To do this, simply add another entry in the networks array. The following example connects the server to 3 different networks:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
  networks:
  - filter:
      name: myNetwork
      tags: myTag
  - uuid: your_network_id
  - subnet_id: your_subnet_id

Subnet Filters

Rather than just using a network, you have the option of specifying a specific subnet to connect your server to. The following is an example of how to specify a specific subnet of a network to use for your server.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
  networks:
  - filter:
      name: <network-name>
    subnets:
    - filter:
       name: <subnet-name>

Ports

A server can also be connected to networks by describing what ports to create. Describing a server’s connection with ports allows for finer and more advanced configuration. For example, you can specify per-port security groups, fixed IPs, VNIC type or profile.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
  ports:
  - network:
      id: <your-network-id>
    nameSuffix: <your-port-name>
    description: <your-custom-port-description>
    vnicType: normal
    fixedIPs:
    - subnet:
        id: <your-subnet-id>
      ipAddress: <your-fixed-ip>
    - subnet:
        name: <your-subnet-name>
        tags:
          - tag1
          - tag2
    securityGroups:
    - <your-security-group-id>
    profile:
      capabilities:
        - <capability>

Any such ports are created in addition to ports used for connections to networks or subnets.

Also, port security can be applied to specific port to enable/disable the port security on that port; When not set, it takes the value of the corresponding field at the network level.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
  ports:
  - networkId: <your-network-id>
    ...
    disablePortSecurity: true
    ...

Security groups

Security groups are used to determine which ports of the cluster nodes are accessible from where.

If spec.managedSecurityGroups of OpenStackCluster is set to true, two security groups named k8s-cluster-${NAMESPACE}-${CLUSTER_NAME}-secgroup-controlplane and k8s-cluster-${NAMESPACE}-${CLUSTER_NAME}-secgroup-worker will be created and added to the control plane and worker nodes respectively.

By default, these groups have rules that allow the following traffic:

• Control plane nodes
  • API server traffic from anywhere
  • Etcd traffic from other control plane nodes
  • Kubelet traffic from other cluster nodes
  • Calico CNI traffic from other cluster nodes
• Worker nodes
  • Node port traffic from anywhere
  • Kubelet traffic from other cluster nodes
  • Calico CNI traffic from other cluster nodes

To use a CNI other than Calico, the flag OpenStackCluster.spec.allowAllInClusterTraffic can be set to true. With this flag set, the rules for the managed security groups permit all traffic between cluster nodes on all ports and protocols (API server and node port traffic is still permitted from anywhere, as with the default rules).

If this is not flexible enough, pre-existing security groups can be added to the spec of an OpenStackMachineTemplate, e.g.:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: ${CLUSTER_NAME}-control-plane
spec:
  template:
    spec:
      securityGroups:
      - name: allow-ssh

Tagging

You have the ability to tag all resources created by the cluster in the OpenStackCluster spec. Here is an example how to configure tagging:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackCluster
metadata:
  name: <cluster-name>
  namespace: <cluster-name>
spec:
  tags:
  - cluster-tag

To tag resources specific to a machine, add a value to the tags field in the OpenStackMachineTemplate spec like this:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
  tags:
  - machine-tag

Metadata

You also have the option to add metadata to instances. Here is a usage example:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
  serverMetadata:
    name: bob
    nickname: bobbert

Boot From Volume

For example in OpenStackMachineTemplate set spec.rootVolume.diskSize to something greater than 0 means boot from volume.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha6
kind: OpenStackMachineTemplate
metadata:
  name: <cluster-name>-controlplane
  namespace: <cluster-name>
spec:
...
  rootVolume:
    diskSize: <image size>
    volumeType: <a cinder volume type (*optional)>
    availabilityZone: <the cinder availability zone for the root volume (*optional)>
...

If volumeType is not specified, cinder will use the default volume type.

If availabilityZone is not specified, the volume will be created in the cinder availability zone specified in the MachineSpec’s failureDomain. This same value is also used as the nova availability zone when creating the server. Note that this will fail if cinder and nova do not have matching availability zones. In this case, cinder availabilityZone must be specified explicitly on rootVolume.

Timeout settings

The default timeout for instance creation is 5 minutes. If creating servers in your OpenStack takes a long time, you can increase the timeout. You can set a new value, in minutes, via the envorinment variable CLUSTER_API_OPENSTACK_INSTANCE_CREATE_TIMEOUT in your Cluster API Provider OpenStack controller deployment.

Custom pod network CIDR

If 192.168.0.0/16 is already in use within your network, you must select a different pod network CIDR. You have to replace the CIDR 192.168.0.0/16 with your own in the generated file.

Accessing nodes through the bastion host via SSH

Enabling the bastion host

To configure the Cluster API Provider for OpenStack to create a SSH bastion host, add this line to the OpenStackCluster spec after clusterctl generate cluster was successfully executed:

spec:
  ...
  bastion:
    enabled: true
    instance:
      flavor: <Flavor name>
      image:  <Image name>
      sshKeyName: <Key pair name>

All parameters are mutable during the runtime of the bastion host. The bastion host will be re-created if it’s enabled and the instance spec has been changed. This is done by a simple checksum validation of the instance spec which is stored in the OpenStackCluster annotation infrastructure.cluster.x-k8s.io/bastion-hash.

A floating IP is created and associated to the bastion host automatically, but you can add the IP address explicitly:

spec:
  ...
  bastion:
    ...
        floatingIP: <Floating IP address>

If managedSecurityGroups: true, security group rule opening 22/tcp is added to security groups for bastion, controller, and worker nodes respectively. Otherwise, you have to add securityGroups to the bastion in OpenStackCluster spec and OpenStackMachineTemplate spec template respectively.

Obtain floating IP address of the bastion node

Once the workload cluster is up and running after being configured for an SSH bastion host, you can use the kubectl get openstackcluster command to look up the floating IP address of the bastion host (make sure the kubectl context is set to the management cluster). The output will look something like this:

$ kubectl get openstackcluster
NAME    CLUSTER   READY   NETWORK                                SUBNET                                 BASTION
nonha   nonha     true    2e2a2fad-28c0-4159-8898-c0a2241a86a7   53cb77ab-86a6-4f2c-8d87-24f8411f15de   10.0.0.213

Topics

Table of Contents generated with DocToc

• External Cloud Provider
  • Steps

External Cloud Provider

To deploy a cluster using external cloud provider, create a cluster configuration with the external cloud provider template or refer to helm chart.

Steps of using external cloud provider template

• After control plane is up and running, retrieve the workload cluster Kubeconfig:

  clusterctl get kubeconfig ${CLUSTER_NAME} --namespace default > ./${CLUSTER_NAME}.kubeconfig
  

• Deploy a CNI solution (using Calico now)

  Note: choose desired version by replace <v3.23> below

  kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f https://docs.projectcalico.org/archive/v3.23/manifests/calico.yaml
  

• Create a secret containing the cloud configuration

  templates/create_cloud_conf.sh <path/to/clouds.yaml> <cloud> > /tmp/cloud.conf
  

  kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig create secret -n kube-system generic cloud-config --from-file=/tmp/cloud.conf
  

  rm /tmp/cloud.conf
  

• Create RBAC resources and openstack-cloud-controller-manager deamonset

  kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-roles.yaml
  kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml
  kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
  

• Waiting for all the pods in kube-system namespace up and running

  $ kubectl --kubeconfig=./${CLUSTER_NAME}.kubeconfig get pod -n kube-system
  NAME                                                    READY   STATUS    RESTARTS   AGE
  calico-kube-controllers-5569bdd565-ncrff                1/1     Running   0          20m
  calico-node-g5qqq                                       1/1     Running   0          20m
  calico-node-hdgxs                                       1/1     Running   0          20m
  coredns-864fccfb95-8qgp2                                1/1     Running   0          109m
  coredns-864fccfb95-b4zsf                                1/1     Running   0          109m
  etcd-mycluster-control-plane-cp2zw                      1/1     Running   0          108m
  kube-apiserver-mycluster-control-plane-cp2zw            1/1     Running   0          110m
  kube-controller-manager-mycluster-control-plane-cp2zw   1/1     Running   0          109m
  kube-proxy-mxkdp                                        1/1     Running   0          107m
  kube-proxy-rxltx                                        1/1     Running   0          109m
  kube-scheduler-mycluster-control-plane-cp2zw            1/1     Running   0          109m
  openstack-cloud-controller-manager-rbxkz                1/1     Running   8          18m
  

Table of Contents generated with DocToc

• Pre-condition
• Install OpenStack Cluster API provider into target cluster
• Move objects from bootstrap cluster into target cluster.
• Check cluster status

This documenation describes how to move Cluster API related objects from bootstrap cluster to target cluster. Check clusterctl move for further information.

Pre-condition

Bootstrap cluster

# kubectl get pods --all-namespaces
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-68cfd4c5b8-mjq75       2/2     Running   0          27m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-848575ccb7-m672j   2/2     Running   0          27m
capi-system                         capi-controller-manager-564d97d59c-2t7sl                         2/2     Running   0          27m
capi-webhook-system                 capi-controller-manager-9c8b5d6d4-49czx                          2/2     Running   0          28m
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-7dff4b8c7-8w9sq        2/2     Running   1          27m
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-74c99998d-bftbn    2/2     Running   0          27m
capi-webhook-system                 capo-controller-manager-7d7bfc856b-5ttw6                         2/2     Running   0          24m
capo-system                         capo-controller-manager-5fb48fcb4c-ttkpv                         2/2     Running   0          25m
cert-manager                        cert-manager-544d659678-l9pjb                                    1/1     Running   0          29m
cert-manager                        cert-manager-cainjector-64c9f978d7-bjxkg                         1/1     Running   0          29m
cert-manager                        cert-manager-webhook-5855bb8c8c-8hb9w                            1/1     Running   0          29m
kube-system                         coredns-66bff467f8-ggn54                                         1/1     Running   0          40m
kube-system                         coredns-66bff467f8-t4bqr                                         1/1     Running   0          40m
kube-system                         etcd-kind-control-plane                                          1/1     Running   1          40m
kube-system                         kindnet-ng2gf                                                    1/1     Running   0          40m
kube-system                         kube-apiserver-kind-control-plane                                1/1     Running   1          40m
kube-system                         kube-controller-manager-kind-control-plane                       1/1     Running   1          40m
kube-system                         kube-proxy-h6rmz                                                 1/1     Running   0          40m
kube-system                         kube-scheduler-kind-control-plane                                1/1     Running   1          40m
local-path-storage                  local-path-provisioner-bd4bb6b75-ft7wh                           1/1     Running   0          40m

Target cluster (Below is an example of external cloud provider)

# kubectl get pods --kubeconfig capi-openstack-3.kubeconfig --all-namespaces
NAMESPACE     NAME                                                         READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-59b699859f-djqd7                     1/1     Running   0          6m2s
kube-system   calico-node-szp44                                            1/1     Running   0          6m2s
kube-system   calico-node-xhgzr                                            1/1     Running   0          6m2s
kube-system   coredns-6955765f44-wk2vq                                     1/1     Running   0          21m
kube-system   coredns-6955765f44-zhps9                                     1/1     Running   0          21m
kube-system   etcd-capi-openstack-control-plane-82xck                      1/1     Running   0          22m
kube-system   kube-apiserver-capi-openstack-control-plane-82xck            1/1     Running   0          22m
kube-system   kube-controller-manager-capi-openstack-control-plane-82xck   1/1     Running   2          22m
kube-system   kube-proxy-4f9k8                                             1/1     Running   0          21m
kube-system   kube-proxy-gjd55                                             1/1     Running   0          21m
kube-system   kube-scheduler-capi-openstack-control-plane-82xck            1/1     Running   2          22m
kube-system   openstack-cloud-controller-manager-z9jtc                     1/1     Running   1          4m9s

Install OpenStack Cluster API provider into target cluster

You need install OpenStack cluster api providers into target cluster first.

# clusterctl --kubeconfig capi-openstack-3.kubeconfig  init --infrastructure openstack
Fetching providers
Installing cert-manager
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.8" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.8" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.8" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-openstack" Version="v0.3.1" TargetNamespace="capo-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

Move objects from bootstrap cluster into target cluster.

CRD, objects such as OpenStackCluster, OpenStackMachine etc need to be moved.

# clusterctl move --to-kubeconfig capi-openstack-3.kubeconfig -v 10
No default config file available
Performing move...
Discovering Cluster API objects
Cluster Count=1
KubeadmConfig Count=2
KubeadmConfigTemplate Count=1
KubeadmControlPlane Count=1
MachineDeployment Count=1
Machine Count=2
MachineSet Count=1
OpenStackCluster Count=1
OpenStackMachine Count=2
OpenStackMachineTemplate Count=2
Secret Count=9
Total objects Count=23
Moving Cluster API objects Clusters=1
Pausing the source cluster
Set Cluster.Spec.Paused Paused=true Cluster="capi-openstack-3" Namespace="default"
Creating target namespaces, if missing
Creating objects in the target cluster
Creating Cluster="capi-openstack-3" Namespace="default"
Creating OpenStackMachineTemplate="capi-openstack-control-plane" Namespace="default"
Creating OpenStackMachineTemplate="capi-openstack-md-0" Namespace="default"
Creating OpenStackCluster="capi-openstack-3" Namespace="default"
Creating KubeadmConfigTemplate="capi-openstack-md-0" Namespace="default"
Creating MachineDeployment="capi-openstack-md-0" Namespace="default"
Creating KubeadmControlPlane="capi-openstack-control-plane" Namespace="default"
Creating Secret="capi-openstack-3-etcd" Namespace="default"
Creating Machine="capi-openstack-control-plane-n2kdq" Namespace="default"
Creating Secret="capi-openstack-3-sa" Namespace="default"
Creating Secret="capi-openstack-3-kubeconfig" Namespace="default"
Creating Secret="capi-openstack-3-proxy" Namespace="default"
Creating MachineSet="capi-openstack-md-0-dfdf94979" Namespace="default"
Creating Secret="capi-openstack-3-ca" Namespace="default"
Creating Machine="capi-openstack-md-0-dfdf94979-656zq" Namespace="default"
Creating KubeadmConfig="capi-openstack-control-plane-xzj7x" Namespace="default"
Creating OpenStackMachine="capi-openstack-control-plane-82xck" Namespace="default"
Creating Secret="capi-openstack-control-plane-xzj7x" Namespace="default"
Creating OpenStackMachine="capi-openstack-md-0-bkwhh" Namespace="default"
Creating KubeadmConfig="capi-openstack-md-0-t44gj" Namespace="default"
Creating Secret="capi-openstack-md-0-t44gj" Namespace="default"
Deleting objects from the source cluster
Deleting Secret="capi-openstack-md-0-t44gj" Namespace="default"
Deleting Secret="capi-openstack-control-plane-xzj7x" Namespace="default"
Deleting OpenStackMachine="capi-openstack-md-0-bkwhh" Namespace="default"
Deleting KubeadmConfig="capi-openstack-md-0-t44gj" Namespace="default"
Deleting Machine="capi-openstack-md-0-dfdf94979-656zq" Namespace="default"
Deleting KubeadmConfig="capi-openstack-control-plane-xzj7x" Namespace="default"
Deleting OpenStackMachine="capi-openstack-control-plane-82xck" Namespace="default"
Deleting Secret="capi-openstack-3-etcd" Namespace="default"
Deleting Machine="capi-openstack-control-plane-n2kdq" Namespace="default"
Deleting Secret="capi-openstack-3-sa" Namespace="default"
Deleting Secret="capi-openstack-3-kubeconfig" Namespace="default"
Deleting Secret="capi-openstack-3-proxy" Namespace="default"
Deleting MachineSet="capi-openstack-md-0-dfdf94979" Namespace="default"
Deleting Secret="capi-openstack-3-ca" Namespace="default"
Deleting OpenStackMachineTemplate="capi-openstack-control-plane" Namespace="default"
Deleting OpenStackMachineTemplate="capi-openstack-md-0" Namespace="default"
Deleting OpenStackCluster="capi-openstack-3" Namespace="default"
Deleting KubeadmConfigTemplate="capi-openstack-md-0" Namespace="default"
Deleting MachineDeployment="capi-openstack-md-0" Namespace="default"
Deleting KubeadmControlPlane="capi-openstack-control-plane" Namespace="default"
Deleting Cluster="capi-openstack-3" Namespace="default"
Resuming the target cluster
Set Cluster.Spec.Paused Paused=false Cluster="capi-openstack-3" Namespace="default"

Check cluster status

# kubectl get openstackcluster --kubeconfig capi-openstack-3.kubeconfig --all-namespaces
NAMESPACE   NAME               CLUSTER            READY   NETWORK                                SUBNET
default     capi-openstack-3   capi-openstack-3   true    4a6f2d57-bb3d-44f4-a28a-4c94a92e41d0   1a1a1d9d-5258-42cb-8756-fa4c648af72b

# kubectl get openstackmachines --kubeconfig capi-openstack-3.kubeconfig --all-namespaces
NAMESPACE   NAME                                 CLUSTER            STATE    READY   INSTANCEID                                         MACHINE
default     capi-openstack-control-plane-82xck   capi-openstack-3   ACTIVE   true    openstack:///f29007c5-f672-4214-a508-b7cf4a17b3ed   capi-openstack-control-plane-n2kdq
default     capi-openstack-md-0-bkwhh            capi-openstack-3   ACTIVE   true    openstack:///6e23324d-315a-4d75-85a9-350fd1705ab6   capi-openstack-md-0-dfdf94979-656zq

Table of Contents generated with DocToc

• Troubleshooting
  • Get logs of Cluster API controller containers
  • Master failed to start with error: node xxxx not found
  • providerClient authentication err
  • Fails in creating floating IP during cluster creation.

Troubleshooting

This guide (based on Minikube but others should be similar) explains general info on how to debug issues if a cluster creation fails.

Get logs of Cluster API controller containers

kubectl --kubeconfig minikube.kubeconfig -n capo-system logs -l control-plane=capo-controller-manager -c manager

Similarly, the logs of the other controllers in the namespaces capi-system and cabpk-system can be retrieved.

Master failed to start with error: node xxxx not found

Sometimes the master machine is created but fails to startup, take Ubuntu as example, open /var/log/messages and if you see something like this:

Jul 10 00:07:58 openstack-master-5wgrw kubelet: E0710 00:07:58.444950 4340 kubelet.go:2248] node "openstack-master-5wgrw" not found
Jul 10 00:07:58 openstack-master-5wgrw kubelet: I0710 00:07:58.526091 4340 kubelet_node_status.go:72] Attempting to register node openstack-master-5wgrw
Jul 10 00:07:58 openstack-master-5wgrw kubelet: E0710 00:07:58.527398 4340 kubelet_node_status.go:94] Unable to register node "openstack-master-5wgrw" with API server: nodes "openstack-master-5wgrw" is forbidden: node "openstack-master-5wgrw.novalocal" is not allowed to modify node "openstack-master-5wgrw"

This might be caused by This issue, try the method proposed there.

providerClient authentication err

If you are using https, you must specify the CA certificate in your clouds.yaml file, and when you encounter issue like:

kubectl --kubeconfig minikube.kubeconfig logs -n capo-system logs -l control-plane=capo-controller-manager
...
E0814 04:32:52.688514       1 machine_controller.go:204] Failed to check if machine "openstack-master-hxk9r" exists: providerClient authentication err: Post https://xxxxxxxxxxxxxxx:5000/v3/auth/tokens: x509: certificate signed by unknown authority
...

you can also add verify: false into clouds.yaml file to solve the problem.

clouds:
  openstack:
    auth:
        ....
    region_name: "RegionOne"
    interface: "public"
    identity_api_version: 3
    cacert: /etc/certs/cacert
    verify: false

Fails in creating floating IP during cluster creation.

If you encounter rule:create_floatingip and rule:create_floatingip:floating_ip_address is disallowed by policy when create floating ip, check with your openstack administrator, you need to be authorized to perform those actions, see issue 572 for more detailed information.

Refer to rule:create_floatingip and rule:create_floatingip:floating_ip_address for further policy information.

An alternative is to create the floating IP before create the cluster and use it.

Table of Contents generated with DocToc

• CRD Changes
  • Conversions

CRD Changes

Conversions

CAPO is able to automatically convert your old resources into new API versions.

Table of Contents generated with DocToc

• v1alpha4 compared to v1alpha5
  • Migration
  • API Changes
    • OpenStackCluster
      • Managed API LoadBalancer
    • OpenStackMachine
      • Major Changes to Ports and Networks
      • Rename of status.error{Reason,Message} to status.failure{Reason,Message}
      • Changes to rootVolume

v1alpha4 compared to v1alpha5

Migration

All users are encouraged to migrate their usage of the CAPO CRDs from older versions to v1alpha5. This includes yaml files and source code. As CAPO implements automatic conversions between the CRD versions, this migration can happen after installing the new CAPO release.

API Changes

This only documents backwards incompatible changes. Fields that were added to v1alpha5 are not listed here.

OpenStackCluster

Managed API LoadBalancer

The fields related to the managed API LoadBalancer were moved into a seperate object:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: OpenStackCluster
spec:
  managedAPIServerLoadBalancer: true
  apiServerLoadBalancerAdditionalPorts: [443]

becomes:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha5
kind: OpenStackCluster
spec:
  apiServerLoadBalancer:
    enabled: true
    additionalPorts: [443]

OpenStackMachine

Major Changes to Ports and Networks

When using Ports it is now possible to specify network and subnet by filter instead of just ID. As a consequence, the relevant ID fields are now moved into the new filter specifications:

ports:
  - networkId: d-e-a-d
    fixedIPs:
      - subnetId: b-e-e-f

becomes:

ports:
  - network:
      id: d-e-a-d
    fixedIPs:
      subnet:
        id: b-e-e-f

Networks are now deprecated. With one exception, all functionality of Networks is now available for Ports. Consequently, Networks will be removed from the API in a future release.

The ability of a Network to add multiple ports with a single directive will not be added to Ports. When moving to Ports, all ports must be added explicitly. Specifically, when evaluating the network or subnet filter of a Network, if there are multiple matches we will add all of these to the server. By contrast we raise an error if the network or subnet filter of a Port does not return exactly one result.

tenantId was previously a synonym for projectId in both network and subnet filters. This has now been removed. Use projectId instead.

The following fields are removed from network and subnet filters without replacement:

• status
• adminStateUp
• shared
• marker
• limit
• sortKey
• sortDir
• subnetPoolId

Rename of status.error{Reason,Message} to status.failure{Reason,Message}

The actual fields were previously already renamed, but we still used the error prefix in JSON. This was done to align with CAPI, where these fields were renamed in v1alpha3.

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: OpenStackMachine
status:
  errorReason: UpdateError
  errorMessage: Something when wrong

becomes:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha5
kind: OpenStackMachine
status:
  failureReason: UpdateError
  failureMessage: Something when wrong

Changes to rootVolume

The following fields were removed without replacement:

• rootVolume.deviceType
• rootVolume.sourceType

Additionally, rootVolume.sourceUUID has been replaced by using ImageUUID or Image from the OpenStackMachine as appropriate.

Table of Contents generated with DocToc

• v1alpha5 compared to v1alpha6
  • Migration
  • API Changes
    • OpenStackCluster
    • OpenStackMachine

v1alpha5 compared to v1alpha6

⚠️ v1alpha6 has not been released yet.

Migration

All users are encouraged to migrate their usage of the CAPO CRDs from older versions to v1alpha6. This includes yaml files and source code. As CAPO implements automatic conversions between the CRD versions, this migration can happen after installing the new CAPO release.

API Changes

This only documents backwards incompatible changes. Fields that were added to v1alpha6 are not listed here.

OpenStackCluster

OpenStackMachine

Table of Contents generated with DocToc

• Development Guide
  • Using your own capi-openstack controller image for testing cluster creation or deletion
    • Building and upload your own capi-openstack controller image
    • Using your own capi-openstack controller image
  • Developing with Tilt
  • Running E2E tests locally
    • Support for clouds using SSL
    • Support for clouds with multiple external networks
    • OpenStack prerequisites
  • Running E2E tests using rootless podman
    • Host configuration
    • Running podman system service to emulate docker daemon

Development Guide

This document explains how to develop Cluster API Provider OpenStack.

Using your own capi-openstack controller image for testing cluster creation or deletion

You need to create your own openstack-capi controller image for testing cluster creation or deletion by your code. The image is stored in the docker registry. You need to create an account of Docker registry in advance.

Building and upload your own capi-openstack controller image

Log in to your registry account. Export the following environment variables which will be used by the Makefile.

Execute the command to build and upload the image to the Docker registry.

make docker-build docker-push

Using your own capi-openstack controller image

After generating infrastructure-components.yaml, replace the us.gcr.io/k8s-artifacts-prod/capi-openstack/capi-openstack-controller:v0.3.4 with your image.

Developing with Tilt

We have support for using Tilt for rapid iterative development. Please visit the Cluster API documentation on Tilt for information on how to set up your development environment.

Running E2E tests locally

You can run the E2E tests locally with:

make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=mycloud

where mycloud is an entry in clouds.yaml.

The E2E tests:

• Build a CAPO image from the local working directory
• Create a kind cluster locally
• Deploy downloaded CAPI, and locally-build CAPO to kind
• Create an e2e namespace per-test on the kind cluster
• Deploy cluster templates to the test namespace
• Create test clusters on the target OpenStack

Support for clouds using SSL

If your cloud requires a cacert you must also pass this to make via OPENSTACK_CLOUD_CACERT_B64, i.e.:

make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=my_cloud \
              OPENSTACK_CLOUD_CACERT_B64=$(base64 -w0 /path/to/mycloud-ca.crt)

CAPO deployed in the local kind cluster will automatically pick up a cacert defined in your clouds.yaml so you will see servers created in OpenStack without specifying OPENSTACK_CLOUD_CACERT_B64. However, the cacert won’t be deployed to those servers, so kubelet will fail to start.

Support for clouds with multiple external networks

If your cloud contains only a single external network CAPO will automatically select that network for use by a deployed cluster. However, if there are multiple external networks CAPO will log an error and fail to create any machines. In this case you must pass the id of an external network to use explicitly with OPENSTACK_EXTERNAL_NETWORK_ID, i.e.:

make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=my_cloud \
              OPENSTACK_EXTERNAL_NETWORK_ID=27635f93-583d-454e-9c6d-3d305e7f8a22

OPENSTACK_EXTERNAL_NETWORK_ID must be specified as a uuid. Specifying by name is not supported.

You can list available external networks with:

$ openstack network list --external
+--------------------------------------+----------+--------------------------------------+
| ID                                   | Name     | Subnets                              |
+--------------------------------------+----------+--------------------------------------+
| 27635f93-583d-454e-9c6d-3d305e7f8a22 | external | be64cd07-f8b7-4705-8446-26b19eab3914 |
| cf2e83dc-545d-490f-9f9c-4e90927546f2 | hostonly | ec95befe-72f4-4af6-a263-2aec081f47d3 |
+--------------------------------------+----------+--------------------------------------+

OpenStack prerequisites

The file test/e2e/data/e2e_conf.yaml and the test templates under test/e2e/data/infrastructure-openstack reference several OpenStack resources which must exist before running the test:

• System requirements
  • Multiple nodes
  • controller: 16 CPUs / 64 GB RAM
  • worker: 8 CPUs / 32 GB RAM
• Availability zones (for multi-AZ tests)
  • testaz1: used by all test cases
  • testaz2: used by multi-az test case
• Services (Additional services to be enabled)
  • Octavia
  • Network trunking (neutron-trunk)
  • see Configration for more details.
• Glance images
  • cirros-0.5.1-x86_64-disk
    • Download from https://docs.openstack.org/image-guide/obtain-images.html
  • ubuntu-2004-kube-v1.18.15
    • Download from https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/ubuntu/2021-03-27/ubuntu-2004-kube-v1.18.15.qcow2
    • Or generate using the images/capi directory from https://github.com/kubernetes-sigs/image-builder
      • Boot volume size must be less than 15GB
• Flavors
  • m1.medium: used by control plane
  • m1.small: used by workers
  • m1.tiny: used by bastion
• clouds.yaml
  • capo-e2e: for general user authorization
  • capo-e2e-admin: for administrator user authorization
  • i.e.:

    clouds:
      capo-e2e:
        auth:
          auth_url: http://Node-Address/identity
          project_name: demo
          project_domain_name: Default
          user_domain_name: Default
          username: demo
          password: secret
        region_name: RegionOne
    
      capo-e2e-admin:
        auth:
          auth_url: http://Node-Address/identity
          project_name: demo
          project_domain_name: Default
          user_domain_name: Default
          username: admin
          password: secret
        region_name: RegionOne
    

You can also use Hacking CI scripts to automatically create OpenStack environment.

Running E2E tests using rootless podman

You can use unprivileged podman to:

• Build the CAPO image
• Deploy the kind cluster

To do this you need to configure the host appropriately and pass PODMAN=1 to make, i.e.:

make test-e2e OPENSTACK_CLOUD_YAML_FILE=/path/to/clouds.yaml OPENSTACK_CLOUD=my_cloud \
              PODMAN=1

Host configuration

Firstly, you must be using kernel >=5.11. If you are using Fedora, this means Fedora >= 34.

You must configure systemd and iptables as described in https://kind.sigs.k8s.io/docs/user/rootless/. There is no need to configure cgroups v2 on Fedora, as it uses this by default.

You must install the podman-docker package to emulate the docker cli tool. However, this is not sufficient on its own as described below.

Running podman system service to emulate docker daemon

While kind itself supports podman, the cluster-api test framework does not. This framework is used by the CAPO tests to push test images into the kind cluster. Unfortunately the cluster-api test framework explicitly connects to a running docker daemon, so cli emulation is not sufficient for compatibility. This issue is tracked in https://github.com/kubernetes-sigs/cluster-api/issues/5146, and the following workaround can be ignored when this is resolved.

podman includes a ‘system service’ which emulates docker. For the tests to work, this service must be running and listening on a unix socket at /var/run/docker.sock. You can achieve this with:

$ podman system service -t 0 &
$ sudo rm /var/run/docker.sock
$ sudo ln -s /run/user/$(id -u)/podman/podman.sock /var/run/docker.sock

Table of Contents generated with DocToc

• Hacking CI for the E2E tests
  • Prow
  • DevStack
    • Configuration
    • Build order
    • Networking
    • Availability zones
  • Connecting to DevStack

Hacking CI for the E2E tests

Prow

CAPO tests are executed by Prow. They are defined in the Kubernetes test-infra repository. The E2E tests run as a presubmit. They run in a docker container in Prow infrastructure which contains a checkout of the CAPO tree under test. The entry point for tests is scripts/ci-e2e.sh, which is defined in the job in Prow.

DevStack

The E2E tests require an OpenStack cloud to run against, which we provision during the test with DevStack. The project has access to capacity on GCP, so we provision DevStack on 2 GCP instances.

The entry point for the creation of the test DevStack is hack/ci/create_devstack.sh, which is executed by scripts/ci-e2e.sh. We create 2 instances: controller and worker. Each will provision itself via cloud-init using config defined in hack/ci/cloud-init.

DevStack OS

In GCE, DevStack is installed on a community-maintained Ubuntu 20.04 LTS cloud image. The cloud-init config is also intended to work on CentOS 8, and this is known to work as of 2021-01-12. However, note that this is not regularly tested. See the comment in hack/ci/gce-project.sh for how to deploy on CentOS.

It is convenient to the project to have a viable second OS option as it gives us an option to work around issues which only affect one or the other. This is most likely when enabling new DevStack features, but may also include infrastructure issues. Consequently, when making changes to cloud-init, try not to use features specific to Ubuntu or CentOS. DevStack already supports both operating systems, so we just need to be careful in our periferal configuration, for example by using cloud-init’s packages module rather than manually invoking apt-get or yum. Fortunately package names tend to be consistent across the two distributions.

Configuration

We configure a 2 node DevStack. controller is running:

• All control plane services
• Nova: all services, including compute
• Glance: all services
• Octavia: all services
• Neutron: all services with ML2/OVS, including L3 agent
• Cinder: all services, including volume with default LVM/iSCSI backend

worker is running:

• Nova: compute only
• Neutron: agent only (not L3 agent)
• Cinder: volume only with default LVM/iSCSI backend

controller is using the n2-standard-16 machine type with 16 vCPUs and 64 GB RAM. worker is using the n2-standard-8 machine type with 8 vCPUs and 32 GB RAM. Each job has a quota limit of 24 vCPUs.

Build order

We build controller first, and then worker. We let worker build asynchronously because tests which don’t require a second AZ can run without it while it builds. A systemd job defined in the cloud-init of controller polls for worker coming up and automatically configures it.

Networking

Both instances share a common network which uses the CIDR defined in PRIVATE_NETORK_CIDR in hack/ci/create_devstack.sh. Each instance has a single IP on this network:

• controller: 10.0.3.15
• worker: 10.0.3.16

In addition, DevStack will create a floating IP network using CIDR defined in FLOATING_RANGE in hack/ci/create_devstack.sh. As the neutron L3 agent is only running on the controller, all of this traffic is handled on the controller, even if the source is an instance running on the worker. The controller creates iptables rules to NAT this traffic.

The effect of this is that instances created on either controller or worker can get a floating ip from the public network. Traffic using this floating IP will be routed via controller and externally via NAT.

Availability zones

We are running nova compute and cinder volume on each of controller and worker. Each nova compute and cinder volume are configured to be in their own availability zone. The names of the availability zones are defined in OPENSTACK_FAILURE_DOMAIN and OPENSTACK_FAILURE_DOMAIN_ALT in test/e2e/data/e2e_conf.yaml, with the services running on controller being in OPENSTACK_FAILURE_DOMAIN and the services running on worker being in OPENSTACK_FAILURE_DOMAIN_ALT.

This configuration is intended only to allow the testing of functionality related to availability zones, and does not imply any robustness to failure.

Nova is configured (via [DEFAULT]/default_schedule_zone) to place all workloads on the controller unless they have an explicit availability zone. The intention is that controller should have the capacity to run all tests which are agnostic to availability zones. This means that the explicitly multi-az tests do not risk failure due to capacity issues.

However, this is not sufficient because by default CAPI explicitly schedules the control plane across all discovered availability zones. Consequently we explicitly confine all clusters to OPENSTACK_FAILURE_DOMAIN (controller) in the test cluster definitions in test/e2e/data/infrastructure-openstack.

Connecting to DevStack

The E2E tests running in Prow create a kind cluster. This also running in Prow using Docker in Docker. The E2E tests configure this cluster with clusterctl, which is where CAPO executes.

create_devstack.sh wrote a clouds.yaml to the working directory, which is passed to CAPO via the cluster definitions in test/e2e/data/infrastructure-openstack. This clouds.yaml references the public, routable IP of controller. However, DevStack created all the service endpoints using controller‘s private IP, which is not publicly routable. In addition, the tests need to be able to SSH to the floating IP of the Bastion. This floating IP is also allocated from a range which is not publicly routable.

To allow this access we run sshuttle from create_devstack.sh. This creates an SSH tunnel and routes traffic for PRIVATE_NETWORK_CIDR and FLOATING_RANGE over it.

Note that the semantics of a sshuttle tunnel are problematic. While they happen to work currently for DinD, Podman runs the kind cluster in a separate network namespace. This means that kind running in podman cannot route over sshuttle running outside the kind cluster. This may also break in future versions of Docker.