Oxide Cloud Controller Manager

The Oxide Cloud Controller Manager is a Kubernetes control plane component that embeds Oxide specific control logic, allowing Kubernetes clusters running on Oxide to integrate with the Oxide API via the Cloud Controller Manager architecture.

A cloud controller manager is free to embed any cloud-specific control logic it needs. However, cloud controller manager implementations generally embed the following control logic by implementing the cloudprovider.Interface.

• Node Controller: Manages Node resources based on the information returned from the cloud provider API (e.g., labels, addresses, node health).

• Route Controller: Configures routes in the cloud provider so pods running on different Kubernetes nodes can communicate with one another.

• Service Controller: Ensures cloud provider infrastructure (e.g., load balancer, IP addresses) exists for a Service of type LoadBalancer.

The Oxide Cloud Controller Manager implements the following Oxide specific control logic.

• Node Controller

Usage

Please note the following before using the Oxide Cloud Controller Manager.

• The cloud controller manager can only manage a single Kubernetes cluster with all its nodes running in the same Oxide silo and project. This may be expanded in the future.

• The kubelet, kube-apiserver, and kube-controller-manager must be run with --cloud-provider=external to configure the Kubernetes cluster to use a cloud controller manager. This process differs depending on your Kubernetes distribution of choice.

• Nodes joining a Kubernetes cluster configured to use a cloud controller manager will have a taint node.cloudprovider.kubernetes.io/uninitialized with effect NoSchedule. This taint will be removed by the node controller within the cloud controller manager.

With the above noted, let’s run the Oxide Cloud Controller Manager in your Kubernetes cluster.

Helm Chart

Create a Secret to hold the Oxide credentials. The secret name must match the Helm release’s full name, which defaults to <RELEASE_NAME>-oxide-cloud-controller-manager.

kubectl create secret generic <RELEASE_NAME>-oxide-cloud-controller-manager \
  --namespace kube-system \
  --from-literal=oxide-host=https://oxide.sys.example.com \
  --from-literal=oxide-token=oxide-token-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
  --from-literal=oxide-project=example

Install the Helm chart.

helm install <RELEASE_NAME> \
  oci://ghcr.io/oxidecomputer/charts/oxide-cloud-controller-manager \
  --version X.Y.Z \
  --namespace kube-system \
  --create-namespace

Kubernetes Manifest

Create the following Secret to hold the Oxide credentials.

---
apiVersion: v1
kind: Secret
metadata:
  name: oxide-cloud-controller-manager
  namespace: kube-system
type: Opaque
stringData:
  oxide-host: "https://oxide.sys.example.com"
  oxide-token: "oxide-token-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
  oxide-project: "example"

Apply the oxide-cloud-controller-manager.yaml Kubernetes manifest. This manifest is generated from the Helm chart in charts/oxide-cloud-controller-manager.

kubectl apply -f oxide-cloud-controller-manager.yaml

Development

The Makefile is the primary method of interfacing with this project. Refer to its targets for more information. The build artifact is a container image to be run either inside or outside the Kubernetes cluster it’s meant to manage.

Running Locally

Build the container image.

make build

Determine if you want to run the cloud controller manager inside or outside the Kubernetes cluster it’s meant to manage.

To run the cloud controller manager inside the Kubernetes cluster, refer to Usage.

To run the cloud controller manager outside the Kubernetes cluster, run the container image with a kubeconfig for the cluster you want to manage.

podman run \
	--env OXIDE_HOST \
	--env OXIDE_TOKEN \
	--env OXIDE_PROJECT \
	--volume ./kubeconfig.yaml:/tmp/kubeconfig.yaml:ro \
	ghcr.io/oxidecomputer/oxide-cloud-controller-manager:TAG \
	--cloud-provider oxide \
	--kubeconfig /tmp/kubeconfig.yaml

Release Process

The release process is manual and runs from a developer’s workstation for now.

1. Check out the revision to be released. Ensure the working copy is clean.

2. Build the Helm chart.

   make helm-package

3. Push the Helm chart.

   make helm-push

4. Build the container image.

   RELEASE=true make build

5. Push the container image.

   RELEASE=true make push

6. Create a GitHub release.

   1. Create and push a Git tag with the v-prefixed VERSION (e.g., vX.Y.Z).

   2. Create the GitHub release for the newly pushed tag. Automatically generate the release notes.

7. Open a pull request with the following changes.

   1. Update the VERSION within the Makefile to the next version.

   2. Run make manifest to generate a new manifests/oxide-cloud-controller-manager.yaml.