Write Controllers on Top of Metal-Stack by Using Kubebuilder - II

This is the second part of the series Write Controllers on Top of Metal-Stack by Using Kubebuilder. Make sure you went through the first part.

kubebuilder markers for CRD

kubebuilder provides lots of handful markers. Here are some examples:

API Resource Type
```
// +kubebuilder:object:root=true
```
The go struct under this marker will be an API resource type in the url. For example, the url path to XCluster instance myxcluster would be
```
/apis/cluster.www.x-cellent.com/v1/namespaces/myns/xclusters/myxcluster
```
API Subresource
```
// +kubebuilder:subresource:status
```
The go struct under this marker contains API subresource status. For the last example, the url path to the status of the instance would be:
```
/apis/cluster.www.x-cellent.com/v1/namespaces/myns/xclusters/myxcluster/status
```
Terminal Output
```
// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.ready`
```
This specifies an extra column of output on terminal when you do kubectl get.

Wire up metal-api client metalgo.Driver

metalgo.Driver is the client in go code for talking to metal-api. To enable both controllers of XCluster and XFirewall to do that, we created a metalgo.Driver named metalClient and set field Driver of both controllers as shown in the following snippet from main.go.

	if err = (&controllers.XClusterReconciler{
		Client: mgr.GetClient(),
		Driver: metalClient,
		Log:    ctrl.Log.WithName("controllers").WithName("XCluster"),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "XCluster")
		os.Exit(1)
	}

Role-based access control (RBAC)

With the following lines in xcluster_controller.go and the equivalent lines in xfirewall_controller.go (in our case overlapped), kubebuilder generates role.yaml and wire up everything for your xcluster-controller-manager pod when you do make deploy. The verbs are the actions your pod is allowed to perform on the resources, which are xclusters and xfirewalls in our case.

// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xclusters/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xfirewalls,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xfirewalls/status,verbs=get;update;patch

Finalizer

When you want to do some clean-up before the kubernetes api-server deletes your resource in no time upon kubectl delete, finalizers come in handy. A finalizer is simply a string constant stored in field finalizers of a Kubernetes resource instance's metadata. For example, the finalizer of XCluster in xcluster_types.go:

const XClusterFinalizer = "xcluster.finalizers.cluster.www.x-cellent.com"

The api-server will not delete the instance before its finalizers are all removed from the resource instance. For example, in xcluster_controller.go we add the above finalizer to the XCluster instance, so later when the instance is about to be deleted, the api-server can't delete the instance before we've freed the metal-stack network and then removed the finalizer from the instance. We can see that in action in the following listing. We use the Driver mentioned earlier to ask metal-api if the metal-stack network we allocated is still there. If so, we use the Driver to free it and then remove the finalizer of XCluster.

	resp, err := r.Driver.NetworkFind(&metalgo.NetworkFindRequest{
		ID:        &cl.Spec.PrivateNetworkID,
		Name:      &cl.Spec.Partition,
		ProjectID: &cl.Spec.ProjectID,
	})

	if err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to list metal-stack networks: %w", err)
	}

	if len := len(resp.Networks); len > 1 {
		return ctrl.Result{}, fmt.Errorf("more than one network listed: %w", err)
	} else if len == 1 {
		if _, err := r.Driver.NetworkFree(cl.Spec.PrivateNetworkID); err != nil {
			return ctrl.Result{Requeue: true}, nil
		}
	}
	log.Info("metal-stack network freed")

	cl.RemoveFinalizer(clusterv1.XFirewallFinalizer)
	if err := r.Update(ctx, cl); err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to remove xcluster finalizer: %w", err)
	}
	r.Log.Info("finalizer removed")

Likewise, in xfirewall_controller.go we add the finalizer to XFirewall instance. The api-server can't delete the instance before we clean up the underlying metal-stack firewall (r.Driver.MachineDelete(fw.Spec.MachineID) in the following listing) and then remove the finalizer from the instance:

func (r *XFirewallReconciler) DeleteFirewall(ctx context.Context, fw *clusterv1.XFirewall, log logr.Logger) (ctrl.Result, error) {
	if _, err := r.Driver.MachineDelete(fw.Spec.MachineID); err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to delete firewall: %w", err)
	}
	log.Info("states of the machine managed by XFirewall reset")

	fw.RemoveFinalizer(clusterv1.XFirewallFinalizer)
	if err := r.Update(ctx, fw); err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to remove XFirewall finalizer: %w", err)
	}
	r.Log.Info("finalizer removed")

	return ctrl.Result{}, nil
}

func errors.IsNotFound and client.IgnoreNotFound

When you have different handlers depending on whether the error is the instance not found, you can consider using errors.IsNotFound(err) as follows from xcluster_controller.go:

	fw := &clusterv1.XFirewall{}
	if err := r.Get(ctx, req.NamespacedName, fw); err != nil {
		// errors other than `NotFound`
		if !errors.IsNotFound(err) {
			return ctrl.Result{}, fmt.Errorf("failed to fetch XFirewall instance: %w", err)
		}

		// Create XFirewall instance
		fw = cl.ToXFirewall()

If we can do nothing against the error the instance not found, we might simply stop the reconciliation without requeueing the request as follows:

	cl := &clusterv1.XCluster{}
	if err := r.Get(ctx, req.NamespacedName, cl); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

Exponential Back-Off

As far as requeue is concerned, returning ctrl.Result{}, err and ctrl.Result{Requeue: true}, nil are the same as shown in this if clause and this else if clause in the source code. Moreover, exponential back-off can be observed in the source code where dependencies of a controller are set and where func workqueue.DefaultControllerRateLimiter is defined.

ControllerReference

ControllerReference is a kind of OwnerReference that enables the garbage collection of the owned instance (XFirewall) when the owner instance (XCluster) is deleted. We demonstrate that in xcluster_controller.go by using the function SetControllerReference.

		if err := controllerutil.SetControllerReference(cl, fw, r.Scheme); err != nil {
			return ctrl.Result{}, fmt.Errorf("failed to set the owner reference of the XFirewall: %w", err)
		}

Since XCluster owns XFirewall instance, we have to inform the manager that it should reconcile XCluster upon any change of an XFirewall instance:

func (r *XClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&clusterv1.XCluster{}).
		Owns(&clusterv1.XFirewall{}).
		Complete(r)
}

Wrap-up

Check out the code in this project for more details. If you want a fully-fledged implementation, stay tuned! Our cluster-api-provider-metalstack is on the way. If you want more blog posts about metal-stack and kubebuilder, let us know! Special thanks go to Grigoriy Mikhalkin.

← x-cellent technologies GmbH Blog