Write Controllers on Top of Metal-Stack by Using Kubebuilder - II

06.04.2021

Fensterer

Geschäftsführer

This is the second part of the series Write Controllers on Top of Metal-Stack by Using Kubebuilder. Make sure you went through the first part.

kubebuilder markers for CRD

kubebuilder provides lots of handful markers. Here are some examples:

API Resource Type
```
// +kubebuilder:object:root=true
```

The gostruct under this marker will be an API resource type in the url. For example, the url path to XCluster instance myxcluster would be

/apis/cluster.www.x-cellent.com/v1/namespaces/myns/xclusters/myxcluster

API Subresource
```
// +kubebuilder:subresource:status
```

The gostruct under this marker contains API subresource status. For the last example, the url path to the status of the instance would be:

/apis/cluster.www.x-cellent.com/v1/namespaces/myns/xclusters/myxcluster/status

Terminal Output

// +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.ready`

This specifies an extra column of output on terminal when you do kubectl get.

Wire up metal-api client metalgo.Driver

metalgo.Driver is the client in go code for talking to metal-api. To enable both controllers of XCluster and XFirewall to do that, we created a metalgo.Driver named metalClient and set field Driver of both controllers as shown in the following snippet from main.go.

    if err = (&controllers.XClusterReconciler{
        Client: mgr.GetClient(),
        Driver: metalClient,
        Log:    ctrl.Log.WithName("controllers").WithName("XCluster"),
        Scheme: mgr.GetScheme(),
    }).SetupWithManager(mgr); err != nil {
        setupLog.Error(err, "unable to create controller", "controller", "XCluster")
        os.Exit(1)
    }

Role-based access control (RBAC)

With the following lines in xcluster_controller.go and the equivalent lines in xfirewall_controller.go (in our case overlapped), kubebuilder generates role.yaml and wire up everything for your xcluster-controller-manager pod when you do make deploy. The verbs are the actions your pod is allowed to perform on the resources, which are xclusters and xfirewalls in our case.

// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xclusters,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xclusters/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xfirewalls,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cluster.www.x-cellent.com,resources=xfirewalls/status,verbs=get;update;patch

Finalizer

When you want to do some clean-up before the kubernetes api-server deletes your resource in no time upon kubectl delete, finalizers come in handy. A finalizer is simply a string constant stored in field finalizers of a Kubernetes resource instance's metadata. For example, the finalizer of XCluster in xcluster_types.go:

const XClusterFinalizer = "xcluster.finalizers.cluster.www.x-cellent.com"

The api-server will not delete the instance before its finalizers are all removed from the resource instance. For example, in [**xcluster_controller.go](https://github.com/LimKianAn/xcluster/blob/main/controllers/xcluster_controller.go) we add the above finalizer to the XCluster instance, so later when the instance is about to be deleted, the api-server can't delete the instance before we've freed the metal-stack network and then removed the finalizer from the instance. We can see that in action in the following listing. We use the Driver mentioned earlier to ask metal-api if the metal-stack network we allocated is still there. If so, we use the Driver to free it and then remove the finalizer of XCluster.

    resp, err := r.Driver.NetworkFind(&metalgo.NetworkFindRequest{
        ID:        &cl.Spec.PrivateNetworkID,
        Name:      &cl.Spec.Partition,
        ProjectID: &cl.Spec.ProjectID,
    })

    if err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to list metal-stack networks: %w", err)
    }

    if len := len(resp.Networks); len > 1 {
        return ctrl.Result{}, fmt.Errorf("more than one network listed: %w", err)
    } else if len == 1 {
        if _, err := r.Driver.NetworkFree(cl.Spec.PrivateNetworkID); err != nil {
            return ctrl.Result{Requeue: true}, nil
        }
    }
    log.Info("metal-stack network freed")

    cl.RemoveFinalizer(clusterv1.XFirewallFinalizer)
    if err := r.Update(ctx, cl); err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to remove xcluster finalizer: %w", err)
    }
    r.Log.Info("finalizer removed")

Likewise, in xfirewall_controller.go we add the finalizer to XFirewall instance. The api-server can't delete the instance before we clean up the underlying metal-stack firewall (r.Driver.MachineDelete(fw.Spec.MachineID) in the following listing) and then remove the finalizer from the instance:

func (r *XFirewallReconciler) DeleteFirewall(ctx context.Context, fw *clusterv1.XFirewall, log logr.Logger) (ctrl.Result, error) {
    if _, err := r.Driver.MachineDelete(fw.Spec.MachineID); err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to delete firewall: %w", err)
    }
    log.Info("states of the machine managed by XFirewall reset")

    fw.RemoveFinalizer(clusterv1.XFirewallFinalizer)
    if err := r.Update(ctx, fw); err != nil {
        return ctrl.Result{}, fmt.Errorf("failed to remove XFirewall finalizer: %w", err)
    }
    r.Log.Info("finalizer removed")

    return ctrl.Result{}, nil
}

func errors.IsNotFound and client.IgnoreNotFound

When you have different handlers depending on whether the error is the instance not found, you can consider using errors.IsNotFound(err) as follows from xcluster_controller.go:

    fw := &clusterv1.XFirewall{}
    if err := r.Get(ctx, req.NamespacedName, fw); err != nil {
        // errors other than `NotFound`
        if !errors.IsNotFound(err) {
            return ctrl.Result{}, fmt.Errorf("failed to fetch XFirewall instance: %w", err)
        }

        // Create XFirewall instance
        fw = cl.ToXFirewall()

If we can do nothing against the error the instance not found, we might simply stop the reconciliation without requeueing the request as follows:

    cl := &clusterv1.XCluster{}
    if err := r.Get(ctx, req.NamespacedName, cl); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

Exponential Back-Off

As far as requeue is concerned, returning ctrl.Result{}, err and ctrl.Result{Requeue: true}, nil are the same as shown in this if clause and this else if clause in the source code. Moreover, exponential back-off can be observed in the source code where dependencies of a controller are set and where func workqueue.DefaultControllerRateLimiter is defined.

ControllerReference

ControllerReference is a kind of OwnerReference that enables the garbage collection of the owned instance (XFirewall) when the owner instance (XCluster) is deleted. We demonstrate that in xcluster_controller.go by using the function SetControllerReference.

        if err := controllerutil.SetControllerReference(cl, fw, r.Scheme); err != nil {
            return ctrl.Result{}, fmt.Errorf("failed to set the owner reference of the XFirewall: %w", err)
        }

Since XCluster owns XFirewall instance, we have to inform the manager that it should reconcile XCluster upon any change of an XFirewall instance:

func (r *XClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&clusterv1.XCluster{}).
        Owns(&clusterv1.XFirewall{}).
        Complete(r)
}

Wrap-up

Check out the code in this project for more details. If you want a fully-fledged implementation, stay tuned! Our cluster-api-provider-metalstack is on the way. If you want more blog posts about metal-stack and kubebuilder, let us know! Special thanks go to Grigoriy Mikhalkin.