This is the second part of the series Write Controllers on Top of Metal-Stack by Using Kubebuilder. Make sure you went through the first part.

kubebuilder markers for CRD

kubebuilder provides lots of handful markers. Here are some examples:

  1. API Resource Type

    // +kubebuilder:object:root=true

    The go struct under this marker will be an API resource type in the url. For example, the url path to XCluster instance myxcluster would be

  2. API Subresource

    // +kubebuilder:subresource:status

    The go struct under this marker contains API subresource status. For the last example, the url path to the status of the instance would be:

  3. Terminal Output

    // +kubebuilder:printcolumn:name="Ready",type=string,JSONPath=`.status.ready`

    This specifies an extra column of output on terminal when you do kubectl get.

Wire up metal-api client metalgo.Driver

metalgo.Driver is the client in go code for talking to metal-api. To enable both controllers of XCluster and XFirewall to do that, we created a metalgo.Driver named metalClient and set field Driver of both controllers as shown in the following snippet from main.go.

	if err = (&controllers.XClusterReconciler{
		Client: mgr.GetClient(),
		Driver: metalClient,
		Log:    ctrl.Log.WithName("controllers").WithName("XCluster"),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "XCluster")

Role-based access control (RBAC)

With the following lines in xcluster_controller.go and the equivalent lines in xfirewall_controller.go (in our case overlapped), kubebuilder generates role.yaml and wire up everything for your xcluster-controller-manager pod when you do make deploy. The verbs are the actions your pod is allowed to perform on the resources, which are xclusters and xfirewalls in our case.



When you want to do some clean-up before the kubernetes api-server deletes your resource in no time upon kubectl delete, finalizers come in handy. A finalizer is simply a string constant stored in field finalizers of a Kubernetes resource instance's metadata. For example, the finalizer of XCluster in xcluster_types.go:

const XClusterFinalizer = ""

The api-server will not delete the instance before its finalizers are all removed from the resource instance. For example, in xcluster_controller.go we add the above finalizer to the XCluster instance, so later when the instance is about to be deleted, the api-server can't delete the instance before we've freed the metal-stack network and then removed the finalizer from the instance. We can see that in action in the following listing. We use the Driver mentioned earlier to ask metal-api if the metal-stack network we allocated is still there. If so, we use the Driver to free it and then remove the finalizer of XCluster.

	resp, err := r.Driver.NetworkFind(&metalgo.NetworkFindRequest{
		ID:        &cl.Spec.PrivateNetworkID,
		Name:      &cl.Spec.Partition,
		ProjectID: &cl.Spec.ProjectID,

	if err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to list metal-stack networks: %w", err)

	if len := len(resp.Networks); len > 1 {
		return ctrl.Result{}, fmt.Errorf("more than one network listed: %w", err)
	} else if len == 1 {
		if _, err := r.Driver.NetworkFree(cl.Spec.PrivateNetworkID); err != nil {
			return ctrl.Result{Requeue: true}, nil
	log.Info("metal-stack network freed")

	if err := r.Update(ctx, cl); err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to remove xcluster finalizer: %w", err)
	r.Log.Info("finalizer removed")

Likewise, in xfirewall_controller.go we add the finalizer to XFirewall instance. The api-server can't delete the instance before we clean up the underlying metal-stack firewall (r.Driver.MachineDelete(fw.Spec.MachineID) in the following listing) and then remove the finalizer from the instance:

func (r *XFirewallReconciler) DeleteFirewall(ctx context.Context, fw *clusterv1.XFirewall, log logr.Logger) (ctrl.Result, error) {
	if _, err := r.Driver.MachineDelete(fw.Spec.MachineID); err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to delete firewall: %w", err)
	log.Info("states of the machine managed by XFirewall reset")

	if err := r.Update(ctx, fw); err != nil {
		return ctrl.Result{}, fmt.Errorf("failed to remove XFirewall finalizer: %w", err)
	r.Log.Info("finalizer removed")

	return ctrl.Result{}, nil

func errors.IsNotFound and client.IgnoreNotFound

When you have different handlers depending on whether the error is the instance not found, you can consider using errors.IsNotFound(err) as follows from xcluster_controller.go:

	fw := &clusterv1.XFirewall{}
	if err := r.Get(ctx, req.NamespacedName, fw); err != nil {
		// errors other than `NotFound`
		if !errors.IsNotFound(err) {
			return ctrl.Result{}, fmt.Errorf("failed to fetch XFirewall instance: %w", err)

		// Create XFirewall instance
		fw = cl.ToXFirewall()

If we can do nothing against the error the instance not found, we might simply stop the reconciliation without requeueing the request as follows:

	cl := &clusterv1.XCluster{}
	if err := r.Get(ctx, req.NamespacedName, cl); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)

Exponential Back-Off

As far as requeue is concerned, returning ctrl.Result{}, err and ctrl.Result{Requeue: true}, nil are the same as shown in this if clause and this else if clause in the source code. Moreover, exponential back-off can be observed in the source code where dependencies of a controller are set and where func workqueue.DefaultControllerRateLimiter is defined.


ControllerReference is a kind of OwnerReference that enables the garbage collection of the owned instance (XFirewall) when the owner instance (XCluster) is deleted. We demonstrate that in xcluster_controller.go by using the function SetControllerReference.

		if err := controllerutil.SetControllerReference(cl, fw, r.Scheme); err != nil {
			return ctrl.Result{}, fmt.Errorf("failed to set the owner reference of the XFirewall: %w", err)

Since XCluster owns XFirewall instance, we have to inform the manager that it should reconcile XCluster upon any change of an XFirewall instance:

func (r *XClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).


Check out the code in this project for more details. If you want a fully-fledged implementation, stay tuned! Our cluster-api-provider-metalstack is on the way. If you want more blog posts about metal-stack and kubebuilder, let us know! Special thanks go to Grigoriy Mikhalkin.