Series overview:

• Introduction & Environment Bootstrap
• Implementing Main Reconcile Logic
• Implementing Resource Cleanup
• Implementing Webhooks
• Testing Your Operator (this post)

In previous posts, we covered bootstrapping an operator, implementing the reconciliation loop, handling cleanup logic using finalizers and using webhooks. Now let's look at how we can implement end-to-end testing for newly created Tenant Operator.

Tools we'll use:

1. Envtest: Part of the Kubebuilder suite, it helps in setting up a test control plane for unit tests.
2. Ginkgo: A BDD (Behavior-Driven Development) testing framework for Go.

Setting up your Development Environment

Installing Envtest

First thing you need to do is to ensure that envtest binaries are installed by running:

$ make envtest
test -s /Users/codereliant/dev/sample-tenant-operator/bin/setup-envtest || GOBIN=/Users/codereliant/dev/sample-tenant-operator/bin go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest

Envtest is a testing utility provided by the Kubernetes Controller Runtime library. It allows spinning up a local Kubernetes API server and etcd instance for running integration tests against the Kubernetes API:

• Provides a local control plane for testing without needing a real cluster
• Automatically registers CRDs
• Exposes Kubernetes client config for test clients
• Manages lifecycle of API server and etcd instances

In a typical test suite using Envtest, you would:

1. Initialize an Envtest instance
2. Call Start() to bootstrap the local control plane
3. Use the client config to create a Kubernetes client
4. Write test cases that interact with the API server using the client
5. Call Stop() to tear down Envtest after tests finish

This allows you to test controllers, webhooks, operators etc without requiring an actual remote cluster. Everything runs locally against API objects in memory.

Preparing the Test Suite

As mentioned above, we will utilize Ginkgo, which provides a structured way to write and execute Go tests that is easy to read and maintain. The BDD style lends itself well to both unit and integration tests and is commonly used for testing Kubernetes controllers and operators developed with Kubebuilder.

High level Ginkgo test workflow looks like this:

We'll go over these steps below.

The operator scaffolding has generated a basic test file internal/controller/suite_test.go that we can build on. Let's quickly go through what it already has:

var _ = BeforeSuite(func() {
    ...

  	By("bootstrapping test environment")
	testEnv = &envtest.Environment{
		CRDDirectoryPaths:     []string{filepath.Join("..", "..", "config", "crd", "bases")},
		ErrorIfCRDPathMissing: true,
	}

	var err error
	// cfg is defined in this file globally.
	cfg, err = testEnv.Start()
	Expect(err).NotTo(HaveOccurred())
	Expect(cfg).NotTo(BeNil())

	err = multitenancyv1.AddToScheme(scheme.Scheme)
	Expect(err).NotTo(HaveOccurred())

	//+kubebuilder:scaffold:scheme

	k8sClient, err = client.New(cfg, client.Options{Scheme: scheme.Scheme})
	Expect(err).NotTo(HaveOccurred())
	Expect(k8sClient).NotTo(BeNil())
}

var _ = AfterSuite(func() {
	By("tearing down the test environment")
	err := testEnv.Stop()
	Expect(err).NotTo(HaveOccurred())
})

This code is setting up the test environment and clients for running the Ginkgo test suite. Let's go through it step-by-step:

First in the BeforeSuite function:

• testEnv is initialized as an envtest.Environment instance. This will represent the local control plane.
• CRDDirectoryPaths tells Envtest where to load CRDs from. This ensures our custom resources are registered when control plane starts.
• testEnv.Start() boots up the local control plane by starting etcd and the API server.
• cfg contains the Kubernetes client config for connecting to the API server.
• AddToScheme() registers our CRD kinds with the client scheme. This is needed to use our custom resources.
• A Kubernetes client k8sClient is initialized using the client config. We'll use this in our tests to make API calls.
• Expect() calls are making assertions that everything initialized correctly.

In the AfterSuite:

• testEnv.Stop() shuts down the API server and etcd, cleaning up the local environment.

Now what's missing in this code is start of our Tenant controller, so let's implement it. This can be done similar to how we already do it in our main.go:

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:             scheme.Scheme,
		MetricsBindAddress: ":8080",
		Port:               9443,
	})
	Expect(err).NotTo(HaveOccurred())

	err = (&TenantReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr)
	Expect(err).ToNot(HaveOccurred())

	go func() {
		defer GinkgoRecover()
		err = mgr.Start(ctrl.SetupSignalHandler())
		Expect(err).ToNot(HaveOccurred(), "failed to run manager")
	}()

Now we are ready to write our tests.

Writing Test Cases

Ginkgo basics

We can now create individual tests (or specs) using Ginkgo. For our Tenant Operator, we will have tenant_controller_test.go stored in the same directory as tenant_controller.go

Ginkgo uses a specific structure for organizing test specs:

• Describe blocks define a test suite
• It blocks define a test spec
• BeforeEach/AfterEach configure per-test setup/teardown

For example:

Describe("Tenant controller", func() {

  BeforeEach(func() {
    // Common setup 
  })

  It("should create a namespace", func() {
    // Test logic  
  })

})

It blocks should read like sentences describing the expected behavior.

Tenant Controller Tests

Let's walk through the test code for a Tenant controller:

// tenant_controller_test.go

var _ = Describe("Tenant controller", func() {

  // Variables for common test data
  const tenantName = "test-tenant"

  // Shared test context
  ctx := context.Background()

  // Helper functions for fetching resources
  func fetchNamespace(name string) {...}

  func fetchRoleBinding(name string) {...}

  Context("When reconciling a Tenant", func() {

    It("should create corresponding namespaces and rolebindgings", func() {
      
      // Arrange
      tenant := createTestTenant(tenantName)

      // Act
      reconciler.Reconcile(tenant)  

      // Assert
      Expect(fetchNamespace("ns1")).ToNot(BeNil())
    })
})

The overall structure separates the 3 A's:

• Arrange - Initialize test objects
• Act - Call reconciliation logic
• Assert - Verify expected state

Now let's create some help functions we will use in our test:

 func createTestTenant(name string) *multitenancyv1.Tenant {
	return &multitenancyv1.Tenant{
		ObjectMeta: metav1.ObjectMeta{
			Name: name,
		},
		Spec: multitenancyv1.TenantSpec{
			AdminEmail:  "test@example.com",
			Namespaces:  []string{"test-namespace"},
			AdminGroups: []string{"test-admin-group"},
			UserGroups:  []string{"test-user-group"},
		},
	}
}

func fetchNamespace(ctx context.Context, name string, k8sClient client.Client) *corev1.Namespace {
	ns := &corev1.Namespace{}
	err := k8sClient.Get(ctx, client.ObjectKey{Name: name}, ns)
	Expect(err).ToNot(HaveOccurred(), "Failed to fetch namespace: %s", name)
	return ns
}

func fetchRoleBinding(ctx context.Context, nsName, roleName string, k8sClient client.Client) *rbacv1.RoleBinding {
	rb := &rbacv1.RoleBinding{}
	err := k8sClient.Get(ctx, client.ObjectKey{Namespace: nsName, Name: roleName}, rb)
	Expect(err).ToNot(HaveOccurred(), "Failed to fetch RoleBinding in namespace %s with name %s", nsName, roleName)
	return rb

And finally implement the test itself:

• We create a new tenant based on predefined test
• Run a reconcile operation
• Ensure that requested namespaces were created, together with required rolebindings

var _ = Describe("Tenant controller", func() {
	const (
		TenantName = "test-tenant"
	)

	ctx := context.Background()

	Context("When reconciling a Tenant", func() {
		// Tests the tenant creation process.
		It("should create corresponding namespaces and rolebindgings", func() {
			tenant := createTestTenant(TenantName)
			Expect(k8sClient.Create(ctx, tenant)).Should(Succeed())

			reconciler := &TenantReconciler{
				Client: k8sClient,
			}

			_, err := reconciler.Reconcile(ctx, ctrl.Request{
				NamespacedName: client.ObjectKey{Name: TenantName},
			})
			Expect(err).ToNot(HaveOccurred())

			for _, ns := range tenant.Spec.Namespaces {
				// Checking the annotations of the namespace.
				namespace := fetchNamespace(ctx, ns, k8sClient)
				Expect(namespace.Annotations["adminEmail"]).To(Equal(tenant.Spec.AdminEmail), "Expected adminEmail annotation to match")

				// Verifying the admin RoleBinding exists.
				adminRoleBinding := fetchRoleBinding(ctx, ns, fmt.Sprintf("%s-admin-rb", ns), k8sClient)
				Expect(adminRoleBinding).NotTo(BeNil(), "Expected admin RoleBinding to exist")

				// Verifying the user RoleBinding exists.
				userRoleBinding := fetchRoleBinding(ctx, ns, fmt.Sprintf("%s-edit-rb", ns), k8sClient)
				Expect(userRoleBinding).NotTo(BeNil(), "Expected user RoleBinding to exist")
			}
		})
	})
})

Executing the test

The tests are run with:

$ make test
KUBEBUILDER_ASSETS="/Users/codereliant/dev/sample-tenant-operator/bin/k8s/1.27.1-darwin-arm64" go test ./... -coverprofile cover.out
?   	codereliant.io/tenant/cmd	[no test files]
ok  	codereliant.io/tenant/api/v1	0.356s	coverage: 1.1% of statements
ok  	codereliant.io/tenant/internal/controller	6.822s	coverage: 54.0% of statements

This will spin up the Envtest control plane, execute the Ginkgo test suites, and report results. In case any errors occurs you will see which tests failed and it'll be enriched with control plane logs.

Wrapping up

In this multi-part blog series, we walked through the hands-on process of implementing a Tenant Operator from scratch using the Kubernetes Operator SDK. I hope this series provided a solid foundation for implementing robust Kubernetes operators in Go. If you like this series - please subscribe to our free newsletter to stay in the loop.

The code discussed in this blog series can be found in this github repo.