How I built a Kubernetes cluster so my coworkers could deploy apps faster

As a full-stack developer, I‘m always looking for ways to enable my team to build and ship products faster. Removing barriers and automating manual processes is key to moving quickly and encouraging experimentation.

One area my team was getting bogged down was application deployment. We build a lot of web and mobile apps, which means a variety of backend services to deploy and manage. While we had a standard deployment pipeline, it still required a ticketing process and manual work from the ops team to provision servers and deploy updates. This meant lead times of days or sometimes weeks to get an app deployed.

To solve this problem and empower the dev team, we decided to set up a Kubernetes cluster that developers could deploy apps to directly. Kubernetes, for those not familiar, is an open-source container orchestration platform originally developed by Google. It provides powerful automation capabilities for deploying, scaling, and managing containerized applications.

By running a Kubernetes cluster, we could abstract away the underlying infrastructure. Developers could simply build container images for their apps and deploy them to the cluster without worrying about the individual servers underneath. Kubernetes would handle scheduling the containers, setting up networking and storage, and keeping the apps running.

However, we didn‘t want the Kubernetes cluster to become a free-for-all. We needed to maintain security and ensure proper isolation between different apps and teams. At the same time, we wanted to minimize the administrative overhead required to grant developers access to deploy.

After researching the options, we settled on using GitHub authentication to enable developer access to the cluster. We were already heavy GitHub users, so it made sense to leverage the existing accounts and authentication. Here‘s how we set it up:

Authentication

Kubernetes supports a variety of authentication methods, including client certificates, static tokens, and external identity providers. We considered a few different options:

Certificates: With certificate-based authentication, each user has an X.509 cert that they present when accessing the cluster API. While very secure, this requires a lot of overhead to manage a private key infrastructure (PKI) and distribute certs to each user. Definitely not self-service.

Static tokens: Kubernetes lets you define a static file mapping users to tokens. When a user connects with that token, they are authenticated as that user. More manageable than certs, but still requires an admin to create accounts and distribute tokens to each user. Tokens can also easily be leaked.

OpenID Connect (OIDC): Kubernetes has built-in support for OIDC identity providers, which provide OAuth2-style "Log In With X" functionality. At the time we were setting this up, the Kubernetes OIDC support was fairly new, so we were hesitant to use it for such a critical system. But it‘s a good option with better industry support these days.

Webhook token authentication: Kubernetes provides a mechanism for verifying bearer tokens by making a webhook call to an external service. This external service can use whatever means it wants to verify the token and return the authenticated user info.

We went with the webhook token approach, using GitHub personal access tokens. Since all our developers already have GitHub accounts, we could leverage those for authentication. Developers could generate access tokens in their GitHub account settings and use them to access the Kubernetes cluster.

I implemented the webhook authentication service in Go. It exposes a single endpoint that the Kubernetes API server calls with a provided bearer token. The webhook handler takes the token and validates it against the GitHub API. If the token is valid, it returns the associated GitHub username.

With this in place, developers could now authenticate to Kubernetes by passing their GitHub token in the Authorization header or by using the kubectl –token flag. No admin setup required.

Authorization

Authentication is only half the story. Once authenticated, we still needed to define what actions developers could perform on the cluster. Enter role-based access control, or RBAC.

RBAC is a powerful Kubernetes feature that lets you define fine-grained permissions using roles and role bindings. A role defines a set of permissions on resources, like "get", "list", "create", etc. A role binding grants those permissions to a user or group.

To enable developer self-service, we set up a few standard roles:

Application developer: Can deploy and manage applications within their own namespace. Can create deployments, services, etc. but cannot access cluster-level resources or other namespaces.

Project lead: Can manage RBAC permissions for their project namespace. Can onboard new developers to the project by granting them the appropriate roles.

Cluster admin: Can perform any action on any resource across the entire cluster. Reserved for the ops team.

We then wrote a service that automatically creates a namespace and role binding when a new GitHub team is added to our org. The namespace is named after the team, and all members of the GitHub team are bound to the application developer role in that namespace. Project leads are identified by a special GitHub team and are bound to the project lead role.

With this setup, developers can now create a new GitHub team for their project, add their team members, and immediately get a namespace provisioned where they can deploy their apps. No more ticketing or manual setup required.

A few bumps along the road

While the GitHub authentication and RBAC setup has worked well, we did hit a few snags along the way. One issue was that Kubernetes expects the webhook service to be accessible at all times. If the GitHub API is down or rate limiting us, authentication to the Kubernetes cluster breaks.

To mitigate this, we now run several replicas of the webhook service behind a load balancer. We‘ve also tweaked the service to gracefully handle and cache GitHub API outages to prevent full authentication failure.

We also had to educate developers about RBAC permissions and the dangers of overly broad roles. It‘s easy to fall into the trap of granting wide permissions to make things "just work." But that defeats the purpose of RBAC isolation. We now have better documentation and on-boarding materials to help developers understand how to define least-privilege RBAC roles.

The payoff

Setting up the Kubernetes cluster with GitHub auth and RBAC has been a huge enabler for our team. Instead of waiting days or weeks for deployment, developers can now ship new apps and features in minutes.

The lowered barrier to deployment has led to an explosion in experimentation and new projects. Developers are free to try out new ideas without jumping through hoops or worrying about wasting costly resources.

For the ops team, we‘ve been able to focus on higher-level cluster management and improvements instead of being bogged down in manual deployment work. We still maintain oversight via RBAC controls, but much of the day-to-day deployment work is self-service.

Looking ahead

With the success of the new cluster setup, we‘re looking at ways to extend it further. Some ideas on the roadmap:

Automatic RBAC sync: While new team namespaces are automatic, RBAC role bindings are still manual. We‘re working on tooling to detect GitHub team membership changes and automatically sync them to the corresponding RBAC bindings.

Monitoring and auditing: With more activity on the cluster, we need better visibility. We‘re setting up centralized logging, monitoring, and auditing to keep an eye on cluster health and usage. We want to make sure any security-sensitive actions are logged for traceability.

Continuous deployment: Right now, developers still need to manually deploy new container images via kubectl. The next step is to hook up the cluster to a continuous deployment pipeline so that new commits automatically trigger container builds and rolls out to the appropriate environments.

In summary, adopting Kubernetes and enabling developer self-service has been a huge boost to our team‘s productivity and experimentation. If you‘re looking to achieve similar results, I highly recommend looking into Kubernetes with GitHub authentication. The setup takes some work, but the payoff in speed and flexibility is well worth it.