Blue/Green Message Bus Handling on Kubernetes with Shawarma

In the modern world of cloud-based microservices and continuous delivery, blue/green deployments (a.k.a red/black deployments) have become commonplace. This approach provides a way to safely deploy an application with zero down time, and quickly roll back to the previous version if there is a problem. If you’re deploying in the cloud and not using blue/gren deployments, stop reading now and go find out what you’re missing.

At CenterEdge Software our primary deployment strategy is blue/green deployments to Kubernetes using Spinnaker. However, in practice we quickly discovered a limitation of the typical blue/green deployment strategy.

TL;DR

We’ve solved the problem of blue/green deployments of message bus consumers on Kubernetes using a lightweight sidecar and a simple application plugin. We call the system Shawarma. (Yes, it’s an Avengers MCU reference)

The Problem

The key to blue/green deployments is the ability to route incoming requests to just the active version of the service, leaving the inactive version idle. This works great for incoming HTTP requests, as the blue/green deployment pipeline will put the active service on the load balancer and remove the inactive service. You’ll only have a brief window where both versions are serving HTTP requests.

However, queued messages being processed from your message bus, such as Kafka or RabbitMQ, should be restricted as well. These also represent a form of input, like HTTP requests, which should only be processed by the active version. However, message bus processing is normally a pull process, not push, which makes the problem more complicated.

Possible Solutions

There are several possible approaches to solving the problem, but many suffer from too much complexity. For example, one approach is to separate message bus processing from the service and forward requests to the service over HTTP. I’ll call this the Redirection Approach.

This approach helps to maintain separation of concerns, since the service itself isn’t required to know about its deployment or understand if it’s the active or inactive version. However, it adds a lot of complexity, requires deploying an additional service, and results in a signifcant performance hit for high-volume queues.

Another approach, which I’ll call the Colorful Service Approach, requires that the service know about its deployment status by monitoring the load balancer.

The Colorful Service Approach is simpler in many ways, but now our concerns are not separate. The service itself needs to know about the deployment environment, how to check the status of the load balancer, and more. While this works, it doesn’t make for a very Evolutionary Architecture.

The Shawarma Solution

At CenterEdge, we’ve created our own, open source solution to this problem, Shawarma. Shawarma is specifically designed for Kubernetes, and works via a modified version of the Colorful Service Approach. It simplifies the design even further, and maintains separation of concerns by keeping the application itself agnostic. It takes advatange of the Kubernetes API and built-in support for watch commands to receive notifications about state changes within the Kubernetes cluster.

In the Shawarma Approach, each service gets a single additional HTTP endpoint which receives POST operations informing the service of its state, active or inactive. It doesn’t know anything at all about the deployment environment, it only knows this single, basic data point.

The brains of the operation is a Shawarma sidecar deployed within the Kubernetes pod. This sidecar container knows about Kubernetes and its Services and Endpoints, and monitors Kubernetes to know if the application is active or inactive on the assigned load balancer. When the status changes, it makes an HTTP POST calls to notify the main service. The main service can then discontinue or restart any background processing, including message bus processing.

Shawarma Implementation

The Shawarma sidecar is implemented as a very lightweight Go application, available on Docker Hub. At this point it’s in beta testing at CenterEdge, but the early signs are very encouraging.

Additionally, we recognized very quickly that deployments needed to be kept as simple as possible. The need to configure dozens of microservices with a sidecar container was daunting. To that end, we’ve created a simple Kubernetes MutatingAdmissionWebhook. When the Shawarma Webhook is deployed, a simple annotation on a Pod (or Deployment template) will automatically configure the sidecar.

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: shawarma-example
    active: 'true'
  annotations:
    # The annotation value is name of the service for Shawarma to monitor
    shawarma.centeredge.io/service-name: shawarma-example
spec:
  # ...

A full example set can be found here.

Application State in ASP.NET Core

Since CenterEdge is a .NET shop, we’ve also implemented some supporting NuGet packages to handle the application state.

Package	Description
Shawarma.Abstractions	Basic abstractions and models
Shawarma.AspNetCore	.NET Core middleware for receiving and handling application state
Shawarma.Hosting	The IShawarmaService system, an equivalent to IHostedService which starts and stops based on application state

Using these SDK packages, adding slim, agnostic handling of Shawarma application state is easy.

public void ConfigureServices(IServiceCollection services)
{
    services
        .AddMvc()
        .AddShawarmaHosting()
        // Add any IShawarmaService instances to be managed
        .AddShawarmaService<TestService>();
}

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

    // Add before MVC or other handlers
    app
        .UseShawarma()
        .UseMvc();
}

public class TestService : GenericShawarmaService
{
    public TestService(ILogger<TestService> logger)
        : base(logger)
    {
    }

    protected override Task StartInternalAsync(CancellationToken cancellationToken)
    {
        // Start doing work here
        return Task.CompletedTask;
    }

    protected override Task StopInternalAsync(CancellationToken cancellationToken)
    {
        // Stop doing work here
        return Task.CompletedTask;
    }
}

Conclusion

Hopefully, the Shawarma project will be helpful to other services deploying microservices to Kubernetes. We’re very open to feedback, issues may be filed at GitHub. We’d also love some help from the community developing application state SDKs for other frameworks such as Java and Node.

It’s also worth nothing that the Shawarma Approach is useful for more than just message bus processing. Any kind of background work being handled outside of HTTP requests can benefit, such as scheduled tasks or polling remote data sources.