--- title: Zero Downtime Rails Deployments with Kubernetes teaser: 'If you''re deploying with downtime you don''t have continuous delivery, you have continuous downtime. ' tags: kubernetes,devops,rails author: Joe Ferris published_on: 2021-01-11 --- ![Rollout in Action](https://images.thoughtbot.com/blog-vellum-image-uploads/M0f6rEeqTZ21smXm5Z4V_k8s-deploy.gif) If you practice continuous deployment, it's important to deploy without downtime. Otherwise, you don't have continuous delivery; you have continuous downtime. If you're deploying your application to Kubernetes, you already have all the pieces you need to ensure your deployments run without downtime. However, figuring out how to piece them all together for a Rails application can be something of a challenge! This post will walk through the major steps to get a Rails application running in Kubernetes, deployed without downtime. ## Containers Kubernetes runs containers using images built according to the OCI container format. These images are most commonly built using [Docker], but other tools like [buildpacks] and [podman] are available as well. [Docker]: https://docs.docker.com/engine/ [buildpacks]: https://buildpacks.io/ [podman]: https://podman.io/ To deploy a Rails application to a Kubernetes cluster, the first thing you'll need is a container image for your application. Images for your application will live in a container image repository with a URL, like: docker.io/mycompany/myapplication:tag Each time you build an image for your application, you can give it a new tag. I recommend tagging your images to match the Git commit the image is built from, so you'll end up with image URLs that look like this: docker.io/mycompany/myapplication:abcd123 To build a container image using Docker, you'll need a [Dockerfile]. After that, you can build, tag, and push your image: docker build . --tag myapplication:latest docker tag myapplication:latest docker.io/mycompany/myapplication:abcd123 docker push docker.io/mycompany/myapplication:abcd123 [Dockerfile]: https://github.com/thoughtbot/suspenders/blob/81e4b52fd7d3238c4d277186ccdd7a5e85dc6631/Dockerfile Most teams will want to automatically build these images during Continuous Integration. ## Deployment Once you have an image built, you can tell Kubernetes you want to run pods using that image: ``` yaml # deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: rails # ... spec: # ... template: # ... spec: containers: - name: main image: docker.io/mycompany/myapplication:abcd123 ``` kubectl apply -f deployment.yaml Whenever you want to deploy a new version of your application, you can update the manifest to use the latest tag of your container image, and Kubernetes will perform a rollout of the new version. ## Rollout Kubernetes will roll out the release for you by creating and scaling replica sets. The rollout strategy is customizable, but Kubernetes will utilize a rolling release by default. A simple rolling release looks like this: 1. Kubernetes creates a new replica set matching the updated deployment. 2. Kubernetes will launch a new pod matching the new replica set. 3. Once the pod is marked as ready, the pod will begin receiving production traffic. 4. Kubernetes will terminate a pod from the old replica set. 5. Steps 2-4 are repeated until the new replica set is running the desired number of pods and the old replica set is scaled down completely. This process means that new containers are slowly brought online and traffic gradually shifts over to the latest version. This avoids a period during the deploy where pods are unavailable to serve traffic, all traffic shifts at once to pods with cold caches, or many extra containers boot at the same time, overwhelming the database with new connections. This is a good, default recipe for a zero downtime deploy. However, you'll need to provide Kubernetes some extra information to make sure this process runs smoothly for your Rails migrations. ## Readiness Probes Kubernetes needs to know when your pods are ready to begin receiving production traffic. There will be a period after starting your new container when it's loading application code, connecting to databases, and otherwise getting ready to serve traffic. Kubernetes to know when this period is over to avoid prematurely moving forward in the deployment. For a Rails application, making an HTTP request is a good safe bet: ``` yaml spec: containers: - name: main image: docker.io/mycompany/myapplication:abcd123 readinessProbe: httpGet: path: /robots.txt port: 3000 scheme: HTTP ``` After updating your `deployment.yaml` file, you can re-apply it to update: kubectl apply -f deployment.yaml Adding this to your deployment tells Kubernetes to wait until a request to `/robots.txt` responds with a 200 before marking the pod as ready. This will pause the deploy until your application server (such as Puma) has loaded your application and is ready to respond to user requests. If something prevents your application from reaching the point where it can fulfill this readiness probe, the rollout won't move forward and the old version will continue running uninterrupted. This means that you won't bring down production by forgetting to add a new environment variable. The rollout will keep trying to restart and probe the container until it responds successfully. ## Migrate Job Most Rails applications will utilize a database like Postgres, and this means keeping the database schema synchronized with the application. This is easy to do with Rails migrations, but you need a way to tell Kubernetes to run the migrations. One good way to do this is by using a Job: ``` yaml apiVersion: batch/v1 kind: Job metadata: generateName: db-migrate- spec: template: spec: containers: - name: main command: - rails - db:migrate image: docker.io/mycompany/myapplication:abcd123 ``` A Job works like a Deployment, but rather than keeping a certain number of pods running at all times, it runs your pod until it completes successfully and then stops. You can create a new migration job for each new version in your CD pipeline. ## Init Containers You can start your migration job at the same time as you update your deployment to use your new image tag, but you'll need to make sure the migrations complete successfully before trying to start the Rails application. Otherwise, the new code will attempt to use database columns that don't exist yet, which will cause application errors. You can make sure your containers wait for migrations to complete by adding an initContainer to your deployment manifest: ``` yaml spec: containers: - name: main image: docker.io/mycompany/myapplication:abcd123 initContainers: - name: migrations image: docker.io/mycompany/myapplication:abcd123 command: - rake - db:abort_if_pending_migrations ``` Kubernetes will run the `migrations` container before trying to start the `main` container. This built-in rake task will exit with an error if migrations are pending. If an [initContainer](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) fails, Kubernetes will restart it until it completes successfully. This means that your main container won't start until migrations complete successfully, and the pod won't be marked as ready until the database schema it needs is in place. ## Migration Gotchas One downside to using a rolling release is that multiple versions of your application will be running at the same time during a rollout. If you run migrations while old versions of your application are running, this means your migrations must be backwards compatible for at least one deploy. Most migrations will work fine as-is, but it means handling some changes with care: * If you add a new, non-nullable column, you'll have to provide a default value. * Deleting a column that's still in use will cause application errors. Release a version that doesn't use the column first, and then remove it in a subsequent release. * Renaming tables or columns is likely to cause issues. You may want to deploy code that uses an alias first, and then rename it in a subsequent release. This requires a little extra vigilance, but it makes for effortless zero downtime deploys. As a bonus, it makes it easier to roll back a release in case something goes wrong as well. ## Putting It All Together With all these pieces in place, you can deploy continuously without interruptions or errors, automatically running database migrations as you go. In case you're looking for a set of manifests to serve as the base of your own deployment, I've put together a [Gist] containing a fuller example. [Gist]: https://gist.github.com/jferris/1aba7433f5318715bda66b98c1d953f0