The project we’re working on is currently set to have new code released following a schedule, twice a week and the process is heavily manual. A developer needs to be online at specific times, after-hours, to limit the impact on the users. By impact we mean that the site has to be taken into maintenance mode, preventing the users from accessing the app while new code is being deployed.
The amount of time it takes to do a release varies, and there are several steps involved, which are all documented, but the list is cumbersome and can be prone to errors. When an automatic system can do it, why should a developer do it? It is a cost for the client, where they could employ our developer time for better tasks than releasing code. It is a cost for the developers, in terms of context-switching, paying attention to the list of steps of the manual process, and impacting their work-life balance, having to be available after their regular working schedule.
Why are we making this change?
Achieve zero-downtime deploys
The main reason for making this change is to allow us to have zero-downtime when we deploy the application to our servers.
Currently we use a command, sudo systemctl restart efg-puma
as part of our deploy script to perform a full restart of puma
(the web-server we use.)
This causes downtime and forces us to put up a user-facing “maintenance
page” during deploys.
Our intention is to replace the sudo systemctl restart efg-puma
command with sudo systemctl reload efg-puma. This hot-reloads the
puma server and does not cause any downtime.
However, we cannot do this whilst using environment variables, as changes to the environment variables are only picked up when you restart the server and not when you simply reload the server.
Therefore if we were to make a change to an environment variable on the server and only reload the server, the app would still be reading the environment variable at its previous value.
Moving to Rails Credentials allows us to reload the server during deploys because changes to Rails Credentials will be picked up when only reloading the server.
Commit to source control
Another benefit of using Rails Credentials is that we actually commit the key value pairs to source control. We can do this because they are encrypted.
Environment variables are not encrypted and therefore they can not be stored in any form of version control.
The advantage of this is that it makes it easier for developers to view
the key value pairs for each environment in the codebase and also
makes it easier to add new credentials because you don’t need to
tell other developers to add them to their own .env file.
It is also easier to deploy a feature which relies on a credential
because we store that credential in the codebase along with that
feature and do not have to separately add the environment variable
to the .env file on each server.
Tradeoff
It is worth noting that as with any technical choice there is always a tradeoff. There are some things that environment variables are better at doing than credentials and some things credentials are better at.
Ultimately, the main driving factor is the benefit of zero-downtime so we can get away from deployment windows outside of hours and head towards continuous delivery, but we may discover that Rails Credentials could make some things more tricky.