Terraform Deployment Orchestration Tool

Challenge

While implementing the cloud and terraform setup for planblues data pipeline architecture, not only deployment time, but also the complexity of the deployment process grew. It was important to keep the terraform setup clean and sliced into isolated stacks but also improve the daily workflow of the team as well as the CI/CD process.

My Contribution

I designed and implemented a cli tool to deploy all the stacks in a selective, structured, yet parallelized and hands-off way. To keep maintainability as high as it gets, i designed the tool to be an acceleration layer for the terraform setup. Each of the stacks should be still deployable with plain terraform.

Deploying services went from “Deep dive into the directory structure to” to simply running a dedicated command.

Key features included:

• Selection of stacks to deploy
• Creation and execution of terraform commands (init, plan, apply, destroy…) for selected stacks
• Stack dependency graph visualization based on terraform_remote_state
• Hierarchical parallelized deployment of stacks
• Secrets and env var management
  • e.g. env vars for Mongo Atlas access
• Multi backend and tfvars configuration for multiple environments
• Shipping the tool as a docker image to enable ci/cd deployment

Impact

The new deployment tool enabled the team to deploy services hands-off and with minimal manual intervention. It also reduced the complexity of the deployment process and improved the overall developer experience.

pythontyperterraformci/cddocker