How do you mitigate deployment risks?

If you’re looking for ways to mitigate deployment risks, you’ve just found the right place. You can do this by adopting approaches that provide fast feedback on quality and enable rapid recovery when recently introduced changes have undesired outcomes. In this article, based on AWS Best Practices, we will explore different ways in which you can mitigate the impact of any issues that might be introduced through the deployment of changes.


Planning for unsuccessful changes

It goes without saying that, whenever you’re ready to introduce changes, you expect things to work according to plan and to have a positive experience. However, it’s best practice to always plan for any scenario in which the opposite is true. Before implementing any changes, you should assess what could go wrong and plan accordingly. In the event of unsuccessful changes, you can reduce recovery time through faster responses by having a plan to revert to a known good state or remediate the issue in the production environment.


Testing and validating changes

The risks and impact of failed deployments can be minimized by introducing solid testing. All changes should be tested in all stages of their lifecycle. If you’re able to validate the results of testing during the development, test, and production stages, you can confirm whether the changes will have the desired impact and obtain more information to plan for the worst-case scenario.

Useful resources:

AWS Cloud9

What is AWS Cloud9?

How to test and debug AWS CodeDeploy locally before you ship your code


Using deployment management systems

It’s best practice to track and implement change using deployment management systems. By doing this, you can reduce errors introduced by manual processing and reduce the level of effort it takes to deploy changes. If you’re looking to reduce lead time, enable increased frequency of change and, at the same time, further reduce the level of effort required, you should be looking into introducing automation. You can automate the integration and deployment pipeline from code check-in through testing, deployment, and validation by using deployment management systems, thus, contributing to mitigating deployment risks.

Useful resources:

Introduction to AWS CodeDeploy - automated software deployment with Amazon Web Services

What is AWS CodeDeploy?

What is AWS Elastic Beanstalk?

What is Amazon API Gateway?


Using limited deployments when testing

If you’re looking to mitigate deployment risks, another useful way of doing so can be to confirm desired outcomes prior to full-scale deployment by testing with limited deployments alongside existing systems. Some of the ways in which you can achieve this are using deployment canary testing or one-box deployments.

Useful resources:

Working with deployment configurations in AWS CodeDeploy

Blue/Green deployments with AWS Elastic Beanstalk

Set up an API Gateway canary release deployment


Using parallel environments for deployment

Implementing changes onto parallel environments to obtain confirmation of successful deployment before transitioning over to the new environment can be a great strategy to minimise deployment risks. This process allows you to enable rollback to the previous environment, thus, minimising recovery time. You can do this, for example, by using immutable infrastructures with blue/green deployments.

Useful resources:

Working with deployment configurations in AWS CodeDeploy

Blue/Green deployments with AWS Elastic Beanstalk

Set up an API Gateway canary release deployment


Deploying frequent, small, reversible change

You can minimize deployment risks by using frequent, small, and reversible changes. This reduces the scope of change, which results in easier troubleshooting and faster remediation enabled by always having the option to roll back change without creating big-scale issues.


Fully automating integration and deployment

Build and deployment management systems that track and implement change will allow you to fully automate integration and deployment. This can reduce human error as well as decrease the level of effort required to introduce changes. If you’re looking to reduce lead time, enable increased frequency of change, and minimize effort, the way to go is to fully automate the integration and deployment pipeline from code check-in through build, testing, deployment, and validation.

Useful resources:

What is AWS CodeBuild?

Continuous integration best practices for software development

Slalom: CI/CD for serverless applications on AWS

Introduction to AWS CodeDeploy - automated software deployment with Amazon Web Services


Automating testing and rollback

Testing of deployed environments can be automated to confirm desired outcomes, thus, mitigating deployment risks. You can also automate rollback to a prior known good state in case your desired outcomes are not achieved. This brings recovery time to an all-time low while, at the same time, reducing the chance of introducing issues by human error. Examples of this strategy can be to perform detailed synthetic user transactions following deployment, verify the results, and roll back in the event of failure.

Useful resources:

Redeploy and roll back a deployment with AWS CodeDeploy