WIREDGOV

Blog posted by: Oswald Quek, Senior Developer, Government Digital Service, 26 July 2023 – Categories: Posts for Tech Specialists.

This is part of our blog series intended for a technical audience. While we try our best to explain things clearly, we will use technical terms in common use in the industry. As part of our practice of working in the open, GDS likes to write about our technical work, so that we can share and connect with technical specialists around the world.

On GOV.UK Pay, we're evaluating options to improve our monitoring and observability capabilities which help us ensure GOV.UK Pay is efficient and reliable.

In this post we show how to set up a new AWS feature called CloudWatch cross-account observability using Terraform.

GOV.UK Pay runs on AWS and like many organisations, it has multiple AWS accounts to separate our test, staging and live environments, as well as separating our deployment and testing infrastructure. We use a combination of Cloudwatch for our AWS generated Infrastructure metrics and a SaaS product to aggregate our application metrics. One of the downsides with this setup is that engineers need to log into multiple AWS accounts and the SaaS UI to view metrics, making it difficult to correlate behaviour across environments. We're considering ways to improve our monitoring systems and a recently launched AWS feature called "cross-account observability" looked like it could help fix this problem. It allows a single AWS account to access instrumentation data from multiple AWS accounts. The potential benefits of this are: 

• it gives all technical staff on the team access to CloudWatch monitoring in one place, removing the need to have production access in order to view production metrics
• it avoids the need to log in and out of different accounts to view metrics
• we can potentially codify all CloudWatch alarms and dashboards in one place which could make for simpler code organisation
• we could make correlation and contextualisation of data easier
• there's no extra cost for logs and metrics — traces can be shared with one monitoring account at no cost, but will incur costs if shared with additional monitoring accounts — this will provide cost savings for us as our monitoring SaaS product pulls data from the CloudWatch API which comes at a cost

When exploring CloudWatch cross-account observability, we wanted to configure everything using Infrastructure as Code using Terraform and avoid having to manually configure ("click ops") anything at all. Unfortunately, all the examples found on the internet used the click ops method, and there is sparse documentation on how to configure it in Terraform. Thankfully, we were able to configure everything in Terraform and thought it would be good to share our solution with the world!

Click here for the full blog post