Government Digital Service (GDS)
 Printable version
|
How we migrated our PostgreSQL database with 11 seconds downtime
- Also published by:
- Cabinet Office
Blog posted by: David McDonald, 17 January 2024 – Categories: GOV.UK Notify, Posts for Tech Specialists.
GOV.UK Notify is hosted on the GOV.UK Platform as a Service (PaaS). The PaaS is being retired, so we are migrating all of our infrastructure into our own Amazon Web Services (AWS) account. This blog post explains how we migrated our PostgreSQL database with minimal downtime.
Migrating our database
The PaaS provides a database for us and we use it to store all of our data - from data about each notification we send to the content of the hundreds of thousands of templates service teams use to send those notifications. This is an AWS RDS PostgreSQL database and it lives in the PaaS’ AWS account. Our apps that run in the PaaS talk to this database. We are going to call this database our ‘source database’.
We needed to set up a new database in our own AWS account, and get all of our apps to talk to the new database. We are going to call this new database our ‘target database’.
Creating a new PostgreSQL database in our own AWS account is not too difficult. The hard part is transferring all of our data and getting our apps to use this new database, whilst incurring minimal downtime.
A bit more about our source database
Our source database is about 400GB in size. It has about 1.3 billion rows, 85 tables, 185 indexes and 120 foreign keys. It is PostgreSQL version 11.
On a usual weekday, we do somewhere in the region of 1,000 inserts or updates per second (sometimes much lower, sometimes much higher), plus a similar number of reads.
GOV.UK Notify sends millions of important and timely notifications each day, from flood alerts to updating users about their passport applications . Every notification we send requires talking to our database. Therefore it’s important that we minimise any downtime.
AWS Database Migration Service
The PaaS team offered us the ability to migrate databases using AWS Database Migration Service (DMS).
DMS is responsible for transferring data from our source database to our target database. It can be run in either the source or target AWS account.
DMS works by:
- Copying across all of the data, table by table, up to a specific point in time. This is known as the ‘full load’ task.
- Entering replication mode, where it ensures that all new transactions on the source database are replayed onto the target database, so that the 2 databases are in sync.
We would then be responsible for getting our apps to stop talking to the source database and start talking to the target database.
Click here for the full blog post
Original article link: https://gds.blog.gov.uk/2024/01/17/how-we-migrated-our-postgresql-database-with-11-seconds-downtime/
