Amazon Aurora HA and DR Design Patterns for Global Resilience (DAT442)

Session Notes

Aurora's Approach to Performance

Adding replicas for performance purposes

• Offloads read traffic to replicas, reducing impact on the primary writer.
• Allows separation of noisy neighbors by dedicating replicas to specific workloads (e.g., analytics).
• Up to 15 replicas can be added for fine-grained performance management.
• Storage Scaling:
• Storage capacity and performance scale automatically in Aurora, requiring no manual provisioning.
• Instance Scaling:
• Instances can be added in various sizes (e.g., 16XL, 8XL, 4XL) for failover and performance management.
• DB instances are elastic, scaling from 0 to 256 Aurora capacity units.
• Failover tiers are defined to manage instance failover order based on size and capacity.
• Elastic instances help manage spikes in workload without long-term cost implications.

Challenge: Connection Management in Aurora

• Managing connections to multiple instances, especially identifying the writer for transactions.
• Techniques:
• Pooling and limiting connections per DB instance, avoiding reconnections.
• Driver Frameworks:
• Standard practices like Spring and JDBC may not handle clustered databases effectively.

Aurora Endpoints

Writer Endpoint

• Always points to the current writer node, facilitating consistent transaction routing.
• Reader Endpoint:
• Points to a pool of instances for read operations using DNS round robin for coarse load balancing.
• Custom Endpoints:
• Option to create custom endpoints for specific use cases (e.g., analytics).

AWS Advanced Wrapper Drivers

Rapid Drivers

• Enhanced drivers with intimate knowledge of cluster state, facilitating quicker failover.
• RDS Proxy:
• Additional tool for managing connections and reducing failover time.
• Plugins:
• Enhanced failover, rewrite splitting, blue-green deployments.

Multi-Region Resilience

• Challenge: Ensuring business continuity if a region fails or is cut off from the network.
• Solution: Maintaining a copy in another region, keeping it up-to-date with minimal cost, and utilizing it for various purposes.
• Multi-Region Resilience with Aurora Global Database:
• Aurora Global Database allows up to 10 secondary regions for AMS and APG.
• The internal replication server handles asynchronous physical replication between regions.
• Head nodes are not involved, avoiding performance associated with logical replication.
• Configuration and Benefits:
• Region A: Primary region with replicas and storage.
• Region B: Secondary region with storage initially, can add read-only instances for local reads.
• Asymmetric configurations are possible; instances in secondary regions do not need to match primary.
• RPO lag is typically about 1 second, depending on region pair distance.
• Global Endpoint and Failover:
• Global endpoint manages primary region identification.
• In case of failure in Region A, failover to Region B is managed, with potential data loss due to asynchronous replication.
• Global endpoint is repointed using Route 53 data planes for quick redirection.
• Switchover:
• Switchover is performed when both regions are healthy.
• Region A is told to stop being primary, and Region B becomes primary.
• Allow quick transition between regions without disruption.
• Switchover Process:
• Inserts a special log record (marker log record) into the log record stream.
• The other side recognizes the marker and becomes the primary at the specified log time.
• Region A stops being primary at the same time, ensuring a clean handoff.
• Results in zero RPO and very low RTO (about 30 seconds).

Maintenance and Upgrades

• Version upgrades (e.g., PostgreSQL 16 to PostgreSQL 17) and OS patches are necessary.
• Self-managed systems require downtime, snapshots for rollback, compatibility testing, and performance testing.
• Aurora provides a fully managed and automated minor version and patch upgrade experience.
• Rolling operating system upgrades are now available, increasing availability.
• Maintenance actions can be defined within a specified window or automatically managed by Aurora.
• AWS Health Dashboard provides maintenance notifications.
• Opt-in automation for maintenance is strongly recommended.
• AWS Organization's Upgrade Roll-out Policy:
• New feature to manage upgrades across multiple DB clusters in development, QA, and production environments.
• Simplifies the automation of the lifecycle management for fleets of DB clusters.
• Allows setting upgrade policies based on tags across multiple resources in an AWS organization.
• Example:
• Resources tagged Prod are upgraded last; dev resources are upgraded first.
• Default behavior: Resources without recognized tags are upgraded second.
• Upgrade rollout involves assigning the policy to resources, determining upgrade order, and proceeding in waves within maintenance windows.
• Provides flexibility to halt upgrades if issues are detected during the maintenance window.
• In-Place Upgrades:
• For major upgrades incompatible with open source, in-place upgrades are performed.
• Involves creating a clone of the database cluster, upgrading the clone, and testing with the application.
• Once validated, the clone is discarded, and the upgrade is applied to the production cluster.
• Blue-Green Deployments:
• More flexible approach for major upgrades, schema changes, or static parameter changes.
• Involves creating a new environment ( Green ) while keeping the old environment ( Blue ) in sync using logical replication.
• Allows testing the upgrade in the green environment before promoting it to production.
• Switchover renames resources to maintain consistency, completing in as little as a minute.
• Blue-green deployments are now available for global databases across multiple regions.
• Summary of Aurora APG and AMS Features:
• Addition of replicas to increase availability and enable failover.
• Use of AWS Advanced Rapid Drivers for smoother failover processes.
• Global Database for durability and availability across up to 10 secondary regions.
• Low latency local region reads and simplified endpoint management with global endpoints.
• Managed and automated cluster-wide upgrades, fast clones, and blue-green deployments for maintenance.

Introduction to Aurora DSQL

• Amazon Aurora DSQL is the fastest serverless distributed SQL database for always available applications. Aurora DSQL offers the fastest multi-Region reads and writes. It makes it effortless for customers to scale to meet any workload demand with zero infrastructure management and zero downtime maintenance.
• Combines the best of relational databases, distributed database architectures, and serverless services.
• Designed to run complex queries, scale out, and offer pay-per-use pricing.
• DSQL Architecture Overview:
• Comparison with self-managed PostgreSQL running on EC2.
• DSQL uses a query processor per connection, similar to PostgreSQL backend processes.
• Read queries are sent directly to the DSQL storage engine.
• Writes are buffered in memory on the query processor until commit.
• Upon commit, transactions are sent to the Adjudicator for concurrency control and then to the Journal for durability, replicating to at least 2 AZs.
• The Journal is used to update storage, similar to traditional PostgreSQL.
• DSQL is an active-active service, allowing any connection to read or write data at any time.
• Query processors (QPs) can run on different hosts or availability zones.
• Availability in DSQL:
• If a database node fails, the Query processors (QPs) detect the failure and shift traffic to a healthy replica.
• Replicas can be in the same or different availability zones.
• In-flight operations may experience a small increase in latency due to retries.
• DSQL continuously backs up data and automatically replaces failed nodes.
• Replacement nodes connect to the Journal to catch up with the latest data.
• Traffic is shifted back to the local replica for optimal performance.
• Concurrency Control and Durability:
• The Adjudicator handles concurrency control and conflict resolution.
• A single Adjudicator allows for quick conflict resolution without network coordination.
• Standby adjudicators are kept in other availability zones for failover.
• In case of failure, standby adjudicators run a leadership protocol to elect a new leader.
• Query processors retry in-flight commits against the new leader adjudicator.
• The process is seamless and requires no action from the application.

High Availability in DSQL

• DSQL is designed with no single point of failure.
• Active-active architecture provides high availability out of the box.
• DSQL is designed to scale automatically to handle increasing load.
• Concurrency Control Protocol:
• DSQL uses optimistic concurrency control, assuming transactions seldom conflict.
• Conflicts result in a serialization failure error and require the client to retry the transaction.
• Applications should handle retries and optimistic concurrency control issues.
• Recommended to build helper functions or incorporate handling into the lowest application layer.
• Avoiding Unnecessary Conflicts:
• Design applications to minimize unnecessary conflicts to reduce retries.
• Transactions updating different rows can commit simultaneously without conflict.
• Internal Sharding and Scaling:
• DSQL automatically shards itself internally to handle increased load on the adjudicator and journal.
• Query processors direct commits to the appropriate shard.
• Optimized two-phase commit ensures atomic transactions across multiple shards.
• DSQL automatically scales out by bringing new nodes online and load balancing across replicas.
• Replicas in DSQL are always strongly consistent, enabling seamless scaling.
• Consistency in DSQL:
• DSQL ensures strong consistency through a time-based synchronization protocol.
• Uses EC2 Time Sync Service for microsecond-accurate timestamps and Clockbound library to correct measurement errors.
• Guarantees a linearizable timestamp that is always after any previously committed timestamp.
• Storage waits for updates from the journal before returning rows to ensure consistency.

Two-Phase Commit (2PC) Protocol Overview

• 2PC is a distributed system protocol ensuring all participants in a transaction either commit or abort together.
• It consists of two phases: Prepare and Commit/Abort.
• Phase 1: Prepare Phase:
• A coordinator sends a "prepare" request to all participating nodes.
• Each participant performs the transaction's work, acquires necessary locks, and writes a "prepared" record to its transaction log.
• Participants send a "yes" or "no" vote back to the coordinator.

Phase 2: Commit/Abort Phase

If all participants voted "yes"

• The coordinator writes a "commit" record to its log and sends a "commit" message to all participants.
• Participants commit the transaction, release locks, and send an acknowledgement to the coordinator.
• If any participant voted "no":
• The coordinator writes an "abort" record to its log and sends an "abort" message to all participants.
• Participants roll back the transaction and release any locks.
• Key Considerations: Atomicity:
• 2PC guarantees that a transaction is atomic, either complete on all nodes or a complete failure on all nodes.
• Consistency:
• Ensures data consistency across multiple nodes, but may impact availability due to blocking and waiting for a decision.
• Failure Handling:
• Includes mechanisms to recover from failures; a coordinator failure can lead to an "in-doubt" state requiring manual intervention.

Connection Management in DSQL

• Connections to DSQL are managed by the session routing layer.
• Ensures quick availability of connections, even during network events causing fluctuations.
• Maintains a warm pool of query processors (QPs) ready for immediate use.
• When a connection is requested, DSQL assigns an available QP from the pool.
• Handling Connection Failures:
• In the event of a connection failure, the application must detect the failure and retry.
• The retry handler from the optimistic concurrency control section is useful for this purpose.
• Applications should implement retry logic for connection errors to maintain availability.
• Unlike storage and adjudicator failures, applications must manually reconnect failed connections.
• DSQL enforces a 1-hour maximum lifetime for connections to encourage proper connection management.
• Best Practices for Connection Management:
• Use client-side connection pooling libraries (e.g., Java C3PO) with health checks and maximum connection age.
• DSQL inherently includes connection pooling, making server-side pooling (e.g., PG Bouncer, RDS Proxy) unnecessary.
• Recommended to connect to QPs in the same availability zone for lower latency.
• DSQL automatically fails over to a healthy zone if the current zone is unavailable.

Multi-Region Clusters in DSQL

• DSQL supports multi-region clusters by creating clusters in two regions and designating a third region as the witness region.
• The witness region holds a copy of the journal to facilitate quorum-based protocol for regional failures.
• Applications can remain active across regions, with both regions capable of handling reads and writes.
• A quorum-based protocol
• is a method used in distributed systems to ensure data consistency, fault tolerance, and reliability by requiring a minimum number of participating nodes (a quorum) to agree on an operation before it is considered valid.
• S3 Quorum-based Protocol:
• In distributed systems, "Quorum" means "a minimum number of members required to constitute a meeting."
• The Quorum mechanism is a voting algorithm for ensuring data redundancy and consistency in distributed storage systems like Amazon S3.
• Core Concepts and Terminology:
• Read Quorum (Vr): Minimum number of read votes
• Write Quorum (Vw): Minimum number of write votes
• Total Votes (V)
• Working Principle:
• Assume there are V data replicas in a distributed system.
• To ensure data consistency, read and write operations must obtain a "quorum" of node confirmations.
• Core rules:
• Vr + Vw > V: Ensures the same data is not read and written simultaneously, avoiding read-write conflicts.
• Vw > V / 2: Ensures serialized data modifications, preventing two write operations from modifying data simultaneously.
• Adjusting Vr and Vw allows the system to balance between read and write performance.
• About Amazon S3:
• Amazon S3 announced in 2020 that all GET, PUT, and LIST operations have strong consistency.
• Data written can be immediately read as the latest version, simplifying big data workloads.
• S3 guarantees high durability and availability through its architecture, which stores data replicas across at least three availability zones.

Handling Regional Failures

• In the event of a regional failure (e.g., region B fails), the witness region helps disambiguate the failure and votes to exclude the failed region from the quorum.
• DSQL automatically reconfigures the journal to exclude the failed region, ensuring the cluster remains available in the healthy region (region A).
• There is no data loss, and transactions continue to be replicated to at least one other AWS region.
• When the failed region (region B) comes back online, DSQL detects it, updates it to the latest state, and restores the multi-region configuration.
• Recommended pattern: Deploy applications alongside clusters in both regions for active redundancy.
• Benefits of Active Redundancy:
• Deploying applications alongside clusters in multiple regions provides low latency and continuous validation of application functionality.
• Use a global endpoint (e.g., Route 53 DNS record) to direct traffic to the closest available healthy region.
• Handling Regional Failures with Global Endpoint:
• In the event of a regional failure, data is not lost, and DNS automatically directs traffic to the healthy region.
• Continuous validation ensures the healthy region is operational.
• Maintenance in DSQL:
• DSQL is a fully managed service with no maintenance required from customers.
• Query processes are periodically replaced with updated versions, ensuring the latest OS, security patches, performance improvements, and features.
• Customers do not need to perform any maintenance tasks.
• Takeaways:
• Aurora offers three engines: PostgreSQL, MySQL, and DSQL.
• All engines provide 3 AZ durability out of the box.
• Scaling options vary: APG and AMS scale up for writes and out for reads; DSQL offers hands-free, automatic scaling.
• Multi-region considerations: APG and AMS allow testing failover without data loss; DSQL supports active-active across regions.
• All engines offer easy maintenance with automated security updates and minor version upgrades.