Nasdaq: Build Resilient Infrastructure for Global Financial Services (HMC327)

Session Notes

NASDAQ's Cloud Journey and Resilient System Architecture

• building resilient, high-performance systems for financial services.
• Overview of NASDAQ's cloud journey, disaster recovery strategies, hybrid infrastructure solutions, architectural considerations, and future outlook.
• NASDAQ's Background:
• Pioneer in electronic exchanges, operating over 30 exchanges globally.
• Provides software to over 130 marketplaces, including trading, clearing, risk management, and anti-financial crime solutions.
• Offers solutions as traditional managed software, managed services, and SaaS.
• Cloud Journey:
• Began 15 years ago, gradually migrating critical systems like matching engines and market systems.
• Progressed from T+1 backups to hard real-time systems, handling ultra-low latency transactions.
• Moved from less critical to systemically important systems for national economies.
• Challenges:
• Ultra-low latency transactions measured in microseconds and nanoseconds.
• Specific physical location requirements due to proximity and latency considerations.

Disaster Recovery Strategies and Hybrid Infrastructure

Resiliency and Regulatory Requirements

• High resiliency target with customer expectation of 100% uptime.
• Regulatory oversight across multiple jurisdictions globally.
• Disaster Recovery Spectrum:
• Ranges from backup and restore to multi-site, active-active, multi-region systems.
• Importance of aligning on RPO and RTO targets and understanding system responses to disasters.
• Focus on Critical Systems:
• Centered on systems that must live all the time and cannot go down.
• Hybrid compute with Outposts, failure domains, static stability, and disconnected operation.
• Hybrid Compute with Outposts:
• Use of Outposts to place workloads in data centers while managing them with EC2 API and Amazon infrastructure.
• Direct Connect circuits for connectivity between region and data center.
• Use of EC2 servers in data center racks, managed as a service.
• Addition of ultra-low latency network interface cards for market systems.
• Implementation of bare metal servers and networks for fully physically separate data planes.
• Bare metal servers
• physical, single-tenant server that provides a client with direct, exclusive access to all hardware resources, unlike virtual servers which share hardware via a hypervisor. This setup offers maximum performance, consistent I/O, and complete control over the software stack, making it ideal for high-performance computing, demanding applications like gaming or AI/ML, and workloads that require physical isolation or specific hardware configurations.

Failure Domains and Static Stability

Failure Domains

• Concept of failure domains to understand shared fate among system components.
• EC2 servers running software with multiple instances and hot spares (pre-provisioned EC2 instances) for redundancy.
• AB pairs of every software component, ensuring backup components are not co-located on the same instance.
• Rack Level Failure Domains:
• Consideration of potential rack-level failures (e.g., power loss).
• rack-level failures: a whole server rack goes offline due to power loss, cooling failure, network switch failure, or physical damage, impacting all servers within it.
• Multiple racks are used, with AB pairs of software components spread across different racks to avoid simultaneous loss of both components.
• Site Level Failure Domains:
• Multiple physical sites for data centers to ensure resilience.
• Primary site in New Jersey and secondary site in Chicago for US markets.
• Static Stability:
• Use of hot spares (pre-provisioned EC2 instances) to ensure immediate availability of backup resources.
• AB pairs of software components spread across racks and sites to maintain system integrity in case of failures.
• hot spares:
• pre-provisioned EC2 instances on AWS are primarily implemented through the EC2 Auto Scaling warm pools feature. This functionality allows you to maintain a pool of instances that are ready to be put into service immediately when your application needs to scale out.
• Disaster Recovery (DR) Strategy: Warm pools are an integral part of "warm standby" or "pilot light" disaster recovery strategies
• Faster Scaling: Instances in a warm pool are pre-initialized, meaning time-consuming application startup and configuration processes have already run.

Static Stability and Disconnected Operation

Static Stability

• Concept of static stability to ensure system resilience during disruptions.
• Use of hot spares (pre-provisioned DC2 instances) to avoid dependency on EC2 control plane during failures.
• Aim to maintain system functionality without changes during disruptions, relying on pre-configured resources.
• Disconnected Operation:
• Preparation for scenarios where connectivity to AWS services might be cut off (e.g., optical cable damage).
• Ensuring market systems can continue operating without relying on external services during such events.
• Use of hot spares and pre-configured resources to maintain functionality in disconnected scenarios.
• Operational Runbooks:
• Importance of having well-defined operational runbooks for each level of failure domains.
• Procedures for responding to server, rack, and site-level failures should be choreographed and familiar to operations staff.
• The goal is to have predictable and well-practiced responses to disasters, minimizing surprises during actual events.

Disconnected Operation and Testing

Disconnected Operation

• Preparation for scenarios where connectivity to AWS services is lost (e.g., network line cut).
• Use of local gateway for break glass access to manage systems locally without reliance on external services.
• Awareness of internal dependencies and operations that may require external connectivity, ensuring they are addressed or avoided.
• Testing Failure Modes:
• Conducting thorough testing by disconnecting outposts for extended periods to observe system behavior.
• Identifying and addressing operations that may fail during outages (e.g., SSL checks, IAM interactions).
• Ensuring local gateway remains accessible even during regional or service link issues.
• Summary of Strategies:
• Use of outposts in data centers for hybrid computers.
• Analysis of failure domains to understand component interdependencies and shared fate.
• Design and testing for static stability to maintain system functionality without changes during disruptions.
• Focus on disconnected operation to ensure systems can continue running locally if cut off from AWS services.
• Emphasis on careful planning, well-defined operational runbooks, and extensive testing to achieve resilience and avoid surprises during disasters.
• Break glass
• Access is a security measure that allows authorized personnel to bypass normal access controls and gain emergency access to critical systems, data, or functions. The term comes from the concept of breaking glass to access emergency equipment like a fire alarm. It is a fail-safe protocol used when standard methods fail due to incidents like security breaches, system malfunctions, or password lockouts. Break glass accounts are pre-configured with high privileges but are kept secure and dormant until an emergency arises, and their usage should be strictly monitored and documented.
• Emergency access: When a crisis occurs and standard access methods are unavailable.
• Bypassing controls: These accounts are designed to circumvent normal identity and access management (IAM).
• High-level privileges: The accounts are pre-configured with elevated privileges.
• Strict management: To prevent misuse, break glass credentials are managed with extreme care.

Future Directions and Enhancements

Dynamic Deployments

• Planning for more dynamic deployments with rules-based setups.
• Use of affinity and anti-affinity patterns to balance performance and resiliency.
• Anti-affinity patterns are rules in cloud/virtualization (like Kubernetes, VMware, OpenStack) that prevent specific workloads (VMs, Pods) from running together on the same physical host or in the same failure domain, ensuring high availability, fault tolerance, and load balancing by spreading them out, contrasting with affinity which tries to colocate things for performance.
• Goal to scale systems without downtime as markets move to public cloud.
• Experimentation with Advanced Hardware:
• Positive experiences with Graviton 4, collaborating with Amazon for next-generation CPUs.
• Testing ultra-low latency network cards for Graviton and other advancements.
• Exploration of accelerated compute with FPGAs and GPUs for real-time risk analysis.
• An FPGA (Field-Programmable Gate Array) is a reconfigurable integrated circuit that can be programmed to perform a wide range of digital logic functions after manufacturing.
• Expanding Outpost Capabilities:
• Interest in bringing more advanced components (custom silicon, FPGAs, GPUs) into outposts.
• Selling systems to third parties, aiming to offer more functionality and services.
• Careful Service Integration:
• Cautious approach to adding more services to outposts, conducting deep dives with service teams.
• Focus on understanding failure modes and ensuring high service availability and functionality.
• Conclusion:
• Continued efforts to enhance system resilience, performance, and availability.
• Emphasis on meticulous planning, testing, and collaboration with AWS to achieve future goals.
• Appreciation for the audience's engagement and encouragement to fill out the form for more information.

AWS Outposts within a Nasdaq data center to support ultra-low latency trading operations. The

main components and their functions are

• AWS Cloud and Region: The standard AWS environment providing services across multiple, isolated Availability Zones (AZs).
• AWS Direct Connect: A dedicated private network connection that links the Nasdaq data center to the main AWS region, used for control plane and service link connectivity to manage the Outposts.
• Nasdaq Data Center (On-premises): The physical location hosting the AWS Outposts, allowing workloads to run locally for low latency and regulatory compliance.
• AWS Outposts: Deployed within the Nasdaq data center, these are fully managed AWS compute and storage capacities that extend AWS infrastructure, services, and APIs to the on-premises environment.
• Amazon EC2 Instances with Nitro and ULL NICs: The computer instances running the trading applications. They leverage the AWS Nitro System for enhanced performance and security and use Ultra-Low Latency (ULL) Network Interface Cards (NICs) to meet the strict latency requirements of financial trading.
• Local Gateway: This component of the Outpost connects the on-premises network to the instances within the Outpost's VPC subnet, facilitating communication with existing Nasdaq infrastructure.

Nasdaq Networks

• ULL Trading Network: A physically separate network designed for ultra-low latency, directly interfacing with the ULL NICs on the EC2 instances for rapid transaction processing.