Improving Distributed Systems with Amazon Time Sync Featuring Nasdaq

Session Notes

• Quote from Barbara Lipskovoff: Synchronized clocks can improve distributed system performance by reducing communication.
• Purpose of Clocks
• Clocks as a common measurement scale.
• Traditional use in distributed systems: not trusted, used for human observability.
• Applications:
• Log messages, metrics, monitoring, UI dashboards.
• Auditing in industries like financial services, healthcare, broadcast media, online gaming.
• Potential in distributed systems to simplify operations, reduce locking, and leader election with reliable clocks.

What is a Clock?

Basics of building a computer clock

• Requires something that ticks periodically and something that counts these ticks.
• Typically a single hardware piece combining both functions.
• Equation Representation:
• At time T, clock C can be read as the number of elapsed cycles divided by frequency and a constant.
• Simple Example:
• Counting 1000 cycles with hardware ticking at 1 cycle per second results in 1000 seconds elapsed.
• Conclusion:
• Fundamental component for building a clock on a computer.

Hardware Options for Clocks

Range of Hardware

• From small form factor crystal oscillators to large equipment like caesium beams, hydrogen masers, and upcoming optical clocks.
• Form factors vary from a few millimeters to cabinet-sized equipment.
• Price Points:
• small form factors are more cost-effective for cloud computing.
• large equipment can cost hundreds of thousands of dollars.
• Quality and Accuracy:
• Accuracy in maintaining time within one microsecond varies:
• Small form factor: accurate for a couple of seconds.
• Hydrogen masers and optical clocks: accurate for years, potentially centuries.

Clock Drift

All Oscillators Drift

• Even the most accurate atomic clocks experience drift.
• Drift at the microsecond level may take centuries to observe in highly accurate clocks.
• Drift Causes:
• Manufacturing process, aging (concerns for physicists).
• For cloud providers: small differences in power supply or temperature variations due to workload impact on CPUs and servers.
• Impact of Drift:
• Frequency of cycles changes over time, leading to clock drift.
• Example: An EC2 instance running for 5 days showed a 75 millisecond drift without correction.
• Addressing Clock Drift
• Tracking and Correction:
• Similar to humans adjusting a wristwatch, computers need to track and correct for drift.
• Network Time Protocol (NTP):
• Used for decades to synchronize clocks.
• Involves a client clock sending a message to a server (reference clock) to adjust time.

Challenges in Clock Synchronization

Continuous Adjustment Needed

• The frequency of clock cycles changes constantly.
• Requires repeatedly asking for the correct time, not just once.
• Network Communication Issues:
• Messages exchanged over networks have varying delays.
• Risk of message failure when they are dropped.

Network as Part of the Problem

Viability and Delays

• Noisy communication delays make it hard to discern actual clock drift.
• Network Asymmetry:
• The network is not symmetrical, leading to differences in time observed even with perfect clocks.
• Difficult to distinguish between clock error and network asymmetry.

AWS Solution: Amazon Time Sync Service

• Address the complexities of clock synchronization over non-deterministic networks for customers.
• Implementation:
• Introduced in 2017.
• Utilizes NTP (Network Time Protocol) endpoint.
• Available locally for every EC2 instance worldwide.
• Investment:
• Significant engineering and hardware investment to support this service globally.

Amazon Time Sync Service

Basic Setup

• EC2 instance with Nitro system and software time daemon.
• NTP packet from AWS distribution system through Nitro system to correct server clock.
• Accuracy:
• NTP service accuracy: under 1 millisecond, typically 500-700 microseconds.
• Customer Benefit:
• Easy, hands-off service; no need for customers to understand underlying technology.

Challenges and Solutions for Nanosecond Accuracy

Customer Demand

• Customers sought better than 1 millisecond accuracy for distributed systems.
• AWS Approach:
• Invested in hardware for nanosecond-level accuracy.
• Used dedicated cables from GPS and redundant atomic hardware clocks.
• Signal (GPS) sent directly to EC2 instances, bypassing AWS networks.
• Dedicated Clock Synchronization:
• Signal for clock synchronization is entirely dedicated, not over data or control plane.

Nitro-Based System

• Nitro clock: hardware reference clock.
• NTP packet communication: directly to Nitro card and back, no network involvement.
• PTP Hardware Clock:
• Added Precision Time Protocol (PTP) hardware clock in Nitro system.
• Users read PTP hardware clock (PHC) to correct system clock locally.
• Accuracy Levels:
• NTP: under 100 microseconds at software level.
• PTP: around 20 microseconds at software level.
• Widespread Implementation:
• Reference PTP hardware clocks added across various server types (Graviton, Intel, AMD, GPUs).
• Based on Nitro system hardware, supporting global infrastructure investment.

Hardware Packet Timestamping

Key Application and Benefits

Purpose

• Enables nanosecond-level accuracy for timestamping network packets.
• Benefits:
• Improved observability of network performance for distributed workloads.
• Allows ordering of messages received by the server with nanosecond resolution.
• Built-in facility in the driver, accessible via classic socket API or DPDK toolkit.

How It Works

Before Feature Release

• Application layer timestamping:
• Message arrives, passes through Nitro system, reaches instance.
• Application reads time from system clock and attaches timestamp to message.
• Use 4 lines of C code using socket API.
• After Feature Release (Hardware Packet Timestamping):
• Nitro system and Nitro clock:
• Message arrives, packet is timestamped by Nitro system and Nitro clock before reaching instance.
• Timestamped messages are passed to application layers.

Implementation and Access

Access Methods

• Classic socket API.
• DPDK (Data Plane Development Kit) toolkit for direct packet access bypassing kernel infrastructure.
• Resolution:
• Provides nanosecond resolution on received timestamps for network packets.

Example Comparison

Before

• Simple process involving reading time from system clock after message reception.
• After:
• Enhanced process with packet timestamping done by hardware before the message reaches application, providing more accurate timestamps.

Enhanced Benefits of Hardware Packet Timestamping

Increased Accuracy

• Provides a more accurate view of network performance.
• Example: Measuring round trip time between two instances.
• User space application measurement: ~400 microseconds.
• Hardware packet timestamping measurement: ~100-150 microseconds.
• Observable Difference:
• Difference of ~250 microseconds attributed to time spent in the operating system and network stack.
• Represents about 2/3 of the round trip time, highlighting more precise observability.
• Crisper Network Performance:
• Allows identification and breakdown of delays between application and network.

Introduction to NASDAQ's Use Case

Objective

• To demonstrate NASDAQ's progress towards running markets in a public cloud environment using AWS's precision time technology.

NASDAQ Background

Innovator in Financial Markets

• World's first electronic exchange, created in 1971.
• Operates over 30 markets globally, including stocks, bonds, options, and derivatives.
• Known for hosting top technology companies by market cap, including Amazon.
• Technology Provider:
• Serves over 130 market operators and 2300 financial institutions.
• Supports over 44,400 public companies listed on NASDAQ exchanges globally.
• Provides technology to 6000 corporates for capital market lifecycle support.

NASDAQ's Cloud Journey

Started Over 10 Years Ago

• Began with data repositories and data warehousing using AWS Redshift (2012-2013).
• Currently stores about 60 petabytes of data, supporting back-office functions.
• Evolution to Real-Time Processing:
• Market surveillance using streaming real-time market information.
• Market data distribution available through the public cloud.
• Launch of Options Exchange on AWS Outpost:
• In 2022, launched NASDAQ MRX on AWS Outpost.
• Currently runs 6+ market systems on Outpost.
• Critical Systems in Public Cloud:
• Hosts systems providing real-time insights to market participants.
• Leverages cloud scalability, performance, and right-sizing for large message volumes and peak managed capacity.

Specific Challenges in Moving to the Cloud

Geographic Constraints

• Trading primarily occurs in northern New Jersey for equities and equity options.
• Latency dependencies require computer deployment in specific locations.
• Ultra Low Latency Transactions:
• Transactions measured in order-to-act time of ~20 microseconds.
• High Message Volumes:
• Processes north of 100 billion messages a day.
• Specific exchange systems handle up to 2-3 million messages per second.
• High Resiliency and Uptime Expectations:
• Critical for efficient and well-operating capital markets.
• Primary listing venue for numerous public companies.

NASDAQ Exchange System Architecture

Matching Engine

• Handles primary logic for booking orders, trades, and matching buyers and sellers.
• Order Ports:
• Interface for customers sending order flow to the exchange.
• Performs validation, pre-trade risk, and other functions.
• Public Market Data Feeds:
• Publishes view of matching engine activity to the market (e.g., NASDAQ I).
• Clients:
• Provide order flow to order ports and matching engine through inbound transactions.

Traditional On-Premises Model

Objective

• Provide a deterministic and statistically fair market to participants.
• Reliance on Physical Characteristics:
• Lengths of fibers in the data center.
• Number of switch hops between system components and customers.
• Sources of Network Latency:
• D1 and D2:
• Primary sources of network latency due to physical setup.
• D3:
• Encompasses both network and processing components.
• Approximately half of the round trip latency, roughly 10 microseconds.
• Orderly Processing:
• Sequential processing of the matching engine ensures orders are processed consistently and repeatably as submitted by customers and returned to them and the public market data feed.

Challenges in Public Cloud

• Unpredictable Network Conditions.
• No fixed cable lengths or known switch hops.
• Potential delays from other network traffic or unpredictable node placements.

Solution: Hardware Timestamps and Reordering Buffer

Key Time Periods

• Focus on D1, D2, and D3 to understand transaction times through the network.
• Hardware Timestamps:
• Record time of each transaction at each network point using nanosecond precision.
• Provided through Nitro and Amazon Time Sync Service.
• Reordering Buffer:
• Sits in front of the matching engine.
• Focuses on ordering inbound transactions (similar techniques apply for outbound flow).
• Time-Based Boundary:
• Define using hardware timestamps (e.g., T1, T2, T3 for orders from clients).
• Ensures transactions are processed in the order they originated.
• Patented Technique:
• Foundational for evolving NASDAQ's posture towards public cloud for low latency systems.

Prototype and Experimentation

AWS Instances Used

• M7I.2XL instances.
• Proximity Placement Groups:
• Ensured compute nodes were placed close together within a few racks or the same rack.
• Hardware Timestamps for Reordering:
• Used to reorder packets based on timestamps.
• Testing with Time Boundaries:
• The 5 microsecond boundary reordered ~25% of packets.
• 50 microsecond boundary reordered up to 84% or higher of packets.
• Iteration and Evolution:
• Continued work on proximity placement functions and Amazon Time Sync Service for greater granularity.

Strategic Vision

Scalable Trading Systems

• Aim to develop scalable and high-performance systems whether deployed in public cloud or on-premises.
• Innovation in Public Cloud:
• Embrace cloud innovation, including AI and data migration to public cloud.
• Enhanced Security and Reliability:
• High expectations for availability, security, and reliability.
• Public cloud offers significant advantages if built responsibly with resiliency considerations.
• Leveraging Market Expertise:
• As a leader operating 30 exchanges globally and supplying technology to 130 marketplaces, NASDAQ thinks strategically about applying technology.

Leveraging Good Clocks in Software

• Emphasizes reducing communication in distributed systems using synchronized clocks.
• AWS Time Sync Service:
• Provides nanosecond-range clocks with microsecond accuracy in software.
• Hardware Packet Timestamping:
• Allows applications to benefit from more accurate timing, enhancing performance and observability.

Challenges in Distributed Systems

Multiple Clocks

• Distributed systems have multiple servers, each with its own clock.
• Clocks are not perfectly synchronized, leading to complexity in ordering events.
• Traditional Solutions:
• Distributed locks, leader election, and consensus algorithms used for ordering.
• These algorithms limit performance and increase complexity as systems grow.

Clock Uncertainty and Margin of Error

Human Perspective

• Humans think of time as precise (e.g., 2 p.m.), but clocks have uncertainty.
• Precision Clocks:
• AWS precision clocks in the software layer have uncertainty (e.g., ±20 microseconds).
• Margin of Error in Applications:
• Applications must factor in this margin of error.
• Example: Bank account transactions require correct ordering to avoid overdrafts.

Estimating Clock Uncertainty

Complex Question

• Customers need to understand the intervals of uncertainty for their clocks.
• Example:
• Inspect time synchronization daemon on an EC2 instance to estimate error.
• Sources of time synchronization:
• Internet NTP servers (e.g., time.aws.com): ~395 microseconds uncertainty.
• Local NTP servers via Nitro system: ~90 microseconds uncertainty.
• PTP hardware clock (enhanced Amazon Time Sync Service via Nitro system): More accurate.

AWS Simplification

• AWS aims to simplify understanding and managing clock uncertainty for customers.
• Detailed explanations provided to help customers without needing deep technical knowledge.

AWS Clockbound Software

Purpose

• Simplifies understanding and managing clock uncertainty for customers.
• Availability:
• Open-source, available on GitHub since 2021.
• Key Features:
• Provides three pieces of information in one operation:
• Current Time: E.g., Monday at 2:00 p.m.
• Window of Uncertainty: E.g., ±20 microseconds.
• Clock Status: Initialized, free running, or synchronized.
• Support and Scalability:
• Supports PTP hardware and NTP sources.
• Scales to demanding workloads, capable of millions of queries per second.
• Language and Compatibility:
• Written in Rust for performance and memory safety.
• Available in RSS as a cloud and for C applications.

Clock Error Bound

• The tightest bound on the worst-case clock error due to oscillator drift and communication delays.
• Clockbound's Role:
• Provides accurate answers despite accumulated errors, shining in nanosecond and microsecond level accuracy scenarios.
• YugabyteDB Usage of Clockbound
• Open-source database using AWS's open-source Clockbound software.
• Hybrid Logical Clock System:
• Combines traditional logical ordering with precision clocks.
• Benefits Observed:
• Reduced Latency: Customers experienced 3 times less latency.
• Increased Throughput: Transaction throughput doubled.
• Fewer Retries: 1000 times fewer retries due to reduced conflicts between transactions (e.g., read and write).
• Improved Customer Experience:
• Enhanced consistency, isolation, and overall user experience in the database.

AWS's Use of Precision Clocks

Services Utilizing Precision Clocks

• Aurora DSQL: Multi-region, global-scale database.
• Dynamo DB Global Tables: Multi-region, global-scale database.
• Reason for Use:
• Network latencies across regions (e.g., 70 milliseconds between US West 2 and US East 1).
• Advantage of Precision Clocks:
• Clocks in microseconds to nanosecond range allow for faster transaction ordering than sending packets over the network.

AWS's Use of Precision Clocks in Aurora DSQL

Multi-Region Database Consistency

• Critical for maintaining consistency and isolation in a multi-region database.
• Clockbound's Uncertainty Window:
• Used to manage events across different regions.
• Example: Event 1 is guaranteed to be in the past compared to Event 2 or 3.
• Events 2 and 3 have overlap and need careful processing to avoid retries.
• Possible Solutions:
• Delaying the start of an event (e.g., Event 3) to prevent retries.
• Database Design Considerations:
• Depends on system design, type of events, and customer requirements.

New EST Daemon Release

Purpose

• To simplify the use of advanced timing features for customers.
• Improvements:
• Built under the same software stack as the existing Clockbound software.
• Significant improvements for higher accuracy and reliability.
• Customer Trust:
• AWS aims to maintain and improve customer trust and experience over time.
• Stand-Alone Time Client:
• Clockbound now includes a stand-alone time client and daemon.
• Synchronization and Information Provision:
• Synchronizes the operating system clock.
• Provides current time, size of uncertainty window, and clock status to applications.
• Regulatory and Observability Reasons:
• Important for regulatory compliance and system observability.
• Built-In Metrics and Observability:
• Observability is a first-class citizen with built-in metrics.
• High Performance and Memory Safety:
• Built on the same Rust software stack for memory safety and high performance.
• Achieves high performance through a shared memory segment for inter-process communication.

Cloud-First Design Paradigm Shift

• Redesigned system to be cloud-first.
• Moves away from traditional feedback loop systems to a feed-forward information system, better suited for cloud applications.
• Simulator for Testing
• New software stack ships with a simulator.
• Allows users to test the software, report issues, and suggest improvements.
• Cloud-Native Synchronization
• Availability is a top priority across all layers of virtualization.
• Contributed a new feature to the Linux kernel called VM clock (virtual machine clock).
• Combination with New Clockbound Release:
• Allows direct transfer of time from the Nitro clock into the instance.
• Improves performance during maintenance events, ensuring smooth workload operation.
• Performance of the New System
• The system manages to track and correct drift to within ±1 microsecond over multiple hours.
• Window of Uncertainty:
• The size of the window of uncertainty plotted by querying the Clockbound daemon.
• Sawtooth behavior: grows and shrinks as the clock gets updated.
• Achieved window of uncertainty: between 20 and 26 microseconds within the application.

Equities and options in Northern New Jersey are significant due to its role as a major financial data hub (like Equinix in Secaucus) powering Wall Street trading, proximity attracting financial professionals, and a robust local economy with companies like NJR offering stable dividends, all supported by strong infrastructure and commuter links to NYC, creating a diverse investment

landscape from high-tech data centers to established companies.

Why Northern NJ is a Hub for Equities & Options

• Data Center Powerhouse: Secaucus, NJ, hosts key data centers (like Equinix NY4) that are critical for high-frequency trading, moving activity from Manhattan and making NJ a central location for financial tech.
• NYC Proximity: Excellent transportation (trains, ferries) connects NJ to NYC, attracting financial professionals and making it a prime residential area for those working in Manhattan's financial district, boosting local economies in places like Bergen & Hudson Counties.
• Strong Local Companies: New Jersey has established companies, such as New Jersey Resources (NJR), that are attractive for dividend-focused equity investors, noted for consistent dividends and value metrics.
• Investment Ecosystem: The region benefits from strong real estate appreciation (a form of equity), stable demand, and a growing "innovation ecosystem" supported by the NJEDA (New Jersey Economic Development Authority), fostering both traditional and tech-driven investment.
• Tax & Regulatory Environment: New Jersey has specific regulations, like those for nonresident stock option income, indicating a sophisticated financial regulatory framework that impacts investors.
• What this means for Investors
• Data & Tech Focus: Opportunities exist in tech-related areas and data infrastructure supporting finance.
• Value & Income: Look for stable, dividend-paying companies based in or near the state, alongside property investment (real estate equity).
• Commuter Appeal: Areas with strong commutes to NYC (e.g., Weehawken, Bergen County) see high demand, representing a solid real estate equity play.

Equinix's Secaucus, NJ campus (NY2, NY4, NY5, NY6) is a crucial low-latency hub for Wall

Street, offering direct connections to major financial exchanges and trading ecosystems,

housing numerous financial firms, and providing high-security, reliable colocation for

high-frequency trading (HFT) and essential data services like Dow Jones news feeds, ensuring

speed and proximity to markets. These facilities act as the digital backbone, allowing firms to

place their servers close to exchanges for faster trading.

Why it Matters for Wall Street

• Low Latency: Physical proximity to exchanges minimizes data travel time, critical for HFT.
• Market Access: Direct connections to trading venues, data providers (like Dow Jones), and other financial participants.
• Reliability: Designed with robust power, cooling, and security for uninterrupted financial operations.

Amazon Aurora HA and DR design patterns for global resilience

(DAT442)

Resilience definition

• Ability of a workload to recover from disruptions
• Involves dynamic acquisition of resources and mitigation of disruptions
• High Availability (HA):
• Proportion of time a workload is available
• Measured historically (e.g., 5 9s availability)
• Disaster Recovery (DR):
• Techniques to recover workload after incidents
• Recovery Time Objective (RTO): Maximum time to resume service
• Recovery Point Objective (RPO): Maximum data loss acceptable
• Focus on Aurora:
• Aurora PostgreSQL (APG) and Aurora MySQL (AMS)
• Similar features and resilience capabilities

High Availability (HA)

• Goal: Keep systems running continuously, minimize disruptions, and avoid single points of failure.
• How: Redundancy, automatic failover, load balancing, often within the same data center or region.
• Examples: A web server with multiple instances, a database cluster with automatic switchover if one node fails.
• Focus: Fault tolerance for minor, frequent issues (e.g., hardware failure, maintenance).
• Disaster Recovery (DR)
• Goal: Restore business operations after a major disaster (natural disaster, massive data corruption).
• How: Backups, replication, offsite data centers, defined recovery procedures (RTO/RPO).
• Examples: Restoring data from cloud backups, failing over to a secondary region after a regional outage.
• Focus: Resiliency against large-scale, catastrophic events.
• Key Differences & Relationship
• Scope: HA handles small failures; DR handles big ones.
• Approach: HA is proactive; DR is reactive.
• Integration: They are complementary; robust HA reduces the need for DR, while DR provides the ultimate safety net for when HA isn't enough.

What if my database node fails?

Self-managed database

• Single node with combined database engine and storage
• Potential performance issues during backups
• RPO may be as high as 24 hours
• RTO could be many hours if the node fails
• Aurora durability:
• Starts with a single database cluster in one availability zone
• Utilizes Aurora storage for data replication
• Multi-tenant storage fleet with nodes spread across 3 AZs
• Each write is replicated 6 times across 3 AZs
• Users pay for only one copy
• RPO is 0 with no asynchronous replication
• Aurora writes only log records, not pages, enhancing durability and performance
• Backup and Restore in Aurora:
• The storage engine continuously backs up data to S3 without impacting performance.
• Point-in-time restore (PITR) is available from the earliest restorable time (up to 35 days) to the latest (about 5 minutes ago for a busy database).
• The restore process involves pulling data from S3, applying internal log records, and starting up an engine for transaction recovery.
• Database Instance Availability:
• Basic configuration includes one instance; failure requires 10-15 minutes to replace.
• Adding a standby instance improves availability; Aurora keeps it in sync with the primary.
• Failover to the standby instance occurs within seconds, maintaining the same size and load capacity.
• Results in an RPO of 0 and an RTO typically around 30 seconds, achieving a multi-AZ availability pattern.

Performance Resilience

• Performance can fluctuate based on application demands, leading to potential wasted resources.
• Open-source solutions offer scaling up (hardware upgrade) or scaling out (dividing database into pieces), both challenging for consistency and noisy neighbors.