Coinbase re:Invent Recap (IND3312)

Session Notes

Scaling support, compliance, and productivity with conversational

Overview of Gen AI Development in Financial Services

2023: Introduction of LLMs

• LLMs for customer and enterprise applications began to enter the market.
• Initial focus on trust and security of data.
• Questions about data safety and utility arose.
• Importance of Retrieval Augmented Generation (RAG) for businesses.
• FSI customers had valuable data but were uncertain about safe utilization.

Customer Support Chatbot

• Over 50% of customers prefer chat as their support channel.
• Trained on Coinbase's extensive knowledge base.
• Provides instant, accurate assistance around the clock, adapting to market volatility.

FAQ Style RAG Responses

• Simple, FAQ-style responses to automate common queries (e.g., sign-in issues, two-factor authentication).
• Focuses on accuracy to build a strong foundation.

Cohere's Rerank model enhances search relevance

• Semantically reorders initial search results
• Understands query-document context beyond keywords
• Supports over 100 languages
• Handles complex data like JSON and code
• Improves Retrieval-Augmented Generation (RAG) systems
• Increases enterprise search accuracy
• Provides faster user experiences
• Functions as a powerful second stage in search pipelines
• Sorts results from keyword or vector search by deep relevance scores
• Key Features
• Rerank-v3.5: state-of-the-art performance, enhanced reasoning, and broad data compatibility (text, JSON, code).
• Rerank-english-v3.0: English documents and semi-structured data.
• Rerank-multilingual-v3.0: For non-English content, supporting over 100 languages.
• Rerank 3 Nimble: Prioritize efficiency and lower latency for high-volume use cases like e-commerce search, improving conversion rates.

Cross-encoder architecture

• A model that processes two inputs (like a query and a document) jointly in a single sequence, using cross-attention mechanisms between all tokens to determine their relevance.
• Key Characteristics
• Joint Processing
• Cross-Attention
• Single Output
• First Stage (Bi-encoder)
• Second Stage (Cross-encoder)

Compliance Processes

• Implementation of KYC, KYB, and TMS.
• Heavy reliance on human resources and thorough investigations.
• Need for detailed explainability and adaptability to diverse regulatory requirements across countries.
• Strategy for Compliance
• Leverage of the internal platform for accessing LLMs and data through MCP interfaces.
• Inclusion of a traditional machine learning model to detect high-risk compliance cases, built on the AnyScale-based ML platform.

Ray-powered Scalability

• Optimizes and manages Ray clusters.
• Efficient distribution and scaling across thousands
• of CPUs and GPUs.
• Handles massive datasets and complex models.
• Ray AI Libraries
• Ray Data: For scalable data processing, particularly suited for unstructured data like images, audio, and video.
• Ray Train: For distributed model training, with integrations for frameworks like
• PyTorch, PyTorch Lightning, and Hugging Face Transformers.
• Ray Tune: A framework for scalable and fault-tolerant hyperparameter tuning and experiment execution.
• Ray Serve: For deploying and serving machine learning models as scalable, programmable APIs with low latency.
• RLlib: An industry-grade library for reinforcement learning, supporting a wide range of applications from robotics to finance.

Compliance Auto Resolution Engine (CAR)

• Central to the solution, acting as the orchestrator for the holistic review process.
• Mobilizes both automation and human expertise in an agentic AI-driven workflow.
• Human-in-the-Loop Process
• Interaction with two human personas:
• Compliance Operations Agents: Review AI system findings and provide feedback for improvement.
• End Customers: Reach out for additional information when necessary.

Coinbase re:Invent recap (IND3312)

Scaling support, compliance, and productivity with conversational AI at Coinbase

Customer Support Chatbot

• Over 50% of customers prefer chat as their support channel.
• Trained on Coinbase's extensive knowledge base.
• Provides instant, accurate assistance around the clock, adapting to market volatility.

FAQ Style RAG Responses

• Simple, FAQ-style responses to automate common queries (e.g., sign-in issues, two-factor
• Focuses on accuracy to build a strong foundation authentication).

Cohere's Rerank model enhances search relevance

• Semantically reorders initial search results
• Understands query-document context beyond keywords
• Supports over 100 languages
• Handles complex data like JSON and code
• Improves Retrieval-Augmented Generation (RAG) systems
• Increases enterprise search accuracy
• Provides faster user experiences
• Functions as a powerful second stage in search pipelines
• Sorts results from keyword or vector search by deep relevance scores

Cohere's Rerank model Key Features

• Rerank-v3.5: state-of-the-art performance, enhanced reasoning, and broad data
• Rerank-english-v3.0: English documents and semi-structured data. compatibility (text, JSON, code).
• Rerank-multilingual-v3.0: For non-English content, supporting over 100 languages.- Rerank 3 Nimble: Prioritize efficiency and lower latency for high-volume use cases like e-commerce search, improving conversion rates.

Cross-encoder architecture

A model that processes two inputs (like a query and a document) jointly in a single

sequence, using cross-attention mechanisms between all tokens to determine their relevance.

Key Characteristics

• Joint Processing
• Cross-Attention
• Single Output
• First Stage (Bi-encoder)
• Second Stage (Cross-encoder)

Compliance Processes

• Implementation of KYC, KYB, and TMS.
• Heavy reliance on human resources and thorough investigations.
• Need for detailed explainability and adaptability to diverse regulatory requirements across

countries.

🛠Strategy for Compliance

• Leverage of the internal platform for accessing LLMs and data through MCP interfaces.
• Inclusion of a traditional machine learning model to detect high-risk compliance cases, built

on the AnyScale-based ML platform.

Ray-powered Scalability

• Optimizes and manages Ray clusters.
• Efficient distribution and scaling across thousands
• of CPUs and GPUs.
• Handles massive datasets and complex models.

Ray AI Libraries

• Ray Data: For scalable data processing, particularly suited for unstructured data like images,
• Ray Train: For distributed model training, with integrations for frameworks like PyTorch,

audio, and video.

• Ray Tune: A framework for scalable and fault-tolerant hyperparameter tuning and

PyTorch Lightning, and Hugging Face Transformers.

• Ray Serve: For deploying and serving machine learning models as scalable, programmable

experiment execution.

• RLlib: An industry-grade library for reinforcement learning, supporting a wide range of

APIs with low latency.

applications from robotics to finance.

Compliance Auto Resolution Engine (CAR)

• Central to the solution, acting as the orchestrator for the holistic review process.
• Mobilizes both automation and human expertise in an agentic AI-driven workflow.

Human-in-the-Loop Process

Interaction with two human personas

• Compliance Operations Agents: Review AI system findings and provide feedback for
• End Customers: Reach out for additional information when necessary improvement.

Coinbase re:Invent Recap (IND3312)

Scaling Support, Compliance, and Productivity with Conversational AI

Customer Support Chatbot

• Over 50% of customers prefer chat as their support channel.
• Trained on Coinbase's extensive knowledge base.
• Provides instant, accurate assistance around the clock while adapting to market volatility.

FAQ-Style RAG Responses

• Simple FAQ-style responses automate common queries, such as sign-in issues and two-factor authentication.
• Focuses on accuracy to build a strong foundation.

Cohere's Rerank Model Improves Search Relevance

• Semantically reorders initial search results.
• Understands query-document context beyond keywords.
• Supports more than 100 languages.
• Handles complex data such as JSON and code.
• Improves Retrieval-Augmented Generation (RAG) systems.
• Increases enterprise search accuracy.
• Provides faster user experiences.
• Functions as a powerful second stage in search pipelines.
• Sorts keyword or vector search results by deep relevance scores.

Key Features of Cohere's Rerank Model

• Rerank-v3.5: State-of-the-art performance, enhanced reasoning, and broad data compatibility across text, JSON, and code.
• Rerank-english-v3.0: For English documents and semi-structured data.
• Rerank-multilingual-v3.0: For non-English content, supporting more than 100 languages.
• Rerank 3 Nimble: Prioritizes efficiency and lower latency for high-volume use cases such as e-commerce search, improving conversion rates.

Cross-Encoder Architecture

A model that jointly processes two inputs, such as a query and a document, in a single sequence and uses cross-attention across all tokens to determine their relevance.

Key Characteristics

• Joint processing
• Cross-attention
• Single output
• First stage (bi-encoder)
• Second stage (cross-encoder)

Compliance Processes

• Implements KYC, KYB, and TMS.
• Relies heavily on human resources and thorough investigations.
• Requires detailed explainability and the ability to adapt to different regulatory requirements across countries.

🛠 Compliance Strategy

• Uses the internal platform to access LLMs and data through MCP interfaces.
• Includes a traditional machine learning model, built on the AnyScale-based ML platform, to detect high-risk compliance cases.

Ray-Powered Scalability

• Optimizes and manages Ray clusters.
• Efficiently distributes and scales workloads across thousands
• of CPUs and GPUs.
• Handles massive datasets and complex models.

Ray AI Libraries

• Ray Data: For scalable data processing, especially unstructured data such as images, audio, and video.
• Ray Train: For distributed model training, with integrations for PyTorch, PyTorch Lightning, Hugging Face Transformers, and other frameworks.
• Ray Tune: A scalable, fault-tolerant framework for hyperparameter tuning and experiment execution.
• Ray Serve: Deploys and serves machine learning models as scalable, programmable, low-latency APIs.
• RLlib: An industry-grade reinforcement learning library supporting applications from robotics to finance.

Compliance Auto Resolution Engine (CAR)

• Central to the solution, acting as the orchestrator for the holistic review process.
• Mobilizes automation and human expertise in an agentic AI-driven workflow.

👥 Human-in-the-Loop Process

Interaction with Two Human Roles

• Compliance operations agents: Review AI system findings and provide feedback for improvement.
• End customers: Are contacted for additional information when necessary.

2024: Adoption and Scaling

• Release of Amazon Bedrock and other solutions for secure data access and management.
• Customers began using AI in key areas like customer support.
• Focus shifted to scale, capacity, and granular security.
• Exploration of applying well-architected frameworks to AI for future growth.
• 2025: Advanced AI Applications
• Emergence of multi-agent autonomous applications.
• Customers aim to transform core business operations and departments.
• Emphasis on measurable impact and business KPIs.
• Shift from singular tools to revolutionizing customer and employee workflows.

AWS Services for Gen AI

AWS Offerings

• Pre-built agents for quick deployment.
• Experimentation with open-source tools.
• Building a custom agent.
• Provision of models, tools, infrastructure, and expertise.
• AI Infrastructure and Tools
• Robust AI infrastructure, custom silicon, and data foundation tools.
• Decades of experience in secure, resilient, and globally scaled infrastructure.
• Advanced controls for data privacy, access management, monitoring, and observability.
• AWS Transform
• Automation of legacy workload modernization (e.g., .NET frameworks, VMware).
• Agents in Amazon Connect and tools in the marketplace.
• Investment in interfaces, protocols, security, and network enhancements for AI.

Amazon Bedrock

Service Overview

• Fully managed service for building, deploying, and operating Gen AI applications.
• Access to leading foundation models (Anthropic, Meta, Mistral, Amazon) via a single API.
• Continuous addition of new models.
• Customization and Safety
• Tools for privately customizing models and applications with data.
• Application of safety guardrails, cost optimization, and latency reduction.
• Rapid iteration capabilities.
• AgentCore:
• Set of services for secure, scalable agent deployment and operation.
• Serverless nature eliminates infrastructure management needs.

Challenges in Agent Deployment

Gartner Prediction

• Over 40% of agentic AI projects are predicted to be canceled by 2027.
• Reasons: growing costs, unclear business value, insufficient security.
• Key Challenges
• Specialized Runtime: Need for a runtime environment unlike typical applications.
• Stateless LLMs: Agents require memory to retain context and provide personalized experiences.
• Fine-Trained Identity: Access control to ensure agents only access appropriate systems and data.
• Custom Tool Interaction: Ensuring AI agents can discover and interact with custom tools and data sources with proper identity controls.

Autonomous Multi-Step Workflows

• Need for agents to execute complex, multi-step workflows independently.
• Requirement for common tools in workflows that are either shared or restricted.
• Specialized Monitoring
• Necessity for monitoring systems specifically designed for agentic AI.
• Differences between prototyping and production deployment of agents.

Amazon Bedrock AgentCore

Modular and Fast Iteration Approach

• Designed for building scalable production agents.
• Agent Core Runtime
• Secure, serverless runtime for deploying and scaling dynamic agents and tools.
• Supports various frameworks, protocols, and model choices.
• Reliable execution of multimodal, real-time, and long-running agents (up to 8 hours).
• Features checkpointing and recovery capabilities for graceful handling of interruptions or failures.
• Agent Core Gateway
• Facilitates integration of MCP servers and APIs with agents.
• Empowers agents with a variety of tools.
• Agent Core Browser and Code Interpreter
• Allows agents to act autonomously in the browser or execute code under specified terms and rules.

Supporting Tools

Agent Core Identity

• Builds enterprise-ready AI agents with security and identity features.
• Includes standards-based authentication, compatibility with existing identity providers, and OAuth support.
• Features a secure token vault for frictionless user experiences.
• Agent Core Memory
• Enables agents to store and retrieve short and long-term memory for complex workflows.
• Facilitates continuous use of user input and history in shared memory.
• Observability
• Centralizes observability stack for Gen AI, combining logs, traces, and metrics.
• Measures performance and accuracy at scale.

Coinbase's Use of Generative AI

Crypto and Economic Freedom

• Crypto empowers economic freedom and fair participation in the global economy.
• Coinbase’s mission is to expand economic freedom to over a billion people.
• Key Pillars at Coinbase
• Building Trust
• Safeguarding users' assets with top-tier security.
• Implementing strong protections against fraud and account takeovers.
• Making Crypto Accessible
• Designing intuitive and personalized experiences using AI and machine learning.
• Simplifying complicated financial products.
• Scaling Globally
• Operating efficiently and quickly to support millions of users across 100+ countries.
• Managing billions of dollars in trading volume.

User Journey and ML Applications

Login Security

• ML models defend against account takeovers during user login.
• Fiat to Crypto Conversion
• ML models evaluate the legitimacy of users and the risk of credit defaults when moving currency into the crypto exchange.
• Funds Transfer to Blockchain
• ML models assess risks of reversals before releasing funds onto the blockchain.
• Personalization of User Experience
• ML-Powered Personalization
• Personalized search results, tailored news feeds, intuitive recommendations, and real-time price alerts.
• Use of advanced deep learning models trained on massive datasets.
• Adaptive Risk Scoring System
• Built using graph neural networks to score blockchain addresses.
• Detects and blocks malicious blockchain addresses, protecting customers.
• ERC20 Scam Token Detection
• Combines smart contract auditing and machine learning.
• Reviews assets before listing on the exchange, providing additional customer protection.

Predictive Machine Learning Model

Volatility of Crypto Market

• Crypto markets are highly volatile, requiring a reliable infrastructure.
• Predictive Models
• Used to scale backend databases before surges occur, ensuring reliable performance during market volatility.

New Generation of AI Investments

Large Language Models (LLMs)

• Transformative impact since November 2022, marking a historic moment in AI.
• Key Areas of Application
• Customer Support: AI-powered virtual assistants and advanced tools for human agents.
• Compliance: Generative AI (GenAI) streamlines complex investigations and navigates regulatory landscapes.
• Developer Productivity: Improving developer productivity at Coinbase.
• Challenges in Crypto Customer Support
• Volatility: User activity can fluctuate by up to 50% within a month, requiring scalable support systems.
• Trust: Customers need to feel safe and supported regardless of market conditions.
• Global Operations: Supporting users across different languages, regulations, and expectations.

GenAI Strategy

• Internal GenAI platform integrating multiple LLMs and data sources via standard interfaces (OpenAI API standards
• and Model Context Protocol).
• Depth
• Targeted investments in high-impact areas within customer support:
• Automating customer-facing workflows with conversational chatbots.
• Providing human support agents with intelligent tools to improve effectiveness.
• Extracting valuable insights from support tickets to continually improve products and services.
• Use Cases
• Customers requires help, initiating a cycle of support interactions enhanced by GenAI.

Customer Support Chatbot

• Over 50% of customers prefer chat as their support channel.
• Trained on Coinbase's extensive knowledge base.
• Provides instant, accurate assistance around the clock, adapting to market volatility.

Development of the Chatbot

FAQ Style RAG Responses

• Simple, FAQ-style responses to automate common queries (e.g., sign-in issues, two-factor authentication).
• Focuses on accuracy to build a strong foundation.

Agentic Architecture

User Input and Conversation History

• Stored in a vector database serving as the memory layer.
• RAG Retriever
• Uses Bedrock knowledge bases to vectorize and store Coinbase help articles.
• Employs Cohere’s rerank models for high accuracy in retrieved articles.
• Response Generation
• Mixture of LLMs, including cloud models served through Bedrock.
• Involves a sub-agent developed in an actor-critic architecture for precise response generation.
• Guardrails
• Input and output guardrails powered by Bedrock guardrails.
• Protects against harmful content and sensitive PII leakage.
• Custom domain-specific filters to minimize prompt injection.
• Custom rules to keep responses grounded and reduce hallucinations.

Next Layer of Development

• Industry and LLM Capability Evolution
• Recognition of the need to achieve more than the initial RAG layer.

Cohere's Rerank model enhances search relevance

• Semantically reorders initial search results
• Understands query-document context beyond keywords
• Supports over 100 languages
• Handles complex data like JSON and code
• Improves Retrieval-Augmented Generation (RAG) systems
• Increases enterprise search accuracy
• Provides faster user experiences
• Functions as a powerful second stage in search pipelines
• Sorts results from keyword or vector search by deep relevance scores Key Features & Models
• Rerank-v3.5: state-of-the-art performance, enhanced reasoning, and broad data compatibility (text, JSON, code).
• Rerank-english-v3.0: English documents and semi-structured data.
• Rerank-multilingual-v3.0: For non-English content, supporting over 100 languages.
• Rerank 3 Nimble: Prioritize efficiency and lower latency for high-volume use cases like e-commerce search, improving conversion rates.
• How They Work
• Input: A user query and an initial list of documents (from a keyword search like Elasticsearch or a vector search).
• Processing: The Rerank model, using a cross-encoder architecture, jointly processes the query and each document to understand their semantic relationship.
• Output: An ordered list of documents, each assigned a relevance score, with the most contextually relevant results at the top.
• Use Cases & Benefits
• Enhanced Enterprise Search: Surfaces highly relevant information from complex internal data (invoices, code).
• Improved RAG Systems: Ensures factual grounding by providing the most pertinent snippets to LLMs.
• Multilingual Applications: Semantic search across diverse languages.
• Cost & Complexity Reduction: Integrates easily with existing search stacks (e.g., Elasticsearch) via a single API call, adding semantic power without a full overhaul.

Cross-encoder architecture

• A model that processes two inputs (like a query and a document) jointly in a single sequence, using cross-attention mechanisms between all tokens to determine their relevance.
• Key Characteristics
• Joint Processing: The core feature is that the query and the candidate document are concatenated into a single input sequence, separated by a special token.
• Cross-Attention: Within the transformer layers, the model's self-attention mechanism can directly compare every token in the query with every token in the document. This fine-grained interaction allows for a deep, nuanced understanding of their relationship and semantic relevance.
• Single Output: The model typically outputs a single relevance score (e.g., a number between 0.0 and 1.0) for the pair, often using the embedding from the token passed through a classification layer.
• Role in Modern AI Systems:
• They are primarily deployed as re-rankers in a two-stage retrieval process, particularly in Retrieval-Augmented Generation (RAG) systems:
• First Stage (Bi-encoder): A fast, efficient bi-encoder model quickly narrows down a vast corpus of documents to a smaller, manageable list of potentially relevant candidates (e.g., the top 100).
• Second Stage (Cross-encoder): The cross-encoder then meticulously re-evaluates and re-ranks this smaller subset of candidates with high precision, ensuring the most accurate and contextually appropriate results are presented to the final LLM for generating a response.

Business Procedure Layer

• Enhanced Chatbot Capabilities
• As LLM capabilities improved, the chatbot was enhanced to follow business procedures autonomously.
• Collects information conversationally and takes direct actions on behalf of the user.
• Business procedures serve as a single source of truth, used by both humans and AI agents.
• Automates queries about user accounts and pending transactions.

Agentic Architecture

• Introduction of the Business
• Procedure Classifier.
• Routes queries to the correct sub-agent emulating specific business procedures.
• RAG agent as a specialized sub-agent executing procedures involving the RAG knowledge base.

Proactive Issue Resolution Layer (Anticipatory Capabilities)

• AI agents anticipate common user problems using user signals and active incidents on the platform.
• Proactive issue resolution layer added to the agentic design.
• Proactive agents fall back to the business procedure classifier if no proactive resolution is found.
• Data access for proactive agents powered through standardized MCP servers.

Key Design Principles

Model Selection

• Balanced approach considering accuracy, latency, and scalability.
• Continuous reevaluation as model capabilities evolve.
• Tool Standardization
• Standardization of tool access through
• Model Context Protocol (MCP).
• Strong foundation for customer support and other domains.
• Business Adaptability
• Single source of truth for business procedures powering both humans and AI.
• Enhanced business adaptability.
• Grounded Responses
• Emphasis on factual correctness in responses.

Monitoring and Evaluation

• LLM as a Judge Evaluations
• Assessment of chatbot responses for relevancy, accuracy, potential bias, hallucinations, and more.
• Active tracking of quality metrics and trends to quickly identify anomalies.

Agent Assist Tool

Purpose

• Designed to help human customer support agents handle complex issues.
• Provides real-time assistance for diagnosing and mitigating customer issues.
• Features
• Draws from account signals, ongoing incident data, and past customer support tickets.
• Offers personalized guidance and generates precise responses in multiple languages.
• Impact
• Faster resolutions, happier customers, and improved efficiency for support teams.
• Close to 65% of customer contacts are handled automatically by AI systems.
• Annual savings of nearly 5 million employee hours and significant cost savings.
• 65% of automated cases are resolved in a single interaction, enhancing user experience.

Importance of Compliance

• Critical for a fintech entity like Coinbase to uphold standards in AML/CFT (Anti-Money
• Laundering/Countering Terrorist Financing).
• Commitment to preventing malpractices on the platform.
• Compliance Processes
• Implementation of KYC, KYB, and TMS.
• Heavy reliance on human resources and thorough investigations.
• Need for detailed explainability and adaptability to diverse regulatory requirements across countries.
• Strategy for Compliance
• Leverage of the internal platform for accessing LLMs and data through MCP interfaces.
• Inclusion of a traditional machine learning model to detect high-risk compliance cases, built on the AnyScale-based ML platform.
• Depth
• Advanced deep learning models for detecting high-risk cases
• in KYC, KYB, and TMS.
• Automation and acceleration of complex investigations through AI.
• Gathering and synthesizing data from various sources for rigorous investigations across compliance workflows.

Compliance Assist Tool

Purpose

• Assistive tool for compliance agents, analogous to the customer support agent assist tool.
• Features
• AI-powered gathering and synthesis of information from internal systems and open-source intelligence.
• Composition of a narrative summary and clear risk review signals.
• Provides compliance agents with a fully documented, explainable investigation for well-informed, confident decision-making.
• Impact
• Empowerment of compliance teams and elevation of compliance operation standards.

Ray-powered Scalability

• Optimizes and manages Ray clusters.
• Efficient distribution and scaling across thousands
• of CPUs and GPUs.
• Handles massive datasets and complex models.
• Unified Developer Experience:
• Streamlined workflow from experimentation to production.
• Interactive development consoles.
• Built-in IDEs.
• Tools for debugging distributed workloads.
• Aims to accelerate developer productivity.
• Production Readiness:
• Adds enterprise-grade features on top of Ray.
• Advanced observability.
• Data governance.
• Cost tracking.
• Robust administration tools.
• Ensures reliability, security, and performance of ML models.
• Cloud Agnostic Deployment:
• Supports deployment across various cloud providers
• (AWS, Google Cloud).
• Supports on-premise Kubernetes clusters.
• Allows organizations to leverage existing infrastructure.
• Maintains control over data and resources.
• Optimizations and Integrations:
• Includes optimizations for performance and cost reduction.
• Integrations with essential ML tools:
• Weights & Biases for experiment tracking.
• Astronomer for workflow orchestration (using
• Apache Airflow).
• Focus on AI Democratization:
• Simplifies complexities of distributed computing.
• Aims to make advanced AI and ML capabilities more accessible.
• Ray AI Libraries
• Ray Data: For scalable data processing, particularly suited for unstructured data like images, audio, and video.
• Ray Train: For distributed model training, with integrations for frameworks like
• PyTorch, PyTorch Lightning, and Hugging Face Transformers.
• Ray Tune: A framework for scalable and fault-tolerant hyperparameter tuning and experiment execution.
• Ray Serve: For deploying and serving machine learning models as scalable, programmable APIs with low latency.
• RLlib: An industry-grade library for reinforcement learning, supporting a wide range of applications from robotics to finance.

Agentic Architecture for Compliance

Initiation

• The process begins when deep learning compliance risk models trigger alerts on high-risk cases.
• Initiates a holistic review, a comprehensive investigation of high-risk compliance cases.
• Compliance Auto Resolution Engine (CAR)
• Central to the solution, acting as the orchestrator for the holistic review process.
• Mobilizes both automation and human expertise in an agentic AI-driven workflow.
• Human-in-the-Loop Process
• Interaction with two human personas:
• Compliance Operations Agents: Review AI system findings and provide feedback for improvement.
• End Customers: Reach out for additional information when necessary.
• Data Aggregation and Synthesis
• Engine aggregates and synthesizes data from various sources (internal, external, open-source intelligence) via standardized MCP data connectors.
• Results in a robust AI-generated report called the narrative summary.
• Final Decision
• Human compliance operations agent reviews AI's reasoning and compliance evidence.
• Determines whether a case warrants filing a suspicious activity report with government authorities.

Developer Productivity

AI in SDLC

• AI agents assist engineering teams at various stages of the software development life cycle (SDLC).
• Consolidates tasks like coding, ticketing, documenting, pull request creation, and reviewing pull requests.
• Focus Areas
• Code-Authoring
• Code Reviewing
• Quality Assurance

Code-Authoring Tools

Cloud Code

• State-of-the-art coding assistant integrating directly into IDEs or command line.
• Cursor
• Intelligent, context-aware IDE designed for faster, more collaborative, and efficient software development.
• Offered as an option for developers to choose based on their preferences.
• Powering Models
• Tools powered by Anthropic models served through Bedrock.

Pull Request and Code Reviewing

Homegrown Tool

• Adapted from an open-source tool and enhanced with models from Bedrock.
• Implemented as an AI-powered GitHub action to automate PR reviews.
• Definition:
• A "Homegrown Tool" is software, applications, or solutions developed internally by a company or community to address specific, unique problems.
• Development Approach:
• Often created using custom scripts.
• Utilizes existing platforms like Visio or Excel.
• Avoids reliance on off-the-shelf products.
• Features
• Summarizes pull requests and underlying code changes.
• Generates clear natural language review comments.
• Enforces coding conventions.
• Highlights gaps in unit testing coverage.
• Provides tips for debugging CI failures.
• Impact
• Automates routine aspects of code review, allowing human developers to focus on nuanced aspects and deliver higher value.

Quality Assurance

Automated UI Testing Tool

• Homegrown AI-powered tool for end-to-end quality assurance automation for web and mobile UI.
• Converts natural language test descriptions into autonomous browser actions.
• Features
• Performs browser actions in various form factors using services like BrowserStack.
• Emulates browser actions through frameworks like Playwright.
• Benefits
• Allows testing UI like a human would, enhancing the effectiveness of quality assurance processes.

Impact of AI on Developer Productivity

Code Generation

• Nearly 40% of all code written daily at Coinbase is AI-generated or AI-influenced.
• Goal to surpass 50% soon, with human review and understanding still required.
• Automated PR Reviews
• Estimated annual savings of about 75,000 hours.
• Raises code quality by enforcing coding conventions.
• Quality Assurance
• AI-powered UI testing tool achieves accuracy on par with human testers.
• Detects three times as many bugs as a human in the same amount of time.
• Dramatically speeds up the process of introducing new tests (as little as 15 minutes).
• 86% cost reduction compared to traditional manual testing.