Goldman Sachs: Fast Track your applications onto Cloud - AWS FSI Meetup Q1/2023

Session Notes

Introduction by Ragini Lalwani:

• Leads public cloud onboarding and migration at Goldman Sachs.
• Wishes everyone a Happy Women's Day and Holi.
• Has worked at Goldman Sachs for almost 15 years, starting as an intern.
• Has experience with various cloud platforms: private, hybrid, and public.
• Discussion on Goldman Sachs' approach to ensuring rapid and secure cloud use by developers in a regulated industry.

Introduction by Aditya:

• Leads finance planning engineering teams and is the lead architect for FP&A at Goldman Sachs.
• Also has close to 15 years with the firm, starting with web app development, now focused on enterprise systems.
• Discussion on the firm's unique phase of cloud onboarding and the transition from on-prem to cloud.
• Will talk about onboarding experience with public cloud, secure tooling, acceleration of cloud onboarding, and key highlights and learnings.

Overview of Goldman Sachs:

• A leading financial institution offering a broad range of services including investment banking, securities, investment management, and consumer banking.
• Serves a large and diversified client base including corporations, financial institutions, governments, and individuals.
• Founded in 1869, headquartered in New York, with offices across major financial centers globally.
• Focus on accelerating Goldman Sachs' cloud journey with a security-first mindset.

Fast Track Section Overview:

• Emphasizes a customer-centric mindset using Amazon's "working backwards" process.

Problem Statement from the Customer's Perspective (Application Developer):

• New developers need to understand the firm's security approach, posture, and modules.
• Deployment requires passing through security reviews, which can be manual and time-consuming.
• Security reviews are required not only at the initial deployment but also for subsequent changes, leading to a potentially slow and iterative process.
• Developers may receive alerts after deployment indicating incorrect configurations, highlighting the need for preventive controls rather than detective controls.
• Developers, skilled in application development, may not have the expertise in cloud security, creating a dichotomy between development and security requirements.

Tenets for Addressing the Problem Statement:

• Implement guardrails to ensure secure-by-design deployments, automating the security baseline.
• Establish a clear shared responsibility model delineating the roles of the cloud service provider, Goldman Sachs Cloud teams, and application developers.
• Utilize cloud-native technologies to minimize the need for learning firm-specific tooling, allowing developers to leverage existing knowledge of AWS CDK or Kubernetes.
• Aim for self-service capabilities to reduce the need for ticket raising or manual reviews, enabling more developers to access the cloud without hurdles.

Introduction to Fast Track:

• Fast Track is the solution developed in response to the identified problem statement.
• It involves a technical deep dive into the implementation details.
• The goal is to provide a secure, efficient, and self-service-oriented cloud onboarding experience for developers.

Fast Track Overview:

• A fully managed service enabling Goldman Sachs developers to rapidly develop and deploy software to AWS.
• Enforces the firm's security, regulatory, and compliance requirements through guardrails.
• Enhances the security posture by programmatically and proactively enforcing security policies.

Key Features of Fast Track:

• Handles account creation and pipeline management behind the scenes.
• Users interact with Fast Track via CLI or API to create, delete, or share pipelines.
• Pipeline creation triggers an account provisioning workflow, creating a service account for AWS resources and a pipeline account for management.
• Users push code changes, which run through a deployment pipeline and are checked against guardrails.
• Guardrails enforce the firm's baseline security policy, ensuring resources meet security requirements before deployment.

Simplified User Journey:

• User creates a Fast Track pipeline and associates it with a GitLab repo.
• Writes infrastructure as code using AWS CDK.
• Pipeline synthesizes CDK code into CloudFormation templates.
• Guardrails run against these templates, and the policy engine determines if the deployment can proceed ("go") or not ("no go").

Guardrails Details:

• Guardrails follow a default-deny policy.
• Written using Open Policy Agent (OPA).
• Run against standard CloudFormation templates.
• Example guardrail: Ensures that malicious actors cannot access unencrypted objects.

Threat Modeling and Control Definition:

• Uses NIST 800-53 guidelines to identify threats for specific technologies.
• ISO 27001 is an international standard for information security management systems, while NIST SP 800-53 is a U.S. government standard for security and privacy controls. 2. ISO 27001 focuses on the management of information security, while NIST SP 800-53 focuses on the technical security controls.
• Example threat: Malicious actors can access an unencrypted object.
• Control definition: Encrypt the object.
• Controls are defined and used as OPA policies, reviewed, and unit tested by the security team.

Collaboration and Release of Controls:

• Collaboration between the Cloud enablement guardrail development team and security teams.
• Controls are released as guardrails, creating a secure environment but potentially posing hurdles for developers.

Accelerators:

• Built to work with guardrails, allowing developers to quickly deploy resources that comply with all guardrails.
• Example: Fast Track S3 bucket that works with all 10 guardrails blocking S3.

Sample Code and Starter Projects:

• Provided to help users bootstrap their cloud journey.

Fast Track Launch and Production Use:

• Fast Track has been launched and is available, with several applications in production and many more underway.

Organizational Alignment and Model:

• Goldman Sachs recognized the need for a first-class developer experience without compromising security.
• Fast Track was deemed a flagship project, with Cloud enablement leading the development in collaboration with technology risk teams.
• Guardrail development involved a mix of developers, cloud architects, and security engineers.
• A guardrail contribution model ensures that specific technology teams (e.g., storage engineering) review and contribute to relevant guardrails.

Response to Question on Organizational Structure:

• Acknowledges the typical siloed model in large enterprises.
• Emphasizes the need for a collaborative approach to balance developer experience and security.
• Highlights the importance of a mandate from management to foster cross-team collaboration.
• Notes the constraint that not all Cloud enablement team members are experts in every cloud service (e.g., S3).
• The mix of skills required for guardrail development and the contribution model for specialized reviews.

Clarification on Ownership and Responsibilities in the Cloud:

• In the cloud, the application team owns all their cloud resources, including computers and databases, which are deployed into their VPC.
• The database team is responsible for working with Cloud enablement to establish access control, policy management, and audit-related pieces.
• On-premises, the database team owns the databases, but in the cloud, the customer (application team) owns the databases running in AWS.

Summary of Organizational Model:

• A central Cloud team establishes the guardrails framework and is the authority around cloud development.
• The CIO, CTO, and Tech Risk Office sponsor new guardrail development and policies around cloud onboarding and governance.
• Associated teams (DB teams, IAM teams, etc.) work with the central Cloud team to onboard their services and policies.
• Application development teams implement their apps using Fast Track and may request additional services, which are then worked on with related teams.

Collaborative Model:

• Multiple teams play their part in the cloud onboarding process, ensuring a collaborative approach to cloud development and security.

Hands-On Approach for the First Application:

• The initial application on Fast Track required a more hands-on approach as the team was learning about the customer journey.
• Future applications aim to leverage existing guardrails, allowing customers to use the service independently.
• Investment in tutorials and documentation to support self-service usage.

Platform Team's Role:

• The platform team provides guidance and builds security controls (guardrails) for resources like Lambda, DynamoDB, and RDS.
• Developers are free to choose the resources they need, with the platform team helping to build the necessary security controls.
• Accelerators are provided to make it easier for developers to deploy resources safely.

Guardrails in Development vs. Production:

• The same guardrails are applied in both development (sandbox) and production environments.
• Guardrails are in blocking mode in all environments to maintain a consistent security posture.
• Fine-grained control allows for specific guardrail bypasses on a per-pipeline level, subject to tech risk review.

Technical Implementation of Guardrails:

• Guardrails are checked via an API that the pipeline interacts with.
• The pipeline synthesizes infrastructure as code (e.g., AWS CDK) into CloudFormation templates.
• These templates are then checked against the guardrails and run using Open Policy Agent (OPA).
• The policy engine determines if the deployment can proceed ("go") or not ("no go").

Technical Implementation of Guardrails (Continued)

• Additionally, an IDE plugin is available to provide developers with feedback sooner, without the need to deploy the pipeline.
• Developers can see the feedback as they develop in their IDE, helping them adhere to security standards early in the development process.

Question on NIST Implementation and Threat Database:

• The project started with NIST 800-53 as a baseline.
• The firm's cyber defense and Intel team keeps the team informed about new threats.
• Decisions on managing new threats are made based on the vulnerability level (critical or not) and integrated into the software development lifecycle (SDLC).
• Detective controls complement the preventative controls provided by the guardrails.

Introduction to FP&A (Finance Planning and Analysis):

• FP&A is responsible for firm-wide business planning, cost allocations, and driving financial insights.
• FP&A is pursuing a cloud-first strategy, aiming to move 100% of on-prem and new applications to the cloud by the end of 2024.
• Reasons for adopting the cloud include monthly compute usage peaks and troughs, limitations of current on-prem solutions (installed and run on an organization's own physical hardware and servers, located within their own facilities), and the desire to reduce engineering time spent on maintenance.
• FP&A has aggressively decoupled legacy on-prem infrastructure, with approximately 70% completed, and has built new applications with cloud considerations in mind over the past three years.

FP&A's Cloud Strategy and Adoption:

• FP&A has been building microservice-friendly, cloud-ready architectures and heavily investing in Infrastructure as Code (IaC) even on-prem.
• The adoption of the cloud is driven by the low cost of compute, increased productivity, and reduced build and maintenance cycles.
• Fast Track, with sponsorship from engineering leaders, became the strategic platform for the firm.

First Application: FBA Central:

• A simple gateway application was chosen as the first app to learn AWS and Fast Track.
• Developed in three to four months, went live with an MVP in September-October.
• Representative microservice app with Spring Boot APIs, deployed using ECS Fargate and DynamoDB, with a React.js UI.

Subsequent Applications:

• Built more complex applications with increased microservices, on-prem connectivity, and workflow solutions.
• Deployed the first app in a dedicated FP&A-specific VPC, a first for the firm.
• Despite increased application complexity, Fast Track enabled increased productivity and decreased delivery times.
• Teams became more skilled over time, further reducing timelines.

Summary:

• FP&A's cloud-first strategy, combined with Fast Track, has led to successful onboarding of applications with increasing complexity, improved productivity, and reduced delivery times.

FP&A's Cloud Journey and Future Focus:

• Deployed three cloud-native apps with eight plus services.
• Learned to build software, undergo cloud onboarding, tech risk reviews, architecture bar raising, and SDLC processes.
• Trained about half of FP&A engineers.
• Key focus for 2023: high-usage apps and data-intensive calculations, processing about 6 billion records daily.
• Will explore serverless offerings (Lambda, Step Functions) and data processing/analytics services (Glue, Athena, S3, Snowflake).
• Aim to build FP&A-centric accelerators using CDK constructs to further accelerate development.

Key Decisions and Success Factors:

• Formation of a core team of four developers from different regions to train scrum teams, accelerating development.
• Choice of Fast Track for secure cloud onboarding, dealing with sensitive data.
• Fast Track's guardrails are tech risk-approved and continuously executed at every pipeline, providing a secure and up-to-date profile.
• Fast Track's CDK accelerators and common patterns helped improve developer productivity.

Deployment and Learnings:

• Deployed the first simple app, FBA Central, using AWS components like DynamoDB, ECS Fargate, and ALB.
• Underwent rigorous tech risk reviews and architecture bar raising.
• Learned about gold live mechanisms, environment creation, observability, logging, and using CloudWatch and PagerDuty.
• Decommissioned the on-prem app, becoming fully cloud-native for FP&A.

Cloud Learnings:

• Cloud provides a broader and more sophisticated set of options to closely match application compute consumption needs.
• Encourages organizations starting their cloud journey to adapt the playbook and learn from FP&A's experience.

Importance of Design Choices:

• Engineers must understand the consequences of their design choices, which impact operational costs and maintenance.
• Serverless options offer lower costs, easy scaling, and low maintenance.
• Infrastructure as Code (IaC) allows for faster innovation and toggling between different compute and cost profiles.

Benefits of IaC:

• Enables faster innovation and optimization for cloud costs.
• Allows for different costs and compute profiles based on architecture choices.
• Business Continuity Planning (BCP) is more straightforward with cloud strategies compared to on-prem.

Increased Developer Productivity and Cost Optimization:

• Developer productivity increased with the ability to manage costs on a daily basis.
• The maturing ecosystem and Fast Track have facilitated the onboarding of applications.

Resources for Further Learning:

• Goldman Sachs Engineering developer blog for insights on cloud and related builds.

Clarification on Fast Track and Open Source:

• Fast Track itself has not been open-sourced.
• Goldman Sachs has an open-source team and has open-sourced various projects.
• The guardrails for Fast Track were initially built in partnership with AWS and have been shared back with AWS, who are using them as part of CFN Card.
• Goldman Sachs does not consider their guardrails a competitive advantage and is open to sharing them.

SkyPath: Cloud to On-Prem Integration:

• SkyPath is a connectivity solution that allows applications in different trust zones to connect.
• It addresses the challenge of connecting on-prem and cloud applications in a hybrid environment.
• SkyPath leverages a Service Identity Framework (SIF) to establish a unique identity for each endpoint before onboarding to SkyPath.
• This ensures that only endpoints with a clearly defined identity can connect, addressing potential identity challenges in the integration.

Service Identity Framework (SIF):

• SIF requires every endpoint to have a unique identity defined before it can onboard to SkyPath.
• This acts as a pseudo-cloud identity for on-prem services, ensuring secure and controlled access.
• Establishing a service identity is a prerequisite for enabling access to on-prem APIs via SkyPath.

Conclusion:

• The discussion highlights the importance of secure and controlled integration between on-prem and cloud environments, facilitated by solutions like SkyPath and the Service Identity Framework.