Run OSS LLMs on a Single H100 Smarter, Cheaper, Faster

Session Notes

Introduction to P5.4xlarge Instance:

• AWS introduces P5.4xlarge, enabling the hosting of powerful open-source LLMs (like Qwen-32B, Mistral, and LLaMA 2) on a single NVIDIA H100 GPU.
• Addresses the previous need for overprovisioning large, expensive multi-GPU clusters. Target Audience:
• Engineers, startups, and individual tinkerers who want to host OSS LLMs without high costs or performance compromises.
• Teams looking to scale smart rather than scale up with massive clusters. Key Benefits:
• Cost-Efficiency: Reduces the need for overprovisioning and managing complex multi-GPU clusters.
• Performance: Offers no compromises on memory or compute power with a single H100 GPU.
• Accessibility: Makes powerful GenAI capabilities available to more builders. Session Highlights:
• Exploration of real-world benchmarks for OSS models on P5.4xlarge.
• Discussion on agentic GenAI workflows and cost-saving strategies.
• Practical deployment tips for models like Hugging Face transformers and low-latency chatbots. Practical Outcomes:
• Understanding which OSS models run efficiently on a single H100.
• Expectations for performance on P5.4xlarge.
• Strategies for designing infrastructure that matches specific needs without overprovisioning. Conclusion:
• “Most of us don’t need an 8-GPU monster to ship a useful chatbot or GenAI app, but we still end up paying for one.”
• Empower small teams and individual engineers to deploy OSS LLMs effectively and cost-efficiently using AWS P5.4xlarge. – Introduce the Need:
• “Imagine if, instead of paying for and managing 8 GPUs just to get access to one H100, you could pick an instance size that finally matches your workload.”
• Problem:
• Inefficiency and cost of overprovisioning large GPU clusters. Announce the Breakthrough:
• New Offering:
• “AWS recently introduced new single-GPU P5 instance sizes: p5.4xlarge gives you the latest NVIDIA H100—no more overbuying, no waste.”
• Solution:
• Emphasizes the elimination of unnecessary expenses and complexity. Highlight the Specs and Simplicity:
• Instance Specifications:
• “p5.4xlarge packs everything you need: 1×H100 GPU, 16 vCPUs, 256 GiB RAM, nearly 4 TB NVMe SSD, and 100 Gbps network.”
• Benefits:
• “All the power of a flagship GPU with simple deployment and a lower bill.” Make the Price Contrast Real:
• Cost Comparison:
• “You can rent this for around $3.90 per hour in some regions, compared to paying for all 8 GPUs on bigger P5s.”
• Impact:
• “This significantly lowers the bar for experiments, demos, and even production for small teams.” Before and after:
• Previous Scenario:
• “Before, H100 meant ‘enterprise scale’ and massive upfront cost.”
• Current Scenario:
• “Now, with a single click, you have pro-grade capability—ideal for single-tenant APIs, agentic RAG, internal tools, and startups moving fast.”
• Transformation:
• Focuses on making high-performance GPU capabilities accessible and affordable. What You Want the Audience to Take Away:
• Accessibility:
• “p5.4xlarge puts NVIDIA H100 power in reach for ‘regular’ use cases—no need for massive clusters.”
• Efficiency:
• “Simpler, cheaper, and perfectly sized for most open-source LLM serving and experimentation.”
• Optimal Solution:
• “If you want the best GPU for GenAI, now you can get just one—no clusters required.” – What really fits on a single H100? Qwen3‑32B for General Chat and Strong Reasoning:
• “Qwen3‑32B is a dense, 32.8B parameter model with up to 32k token context—so actual document chat, coding, even agentic use-cases work great.”
• Performance:
• “On a single H100, it cruises at ~1,500 tokens/sec, serving a few simultaneous chats with headroom.” Mistral/Mixtral for Efficiency:
• “Mistral 7B and Mixtral‑7x8B: these are optimized for inference, even outperforming 70B dense models in some benchmarks—despite activating way fewer parameters per token.”
• Performance:
• “TensorRT‑LLM on H1100 gives about 3x the throughput of A100. That means you get near-70B performance for a fraction of the hardware.” Llama 2‑13B / 70B: Perspective and When Single Isn’t Enough:
• “Llama 2‑13B flies on a single H100 (5,000+ tokens/sec); 70B can work but for true at-scale, sometimes you do need multiple GPUs.”
• Use Cases:
• “For most chat and RAG products, 13–32B fits and flies. 70B is only a must for the highest quality or biggest models.”
• “If you need ≤32B dense (or 13B active MoE), one H100 is enough—for real-time chatbots and GenAI APIs.” – Framing the Value:
• “Let’s look at three common GenAI architectures that work beautifully on a single H100—without distributed infrastructure or cluster headaches.” Pattern 1 – Low-latency chat/inference API:
• “You can serve real chatbots and inference APIs on just one server: load models with Hugging Face Transformers, serve with vLLM or TensorRT‑LLM, and front it with a simple API gateway.”
• Benefits:
• “H100’s grunt means higher concurrency and less code complexity—no sharding or parallelism tricks needed.” Pattern 2 – Retrieval-Augmented Generation (RAG):
• “For RAG, keep your vector DB and docs off-GPU, let the H100 do the heavy LLM generation.”
• Benefits:
• “This keeps costs down and performance up. Modern MoE models even fit in one H100 with smart quantization—no special hardware hackery needed.” Pattern 3 – Agentic workflows:
• “Want to chain actions and make agents? One H100 runs a 32B planner model, which calls tools (via HTTP, Lambda, containers etc).”
• Benefits:
• “The GPU is only busy ‘thinking’, so you can power multiple agent flows and users at once, per instance.” Summary:
• Build most GenAI products—chat, APIs, RAG, and agents—using just one H100 instance.” – Framing the Comparison:
• “Let’s translate everything so far into a simple decision: when does one H100 make sense, and when is a cluster truly warranted?” Efficiency:
• “Studies show multi-GPU clusters rarely give perfectly linear scaling. With 8 GPUs, you often get just 75–85% of the speed you expect—inter-GPU communication slows things down, especially for real-time inference.” Cost and Agility Advantage:
• “With p5.4xlarge pricing around $3.90/hr, small teams can run production-grade GenAI for a few dollars, no need for huge cluster commitments.” Single H100 Use Cases:
• “If your model is 13–32B or an MoE with ~13B params active, a single H100 delivers: dev, internal tools, early production, moderate traffic—all without cluster headaches.” Multi-GPU Use Cases:
• “Only go multi-GPU if you need true 70B+ scale or must support huge traffic.” Future Reference:
• “If you ever push past what a single H100 can do, the H200 is on the horizon with higher throughput.”
• “But for nearly everyone today, H100 is the sweet spot.” – Summarize the Key Takeaways:
• “For most open-source LLM workloads, you don’t need a giant cluster—just a single H100 p5.4xlarge tuned to your needs.” Models That Fit:
• “We’ve seen the models that fit—Qwen3‑32B, Mistral, Llama2‑13B, Mixtral—all run smoothly on a single H100.” Modern Architectures:
• “We explored three modern architectures—chat APIs, RAG, agentic workflows—each simplified and scalable with one H100.” Cost and Complexity Savings:
• “And you’re saving money and complexity, only moving to clusters for ultra-high concurrency or largest models.”

26

Adit Modi

Run OSS LLMs on a Single H100 Smarter, Cheaper, Faster

Why this session and who it’s for

• Do you really need 8 GPUs to run GenAI?
• Stop overpaying and over-engineering your LLM infrastructure. Who is this Session For?
• Small teams, startups, and individual engineers tired of overprovisioning and managing bulky multi-GPU clusters.
• Builders who want to run open-source LLMs on their own terms—without becoming full-time cluster admins.
• Anyone curious how to get big-model performance for chatbots, APIs, and RAG without huge bills. Who Is This Session NOT For?
• Teams who use managed, hosted LLM platforms like Bedrock, SageMaker JumpStart, or prefer “LLM-as-a-service” with no infrastructure control.
• Those building ONLY with proprietary cloud APIs (e.g., OpenAl, Anthropic) with no plan to self-host models.
• Anyone who needs 70B+ parameter models at high QPS and is comfortable with large, distributed clusters. The “overkill cluster” problem for OSS LLMs Most GenAI use cases don’t need this much! So is there a better way?
• Teams default to massive, multi-GPU cloud setups for open-source LLMs—just to be safe.
• This brings complex distributed inference, slowdowns, and big costs, often for workloads that are actually light.
• In reality, most modern product use cases run faster and cheaper with a single strong GPU and a smaller 7B/13B/32B model.
• Right-sizing hardware means lower latency, less hassle, and big savings. Running LLMs doesn’t have to mean complex, multi-GPU clusters.
• Overprovisioning adds cost and complexity but is the default for many teams.
• For most modern OSS models and typical product workloads, a single well-equipped GPU is all you need for speed, simplicity, and savings. Meet p5.4xlarge: a single H100 for real workloads One Good GPU—Finally Within Reach
• Amazon Web Services recently introduced new single-GPU P5 instance sizes:
• p5.4xlarge gives you the latest NVIDIA H100—no more overbuying, no waste p5.4xlarge: Specs & Simplicity
• p5.4xlarge packs everything you need:
• 1xH100 GPU, 16 vCPUs, 256 GiB RAM, nearly 4 TB NVMe SSD, and 100 Gbps network. All the power of a flagship GPU with simple deployment and a lower bill.
• You can rent this for around $3.90 per hour in some regions, compared to paying for all 8 GPUs on bigger P5s.

What fits on one H100? (Qwen-32B, Mistral, Llama-2 etc)

Qwen3-32B — Big Reasoning, Single GPU

• Qwen3-32B: 32.8B dense parameters, 32k context, strong chat/coding
• Runs on a single H100 at ~1,500–1,800 tokens/sec
• Ideal for agentic chatbots and API workloads (request batching possible) Mistral/Mixtral — Efficient Power
• Mistral 7B: flies on H100, 3x throughput vs. A100 using TensorRT-LLM
• Mixtral-7x8B (MoE): near-70B performance, much lower VRAM use
• Efficient models, great for cost and speed — perfect for production Llama 2 (13B/70B) — Reality Check
• Llama 2-13B: 5,000–6,000 tokens/sec on single H100 (with optimizations)
• Llama 2-70B: can run, but high latency — best for true 70B+ needs only
• Most products get great results with 13B– 32B on 1 GPU Rule of Thumb—What Can One H100 Do?
• If your LLM is <32B dense or an MoE with ~13B active params, a single H100 is enough for responsive, multi-user chat and RAG APIs H100: The new sweet spot for OSS GenAI serving
• Design patterns on one GPU: APIs, RAG, and agents Chat & API Serving Patterns
• Run LLM APIs on a single H100: Hugging Face + vLLM/TensorRT-LLM
• Serve multiple users at low latency—no cluster ops or sharding
• Simple, fast, production- ready deployment Retrieval-Augmented Generation (RAG)
• Store vector DB/docs off-GPU; use H100 for LLM generation
• Single H100 gives high throughput & rapid GenAI for RAG APIs
• Mixtral/MoE models: fit single GPU with quantization Agentic Workflows
• “Planner” LLM (Qwen/Mistral) on H100 routes tasks to external tools
• GPU used for thinking, tools run elsewhere (HTTP, Lambda, containers)
• Supports multi-agent, multi-user flows—on one instance Recap—One H100, Many Patterns
• Build most GenAI products—chat, APIs, RAG, and agents— using just one H100 instance.
• Cost and right-sizing: when one H100 is enough

One H100 vs Multi-GPU Clusters

• Multi-GPU clusters: 75–85% efficiency, rarely 8x speedup
• Inter-GPU communication slows scaling—more hardware, more complexity
• A single H100 delivers strong performance for most OSS models Cost: p5.4xlarge is Budget Friendly
• Only $3.90/hour (Capacity Blocks, select regions)
• Run experiments, dev, or production— no need to pay for 8 GPUs
• Great for small teams, startups, and internal tools When to Choose H100 vs Multi-GPU
• Single H100: ideal for 7B– 32B dense or ~13B active MoE LLMs, modest QPS
• Multi-GPU: Required for true 70B+ models or large- scale fine-tuning
• H200: Next step up for ultra-high throughput (future ready) Recap & Key Takeaways
• Most OSS LLM workloads: Single H100 p5.4xlarge is the smart move
• No cluster complexity, no wasted spend—just run, tune, and deploy Three Keys to Remember
• Which models fit? Qwen3-32B, Llama 2-13B, Mistral, Mixtral—up to 32B
• Which patterns? Chat API, RAG, agentic workflows—all on one GPU
• When to use: Development, internal tools, early-stage products, and modest concurrency Try it Yourself + Reference Architecture
• Spin up Amazon EC2 p5.4xlarge; deploy with vLLM/TensorRT-LLM
• Measure your own latency, throughput, and cost savings