Guidance For Scalable Model Inference And Agentic Ai On Amazon Eks

Cost Effective and Scalable Model Inference and Agentic AI on Amazon EKS

Table of Contents

• Overview
• Architecture
• Architecture Steps
• Plan Your Deployment
  • Cost
  • Sample Cost Table
  • Security
  • Supported AWS regions
  • Service Quotas
  • Third party dependencies disclaimer
• Quick Start Guide
  • Important Setup Instructions
• Important Notes
  • Architecture Benefits
  • Key Improvements
• Notices

Overview

This solution implements a comprehensive, scalable ML inference architecture using Amazon EKS with AutoMode, leveraging AWS Graviton processors for cost-effective CPU-based inference and GPU and AWS Inferentia instances for accelerated inference. The system provides a complete end-to-end platform for deploying large language models with Agentic AI capabilities, including RAG (Retrieval Augmented Generation) and intelligent document processing.

Architecture

Infrastructure Architecture

 _Figure1. Reference Architecture for EKS cluster with add-ons_

Agentic Application Architecture

 _Figure2. Reference Architecture Gen AI applications deployed on the EKS cluster_

The architecture diagrams illustrate our scalable ML inference solution with the following components:

1. **Amazon EKS Cluster**: The foundation of our architecture, providing a managed Kubernetes environment with automated provisioning and configuration.

2. **Managed Karpenter Auto-scaling**: Dynamically provision and scale compute resources based on workload demands across