Evaluate, Design & Deploy Inference Environments
You have a model - or an application idea. Where should it run on AWS? What will it cost for your traffic? Can it meet your response-time needs?
Answering those questions usually means piecing together model documentation, pricing, service limits and deployment scripts. This accelerator brings that work into one workspace.
Benchmark example: historical API costs, not current AWS quotes. How to read this comparison.
- Discover: browse Amazon Bedrock models or inspect a Hugging Face model.
- Compare: check supported Bedrock API, Custom Model Import and SageMaker options using AWS pricing and a deterministic solver.
- Understand: open the decision map to see costs, exclusions and missing evidence. An optional Advisor helps when you get stuck.
- Size: explore CPU or GPU experiments with model memory, traffic units, explicit assumptions and a recorded CPU speech example.
- Tokenomics: compare public On-Demand and 1-year/3-year Savings Plan costs for GPU pools, including upfront payments and the full commitment.
- Evaluate: explore published coding benchmarks, or test your own workload for quality, latency, throughput and cost. Inspect the evidence and exclusions behind each comparison.
- Try and remove: approve a supported, budget- and time-limited SageMaker trial, send authenticated requests, and track resource removal.
Saved projects let you move freely between these steps. Missing measurements stay unverified.
A deployment recipe is the code and configuration that packages a model, starts its serving software, and runs an inference trial with a budget and time limit. Included recipes support small Qwen2.5 text models, compatible fine-tuned Qwen2 checkpoints, and Magpie speech on CPU. Inspecting a model does not automatically make it deployable. See Deployment recipes to enable an included recipe or add support for another model.
The Advisor uses skill files for decision methods and AWS Knowledge MCP for current AWS documentation. Prices and account facts come from AWS APIs; unavailable checks stay unverified.
Prepare AWS CLI v2 credentials, Python 3, Node.js/npm, and a running Docker engine
with Buildx. See the deployment prerequisites.
Replace YOUR_ACCOUNT_ID with your target AWS account:
git clone https://github.com/aws-samples/sample-open-weight-inference.git
cd sample-open-weight-inference
./deploy.sh --region us-east-1 --expect-account YOUR_ACCOUNT_IDThe script prepares local dependencies, runs checks, deploys with CloudFormation, and prints your application URL. Create your sign-in to start your first project.
AWS charges apply. Remove inference trials in the app before using destroy.sh to remove the application. See costs and cleanup.
This is sample code, for non-production usage. You should work with your security and legal teams to meet your organizational security, regulatory and compliance requirements before deployment
Cognito MFA is optional, not enforced. Grant deployment and approval permissions deliberately, use synthetic data, and review the security controls and deployment settings before installing. Report vulnerabilities privately as described in SECURITY.md.
The Advisor explains; the solver decides. CloudFront serves the frontend, while authenticated API requests go directly to AgentCore.
| Task | Guide |
|---|---|
| Deploy, sign in, or remove the application | Getting started |
| Understand hosting paths and deployment decisions | Architecture, decision map |
| Bring your own fine-tuned model | Fine-tuned checkpoints |
| Enable trials or support another model | Deployment recipes |
| Compare public benchmarks or test your own workload | Evaluation and benchmark guide |
| Size compute and compare hosting costs | CPU and GPU planning, cost model, Tokenomics |
| Understand Advisor sources and decision methods | AWS documentation MCP, inference skills |
| Review access controls and security settings | Security considerations |
| Develop locally, test, or contribute | Contributing |
Licensed under MIT-0. Third-party terms are listed in NOTICE.