rai vs guidance-for-scalable-model-inference-and-agentic-ai-on-amazon-eks

CPU-only LLM inference engine in pure Rust — 4-bit quantized models, hand-written AVX2 kernels, speculative decoding, and a local HTTP/MCP server. No GPU, no Python runtime.

Comprehensive, scalable ML inference architecture using Amazon EKS, leveraging Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and M

MCP Servers

MCP Servers

10

—

—

Rust

—

Apache-2.0

—

—

—

16

0