rai

10

CPU-only LLM inference engine in pure Rust — 4-bit quantized models, hand-written AVX2 kernels, speculative decoding, and a local HTTP/MCP server. No GPU, no Python runtime.

Visit Site GitHub
Category MCP Servers
Language Rust
License Apache-2.0
Added Oct 3, 2026
Views 15

About

CPU-only LLM inference engine in pure Rust — 4-bit quantized models, hand-written AVX2 kernels, speculative decoding, and a local HTTP/MCP server. No GPU, no Python runtime.

Is this your product? Claim this listing with an email at your domain to post updates.

Is this your project?

Claim this listing to manage your page, access analytics, and unlock upgrades. Verification takes 60 seconds.

Log In to Claim

Share This Project

Share on X Share on LinkedIn

Embed Badge

Add this badge to your README:

[![Listed on AiList](https://hifriendbot.com/ai-list/badge/rai.svg)](https://hifriendbot.com/ai-list/rai/)
Listed on AiList

Compare

rai vs flameox rai vs genpark-speculative-decoding-compound-ai-serving-runtime-skill rai vs guidance-for-scalable-model-inference-and-agentic-ai-on-amazon-eks

Similar Projects

flameox

7

Runtime evidence that helps agents trace, profile, and burn down hotspots in application and native code, GPU kernels, and inference stacks.

MCP Servers

genpark-speculative-decoding-compound-ai-serving-runtime-skill

8

Speculative decoding compound AI high-throughput serving runtime (Fireworks AI style)

MCP Servers

guidance-for-scalable-model-inference-and-agentic-ai-on-amazon-eks

21

Comprehensive, scalable ML inference architecture using Amazon EKS, leveraging Graviton processors for cost-effective CPU-based inference and GPU instances for accelerated inference. Guidance provides a complete end-to-end platform for deploying LLMs with agentic AI capabilities, including RAG and M

MCP Servers

List Your Project

Join the directory Ai agents read. Free forever.

Submit Your Project