Hypura: A Storage‑Tier‑Aware LLM Inference Scheduler for Apple Silicon

Hypura is a storage‑tier‑aware LLM inference scheduler that lets Apple Silicon devices run language models far larger than their physical memory, delivering usable token‑per‑second rates without crashing.

What Is Hypura?

Developed as an open‑source project on GitHub, Hypura targets the growing community of developers, AI researchers, and tech enthusiasts who own MacBook Pro, Mac Studio, or Mac Mini machines equipped with Apple Silicon (M1, M2, M1 Max, M2 Pro, etc.). These devices combine fast unified memory with high‑throughput NVMe storage, yet their limited RAM often prevents loading state‑of‑the‑art models such as Mixtral 8×7B or Llama 70B.

By intelligently placing model tensors across three storage tiers—GPU, RAM, and NVMe—Hypura transforms “out‑of‑memory” errors into smooth, interactive inference sessions. The scheduler is built in Rust, leverages Metal for GPU acceleration, and follows a storage‑tier‑aware placement algorithm that respects access patterns, bandwidth costs, and hardware capabilities.

Core Features & Architecture

1. Tier‑Aware Tensor Placement

• GPU tier: Critical tensors (attention matrices, embeddings, layer norms) stay on the Metal GPU for nanosecond‑scale access.
• RAM tier: Overflow layers that exceed GPU capacity are memory‑mapped, allowing fast CPU‑side reads.
• NVMe tier: The bulk of model weights (e.g., dense FFN slices) stream directly from the SSD using pread() with F_NOCACHE, eliminating OS‑level swapping.

2. Expert‑Streaming for MoE Models

Mixtral‑style Mixture‑of‑Experts (MoE) models fire only a subset of experts per token. Hypura’s router interception identifies the active experts and streams just those slices from NVMe, cutting I/O by up to 75 % and achieving a 99.5 % cache‑hit rate with its neuron cache.

3. Dynamic Prefetch & Pool Buffers

For dense models like Llama 70B, Hypura creates a dynamically sized pool buffer that prefetches upcoming FFN layers based on a look‑ahead depth automatically calculated from available memory. This ensures the GPU never stalls waiting for data.

4. Automatic Hardware Profiling

On first run, Hypura profiles the device’s GPU working‑set size, RAM bandwidth, and NVMe sequential read speed. The profiling data drives a linear‑programming optimizer that produces the optimal placement plan—no manual tuning required.

5. Ollama‑Compatible HTTP API

Hypura exposes a drop‑in Ollama‑compatible server, making it instantly usable with tools like AI Chatbot template, OpenClaw, or any client that speaks the Ollama protocol.

6. Seamless Integration with UBOS

Because Hypura is a Rust library with a clean C‑FFI layer, it can be wrapped inside the UBOS platform overview. This enables developers to embed high‑performance inference into UBOS‑hosted web apps, workflow automations, or AI marketing agents.

Performance Benchmarks

All tests were performed on an M1 Max Mac Mini with 32 GB unified memory and a 5.1 GB/s NVMe SSD. Results illustrate the stark contrast between naive llama.cpp loading and Hypura’s tier‑aware scheduling.

Key takeaways:

• When a model fits entirely in GPU + RAM, Hypura adds zero latency overhead.
• For MoE models, expert‑streaming reduces I/O by up to three‑quarters, delivering token rates that make real‑time chat feasible.
• Dense‑FFN streaming extends the same principle to non‑MoE giants, turning previously “OOM‑only” workloads into productive inference pipelines.

Installation & Quick‑Start Guide

Hypura is distributed as a Cargo workspace. Follow these steps to get up and running on your Apple Silicon machine.

Prerequisites

• Rust 1.75+ (rustup install stable)
• CMake (required for the vendored llama.cpp library)
• Git with submodule support

Step‑by‑Step Installation

git clone --recurse-submodules https://github.com/hypura/hypura.git
cd hypura
cargo build --release
# Binary will appear at target/release/hypura

Future releases will be available via a Homebrew tap, simplifying the process for non‑technical users.

First Run – Profiling Your Hardware

./target/release/hypura profile

This command runs once, caches the hardware profile, and informs the scheduler of GPU working‑set limits, RAM bandwidth, and NVMe read speed.

Running Inference

# Simple one‑off generation
./target/release/hypura run ./model.gguf --prompt "Explain Hypura in one sentence."

# Interactive chat mode
./target/release/hypura run ./model.gguf --interactive

# Benchmark against naive baseline
./target/release/hypura bench ./model.gguf

Serving via Ollama‑Compatible API

./target/release/hypura serve ./model.gguf --port 8080
# API endpoint: http://127.0.0.1:8080/api/generate

Clients such as AI Chatbot template can now point to this endpoint and enjoy high‑performance inference without code changes.

How Does Hypura Compare?

Several open‑source inference engines target Apple Silicon, but they differ in memory handling, performance, and ease of integration.

llama.cpp (Metal backend)

• Excellent raw GPU speed when the model fits entirely in memory.
• Fails catastrophically for models > GPU + RAM capacity (OOM or OS swapping).
• No built‑in tier‑aware scheduler; developers must manually chunk models.

vLLM (CUDA‑only)

• Optimized for NVIDIA GPUs; not applicable to Apple Silicon.
• Relies on GPU memory paging, which Apple’s unified memory architecture does not expose.

DeepSpeed (ZeRO‑Offload)

• Provides CPU‑offload for large models on Linux/Windows.
• Requires extensive configuration and is not tuned for NVMe‑based streaming on macOS.

Hypura’s Unique Edge

• Storage‑tier awareness: Explicitly models GPU, RAM, and NVMe as separate tiers.
• MoE‑specific expert streaming that leverages sparsity for massive speed gains.
• Zero‑overhead mode when the model fits, preserving the raw Metal performance of llama.cpp.
• Automatic hardware profiling eliminates manual tuning.
• Ollama‑compatible API for plug‑and‑play integration with existing AI tooling.

Explore the Source Code

All of Hypura’s logic, from the scheduler to the NVMe backend, is openly available on GitHub. Developers can clone the repository, contribute improvements, or fork it for custom extensions.

Visit the official repository here: Hypura GitHub repo.

Why UBOS Is the Perfect Companion

UBOS provides a unified platform for building, deploying, and scaling AI‑powered applications. By pairing Hypura with UBOS, you gain:

• Rapid prototyping via the UBOS templates for quick start, including the AI SEO Analyzer and AI Article Copywriter.
• Integration with AI Chatbot template to expose Hypura’s Ollama API as a conversational assistant.
• Automation of data pipelines using the Workflow automation studio, perfect for batch inference jobs.
• Scalable deployment on the Enterprise AI platform by UBOS, ensuring enterprise‑grade security and monitoring.
• Cost‑effective plans via the UBOS pricing plans, suitable for startups and SMBs alike.

Whether you are a solo developer or part of a growing startup, the UBOS for startups program offers mentorship, credits, and a community of AI innovators.

Explore real‑world use cases in the UBOS portfolio examples and see how other teams have leveraged AI inference at scale.

For a deeper dive into building AI agents, check out the AI marketing agents page, which demonstrates how Hypura‑powered models can generate personalized campaigns in seconds.