Overview
Yanah Chat is a free, zero-configuration desktop app for Apple Silicon Macs that runs large Mixture-of-Experts (MoE) models locally by streaming weights from the internal SSD instead of holding them all in RAM.
- Delivery: one
.dmginstaller with the runtimes pre-wired; the AI models download automatically during install. The installer offers two options: just the 9B model (about 25GB), or all four models (about 150GB). No CLI, no Swift compilation, no manual model paths, no quantization tuning. - Target hardware: Apple Silicon, from base 8GB machines (proof of concept) to 16GB+ Pro/Max machines (production speed).
- Design priority: zero lag. The goal is a conversation that feels like talking, not like emailing an assistant.
MoE + SSD streaming
Only the experts a token needs are loaded into memory; the rest of the weights stay on the SSD until called.
Why dense models can't do this. A dense model computes every parameter for every token, so all weights must sit in RAM. A dense 26B model would crash an 8GB machine, or stall if forced through an SSD swap file.
Why MoE can. An MoE model activates only a small slice per token. Gemma 4 26B-A4B activates about 4B of its 26B parameters per token; A3B/A4B variants generally run 3B–4B active out of 35B–80B total.
How the streaming works (example: TurboFieldfare running Gemma 4 26B-A4B):
- A small shared core (about 1.35 GB) and a 4K KV cache stay resident in unified memory.
- The full 14.3 GB weight file stays on the SSD rather than being loaded into RAM.
- A custom Swift/Metal engine uses an LFU cache and chunked prefill to page in only the active experts, on demand.
- Most experts are cold at any given moment, so paging them from Apple Silicon NVMe costs latency rather than gigabytes of RAM.
Why the storage bus doesn't choke. A full-model swap moves the entire model; expert paging moves only the active slice, and the LFU cache keeps frequently used experts warm.
Runtime stack and backends
Yanah bundles several inference backends under one desktop UI.
| Backend | Role |
|---|---|
| llama.cpp | Memory-mapped (--mmap) loading and chunked prefill; runs the 177B MoE flagship |
| Swiftlet / sushi | Native Swift/Metal runtimes for Apple Silicon that stream MoE experts from SSD (sushi is built on MLX) |
| TurboFieldfare | SSD streaming framework (shared core + LFU expert paging) |
| EXA | Live web search when online (50 free searches per day) |
Request path:
- You type a prompt in the desktop app.
- The app hands it to the local inference engine (llama.cpp, Swiftlet or sushi).
- The engine pulls active expert weights through the SSD streaming layer.
- Tokens stream back to the app.
Memory system
Yanah's long-term memory is separate from the KV cache: plain Markdown files hold the memories, and a local SQLite index makes them fast to retrieve.
- Ground truth in
.mdfiles. Human-readable; you can open, grep, orgit diffthem. - SQLite as the retrieval layer. A derived index for low-latency lookup, without a heavy vector database.
- Designed by the AI that uses it. The AI built and tested the system, with repeated adjustments until it was satisfied with how it worked.
- Built for zero lag. Avoids the mid-conversation round trips that make many local agent frameworks feel slow.
Quantization, context and connectivity
Quantization: Q4 standard. Yanah aims for 4-bit quantization, the widely used sweet spot where output quality stays close to the original FP16 weights while files shrink enough for consumer hardware. Aggressive sub-2-bit quantization is avoided because it degrades reasoning.
KV cache. The streaming example keeps a 4K KV cache resident. For long sessions, sliding-window attention or a quantized (e.g. 4-bit) KV cache keeps context from consuming the unified memory pool.
Connectivity has two modes:
- Online: EXA web search, up to 50 free searches per day.
- Offline: chat and memory keep working, including reasoning and analysis, writing, help with code, summarizing and brainstorming.
Privacy design. Local storage (Markdown + SQLite), no outbound telemetry, no background phone-home daemons. Apart from the one-time model download during install, the only network traffic is EXA web searches.
SSD wear. Weights are written once at install and then only read during inference via memory mapping, so they add essentially no write wear.
What's in Yanah today
Yanah Chat is completely free: every model tier, no subscription.
- Chat with a local model sized to your Mac (9B, 26B, 80B or 177B).
- Memory that carries across conversations.
- Web search: 50 free EXA searches per day.
- No file-system access: Yanah is a sandboxed chat app. It doesn't read or change your files.
Open-core split:
- Open source: the glue code (for credibility and auditability)
- Proprietary: the Markdown + SQLite memory system
For what's coming next, see Thinking Forward.