Mac LLM cluster: the complete 2026 guide to home Mac inference#

A 4-node Mac Studio M3 Ultra cluster with RDMA over Thunderbolt 5 and EXO 1.0 runs Kimi K2 Thinking (1T params, 32B active) at about 25 tok/s, DeepSeek V3.1 671B at 32.5 tok/s, and Qwen3 235B-A22B at 31.9 tok/s. The rig costs under $40,000, draws 600 to 800 W under load, and fits on a single 15 A residential circuit. An equivalent-VRAM NVIDIA build typically requires a 240 V dryer-circuit run or sub-panel addition. Two December 2025 releases made this happen: macOS Tahoe 26.2 shipped native RDMA over Thunderbolt 5, and EXO Labs released 1.0 with RDMA support. This is the full 2026 reference on when that math wins, when it doesn't, and how to build the rig.

The framing#

A Mac cluster is a memory-capacity machine, not a throughput machine. Apple Silicon's unified memory lets a 2- to 4-node cluster fit models no single consumer NVIDIA card can hold, at a fraction of the wall draw, panel-circuit load, and noise. At batch_size=1 inference is bandwidth-bound, so a Mac with 819 GB/s is competitive on decode. At batch_size>1 a single H100 or RTX Pro 6000 Blackwell out-throughputs a small Mac cluster by a wide margin because the bottleneck moves to FP16 compute, where NVIDIA wins by 10x or more.

The buying decision turns on three questions: which model must you run, how much do you care about wall-plug power and acoustics, and is your workload continuous single-user interactive or many-concurrent batched.

Pipeline vs tensor parallelism#

Distributing inference across a Mac cluster uses one of two parallelism strategies.

Pipeline parallelism (PP) splits layers sequentially. Token activations flow node to node at layer-group boundaries. Activations are tiny (under 4 KB per token for Llama 3.2 3B), so PP tolerates slow networks. The cost: PP does not accelerate single-stream tok/s. It merely lets you run models too large for one node. Single-stream throughput typically drops slightly with additional nodes (network overhead, no compute gain), but aggregate throughput across concurrent requests scales nearly linearly.

Tensor parallelism (TP) splits each layer's weight matrices across nodes. Every device participates in every forward pass. TP can yield near-linear N-times speedups, but requires microsecond-scale latency interconnects. The synchronization volume makes TP impractical without RDMA. The side-by-side comparison of pipeline vs tensor parallelism on a Mac cluster walks the math and benchmark numbers in detail.

The practical decision rule: if a model fits on one Mac, do not cluster it for single-stream interactive use. Performance gets worse, not better. Cluster for capacity (model does not fit on one node), or for batched workloads where PP aggregate throughput pays off.

The Qwen3 235B-A22B numbers make the point. On llama.cpp RPC across an M3 Ultra cluster, throughput went 20.4 to 17.2 to 15.2 tok/s as nodes were added. On the same hardware with EXO plus RDMA: 19.5 to 26.2 to 31.9.

The framework landscape#

Five tools matter and they are not interchangeable.

EXO (github.com/exo-explore/exo, 1.0 December 2025) is the production tool for cross-Mac clusters. PP by default with automatic mDNS-based topology discovery: drop EXO on every node and they self-organize. EXO 1.0's December 2025 release notes detail the two structural additions: RDMA-over-TB5 support and "disaggregated prefill/decode," routing compute-bound prompt processing to one device class and bandwidth-bound token generation to another. Public benchmarks at benchmarks.exolabs.net.

MLX and mx.distributed is Apple's array framework, the official Apple Silicon ML path. Two distributed backends: Ring (TCP-based PP) and JACCL (TP via RDMA), every rank in JACCL holds a shard of every layer. Jobs launch via mlx.launch; primitives like mx.distributed.all_sum() are exposed. The companion vllm-mlx project (arxiv 2511.05502) reports 525 tok/s on a single M4 Max, 21 to 87% higher throughput than llama.cpp on Apple Silicon, and up to 4.3x aggregate throughput at 16 concurrent requests via continuous batching.

llama.cpp RPC (github.com/ggml-org/llama.cpp) runs rpc-server workers exposing GPU memory over TCP. The only viable option for heterogeneous topologies (Mac plus NVIDIA box plus Framework Desktop). It does not speed up as you add nodes; it slows down. The llama.cpp RPC capacity-unlock pattern covers the worker memory leak, the primary-node restart pitfall, and the recent regressions that produce invalid tensor buffers and crash on load.

Ollama has been the most common single-node entry point. On March 30, 2026, the project announced a preview switch to MLX. Community benchmarks on an M4 Pro Mac Mini running Qwen3-Coder-30B-A3B showed ~130 tok/s on MLX versus 43 tok/s on llama.cpp. The 3x delta explains the move.

dnet (github.com/firstbatchxyz/dnet) builds distributed primitives directly atop MLX with a dynamic-topology approach: an API node solves for optimal layer distribution and instructs worker shards which weights to load.

Failure modes worth engineering around#

EXO has a mesh-recovery anomaly. When a node crashes and rejoins, auto-discovery can fail to re-prefer the high-speed Thunderbolt link and instead route tensor synchronizations over the fallback mesh VPN. Inter-node latency spikes from sub-ms to 40 to 70 ms, collapsing throughput. Tracked as issue #1723.

MLX has a Metal command-buffer timeout trap. mx.distributed.send and recv schedule on the GPU stream by default, and Metal enforces a strict ~5-second timeout. If a receiver waits longer than 5 seconds for incoming RDMA data, Metal terminates with a GPU timeout and crashes inference. The sender stays alive because send completes locally as soon as data hits the local RDMA buffer. Workaround: force RDMA send/receive onto the CPU stream with stream=mx.cpu.

The interconnect latency hierarchy#

Network bandwidth is rarely the binding constraint. Latency is. Three regimes coexist in mid-2026.

Ethernet (TCP/IP). Mac Studios ship with 10 GbE; Mac Minis with 1 GbE plus a $100 BTO 10 GbE upgrade. Round-trip latency runs ~300+ µs on 1 GbE, ~200+ µs on 2.5 GbE, sub-ms on 10 GbE direct-attach. For Mac Mini PP clusters, 2.5 GbE suffices because per-token activations are KB-scale.

Thunderbolt Bridge. macOS's IP-over-Thunderbolt feature gives a point-to-point link, but the protocol underneath caps at USB 3.x speeds (~10 Gbps) even on TB4/TB5 cables. Latency: ~100 to 200 µs on TB4, ~50 to 100 µs on TB5. The 2024-era Mac Mini cluster standard, now a fallback when RDMA is unavailable.

RDMA over Thunderbolt 5 (macOS 26.2, December 2025). The structural breakthrough. Apple's technote is TN3205. Available only on TB5 Macs (M3 Ultra Mac Studio and M4 Pro/Max with TB5 ports). Enabling RDMA over Thunderbolt 5 per node takes one rdma_ctl enable command in Recovery Mode plus a reboot, after which TB ports appear to the kernel as InfiniBand interfaces (rdma_en2, etc.) usable via ibv_devices and ibv_devinfo. Round-trip latency drops from ~300 µs (TCP) to 5 to 50 µs, with hot-path measurements as low as 3 µs. Bandwidth is 80 Gbps symmetric per TB5 port, bypassing the kernel network stack entirely. As of mid-2026, only EXO 1.0+ and MLX's JACCL backend consume RDMA. llama.cpp RPC and distributed-llama still use TCP.

The no-TB5-switch ceiling#

There is no Thunderbolt 5 switch on the market, and none credibly announced. All-to-all RDMA requires every Mac to physically cable to every other Mac. The M3 Ultra has five TB5 ports, so a full mesh is possible up to about 5 nodes: a 4-node mesh needs six cables, a 5-node mesh needs ten. Beyond that, operators daisy-chain (latency and partial-bandwidth penalties on multi-hop paths) or fall back to Ethernet PP. The full cabling math and why no vendor ships a TB5 switch explains the residential ceiling in detail.

This is the single biggest structural limit on residential Mac cluster scale today. Unchanged in 18 months. Unlikely to change in the next 12.

Memory bandwidth defines decode#

Token generation at batch_size=1 is memory-bandwidth-bound. Achievable tok/s is approximately memory_bandwidth ÷ active_parameter_bytes. Bandwidth is the most important hardware spec for single-stream inference.

A single M3 Ultra has ~80% of an RTX 4090's bandwidth with 21x the memory capacity (512 GB vs 24 GB). For models that fit in 24 GB the 4090 wins on raw tok/s; for 40 to 700 GB models the Mac is often the only single-box option. The M4 Pro's 273 GB/s is also why base Mac Mini clusters cap at 4 to 8 tok/s on 70B-class: per-node bandwidth, not interconnect, is the wall. Alex Cheema's iconic July 2024 demo of 23 base M2 Mac Minis (368 GB pooled) ran Llama 3 405B Q4_K_S at sub-1 tok/s for exactly this reason. Proof of concept, not production.

Real-world performance#

The pattern: dense models scale via PP for aggregate throughput but not single-stream latency. MoE models with low active-parameter counts get the best clustering speedups because per-token bandwidth is dominated by the active experts, not the full parameter pool.

Residential power: the NEC continuous-load rule#

The structural advantage of Apple Silicon for residential AI is wall-plug efficiency. Apple's published figures per support.apple.com/102027: Mac Studio M3 Ultra 512 GB idles at 9 W and peaks at 270 W; M4 Max Mac Studio peaks at 145 W; M4 Pro Mac Mini PSU rating is 140 W (rarely reached). Real-world LLM inference draw sits well below "max" because decode at batch_size=1 saturates bandwidth, not compute. Independent testing of an M3 Ultra running DeepSeek R1 measured wall draw under 200 W during 671B inference. A 4x M3 Ultra cluster at full inference draws 600 to 800 W, comparable to a single RTX 4090 system.

NEC section 220.41 requires continuous loads (sustained ≥3 hours) to not exceed 80% of breaker capacity. A 24/7 inference cluster is continuous by definition. The 80% rule gives 1,440 W on a 15 A/120 V circuit, 1,920 W on a 20 A circuit, 5,760 W on a 30 A/240 V circuit. About 4 maxed Mac Studio M3 Ultras fit on a 15 A circuit; 6 to 7 fit on a 20 A circuit with headroom. A $40K four-node M3 Ultra cluster running trillion-parameter models comfortably fits on an existing 15 A residential circuit without tripping a breaker. An NVIDIA build of equivalent VRAM capacity typically requires a 240 V dryer-circuit run or sub-panel addition. Easily $1,000 to $3,000 of electrician work, and often a non-starter in rentals.

Always terminate the cluster into a line-interactive UPS. Residential grids exhibit micro-sags that can trigger kernel panics on Apple Silicon SoCs. A 1500 VA / 1000 W unit gives 5 to 8 minutes for graceful shutdown of a 4-node cluster; a 3000 VA double-conversion unit gives 15 to 30 minutes of ride-through but needs its own 20 A circuit.

SSD wear: the most-feared 24/7 failure mode#

Macs ship with soldered, non-replaceable internal SSDs. Failure permanently kills the entire compute node.

macOS swaps aggressively. Inference that pushes close to physical memory limits causes macOS to swap GB/hour to the internal SSD. Mac Studio internal SSDs are rated by community measurement at ~1,200 to 2,400 TBW lifetime. Sustained 5 GB/min swap writes burn through that in 4 to 8 months.

The single most important mitigation: ensure model + KV cache + OS fits comfortably in unified memory. Use sudo sysctl iogpu.wired_limit_mb=458752 on a 512 GB M3 Ultra to allocate ~448 GB to GPU-wired memory, raising the ceiling from the default ~384 GB. Then map highly-active Docker volumes, caches, and model download directories to external Thunderbolt NVMe, curtail non-critical logging, and monitor wear via smartctl -a /dev/disk0 or DriveDx. The Apple Silicon SSD-wear monitoring guide for cluster nodes covers each field, including what Percentage Used and Available Spare mean when planning retirement. Plan for whole-Mac retirement at 80 to 90% wear. The detailed wear math, monitoring procedure, and prevention checklist for a 24/7 Mac inference server walks every step. This setting is the difference between a cluster that runs for years and one that burns through SSDs in eight months.

Headless Mac gotchas#

macOS was not designed as a headless server OS. The headless Mac homelab stack covers Tailscale, dummy plugs, and pmset in detail; the minimum viable config:

• Tailscale via Homebrew (sudo brew services start tailscale), not the App Store app. The latter has a documented "sleep death" failure mode. Use sudo tailscale up --auth-key tskey-... rather than the browser flow.
• HDMI dummy plugs ($5 to $10 EDID emulators). A Mac without a display may not initialize the GPU rendering pipeline, may display 640x480 over Screen Sharing, and may stall Metal compute tasks.
• pmset: sudo pmset -a sleep 0 disksleep 0 displaysleep 0 powernap 0 autorestart 1 womp 1. Enable "Start up automatically after a power failure" in System Settings then Energy.
• FileVault halts at a pre-boot password prompt on every reboot until someone types it on a physical keyboard. Most homelabs in locked residences disable it.
• launchd, not cron, for supervision. Plists in /Library/LaunchDaemons/ with KeepAlive plus a watchdog that pings /health every 30 to 60 seconds and triggers sudo shutdown -r now on no response.
• macOS will not run a system upgrade over SSH. Pin to a known-good macOS version for the deployment.

Economics#

The M3 Ultra 512 GB at $18.6/GB is the most cost-effective dollar-per-GB unified memory on the market. NVIDIA at the same capacity tier: RTX 4090 full system $125 to $167/GB, RTX Pro 6000 Blackwell $208 to $229/GB, H100 SXM $500+/GB. The full Mac Studio vs NVIDIA dollar-per-GB breakdown across every tier shows the M3 Ultra 512 GB is 3 to 4x cheaper per GB than a used quad-3090 rig, 6 to 9x cheaper than dual RTX Pro 6000 Blackwell, 20 to 27x cheaper than H100 systems. Mac GB is not equal to NVIDIA GB in throughput, but it matters when you need to fit the model at all.

For 671B single-stream, tok/s per $1,000: 1.9 on the single 512 GB Mac Studio, 0.81 on the 4x cluster, 0.37 on the 8x M4 Pro Mac Mini setup. The 4x cluster doubles throughput at roughly 4x the cost. Worth it only for concurrent users or larger models. Within the Mac ecosystem, scale-up usually wins over scale-out: four Mac Mini M4 Pros at $10,000 give 256 GB pooled but are bandwidth-limited per node, while a single Mac Studio M3 Ultra 256 GB at $5,999 gives 800+ GB/s on one node, no inter-node communication, and a single failure domain.

Cloud break-even#

Reference cloud prices (May 2026): A100 80 GB at $1.07 to $1.90/hr, H100 SXM at $2.01 to $6.88/hr. Managed inference: Together/Fireworks/DeepInfra at $0.30 to $0.90/M tokens; Groq Llama 70B at ~$0.59/M.

A single M3 Ultra 256 GB ($6,000) amortized over 3 years at ~50% residual, plus $500 electricity and AppleCare ($170), nets to ~$103/month. Break-even vs Groq Llama 70B at $0.59/M is 175 million tokens/month. At 20 tok/s sustained the Mac produces ~52 million tokens/day, ~1.5 billion/month. A Mac running continuously beats cloud the moment you actually use it.

For a $40K 4x M3 Ultra cluster amortized over 3 years (~$13K/year + ~$600 power), beating Together/DeepInfra DeepSeek pricing needs ~20 billion tokens/year. At 32 tok/s single-stream that is ~1 billion tokens/year of natural production: a 20x utilization gap. The cloud GPU vs local Mac cluster break-even math walks the amortization across single-Mac and full-cluster configurations against Together, Fireworks, and Groq pricing. Multi-stream and continuous serving justify the local cluster; episodic single-developer use does not. The primary motivation for a home Mac cluster is privacy, control, and unlimited iteration, not pure cost savings.

Macs also hold value: a 2-year-old M2 Ultra 192 GB sells used at $3,500 to $4,500 (vs $4,800 original), versus 2024-vintage H100s collapsing from $30K+ peak to under $20K. Resale resilience is a real $1,500 to $3,000 per node TCO advantage for Macs over 3 years.

Build recipes#

Trillion-parameter capability ($19,100): 2x Mac Studio M3 Ultra 512 GB ($9,499 each), 1x TB5 cable ($60), 10 GbE direct-attach ($30 DAC, no switch), macOS 26.2+, EXO 1.0 with PP + RDMA. Fits a 15 A circuit. Runs Kimi K2 Thinking at ~24 tok/s.

405B under $10K: 1x M3 Ultra 256 GB ($5,999) primary, 1x M4 Max 128 GB ($3,499) for prefill assist via EXO disaggregated prefill/decode, 1x TB5 cable. Total ~$9,500. Fits 405B Q4 with 128 GB headroom.

Mac Mini cluster (budget): 4x Mac Mini M4 Pro 64 GB ($7,996), 4x TB5 passive cables ($200), 8-port 2.5 GbE switch ($100). Total ~$8,300. 256 GB pooled. 70B-class at ~8 tok/s. Per-node 273 GB/s bandwidth makes this a worse value than two Mac Studios for the same money.

Beyond 4 Mac Studios on TB5 RDMA, or 8 to 10 Macs on 10 GbE PP, the economics and complexity argue for a single big NVIDIA system: a used 4x H100 SXM5 node from a deprecating cloud provider trades at ~$80 to $120K and delivers far more throughput than 10 to 15 Mac Studios, at the cost of 240 V, ~3 kW continuous, and datacenter-grade fan noise.

Timeline of what changed (2024 to 2026)#

• Q3 2024: Alex Cheema's 23-Mini cluster goes viral. EXO trends on GitHub.
• March 2025: Mac Studio M3 Ultra ships with 512 GB ceiling and TB5 ports (no RDMA stack yet). DeepSeek R1 671B on a single Mac becomes the defining demo.
• December 2025: macOS Tahoe 26.2 ships with RDMA over TB5. EXO 1.0 ships with RDMA support. Canonical 4x M3 Ultra benchmarks published.
• March 2026: Ollama announces MLX backend preview.
• April 2026: Apple briefly removes 512 GB Mac Studio; 256 GB upgrade pricing increases. DRAM market signal; pricing remains volatile.

What has not changed in 18 months and is unlikely to change in the next 12: there is still no Thunderbolt 5 switch.

The decision framework#

1. Specific 400 to 700B model, single-stream, capacity over throughput. Buy a single M3 Ultra 512 GB Mac Studio for DeepSeek R1 671B local inference ($9,499). Do not cluster. It runs DeepSeek R1, Llama 3.1 405B, and any sub-trillion model on a single 9 W idle / 270 W max box.

2. Trillion-parameter capability or aggregate throughput for concurrent users. Build a 2- or 4-node M3 Ultra cluster with TB5 RDMA on macOS 26.2 and EXO 1.0. Budget $20K to $40K. Plan around the no-TB5-switch ceiling, residential 15/20 A circuit fit, and EXO's evolving stability surface.

3. 70B-class single-stream and you have cheap used 3090s. Build a dual or quad RTX 3090 NVLink rig. Louder, hotter, needs a basement, garage, or dedicated 240 V circuit. $/tok at 70B and below is unmatched. The Mac case is weaker here.

4. Bursty workload, no data-residency constraint. Stay on cloud. Local hardware does not pay back at <10% sustained utilization for general-purpose inference.

What this means for the buyer#

The case for Macs at home is real but specific. The operator who values silence, fits-in-the-living-room form factor, residential-grade power, and the ability to load and iterate on the very largest open-weight models. The number that matters most for the buying decision is not $/tok. It is how often you actually run inference, what the largest model you need to run is, and whether you can install 240 V in your home.

For the operator who has decided on Macs, the 2026 default build is one or two M3 Ultra Mac Studios at the highest unified-memory tier affordable, on macOS 26.2 or later, running Ollama (transitioning to MLX) or EXO 1.0 with RDMA over Thunderbolt 5, fronted by Tailscale, supervised by launchd watchdogs, with model weights staged from a NAS over 10 GbE.

Buying used, especially a 192 or 256 GB M2 Ultra at $3,500 to $4,500, is one of the strongest secondary-market buys of 2026. The catch is that the secondary market is full of ex-mining, ex-render-farm, and ex-server units whose internal SSD wear, kernel-panic history, and prior heat-cycling history are invisible from a photo. The seller saying "barely used" does not tell you whether the SSD has 12 TBW left or 1,200. That is the exact gap a full Mac diagnostic stack closes: a buyer-verifiable, device-bound report that pulls the real SMART counters off the machine before money moves.