Keep a Mac MLX box stable

cd llm-survival
uv venv
uv add torch mlx mlx-lm
uv run python -c "import mlx.core as mx; print(mx.device_info())"

What people report

This lesson is for teams that run MLX as a local service on a Mac: a coding agent, a nightly batch job, or a small team server. The issue mlx #3186 reports a kernel panic, "completeMemory() prepare count underflow" @IOGPUMemory.cpp:550, during a very long prefill (about 173,000 tokens) on an M4 Max. Other people added the same panic on other Macs, and a separate report (mlx #3346) was merged into it. The panic is inside Apple's IOGPU kernel extension. MLX maintainers said they would check internally. So no setting in this lesson is a proven fix. These steps lower the risk and make memory use visible.

Step 1: read the limits

import mlx.core as mx

info = mx.device_info()
total = info["memory_size"]
working_set = info["max_recommended_working_set_size"]
print(f"total RAM: {total / 2**30:.1f} GiB")
print(f"recommended GPU working set: {working_set / 2**30:.1f} GiB")

Run this on the Mac itself. The keys memory_size and max_recommended_working_set_size are the ones the set_wired_limit docs tell you to use.

Step 2: set a wired limit and a cache limit

The wired limit is how many bytes MLX keeps resident. The docs say: the default is 0, the limit must stay strictly less than total memory, and a value larger than the system wired limit is an error.

import mlx.core as mx

def configure_mlx_memory(share_of_working_set: float = 0.85) -> None:
    info = mx.device_info()
    working_set = info["max_recommended_working_set_size"]
    wired = int(working_set * share_of_working_set)
    previous = mx.set_wired_limit(wired)
    # Limit the buffer cache too, so freed buffers do not pile up for hours.
    mx.set_cache_limit(int(wired * 0.1))
    print(f"wired limit {wired / 2**30:.1f} GiB (was {previous / 2**30:.1f} GiB)")

configure_mlx_memory()

The 0.85 share is our starting choice, not an MLX number. It leaves room for the rest of the system. The docs also show how to raise the system limit with sudo sysctl iogpu.wired_limit_mb=.... Do not raise it on a shared machine unless you have tested the result for hours.

Step 3: cap the prompt

The reports in #3186 tie the panic to very long prompts. Put a hard cap in your service so one request cannot start a giant prefill.

from mlx_lm import load, generate

MAX_PROMPT_TOKENS = 32_000   # pick from your own tests

model, tokenizer = load("mlx-community/Qwen3-4B-4bit")

def answer(prompt: str, max_tokens: int = 512) -> str:
    n = len(tokenizer.encode(prompt))
    if n > MAX_PROMPT_TOKENS:
        raise ValueError(f"prompt has {n} tokens, limit is {MAX_PROMPT_TOKENS}")
    return generate(model, tokenizer, prompt=prompt, max_tokens=max_tokens)

Step 4: one process, and watch the real footprint

In #3186, one user traced two trigger paths in their own logs: a generate call on a worker thread while the main thread cleared the cache, and two Python processes running MLX at the same time. So run one MLX process per Mac, and do not call mx.clear_cache() from another thread while generation runs.

Do not trust mx.get_peak_memory() alone for long-running services. In mlx #3896, it reported about 46 GB while the OS showed about 110 GB. A maintainer suggested get_active_memory() + get_cache_memory() for long or parallel serving. Also check from outside the process:

# Real footprint of the service process, as the OS sees it.
footprint $(pgrep -f my_mlx_service.py)
# Swap and compressor growth are early warnings.
vm_stat 5

import mlx.core as mx

def memory_report() -> dict:
    return {
        "active_gib": mx.get_active_memory() / 2**30,
        "cache_gib": mx.get_cache_memory() / 2**30,
        "peak_gib": mx.get_peak_memory() / 2**30,
    }

Log the report after each request. If active plus cache keeps going up across requests, you have a leak in your own code or a growing cache. For an iOS app that uses MLX Swift, the related problem is GPU work while the app is in the background. The MLX Swift in Production course covers that.

Sources

• MLX docs: mlx.core.set_wired_limit
• MLX docs: mlx.core.metal.device_info