---
name: on-device-llm-survival
description: >-
  Harden an on-device LLM feature for real phones and Macs. Memory check before load, single model owner, thermal policy, resumable verified downloads, process-death recovery, and MLX wired limits. Swift for iOS, Kotlin for Android, Python for MLX.
---
# Keep On-Device LLMs Alive on Real Phones

Keep your local AI feature running after launch day, so users do not see the app vanish, slow to a crawl, or lose a half-downloaded model.

Course home: https://edgefde.dev/courses/on-device-llm-survival/

## When to use

Use when a local LLM works in a demo but users report crashes, jetsam kills, slow answers after a few minutes, broken model downloads, or Mac reboots during long MLX runs.

## Environment

Xcode 27 with an iOS 27 device. Android Studio, Kotlin, minSdk 30+, real phone. Python 3.11+ with uv: `uv init --python 3.11`, `uv venv`, `uv add torch mlx mlx-lm`, `uv run ...`. PyTorch uses MPS on Apple silicon, CPU elsewhere.

## Recipe

1. **Fit the model in the memory budget.** Add a memory gate that decides, before each load, whether a model fits in the app's current memory budget. Brief: `01-memory-budget-and-jetsam.llms.md`.
2. **Load one model at a time.** Put all model loads behind one serialized owner that frees memory first and unloads on memory pressure. Brief: `02-one-model-load-at-a-time.llms.md`.
3. **Plan for heat: burst vs sustained.** Map device thermal state to a generation policy on iOS and Android, and log it for every answer. Brief: `03-thermal-sustained-vs-burst.llms.md`.
4. **Survive downloads and process death.** Download a large model in a way that resumes after failures and process death, check it, and restore the chat after the app is killed. Brief: `04-downloads-and-process-death.llms.md`.
5. **Keep a Mac MLX box stable.** Run an MLX service on a Mac with a set wired limit, a cache limit, a prompt cap, one process, and real footprint logging. Brief: `05-mac-mlx-wired-memory.llms.md`.

## Acceptance checks

- The app logs available memory and the model's measured footprint before every load, and refuses the load when the check fails.
- Only one code path can load or unload a model, and it is an actor (Swift) or a Mutex-guarded class (Kotlin).
- The app reacts to thermal state changes with a documented policy table.
- Killing the app during a download and relaunching resumes the download; a corrupted file fails the checksum and is never loaded.
- The Mac service sets a wired limit below max_recommended_working_set_size, caps prompt tokens, and samples the process footprint.

## Key concepts

- The limit that kills your app is the per-app memory limit, not the phone's RAM. Ask the OS how much is left before you load.
- Load, warm up, and unload models through one owner, one at a time.
- Heat lowers speed over minutes. Design a short burst mode and a slower sustained mode.
- A model download must survive app kills, network loss, and reboots, and must be checked before use.
- On a Mac, keep MLX inside the GPU working set and cap context length. Kernel panics in IOGPU are an OS bug you can only avoid, not fix.

## Takeaways

- You can refuse a model load with a clear message instead of a silent jetsam kill.
- You can fall back to a smaller model or a server path when memory or heat says no.
- You can resume a 2 GB download instead of starting again, and you never load a half-written file.
- You can run a Mac MLX service for hours with a set wired limit, a context cap, and a real footprint check.

## Sources

- Apple: [os_proc_available_memory()](https://developer.apple.com/documentation/os/os_proc_available_memory), [Increased Memory Limit entitlement](https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.kernel.increased-memory-limit), [Extended Virtual Addressing entitlement](https://developer.apple.com/documentation/bundleresources/entitlements/com.apple.developer.kernel.extended-virtual-addressing), [jetsam event reports](https://developer.apple.com/documentation/xcode/identifying-high-memory-use-with-jetsam-event-reports)
- Apple: [ProcessInfo.thermalState](https://developer.apple.com/documentation/foundation/processinfo/thermalstate-swift.property), [URLSession downloadTask(withResumeData:)](https://developer.apple.com/documentation/foundation/urlsession/downloadtask(withresumedata:)), [didReceiveMemoryWarningNotification](https://developer.apple.com/documentation/uikit/uiapplication/didreceivememorywarningnotification)
- Android: [ADPF Thermal API](https://developer.android.com/games/optimize/adpf/thermal)
- [LiteRT-LM #2545](https://github.com/google-ai-edge/LiteRT-LM/issues/2545): a large model section could not get one contiguous memory map inside a big iOS app, the engine still said ready, then inference failed
- MLX: [mx.set_wired_limit](https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.set_wired_limit.html), [mlx #3186](https://github.com/ml-explore/mlx/issues/3186) (IOGPU kernel panic during long prefill), [mlx #3896](https://github.com/ml-explore/mlx/issues/3896) (peak memory counter under-reports the real footprint)