EDGE FDE
Train local. Bake it. Run on the edge.
Edge AI Skills that Ship
Lessons, tips, and tricks for forward-deployed engineers who ship AI at the edge. Start simple. Run the model at the edge.
Available courses
34 coursesBeginner
Model Architectures in Plain English (Edge AI)
Model architectures in plain English. A simple look at language and image model shapes, tied to the clothing-classifier course and From the metal to the model.
Apple coreai-models Catalog, Typed
Core AI models, typed. A simple, typed look at Apple's official coreai-models catalog: LLMs, diffusion, SAM, ASR, detectors, and more.
Core AI vs Core ML vs MLX — Which to Use
Core AI vs Core ML vs MLX. Choose the Apple AI stack that matches your OS, model, and job: Core ML, Core AI, or MLX.
On-Device Semantic Cache for Apple AI
Lower latency and battery use by returning a past answer when the new question is close enough.
Embedding similarity cache + hit-rate tracking for on-device sessions.
Reuse answers that are almost the same. Build a small on-device cache that embeds the question, checks nearby past answers, and records hit rate. iOS 27 and macOS 27 only.
On-Device Prompt Registry & Dynamic Instructions
Roll back a bad prompt in minutes instead of hunting through app builds.
Version prompts like product copy, then route and roll them back with Dynamic Profiles and Dynamic Instructions.
Roll back a bad prompt in minutes instead of hunting through app builds. Version prompts like product copy, then route and roll them back with Dynamic Profiles and Dynamic Instructions.
Intermediate
PyTorch to Core AI in Xcode (Fashion-MNIST)
PyTorch to Core AI in Xcode. Train a clothing classifier on a Mac with MPS or CPU. Convert it with coreai-torch. Then show labels on the device in a SwiftUI Core AI app.
PyTorch to Android with LiteRT
Train the same clothing classifier on a Mac (MPS or CPU). Export it, convert it to LiteRT/TFLite, and show labels on the device in a Kotlin Android app.
From Metal to Core AI Model
From the metal to the model. Start at Metal shaders. Walk the clothing-classifier stack backwards until you return to the original pipeline lesson.
On-Device Context Assembler (Token Budget)
Cut wasted tokens and wrong answers by packing only the memory, search hits, and tools the model needs right now.
Build a token-budgeted context assembler for LanguageModelSession / Dynamic Profiles.
Build a context packer for on-device AI. Rank memory, search hits, and tools into a token budget, then hand that pack to a LanguageModelSession through Dynamic Profiles. iOS 27 and macOS 27 only.
On-Device Retrieval: BM25 + Core AI Embeddings
Let users get answers from their own notes without sending data to a public cloud.
Chunk + BM25 + dense search + rerank for on-device / Core AI embeddings path (honest about APIs).
Search private files on the phone. Index private notes on the device. Combine word search, Core AI embeddings, and a rerank step. Be honest about Spotlight versus a stack you own. iOS 27 and macOS 27 only.
On-Device Model Router (Apple FM)
Spend less and stay fast by sending each task to the smallest model that still works (on-device vs Private Cloud Compute vs custom Core AI).
Model router with cost/latency/quality rules using Dynamic Profiles and model modifiers.
Route work to the right Apple model. Write rules that pick SystemLanguageModel, PrivateCloudComputeLanguageModel, or CoreAILanguageModel, then apply them with Dynamic Profile modifiers. iOS 27 and macOS 27 only.
On-Device Agent Orchestrator
Make agent behavior predictable so product and support can trust what the app will do next.
Deterministic orchestrator with Dynamic Profiles (one active Profile at a time), no LangChain.
Run an agent with a state machine. Own the loop yourself. One Dynamic Profile is active at a time. Tools change state. The app decides the next step. iOS 27 and macOS 27 only.
On-Device Streaming Metrics (TTFT)
Users feel the app is faster when the first token arrives sooner. You can prove it with numbers.
Stream LanguageModelSession tokens and measure TTFT and inter-token latency for on-device and Private Cloud Compute.
Stream tokens and measure speed. Users feel the app is faster when the first token arrives sooner. You can prove it with numbers. Learn the streaming path plus time to first token and inter-token latency for on-device and Private Cloud Compute sessions.
On-Device LLM Tracer (OpenTelemetry-style)
Debug a slow or wrong answer by seeing each hop instead of one opaque duration.
OpenTelemetry-style spans around context pack, tool, model, and guardrail next to Instruments.
Trace every step of an AI call. Debug a slow or wrong answer by seeing each hop: context pack, tool call, model, and guardrail. Learn OpenTelemetry-style spans for on-device AI pipelines.
On-Device AI Eval Harness for CI
Catch regressions in CI so a prompt change does not quietly break the product.
Evaluations harness with trajectory grading and Swift Testing CI gates.
Test AI behavior before you ship. Catch regressions in CI so a prompt change does not quietly break the product. Build an eval harness with trajectory grading and CI gates using Apple's Evaluations framework.
On-Device MCP for Foundation Models
Plug new tools into your Apple AI app without rewriting the chat loop.
Raw JSON-RPC MCP server and client mapped into Foundation Models Tool types.
Talk to tools with MCP. Plug new tools into your Apple AI app without rewriting the chat loop. You will build a raw JSON-RPC MCP server and client, then map those tools into Foundation Models.
Sandboxed On-Device Tool Execution
Protect user data and device resources when the model calls tools.
Isolated tool executor with resource limits, honest about Apple sandbox options.
Run tools in a safe box. Protect user data and device resources when the model calls tools. You will build an isolated tool executor with resource limits, and you will hear what Apple sandbox options actually do on iOS 27 and macOS 27.
On-Device AI Guardrails (Injection + PII)
Keep private data private and block prompt-injection before it reaches the model or leaves the device.
Guardrails middleware with injection checks and PII redaction around LanguageModelSession.
Stop prompt attacks and leaks. Keep private data private and block common prompt-injection tricks before they reach the model or leave the device. You will add guardrails middleware with injection checks and PII redaction for on-device pipelines.
Agent Harness: Build and Improve the Loop
Ship agents that stay safe and get better from real runs, not from model hype.
Define a harness, name its six layers, sketch a small harness, and run an improve loop using traces + golden checks.
The agent harness (and how to improve it). Learn what wraps the model into a real agent, then run a production→trace→eval loop that hardens the harness without fine-tuning first.
MLX Swift in Production (Gemma + scenePhase)
Keep on-device Swift inference stable across app lifecycle and newer model families.
Register a custom MLX model architecture in Swift, load weights safely, and gate inference on scenePhase so Metal is not used while backgrounded.
MLX Swift in production. Ship MLX Swift models that register custom architectures and stop Metal work when the app backgrounds, so inference does not die mid-session.
Keep On-Device LLMs Alive on Real Phones
Keep your local AI feature running after launch day, so users do not see the app vanish, slow to a crawl, or lose a half-downloaded model.
Memory budgets, one-at-a-time model loads, thermal policy, resumable downloads, process-death recovery, and MLX wired limits on a Mac.
Keep your local LLM alive on real phones. Check memory before load, serialize model init, react to heat on iOS and Android, resume model downloads, recover from process death, and set MLX wired limits on a Mac.
Reliable Local Agents (llama.cpp + Ollama)
Make local agents fast and stop one bad reply from breaking the whole chat.
Prompt cache order, safe tool calls, a reply parser for thinking models, JSON vs tools, and sizing one server for several agents.
Make local agent loops fast and safe. Keep the prompt cache warm, validate tool calls, catch thinking output in the wrong field, keep JSON mode and tools apart, and run several agents on one box with llama.cpp or Ollama.
Local LLM Runtime Traps (Ollama + llama.cpp)
Stop quiet runtime problems from making your local model worse without any error.
A CI context check, context and KV memory sizing, a speculative decoding A/B test, and a pin-and-smoke-test upgrade path.
Avoid quiet local runtime problems. Catch Ollama context truncation in CI, size context and KV memory, measure speculative decoding before you trust it, and pin versions so new models and upgrades do not break your app.
Test Local Decision Models (/v1/systemone)
Let a small local model make routine decisions only when it is sure, and send the rest to a person.
A labelled test set, an order-bias check, a quantization comparison, a confidence threshold with human fallback, and latency numbers.
Test a local decision model before it runs your pipeline. Call /v1/systemone on llama.cpp or Ollama, measure accuracy and option order bias, compare quants, set a confidence threshold with a human fallback, and measure latency.
On-Device Search with EmbeddingGemma 2
Get on-device search results you can trust, and catch quiet embedding mistakes before users do.
Correct task prefixes, safe number formats, truncation with renormalization, a setup-checked index, and a recall test.
Build on-device search that does not fail quietly. Use the right EmbeddingGemma 2 prefixes, avoid float16, renormalize truncated vectors, keep one model per index, and test recall.
Advanced
Diffusion LoRA to Core AI On Device
Diffusion LoRA to Core AI on device. Train a subject or style LoRA on a Mac. Merge it into a catalog diffusion base. Export a baked .aimodel and generate on the phone. There is no documented hot-swap LoRA.
LLM Writing-Style LoRA to Core AI
LLM LoRA for your writing style. Prepare your own writing. Train a LoRA on a Mac. Merge it into a small instruct LLM. Export a baked .aimodel plus tokenizer, and chat on the phone. There is no documented hot-swap LoRA.
On-Device Data Flywheel to Core AI LoRA
Improve quality from real use without sending raw user text to a public trainer by default.
Signals to synthetic data to LoRA to a baked Core AI export, with honest privacy limits.
Turn feedback into a better local model. Improve quality from real use without sending raw user text to a public trainer by default. Close the loop from feedback to synthetic data to LoRA to a Core AI export, with honest privacy limits.
On-Device Multi-Agent Consensus
Reduce costly mistakes by requiring agreement, or a judge, before the app acts.
Weighted voting, a judge session, and escalation with Dynamic Profiles and multiple sessions.
Make several agents agree first. Reduce costly mistakes by requiring agreement, or a judge, before the app acts. You will build weighted voting, a judge session, and escalation with Dynamic Profiles and multiple sessions.
Durable On-Device AI Workflows
Survive app backgrounding and retries so long tasks finish without starting over.
Checkpoint and resume mini workflow engine for on-device agent work.
Pause and resume long AI jobs. Survive app backgrounding and retries so long tasks finish without starting over. You will build a checkpoint and resume mini workflow engine for on-device agent work.
AI Full-Stack Operator: Train, Convert, Ship On-Device
Build progressively realistic replicas of the whole stack until you can reason about tokens, throughput, and hardware without hand-waving.
Explain transformer internals, train small LMs, run SFT and preference tuning, serve with vLLM, write basic Triton, and hold earned opinions on clusters and the semiconductor supply chain.
Twelve weeks of replicas: linear algebra through transformers, training, post-training, vLLM, Triton, clusters, and the GPU supply chain. You do not need to pretrain an 800B model to understand the shape of the problem.
Premium. Join the waitlist for paid access notice.
Ship an LLM on Android and iOS with LiteRT-LM
Ship a LiteRT-LM chat feature that passes release builds and does not give users wrong answers with no error.
Conversation lifecycle, R8 keep rules, maxNumTokens vs the baked KV cache, NPU prefill loss, a CPU vs GPU parity harness, and an iOS release checklist.
Ship a local LLM with LiteRT-LM without crashes or silent wrong answers. Close Conversations safely, add R8 keep rules, match maxNumTokens to the model, catch silent NPU and GPU errors with a CPU vs GPU parity test, and pass an iOS release checklist.
Core AI Fast First Token and the Right Compute Unit
Make the first answer from your Core AI model arrive fast, on the compute unit you planned, without silent slow or wrong runs.
Specialization and caching, ahead-of-time .aimodelc, prefill vs decode functions, Neural Engine fallback checks, a .cpuOnly safe path, and runtime memory.
Get a fast first token from Core AI. Measure specialization, load, and inference, specialize ahead of time, use prefill functions for long prompts, catch silent Neural Engine fallback, and keep a .cpuOnly safe path.
Run One Large LLM Across Several Macs with MLX
Run a frontier-class open model on your own Macs, for private code and documents, when it is too big for one machine.
Pipeline vs tensor parallel, memory per Mac, Thunderbolt 5 mesh, RDMA and the JACCL backend, link tests, known failures, ring fallback, and serving agents.
Run one model that is too big for a single Mac. Pick pipeline or tensor parallel, size memory per Mac, cable a Thunderbolt 5 mesh with RDMA, test the link with mlx.launch, recover from JACCL errors, and serve several agents.