Pick a base and LoRA idea

Why the base model choice matters

The base you train against is the base you must bake and ship. A checkpoint that has no Core AI recipe is only useful in the lab. If you put the model on a phone, start from a model the official catalog already knows how to export. Then personalize it locally so the device never needs the training set.

Privacy starts here. If the subject is a person, you need their permission and a plan for the raw photos after export. The baked model still contains the concept. Treat the resource folder as sensitive.

The steps, one box at a time

First pick the catalog model. This course uses sd-1.5 or sd-2.1 from apple/coreai-models. The typed catalog course lists sd-3.5-medium too. Leave that for a later experiment. Smaller, older Stable Diffusion graphs are the first export that works.

Then name the behavior. Subject or style. One sentence. Not both in the first run.

Then write down how you will test it. Held-out prompts the trainer never sees. Same prompts against the untouched base and the merged result.

Subject versus style

A subject LoRA needs varied views of one identifiable thing. Change clothes, lighting, and background in the training set so the model does not memorize one kitchen. Captions should describe what changes around the subject. Keep a rare token or unique name in the caption if your recipe uses one.

A style LoRA needs a consistent treatment across many subjects. If every training image is the same building, you trained one subject, not a style.

Write the success sentence before you gather files. Examples that work: "the same mug appears in three unseen settings" or "the scene keeps its objects while the rendering looks like the reference prints."

Worked path

Reuse the project from lesson 01. If it is missing, create it before any other command.

curl -LsSf https://astral.sh/uv/install.sh | sh
mkdir -p diffusion-lora-coreai && cd diffusion-lora-coreai
uv init --python 3.11
uv venv
uv add torch torchvision diffusers peft transformers accelerate datasets safetensors coreai-torch
uv run python -c "import torch; print(torch.__version__); print('MPS:', torch.backends.mps.is_available())"

List the catalog short-names the installed export tool actually knows. Confirm your chosen name is present before you download weights.

uv run coreai.diffusion.export --help
uv run coreai.model.registry --help
# Confirm sd-1.5 or sd-2.1 against the installed apple/coreai-models list.
# Pin the Hugging Face id and revision the short-name resolves to.

Lay out a local dataset you own or have a license to use. Keep a holdout folder the trainer will not read.

mkdir -p data/train data/holdout artifacts
# data/train: 12 to 40 clean images plus matching .txt captions
# data/holdout: 4 to 8 images or prompt-only tests the trainer never sees
ls data/train data/holdout

A caption file next to mug-03.jpg can be as plain as this:

a photo of sks mug on a wooden table, natural window light, no logo

Record the contract (your short written plan). Fill every field. Leave guesses blank until you look them up.

{
  "base_short_name": "sd-1.5",
  "base_hf_id": "runwayml/stable-diffusion-v1-5",
  "base_revision": "pin-this",
  "license": "read-before-ship",
  "goal": "subject",
  "trigger_token": "sks mug",
  "success": "The same mug appears in three held-out kitchen scenes.",
  "holdout_prompts": [
    "sks mug on a hotel nightstand, warm lamp, 35mm",
    "sks mug on a picnic blanket, overcast daylight",
    "sks mug next to a laptop, office fluorescent light"
  ],
  "runtime_lora_swap": false
}

Do not assume that every scheduler, text encoder, or community checkpoint will export. The selected Core AI recipe decides the supported graph. Read the diffusion family notes in Core AI models, typed and the image-model shape in Model architectures in plain English.

Failure modes

• Picking a base with no export recipe. You will train a LoRA you cannot bake for Core AI.
• Mixing subject and style in one first adapter. You will not know which signal won.
• No holdout. Training previews will look fine while the concept is just memorized backgrounds.
• Unlicensed or sensitive photos. The baked .aimodel still carries the concept after you delete the folder.
• Captions that only repeat the trigger. Those captions teach the model nothing about the scene you want later.

Done when

• The contract names only sd-1.5 or sd-2.1, with revision and license filled in.
• The goal is subject or style, not both, plus one success sentence.
• data/train and data/holdout exist, and holdout prompts are written down.
• The contract still says there is no runtime LoRA swap.

Keep learning

PyTorch to Core AI in Xcode · From the metal to the model · Core AI models, typed · Model architectures in plain English · LLM LoRA for your writing style