Plan for heat: burst vs sustained

The problem in numbers you can see

Your benchmark runs for 30 seconds on a cool phone. A real user chats for 15 minutes on a warm day with the phone charging. Speed in the second case can be much lower, because the system cuts clock speeds to protect the device. Nothing crashes. The app just feels slow.

Do not guess. Run your own test: 20 minutes of back-to-back answers, and log tokens per second and thermal state every answer. That chart tells you how much slower sustained mode really is on each device.

iOS: ProcessInfo.thermalState

ProcessInfo.processInfo.thermalState has four values: .nominal, .fair, .serious, and .critical. The system posts thermalStateDidChangeNotification when it changes.

import Foundation

struct GenerationPolicy: Equatable {
    var maxOutputTokens: Int
    var allowBackgroundJobs: Bool
    var preferSmallModel: Bool
}

func policy(for state: ProcessInfo.ThermalState) -> GenerationPolicy {
    switch state {
    case .nominal, .fair:
        return GenerationPolicy(maxOutputTokens: 1024, allowBackgroundJobs: true,  preferSmallModel: false)
    case .serious:
        return GenerationPolicy(maxOutputTokens: 384,  allowBackgroundJobs: false, preferSmallModel: true)
    case .critical:
        return GenerationPolicy(maxOutputTokens: 128,  allowBackgroundJobs: false, preferSmallModel: true)
    @unknown default:
        return GenerationPolicy(maxOutputTokens: 384,  allowBackgroundJobs: false, preferSmallModel: true)
    }
}

@MainActor
final class ThermalWatcher: ObservableObject {
    @Published private(set) var current = policy(for: ProcessInfo.processInfo.thermalState)
    private var token: NSObjectProtocol?

    init() {
        token = NotificationCenter.default.addObserver(
            forName: ProcessInfo.thermalStateDidChangeNotification,
            object: nil, queue: .main
        ) { [weak self] _ in
            let state = ProcessInfo.processInfo.thermalState
            print("thermal:", state.rawValue)
            self?.current = policy(for: state)
        }
    }
}

The numbers in the table are examples. Pick yours from your 20 minute test.

Android: thermal headroom

Android's Thermal API (API level 30) gives PowerManager.getThermalHeadroom(forecastSeconds). The value goes from 0.0 (no throttling) to 1.0 (severe throttling). Google's page lists some rules that are easy to miss:

• Do not call getThermalHeadroom more than once every 10 seconds, or it returns NaN.
• If the first value is NaN or 0, the device may not support the API.
• Some devices always report THERMAL_STATUS_NONE from getCurrentThermalStatus(), so Google suggests you trust the headroom value.

import android.content.Context
import android.os.PowerManager
import android.os.SystemClock

data class GenerationPolicy(val maxOutputTokens: Int, val allowBackgroundJobs: Boolean, val preferSmallModel: Boolean)

class ThermalPolicy(context: Context) {
    private val power = context.getSystemService(Context.POWER_SERVICE) as PowerManager
    private var lastReadMs = 0L
    private var lastHeadroom = 0f

    /** Reads at most once every 10 seconds, as the Thermal API docs ask. */
    private fun headroom(): Float {
        val now = SystemClock.elapsedRealtime()
        if (now - lastReadMs >= 10_000) {
            lastHeadroom = power.getThermalHeadroom(10)   // forecast 10 seconds ahead
            lastReadMs = now
        }
        return lastHeadroom
    }

    fun current(): GenerationPolicy {
        val h = headroom()
        return when {
            h.isNaN() -> GenerationPolicy(512, false, false)          // API not usable, stay careful
            h >= 0.95f -> GenerationPolicy(128, false, true)
            h >= 0.85f -> GenerationPolicy(384, false, true)
            else -> GenerationPolicy(1024, true, false)
        }
    }

    fun listen(onChange: (Int) -> Unit) {
        power.addThermalStatusListener { status -> onChange(status) }
    }
}

The 0.85 and 0.95 levels come from the heuristic example on Google's Thermal API page. Google also has the Android Dynamic Performance Framework (ADPF) performance hint API for games. For LLM work, the thermal headroom is the part you need first.

Burst vs sustained, in the UI

• Burst: the first answers in a session, on a cool device. Full model, long answers.
• Sustained: after the state rises. Shorter answers, the smaller model, no background summaries. Tell the user in a small line of text: "Shorter answers while your phone cools down."

Showing the mode is better than a slow answer with no reason. To measure speed per answer, the Streaming Metrics course shows how to log time to first token and tokens per second.

Sources

• Apple: ProcessInfo.thermalState
• Android: ADPF Thermal API