Profile and verify on Android · PyTorch to Android with LiteRT

Environment setup

Use the same Android Studio project and a debug APK. When you measure latency, use a physical device. An emulator is fine for checking that the app works, but it is not a good stand-in for phone speed. Keep the CPU path first. Only compare accelerators after the labels are correct.

./gradlew :app:assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb logcat -c
adb logcat | rg "Clothing|LiteRT|inference"

On a real device, turn on Developer options and USB debugging. The current CompiledModel API can use CPU, GPU, or NPU when the runtime and the device support them. Optional accelerators are an experiment. Try them only after the CPU path works.

Measure cold and warm inference

private fun measure(classifier: ClothingClassifier, bitmap: Bitmap) {
    val coldStart = System.nanoTime()
    classifier.classify(bitmap)
    val coldMs = (System.nanoTime() - coldStart) / 1_000_000.0

    val samples = LongArray(30)
    repeat(samples.size) { index ->
        val start = System.nanoTime()
        classifier.classify(bitmap)
        samples[index] = System.nanoTime() - start
    }
    val warmMeanMs = samples.average() / 1_000_000.0
    val warmP50Ms = samples.sorted()[samples.size / 2] / 1_000_000.0
    Log.i("Clothing", "cold=${"%.2f".format(coldMs)}ms " +
        "warmMean=${"%.2f".format(warmMeanMs)}ms " +
        "warmP50=${"%.2f".format(warmP50Ms)}ms")
}

When you want to go faster, time image prep separately from compiledModel.run. Write down the device model, Android version, runtime version, accelerator option, image size, and whether the measurement is cold (first run) or warm (later runs).

Try an accelerator only after the baseline

import com.google.ai.edge.litert.Accelerator
import com.google.ai.edge.litert.CompiledModel

// Replace CPU only after the CPU result and label are correct.
val options = CompiledModel.Options(Accelerator.GPU)
val accelerated = CompiledModel.create(modelFile.absolutePath, options)
// Allocate matching buffers and run the same FloatArray contract.
// If creation fails or the device lacks support, keep the CPU model.

Do not build a GPU or NPU fallback by mixing old TensorFlow Lite delegates into this course. Use LiteRT’s current accelerator options. If creation fails, go back to the CPU path you already trust. Hardware support and speed vary by device.

Sanity checks

Run a fixed bundled sample more than once. The predicted index should stay the same.
Log the output length and check that the values are finite (real numbers, not NaN). NaN or a surprising length means the model file or the buffers do not match.
Compare one sample in Python and Android. The top class should usually match. Small float differences are fine.
Temporarily feed an all-zero image and a full-one image. This helps you catch stale buffers and channel-order mistakes.

Failure modes

All predictions look wrong: confirm the app writes grayscale 0–1 values, not 0–255, and does not accidentally use NHWC.
Every result is class zero: confirm the output buffer is read after run and that the APK contains the intended model.
App cannot find the model: check the exact asset name and the one-time copy from assets.
Confidence looks wrong: the model returns logits, not probabilities. Apply softmax only for display.
GPU creation fails: return to CPU and write down the device and runtime combination. Accelerator support is not guaranteed.

Key concepts

Measure cold latency on the first call and warm latency on later calls.
Use a physical device for timing. The emulator is for functional checks.
Try GPU or NPU only after CPU labels are correct. Fall back to CPU if creation fails.
Check a fixed sample, finite outputs, and Python versus Android top-class match.

Takeaways

Time image prep separately from compiledModel.run when you optimize.
Do not mix old TensorFlow Lite delegates into this course.
Record device, Android version, runtime, accelerator, and cold versus warm in your notes.