Working notes from shipping machine learning on Apple Silicon: what runs fast on the Neural Engine, what the benchmarks actually say, and the dead ends worth documenting. The consulting side lives at aszc.dev.
The energy post left one thread hanging. Running the SD1.5 UNet on
the Neural Engine buys 6-7x lower energy at the same speed as GPU/MPS, but the
SPLIT_EINSUM_V2 fp16 conversion lands 0.31% off an fp32 reference in cosine
terms — “a small numerical catch.” That number is a tensor distance. It says the
latents drift; it does not say whether a human looking at the decoded image would
ever notice, or which of the three converted components is to blame.
I am building a macOS menu-bar app that listens to live disk I/O and plays back
the mechanical chatter of a 1990s hard drive in response. The synthesis engine is
the whole point, and my first one was wrong in a way that took a while to hear:
it modelled the wrong motor. This is the log of getting from that first engine to
one that actually clicks.
If you run Stable Diffusion 1.5 on Apple Silicon in a loop — anything that bills
you by the watt — the question that matters is not which backend is fastest. They
are all within a hair of each other on latency. The question is which one spends
the least energy doing it, and the answer is lopsided: on an M2 Pro the Neural
Engine draws 6 to 7x less energy per UNet step than the same model on the GPU or
MPS, at the same wall-clock speed. The catch is numerical, not temporal, and it is
small enough that most generation pipelines will not care.