New M5 Ultra and neural accelerators

Apple just dropped two new chipsets: the much-awaited M6 and the new M5 Ultra SoCs. This is a particularly exciting time to be building on-device ML/AI systems because of all the advancements happening with Apple Silicon and hardware. These new chips were launched with the new Mac Mini and the new Mac Studios (which recently saw price hikes along with a lot of other hardware) and are available for pre-order now, with release set for 22nd September. These new chips come packing with a lot of powerful new upgrades - and Apple has not shied away from positioning these machines/chips as AI powerhouses. The one we are most excited about is the presence of neural accelerators in each GPU core. Before we get there, let's look at these chip launches individually.

M6: Optimised Design and Enhanced Performance

Apple's base M-series chips are usually targeted towards students, hobbyists and normal folks who use the computer to do, well, normal computing: browsing email, watching Netflix, the usual. Apple has brought a whole new 2nm process for greater transistor packing density. It also features a new dual 16-core Neural Engine, which is said to boost peak ML workload performance by 2x compared to the previous generation.

The CPU itself is stunning: 12 cores consisting of 2 "super" cores, 4 performance cores and 6 efficiency cores. Apple claims it's 20% faster than M5 - giving it the "world's fastest single-threaded performance". But what caught my interest was the new GPU. The M5 chip present in the MacBook Pro did not have neural accelerators - these were components only present in the M5 Pro/Max chipsets. But now, they have arrived with the M6 chip.

M6 has a 12-core GPU (2 more than M5) with a neural accelerator present in each core. This allows for a 30% increase in peak GPU compute for AI compared to M5. We can very likely witness massive prompt processing gains when using a local LLM. M6 also has support for up to 32GB of unified memory and up to 170GB/s memory bandwidth - a 10% improvement over the M5.

Neural Accelerators

Think of neural accelerators as a tiny version of the Apple Neural Engine (ANE). The M-series Apple Silicon chips are SoCs (systems on a chip) - meaning they pack the CPU, GPU and memory together on one die. And this includes the ANE - a neural processor designed to perform heavy matrix multiplication workloads efficiently. While the GPU is a powerhouse at handling these ML workloads, the ANE does the same thing with much greater efficiency (and much lower power draw). But the downside is that the ANE is heavily guarded by Apple's internal APIs, and to utilize it from an external application standpoint, you have to use either the primitive CoreML-based workflows or the new CoreAI-based .aimodel conversions. And then you can get the ANE to pick up these workloads to free your GPU for other tasks.

The neural accelerators do the same thing - but at the GPU-core level. This has no impact on the actual full-sized ANE, though; there is still an ANE to handle all of the useful under-the-hood Apple stuff like computational photography, spatial audio, Face ID and things like that. It's particularly exciting to see Apple put these new accelerators on the base M6 too - making it truly a powerhouse for ML/AI workflows. The area where we'd see the greatest impact would be in prefills when running a large language model - because a prefill step is compute-bound (unlike GPU caches, which are bandwidth-heavy).

For further reading, I highly recommend watching this YouTube video from Apple Developer: Accelerate your machine learning workloads with the M5 and A19 GPUs

M5 Ultra: The Ultimate Powerhouse for Pro Workflows

As always, the Ultra line-up of chips is geared towards the pro-est of pros, hardcore video editors, people serious about AI and those working with complex computing and GPU workloads. The M5 Ultra features a new quad-die architecture that fuses two dual-die M5 Max chips, achieving an inter-die bandwidth of over 4.4TB/s. The M5 Ultra features up to a 36-core CPU (12 super and 24 performance cores) and delivers 1.25x single-threaded and 1.3x multi-threaded performance over the M3 Ultra.

And again, the GPU is where things blow out of proportion. The M5 Ultra now features up to a whopping 80 GPU cores with a neural accelerator in each core (similar to the M6) - churning out an insane 4.5x peak GPU compute for AI compared to the M3 Ultra. The GPU includes Apple’s latest shader core with second-generation Dynamic Caching, as well as hardware-accelerated mesh shading and third-generation ray tracing. It also features a more capable Media Engine for high-resolution video editing. The most exciting part is that it includes a 32-core ANE for on-device AI/ML workflows enabled by the system or through CoreML/CoreAI.

It features up to 512GB of unified memory (no increase this year) but delivers a staggering 1.2TB/s of unified bandwidth (a 50% increase over M3 Ultra's 819 GB/s), which determines how quickly decoding happens when running an LLM - directly impacting the TPS (tokens per second) of the model. This is going to enable running really large models on the Mac Studio and even larger ones when you daisy-chain two or more of these beasts together using Thunderbolt cables (distributed inference through RDMA).

---

Overall, these are some terrific advancements in hardware and Apple Silicon. With Ternus becoming CEO, we are hopeful that the future is bright for the hardware Apple will churn out.