WWDC26: CoreAI, Foundation Models and the future of AI

WWDC26 has been crazy this year. Apple focused on three main categories for the main keynote: Platform improvements (across iOS/macOS and all other operating systems), Trust and safety, Apple Intelligence and Siri. Apple brushed past the OS upgrades (which are usually split into its own sections: iOS, macOS etc) in just around 10 minutes of the keynote. Trust and safety took about the next 10 minutes, and finally, the most important section of this keynote: AI and Siri, took up the majority of the it, close to 45-50 minutes.

There were several quality of life features that were introduced in platform upgrades, including but not limited to, the operating systems becoming much faster. As an Apple geek, I installed the betas for ALL of my Apple products (daily driver devices) on day 0, along with a few other geeks that I know, and can verify that some of these operating systems are MUCH smoother and faster than their stable 26.0 counterparts. The downside being - there are some obvious bugs (not related to performance, just general bugs). Overall - it has been a solid developer beta 1, compared to previous releases.

Trust and safety addressed a lot about children and how parents can let them safely interact with content online - and with new features to screentime and permissions. They have also exposed several developer focused APIs for you to get started with building apps that are targeted at kids.

Siri AI

Siri gets a fresh new look - now called "Siri AI", the new siri plunges in right from the Dynamic Island. It has a GORGEOUS new UI and finally gets its language model improvements. But the new Siri is just the outer most layer on top of everything that's happening. Beneath this layer, Apple uses your personal context, world knowledge, app actions, on screen awareness and bundles all that using a "System Orchestrator" so that Siri can surface and reply with the most insightful responses.

Behind all of this is the new series of Apple Foundation Models. Apple Machine Learning Research has published a detailed blog post about their new suite of models - AFM 3, including a shocking new 20 Billion parameter model that runs on-Device (if you know your ML, this is CRAZY). They use a custom Mixture-of-Experts architecture to make this feasible. We plan to share an in-depth analysis on this soon. There's also their cloud range of models, the AFM cloud, for which they have partnered with Google and NVIDIA to extend their Private Cloud Compute (PCC) on to NVIDIA GPUs in Google Cloud.

This is not the first time Apple has used third party hardware to run their software. However, this is definitely surprising, considering how efficient the current Apple Silicon is. This has me wondering if they would ever go on to release an ultra-pro version of the Apple Silicon that's incredibly powerful (at the cost of being energy efficient of course).

What's in it for the Devs?

The Platforms State of the Union (which followed the keynote) summarized the vast majority of the developer tools that got released this DubDub. The highlight of the show, was of course, foundation models and core AI. AFM (Apple foundation models) are placed at the core of it all. There are so many things happening with new AI and foundation models on apple platforms.

Number one: AFM 3 suite of models is released.

As mentioned above, Apple ML Research has dropped a new version of Apple Foundation Models - these large language models are pivotal to how Siri operates - combining personal context and world knowledge into useful answers that Siri can provide. AFM can be accessed via the terminal using the new fm command-line interface. And most importantly, Apple has claimed that the foundation models WILL be open-sourced soon!

The compatibility here is the biggest issue. The new 20B model is only available on TWO iPhones: iPhone Air and iPhone 17 Pro/Pro Max. iPads need M4 or later, Macs M3 or later, and 12GB of ram is mandatory. As Awni Hannun said:

"You can't put 20B parameters in RAM at any reasonable precision. To make it work they are using pretty exotic architecture by today's standards."

Number two: CoreAI framework is released.

CoreAI - the successor to the age-old CoreML - has now been released. This Apple proprietary framework helps developers take an existing PyTorch based AI Model and run it directly on swift. Swift provides a memory-safe interface to run this model, as well as an interactive debugger to visualize your model's tensors directly within Xcode - if you want to trace a bug.

MLX LM and MLXSwift have been around for quite some time - letting you run inference on local models right on your iPhone. The new CoreAI has made things so much simpler (and the whole language model inference paradigm has been pulled in-house as opposed to open source). This is reflected in the new WWDC session about MLXSwift, which barely talks about language models and inference, instead just touching on GPU-heavy ops (tensor ops) for various other use-cases. One huge caveat with MLX: it doesn't use the Apple Neural Engine (ANE) for running ML workloads - while CoreAI DOES indeed use the ANE for this, making local models much, much faster.

Number three: App Intents / App actions

App Intents has been on Apple platforms for all these years. And just as the community has been predicting - this year, it gets a huge upgrade by letting Siri AI interface with these app intents. This happens using the new entity enhancements released this year, that lets developers add elegant App Intents in their apps that can be indexed by spotlight and be surfaced by Siri as entities or actions.

Using Foundation Models in Swift

New AFM framework in Swift comes feature-packed this year. To kick things off, Foundation Models now support multi-modality (images) so, providing it image data is as simple as a few lines of code:

  let response = try await session.respond {
      "What animal is this?"
      Attachment(UIImage(...))
  }

There are also new Dynamic Profiles that let you create different "agents" with varying system prompts, reasoning level, tools etc, based on the task at hand. The profiles can also be swapped automatically based on the task's requirement.

  struct CraftProfile: LanguageModelSession.DynamicProfile {
      let states: CraftProjectStates

      var body: some DynamicProfile {
          switch states.mode {
          case .craftAnalysis:
              Profile {
                  Instructions { /* ... */ }
                  RecordImageAnalysisTool()
                  SwitchModeTool(states: states)
              }
          case .brainstorm:
              Profile {
                  Instructions { /* ... */ }
                  BrainstormRecordTool()
              }
              .model(states.privateCloudCompute)
              .reasoningLevel(.deep)
          }
      }
  }

You can use custom models via Foundation Models framework by conforming your model to the new LanguageModel protocol and it's as simple as that. There's also a new python library, apple_fm_sdk, for interacting with AFM.

Do note that a lot of developer tools released on day one are subject to change as time progresses - this has happened a lot previously, so do keep an eye out for the official GitHub repos for open source frameworks and Apple Developer Documentation for their proprietary toolkits and APIs.

That's a wrap

This DubDub was PACKED with a lot of intelligence related features. New Siri AI, Apple Foundation Models, and advancements across all their platforms. The best part about it all for developers is the rich suite of APIs released, including CoreAI to build on top of the intelligence layer that Apple has provided system wide. The future of AI on Apple Silicon begins here.