Running Bonsai 27B as an agent on my phone

Prism dropped Bonsai 27B and I couldn't stop thinking about it. A 27 billion parameter model, quantized down to roughly one bit per weight, landing at about 4GB on disk. That's a phone-sized download for a model that has no business being phone-sized.

Everyone's first instinct with a number like that is to check whether it's real - load it up, ask it a riddle, watch it produce a paragraph, nod, close the terminal. I've done that dance enough times. It tells you almost nothing. A model that can autocomplete a haiku on your laptop is not the same thing as a model that can do something for you.

So I wanted a different test. Not "can it talk," but: can it hold a job? Can it look at a real request, figure out who it's about, go find that person in my actual contacts, write a real email, and send it - on device, no network, no cloud fallback, all the way through to the thing actually landing in someone's inbox?

And it turns out I'd spent the last few weeks building exactly the harness to find out.

A bit about AgentFlow

I've been building a package called AgentFlow (open-sourcing it soon). The short version: it's agent orchestration for Apple platforms, built on top of the Foundation Models framework. Steps, routing, tool calls, transcripts, the usual agentic furniture.

The part that matters for this post is one design decision. In AgentFlow, an agent declares what it is - its instructions, its tools, its config - and a completely separate thing called an AgentModel decides who actually runs it. There's a seam between the two.

I built that seam so I could swap in a test double for unit tests without a real model. But a seam doesn't care why you built it. If an agent doesn't know who's running it, then in principle you can point one agent at Apple's on-device model and the agent right next to it at something else entirely, in the same orchestration, and neither of them has to know.

Which raises the obvious question.

Can Bonsai go in there?

The bar I set: you should be able to type a HuggingFace ID and have inference just happen. Not a Bonsai adapter. Not a special case with if modelID.contains("bonsai") buried in it. If it only worked for Bonsai it wasn't worth building, because Prism ships a whole suite of these and someone else is going to ship the next one.

What I wanted to end up with was this and nothing more:

let agent = AgentDefinition(
    id: "email-agent",
    instructions: "You are a precise assistant helping compose and address emails.",
    modelConfig: ModelConfig(model: .mlx("prism-ml/Bonsai-27B-mlx-1bit"),
                             temperature: 0.4)
)
let session = try AgentSession(agent: agent, model: MLXAgentModel())

One line names the model. Everything else is the same code you'd write against Apple's model.

Getting there took three fights.

Fight one: the right way doesn't work

There are two ways to put a non-Apple model behind that seam.

The good one: iOS 27 exposes a LanguageModel protocol. Conform MLX to it, and MLX inherits the entire FoundationModels stack for free - sessions, @Generable structured output, tool calling, transcripts. Everything Apple built, pointed at your weights. That's the version you'd want to write.

The other one: skip all that, run MLX directly behind the seam, and reimplement the bits you need.

I went for the good one first. It doesn't work. MLX drives its own Metal execution, and Foundation Models wants to dispatch your model through its own custom executor. Put those two together and you get an EXC_BAD_ACCESS at address 0x0 - a null-pointer dereference deep in the stack, nowhere near any code I wrote.

I spent an embarrassing amount of time on this before accepting it. I tried it every way I could think of. The conclusion I landed on is that these two execution models are not currently compatible, full stop, and no amount of cleverness on my side was going to fix it from the outside. So I deleted the whole conformance and went the other way: MLX runs directly behind the seam, bypassing Foundation Models entirely.

Which is fine, honestly. That's what the seam is for. But if you're about to go down that road on iOS 27, save yourself the weekend.

Fight two: MLX won't go below 2 bits

Loaded the ternary model. Worked. Loaded the 1-bit model, and:

Fatal error: [quantize] The requested number of bits 1 is not supported

Upstream mlx-swift just doesn't quantize below 2 bits. The kernels aren't there. Which is a little funny, because the entire pitch of the model is that it's one bit.

Prism ships their own fork of mlx-swift with the low-bit kernels in it. So the fix is to use theirs instead of upstream's. My first instinct was that this wasn't possible - mlx-swift-lm hard-codes ml-explore/mlx-swift as its own dependency, and SwiftPM keys package identity off the URL, so surely you'd get a conflict.

I was wrong, and it's worth knowing why. SwiftPM resolves identity conflicts in favour of the root package's declaration. If your root package declares the fork under the same identity, the fork wins, and every transitive dependency quietly gets the fork instead:

.package(url: "https://github.com/ml-explore/mlx-swift-lm", from: "3.31.0"),
// Prism's fork, in place of ml-explore's: it carries the low-bit kernels
.package(url: "https://github.com/PrismML-Eng/mlx-swift", branch: "v0.31.6_prism"),

You get a warning about the conflicting identity. It works anyway.

The gotcha that ate an hour: "root" is literal. My package declared the fork, but the app consuming my package didn't - and from the app's perspective, both were transitive, so neither was root, and it happily resolved back to upstream. The app has to declare it too. Every root in the graph.

Fight three: you get one model, not two

The first thing I wanted to build was a map-reduce - chunk the input, run a few generations in parallel, merge. Standard shape.

You can't. Not with a 27B. The weights are resident and there is no headroom for a second concurrent generation. My first attempt to run two at once got the process killed outright by the memory watchdog.

This is the part I think people underestimate about on-device agents. In the cloud, parallelism is a cost decision. On device, it's a physics decision, and it reshapes your architecture. Every design I'd sketched assuming I could fan out had to become a sequence.

So: one session, one model, strictly sequential steps, each one feeding the next.

What it does

Four steps, all on device:

1. Find contacts - reads name and email out of CNContactStore.
2. Get email from contact - Bonsai reads the request and says who it's about; that name gets matched against my actual contacts.
3. Write email content - Bonsai drafts a subject and body.
4. Send email - confirmation alert, then SMTP.

(I tweeted about this here)

Type "Email Kautuk, asking him to be prepared for the 5PM investor meeting today.", tap Run, and it goes and does it. Contacts never leave the phone. The draft never leaves the phone. The only thing that goes over the wire is the finished email, and only after I tap Send. Nothing sends without the tap - I was not about to build a thing that autonomously emails people.

One unglamorous but load-bearing detail: you have to be defensive about what comes back. It's a reasoning model, so it emits reasoning, and you're pulling a subject line out of the middle of that. I take the last Subject: line after stripping the thinking block, and if there isn't one, or the body is too short, or it's obviously been cut off mid-sentence - I refuse to send it and fail the step. A demo prints whatever the model said. A thing you let touch your mail does not.

And it worked! Actual email, actual inbox, written by a 1-bit 27B model sitting in my pocket.

It was also unbearably slow.

Figuring out what went wrong

That last step - "Write email content" - took minutes. Not a beat, not a spinner-and-done, but long enough that I'd put the phone down and go do something else. The prompt goes in, and then nothing. Forever. Then an email.

Here's the thing though: I couldn't tell you why, because I had nothing to look at. A spinner is not data. I didn't know if it was slow to start or slow throughout, whether it was choking on the prompt or the answer, whether it was thrashing memory or just... thinking. I had a vibe and a stopwatch.

So I stopped guessing and built the instruments. Streaming, so I could watch tokens land one at a time instead of staring at a spinner. Time-to-first-token. Prefill and decode speed, measured separately. A stop reason, so I could tell "finished its sentence" apart from "ran out of budget mid-word." Live memory footprint.

The moment I turned streaming on, the bug diagnosed itself. Because what I watched scroll past, for forty straight seconds, was the model thinking. Reasoning. Deliberating. About an email reminding a guy about a 5PM meeting.

The flag

Bonsai is a Qwen3-family model, so it uses a Qwen3-family chat template. Here's the end of it:

{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking is false %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- else %}
        {{- '<think>\n' }}
    {%- endif %}
{%- endif %}

Read that carefully, because it took me longer than I'd like to admit.

The template opens a <think> block in the generation prompt on every single turn. The model isn't deciding to reason. It's handed a prompt that already ends mid-thought, and the only thing it can possibly do next is think. The <think> tag isn't in the model's output - it's in the input. That's why nothing I did to the prompt ever made it stop.

Set enable_thinking to false and the template instead emits a pre-closed empty block, and the model comes out swinging with the answer.

(I've been bitten by this exact family of template behaviour before, when a stray empty <think></think> block quietly wrecked a LoRA run. Chat templates are load-bearing and nobody reads them.)

MLX exposes this as additionalContext on the session, so I plumbed it through the seam as generic chat-template context - it's not a Bonsai concept, it's whatever variables your model's template happens to read:

MLXAgentModel(chatTemplateContext: ["enable_thinking": false])

Then I ran the same two turns, same weights, same machine, with one flag flipped.

The numbers

These are from my Mac, running the identical pipeline natively - I wanted a clean read without the memory watchdog in the mix:

	Reasoning ON	Reasoning OFF
Compose step - tokens decoded	1,221	60
Compose step - decode time	41.9s	2.1s
Decode speed	29.1 tok/s	28.7 tok/s
Both turns, end to end	52.3s	3.1s

Seventeen times faster. And look at the row that didn't move: decode speed. 29.1 versus 28.7 tokens per second. Identical.

The model was never slow. It was doing 29 tok/s the whole time, cheerfully, exactly as advertised. It was just generating twenty times more tokens than the answer needed. Eleven hundred tokens of deliberation to produce a sixty token email. I wasn't waiting on a slow model. I was waiting on an essay I never asked for and never got to read.

The other number worth staring at: prefill runs around 190 tok/s, decode around 29. Prefill is roughly six and a half times faster than decode. Which means on-device latency is almost entirely an output length problem. Every instinct I had from prompt engineering - trim the prompt, tighten the context, save tokens on input - was aimed at the cheap half. The expensive half is what the model says back.

Now, the honest caveat: reasoning off is right for this task. Pull a name out of a sentence, write four sentences back. There's no reasoning to be done, so I'm paying 1,161 tokens for nothing. Give it something that actually needs deliberation and I'd want it back on. The point isn't "turn thinking off." The point is that this was costing me 17x and I had no way to see it. It looked exactly like a slow model, and I'd have spent the next week optimizing the wrong thing.

If you're running any Qwen3-derived model locally and you've concluded 27B just isn't fast enough to be interactive - check your template before you believe yourself. You might be measuring a default.

That's a wrap

A 27B reasoning model, on device, driving a four step agent with real tool calls and a real side effect, in about three seconds of model time. Not a tech demo, not a haiku - it read my contacts, found the guy, wrote the email, and sent it.

Two things I'd tell past-me. Build the instruments first - I burned days on a problem that took thirty seconds to spot once tokens were visible on screen, and "it's slow" is not a bug report, it's a feeling. And prefill and decode are different problems with opposite fixes - one aggregate number tells you nothing and points you at the wrong lever.

AgentFlow's going up on GitHub soon. If anyone's actually gotten MLX running under Apple's LanguageModel protocol on 27 - please, genuinely, tell me how.