盆

Bonsai WebGPU

2 billion parameters. In your browser.

PTQ1 · packed ternary

Keep the model on this device

429 MB once. Prompts and inference stay here.

Checking WebGPU and cached model…

Pixel 7, phase-split Chat: 13.8 decode tok/s · 0.6 s to first token

—prefill tok/s
—decode tok/s
—first token
—new tokens

Download the model, then send a message. No inference server, no CDN.

Which Chat route is fastest depends on the device. On desktop GPUs, Chat standard is fastest and is selected first: on a Radeon 780M it decodes about 87 tok/s, against 77–83 for accelerated and phase-split. Phones start on phase-split. On a Pixel 7 it decodes 13.8 tok/s against 13.5 for standard, reads the prompt at about 55 tok/s and shows its first token in about 0.6 s. Accelerated is about as fast; it and phase-split need WebGPU subgroups. The prefill experiments (P06, P07, P09) are optional, and whether they help depends on prompt length. Compare routes on your own device with the numbers above.

Follow-up messages resume from the model state saved at the end of the previous reply, so each turn processes only what is new instead of the whole conversation. The output is identical either way.

Copy modes verify prompt-matched drafts with the full model and help when the reply repeats the prompt. Low memory uses 7 drafts; wide tries 15 using 32-row state and is opt-in. Extra GPU memory is allocated at first verification, with no extra model weights. Changing mode releases the previous engine and reloads from device cache.

Device & speed lab

Detecting adapter…

Microbenchmarks measure streaming read/write bandwidth and dispatch wall time. Chat metrics include real graph execution, sampling and GPU readback. Cached loads still upload weights and compile shaders. Browser storage can be evicted unless persistence is granted.