cd /tmp
# Get the Prism macOS runtime
curl -fL https://github.com/PrismML-Eng/llama.cpp/releases/download/prism-b10685-7dffb15/llama-prism-b10685-7dffb15-bin-macos-arm64.tar.gz -o bonsai-runtime.tar.gz
tar -xzf bonsai-runtime.tar.gz
# Get the ~5.95 GB GGUF model:
curl -fL https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/resolve/main/Ternary-Bonsai-2-27B-PTQ1_0.gguf -o Ternary-Bonsai-2-27B-PTQ1_0.gguf
# Run the server, I used port 8331
./llama-prism-b10685-7dffb15/llama-server \
-m Ternary-Bonsai-2-27B-PTQ1_0.gguf \
--port 8331 -ngl 99 -fa on -c 32768
Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this:
uvx llm openai endpoint http://127.0.0.1:8331/v1 \
--model bonsai-2-27b --responses hi
That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".
simonw 2 days ago [-]
I used that to Generate an SVG of a pelican riding a bicycle:
Honestly looks pretty good except whatever is going on with its booty. Is that an ass helmet? I cannot parse what's going on there.
tomcam 2 days ago [-]
Like you've never worn an ass helmet
kadoban 2 days ago [-]
Only because I hadn't previously thought of it xD Step up from the standard ass-hat for sure.
Forgeties79 2 days ago [-]
I think it’s supposed to be a wing
bigwheels 2 days ago [-]
I like the lens effect behind the rear tire.
shmoil 1 days ago [-]
Can you ask it for an SVG of a bicycle riding a pelican? Thanks.
wombat23 1 days ago [-]
I managed to run it with RTX 3070 (8GB VRAM) following the "Quickstart" on HF model card with minor modifications (modify the architecture 86 for your own hardware):
the max context size i could serve is 64K on GPU only (the -ngl 99 setting). tested with pi harness and it is very fast. the /thinking level always gets reset to off though and it is not very smart like this. haven't figure out a way to fix that.
aktenlage 1 days ago [-]
Would it speed up prompt processing if you increased the -ub (and -b) parameters.
wombat23 1 days ago [-]
I don't see any significant speed up. also with defaults according to --help
-b, --batch-size N logical maximum batch size (default: 2048)
-ub, --ubatch-size N physical maximum batch size (default: 512)
I also tried double the default. that also means that they can be left to default settings, apparently.
jimmySixDOF 1 days ago [-]
hummm wonder what the context length limit will be like this
ekianjo 1 days ago [-]
wow why is prompt processing so slow?
francisjp 2 days ago [-]
Thanks for all of your exploration in public Simon.
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
That kind of issue is exactly why Im so happy to have LLMs, let it take one hour or trial and error instead of me spending a day digging traces
rahimnathwani 2 days ago [-]
If you want to download the gguf to your regular huggingface cache directory instead of to /tmp, you can download the model and run the server in one step:
export HF_TOKEN=xxx # optional, speeds up the download
./llama-prism-b10685-7dffb15/llama serve \
-hf prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0 \
--port 8331 -ngl 99 -fa on -c 32768
ithkai92 13 hours ago [-]
Thanks for the headstart, I saw hf also has PQ2_0 and able to finetune the command and in a MBA M4 24GB averages around 10t/s with the command.
The release tag and weight file you suggest doesn’t match what they wrote.
simonw 2 days ago [-]
I figured them out, starting from the GGUF on Hugging Face.
If you have found better instructions and they work then use those instead!
Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
refibrillator 2 days ago [-]
Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
fnordpiglet 1 days ago [-]
They’re fairly useful as constrain domain classifiers due to the low memory requirements means you can stuff a lot of them into a less expensive GPU farm and get really decent throughout with pretty good results over all. At least that’s my experience. I wouldn’t bother using a tiny model for coding - but the world is full of abductive reasoning tasks that don’t involve coding.
nikwen 2 days ago [-]
It would be great to have upstream llama.cpp support for this!
iJohnDoe 2 days ago [-]
Agreed. They always sound exciting to try out but are such a pain to get working.
Zetaphor 2 days ago [-]
I always just throw an agent at it. Is this the RSI I keep hearing about
hedgehog 2 days ago [-]
RSI saves you from RSI
ricardobayes 1 days ago [-]
Thanks for this, I never knew llama-server has a web ui until now.
jakswa 1 days ago [-]
I love their web UI so much that I had AI slop all over it in a fit of fanboy-ism https://inkcap.click
2 days ago [-]
miffy900 2 days ago [-]
I really wish people would stop saying N times smaller than something when making a comparison; that makes no sense - it's 1/9th (11.11%) the size. You don't get a smaller quantity by multiplying by a number greater than 1.0. You could instead reverse the subjects being compared - "the original model is 9x bigger than this new smaller, efficient model" or some such. That makes sense.
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
zamadatix 2 days ago [-]
I agree it makes little sense in a literal mathematical take but "it's 9x smaller" or is too much of linguistic advantage compared to "the original is 9x larger" or "it's 1/9th as large" to expect a change with. You don't have to invoke fractions, it keeps the thing in focus as the first subject, it matches the pattern of the inverse statement, and it's just plain short... so that's what people will adapt and interpret the meaning to be.
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then "smaller"/"larger" tells you which end of that separation the subject is (rather than specifying whether the multiplication builds up or down).
kevinwang 2 days ago [-]
"it's 11% as large" avoids fractions but is much more clear (IMO) than "9x smaller" which my brain doesn't understand.
DoctorOetker 3 hours ago [-]
I don't see the issue at all, consider whatever as a fraction, or just add a denominator of 1:
"9 times larger" => multiply the numerator of the fraction with 9
"9 times smaller" => multiply the denominator of the fraction with 9
Consider a bag of identical resistors with resistance R
Add 9 of them in series: the resistance is 9 times larger.
Add 9 of them in parallel: the resistance is 9 times smaller.
Its the series vs parallel dictionary wars all over again...
Check my other comment.
Taek 1 days ago [-]
It's an idiom, much like "you are pulling my hair". Are you actually pulling my hair? Of course not; taken literally, many English phrases make no sense. But they are common elements of the language and everyone understands them, so they aren't problematic.
This is the same. Taken literally, "9x smaller" is nonsense, but everyone who hears that phrase knows exactly what mathematical operation you are referring to, thus it's a totally acceptable way to express that you mean to say 11.11% as large.
asterix_pano 1 days ago [-]
When I hear 9x smaller, I visualize 9 small items taken as much space as the big one, I find it quite clear.
zamadatix 2 days ago [-]
11% is also a really saying a fraction, 11 per-cent or 11/100, but there's nothing wrong with that feeling more natural to some and it is at least a nice shorthand way for the written form. "A ninth the size" is a similar alternative. All are really fine, there's always someone who has trouble with a given representation compared to another.
fwip 1 days ago [-]
"The original is 9x larger"
This is another pet peeve that I have with the way we phrase these things: It should be 9x "as large" and 8x "larger."
The way I'd usually phrase this, to avoid ambiguity, is "this new model is 11% the size (of the original)." Or, the other way, "the old model is 9x the size".
hamandcheese 2 days ago [-]
If we were talking about speed instead of size, i think it would be perfectly reasonable to say 9x faster. I'm not sure I agree that 9x smaller is unintuitive. It makes sense to me.
simondotau 2 days ago [-]
"Nine times" literally means multiplied by nine, but here we're dividing by nine. It's not unintelligible (because the corrupted verbiage is so commonplace) but it is needlessly awkward. Like saying "resulted in a size reduction increase of 10 megabytes."
D-Machine 2 days ago [-]
> "Nine times" literally means multiplied by nine
Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO.
If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
simondotau 1 days ago [-]
[flagged]
dools 1 days ago [-]
Saying something is 9 times bigger is also a shorthand because “times” is a shorthand for repeated addition. You are adding a number “9 times” to make it “9 times larger”. But we don’t say the “adding” part.
Likewise if I repeatedly subtract a number 9 times to make it “9 times smaller” I can omit the subtracted bit.
simondotau 1 days ago [-]
That interpretation doesn’t make sense. Repeated subtraction does not correspond to division in the way repeated addition corresponds to multiplication.
dools 1 days ago [-]
Que? Division is the opposite of multiplication of course it does.
If you divide 81 by 9 you subtract 9 from 81 until you get to 0. The number of TIMES you do that is the result “81 divided by 9”.
simondotau 23 hours ago [-]
This isn’t analogous to repeated addition giving multiplication. You’ve switched from applying an operation nine times to counting how many times an operation can be applied.
You chose 81 because it’s the one number that makes your argument appear to work. Try the same reasoning with 80 ÷ 9.
dools 21 hours ago [-]
[dead]
_carbyau_ 2 days ago [-]
"faster" relates to speed. Speed is related to time and speed of a thing is usually defined by time. 9x faster speed translates to time/9. There is an extra step of related conversion there.
Conversely, 9x [filesize/natural number] is bigger. Every time. At least in the basic maths used by most people. There is no conversion into other units.
Therefore "9x smaller" when talking about a natural number like filesize is a nonsense statement in logic terms. If you strive for unambiguous phrasing - which is a significant part of the programming experience - this logical nonsense might well perturb you.
But english language is a flexible thing and if the phrase communicates your intent to your audience then that's fine by me.
jonhohle 2 days ago [-]
I think more intuitively for speed - you are multiplying the things per period. If I’m making widgets 9x faster I have 9 x the previous number of units per period of time. To get the amount of time, you invert the ratio to period over units.
Multiplying some scale by units per period makes sense and is both linguistically and mathematically sound.
2 days ago [-]
2 days ago [-]
peey 2 days ago [-]
It's simple
If 9 is "9 times greater" than 1 then 1 must be "9 times smaller" than 9
It'd help if you read "9 times" with the operator which is what's being flipped instead of with the number
a3w 1 days ago [-]
Its complicated:
50 percent smaller means either half, or two-thirds the size, depending on Apple Marketing doing the math.
0.999 times smaller means size is nearly zero. 9 times smaller means you get negative memory from loading it.
nicbor 2 days ago [-]
This is widespread usage.
I don't see what makes it hard to understand.
DoctorOetker 1 days ago [-]
Define the "parallel sum"
a∥b=((a*b)/(a+b)
A normal sum has the associative property:
a+(b+c)=(a+b)+c which justifies dropping the parentheses
a+(b+c)=a+b+c=(a+b)+c
It is just an exercise for the reader that the parallel sum ∥ operation is associative:
a∥(b∥c) = (a∥b)∥c
multiplication is typically defined as repeatedly adding:
a x b = b+b+...+b+b (a times b)
similarily one can define parallel multiplication xx :
a xx b = b∥b∥...∥b∥b (a parallel-times b)
The reader can verify that 9 parallel-times size
9 xx size = size∥size∥size∥size∥size∥size∥size∥size∥size = size / 9
So I just think the Bonsai designers meant it was 9 parallel-times smaller, which checks out...
x∥x=(xx)/(x+x)= x/2
x∥x∥x =(x/2
x)/(2/x+x)=(x/2)/(3/2)=x/3
x∥x∥x∥x = x/4
foobarbecue 2 days ago [-]
But we're cutting drug prices 500, 800, 1700%! Numbers nobody thought were possible.
dofm 2 days ago [-]
It was hilarious that this is the only time his, er, meta-imaginary-gains intensifier made the statement literally true.
UI_at_80x24 2 days ago [-]
Me too!! It's a huge pet peeve. And it's so hard to get people to see how it's linguistically AND mathematically WRONG.
Taek 1 days ago [-]
I wrote this elsewhere but I felt it was worth saying again. "9x smaller" is an idiom, much like "spill the beans" or "it costs an arm and a leg".
Idioms don't have to make literal sense or be linguistically/mathematically correct to be useful. All that matters is that other people know exactly what you mean when you say it.
And, pretty much universally, if I tell someone "the compressed file is 10x smaller than the original", they are going to know what I mean is that the byte size is 10% of the size of the original.
That makes it an idiom that is perfectly okay for everyday use.
Kinrany 2 days ago [-]
I disagree completely and I'd be happy to be convinced otherwise
derefr 2 days ago [-]
They probably rephrased it from some more technical form like "we compressed the model by a factor of 9" or "we've improved the packing efficiency of the model by 9x". Where these are measurements of the transformation the model is undergoing, not measurements of the resulting model.
stkdump 2 days ago [-]
Also, X is 9 times larger than Y and X is 9 times as large as Y are two different statements.
so-cal-schemer 1 days ago [-]
to spell it out:
200% as large as 1 is 2.
200% larger than 1 is 3..
so-cal-schemer 24 hours ago [-]
200% (or 2x) smaller than 3 is.. ?
rpdillon 2 days ago [-]
Eh, when I read smaller with an integer multiplier, I mentally switch to the reciprocal. Easier than convincing the world not to use "9x smaller". Do you feel the same way about "9x faster"? What you're actually measuring is time, and "faster" is the reciprocal of time, similarly to "smaller" being the reciprocal of size.
miffy900 2 days ago [-]
yes, actually I do think the phrase 'N times faster' is sensible and logical
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster.
i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
atiedebee 1 days ago [-]
I feel like this becomes less clear if you're talking about "one-time" events. "Our website loads 2x faster now" doesn't mean that they are sending more data per second, and the user won't be doing more website loads. The website load time going from 1 second to 0.5 seconds is "2x faster" and I think that that is a very common way of communicating such performance increases too.
fwip 1 days ago [-]
It could be sending more data per second if it's the result of a network upgrade at their datacenter. :)
rpdillon 2 days ago [-]
Ah, but in this case the customer is buying the coffee for the caffeine content. And you've doubled the caffeine content per volume! Twice as efficient a delivery mechanism, similar to this model!
Kinrany 2 days ago [-]
This sounds like the same kind of error as writing "0.10 cents" because it's less than a dollar when the number is in dollars regardless of how big or small it is
a3w 1 days ago [-]
Yes, 50 percent smaller means half the size. 1 times smaller means size zero, so nine times smaller is impossible.
comradesmith 2 days ago [-]
I completely agree and this is a pet peeve of mine so it's nice to be validated :]
trentor 2 days ago [-]
[dead]
verytrivial 2 days ago [-]
There's a chap called Bijian Bowen who does very quick agentic coding challenges for new models (very soon after release!) mainly for toy games or websites. He just did one for this model and included a comparison with the base model Qwen 3.8 which shows the "near-lossless" claim should be taken with a grain of salt. It is an interesting model if you are GPU starved and want local, but you might have trouble finding things it is good at.
Aurornis 1 days ago [-]
> which shows the "near-lossless" claim should be taken with a grain of salt
I agree. I don’t know how they get such good results on these benchmarks because using them gives a very different experience. They’re kind of cool for doing short free form outputs in memory constrained systems, but I don’t think they’re useful as coding agents.
Remember to clear the downloaded weights afterward.
Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
outofpaper 2 days ago [-]
So you have some fun examples?
Aurornis 1 days ago [-]
Maybe I oversold the fun-ness of it. The most common failure mode is that it goes into loops and you come back to find it exhausted the output length without getting anywhere.
aschobel 1 days ago [-]
40 tokens a second on my M4 Max (Safari). Wild times.
SXX 2 days ago [-]
Sadly crashing on Pixel 9 Pro, but I guess phone GPU with 16GB RAM total wouldnt be enough anyway.
14u2c 2 days ago [-]
Runs on my 16GB M2 Air (firefox). ~7 tok/s
jeroenhd 1 days ago [-]
3/16 GiB of RAM on the P9Pro is dedicated towards on-device AI models, which WebGPU probably can't access.
throwa356262 1 days ago [-]
To be fair, the GPU in pixel 9 and 10 is pretty bad.
I really hope they stop using PowerVR in pixel 11.
trvz 2 days ago [-]
It should be though.
SXX 2 days ago [-]
Even native AI gallery uses smaller Gemma models. I guess whatever Chrome is using for WebGPU compute on Android is just adding too much overhead.
adrian17 2 days ago [-]
> Ternary Bonsai 2 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, for 1.76 effective bits per weight
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?
The table claims it performs on par with UD-Q4_K_XL except on OCR.
Balinares 2 days ago [-]
I wonder how well it performs in practice, because I can't help seriously doubting those benchmarks. That would put this 6GB model in Opus 4.6+ ballpark. Granted, that's mostly to Qwen 3.8's credit, but it's hard to believe that Qwen's already unbelievable capability density can still be compressed this much more.
edflsafoiewq 2 days ago [-]
I think the general idea is naive quantization falls apart below 4bpw but you can go lower with more sophisticated QAT-adjacent methods. Bonsai's quantization method is proprietary though.
yowlingcat 2 days ago [-]
That's correct. I think there can certainly be issues even with 4bpw with naive quantization (IE you'll notice far better results from a QAT 4bpw vs a naive 4bpw).
One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw:
Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
om8 2 days ago [-]
Could've been better if GGUF implemented QTIP format. GGUF representation is a major limitation for llama.cpp quantization performance
edflsafoiewq 2 days ago [-]
They use their own llama fork anyway, so that shouldn't matter.
0x457 2 days ago [-]
1.76 bpw number is kinda misleading if you compare it directly to IQ2/Q2. The encoding is ternary, but the quantization procedure is way more sophisticated than "round Qwen weights to {-1,0,+1}."
They rotate the weights into a quantization-friendly basis first, then ternarize with per-group scales and error compensation.
smallerize 1 days ago [-]
I think the 1.76 includes that. Plain ternary packed into bytes would be around 1.58.
Chance-Device 1 days ago [-]
Let’s see, so if you get the same 1/9th the size compression ratio with GLM-5.3-Flash, then you’d end up with a ~72GB model that’s about as good as GPT-5.6 Sol (high), according to artificialanalysis.ai
Which is within reach of some higher end consumer hardware, especially with layer offloading.
You have to wonder what kind of trouble the “labs” are in when this is becoming possible. Lots of money, where’s the moat?
indy 1 days ago [-]
They're trying to build a moat with legislation, using fear over 'safety' as an excuse to ban these open models
ctolsen 1 days ago [-]
If we go with AA's benchmarks Qwen 3.8 27B is already slightly below Luna level which is in itself impressive, but with this compression it should be just slightly more below Luna level and could run on my old GTX 1070 that I'm now tempted to fire up. That's kinda nuts even allowing for small-model problems that I'm sure I'd see quite clearly.
kllrnohj 1 days ago [-]
The labs still have performance as a differentiator for coding usages and similar, and for other things there's still all the same reasons people switched cloud hosted stuff in the first place. AWS & friends didn't get popular because the hardware was out of reach, after all.
Chance-Device 1 days ago [-]
That’s an argument for a cloud LLM service, sure, but the hyperscalers can do that by themselves with the weights.
What’s the moat for trillion dollar AI companies? Access-anywhere convenience for models as good as everyone else’s?
kllrnohj 1 days ago [-]
the trillion dollar AI companies will be the chip designers for the hyperscalers, and probably not worth a trillion dollars as a result
I just checked that model (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S) and it does much much worse on the "Please recite Jabberwocky" test than the original bf16 does.
The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
sorenjan 1 days ago [-]
I think it should be expected that a smaller model is worse at reciting memorized data than a big one. I also don't think it's a good use case of small local models. Can it find and recite Jabberwocky if given access to a web search tool?
0xbadcafebee 1 days ago [-]
Forgetting things isn't lossless though is it? Makes the benchmark and the finding quite suspect
sorenjan 7 hours ago [-]
It depends on what you mean by lossless. If both models can perform the same tasks it can be considered lossless for those tasks. That task might be more related to language understanding rather than memorizing, they have several benchmarks in the article.
WithinReason 1 days ago [-]
I tried and ended up with:
I'm going to stop here and be direct: I'm having trouble recalling the exact text, and every attempt above is me guessing. Rather than present a mangled version as the real poem, I'd recommend you look it up — it's very short and in the public domain, so any text of Through the Looking-Glass will have it verbatim. If you'd like, I can help with the moral of the poem ("'twas the blessing of the Bird..."), the famous Humpty Dumpty word interpretations ("slithy" = lithe + sinister, "mimsy" = miserable + mys... etc.), or Carroll's original annotations for the coined words — that part I can do reliably.
Oh, wow, they think it's just a smidge below the q4? That's crazy good if true.
anana_ 2 days ago [-]
The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
SillyUsername 2 days ago [-]
Yep more hops from the lower Q is likely going to skew the vectors further over time.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
2 days ago [-]
0xbadcafebee 2 days ago [-]
Came to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).
blactuary 2 days ago [-]
What is never totally clear with a lot of these releases is the scope of what it's good at. Models that can run with good speed on affordable consumer hardware for coding only is the dream. I am never going to use this for writing, images, or "general knowledge". Coding only
nullbio 1 days ago [-]
Any time I'm doing frontend work it involves images. I think images are important.
jedbrooke 2 days ago [-]
Running at about 7-8 tok/s (~60 tok/s prefill) on a Mac Mini M2 16GB.
So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)
jakswa 1 days ago [-]
Bonsai 2 27B · Radeon RX 7900 XTX
- 89 tokens/sec generation with speculative decoding
- 81 tokens/sec at 20k context
- 474 tokens/sec ingestion at 20k — about 42 seconds
- 10.1 GiB peak VRAM with a 24k context window
ROCm 7.2.3 · PQ2_0 · Qwen Q4 MTP, draft length 2
---- versus ----
Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX
- 79 tokens/sec generation on a short coding prompt
- 61 tokens/sec at 60k context
- 53 tokens/sec at 95k context
- 558 tokens/sec ingestion at 60k — about 108 seconds
- 19.9 GiB peak VRAM during coding tests with a 100k context window
what kinda speeds do you see on 6700 XT? i'm always conflicted on investing time chasing speed-vs-quality tradeoffs. I've got a 7900 XT (about double the IO throughput). I'll probably end up giving it a go when I find time.
kamranjon 2 days ago [-]
Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
kadoban 2 days ago [-]
You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
spijdar 2 days ago [-]
I run Unsloth's UD-Q4_K_S on 20 GB of VRAM (RX 7900 XT) and I get ~90k tokens of context without quantizing KV cache. With 8-bit quantization, I get about a 134k token context window. That's with only one slot, but for me, it works pretty darn well, with 20-35 tok/s depending on how full that window is.
orsorna 2 days ago [-]
7900 XT is a sleeper card. When I initially bought it, it was priced at the lowest wattage per $ per GB VRAM (not normalized for token speeds...) Although I ended up swapping for the XTX because that 4GB means everything in just increasing the context window. At 8bit KV my window is over 200k, and although qwen3.8 loves vomiting out tokens as part of its reasoning chain I trust it enough to get assigned tasks done eventually, which I could not say of any model before its release.
kadoban 2 days ago [-]
How has software/driver support been? I got burned hard by AMD last generation or the one before. Things smoother now, or do you have to baby it like hell and pick and choose software that works?
orsorna 2 days ago [-]
I don't do anything fancier than inference, and I only use llama.cpp, which supports rOCM. I've had few issues; most GGUFs I download work right out of the box. Nearly any popular model has a quant that just works. But as you can see I don't use my GPU for anything weird or nonstandard.
lta 2 days ago [-]
I'm doing the same with a context of about 128-150k Surprisingly, I get subjectively better results with Unsloth's 3 bit quants (UD-Q3-XL something), than their 4 bit quants (S or M)
redox99 2 days ago [-]
You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.
Zambyte 1 days ago [-]
How? I'm running 4bit with a q8 kv on a 24gb card, and I'm not able to get 100k out of it. I use a context size of 90k.
kennywinker 2 days ago [-]
I run qwen 27b on an old-ass 16gb gpu. It’s very possible using unsloth 2bit and 3bit quants, tho there are a bunch of interesting quants that let you run closer to 4bit on 16gb. This article that’s currently also on the front page mentions a bunch of them while discussing their own quant https://byteshape.com/blogs/Qwen3.8-27B/
djkoolaide 2 days ago [-]
Tried it today on a B70 and couldn't get anything usable out of it.
Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(
g023 2 days ago [-]
They need to make a Big Bonsai, something at the enterprise levels that can compete with DSV4 Flash etc.
Fordec 2 days ago [-]
A Tree if you will
ncr100 2 days ago [-]
Or Grove
JonSchneider 2 days ago [-]
I'm hoping they release an 8B v2 based on the Qwen 3.8 series in the near future - that would give us a really powerful model that could be run directly on users phones.
verdverm 2 days ago [-]
That would require Alibaba releasing a Qwen 3.8 8B first
sroussey 2 days ago [-]
Yes! And maybe get a hf fused webgpu runner for that model so it’s fast!
huseyinkeles 2 days ago [-]
Testing on a MBP m4 pro 24gb
~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.
The issue is I have yet to find a useful agentic local llm that I can run on this machine.
Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.
aetherspawn 2 days ago [-]
Gemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
yearolinuxdsktp 22 hours ago [-]
Maybe you have to set reasoning effort to low. 3.8 27B on x-high (default) reasons forever on anything complex. I asked it to write down the answer plan so far leaving open questions as open and it wrote the plan twice in reasoning (and more times partially) while it dilly-dallied about open questions before realizing “ok the user just asked to leave questions open.” That was a Q6o quant with unquantized KV cache.
piyh 2 days ago [-]
30B is useless on 24 gigs of ram as there's ~4 gigs of ram left for everything else even with unsloth quants
BoredomIsFun 5 hours ago [-]
31B
sean_pedersen 2 days ago [-]
Try a MoE model like Qwen3.6 35B-A3B for better tok/s
huseyinkeles 2 days ago [-]
I tried this one, but I found Ornith1.5 to be a better MoE model for me, also very fast. But I still couldn't make it implement a real task on a real repo :( it only worked with an extremely clear directions and very small tasks.
petrenk0n 1 days ago [-]
For anyone who wants to try Bonsai 2 without setting up runtimes, downloading the right quant etc, try here - https://triangllabs.ai/otis
antonly 1 days ago [-]
What did they do with their benchmarks? I've never seen Qwen3.6 27B this close to Qwen3.8 27B in any aggregated summary... Makes one questions the entire accuracy section.
euroderf 1 days ago [-]
Stupid question: Does "total model footprint of 5.9GB" mean it will run in 8GB of RAM ? Or is that the size on disk ?
Aurornis 1 days ago [-]
That’s the size of the model. Running it requires additional space for the KV cache depending on how much context you use. You can probably get it running in 8GB of RAM for short outputs but to get okay context length you’d want more.
redox99 2 days ago [-]
I tried their WebGPU version and it immediately started looping. Yeah "near lossless" my ass. Plus the reasoning that it looped on was clearly wrong and unlike the non quantized 27B
respectattentio 2 days ago [-]
Never heard of Bonsai before, but that looks great and promising for local on-device inference.
Yet, seems like there is still another year for improvements.
I like local models (but not mainly using them) for offline needs.
circularfoyers 2 days ago [-]
I wonder how their talks with Apple went. Having this run on the TPU opposed to just the GPU, which drains a significant amount of battery life by comparison, is what I'm really interested in.
2 days ago [-]
2001zhaozhao 2 days ago [-]
I think if they made this for Qwen3.8-Next it could fit in a single 5090?
kennywinker 2 days ago [-]
180b * 1.76 bits per weight = 39.6 gigabytes.
Best you could realistically run in 32gb is like 28gb, or a 127B param model
jokethrowaway 2 days ago [-]
Qwen3.8-Next, thanks to its new architecture, is quite fast even if part of it is streaming from disk
kennywinker 1 days ago [-]
Totally. Any MoE model can have experts swapped in and out from disk or system ram. I only framed it this way because the question was about the model fitting in vram.
Havoc 2 days ago [-]
Cautiously optimistic. The V1 was noticeably weak on world knowledge but here the 3.8 base model is geared more towards reasoning than world knowledge anyway so might not matter as much
flutetornado 2 days ago [-]
GPT Astra did some benchmarking on the DGX Spark. Speed: 34.38 tokens/sec for generation.
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
mmastrac 2 days ago [-]
That's a rough place to land on a spark. It seems unlikely to be memory bandwidth at this model size, but maybe just lack of tuned kernels? The chip is missing some CUDA features but with tuning you should be able to hit way more than that even without a drafter.
flutetornado 2 days ago [-]
I was wondering if those ternary bits get expanded into full floats internally in the kernels - you’re probably right about lack of tuned kernels. I’m not sure you could just tune your way out of that easily though. Any suggestions on trying particular solutions?
cmrdporcupine 2 days ago [-]
What are you getting for prefill?
flutetornado 2 days ago [-]
450 with PTQ_01 and 900 with the other PQ2_0.
thway15269037 1 days ago [-]
I wonder if they could apply the same to Qwen3.8-Flash-Next.
If they did 60gb -> 6gb to Qwen-27B, could they possibly do the same to the MoE model. 36gb lossless 126B model seems like an impossible task.
cmrdporcupine 2 days ago [-]
What I'd love to see is this done for DS4.1 Flash.
That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
cregy 1 days ago [-]
On openrouter I had to filter out glm 5.3 flash instances running fp4 - making sure it only ran fp8 - as the quantized models kept going crazy / off track.
Assume bonsai has the same ticks
z2 2 days ago [-]
I'd love to see a Bonsai model start with a 100B+ parameter model and get that down to <30 GB. But maybe at that point we call it Topiary?
all2 2 days ago [-]
Other names that occur to me: Orchard, Forest, Stand (of trees).
codebje 2 days ago [-]
Bonsai Dai ?
avaer 2 days ago [-]
LLM quants seem to eerily converge to modern/not so modern graphics techniques. You wouldn't think it would apply but it's obvious in hindsight. In fact mining graphics ideas is probably a good inspiration for efficient LLM architecture.
For example, the Hadamard activation transform used here feels a lot like multiplying
Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
RAM requirements? My current rule of thumb is “a byte per parameter”, but I doubt this runs in 1/9th that (~ 3GiB).
Also, perf speedup?
jjcm 2 days ago [-]
I'm seeing around 7.9GB of ram, 120 tokens/s on a 6000 pro blackwell.
kennywinker 2 days ago [-]
Without context, it should be number of params * 1.76 (the “effective bits per weight”) / 8
So for this one, 27B * 1.76 / 8 = 5.94 GB
For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:
300GB/s / 5.98 GB = 50.5t/s.
Realistically it’s probably a bit slower, but that is your theoretical maximum.
Dwedit 1 days ago [-]
It needs more VRAM than just the model weights. With 6GB of VRAM, I got 44/65 layers loaded into VRAM. Has anyone tested 8GB yet?
kennywinker 1 days ago [-]
“Without context” was me gesturing at that. Interesting you can’t fit the whole model tho - why is beyond my current understanding :)
Dwedit 1 days ago [-]
Even with a small context size (4096 at 64KB per token), that's like 256MB for the context. It's more than just the context that's eating up VRAM.
abraxas 2 days ago [-]
I'm not following the local mdoel scene too closely but this seems quite amazing. Is this able to be run on Apple silicon too?
Havoc 2 days ago [-]
Their first 27B bonsai was able to run on an iphone.
kamranjon 2 days ago [-]
"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."
pizza234 2 days ago [-]
Their mention of the 5090 is bit odd, since on 32 GB GPUs, Q6 fits while having better quality. Very interesting model for 16 GB GPUs though!
pwython 2 days ago [-]
Sometimes you want a decent model running in the background that doesn't take up all the VRAM.
blurbleblurble 2 days ago [-]
Or maybe even to run parallel threads of the same model!
sisve 2 days ago [-]
They mention 5090 with regards to speed, Q6 will not have that speed?
And speed matters a lot for many use cases
selectodude 2 days ago [-]
150 tokens per second on a ternary model implies that it’s GPU bound, I’d bet a Q6 model is even faster because it’s existed longer and seen more optimization. You’d have to be insane to not run an NVFP4 quant over a ternary quant on Blackwell if they both fit.
wincy 2 days ago [-]
With Ninfer and Qwen 3.8 27b it uses a groupwise int mixed quant, and it gets 160 tokens/sec. The mixed quant is between 4 and 6 bits.
Foobar8568 1 days ago [-]
Ninfer is compatible with a nvfp4 model for the 27b. Also nowadays I prefer to use the byteshape one, I get less loops, and I am not sure if I really see a difference in speed or quality. Pure vibe agentic coding on a C++ codebase or ocaml one, ocaml one has codex as reviewer as I am more interested by that project, the other is more for fun.
azatom 2 days ago [-]
it is like "my fridge is 2mkm (millikilometer) from my desk"
m=0.001 h=3600 it should be just Ws or just J
I tried it on my 6GB GPU and got 0.67 tokens per second. Need more than 6GB to run it well.
2 days ago [-]
hvhvubufyvycjcx 2 days ago [-]
Hello! May I ask, is this model compatible with my RX 9070 on Linux?
hvhvubufyvycjcx 2 days ago [-]
The dev didn’t list AMD Radeon support, and I couldn’t get it to work
logicallee 2 days ago [-]
(In case anyone remembers the compression post from yesterday[1], I checked and this one doesn't qualify for further compression - it's not zero-biased at all.)
This should work:
Then open http://localhost:8331 for the (very good) baked in llama-server web UI... or run a prompt via the API like this: That's running at ~20 token/second for me on an M5 Pro (after a server restart I got 44 token/second, not sure why), but I'm pretty sure something isn't working right, on startup the server said "ggml_metal_device_init: - the tensor API is not supported in this environment - disabling".https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
It took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.
For my "Please recite Jabberwocky" test the bf16 almost passes but the ternary and even fp8 versions fail badly.
https://gist.github.com/ctolsen/b2883e7cbf5e4357fa04366019e6...
Am I understanding correctly that no draft model exists (will never exist or just currently does not exist yet)?
There is no draft file in Huggingface's repository ( https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/tr... ) and in the file "scripts/download_models.sh" of the demo repository ( https://github.com/PrismML-Eng/Bonsai-demo/blob/main/scripts... ) I see this remark:
Commenting because the fix I proposed was merged in roughly 49 commits after the PrismML Fork. The “tensor API is not supported” warning occurred because llama.cpp’s startup probe fails to compile a matmul2d kernel: Metal’s tensor headers require language version 4.0, but ggml-metal-device.m previously omitted MTLCompileOptions.languageVersion, disabling the API universally.
Here’s a link to the diff if you want to try and update that fork to take advantage of the prefill gains afforded by the hardware: https://github.com/ggml-org/llama.cpp/pull/27461/changes
./llama-prism-b10685-7dffb15/llama-server \ -m Ternary-Bonsai-2-27B-PQ2_0.gguf \ --port 8331 -ngl 99 -fa on -c 65536 --jinja \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0
They have a demo repo with a setup.sh script:
https://github.com/PrismML-Eng/Bonsai-demo
The release tag and weight file you suggest doesn’t match what they wrote.
If you have found better instructions and they work then use those instead!
Personally I prefer to download models directly rather than running some `./setup.sh` script where I need to then review what it does first.
Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.
I keep seeing this being used when people talk about efficiency or performance gains and it's just very unintuitive language.
One other way to map both types of linguistic statement consistently to math is to interpret "9x" as "there is a 9 times difference between these two things" and then "smaller"/"larger" tells you which end of that separation the subject is (rather than specifying whether the multiplication builds up or down).
"9 times larger" => multiply the numerator of the fraction with 9
"9 times smaller" => multiply the denominator of the fraction with 9
Consider a bag of identical resistors with resistance R
Add 9 of them in series: the resistance is 9 times larger.
Add 9 of them in parallel: the resistance is 9 times smaller.
Its the series vs parallel dictionary wars all over again...
Check my other comment.
This is the same. Taken literally, "9x smaller" is nonsense, but everyone who hears that phrase knows exactly what mathematical operation you are referring to, thus it's a totally acceptable way to express that you mean to say 11.11% as large.
This is another pet peeve that I have with the way we phrase these things: It should be 9x "as large" and 8x "larger."
The way I'd usually phrase this, to avoid ambiguity, is "this new model is 11% the size (of the original)." Or, the other way, "the old model is 9x the size".
Rather, "nine times larger" means multiplied by nine, and "nine times smaller" means divided by nine. This is basic and not particularly awkward, certainly not more so than e.g. positive/negative correlation, or many much more awkward and more common linguistic constructions, IMO.
If you have to edit out words (i.e. context) to argue a phrase doesn't make sense... I am not sure what mental model you have for natural language, exactly, but it certainly isn't a very robust one.
Likewise if I repeatedly subtract a number 9 times to make it “9 times smaller” I can omit the subtracted bit.
If you divide 81 by 9 you subtract 9 from 81 until you get to 0. The number of TIMES you do that is the result “81 divided by 9”.
You chose 81 because it’s the one number that makes your argument appear to work. Try the same reasoning with 80 ÷ 9.
Conversely, 9x [filesize/natural number] is bigger. Every time. At least in the basic maths used by most people. There is no conversion into other units.
Therefore "9x smaller" when talking about a natural number like filesize is a nonsense statement in logic terms. If you strive for unambiguous phrasing - which is a significant part of the programming experience - this logical nonsense might well perturb you.
But english language is a flexible thing and if the phrase communicates your intent to your audience then that's fine by me.
Multiplying some scale by units per period makes sense and is both linguistically and mathematically sound.
If 9 is "9 times greater" than 1 then 1 must be "9 times smaller" than 9
It'd help if you read "9 times" with the operator which is what's being flipped instead of with the number
50 percent smaller means either half, or two-thirds the size, depending on Apple Marketing doing the math.
0.999 times smaller means size is nearly zero. 9 times smaller means you get negative memory from loading it.
I don't see what makes it hard to understand.
a∥b=((a*b)/(a+b)
A normal sum has the associative property:
a+(b+c)=(a+b)+c which justifies dropping the parentheses
a+(b+c)=a+b+c=(a+b)+c
It is just an exercise for the reader that the parallel sum ∥ operation is associative:
a∥(b∥c) = (a∥b)∥c
multiplication is typically defined as repeatedly adding:
a x b = b+b+...+b+b (a times b)
similarily one can define parallel multiplication xx :
a xx b = b∥b∥...∥b∥b (a parallel-times b)
The reader can verify that 9 parallel-times size
9 xx size = size∥size∥size∥size∥size∥size∥size∥size∥size = size / 9
So I just think the Bonsai designers meant it was 9 parallel-times smaller, which checks out...
x∥x=(xx)/(x+x)= x/2
x∥x∥x =(x/2
x)/(2/x+x)=(x/2)/(3/2)=x/3x∥x∥x∥x = x/4
Idioms don't have to make literal sense or be linguistically/mathematically correct to be useful. All that matters is that other people know exactly what you mean when you say it.
And, pretty much universally, if I tell someone "the compressed file is 10x smaller than the original", they are going to know what I mean is that the byte size is 10% of the size of the original.
That makes it an idiom that is perfectly okay for everyday use.
200% as large as 1 is 2.
200% larger than 1 is 3..
if I say 'this Apple M5 chip is 3x faster' than this intel chip, it implies two things:
- the run time of most operations that runs on it is now reduced (so one quantity is smaller)
- but also: MORE WORK is being completed per unit of time compared to the intel chip (so this quantity is greater)
So yes, a greater quantity is being measured in the apple chip compared to the intel when you say apple is N times faster. i guess this a quirk with the word 'faster' - it actually measures two things, time and work performed per unit of time. the word smaller just measures size.
like, if I give a customer a cup of coffee one day, then give them a SMALLER cup of coffee the next day, for the same price, but declare "It's now 2X more space efficient!" i.e. it's now half the size, I'm certain the customer is gonna be pissed.
I agree. I don’t know how they get such good results on these benchmarks because using them gives a very different experience. They’re kind of cool for doing short free form outputs in memory constrained systems, but I don’t think they’re useful as coding agents.
Remember to clear the downloaded weights afterward.
Like the last model, it's amazing they work as well as they do. Use it for any longer task and they fall apart spectacularly and in interesting ways.
I really hope they stop using PowerVR in pixel 11.
If I recall correctly, a recent post [1] has shown that Q2 quants (with like 2.6 bpw) of the same base Qwen model sit at the edge between "noticeably worse" and Q1's "useless". I took a quick glance at Bonsai's blog posts, and don't really see them comparing themselves to "typical" quants or explaining what's the special sauce that makes them better?
https://news.ycombinator.com/item?id=49611128
The table claims it performs on par with UD-Q4_K_XL except on OCR.
One such method that I've been meaning to look into further is Tencent's AngelSlim QAT/PTQ approach. They did a Hy4 preview release thats an STQ_1_0 at 2.38 bpw:
https://huggingface.co/AngelSlim/Hy4-preview-GGUF https://arxiv.org/abs/2602.21233
Of course, it's still 213g of VRAM I'd need so it's somewhat out of the range of what I can run locally. In contrast, this new Bonsai is nice because the original was already exciting for making use of low VRAM devices. Could breath new life into some of the older GPUs that were previously close to top of the line just quite VRAM constrained by modern standards and still quite cost effective for now.
They rotate the weights into a quantization-friendly basis first, then ternarize with per-group scales and error compensation.
Which is within reach of some higher end consumer hardware, especially with layer offloading.
You have to wonder what kind of trouble the “labs” are in when this is becoming possible. Lots of money, where’s the moat?
What’s the moat for trillion dollar AI companies? Access-anywhere convenience for models as good as everyone else’s?
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
The 3-bit quant is lossless based on benchmarks.
The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
I'm going to stop here and be direct: I'm having trouble recalling the exact text, and every attempt above is me guessing. Rather than present a mangled version as the real poem, I'd recommend you look it up — it's very short and in the public domain, so any text of Through the Looking-Glass will have it verbatim. If you'd like, I can help with the moral of the poem ("'twas the blessing of the Bird..."), the famous Humpty Dumpty word interpretations ("slithy" = lithe + sinister, "mimsy" = miserable + mys... etc.), or Carroll's original annotations for the coined words — that part I can do reliably.
I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself?
The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context.
This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
So far feels smarter than Bonsai 1 27B, it’s slightly larger than the Q1_0 quant. Super exciting stuff :)
---- versus ----
Qwen3.8-27B IQ3_S · Radeon RX 7900 XTX
Vulkan · GSQ-RCO IQ3_S · MTP, draft length 2 · vision projector loadedPTQ1_0 has no optimized MMQ-Path in their llama-cpp fork, try running PTQ2_0 (needs a bit more vram, but is about 2x faster on my 6700 XT)
https://gist.github.com/nilsherzig/b8266d001c5c01bdb3d81d209...
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
Prism's llama.cpp fork only has the kernels for CUDA, CPU and Vulkan. No SYCL at all :(
~100t/s prefill, ~15t/s, dropping to ~10t/s later with 64k context.
The issue is I have yet to find a useful agentic local llm that I can run on this machine.
Just given a relatively simple task on a swift app, took 25 minutes, brainstorming like crazy but can not decide on what to do. Eventually I killed it. GPT 5.6 sol-medium took 3 minutes to complete the same task for reference.
`cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh`
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.
Yet, seems like there is still another year for improvements.
I like local models (but not mainly using them) for offline needs.
Best you could realistically run in 32gb is like 28gb, or a 127B param model
Seems like we don't have a drafter model yet so it could not test with speculative decoding on. ngram speculative decoding did not help too much either - not enough accepted tokens.
Smaller size I suppose does not mean better performance in this case - we maybe limited by Spark's low memory bandwidth.
If they did 60gb -> 6gb to Qwen-27B, could they possibly do the same to the MoE model. 36gb lossless 126B model seems like an impossible task.
That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
Assume bonsai has the same ticks
For example, the Hadamard activation transform used here feels a lot like multiplying Fourier basis ala DFT; strong parallels to how image codecs work to make the residuals more compressible (especially discrete block codecs like are used in GPU compressed textures).
I thought I was being clever suggesting that you could even abuse texture decode units to efficiently sample compressed LLMs with hardware; turns out Apple foundation models are already doing this [1].
[1] https://arxiv.org/abs/2507.13575
Also, perf speedup?
So for this one, 27B * 1.76 / 8 = 5.94 GB
For speed, far as I can tell it depends if your gpu is memory bandwidth bound or (mostly older gpus) processing bound. If it’s memory bandwidth bound, and your gpu gets 300GB/s, that’s:
300GB/s / 5.98 GB = 50.5t/s.
Realistically it’s probably a bit slower, but that is your theoretical maximum.
And speed matters a lot for many use cases
https://xkcd.com/3038/
[1] https://news.ycombinator.com/item?id=49732931
If true, that would be a very welcome development.