Re: Stop false review statements

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

On Sun, 17 May 2026 14:57:01 -0400
"Theodore Tso" <tytso@mit.edu> wrote:

> On Sun, May 17, 2026 at 11:17:06AM -0700, Roman Gushchin wrote:
> > 
> > I actually tried to run it with ollama on my
> > personal framework 13. Adding nominal support is trivial, but the
> > whole thing is not really useful: I can get maybe few hundreds
> > tokens per second using a quantified model with reduced quality; an
> > average sashiko review is consuming 3.5 millions tokens (with Gemini
> > 3.1 pro, it’s also model-dependent).  
> 
> I'm curious.  What hardware and LLM model were you using?  A few
> hundred tokens per second seems surprising high.  My initial
> research[1] showes that an M5 Max Macbook Pro costing 5 or 6 kilobucks
> can do 31.6 tokens/second on a 27B 4-bit Quanitized model (Qwen 3.5).
> 
> [1] https://www.reddit.com/r/LocalLLaMA/comments/1rzkw4x/m5_max_128g_performance_tests_i_just_got_my_new/
> 
> The model matters of course.  With Gemma 3 27B and a 6-bit
> quantization, it's 21 tokens/s, and with Deepseek R1 8B Q6_K, it's
> 72.8 tokens/second.  But unless you're using a really low-end model,
> or a really expensive, splufty hardware platform, I haven't seen
> reports of hundreds of tokens per second on hardware costing a
> reasonable amount of memory.  (I'll set aside the question of whether
> spending $6k for a fully spec'ed out M5 Max Macbook Pro, or $15k for a
> fully spec'ed out M3 Ultra Mac Studio is "reasonable".)

Ted,

Here, I'm using a RX9060XT, with is a relatively budget hardware.

It is also at the range of dozens of tokens per second. If you're
interested, I ran a benchmark this weekend with 3 models (just
for the sake of testing a set of turboquant patches - those aren't
the models I normally use).

You can see results here:
	https://github.com/ollama/ollama/pull/15505#issuecomment-4467278354

llama3.2:3b with f16 speed gives 72.5 decode tokens/s, and 37 decode tokens/s
with tq4 (actually a modified version of it) which, according with the
PR author, has quality almost identical to f16.

The main issue on such hardware is to have only 16 GB VRAM, making
it a little bit slow for models like qwen3.6:35b, as it will partially 
use CPU. Still, you can get a pretty decent answer in a couple of
minutes, with thinking enabled.

> As a result I'm not entirely sure how realistic it is to do reviews
> using "free" (you still have to pay $$$ for the hardware) local,
> open-weight LLM's if an average review requires around 3.5 million
> tokens.

Yes, 3.5 million tokens is indeed too much. I wonder why. Maybe
Gemini spreads the same query to multiple instances, making it 
spend a lot more tokens?

Here, I did some tests asking some LLM models to review code,
getting answers on a reasonable time (but didn't try to use sashiko
prompts).

Thanks,
Mauro