Re: [MAINTAINERS SUMMIT] Other LLM-related topics - tags, newcomers, etc

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

On Thu, 16 Jul 2026 19:59:02 -0400
"Theodore Tso" <tytso@mit.edu> wrote:

> On Thu, Jul 16, 2026 at 09:53:42PM -0500, Mauro Carvalho Chehab wrote:
> > > What
> > > will we do when the current round of corporate generosity ends and that
> > > tool goes away?  Maybe I'm worrying too much, but this does seem, to me,
> > > like a possibility we should keep in mind.  
> > 
> > This is a serious concern. It sounds risky to rely on that, as there's
> > no free lunch. We need to rely on something that can be managed in
> > an affordable way, prioritizing models that can run on affortable GPUs
> > and are open source.  
> 
> This gets tricky.  We can divide ML models into a couple of
> different classes:
> 
>      1. Those that can fit on a mobile phone
>      2. Those that can fit on low-end developer machine (16GB-32GB
>         unified memory, or 16GB of VRAM in your GPU)
>      3. Those that fit in High-end developer machines (128GB unified
>         memory, such as could be found in a M5 Max Macbook Pro, a DGX
>         Spark, or an AMD Strix Halo)
>      4. Those that fit into one or more Enterprise servers with 8 H100
>         GPU's --- that is, frontier models.
> 
> Machines in category 3 run about $4k (on the low-end, without a
> monitor) and go up from there.  About six weeks ago, I invested in a
> M5 Max Macbook Pro with 128GB, and it set me back $6,214 USD
> (including tax).  When Apple increased their prices due to the
> DRAMpocalypse, for the first time, I've seen a computer *appreciate*
> in value after being purchased --- by $1,400 USD.  :-)   So questions of
> access equity is already an issue with machines in this category.

I'd say the best would be to aim on (3) and (4). Not as powerful
as H100 GPUs, but it may end work.

> Machines in category 4 run around $400,000 each.  (Of course, after
> the dot.COM bubble implosion, Sunfire E10K's that startups paid
> $100,000 ended up selling for pennies on the dollar.  So after a
> Neocloud company go out of business, maybe thse machines will be
> affordable by individual developers.  However, even then your partner
> might not be enthusiastic about the heat and sound from one of these
> data center servers being run in your office or living room --- not to
> mention the electricity bill.  :-)

Those are really noisy. I don't think it is realistic to use it.
Maybe there is an alternative, but I've no idea about how it
actually works: having multiple machines sharing the same model.
I heard some people using DGX Spark and AMD Strix Halo on such
configurations, but I suspect that performance would seriously
drop.

> Now, a H100 has 80 GB of High Bandwidth Memory (HBM) which has a
> bandwidth of 2 TB/s.  So a server with 8 H100's has a 640 GB of HBM
> and an aggregate bandwidth of 16 TB/s.  In contrast, a M5 Max with
> 128 GB and a 40-core GPU has a memory bandwidth of 614 GB/s.  (A
> normal M5 Macbook Pro has 153 GB/s memory bandwidth and maxes out at
> 32GB.)
> 
> Running the 671 billion parameter Deepseek R1 model at full precision
> requires 1.5TB of VRAM --- so two of these 8x H100 servers.  Of could
> try to run them using quantization techniques, by collapsing each
> parameter from 16 bits to 4 bits, or even 1 bit to reduce the size of
> the ML rig required.  However, you lose a lot of accuracy when you do
> that, and when models are much more prone to hallucinate when you use
> the more aggressive levels of quantization.

There are kv-algorithms that provide a good compromise, like turbo-quant.
the precision is still good with 4 or 3 bits.

> So it's one say to say, we should figure out how to try to run Sashiko
> on a local LLM, using open-weight models.  But it's going to be a lot
> easier to propose such a thing than to actually do it.

The main point is: do we really need 671B parameters? Those models
speak a lot of different languages, have medical databases, and a lot
of other random knowledge that are useless for kernel development.

I've been playing for a while with qwen 3.6 with 24KB context size,
36B parameters (3B activated), 4bits kv quantization and it does produce
some decent results. The main limitation is the context size: it is
probably not big enough to test big files (*)

(*) my GPU has 16GB and it is not dedicated to LLM - still, it does
    present results on a reasonable time (a couple of minutes) and
    with decent precision.

Sure, while we have free tokens for sashiko, we can afford using
bigger models, but at the same time we should invest some effort
to make it viable on smaller models as well.

> What we *might* be able to try doing is to take an open-weight model
> that can fit on a 128GB machine, and then fine-tuning it by feeding it
> several years of LKML archives which we convniently have in public
> inbox format.  This might allow us to create a specialized model which
> is optimized for the Linux kernel, and it would be interesting to see
> how this compares with a general frontier model.
> 
> But even if we did that, it use would be limited to those people who
> can afford one of these 128GB unified memory machines or can otherwise
> get access to one of these machines.

I think we should aim on training an open-wight model up to 36B parameters
with 3-4B parameters activated. Those will run easily on 64B unified 
memory even with a big context and may still work on machines with
16GB or with 32GB VRAM GPU (with partial CPU offload - specially
with 16GB).

Thanks,
Mauro