Re: [MAINTAINERS SUMMIT] Other LLM-related topics - tags, newcomers, etc

["Theodore Tso" <tytso@mit.edu>]

On Thu, Jul 16, 2026 at 09:53:42PM -0500, Mauro Carvalho Chehab wrote:
> > What
> > will we do when the current round of corporate generosity ends and that
> > tool goes away?  Maybe I'm worrying too much, but this does seem, to me,
> > like a possibility we should keep in mind.
> 
> This is a serious concern. It sounds risky to rely on that, as there's
> no free lunch. We need to rely on something that can be managed in
> an affordable way, prioritizing models that can run on affortable GPUs
> and are open source.

This gets tricky.  We can divide ML models into a couple of
different classes:

     1. Those that can fit on a mobile phone
     2. Those that can fit on low-end developer machine (16GB-32GB
        unified memory, or 16GB of VRAM in your GPU)
     3. Those that fit in High-end developer machines (128GB unified
        memory, such as could be found in a M5 Max Macbook Pro, a DGX
        Spark, or an AMD Strix Halo)
     4. Those that fit into one or more Enterprise servers with 8 H100
        GPU's --- that is, frontier models.

Machines in category 3 run about $4k (on the low-end, without a
monitor) and go up from there.  About six weeks ago, I invested in a
M5 Max Macbook Pro with 128GB, and it set me back $6,214 USD
(including tax).  When Apple increased their prices due to the
DRAMpocalypse, for the first time, I've seen a computer *appreciate*
in value after being purchased --- by $1,400 USD.  :-)   So questions of
access equity is already an issue with machines in this category.

Machines in category 4 run around $400,000 each.  (Of course, after
the dot.COM bubble implosion, Sunfire E10K's that startups paid
$100,000 ended up selling for pennies on the dollar.  So after a
Neocloud company go out of business, maybe thse machines will be
affordable by individual developers.  However, even then your partner
might not be enthusiastic about the heat and sound from one of these
data center servers being run in your office or living room --- not to
mention the electricity bill.  :-)

Now, a H100 has 80 GB of High Bandwidth Memory (HBM) which has a
bandwidth of 2 TB/s.  So a server with 8 H100's has a 640 GB of HBM
and an aggregate bandwidth of 16 TB/s.  In contrast, a M5 Max with
128 GB and a 40-core GPU has a memory bandwidth of 614 GB/s.  (A
normal M5 Macbook Pro has 153 GB/s memory bandwidth and maxes out at
32GB.)

Running the 671 billion parameter Deepseek R1 model at full precision
requires 1.5TB of VRAM --- so two of these 8x H100 servers.  Of could
try to run them using quantization techniques, by collapsing each
parameter from 16 bits to 4 bits, or even 1 bit to reduce the size of
the ML rig required.  However, you lose a lot of accuracy when you do
that, and when models are much more prone to hallucinate when you use
the more aggressive levels of quantization.

So it's one say to say, we should figure out how to try to run Sashiko
on a local LLM, using open-weight models.  But it's going to be a lot
easier to propose such a thing than to actually do it.

What we *might* be able to try doing is to take an open-weight model
that can fit on a 128GB machine, and then fine-tuning it by feeding it
several years of LKML archives which we convniently have in public
inbox format.  This might allow us to create a specialized model which
is optimized for the Linux kernel, and it would be interesting to see
how this compares with a general frontier model.

But even if we did that, it use would be limited to those people who
can afford one of these 128GB unified memory machines or can otherwise
get access to one of these machines.

Cheers,

					- Ted