Re: live kernel upgrades (was: live kernel patching design)

Ingo Molnar
2015-02-24T10:23:36Z

* Josh Poimboeuf <jpoimboe@redhat.com> wrote:

> Your upgrade proposal is an *enormous* disruption to the
> system:
>
> - a latency of "well below 10" seconds is completely
> unacceptable to most users who want to patch the kernel
> of a production system _while_ it's in production.

I think this statement is false for the following reasons.

- I'd say the majority of system operators of production
systems can live with a couple of seconds of delay at a
well defined moment of the day or week - with gradual,
pretty much open ended improvements in that latency
down the line.

- I think your argument ignores the fact that live
upgrades would extend the scope of 'users willing to
patch the kernel of a production system' _enormously_.

For example, I have a production system with this much
uptime:

10:50:09 up 153 days, 3:58, 34 users, load average: 0.00, 0.02, 0.05

While currently I'm reluctant to reboot the system to
upgrade the kernel (due to a reboot's intrusiveness),
and that is why it has achieved a relatively high
uptime, but I'd definitely allow the kernel to upgrade
at 0:00am just fine. (I'd even give it up to a few
minutes, as long as TCP connections don't time out.)

And I don't think my usecase is special.

What gradual improvements in live upgrade latency am I
talking about?

- For example the majority of pure user-space process
pages in RAM could be saved from the old kernel over
into the new kernel - i.e. they'd stay in place in RAM,
but they'd be re-hashed for the new data structures.
This avoids a big chunk of checkpointing overhead.

- Likewise, most of the page cache could be saved from an
old kernel to a new kernel as well - further reducing
checkpointing overhead.

- The PROT_NONE mechanism of the current NUMA balancing
code could be used to transparently mark user-space
pages as 'checkpointed'. This would reduce system
interruption as only 'newly modified' pages would have
to be checkpointed when the upgrade happens.

- Hardware devices could be marked as 'already in well
defined state', skipping the more expensive steps of
driver initialization.

- Possibly full user-space page tables could be preserved
over an upgrade: this way user-space execution would be
unaffected even in the micro level: cache layout, TLB
patterns, etc.

There's lots of gradual speedups possible with such a model
IMO.

With live kernel patching we run into a brick wall of
complexity straight away: we have to analyze the nature of
the kernel modification, in the context of live patching,
and that only works for the simplest of kernel
modifications.

With live kernel upgrades no such brick wall exists, just
about any transition between kernel versions is possible.

Granted, with live kernel upgrades it's much more complex
to get the 'simple' case into an even rudimentarily working
fashion (full userspace state has to be enumerated, saved
and restored), but once we are there, it's a whole new
category of goodness and it probably covers 90%+ of the
live kernel patching usecases on day 1 already ...

Thanks,

Ingo