live kernel upgrades (was: live kernel patching design)

Ingo Molnar
2015-02-22T09:46:48Z

* Ingo Molnar <mingo@kernel.org> wrote:

> Anyway, let me try to reboot this discussion back to
> technological details by summing up my arguments in
> another mail.

So here's how I see the kGraft and kpatch series. To not
put too fine a point on it, I think they are fundamentally
misguided in both implementation and in design, which turns
them into an (unwilling) extended arm of the security
theater:

- kGraft creates a 'mixed' state where old kernel
functions and new kernel functions are allowed to
co-exist, furthermore there's no guarantee currently at
attempting to get the patching done within a bound
amount of time.

- kpatch uses kernel stack backtraces to determine whether
a task is executing a function or not - which IMO is
fundamentally fragile as kernel stack backtraces are
'debug info' and are maintained and created as such:
we've had long lasting stack backtrace bugs which would
now be turned into 'potentially patching a live
function' type of functional (and hard to debug) bugs.
I didn't see much effort that tries to turn this
equation around and makes kernel stacktraces more
robust.

- the whole 'consistency model' talk both projects employ
reminds me of how we grew 'security modules': where
people running various mediocre projects would in the
end not seek to create a superior upstream project, but
would seek the 'consensus' in the form of cross-acking
each others' patches as long as their own code got
upstream as well ...

I'm not blaming Linus for giving in to allowing security
modules: they might be the right model for such a hard
to define and in good part psychological discipline as
'security', but I sure don't see the necessity of doing
that for 'live kernel patching'.

More importantly, both kGraft and kpatch are pretty limited
in what kinds of updates they allow, and neither kGraft nor
kpatch has any clear path towards applying more complex
fixes to kernel images that I can see: kGraft can only
apply the simplest of fixes where both versions of a
function are interchangeable, and kpatch is only marginally
better at that - and that's pretty fundamental to both
projects!

I think all of these problems could be resolved by shooting
for the moon instead:

- work towards allowing arbitrary live kernel upgrades!

not just 'live kernel patches'.

Work towards the goal of full live kernel upgrades between
any two versions of a kernel that supports live kernel
upgrades (and that doesn't have fatal bugs in the kernel
upgrade support code requiring a hard system restart).

Arbitrary live kernel upgrades could be achieved by
starting with the 'simple method' I outlined in earlier
mails, using some of the methods that kpatch and kGraft are
both utilizing or planning to utilize:

- implement user task and kthread parking to get the
kernel into quiescent state.

- implement (optional, thus ABI-compatible)
system call interruptability and restartability
support.

- implement task state and (limited) device state
snapshotting support

- implement live kernel upgrades by:

- snapshotting all system state transparently

- fast-rebooting into the new kernel image without
shutting down and rebooting user-space, i.e. _much_
faster than a regular reboot.

- restoring system state transparently within the new
kernel image and resuming system workloads where
they were left.

Even complex external state like TCP socket state and
graphics state can be preserved over an upgrade. As far as
the user is concerned, nothing happened but a brief pause -
and he's now running a v3.21 kernel, not v3.20.

Obviously one of the simplest utilizations of live kernel
upgrades would be to apply simple security fixes to
production systems. But that's just a very simple
application of a much broader capability.

Note that if done right, then the time to perform a live
kernel upgrade on a typical system could be brought to well
below 10 seconds system stoppage time: adequate to the vast
majority of installations.

For special installations or well optimized hardware the
latency could possibly be brought below 1 second stoppage
time.

This 'live kernel upgrades' approach would have various
advantages:

- it brings together various principles working towards
shared goals:

- the boot time reduction folks
- the checkpoint/restore folks
- the hibernation folks
- the suspend/resume and power management folks
- the live patching folks (you)
- the syscall latency reduction folks

if so many disciplines are working together then maybe
something really good and long term maintainble can
crystalize out of that effort.

- it ignores the security theater that treats security
fixes as a separate, disproportionally more important
class of fixes and instead allows arbitrary complex
changes over live kernel upgrades.

- there's no need to 'engineer' live patches separately,
there's no need to review them and their usage sites
for live patching relevant side effects. Just create a
'better' kernel as defined by users of that kernel:

- in the enterprise distro space create a more stable
kernel and allow transparent upgrades into it.

- in the desktop distro space create a kernel that
will contain fixes and support for latest hardware.

- etc.

there's the need to engineer c/r and device state
support, but that's a much more concentrated and
specific field with many usecases beyond live
kernel upgrades.

We have many of the building blocks in place and have them
available:

- the freezer code already attempts at parking/unparking
threads transparently, that could be fixed/extended.

- hibernation, regular suspend/resume and in general
power management has in essence already implemented
most building blocks needed to enumerate and
checkpoint/restore device state that otherwise gets
lost in a shutdown/reboot cycle.

- c/r patches started user state enumeration and
checkpoint/restore logic

A feature like arbitrary live kernel upgrades would be well
worth the pain and would be worth the complications, and
it's actually very feasible technically.

The goals of the current live kernel patching projects,
"being able to apply only the simplest of live patches",
which would in my opinion mostly serve the security
theater? They are not forward looking enough, and in that
sense they could even be counterproductive.

Thanks,

Ingo