Re: A system for rebootless kernel security updates

Re: A system for rebootless kernel security updates

[Tomasz Chmielewski]

Jeff Arnold wrote:

> I've put together an automatic system for applying kernel security patches 
> to the Linux kernel without rebooting it, and I wanted to share this 
> system with the community in case others find it useful or interesting.

Hmm, the idea seem to be patented by Microsoft, i.e. this patent from 
December 2002:

http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching

(and other patents by Microsoft if you search for "hotpatching").


And those patent descriptions, by the way, remind the way kexec works 
("A software module is hotpatched by loading a patch into memory and 
modifying an instruction in the original module to jump to the patch"), 
which was released much earlier... In essence, they patented kexec ;)


-- 
Tomasz Chmielewski
http://wpkg.org

Re: A system for rebootless kernel security updates

[Andi Kleen]

> And those patent descriptions, by the way, remind the way kexec works
> ("A software module is hotpatched by loading a patch into memory and
> modifying an instruction in the original module to jump to the patch"),
> which was released much earlier... In essence, they patented kexec ;)

The basic patching idea is old and has been used many times, long
predating kexec. e.g. it's a common way to implement incremental linkers
too.

-Andi

Re: A system for rebootless kernel security updates

[Pavel Machek]

On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
> Jeff Arnold wrote:
> 
> >I've put together an automatic system for applying 
> >kernel security patches to the Linux kernel without 
> >rebooting it, and I wanted to share this system with 
> >the community in case others find it useful or 
> >interesting.
> 
> Hmm, the idea seem to be patented by Microsoft, i.e. 
> this patent from December 2002:
> 
> http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> 
> (and other patents by Microsoft if you search for 
> "hotpatching").

...so US will not be able to fix security holes without reboot, good.
Perhaps they fix their stupid laws after next worm outbreak...

							Pavel
-- 				
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: A system for rebootless kernel security updates

[Gilles Espinasse]

----- Original Message ----- 
From: "Pavel Machek" <pavel@ucw.cz>
To: "Tomasz Chmielewski" <mangoo@wpkg.org>
Cc: "LKML" <linux-kernel@vger.kernel.org>; <jbarnold@MIT.EDU>;
<francois.cami@free.fr>; "Andi Kleen" <andi@firstfloor.org>;
<mail@earthworm.de>
Sent: Sunday, April 27, 2008 12:17 PM
Subject: Re: A system for rebootless kernel security updates


> On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
> > Jeff Arnold wrote:
> >
> > >I've put together an automatic system for applying
> > >kernel security patches to the Linux kernel without
> > >rebooting it, and I wanted to share this system with
> > >the community in case others find it useful or
> > >interesting.
> >
> > Hmm, the idea seem to be patented by Microsoft, i.e.
> > this patent from December 2002:
> >
> > http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> >
> > (and other patents by Microsoft if you search for
> > "hotpatching").
>
> ...so US will not be able to fix security holes without reboot, good.
> Perhaps they fix their stupid laws after next worm outbreak...
>
I have hot-patched on PDP-11 without reboot over the phone line many years
before Microsoft patent.
In the time of MS patent, the machines I care were no more running.

Just another stupid patent.

Gilles

Re: A system for rebootless kernel security updates

[Willy Tarreau]

On Sun, Apr 27, 2008 at 12:17:00PM +0200, Pavel Machek wrote:
> On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
> > Jeff Arnold wrote:
> > 
> > >I've put together an automatic system for applying 
> > >kernel security patches to the Linux kernel without 
> > >rebooting it, and I wanted to share this system with 
> > >the community in case others find it useful or 
> > >interesting.
> > 
> > Hmm, the idea seem to be patented by Microsoft, i.e. 
> > this patent from December 2002:
> > 
> > http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> > 
> > (and other patents by Microsoft if you search for 
> > "hotpatching").
> 
> ...so US will not be able to fix security holes without reboot, good.
> Perhaps they fix their stupid laws after next worm outbreak...

Sounds like a bullshit patent. I remember having loaded a lot of NLM
patches under netware 4.0 in 96-97 without ever rebooting. I think
that the patches only redefined the faulty symbol(s) they wanted to
patch. That was pretty convenient because when in doubt, you could
simply unload the modules and get back to previous situation.

> 							Pavel

Willy

Re: A system for rebootless kernel security updates

[Oliver Pinter]

and if i good know, the macos make too the kernel updates

On 4/27/08, Willy Tarreau <w@1wt.eu> wrote:
> On Sun, Apr 27, 2008 at 12:17:00PM +0200, Pavel Machek wrote:
> > On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
> > > Jeff Arnold wrote:
> > >
> > > >I've put together an automatic system for applying
> > > >kernel security patches to the Linux kernel without
> > > >rebooting it, and I wanted to share this system with
> > > >the community in case others find it useful or
> > > >interesting.
> > >
> > > Hmm, the idea seem to be patented by Microsoft, i.e.
> > > this patent from December 2002:
> > >
> > > http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> > >
> > > (and other patents by Microsoft if you search for
> > > "hotpatching").
> >
> > ...so US will not be able to fix security holes without reboot, good.
> > Perhaps they fix their stupid laws after next worm outbreak...
>
> Sounds like a bullshit patent. I remember having loaded a lot of NLM
> patches under netware 4.0 in 96-97 without ever rebooting. I think
> that the patches only redefined the faulty symbol(s) they wanted to
> patch. That was pretty convenient because when in doubt, you could
> simply unload the modules and get back to previous situation.
>
> > 							Pavel
>
> Willy
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


-- 
Thanks,
Oliver

Re: A system for rebootless kernel security updates

[Jesper Juhl]

2008/4/27 Willy Tarreau <w@1wt.eu>:
> On Sun, Apr 27, 2008 at 12:17:00PM +0200, Pavel Machek wrote:
>  > On Thu 2008-04-24 16:26:44, Tomasz Chmielewski wrote:
>  > > Jeff Arnold wrote:
>  > >
>  > > >I've put together an automatic system for applying
>  > > >kernel security patches to the Linux kernel without
>  > > >rebooting it, and I wanted to share this system with
>  > > >the community in case others find it useful or
>  > > >interesting.
>  > >
>  > > Hmm, the idea seem to be patented by Microsoft, i.e.
>  > > this patent from December 2002:
>  > >
>  > > http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
>  > >
>  > > (and other patents by Microsoft if you search for
>  > > "hotpatching").
>  >
>  > ...so US will not be able to fix security holes without reboot, good.
>  > Perhaps they fix their stupid laws after next worm outbreak...
>
>  Sounds like a bullshit patent. I remember having loaded a lot of NLM
>  patches under netware 4.0 in 96-97 without ever rebooting. I think
>  that the patches only redefined the faulty symbol(s) they wanted to
>  patch. That was pretty convenient because when in doubt, you could
>  simply unload the modules and get back to previous situation.
>

And then there's 'alternatives' that patch running code, there's kexec
and I guess you could even say that various root kits that patch the
running kernel get prior art on that patent ;)

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please http://www.expita.com/nomime.html

Re: A system for rebootless kernel security updates

[Bill Davidsen]

Tomasz Chmielewski wrote:
> Jeff Arnold wrote:
> 
>> I've put together an automatic system for applying kernel security 
>> patches to the Linux kernel without rebooting it, and I wanted to 
>> share this system with the community in case others find it useful or 
>> interesting.
> 
> Hmm, the idea seem to be patented by Microsoft, i.e. this patent from 
> December 2002:
> 
> http://www.google.com/patents?id=cVyWAAAAEBAJ&dq=hotpatching
> 
> (and other patents by Microsoft if you search for "hotpatching").
> 
> 
> And those patent descriptions, by the way, remind the way kexec works 
> ("A software module is hotpatched by loading a patch into memory and 
> modifying an instruction in the original module to jump to the patch"), 
> which was released much earlier... In essence, they patented kexec ;)
> 
I think you will find prior art all the way back to the PDP-8 (or 11) 
and if memory serves DTSS, which was in the 1960's. I think MULTICS 
allowed that as well, by patching the library dispatch table (sort of 
like TLB) to map a virtual address of the entry point to a new location.

All of which doesn't matter, of course, because no one has the money to 
challenge MSFT, and it would be in court until the heat death of the 
universe anyway.

-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: A system for rebootless kernel security updates

[Jeff Arnold]

> i.e. this patent from December 2002

That's a patent application, not a patent.  That application has received 
a "final rejection" from the U.S. patent office (Microsoft is appealing, 
but so far their attempts at arguing the merits of the application have 
not been successful).  You can browse the relevant documents at

http://portal.uspto.gov/external/portal/pair

(The application number is 10/307,902).

Jeff Arnold
jbarnold@mit.edu

A system for rebootless kernel security updates

[Jeff Arnold]

Hello,

I've put together an automatic system for applying kernel security patches 
to the Linux kernel without rebooting it, and I wanted to share this 
system with the community in case others find it useful or interesting.

Here's the summary:  The system takes as input a kernel security patch 
(which can be a unified diff taken directly from Linus' GIT tree) and the 
source code corresponding to the running kernel, and it automatically 
creates a set of kernel modules to perform the update.  The running kernel 
does not need to have been customized in advance in any way.  To be fully 
automatic, the system cannot be used to apply patches that introduce 
semantic changes to data structures, but most Linux kernel security 
patches don't make these kinds of changes.  I've evaluated the system 
against various kernel versions and security vulnerabilities, and the 
system can automatically apply 84% of the significant kernel security 
patches from May 2005 through December 2007.

I've been pursuing this project because I don't like dealing with reboots 
whenever a new local kernel security vulnerability is discovered.  The 
rebootless update practices/systems that are already out there require 
manually constructing an update (through a process that can be tricky and 
error-prone), and they tend to have other disadvantages as well (such as 
requiring a custom kernel, not handling inline functions properly, etc). 
This new system works on existing kernels, and it simply takes a unified 
diff as input and does the rest on its own.

The system's website is http://web.mit.edu/ksplice.

The GIT repository, code tarball, and binary tarballs are available here:
http://web.mit.edu/ksplice/ksplice.git
http://web.mit.edu/ksplice/dist/ksplice-src.tar.gz
http://web.mit.edu/ksplice/dist/ksplice-bin-i386.tar.gz
http://web.mit.edu/ksplice/dist/ksplice-bin-x86_64.tar.gz

A document describing how the system works is available here: 
http://web.mit.edu/ksplice/doc/ksplice.pdf

Any feedback would be appreciated.

Jeff Arnold
jbarnold@mit.edu

Re: A system for rebootless kernel security updates

[FD Cami]

On Wed, 23 Apr 2008 14:59:05 -0400 (EDT)
Jeff Arnold <jbarnold@MIT.EDU> wrote:

> Hello,

Hi Jeff,

> I've put together an automatic system for applying kernel security patches 
> to the Linux kernel without rebooting it, and I wanted to share this 
> system with the community in case others find it useful or interesting.

(reading on)

> Here's the summary:  The system takes as input a kernel security patch 
> (which can be a unified diff taken directly from Linus' GIT tree) and the 
> source code corresponding to the running kernel, and it automatically 
> creates a set of kernel modules to perform the update.  The running kernel 
> does not need to have been customized in advance in any way.  To be fully 
> automatic, the system cannot be used to apply patches that introduce 
> semantic changes to data structures, but most Linux kernel security 
> patches don't make these kinds of changes.  I've evaluated the system 
> against various kernel versions and security vulnerabilities, and the 
> system can automatically apply 84% of the significant kernel security 
> patches from May 2005 through December 2007.

Awesome. Please note that reading this, I thought at first that the set of
kernel modules were in fact, updated kernel modules (i.e. necessary unloading/
loading of modules) which I understood was not the case after reading your
PDF. After checking with a friend of mine, he understood it like I did.
Perhaps :

- it automatically creates a set of kernel modules to perform the update.
+ it automatically creates a set of kernel modules containing the kernel
+ functions touched by the update, and arranges for the running kernel to
+ use the new functions from now on.

would be better.

> I've been pursuing this project because I don't like dealing with reboots 
> whenever a new local kernel security vulnerability is discovered.  The 
> rebootless update practices/systems that are already out there require 
> manually constructing an update (through a process that can be tricky and 
> error-prone), and they tend to have other disadvantages as well (such as 
> requiring a custom kernel, not handling inline functions properly, etc). 
> This new system works on existing kernels, and it simply takes a unified 
> diff as input and does the rest on its own.

It really looks like a non intrusive way of achieving superior uptime.

Congrats ! 

Best,

Francois

Re: A system for rebootless kernel security updates

[Andi Kleen]

Jeff Arnold <jbarnold@MIT.EDU> writes:

> I've put together an automatic system for applying kernel security
> patches to the Linux kernel without rebooting it, and I wanted to
> share this system with the community in case others find it useful or
> interesting.

Sounds interesting.  You should consider bringing the kernel module
up to kernel coding standards and submitting it for mainline. That
would be the best way to get more users and testers.

Have you also considered to turn it into a generic incremental binary
linker too? I guess most of the infrastructure needed for it is
already there. Even though gold looks very promising long link times
can be still a problem in development.

-Andi

Re: A system for rebootless kernel security updates

[Jeff Arnold]

> You should consider bringing the kernel module up to kernel coding 
> standards and submitting it for mainline. That would be the best way to 
> get more users and testers.

I'm certainly interested in bringing the code up to kernel coding 
standards (for example, I'd be happy to address any issues with the code 
that are brought to my attention).  I'm not sure whether submitting it for 
mainline makes sense since the software doesn't significantly benefit from 
being bundled with the kernel.

Instead, it might be more important to 1) package the userspace 
update-construction software for common Linux distributions to make it 
easily available to interested users, and 2) to provide binary kernel 
updates for common distribution kernels so that users can simply sign up 
and get fewer "your machine needs to be rebooted now for an update to take 
effect" notifications.

I'm open to suggestions about how I should proceed, however.

> Have you also considered to turn it into a generic incremental binary
> linker too?

I've considered modifying the system for use in other contexts, but I 
don't think that I'll get around to such a project anytime soon; my 
current focus is on making it easier for people to perform rebootless 
kernel security updates.

Jeff Arnold
jbarnold@mit.edu

Re: A system for rebootless kernel security updates

[Andi Kleen]

Jeff Arnold wrote:

> I'm certainly interested in bringing the code up to kernel coding
> standards (for example, I'd be happy to address any issues with the code
> that are brought to my attention).  I'm not sure whether submitting it
> for mainline makes sense since the software doesn't significantly
> benefit from being bundled with the kernel.

To be honest you weren't the first to come up with something like this
(although you're the first to post to l-k as far as I know). But the
usual problem of something that is kept out of tree is that it
eventually bitrots and gets forgotten. The only sane way to make such
extensions a generically usable linux feature is to merge them to mainline.

> Instead, it might be more important to 1) package the userspace
> update-construction software for common Linux distributions to make it
> easily available to interested users, and 2) to provide binary kernel
> updates for common distribution kernels so that users can simply sign up
> and get fewer "your machine needs to be rebooted now for an update to
> take effect" notifications.

(2) is a incredibly large amount of work longer time. And when
distributions merge your feature they become committed to it so even if
you go away they would still need to maintain it on their own. Since
they understand how much work this is they likely won't do it in the
first place. Really it's far  better to just merge if you want it to
make it out of the "toy" stage.

-Andi

Re: A system for rebootless kernel security updates

[Jeff Arnold]

> The only sane way to make such extensions a generically usable linux 
> feature is to merge them to mainline.

> Really it's far better to just merge if you want it to make it out of 
> the "toy" stage.

You're probably right.

I'm willing to undertake the project of bringing the code up to kernel 
coding standards so that it can eventually be considered for mainline. 
I'll plan on undertaking this project if I don't receive feedback that I 
shouldn't do so.

If people have concerns about the high-level design of the system, it 
would be useful for me to know that information sooner rather than later.

If there's anything that I can do to facilitate feedback at this stage, 
please let me know.

Thanks for the advice.

Jeff Arnold
jbarnold@mit.edu

Re: A system for rebootless kernel security updates

[Dan Noe]

Jeff Arnold wrote:
> If there's anything that I can do to facilitate feedback at this stage, 
> please let me know.

Is there a ksplice mailing list?  I did not see mention of one on the 
web page.

Cheers,
Dan

-- 
                     /--------------- - -  -  -   -   -
                     |  Dan Noe
                     |  http://isomerica.net/~dpn/

Re: A system for rebootless kernel security updates

[Jeff Arnold]

>> If there's anything that I can do to facilitate feedback at this stage, 
>> please let me know.

> Is there a ksplice mailing list?

I've now created one: 
http://mailman.mit.edu/mailman/listinfo/ksplice-devel

Jeff Arnold
jbarnold@mit.edu

Re: A system for rebootless kernel security updates

[Enrico Weigelt]

* Jeff Arnold <jbarnold@MIT.EDU> wrote:

Hi,

> I'm willing to undertake the project of bringing the code up to kernel 
> coding standards so that it can eventually be considered for mainline. 
> I'll plan on undertaking this project if I don't receive feedback that I 
> shouldn't do so.

Great think :)
I'd actually like to see it mainline tree (I prefer vanilla kernel
instead of distro specific). 

> If people have concerns about the high-level design of the system, it 
> would be useful for me to know that information sooner rather than later.

I didn't have the time for an deeper study yet, but as you already
mentioned, there're lots of limitations which can make it harmful:
as soon as interfaces chance, you're in *big* trouble. There should
be a way for finding them (automatically). Maybe extract the 
interface signatures (including structs!) so some appropriate place
next to the kernel, so they can be checked before (re)loading the
module.

Ah, of course you can't change code that's not an dynamic module :(


Even this goes OT now - I'd really prefer more things in userland,
eg. network- or synthetic filesystems, crypt stuff, etc - so 
there would be less to update within the kernel ;-o

cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: A system for rebootless kernel security updates

[David Collier-Brown]

Enrico Weigelt wrote:
> I didn't have the time for an deeper study yet, but as you already
> mentioned, there're lots of limitations which can make it harmful:
> as soon as interfaces chance, you're in *big* trouble. There should
> be a way for finding them (automatically). Maybe extract the 
> interface signatures (including structs!) so some appropriate place
> next to the kernel, so they can be checked before (re)loading the
> module.

  Actually there is a mature discipline around data changes that
dates back to before Unix v7, but it's off-topic for this
discussion: it would make the changes too big (;-))

  If anyone is interested, see http://multicians.org/stachour.html
and/or send me mail

--dave (reading LKML when he should be working) c-b
-- 
David Collier-Brown            | Always do right. This will gratify
Sun Microsystems, Toronto      | some people and astonish the rest
davecb@sun.com                 |                      -- Mark Twain
(905) 943-1983, cell: (647) 833-9377, (800) 555-9786 x56583
bridge: (877) 385-4099 code: 506 9191#

Re: A system for rebootless kernel security updates

[Christian Hesse]

On Wednesday 23 April 2008, Jeff Arnold wrote:
> Hello,
>
> I've put together an automatic system for applying kernel security patches
> to the Linux kernel without rebooting it, and I wanted to share this
> system with the community in case others find it useful or interesting.

This sounds really interesting. I just wanted to test on my Gentoo system with 
kernel 2.6.25. The tarball creation works, but applying the change brings an 
error to syslog:

ksplice_242e9pfc_helper: Unknown symbol init_mm

Any chance to get a fix for that?
-- 
Regards,
Chris

Re: A system for rebootless kernel security updates

[Jeff Arnold]

> I just wanted to test on my Gentoo system with kernel 2.6.25. The 
> tarball creation works, but applying the change brings an error to 
> syslog

Sorry.  (In 2.6.25, init_mm is marked as an unused symbol, and so whether 
it is exported depends on whether CONFIG_UNUSED_SYMBOLS is enabled).

I've committed a change which should fix this problem.  You can get get 
the new version (0.8.1) from http://web.mit.edu/ksplice/dist/ or from Git. 
(I haven't yet made the new version the default, so you won't receive it 
from the main Ksplice page).

Jeff Arnold
jbarnold@mit.edu

On Thu, 24 Apr 2008, Christian Hesse wrote:

> On Wednesday 23 April 2008, Jeff Arnold wrote:
>> Hello,
>>
>> I've put together an automatic system for applying kernel security patches
>> to the Linux kernel without rebooting it, and I wanted to share this
>> system with the community in case others find it useful or interesting.
>
> This sounds really interesting. I just wanted to test on my Gentoo system with
> kernel 2.6.25. The tarball creation works, but applying the change brings an
> error to syslog:
>
> ksplice_242e9pfc_helper: Unknown symbol init_mm
>
> Any chance to get a fix for that?
> --
> Regards,
> Chris
>

Re: A system for rebootless kernel security updates

[Christian Hesse]

On Thursday 24 April 2008, Jeff Arnold wrote:
> > I just wanted to test on my Gentoo system with kernel 2.6.25. The
> > tarball creation works, but applying the change brings an error to
> > syslog
>
> Sorry.  (In 2.6.25, init_mm is marked as an unused symbol, and so whether
> it is exported depends on whether CONFIG_UNUSED_SYMBOLS is enabled).
>
> I've committed a change which should fix this problem.  You can get get
> the new version (0.8.1) from http://web.mit.edu/ksplice/dist/ or from Git.
> (I haven't yet made the new version the default, so you won't receive it
> from the main Ksplice page).

This time I get the following in syslog:

ksplice_h: Preparing and checking ksplice_oq7ne7l6
ksplice: Aborted. (stack check: to-be-replaced code is busy)

I tried the example from your website (modifying printk).
-- 
Regards,
Chris

Re: A system for rebootless kernel security updates

[Jeff Arnold]

> On Thu, 24 Apr 2008, Christian Hesse wrote:
> This time I get the following in syslog:

This problem has now been resolved (a broken wireless card apparently left 
a kernel thread in an unusual state, which led to Ksplice aborting the 
upgrade).  Christian now has Ksplice working on his system.

If anyone else tries the software and encounters trouble, please let me 
know.

Thanks,

Jeff Arnold
jbarnold@mit.edu