[patch] Add basic sanity checks to the syscall execution patch

[patch] Add basic sanity checks to the syscall execution patch

[Arjan van de Ven]

Add basic sanity checks to the syscall execution patch

Several pieces of malware (rootkits etc) have the nasty habbit
of putting their own pointers into the syscall table.
For example, the recently "hot in the news" phalanx rootkit does this.

The patch below, while obviously not perfect protection against malware,
adds some cheap sanity checks to the syscall path to verify the
system call is actually still in the kernel code region and not some
external-to-this region such as a rootkit.

The overhead is very minimal; measured at 2 cycles or less.
(this is because the branches get predicted right and the rest of the
code is almost perfectly parallelizable... and an indirect function call
is a branch issue anyway)

with eyes-on-the-code help from Peter
the idea is from Ben Herrenschmidt 

Signed-off-by: Arjan van de Ven

diff --git a/arch/x86/kernel/entry_32.S b/arch/x86/kernel/entry_32.S
index 109792b..f25c0a1 100644
--- a/arch/x86/kernel/entry_32.S
+++ b/arch/x86/kernel/entry_32.S
@@ -347,7 +347,12 @@ sysenter_past_esp:
 sysenter_do_call:
 	cmpl $(nr_syscalls), %eax
 	jae syscall_badsys
-	call *sys_call_table(,%eax,4)
+	mov sys_call_table(,%eax,4), %eax
+	cmp $_stext, %eax
+	jb syscall_badsys
+	cmp $_etext, %eax
+	jae syscall_badsys
+	call *%eax
 	movl %eax,PT_EAX(%esp)
 	LOCKDEP_SYS_EXIT
 	DISABLE_INTERRUPTS(CLBR_ANY)
@@ -426,7 +431,12 @@ ENTRY(system_call)
 	cmpl $(nr_syscalls), %eax
 	jae syscall_badsys
 syscall_call:
-	call *sys_call_table(,%eax,4)
+	mov sys_call_table(,%eax,4), %eax
+	cmp $_stext, %eax
+	jb syscall_badsys
+	cmp $_etext, %eax
+	jae syscall_badsys
+	call *%eax
 	movl %eax,PT_EAX(%esp)		# store the return value
 syscall_exit:
 	LOCKDEP_SYS_EXIT
diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S
index 89434d4..be42486 100644
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -360,8 +360,13 @@ ENTRY(system_call_after_swapgs)
 system_call_fastpath:
 	cmpq $__NR_syscall_max,%rax
 	ja badsys
+	mov sys_call_table(,%rax,8), %rax
+	cmpq $_stext, %rax
+	jb badsys	
+	cmpq $_etext, %rax
+	jae badsys	
 	movq %r10,%rcx
-	call *sys_call_table(,%rax,8)  # XXX:	 rip relative
+	call *%rax  # XXX:	 rip relative
 	movq %rax,RAX-ARGOFFSET(%rsp)
 /*
  * Syscall return path ending with SYSRET (fast path)

Re: [patch] Add basic sanity checks to the syscall execution patch

[Andi Kleen]

Arjan van de Ven <arjan@infradead.org> writes:

> Add basic sanity checks to the syscall execution patch

This just means that the root kits will switch to patch
the first instruction of the entry points instead.

So the protection will be zero to minimal, but the overhead will
be there forever.

Now that I said this I expect it to go in yesterday.

-Andi

Re: [patch] Add basic sanity checks to the syscall execution patch

[Alan Cox]

On Thu, 04 Sep 2008 14:01:46 +0200
Andi Kleen <andi@firstfloor.org> wrote:

> Arjan van de Ven <arjan@infradead.org> writes:
> 
> > Add basic sanity checks to the syscall execution patch
> 
> This just means that the root kits will switch to patch
> the first instruction of the entry points instead.
> 
> So the protection will be zero to minimal, but the overhead will
> be there forever.

Agreed entirely. This is a waste of time and a game not worth playing.
The only place you can expect to make a difference here is in virtualised
environments by teaching KVM how to provide 'irrevocably read only' pages
to guests where the guest OS isn't permitted to change the rights back or
the virtual mapping of that page.

Alan

Re: [patch] Add basic sanity checks to the syscall execution patch

[Arjan van de Ven]

On Thu, 04 Sep 2008 14:01:46 +0200
Andi Kleen <andi@firstfloor.org> wrote:

> Arjan van de Ven <arjan@infradead.org> writes:
> 
> > Add basic sanity checks to the syscall execution patch
> 
> This just means that the root kits will switch to patch
> the first instruction of the entry points instead.
> 
> So the protection will be zero to minimal, but the overhead will
> be there forever.
> 
> Now that I said this I expect it to go in yesterday.
>

I'd have considered taking your email serious if you had left out the
uncalled and unneeded sarcasm line at the end.



-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [patch] Add basic sanity checks to the syscall execution patch

[Andi Kleen]

On Thu, Sep 04, 2008 at 01:34:19PM +0100, Alan Cox wrote:
> On Thu, 04 Sep 2008 14:01:46 +0200
> Andi Kleen <andi@firstfloor.org> wrote:
> 
> > Arjan van de Ven <arjan@infradead.org> writes:
> > 
> > > Add basic sanity checks to the syscall execution patch
> > 
> > This just means that the root kits will switch to patch
> > the first instruction of the entry points instead.
> > 
> > So the protection will be zero to minimal, but the overhead will
> > be there forever.
> 
> Agreed entirely. This is a waste of time and a game not worth playing.
> The only place you can expect to make a difference here is in virtualised

Even that can be circumvented by patching indirect pointers (or pointer 
to objects with indirect pointers) in any writable object. Or in
a couple of other ways.

But yes it would still seem like a reasonable useful improvement.

-Andi

-- 
ak@linux.intel.com

Re: [patch] Add basic sanity checks to the syscall execution patch

[pageexec]

On 4 Sep 2008 at 5:44, Arjan van de Ven wrote:

> On Thu, 04 Sep 2008 14:01:46 +0200
> Andi Kleen <andi@firstfloor.org> wrote:
> 
> > Arjan van de Ven <arjan@infradead.org> writes:
> > 
> > > Add basic sanity checks to the syscall execution patch
> > 
> > This just means that the root kits will switch to patch
> > the first instruction of the entry points instead.
> > 
> > So the protection will be zero to minimal, but the overhead will
> > be there forever.
> > 
> > Now that I said this I expect it to go in yesterday.
> >
> 
> I'd have considered taking your email serious if you had left out the
> uncalled and unneeded sarcasm line at the end.

consider how your whole patch is based on one big self-contradiction.
you already assume that the attacker *can* modify arbitrary kernel memory
(even the otherwise *read-only* syscall table at that), but at the very
same time you're saying he *can't* use the same powers to patch out your
'protection' or do many other things to evade it. as it is, it's cargo cult
security at its best, reminding one on the Vista kernel's similar 'protection' 
mechanism for the service descriptor tables...

Re: [patch] Add basic sanity checks to the syscall execution patch

[Benjamin Herrenschmidt]

On Fri, 2008-09-05 at 11:43 +0200, pageexec@freemail.hu wrote:
> > I'd have considered taking your email serious if you had left out the
> > uncalled and unneeded sarcasm line at the end.
> 
> consider how your whole patch is based on one big self-contradiction.
> you already assume that the attacker *can* modify arbitrary kernel memory
> (even the otherwise *read-only* syscall table at that), but at the very
> same time you're saying he *can't* use the same powers to patch out your
> 'protection' or do many other things to evade it. as it is, it's cargo cult
> security at its best, reminding one on the Vista kernel's similar 'protection' 
> mechanism for the service descriptor tables...

Well, I see it a different way ... it will once for all screw up
binary modules that try to add syscalls :-)

Ben.

Re: [patch] Add basic sanity checks to the syscall execution patch

[pageexec]

On 5 Sep 2008 at 20:14, Benjamin Herrenschmidt wrote:

> On Fri, 2008-09-05 at 11:43 +0200, pageexec@freemail.hu wrote:
> > > I'd have considered taking your email serious if you had left out the
> > > uncalled and unneeded sarcasm line at the end.
> > 
> > consider how your whole patch is based on one big self-contradiction.
> > you already assume that the attacker *can* modify arbitrary kernel memory
> > (even the otherwise *read-only* syscall table at that), but at the very
> > same time you're saying he *can't* use the same powers to patch out your
> > 'protection' or do many other things to evade it. as it is, it's cargo cult
> > security at its best, reminding one on the Vista kernel's similar 'protection' 
> > mechanism for the service descriptor tables...
> 
> Well, I see it a different way ... it will once for all screw up
> binary modules that try to add syscalls :-)

and that'd be because at the same time they patch the syscall table (remember,
they already have to go to length to get around the read-only pages), they
can't also patch this 'protection'? sounds really plausible, right :).

[fixed hpa's address, .org bounces.]

Re: [patch] Add basic sanity checks to the syscall execution patch

[Benjamin Herrenschmidt]

> and that'd be because at the same time they patch the syscall table (remember,
> they already have to go to length to get around the read-only pages), they
> can't also patch this 'protection'? sounds really plausible, right :).
> 
> [fixed hpa's address, .org bounces.]

Sure, they can :-)

It's just an idea I had on irc but I tend to agree that it wouldn't have
much effect in practice... regarding security, it will break some
existing rootkits ... until updated ones show up.

Cheers,
Ben.

Re: [patch] Add basic sanity checks to the syscall execution patch

[Ingo Molnar]

* Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> 
> > and that'd be because at the same time they patch the syscall table (remember,
> > they already have to go to length to get around the read-only pages), they
> > can't also patch this 'protection'? sounds really plausible, right :).
> > 
> > [fixed hpa's address, .org bounces.]
> 
> Sure, they can :-)
> 
> It's just an idea I had on irc but I tend to agree that it wouldn't 
> have much effect in practice... regarding security, it will break some 
> existing rootkits ... until updated ones show up.

at which point we are left with a change that has no relevance to 
updated rootkits (they circumvent it just fine), while the kernel 
syscall entry path is left with 2 cycles (or more) overhead, forever.

Not a good deal.

We introduced the read-only syscall table because it has debugging and 
robustness advantages, with near zero cost. This change is not zero cost 
- it's ~1% of our null syscall latency. (which is ~100 nsecs, the cost 
of this check is ~1 nsec)

The other, more fundamental problem that nobody has mentioned so far is 
that the check returns -ENOSYS and thus makes rootkit attacks _more 
robust_ and hence more likely!

The far better solution would be to insert uncertainty into the picture: 
some sort of low-frequency watchdog [runs once a second or so] that 
tries to hide itself from the general kernel scope as much as possible, 
perhaps as ELF-PIC code at some randomized location, triggered by some 
frequently used and opaque kernel facility that an attacker can not 
afford to block or fully filter, and which would just check integrity 
periodically and with little cost.

When it finds a problem it immediately triggers a hard to block/filter 
vector of alert (which can be a silent alarm over the network or to the 
screen as well).

that method does not prevent rootkits in general (nothing can), but sure 
makes their life more risky in practice - and a guaranteed livelihood 
and risk reduction is what typical criminals are interested in 
primarily, not whether they can break into a particular house.

If we implement it then it should not be present in distro .config's, 
etc. - it should be as invisible as possible - perhaps only be part of 
the kernel image .init.data section in some unremarkably generic manner. 

[ It would be nice to have a 'randomize instruction scheduling' option 
  for gcc, to make automated attacks that recognize specific instruction 
  patterns less reliable. ]

A good benchmark for such a silent alarm facility would be whether an 
experienced kernel developer could reliably tell it via a kgdb session 
and full access to memory and system symbols that such a silent alarm is 
running on a box. If he cannot do it reliably then there's probably no 
good way for an attacker either.

And of course all the other layers of security play a bigger role: an 
attacker should not get to (native) kernel level access to begin with.

	Ingo

Re: [patch] Add basic sanity checks to the syscall execution patch

[pageexec]

On 5 Sep 2008 at 13:42, Ingo Molnar wrote:
> The other, more fundamental problem that nobody has mentioned so far is 
> that the check returns -ENOSYS and thus makes rootkit attacks _more 
> robust_ and hence more likely!
> 
> The far better solution would be to insert uncertainty into the picture: 
> some sort of low-frequency watchdog [runs once a second or so] that 
> tries to hide itself from the general kernel scope as much as possible, 
> perhaps as ELF-PIC code at some randomized location, triggered by some 
> frequently used and opaque kernel facility that an attacker can not 
> afford to block or fully filter, and which would just check integrity 
> periodically and with little cost.

there's that adage about history being repeated by those not knowing it ;)
for details see the series based around bypassing Vista's PatchGuard at:

  http://uninformed.org/?v=3
  http://uninformed.org/?v=6
  http://uninformed.org/?v=8

> A good benchmark for such a silent alarm facility would be whether an 
> experienced kernel developer could reliably tell it via a kgdb session 
> and full access to memory and system symbols that such a silent alarm is 
> running on a box. If he cannot do it reliably then there's probably no 
> good way for an attacker either.

i believe the above mentioned papers prove that it's not a good benchmark ;)

Re: [patch] Add basic sanity checks to the syscall execution patch

[Andi Kleen]

First as a minor pedantic correction (sorry!): the ro syscall table is not 
fully free.  It means you cannot use 2MB pages anymore to map it, which costs
you in TLB misses.

> [ It would be nice to have a 'randomize instruction scheduling' option 
>   for gcc, to make automated attacks that recognize specific instruction 
>   patterns less reliable. ]

One way to do that today is to feed gcc random data for profile feedback.
But your performance will probably suffer.

> experienced kernel developer could reliably tell it via a kgdb session 
> and full access to memory and system symbols that such a silent alarm is 
> running on a box. If he cannot do it reliably then there's probably no 
> good way for an attacker either.

Game copy protections have been playing similar games for decades. While
I'm sure it was endless fun for both sides afaik the crackers tended to 
ultimatively win. And all of these things also make the kernel more 
fragile which is not good. Likely a case of "the only way to win is not to play"

I liked Alan's proposal of using hypervisor support for truly ro pages,  
although even that is not fully hole proof because of indirect pointers. 
But at least it would make it generally harder to inject code.

-Andi

-- 
ak@linux.intel.com

Re: [patch] Add basic sanity checks to the syscall execution patch

[Ingo Molnar]

* pageexec@freemail.hu <pageexec@freemail.hu> wrote:

> On 5 Sep 2008 at 13:42, Ingo Molnar wrote:
> > The other, more fundamental problem that nobody has mentioned so far is 
> > that the check returns -ENOSYS and thus makes rootkit attacks _more 
> > robust_ and hence more likely!
> > 
> > The far better solution would be to insert uncertainty into the 
> > picture: some sort of low-frequency watchdog [runs once a second or 
> > so] that tries to hide itself from the general kernel scope as much 
> > as possible, perhaps as ELF-PIC code at some randomized location, 
> > triggered by some frequently used and opaque kernel facility that an 
> > attacker can not afford to block or fully filter, and which would 
> > just check integrity periodically and with little cost.
> 
> there's that adage about history being repeated by those not knowing it ;)
> for details see the series based around bypassing Vista's PatchGuard at:
> 
>   http://uninformed.org/?v=3
>   http://uninformed.org/?v=6
>   http://uninformed.org/?v=8

i think Linux is fundamentally different here as we have the source 
code, and could apply the randomization technique i mentioned:

> > [ It would be nice to have a 'randomize instruction scheduling' 
> >   option for gcc, to make automated attacks that recognize specific 
> >   instruction patterns less reliable. ]

and every box where it matters we could have a _per box_ randomized 
kernel image in essence, with non-essential symbols thrown away, and 
with a few checks inserted in random locations - inlined and in essence 
unrecognizable from the general entropy of randomization.

Not that a randomizing compiler which inserts true, hard to eliminate 
entropy would be easy to implement. But once done, the cat and mouse 
game is over and the needle is hidden in the hay-stack. At least as long 
as transparent rootkits are involved.

a successful attack that wants to disable the checks reliably would have 
to patch the IDT and would have to emulate full kernel execution and 
would have to detect the pattern of an alert on the hardware API level - 
as that would be the only reliably observable output of the system. 
Besides being impractical at best, at minimum a huge slow-down would 
occur.

the only other option would be for a rootkit to transparently switch to 
another, new, non-checked kernel image on the fly, while keeping all 
user-space context safe. That's a feature Linux would like to have 
anyway ;-) [and this could be made really difficult as well if gcc 
inserted a modest amount of per kernel random noise in the layout of all 
data structures / field offsets.]

	Ingo

Re: [patch] Add basic sanity checks to the syscall execution patch

[Arjan van de Ven]

On Fri, 05 Sep 2008 11:43:31 +0200
pageexec@freemail.hu wrote:

> consider how your whole patch is based on one big self-contradiction.
> you already assume that the attacker *can* modify arbitrary kernel
> memory (even the otherwise *read-only* syscall table at that), but at
> the very same time you're saying he *can't* use the same powers to
> patch out your 'protection' or do many other things to evade it. as
> it is, it's cargo cult security at its best, reminding one on the
> Vista kernel's similar 'protection' mechanism for the service
> descriptor tables...


so I'm not going to say that the patch is important or good;
it's the result of ben mentioning the idea on irc and me thinking "sure
lets see what it would take and cost".
Nothing more than that


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [patch] Add basic sanity checks to the syscall execution patch

[pageexec]

On 5 Sep 2008 at 17:42, Ingo Molnar wrote:

> * pageexec@freemail.hu <pageexec@freemail.hu> wrote:
> 
> > there's that adage about history being repeated by those not knowing it ;)
> > for details see the series based around bypassing Vista's PatchGuard at:
> > 
> >   http://uninformed.org/?v=3
> >   http://uninformed.org/?v=6
> >   http://uninformed.org/?v=8
> 
> i think Linux is fundamentally different here as we have the source 
> code, and could apply the randomization technique i mentioned:

how's that supposed to work for the binary distros, i.e., the majority of
end users? and who would look at all the bugreports from such kernels?

> > > [ It would be nice to have a 'randomize instruction scheduling' 
> > >   option for gcc, to make automated attacks that recognize specific 
> > >   instruction patterns less reliable. ]
> 
> and every box where it matters we could have a _per box_ randomized 
> kernel image in essence, with non-essential symbols thrown away, and 
> with a few checks inserted in random locations - inlined and in essence 
> unrecognizable from the general entropy of randomization.
> 
> Not that a randomizing compiler which inserts true, hard to eliminate 
> entropy would be easy to implement. But once done, the cat and mouse 
> game is over and the needle is hidden in the hay-stack. At least as long 
> as transparent rootkits are involved.
> 
> a successful attack that wants to disable the checks

why do you assume that an attacker wants to do that? it's equally possible,
and there's even academic research on this in addition to the underground
cracking scene, that one simply hides the modifications from the checker.

from marking your patched code as unreadable to executing it from a different
place than what the checker checks, there're many ways to trick such checkers.
as far as reality goes, it's never been game over ;).

Re: [patch] Add basic sanity checks to the syscall execution patch

[Ingo Molnar]

* pageexec@freemail.hu <pageexec@freemail.hu> wrote:

> > i think Linux is fundamentally different here as we have the source 
> > code, and could apply the randomization technique i mentioned:
> 
> how's that supposed to work for the binary distros, i.e., the majority 
> of end users? [...]

it takes less than 10 minutes to build a full kernel on recent hardware. 
Can be done in the background after install or so.

> [...] and who would look at all the bugreports from such kernels?

yes, in this area debuggability is in straight conflict. Since we can 
assume that both attacker and owner has about the same level of access 
to the system, making the kernel less accessible to an attacker makes it 
less accessible/debuggable to the owner as well.

> > > > [ It would be nice to have a 'randomize instruction scheduling' 
> > > >   option for gcc, to make automated attacks that recognize specific 
> > > >   instruction patterns less reliable. ]
> > 
> > and every box where it matters we could have a _per box_ randomized 
> > kernel image in essence, with non-essential symbols thrown away, and 
> > with a few checks inserted in random locations - inlined and in essence 
> > unrecognizable from the general entropy of randomization.
> > 
> > Not that a randomizing compiler which inserts true, hard to eliminate 
> > entropy would be easy to implement. But once done, the cat and mouse 
> > game is over and the needle is hidden in the hay-stack. At least as long 
> > as transparent rootkits are involved.
> > 
> > a successful attack that wants to disable the checks
> 
> why do you assume that an attacker wants to do that? it's equally 
> possible, and there's even academic research on this in addition to 
> the underground cracking scene, that one simply hides the 
> modifications from the checker.
> 
> from marking your patched code as unreadable to executing it from a 
> different place than what the checker checks, there're many ways to 
> trick such checkers. as far as reality goes, it's never been game over 
> ;).

well at least in the case of Linux we have a fairly good tally of what 
kernel code is supposed to be executable at some given moment after 
bootup, and can lock that list down permanently until the next reboot, 
and give the list to the checker to verify every now and then? Such a 
verification pass certainly wouldnt be cheap though: all kernel 
pagetables have to be scanned and verified, plus all known code (a few 
megabytes typically), and the key CPU data structures.

	Ingo

Re: [patch] Add basic sanity checks to the syscall execution patch

[Andi Kleen]

First such checkers already exist -- they are called root kit checkers.
There are various around. 
Usually operate from user space. You could run them in a cron job.

> well at least in the case of Linux we have a fairly good tally of what 
> kernel code is supposed to be executable at some given moment after 
> bootup, and can lock that list down permanently until the next reboot, 
> and give the list to the checker to verify every now and then? Such a 

Doing it in a hypervisor implicitely like Alan proposed would seem much 
stronger and also somewhat cleaner than doing it delayed.

> verification pass certainly wouldnt be cheap though: all kernel 
> pagetables have to be scanned and verified, plus all known code (a few 
> megabytes typically), and the key CPU data structures.

The issue is that a lot of non key data structures all over the memory
have function pointers (or pointers to function pointers) too.
So if you protect syscall table they are just going to patch some dentry
instead. Still if it's reasonable clean it might be still useful to
raise the bar a bit, but I'm not sure a checker qualifies for that.

-Andi

Re: [patch] Add basic sanity checks to the syscall execution patch

[pageexec]

On 5 Sep 2008 at 18:52, Ingo Molnar wrote:

> * pageexec@freemail.hu <pageexec@freemail.hu> wrote:
> 
> > > i think Linux is fundamentally different here as we have the source 
> > > code, and could apply the randomization technique i mentioned:
> > 
> > how's that supposed to work for the binary distros, i.e., the majority 
> > of end users? [...]
> 
> it takes less than 10 minutes to build a full kernel on recent hardware. 

provided the end user wants/needs to have the whole toolchain on his boxes
at all. how many really do?

> Can be done in the background after install or so.

it's not only installation time (if you meant 'installing the box' itself),
but every time the kernel is updated, so the toolchain will be there forever.

> > [...] and who would look at all the bugreports from such kernels?
> 
> yes, in this area debuggability is in straight conflict. Since we can 
> assume that both attacker and owner has about the same level of access 
> to the system, making the kernel less accessible to an attacker makes it 
> less accessible/debuggable to the owner as well.

in other words, it's a permanently unsolved problem ;). somehow i don't see
Red Hat selling RHEL for production boxes with the tag 'we do not debug crashes
here because we cannot' attached.

> > > a successful attack that wants to disable the checks
> > 
> > why do you assume that an attacker wants to do that? it's equally 
> > possible, and there's even academic research on this in addition to 
> > the underground cracking scene, that one simply hides the 
> > modifications from the checker.
> > 
> > from marking your patched code as unreadable to executing it from a 
> > different place than what the checker checks, there're many ways to 
> > trick such checkers. as far as reality goes, it's never been game over 
> > ;).
> 
> well at least in the case of Linux we have a fairly good tally of what 
> kernel code is supposed to be executable at some given moment after 
> bootup, and can lock that list down permanently until the next reboot, 

so no module support? what about kprobes and/or whatever else that generates
code at runtime?

> and give the list to the checker to verify every now and then?

so good-bye to large page support for kernel code? else there's likely
enough unused space left in the large pages for a rootkit to hide.

what if the rootkit finds unused pieces of actual code and replaces
that (bound to happen with those generic distro configs, especially
if you have to go with a non-modular kernel)?

last but not least, how would that 'lock that list down' work exactly?
what would prevent a rootkit from locating and modifying it as well?

> Such a verification pass certainly wouldnt be cheap though: all kernel 
> pagetables have to be scanned and verified, plus all known code (a few 
> megabytes typically), and the key CPU data structures.

what would you verify on the code? it's obfuscated so you can't really
analyze it (else you've just solved the attacker's problem), all you can
do is probably compute hashes but then you'll have to take care of kernel
self-patching and also protecting the hashes somehow.

Re: [patch] Add basic sanity checks to the syscall execution patch

[pageexec]

On 5 Sep 2008 at 19:26, Andi Kleen wrote:

> 
> First such checkers already exist -- they are called root kit checkers.
> There are various around. 
> Usually operate from user space. You could run them in a cron job.

how trivial do you think it is for *kernel* code to evade *userland*
checking it? ;) otherwise agreed with rest.

Re: [patch] Add basic sanity checks to the syscall execution patch

[Willy Tarreau]

On Fri, Sep 05, 2008 at 01:42:33PM +0200, Ingo Molnar wrote:
> The far better solution would be to insert uncertainty into the picture: 

till there OK :-)

> some sort of low-frequency watchdog [runs once a second or so] that 
> tries to hide itself from the general kernel scope as much as possible, 
> perhaps as ELF-PIC code at some randomized location, triggered by some 
> frequently used and opaque kernel facility that an attacker can not 
> afford to block or fully filter, and which would just check integrity 
> periodically and with little cost.

"can not" above is the unrealistic requirement unfortunately.

> When it finds a problem it immediately triggers a hard to block/filter 
> vector of alert (which can be a silent alarm over the network or to the 
> screen as well).
> 
> that method does not prevent rootkits in general (nothing can), but sure 
> makes their life more risky in practice - and a guaranteed livelihood 
> and risk reduction is what typical criminals are interested in 
> primarily, not whether they can break into a particular house.
> 
> If we implement it then it should not be present in distro .config's, 
> etc. - it should be as invisible as possible - perhaps only be part of 
> the kernel image .init.data section in some unremarkably generic manner. 

Then they will simply proceed like this :
  - patch /boot/vmlinuz
  - sync
  - crash system

=> user says "oh crap" and presses the reset button. Patched kernel boots.
   Game over. Patching vmlinuz for known targetted distros is even easier
   because the attacker just has to embed binary changes for the most
   common distro kernels.

Clearly all this is a waste of developer time, CPU cycles, memory,
reliability and debugging time. All that time would be more efficiently
spent auditing and debugging existing code to reduce the attack surface,
and CPU cycles + memory would be better spent adding double checks to
most sensible functions' entry points and user data processing.

Regards,
Willy

Re: [patch] Add basic sanity checks to the syscall execution patch

[Andi Kleen]

On Fri, Sep 05, 2008 at 09:42:48PM +0200, pageexec@freemail.hu wrote:
> On 5 Sep 2008 at 19:26, Andi Kleen wrote:
> 
> > 
> > First such checkers already exist -- they are called root kit checkers.
> > There are various around. 
> > Usually operate from user space. You could run them in a cron job.
> 
> how trivial do you think it is for *kernel* code to evade *userland*
> checking it? ;) otherwise agreed with rest.

It depends on where the userland runs. e.g. if it's under a hypervisor
and in a separate domain it should be reasonably safe.

And then I don't think it is much difference between Ingo's kernel
checker and a user land checker. Both can be disabled it you know
about them.

-Andi

-- 
ak@linux.intel.com

Re: [patch] Add basic sanity checks to the syscall execution patch

[Ingo Molnar]

* pageexec@freemail.hu <pageexec@freemail.hu> wrote:

> On 5 Sep 2008 at 18:52, Ingo Molnar wrote:
> 
> > * pageexec@freemail.hu <pageexec@freemail.hu> wrote:
> > 
> > > > i think Linux is fundamentally different here as we have the source 
> > > > code, and could apply the randomization technique i mentioned:
> > > 
> > > how's that supposed to work for the binary distros, i.e., the majority 
> > > of end users? [...]
> > 
> > it takes less than 10 minutes to build a full kernel on recent hardware. 
> 
> provided the end user wants/needs to have the whole toolchain on his 
> boxes at all. how many really do?

it's minimal and easy. It really works to operate on the source code - 
this 'open source' thing ;-) We just still tend to think in terms of 
binary software practices that have been established in the past few 
decades.

> > Can be done in the background after install or so.
> 
> it's not only installation time (if you meant 'installing the box' 
> itself), but every time the kernel is updated, so the toolchain will 
> be there forever.

not a problem really, it is rather small compared to all the stuff that 
is in a typical disto install. I like the fundamental message as well: 
"If you want to be more secure, you've got to have the source code, and 
you've got to be able to build it."

> > > [...] and who would look at all the bugreports from such kernels?
> > 
> > yes, in this area debuggability is in straight conflict. Since we 
> > can assume that both attacker and owner has about the same level of 
> > access to the system, making the kernel less accessible to an 
> > attacker makes it less accessible/debuggable to the owner as well.
> 
> in other words, it's a permanently unsolved problem ;). somehow i 
> don't see Red Hat selling RHEL for production boxes with the tag 'we 
> do not debug crashes here because we cannot' attached.

it's not an unsolvable problem. The debug info can be on a separate box, 
encrypted, etc. etc - depending on your level of paranoia. The need to 
debug kernel crashes is a relatively rare event - especially on a box 
that has such high security constraints, fortunately :-)

> > well at least in the case of Linux we have a fairly good tally of 
> > what kernel code is supposed to be executable at some given moment 
> > after bootup, and can lock that list down permanently until the next 
> > reboot,
> 
> so no module support? [...]

why no module support? Once the system has booted up all necessary 
modules are loaded and the ability to load new ones is locked down as 
well. This also makes it harder to inject rootkits btw. (combined with 
signed modules - patches exist for that)

> [...] what about kprobes and/or whatever else that generates code at 
> runtime?

you dont need that in general on a perimeter box. If you need it, you 
open that locked box with the debug info and make the system more 
patchable/debuggable - at the risk of exposing same information to 
attackers (were they gain the same level of access).

> > and give the list to the checker to verify every now and then?
> 
> so good-bye to large page support for kernel code? else there's likely 
> enough unused space left in the large pages for a rootkit to hide.
> 
> what if the rootkit finds unused pieces of actual code and replaces 
> that (bound to happen with those generic distro configs, especially if 
> you have to go with a non-modular kernel)?

are you now talking about the randomized kernel image? The whole point 
why i proposed it was to hide the checking functionality in it, not to 
make it harder for the attacker to place the rootkit.

Once the identity of the checking code is randomized reasonably, we can 
assume it will run every now and then, and would expose any 
modifications of 'unused' kernel functions. (which the attacker would 
have to filter out of the randomized image to begin with)

> last but not least, how would that 'lock that list down' work exactly? 
> what would prevent a rootkit from locating and modifying it as well?

best would be hardware support for mark-read-only-permanently, but once 
the checker functionality is reasonably randomized, its data structure 
can be randomized as well.

> > Such a verification pass certainly wouldnt be cheap though: all 
> > kernel pagetables have to be scanned and verified, plus all known 
> > code (a few megabytes typically), and the key CPU data structures.
> 
> what would you verify on the code? it's obfuscated so you can't really 
> analyze it (else you've just solved the attacker's problem), all you 
> can do is probably compute hashes but then you'll have to take care of 
> kernel self-patching and also protecting the hashes somehow.

yes, hashes. The point would be to make the true characteristics of the 
checker a random, per system property. True, it has many disadvantages 
such as the inevitable slowdown from a randomized kernel image, the 
restrictions on debuggability, etc. - but it can serve its purpose if 
someone is willing to pay that price.

best (and most practical) tactics would still be to allow the kernel to 
be locked down, in terms of not allowing new (non-authorized) kernel 
code to be executed: signed modules and properly locked down debug APIs, 
so that the only vector of code insertion is a not yet fixed kernel 
space security hole.

	Ingo

Re: [patch] Add basic sanity checks to the syscall execution patch

[Ingo Molnar]

* Willy Tarreau <w@1wt.eu> wrote:

> Then they will simply proceed like this :
>   - patch /boot/vmlinuz
>   - sync
>   - crash system
> 
> => user says "oh crap" and presses the reset button. Patched kernel boots.
>    Game over. Patching vmlinuz for known targetted distros is even easier
>    because the attacker just has to embed binary changes for the most
>    common distro kernels.

a reboot often raises attention. But yes, in terms of end user boxes, 
probably not. Anyway, my points were about transparent rootkits 
installed on a running system without anyone noticing - obviously if the 
attacker can modify the kernel image and the user does not mind a reboot 
it's game over.

	Ingo

Re: [patch] Add basic sanity checks to the syscall execution patch

[Jeroen van Rijn]

> a reboot often raises attention. But yes, in terms of end user boxes,
> probably not. Anyway, my points were about transparent rootkits
> installed on a running system without anyone noticing - obviously if the
> attacker can modify the kernel image and the user does not mind a reboot
> it's game over.
>

Hi,

can't then, in this scenario, the VFS keep tabs on /boot/vmlinuz and
only allow modification when the process in question properly
authenticates itself. As long as we're talking signed modules, why not
lock certain files down as well?

e.g. hand the kernel a signed list of files to watch write access to,
and allow only after the process auths via a private key.

-- Jeroen.

n.b. I understand this would slow down things more, but if we're
talking about taking extreme measures...

Re: [patch] Add basic sanity checks to the syscall execution patch

[pageexec]

On 6 Sep 2008 at 17:42, Ingo Molnar wrote:
> * pageexec@freemail.hu <pageexec@freemail.hu> wrote:
> 
> > On 5 Sep 2008 at 18:52, Ingo Molnar wrote:
> > 
> > provided the end user wants/needs to have the whole toolchain on his 
> > boxes at all. how many really do?
> 
> it's minimal and easy. It really works to operate on the source code - 
> this 'open source' thing ;-) We just still tend to think in terms of 
> binary software practices that have been established in the past few 
> decades.

the question wasn't whether it was minimal or easy but whether end users
want to have the toolchain on their production boxes, especially on these
supposedly secure ones. industry wisdom says that they'd rather not.

> > > Can be done in the background after install or so.
> > 
> > it's not only installation time (if you meant 'installing the box' 
> > itself), but every time the kernel is updated, so the toolchain will 
> > be there forever.
> 
> not a problem really, it is rather small compared to all the stuff that 
> is in a typical disto install. I like the fundamental message as well: 
> "If you want to be more secure, you've got to have the source code, and 
> you've got to be able to build it."

the point is not the size of the toolchain, i don't think anyone cares
about that in the days of TB disks. the more fundamental issue is that
the toolchain doesn't normally belong to production boxes and if the
sole reason to have it is this kernel image randomization feature, then
it may not be as easy a sell as you think as there're better alternatives
that work without having the toolchain there.

> > > > [...] and who would look at all the bugreports from such kernels?
> > > 
> > > yes, in this area debuggability is in straight conflict. Since we 
> > > can assume that both attacker and owner has about the same level of 
> > > access to the system, making the kernel less accessible to an 
> > > attacker makes it less accessible/debuggable to the owner as well.
> > 
> > in other words, it's a permanently unsolved problem ;). somehow i 
> > don't see Red Hat selling RHEL for production boxes with the tag 'we 
> > do not debug crashes here because we cannot' attached.
> 
> it's not an unsolvable problem. The debug info can be on a separate box, 
> encrypted, etc. etc - depending on your level of paranoia.

what does having the debug info available in whatever form help you in
the debugging process that doesn't at the same time help an attacker?

remember, the assumption is that the attacker is already on the box (and
as root at that), trying to get his kernel rootkit to work, so you'll
have to come up with a debugging procedure where he can't leverage that
local acccess to pry the debug info out of your hands as you're trying
to diagnose a problem. e.g., you can't just disconnect the box from the
network if you need remote access yourself or reproducing the problem
does.

> The need to 
> debug kernel crashes is a relatively rare event - especially on a box 
> that has such high security constraints, fortunately :-)

how are the security constraints of the box related to its kernel's
susceptibility to crashes/oopes/etc?

> > > well at least in the case of Linux we have a fairly good tally of 
> > > what kernel code is supposed to be executable at some given moment 
> > > after bootup, and can lock that list down permanently until the next 
> > > reboot,
> > 
> > so no module support? [...]
> 
> why no module support? Once the system has booted up all necessary 
> modules are loaded and the ability to load new ones is locked down as 
> well. This also makes it harder to inject rootkits btw. (combined with 
> signed modules - patches exist for that)

and this also makes it impossible to load newer versions of modules,
which will now require a full reboot. i'm sure management will like the
idea ;).

> > [...] what about kprobes and/or whatever else that generates code at 
> > runtime?
> 
> you dont need that in general on a perimeter box. If you need it, you 
> open that locked box with the debug info and make the system more 
> patchable/debuggable - at the risk of exposing same information to 
> attackers (were they gain the same level of access).

so all an attacker needs to do is induce some kernel problems (due to
the underlying assumption, he can easily do that), wait for you guys
come in and have a field day with the debug info? ;)

> > > and give the list to the checker to verify every now and then?
> > 
> > so good-bye to large page support for kernel code? else there's likely 
> > enough unused space left in the large pages for a rootkit to hide.
> > 
> > what if the rootkit finds unused pieces of actual code and replaces 
> > that (bound to happen with those generic distro configs, especially if 
> > you have to go with a non-modular kernel)?
> 
> are you now talking about the randomized kernel image? The whole point 
> why i proposed it was to hide the checking functionality in it, not to 
> make it harder for the attacker to place the rootkit.

i was reflecting to your saying that:

> well at least in the case of Linux we have a fairly good tally of 
> what kernel code is supposed to be executable at some given moment 
> after bootup, and can lock that list down permanently until the next 
> reboot,

and was pointing out that you don't actually have such a good tally unless
you're willing to give up large page support for kernel code, and even if
you go for 4k pages you'll be in trouble because a generic kernel like
those used in distros is bound to have unused regions of code. and i base
this on the assumption that your randomization cannot fundamentally change
function boundaries (i.e., randomizing code placement at the basic block
level) without killing the branch predictor for good. the short of it is
that your list of 'kernel code pages' is useless without ensuring that the
attacker cannot place his code into those same kernel code pages.

> Once the identity of the checking code is randomized reasonably, we can 
> assume it will run every now and then, and would expose any 
> modifications of 'unused' kernel functions. (which the attacker would 
> have to filter out of the randomized image to begin with)

as i indicated at the beginning, you're assuming that the attacker will
try to disable the checking mechanism, whereas he can equally neutralize
it by hiding his modifications to the kernel from the checker. recent years
saw a few academic papers on creating & defeating self-checksumming code,
and from what i recall now, it didn't look too well for the defender side.

> > last but not least, how would that 'lock that list down' work exactly? 
> > what would prevent a rootkit from locating and modifying it as well?
> 
> best would be hardware support for mark-read-only-permanently, but once 
> the checker functionality is reasonably randomized, its data structure 
> can be randomized as well.

what does marking it read-only help when the attacker can just remap the
virtual addresses to some other page under his control? i.e., you want to
lock some TLB entries hard, something not possible on contemporary i386
and amd64. you can sort of simulate it with a hypervisor though but then
you don't need any of this randomization stuff. or in other words, if you
have such a hw capability to mark-read-only-permanently, you might as well
use it for the kernel code itself and not bother with all this 'rootkit
patches kernel' problem.

> > what would you verify on the code? it's obfuscated so you can't really 
> > analyze it (else you've just solved the attacker's problem), all you 
> > can do is probably compute hashes but then you'll have to take care of 
> > kernel self-patching and also protecting the hashes somehow.
> 
> yes, hashes. The point would be to make the true characteristics of the 
> checker a random, per system property. True, it has many disadvantages 
> such as the inevitable slowdown from a randomized kernel image, the 
> restrictions on debuggability, etc. - but it can serve its purpose if 
> someone is willing to pay that price.

the question for the end user is what he gets for that price and whether
he could get it or better for less. as it stands, your idea is both too
expensive and doesn't quite deliver yet, not an easy sell ;).

> best (and most practical) tactics would still be to allow the kernel to 
> be locked down, in terms of not allowing new (non-authorized) kernel 
> code to be executed: signed modules and properly locked down debug APIs, 
> so that the only vector of code insertion is a not yet fixed kernel 
> space security hole.

i thought the whole discussion was about 0-day ;), is there any attacker
that doesn't use that vector to get into the kernel? it's the most generic
method, no need to bother with silly kernel 'protection' features when
exploitable kernels bugs abound (ok, let's not get into that discussion
again ;P).

Re: [patch] Add basic sanity checks to the syscall execution patch

[Pavel Machek]

On Sat 2008-09-06 17:45:51, Ingo Molnar wrote:
> 
> * Willy Tarreau <w@1wt.eu> wrote:
> 
> > Then they will simply proceed like this :
> >   - patch /boot/vmlinuz
> >   - sync
> >   - crash system
> > 
> > => user says "oh crap" and presses the reset button. Patched kernel boots.
> >    Game over. Patching vmlinuz for known targetted distros is even easier
> >    because the attacker just has to embed binary changes for the most
> >    common distro kernels.
> 
> a reboot often raises attention. But yes, in terms of end user boxes, 
> probably not. Anyway, my points were about transparent rootkits 
> installed on a running system without anyone noticing - obviously if the 
> attacker can modify the kernel image and the user does not mind a reboot 
> it's game over.

Well, install a rootkit in /boot/vmlinuz, sync, then wait for user to
reboot its system?

Even well-kept servers are rebooted from time to time.

I agree -- the only way to win is not to play this game.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html