new module to check constant memory for corruption

new module to check constant memory for corruption

[Alexander.Kleinsorge]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

ramcheck kernel module
new module to check constant memory for corruption

detect corruption of constant kernel memory (text and data) periodically.
runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
which is distributed over 8 (BLOCKS) time partitions (less than half ms per sec).
in case of checksum (xor) error, an kernel log is posted.
manual trigger via /proc/ramcheck is possible.
range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")

this new module helps against two scenarios:
1) bit flip on system without crc (typical pc)

more often than you think:
DRAM Errors in the Wild: A Large-Scale Field Study, Mai 2009.
http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf

2) architectures without memory protection plus attack or out-of-bound-writes.
x86 and x64 using it (write protection), but missing (or not enabled) on most other arch.

see: ramcheck.tgz

[-- Attachment #2: ramcheck.tgz --]
[-- Type: application/octet-stream, Size: 4206 bytes --]

[-- Attachment #3: ramcheck.c --]
[-- Type: text/plain, Size: 9614 bytes --]

/* ramcheck.c - checks predefined kernel memory region
*
* KernelRamCheck by
*
* Copyright 2014 Alexander Kleinsorge <alexander[dot]kleinsorge[at]gmx[dot]de>
* Copyright 2014 Benjamin Schroedl <benjamin[at]dev-tec[dot]de>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
*
*/

// Defining __KERNEL__ and MODULE allows us to access kernel-level code not usually available to userspace programs.
/*
#undef __KERNEL__
#define __KERNEL__

#undef MODULE
#define MODULE
*/
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kallsyms.h>
#include <linux/kernel.h>
#include <linux/pfn.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/timer.h>
#include <linux/delay.h>
#include <crypto/algapi.h>
#include <asm/uaccess.h>
#include <asm/sections.h>
#include <linux/time.h>


#define DRIVER_AUTHOR    "Alexander Kleinsorge <alexander[dot]kleinsorge[at]gmx[dot]de> and Benjamin Schroedl <benjamin[at]dev-tec[dot]de>"
#define DRIVER_DESC      "checks predefined memory kernel region"
#define proc_fs_name     "ramcheck"
#define MODNAME proc_fs_name" "

#define CHECK_TIMER_MS   1000
#define BLOCK ((sizeof(long) <= 4) ? 1u << 20 : 2u << 20) /* 64bit CPU is often faster (bigger blocks at once) */
#define BLOCKS 8

MODULE_LICENSE("GPL");
MODULE_AUTHOR(DRIVER_AUTHOR);
MODULE_DESCRIPTION(DRIVER_DESC);


static void ram_check_exit(void);
static int  ram_check_init(void);
static int  ram_check_print(struct seq_file *m, void *v);
static int  ram_check_run(void);
void        ram_check_timer_callback(unsigned long data);
static int  ram_check_open(struct inode *inode, struct file *file);

static struct timer_list ram_check_timer;
struct proc_dir_entry *ram_check_proc_file;
struct file_operations proc_fops = {
	.open     = ram_check_open,
	.read     = seq_read,
	.llseek   = seq_lseek,
	.release  = single_release,
};


/* Global Variables */

static unsigned long g_RangeSize = 0;
static unsigned long g_RangeStart = 0;

static unsigned int g_TempCount = 0;
static unsigned int g_Timers = 0;
static unsigned int g_TimerActive = 0;

static unsigned long g_SumFirst = 0;
static unsigned long g_SumLast = 0;

static unsigned long g_BlockSum[BLOCKS];


/* Functions */

/* simple checksum/hash, xor is fastest (~1-6 MB/ms) */
static unsigned long calcChecksumXor(const void* ptr, const unsigned long bytes)
{
	unsigned long ret, i;
	const unsigned long long* ptr64 = (const unsigned long long*) ptr;
	const unsigned long len = bytes / sizeof(*ptr64);
	unsigned long long sum64 = 0u;

	for (i=0; i < len; i++) {
		sum64 ^= ptr64[i];
	}

	/* return 32 or 64 bit possible */
	ret = ((unsigned long) sum64) ^ ((unsigned long) (sum64>>32));

	return ret;
}

/* merge 4 to 1 byte ==> less info (security reason) */
static unsigned char shrink2byte(unsigned long u)
{
	unsigned long ret = (u) ^ (u>>8) ^ (u>>16) ^ (u>>24);
	/* (lowest) 32 bit --> 8 bit */
	return (unsigned char) ret;
}

static unsigned long long getTimeUS(void)
{
	struct timespec ts;
	ktime_get_ts( &ts );

	return (ts.tv_sec*USEC_PER_SEC) + (ts.tv_nsec/1000u);
}

// check for const range (in case of strange checksum)
static int check_range(void)
{
	unsigned long start = kallsyms_lookup_name("_text");
	unsigned long size  = kallsyms_lookup_name("__end_rodata") - start;
	int ret = 0;

	if (g_RangeSize && ((start != g_RangeStart) || (size != g_RangeSize))) {
		printk(KERN_WARNING MODNAME "error: const kernel memory address changed (%08lx + %lu K), stop!\n", start, size/1024u);
		g_RangeSize = 0; /* stop working */
		ret = -1;
	}

	return ret;
}

/* partitions to limit runtime to 1 ms (1..6 MB) */
static unsigned long check_block(unsigned long blk, unsigned long lastblk)
{
	const unsigned long offset = blk * BLOCK;
	unsigned long start = g_RangeStart + offset;
	unsigned long size = BLOCK;
	unsigned long ret;

	if (blk >= lastblk) {
		size = g_RangeSize % BLOCK;
		if (blk > lastblk) {
			return (~0u); /* should never happen */
		}
	}
	ret = calcChecksumXor((const void *) start, size);

	return ret;
}

static int ram_check_init(void)
{
	int ret;
	unsigned long start, size, end;
	unsigned long b, t, bps;

	for (b=0; b<BLOCKS; b++) {
		g_BlockSum[b] = 0u;
	}

	ram_check_proc_file = proc_create(proc_fs_name, 0, NULL, &proc_fops);

	if (ram_check_proc_file == NULL) {
		remove_proc_entry(proc_fs_name, NULL);
		printk(KERN_ALERT MODNAME "error: Could not initialize /proc/%s\n", proc_fs_name);

		return -ENOMEM;
	}

	start = kallsyms_lookup_name("_text");
	end   = kallsyms_lookup_name("__end_rodata");
	/* kallsyms_lookup_name("__end_rodata_hpage_align"); ==> Bad Address! */
	size  = (end - start);

	/* error handling on illegal size and ptr or size to big */
	if ((start == 0) || (end == 0) || (end <= start) || (size > (1000u << 20))) {
		printk(KERN_ALERT MODNAME "error: bad start or end adress [%lx + %lx]\n", start, size);

		return -EFAULT;
	}
	if ((size < (1u << 20)) || (size > (200u << 20))) {
		printk(KERN_INFO MODNAME "warning: strange const size = %lu MB\n", size>>20);
	}

	g_RangeStart = start;
	g_RangeSize  = size;

	t = (unsigned long) getTimeUS();
	for (b = 0; b <= size/BLOCK; b++) {
		g_BlockSum[b % BLOCKS] ^= check_block(b, size/BLOCK);
	}
	t = (unsigned long) getTimeUS() - t; /* ca. 1k..7k Byte/us */
	for (b = 0; b < BLOCKS; b++) {
		g_SumFirst ^= g_BlockSum[b]; /* all together */
	}

	bps = (t > 0) ? (size / t) : 0;
	printk(KERN_INFO MODNAME "init: %lu B/us, %lu kB, %d bit, (0x%02x, %u)\n",
		bps, size>>10, (int)sizeof(g_SumFirst)*8, shrink2byte(g_SumFirst), BLOCKS);

	/* TIMER init */
	setup_timer(&ram_check_timer, ram_check_timer_callback, 0);

	ret = mod_timer(&ram_check_timer, jiffies + msecs_to_jiffies(CHECK_TIMER_MS));
	if (ret) {
		printk(KERN_ALERT MODNAME "error: Failure in mod_timer\n");

		return -EFAULT;
	}
	g_TimerActive = 1;

	return 0;
}


static void ram_check_exit(void)
{
	int ret;
	g_RangeSize = 0; /* stop work */

	remove_proc_entry(proc_fs_name, NULL);
	check_range();

	for (ret = 0; (g_TimerActive > 0) && (ret < 1000/20); ret ++) {
		mdelay(20);
	}

	ret = del_timer_sync(&ram_check_timer);

	if (ret) {
		printk(KERN_WARNING MODNAME "warning: timer still in use on exit...(%d,%u)\n", ret, g_Timers);
	} else {
		printk(KERN_INFO MODNAME ": exit (%lx,%u)\n", g_SumLast, g_Timers);
	}
}

/* file handler /proc/.. */
static int ram_check_print(struct seq_file *m, void *v)
{
	int ret = 0;
	unsigned long t = 0;

	ret = check_range();
	if (ret) {
		seq_printf(m, "ERROR: const kernel memory address changed (%08lx), stop!\n", g_SumLast);

		return -1;
	}

	if (g_SumLast != 0u) {
		seq_printf(m, "ERROR: const kernel memory was earlier broken (%08lx), no check again!\n", g_SumLast);

		return -2;
	}

	if (g_TempCount <= 2) { /* max 2 calls per timer (1 sec) */
		unsigned long xor;
		g_TempCount ++;
		t = (unsigned long) getTimeUS();
		xor = calcChecksumXor((const void *) g_RangeStart, g_RangeSize);
		t = (unsigned long) getTimeUS() - t;
		xor = g_SumFirst ^ xor;
		if (xor) g_SumLast = xor; /* only write bad events */
	} else {
		g_TempCount ++;
		seq_printf(m, "warning: too much triggers (%u) for ram_check (%08lx) !\n", g_TempCount, g_SumLast);
	}

	if (!g_SumLast) {
		seq_printf(m, "const kernel memory is OK, size = %lu MB, t = %lu us (%u)\n", g_RangeSize>>20, t, g_Timers);
	} else {
		printk(KERN_EMERG MODNAME "error: const kernel memory is broken (%08lx != 0), please reboot!", g_SumLast);
		seq_printf(m, "ERROR: const kernel memory is broken (%08lx != 0), please reboot!\n", g_SumLast);
		ret = -3;
		/* !! kernel panic (ram error) !! */
	}

	return ret;
}

/* only inside timer (1 sec) */
static int ram_check_run(void)
{
	const unsigned int lastblk = (unsigned int) (g_RangeSize / BLOCK);
	const unsigned int b = g_Timers % BLOCKS;
	unsigned int  l;
	unsigned long x = 0;

	if (0 == b) {
		check_range(); /* check sometimes */
	}

	for (l = b; l <= lastblk; l += BLOCKS) {
		x ^= check_block(l, lastblk);
	}
	x ^= g_BlockSum[b];

	if (x != 0u) {
		check_range();
		g_SumLast = x;
		/* !! kernel panic (ram error) !! */
		printk(KERN_EMERG MODNAME "error: const kernel memory broken (%08lx != 0), timer(%u, %u) !", x, g_Timers, b);
	}

	return 0;
}

/* Timer Callback */
void ram_check_timer_callback(unsigned long data)
{
	int ret;

	g_Timers ++;
	if (0 == g_RangeSize) {
		g_TimerActive = 0;

		return;
	}

	if (g_SumLast != 0u) { /* failure earlier detected, skip */
		if (0 == (g_Timers % 32u)) {
			printk(KERN_WARNING MODNAME "error: const kernel memory earlier broken (%08lx), skip !\n", g_SumLast);
		}

		return;
	}

	ram_check_run();

	ret = mod_timer(&ram_check_timer, jiffies + msecs_to_jiffies(CHECK_TIMER_MS));
	if (ret) {
		printk(KERN_ALERT MODNAME "error: in mod_timer (%u, %d)\n", g_Timers, ret);
	}

	g_TimerActive = 1; /* is already set, but be safe */
	g_TempCount = 0;
}


static int ram_check_open(struct inode *inode, struct file *file)
{
	return single_open(file, ram_check_print, NULL);
}


module_init(ram_check_init);
module_exit(ram_check_exit);

Re: new module to check constant memory for corruption

[Andi Kleen]

Alexander.Kleinsorge@gmx.de writes:

> ramcheck kernel module
> new module to check constant memory for corruption
>
> detect corruption of constant kernel memory (text and data) periodically.
> runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
> which is distributed over 8 (BLOCKS) time partitions (less than half
> ms per sec).
> in case of checksum (xor) error, an kernel log is posted.
> manual trigger via /proc/ramcheck is possible.
> range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")


Can you explain how this works? How does it handle legal writes?

If it just checks its own memory it could be done in user space.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only

Aw: Re: new module to check constant memory for corruption

[Alexander.Kleinsorge]

Hi Andi,

the module considers only the adress range between: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata").
this range has a typical size of 10..20 mb (depending on kernel-version and arch). 
see files: linux-3.*\arch\x86\mm\init_32.c + init_64.c
function: void mark_rodata_ro(void)
"Write protecting the kernel text: %luk\n"
"Write protecting the kernel read-only data: %luk\n"
dmesg | grep protecting

your question: there are no writes in this write protected adress range (e.g. kernel code).
my idea is to calculate a checksum (xor is fastest) over this range and check later (periodically) if its unchanged.
see source code download (5 KB): http://tauruz.homeip.net/ramcheck.tgz
the code is working fine and the checksum is (as expected) constant (at least for many hours).

regards, Alexander


Gesendet: Sonntag, 13. April 2014 um 05:00 Uhr
Von: "Andi Kleen" <andi@firstfloor.org>
An: Alexander.Kleinsorge@gmx.de
Cc: linux-kernel@vger.kernel.org
Betreff: Re: new module to check constant memory for corruption
Alexander.Kleinsorge@gmx.de writes:

> ramcheck kernel module
> new module to check constant memory for corruption
>
> detect corruption of constant kernel memory (text and data) periodically.
> runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
> which is distributed over 8 (BLOCKS) time partitions (less than half
> ms per sec).
> in case of checksum (xor) error, an kernel log is posted.
> manual trigger via /proc/ramcheck is possible.
> range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")


Can you explain how this works? How does it handle legal writes?

If it just checks its own memory it could be done in user space.

-Andi

--
ak@linux.intel.com -- Speaking for myself only

Re: Re: new module to check constant memory for corruption

[Richard Weinberger]

On Sun, Apr 13, 2014 at 12:14 PM,  <Alexander.Kleinsorge@gmx.de> wrote:
> Hi Andi,
>
> the module considers only the adress range between: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata").
> this range has a typical size of 10..20 mb (depending on kernel-version and arch).
> see files: linux-3.*\arch\x86\mm\init_32.c + init_64.c
> function: void mark_rodata_ro(void)
> "Write protecting the kernel text: %luk\n"
> "Write protecting the kernel read-only data: %luk\n"
> dmesg | grep protecting
>
> your question: there are no writes in this write protected adress range (e.g. kernel code).

And what happens if one enables dynamic ftrace or other kernel
features which modify kernel code?

> my idea is to calculate a checksum (xor is fastest) over this range and check later (periodically) if its unchanged.
> see source code download (5 KB): http://tauruz.homeip.net/ramcheck.tgz
> the code is working fine and the checksum is (as expected) constant (at least for many hours).
>
> regards, Alexander
>
>
> Gesendet: Sonntag, 13. April 2014 um 05:00 Uhr
> Von: "Andi Kleen" <andi@firstfloor.org>
> An: Alexander.Kleinsorge@gmx.de
> Cc: linux-kernel@vger.kernel.org
> Betreff: Re: new module to check constant memory for corruption
> Alexander.Kleinsorge@gmx.de writes:
>
>> ramcheck kernel module
>> new module to check constant memory for corruption
>>
>> detect corruption of constant kernel memory (text and data) periodically.
>> runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
>> which is distributed over 8 (BLOCKS) time partitions (less than half
>> ms per sec).
>> in case of checksum (xor) error, an kernel log is posted.
>> manual trigger via /proc/ramcheck is possible.
>> range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")
>
>
> Can you explain how this works? How does it handle legal writes?
>
> If it just checks its own memory it could be done in user space.
>
> -Andi
>
> --
> ak@linux.intel.com -- Speaking for myself only
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/



-- 
Thanks,
//richard

Re: new module to check constant memory for corruption

[Richard Weinberger]

Alex,

please don't crop the recipient list.

Am 13.04.2014 18:26, schrieb Alexander.Kleinsorge@gmx.de:
> Hi Richard,
> 
> I updated my code and check for ftrace (cat /proc/sys/kernel/ftrace_enabled > 0).
> http://tauruz.homeip.net/ramcheck.tgz
> 
> On a 24/7 server ftrace is not a common thing anyway.

I disagree with you.
perf, systemtap and other tools are widely used on "24/7" servers.
As Andi noted, you have to hook all places where kernel code get's modified.

Thanks,
//richard

> Thx, Alex
>  
> 
> Gesendet: Sonntag, 13. April 2014 um 12:26 Uhr
> Von: "Richard Weinberger" <richard.weinberger@gmail.com>
> An: Alexander.Kleinsorge@gmx.de
> Cc: "Andi Kleen" <andi@firstfloor.org>, LKML <linux-kernel@vger.kernel.org>
> Betreff: Re: Re: new module to check constant memory for corruption
> On Sun, Apr 13, 2014 at 12:14 PM, <Alexander.Kleinsorge@gmx.de> wrote:
>> Hi Andi,
>>
>> the module considers only the adress range between: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata").
>> this range has a typical size of 10..20 mb (depending on kernel-version and arch).
>> see files: linux-3.*\arch\x86\mm\init_32.c + init_64.c
>> function: void mark_rodata_ro(void)
>> "Write protecting the kernel text: %luk\n"
>> "Write protecting the kernel read-only data: %luk\n"
>> dmesg | grep protecting
>>
>> your question: there are no writes in this write protected adress range (e.g. kernel code).
> 
> And what happens if one enables dynamic ftrace or other kernel
> features which modify kernel code?
> 
>> my idea is to calculate a checksum (xor is fastest) over this range and check later (periodically) if its unchanged.
>> see source code download (5 KB): http://tauruz.homeip.net/ramcheck.tgz
>> the code is working fine and the checksum is (as expected) constant (at least for many hours).
>>
>> regards, Alexander
>>
>>
>> Gesendet: Sonntag, 13. April 2014 um 05:00 Uhr
>> Von: "Andi Kleen" <andi@firstfloor.org>
>> An: Alexander.Kleinsorge@gmx.de
>> Cc: linux-kernel@vger.kernel.org
>> Betreff: Re: new module to check constant memory for corruption
>> Alexander.Kleinsorge@gmx.de writes:
>>
>>> ramcheck kernel module
>>> new module to check constant memory for corruption
>>>
>>> detect corruption of constant kernel memory (text and data) periodically.
>>> runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
>>> which is distributed over 8 (BLOCKS) time partitions (less than half
>>> ms per sec).
>>> in case of checksum (xor) error, an kernel log is posted.
>>> manual trigger via /proc/ramcheck is possible.
>>> range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")
>>
>>
>> Can you explain how this works? How does it handle legal writes?
>>
>> If it just checks its own memory it could be done in user space.
>>
>> -Andi
>>
>> --
>> ak@linux.intel.com -- Speaking for myself only
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html[http://vger.kernel.org/majordomo-info.html]
>> Please read the FAQ at http://www.tux.org/lkml/[http://www.tux.org/lkml/]
> 
> 
> 
> --
> Thanks,
> //richard
> 

Re: new module to check constant memory for corruption

[Jiri Kosina]

On Sun, 13 Apr 2014, Richard Weinberger wrote:

> > Hi Richard,
> > 
> > I updated my code and check for ftrace (cat /proc/sys/kernel/ftrace_enabled > 0).
> > http://tauruz.homeip.net/ramcheck.tgz
> > 
> > On a 24/7 server ftrace is not a common thing anyway.
> 
> I disagree with you.
> perf, systemtap and other tools are widely used on "24/7" servers.

Plus the self-modifying code is present in other parts of kernel that have 
nothing to do with tracing as well. Consider static keys for example; they 
are used heavily in networking stack, scheduler code, etc.

-- 
Jiri Kosina
SUSE Labs

Aw: Re: Re: new module to check constant memory for corruption

[Alexander.Kleinsorge]

[-- Attachment #1: Type: text/plain, Size: 3745 bytes --]

1. KERN_EMERG is now KERN_INFO or ALERT (depending on ftrace_enabled).
2. there is no trigger for real ram errors possible! (only for out of bound writes)
Therefore it is checked periodically and can also be triggered by /proc/ramcheck

Alex

 On Sun, 13 Apr 2014 03:33:10 +0200, Alexander.Kleinsorge [at] gmx said:

> printk(KERN_EMERG MODNAME "error: const kernel memory is broken (%08lx != 0), please reboot!", g_SumLast);

Make a list of all the things that can dynamically modify kernel text while
it's running, starting with the alternatives code, and adding in the various
tracing and debugging tools, and ask yourself if KERN_EMERG is the right
choice here...

Also, that printk is in the wrong place - it only fires if somebody reads
the /proc file, and it *should* trigger as soon as practical after a
problem has been detected.

Bonus points for computing what percent of single, double, triple, and
other error syndromes your xor detects (hint - what is the behavior of a
dead row or column of bits?). 
 

Gesendet: Sonntag, 13. April 2014 um 12:26 Uhr
Von: "Richard Weinberger" <richard.weinberger@gmail.com>
An: Alexander.Kleinsorge@gmx.de
Cc: "Andi Kleen" <andi@firstfloor.org>, LKML <linux-kernel@vger.kernel.org>
Betreff: Re: Re: new module to check constant memory for corruption
On Sun, Apr 13, 2014 at 12:14 PM, <Alexander.Kleinsorge@gmx.de> wrote:
> Hi Andi,
>
> the module considers only the adress range between: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata").
> this range has a typical size of 10..20 mb (depending on kernel-version and arch).
> see files: linux-3.*\arch\x86\mm\init_32.c + init_64.c
> function: void mark_rodata_ro(void)
> "Write protecting the kernel text: %luk\n"
> "Write protecting the kernel read-only data: %luk\n"
> dmesg | grep protecting
>
> your question: there are no writes in this write protected adress range (e.g. kernel code).

And what happens if one enables dynamic ftrace or other kernel
features which modify kernel code?

> my idea is to calculate a checksum (xor is fastest) over this range and check later (periodically) if its unchanged.
> see source code download (5 KB): http://tauruz.homeip.net/ramcheck.tgz
> the code is working fine and the checksum is (as expected) constant (at least for many hours).
>
> regards, Alexander
>
>
> Gesendet: Sonntag, 13. April 2014 um 05:00 Uhr
> Von: "Andi Kleen" <andi@firstfloor.org>
> An: Alexander.Kleinsorge@gmx.de
> Cc: linux-kernel@vger.kernel.org
> Betreff: Re: new module to check constant memory for corruption
> Alexander.Kleinsorge@gmx.de writes:
>
>> ramcheck kernel module
>> new module to check constant memory for corruption
>>
>> detect corruption of constant kernel memory (text and data) periodically.
>> runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
>> which is distributed over 8 (BLOCKS) time partitions (less than half
>> ms per sec).
>> in case of checksum (xor) error, an kernel log is posted.
>> manual trigger via /proc/ramcheck is possible.
>> range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")
>
>
> Can you explain how this works? How does it handle legal writes?
>
> If it just checks its own memory it could be done in user space.
>
> -Andi
>
> --
> ak@linux.intel.com -- Speaking for myself only
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html[http://vger.kernel.org/majordomo-info.html]
> Please read the FAQ at http://www.tux.org/lkml/[http://www.tux.org/lkml/]



--
Thanks,
//richard

[-- Attachment #2: ramcheck.c --]
[-- Type: text/plain, Size: 11941 bytes --]

/* ramcheck.c - checks predefined kernel memory region
*
* KernelRamCheck by
*
* Copyright 2014 Alexander Kleinsorge <alexander[dot]kleinsorge[at]gmx[dot]de>
* Copyright 2014 Benjamin Schroedl <benjamin[at]dev-tec[dot]de>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 2 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
*
*/

// Defining __KERNEL__ and MODULE allows us to access kernel-level code not usually available to userspace programs.
/*
#undef __KERNEL__
#define __KERNEL__

#undef MODULE
#define MODULE
*/
#include <linux/init.h>
#include <linux/module.h>
#include <linux/kallsyms.h>
#include <linux/kernel.h>
#include <linux/pfn.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
#include <linux/timer.h>
#include <linux/delay.h>
#include <crypto/algapi.h>
#include <asm/uaccess.h>
#include <asm/sections.h>
#include <linux/time.h>
#include <linux/ftrace.h>
#include <asm/cacheflush.h>
/* read file: /proc/sys/kernel/ftrace_enabled */
#include <linux/fs.h>
#include <asm/segment.h>
#include <asm/uaccess.h>
#include <linux/buffer_head.h>

#ifdef CONFIG_DYNAMIC_FTRACE
# warning "RAMCHECK: using FTRACE can cause false alarm."
/* cat /proc/sys/kernel/ftrace_enabled (default 1) */
#endif

#define DRIVER_AUTHOR    "Alexander Kleinsorge <alexander[dot]kleinsorge[at]gmx[dot]de> and Benjamin Schroedl <benjamin[at]dev-tec[dot]de>"
#define DRIVER_DESC      "checks predefined memory kernel region"
#define proc_fs_name     "ramcheck"
#define MODNAME proc_fs_name" "

#define CHECK_TIMER_MS   1000
#define BLOCK ((sizeof(long) <= 4) ? 1u << 20 : 2u << 20) /* 64bit CPU is often faster (bigger blocks at once) */
#define BLOCKS 8

MODULE_LICENSE("GPL");
MODULE_AUTHOR(DRIVER_AUTHOR);
MODULE_DESCRIPTION(DRIVER_DESC);


static void ram_check_exit(void);
static int  ram_check_init(void);
static int  ram_check_print(struct seq_file *m, void *v);
static int  ram_check_run(void);
void        ram_check_timer_callback(unsigned long data);
static int  ram_check_open(struct inode *inode, struct file *file);

static struct timer_list ram_check_timer;
struct proc_dir_entry *ram_check_proc_file;
struct file_operations proc_fops = {
	.open     = ram_check_open,
	.read     = seq_read,
	.llseek   = seq_lseek,
	.release  = single_release,
};


/* Global Variables */

struct s_ramcheck_init
{
	unsigned long RangeSize;
	unsigned long RangeStart;
	unsigned long SumFirst;
	unsigned long BlockSum[BLOCKS];
	unsigned int  TimerActive;
	unsigned int  dummy; // allign 64 bit
};
static struct s_ramcheck_init g_init = {0u,0u,0u, {0},0u,0u};

static unsigned long g_SumLast = 0;
static unsigned int g_TempCount = 0;
static unsigned int g_Timers = 0;


/* Functions */

/* simple checksum/hash, xor is fastest (~1-6 MB/ms) */
static unsigned long calcChecksumXor(const void* ptr, const unsigned long bytes)
{
	unsigned long ret, i;
	const unsigned long long* ptr64 = (const unsigned long long*) ptr;
	const unsigned long len = bytes / sizeof(*ptr64);
	unsigned long long sum64 = 0u;

	for (i=0; i < len; i++) {
		sum64 ^= ptr64[i];
	}

	/* return 32 or 64 bit possible */
	ret = ((unsigned long) sum64) ^ ((unsigned long) (sum64>>32));

	return ret;
}

/* merge 4 to 1 byte ==> less info (security reason) */
static unsigned char shrink2byte(unsigned long u)
{
	unsigned long ret = (u) ^ (u>>8) ^ (u>>16) ^ (u>>24);
	/* (lowest) 32 bit --> 8 bit */
	return (unsigned char) ret;
}

static unsigned long long getTimeUS(void)
{
	struct timespec ts;
	ktime_get_ts( &ts );

	return (ts.tv_sec*USEC_PER_SEC) + (ts.tv_nsec/1000u);
}

// check for const range (in case of strange checksum)
static int check_range(void)
{
	unsigned long start = kallsyms_lookup_name("_text");
	unsigned long size  = kallsyms_lookup_name("__end_rodata") - start;
	int ret = 0;

	if (g_init.RangeSize && ((start != g_init.RangeStart) || (size != g_init.RangeSize))) {
		printk(KERN_WARNING MODNAME "error: const kernel memory address changed (%08lx + %lu K), stop!\n", start, size/1024u);
		g_init.RangeSize = 0; /* stop working */
		ret = -1;
	}

	return ret;
}


static int fread_ftrace(void)
{
	int rights = 0, flags = O_RDONLY;
	const char path[64] = "/proc/sys/kernel/ftrace_enabled";
	char data[4] = "\0\0\0";
	struct file* filp = NULL;
	mm_segment_t oldfs;
	int err = 0, ret;
	unsigned long long offset = 0u;

	oldfs = get_fs();
	set_fs(get_ds());
	filp = filp_open(path, flags, rights);
	set_fs(oldfs);
	if (IS_ERR(filp)) {
		err = PTR_ERR(filp);
		return -1;
	}

	oldfs = get_fs();
	set_fs(get_ds());
	ret = vfs_read(filp, data, 1, &offset);
	set_fs(oldfs);
	filp_close(filp, NULL);

	if (!ret) return -2;
	if ('1' == data[0]) return 1;

	return 0;
}

static int check_ftrace(void)
{
	int ft = fread_ftrace();
	
#ifdef CONFIG_DYNAMIC_FTRACE
	/* unknown: function_trace_stop, ftrace_enabled, ftrace_nr_registered_ops() */
	if (ft > 0)
		printk(KERN_WARNING MODNAME ": ftrace state = %d (enabled), false alarm possible!\n", ft);
	else
        printk(KERN_INFO MODNAME ": ftrace state = %d (disabled).\n", ft);
#else
	if (ft >= 0) 
		printk(KERN_WARNING MODNAME ": CONFIG_DYNAMIC_FTRACE not defined, but useable (%d)!\n", ft);
#endif
	return 0;
}

/* partitions to limit runtime to 1 ms (1..6 MB) */
static unsigned long check_block(unsigned long blk, unsigned long lastblk)
{
	const unsigned long offset = blk * BLOCK;
	unsigned long start = g_init.RangeStart + offset;
	unsigned long size = BLOCK;
	unsigned long ret;

	if (blk >= lastblk) {
		size = g_init.RangeSize % BLOCK;
		if (blk > lastblk) {
			return (~0u); /* should never happen */
		}
	}
	ret = calcChecksumXor((const void *) start, size);

	return ret;
}

/* INIT: call not before mark_rodata_ro(void) */
static int ram_check_init(void)
{
	int ret;
	unsigned long start, size, end;
	unsigned long b, t, bps;

	for (b=0; b<BLOCKS; b++) {
		g_init.BlockSum[b] = 0u;
	}

	ram_check_proc_file = proc_create(proc_fs_name, 0, NULL, &proc_fops);

	if (ram_check_proc_file == NULL) {
		remove_proc_entry(proc_fs_name, NULL);
		printk(KERN_ALERT MODNAME "error: Could not initialize /proc/%s\n", proc_fs_name);

		return -ENOMEM;
	}

	start = kallsyms_lookup_name("_text");
	end   = kallsyms_lookup_name("__end_rodata");
	/* kallsyms_lookup_name("__end_rodata_hpage_align"); ==> Bad Address! */
	size  = (end - start);

	/* error handling on illegal size and ptr or size to big */
	if ((start == 0) || (end == 0) || (end <= start) || (size > (1000u << 20))) {
		printk(KERN_ALERT MODNAME "error: bad start or end adress [%lx + %lx]\n", start, size);

		return -EFAULT;
	}
	if ((size < (1u << 20)) || (size > (200u << 20))) {
		printk(KERN_INFO MODNAME "warning: strange const size = %lu MB\n", size>>20);
	}

	check_ftrace();

	g_init.RangeStart = start;
	g_init.RangeSize  = size;

	t = (unsigned long) getTimeUS();
	for (b = 0; b <= size/BLOCK; b++) {
		g_init.BlockSum[b % BLOCKS] ^= check_block(b, size/BLOCK);
	}
	t = (unsigned long) getTimeUS() - t; /* ca. 1k..7k Byte/us */
	for (b = 0; b < BLOCKS; b++) {
		g_init.SumFirst ^= g_init.BlockSum[b]; /* all together */
	}

	bps = (t > 0) ? (size / t) : 0;
	printk(KERN_INFO MODNAME "init: %lu B/us, %lu kB, %d bit, (0x%02x, %u)\n",
		bps, size>>10, (int)sizeof(g_init.SumFirst)*8, shrink2byte(g_init.SumFirst), BLOCKS);

	/* TIMER init */
	setup_timer(&ram_check_timer, ram_check_timer_callback, 0);

	ret = mod_timer(&ram_check_timer, jiffies + msecs_to_jiffies(CHECK_TIMER_MS));
	if (ret) {
		printk(KERN_ALERT MODNAME "error: Failure in mod_timer\n");

		return -EFAULT;
	}
	g_init.TimerActive = 1;
	/* optional: clflush_cache_range or flush_icache_range (&g_init, &g_init + sizeof(g_init)); */

	return 0;
}


static void ram_check_exit(void)
{
	int ret;
	g_init.RangeSize = 0; /* stop work */

	remove_proc_entry(proc_fs_name, NULL);
	check_range();

	for (ret = 0; (g_init.TimerActive > 0) && (ret < 1000/20); ret ++) {
		mdelay(20);
	}

	ret = del_timer_sync(&ram_check_timer);

	if (ret) {
		printk(KERN_WARNING MODNAME "warning: timer still in use on exit...(%d,%u)\n", ret, g_Timers);
	} else {
		printk(KERN_INFO MODNAME ": exit (%lx,%u)\n", g_SumLast, g_Timers);
	}
}

/* file handler /proc/.. */
static int ram_check_print(struct seq_file *m, void *v)
{
	int ret = 0;
	unsigned long t = 0;

	ret = check_range();
	if (ret) {
		seq_printf(m, "ERROR: const kernel memory address changed (%08lx), stop!\n", g_SumLast);

		return -1;
	}

	if (g_SumLast != 0u) {
		seq_printf(m, "ERROR: const kernel memory was earlier broken (%08lx), no check again!\n", g_SumLast);

		return -2;
	}

	if (g_TempCount <= 2) { /* max 2 calls per timer (1 sec) */
		unsigned long xor;
		g_TempCount ++;
		t = (unsigned long) getTimeUS();
		xor = calcChecksumXor((const void *) g_init.RangeStart, g_init.RangeSize);
		t = (unsigned long) getTimeUS() - t;
		xor = g_init.SumFirst ^ xor;
		if (xor) g_SumLast = xor; /* only write bad events */
	} else {
		g_TempCount ++;
		seq_printf(m, "warning: too much triggers (%u) for ram_check (%08lx) !\n", g_TempCount, g_SumLast);
	}

	if (!g_SumLast) {
		seq_printf(m, "const kernel memory is OK, size = %lu MB, t = %lu us (%u)\n", g_init.RangeSize>>20, t, g_Timers);
	} else {
		int ft = fread_ftrace();
		if (ft > 0) {
			printk(KERN_INFO MODNAME "warning: const kernel memory is broken (%08lx != 0), ftrace used (%d)?\n", g_SumLast, ft);
			seq_printf(m, "WARNING: const kernel memory is broken (%08lx != 0), ftrace used (%d)?\n", g_SumLast, ft);
		} else {
			printk(KERN_ALERT MODNAME "error: const kernel memory is broken (%08lx != 0), please reboot!\n", g_SumLast);
			seq_printf(m, "ERROR: const kernel memory is broken (%08lx != 0), please reboot!\n", g_SumLast);
		}
		ret = -3;
		/* if (!FTRACE) : kernel panic (ram error) !! */
	}

	return ret;
}

/* only inside timer (1 sec) */
static int ram_check_run(void)
{
	const unsigned int lastblk = (unsigned int) (g_init.RangeSize / BLOCK);
	const unsigned int b = g_Timers % BLOCKS;
	unsigned int  l;
	unsigned long x = 0;

	if (0 == b) {
		check_range(); /* check sometimes */
	}

	for (l = b; l <= lastblk; l += BLOCKS) {
		x ^= check_block(l, lastblk);
	}
	x ^= g_init.BlockSum[b];

	if (x != 0u) {
		int ft = fread_ftrace();
		check_range();
		g_SumLast = x;
		if (ft > 0) {
			printk(KERN_ALERT MODNAME "error: const kernel memory broken (%08lx != 0), timer(%u, %u), ftrace=on(%d)!\n", x, g_Timers, b, ft);
			/* kernel panic (ram error) !! */
		} else {
			printk(KERN_WARNING MODNAME "warning: const kernel memory broken (%08lx != 0), timer(%u, %u), ftrace=off(%d)!\n", x, g_Timers, b, ft);
		}
	}

	return 0;
}

/* Timer Callback */
void ram_check_timer_callback(unsigned long data)
{
	int ret;

	g_Timers ++;
	if (0 == g_init.RangeSize) {
		g_init.TimerActive = 0;

		return;
	}

	if (g_SumLast != 0u) { /* failure earlier detected, skip */
		if (0 == (g_Timers % 32u)) {
			printk(KERN_WARNING MODNAME "error: const kernel memory earlier broken (%08lx), skip !\n", g_SumLast);
		}

		return;
	}

	ram_check_run();

	ret = mod_timer(&ram_check_timer, jiffies + msecs_to_jiffies(CHECK_TIMER_MS));
	if (ret) {
		printk(KERN_ALERT MODNAME "error: in mod_timer (%u, %d)\n", g_Timers, ret);
	}

	g_init.TimerActive = 1; /* is already set, but be safe */
	g_TempCount = 0;
}


static int ram_check_open(struct inode *inode, struct file *file)
{
	return single_open(file, ram_check_print, NULL);
}


module_init(ram_check_init);
module_exit(ram_check_exit);


[-- Attachment #3: ramcheck.tgz --]
[-- Type: application/octet-stream, Size: 4959 bytes --]

Re: Re: new module to check constant memory for corruption

[Andi Kleen]

> your question: there are no writes in this write protected adress range (e.g. kernel code).

It's actually not true, Linux changes r/o code. But you could
handle that by hooking into the right places.

> my idea is to calculate a checksum (xor is fastest) over this range and check later (periodically) if its unchanged.
> see source code download (5 KB): http://tauruz.homeip.net/ramcheck.tgz
> the code is working fine and the checksum is (as expected) constant (at least for many hours).
> 

So is the goal security or reliability or debugging?

Reliability:
I have doubts it makes sense for that. On most system the code
is only a very small part of the total memory. So you wouldn't
cover most data.

Also if something corrupts the code we likely already detect it
eventually by crashing. Your module would need to panic too in this
case.

Security: If someone can change the code what stops them from changing
the checksum module too?

Also if you use a poor (= fast) checksum it's likely easy to construct
a valid patch that does not change the checksum.

Debugging:
Maybe, but I have never seen a bug where code got corrupted.

The user program technique works reasonably well for finding bad
pointers. Write a program that allocates a lot of memory. Regularly
checksum and recheck all its memory.


-Andi

Aw: Re: Re: new module to check constant memory for corruption

[Alexander.Kleinsorge]

Hi Andi,
 
1. I build in a check if ftrace is enabled. (like: cat /debug/tracing/tracing_enabled != 1)

2. Main goal is to detect: real ram errors (non crc systems = normal pc). This happens more often than you think:
DRAM Errors in the Wild: A Large-Scale Field Study, Mai 2009 (http://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf)
The rate is about 1 bit per GB per year!

3. I read a blog some weeks ago where an Android developer had problem with accidentally corrupted kernel memory (on ARM).
And he asked for the x86 feature [arch\x86\mm\init_32.c + init_64.c / void mark_rodata_ro()]. He wasted long time debugging it.

4. If kernel text or data is wrong by 1 bit (ram error) system can crash, but does not have to!
The worst case: this single bit is producing similar but runnable code.
But this range of memory is the most critical (even if only 1% of all ram).

Alex

-----------------------------------------------------

Gesendet: Sonntag, 13. April 2014 um 17:55 Uhr
Von: "Andi Kleen" <andi@firstfloor.org>
An: Alexander.Kleinsorge@gmx.de
Cc: "Andi Kleen" <andi@firstfloor.org>, linux-kernel@vger.kernel.org
Betreff: Re: Re: new module to check constant memory for corruption
> your question: there are no writes in this write protected adress range (e.g. kernel code).

It's actually not true, Linux changes r/o code. But you could
handle that by hooking into the right places.

> my idea is to calculate a checksum (xor is fastest) over this range and check later (periodically) if its unchanged.
> see source code download (5 KB): http://tauruz.homeip.net/ramcheck.tgz
> the code is working fine and the checksum is (as expected) constant (at least for many hours).
>

So is the goal security or reliability or debugging?

Reliability:
I have doubts it makes sense for that. On most system the code
is only a very small part of the total memory. So you wouldn't
cover most data.

Also if something corrupts the code we likely already detect it
eventually by crashing. Your module would need to panic too in this
case.

Security: If someone can change the code what stops them from changing
the checksum module too?

Also if you use a poor (= fast) checksum it's likely easy to construct
a valid patch that does not change the checksum.

Debugging:
Maybe, but I have never seen a bug where code got corrupted.

The user program technique works reasonably well for finding bad
pointers. Write a program that allocates a lot of memory. Regularly
checksum and recheck all its memory.


-Andi

Aw: Re: new module to check constant memory for corruption

[Alexander.Kleinsorge]

Hi Andi,

I would like to ask for your help.
Is there a central kernel function that enables/disables write-protection for kernel memory?
I only know the place, where its write protected at booting for x86/x64.
If there is one place, it would be much easier, as to hook into many places (e.g. ftrace, etc). 

My goal is to verify kernel memory (the constant part), not to rebuild memcheck as a process.
In my Ubuntu 12.04 system, this range has never changed during module-runtime (many hours).
I think in >90% of all systems is no ftrace (etc) changing this range.

So I need either a trigger when to update my reference sum, or to know when ftrace (etc) are really working.
ftrace_enabled = 1 (default) is no indicator, because it is typically on.

Thanks a lot,
Alexander


Gesendet: Sonntag, 13. April 2014 um 05:00 Uhr
Von: "Andi Kleen" <andi@firstfloor.org>
An: Alexander.Kleinsorge@gmx.de
Cc: linux-kernel@vger.kernel.org
Betreff: Re: new module to check constant memory for corruption
Alexander.Kleinsorge@gmx.de writes:

> ramcheck kernel module
> new module to check constant memory for corruption
>
> detect corruption of constant kernel memory (text and data) periodically.
> runtime costs about 1..2 ms per sec (about 10 mb with 5 mb/ms),
> which is distributed over 8 (BLOCKS) time partitions (less than half
> ms per sec).
> in case of checksum (xor) error, an kernel log is posted.
> manual trigger via /proc/ramcheck is possible.
> range: kallsyms_lookup_name("_text") .. kallsyms_lookup_name("__end_rodata")


Can you explain how this works? How does it handle legal writes?

If it just checks its own memory it could be done in user space.

-Andi

--
ak@linux.intel.com -- Speaking for myself only

Re: new module to check constant memory for corruption

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

On Sun, 13 Apr 2014 03:33:10 +0200, Alexander.Kleinsorge@gmx.de said:

> 		printk(KERN_EMERG MODNAME "error: const kernel memory is broken (%08lx != 0), please reboot!", g_SumLast);

Make a list of all the things that can dynamically modify kernel text while
it's running, starting with the alternatives code, and adding in the various
tracing and debugging tools, and ask yourself if KERN_EMERG is the right
choice here...

Also, that printk is in the wrong place - it only fires if somebody reads
the /proc file, and it *should* trigger as soon as practical after a
problem has been detected.

Bonus points for computing what percent of single, double, triple, and
other error syndromes your xor detects (hint - what is the behavior of a
dead row or column of bits?).

[-- Attachment #2: Type: application/pgp-signature, Size: 848 bytes --]