RE: iptables/vmalloc issues on alchemy

RE: iptables/vmalloc issues on alchemy

[Christian Gan]

Could this also related to this problem I posted a long time ago?  I
haven't had a chance to revisit things for a while...

-----Original Message-----
Just a little update on this:

I turned off 36 bit address support for the MIPS32
(CONFIG_64BIT_PHYS_ADDR) and found that my tests for vmalloc work.

Can anyone who knows better point me in a direction of where I should
start?

Thanks!

Christian

-----Original Message-----
From: linux-mips-bounce@linux-mips.org
[mailto:linux-mips-bounce@linux-mips.org] On Behalf Of Christian Gan
Sent: Tuesday, January 04, 2005 3:25 PM
To: linux-mips@linux-mips.org
Subject: vmalloc memory corruption

Hey all,

I'm running into a strange memory corruption problem while trying to get
a driver to load firmware using hotplug support on a DBAU1550 eval
board.  This board has a single AU1550 MIPS32 processor.  I'm running
kernel version 2.6.10-rc3.

I've attached a code snippet below that I'd like somebody to try to
verify for me so I can determine whether or not it is a problem with my
kernel/hardware.  

The test code is a simplified version of the interaction between the
driver and the firmware/hotplug support of the kernel.  The firmware
hotplug support (linux/drivers/base/firmware_class.c) if fed data from a
file and places it into a buffer, this buffer gets reallocated when
required and grows as more data is read in.  Old data is memcopied into
the new reallocated buffer and newly read data is appended to the end.
What I'm finding is that eventually this data is corrupted.

Compile the test snippet below as a module and insmod it.  If things are
successful you should see results like this:

# insmod vmalloctest.ko
Using vmalloctest.ko
testing vmalloc
0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x00 0x01 0x02 0x03
0x04 0x05 0x06 0x07 0x08 0x09 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07
0x08 0x09 ...
0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09

But if data changes at any time, you've got problems like me.  When I
run this on my platform eventually the data becomes all 0x00.

Notice that if you use kmalloc instead of vmalloc (define USE_KMALLOC),
everything works.

Thanks!

Christian Gan

// Start vmalloctest.c

#ifndef __KERNEL__
#define __KERNEL__
#endif

#include <linux/config.h>
#include <linux/version.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/errno.h>
#include <linux/string.h>
#include <linux/mm.h>
#include <linux/delay.h>
#include <linux/init.h>
#include <linux/vmalloc.h>

#include <asm/io.h>

//#define USE_KMALLOC

void fillbuf(char * buf, unsigned long len) {
	int i = 0;
	for (i = 0; i < len; i++)
	{
		buf[i] = 0xFF & i;
	}
}

int __init vmalloctest_init(void)
{
    char * buf = NULL;
	int i = 0;

	printk("testing vmalloc\n");
	
	for (i = 0; i < 50; i++)
	{
		int j = 0;
#ifndef USE_KMALLOC		
		char * newbuf = vmalloc( (i+1)*PAGE_SIZE ); // allocate
a new buffer that grows in size #else
		char * newbuf = kmalloc( (i+1)*PAGE_SIZE, GFP_KERNEL );
// allocate a new buffer that grows in size
#endif		
		if (!newbuf)
		{
			printk("Could not allocate memory: %ld bytes\n",
(i+1)*PAGE_SIZE);
			break;
		}
				
		if (i==0)
		{
			fillbuf(newbuf, PAGE_SIZE );		// fill
the original buffer with some data... any will suffice
		}
		else
		{
			memcpy(newbuf, buf, i*PAGE_SIZE);	// copy
the contents of the old buffer to the new	
		}
		
#ifndef USE_KMALLOC
		vfree(buf);		// free the old buffer
#else
		kfree(buf);		// free the old buffer
#endif		

		buf = newbuf;
		
		// print out the first few bytes
		for (j=0; j < 10; j++)
		{
			printk("0x%02X ", 0xff & buf[j]);
		}
		printk("\n");
	}

#ifndef USE_KMALLOC
	vfree(buf);
#else
	kfree(buf);
#endif		

	return 0;
}
module_init(vmalloctest_init);

MODULE_LICENSE("GPL");

RE: iptables/vmalloc issues on alchemy

[Christian Gan]

Could this also related to this problem I posted a long time ago?  I
haven't had a chance to revisit things for a while...

-----Original Message-----
Just a little update on this:

I turned off 36 bit address support for the MIPS32
(CONFIG_64BIT_PHYS_ADDR) and found that my tests for vmalloc work.

Can anyone who knows better point me in a direction of where I should
start?

Thanks!

Christian

-----Original Message-----
From: linux-mips-bounce@linux-mips.org
[mailto:linux-mips-bounce@linux-mips.org] On Behalf Of Christian Gan
Sent: Tuesday, January 04, 2005 3:25 PM
To: linux-mips@linux-mips.org
Subject: vmalloc memory corruption

Hey all,

I'm running into a strange memory corruption problem while trying to get
a driver to load firmware using hotplug support on a DBAU1550 eval
board.  This board has a single AU1550 MIPS32 processor.  I'm running
kernel version 2.6.10-rc3.

I've attached a code snippet below that I'd like somebody to try to
verify for me so I can determine whether or not it is a problem with my
kernel/hardware.  

The test code is a simplified version of the interaction between the
driver and the firmware/hotplug support of the kernel.  The firmware
hotplug support (linux/drivers/base/firmware_class.c) if fed data from a
file and places it into a buffer, this buffer gets reallocated when
required and grows as more data is read in.  Old data is memcopied into
the new reallocated buffer and newly read data is appended to the end.
What I'm finding is that eventually this data is corrupted.

Compile the test snippet below as a module and insmod it.  If things are
successful you should see results like this:

# insmod vmalloctest.ko
Using vmalloctest.ko
testing vmalloc
0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x00 0x01 0x02 0x03
0x04 0x05 0x06 0x07 0x08 0x09 0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07
0x08 0x09 ...
0x00 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09

But if data changes at any time, you've got problems like me.  When I
run this on my platform eventually the data becomes all 0x00.

Notice that if you use kmalloc instead of vmalloc (define USE_KMALLOC),
everything works.

Thanks!

Christian Gan

// Start vmalloctest.c

#ifndef __KERNEL__
#define __KERNEL__
#endif

#include <linux/config.h>
#include <linux/version.h>
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/errno.h>
#include <linux/string.h>
#include <linux/mm.h>
#include <linux/delay.h>
#include <linux/init.h>
#include <linux/vmalloc.h>

#include <asm/io.h>

//#define USE_KMALLOC

void fillbuf(char * buf, unsigned long len) {
	int i = 0;
	for (i = 0; i < len; i++)
	{
		buf[i] = 0xFF & i;
	}
}

int __init vmalloctest_init(void)
{
    char * buf = NULL;
	int i = 0;

	printk("testing vmalloc\n");
	
	for (i = 0; i < 50; i++)
	{
		int j = 0;
#ifndef USE_KMALLOC		
		char * newbuf = vmalloc( (i+1)*PAGE_SIZE ); // allocate
a new buffer that grows in size #else
		char * newbuf = kmalloc( (i+1)*PAGE_SIZE, GFP_KERNEL );
// allocate a new buffer that grows in size
#endif		
		if (!newbuf)
		{
			printk("Could not allocate memory: %ld bytes\n",
(i+1)*PAGE_SIZE);
			break;
		}
				
		if (i==0)
		{
			fillbuf(newbuf, PAGE_SIZE );		// fill
the original buffer with some data... any will suffice
		}
		else
		{
			memcpy(newbuf, buf, i*PAGE_SIZE);	// copy
the contents of the old buffer to the new	
		}
		
#ifndef USE_KMALLOC
		vfree(buf);		// free the old buffer
#else
		kfree(buf);		// free the old buffer
#endif		

		buf = newbuf;
		
		// print out the first few bytes
		for (j=0; j < 10; j++)
		{
			printk("0x%02X ", 0xff & buf[j]);
		}
		printk("\n");
	}

#ifndef USE_KMALLOC
	vfree(buf);
#else
	kfree(buf);
#endif		

	return 0;
}
module_init(vmalloctest_init);

MODULE_LICENSE("GPL");

Re: iptables/vmalloc issues on alchemy

[Dan Malek]

On Apr 28, 2005, at 4:11 PM, Christian Gan wrote:

> Could this also related to this problem I posted a long time ago?  I
> haven't had a chance to revisit things for a while...

I've just been talking about 2.6, so "long time ago" can't be
that long :-)  I have the updates to the exception handler so
the PTEs get loaded correctly, that's on the way.  I think 2.4
should be OK if anyone is using that.

The one that bothers me is this patch I've been sitting on for
quite some time.  It's not specific to Alchemy, it's a generic
MIPS page table issue.  The problem started when someone
loaded very large modules which caused a new kernel page
table to be allocated.  For some reason I don't remember,
the vmalloc_fault fixup didn't handle this, and the applications
would crash the system because their pgd page  didn't
get updated to reflect this.  The address must have been in
the vmalloc space, but I no longer have the system for testing
(but the code is running in a several thousand real products).
The only way I could make this work is to test the fault address,
the most significant bit anyway, in the TLB miss handler to
see if it was a user or kernel address.  If it was a kernel address,
I loaded the init_mm pgd instead of the thread pgd.  Just one
test and a couple of instructions, but it was necessary.  I'm
still puzzling, but will remember :-)

Thanks.


	-- Dan

Re: iptables/vmalloc issues on alchemy

[Dan Malek]

On Apr 28, 2005, at 4:11 PM, Christian Gan wrote:

> Could this also related to this problem I posted a long time ago?  I
> haven't had a chance to revisit things for a while...

I've just been talking about 2.6, so "long time ago" can't be
that long :-)  I have the updates to the exception handler so
the PTEs get loaded correctly, that's on the way.  I think 2.4
should be OK if anyone is using that.

The one that bothers me is this patch I've been sitting on for
quite some time.  It's not specific to Alchemy, it's a generic
MIPS page table issue.  The problem started when someone
loaded very large modules which caused a new kernel page
table to be allocated.  For some reason I don't remember,
the vmalloc_fault fixup didn't handle this, and the applications
would crash the system because their pgd page  didn't
get updated to reflect this.  The address must have been in
the vmalloc space, but I no longer have the system for testing
(but the code is running in a several thousand real products).
The only way I could make this work is to test the fault address,
the most significant bit anyway, in the TLB miss handler to
see if it was a user or kernel address.  If it was a kernel address,
I loaded the init_mm pgd instead of the thread pgd.  Just one
test and a couple of instructions, but it was necessary.  I'm
still puzzling, but will remember :-)

Thanks.


	-- Dan

RE: iptables/vmalloc issues on alchemy

[Christian Gan]

Sorry, I should have been more clear:  The little test module I created
worked without CONFIG_64BIT_PHYS_ADDR, I realize that it would fail for
the PCI device though.

Thanks!

Christian

-----Original Message-----
From: Dan Malek [mailto:dan@embeddedalley.com] 
Sent: Thursday, April 28, 2005 5:17 PM
To: Christian Gan
Cc: Josh Green; Herbert Valerio Riedel; linux-mips@linux-mips.org; Pete
Popov
Subject: Re: iptables/vmalloc issues on alchemy


On Apr 28, 2005, at 5:22 PM, Christian Gan wrote:

> ..... Again it would be okay if I used
> kmalloc or if I disabled CONFIG_64BIT_PHYS_ADDR.

An Au1500 or Au1550 isn't likely to work with this disabled.
PCI and other peripherals exist in the 36-bit space, unless you
disable them.  I suspect all of this got broken with the dynamic
exception handler building.  Prior to that, I suspect it works
fine.  I guess we need to do some regression testing ....

Thanks.


	-- Dan

RE: iptables/vmalloc issues on alchemy

[Christian Gan]

Sorry, I should have been more clear:  The little test module I created
worked without CONFIG_64BIT_PHYS_ADDR, I realize that it would fail for
the PCI device though.

Thanks!

Christian

-----Original Message-----
From: Dan Malek [mailto:dan@embeddedalley.com] 
Sent: Thursday, April 28, 2005 5:17 PM
To: Christian Gan
Cc: Josh Green; Herbert Valerio Riedel; linux-mips@linux-mips.org; Pete
Popov
Subject: Re: iptables/vmalloc issues on alchemy


On Apr 28, 2005, at 5:22 PM, Christian Gan wrote:

> ..... Again it would be okay if I used
> kmalloc or if I disabled CONFIG_64BIT_PHYS_ADDR.

An Au1500 or Au1550 isn't likely to work with this disabled.
PCI and other peripherals exist in the 36-bit space, unless you
disable them.  I suspect all of this got broken with the dynamic
exception handler building.  Prior to that, I suspect it works
fine.  I guess we need to do some regression testing ....

Thanks.


	-- Dan

RE: iptables/vmalloc issues on alchemy

[Christian Gan]

Yes "long time ago" is a very loose and relative term :).  My version at
the time was 2.6.10 (December-ish).  I was seeing very similar symptoms
to Herbert:

- loading a firmware module (for a prism 802.11b mini PCI card) through
the hotplug support would fail when using vmalloc, but not kmalloc.
- I put together a dirty tester that kept increasing page requests to
vmalloc until corruption was detected.  Again it would be okay if I used
kmalloc or if I disabled CONFIG_64BIT_PHYS_ADDR.

By the way I didn't mention but my original post was attached to that
last post of mine if people wanted to glance over it.  Unfortunately I'm
in the same boat as Dan where I don't have my setup anymore at this
time.

Christian Gan
Software Developer
LibreStream Technologies
200-55 Rothwell Road
Winnipeg, MB, Canada R3P-2M5
christian.gan@librestream.com
Ph: (204) 487-0612 x209
Fx: (204) 487-0914

-----Original Message-----
From: Dan Malek [mailto:dan@embeddededge.com] 
Sent: Thursday, April 28, 2005 3:57 PM
To: Christian Gan
Cc: linux-mips@linux-mips.org; Herbert Valerio Riedel; Josh Green; Pete
Popov
Subject: Re: iptables/vmalloc issues on alchemy


On Apr 28, 2005, at 4:11 PM, Christian Gan wrote:

> Could this also related to this problem I posted a long time ago?  I
> haven't had a chance to revisit things for a while...

I've just been talking about 2.6, so "long time ago" can't be
that long :-)  I have the updates to the exception handler so
the PTEs get loaded correctly, that's on the way.  I think 2.4
should be OK if anyone is using that.

The one that bothers me is this patch I've been sitting on for
quite some time.  It's not specific to Alchemy, it's a generic
MIPS page table issue.  The problem started when someone
loaded very large modules which caused a new kernel page
table to be allocated.  For some reason I don't remember,
the vmalloc_fault fixup didn't handle this, and the applications
would crash the system because their pgd page  didn't
get updated to reflect this.  The address must have been in
the vmalloc space, but I no longer have the system for testing
(but the code is running in a several thousand real products).
The only way I could make this work is to test the fault address,
the most significant bit anyway, in the TLB miss handler to
see if it was a user or kernel address.  If it was a kernel address,
I loaded the init_mm pgd instead of the thread pgd.  Just one
test and a couple of instructions, but it was necessary.  I'm
still puzzling, but will remember :-)

Thanks.


	-- Dan

RE: iptables/vmalloc issues on alchemy

[Christian Gan]

Yes "long time ago" is a very loose and relative term :).  My version at
the time was 2.6.10 (December-ish).  I was seeing very similar symptoms
to Herbert:

- loading a firmware module (for a prism 802.11b mini PCI card) through
the hotplug support would fail when using vmalloc, but not kmalloc.
- I put together a dirty tester that kept increasing page requests to
vmalloc until corruption was detected.  Again it would be okay if I used
kmalloc or if I disabled CONFIG_64BIT_PHYS_ADDR.

By the way I didn't mention but my original post was attached to that
last post of mine if people wanted to glance over it.  Unfortunately I'm
in the same boat as Dan where I don't have my setup anymore at this
time.

Christian Gan
Software Developer
LibreStream Technologies
200-55 Rothwell Road
Winnipeg, MB, Canada R3P-2M5
christian.gan@librestream.com
Ph: (204) 487-0612 x209
Fx: (204) 487-0914

-----Original Message-----
From: Dan Malek [mailto:dan@embeddededge.com] 
Sent: Thursday, April 28, 2005 3:57 PM
To: Christian Gan
Cc: linux-mips@linux-mips.org; Herbert Valerio Riedel; Josh Green; Pete
Popov
Subject: Re: iptables/vmalloc issues on alchemy


On Apr 28, 2005, at 4:11 PM, Christian Gan wrote:

> Could this also related to this problem I posted a long time ago?  I
> haven't had a chance to revisit things for a while...

I've just been talking about 2.6, so "long time ago" can't be
that long :-)  I have the updates to the exception handler so
the PTEs get loaded correctly, that's on the way.  I think 2.4
should be OK if anyone is using that.

The one that bothers me is this patch I've been sitting on for
quite some time.  It's not specific to Alchemy, it's a generic
MIPS page table issue.  The problem started when someone
loaded very large modules which caused a new kernel page
table to be allocated.  For some reason I don't remember,
the vmalloc_fault fixup didn't handle this, and the applications
would crash the system because their pgd page  didn't
get updated to reflect this.  The address must have been in
the vmalloc space, but I no longer have the system for testing
(but the code is running in a several thousand real products).
The only way I could make this work is to test the fault address,
the most significant bit anyway, in the TLB miss handler to
see if it was a user or kernel address.  If it was a kernel address,
I loaded the init_mm pgd instead of the thread pgd.  Just one
test and a couple of instructions, but it was necessary.  I'm
still puzzling, but will remember :-)

Thanks.


	-- Dan

Re: iptables/vmalloc issues on alchemy

[Dan Malek]

On Apr 28, 2005, at 5:22 PM, Christian Gan wrote:

> ..... Again it would be okay if I used
> kmalloc or if I disabled CONFIG_64BIT_PHYS_ADDR.

An Au1500 or Au1550 isn't likely to work with this disabled.
PCI and other peripherals exist in the 36-bit space, unless you
disable them.  I suspect all of this got broken with the dynamic
exception handler building.  Prior to that, I suspect it works
fine.  I guess we need to do some regression testing ....

Thanks.


	-- Dan

Re: iptables/vmalloc issues on alchemy

[Dan Malek]

On Apr 28, 2005, at 5:22 PM, Christian Gan wrote:

> ..... Again it would be okay if I used
> kmalloc or if I disabled CONFIG_64BIT_PHYS_ADDR.

An Au1500 or Au1550 isn't likely to work with this disabled.
PCI and other peripherals exist in the 36-bit space, unless you
disable them.  I suspect all of this got broken with the dynamic
exception handler building.  Prior to that, I suspect it works
fine.  I guess we need to do some regression testing ....

Thanks.


	-- Dan

Re: iptables/vmalloc issues on alchemy

[Thiemo Seufer]

Dan Malek wrote:
> 
> On Apr 28, 2005, at 5:22 PM, Christian Gan wrote:
> 
> >..... Again it would be okay if I used
> >kmalloc or if I disabled CONFIG_64BIT_PHYS_ADDR.
> 
> An Au1500 or Au1550 isn't likely to work with this disabled.
> PCI and other peripherals exist in the 36-bit space, unless you
> disable them.  I suspect all of this got broken with the dynamic
> exception handler building.  Prior to that, I suspect it works
> fine.  I guess we need to do some regression testing ....

I took the code from 2.4 for 36bit support, without the means of testing
the result. There was no support for it in the 2.6 linux-mips.org CVS at
that time, but some people apparently had uncontributed patches.


Thiemo

iptables/vmalloc issues on alchemy

[Herbert Valerio Riedel]

hello!

I'm seeing similiar problems to the ones that Josh Green reported some
time ago on this list (but alas didn't seem to get any further
attention...)

The problem seems to be so far, that when modifying the iptables
structures by adding/flushing the rules, a state can be reached sooner
or later (indeterministic! smells like race) where the data structure
becomes invalid (although there are checks in the kernel which would
prevent that); the result is either ip_tables.c:ipt_do_tables() causing
an oops due to bad pointer dereferencing (or the kernel freezing w/o
further notice at all), or the iptables tool being unable to
retrieve/modify the rules from the kernel (and getting ENOMEM/EINVAL) or
simply segfaulting due to other inconsistencies with the data...

I was able to avoid corruption by replacing all vmalloc()/vfree() calls
by kmalloc()/kfree()'s respectively in ip_tables.c; another variant
which I was suggested to try helped as well: inserting flush_tlb_all()
calls after every vmalloc() in said source file;

I assumed so far, the issue must be alchemy specific, as I experienced
this on a DbAu1550 board; and Josh had it on a DbAu1100; As it's a
rather easy to trigger bug, I would have expected to see more bug
reports if it affected more than just the alchemy's on 2.6.x;
I tried it with a few kernel revisions from linux-mips' cvs (from 2.6.10
upto 2.6.12rc2); also tried different compilers (debian's cross-gcc's
3.4.4 and 3.3.3), even switched the alchemy to little endian
operation... all the same; everything else I use on that board has been
rather stable so far, iptables are the only part which show up this
vmalloc-issue so far...

as to reproducing the bug, it's rather easy for me:

a script repeatedly performing rule modifications should trigger the
issue rather easily (possibly called over a remote ssh/telnet session,
in order to create a bit of traffic as well...)

set -x
while :
do
  iptables -F || exit 1
  iptables -A INPUT -i lo -j ACCEPT || exit 1
  iptables -A INPUT -p udp -i eth0 --dport domain -j ACCEPT || exit 1
  iptables -A INPUT -p udp -i eth0 --dport 6666 -j ACCEPT || exit 1
  iptables -A INPUT -p tcp -i eth0 --dport ssh -j ACCEPT || exit 1
  iptables -A INPUT -p tcp -i eth0 --dport http -j ACCEPT || exit 1
  iptables -A INPUT -p tcp -i eth0 --dport https -j ACCEPT || exit 1
  iptables -A INPUT -p tcp -i eth0 --dport 3496 -j ACCEPT || exit 1
done

or alternatively (requiring state & multiport helpers)

while :
do
  iptables -F
  iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT  || exit 1
  iptables -A INPUT -i lo -j ACCEPT || exit 1
  iptables -A INPUT -p udp -i eth0 -m multiport --dports domain,6666 -j ACCEPT || exit 1
  iptables -A INPUT -p tcp -i eth0 -m multiport --dports ssh,http,https,3496 -j ACCEPT || exit 1
  iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT || exit 1
  iptables -A OUTPUT -o lo -j ACCEPT || exit 1
  iptables -A OUTPUT -p udp -o eth0 -m multiport --dports ntp -j ACCEPT || exit 1
  iptables -A OUTPUT -p tcp -o eth0 -m multiport --dports www,https,8444,8445,8446,8454,8455,8456,8464,8465,8466,9225 -j ACCEPT || exit 1
done

regards,
-- 
Herbert Valerio Riedel <hvr@hvrlab.org>

Re: iptables/vmalloc issues on alchemy

[Josh Green]

[-- Attachment #1: Type: text/plain, Size: 4087 bytes --]

In my case it seemed I was able to achieve a stable condition by loading
modules in a specific order.  I am not doing a lot of iptables rule
modifications currently though, but will be in the future.  I was
planning on doing some additional gdb debugging of the failure
(especially the initial large MMAP attempt by iptables, which was 1.5GB
in my case).  I wont be doing anything on this for quite a while though,
since I am currently away from the board I was doing this work on.  I'm
currently traveling so I'm not on the Linux MIPS list.  I would be
interested in any fixes to this problem though, so please CC me :)
Cheers.
	Josh Green

On Tue, 2005-04-26 at 10:43 +0200, Herbert Valerio Riedel wrote:
> hello!
> 
> I'm seeing similiar problems to the ones that Josh Green reported some
> time ago on this list (but alas didn't seem to get any further
> attention...)
> 
> The problem seems to be so far, that when modifying the iptables
> structures by adding/flushing the rules, a state can be reached sooner
> or later (indeterministic! smells like race) where the data structure
> becomes invalid (although there are checks in the kernel which would
> prevent that); the result is either ip_tables.c:ipt_do_tables() causing
> an oops due to bad pointer dereferencing (or the kernel freezing w/o
> further notice at all), or the iptables tool being unable to
> retrieve/modify the rules from the kernel (and getting ENOMEM/EINVAL) or
> simply segfaulting due to other inconsistencies with the data...
> 
> I was able to avoid corruption by replacing all vmalloc()/vfree() calls
> by kmalloc()/kfree()'s respectively in ip_tables.c; another variant
> which I was suggested to try helped as well: inserting flush_tlb_all()
> calls after every vmalloc() in said source file;
> 
> I assumed so far, the issue must be alchemy specific, as I experienced
> this on a DbAu1550 board; and Josh had it on a DbAu1100; As it's a
> rather easy to trigger bug, I would have expected to see more bug
> reports if it affected more than just the alchemy's on 2.6.x;
> I tried it with a few kernel revisions from linux-mips' cvs (from 2.6.10
> upto 2.6.12rc2); also tried different compilers (debian's cross-gcc's
> 3.4.4 and 3.3.3), even switched the alchemy to little endian
> operation... all the same; everything else I use on that board has been
> rather stable so far, iptables are the only part which show up this
> vmalloc-issue so far...
> 
> as to reproducing the bug, it's rather easy for me:
> 
> a script repeatedly performing rule modifications should trigger the
> issue rather easily (possibly called over a remote ssh/telnet session,
> in order to create a bit of traffic as well...)
> 
> set -x
> while :
> do
>   iptables -F || exit 1
>   iptables -A INPUT -i lo -j ACCEPT || exit 1
>   iptables -A INPUT -p udp -i eth0 --dport domain -j ACCEPT || exit 1
>   iptables -A INPUT -p udp -i eth0 --dport 6666 -j ACCEPT || exit 1
>   iptables -A INPUT -p tcp -i eth0 --dport ssh -j ACCEPT || exit 1
>   iptables -A INPUT -p tcp -i eth0 --dport http -j ACCEPT || exit 1
>   iptables -A INPUT -p tcp -i eth0 --dport https -j ACCEPT || exit 1
>   iptables -A INPUT -p tcp -i eth0 --dport 3496 -j ACCEPT || exit 1
> done
> 
> or alternatively (requiring state & multiport helpers)
> 
> while :
> do
>   iptables -F
>   iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT  || exit 1
>   iptables -A INPUT -i lo -j ACCEPT || exit 1
>   iptables -A INPUT -p udp -i eth0 -m multiport --dports domain,6666 -j ACCEPT || exit 1
>   iptables -A INPUT -p tcp -i eth0 -m multiport --dports ssh,http,https,3496 -j ACCEPT || exit 1
>   iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT || exit 1
>   iptables -A OUTPUT -o lo -j ACCEPT || exit 1
>   iptables -A OUTPUT -p udp -o eth0 -m multiport --dports ntp -j ACCEPT || exit 1
>   iptables -A OUTPUT -p tcp -o eth0 -m multiport --dports www,https,8444,8445,8446,8454,8455,8456,8464,8465,8466,9225 -j ACCEPT || exit 1
> done
> 
> regards,

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: iptables/vmalloc issues on alchemy

[Dan Malek]

On Apr 27, 2005, at 2:49 PM, Josh Green wrote:

> ...... I was
> planning on doing some additional gdb debugging of the failure
> (especially the initial large MMAP attempt by iptables, which was 1.5GB
> in my case).

Oh wait ....  I found a bug a while ago from someone trying to load
large modules.  There is a problem if the kernel grows to need
additional PTE tables, the top level pointers don't get propagated
correctly and subsequent access by a thread that didn't actually
do the allocation would fail.  I'm looking into this, including your
past message about 64-bit PTEs.

Thanks.


	-- Dan

Re: iptables/vmalloc issues on alchemy

[Herbert Valerio Riedel]

On Wed, 2005-04-27 at 15:06 -0400, Dan Malek wrote:
> On Apr 27, 2005, at 2:49 PM, Josh Green wrote:
> 
> > ...... I was
> > planning on doing some additional gdb debugging of the failure
> > (especially the initial large MMAP attempt by iptables, which was 1.5GB
> > in my case).
> 
> Oh wait ....  I found a bug a while ago from someone trying to load
> large modules.  There is a problem if the kernel grows to need
> additional PTE tables, the top level pointers don't get propagated
> correctly and subsequent access by a thread that didn't actually
> do the allocation would fail.  I'm looking into this, including your
> past message about 64-bit PTEs.

additional note:

the problem only shows up for me only when enabling
CONFIG_64BIT_PHYS_ADDR, in case someone had problems reproducing the
issue...

regards,
-- 
Herbert Valerio Riedel <hvr@hvrlab.org>