Last night Linus bk - netfilter busted?

Last night Linus bk - netfilter busted?

[Dmitry Torokhov]

Hi,

My box gets stuck while booting (actually starting ntpd) whith tonight
pull from Linus. It looks like it is spinning in ipt_do_table when I do
SysRq-P. No call trace though.

Anyone else seeing it? Any ideas?

-- 
Dmitry

[Announce] Stream line modules from .config with streamline_config.pl

[Steven Rostedt]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2651 bytes --]


Hi all,

I'm not sure if someone else did this, but I wrote this simple script that
turns off the modules from your config file that you are not currently
using.  After downloading a new kernel I usually use the .config from my
Debian distribution.  But this usually has way too many modules turned on
and my compile takes much longer than I prefer. So instead of searching
for all the modules that I need in "make *config" I wrote this script to
do it for me.

It's a small simple perl script (that's why it's attached and not on some
web site), and it can easily have options added to it, but it's good
enough for me, so that's where I left it.

Here's how it works and what it does.

1. Boot up the kernel that has the modules you want to load.
2. cd to the directory that holds the source of the kernel you
    just booted.
3. If it's not already there, copy the config file you want to start
    with to this directory, as .config.
4. Make sure all the modules you want/need are loaded.
5. run this script redirecting the output to another file.
6. copy this other file to .config (backing up your old one if you want).
7. Run "make oldconfig".

What this script does is simply, reads all the makefiles in the current
directory tree (find . -name Makefile). Searches each of these Makefiles
for the string "obj-$(CONFIG.*) += ..." (the real regular expression is
more complex, and the file is attached if you want to see it). It then
stores the object files associated to the CONFIG_.* and it handles
multiple lines that end with "\".

Then it runs "lsmod" to get what modules are loaded.

Finally it reads the .config file in the directory and prints it to
standard output.  When it finds a CONFIG_.*=m it checks to see if that
config had an object from lsmod associated to it, if so, then it prints it
as is, otherwise it turns it off.

Here's what I did with my Debian distribution:

  cd /usr/src/linux-2.6.10
  cp /boot/config-2.6.10-1-686-smp .config
  ~/bin/streamline_config > config_strip
  mv .config config_sav
  mv config_strip .config
  make oldconfig

Now this is the config file that I start with when downloading other
kernels.

Obviously if you don't load all the modules you want, or later buy a
new device that needs a module that wasn't loaded, you will need to figure
out what module to add and compile it.  Or use the saved config again
(you did save it?) and do this all over.

Well, do what you want with this, it's an unrescricted license. Comments?

If someone else had already done something like this, let me know. But I
wrote this and a colleage of mine suggested to send it here. So here it
is.

Cheers,

-- Steve


[-- Attachment #2: Type: TEXT/x-perl, Size: 2975 bytes --]

#!/usr/bin/perl -w
#
# Copywrite 2005 - Steven Rostedt
#
# This code has no restrictions and NO WARRANTY. 
#  Use it at your own risk, do what you want with it,
#  Just don't blame me.
#
#  It's simple enough to figure out how this works.
#  If not, then you can ask me at stripconfig@goodmis.org
#  
# What it does?
#
#   If you have installed a Linux kernel from a distribution
#   that turns on way too many modules than you need, and 
#   you only want the modules you use, than this program
#   is perfect for you.
#
#   It gives you the ability to turn off all the modules that are
#   not loaded on your system. 
#
# Howto:
#
#  1. Boot up the kernel that you want to stream line the config on.
#  2. Change directory to the directory holding the source of the 
#       kernel that you just booted.
#  3. Copy the configuraton file to this directory as .config
#  4. Have all your devices that you need modules for connected and
#      operational (make sure that their corresponding modules are loaded)
#  5. Run this script redirecting the output to some other file
#       like config_strip.
#  6. Back up your old config (if you want too).
#  7. copy the config_strip file to .config
#  8. Run "make oldconfig"
#  
#  Now your kernel is ready to be built with only the modules that
#  are loaded.
#
# Here's what I did with my Debian distribution.
#
#    cd /usr/src/linux-2.6.10
#    cp /boot/config-2.6.10-1-686-smp .config
#    ~/bin/streamline_config > config_strip
#    mv .config config_sav
#    mv config_strip .config
#    make oldconfig
# 
my $config = ".config";
my $linuxpath = ".";

open(CIN,$config) || die "Can't open current config file: $config";
my @makefiles = `find $linuxpath -name Makefile`;

my %objects;
my $var;
my $cont = 0;

foreach my $makefile (@makefiles) {
	chomp $makefile;
	
	open(MIN,$makefile) || die "Can't open $makefile";
	while (<MIN>) {
		my $catch = 0;
		
		if ($cont && /(\S.*)$/) {
			$objs = $1;
			$catch = 1;
		}
		$cont = 0;
		
		if (/obj-\$\((CONFIG_[^)]*)\)\s*[+:]?=\s*(.*)/) {
			$var = $1;
			$objs = $2;
			$catch = 1;
		}
		if ($catch) {
			if ($objs =~ m,(.*)/$,) {
				$objs = $1;
				$cont = 1;
			}
			
			foreach my $obj (split /\s+/,$objs) {
				$obj =~ s/-/_/g;
				if ($obj =~ /(.*)\.o$/) {
					$objects{$1} = $var;
				}
			}
		}
	}
	close(MIN);
}

my %modules;

open(LIN,"/sbin/lsmod|") || die "Cant lsmod";
while (<LIN>) {
	next if (/^Module/);  # Skip the first line.
	if (/^(\S+)/) {
		$modules{$1} = 1;
	}
}
close (LIN);

my %configs;
foreach my $module (keys(%modules)) {
	if (defined($objects{$module})) {
		$configs{$objects{$module}} = $module;
	} else {
		print STDERR "$module config not found!!\n";
	}
}

while(<CIN>) {
	if (/^(CONFIG.*)=m/) {
		if (defined($configs{$1})) {
			print;
		} else {
			print "# $1 is not set\n";
		}
	} else {
		print;
	}
}
close(CIN);

Re: Last night Linus bk - netfilter busted?

[Patrick McHardy]

Dmitry Torokhov wrote:
> My box gets stuck while booting (actually starting ntpd) whith tonight
> pull from Linus. It looks like it is spinning in ipt_do_table when I do
> SysRq-P. No call trace though.

Please post your ruleset and .config. A backtrace would also be
useful.

> Anyone else seeing it? Any ideas?

Works fine here. You could try if reverting one of these two patches
helps (second one only if its a SMP box).

ChangeSet@1.2010, 2005-03-09 20:28:17-08:00, bdschuym@pandora.be
   [NETFILTER]: Reduce call chain length in netfilter (take 2)

ChangeSet@1.1982.114.20, 2005-03-03 23:15:48+01:00, ak@suse.de
   [NETFILTER]: Reduce netfilter memory use on MP systems

Regards
Patrick

Re: Last night Linus bk - netfilter busted?

[David S. Miller]

On Fri, 11 Mar 2005 15:00:56 +0100
Patrick McHardy <kaber@trash.net> wrote:

> Works fine here. You could try if reverting one of these two patches
> helps (second one only if its a SMP box).
> 
> ChangeSet@1.2010, 2005-03-09 20:28:17-08:00, bdschuym@pandora.be
>    [NETFILTER]: Reduce call chain length in netfilter (take 2)

It's this change, I know it is, because Linus sees the same problem
on his workstation.

You wouldn't happen to be seeing this problem on a PPC box would
you?  Since Linus's machine is a PPC machine too, that would support
my theory that this could be a compiler issue on that platform.

Damn, wait, Patrick, I think I know what's happening.  The iptables
IPT_* verdicts are dependant upon the NF_* values, and they don't
cope with Bart's changes I bet.  Can you figure out what the exact
error would be?  This kind of issue would explain the looping inside
of ipt_do_table(), wouldn't it?

Re: Last night Linus bk - netfilter busted?

[Dmitry Torokhov]

On Friday 11 March 2005 13:51, David S. Miller wrote:
> On Fri, 11 Mar 2005 15:00:56 +0100
> Patrick McHardy <kaber@trash.net> wrote:
> 
> > Works fine here. You could try if reverting one of these two patches
> > helps (second one only if its a SMP box).
> > 
> > ChangeSet@1.2010, 2005-03-09 20:28:17-08:00, bdschuym@pandora.be
> >    [NETFILTER]: Reduce call chain length in netfilter (take 2)
> 
> It's this change, I know it is, because Linus sees the same problem
> on his workstation.
> 
> You wouldn't happen to be seeing this problem on a PPC box would
> you?  Since Linus's machine is a PPC machine too, that would support
> my theory that this could be a compiler issue on that platform.
> 

No, it is regular PIII laptop (preempt, UP).

> Damn, wait, Patrick, I think I know what's happening.  The iptables
> IPT_* verdicts are dependant upon the NF_* values, and they don't
> cope with Bart's changes I bet.  Can you figure out what the exact
> error would be?  This kind of issue would explain the looping inside
> of ipt_do_table(), wouldn't it?
> 

-- 
Dmitry

Re: Last night Linus bk - netfilter busted?

[Sergey Vlasov]

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

On Fri, 11 Mar 2005 10:51:36 -0800 David S. Miller wrote:

> On Fri, 11 Mar 2005 15:00:56 +0100
> Patrick McHardy <kaber@trash.net> wrote:
> 
> > Works fine here. You could try if reverting one of these two patches
> > helps (second one only if its a SMP box).
> > 
> > ChangeSet@1.2010, 2005-03-09 20:28:17-08:00, bdschuym@pandora.be
> >    [NETFILTER]: Reduce call chain length in netfilter (take 2)
> 
> It's this change, I know it is, because Linus sees the same problem
> on his workstation.
> 
> You wouldn't happen to be seeing this problem on a PPC box would
> you?  Since Linus's machine is a PPC machine too, that would support
> my theory that this could be a compiler issue on that platform.
> 
> Damn, wait, Patrick, I think I know what's happening.  The iptables
> IPT_* verdicts are dependant upon the NF_* values, and they don't
> cope with Bart's changes I bet.  Can you figure out what the exact
> error would be?  This kind of issue would explain the looping inside
> of ipt_do_table(), wouldn't it?

This is not just some buggy code - that patch also breaks interfaces:

include/linux/netfilter_ipv4/ip_tables.h:
#define IPT_RETURN (-NF_MAX_VERDICT - 1)

And this value is visible in userspace.  Therefore we cannot modify
NF_MAX_VERDICT without breaking all existing iptables binaries.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Last night Linus bk - netfilter busted?

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]

David S. Miller wrote:
> Damn, wait, Patrick, I think I know what's happening.  The iptables
> IPT_* verdicts are dependant upon the NF_* values, and they don't
> cope with Bart's changes I bet.  Can you figure out what the exact
> error would be?  This kind of issue would explain the looping inside
> of ipt_do_table(), wouldn't it?

You're right, good catch. IPT_RETURN is interpreted internally by
ip_tables, but since the value changed it isn't recognized by ip_tables
anymore and returned to nf_iterate() as NF_REPEAT. This patch restores
the old value.


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 1594 bytes --]

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2005/03/11 21:41:01+01:00 kaber@coreworks.de 
#   [NETFILTER]: Fix iptables userspace compatibility breakage
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
# include/linux/netfilter_ipv6/ip6_tables.h
#   2005/03/11 21:40:52+01:00 kaber@coreworks.de +1 -1
#   [NETFILTER]: Fix iptables userspace compatibility breakage
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
# include/linux/netfilter_ipv4/ip_tables.h
#   2005/03/11 21:40:52+01:00 kaber@coreworks.de +1 -1
#   [NETFILTER]: Fix iptables userspace compatibility breakage
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
diff -Nru a/include/linux/netfilter_ipv4/ip_tables.h b/include/linux/netfilter_ipv4/ip_tables.h
--- a/include/linux/netfilter_ipv4/ip_tables.h	2005-03-11 21:41:32 +01:00
+++ b/include/linux/netfilter_ipv4/ip_tables.h	2005-03-11 21:41:32 +01:00
@@ -166,7 +166,7 @@
 #define IPT_CONTINUE 0xFFFFFFFF
 
 /* For standard target */
-#define IPT_RETURN (-NF_MAX_VERDICT - 1)
+#define IPT_RETURN (-NF_REPEAT - 1)
 
 /* TCP matching stuff */
 struct ipt_tcp
diff -Nru a/include/linux/netfilter_ipv6/ip6_tables.h b/include/linux/netfilter_ipv6/ip6_tables.h
--- a/include/linux/netfilter_ipv6/ip6_tables.h	2005-03-11 21:41:32 +01:00
+++ b/include/linux/netfilter_ipv6/ip6_tables.h	2005-03-11 21:41:32 +01:00
@@ -166,7 +166,7 @@
 #define IP6T_CONTINUE 0xFFFFFFFF
 
 /* For standard target */
-#define IP6T_RETURN (-NF_MAX_VERDICT - 1)
+#define IP6T_RETURN (-NF_REPEAT - 1)
 
 /* TCP matching stuff */
 struct ip6t_tcp

Re: Last night Linus bk - netfilter busted?

[Herbert Xu]

Patrick McHardy <kaber@trash.net> wrote:
> 
> You're right, good catch. IPT_RETURN is interpreted internally by
> ip_tables, but since the value changed it isn't recognized by ip_tables
> anymore and returned to nf_iterate() as NF_REPEAT. This patch restores
> the old value.

Please fix netfilter_arp while you're at it since it does exactly
the same thing.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Last night Linus bk - netfilter busted?

[Patrick McHardy]

[-- Attachment #1: Type: text/plain, Size: 418 bytes --]

Herbert Xu wrote:
> Patrick McHardy <kaber@trash.net> wrote:
> 
>>You're right, good catch. IPT_RETURN is interpreted internally by
>>ip_tables, but since the value changed it isn't recognized by ip_tables
>>anymore and returned to nf_iterate() as NF_REPEAT. This patch restores
>>the old value.
> 
> 
> Please fix netfilter_arp while you're at it since it does exactly
> the same thing.

New patch attached, thanks.


[-- Attachment #2: x --]
[-- Type: text/plain, Size: 2287 bytes --]

# This is a BitKeeper generated diff -Nru style patch.
#
# ChangeSet
#   2005/03/11 23:54:54+01:00 kaber@coreworks.de 
#   [NETFILTER]: Fix iptables userspace compatibility breakage
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
# include/linux/netfilter_ipv6/ip6_tables.h
#   2005/03/11 23:54:44+01:00 kaber@coreworks.de +1 -1
#   [NETFILTER]: Fix iptables userspace compatibility breakage
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
# include/linux/netfilter_ipv4/ip_tables.h
#   2005/03/11 23:54:44+01:00 kaber@coreworks.de +1 -1
#   [NETFILTER]: Fix iptables userspace compatibility breakage
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
# include/linux/netfilter_arp/arp_tables.h
#   2005/03/11 23:54:44+01:00 kaber@coreworks.de +1 -1
#   [NETFILTER]: Fix iptables userspace compatibility breakage
#   
#   Signed-off-by: Patrick McHardy <kaber@trash.net>
# 
diff -Nru a/include/linux/netfilter_arp/arp_tables.h b/include/linux/netfilter_arp/arp_tables.h
--- a/include/linux/netfilter_arp/arp_tables.h	2005-03-11 23:55:09 +01:00
+++ b/include/linux/netfilter_arp/arp_tables.h	2005-03-11 23:55:09 +01:00
@@ -154,7 +154,7 @@
 #define ARPT_CONTINUE 0xFFFFFFFF
 
 /* For standard target */
-#define ARPT_RETURN (-NF_MAX_VERDICT - 1)
+#define ARPT_RETURN (-NF_REPEAT - 1)
 
 /* The argument to ARPT_SO_GET_INFO */
 struct arpt_getinfo
diff -Nru a/include/linux/netfilter_ipv4/ip_tables.h b/include/linux/netfilter_ipv4/ip_tables.h
--- a/include/linux/netfilter_ipv4/ip_tables.h	2005-03-11 23:55:09 +01:00
+++ b/include/linux/netfilter_ipv4/ip_tables.h	2005-03-11 23:55:09 +01:00
@@ -166,7 +166,7 @@
 #define IPT_CONTINUE 0xFFFFFFFF
 
 /* For standard target */
-#define IPT_RETURN (-NF_MAX_VERDICT - 1)
+#define IPT_RETURN (-NF_REPEAT - 1)
 
 /* TCP matching stuff */
 struct ipt_tcp
diff -Nru a/include/linux/netfilter_ipv6/ip6_tables.h b/include/linux/netfilter_ipv6/ip6_tables.h
--- a/include/linux/netfilter_ipv6/ip6_tables.h	2005-03-11 23:55:09 +01:00
+++ b/include/linux/netfilter_ipv6/ip6_tables.h	2005-03-11 23:55:09 +01:00
@@ -166,7 +166,7 @@
 #define IP6T_CONTINUE 0xFFFFFFFF
 
 /* For standard target */
-#define IP6T_RETURN (-NF_MAX_VERDICT - 1)
+#define IP6T_RETURN (-NF_REPEAT - 1)
 
 /* TCP matching stuff */
 struct ip6t_tcp

Re: Last night Linus bk - netfilter busted?

[Dmitry Torokhov]

On Friday 11 March 2005 17:55, Patrick McHardy wrote:
> Herbert Xu wrote:
> > Patrick McHardy <kaber@trash.net> wrote:
> > 
> >>You're right, good catch. IPT_RETURN is interpreted internally by
> >>ip_tables, but since the value changed it isn't recognized by ip_tables
> >>anymore and returned to nf_iterate() as NF_REPEAT. This patch restores
> >>the old value.
> > 
> > 
> > Please fix netfilter_arp while you're at it since it does exactly
> > the same thing.
> 
> New patch attached, thanks.
> 

If this is of any interest, yesterday's pull from Linux plus this patch
seem to be working fine here.

Thank you.

-- 
Dmitry