iptables not working with monolithic 2.6.26.5 kernel on ARM

iptables not working with monolithic 2.6.26.5 kernel on ARM

[Martin Schwenke]

I'm seeing the following error on my Buffalo Linkstation Pro:

  # iptables -L
  iptables v1.3.6: can't initialize iptables table `filter': No chain/target/match by that name
  Perhaps iptables or your kernel needs to be upgraded.

This is with a monolithic 2.6.26.5 kernel:

  root@slinky:~# uname -a
  Linux slinky 2.6.26.5-dirty #9 PREEMPT Fri Oct 3 13:26:38 EST 2008 armv5tel GNU/Linux

* I think the "-dirty" comes from me having edited
  net/ipv4/netfilter/ip_tables.c to uncomment the #define for
  DEBUG_IP_FIREWALL_USER (see below).  That's certainly the only thing
  different to the standard 2.6..26.5 kernel...

* As you can see, this is an ARM box.

* I've been seeing this problem since I first compiled a 2.6.22 kernel
  for this box.

* I've tried iptables-1.3.8 and it didn't help.  I did a quick diff
  with the latest version iptables/libiptc/libiptc.c in git and can't
  see any changes that would help.

When I strace the above command, I see this:

  ...
  socket(PF_INET, SOCK_RAW, IPPROTO_RAW)  = 3
  getsockopt(3, SOL_IP, 0x40 /* IP_??? */, "filter\0\300\244\306\6\300D\0\0\0\200\\M\304\234\\M\304"..., [84]) = 0
  getsockopt(3, SOL_IP, 0x41 /* IP_??? */, 0x1c0c0, 0xbea17c30) = -1 EINVAL (Invalid argument)
  close(3)                                = 0
  open("/proc/sys/kernel/modprobe", O_RDONLY) = -1 ENOENT (No such file or directory)
  ...

This suggests to me the 2nd call to getsockopt in
iptables/libiptc/libiptc.c:TC_INIT() is failing.

So, I start to poke around.  Does this mean that the table exists?

  # cat /proc/net/ip_tables_names 
  filter

So, I define DEBUG_IP_FIREWALL_USER in the kernel in
net/ipv4/netfilter/ip_tables.c and try again.  Now I get this debug:

  get_entries: 668 != 672

This is generated by this code in get_entries():

        if (*len != sizeof(struct ipt_get_entries) + get.size) {
                duprintf("get_entries: %u != %zu\n",
                         *len, sizeof(get) + get.size);
                return -EINVAL;
        }

I can see code in libiptc.c calculating and passing *len but I don't
understand what's going on enough to know how to debug it.

Any suggestions?

(I'm not on the list, please CC: me...  :-)

peace & happiness,
martin

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Patrick McHardy]

Martin Schwenke wrote:
> I'm seeing the following error on my Buffalo Linkstation Pro:
>
>   # iptables -L
>   iptables v1.3.6: can't initialize iptables table `filter': No chain/target/match by that name
>   Perhaps iptables or your kernel needs to be upgraded.
>
> This is with a monolithic 2.6.26.5 kernel:
>
>   root@slinky:~# uname -a
>   Linux slinky 2.6.26.5-dirty #9 PREEMPT Fri Oct 3 13:26:38 EST 2008 armv5tel GNU/Linux
>
> * I think the "-dirty" comes from me having edited
>   net/ipv4/netfilter/ip_tables.c to uncomment the #define for
>   DEBUG_IP_FIREWALL_USER (see below).  That's certainly the only thing
>   different to the standard 2.6..26.5 kernel...
>
> * As you can see, this is an ARM box.
>
> * I've been seeing this problem since I first compiled a 2.6.22 kernel
>   for this box.
>
> * I've tried iptables-1.3.8 and it didn't help.  I did a quick diff
>   with the latest version iptables/libiptc/libiptc.c in git and can't
>   see any changes that would help.
>
> When I strace the above command, I see this:
>
>   ...
>   socket(PF_INET, SOCK_RAW, IPPROTO_RAW)  = 3
>   getsockopt(3, SOL_IP, 0x40 /* IP_??? */, "filter\0\300\244\306\6\300D\0\0\0\200\\M\304\234\\M\304"..., [84]) = 0
>   getsockopt(3, SOL_IP, 0x41 /* IP_??? */, 0x1c0c0, 0xbea17c30) = -1 EINVAL (Invalid argument)
>   close(3)                                = 0
>   open("/proc/sys/kernel/modprobe", O_RDONLY) = -1 ENOENT (No such file or directory)
>   ...
>
> This suggests to me the 2nd call to getsockopt in
> iptables/libiptc/libiptc.c:TC_INIT() is failing.
>
> So, I start to poke around.  Does this mean that the table exists?
>
>   # cat /proc/net/ip_tables_names 
>   filter
>
> So, I define DEBUG_IP_FIREWALL_USER in the kernel in
> net/ipv4/netfilter/ip_tables.c and try again.  Now I get this debug:
>
>   get_entries: 668 != 672
>
> This is generated by this code in get_entries():
>
>         if (*len != sizeof(struct ipt_get_entries) + get.size) {
>                 duprintf("get_entries: %u != %zu\n",
>                          *len, sizeof(get) + get.size);
>                 return -EINVAL;
>         }
>
> I can see code in libiptc.c calculating and passing *len but I don't
> understand what's going on enough to know how to debug it.
>
> Any suggestions?
>
> (I'm not on the list, please CC: me...  :-)

If I remember correctly we had a similar reports that were related to ABI
mismatches. Please verify that both userspace and kernel are using the
same ABI, in case they are please post your .config.

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Jan Engelhardt]

On Friday 2008-10-03 08:56, Patrick McHardy wrote:
>>
>> So, I define DEBUG_IP_FIREWALL_USER in the kernel in
>> net/ipv4/netfilter/ip_tables.c and try again.  Now I get this debug:
>>
>>   get_entries: 668 != 672
>>
>> This is generated by this code in get_entries():
>>
>>         if (*len != sizeof(struct ipt_get_entries) + get.size) {
>>                 duprintf("get_entries: %u != %zu\n",
>>                          *len, sizeof(get) + get.size);
>>                 return -EINVAL;
>>         }
>>
>> I can see code in libiptc.c calculating and passing *len but I don't
>> understand what's going on enough to know how to debug it.
>>
>> Any suggestions?
>>
>> (I'm not on the list, please CC: me...  :-)
>
> If I remember correctly we had a similar reports that were related to ABI
> mismatches. Please verify that both userspace and kernel are using the
> same ABI, in case they are please post your .config.

Well, the kernel/userspace certainly has support for mixed 64/32 ABI,
so calling for both being the same seems like an unnecessary push.

I might wonder; what does

	#include <stdio.h>
	/* userspace program here */
	printf("%Zu\n", __alignof__(uint64_t));

give on ARM?

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Martin Schwenke]

>>>>> "Jan" == Jan Engelhardt <jengelh@medozas.de> writes:

    Jan> Well, the kernel/userspace certainly has support for mixed
    Jan> 64/32 ABI, so calling for both being the same seems like an
    Jan> unnecessary push.

    Jan> I might wonder; what does

    Jan> 	#include <stdio.h>
    Jan> 	/* userspace program here */
    Jan> 	printf("%Zu\n", __alignof__(uint64_t));

    Jan> give on ARM?

If I compile using the native Debian etch gcc:

  gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)

then it prints 4.

If I use the cross-compiler that I use to compile kernels (and use
--static) then it prints 8.  The version is:

  arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2008q1-126) 4.2.3

So there does seem to be a mismatch.

However, compiling iptables statically with the cross-compiler results
in:

  # ./iptables -L
  getsockopt failed strangely: No such file or directory

Of course, I can't strace it because it is static.  I could compile a
debug version and use gdb... but I'm not clued up enough on ABI issues
to know what to look for...  and I suspect that using gdb when I have
ABI issues is just going to confuse me.

I'm pretty sure I once tried compiling a kernel with the native 4.1.2
GCC but it wouldn't boot... which is annoying because trying it and
failing involves fixing it with a screwdriver!  :-(

Any suggestions before I try that again?  If all common sense points
to me needing to compile kernel's natively then I'll obviously give it
a try.  Alternatively, if there's a small hack that isn't obvious to
me then I'd be happy to hear about it.  That said, I suspect that if I
have ABI issues then I shouldn't be trying small hacks... but then
everything else on the box *seems* to work just fine...  yep, famous
last words...  ;-)

Thanks for any suggestions...

peace & happiness,
martin

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Jan Engelhardt]

On Saturday 2008-10-04 00:21, Martin Schwenke wrote:

>    Jan> 	#include <stdio.h>
>    Jan> 	/* userspace program here */
>    Jan> 	printf("%Zu\n", __alignof__(uint64_t));
>
>    Jan> give on ARM?
>
>If I compile using the native Debian etch gcc:
>
>  gcc (GCC) 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)
>
>then it prints 4.

It's what I thought. There lies the reason for all those strange issues on
embedded devices ;-)

You can change the XT_ALIGN macro in the
kernel/include/linux/netfilter/x_tables.h and
iptables/include/linux/netfilter/x_tables.h to read

	#ifndef roundup
	#define roundup(x, y) ((((x) + ((y) - 1)) / (y)) * (y))
	#endif
	#define XT_ALIGN(s) roundup(s, sizeof(uint64_t))

and try again. There might be more alignment issues hiding, so
don't expect it to fix it all in a magic whizz.

>If I use the cross-compiler that I use to compile kernels (and use
>--static) then it prints 8.  The version is:
>
>  arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2008q1-126) 4.2.3
>
>So there does seem to be a mismatch.
>
>However, compiling iptables statically with the cross-compiler results
>in:
>
>  # ./iptables -L
>  getsockopt failed strangely: No such file or directory
>
>Of course, I can't strace it because it is static.

stracing is independent of staticness/dynamicness.

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Patrick McHardy]

Jan Engelhardt wrote:
> On Friday 2008-10-03 08:56, Patrick McHardy wrote:
>> If I remember correctly we had a similar reports that were related to ABI
>> mismatches. Please verify that both userspace and kernel are using the
>> same ABI, in case they are please post your .config.
> 
> Well, the kernel/userspace certainly has support for mixed 64/32 ABI,
> so calling for both being the same seems like an unnecessary push.

My knowledge of ARM is quite limited, so you're probably right
that its not necessary to have all of userspace match the kernel
ABI. But I think its necessary for iptables to avoid alignment
differences.

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Patrick McHardy]

Jan Engelhardt wrote:
> You can change the XT_ALIGN macro in the
> kernel/include/linux/netfilter/x_tables.h and
> iptables/include/linux/netfilter/x_tables.h to read
> 
> 	#ifndef roundup
> 	#define roundup(x, y) ((((x) + ((y) - 1)) / (y)) * (y))
> 	#endif
> 	#define XT_ALIGN(s) roundup(s, sizeof(uint64_t))
> 
> and try again. There might be more alignment issues hiding, so
> don't expect it to fix it all in a magic whizz.

An idea to avoid these problems in the future would be to change
the kernel so it accepts stricter alignment than what we currently
use and after a grace period (maybe 1-2 years) change XT_ALIGN in
userspace so it starts using u64 or simply 8 byte alignment
everywhere.

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Jan Engelhardt]

On Sunday 2008-10-05 09:24, Patrick McHardy wrote:
> Jan Engelhardt wrote:
>> You can change the XT_ALIGN macro in the
>> kernel/include/linux/netfilter/x_tables.h and
>> iptables/include/linux/netfilter/x_tables.h to read
>> 
>>  #ifndef roundup
>>  #define roundup(x, y) ((((x) + ((y) - 1)) / (y)) * (y))
>>  #endif
>>  #define XT_ALIGN(s) roundup(s, sizeof(uint64_t))
>> 
>> and try again. There might be more alignment issues hiding, so
>> don't expect it to fix it all in a magic whizz.
>
> An idea to avoid these problems in the future would be to change
> the kernel so it accepts stricter alignment than what we currently
> use and after a grace period (maybe 1-2 years) change XT_ALIGN in
> userspace so it starts using u64 or simply 8 byte alignment
> everywhere.

I'd rather be interested in what architectures actually have
alignof(uint64_t) == 4, besides the native ARM gcc that's used here.
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Patrick McHardy]

Jan Engelhardt wrote:
> On Sunday 2008-10-05 09:24, Patrick McHardy wrote:
>> Jan Engelhardt wrote:
>>> You can change the XT_ALIGN macro in the
>>> kernel/include/linux/netfilter/x_tables.h and
>>> iptables/include/linux/netfilter/x_tables.h to read
>>>
>>>  #ifndef roundup
>>>  #define roundup(x, y) ((((x) + ((y) - 1)) / (y)) * (y))
>>>  #endif
>>>  #define XT_ALIGN(s) roundup(s, sizeof(uint64_t))
>>>
>>> and try again. There might be more alignment issues hiding, so
>>> don't expect it to fix it all in a magic whizz.
>> An idea to avoid these problems in the future would be to change
>> the kernel so it accepts stricter alignment than what we currently
>> use and after a grace period (maybe 1-2 years) change XT_ALIGN in
>> userspace so it starts using u64 or simply 8 byte alignment
>> everywhere.
> 
> I'd rather be interested in what architectures actually have
> alignof(uint64_t) == 4, besides the native ARM gcc that's used here.

CRIS isn't requiring any alignment AFAIK, so everything is just
byte aligned. I don't know about any other architectures not
using 8 byte alignment.

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Martin Schwenke]

>>>>> "Jan" == Jan Engelhardt <jengelh@medozas.de> writes:

    Jan> You can change the XT_ALIGN macro in the
    Jan> kernel/include/linux/netfilter/x_tables.h and
    Jan> iptables/include/linux/netfilter/x_tables.h to read

    Jan> 	#ifndef roundup
    Jan> 	#define roundup(x, y) ((((x) + ((y) - 1)) / (y)) * (y))
    Jan> 	#endif
    Jan> 	#define XT_ALIGN(s) roundup(s, sizeof(uint64_t))

    Jan> and try again. There might be more alignment issues hiding,
    Jan> so don't expect it to fix it all in a magic whizz.

Hmmm... that didn't do it.  I'm still seeing the same problem.

I've made the above change in both places (kernel tree, iptables
source).  I compiled the kernel with the cross compiler, and tried
both a statically linked, cross-compiled iptables binary, and natively
compiled complete iptables Debian package.

Weird...  but thanks for the possible workaround!

    >> Of course, I can't strace it because it is static.

    Jan> stracing is independent of staticness/dynamicness.

Oops... could I be thinking of ltrace or should I shut up before I
embarrass myself more?  ;-)  Whatever the case, strace doesn't work
on my static cross-compiled binary...  :-(

peace & happiness,
martin

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Jan Engelhardt]

On Sunday 2008-10-05 21:34, Martin Schwenke wrote:
>    >> Of course, I can't strace it because it is static.
>
>    Jan> stracing is independent of staticness/dynamicness.
>
>Oops... could I be thinking of ltrace or should I shut up before I
>embarrass myself more?  ;-)  Whatever the case, strace doesn't work
>on my static cross-compiled binary...  :-(

Hey, does your kernel's .config even have a CONFIG_COMPAT?

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Martin Schwenke]

>>>>> "Jan" == Jan Engelhardt <jengelh@medozas.de> writes:

    Jan> Hey, does your kernel's .config even have a CONFIG_COMPAT?

Nope.

I spent some time the other day looking at what setting CONFIG_COMPAT
means for netfilter.  I then tried to work out how to get it set on
the off chance that it might fix my problems... but I can't find any
info about it...  :-(

What does CONFIG_COMPAT mean?

peace & happiness,
martin

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Jan Engelhardt]

On Sunday 2008-10-05 23:02, Martin Schwenke wrote:
>
>What does CONFIG_COMPAT mean?

Support of an ABI (specifically one only) that is different
from the kernel. Usually used to implement 32-bit userspace
for 64-bit kernels.

Re: iptables not working with monolithic 2.6.26.5 kernel on ARM

[Martin Schwenke]

>>>>> "Jan" == Jan Engelhardt <jengelh@medozas.de> writes:

    Jan> On Sunday 2008-10-05 23:02, Martin Schwenke wrote:

    >> What does CONFIG_COMPAT mean?

    Jan> Support of an ABI (specifically one only) that is different
    Jan> from the kernel. Usually used to implement 32-bit userspace
    Jan> for 64-bit kernels.

That's why I thought you asked...  :-)

I thought that some of my problems stemmed from the cross-compiler and
native compiler doing things differently.  However, I'm now running a
natively compiled kernel (yep, it took a while to build, but it
boots... so no screwdriver ;-) and am still seeing the same ABI
problems, with the same debug message:

  get_entries: 668 != 672

The ARM CPU looks to be 32-bit (Wikipedia suggests that all ARM CPUs
32-bit).  The vmlinux that gets generated by both the cross-compiler
and the native compiler is 32-bit.  So, I guess this has nothing to do
with 64-bit-ness?

So, if its possible, I'm more confused than I was before!  :-)

This has obviously got something to do with alignment.  However, I'm a
bit too naive to understand how the alignment is getting messed up
when the kernel and userspace are built with the same compiler.  That
said, I don't know what compiler libc was built with... but I assume
the Debian guys built it with the stock compiler for the distro...

Any other ideas?  :-)

peace & happiness,
martin