bugreport ipt_CLUSTERIP

bugreport ipt_CLUSTERIP

[Maik Hentsche]

Hi!
I tried ipt_CLUSTERIP and it crashed with the appended error message. 
Unfortunatelly, I was not able to get an account for bugzilla (neither 
automaticall nor by writing the administrator) for 2 weeks, so I try to 
put it here.

so long
Maik


Unable to handle kernel NULL pointer dereference at 0000000000000053 RIP:
<ffffffff880c46da>{:ip_tables:ipt_do_table+188}
PGD 7e53c067 PUD 7d172067 PMD 0
Oops: 0000 [1] SMP
CPU 1
Modules linked in: af_packet ipt_CLUSTERIP iptable_filter ip_tables 
x_tables ipv6 hw_random i2c_amd756 i2c_amd8111 i2c_core generic amd74xx 
e1000 tg3 ide_cd cdrom rtc unix
Pid: 3068, comm: rcp.statd Not tainted 2.6.17.3-egde #1
RIP: 0010:[<ffffffff880c46da>]

Re: bugreport ipt_CLUSTERIP

[Patrick McHardy]

Maik Hentsche wrote:
> Hi!
> I tried ipt_CLUSTERIP and it crashed with the appended error message.
> Unfortunatelly, I was not able to get an account for bugzilla (neither
> automaticall nor by writing the administrator) for 2 weeks, so I try to
> put it here.

Crash reports are better sent to mailing lists anyway

> Unable to handle kernel NULL pointer dereference at 0000000000000053 RIP:
> <ffffffff880c46da>{:ip_tables:ipt_do_table+188}
> PGD 7e53c067 PUD 7d172067 PMD 0
> Oops: 0000 [1] SMP
> CPU 1
> Modules linked in: af_packet ipt_CLUSTERIP iptable_filter ip_tables
> x_tables ipv6 hw_random i2c_amd756 i2c_amd8111 i2c_core generic amd74xx
> e1000 tg3 ide_cd cdrom rtc unix
> Pid: 3068, comm: rcp.statd Not tainted 2.6.17.3-egde #1
> RIP: 0010:[<ffffffff880c46da>]

Are you sure this is caused by ipt_CLUSTERIP? The crash is within
iptables. What options did you use to trigger this? Did you apply
any patches to your kernel?

Re: bugreport ipt_CLUSTERIP

[Maik Hentsche]

Zitat von Patrick McHardy <kaber@trash.net>:

> Crash reports are better sent to mailing lists anyway

I'll keep that in mind.

> Are you sure this is caused by ipt_CLUSTERIP?

No, I'm not.

> What options did you use to trigger this?

iptables -D INPUT -i eth1 -d 172.20.251.25  -j CLUSTERIP --new
--hashmode sourceip-sourceport --clustermac 01:00:5e:12:34:56
--total-nodes 2 --local-node 2

The first time, I tried this to delete the appropiate rule, it directly
crashed. Unfortunatelly, I do not have the related Oops message, since
I thought it might occur again and wanted to log via serial console.
The second time, the already mentioned error occured, when I rebooted
my machine and unfortunatelly after I had plugged the serial cable off
:o(

> Did you apply any patches to your kernel?

No, plain vanilla kernel 2.6.17.3.

so long
Maik

Re: bugreport ipt_CLUSTERIP

[Patrick McHardy]

Maik Hentsche wrote:
> Zitat von Patrick McHardy <kaber@trash.net>:
> 
>>What options did you use to trigger this?
> 
> 
> iptables -D INPUT -i eth1 -d 172.20.251.25  -j CLUSTERIP --new
> --hashmode sourceip-sourceport --clustermac 01:00:5e:12:34:56
> --total-nodes 2 --local-node 2
> 
> The first time, I tried this to delete the appropiate rule, it directly
> crashed. Unfortunatelly, I do not have the related Oops message, since
> I thought it might occur again and wanted to log via serial console.
> The second time, the already mentioned error occured, when I rebooted
> my machine and unfortunatelly after I had plugged the serial cable off
> :o(


I tried to reproduce this (by first adding your rule, then deleting it,
adding it again, rebooting), but got no crash. CLUSTERIP keeps global
state, so it might be that some previous actions using it contributed
to the crash, although its strange that it crashed in ip_tables, not
in CLUSTERIP itself. Did you added/deleted/flushed/... any CLUSTERIP
rules before the crashes? If you can find a way to reproducibly cause
the crash it would help a lot.

Re: bugreport ipt_CLUSTERIP

[Maik Hentsche]

Patrick McHardy <kaber@trash.net> wrote:

> I tried to reproduce this (by first adding your rule, then deleting
> it, adding it again, rebooting), but got no crash.

It did not happen again for me either, even though, I tried a lot to
reproduce it.

> Did you added/deleted/flushed/... any CLUSTERIP rules before the crashes?

I added the rule, then tried to ping the virtual IP, which did not
work (no arp response). Then I deleted the rule and did nothing else
with CLUSTERIP. The machine was used by another developer a week ago
and I do not know, if he did anything with iptables, but he hardly used
CLUSTERIP. I'm very sorry to not be able to help more.

so long
Maik

-- 
Der Verstand ist wie eine Fahrkarte. Sie hat nur Sinn wenn man sie
benutzt. (Ernst R. Hauschka (*1926), deutscher Essayist,
Aphoristiker und Bibliothekar)

Re: bugreport ipt_CLUSTERIP

[Patrick McHardy]

Maik Hentsche wrote:
> I added the rule, then tried to ping the virtual IP, which did not
> work (no arp response). Then I deleted the rule and did nothing else
> with CLUSTERIP. The machine was used by another developer a week ago
> and I do not know, if he did anything with iptables, but he hardly used
> CLUSTERIP. I'm very sorry to not be able to help more.


Thanks anyway. I'll try some more to reproduce it ..

Re: bugreport ipt_CLUSTERIP

[Maik Hentsche]

Zitat von Patrick McHardy <kaber@trash.net>:

> Thanks anyway. I'll try some more to reproduce it ..

It happened again and now I have a clue, what _might_ trigger it. I did
a few (around 15) inserting/deleting/inserting/flushing-cycles. All the
time, a ping to the virtual IP was running. Most important, another
programm held the virtual IP when it crashed (twice), once keepalived
and once heartbeatd. Without them, the error did not occur. I appended
a console log. Note, that, even though, there is a "hash=1 ct_hash=1
responsible", which seems to come from CLUSTERIP, I do not get an arp
response (and as far as I understood, CLUSTERIP is supposed to send
one).

HTH & so long
Maik


hash=1 ct_hash=1 responsible
hash=1 ct_hash=1 responsible
IPVS: Registered protocols (TCP, UDP, AH, ESP)
IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
IPVS: ipvs loaded.
Unable to handle kernel NULL pointer dereference at 0000000000000000
RIP:
<ffffffff88135022>{:ipt_CLUSTERIP:__clusterip_config_find+34}
PGD 798d2067 PUD 7c090067 PMD 0
Oops: 0000 [1] PREEMPT SMP
CPU 1
Modules linked in: ip_vs ipt_CLUSTERIP ipt_REJECT xt_state ip_conntrack
nfnetlink xt_tcpudp iptable_filter ip_tables ipt_ULOG x_tables ipv3Pid:
0, comm: swapper Not tainted 2.6.17.3 #1
RIP: 0010:[<ffffffff88135022>]
<ffffffff88135022>{:ipt_CLUSTERIP:__clusterip_config_find+34}
RSP: 0018:ffff810001377d60  EFLAGS: 00010a16
RAX: 0000000000000000 RBX: 0000000016fb14ac RCX: ffff81007cb50000
RDX: 0000000000000000 RSI: ffff810001377e48 RDI: 0000000016fb14ac
RBP: ffff81007d2e3410 R08: ffffffff803aadc0 R09: 0000000000000205
R10: ffff81007cb50168 R11: 0000000000000001 R12: ffff81007cb50000
R13: ffff81007d2e3418 R14: 0000000000000000 R15: 0000000000000001
FS:  00002b8f4febc6d0(0000) GS:ffff81007e3afa40(0000)
knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000007d3d4000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffff810001370000, task
ffff810001344080)
Stack: ffffffff881357a4 ffff81000132bac0 ffff810001377df8
ffffffff8052b6f0
       0000000080000000 ffff81007cb50000 ffffffff803bce1e
ffff810001377e48
       0000000000000001 ffff810001377e48
Call Trace: <IRQ> <ffffffff881357a4>{:ipt_CLUSTERIP:arp_mangle+116}
       <ffffffff803bce1e>{nf_iterate+94}
<ffffffff803aadc0>{dev_queue_xmit+0}
       <ffffffff803bcee7>{nf_hook_slow+135}
<ffffffff803aadc0>{dev_queue_xmit+0}
       <ffffffff803e690e>{arp_xmit+62}
<ffffffff803e628c>{arp_solicit+396}
       <ffffffff803b040e>{neigh_timer_handler+654}
<ffffffff802454d6>{hrtimer_run_queues+214}
       <ffffffff803b0180>{neigh_timer_handler+0}
<ffffffff80237777>{run_timer_softirq+375}
       <ffffffff8023301b>{__do_softirq+91}
<ffffffff80207ae0>{default_idle+0}
       <ffffffff8020ade2>{call_softirq+30}
<ffffffff8020cc51>{do_softirq+49}
       <ffffffff8023317f>{irq_exit+63}
<ffffffff80207ae0>{default_idle+0}
       <ffffffff8020a786>{apic_timer_interrupt+98} <EOI>
<ffffffff80412105>{_spin_unlock_irq+21}
       <ffffffff8041021a>{thread_return+186}
<ffffffff80207b0f>{default_idle+47}
       <ffffffff80207d08>{cpu_idle+104}

Code: 48 8b 00 0f 18 08 48 81 fa 00 75 13 88 75 e5 31 c0 c3 66 66
RIP <ffffffff88135022>{:ipt_CLUSTERIP:__clusterip_config_find+34} RSP
<ffff810001377d60>
CR2: 0000000000000000
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!