kernel oops - do_ip_vs_get_ctl

kernel oops - do_ip_vs_get_ctl

[Ryan O'Hara]

[-- Attachment #1: Type: text/plain, Size: 246 bytes --]


I frequently get a kernel oops in do_ip_vs_get_ctl when starting 
keepalived on a 3.3.0 kernel. I'm attaching the trace from 
/var/log/messages. Has anyone encountered this problem and if so is 
there a patch available? Much appreciated.

Ryan


[-- Attachment #2: oops.txt --]
[-- Type: text/plain, Size: 5490 bytes --]

Mar 25 22:04:32 rocket-01 kernel: [  232.003309] nf_conntrack version 0.5.0 (7969 buckets, 31876 max)
Mar 25 22:04:32 rocket-01 kernel: [  232.015724] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Mar 25 22:04:32 rocket-01 kernel: [  232.019222] BUG: unable to handle kernel NULL pointer dereference at 00000000000005f8
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] IP: [<ffffffffa00bf6b9>] do_ip_vs_get_ctl+0x5d9/0x960 [ip_vs]
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] PGD 3b98e067 PUD 3b999067 PMD 0 
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] Oops: 0000 [#1] SMP 
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] CPU 0 
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] Modules linked in: ip_vs(+) nf_conntrack libcrc32c lockd i2c_piix4 joydev i2c_core virtio_net virtio_balloon microcode sunrpc virtio_blk [last unloaded: scsi_wait_scan]
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] 
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] Pid: 868, comm: keepalived Not tainted 3.3.0-4.fc16.x86_64 #1 Red Hat KVM
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] RIP: 0010:[<ffffffffa00bf6b9>]  [<ffffffffa00bf6b9>] do_ip_vs_get_ctl+0x5d9/0x960 [ip_vs]
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] RSP: 0018:ffff88003b9f3ca8  EFLAGS: 00010293
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] RAX: 0000000000001000 RBX: 0000000000000481 RCX: 0000000000000001
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] RDX: ffff88003b9f3fd8 RSI: 000000000062ba20 RDI: ffffffffa00d10c0
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] RBP: ffff88003b9f3e48 R08: 0000000000000000 R09: 0000000000000000
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] R10: 0000000000000074 R11: 0000000000001000 R12: 0000000000000000
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] R13: ffffffff81f72240 R14: 00000000fffffe00 R15: 000000000062ba14
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] FS:  00007f53e38277c0(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] CR2: 00000000000005f8 CR3: 000000003b99c000 CR4: 00000000000006f0
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] Process keepalived (pid: 868, threadinfo ffff88003b9f2000, task ffff88003b924590)
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] Stack:
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  ffff88000000000c ffffffff00000001 ffff88003b9f3ed4 ffffffff814d3173
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  0000100000010201 ffff88003c826700 0000000000000481 00007fff5552827c
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  ffff88003b9f3ea8 ffffffff81522b8d ffff88003b9f3b78 000000003b9f3f18
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] Call Trace:
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff814d3173>] ? release_sock+0xe3/0x110
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff81522b8d>] ? do_ip_getsockopt.part.1+0x7d/0x650
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff8114671d>] ? handle_pte_fault+0x2ad/0xad0
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff814ced7b>] ? move_addr_to_user+0xbb/0xd0
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff8116c22f>] ? kmem_cache_free+0x2f/0x130
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff811bf5a1>] ? fsnotify_clear_marks_by_inode+0x31/0xf0
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff8116c22f>] ? kmem_cache_free+0x2f/0x130
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff8126be84>] ? avc_has_perm_flags+0x74/0x90
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff81512ce6>] ? nf_sockopt_find.part.0+0xa6/0x120
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff81512f38>] nf_sockopt+0x98/0xc0
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff81512f78>] nf_getsockopt+0x18/0x20
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff81523335>] ip_getsockopt+0xa5/0x110
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff81543716>] raw_getsockopt+0x16/0x30
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff814d2924>] sock_common_getsockopt+0x14/0x20
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff814d1db3>] sys_getsockopt+0x73/0xe0
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  [<ffffffff815fc029>] system_call_fastpath+0x16/0x1b
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] Code: 49 81 f8 00 01 00 00 0f 85 18 ff ff ff 45 31 f6 e9 45 fb ff ff 8b 05 17 28 01 00 c7 85 80 fe ff ff 01 02 01 00 89 85 84 fe ff ff <41> 8b 84 24 f8 05 00 00 89 85 88 fe ff ff e9 91 fc ff ff 48 8d 
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] RIP  [<ffffffffa00bf6b9>] do_ip_vs_get_ctl+0x5d9/0x960 [ip_vs]
Mar 25 22:04:32 rocket-01 kernel: [  232.020016]  RSP <ffff88003b9f3ca8>
Mar 25 22:04:32 rocket-01 kernel: [  232.020016] CR2: 00000000000005f8
Mar 25 22:04:32 rocket-01 kernel: [  232.086854] ---[ end trace 3203677ddb800294 ]---
Mar 25 22:04:32 rocket-01 kernel: [  232.088977] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Mar 25 22:04:32 rocket-01 kernel: [  232.090732] IPVS: Creating netns size=2088 id=0
Mar 25 22:04:32 rocket-01 kernel: [  232.092253] IPVS: ipvs loaded.

Re: kernel oops - do_ip_vs_get_ctl

[Julian Anastasov]

	Hello,

On Fri, 20 Apr 2012, Ryan O'Hara wrote:

> 
> I frequently get a kernel oops in do_ip_vs_get_ctl when starting keepalived on
> a 3.3.0 kernel. I'm attaching the trace from /var/log/messages. Has anyone
> encountered this problem and if so is there a patch available? Much
> appreciated.

	Is it happening while ip_vs module is loading?
nf_register_sockopt is one of the first thing that is
initialized, may be the GET operation accesses something
from net->ipvs that is not initialized yet. As I can not
see properly the exact place from the oops, I'll try to
analyze do_ip_vs_get_ctl for such problem. May be other
folks will be faster in tracking the right command and
position in the source code.

	May be we have to split ip_vs_control_init to
two parts so that sockopts are the last thing to register
on init.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re[3]: kernel oops - do_ip_vs_get_ctl

[Hans Schillstrom]

Hello 
>Subject: Re[2]: kernel oops - do_ip_vs_get_ctl
>
>Hello,
>
>On Fri, 20 Apr 2012, Hans Schillstrom wrote:
>
>> >On Fri, 20 Apr 2012, Ryan O'Hara wrote:
>> >
>> >> 
>> >> I frequently get a kernel oops in do_ip_vs_get_ctl when starting keepalived on
>> >> a 3.3.0 kernel. I'm attaching the trace from /var/log/messages. Has anyone
>> >> encountered this problem and if so is there a patch available? Much
>> >> appreciated.
>> >
[snip]

>> Do you prepare a patch or should I do it  ?
>
>	I'm stopping doing more patches until Simon takes
>the previous changes, so that we can use some fresh tree.
>You can try fixing this problem if you think you have
>recent changes.
>
OK I'll take care of this and double check with Simon that we have the same view of patches

Regards 
Hans Schillstrom <hans@schillstrom.com>

Re: kernel oops - do_ip_vs_get_ctl

[Simon Horman]

On Sat, Apr 21, 2012 at 12:11:49PM +0200, Hans Schillstrom wrote:
> Hello 
> >Subject: Re[2]: kernel oops - do_ip_vs_get_ctl
> >
> >Hello,
> >
> >On Fri, 20 Apr 2012, Hans Schillstrom wrote:
> >
> >> >On Fri, 20 Apr 2012, Ryan O'Hara wrote:
> >> >
> >> >> 
> >> >> I frequently get a kernel oops in do_ip_vs_get_ctl when starting keepalived on
> >> >> a 3.3.0 kernel. I'm attaching the trace from /var/log/messages. Has anyone
> >> >> encountered this problem and if so is there a patch available? Much
> >> >> appreciated.
> >> >
> [snip]
> 
> >> Do you prepare a patch or should I do it  ?
> >
> >	I'm stopping doing more patches until Simon takes
> >the previous changes, so that we can use some fresh tree.
> >You can try fixing this problem if you think you have
> >recent changes.
> >
> OK I'll take care of this and double check with Simon that we have the
> same view of patches

Hi,

sorry for not being a little more attentive to patches.

I have now picked up all the patches that seem to have consensus.
Those that seem critical I have pushed into ipvs with a CC: stable@
and sent a pull request to Pablo to consider them for 3.4.

There are two such patches and the head of the ipvs tree now looks like
this:

0cc4789 ipvs: fix crash in ip_vs_control_net_cleanup on unload
bd7dc1c netfilter: ipvs: Verify that IP_VS protocol has been registered

Those that seemed less critical where the GFP_ATOMIC changes, one
from Sasha and 6 from Julian. The head of the ipvs-next tree now looks like
this:

663f4b2 netfilter: ipvs: use GFP_KERNEL allocation where possible
b5cfd04 ipvs: SH scheduler does not need GFP_ATOMIC allocation
5b3b290 ipvs: LBLCR scheduler does not need GFP_ATOMIC allocation on init
c087c6f ipvs: WRR scheduler does not need GFP_ATOMIC allocation
8cfaf8d ipvs: DH scheduler does not need GFP_ATOMIC allocation
e7c6390 ipvs: LBLC scheduler does not need GFP_ATOMIC allocation on init
8f78609 ipvs: timeout tables do not need GFP_ATOMIC allocation

Please let me know if there are any other patches you would like
merged at this time.

Re: kernel oops - do_ip_vs_get_ctl

[Julian Anastasov]

	Hello,

On Mon, 23 Apr 2012, Simon Horman wrote:

> Hi,
> 
> sorry for not being a little more attentive to patches.
> 
> I have now picked up all the patches that seem to have consensus.
> Those that seem critical I have pushed into ipvs with a CC: stable@
> and sent a pull request to Pablo to consider them for 3.4.
> 
> There are two such patches and the head of the ipvs tree now looks like
> this:
> 
> 0cc4789 ipvs: fix crash in ip_vs_control_net_cleanup on unload
> bd7dc1c netfilter: ipvs: Verify that IP_VS protocol has been registered
> 
> Those that seemed less critical where the GFP_ATOMIC changes, one
> from Sasha and 6 from Julian. The head of the ipvs-next tree now looks like
> this:
> 
> 663f4b2 netfilter: ipvs: use GFP_KERNEL allocation where possible
> b5cfd04 ipvs: SH scheduler does not need GFP_ATOMIC allocation
> 5b3b290 ipvs: LBLCR scheduler does not need GFP_ATOMIC allocation on init
> c087c6f ipvs: WRR scheduler does not need GFP_ATOMIC allocation
> 8cfaf8d ipvs: DH scheduler does not need GFP_ATOMIC allocation
> e7c6390 ipvs: LBLC scheduler does not need GFP_ATOMIC allocation on init
> 8f78609 ipvs: timeout tables do not need GFP_ATOMIC allocation
> 
> Please let me know if there are any other patches you would like
> merged at this time.

	These two patches are also fixes but may be the 2nd
patch is too long for fix:

ipvs: reset ipvs pointer in netns
ipvs: fix app registration in netns

	If it looks too long for fix we can push some
simple check for net->ipvs in __ip_vs_ftp_init, so that
we do not oops when IPVS core is compiled in kernel.
Even if smaller version is sent to stable kernels,
I prefer "ipvs: fix app registration in netns" to be
applied at least for net-next. May be I should split
this 2nd patch as two-line fix + additional change
for net-next?

	Hans should provide similar two-line fixes for
LBLC and LBLCR. And one for latest crash report.

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: kernel oops - do_ip_vs_get_ctl

[Simon Horman]

On Mon, Apr 23, 2012 at 10:55:30PM +0300, Julian Anastasov wrote:
> 
> 	Hello,
> 
> On Mon, 23 Apr 2012, Simon Horman wrote:
> 
> > Hi,
> > 
> > sorry for not being a little more attentive to patches.
> > 
> > I have now picked up all the patches that seem to have consensus.
> > Those that seem critical I have pushed into ipvs with a CC: stable@
> > and sent a pull request to Pablo to consider them for 3.4.
> > 
> > There are two such patches and the head of the ipvs tree now looks like
> > this:
> > 
> > 0cc4789 ipvs: fix crash in ip_vs_control_net_cleanup on unload
> > bd7dc1c netfilter: ipvs: Verify that IP_VS protocol has been registered
> > 
> > Those that seemed less critical where the GFP_ATOMIC changes, one
> > from Sasha and 6 from Julian. The head of the ipvs-next tree now looks like
> > this:
> > 
> > 663f4b2 netfilter: ipvs: use GFP_KERNEL allocation where possible
> > b5cfd04 ipvs: SH scheduler does not need GFP_ATOMIC allocation
> > 5b3b290 ipvs: LBLCR scheduler does not need GFP_ATOMIC allocation on init
> > c087c6f ipvs: WRR scheduler does not need GFP_ATOMIC allocation
> > 8cfaf8d ipvs: DH scheduler does not need GFP_ATOMIC allocation
> > e7c6390 ipvs: LBLC scheduler does not need GFP_ATOMIC allocation on init
> > 8f78609 ipvs: timeout tables do not need GFP_ATOMIC allocation
> > 
> > Please let me know if there are any other patches you would like
> > merged at this time.
> 
> 	These two patches are also fixes but may be the 2nd
> patch is too long for fix:
> 
> ipvs: reset ipvs pointer in netns
> ipvs: fix app registration in netns

Thanks, I have these now.

> 	If it looks too long for fix we can push some
> simple check for net->ipvs in __ip_vs_ftp_init, so that
> we do not oops when IPVS core is compiled in kernel.
> Even if smaller version is sent to stable kernels,
> I prefer "ipvs: fix app registration in netns" to be
> applied at least for net-next. May be I should split
> this 2nd patch as two-line fix + additional change
> for net-next?
> 
> 	Hans should provide similar two-line fixes for
> LBLC and LBLCR. And one for latest crash report.

And I am awaiting consensus on the following fixes from Hans.

ipvs: null check of net->ipvs in lblc(r) shedulers
ipvs: take care of return value from protocol init_netns
ipvs: kernel oops - do_ip_vs_get_ctl