Re: BUG: 4.14.11 unable to handle kernel NULL pointer dereference in xfrm_lookup

[Tobias Hommel <netdev-list@genoetigt.de>]

[-- Attachment #1: Type: text/plain, Size: 2375 bytes --]

On Tue, Jan 09, 2018 at 03:49:21PM +0100, Tobias Hommel wrote:
> On Tue, Jan 09, 2018 at 10:26:24AM +0100, Steffen Klassert wrote:
> > On Tue, Jan 09, 2018 at 10:06:51AM +0100, Tobias Hommel wrote:
> > > > 
> > > > You have CONFIG_INET_ESP_OFFLOAD enabled, this is new maybe it
> > > > still has some problems. You should not hit an offload codepath
> > > > because all your SAs are configured with UDP encapsulation which
> > > > is still not supported with offload.
I ran some new tests with 4.14.12. This time I removed encap=yes from the
strongswan config so I have plain ESP tunnels, without UDP encapsulation. Just
to be sure. It still crashes, the attached panic.noencap.log is pretty much
the same as the logs before.

> > > > 
> > > > Please try to disable GRO on both interfaces and see what happens:
> > > > 
> > > > ethtool -K eth0 gro off
> > > > ethtool -K eth1 gro off
> > > I actually already tried that with only eth1 off, to verify I turned offloading
> > > off for both interfaces. The same problem: see attached panic.gro_off.log
> > > 
> > > > 
> > > > Then disable CONFIG_INET_ESP_OFFLOAD and try again.
> > > Rebuild with CONFIG_INET_ESP_OFFLOAD disabled, same problem: see attached
> > > panic.esp_offload_disabled.log
> > 
> > So ESP offload is not the problem. Next thing that comes to my mind
> > is the flowcache removal, this was introduced with v4.14.
> > 
> > > 
> > > > 
> > > > This should show us if this feature is responsible for the bug.
> > > > 
> > > 
> > > I will try narrowing down the problem by trying out some older kernels for now.
> > 
> > Thanks!
> > 
> > Let me know about the results.
> 
> I copied the config from my 4.14.12 sources to a fresh 4.13.16 source tree, ran
> `make olddefconfig` and built a new kernel.
> The kernel config is attached as kernel-4.13.16.config.
> The panic*.log files are kernel logs from different crashes of this 4.13.16
> kernel, but all from the same scenario as before.
> I also enabled CONFIG_DEBUG_INFO, so if any disassemblies are required, I'd be
> happy to provide them.
> 
> So, the system still crashes, but the traces are completely different from
> those with 4.14.12. This time there are also WARNINGs and "refcnt: -1" messages
> sometimes before the actual panic, so not sure if there is maybe some other
> problem. Still, the crashes all seem to be related to ip routing somehow.

[-- Attachment #2: panic.noencap.log --]
[-- Type: text/plain, Size: 2703 bytes --]

[ 2298.720212] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
[ 2298.728193] IP: xfrm_lookup+0x2a/0x7d0
[ 2298.731986] PGD 0 P4D 0 
[ 2298.734535] Oops: 0000 [#1] SMP PTI
[ 2298.738035] Modules linked in:
[ 2298.741121] CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 4.14.12 #3
[ 2298.747362] Hardware name: To be filled by O.E.M. CAR-2051/CAR, BIOS 1.01 07/11/2016
[ 2298.755136] task: ffffa0dafb08dc00 task.stack: ffffa211c0040000
[ 2298.761091] RIP: 0010:xfrm_lookup+0x2a/0x7d0
[ 2298.765403] RSP: 0018:ffffa211c0043ad0 EFLAGS: 00010246
[ 2298.770656] RAX: 0000000000000000 RBX: ffffffff87074080 RCX: 0000000000000000
[ 2298.777851] RDX: ffffa211c0043b48 RSI: 0000000000000000 RDI: ffffffff87074080
[ 2298.785025] RBP: ffffffff87074080 R08: 0000000000000002 R09: 0000000000000000
[ 2298.792184] R10: 0000000000000020 R11: 0000000000000020 R12: ffffa211c0043b48
[ 2298.799351] R13: 0000000000000000 R14: 0000000000000002 R15: ffffa0dafb240078
[ 2298.806511] FS:  0000000000000000(0000) GS:ffffa0daffc00000(0000) knlGS:0000000000000000
[ 2298.814647] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2298.820428] CR2: 0000000000000020 CR3: 0000000177dcc000 CR4: 00000000001006f0
[ 2298.827587] Call Trace:
[ 2298.830045]  __xfrm_route_forward+0xa4/0x110
[ 2298.834340]  ip_forward+0x3da/0x450
[ 2298.837851]  ? ip_rcv_finish+0x61/0x390
[ 2298.841708]  ip_rcv+0x2b5/0x380
[ 2298.844871]  ? inet_del_offload+0x30/0x30
[ 2298.848910]  __netif_receive_skb_core+0x751/0xb00
[ 2298.853640]  ? netif_receive_skb_internal+0x47/0xf0
[ 2298.858573]  ? inet_gro_receive+0x1fa/0x2a0
[ 2298.862785]  netif_receive_skb_internal+0x47/0xf0
[ 2298.867523]  dev_gro_receive+0x270/0x440
[ 2298.871487]  napi_gro_receive+0x28/0x90
[ 2298.875350]  igb_poll+0x600/0xe80
[ 2298.878695]  net_rx_action+0x1fc/0x310
[ 2298.882478]  __do_softirq+0xd5/0x1cf
[ 2298.886064]  run_ksoftirqd+0x14/0x30
[ 2298.889670]  smpboot_thread_fn+0xf9/0x150
[ 2298.893707]  kthread+0xf2/0x130
[ 2298.896869]  ? sort_range+0x20/0x20
[ 2298.900387]  ? kthread_park+0x60/0x60
[ 2298.904080]  ret_from_fork+0x1f/0x30
[ 2298.907684] Code: 00 41 57 41 56 45 89 c6 41 55 41 54 49 89 f5 55 53 49 89 d4 48 89 fb 48 83 ec 40 65 48 8b 04 25 28 00 00 00 48 89 44 24 38 31 c0 <48> 8b 46 20 48 85 c9 44 0f b7 38 c7 44 24 0c 00 00 00 00 0f 84 
[ 2298.926666] RIP: xfrm_lookup+0x2a/0x7d0 RSP: ffffa211c0043ad0
[ 2298.932447] CR2: 0000000000000020
[ 2298.935792] ---[ end trace 4045e2796d0dd0c8 ]---
[ 2298.940446] Kernel panic - not syncing: Fatal exception in interrupt
[ 2298.946937] Kernel Offset: 0x5000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 2298.957711] Rebooting in 10 seconds..