[bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Timo Teras]

The oops happens when forwarding traffic between ethX <-> gre. Where
the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in
transport mode. It seems that locally originating traffic to gre is not
affected.

The oops also goes away if GRO is turned off for gre tunnel device.

I have bisected this always reproducible oops to commit:

8a29111c7ca68d928dfab58636f3f6acf0ac04f7 is the first bad commit

commit 8a29111c7ca68d928dfab58636f3f6acf0ac04f7
Author: Eric Dumazet <edumazet@google.com>
Date: Oct 8 09:02:23 2013 -0700
net: gro: allow to build full sized skb

This oops backtrace is from vanilla 3.14.4 kernel, but it is identical
up to the offending commit.

[  286.927713] BUG: unable to handle kernel paging request at 2b90bdc8
[  286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453
[  286.930813] *pde = 00000000 
[  286.930813] Oops: 0000 [#1] SMP 
[  286.930813] Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
deflate ctr twofish_generic twofish_i586 twofish_common camellia_generic
serpent_sse2_i586 xts lrw gf128mul serpent_generic glue_helper ablk_helper cryptd
blowfish_generic blowfish_common cast5_generic cast_common des_generic cbc cmac
xcbc rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre ip_tunnel
nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_raw ipt_MASQUERADE
iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_helper nf_conntrack_ipv4
nf_defrag_ipv4 iptable_filter ip_tables nf_conntrack_ftp nf_conntrack_sip
xt_CT ip6table_raw xt_LOG xt_limit xt_policy xt_tcpudp nf_conntrack_ipv6
nf_defrag_ipv6 xt_recent xt_multiport xt_conntrack nf_conntrack ip6table_filter
ip6_tables x_tables ipv6 af_packet mousedev via_rng rng_core via_cputemp hwmon
hwmon_vid padlock_aes padlock_sha serio_raw psmouse pcspkr shpchp i2c_viapro
i2c_core via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus
snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore firewire_ohci
firewire_core crc_itu_t via_agp agpgart r8169 firmware_class mii fan evdev
parport_pc parport thermal button acpi_cpufreq processor nls_utf8 nls_cp437
vfat fat sata_via ehci_pci ehci_hcd uhci_hcd pata_via pata_acpi ata_generic
libata usb_storage usbcore usb_common sd_mod scsi_mod crc_t10dif
crct10dif_common squashfs loop
[  286.930813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.4 #1-fragbisect
[  286.930813] Hardware name:    /CN700-8237, BIOS 6.00 PG 08/06/2008
[  286.930813] task: c13e8930 ti: f6408000 task.ti: c13de000
[  286.930813] EIP: 0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0
[  286.930813] EIP is at skb_gro_receive+0x118/0x453
[  286.930813] EAX: 2b90bdc8 EBX: e9f669c0 ECX: 00000034 EDX: 00000596
[  286.930813] ESI: e9f669c0 EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c
[  286.930813]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[  286.930813] CR0: 8005003b CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690
[  286.930813] Stack:
[  286.930813]  c120df4b e9f4f796 00000046 00000000 f648f134 c013a600 0000007a e9f4fd40
[  286.930813]  2b90bdc8 00000034 e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f
[  286.930813]  00000000 00000020 00000046 2e001080 e9f4ed96 00000562 e9f663c0 e9f4f782
[  286.930813] Call Trace:
[  286.930813]  [<c120df4b>] ? csum_partial_ext+0x16/0x18
[  286.930813]  [<c125cc6f>] tcp_gro_receive+0x1a8/0x218
[  286.930813]  [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8
[  286.930813]  [<c1268439>] inet_gro_receive+0x1c5/0x1df
[  286.930813]  [<c1219a15>] dev_gro_receive+0x231/0x393
[  286.930813]  [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore]
[  286.930813]  [<c1219c1c>] napi_gro_receive+0xb/0x5e
[  286.930813]  [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel]
[  286.930813]  [<c121a2da>] net_rx_action+0xb0/0x14d
[  286.930813]  [<c1030d43>] __do_softirq+0xb8/0x1a5
[  286.930813]  [<c1030c8b>] ? cpu_callback+0xec/0xec
[  286.930813]  <IRQ> 
[  286.930813]  [<c1030fa5>] ? irq_exit+0x44/0x81
[  286.930813]  [<c1002da0>] ? do_IRQ+0x9f/0xb3
[  286.930813]  [<c106e642>] ? clockevents_notify+0x10f/0x116
[  286.930813]  [<c1298d33>] ? common_interrupt+0x33/0x40
[  286.930813]  [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3
[  286.930813]  [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6
[  286.930813]  [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c
[  286.930813]  [<c1060542>] ? cpu_startup_entry+0xf3/0x159
[  286.930813]  [<c128b36b>] ? rest_init+0x5d/0x5f
[  286.930813]  [<c1415a17>] ? start_kernel+0x3b2/0x3b8
[  286.930813]  [<c141549b>] ? repair_env_string+0x51/0x51
[  286.930813]  [<c14152d0>] ? i386_start_kernel+0x7a/0x7e
[  286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00
e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00
<8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05
[  286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP 0068:f6409e7c
[  286.930813] CR2: 000000002b90bdc8
[  286.930813] ---[ end trace ac04411e60d3534c ]---
[  286.930813] Kernel panic - not syncing: Fatal exception in interrupt
[  286.930813] Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf7ffdfff)

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Eric Dumazet]

On Fri, 2014-05-16 at 10:40 +0300, Timo Teras wrote:
> The oops happens when forwarding traffic between ethX <-> gre. Where
> the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in
> transport mode. It seems that locally originating traffic to gre is not
> affected.
> 
> The oops also goes away if GRO is turned off for gre tunnel device.
> 
> I have bisected this always reproducible oops to commit:
> 
> 8a29111c7ca68d928dfab58636f3f6acf0ac04f7 is the first bad commit
> 
> commit 8a29111c7ca68d928dfab58636f3f6acf0ac04f7
> Author: Eric Dumazet <edumazet@google.com>
> Date: Oct 8 09:02:23 2013 -0700
> net: gro: allow to build full sized skb
> 
> This oops backtrace is from vanilla 3.14.4 kernel, but it is identical
> up to the offending commit.
> 
> [  286.927713] BUG: unable to handle kernel paging request at 2b90bdc8
> [  286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453
> [  286.930813] *pde = 00000000 
> [  286.930813] Oops: 0000 [#1] SMP 
> [  286.930813] Modules linked in: sha1_generic authenc esp4 xfrm4_mode_transport
> deflate ctr twofish_generic twofish_i586 twofish_common camellia_generic
> serpent_sse2_i586 xts lrw gf128mul serpent_generic glue_helper ablk_helper cryptd
> blowfish_generic blowfish_common cast5_generic cast_common des_generic cbc cmac
> xcbc rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre ip_tunnel
> nf_conntrack_netbios_ns nf_conntrack_broadcast iptable_raw ipt_MASQUERADE
> iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT xt_helper nf_conntrack_ipv4
> nf_defrag_ipv4 iptable_filter ip_tables nf_conntrack_ftp nf_conntrack_sip
> xt_CT ip6table_raw xt_LOG xt_limit xt_policy xt_tcpudp nf_conntrack_ipv6
> nf_defrag_ipv6 xt_recent xt_multiport xt_conntrack nf_conntrack ip6table_filter
> ip6_tables x_tables ipv6 af_packet mousedev via_rng rng_core via_cputemp hwmon
> hwmon_vid padlock_aes padlock_sha serio_raw psmouse pcspkr shpchp i2c_viapro
> i2c_core via_rhine snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus
> snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore firewire_ohci
> firewire_core crc_itu_t via_agp agpgart r8169 firmware_class mii fan evdev
> parport_pc parport thermal button acpi_cpufreq processor nls_utf8 nls_cp437
> vfat fat sata_via ehci_pci ehci_hcd uhci_hcd pata_via pata_acpi ata_generic
> libata usb_storage usbcore usb_common sd_mod scsi_mod crc_t10dif
> crct10dif_common squashfs loop
> [  286.930813] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.4 #1-fragbisect
> [  286.930813] Hardware name:    /CN700-8237, BIOS 6.00 PG 08/06/2008
> [  286.930813] task: c13e8930 ti: f6408000 task.ti: c13de000
> [  286.930813] EIP: 0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0
> [  286.930813] EIP is at skb_gro_receive+0x118/0x453
> [  286.930813] EAX: 2b90bdc8 EBX: e9f669c0 ECX: 00000034 EDX: 00000596
> [  286.930813] ESI: e9f669c0 EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c
> [  286.930813]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
> [  286.930813] CR0: 8005003b CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690
> [  286.930813] Stack:
> [  286.930813]  c120df4b e9f4f796 00000046 00000000 f648f134 c013a600 0000007a e9f4fd40
> [  286.930813]  2b90bdc8 00000034 e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f
> [  286.930813]  00000000 00000020 00000046 2e001080 e9f4ed96 00000562 e9f663c0 e9f4f782
> [  286.930813] Call Trace:
> [  286.930813]  [<c120df4b>] ? csum_partial_ext+0x16/0x18
> [  286.930813]  [<c125cc6f>] tcp_gro_receive+0x1a8/0x218
> [  286.930813]  [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8
> [  286.930813]  [<c1268439>] inet_gro_receive+0x1c5/0x1df
> [  286.930813]  [<c1219a15>] dev_gro_receive+0x231/0x393
> [  286.930813]  [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore]
> [  286.930813]  [<c1219c1c>] napi_gro_receive+0xb/0x5e
> [  286.930813]  [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel]
> [  286.930813]  [<c121a2da>] net_rx_action+0xb0/0x14d
> [  286.930813]  [<c1030d43>] __do_softirq+0xb8/0x1a5
> [  286.930813]  [<c1030c8b>] ? cpu_callback+0xec/0xec
> [  286.930813]  <IRQ> 
> [  286.930813]  [<c1030fa5>] ? irq_exit+0x44/0x81
> [  286.930813]  [<c1002da0>] ? do_IRQ+0x9f/0xb3
> [  286.930813]  [<c106e642>] ? clockevents_notify+0x10f/0x116
> [  286.930813]  [<c1298d33>] ? common_interrupt+0x33/0x40
> [  286.930813]  [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3
> [  286.930813]  [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6
> [  286.930813]  [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c
> [  286.930813]  [<c1060542>] ? cpu_startup_entry+0xf3/0x159
> [  286.930813]  [<c128b36b>] ? rest_init+0x5d/0x5f
> [  286.930813]  [<c1415a17>] ? start_kernel+0x3b2/0x3b8
> [  286.930813]  [<c141549b>] ? repair_env_string+0x51/0x51
> [  286.930813]  [<c14152d0>] ? i386_start_kernel+0x7a/0x7e
> [  286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00
> e9 09 03 00 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00
> <8a> 00 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05
> [  286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP 0068:f6409e7c
> [  286.930813] CR2: 000000002b90bdc8
> [  286.930813] ---[ end trace ac04411e60d3534c ]---
> [  286.930813] Kernel panic - not syncing: Fatal exception in interrupt
> [  286.930813] Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf7ffdfff)
> --

Unffortunately bisecting wont help a lot, as we knew the commit was
faulty.

Please check 9d8506cc2d7ea1f911c72c100193a3677f6668c3

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Timo Teras]

On Fri, 16 May 2014 05:59:17 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Fri, 2014-05-16 at 10:40 +0300, Timo Teras wrote:
> > The oops happens when forwarding traffic between ethX <-> gre. Where
> > the GRE tunnel is an NBMA tunnel and the GRE traffic is IPsec'ed in
> > transport mode. It seems that locally originating traffic to gre is
> > not affected.
> > 
> > The oops also goes away if GRO is turned off for gre tunnel device.
> > 
> > I have bisected this always reproducible oops to commit:
> > 
> > 8a29111c7ca68d928dfab58636f3f6acf0ac04f7 is the first bad commit
> > 
> > commit 8a29111c7ca68d928dfab58636f3f6acf0ac04f7
> > Author: Eric Dumazet <edumazet@google.com>
> > Date: Oct 8 09:02:23 2013 -0700
> > net: gro: allow to build full sized skb
> > 
> > This oops backtrace is from vanilla 3.14.4 kernel, but it is
> > identical up to the offending commit.
> > 
> > [  286.927713] BUG: unable to handle kernel paging request at
> > 2b90bdc8 [  286.930813] IP: [<c120fc88>] skb_gro_receive+0x118/0x453
> > [  286.930813] *pde = 00000000 
> > [  286.930813] Oops: 0000 [#1] SMP 
> > [  286.930813] Modules linked in: sha1_generic authenc esp4
> > xfrm4_mode_transport deflate ctr twofish_generic twofish_i586
> > twofish_common camellia_generic serpent_sse2_i586 xts lrw gf128mul
> > serpent_generic glue_helper ablk_helper cryptd blowfish_generic
> > blowfish_common cast5_generic cast_common des_generic cbc cmac xcbc
> > rmd160 sha512_generic hmac crypto_null af_key xfrm_algo ip_gre
> > ip_tunnel nf_conntrack_netbios_ns nf_conntrack_broadcast
> > iptable_raw ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat
> > ipt_REJECT xt_helper nf_conntrack_ipv4 nf_defrag_ipv4
> > iptable_filter ip_tables nf_conntrack_ftp nf_conntrack_sip xt_CT
> > ip6table_raw xt_LOG xt_limit xt_policy xt_tcpudp nf_conntrack_ipv6
> > nf_defrag_ipv6 xt_recent xt_multiport xt_conntrack nf_conntrack
> > ip6table_filter ip6_tables x_tables ipv6 af_packet mousedev via_rng
> > rng_core via_cputemp hwmon hwmon_vid padlock_aes padlock_sha
> > serio_raw psmouse pcspkr shpchp i2c_viapro i2c_core via_rhine
> > snd_via82xx snd_ac97_codec snd_pcm snd_timer ac97_bus
> > snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore
> > firewire_ohci firewire_core crc_itu_t via_agp agpgart r8169
> > firmware_class mii fan evdev parport_pc parport thermal button
> > acpi_cpufreq processor nls_utf8 nls_cp437 vfat fat sata_via
> > ehci_pci ehci_hcd uhci_hcd pata_via pata_acpi ata_generic libata
> > usb_storage usbcore usb_common sd_mod scsi_mod crc_t10dif
> > crct10dif_common squashfs loop [  286.930813] CPU: 0 PID: 0 Comm:
> > swapper/0 Not tainted 3.14.4 #1-fragbisect [  286.930813] Hardware
> > name:    /CN700-8237, BIOS 6.00 PG 08/06/2008 [  286.930813] task:
> > c13e8930 ti: f6408000 task.ti: c13de000 [  286.930813] EIP:
> > 0060:[<c120fc88>] EFLAGS: 00210202 CPU: 0 [  286.930813] EIP is at
> > skb_gro_receive+0x118/0x453 [  286.930813] EAX: 2b90bdc8 EBX:
> > e9f669c0 ECX: 00000034 EDX: 00000596 [  286.930813] ESI: e9f669c0
> > EDI: f648f134 EBP: f6409eb4 ESP: f6409e7c [  286.930813]  DS: 007b
> > ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [  286.930813] CR0: 8005003b
> > CR2: 2b90bdc8 CR3: 2a731000 CR4: 00000690 [  286.930813] Stack:
> > [  286.930813]  c120df4b e9f4f796 00000046 00000000 f648f134
> > c013a600 0000007a e9f4fd40 [  286.930813]  2b90bdc8 00000034
> > e9f663c0 e9f663c0 e9f669c0 f648f134 f6409ee0 c125cc6f
> > [  286.930813]  00000000 00000020 00000046 2e001080 e9f4ed96
> > 00000562 e9f663c0 e9f4f782 [  286.930813] Call Trace:
> > [  286.930813]  [<c120df4b>] ? csum_partial_ext+0x16/0x18
> > [  286.930813]  [<c125cc6f>] tcp_gro_receive+0x1a8/0x218
> > [  286.930813]  [<c125cd9f>] tcp4_gro_receive+0xc0/0xc8
> > [  286.930813]  [<c1268439>] inet_gro_receive+0x1c5/0x1df
> > [  286.930813]  [<c1219a15>] dev_gro_receive+0x231/0x393
> > [  286.930813]  [<f80e006b>] ? rh_timer_func+0x8/0xa [usbcore]
> > [  286.930813]  [<c1219c1c>] napi_gro_receive+0xb/0x5e
> > [  286.930813]  [<f8650056>] gro_cell_poll+0x56/0x73 [ip_tunnel]
> > [  286.930813]  [<c121a2da>] net_rx_action+0xb0/0x14d
> > [  286.930813]  [<c1030d43>] __do_softirq+0xb8/0x1a5
> > [  286.930813]  [<c1030c8b>] ? cpu_callback+0xec/0xec
> > [  286.930813]  <IRQ> [  286.930813]  [<c1030fa5>] ?
> > irq_exit+0x44/0x81 [  286.930813]  [<c1002da0>] ? do_IRQ+0x9f/0xb3
> > [  286.930813]  [<c106e642>] ? clockevents_notify+0x10f/0x116
> > [  286.930813]  [<c1298d33>] ? common_interrupt+0x33/0x40
> > [  286.930813]  [<c11f0292>] ? cpuidle_enter_state+0x39/0xa3
> > [  286.930813]  [<c11f03a5>] ? cpuidle_idle_call+0xa9/0xe6
> > [  286.930813]  [<c1007f8f>] ? arch_cpu_idle+0x8/0x1c
> > [  286.930813]  [<c1060542>] ? cpu_startup_entry+0xf3/0x159
> > [  286.930813]  [<c128b36b>] ? rest_init+0x5d/0x5f [  286.930813]
> > [<c1415a17>] ? start_kernel+0x3b2/0x3b8 [  286.930813]
> > [<c141549b>] ? repair_env_string+0x51/0x51 [  286.930813]
> > [<c14152d0>] ? i386_start_kernel+0x7a/0x7e [  286.930813] Code: 54
> > 29 56 50 c7 46 54 00 00 00 00 29 d1 89 8e b4 00 00 00 e9 09 03 00
> > 00 8b 45 f0 f6 80 87 00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00
> > 88 45 d8 0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05
> > [  286.930813] EIP: [<c120fc88>] skb_gro_receive+0x118/0x453 SS:ESP
> > 0068:f6409e7c [  286.930813] CR2: 000000002b90bdc8 [  286.930813]
> > ---[ end trace ac04411e60d3534c ]--- [  286.930813] Kernel panic -
> > not syncing: Fatal exception in interrupt [  286.930813] Kernel
> > Offset: 0x0 from 0xc1000000 (relocation range:
> > 0xc0000000-0xf7ffdfff) --
> 
> Unffortunately bisecting wont help a lot, as we knew the commit was
> faulty.
> 
> Please check 9d8506cc2d7ea1f911c72c100193a3677f6668c3

Ah, did not notice that one. So I'm adding Herbert as explicit cc.

$ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3
v3.13-rc1~7^2

But as you may have noticed, the attached oops is from 3.14.4 that
includes the fix commit. While bisecting, I tested multiple versions
between 3.12 and 3.14. Bisecting clearly points this is the commit that
introduces this oops, and it is not fixed in any version up until
3.14.4.

Also the forward path from gre->ethX goes to eth that does not have
TSO/GSO support. So I'd assume that the fix commit is not that
relevant in my specific test case.

The faulty commit must be introducing additional bug, and the commit
you refer did not fix it.

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Eric Dumazet]

On Fri, 2014-05-16 at 17:34 +0300, Timo Teras wrote:

> Ah, did not notice that one. So I'm adding Herbert as explicit cc.
> 
> $ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3
> v3.13-rc1~7^2
> 
> But as you may have noticed, the attached oops is from 3.14.4 that
> includes the fix commit. While bisecting, I tested multiple versions
> between 3.12 and 3.14. Bisecting clearly points this is the commit that
> introduces this oops, and it is not fixed in any version up until
> 3.14.4.
> 
> Also the forward path from gre->ethX goes to eth that does not have
> TSO/GSO support. So I'd assume that the fix commit is not that
> relevant in my specific test case.
> 
> The faulty commit must be introducing additional bug, and the commit
> you refer did not fix it.

Right. The commit could expose a dormant bug in a driver.

For example, some drivers might forget to use skb_put()

Please give us more details.

What is the driver receiving ethernet frames ?

Is it using GRO ?

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Timo Teras]

On Fri, 16 May 2014 09:29:29 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Fri, 2014-05-16 at 17:34 +0300, Timo Teras wrote:
> 
> > Ah, did not notice that one. So I'm adding Herbert as explicit cc.
> > 
> > $ git describe --contains 9d8506cc2d7ea1f911c72c100193a3677f6668c3
> > v3.13-rc1~7^2
> > 
> > But as you may have noticed, the attached oops is from 3.14.4 that
> > includes the fix commit. While bisecting, I tested multiple versions
> > between 3.12 and 3.14. Bisecting clearly points this is the commit
> > that introduces this oops, and it is not fixed in any version up
> > until 3.14.4.
> > 
> > Also the forward path from gre->ethX goes to eth that does not have
> > TSO/GSO support. So I'd assume that the fix commit is not that
> > relevant in my specific test case.
> > 
> > The faulty commit must be introducing additional bug, and the commit
> > you refer did not fix it.
> 
> Right. The commit could expose a dormant bug in a driver.
> 
> For example, some drivers might forget to use skb_put()
> 
> Please give us more details.
> 
> What is the driver receiving ethernet frames ?
> 
> Is it using GRO ?

Both the gre bound interface, and the inside interface are identical.

And you are right. It might be driver / hw / gro capability specific
issue. I had also report from a user using similar gre/ipsec/forwarding
setup, that 3.13-stable kernels works on his hardware.

dmesg tells:

[   23.946984] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   23.978145] r8169 0000:00:09.0 (unregistered net_device): not PCI Express
[   23.997147] r8169 0000:00:09.0 eth0: RTL8169sc/8110sc at 0xf8358000, 00:30:18:a8:14:ac, XID 18000000 IRQ 18
[   24.007012] r8169 0000:00:09.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]
[   24.032389] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   24.039530] r8169 0000:00:0b.0 (unregistered net_device): not PCI Express
[   24.057092] r8169 0000:00:0b.0 eth1: RTL8169sc/8110sc at 0xf835a000, 00:30:18:ab:69:4b, XID 18000000 IRQ 19
[   24.066955] r8169 0000:00:0b.0 eth1: jumbo features [frames: 7152 bytes, tx checksumming: ok]
[   24.092330] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   24.098745] r8169 0000:00:0c.0 (unregistered net_device): not PCI Express
[   24.117118] r8169 0000:00:0c.0 eth2: RTL8169sc/8110sc at 0xf835c000, 00:30:18:a8:14:ad, XID 18000000 IRQ 16
[   24.126983] r8169 0000:00:0c.0 eth2: jumbo features [frames: 7152 bytes, tx checksumming: ok]

And, ethtool -k eth0

Features for eth0:
rx-checksumming: on
tx-checksumming: off
	tx-checksum-ipv4: off
	tx-checksum-ip-generic: off [fixed]
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: off
	tx-scatter-gather: off
	tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
	tx-tcp-segmentation: off
	tx-tcp-ecn-segmentation: off [fixed]
	tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

Using unmodified in-tree r8169 module.

Please let me know if additional information is needed.

Thanks.
Timo

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Eric Dumazet]

On Fri, 2014-05-16 at 19:40 +0300, Timo Teras wrote:

> Both the gre bound interface, and the inside interface are identical.
> 
> And you are right. It might be driver / hw / gro capability specific
> issue. I had also report from a user using similar gre/ipsec/forwarding
> setup, that 3.13-stable kernels works on his hardware.
> 

> Using unmodified in-tree r8169 module.
> 
> Please let me know if additional information is needed.

What happens if you disable gro on eth0 ?

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Timo Teras]

On Fri, 16 May 2014 09:50:27 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Fri, 2014-05-16 at 19:40 +0300, Timo Teras wrote:
> 
> > Both the gre bound interface, and the inside interface are
> > identical.
> > 
> > And you are right. It might be driver / hw / gro capability specific
> > issue. I had also report from a user using similar
> > gre/ipsec/forwarding setup, that 3.13-stable kernels works on his
> > hardware.
> > 
> 
> > Using unmodified in-tree r8169 module.
> > 
> > Please let me know if additional information is needed.
> 
> What happens if you disable gro on eth0 ?

I believe it crashes, but cannot say with 100% certainty. I can
test later on if needed. Based on earlier experiment, the only way to
avoid the crash was to turn off gro on gre1.

In any case I don't think it should make any effect, since the GRE
packets arrive IPseced. eth0 is receiving ESP packets - and I believe
there's no ESP GRO support. Only after decryption they go to gre1, so
gre1 gro is basically the place where received packets are coalesced.

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Eric Dumazet]

On Fri, 2014-05-16 at 20:01 +0300, Timo Teras wrote:
Documentation/oops-tracing.txtscripts/decodecode
> I believe it crashes, but cannot say with 100% certainty. I can
> test later on if needed. Based on earlier experiment, the only way to
> avoid the crash was to turn off gro on gre1.
> 
> In any case I don't think it should make any effect, since the GRE
> packets arrive IPseced. eth0 is receiving ESP packets - and I believe
> there's no ESP GRO support. Only after decryption they go to gre1, so
> gre1 gro is basically the place where received packets are coalesced.

Oh this is definitely useful.

I thought we might have a problem now with have GRE enabled GRO in
native GRO layer. I was wondering if the double attempt to GRO packets a
second time was a problem.

So it looks like ESP provides packets that are not very gentle for
skb_gro_receive()...

Could you provide scripts/decodecode output ?

Thanks

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Timo Teras]

On Fri, 16 May 2014 10:13:50 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Fri, 2014-05-16 at 20:01 +0300, Timo Teras wrote:
> Documentation/oops-tracing.txtscripts/decodecode
> > I believe it crashes, but cannot say with 100% certainty. I can
> > test later on if needed. Based on earlier experiment, the only way
> > to avoid the crash was to turn off gro on gre1.
> > 
> > In any case I don't think it should make any effect, since the GRE
> > packets arrive IPseced. eth0 is receiving ESP packets - and I
> > believe there's no ESP GRO support. Only after decryption they go
> > to gre1, so gre1 gro is basically the place where received packets
> > are coalesced.
> 
> Oh this is definitely useful.
> 
> I thought we might have a problem now with have GRE enabled GRO in
> native GRO layer. I was wondering if the double attempt to GRO
> packets a second time was a problem.
> 
> So it looks like ESP provides packets that are not very gentle for
> skb_gro_receive()...
> 
> Could you provide scripts/decodecode output ?

[ 286.930813] Code: 54 29 56 50 c7 46 54 00 00 00 00 29
d1 89 8e b4 00 00 00 e9 09 03 00 00 8b 45 f0 f6 80 87
00 00 00 01 8b 45 e8 0f 84 cf 00 00 00 <8a> 00 88 45 d8
0f b6 d0 8b 45 f0 8b 80 ac 00 00 00 89 45 d4 05

All code
========
   0:	54                   	push   %esp
   1:	29 56 50             	sub    %edx,0x50(%esi)
   4:	c7 46 54 00 00 00 00 	movl   $0x0,0x54(%esi)
   b:	29 d1                	sub    %edx,%ecx
   d:	89 8e b4 00 00 00    	mov    %ecx,0xb4(%esi)
  13:	e9 09 03 00 00       	jmp    0x321
  18:	8b 45 f0             	mov    -0x10(%ebp),%eax
  1b:	f6 80 87 00 00 00 01 	testb  $0x1,0x87(%eax)
  22:	8b 45 e8             	mov    -0x18(%ebp),%eax
  25:	0f 84 cf 00 00 00    	je     0xfa
  2b:*	8a 00                	mov    (%eax),%al		<-- trapping instruction
  2d:	88 45 d8             	mov    %al,-0x28(%ebp)
  30:	0f b6 d0             	movzbl %al,%edx
  33:	8b 45 f0             	mov    -0x10(%ebp),%eax
  36:	8b 80 ac 00 00 00    	mov    0xac(%eax),%eax
  3c:	89 45 d4             	mov    %eax,-0x2c(%ebp)
  3f:	05                   	.byte 0x5

Code starting with the faulting instruction
===========================================
   0:	8a 00                	mov    (%eax),%al
   2:	88 45 d8             	mov    %al,-0x28(%ebp)
   5:	0f b6 d0             	movzbl %al,%edx
   8:	8b 45 f0             	mov    -0x10(%ebp),%eax
   b:	8b 80 ac 00 00 00    	mov    0xac(%eax),%eax
  11:	89 45 d4             	mov    %eax,-0x2c(%ebp)
  14:	05                   	.byte 0x5

Unfortunately, I do not have fully matching identical .s, or debug
built kernel. I can do that on Monday if this does not help.

Additionally, the .s file from separate, but similar build, that seems
to have matching assembly is as follows:

	movl	168(%esi), %edx	# MEM[(const struct sk_buff *)skb_19(D)].end, D.55920
	subl	172(%esi), %edx	# MEM[(const struct sk_buff *)skb_19(D)].head, D.55920
	movb	$1, 43(%esi)	#, MEM[(struct napi_gro_cb *)skb_19(D) + 24B].free
	leal	-384(%ecx), %eax	#, delta_truesize
	subl	%edx, %eax	# D.55920, delta_truesize
	movl	84(%esi), %edx	# skb_19(D)->data_len, D.55922
	subl	%edx, 80(%esi)	# D.55922, skb_19(D)->len
	movl	$0, 84(%esi)	#, skb_19(D)->data_len
	subl	%edx, %ecx	# D.55922, tmp300
	movl	%ecx, 180(%esi)	# tmp300, skb_19(D)->truesize
	jmp	.L572	#
.L568:
	movl	-16(%ebp), %eax	# %sfp, skb
	testb	$1, 135(%eax)	#, *skb_19(D)
	movl	-24(%ebp), %eax	# %sfp, D.55921
	je	.L573	#,
## Crash on the next opcode
	movb	(%eax), %al	# MEM[(struct skb_shared_info *)_29].nr_frags, D.55923
	movb	%al, -40(%ebp)	# D.55923, %sfp
	movzbl	%al, %edx	# D.55923, nr_frags
	movl	-16(%ebp), %eax	# %sfp, skb
	movl	172(%eax), %eax	# skb_19(D)->head, skb_19(D)->head
	movl	%eax, -44(%ebp)	# skb_19(D)->head, %sfp
	addl	$1073741824, %eax	#, page
	shrl	$12, %eax	#, page
	sall	$5, %eax	#, page
	addl	mem_map, %eax	# mem_map, page
	movl	(%eax), %ecx	# MEM[(const long unsigned int *)page_213], D.55925
	movl	%eax, %edi	# page, D.55929
	andb	$128, %ch	#, D.55925
	je	.L574	#,
	movl	28(%eax), %esi	# page_213->D.14510.first_page, head
	movl	(%eax), %ecx	# MEM[(const long unsigned int *)page_213], D.55925
	andb	$128, %ch	#, D.55925
	je	.L587	#,
	movl	%esi, %edi	# head, D.55929
	jmp	.L574	#

Re: [bisected] [oops] gre/gro oops in skb_gro_receive+0x118/0x453

[Eric Dumazet]

On Fri, 2014-05-16 at 20:38 +0300, Timo Teras wrote:


> Unfortunately, I do not have fully matching identical .s, or debug
> built kernel. I can do that on Monday if this does not help.
> 
> Additionally, the .s file from separate, but similar build, that seems
> to have matching assembly is as follows:

Thanks a lot, I found the bug, its yet another case of skb->cb[] being
not clean ;)

I am testing a patch atm.

[PATCH] net: gro: make sure skb->cb[] initial content has not to be zero

[Eric Dumazet]

From: Eric Dumazet <edumazet@google.com>

Starting from linux-3.13, GRO attempts to build full size skbs.

Problem is the commit assumed one particular field in skb->cb[]
was clean, but it is not the case on some stacked devices.

Timo reported a crash in case traffic is decrypted before
reaching a GRE device.

Fix this by initializing NAPI_GRO_CB(skb)->last at the right place,
this also removes one conditional.

Thanks a lot to Timo for providing full reports and bisecting this.

Fixes: 8a29111c7ca6 ("net: gro: allow to build full sized skb")
Bisected-by: Timo Teras <timo.teras@iki.fi>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
Please Timo test this patch so that we confirm it fixes the bug for you.

 net/core/dev.c    |    1 +
 net/core/skbuff.c |    4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/net/core/dev.c b/net/core/dev.c
index 6da649bde4f7..ed928e846559 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -3951,6 +3951,7 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 	}
 	NAPI_GRO_CB(skb)->count = 1;
 	NAPI_GRO_CB(skb)->age = jiffies;
+	NAPI_GRO_CB(skb)->last = skb;
 	skb_shinfo(skb)->gso_size = skb_gro_len(skb);
 	skb->next = napi->gro_list;
 	napi->gro_list = skb;
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index 1b62343f5837..8383b2bddeb9 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -3076,7 +3076,7 @@ int skb_gro_receive(struct sk_buff **head, struct sk_buff *skb)
 	if (unlikely(p->len + len >= 65536))
 		return -E2BIG;
 
-	lp = NAPI_GRO_CB(p)->last ?: p;
+	lp = NAPI_GRO_CB(p)->last;
 	pinfo = skb_shinfo(lp);
 
 	if (headlen <= offset) {
@@ -3192,7 +3192,7 @@ merge:
 
 	__skb_pull(skb, offset);
 
-	if (!NAPI_GRO_CB(p)->last)
+	if (NAPI_GRO_CB(p)->last == p)
 		skb_shinfo(p)->frag_list = skb;
 	else
 		NAPI_GRO_CB(p)->last->next = skb;

Re: [PATCH] net: gro: make sure skb->cb[] initial content has not to be zero

[Timo Teras]

On Fri, 16 May 2014 11:34:37 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> From: Eric Dumazet <edumazet@google.com>
> 
> Starting from linux-3.13, GRO attempts to build full size skbs.
> 
> Problem is the commit assumed one particular field in skb->cb[]
> was clean, but it is not the case on some stacked devices.
> 
> Timo reported a crash in case traffic is decrypted before
> reaching a GRE device.
> 
> Fix this by initializing NAPI_GRO_CB(skb)->last at the right place,
> this also removes one conditional.
> 
> Thanks a lot to Timo for providing full reports and bisecting this.
> 
> Fixes: 8a29111c7ca6 ("net: gro: allow to build full sized skb")
> Bisected-by: Timo Teras <timo.teras@iki.fi>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> ---
> Please Timo test this patch so that we confirm it fixes the bug for
> you.

Yes, everything looks green now! (The test case reproduced the crash
reliably and immediately. Now it's been up and working for more than
tens minutes already :)

Tested-by: Timo Teräs <timo.teras@iki.fi>

Needless to say, this should go to 3.14-stable as well.

Thanks a lot Eric!

Re: [PATCH] net: gro: make sure skb->cb[] initial content has not to be zero

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Fri, 16 May 2014 11:34:37 -0700

> From: Eric Dumazet <edumazet@google.com>
> 
> Starting from linux-3.13, GRO attempts to build full size skbs.
> 
> Problem is the commit assumed one particular field in skb->cb[]
> was clean, but it is not the case on some stacked devices.
> 
> Timo reported a crash in case traffic is decrypted before
> reaching a GRE device.
> 
> Fix this by initializing NAPI_GRO_CB(skb)->last at the right place,
> this also removes one conditional.
> 
> Thanks a lot to Timo for providing full reports and bisecting this.
> 
> Fixes: 8a29111c7ca6 ("net: gro: allow to build full sized skb")
> Bisected-by: Timo Teras <timo.teras@iki.fi>
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> ---
> Please Timo test this patch so that we confirm it fixes the bug for you.

Applied and queued up for -stable, thanks everyone.