From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Magnus Karlsson <magnus.karlsson@gmail.com>
Cc: Lorenzo Bianconi <lorenzo@kernel.org>, bpf <bpf@vger.kernel.org>,
Network Development <netdev@vger.kernel.org>,
"David S. Miller" <davem@davemloft.net>,
Jakub Kicinski <kuba@kernel.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
John Fastabend <john.fastabend@gmail.com>,
David Ahern <dsahern@kernel.org>,
Eelco Chaudron <echaudro@redhat.com>,
Jason Wang <jasowang@redhat.com>,
Alexander Duyck <alexander.duyck@gmail.com>,
Saeed Mahameed <saeed@kernel.org>,
"Fijalkowski, Maciej" <maciej.fijalkowski@intel.com>,
Tirthendu <tirthendu.sarkar@intel.com>,
brouer@redhat.com
Subject: Crash for i40e on net-next (was: [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support)
Date: Thu, 22 Apr 2021 17:05:08 +0200 [thread overview]
Message-ID: <20210422170508.22c58226@carbon> (raw)
In-Reply-To: <20210422164223.77870d28@carbon>
On Thu, 22 Apr 2021 16:42:23 +0200
Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> On Thu, 22 Apr 2021 12:24:32 +0200
> Magnus Karlsson <magnus.karlsson@gmail.com> wrote:
>
> > On Wed, Apr 21, 2021 at 5:39 PM Jesper Dangaard Brouer
> > <brouer@redhat.com> wrote:
> > >
> > > On Wed, 21 Apr 2021 16:12:32 +0200
> > > Magnus Karlsson <magnus.karlsson@gmail.com> wrote:
> > >
> [...]
> > > > more than I get.
> > >
> > > I clearly have a bug in the i40e driver. As I wrote later, I don't see
> > > any packets transmitted for XDP_TX. Hmm, I using Mel Gorman's tree,
> > > which contains the i40e/ice/ixgbe bug we fixed earlier.
>
> Something is wrong with i40e, I changed git-tree to net-next (at
> commit 5d869070569a) and XDP seems to have stopped working on i40e :-(
Renamed subj as this is without this patchset applied.
> $ uname -a
> Linux broadwell 5.12.0-rc7-net-next+ #600 SMP PREEMPT Thu Apr 22 15:13:15 CEST 2021 x86_64 x86_64 x86_64 GNU/Linux
>
> When I load any XDP prog almost no packets are let through:
>
> [kernel-bpf-samples]$ sudo ./xdp1 i40e2
> libbpf: elf: skipping unrecognized data section(16) .eh_frame
> libbpf: elf: skipping relo section(17) .rel.eh_frame for section(16) .eh_frame
> proto 17: 1 pkt/s
> proto 0: 0 pkt/s
> proto 17: 0 pkt/s
> proto 0: 0 pkt/s
> proto 17: 1 pkt/s
Trying out xdp_redirect:
[kernel-bpf-samples]$ sudo ./xdp_redirect i40e2 i40e2
input: 7 output: 7
libbpf: elf: skipping unrecognized data section(20) .eh_frame
libbpf: elf: skipping relo section(21) .rel.eh_frame for section(20) .eh_frame
libbpf: Kernel error message: XDP program already attached
WARN: link set xdp fd failed on 7
ifindex 7: 7357 pkt/s
ifindex 7: 7909 pkt/s
ifindex 7: 7909 pkt/s
ifindex 7: 7909 pkt/s
ifindex 7: 7909 pkt/s
ifindex 7: 7909 pkt/s
ifindex 7: 6357 pkt/s
And then it crash (see below) at page_frag_free+0x31 which calls
virt_to_head_page() with a wrong addr (I guess). This is called by
i40e_clean_tx_irq+0xc9.
$ ./scripts/faddr2line drivers/net/ethernet/intel/i40e/i40e.o i40e_clean_tx_irq+0xc9
i40e_clean_tx_irq+0xc9/0x440:
i40e_clean_tx_irq at /home/jbrouer/git/kernel/net-next/drivers/net/ethernet/intel/i40e/i40e_txrx.c:976
Which is:
/* unmap skb header data */
Line:976 dma_unmap_single(tx_ring->dev,
dma_unmap_addr(tx_buf, dma),
dma_unmap_len(tx_buf, len),
DMA_TO_DEVICE);
[ 935.781751] BUG: unable to handle page fault for address: ffffebde00000008
[ 935.788630] #PF: supervisor read access in kernel mode
[ 935.793766] #PF: error_code(0x0000) - not-present page
[ 935.798906] PGD 0 P4D 0
[ 935.801445] Oops: 0000 [#1] PREEMPT SMP PTI
[ 935.805632] CPU: 4 PID: 113 Comm: kworker/u12:9 Not tainted 5.12.0-rc7-net-next+ #600
[ 935.813460] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0a 08/01/2016
[ 935.820937] Workqueue: events_unbound call_usermodehelper_exec_work
[ 935.827214] RIP: 0010:page_frag_free+0x31/0x70
[ 935.831656] Code: 00 00 80 48 01 c7 72 55 48 b8 00 00 00 80 7f 77 00 00 48 01 c7 48 b8 00 00 00 00 00 ea ff ff 48 c1 ef 0c 48 c1 e7 06 48 01 c7 <48> 8b 47 08 48 8d 50 ff a8 01 48 0f 45 fa f0 ff 4f 34 74 01 c3 48
[ 935.850406] RSP: 0018:ffffc900001c0e50 EFLAGS: 00010286
[ 935.855629] RAX: ffffea0000000000 RBX: ffff88813b258180 RCX: 0000000000000000
[ 935.862764] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffebde00000000
[ 935.869895] RBP: ffff88813b258180 R08: ffffc900001c0f38 R09: 0000000000000180
[ 935.877028] R10: 00000000fffffe18 R11: ffffc900001c0ff8 R12: ffff88813bc403c0
[ 935.884160] R13: 000000000000003c R14: 00000000fffffe18 R15: ffff88813b15b180
[ 935.891295] FS: 0000000000000000(0000) GS:ffff88887fd00000(0000) knlGS:0000000000000000
[ 935.899380] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 935.905126] CR2: ffffebde00000008 CR3: 000000087e810002 CR4: 00000000003706e0
[ 935.912259] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 935.919391] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 935.926526] Call Trace:
[ 935.928980] <IRQ>
[ 935.930999] i40e_clean_tx_irq+0xc9/0x440 [i40e]
[ 935.935653] i40e_napi_poll+0x101/0x410 [i40e]
[ 935.940116] __napi_poll+0x2a/0x140
[ 935.943607] net_rx_action+0x215/0x2d0
[ 935.947358] ? i40e_msix_clean_rings+0x3f/0x50 [i40e]
[ 935.952449] __do_softirq+0xe3/0x2df
[ 935.956028] irq_exit_rcu+0xa7/0xb0
[ 935.959519] common_interrupt+0x83/0xa0
[ 935.963358] </IRQ>
[ 935.965465] asm_common_interrupt+0x1e/0x40
[ 935.969651] RIP: 0010:clear_page_erms+0x7/0x10
[ 935.974096] Code: 48 89 47 18 48 89 47 20 48 89 47 28 48 89 47 30 48 89 47 38 48 8d 7f 40 75 d9 90 c3 0f 1f 80 00 00 00 00 b9 00 10 00 00 31 c0 <f3> aa c3 cc cc cc cc cc cc 0f 1f 44 00 00 48 85 ff 0f 84 f2 00 00
[ 935.992845] RSP: 0018:ffffc900003bfbc8 EFLAGS: 00010246
[ 935.998069] RAX: 0000000000000000 RBX: 0000000000000901 RCX: 0000000000000340
[ 936.005202] RDX: 0000000000002dc2 RSI: 0000000000000000 RDI: ffff88813b4d8cc0
[ 936.012334] RBP: ffffea0004ed3600 R08: ffff88887c625f00 R09: ffffea0004ed3600
[ 936.019467] R10: ffff888000000000 R11: 0000160000000000 R12: ffffea0004ed3640
[ 936.026600] R13: 0000000000000000 R14: 0000000000005c39 R15: ffffc900003bfc50
[ 936.033738] prep_new_page+0x88/0xe0
[ 936.037313] get_page_from_freelist+0x2c6/0x3d0
[ 936.041847] __alloc_pages_nodemask+0x137/0x2e0
[ 936.046380] __vmalloc_node_range+0x14f/0x270
[ 936.050738] copy_process+0x39d/0x1ad0
[ 936.054491] ? kernel_clone+0x8b/0x3c0
[ 936.058244] kernel_clone+0x8b/0x3c0
[ 936.061822] ? dequeue_entity+0xc0/0x270
[ 936.065748] kernel_thread+0x47/0x50
[ 936.069329] ? umh_complete+0x40/0x40
[ 936.072992] call_usermodehelper_exec_work+0x2f/0x90
[ 936.077960] process_one_work+0x1ad/0x380
[ 936.081974] worker_thread+0x50/0x390
[ 936.085638] ? process_one_work+0x380/0x380
[ 936.089824] kthread+0x116/0x150
[ 936.093057] ? kthread_park+0x90/0x90
[ 936.096722] ret_from_fork+0x22/0x30
[ 936.100307] Modules linked in: veth nf_defrag_ipv6 nf_defrag_ipv4 tun bridge stp llc rpcrdma sunrpc rdma_ucm ib_umad coretemp rdma_cm ib_ipoib kvm_intel iw_cm ib_cm kvm mlx5_ib i40iw irqbypass ib_uverbs rapl intel_cstate intel_uncore ib_core pcspkr i2c_i801 bfq i2c_smbus acpi_ipmi wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad sch_fq_codel sd_mod t10_pi ixgbe igb igc mdio mlx5_core i40e mlxfw nfp psample i2c_algo_bit ptp i2c_core pps_core hid_generic [last unloaded: bpfilter]
[ 936.142697] CR2: ffffebde00000008
[ 936.146015] ---[ end trace 1bffa979f2cccd16 ]---
[ 936.156720] RIP: 0010:page_frag_free+0x31/0x70
[ 936.161170] Code: 00 00 80 48 01 c7 72 55 48 b8 00 00 00 80 7f 77 00 00 48 01 c7 48 b8 00 00 00 00 00 ea ff ff 48 c1 ef 0c 48 c1 e7 06 48 01 c7 <48> 8b 47 08 48 8d 50 ff a8 01 48 0f 45 fa f0 ff 4f 34 74 01 c3 48
[ 936.179919] RSP: 0018:ffffc900001c0e50 EFLAGS: 00010286
[ 936.185140] RAX: ffffea0000000000 RBX: ffff88813b258180 RCX: 0000000000000000
[ 936.192275] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffebde00000000
[ 936.199407] RBP: ffff88813b258180 R08: ffffc900001c0f38 R09: 0000000000000180
[ 936.206541] R10: 00000000fffffe18 R11: ffffc900001c0ff8 R12: ffff88813bc403c0
[ 936.213673] R13: 000000000000003c R14: 00000000fffffe18 R15: ffff88813b15b180
[ 936.220804] FS: 0000000000000000(0000) GS:ffff88887fd00000(0000) knlGS:0000000000000000
[ 936.228893] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 936.234638] CR2: ffffebde00000008 CR3: 000000087e810002 CR4: 00000000003706e0
[ 936.241771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 936.248903] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 936.256036] Kernel panic - not syncing: Fatal exception in interrupt
[ 936.262401] Kernel Offset: disabled
[ 936.271867] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
next prev parent reply other threads:[~2021-04-22 15:05 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-08 12:50 [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 01/14] xdp: introduce mb in xdp_buff/xdp_frame Lorenzo Bianconi
2021-04-08 18:17 ` Vladimir Oltean
2021-04-09 16:03 ` Lorenzo Bianconi
2021-04-29 13:36 ` Jesper Dangaard Brouer
2021-04-29 13:54 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 02/14] xdp: add xdp_shared_info data structure Lorenzo Bianconi
2021-04-08 13:39 ` Vladimir Oltean
2021-04-08 14:26 ` Lorenzo Bianconi
2021-04-08 18:06 ` kernel test robot
2021-04-08 18:06 ` kernel test robot
2021-04-08 12:50 ` [PATCH v8 bpf-next 03/14] net: mvneta: update mb bit before passing the xdp buffer to eBPF layer Lorenzo Bianconi
2021-04-08 18:19 ` Vladimir Oltean
2021-04-09 16:24 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 04/14] xdp: add multi-buff support to xdp_return_{buff/frame} Lorenzo Bianconi
2021-04-08 18:30 ` Vladimir Oltean
2021-04-09 16:28 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 05/14] net: mvneta: add multi buffer support to XDP_TX Lorenzo Bianconi
2021-04-08 18:40 ` Vladimir Oltean
2021-04-09 16:36 ` Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 06/14] net: mvneta: enable jumbo frames for XDP Lorenzo Bianconi
2021-04-08 12:50 ` [PATCH v8 bpf-next 07/14] net: xdp: add multi-buff support to xdp_build_skb_from_fram Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 08/14] bpf: add multi-buff support to the bpf_xdp_adjust_tail() API Lorenzo Bianconi
2021-04-08 19:15 ` Vladimir Oltean
2021-04-08 20:54 ` Vladimir Oltean
2021-04-09 18:13 ` Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 09/14] bpd: add multi-buffer support to xdp copy helpers Lorenzo Bianconi
2021-04-08 20:57 ` Vladimir Oltean
2021-04-09 18:19 ` Lorenzo Bianconi
2021-04-08 21:04 ` Vladimir Oltean
2021-04-14 8:08 ` Eelco Chaudron
2021-04-08 12:51 ` [PATCH v8 bpf-next 10/14] bpf: add new frame_length field to the XDP ctx Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 11/14] bpf: move user_size out of bpf_test_init Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 12/14] bpf: introduce multibuff support to bpf_prog_test_run_xdp() Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 13/14] bpf: test_run: add xdp_shared_info pointer in bpf_test_finish signature Lorenzo Bianconi
2021-04-08 12:51 ` [PATCH v8 bpf-next 14/14] bpf: update xdp_adjust_tail selftest to include multi-buffer Lorenzo Bianconi
2021-04-09 0:56 ` [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support John Fastabend
2021-04-09 20:16 ` Lorenzo Bianconi
2021-04-13 15:16 ` Eelco Chaudron
2021-04-16 14:27 ` Magnus Karlsson
2021-04-16 21:29 ` Lorenzo Bianconi
2021-04-16 23:00 ` Daniel Borkmann
2021-04-18 16:18 ` Jesper Dangaard Brouer
2021-04-19 6:20 ` Magnus Karlsson
2021-04-19 6:55 ` Lorenzo Bianconi
2021-04-20 13:49 ` Magnus Karlsson
2021-04-21 12:47 ` Jesper Dangaard Brouer
2021-04-21 14:12 ` Magnus Karlsson
2021-04-21 15:39 ` Jesper Dangaard Brouer
2021-04-22 10:24 ` Magnus Karlsson
2021-04-22 14:42 ` Jesper Dangaard Brouer
2021-04-22 15:05 ` Jesper Dangaard Brouer [this message]
2021-04-23 5:28 ` Crash for i40e on net-next (was: [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support) Magnus Karlsson
2021-04-23 16:43 ` Alexander Duyck
2021-04-25 9:45 ` Magnus Karlsson
2021-04-27 18:28 ` [PATCH v8 bpf-next 00/14] mvneta: introduce XDP multi-buffer support Lorenzo Bianconi
2021-04-28 7:41 ` Magnus Karlsson
2021-04-29 12:49 ` Jesper Dangaard Brouer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210422170508.22c58226@carbon \
--to=brouer@redhat.com \
--cc=alexander.duyck@gmail.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=dsahern@kernel.org \
--cc=echaudro@redhat.com \
--cc=jasowang@redhat.com \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=lorenzo@kernel.org \
--cc=maciej.fijalkowski@intel.com \
--cc=magnus.karlsson@gmail.com \
--cc=netdev@vger.kernel.org \
--cc=saeed@kernel.org \
--cc=tirthendu.sarkar@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.