* Page faults in tracepoint caused by aliased pointer
@ 2024-02-12 22:14 Yan Zhai
2024-02-12 22:55 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 11+ messages in thread
From: Yan Zhai @ 2024-02-12 22:14 UTC (permalink / raw)
To: bpf; +Cc: kernel-team, jakub, ignat
Hello!
We are getting page fault errors inside BPF tracepoint that accessed
not-present pages. This caused kernel panic:
[717542.963064][T897981] BUG: unable to handle page fault for address: ffffffffff600c7d
[717542.975692][T897981] #PF: supervisor read access in kernel mode
[717542.986496][T897981] #PF: error_code(0x0000) - not-present page
[717542.997237][T897981] PGD 1965012067 P4D 1965012067 PUD 1965014067 PMD 1965016067 PTE 0
[717543.009965][T897981] Oops: 0000 [#1] PREEMPT SMP NOPTI
[717543.019835][T897981] CPU: 34 PID: 897981 Comm: warp-service Kdump: loaded Tainted: G O 6.1.74-cloudflare-2024.1.14 #1
[717543.041140][T897981] Hardware name: HYVE EDGE-METAL-GEN11/HS1811D_Lite, BIOS V0.11-sig 12/23/2022
[717543.059260][T897981] RIP: 0010:bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
[717543.071449][T897981] Code: ff eb 07 48 8b bf f8 04 00 00 49 bb 80 0e 00 00 00 80 00 00 4c 39 df 72 0c 49 89 fb 49 81 c3 80 0e 00 00 73 05 45 31 ed eb 07 <4c> 8b af 80 0e 00 00 48 89 ee 48 83 c6 f0 48 bf 00 04 7a 0a 3d 9e
[717543.104780][T897981] RSP: 0018:ffffaece810efab8 EFLAGS: 00010286
[717543.115372][T897981] RAX: 0000000000000000 RBX: ffffcea96b4ae350 RCX: 0000000000000010
[717543.127887][T897981] RDX: 0000000000000030 RSI: ffffffffac168443 RDI: ffffffffff5ffdfd
[717543.140325][T897981] RBP: ffffaece810efb28 R08: ffff9e61e3b27c80 R09: 000000000000e000
[717543.152712][T897981] R10: 0000000000000041 R11: ffffffffff600c7d R12: 00028c9a1e371991
[717543.165011][T897981] R13: 0000000000000000 R14: ffff9e6339dce8c0 R15: ffff9e61e3b27c00
[717543.177253][T897981] FS: 00007f769a1fd6c0(0000) GS:ffff9e6bdfa80000(0000) knlGS:0000000000000000
[717543.194511][T897981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[717543.205261][T897981] CR2: ffffffffff600c7d CR3: 0000003d21706005 CR4: 0000000000770ee0
[717543.217411][T897981] PKRU: 55555554
[717543.224999][T897981] Call Trace:
[717543.232224][T897981] <TASK>
[717543.239016][T897981] ? __die+0x20/0x70
[717543.246661][T897981] ? page_fault_oops+0x150/0x490
[717543.255270][T897981] ? __sk_dst_check+0x39/0xa0
[717543.263548][T897981] ? inet6_csk_route_socket+0x123/0x200
[717543.272622][T897981] ? exc_page_fault+0x67/0x140
[717543.280831][T897981] ? asm_exc_page_fault+0x22/0x30
[717543.289230][T897981] ? tcp_data_queue+0xc03/0xe20
[717543.297374][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
[717543.307555][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x281/0xada
[717543.317638][T897981] ? tcp_data_queue+0xc03/0xe20
[717543.325540][T897981] bpf_trace_run3+0x92/0xc0
[717543.333026][T897981] ? tcp_data_queue+0xc03/0xe20
[717543.340823][T897981] kfree_skb_reason+0x7b/0xd0
[717543.348427][T897981] tcp_data_queue+0xc03/0xe20
[717543.355985][T897981] tcp_rcv_established+0x218/0x740
[717543.363944][T897981] tcp_v4_do_rcv+0x157/0x290
[717543.371315][T897981] tcp_v4_rcv+0xddd/0xf00
[717543.378330][T897981] ? raw_local_deliver+0xc0/0x230
[717543.385973][T897981] ip_protocol_deliver_rcu+0x32/0x200
[717543.393880][T897981] ip_local_deliver_finish+0x73/0xa0
[717543.401616][T897981] __netif_receive_skb_one_core+0x8b/0xa0
[717543.409751][T897981] netif_receive_skb+0x38/0x160
[717543.416920][T897981] tun_get_user+0xbe6/0x1080 [tun]
[717543.424292][T897981] ? mlx5e_handle_rx_dim+0x6b/0x80 [mlx5_core]
[717543.432754][T897981] ? mlx5e_napi_poll+0x710/0x720 [mlx5_core]
[717543.441007][T897981] ? tun_chr_write_iter+0x69/0xb0 [tun]
[717543.448753][T897981] tun_chr_write_iter+0x69/0xb0 [tun]
[717543.456312][T897981] vfs_write+0x2a3/0x3b0
[717543.462722][T897981] ksys_write+0x5f/0xe0
[717543.469018][T897981] do_syscall_64+0x3b/0x90
[717543.475522][T897981] entry_SYSCALL_64_after_hwframe+0x4c/0xb6
[717543.483443][T897981] RIP: 0033:0x7f76b3b3027f
[717543.489848][T897981] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c d5 f8 ff 48
[717543.515551][T897981] RSP: 002b:00007f769a1f9870 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[717543.526219][T897981] RAX: ffffffffffffffda RBX: 0000000000000500 RCX: 00007f76b3b3027f
[717543.536507][T897981] RDX: 0000000000000500 RSI: 00007f761a694a00 RDI: 00000000000015a4
[717543.546815][T897981] RBP: 00007f75f53cf600 R08: 0000000000000000 R09: 00000000000272c8
[717543.557136][T897981] R10: 00000000000075dc R11: 0000000000000293 R12: 00007f76b37b0198
[717543.567447][T897981] R13: 0000000000000000 R14: 00007f76b37a4000 R15: 0000000000000004
[717543.577777][T897981] </TASK>
[717543.583106][T897981] Modules linked in: mptcp_diag raw_diag unix_diag xt_LOG nf_log_syslog overlay nft_compat xt_hashlimit ip_set_hash_netport xt_length esp4 nf_conntrack_netlink nft_fwd_netdev nf_dup_netdev xfrm_interface xfrm6_tunnel nft_numgen nft_log nft_limit dummy xfrm_user xfrm_algo fou6 ip6_tunnel tunnel6 ipip mpls_gso mpls_iptunnel mpls_router sit tunnel4 fou nft_ct nf_tables cls_bpf ip_gre gre ip_tunnel geneve ip6_udp_tunnel udp_tunnel zstd zstd_compress zram zsmalloc sch_ingress tcp_diag veth tun udp_diag inet_diag dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpmss iptable_filter xt_TCPMSS xt_bpf xt_limit xt_multiport xt_NFLOG nfnetlink_log xt_connbytes xt_connlabel xt_statistic xt_mark xt_connmark xt_conntrack iptable_mangle xt_nat iptable_nat nf_nat xt_owner xt_set xt_comment xt_tcpudp xt_CT iptable_raw
[717543.583186][T897981] ip_set_hash_ip ip_set_hash_net ip_set nfnetlink tcp_bbr sch_fq nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 algif_skcipher af_alg raid0 md_mod essiv dm_crypt trusted asn1_encoder tee 8021q garp mrp stp llc nvme_fabrics ipmi_ssif amd64_edac kvm_amd kvm irqbypass crc32_pclmul crc32c_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 mlx5_core aesni_intel acpi_ipmi rapl ipmi_si mlxfw xhci_pci nvme tls ipmi_devintf tiny_power_button xhci_hcd nvme_core psample ccp i2c_piix4 ipmi_msghandler button fuse dm_mod dax efivarfs ip_tables x_tables bcmcrypt(O) crypto_simd cryptd [last unloaded: kheaders]
[717543.774881][T897981] CR2: ffffffffff600c7d
The panic happens as we inspect dropped out of order TCP packets in kfree_skb
tracepoint with a tp_btf program, and try to read out the network namespace
cookie via:
skb->dev->nd_net.net->net_cookie
Code generation looks fine on x86_64 with 4 layer pagetable, but the verifier
placed boundary check is not sufficient to catch the issue: skb->dev is alised
as skb->rbnode in the same union after packets entered TCP state machine, and
the out of order queue is one of such rbnode users:
; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
2bd: movabs $0x800000000010,%r11
2c7: cmp %r11,%r15
2ca: jb 0x000002d8
2cc: mov %r15,%r11
2cf: add $0x10,%r11
2d6: jae 0x000002dc
2d8: xor %edi,%edi
2da: jmp 0x000002e0
2dc: mov 0x10(%r15),%rdi
; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
2e0: movabs $0x8000000004f8,%r11
2ea: cmp %r11,%rdi <--- (1) rdi is a valid rbnode*, not net_device*
2ed: jb 0x000002fb
2ef: mov %rdi,%r11
2f2: add $0x4f8,%r11
2f9: jae 0x000002ff
2fb: xor %edi,%edi
2fd: jmp 0x00000306
2ff: mov 0x4f8(%rdi),%rdi
; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
306: movabs $0x800000000e80,%r11
310: cmp %r11,%rdi <--- (2) rdi is a wild ptr now
313: jb 0x00000321
315: mov %rdi,%r11
318: add $0xe80,%r11
31f: jae 0x00000326
321: xor %r13d,%r13d
324: jmp 0x0000032d
326: mov 0xe80(%rdi),%r13 <--- (3) fault
32d: mov %rbp,%rsi
OOO happens a lot on our servers but this is the first time we noticed
such panic since we had deployed the program for a while. For bpf list
I think the question is mainly about what to do in this scenario:
apparently it is a valid kernel pointer at step (1) above, but it's
just not the type we assumed, which leads to a wild pointer at (2) and
caused fault at (3). I am not aware of a way to determine such aliased
pointer is good or not in general. Is it possible to PF safer in this
case, like returning from PF handler to the end of tracing program?
thanks
Yan
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-12 22:14 Page faults in tracepoint caused by aliased pointer Yan Zhai
@ 2024-02-12 22:55 ` Kumar Kartikeya Dwivedi
2024-02-12 23:15 ` Ignat Korchagin
0 siblings, 1 reply; 11+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2024-02-12 22:55 UTC (permalink / raw)
To: Yan Zhai; +Cc: bpf, kernel-team, jakub, ignat
On Mon, 12 Feb 2024 at 23:14, Yan Zhai <yan@cloudflare.com> wrote:
>
> Hello!
>
> We are getting page fault errors inside BPF tracepoint that accessed
> not-present pages. This caused kernel panic:
>
> [717542.963064][T897981] BUG: unable to handle page fault for address: ffffffffff600c7d
> [717542.975692][T897981] #PF: supervisor read access in kernel mode
> [717542.986496][T897981] #PF: error_code(0x0000) - not-present page
> [717542.997237][T897981] PGD 1965012067 P4D 1965012067 PUD 1965014067 PMD 1965016067 PTE 0
> [717543.009965][T897981] Oops: 0000 [#1] PREEMPT SMP NOPTI
> [717543.019835][T897981] CPU: 34 PID: 897981 Comm: warp-service Kdump: loaded Tainted: G O 6.1.74-cloudflare-2024.1.14 #1
> [717543.041140][T897981] Hardware name: HYVE EDGE-METAL-GEN11/HS1811D_Lite, BIOS V0.11-sig 12/23/2022
> [717543.059260][T897981] RIP: 0010:bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
> [717543.071449][T897981] Code: ff eb 07 48 8b bf f8 04 00 00 49 bb 80 0e 00 00 00 80 00 00 4c 39 df 72 0c 49 89 fb 49 81 c3 80 0e 00 00 73 05 45 31 ed eb 07 <4c> 8b af 80 0e 00 00 48 89 ee 48 83 c6 f0 48 bf 00 04 7a 0a 3d 9e
> [717543.104780][T897981] RSP: 0018:ffffaece810efab8 EFLAGS: 00010286
> [717543.115372][T897981] RAX: 0000000000000000 RBX: ffffcea96b4ae350 RCX: 0000000000000010
> [717543.127887][T897981] RDX: 0000000000000030 RSI: ffffffffac168443 RDI: ffffffffff5ffdfd
> [717543.140325][T897981] RBP: ffffaece810efb28 R08: ffff9e61e3b27c80 R09: 000000000000e000
> [717543.152712][T897981] R10: 0000000000000041 R11: ffffffffff600c7d R12: 00028c9a1e371991
> [717543.165011][T897981] R13: 0000000000000000 R14: ffff9e6339dce8c0 R15: ffff9e61e3b27c00
> [717543.177253][T897981] FS: 00007f769a1fd6c0(0000) GS:ffff9e6bdfa80000(0000) knlGS:0000000000000000
> [717543.194511][T897981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [717543.205261][T897981] CR2: ffffffffff600c7d CR3: 0000003d21706005 CR4: 0000000000770ee0
> [717543.217411][T897981] PKRU: 55555554
> [717543.224999][T897981] Call Trace:
> [717543.232224][T897981] <TASK>
> [717543.239016][T897981] ? __die+0x20/0x70
> [717543.246661][T897981] ? page_fault_oops+0x150/0x490
> [717543.255270][T897981] ? __sk_dst_check+0x39/0xa0
> [717543.263548][T897981] ? inet6_csk_route_socket+0x123/0x200
> [717543.272622][T897981] ? exc_page_fault+0x67/0x140
> [717543.280831][T897981] ? asm_exc_page_fault+0x22/0x30
> [717543.289230][T897981] ? tcp_data_queue+0xc03/0xe20
> [717543.297374][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
> [717543.307555][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x281/0xada
> [717543.317638][T897981] ? tcp_data_queue+0xc03/0xe20
> [717543.325540][T897981] bpf_trace_run3+0x92/0xc0
> [717543.333026][T897981] ? tcp_data_queue+0xc03/0xe20
> [717543.340823][T897981] kfree_skb_reason+0x7b/0xd0
> [717543.348427][T897981] tcp_data_queue+0xc03/0xe20
> [717543.355985][T897981] tcp_rcv_established+0x218/0x740
> [717543.363944][T897981] tcp_v4_do_rcv+0x157/0x290
> [717543.371315][T897981] tcp_v4_rcv+0xddd/0xf00
> [717543.378330][T897981] ? raw_local_deliver+0xc0/0x230
> [717543.385973][T897981] ip_protocol_deliver_rcu+0x32/0x200
> [717543.393880][T897981] ip_local_deliver_finish+0x73/0xa0
> [717543.401616][T897981] __netif_receive_skb_one_core+0x8b/0xa0
> [717543.409751][T897981] netif_receive_skb+0x38/0x160
> [717543.416920][T897981] tun_get_user+0xbe6/0x1080 [tun]
> [717543.424292][T897981] ? mlx5e_handle_rx_dim+0x6b/0x80 [mlx5_core]
> [717543.432754][T897981] ? mlx5e_napi_poll+0x710/0x720 [mlx5_core]
> [717543.441007][T897981] ? tun_chr_write_iter+0x69/0xb0 [tun]
> [717543.448753][T897981] tun_chr_write_iter+0x69/0xb0 [tun]
> [717543.456312][T897981] vfs_write+0x2a3/0x3b0
> [717543.462722][T897981] ksys_write+0x5f/0xe0
> [717543.469018][T897981] do_syscall_64+0x3b/0x90
> [717543.475522][T897981] entry_SYSCALL_64_after_hwframe+0x4c/0xb6
> [717543.483443][T897981] RIP: 0033:0x7f76b3b3027f
> [717543.489848][T897981] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c d5 f8 ff 48
> [717543.515551][T897981] RSP: 002b:00007f769a1f9870 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
> [717543.526219][T897981] RAX: ffffffffffffffda RBX: 0000000000000500 RCX: 00007f76b3b3027f
> [717543.536507][T897981] RDX: 0000000000000500 RSI: 00007f761a694a00 RDI: 00000000000015a4
> [717543.546815][T897981] RBP: 00007f75f53cf600 R08: 0000000000000000 R09: 00000000000272c8
> [717543.557136][T897981] R10: 00000000000075dc R11: 0000000000000293 R12: 00007f76b37b0198
> [717543.567447][T897981] R13: 0000000000000000 R14: 00007f76b37a4000 R15: 0000000000000004
> [717543.577777][T897981] </TASK>
> [717543.583106][T897981] Modules linked in: mptcp_diag raw_diag unix_diag xt_LOG nf_log_syslog overlay nft_compat xt_hashlimit ip_set_hash_netport xt_length esp4 nf_conntrack_netlink nft_fwd_netdev nf_dup_netdev xfrm_interface xfrm6_tunnel nft_numgen nft_log nft_limit dummy xfrm_user xfrm_algo fou6 ip6_tunnel tunnel6 ipip mpls_gso mpls_iptunnel mpls_router sit tunnel4 fou nft_ct nf_tables cls_bpf ip_gre gre ip_tunnel geneve ip6_udp_tunnel udp_tunnel zstd zstd_compress zram zsmalloc sch_ingress tcp_diag veth tun udp_diag inet_diag dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpmss iptable_filter xt_TCPMSS xt_bpf xt_limit xt_multiport xt_NFLOG nfnetlink_log xt_connbytes xt_connlabel xt_statistic xt_mark xt_connmark xt_conntrack iptable_mangle xt_nat iptable_nat nf_nat xt_owner xt_set xt_comment xt_tcpudp xt_CT iptable_raw
> [717543.583186][T897981] ip_set_hash_ip ip_set_hash_net ip_set nfnetlink tcp_bbr sch_fq nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 algif_skcipher af_alg raid0 md_mod essiv dm_crypt trusted asn1_encoder tee 8021q garp mrp stp llc nvme_fabrics ipmi_ssif amd64_edac kvm_amd kvm irqbypass crc32_pclmul crc32c_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 mlx5_core aesni_intel acpi_ipmi rapl ipmi_si mlxfw xhci_pci nvme tls ipmi_devintf tiny_power_button xhci_hcd nvme_core psample ccp i2c_piix4 ipmi_msghandler button fuse dm_mod dax efivarfs ip_tables x_tables bcmcrypt(O) crypto_simd cryptd [last unloaded: kheaders]
> [717543.774881][T897981] CR2: ffffffffff600c7d
>
> The panic happens as we inspect dropped out of order TCP packets in kfree_skb
> tracepoint with a tp_btf program, and try to read out the network namespace
> cookie via:
>
> skb->dev->nd_net.net->net_cookie
>
> Code generation looks fine on x86_64 with 4 layer pagetable, but the verifier
> placed boundary check is not sufficient to catch the issue: skb->dev is alised
> as skb->rbnode in the same union after packets entered TCP state machine, and
> the out of order queue is one of such rbnode users:
>
> ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> 2bd: movabs $0x800000000010,%r11
> 2c7: cmp %r11,%r15
> 2ca: jb 0x000002d8
> 2cc: mov %r15,%r11
> 2cf: add $0x10,%r11
> 2d6: jae 0x000002dc
> 2d8: xor %edi,%edi
> 2da: jmp 0x000002e0
> 2dc: mov 0x10(%r15),%rdi
> ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> 2e0: movabs $0x8000000004f8,%r11
> 2ea: cmp %r11,%rdi <--- (1) rdi is a valid rbnode*, not net_device*
> 2ed: jb 0x000002fb
> 2ef: mov %rdi,%r11
> 2f2: add $0x4f8,%r11
> 2f9: jae 0x000002ff
> 2fb: xor %edi,%edi
> 2fd: jmp 0x00000306
> 2ff: mov 0x4f8(%rdi),%rdi
> ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> 306: movabs $0x800000000e80,%r11
> 310: cmp %r11,%rdi <--- (2) rdi is a wild ptr now
> 313: jb 0x00000321
> 315: mov %rdi,%r11
> 318: add $0xe80,%r11
> 31f: jae 0x00000326
> 321: xor %r13d,%r13d
> 324: jmp 0x0000032d
> 326: mov 0xe80(%rdi),%r13 <--- (3) fault
> 32d: mov %rbp,%rsi
>
> OOO happens a lot on our servers but this is the first time we noticed
> such panic since we had deployed the program for a while. For bpf list
> I think the question is mainly about what to do in this scenario:
> apparently it is a valid kernel pointer at step (1) above, but it's
> just not the type we assumed, which leads to a wild pointer at (2) and
> caused fault at (3). I am not aware of a way to determine such aliased
> pointer is good or not in general. Is it possible to PF safer in this
> case, like returning from PF handler to the end of tracing program?
>
I think it is not supposed to panic, since exception handling for such
PROBE_MEM loads should handle such a case and mark the destination as
zero.
Something must be broken with that.
Which kernel do you observe this problem with? And do you have a
reference version where you do not see it?
Do you have a reduced reproducer for this that I could play with?
Just the part of the tp_btf program necessary to trigger this?
There were some changes made to the JIT code around the bounds
checking to reduce the instruction count.
That was in 90156f4bfa21 ("bpf, x86: Improve PROBE_MEM runtime load check").
Especially when src_reg == dst_reg, the case which happens in the
splat at 0x2ff.
Nothing else comes immediately to mind in terms of changes that could
affect this exception handling stuff.
> thanks
> Yan
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-12 22:55 ` Kumar Kartikeya Dwivedi
@ 2024-02-12 23:15 ` Ignat Korchagin
2024-02-12 23:27 ` Yan Zhai
2024-02-12 23:33 ` Alexei Starovoitov
0 siblings, 2 replies; 11+ messages in thread
From: Ignat Korchagin @ 2024-02-12 23:15 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi; +Cc: Yan Zhai, bpf, kernel-team, jakub
On Mon, Feb 12, 2024 at 10:55 PM Kumar Kartikeya Dwivedi
<memxor@gmail.com> wrote:
>
> On Mon, 12 Feb 2024 at 23:14, Yan Zhai <yan@cloudflare.com> wrote:
> >
> > Hello!
> >
> > We are getting page fault errors inside BPF tracepoint that accessed
> > not-present pages. This caused kernel panic:
> >
> > [717542.963064][T897981] BUG: unable to handle page fault for address: ffffffffff600c7d
> > [717542.975692][T897981] #PF: supervisor read access in kernel mode
> > [717542.986496][T897981] #PF: error_code(0x0000) - not-present page
> > [717542.997237][T897981] PGD 1965012067 P4D 1965012067 PUD 1965014067 PMD 1965016067 PTE 0
> > [717543.009965][T897981] Oops: 0000 [#1] PREEMPT SMP NOPTI
> > [717543.019835][T897981] CPU: 34 PID: 897981 Comm: warp-service Kdump: loaded Tainted: G O 6.1.74-cloudflare-2024.1.14 #1
> > [717543.041140][T897981] Hardware name: HYVE EDGE-METAL-GEN11/HS1811D_Lite, BIOS V0.11-sig 12/23/2022
> > [717543.059260][T897981] RIP: 0010:bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
> > [717543.071449][T897981] Code: ff eb 07 48 8b bf f8 04 00 00 49 bb 80 0e 00 00 00 80 00 00 4c 39 df 72 0c 49 89 fb 49 81 c3 80 0e 00 00 73 05 45 31 ed eb 07 <4c> 8b af 80 0e 00 00 48 89 ee 48 83 c6 f0 48 bf 00 04 7a 0a 3d 9e
> > [717543.104780][T897981] RSP: 0018:ffffaece810efab8 EFLAGS: 00010286
> > [717543.115372][T897981] RAX: 0000000000000000 RBX: ffffcea96b4ae350 RCX: 0000000000000010
> > [717543.127887][T897981] RDX: 0000000000000030 RSI: ffffffffac168443 RDI: ffffffffff5ffdfd
> > [717543.140325][T897981] RBP: ffffaece810efb28 R08: ffff9e61e3b27c80 R09: 000000000000e000
> > [717543.152712][T897981] R10: 0000000000000041 R11: ffffffffff600c7d R12: 00028c9a1e371991
> > [717543.165011][T897981] R13: 0000000000000000 R14: ffff9e6339dce8c0 R15: ffff9e61e3b27c00
> > [717543.177253][T897981] FS: 00007f769a1fd6c0(0000) GS:ffff9e6bdfa80000(0000) knlGS:0000000000000000
> > [717543.194511][T897981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [717543.205261][T897981] CR2: ffffffffff600c7d CR3: 0000003d21706005 CR4: 0000000000770ee0
> > [717543.217411][T897981] PKRU: 55555554
> > [717543.224999][T897981] Call Trace:
> > [717543.232224][T897981] <TASK>
> > [717543.239016][T897981] ? __die+0x20/0x70
> > [717543.246661][T897981] ? page_fault_oops+0x150/0x490
> > [717543.255270][T897981] ? __sk_dst_check+0x39/0xa0
> > [717543.263548][T897981] ? inet6_csk_route_socket+0x123/0x200
> > [717543.272622][T897981] ? exc_page_fault+0x67/0x140
> > [717543.280831][T897981] ? asm_exc_page_fault+0x22/0x30
> > [717543.289230][T897981] ? tcp_data_queue+0xc03/0xe20
> > [717543.297374][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
> > [717543.307555][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x281/0xada
> > [717543.317638][T897981] ? tcp_data_queue+0xc03/0xe20
> > [717543.325540][T897981] bpf_trace_run3+0x92/0xc0
> > [717543.333026][T897981] ? tcp_data_queue+0xc03/0xe20
> > [717543.340823][T897981] kfree_skb_reason+0x7b/0xd0
> > [717543.348427][T897981] tcp_data_queue+0xc03/0xe20
> > [717543.355985][T897981] tcp_rcv_established+0x218/0x740
> > [717543.363944][T897981] tcp_v4_do_rcv+0x157/0x290
> > [717543.371315][T897981] tcp_v4_rcv+0xddd/0xf00
> > [717543.378330][T897981] ? raw_local_deliver+0xc0/0x230
> > [717543.385973][T897981] ip_protocol_deliver_rcu+0x32/0x200
> > [717543.393880][T897981] ip_local_deliver_finish+0x73/0xa0
> > [717543.401616][T897981] __netif_receive_skb_one_core+0x8b/0xa0
> > [717543.409751][T897981] netif_receive_skb+0x38/0x160
> > [717543.416920][T897981] tun_get_user+0xbe6/0x1080 [tun]
> > [717543.424292][T897981] ? mlx5e_handle_rx_dim+0x6b/0x80 [mlx5_core]
> > [717543.432754][T897981] ? mlx5e_napi_poll+0x710/0x720 [mlx5_core]
> > [717543.441007][T897981] ? tun_chr_write_iter+0x69/0xb0 [tun]
> > [717543.448753][T897981] tun_chr_write_iter+0x69/0xb0 [tun]
> > [717543.456312][T897981] vfs_write+0x2a3/0x3b0
> > [717543.462722][T897981] ksys_write+0x5f/0xe0
> > [717543.469018][T897981] do_syscall_64+0x3b/0x90
> > [717543.475522][T897981] entry_SYSCALL_64_after_hwframe+0x4c/0xb6
> > [717543.483443][T897981] RIP: 0033:0x7f76b3b3027f
> > [717543.489848][T897981] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c d5 f8 ff 48
> > [717543.515551][T897981] RSP: 002b:00007f769a1f9870 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
> > [717543.526219][T897981] RAX: ffffffffffffffda RBX: 0000000000000500 RCX: 00007f76b3b3027f
> > [717543.536507][T897981] RDX: 0000000000000500 RSI: 00007f761a694a00 RDI: 00000000000015a4
> > [717543.546815][T897981] RBP: 00007f75f53cf600 R08: 0000000000000000 R09: 00000000000272c8
> > [717543.557136][T897981] R10: 00000000000075dc R11: 0000000000000293 R12: 00007f76b37b0198
> > [717543.567447][T897981] R13: 0000000000000000 R14: 00007f76b37a4000 R15: 0000000000000004
> > [717543.577777][T897981] </TASK>
> > [717543.583106][T897981] Modules linked in: mptcp_diag raw_diag unix_diag xt_LOG nf_log_syslog overlay nft_compat xt_hashlimit ip_set_hash_netport xt_length esp4 nf_conntrack_netlink nft_fwd_netdev nf_dup_netdev xfrm_interface xfrm6_tunnel nft_numgen nft_log nft_limit dummy xfrm_user xfrm_algo fou6 ip6_tunnel tunnel6 ipip mpls_gso mpls_iptunnel mpls_router sit tunnel4 fou nft_ct nf_tables cls_bpf ip_gre gre ip_tunnel geneve ip6_udp_tunnel udp_tunnel zstd zstd_compress zram zsmalloc sch_ingress tcp_diag veth tun udp_diag inet_diag dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpmss iptable_filter xt_TCPMSS xt_bpf xt_limit xt_multiport xt_NFLOG nfnetlink_log xt_connbytes xt_connlabel xt_statistic xt_mark xt_connmark xt_conntrack iptable_mangle xt_nat iptable_nat nf_nat xt_owner xt_set xt_comment xt_tcpudp xt_CT iptable_raw
> > [717543.583186][T897981] ip_set_hash_ip ip_set_hash_net ip_set nfnetlink tcp_bbr sch_fq nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 algif_skcipher af_alg raid0 md_mod essiv dm_crypt trusted asn1_encoder tee 8021q garp mrp stp llc nvme_fabrics ipmi_ssif amd64_edac kvm_amd kvm irqbypass crc32_pclmul crc32c_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 mlx5_core aesni_intel acpi_ipmi rapl ipmi_si mlxfw xhci_pci nvme tls ipmi_devintf tiny_power_button xhci_hcd nvme_core psample ccp i2c_piix4 ipmi_msghandler button fuse dm_mod dax efivarfs ip_tables x_tables bcmcrypt(O) crypto_simd cryptd [last unloaded: kheaders]
> > [717543.774881][T897981] CR2: ffffffffff600c7d
> >
> > The panic happens as we inspect dropped out of order TCP packets in kfree_skb
> > tracepoint with a tp_btf program, and try to read out the network namespace
> > cookie via:
> >
> > skb->dev->nd_net.net->net_cookie
> >
> > Code generation looks fine on x86_64 with 4 layer pagetable, but the verifier
> > placed boundary check is not sufficient to catch the issue: skb->dev is alised
> > as skb->rbnode in the same union after packets entered TCP state machine, and
> > the out of order queue is one of such rbnode users:
> >
> > ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> > 2bd: movabs $0x800000000010,%r11
> > 2c7: cmp %r11,%r15
> > 2ca: jb 0x000002d8
> > 2cc: mov %r15,%r11
> > 2cf: add $0x10,%r11
> > 2d6: jae 0x000002dc
> > 2d8: xor %edi,%edi
> > 2da: jmp 0x000002e0
> > 2dc: mov 0x10(%r15),%rdi
> > ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> > 2e0: movabs $0x8000000004f8,%r11
> > 2ea: cmp %r11,%rdi <--- (1) rdi is a valid rbnode*, not net_device*
> > 2ed: jb 0x000002fb
> > 2ef: mov %rdi,%r11
> > 2f2: add $0x4f8,%r11
> > 2f9: jae 0x000002ff
> > 2fb: xor %edi,%edi
> > 2fd: jmp 0x00000306
> > 2ff: mov 0x4f8(%rdi),%rdi
> > ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> > 306: movabs $0x800000000e80,%r11
> > 310: cmp %r11,%rdi <--- (2) rdi is a wild ptr now
> > 313: jb 0x00000321
> > 315: mov %rdi,%r11
> > 318: add $0xe80,%r11
> > 31f: jae 0x00000326
> > 321: xor %r13d,%r13d
> > 324: jmp 0x0000032d
> > 326: mov 0xe80(%rdi),%r13 <--- (3) fault
> > 32d: mov %rbp,%rsi
> >
> > OOO happens a lot on our servers but this is the first time we noticed
> > such panic since we had deployed the program for a while. For bpf list
> > I think the question is mainly about what to do in this scenario:
> > apparently it is a valid kernel pointer at step (1) above, but it's
> > just not the type we assumed, which leads to a wild pointer at (2) and
> > caused fault at (3). I am not aware of a way to determine such aliased
> > pointer is good or not in general. Is it possible to PF safer in this
> > case, like returning from PF handler to the end of tracing program?
> >
>
> I think it is not supposed to panic, since exception handling for such
> PROBE_MEM loads should handle such a case and mark the destination as
> zero.
> Something must be broken with that.
>
> Which kernel do you observe this problem with? And do you have a
> reference version where you do not see it?
> Do you have a reduced reproducer for this that I could play with?
> Just the part of the tp_btf program necessary to trigger this?
We were able to reproduce this with a simple bpftrace (version 0.17.1):
$ sudo bpftrace -kk -e 'BEGIN { print(*(uint8 *)0xffffffffff600c7d); exit(); }'
WARNING: Addrspace is not set
Attaching 1 probe...
Killed
[288931.216699][T109754] BUG: unable to handle page fault for address:
ffffffffff600c7d
[288931.217143][T109754] #PF: supervisor read access in kernel mode
[288931.217143][T109754] #PF: error_code(0x0000) - not-present page
[288931.217143][T109754] PGD fa5a1e067 P4D fa5a1e067 PUD fa5a20067 PMD
fa5a22067 PTE 0
[288931.217143][T109754] Oops: 0000 [#1] PREEMPT SMP NOPTI
[288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
6.6.16+ #10
[288931.217143][T109754] Hardware name: KubeVirt None/RHEL, BIOS
edk2-20221207gitfff6d81270b5-9.el9 12/07/2022
[288931.217143][T109754] RIP: 0010:copy_from_kernel_nofault+0x89/0xe0
[288931.217143][T109754] Code: 48 83 c3 04 48 83 c5 04 49 83 ec 04 49
83 fc 01 76 13 66 8b 03 66 89 45 00 48 83 c3 02 48 83 c5 02 49 83 ec
02 4d 85 e4 74 05 <8a> 03 88 45 00 65 48 8b 04 25 80 11 03 00 83 a8 54
1b 00 00 01 31
[288931.217143][T109754] RSP: 0018:ffff93d787777d38 EFLAGS: 00010202
[288931.217143][T109754] RAX: ffff910d4830a0c0 RBX: ffffffffff600c7d
RCX: 0000000000000010
[288931.217143][T109754] RDX: 0000000000000030 RSI: 0000000000000001
RDI: ffffffffff600c7d
[288931.217143][T109754] RBP: ffff93d787777d87 R08: 0101010101010101
R09: 00000000fffffdf4
[288931.217143][T109754] R10: 0000000000000000 R11: 0000000000000000
R12: 0000000000000001
[288931.217143][T109754] R13: ffff93d7807eb000 R14: 000000000000000a
R15: 0000000000000000
[288931.217143][T109754] FS: 00007f2818fd79c0(0000)
GS:ffff911c7f800000(0000) knlGS:0000000000000000
[288931.217143][T109754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[288931.217143][T109754] CR2: ffffffffff600c7d CR3: 00000001174f4000
CR4: 0000000000350ee0
[288931.217143][T109754] Call Trace:
[288931.217143][T109754] <TASK>
[288931.217143][T109754] ? __die+0x1f/0x70
[288931.217143][T109754] ? page_fault_oops+0x151/0x480
[288931.217143][T109754] ? srso_return_thunk+0x5/0x10
[288931.217143][T109754] ? alloc_empty_file+0x7a/0x120
[288931.217143][T109754] ? __d_instantiate+0x34/0xf0
[288931.217143][T109754] ? srso_return_thunk+0x5/0x10
[288931.217143][T109754] ? alloc_file+0x9b/0x170
[288931.217143][T109754] ? exc_page_fault+0x68/0x140
[288931.217143][T109754] ? asm_exc_page_fault+0x22/0x30
[288931.217143][T109754] ? copy_from_kernel_nofault+0x89/0xe0
[288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
[288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
[288931.217143][T109754] bpf_prog_27620a19791a7c9c_BEGIN+0x2e/0xe7
[288931.217143][T109754] ? srso_return_thunk+0x5/0x10
[288931.217143][T109754] __bpf_prog_test_run_raw_tp+0x2e/0x90
[288931.217143][T109754] bpf_prog_test_run_raw_tp+0xe6/0x1c0
[288931.217143][T109754] __sys_bpf+0x93a/0x26a0
[288931.217143][T109754] ? srso_return_thunk+0x5/0x10
[288931.217143][T109754] ? __check_object_size+0x16a/0x2c0
[288931.217143][T109754] ? srso_return_thunk+0x5/0x10
[288931.217143][T109754] __x64_sys_bpf+0x1a/0x30
[288931.217143][T109754] do_syscall_64+0x3a/0x90
[288931.217143][T109754] entry_SYSCALL_64_after_hwframe+0x56/0xc0
[288931.217143][T109754] RIP: 0033:0x7f281b320719
[288931.217143][T109754] Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00
00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c
8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b7 06 0d 00 f7
d8 64 89 01 48
[288931.217143][T109754] RSP: 002b:00007ffdab7f0118 EFLAGS: 00000246
ORIG_RAX: 0000000000000141
[288931.217143][T109754] RAX: ffffffffffffffda RBX: 00007ffdab7f01e8
RCX: 00007f281b320719
[288931.217143][T109754] RDX: 0000000000000050 RSI: 00007ffdab7f0120
RDI: 000000000000000a
[288931.217143][T109754] RBP: 00000000024e27a0 R08: 00007f28100cf880
R09: 0000000000000016
[288931.217143][T109754] R10: 0000000000000007 R11: 0000000000000246
R12: 00000000ffffffff
[288931.217143][T109754] R13: 00007ffdab7f10c8 R14: 000000000000000d
R15: 00000000024e27a0
[288931.217143][T109754] </TASK>
[288931.217143][T109754] Modules linked in: xt_bpf xt_conntrack
nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype
nft_compat nf_tables br_netfilter bridge stp llc overlay kvm_amd ccp
kvm irqbypass crc32_pclmul sha512_ssse3 sha256_ssse3 sha1_ssse3
aesni_intel crypto_simd cryptd virtio_balloon virtio_console
tiny_power_button button fuse dm_mod dax configfs nfnetlink efivarfs
ip_tables x_tables virtio_net net_failover virtio_blk virtio_scsi
failover crc32c_intel i2c_i801 virtio_pci i2c_smbus
virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
[288931.217143][T109754] CR2: ffffffffff600c7d
[288931.217143][T109754] ---[ end trace 0000000000000000 ]---
[288931.509063][T109754] RIP: 0010:copy_from_kernel_nofault+0x89/0xe0
[288931.509063][T109754] Code: 48 83 c3 04 48 83 c5 04 49 83 ec 04 49
83 fc 01 76 13 66 8b 03 66 89 45 00 48 83 c3 02 48 83 c5 02 49 83 ec
02 4d 85 e4 74 05 <8a> 03 88 45 00 65 48 8b 04 25 80 11 03 00 83 a8 54
1b 00 00 01 31
[288931.509063][T109754] RSP: 0018:ffff93d787777d38 EFLAGS: 00010202
[288931.509063][T109754] RAX: ffff910d4830a0c0 RBX: ffffffffff600c7d
RCX: 0000000000000010
[288931.509063][T109754] RDX: 0000000000000030 RSI: 0000000000000001
RDI: ffffffffff600c7d
[288931.509063][T109754] RBP: ffff93d787777d87 R08: 0101010101010101
R09: 00000000fffffdf4
[288931.509063][T109754] R10: 0000000000000000 R11: 0000000000000000
R12: 0000000000000001
[288931.509063][T109754] R13: ffff93d7807eb000 R14: 000000000000000a
R15: 0000000000000000
[288931.509063][T109754] FS: 00007f2818fd79c0(0000)
GS:ffff911c7f800000(0000) knlGS:0000000000000000
[288931.509063][T109754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[288931.509063][T109754] CR2: ffffffffff600c7d CR3: 00000001174f4000
CR4: 0000000000350ee0
[288931.509063][T109754] note: bpftrace[109754] exited with irqs disabled
[288932.319062][T109754] note: bpftrace[109754] exited with preempt_count 1
And Jakub CCed here did it for 6.8.0-rc2+
> There were some changes made to the JIT code around the bounds
> checking to reduce the instruction count.
> That was in 90156f4bfa21 ("bpf, x86: Improve PROBE_MEM runtime load check").
> Especially when src_reg == dst_reg, the case which happens in the
> splat at 0x2ff.
> Nothing else comes immediately to mind in terms of changes that could
> affect this exception handling stuff.
>
> > thanks
> > Yan
> >
Ignat
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-12 23:15 ` Ignat Korchagin
@ 2024-02-12 23:27 ` Yan Zhai
2024-02-12 23:33 ` Alexei Starovoitov
1 sibling, 0 replies; 11+ messages in thread
From: Yan Zhai @ 2024-02-12 23:27 UTC (permalink / raw)
To: Ignat Korchagin; +Cc: Kumar Kartikeya Dwivedi, bpf, kernel-team, jakub
On Mon, Feb 12, 2024 at 5:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
>
> On Mon, Feb 12, 2024 at 10:55 PM Kumar Kartikeya Dwivedi
> <memxor@gmail.com> wrote:
> >
> > On Mon, 12 Feb 2024 at 23:14, Yan Zhai <yan@cloudflare.com> wrote:
> > >
> > > Hello!
> > >
> > > We are getting page fault errors inside BPF tracepoint that accessed
> > > not-present pages. This caused kernel panic:
> > >
> > > [717542.963064][T897981] BUG: unable to handle page fault for address: ffffffffff600c7d
> > > [717542.975692][T897981] #PF: supervisor read access in kernel mode
> > > [717542.986496][T897981] #PF: error_code(0x0000) - not-present page
> > > [717542.997237][T897981] PGD 1965012067 P4D 1965012067 PUD 1965014067 PMD 1965016067 PTE 0
> > > [717543.009965][T897981] Oops: 0000 [#1] PREEMPT SMP NOPTI
> > > [717543.019835][T897981] CPU: 34 PID: 897981 Comm: warp-service Kdump: loaded Tainted: G O 6.1.74-cloudflare-2024.1.14 #1
> > > [717543.041140][T897981] Hardware name: HYVE EDGE-METAL-GEN11/HS1811D_Lite, BIOS V0.11-sig 12/23/2022
> > > [717543.059260][T897981] RIP: 0010:bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
> > > [717543.071449][T897981] Code: ff eb 07 48 8b bf f8 04 00 00 49 bb 80 0e 00 00 00 80 00 00 4c 39 df 72 0c 49 89 fb 49 81 c3 80 0e 00 00 73 05 45 31 ed eb 07 <4c> 8b af 80 0e 00 00 48 89 ee 48 83 c6 f0 48 bf 00 04 7a 0a 3d 9e
> > > [717543.104780][T897981] RSP: 0018:ffffaece810efab8 EFLAGS: 00010286
> > > [717543.115372][T897981] RAX: 0000000000000000 RBX: ffffcea96b4ae350 RCX: 0000000000000010
> > > [717543.127887][T897981] RDX: 0000000000000030 RSI: ffffffffac168443 RDI: ffffffffff5ffdfd
> > > [717543.140325][T897981] RBP: ffffaece810efb28 R08: ffff9e61e3b27c80 R09: 000000000000e000
> > > [717543.152712][T897981] R10: 0000000000000041 R11: ffffffffff600c7d R12: 00028c9a1e371991
> > > [717543.165011][T897981] R13: 0000000000000000 R14: ffff9e6339dce8c0 R15: ffff9e61e3b27c00
> > > [717543.177253][T897981] FS: 00007f769a1fd6c0(0000) GS:ffff9e6bdfa80000(0000) knlGS:0000000000000000
> > > [717543.194511][T897981] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [717543.205261][T897981] CR2: ffffffffff600c7d CR3: 0000003d21706005 CR4: 0000000000770ee0
> > > [717543.217411][T897981] PKRU: 55555554
> > > [717543.224999][T897981] Call Trace:
> > > [717543.232224][T897981] <TASK>
> > > [717543.239016][T897981] ? __die+0x20/0x70
> > > [717543.246661][T897981] ? page_fault_oops+0x150/0x490
> > > [717543.255270][T897981] ? __sk_dst_check+0x39/0xa0
> > > [717543.263548][T897981] ? inet6_csk_route_socket+0x123/0x200
> > > [717543.272622][T897981] ? exc_page_fault+0x67/0x140
> > > [717543.280831][T897981] ? asm_exc_page_fault+0x22/0x30
> > > [717543.289230][T897981] ? tcp_data_queue+0xc03/0xe20
> > > [717543.297374][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x326/0xada
> > > [717543.307555][T897981] ? bpf_prog_2eca29f2a4f78ed1_drop_monitor+0x281/0xada
> > > [717543.317638][T897981] ? tcp_data_queue+0xc03/0xe20
> > > [717543.325540][T897981] bpf_trace_run3+0x92/0xc0
> > > [717543.333026][T897981] ? tcp_data_queue+0xc03/0xe20
> > > [717543.340823][T897981] kfree_skb_reason+0x7b/0xd0
> > > [717543.348427][T897981] tcp_data_queue+0xc03/0xe20
> > > [717543.355985][T897981] tcp_rcv_established+0x218/0x740
> > > [717543.363944][T897981] tcp_v4_do_rcv+0x157/0x290
> > > [717543.371315][T897981] tcp_v4_rcv+0xddd/0xf00
> > > [717543.378330][T897981] ? raw_local_deliver+0xc0/0x230
> > > [717543.385973][T897981] ip_protocol_deliver_rcu+0x32/0x200
> > > [717543.393880][T897981] ip_local_deliver_finish+0x73/0xa0
> > > [717543.401616][T897981] __netif_receive_skb_one_core+0x8b/0xa0
> > > [717543.409751][T897981] netif_receive_skb+0x38/0x160
> > > [717543.416920][T897981] tun_get_user+0xbe6/0x1080 [tun]
> > > [717543.424292][T897981] ? mlx5e_handle_rx_dim+0x6b/0x80 [mlx5_core]
> > > [717543.432754][T897981] ? mlx5e_napi_poll+0x710/0x720 [mlx5_core]
> > > [717543.441007][T897981] ? tun_chr_write_iter+0x69/0xb0 [tun]
> > > [717543.448753][T897981] tun_chr_write_iter+0x69/0xb0 [tun]
> > > [717543.456312][T897981] vfs_write+0x2a3/0x3b0
> > > [717543.462722][T897981] ksys_write+0x5f/0xe0
> > > [717543.469018][T897981] do_syscall_64+0x3b/0x90
> > > [717543.475522][T897981] entry_SYSCALL_64_after_hwframe+0x4c/0xb6
> > > [717543.483443][T897981] RIP: 0033:0x7f76b3b3027f
> > > [717543.489848][T897981] Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 39 d5 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 8c d5 f8 ff 48
> > > [717543.515551][T897981] RSP: 002b:00007f769a1f9870 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
> > > [717543.526219][T897981] RAX: ffffffffffffffda RBX: 0000000000000500 RCX: 00007f76b3b3027f
> > > [717543.536507][T897981] RDX: 0000000000000500 RSI: 00007f761a694a00 RDI: 00000000000015a4
> > > [717543.546815][T897981] RBP: 00007f75f53cf600 R08: 0000000000000000 R09: 00000000000272c8
> > > [717543.557136][T897981] R10: 00000000000075dc R11: 0000000000000293 R12: 00007f76b37b0198
> > > [717543.567447][T897981] R13: 0000000000000000 R14: 00007f76b37a4000 R15: 0000000000000004
> > > [717543.577777][T897981] </TASK>
> > > [717543.583106][T897981] Modules linked in: mptcp_diag raw_diag unix_diag xt_LOG nf_log_syslog overlay nft_compat xt_hashlimit ip_set_hash_netport xt_length esp4 nf_conntrack_netlink nft_fwd_netdev nf_dup_netdev xfrm_interface xfrm6_tunnel nft_numgen nft_log nft_limit dummy xfrm_user xfrm_algo fou6 ip6_tunnel tunnel6 ipip mpls_gso mpls_iptunnel mpls_router sit tunnel4 fou nft_ct nf_tables cls_bpf ip_gre gre ip_tunnel geneve ip6_udp_tunnel udp_tunnel zstd zstd_compress zram zsmalloc sch_ingress tcp_diag veth tun udp_diag inet_diag dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_security ip6table_nat ip6_tables ipt_REJECT nf_reject_ipv4 xt_tcpmss iptable_filter xt_TCPMSS xt_bpf xt_limit xt_multiport xt_NFLOG nfnetlink_log xt_connbytes xt_connlabel xt_statistic xt_mark xt_connmark xt_conntrack iptable_mangle xt_nat iptable_nat nf_nat xt_owner xt_set xt_comment xt_tcpudp xt_CT iptable_raw
> > > [717543.583186][T897981] ip_set_hash_ip ip_set_hash_net ip_set nfnetlink tcp_bbr sch_fq nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 algif_skcipher af_alg raid0 md_mod essiv dm_crypt trusted asn1_encoder tee 8021q garp mrp stp llc nvme_fabrics ipmi_ssif amd64_edac kvm_amd kvm irqbypass crc32_pclmul crc32c_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 mlx5_core aesni_intel acpi_ipmi rapl ipmi_si mlxfw xhci_pci nvme tls ipmi_devintf tiny_power_button xhci_hcd nvme_core psample ccp i2c_piix4 ipmi_msghandler button fuse dm_mod dax efivarfs ip_tables x_tables bcmcrypt(O) crypto_simd cryptd [last unloaded: kheaders]
> > > [717543.774881][T897981] CR2: ffffffffff600c7d
> > >
> > > The panic happens as we inspect dropped out of order TCP packets in kfree_skb
> > > tracepoint with a tp_btf program, and try to read out the network namespace
> > > cookie via:
> > >
> > > skb->dev->nd_net.net->net_cookie
> > >
> > > Code generation looks fine on x86_64 with 4 layer pagetable, but the verifier
> > > placed boundary check is not sufficient to catch the issue: skb->dev is alised
> > > as skb->rbnode in the same union after packets entered TCP state machine, and
> > > the out of order queue is one of such rbnode users:
> > >
> > > ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> > > 2bd: movabs $0x800000000010,%r11
> > > 2c7: cmp %r11,%r15
> > > 2ca: jb 0x000002d8
> > > 2cc: mov %r15,%r11
> > > 2cf: add $0x10,%r11
> > > 2d6: jae 0x000002dc
> > > 2d8: xor %edi,%edi
> > > 2da: jmp 0x000002e0
> > > 2dc: mov 0x10(%r15),%rdi
> > > ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> > > 2e0: movabs $0x8000000004f8,%r11
> > > 2ea: cmp %r11,%rdi <--- (1) rdi is a valid rbnode*, not net_device*
> > > 2ed: jb 0x000002fb
> > > 2ef: mov %rdi,%r11
> > > 2f2: add $0x4f8,%r11
> > > 2f9: jae 0x000002ff
> > > 2fb: xor %edi,%edi
> > > 2fd: jmp 0x00000306
> > > 2ff: mov 0x4f8(%rdi),%rdi
> > > ; uint64_t netns_cookie = skb->dev->nd_net.net->net_cookie;
> > > 306: movabs $0x800000000e80,%r11
> > > 310: cmp %r11,%rdi <--- (2) rdi is a wild ptr now
> > > 313: jb 0x00000321
> > > 315: mov %rdi,%r11
> > > 318: add $0xe80,%r11
> > > 31f: jae 0x00000326
> > > 321: xor %r13d,%r13d
> > > 324: jmp 0x0000032d
> > > 326: mov 0xe80(%rdi),%r13 <--- (3) fault
> > > 32d: mov %rbp,%rsi
> > >
> > > OOO happens a lot on our servers but this is the first time we noticed
> > > such panic since we had deployed the program for a while. For bpf list
> > > I think the question is mainly about what to do in this scenario:
> > > apparently it is a valid kernel pointer at step (1) above, but it's
> > > just not the type we assumed, which leads to a wild pointer at (2) and
> > > caused fault at (3). I am not aware of a way to determine such aliased
> > > pointer is good or not in general. Is it possible to PF safer in this
> > > case, like returning from PF handler to the end of tracing program?
> > >
> >
> > I think it is not supposed to panic, since exception handling for such
> > PROBE_MEM loads should handle such a case and mark the destination as
> > zero.
> > Something must be broken with that.
Just to clarify, by exception handling, do you mean there are some
special treatments in the page fault handler? Or do you mean page
fault should not happen in the first place because the BPF verifier
should catch it?
> >
> > Which kernel do you observe this problem with? And do you have a
> > reference version where you do not see it?
We saw this once today on the 6.1.74 kernel with some internally
vendor patches, but none of those ring a bell to this situation. Other
6.1.74 kernel does fine. To be clear, I did see netns cookies being
read as 0 when the skb->dev is NULL. But not sure if it also can deal
with a valid but aliased pointer this way.
> > Do you have a reduced reproducer for this that I could play with?
> > Just the part of the tp_btf program necessary to trigger this?
>
> We were able to reproduce this with a simple bpftrace (version 0.17.1):
>
> $ sudo bpftrace -kk -e 'BEGIN { print(*(uint8 *)0xffffffffff600c7d); exit(); }'
> WARNING: Addrspace is not set
> Attaching 1 probe...
> Killed
> [288931.216699][T109754] BUG: unable to handle page fault for address:
> ffffffffff600c7d
> [288931.217143][T109754] #PF: supervisor read access in kernel mode
> [288931.217143][T109754] #PF: error_code(0x0000) - not-present page
> [288931.217143][T109754] PGD fa5a1e067 P4D fa5a1e067 PUD fa5a20067 PMD
> fa5a22067 PTE 0
> [288931.217143][T109754] Oops: 0000 [#1] PREEMPT SMP NOPTI
> [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
> 6.6.16+ #10
> [288931.217143][T109754] Hardware name: KubeVirt None/RHEL, BIOS
> edk2-20221207gitfff6d81270b5-9.el9 12/07/2022
> [288931.217143][T109754] RIP: 0010:copy_from_kernel_nofault+0x89/0xe0
> [288931.217143][T109754] Code: 48 83 c3 04 48 83 c5 04 49 83 ec 04 49
> 83 fc 01 76 13 66 8b 03 66 89 45 00 48 83 c3 02 48 83 c5 02 49 83 ec
> 02 4d 85 e4 74 05 <8a> 03 88 45 00 65 48 8b 04 25 80 11 03 00 83 a8 54
> 1b 00 00 01 31
> [288931.217143][T109754] RSP: 0018:ffff93d787777d38 EFLAGS: 00010202
> [288931.217143][T109754] RAX: ffff910d4830a0c0 RBX: ffffffffff600c7d
> RCX: 0000000000000010
> [288931.217143][T109754] RDX: 0000000000000030 RSI: 0000000000000001
> RDI: ffffffffff600c7d
> [288931.217143][T109754] RBP: ffff93d787777d87 R08: 0101010101010101
> R09: 00000000fffffdf4
> [288931.217143][T109754] R10: 0000000000000000 R11: 0000000000000000
> R12: 0000000000000001
> [288931.217143][T109754] R13: ffff93d7807eb000 R14: 000000000000000a
> R15: 0000000000000000
> [288931.217143][T109754] FS: 00007f2818fd79c0(0000)
> GS:ffff911c7f800000(0000) knlGS:0000000000000000
> [288931.217143][T109754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [288931.217143][T109754] CR2: ffffffffff600c7d CR3: 00000001174f4000
> CR4: 0000000000350ee0
> [288931.217143][T109754] Call Trace:
> [288931.217143][T109754] <TASK>
> [288931.217143][T109754] ? __die+0x1f/0x70
> [288931.217143][T109754] ? page_fault_oops+0x151/0x480
> [288931.217143][T109754] ? srso_return_thunk+0x5/0x10
> [288931.217143][T109754] ? alloc_empty_file+0x7a/0x120
> [288931.217143][T109754] ? __d_instantiate+0x34/0xf0
> [288931.217143][T109754] ? srso_return_thunk+0x5/0x10
> [288931.217143][T109754] ? alloc_file+0x9b/0x170
> [288931.217143][T109754] ? exc_page_fault+0x68/0x140
> [288931.217143][T109754] ? asm_exc_page_fault+0x22/0x30
> [288931.217143][T109754] ? copy_from_kernel_nofault+0x89/0xe0
> [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
> [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
> [288931.217143][T109754] bpf_prog_27620a19791a7c9c_BEGIN+0x2e/0xe7
> [288931.217143][T109754] ? srso_return_thunk+0x5/0x10
> [288931.217143][T109754] __bpf_prog_test_run_raw_tp+0x2e/0x90
> [288931.217143][T109754] bpf_prog_test_run_raw_tp+0xe6/0x1c0
> [288931.217143][T109754] __sys_bpf+0x93a/0x26a0
> [288931.217143][T109754] ? srso_return_thunk+0x5/0x10
> [288931.217143][T109754] ? __check_object_size+0x16a/0x2c0
> [288931.217143][T109754] ? srso_return_thunk+0x5/0x10
> [288931.217143][T109754] __x64_sys_bpf+0x1a/0x30
> [288931.217143][T109754] do_syscall_64+0x3a/0x90
> [288931.217143][T109754] entry_SYSCALL_64_after_hwframe+0x56/0xc0
> [288931.217143][T109754] RIP: 0033:0x7f281b320719
> [288931.217143][T109754] Code: 08 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00
> 00 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c
> 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d b7 06 0d 00 f7
> d8 64 89 01 48
> [288931.217143][T109754] RSP: 002b:00007ffdab7f0118 EFLAGS: 00000246
> ORIG_RAX: 0000000000000141
> [288931.217143][T109754] RAX: ffffffffffffffda RBX: 00007ffdab7f01e8
> RCX: 00007f281b320719
> [288931.217143][T109754] RDX: 0000000000000050 RSI: 00007ffdab7f0120
> RDI: 000000000000000a
> [288931.217143][T109754] RBP: 00000000024e27a0 R08: 00007f28100cf880
> R09: 0000000000000016
> [288931.217143][T109754] R10: 0000000000000007 R11: 0000000000000246
> R12: 00000000ffffffff
> [288931.217143][T109754] R13: 00007ffdab7f10c8 R14: 000000000000000d
> R15: 00000000024e27a0
> [288931.217143][T109754] </TASK>
> [288931.217143][T109754] Modules linked in: xt_bpf xt_conntrack
> nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack
> nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype
> nft_compat nf_tables br_netfilter bridge stp llc overlay kvm_amd ccp
> kvm irqbypass crc32_pclmul sha512_ssse3 sha256_ssse3 sha1_ssse3
> aesni_intel crypto_simd cryptd virtio_balloon virtio_console
> tiny_power_button button fuse dm_mod dax configfs nfnetlink efivarfs
> ip_tables x_tables virtio_net net_failover virtio_blk virtio_scsi
> failover crc32c_intel i2c_i801 virtio_pci i2c_smbus
> virtio_pci_legacy_dev virtio_pci_modern_dev virtio virtio_ring
> [288931.217143][T109754] CR2: ffffffffff600c7d
> [288931.217143][T109754] ---[ end trace 0000000000000000 ]---
> [288931.509063][T109754] RIP: 0010:copy_from_kernel_nofault+0x89/0xe0
> [288931.509063][T109754] Code: 48 83 c3 04 48 83 c5 04 49 83 ec 04 49
> 83 fc 01 76 13 66 8b 03 66 89 45 00 48 83 c3 02 48 83 c5 02 49 83 ec
> 02 4d 85 e4 74 05 <8a> 03 88 45 00 65 48 8b 04 25 80 11 03 00 83 a8 54
> 1b 00 00 01 31
> [288931.509063][T109754] RSP: 0018:ffff93d787777d38 EFLAGS: 00010202
> [288931.509063][T109754] RAX: ffff910d4830a0c0 RBX: ffffffffff600c7d
> RCX: 0000000000000010
> [288931.509063][T109754] RDX: 0000000000000030 RSI: 0000000000000001
> RDI: ffffffffff600c7d
> [288931.509063][T109754] RBP: ffff93d787777d87 R08: 0101010101010101
> R09: 00000000fffffdf4
> [288931.509063][T109754] R10: 0000000000000000 R11: 0000000000000000
> R12: 0000000000000001
> [288931.509063][T109754] R13: ffff93d7807eb000 R14: 000000000000000a
> R15: 0000000000000000
> [288931.509063][T109754] FS: 00007f2818fd79c0(0000)
> GS:ffff911c7f800000(0000) knlGS:0000000000000000
> [288931.509063][T109754] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [288931.509063][T109754] CR2: ffffffffff600c7d CR3: 00000001174f4000
> CR4: 0000000000350ee0
> [288931.509063][T109754] note: bpftrace[109754] exited with irqs disabled
> [288932.319062][T109754] note: bpftrace[109754] exited with preempt_count 1
>
> And Jakub CCed here did it for 6.8.0-rc2+
>
> > There were some changes made to the JIT code around the bounds
> > checking to reduce the instruction count.
> > That was in 90156f4bfa21 ("bpf, x86: Improve PROBE_MEM runtime load check").
> > Especially when src_reg == dst_reg, the case which happens in the
> > splat at 0x2ff.
> > Nothing else comes immediately to mind in terms of changes that could
> > affect this exception handling stuff.
Thanks, let me check if this is included or helps.
> >
> > > thanks
> > > Yan
> > >
>
> Ignat
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-12 23:15 ` Ignat Korchagin
2024-02-12 23:27 ` Yan Zhai
@ 2024-02-12 23:33 ` Alexei Starovoitov
2024-02-12 23:41 ` Kumar Kartikeya Dwivedi
1 sibling, 1 reply; 11+ messages in thread
From: Alexei Starovoitov @ 2024-02-12 23:33 UTC (permalink / raw)
To: Ignat Korchagin
Cc: Kumar Kartikeya Dwivedi, Yan Zhai, bpf, kernel-team,
Jakub Sitnicki
On Mon, Feb 12, 2024 at 3:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
>
> [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
> 6.6.16+ #10
...
> [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
> [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
>
> And Jakub CCed here did it for 6.8.0-rc2+
I suspect something is broken in your kernels.
Above is doing generic copy_from_kernel_nofault(),
so one should be able to crash the kernel without any bpf.
We have this in selftests/bpf:
__weak noinline struct file *bpf_testmod_return_ptr(int arg)
{
static struct file f = {};
switch (arg) {
case 1: return (void *)EINVAL; /* user addr */
case 2: return (void *)0xcafe4a11; /* user addr */
case 3: return (void *)-EINVAL; /* canonical, but invalid */
case 4: return (void *)(1ull << 60); /* non-canonical and invalid */
case 5: return (void *)~(1ull << 30); /* trigger extable */
case 6: return &f; /* valid addr */
case 7: return (void *)((long)&f | 1); /* kernel tricks */
default: return NULL;
}
}
where we check that extables setup by JIT for bpf progs are working correctly.
You should see the kernel crashing when you just run bpf selftests.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-12 23:33 ` Alexei Starovoitov
@ 2024-02-12 23:41 ` Kumar Kartikeya Dwivedi
2024-02-12 23:52 ` Alexei Starovoitov
0 siblings, 1 reply; 11+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2024-02-12 23:41 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Ignat Korchagin, Yan Zhai, bpf, kernel-team, Jakub Sitnicki
On Tue, 13 Feb 2024 at 00:34, Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon, Feb 12, 2024 at 3:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
> >
> > [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
> > 6.6.16+ #10
>
> ...
> > [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
> > [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
> >
> > And Jakub CCed here did it for 6.8.0-rc2+
>
> I suspect something is broken in your kernels.
> Above is doing generic copy_from_kernel_nofault(),
> so one should be able to crash the kernel without any bpf.
>
> We have this in selftests/bpf:
> __weak noinline struct file *bpf_testmod_return_ptr(int arg)
> {
> static struct file f = {};
>
> switch (arg) {
> case 1: return (void *)EINVAL; /* user addr */
> case 2: return (void *)0xcafe4a11; /* user addr */
> case 3: return (void *)-EINVAL; /* canonical, but invalid */
> case 4: return (void *)(1ull << 60); /* non-canonical and invalid */
> case 5: return (void *)~(1ull << 30); /* trigger extable */
> case 6: return &f; /* valid addr */
> case 7: return (void *)((long)&f | 1); /* kernel tricks */
> default: return NULL;
> }
> }
> where we check that extables setup by JIT for bpf progs are working correctly.
> You should see the kernel crashing when you just run bpf selftests.
I agree, this appears unrelated to BPF since it is happening when
using copy_from_kernel_nofault (which should be jumping to the Efault
label instead of the oops), but I think it's not specific to some
custom kernel. I can reproduce it on my dev machine on top of bpf-next
as well, and another machine with Ubuntu's generic 6.5 kernel for
24.04. And I think Ignat tried it on the mainline 6.8-rc2 as well.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-12 23:41 ` Kumar Kartikeya Dwivedi
@ 2024-02-12 23:52 ` Alexei Starovoitov
2024-02-13 0:21 ` Yan Zhai
0 siblings, 1 reply; 11+ messages in thread
From: Alexei Starovoitov @ 2024-02-12 23:52 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi
Cc: Ignat Korchagin, Yan Zhai, bpf, kernel-team, Jakub Sitnicki
On Mon, Feb 12, 2024 at 3:42 PM Kumar Kartikeya Dwivedi
<memxor@gmail.com> wrote:
>
> On Tue, 13 Feb 2024 at 00:34, Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Mon, Feb 12, 2024 at 3:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
> > >
> > > [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
> > > 6.6.16+ #10
> >
> > ...
> > > [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
> > > [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
> > >
> > > And Jakub CCed here did it for 6.8.0-rc2+
> >
> > I suspect something is broken in your kernels.
> > Above is doing generic copy_from_kernel_nofault(),
> > so one should be able to crash the kernel without any bpf.
> >
> > We have this in selftests/bpf:
> > __weak noinline struct file *bpf_testmod_return_ptr(int arg)
> > {
> > static struct file f = {};
> >
> > switch (arg) {
> > case 1: return (void *)EINVAL; /* user addr */
> > case 2: return (void *)0xcafe4a11; /* user addr */
> > case 3: return (void *)-EINVAL; /* canonical, but invalid */
> > case 4: return (void *)(1ull << 60); /* non-canonical and invalid */
> > case 5: return (void *)~(1ull << 30); /* trigger extable */
> > case 6: return &f; /* valid addr */
> > case 7: return (void *)((long)&f | 1); /* kernel tricks */
> > default: return NULL;
> > }
> > }
> > where we check that extables setup by JIT for bpf progs are working correctly.
> > You should see the kernel crashing when you just run bpf selftests.
>
> I agree, this appears unrelated to BPF since it is happening when
> using copy_from_kernel_nofault (which should be jumping to the Efault
> label instead of the oops), but I think it's not specific to some
> custom kernel. I can reproduce it on my dev machine on top of bpf-next
> as well, and another machine with Ubuntu's generic 6.5 kernel for
> 24.04. And I think Ignat tried it on the mainline 6.8-rc2 as well.
Then it must be vsyscall address that this series are fixing:
https://patchwork.kernel.org/project/netdevbpf/patch/20240202103935.3154011-3-houtao@huaweicloud.com/
We're still waiting on x86 maintainers to ack them.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-12 23:52 ` Alexei Starovoitov
@ 2024-02-13 0:21 ` Yan Zhai
2024-02-13 0:33 ` Kumar Kartikeya Dwivedi
0 siblings, 1 reply; 11+ messages in thread
From: Yan Zhai @ 2024-02-13 0:21 UTC (permalink / raw)
To: Alexei Starovoitov
Cc: Kumar Kartikeya Dwivedi, Ignat Korchagin, bpf, kernel-team,
Jakub Sitnicki
On Mon, Feb 12, 2024 at 5:52 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Mon, Feb 12, 2024 at 3:42 PM Kumar Kartikeya Dwivedi
> <memxor@gmail.com> wrote:
> >
> > On Tue, 13 Feb 2024 at 00:34, Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > >
> > > On Mon, Feb 12, 2024 at 3:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
> > > >
> > > > [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
> > > > 6.6.16+ #10
> > >
> > > ...
> > > > [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
> > > > [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
> > > >
> > > > And Jakub CCed here did it for 6.8.0-rc2+
> > >
> > > I suspect something is broken in your kernels.
> > > Above is doing generic copy_from_kernel_nofault(),
> > > so one should be able to crash the kernel without any bpf.
> > >
> > > We have this in selftests/bpf:
> > > __weak noinline struct file *bpf_testmod_return_ptr(int arg)
> > > {
> > > static struct file f = {};
> > >
> > > switch (arg) {
> > > case 1: return (void *)EINVAL; /* user addr */
> > > case 2: return (void *)0xcafe4a11; /* user addr */
> > > case 3: return (void *)-EINVAL; /* canonical, but invalid */
> > > case 4: return (void *)(1ull << 60); /* non-canonical and invalid */
> > > case 5: return (void *)~(1ull << 30); /* trigger extable */
> > > case 6: return &f; /* valid addr */
> > > case 7: return (void *)((long)&f | 1); /* kernel tricks */
> > > default: return NULL;
> > > }
> > > }
> > > where we check that extables setup by JIT for bpf progs are working correctly.
> > > You should see the kernel crashing when you just run bpf selftests.
> >
> > I agree, this appears unrelated to BPF since it is happening when
> > using copy_from_kernel_nofault (which should be jumping to the Efault
> > label instead of the oops), but I think it's not specific to some
> > custom kernel. I can reproduce it on my dev machine on top of bpf-next
> > as well, and another machine with Ubuntu's generic 6.5 kernel for
> > 24.04. And I think Ignat tried it on the mainline 6.8-rc2 as well.
>
copy_from_kernel_nofault is called in Jakub's reproducer, but the
panic case in our production seems to be direct memory accessing
according to bpftool dumped jited code. Will faults from such
instructions also be caught correctly?
Yan
> Then it must be vsyscall address that this series are fixing:
> https://patchwork.kernel.org/project/netdevbpf/patch/20240202103935.3154011-3-houtao@huaweicloud.com/
>
> We're still waiting on x86 maintainers to ack them.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-13 0:21 ` Yan Zhai
@ 2024-02-13 0:33 ` Kumar Kartikeya Dwivedi
2024-02-21 17:47 ` Ignat Korchagin
2024-02-22 9:27 ` Hou Tao
0 siblings, 2 replies; 11+ messages in thread
From: Kumar Kartikeya Dwivedi @ 2024-02-13 0:33 UTC (permalink / raw)
To: Yan Zhai
Cc: Alexei Starovoitov, Ignat Korchagin, bpf, kernel-team,
Jakub Sitnicki
On Tue, 13 Feb 2024 at 01:21, Yan Zhai <yan@cloudflare.com> wrote:
>
> On Mon, Feb 12, 2024 at 5:52 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Mon, Feb 12, 2024 at 3:42 PM Kumar Kartikeya Dwivedi
> > <memxor@gmail.com> wrote:
> > >
> > > On Tue, 13 Feb 2024 at 00:34, Alexei Starovoitov
> > > <alexei.starovoitov@gmail.com> wrote:
> > > >
> > > > On Mon, Feb 12, 2024 at 3:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
> > > > >
> > > > > [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
> > > > > 6.6.16+ #10
> > > >
> > > > ...
> > > > > [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
> > > > > [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
> > > > >
> > > > > And Jakub CCed here did it for 6.8.0-rc2+
> > > >
> > > > I suspect something is broken in your kernels.
> > > > Above is doing generic copy_from_kernel_nofault(),
> > > > so one should be able to crash the kernel without any bpf.
> > > >
> > > > We have this in selftests/bpf:
> > > > __weak noinline struct file *bpf_testmod_return_ptr(int arg)
> > > > {
> > > > static struct file f = {};
> > > >
> > > > switch (arg) {
> > > > case 1: return (void *)EINVAL; /* user addr */
> > > > case 2: return (void *)0xcafe4a11; /* user addr */
> > > > case 3: return (void *)-EINVAL; /* canonical, but invalid */
> > > > case 4: return (void *)(1ull << 60); /* non-canonical and invalid */
> > > > case 5: return (void *)~(1ull << 30); /* trigger extable */
> > > > case 6: return &f; /* valid addr */
> > > > case 7: return (void *)((long)&f | 1); /* kernel tricks */
> > > > default: return NULL;
> > > > }
> > > > }
> > > > where we check that extables setup by JIT for bpf progs are working correctly.
> > > > You should see the kernel crashing when you just run bpf selftests.
> > >
> > > I agree, this appears unrelated to BPF since it is happening when
> > > using copy_from_kernel_nofault (which should be jumping to the Efault
> > > label instead of the oops), but I think it's not specific to some
> > > custom kernel. I can reproduce it on my dev machine on top of bpf-next
> > > as well, and another machine with Ubuntu's generic 6.5 kernel for
> > > 24.04. And I think Ignat tried it on the mainline 6.8-rc2 as well.
> >
> copy_from_kernel_nofault is called in Jakub's reproducer, but the
> panic case in our production seems to be direct memory accessing
> according to bpftool dumped jited code. Will faults from such
> instructions also be caught correctly?
>
Yep, since faults in both cases end up in the page fault handler.
Once the fix pointed out by Alexei is applied, it should address both scenarios.
> Yan
>
> > Then it must be vsyscall address that this series are fixing:
> > https://patchwork.kernel.org/project/netdevbpf/patch/20240202103935.3154011-3-houtao@huaweicloud.com/
> >
> > We're still waiting on x86 maintainers to ack them.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-13 0:33 ` Kumar Kartikeya Dwivedi
@ 2024-02-21 17:47 ` Ignat Korchagin
2024-02-22 9:27 ` Hou Tao
1 sibling, 0 replies; 11+ messages in thread
From: Ignat Korchagin @ 2024-02-21 17:47 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, Alexei Starovoitov
Cc: Yan Zhai, bpf, kernel-team, Jakub Sitnicki
On Tue, Feb 13, 2024 at 12:34 AM Kumar Kartikeya Dwivedi
<memxor@gmail.com> wrote:
>
> On Tue, 13 Feb 2024 at 01:21, Yan Zhai <yan@cloudflare.com> wrote:
> >
> > On Mon, Feb 12, 2024 at 5:52 PM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > >
> > > On Mon, Feb 12, 2024 at 3:42 PM Kumar Kartikeya Dwivedi
> > > <memxor@gmail.com> wrote:
> > > >
> > > > On Tue, 13 Feb 2024 at 00:34, Alexei Starovoitov
> > > > <alexei.starovoitov@gmail.com> wrote:
> > > > >
> > > > > On Mon, Feb 12, 2024 at 3:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
> > > > > >
> > > > > > [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
> > > > > > 6.6.16+ #10
> > > > >
> > > > > ...
> > > > > > [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
> > > > > > [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
> > > > > >
> > > > > > And Jakub CCed here did it for 6.8.0-rc2+
> > > > >
> > > > > I suspect something is broken in your kernels.
> > > > > Above is doing generic copy_from_kernel_nofault(),
> > > > > so one should be able to crash the kernel without any bpf.
> > > > >
> > > > > We have this in selftests/bpf:
> > > > > __weak noinline struct file *bpf_testmod_return_ptr(int arg)
> > > > > {
> > > > > static struct file f = {};
> > > > >
> > > > > switch (arg) {
> > > > > case 1: return (void *)EINVAL; /* user addr */
> > > > > case 2: return (void *)0xcafe4a11; /* user addr */
> > > > > case 3: return (void *)-EINVAL; /* canonical, but invalid */
> > > > > case 4: return (void *)(1ull << 60); /* non-canonical and invalid */
> > > > > case 5: return (void *)~(1ull << 30); /* trigger extable */
> > > > > case 6: return &f; /* valid addr */
> > > > > case 7: return (void *)((long)&f | 1); /* kernel tricks */
> > > > > default: return NULL;
> > > > > }
> > > > > }
> > > > > where we check that extables setup by JIT for bpf progs are working correctly.
> > > > > You should see the kernel crashing when you just run bpf selftests.
> > > >
> > > > I agree, this appears unrelated to BPF since it is happening when
> > > > using copy_from_kernel_nofault (which should be jumping to the Efault
> > > > label instead of the oops), but I think it's not specific to some
> > > > custom kernel. I can reproduce it on my dev machine on top of bpf-next
> > > > as well, and another machine with Ubuntu's generic 6.5 kernel for
> > > > 24.04. And I think Ignat tried it on the mainline 6.8-rc2 as well.
> > >
> > copy_from_kernel_nofault is called in Jakub's reproducer, but the
> > panic case in our production seems to be direct memory accessing
> > according to bpftool dumped jited code. Will faults from such
> > instructions also be caught correctly?
> >
>
> Yep, since faults in both cases end up in the page fault handler.
> Once the fix pointed out by Alexei is applied, it should address both scenarios.
Just as a follow up the patches do seem to help for x86, but we've
recently encountered a similar problem on arm64 (6.1.74 kernel):
[Wed Feb 21 12:06:33 2024] Unable to handle kernel access to user
memory outside uaccess routines at virtual address 00007fff9959b150
[Wed Feb 21 12:06:33 2024] Mem abort info:
[Wed Feb 21 12:06:33 2024] ESR = 0x000000009600000f
[Wed Feb 21 12:06:33 2024] EC = 0x25: DABT (current EL), IL = 32 bits
[Wed Feb 21 12:06:33 2024] SET = 0, FnV = 0
[Wed Feb 21 12:06:33 2024] EA = 0, S1PTW = 0
[Wed Feb 21 12:06:33 2024] FSC = 0x0f: level 3 permission fault
[Wed Feb 21 12:06:33 2024] Data abort info:
[Wed Feb 21 12:06:33 2024] ISV = 0, ISS = 0x0000000f
[Wed Feb 21 12:06:33 2024] CM = 0, WnR = 0
[Wed Feb 21 12:06:33 2024] user pgtable: 4k pages, 48-bit VAs,
pgdp=00000812b1f69000
[Wed Feb 21 12:06:33 2024] [00007fff9959b150] pgd=08000812b1f72003,
p4d=08000812b1f72003, pud=08000812b1ff2003, pmd=08000855b2eb4003,
pte=0068087760598fc3
[Wed Feb 21 12:06:33 2024] Internal error: Oops: 000000009600000f [#1] SMP
[Wed Feb 21 12:06:33 2024] Modules linked in: nft_compat xt_hashlimit
ip_set_hash_netport xt_length esp4 nf_conntrack_netlink zstd
zstd_compress zram zsmalloc xgene_edac dm_thin_pool dm_persistent_data
dm_bio_prison dm_bufio nft_fwd_netdev nf_dup_netdev xfrm_interface
xfrm6_tunnel mpls_gso mpls_iptunnel mpls_router sit nft_numgen nft_log
nft_limit dummy ipip tunnel4 xfrm_user xfrm_algo nft_ct iptable_raw
iptable_nat iptable_mangle ipt_REJECT nf_reject_ipv4 ip6table_security
xt_CT ip6table_raw xt_nat ip6table_nat nf_nat xt_TCPMSS xt_owner
xt_NFLOG xt_connbytes xt_connlabel xt_statistic xt_connmark
ip6table_mangle xt_limit xt_LOG nf_log_syslog xt_mark xt_tcpudp
xt_conntrack ip6t_REJECT nf_reject_ipv6 xt_multiport xt_set xt_tcpmss
xt_comment ip6table_filter ip6_tables iptable_filter nfnetlink_log
tcp_diag cls_bpf sch_ingress ip_gre gre geneve tun xt_bpf nf_conntrack
nf_defrag_ipv6 nf_defrag_ipv4 fou6 fou ip_tunnel ip6_udp_tunnel
udp_tunnel ip6_tunnel tunnel6 veth nf_tables tcp_bbr sch_fq
[Wed Feb 21 12:06:33 2024] ip_set_hash_ip ip_set_hash_net ip_set
nfnetlink udp_diag inet_diag raid0 md_mod dm_crypt trusted
asn1_encoder tee algif_skcipher af_alg 8021q garp mrp stp llc
nvme_fabrics crct10dif_ce ghash_ce acpi_ipmi mlx5_core sha2_ce
ipmi_ssif sha256_arm64 sha1_ce mlxfw ipmi_devintf arm_spe_pmu
tiny_power_button tls igb xhci_pci nvme psample nvme_core xhci_hcd
ipmi_msghandler i2c_algo_bit button i2c_designware_platform
i2c_designware_core cppc_cpufreq arm_dsu_pmu tpm_tis tpm_tis_core fuse
dm_mod dax efivarfs ip_tables x_tables bcmcrypt(O) aes_neon_bs
aes_neon_blk aes_ce_blk aes_ce_cipher [last unloaded: kheaders]
[Wed Feb 21 12:06:33 2024] CPU: 15 PID: 547138 Comm: nginx-ssl
Tainted: G O 6.1.74-cloudflare-2024.1.14 #1
[Wed Feb 21 12:06:33 2024] Hardware name: GIGABYTE
[Wed Feb 21 12:06:33 2024] pstate: 20400009 (nzCv daif +PAN -UAO -TCO
-DIT -SSBS BTYPE=--)
[Wed Feb 21 12:06:33 2024] pc : 0xffff8000288c0674
[Wed Feb 21 12:06:33 2024] lr : 0xffff8000288c064c
[Wed Feb 21 12:06:33 2024] sp : ffff8000afdd3940
[Wed Feb 21 12:06:33 2024] x29: ffff8000afdd39d0 x28: ffff081142f99f80
x27: ffff8000afdd3940
[Wed Feb 21 12:06:33 2024] x26: 0000000000000000 x25: ffff8000afdd3990
x24: 0000000000000001
[Wed Feb 21 12:06:33 2024] x23: 000000002e4773f7 x22: ffff0800e7078300
x21: ffff08378b4c5180
[Wed Feb 21 12:06:33 2024] x20: 0000000000000000 x19: fffffbff5dc7d548
x18: 0000000000000000
[Wed Feb 21 12:06:33 2024] x17: 0000000000000000 x16: 0000000000000000
x15: ffff081b6e9e8196
[Wed Feb 21 12:06:33 2024] x14: 0000000000000000 x13: 0000000000000000
x12: 0000000000000000
[Wed Feb 21 12:06:33 2024] x11: 0000000000000000 x10: ffffda25e4cc90f0
x9 : ffffda25e4d71074
[Wed Feb 21 12:06:33 2024] x8 : ffff8000afdd3af8 x7 : 0000000000000000
x6 : 0000008124f0e5a3
[Wed Feb 21 12:06:33 2024] x5 : ffff80023c9cd000 x4 : 0000000000001000
x3 : 0000000000000008
[Wed Feb 21 12:06:33 2024] x2 : ffff081142f99f80 x1 : ffffda25e55e76a0
x0 : 00007fff9959a2d0
[Wed Feb 21 12:06:33 2024] Call trace:
[Wed Feb 21 12:06:33 2024] 0xffff8000288c0674
[Wed Feb 21 12:06:33 2024] bpf_trace_run3+0xcc/0x148
[Wed Feb 21 12:06:34 2024] __bpf_trace_kfree_skb+0x14/0x20
[Wed Feb 21 12:06:34 2024] __traceiter_kfree_skb+0x50/0x78
[Wed Feb 21 12:06:34 2024] kfree_skb_reason+0xa8/0x118
[Wed Feb 21 12:06:34 2024] tcp_data_queue+0x9f8/0xe20
[Wed Feb 21 12:06:34 2024] tcp_rcv_established+0x2b4/0x738
[Wed Feb 21 12:06:34 2024] tcp_v4_do_rcv+0x194/0x2d8
[Wed Feb 21 12:06:34 2024] __release_sock+0x90/0x138
[Wed Feb 21 12:06:34 2024] release_sock+0x64/0x120
[Wed Feb 21 12:06:34 2024] tcp_recvmsg+0x80/0x1c8
[Wed Feb 21 12:06:34 2024] inet_recvmsg+0x50/0xf8
[Wed Feb 21 12:06:34 2024] sock_read_iter+0xf4/0x128
[Wed Feb 21 12:06:34 2024] vfs_read+0x27c/0x2b0
[Wed Feb 21 12:06:34 2024] ksys_read+0xe4/0x108
[Wed Feb 21 12:06:34 2024] __arm64_sys_read+0x24/0x38
[Wed Feb 21 12:06:34 2024] invoke_syscall.constprop.0+0x58/0xf8
[Wed Feb 21 12:06:34 2024] do_el0_svc+0x174/0x1a0
[Wed Feb 21 12:06:34 2024] el0_svc+0x38/0xf0
[Wed Feb 21 12:06:34 2024] el0t_64_sync_handler+0xbc/0x138
[Wed Feb 21 12:06:34 2024] el0t_64_sync+0x18c/0x190
[Wed Feb 21 12:06:34 2024] Code: b94096c0 f9001360 f9400ac0 f9427c00 (f9474014)
[Wed Feb 21 12:06:34 2024] ---[ end trace 0000000000000000 ]---
Not sure if there's a similar fix for arm64 pending or is it some kind
more of a cross-platform problem
Ignat
> > Yan
> >
> > > Then it must be vsyscall address that this series are fixing:
> > > https://patchwork.kernel.org/project/netdevbpf/patch/20240202103935.3154011-3-houtao@huaweicloud.com/
> > >
> > > We're still waiting on x86 maintainers to ack them.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: Page faults in tracepoint caused by aliased pointer
2024-02-13 0:33 ` Kumar Kartikeya Dwivedi
2024-02-21 17:47 ` Ignat Korchagin
@ 2024-02-22 9:27 ` Hou Tao
1 sibling, 0 replies; 11+ messages in thread
From: Hou Tao @ 2024-02-22 9:27 UTC (permalink / raw)
To: Kumar Kartikeya Dwivedi, Yan Zhai
Cc: Alexei Starovoitov, Ignat Korchagin, bpf, kernel-team,
Jakub Sitnicki
Hi,
On 2/13/2024 8:33 AM, Kumar Kartikeya Dwivedi wrote:
> On Tue, 13 Feb 2024 at 01:21, Yan Zhai <yan@cloudflare.com> wrote:
>> On Mon, Feb 12, 2024 at 5:52 PM Alexei Starovoitov
>> <alexei.starovoitov@gmail.com> wrote:
>>> On Mon, Feb 12, 2024 at 3:42 PM Kumar Kartikeya Dwivedi
>>> <memxor@gmail.com> wrote:
>>>> On Tue, 13 Feb 2024 at 00:34, Alexei Starovoitov
>>>> <alexei.starovoitov@gmail.com> wrote:
>>>>> On Mon, Feb 12, 2024 at 3:16 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
>>>>>> [288931.217143][T109754] CPU: 4 PID: 109754 Comm: bpftrace Not tainted
>>>>>> 6.6.16+ #10
>>>>> ...
>>>>>> [288931.217143][T109754] ? copy_from_kernel_nofault+0x1d/0xe0
>>>>>> [288931.217143][T109754] bpf_probe_read_compat+0x6a/0x90
>>>>>>
>>>>>> And Jakub CCed here did it for 6.8.0-rc2+
>>>>> I suspect something is broken in your kernels.
>>>>> Above is doing generic copy_from_kernel_nofault(),
>>>>> so one should be able to crash the kernel without any bpf.
>>>>>
>>>>> We have this in selftests/bpf:
>>>>> __weak noinline struct file *bpf_testmod_return_ptr(int arg)
>>>>> {
>>>>> static struct file f = {};
>>>>>
>>>>> switch (arg) {
>>>>> case 1: return (void *)EINVAL; /* user addr */
>>>>> case 2: return (void *)0xcafe4a11; /* user addr */
>>>>> case 3: return (void *)-EINVAL; /* canonical, but invalid */
>>>>> case 4: return (void *)(1ull << 60); /* non-canonical and invalid */
>>>>> case 5: return (void *)~(1ull << 30); /* trigger extable */
>>>>> case 6: return &f; /* valid addr */
>>>>> case 7: return (void *)((long)&f | 1); /* kernel tricks */
>>>>> default: return NULL;
>>>>> }
>>>>> }
>>>>> where we check that extables setup by JIT for bpf progs are working correctly.
>>>>> You should see the kernel crashing when you just run bpf selftests.
>>>> I agree, this appears unrelated to BPF since it is happening when
>>>> using copy_from_kernel_nofault (which should be jumping to the Efault
>>>> label instead of the oops), but I think it's not specific to some
>>>> custom kernel. I can reproduce it on my dev machine on top of bpf-next
>>>> as well, and another machine with Ubuntu's generic 6.5 kernel for
>>>> 24.04. And I think Ignat tried it on the mainline 6.8-rc2 as well.
>> copy_from_kernel_nofault is called in Jakub's reproducer, but the
>> panic case in our production seems to be direct memory accessing
>> according to bpftool dumped jited code. Will faults from such
>> instructions also be caught correctly?
>>
> Yep, since faults in both cases end up in the page fault handler.
> Once the fix pointed out by Alexei is applied, it should address both scenarios.
I didn't get the idea on how the vsyscall patch [1] will fix the
unhandled page fault caused by BTF pointer dereference. In my
understanding, for BTF pointer dereference, x86 JIT checks whether the
address is a kernel space address or not. If it is the kernel space
address, it will setup an exception fix-up entry for its dereference and
will try to do dereference directly. If the address is vsyscall address,
x86 JIT will consider it as kernel space address and will try to
dereference it directly. The dereference of vsyscall page in kernel will
trigger the page fault, handle_page_fault() will be invoked and it will
invoke do_user_addr_fault() and page_fault_oops() accordingly.
[1]:
https://patchwork.kernel.org/project/netdevbpf/patch/20240202103935.3154011-3-houtao@huaweicloud.com/
>
>> Yan
>>
>>> Then it must be vsyscall address that this series are fixing:
>>> https://patchwork.kernel.org/project/netdevbpf/patch/20240202103935.3154011-3-houtao@huaweicloud.com/
>>>
>>> We're still waiting on x86 maintainers to ack them.
> .
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-02-22 9:27 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-02-12 22:14 Page faults in tracepoint caused by aliased pointer Yan Zhai
2024-02-12 22:55 ` Kumar Kartikeya Dwivedi
2024-02-12 23:15 ` Ignat Korchagin
2024-02-12 23:27 ` Yan Zhai
2024-02-12 23:33 ` Alexei Starovoitov
2024-02-12 23:41 ` Kumar Kartikeya Dwivedi
2024-02-12 23:52 ` Alexei Starovoitov
2024-02-13 0:21 ` Yan Zhai
2024-02-13 0:33 ` Kumar Kartikeya Dwivedi
2024-02-21 17:47 ` Ignat Korchagin
2024-02-22 9:27 ` Hou Tao
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox