CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd]

CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd]

[Rin Cat via Bugspray Bot]

Rin Cat writes via Kernel.org Bugzilla:

When enabled LLVM/clang Control Flow Integrity in the kernel config, nfs caused kernel panic due to violate validation.

Linux: 6.1.55
Clang: 16.0.6


[ 1512.331974] CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd] (target: nfs4svc_encode_compoundres+0xb10/0x6f40 [nfsd]; expected type: 0x5d70b2b0)
[ 1512.331991] WARNING: CPU: 6 PID: 16245 at nfsd4_encode_operation+0xa2/0x210 [nfsd]
[ 1512.332000] Modules linked in: macvlan ebtable_filter ebtables ip6table_raw ip6table_mangle ip6table_nat ip6table_filter ip6_tables iptable_raw iptable_mangle iptable_nat iptable_filter ip_tables bpfilter fuse zram wireguard ip6_udp_tunnel udp_tunnel libchacha20poly1305 poly1305_x86_64 chacha_x86_64 curve25519_x86_64 libcurve25519_generic libchacha nfs fscache netfs nfsd auth_rpcgss lockd grace cfg80211 rfkill 8021q mrp garp stp llc sunrpc nft_chain_nat nf_nat nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink msr tcp_bbr sch_fq nls_iso8859_1 vfat fat amdgpu snd_hda_codec_realtek snd_hda_codec_generic drm_buddy ledtrig_audio video drm_ttm_helper snd_hda_codec_hdmi ttm amd64_edac drm_display_helper edac_mce_amd snd_hda_intel snd_intel_dspcfg drm_kms_helper snd_hda_codec kvm_amd gpu_sched mxm_wmi wmi_bmof snd_hda_core kvm snd_hwdep snd_pcm irqbypass crct10dif_pclmul snd_timer drm snd ghash_clmulni_intel backlight soundcore k10temp pcspkr rapl wmi acpi_cpufreq efiv
 arfs
[ 1512.332048]  dm_crypt dm_mod dax sd_mod igb mlx4_en i2c_algo_bit nvme i2c_core nvme_core dca crc32_pclmul t10_pi ptp mlx4_core xhci_pci crc32c_intel ahci xhci_hcd crc64_rocksoft pps_core libahci crc64
[ 1512.332060] CPU: 6 PID: 16245 Comm: nfsd Tainted: P                   6.1.55-x86_64 #1
[ 1512.332062] Hardware name: System manufacturer System Product Name/PRIME X570-PRO, BIOS 4602 02/23/2023
[ 1512.332063] RIP: 0010:nfsd4_encode_operation+0xa2/0x210 [nfsd]
[ 1512.332071] Code: 01 00 00 8d 48 b4 83 f9 b6 0f 86 81 01 00 00 4c 8b 1c c5 d0 d0 ae c0 48 8d 53 20 4c 89 ef 41 ba 50 4d 8f a2 45 03 53 fc 74 02 <0f> 0b 2e e8 e6 df 34 e4 89 43 04 49 83 7f 28 00 75 10 85 c0 74 1b
[ 1512.332073] RSP: 0018:ffffbe7f067bbdc0 EFLAGS: 00010287
[ 1512.332074] RAX: 0000000000000009 RBX: ffffa2a2e30de480 RCX: 00000000ffffffbd
[ 1512.332076] RDX: ffffa2a2e30de4a0 RSI: 0000000000000000 RDI: ffffa2a2e30dc000
[ 1512.332077] RBP: ffffffffc0aec1d8 R08: 0000000000000000 R09: 0000000000000000
[ 1512.332078] R10: 0000000094d577e1 R11: ffffffffc0ab3640 R12: ffffa2a163100000
[ 1512.332080] R13: ffffa2a2e30dc000 R14: ffffa2a3ea430058 R15: ffffa2a163100230
[ 1512.332081] FS:  0000000000000000(0000) GS:ffffa2afeeb80000(0000) knlGS:0000000000000000
[ 1512.332083] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1512.332084] CR2: 00007efee25ff000 CR3: 0000000323380000 CR4: 0000000000350ee0
[ 1512.332085] Call Trace:
[ 1512.332086]  <TASK>
[ 1512.332087]  ? __warn+0xac/0x160
[ 1512.332090]  ? nfsd4_encode_operation+0xa2/0x210 [nfsd]
[ 1512.332098]  ? report_cfi_failure+0x45/0x70
[ 1512.332100]  ? handle_cfi_failure+0x143/0x1d0
[ 1512.332103]  ? nfs4svc_encode_compoundres+0xb10/0x6f40 [nfsd]
[ 1512.332112]  ? handle_bug+0x4f/0xa0
[ 1512.332114]  ? exc_invalid_op+0x16/0x50
[ 1512.332116]  ? asm_exc_invalid_op+0x16/0x20
[ 1512.332119]  ? nfs4svc_encode_compoundres+0xb10/0x6f40 [nfsd]
[ 1512.332128]  ? nfsd4_encode_operation+0xa2/0x210 [nfsd]
[ 1512.332137]  warn_on_nonidempotent_op+0x417d/0x4370 [nfsd]
[ 1512.332146]  nfsd_dispatch+0x170/0x210 [nfsd]
[ 1512.332155]  svc_process+0x3d4/0x780 [sunrpc]
[ 1512.332165]  ? __cfi_nfsd_dispatch+0x10/0x10 [nfsd]
[ 1512.332173]  svc_process+0xdc/0x780 [sunrpc]
[ 1512.332183]  i_am_nfsd+0x109/0x1d0 [nfsd]
[ 1512.332192]  ? i_am_nfsd+0x40/0x1d0 [nfsd]
[ 1512.332200]  kthread+0x114/0x130
[ 1512.332202]  ? __cfi_kthread+0x10/0x10
[ 1512.332204]  ret_from_fork+0x22/0x30
[ 1512.332207]  </TASK>
[ 1512.332207] ---[ end trace 0000000000000000 ]---

View: https://bugzilla.kernel.org/show_bug.cgi?id=217973#c0
You can reply to this message to join the discussion.
-- 
Deet-doot-dot, I am a bot.
Kernel.org Bugzilla (bugspray 0.1-dev)

Re: CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd]

[Chuck Lever via Bugspray Bot]

Chuck Lever writes via Kernel.org Bugzilla:

Can nfsd4_encode_operation+0xa2 be resolved to a source line number in v6.1.55 ?

View: https://bugzilla.kernel.org/show_bug.cgi?id=217973#c1
You can reply to this message to join the discussion.
-- 
Deet-doot-dot, I am a bot.
Kernel.org Bugzilla (bugspray 0.1-dev)

Re: CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd]

[Jeff Layton via Bugspray Bot]

Jeff Layton writes via Kernel.org Bugzilla:

Also, I'm unfamiliar with CFI. What does a CFI failure mean?

View: https://bugzilla.kernel.org/show_bug.cgi?id=217973#c2
You can reply to this message to join the discussion.
-- 
Deet-doot-dot, I am a bot.
Kernel.org Bugzilla (bugspray 0.1-dev)

Re: CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd]

[Rin Cat via Bugspray Bot]

Rin Cat writes via Kernel.org Bugzilla:

I no longer had this issue when I moved to LTS 6.6 kernel from LTS 6.1, so I am not sure where it was fixed.

And for Jeff, CFI (Control Flow Integrity) is LLVM/clang supported runtime type check.
If a pointer or function argument is different or incompatible with the declared type, a kernel warning or panic will be triggered depending on the configuration.

https://clang.llvm.org/docs/ControlFlowIntegrity.html

A CFI failure most likely means some runtime bugs.

View: https://bugzilla.kernel.org/show_bug.cgi?id=217973#c3
You can reply to this message to join the discussion.
-- 
Deet-doot-dot, I am a bot.
Kernel.org Bugzilla (bugspray 0.1-dev)

Re: CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd]

[Chuck Lever via Bugspray Bot]

Chuck Lever writes via Kernel.org Bugzilla:

(In reply to Rin Cat from comment #3)
> I no longer had this issue when I moved to LTS 6.6 kernel from LTS 6.1, so I
> am not sure where it was fixed.

If you want to find the specific commit that resolved the issue, the best you can do is bisect between v6.1 and v6.13, applying the flow integrity check at each step. That shouldn't be more than two dozen steps.

I don't think we would have explicitly fixed a flow integrity bug, our current tooling does not point those out.

 
> And for Jeff, CFI (Control Flow Integrity) is LLVM/clang supported runtime
> type check.
> If a pointer or function argument is different or incompatible with the
> declared type, a kernel warning or panic will be triggered depending on the
> configuration.
> 
> https://clang.llvm.org/docs/ControlFlowIntegrity.html
> 
> A CFI failure most likely means some runtime bugs.

If v6.1.127 still triggers a CFI warning, it would help if you could bisect as described above and report the result here.

Are you able to regularly test upstream kernels or even this branch:

https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/log/?h=nfsd-testing

View: https://bugzilla.kernel.org/show_bug.cgi?id=217973#c4
You can reply to this message to join the discussion.
-- 
Deet-doot-dot, I am a bot.
Kernel.org Bugzilla (bugspray 0.1-dev)

Re: CFI failure at nfsd4_encode_operation+0xa2/0x210 [nfsd]

[Rin Cat via Bugspray Bot]

Rin Cat writes via Kernel.org Bugzilla:

OK, I will check v6.1.127 and see if I can reproduce it.

View: https://bugzilla.kernel.org/show_bug.cgi?id=217973#c5
You can reply to this message to join the discussion.
-- 
Deet-doot-dot, I am a bot.
Kernel.org Bugzilla (bugspray 0.1-dev)