nft_pipapo_avx2_lookup backtrace in linux 5.10

nft_pipapo_avx2_lookup backtrace in linux 5.10

[Arturo Borrero Gonzalez]

[-- Attachment #1: Type: text/plain, Size: 7252 bytes --]

Hi there,

I got this backtrace in one of my servers. I wonder if it is known or fixed 
already in a later version.

My versions:
* kernel 5.10.24
* nft 0.9.6

Also, find attached the ruleset that triggered this.

[Thu May  6 16:20:21 2021] ------------[ cut here ]------------
[Thu May  6 16:20:21 2021] WARNING: CPU: 3 PID: 456 at 
arch/x86/kernel/fpu/core.c:129 kernel_fpu_begin_mask+0xc9/0xe0
[Thu May  6 16:20:21 2021] Modules linked in: binfmt_misc nft_nat nft_chain_nat 
nf_nat nft_counter nft_ct nf_tables nf_conntrack_netlink nfnetlink 8021q garp 
stp mrp llc vrf intel_rapl_msr intel_rapl_common skx_edac nfit libnvdimm 
ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul mgag200 
ghash_clmulni_intel drm_kms_helper cec aesni_intel drm libaes crypto_simd cryptd 
glue_helper mei_me dell_smbios iTCO_wdt evdev intel_pmc_bxt iTCO_vendor_support 
dcdbas pcspkr rapl dell_wmi_descriptor wmi_bmof sg i2c_algo_bit watchdog mei 
acpi_ipmi ipmi_si button nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf 
ipmi_msghandler ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 dm_mod raid10 
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor sd_mod 
t10_pi crc_t10dif crct10dif_generic raid6_pq libcrc32c crc32c_generic raid1 
raid0 multipath linear md_mod ahci libahci tg3 libata xhci_pci libphy xhci_hcd 
ptp usbcore crct10dif_pclmul crct10dif_common bnxt_en crc32c_intel scsi_mod
[Thu May  6 16:20:21 2021]  pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
[Thu May  6 16:20:21 2021] CPU: 3 PID: 456 Comm: jbd2/dm-0-8 Tainted: G        W 
         5.10.0-0.bpo.5-amd64 #1 Debian 5.10.24-1~bpo10+1
[Thu May  6 16:20:21 2021] Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 
2.9.3 09/23/2020
[Thu May  6 16:20:21 2021] RIP: 0010:kernel_fpu_begin_mask+0xc9/0xe0
[Thu May  6 16:20:21 2021] Code: c4 10 5b c3 65 8a 05 5e 21 5e 76 84 c0 74 92 0f 
0b eb 8e f0 80 4f 01 40 48 81 c7 00 14 00 00 e8 dd fb ff ff eb a5 db e3 eb c4 
<0f> 0b e9 7b ff ff ff e8 2b 8a 84 00 66 66 2e 0f 1f 84 00 00 00 00
[Thu May  6 16:20:21 2021] RSP: 0018:ffffbb9700304740 EFLAGS: 00010202
[Thu May  6 16:20:21 2021] RAX: 0000000000000001 RBX: 0000000000000003 RCX: 
0000000000000001
[Thu May  6 16:20:21 2021] RDX: ffffbb9700304970 RSI: ffff922fe1952e00 RDI: 
0000000000000003
[Thu May  6 16:20:21 2021] RBP: ffffbb9700304970 R08: ffff922fc868a600 R09: 
ffff922fc711e462
[Thu May  6 16:20:21 2021] R10: 000000000000005f R11: ffff922ff0b27180 R12: 
ffffbb9700304960
[Thu May  6 16:20:21 2021] R13: ffffbb9700304b08 R14: ffff922fc664b6c8 R15: 
ffff922fc664b660
[Thu May  6 16:20:21 2021] FS:  0000000000000000(0000) GS:ffff92371fec0000(0000) 
knlGS:0000000000000000
[Thu May  6 16:20:21 2021] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Thu May  6 16:20:21 2021] CR2: 0000557a6655bdd0 CR3: 000000026020a001 CR4: 
00000000007706e0
[Thu May  6 16:20:21 2021] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[Thu May  6 16:20:21 2021] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
0000000000000400
[Thu May  6 16:20:21 2021] PKRU: 55555554
[Thu May  6 16:20:21 2021] Call Trace:
[Thu May  6 16:20:21 2021]  <IRQ>
[Thu May  6 16:20:21 2021]  nft_pipapo_avx2_lookup+0x4c/0x1cba [nf_tables]
[Thu May  6 16:20:21 2021]  ? sched_clock+0x5/0x10
[Thu May  6 16:20:21 2021]  ? sched_clock_cpu+0xc/0xb0
[Thu May  6 16:20:21 2021]  ? record_times+0x16/0x80
[Thu May  6 16:20:21 2021]  ? plist_add+0xc1/0x100
[Thu May  6 16:20:21 2021]  ? psi_group_change+0x47/0x230
[Thu May  6 16:20:21 2021]  ? skb_clone+0x4d/0xb0
[Thu May  6 16:20:21 2021]  ? enqueue_task_rt+0x22b/0x310
[Thu May  6 16:20:21 2021]  ? bnxt_start_xmit+0x1e8/0xaf0 [bnxt_en]
[Thu May  6 16:20:21 2021]  ? packet_rcv+0x40/0x4a0
[Thu May  6 16:20:21 2021]  nft_lookup_eval+0x59/0x160 [nf_tables]
[Thu May  6 16:20:21 2021]  nft_do_chain+0x350/0x500 [nf_tables]
[Thu May  6 16:20:21 2021]  ? nft_lookup_eval+0x59/0x160 [nf_tables]
[Thu May  6 16:20:21 2021]  ? nft_do_chain+0x364/0x500 [nf_tables]
[Thu May  6 16:20:21 2021]  ? fib4_rule_action+0x6d/0x80
[Thu May  6 16:20:21 2021]  ? fib_rules_lookup+0x107/0x250
[Thu May  6 16:20:21 2021]  nft_nat_do_chain+0x8a/0xf2 [nft_chain_nat]
[Thu May  6 16:20:21 2021]  nf_nat_inet_fn+0xea/0x210 [nf_nat]
[Thu May  6 16:20:21 2021]  nf_nat_ipv4_out+0x14/0xa0 [nf_nat]
[Thu May  6 16:20:21 2021]  nf_hook_slow+0x44/0xc0
[Thu May  6 16:20:21 2021]  ip_output+0xd2/0x100
[Thu May  6 16:20:21 2021]  ? __ip_finish_output+0x210/0x210
[Thu May  6 16:20:21 2021]  ip_forward+0x37d/0x4a0
[Thu May  6 16:20:21 2021]  ? ip4_key_hashfn+0xb0/0xb0
[Thu May  6 16:20:21 2021]  ip_sublist_rcv_finish+0x4f/0x60
[Thu May  6 16:20:21 2021]  ip_sublist_rcv+0x196/0x220
[Thu May  6 16:20:21 2021]  ? ip_rcv_finish_core.isra.22+0x400/0x400
[Thu May  6 16:20:21 2021]  ip_list_rcv+0x137/0x160
[Thu May  6 16:20:21 2021]  __netif_receive_skb_list_core+0x29b/0x2c0
[Thu May  6 16:20:21 2021]  netif_receive_skb_list_internal+0x1a6/0x2d0
[Thu May  6 16:20:21 2021]  gro_normal_list.part.156+0x19/0x40
[Thu May  6 16:20:21 2021]  napi_complete_done+0x67/0x170
[Thu May  6 16:20:21 2021]  bnxt_poll+0x105/0x190 [bnxt_en]
[Thu May  6 16:20:21 2021]  ? irqentry_exit+0x29/0x30
[Thu May  6 16:20:21 2021]  ? asm_common_interrupt+0x1e/0x40
[Thu May  6 16:20:21 2021]  net_rx_action+0x144/0x3c0
[Thu May  6 16:20:21 2021]  __do_softirq+0xd5/0x29c
[Thu May  6 16:20:21 2021]  asm_call_irq_on_stack+0xf/0x20
[Thu May  6 16:20:21 2021]  </IRQ>
[Thu May  6 16:20:21 2021]  do_softirq_own_stack+0x37/0x40
[Thu May  6 16:20:21 2021]  irq_exit_rcu+0x9d/0xa0
[Thu May  6 16:20:21 2021]  common_interrupt+0x78/0x130
[Thu May  6 16:20:21 2021]  asm_common_interrupt+0x1e/0x40
[Thu May  6 16:20:21 2021] RIP: 0010:crc_41+0x0/0x1e [crc32c_intel]
[Thu May  6 16:20:21 2021] Code: ff ff f2 4d 0f 38 f1 93 a8 fe ff ff f2 4c 0f 38 
f1 81 b0 fe ff ff f2 4c 0f 38 f1 8a b0 fe ff ff f2 4d 0f 38 f1 93 b0 fe ff ff 
<f2> 4c 0f 38 f1 81 b8 fe ff ff f2 4c 0f 38 f1 8a b8 fe ff ff f2 4d
[Thu May  6 16:20:21 2021] RSP: 0018:ffffbb97008dfcd0 EFLAGS: 00000246
[Thu May  6 16:20:21 2021] RAX: 000000000000002a RBX: 0000000000000400 RCX: 
ffff922fc591dd50
[Thu May  6 16:20:21 2021] RDX: ffff922fc591dea0 RSI: 0000000000000a14 RDI: 
ffffffffc00dddc0
[Thu May  6 16:20:21 2021] RBP: 0000000000001000 R08: 000000000342d8c3 R09: 
0000000000000000
[Thu May  6 16:20:21 2021] R10: 0000000000000000 R11: ffff922fc591dff0 R12: 
ffffbb97008dfe58
[Thu May  6 16:20:21 2021] R13: 000000000000000a R14: ffff922fd2b91e80 R15: 
ffff922fef83fe38
[Thu May  6 16:20:21 2021]  ? crc_43+0x1e/0x1e [crc32c_intel]
[Thu May  6 16:20:21 2021]  ? crc32c_pcl_intel_update+0x97/0xb0 [crc32c_intel]
[Thu May  6 16:20:21 2021]  ? jbd2_journal_commit_transaction+0xaec/0x1a30 [jbd2]
[Thu May  6 16:20:21 2021]  ? irq_exit_rcu+0x3e/0xa0
[Thu May  6 16:20:21 2021]  ? kjournald2+0xbd/0x270 [jbd2]
[Thu May  6 16:20:21 2021]  ? finish_wait+0x80/0x80
[Thu May  6 16:20:21 2021]  ? commit_timeout+0x10/0x10 [jbd2]
[Thu May  6 16:20:21 2021]  ? kthread+0x116/0x130
[Thu May  6 16:20:21 2021]  ? kthread_park+0x80/0x80
[Thu May  6 16:20:21 2021]  ? ret_from_fork+0x1f/0x30
[Thu May  6 16:20:21 2021] ---[ end trace 081a19978e5f09f6 ]---
[Thu May  6 16:20:21 2021] ------------[ cut here ]------------

[-- Attachment #2: ruleset.nft --]
[-- Type: text/plain, Size: 2877 bytes --]

define routing_source_ip   = 185.15.56.1
define virtual_subnet_cidr = 172.16.0.0/21
define nic_host = "ens2f0np0"
define nic_virt = "ens2f1np1.1107"
define nic_wan = "ens2f1np1.1120"

table inet cloudgw {
    set dmz_cidr_set {
        type ipv4_addr . ipv4_addr
        flags interval
        counter
        elements = {
                    172.16.0.0/21 . 208.80.154.224 ,
                    172.16.0.0/21 . 208.80.154.240 ,
                    172.16.0.0/21 . 208.80.153.224 ,
                    172.16.0.0/21 . 208.80.153.240 ,
                    172.16.0.0/21 . 198.35.26.96 ,
                    172.16.0.0/21 . 198.35.26.112 ,
                    172.16.0.0/21 . 103.102.166.224 ,
                    172.16.0.0/21 . 103.102.166.240 ,
                    172.16.0.0/21 . 91.198.174.192 ,
                    172.16.0.0/21 . 91.198.174.208 ,
                    172.16.0.0/21 . 208.80.154.24 ,
                    172.16.0.0/21 . 208.80.154.143 ,
                    172.16.0.0/21 . 208.80.153.118 ,
                    172.16.0.0/21 . 208.80.153.78 ,
                    172.16.0.0/21 . 208.80.153.107 ,
                    172.16.0.0/21 . 208.80.154.137 ,
                    172.16.0.0/21 . 208.80.154.30 ,
                    172.16.0.0/21 . 208.80.153.42 ,
                    172.16.0.0/21 . 208.80.154.15 ,
                    172.16.0.0/21 . 208.80.154.23 ,
                    172.16.0.0/21 . 208.80.154.132 ,
                    172.16.0.0/21 . 208.80.154.85 ,
                    172.16.0.0/21 . 208.80.153.59 ,
                    172.16.0.0/21 . 208.80.153.75 ,
                    172.16.0.0/21 . 208.80.153.116 ,
                    172.16.0.0/21 . 208.80.153.15 ,
                    172.16.0.0/21 . 208.80.154.252 ,
                    172.16.0.0/21 . 208.80.153.252 ,
                    172.16.0.0/21 . 208.80.155.119 ,
                    172.16.0.0/21 . 208.80.155.126 ,
                    172.16.0.0/21 . 208.80.155.125 ,
                    172.16.0.0/21 . 10.64.37.18/32 ,
                    172.16.0.0/21 . 10.64.37.28/32 ,
                    172.16.0.0/21 . 10.64.37.27/32 ,
                    172.16.0.0/21 . 10.64.37.13/32 ,
                    172.16.0.0/21 . 10.64.4.15/32 ,
                }
    }

    chain prerouting {
        type nat hook prerouting priority dstnat; policy accept;
    }

    chain postrouting {
        type nat hook postrouting priority srcnat; policy accept;

        ip saddr . ip daddr @dmz_cidr_set counter accept comment "dmz_cidr"
        ip saddr $virtual_subnet_cidr counter snat ip to $routing_source_ip comment "routing_source_ip"
    }

    chain forward {
        type filter hook forward priority filter; policy drop;
        # only forward packets in the VRF
        iifname "vrf-cloudgw" oifname { $nic_virt, $nic_wan } counter accept
        counter comment "counter dropped packets"
    }
}

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Florian Westphal]

Arturo Borrero Gonzalez <arturo@netfilter.org> wrote:
> Hi there,
> 
> I got this backtrace in one of my servers. I wonder if it is known or fixed
> already in a later version.
> 
> My versions:
> * kernel 5.10.24
> * nft 0.9.6
> 
> Also, find attached the ruleset that triggered this.
> 
> [Thu May  6 16:20:21 2021] ------------[ cut here ]------------
> [Thu May  6 16:20:21 2021] WARNING: CPU: 3 PID: 456 at
> arch/x86/kernel/fpu/core.c:129 kernel_fpu_begin_mask+0xc9/0xe0
> [Thu May  6 16:20:21 2021] Modules linked in: binfmt_misc nft_nat

Hmm, I suspect this is needed (not even compile tested).

diff --git a/net/netfilter/nft_set_pipapo_avx2.c b/net/netfilter/nft_set_pipapo_avx2.c
--- a/net/netfilter/nft_set_pipapo_avx2.c
+++ b/net/netfilter/nft_set_pipapo_avx2.c
@@ -1105,6 +1105,18 @@ bool nft_pipapo_avx2_estimate(const struct nft_set_desc *desc, u32 features,
 	return true;
 }
 
+static void nft_pipapo_avx_begin(void)
+{
+	local_bh_disable();
+	kernel_fpu_begin();
+}
+
+static void nft_pipapo_avx_end(void)
+{
+	kernel_fpu_end();
+	local_bh_enable();
+}
+
 /**
  * nft_pipapo_avx2_lookup() - Lookup function for AVX2 implementation
  * @net:	Network namespace
@@ -1134,11 +1146,11 @@ bool nft_pipapo_avx2_lookup(const struct net *net, const struct nft_set *set,
 	m = rcu_dereference(priv->match);
 
 	/* This also protects access to all data related to scratch maps */
-	kernel_fpu_begin();
+	nft_pipapo_avx_begin();
 
 	scratch = *raw_cpu_ptr(m->scratch_aligned);
 	if (unlikely(!scratch)) {
-		kernel_fpu_end();
+		nft_pipapo_avx_end();
 		return false;
 	}
 	map_index = raw_cpu_read(nft_pipapo_avx2_scratch_index);
@@ -1217,7 +1229,7 @@ bool nft_pipapo_avx2_lookup(const struct net *net, const struct nft_set *set,
 out:
 	if (i % 2)
 		raw_cpu_write(nft_pipapo_avx2_scratch_index, !map_index);
-	kernel_fpu_end();
+	nft_pipapo_avx_end();
 
 	return ret >= 0;
 }

kernel_fpu_begin() disables preemption, but we can still reenter via
softirq.

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Stefano Brivio]

Hi Arturo,

On Fri, 7 May 2021 11:26:51 +0200
Arturo Borrero Gonzalez <arturo@netfilter.org> wrote:

> Hi there,
> 
> I got this backtrace in one of my servers. I wonder if it is known or fixed 
> already in a later version.

Not as far as I know. At a glance:

> [...]
>
> [Thu May  6 16:20:21 2021]  nft_pipapo_avx2_lookup+0x4c/0x1cba [nf_tables]
>
> [...]
>
> [Thu May  6 16:20:21 2021]  asm_common_interrupt+0x1e/0x40
> [Thu May  6 16:20:21 2021] RIP: 0010:crc_41+0x0/0x1e [crc32c_intel]

we probably need to add an irq_fpu_usable() check in
nft_pipapo_avx2_lookup() and fall back to nft_pipapo_avx2_lookup_slow()
if that returns false, but I haven't looked into this thoroughly yet.

Can you reproduce this reliably? That would be helpful.

-- 
Stefano

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Stefano Brivio]

On Fri, 7 May 2021 12:35:23 +0200
Florian Westphal <fw@strlen.de> wrote:

> Arturo Borrero Gonzalez <arturo@netfilter.org> wrote:
> > Hi there,
> > 
> > I got this backtrace in one of my servers. I wonder if it is known or fixed
> > already in a later version.
> > 
> > My versions:
> > * kernel 5.10.24
> > * nft 0.9.6
> > 
> > Also, find attached the ruleset that triggered this.
> > 
> > [Thu May  6 16:20:21 2021] ------------[ cut here ]------------
> > [Thu May  6 16:20:21 2021] WARNING: CPU: 3 PID: 456 at
> > arch/x86/kernel/fpu/core.c:129 kernel_fpu_begin_mask+0xc9/0xe0
> > [Thu May  6 16:20:21 2021] Modules linked in: binfmt_misc nft_nat  
> 
> Hmm, I suspect this is needed (not even compile tested).
> 
> diff --git a/net/netfilter/nft_set_pipapo_avx2.c b/net/netfilter/nft_set_pipapo_avx2.c
> --- a/net/netfilter/nft_set_pipapo_avx2.c
> +++ b/net/netfilter/nft_set_pipapo_avx2.c
> @@ -1105,6 +1105,18 @@ bool nft_pipapo_avx2_estimate(const struct nft_set_desc *desc, u32 features,
>  	return true;
>  }
>  
> +static void nft_pipapo_avx_begin(void)
> +{
> +	local_bh_disable();
> +	kernel_fpu_begin();
> +}
>
> [...]
> 
> kernel_fpu_begin() disables preemption, but we can still reenter via
> softirq.

Right... if that's enough (I'm quite convinced), and the overhead is
negligible (not as much... I'll test), I would prefer this to the
fallback option on !irq_fpu_usable() -- it's simpler.

-- 
Stefano

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Arturo Borrero Gonzalez]

On 5/7/21 12:36 PM, Stefano Brivio wrote:
> 
> Can you reproduce this reliably? That would be helpful.
> 

No :-( this is a live production system. The backtrace was triggered by real 
life traffic. Even worse, I can't hack the kernel with a patch ... We should 
stick to debian kernel builds for production systems per our internal policy.

However, the nft ruleset is quite simple. It should be possible for you to grab 
a similar arch CPU, introduce the ruleset and generate some traffic to trigger 
the lookup(), no?

Thanks for your prompt response, also Florian for the patch proposal!

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Stefano Brivio]

On Fri, 7 May 2021 13:12:43 +0200
Arturo Borrero Gonzalez <arturo@netfilter.org> wrote:

> However, the nft ruleset is quite simple. It should be possible for you to grab 
> a similar arch CPU, introduce the ruleset and generate some traffic to trigger 
> the lookup(), no?

Unfortunately, this has little to do with the ruleset, it's rather
about the fact that a kthread using the FPU gets interrupted by
net_rx_action() which ends up in a call to nft_pipapo_avx2_lookup(),
which also uses the FPU.

The amount of luck I'd need to hit this with some ext4 worklaod
together with packet classification discourages me from even trying. ;)
But luckily my mistake here looks simple enough to fix.

-- 
Stefano

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Stefano Brivio]

On Fri, 7 May 2021 12:52:47 +0200
Stefano Brivio <sbrivio@redhat.com> wrote:

> On Fri, 7 May 2021 12:35:23 +0200
> Florian Westphal <fw@strlen.de> wrote:
> 
> > Arturo Borrero Gonzalez <arturo@netfilter.org> wrote:  
> > > Hi there,
> > > 
> > > I got this backtrace in one of my servers. I wonder if it is known or fixed
> > > already in a later version.
> > > 
> > > My versions:
> > > * kernel 5.10.24
> > > * nft 0.9.6
> > > 
> > > Also, find attached the ruleset that triggered this.
> > > 
> > > [Thu May  6 16:20:21 2021] ------------[ cut here ]------------
> > > [Thu May  6 16:20:21 2021] WARNING: CPU: 3 PID: 456 at
> > > arch/x86/kernel/fpu/core.c:129 kernel_fpu_begin_mask+0xc9/0xe0
> > > [Thu May  6 16:20:21 2021] Modules linked in: binfmt_misc nft_nat    
> > 
> > Hmm, I suspect this is needed (not even compile tested).
> > 
> > diff --git a/net/netfilter/nft_set_pipapo_avx2.c b/net/netfilter/nft_set_pipapo_avx2.c
> > --- a/net/netfilter/nft_set_pipapo_avx2.c
> > +++ b/net/netfilter/nft_set_pipapo_avx2.c
> > @@ -1105,6 +1105,18 @@ bool nft_pipapo_avx2_estimate(const struct nft_set_desc *desc, u32 features,
> >  	return true;
> >  }
> >  
> > +static void nft_pipapo_avx_begin(void)
> > +{
> > +	local_bh_disable();
> > +	kernel_fpu_begin();
> > +}
> >
> > [...]
> > 
> > kernel_fpu_begin() disables preemption, but we can still reenter via
> > softirq.  
> 
> Right... if that's enough (I'm quite convinced), and the overhead is
> negligible (not as much... I'll test), I would prefer this to the
> fallback option on !irq_fpu_usable() -- it's simpler.

Hmm, wait, the overhead is actually negligible, but I don't think
calling local_bh_disable() from the lookup function would actually
help: crc32c_pcl_intel_update() runs from a kthread, and while it's
using the FPU the softirq triggers, not the other way around... right?

I think we really need to check that the FPU isn't already in use by
the kernel with irq_fpu_usable() instead, just like
crc32c_pcl_intel_update() does.

Arturo, there's one thing confusing me here: checking 5.10.24, we're
hitting:

	WARN_ON_FPU(this_cpu_read(in_kernel_fpu));

at line 129 of arch/x86/kernel/fpu/core.c, but not:

	WARN_ON_FPU(!irq_fpu_usable());

at line 128? Those should be equivalent in this situation, because
irq_fpu_usable() checks:

	!in_interrupt() -> false (softirq here) ||

	interrupted_user_mode() -> false (judging from backtrace) ||

	interrupted_kernel_fpu_idle()
		== !!this_cpu_read(in_kernel_fpu);
		-> must be true if warning at line 129 triggers

...I see from tainted flags that some warning was already printed,
could it be that you have a warning from arch/x86/kernel/fpu/core.c:128
in your logs, before this one?

Florian, now that set back-ends are built-in, I'd simply go with
something like (oink):

diff --git a/net/netfilter/nft_set_pipapo.c b/net/netfilter/nft_set_pipapo.c
index 528a2d7ca991..dce866d93fee 100644
--- a/net/netfilter/nft_set_pipapo.c
+++ b/net/netfilter/nft_set_pipapo.c
@@ -408,8 +408,8 @@ int pipapo_refill(unsigned long *map, int len, int rules, unsigned long *dst,
  *
  * Return: true on match, false otherwise.
  */
-static bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,
-                             const u32 *key, const struct nft_set_ext **ext)
+bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,
+                      const u32 *key, const struct nft_set_ext **ext)
 {
        struct nft_pipapo *priv = nft_set_priv(set);
        unsigned long *res_map, *fill_map;
diff --git a/net/netfilter/nft_set_pipapo.h b/net/netfilter/nft_set_pipapo.h
index 25a75591583e..d84afb8fa79a 100644
--- a/net/netfilter/nft_set_pipapo.h
+++ b/net/netfilter/nft_set_pipapo.h
@@ -178,6 +178,8 @@ struct nft_pipapo_elem {
 
 int pipapo_refill(unsigned long *map, int len, int rules, unsigned long *dst,
                  union nft_pipapo_map_bucket *mt, bool match_only);
+bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,
+                      const u32 *key, const struct nft_set_ext **ext);
 
 /**
  * pipapo_and_field_buckets_4bit() - Intersect 4-bit buckets
diff --git a/net/netfilter/nft_set_pipapo_avx2.c b/net/netfilter/nft_set_pipapo_avx2.c
index d65ae0e23028..eabdb8d552ee 100644
--- a/net/netfilter/nft_set_pipapo_avx2.c
+++ b/net/netfilter/nft_set_pipapo_avx2.c
@@ -1131,6 +1131,9 @@ bool nft_pipapo_avx2_lookup(const struct net *net, const struct nft_set *set,
        bool map_index;
        int i, ret = 0;
 
+       if (unlikely(!irq_fpu_usable()))
+               return nft_pipapo_lookup(net, set, key, ext);
+
        m = rcu_dereference(priv->match);
 
        /* This also protects access to all data related to scratch maps */


-- 
Stefano

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Arturo Borrero Gonzalez]

On 5/8/21 3:00 AM, Stefano Brivio wrote:
> On Fri, 7 May 2021 13:12:43 +0200
> Arturo Borrero Gonzalez <arturo@netfilter.org> wrote:
> 
>> However, the nft ruleset is quite simple. It should be possible for you to grab
>> a similar arch CPU, introduce the ruleset and generate some traffic to trigger
>> the lookup(), no?
> 
> Unfortunately, this has little to do with the ruleset, it's rather
> about the fact that a kthread using the FPU gets interrupted by
> net_rx_action() which ends up in a call to nft_pipapo_avx2_lookup(),
> which also uses the FPU.
> 
> The amount of luck I'd need to hit this with some ext4 worklaod
> together with packet classification discourages me from even trying. ;)
> But luckily my mistake here looks simple enough to fix.
> 

right

Re: nft_pipapo_avx2_lookup backtrace in linux 5.10

[Arturo Borrero Gonzalez]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

On 5/8/21 3:00 AM, Stefano Brivio wrote:
> ...I see from tainted flags that some warning was already printed,
> could it be that you have a warning from arch/x86/kernel/fpu/core.c:128
> in your logs, before this one?

You are right there is a previous backtrace. Find attached the complete log file.

[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 15154 bytes --]

[709732.358778] ------------[ cut here ]------------
[709732.358791] WARNING: CPU: 3 PID: 456 at arch/x86/kernel/fpu/core.c:128 kernel_fpu_begin_mask+0xae/0xe0
[709732.358793] Modules linked in: binfmt_misc nft_nat nft_chain_nat nf_nat nft_counter nft_ct nf_tables nf_conntrack_netlink nfnetlink 8021q garp stp mrp llc vrf intel_rapl_msr intel_rapl_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul mgag200 ghash_clmulni_intel drm_kms_helper cec aesni_intel drm libaes crypto_simd cryptd glue_helper mei_me dell_smbios iTCO_wdt evdev intel_pmc_bxt iTCO_vendor_support dcdbas pcspkr rapl dell_wmi_descriptor wmi_bmof sg i2c_algo_bit watchdog mei acpi_ipmi ipmi_si button nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor sd_mod t10_pi crc_t10dif crct10dif_generic raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod ahci libahci tg3 libata xhci_pci libphy xhci_hcd ptp usbcore crct10dif_pclmul crct10dif_common bnxt_en crc32c_intel scsi_mod
[709732.358941]  pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
[709732.358957] CPU: 3 PID: 456 Comm: jbd2/dm-0-8 Not tainted 5.10.0-0.bpo.5-amd64 #1 Debian 5.10.24-1~bpo10+1
[709732.358959] Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.9.3 09/23/2020
[709732.358964] RIP: 0010:kernel_fpu_begin_mask+0xae/0xe0
[709732.358969] Code: ae 54 24 04 83 e3 01 75 38 48 8b 44 24 08 65 48 33 04 25 28 00 00 00 75 33 48 83 c4 10 5b c3 65 8a 05 5e 21 5e 76 84 c0 74 92 <0f> 0b eb 8e f0 80 4f 01 40 48 81 c7 00 14 00 00 e8 dd fb ff ff eb
[709732.358972] RSP: 0018:ffffbb9700304740 EFLAGS: 00010202
[709732.358976] RAX: 0000000000000001 RBX: 0000000000000003 RCX: 0000000000000001
[709732.358979] RDX: ffffbb9700304970 RSI: ffff922fe1952e00 RDI: 0000000000000003
[709732.358981] RBP: ffffbb9700304970 R08: ffff922fc868a600 R09: ffff922fc711e462
[709732.358984] R10: 000000000000005f R11: ffff922ff0b27180 R12: ffffbb9700304960
[709732.358987] R13: ffffbb9700304b08 R14: ffff922fc664b6c8 R15: ffff922fc664b660
[709732.358990] FS:  0000000000000000(0000) GS:ffff92371fec0000(0000) knlGS:0000000000000000
[709732.358993] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[709732.358996] CR2: 0000557a6655bdd0 CR3: 000000026020a001 CR4: 00000000007706e0
[709732.358999] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[709732.359001] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[709732.359003] PKRU: 55555554
[709732.359005] Call Trace:
[709732.359009]  <IRQ>
[709732.359035]  nft_pipapo_avx2_lookup+0x4c/0x1cba [nf_tables]
[709732.359046]  ? sched_clock+0x5/0x10
[709732.359054]  ? sched_clock_cpu+0xc/0xb0
[709732.359061]  ? record_times+0x16/0x80
[709732.359068]  ? plist_add+0xc1/0x100
[709732.359073]  ? psi_group_change+0x47/0x230
[709732.359079]  ? skb_clone+0x4d/0xb0
[709732.359085]  ? enqueue_task_rt+0x22b/0x310
[709732.359098]  ? bnxt_start_xmit+0x1e8/0xaf0 [bnxt_en]
[709732.359102]  ? packet_rcv+0x40/0x4a0
[709732.359121]  nft_lookup_eval+0x59/0x160 [nf_tables]
[709732.359133]  nft_do_chain+0x350/0x500 [nf_tables]
[709732.359152]  ? nft_lookup_eval+0x59/0x160 [nf_tables]
[709732.359163]  ? nft_do_chain+0x364/0x500 [nf_tables]
[709732.359172]  ? fib4_rule_action+0x6d/0x80
[709732.359178]  ? fib_rules_lookup+0x107/0x250
[709732.359184]  nft_nat_do_chain+0x8a/0xf2 [nft_chain_nat]
[709732.359193]  nf_nat_inet_fn+0xea/0x210 [nf_nat]
[709732.359202]  nf_nat_ipv4_out+0x14/0xa0 [nf_nat]
[709732.359207]  nf_hook_slow+0x44/0xc0
[709732.359214]  ip_output+0xd2/0x100
[709732.359221]  ? __ip_finish_output+0x210/0x210
[709732.359226]  ip_forward+0x37d/0x4a0
[709732.359232]  ? ip4_key_hashfn+0xb0/0xb0
[709732.359238]  ip_sublist_rcv_finish+0x4f/0x60
[709732.359243]  ip_sublist_rcv+0x196/0x220
[709732.359250]  ? ip_rcv_finish_core.isra.22+0x400/0x400
[709732.359255]  ip_list_rcv+0x137/0x160
[709732.359264]  __netif_receive_skb_list_core+0x29b/0x2c0
[709732.359272]  netif_receive_skb_list_internal+0x1a6/0x2d0
[709732.359280]  gro_normal_list.part.156+0x19/0x40
[709732.359286]  napi_complete_done+0x67/0x170
[709732.359298]  bnxt_poll+0x105/0x190 [bnxt_en]
[709732.359304]  ? irqentry_exit+0x29/0x30
[709732.359309]  ? asm_common_interrupt+0x1e/0x40
[709732.359315]  net_rx_action+0x144/0x3c0
[709732.359322]  __do_softirq+0xd5/0x29c
[709732.359329]  asm_call_irq_on_stack+0xf/0x20
[709732.359332]  </IRQ>
[709732.359339]  do_softirq_own_stack+0x37/0x40
[709732.359346]  irq_exit_rcu+0x9d/0xa0
[709732.359353]  common_interrupt+0x78/0x130
[709732.359358]  asm_common_interrupt+0x1e/0x40
[709732.359366] RIP: 0010:crc_41+0x0/0x1e [crc32c_intel]
[709732.359370] Code: ff ff f2 4d 0f 38 f1 93 a8 fe ff ff f2 4c 0f 38 f1 81 b0 fe ff ff f2 4c 0f 38 f1 8a b0 fe ff ff f2 4d 0f 38 f1 93 b0 fe ff ff <f2> 4c 0f 38 f1 81 b8 fe ff ff f2 4c 0f 38 f1 8a b8 fe ff ff f2 4d
[709732.359373] RSP: 0018:ffffbb97008dfcd0 EFLAGS: 00000246
[709732.359377] RAX: 000000000000002a RBX: 0000000000000400 RCX: ffff922fc591dd50
[709732.359379] RDX: ffff922fc591dea0 RSI: 0000000000000a14 RDI: ffffffffc00dddc0
[709732.359382] RBP: 0000000000001000 R08: 000000000342d8c3 R09: 0000000000000000
[709732.359384] R10: 0000000000000000 R11: ffff922fc591dff0 R12: ffffbb97008dfe58
[709732.359386] R13: 000000000000000a R14: ffff922fd2b91e80 R15: ffff922fef83fe38
[709732.359395]  ? crc_43+0x1e/0x1e [crc32c_intel]
[709732.359403]  ? crc32c_pcl_intel_update+0x97/0xb0 [crc32c_intel]
[709732.359419]  ? jbd2_journal_commit_transaction+0xaec/0x1a30 [jbd2]
[709732.359425]  ? irq_exit_rcu+0x3e/0xa0
[709732.359447]  ? kjournald2+0xbd/0x270 [jbd2]
[709732.359454]  ? finish_wait+0x80/0x80
[709732.359470]  ? commit_timeout+0x10/0x10 [jbd2]
[709732.359476]  ? kthread+0x116/0x130
[709732.359481]  ? kthread_park+0x80/0x80
[709732.359488]  ? ret_from_fork+0x1f/0x30
[709732.359494] ---[ end trace 081a19978e5f09f5 ]---
[709732.359498] ------------[ cut here ]------------
[709732.359503] WARNING: CPU: 3 PID: 456 at arch/x86/kernel/fpu/core.c:129 kernel_fpu_begin_mask+0xc9/0xe0
[709732.359505] Modules linked in: binfmt_misc nft_nat nft_chain_nat nf_nat nft_counter nft_ct nf_tables nf_conntrack_netlink nfnetlink 8021q garp stp mrp llc vrf intel_rapl_msr intel_rapl_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul mgag200 ghash_clmulni_intel drm_kms_helper cec aesni_intel drm libaes crypto_simd cryptd glue_helper mei_me dell_smbios iTCO_wdt evdev intel_pmc_bxt iTCO_vendor_support dcdbas pcspkr rapl dell_wmi_descriptor wmi_bmof sg i2c_algo_bit watchdog mei acpi_ipmi ipmi_si button nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor sd_mod t10_pi crc_t10dif crct10dif_generic raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod ahci libahci tg3 libata xhci_pci libphy xhci_hcd ptp usbcore crct10dif_pclmul crct10dif_common bnxt_en crc32c_intel scsi_mod
[709732.359635]  pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
[709732.359648] CPU: 3 PID: 456 Comm: jbd2/dm-0-8 Tainted: G        W         5.10.0-0.bpo.5-amd64 #1 Debian 5.10.24-1~bpo10+1
[709732.359650] Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.9.3 09/23/2020
[709732.359654] RIP: 0010:kernel_fpu_begin_mask+0xc9/0xe0
[709732.359657] Code: c4 10 5b c3 65 8a 05 5e 21 5e 76 84 c0 74 92 0f 0b eb 8e f0 80 4f 01 40 48 81 c7 00 14 00 00 e8 dd fb ff ff eb a5 db e3 eb c4 <0f> 0b e9 7b ff ff ff e8 2b 8a 84 00 66 66 2e 0f 1f 84 00 00 00 00
[709732.359660] RSP: 0018:ffffbb9700304740 EFLAGS: 00010202
[709732.359663] RAX: 0000000000000001 RBX: 0000000000000003 RCX: 0000000000000001
[709732.359666] RDX: ffffbb9700304970 RSI: ffff922fe1952e00 RDI: 0000000000000003
[709732.359668] RBP: ffffbb9700304970 R08: ffff922fc868a600 R09: ffff922fc711e462
[709732.359671] R10: 000000000000005f R11: ffff922ff0b27180 R12: ffffbb9700304960
[709732.359673] R13: ffffbb9700304b08 R14: ffff922fc664b6c8 R15: ffff922fc664b660
[709732.359676] FS:  0000000000000000(0000) GS:ffff92371fec0000(0000) knlGS:0000000000000000
[709732.359679] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[709732.359681] CR2: 0000557a6655bdd0 CR3: 000000026020a001 CR4: 00000000007706e0
[709732.359684] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[709732.359686] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[709732.359688] PKRU: 55555554
[709732.359690] Call Trace:
[709732.359692]  <IRQ>
[709732.359710]  nft_pipapo_avx2_lookup+0x4c/0x1cba [nf_tables]
[709732.359718]  ? sched_clock+0x5/0x10
[709732.359723]  ? sched_clock_cpu+0xc/0xb0
[709732.359728]  ? record_times+0x16/0x80
[709732.359734]  ? plist_add+0xc1/0x100
[709732.359739]  ? psi_group_change+0x47/0x230
[709732.359743]  ? skb_clone+0x4d/0xb0
[709732.359747]  ? enqueue_task_rt+0x22b/0x310
[709732.359760]  ? bnxt_start_xmit+0x1e8/0xaf0 [bnxt_en]
[709732.359763]  ? packet_rcv+0x40/0x4a0
[709732.359780]  nft_lookup_eval+0x59/0x160 [nf_tables]
[709732.359792]  nft_do_chain+0x350/0x500 [nf_tables]
[709732.359809]  ? nft_lookup_eval+0x59/0x160 [nf_tables]
[709732.359820]  ? nft_do_chain+0x364/0x500 [nf_tables]
[709732.359829]  ? fib4_rule_action+0x6d/0x80
[709732.359833]  ? fib_rules_lookup+0x107/0x250
[709732.359839]  nft_nat_do_chain+0x8a/0xf2 [nft_chain_nat]
[709732.359847]  nf_nat_inet_fn+0xea/0x210 [nf_nat]
[709732.359856]  nf_nat_ipv4_out+0x14/0xa0 [nf_nat]
[709732.359860]  nf_hook_slow+0x44/0xc0
[709732.359866]  ip_output+0xd2/0x100
[709732.359872]  ? __ip_finish_output+0x210/0x210
[709732.359878]  ip_forward+0x37d/0x4a0
[709732.359884]  ? ip4_key_hashfn+0xb0/0xb0
[709732.359889]  ip_sublist_rcv_finish+0x4f/0x60
[709732.359894]  ip_sublist_rcv+0x196/0x220
[709732.359900]  ? ip_rcv_finish_core.isra.22+0x400/0x400
[709732.359906]  ip_list_rcv+0x137/0x160
[709732.359913]  __netif_receive_skb_list_core+0x29b/0x2c0
[709732.359921]  netif_receive_skb_list_internal+0x1a6/0x2d0
[709732.359929]  gro_normal_list.part.156+0x19/0x40
[709732.359935]  napi_complete_done+0x67/0x170
[709732.359946]  bnxt_poll+0x105/0x190 [bnxt_en]
[709732.359951]  ? irqentry_exit+0x29/0x30
[709732.359955]  ? asm_common_interrupt+0x1e/0x40
[709732.359961]  net_rx_action+0x144/0x3c0
[709732.359968]  __do_softirq+0xd5/0x29c
[709732.359974]  asm_call_irq_on_stack+0xf/0x20
[709732.359976]  </IRQ>
[709732.359983]  do_softirq_own_stack+0x37/0x40
[709732.359989]  irq_exit_rcu+0x9d/0xa0
[709732.359994]  common_interrupt+0x78/0x130
[709732.359999]  asm_common_interrupt+0x1e/0x40
[709732.360006] RIP: 0010:crc_41+0x0/0x1e [crc32c_intel]
[709732.360010] Code: ff ff f2 4d 0f 38 f1 93 a8 fe ff ff f2 4c 0f 38 f1 81 b0 fe ff ff f2 4c 0f 38 f1 8a b0 fe ff ff f2 4d 0f 38 f1 93 b0 fe ff ff <f2> 4c 0f 38 f1 81 b8 fe ff ff f2 4c 0f 38 f1 8a b8 fe ff ff f2 4d
[709732.360012] RSP: 0018:ffffbb97008dfcd0 EFLAGS: 00000246
[709732.360016] RAX: 000000000000002a RBX: 0000000000000400 RCX: ffff922fc591dd50
[709732.360018] RDX: ffff922fc591dea0 RSI: 0000000000000a14 RDI: ffffffffc00dddc0
[709732.360021] RBP: 0000000000001000 R08: 000000000342d8c3 R09: 0000000000000000
[709732.360023] R10: 0000000000000000 R11: ffff922fc591dff0 R12: ffffbb97008dfe58
[709732.360025] R13: 000000000000000a R14: ffff922fd2b91e80 R15: ffff922fef83fe38
[709732.360033]  ? crc_43+0x1e/0x1e [crc32c_intel]
[709732.360040]  ? crc32c_pcl_intel_update+0x97/0xb0 [crc32c_intel]
[709732.360054]  ? jbd2_journal_commit_transaction+0xaec/0x1a30 [jbd2]
[709732.360059]  ? irq_exit_rcu+0x3e/0xa0
[709732.360081]  ? kjournald2+0xbd/0x270 [jbd2]
[709732.360086]  ? finish_wait+0x80/0x80
[709732.360102]  ? commit_timeout+0x10/0x10 [jbd2]
[709732.360107]  ? kthread+0x116/0x130
[709732.360111]  ? kthread_park+0x80/0x80
[709732.360117]  ? ret_from_fork+0x1f/0x30
[709732.360121] ---[ end trace 081a19978e5f09f6 ]---
[709732.360144] ------------[ cut here ]------------
[709732.360148] WARNING: CPU: 3 PID: 456 at arch/x86/kernel/fpu/core.c:155 kernel_fpu_end+0x19/0x20
[709732.360281] Modules linked in: binfmt_misc nft_nat nft_chain_nat nf_nat nft_counter nft_ct nf_tables nf_conntrack_netlink nfnetlink 8021q garp stp mrp llc vrf intel_rapl_msr intel_rapl_common skx_edac nfit libnvdimm ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp crc32_pclmul mgag200 ghash_clmulni_intel drm_kms_helper cec aesni_intel drm libaes crypto_simd cryptd glue_helper mei_me dell_smbios iTCO_wdt evdev intel_pmc_bxt iTCO_vendor_support dcdbas pcspkr rapl dell_wmi_descriptor wmi_bmof sg i2c_algo_bit watchdog mei acpi_ipmi ipmi_si button nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor sd_mod t10_pi crc_t10dif crct10dif_generic raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod ahci libahci tg3 libata xhci_pci libphy xhci_hcd ptp usbcore crct10dif_pclmul crct10dif_common bnxt_en crc32c_intel scsi_mod
[709732.360434]  pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
[709732.360446] CPU: 3 PID: 456 Comm: jbd2/dm-0-8 Tainted: G        W         5.10.0-0.bpo.5-amd64 #1 Debian 5.10.24-1~bpo10+1
[709732.360448] Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.9.3 09/23/2020
[709732.360452] RIP: 0010:kernel_fpu_end+0x19/0x20
[709732.360459] Code: ae 47 40 b8 01 00 00 00 c3 0f 0b eb d7 0f 0b eb c9 0f 1f 44 00 00 65 8a 05 fc 24 5e 76 84 c0 74 09 65 c6 05 f0 24 5e 76 00 c3 <0f> 0b eb f3 0f 1f 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53
[709732.360462] RSP: 0018:ffffbb97008dfce8 EFLAGS: 00010246
[709732.360465] RAX: 0000000055e8bc00 RBX: ffff922fc591d000 RCX: ffff922fc591dff0
[709732.360468] RDX: ffff922fc591dea0 RSI: 0000000000001000 RDI: ffff922fc591d000
[709732.360470] RBP: 0000000000001000 R08: 0000000055e8bc95 R09: 0000000087eeaa9c
[709732.360472] R10: 00000000becc42d6 R11: ffff922fc591dff0 R12: ffffbb97008dfe58
[709732.360475] R13: 000000000000000a R14: ffff922fd2b91e80 R15: ffff922fef83fe38
[709732.360478] FS:  0000000000000000(0000) GS:ffff92371fec0000(0000) knlGS:0000000000000000
[709732.360484] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[709732.360486] CR2: 0000557a6655bdd0 CR3: 000000026020a001 CR4: 00000000007706e0
[709732.360489] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[709732.360491] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[709732.360493] PKRU: 55555554
[709732.360494] Call Trace:
[709732.360501]  crc32c_pcl_intel_update+0xa1/0xb0 [crc32c_intel]
[709732.360515]  ? jbd2_journal_commit_transaction+0xaec/0x1a30 [jbd2]
[709732.360521]  ? irq_exit_rcu+0x3e/0xa0
[709732.360541]  ? kjournald2+0xbd/0x270 [jbd2]
[709732.360551]  ? finish_wait+0x80/0x80
[709732.360566]  ? commit_timeout+0x10/0x10 [jbd2]
[709732.360593]  ? kthread+0x116/0x130
[709732.360598]  ? kthread_park+0x80/0x80
[709732.360603]  ? ret_from_fork+0x1f/0x30
[709732.360608] ---[ end trace 081a19978e5f09f7 ]---