TPROXY + Attempt to release TCP socket in state 1

TPROXY + Attempt to release TCP socket in state 1

[Michelle Bies]

Hi

I'm trying to run squid with TPROXY enabled but the system reboots after logging these messages:

  IPv4: Attempt to release TCP socket in state 1 00000000dfe7f997

and after some seconds:

  rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
  rcu:5-....: (6 GPs behind) idle=f9e/0/0x1 softirq=184030/184030 fqs=1901
  (detected by 1, t=6302 jiffies, g=894549, q=17504)
  Sending NMI from CPU 1 to CPUs 5:
  NMI backtrace for cpu 5
  CPU: 5 PID: 0 Comm: swapper/5 Tainted: GO 5.4.181+ #9
  Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.12.1 12/04/2020
  RIP: 0010:__inet_lookup_established+0x4c/0xf7
  Code: 48 89 f5 53 48 c1 e5 20 44 89 cb 48 09 c5 44 09 c3 e8 7b fe ff ff 41 89 c6 41 89 c5 49 8b 07 45 23 77 10 4e 8d 3c f0 49 8b 17 <f6> c2 01 0f 85 89 00 00 00 44 39 6a a0 75 7b 39 5a a4 75 76 48 39
  RSP: 0018:ffffc90000208c48 EFLAGS: 00000a12
  RAX: ffff88844ad00000 RBX: 0000000001bb18e1 RCX: 00000000baf65c60
  RDX: 000000000002b145 RSI: 00000000e5cc7389 RDI: 000000009da543b1
  RBP: 490e1a683c63a8c0 R08: 00000000000018e1 R09: 0000000001bb0000
  R10: ffff8883c96da400 R11: 00000000000018e1 R12: ffffffff822b3740
  R13: 00000000593d39c6 R14: 00000000000139c6 R15: ffff88844ad9ce30
  FS:0000000000000000(0000) GS:ffff88844d940000(0000) knlGS:0000000000000000
  CS:0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007fd0297e8eb0 CR3: 00000003d37d4005 CR4: 00000000003606e0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  Call Trace:
 <IRQ>
 nf_sk_lookup_slow_v4+0x224/0xfbe [nf_socket_ipv4]
 ? do_raw_spin_lock+0x2b/0x52
 socket_match.isra.0+0x2f/0xf9 [xt_socket]
 ipt_do_table+0x26f/0x5c1 [ip_tables]
 ? nf_ct_key_equal+0x38/0x5d [nf_conntrack]
 ? nf_conntrack_in+0x2bd/0x46b [nf_conntrack]
 nf_hook_slow+0x3c/0xb4
 nf_hook.constprop.0+0xa5/0xc8
 ? l3mdev_l3_rcv.constprop.0+0x50/0x50
 ip_rcv+0x41/0x61
 __netif_receive_skb_one_core+0x74/0x95
 process_backlog+0x97/0x122
 net_rx_action+0xf5/0x2a3
 __do_softirq+0xc2/0x1c6
 irq_exit+0x41/0x80
 call_function_single_interrupt+0xf/0x20
 </IRQ>
  RIP: 0010:mwait_idle+0x5f/0x75
  Code: f0 31 d2 48 89 d1 65 48 8b 04 25 40 ac 01 00 0f 01 c8 48 8b 08 48 c1 e9 03 83 e1 01 75 0e e8 2b c3 6c ff 48 89 c8 fb 0f 01 c9 <eb> 01 fb bf 15 00 00 00     65 48 8b 34 25 40 ac 01 00 e9 26 c3 6c ff
  RSP: 0018:ffffc900000f3ee0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
  RAX: 0000000000000000 RBX: ffff88844beee3c0 RCX: 0000000000000000
  RDX: 0000000000000000 RSI: ffff88844beee3c0 RDI: 0000000000000015  
  RBP: 0000000000000000 R08: 0000000000840188 R09: 0000000000000000
  R10: ffff88844d964b80 R11: ffff88844d964bc0 R12: 0000000000000000
  R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
 do_idle+0xcf/0x1da
 ? do_idle+0x2/0x1da 
 cpu_startup_entry+0x1a/0x1c 
 start_secondary+0x14b/0x169 
 secondary_startup_64+0xa4/0xb0
  ------------[ cut here ]------------ 
  NETDEV WATCHDOG: eth2 (igb): transmit queue 4 timed out 
  WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:480 dev_watchdog+0xcf/0x128
  Modules linked in: xt_CLASSIFY xt_nfacct sch_sfq xt_IMQ xt_NFLOG xt_limit xt_pkttype xt_nat xt_MASQUERADE xt_REDIRECT xt_connlimit nf_conncount xt_time xt_geoip(O) xt_iprange xt_NFQUEUE xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 xt_mac xt_mark 8021q garp mrp xt_multiport xt_socket nf_socket_ipv4 nf_socket_ipv6 ebtable_filter ebtable_nat ebtables xt_state xt_conntrack iptable_filter iptable_nat xt_set xt_connlabel xt_connmark iptable_mangle xt_recent iptable_raw sch_htb ip_set_hash_ipportip ip_set_hash_ip nfnetlink_acct nf_nat_pptp nf_conntrack_pptp nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat nfnetlink_log ip_set nfnetlink_queue nf_conntrack_netlink tun nfnetlink intel_lpss_pci intel_lpss imq igb sch_fq_codel nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bonding llc e1000e e1000 ip_tables
  CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O5.4.181+ #9
  Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.12.1 12/04/2020
  RIP: 0010:dev_watchdog+0xcf/0x128
  Code: 4b a1 00 00 75 38 48 89 ef c6 05 39 4b a1 00 01 e8 2e 7b fd ff 44 89 e1 48 89 ee 48 c7 c7 fb e6 0d 82 48 89 c2 e8 a0 98 0d 00 <0f> 0b eb 10 41 ff c4 48 05 40 01 00 00 41 39 f4 75 9d eb 13 48 8b
  RSP: 0018:ffffc90000158ec0 EFLAGS: 00010282
  RAX: 0000000000000000 RBX: ffff88844b234440 RCX: 0000000000000007
  RDX: 00000000000003f2 RSI: ffffc90000158db4 RDI: ffff88844d85b5b0
  RBP: ffff88844b234000 R08: 0000000000000001 R09: 0000000000014600
  R10: 0000000000000000 R11: 000000000000005c R12: 0000000000000004
  R13: ffffc90000158ef8 R14: ffffffff822050c0 R15: 0000000000000002
  FS:0000000000000000(0000) GS:ffff88844d840000(0000) knlGS:0000000000000000 
  CS:0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 00007faad50f2000 CR3: 0000000437d0e005 CR4: 00000000003606e0
  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
  DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
  Call Trace:
 <IRQ> 
 call_timer_fn.isra.0+0x18/0x6f
 ? netif_tx_lock+0x7a/0x7a 
 __run_timers.part.0+0x12d/0x163 
 ? hrtimer_forward+0x73/0x7b 
 ? tick_sched_timer+0x57/0x62
 ? timerqueue_add+0x62/0x68
 run_timer_softirq+0x21/0x43 
 __do_softirq+0xc2/0x1c6 
 irq_exit+0x41/0x80
 smp_apic_timer_interrupt+0x6f/0x7a
 apic_timer_interrupt+0xf/0x20
 </IRQ>
  RIP: 0010:mwait_idle+0x5f/0x75
  Code: f0 31 d2 48 89 d1 65 48 8b 04 25 40 ac 01 00 0f 01 c8 48 8b 08 48 c1 e9 03 83 e1 01 75 0e e8 2b c3 6c ff 48 89 c8 fb 0f 01 c9 <eb> 01 fb bf 15 00 00 00 65 48 8b 34 25 40 ac 01 00 e9 26 c3 6c ff
  RSP: 0018:ffffc900000d3ee0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
  RAX: 0000000000000000 RBX: ffff88844beeaac0 RCX: 0000000000000000
  RDX: 0000000000000000 RSI: ffff88844beeaac0 RDI: 0000000000000015
  RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
  R10: ffff88844d864b80 R11: ffff88844d864bc0 R12: 0000000000000000
  R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
 do_idle+0xcf/0x1da
 cpu_startup_entry+0x1a/0x1c
 start_secondary+0x14b/0x169
 secondary_startup_64+0xa4/0xb0
  ---[ end trace 9e50b2e05e0ee06d ]---

My current kernel is 5.4 and these are my iptables config:

 iptables -t mangle -A PREROUTING -p tcp -m multiport --sport 80 -m socket -m conntrack --ctdir REPLY -j DIVERT
 iptables -t mangle -A DIVERT -j MARK --set-mark 1
 iptables -t mangle -A DIVERT -j ACCEPT
 iptables -t mangle -A PREROUTING -p tcp -m multiport --dports 80 -j TPROXY --tproxy-mark 1 --on-port 3129

Re: TPROXY + Attempt to release TCP socket in state 1

[Michelle Bies]

Hi Eric
unfortunately, nobody response to my problem :(
Did I report my problem to the right mailing list? 


On 05 July 2022 09:52, Michelle Bies wrote:
> Hi
>
> I'm trying to run squid with TPROXY enabled but the system reboots after logging these messages:
> 
>   IPv4: Attempt to release TCP socket in state 1 00000000dfe7f997
> 
> and after some seconds:
> 
>   rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
>   rcu:5-....: (6 GPs behind) idle=f9e/0/0x1 softirq=184030/184030 fqs=1901
>   (detected by 1, t=6302 jiffies, g=894549, q=17504)
>   Sending NMI from CPU 1 to CPUs 5:
>   NMI backtrace for cpu 5
>   CPU: 5 PID: 0 Comm: swapper/5 Tainted: GO 5.4.181+ #9
>   Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.12.1 12/04/2020
>   RIP: 0010:__inet_lookup_established+0x4c/0xf7
>   Code: 48 89 f5 53 48 c1 e5 20 44 89 cb 48 09 c5 44 09 c3 e8 7b fe ff ff 41 89 c6 41 89 c5 49 8b 07 45 23 77 10 4e 8d 3c f0 49 8b 17 <f6> c2 01 0f 85 89 00 00 00 44 39 6a a0 75 7b 39 5a a4 75 76 48 39
>   RSP: 0018:ffffc90000208c48 EFLAGS: 00000a12
>   RAX: ffff88844ad00000 RBX: 0000000001bb18e1 RCX: 00000000baf65c60
>   RDX: 000000000002b145 RSI: 00000000e5cc7389 RDI: 000000009da543b1
>   RBP: 490e1a683c63a8c0 R08: 00000000000018e1 R09: 0000000001bb0000
>   R10: ffff8883c96da400 R11: 00000000000018e1 R12: ffffffff822b3740
>   R13: 00000000593d39c6 R14: 00000000000139c6 R15: ffff88844ad9ce30
>   FS:0000000000000000(0000) GS:ffff88844d940000(0000) knlGS:0000000000000000
>   CS:0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>   CR2: 00007fd0297e8eb0 CR3: 00000003d37d4005 CR4: 00000000003606e0
>   DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>   DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>   Call Trace:
>  <IRQ>
>  nf_sk_lookup_slow_v4+0x224/0xfbe [nf_socket_ipv4]
>  ? do_raw_spin_lock+0x2b/0x52
>  socket_match.isra.0+0x2f/0xf9 [xt_socket]
>  ipt_do_table+0x26f/0x5c1 [ip_tables]
>  ? nf_ct_key_equal+0x38/0x5d [nf_conntrack]
>  ? nf_conntrack_in+0x2bd/0x46b [nf_conntrack]
>  nf_hook_slow+0x3c/0xb4
>  nf_hook.constprop.0+0xa5/0xc8
>  ? l3mdev_l3_rcv.constprop.0+0x50/0x50
>  ip_rcv+0x41/0x61
>  __netif_receive_skb_one_core+0x74/0x95
>  process_backlog+0x97/0x122
>  net_rx_action+0xf5/0x2a3
>  __do_softirq+0xc2/0x1c6
>  irq_exit+0x41/0x80
>  call_function_single_interrupt+0xf/0x20
>  </IRQ>
>   RIP: 0010:mwait_idle+0x5f/0x75
>   Code: f0 31 d2 48 89 d1 65 48 8b 04 25 40 ac 01 00 0f 01 c8 48 8b 08 48 c1 e9 03 83 e1 01 75 0e e8 2b c3 6c ff 48 89 c8 fb 0f 01 c9 <eb> 01 fb bf 15 00 00 00     65 48 8b 34 25 40 ac 01 00 e9 26 c3 6c ff
>   RSP: 0018:ffffc900000f3ee0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
>   RAX: 0000000000000000 RBX: ffff88844beee3c0 RCX: 0000000000000000
>   RDX: 0000000000000000 RSI: ffff88844beee3c0 RDI: 0000000000000015
>   RBP: 0000000000000000 R08: 0000000000840188 R09: 0000000000000000
>   R10: ffff88844d964b80 R11: ffff88844d964bc0 R12: 0000000000000000
>   R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
>  do_idle+0xcf/0x1da
>  ? do_idle+0x2/0x1da
>  cpu_startup_entry+0x1a/0x1c
>  start_secondary+0x14b/0x169
>  secondary_startup_64+0xa4/0xb0
>   ------------[ cut here ]------------
>   NETDEV WATCHDOG: eth2 (igb): transmit queue 4 timed out
>   WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:480 dev_watchdog+0xcf/0x128
>   Modules linked in: xt_CLASSIFY xt_nfacct sch_sfq xt_IMQ xt_NFLOG xt_limit xt_pkttype xt_nat xt_MASQUERADE xt_REDIRECT xt_connlimit nf_conncount xt_time xt_geoip(O) xt_iprange xt_NFQUEUE xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 xt_mac xt_mark 8021q garp mrp xt_multiport xt_socket nf_socket_ipv4 nf_socket_ipv6 ebtable_filter ebtable_nat ebtables xt_state xt_conntrack iptable_filter iptable_nat xt_set xt_connlabel xt_connmark iptable_mangle xt_recent iptable_raw sch_htb ip_set_hash_ipportip ip_set_hash_ip nfnetlink_acct nf_nat_pptp nf_conntrack_pptp nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 nf_nat_sip nf_conntrack_sip nf_nat nfnetlink_log ip_set nfnetlink_queue nf_conntrack_netlink tun nfnetlink intel_lpss_pci intel_lpss imq igb sch_fq_codel nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bonding llc e1000e e1000 ip_tables
>   CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O5.4.181+ #9
>   Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.12.1 12/04/2020
>   RIP: 0010:dev_watchdog+0xcf/0x128
>   Code: 4b a1 00 00 75 38 48 89 ef c6 05 39 4b a1 00 01 e8 2e 7b fd ff 44 89 e1 48 89 ee 48 c7 c7 fb e6 0d 82 48 89 c2 e8 a0 98 0d 00 <0f> 0b eb 10 41 ff c4 48 05 40 01 00 00 41 39 f4 75 9d eb 13 48 8b
>   RSP: 0018:ffffc90000158ec0 EFLAGS: 00010282
>   RAX: 0000000000000000 RBX: ffff88844b234440 RCX: 0000000000000007
>   RDX: 00000000000003f2 RSI: ffffc90000158db4 RDI: ffff88844d85b5b0
>   RBP: ffff88844b234000 R08: 0000000000000001 R09: 0000000000014600
>   R10: 0000000000000000 R11: 000000000000005c R12: 0000000000000004
>   R13: ffffc90000158ef8 R14: ffffffff822050c0 R15: 0000000000000002
>   FS:0000000000000000(0000) GS:ffff88844d840000(0000) knlGS:0000000000000000
>   CS:0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>   CR2: 00007faad50f2000 CR3: 0000000437d0e005 CR4: 00000000003606e0
>   DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>   DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>   Call Trace:
>  <IRQ>
>  call_timer_fn.isra.0+0x18/0x6f
>  ? netif_tx_lock+0x7a/0x7a
>  __run_timers.part.0+0x12d/0x163
>  ? hrtimer_forward+0x73/0x7b
>  ? tick_sched_timer+0x57/0x62
>  ? timerqueue_add+0x62/0x68
>  run_timer_softirq+0x21/0x43
>  __do_softirq+0xc2/0x1c6
>  irq_exit+0x41/0x80
>  smp_apic_timer_interrupt+0x6f/0x7a
>  apic_timer_interrupt+0xf/0x20
>  </IRQ>
>   RIP: 0010:mwait_idle+0x5f/0x75
>   Code: f0 31 d2 48 89 d1 65 48 8b 04 25 40 ac 01 00 0f 01 c8 48 8b 08 48 c1 e9 03 83 e1 01 75 0e e8 2b c3 6c ff 48 89 c8 fb 0f 01 c9 <eb> 01 fb bf 15 00 00 00 65 48 8b 34 25 40 ac 01 00 e9 26 c3 6c ff
>   RSP: 0018:ffffc900000d3ee0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
>   RAX: 0000000000000000 RBX: ffff88844beeaac0 RCX: 0000000000000000
>   RDX: 0000000000000000 RSI: ffff88844beeaac0 RDI: 0000000000000015
>   RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
>   R10: ffff88844d864b80 R11: ffff88844d864bc0 R12: 0000000000000000
>   R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
>  do_idle+0xcf/0x1da
>  cpu_startup_entry+0x1a/0x1c
>  start_secondary+0x14b/0x169
>  secondary_startup_64+0xa4/0xb0
>   ---[ end trace 9e50b2e05e0ee06d ]---
> 
> My current kernel is 5.4 and these are my iptables config:
> 
>  iptables -t mangle -A PREROUTING -p tcp -m multiport --sport 80 -m socket -m conntrack --ctdir REPLY -j DIVERT
>  iptables -t mangle -A DIVERT -j MARK --set-mark 1
>  iptables -t mangle -A DIVERT -j ACCEPT
>  iptables -t mangle -A PREROUTING -p tcp -m multiport --dports 80 -j TPROXY --tproxy-mark 1 --on-port 3129
> 

Re: TPROXY + Attempt to release TCP socket in state 1

[Willy Tarreau]

Hi,

On Sat, Jul 09, 2022 at 06:14:14AM +0000, Michelle Bies wrote:
> Hi Eric
> unfortunately, nobody response to my problem :(
> Did I report my problem to the right mailing list? 

You sent it only 4 days ago! As you might have noticed you're not the
only one to request help, code reviews or anything that requires time
from only a few people. What progress have you made on your side on
the analysis of this problem in during this time, that you could share
to save precious time to those who are going to help you, and to make
your issue more interesting to analyse than other ones ?

Also, I'm seeing that your kernel is tainted by an out-of-tree module:

> >   CPU: 5 PID: 0 Comm: swapper/5 Tainted: GO 5.4.181+ #9
                                             ^^

Most likely it's this "xt_geoip" module but it may also be anything
else I have not spotted in your dump. Have you rechecked without it ?
While unlikely to be related, any out-of-tree code must be handled
with extreme care, as their impact on the rest of the kernel remains
mostly unknown, so they're the first ones to disable during
troubleshooting.

> > My current kernel is 5.4 and these are my iptables config:

Please always mention the exact version in reports. I've seen "5.4.181+"
in your dump, which means 5.4.181 plus extra patches. Have you tried
again without them ?

Hoping this helps,
Willy

Re: TPROXY + Attempt to release TCP socket in state 1

[Eric Dumazet]

On Sat, Jul 9, 2022 at 8:41 AM Willy Tarreau <w@1wt.eu> wrote:
>
> Hi,
>
> On Sat, Jul 09, 2022 at 06:14:14AM +0000, Michelle Bies wrote:
> > Hi Eric
> > unfortunately, nobody response to my problem :(
> > Did I report my problem to the right mailing list?
>
> You sent it only 4 days ago!

As a matter of fact, email did not reach the list because it had html format.

> As you might have noticed you're not the
> only one to request help, code reviews or anything that requires time
> from only a few people. What progress have you made on your side on
> the analysis of this problem in during this time, that you could share
> to save precious time to those who are going to help you, and to make
> your issue more interesting to analyse than other ones ?
>
> Also, I'm seeing that your kernel is tainted by an out-of-tree module:
>
> > >   CPU: 5 PID: 0 Comm: swapper/5 Tainted: GO 5.4.181+ #9
>                                              ^^
>
> Most likely it's this "xt_geoip" module but it may also be anything
> else I have not spotted in your dump. Have you rechecked without it ?
> While unlikely to be related, any out-of-tree code must be handled
> with extreme care, as their impact on the rest of the kernel remains
> mostly unknown, so they're the first ones to disable during
> troubleshooting.
>
> > > My current kernel is 5.4 and these are my iptables config:
>
> Please always mention the exact version in reports. I've seen "5.4.181+"
> in your dump, which means 5.4.181 plus extra patches. Have you tried
> again without them ?
>
> Hoping this helps,
> Willy

Re: TPROXY + Attempt to release TCP socket in state 1

[Michelle Bies]

Hi 

On Sat, Jul 9, 2022 at 8:41 AM Willy Tarreau <w@1wt.eu> wrote:
> 
> You sent it only 4 days ago! As you might have noticed you're not the
> only one to request help, code reviews or anything that requires time

You are completely right. Thank you for your attention and I am sorry for the rush (I am a newbie in the kernel scope).

> ... . What progress have you made on your side on
> the analysis of this problem in during this time, that you could share
> to save precious time to those who are going to help you, and to make
> your issue more interesting to analyse than other ones ?

The problem occurs in a production environment in random periods, sometimes after some hours and sometimes after some days.
I couldn't reproduce it in my test environment, I even tried to simulate the problem under huge load in my development environment but still no crash occurred.
I also tried to attach a serial console but nothing special happens when system reboots.

> Also, I'm seeing that your kernel is tainted by an out-of-tree module:
> 
> > >   CPU: 5 PID: 0 Comm: swapper/5 Tainted: GO 5.4.181+ #9
>                                              ^^
> 
> Most likely it's this "xt_geoip" module but it may also be anything
> else I have not spotted in your dump. Have you rechecked without it ?
> While unlikely to be related, any out-of-tree code must be handled
> with extreme care, as their impact on the rest of the kernel remains
> mostly unknown, so they're the first ones to disable during
> troubleshooting.

I applied IMQ patch so I have a xt_IMQ module.
Could this be root of the problem?

> Please always mention the exact version in reports. I've seen "5.4.181+"
> in your dump, which means 5.4.181 plus extra patches. Have you tried
> again without them ?
You're correct. My kernel is 5.4.181+. I got the tip and I will check all my changes, although I think they are irrelevant.

Thank you again for your time.