commit to kernel fails since Debian 12 (bookworm)

[Markus Wigge <wigge@bht-berlin.de>]

[-- Attachment #1.1: Type: text/plain, Size: 6476 bytes --]

Hello,

we are encountering a strange problem with conntrackd after upgrading to 
Debian 12 (bookworm).
First the logs were flooded with errors like this:
> 2023-10-13T12:49:06.724542+02:00 fw-dc-c conntrackd[421008]: [Fri Oct 13 12:49:06 2023] (pid=421008) [warning] could not add new ct entry: Device or resource busy                            
> 2023-10-13T12:49:06.724690+02:00 fw-dc-c conntrackd[421008]: [Fri Oct 13 12:49:06 2023] (pid=421008) [warning] could not add new ct entry: Device or resource busy                            
> 2023-10-13T12:49:06.724847+02:00 fw-dc-c conntrackd[421008]: [Fri Oct 13 12:49:06 2023] (pid=421008) [warning] could not add new ct entry: Device or resource busy                            
> 2023-10-13T12:49:06.725048+02:00 fw-dc-c conntrackd[421008]: [Fri Oct 13 12:49:06 2023] (pid=421008) [warning] could not update ct entry, even if creating it instead: Device or resource busy
> 2023-10-13T12:49:06.725182+02:00 fw-dc-c conntrackd[421008]: [Fri Oct 13 12:49:06 2023] (pid=421008) [warning] could not update ct entry, even if creating it instead: Device or resource busy
> 2023-10-13T12:49:06.725271+02:00 fw-dc-c conntrackd[421008]: [Fri Oct 13 12:49:06 2023] (pid=421008) [warning] could not update ct entry, even if creating it instead: Device or resource busy
> 2023-10-13T12:49:06.725406+02:00 fw-dc-c conntrackd[421008]: [Fri Oct 13 12:49:06 2023] (pid=421008) [warning] could not update ct entry, even if creating it instead: Device or resource busy


Then with strace we discovered that conntrackd seems not to be able to 
commit received state information to the kernel any more:

> pselect6(18, [3 6 7 8 11 13 17], NULL, NULL, {tv_sec=0, tv_nsec=997962986}, NULL) = 1 (in [6], left {tv_sec=0, tv_nsec=995201883})                                                            
> rt_sigprocmask(SIG_BLOCK, [INT TERM CHLD], NULL, 8) = 0                                                                                                                                       
> recvfrom(6, "\20\0\0<h\233\333k\0\f\0\0\215@\314\v\215@\342+\0\10\0\5\0\0\1\210\0\5\0\2"..., 65536, 0, {sa_family=AF_INET, sin_port=htons(46346), sin_addr=inet_addr("172.23.42.10")}, [16]) =
>  60                                                                                                                                                                                           
> sendto(4, [{nlmsg_len=172, nlmsg_type=NFNL_SUBSYS_CTNETLINK<<8|IPCTNL_MSG_CT_NEW, nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE, nlmsg_seq=0, nlmsg_pid=0}, {nfgen_family=AF_INE
> T, version=NFNETLINK_V0, res_id=htons(0)}, [[{nla_len=52, nla_type=NLA_F_NESTED|0x1}, "\x14\x00\x01\x80\x08\x00\x01\x00\x8d\x40\xcc\x0b\x08\x00\x02\x00\x8d\x40\xe2\x2b\x1c\x00\x02\x80\x05\x0
> 0\x01\x00\x06\x00\x00\x00"...], [{nla_len=52, nla_type=NLA_F_NESTED|0x2}, "\x14\x00\x01\x80\x08\x00\x01\x00\x8d\x40\xe2\x2b\x08\x00\x02\x00\x8d\x40\xcc\x0b\x1c\x00\x02\x80\x05\x00\x01\x00\x0
> 6\x00\x00\x00"...], [{nla_len=8, nla_type=0x3}, "\x00\x00\x01\x88"], [{nla_len=8, nla_type=0x7}, "\x00\x00\x00\x78"], [{nla_len=32, nla_type=NLA_F_NESTED|0x4}, "\x1c\x00\x01\x80\x05\x00\x01\
> x00\x01\x00\x00\x00\x06\x00\x04\x00\x08\x08\x00\x00\x06\x00\x05\x00\x08\x08\x00\x00"]]], 172, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 172                              
> recvfrom(4, [{nlmsg_len=192, nlmsg_type=NLMSG_ERROR, nlmsg_flags=0, nlmsg_seq=0, nlmsg_pid=369424}, {error=-EBUSY, msg=[{nlmsg_len=172, nlmsg_type=NFNL_SUBSYS_CTNETLINK<<8|IPCTNL_MSG_CT_NEW,
>  nlmsg_flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_EXCL|NLM_F_CREATE, nlmsg_seq=0, nlmsg_pid=0}, {nfgen_family=AF_INET, version=NFNETLINK_V0, res_id=htons(0)}, [[{nla_len=52, nla_type=NLA_F_NESTED|0
> x1}, "\x14\x00\x01\x80\x08\x00\x01\x00\x8d\x40\xcc\x0b\x08\x00\x02\x00\x8d\x40\xe2\x2b\x1c\x00\x02\x80\x05\x00\x01\x00\x06\x00\x00\x00"...], [{nla_len=52, nla_type=NLA_F_NESTED|0x2}, "\x14\x
> 00\x01\x80\x08\x00\x01\x00\x8d\x40\xe2\x2b\x08\x00\x02\x00\x8d\x40\xcc\x0b\x1c\x00\x02\x80\x05\x00\x01\x00\x06\x00\x00\x00"...], [{nla_len=8, nla_type=0x3}, "\x00\x00\x01\x88"], [{nla_len=8,
>  nla_type=0x7}, "\x00\x00\x00\x78"], [{nla_len=32, nla_type=NLA_F_NESTED|0x4}, "\x1c\x00\x01\x80\x05\x00\x01\x00\x01\x00\x00\x00\x06\x00\x04\x00\x08\x08\x00\x00\x06\x00\x05\x00\x08\x08\x00\x
> 00"]]]}], 8192, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12]) = 192                                                                                                          
> newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=2298, ...}, 0) = 0                                                                                                      
> getpid()                                = 369424                                                                                                                                              
> write(2, "[Fri Oct 13 12:17:39 2023] (pid="..., 50) = 50                                                                                                                                      
> write(2, "could not add new ct entry: Devi"..., 51) = 51                                                                                                                                      
> write(2, "\n", 1)                       = 1                                                                                                                                                   


Any ideas when and why this broke? Is it a known bug? I was not able to 
find anything about that yet.
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:        12
Codename:       bookworm

# dpkg -l conntrackd
Gewünscht=Unbekannt/Installieren/R=Entfernen/P=Vollständig Löschen/Halten
| Status=Nicht/Installiert/Config/U=Entpackt/halb konFiguriert/
          Halb installiert/Trigger erWartet/Trigger anhängig
|/ Fehler?=(kein)/R=Neuinstallation notwendig (Status, Fehler: 
GROSS=schlecht)
||/ Name           Version      Architektur  Beschreibung
+++-==============-============-============-=================================
ii  conntrackd     1:1.4.7-1+b2 amd64        Connection tracking daemon

# uname -a
Linux fw-dc-c 6.1.0-9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1 
(2023-05-08) x86_64 GNU/Linux


Kind Regards
Markus

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]