From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-vk1-f177.google.com (mail-vk1-f177.google.com [209.85.221.177])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FED9385
	for <regressions@lists.linux.dev>; Fri, 22 Sep 2023 03:06:37 +0000 (UTC)
Received: by mail-vk1-f177.google.com with SMTP id 71dfb90a1353d-49618e09f16so686843e0c.2
        for <regressions@lists.linux.dev>; Thu, 21 Sep 2023 20:06:37 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695351996; x=1695956796; darn=lists.linux.dev;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to;
        bh=P6X0PKOimTbsxZTjYFPpLza08mWiA8aUu9v5BicDRCw=;
        b=O8m8wlOkRsQvKihdBWiM7VZj2KQTi68ZGvW3lyD85/ogcCZiJa5U4q/8hbo5l06yEt
         wn+JW+jeY3BKqAHp+F5asUtpIddfRhcGWTHv5AMdOOZ8jfXc5QQfGA/E+QVqqLe3Qzj3
         E/7259EEZf/tahs3wEVheMbG097xD8horKVKy06ZObqD5Zw63QqU7bn4Vycan9KXsDyD
         j53G3MDUtA0nNULDRCGTw9q7qZbe2k7m1460e1Huk0wFI0UaQLgywaMJikW9gU+yJSGS
         d/NImgCB8/B/gyBadONgItV1SRi90244xBlG0SNYgMAGkodH/v28OoKbvvirGPRGsk38
         saQQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695351996; x=1695956796;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date
         :message-id:reply-to;
        bh=P6X0PKOimTbsxZTjYFPpLza08mWiA8aUu9v5BicDRCw=;
        b=O/GOb6gOw+N5V0ID9IyC8modL47g9Qg+2PSwcd63YGG+uT0xmUMEBI2haPRqISIm2C
         YRxQqQYt6MXm2W69AnjQZvCpFYG/7zseCbZGbT48IgGvck+vZGY6f4n1/QCX7CnNvzS6
         KAOPsd42DaNTAbw8GI5dQxh85dApK+WC1NeTqWAEn6ZFu0SS+y9ez/Fbz5voKRLobtPc
         CGa1LREnh6XDgAuJLAV2GbHrJTVeXMBXI5Agf97Bxp8IsrIlNgbmYZI1wWHJi0j3mhvA
         HUbxCQYY63F1rilBcgSi8UVIxiQuOELM7EPW0npOEp9X0XtzxWsVG7iCeqqHcSXf32LZ
         xILQ==
X-Gm-Message-State: AOJu0Yy6IGBY9OISY6mkeVG83L4HiXZmEBnEMKONlmoIzMncYOUp10JM
	InfAj9TZA2e+nohActLaOzc=
X-Google-Smtp-Source: AGHT+IGguXOsPlNeZ6P6fRewB6MqTQ2QHYfhQiTih2KfqgfrthzJhwuk/JVd2WRsCAk7JPC8igNEng==
X-Received: by 2002:a05:6122:1697:b0:493:d68a:951 with SMTP id 23-20020a056122169700b00493d68a0951mr8895441vkl.14.1695351994464;
        Thu, 21 Sep 2023 20:06:34 -0700 (PDT)
Received: from debian.me ([103.124.138.83])
        by smtp.gmail.com with ESMTPSA id j5-20020aa78d05000000b0068fe7e07190sm2121118pfe.3.2023.09.21.20.06.33
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Thu, 21 Sep 2023 20:06:33 -0700 (PDT)
Received: by debian.me (Postfix, from userid 1000)
	id AD13E81B96D2; Fri, 22 Sep 2023 10:06:31 +0700 (WIB)
Date: Fri, 22 Sep 2023 10:06:31 +0700
From: Bagas Sanjaya <bagasdotme@gmail.com>
To: Martin Zaharinov <micron10@gmail.com>
Cc: Eric Dumazet <edumazet@google.com>, Paolo Abeni <pabeni@redhat.com>,
	netdev <netdev@vger.kernel.org>, patchwork-bot+netdevbpf@kernel.org,
	Jakub Kicinski <kuba@kernel.org>,
	Stephen Hemminger <stephen@networkplumber.org>,
	kuba+netdrv@kernel.org, dsahern@gmail.com,
	Florian Westphal <fw@strlen.de>,
	Pablo Neira Ayuso <pablo@netfilter.org>,
	Thorsten Leemhuis <regressions@leemhuis.info>,
	Wangyang Guo <wangyang.guo@intel.com>,
	Arjan Van De Ven <arjan.van.de.ven@intel.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Linux Regressions <regressions@lists.linux.dev>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
Message-ID: <ZQ0EtyRxjzQTrPNd@debian.me>
References: <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com>
 <ZQqOJOa_qTwz_k0V@debian.me>
 <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
 <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
 <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
 <85F1F301-BECA-4210-A81F-12CAEEC85FD7@gmail.com>
 <be58d429-90d1-42ff-a36b-da318db6ee68@gmail.com>
 <6A98504D-DB99-42A5-A829-B81739822CB2@gmail.com>
Precedence: bulk
X-Mailing-List: regressions@lists.linux.dev
List-Id: <regressions.lists.linux.dev>
List-Subscribe: <mailto:regressions+subscribe@lists.linux.dev>
List-Unsubscribe: <mailto:regressions+unsubscribe@lists.linux.dev>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="CmhnOXDG7xp31zXy"
Content-Disposition: inline
In-Reply-To: <6A98504D-DB99-42A5-A829-B81739822CB2@gmail.com>


--CmhnOXDG7xp31zXy
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, Sep 21, 2023 at 11:13:55AM +0300, Martin Zaharinov wrote:
> Hi Bagas,
>=20
>=20
> Its not easy to make this on production, have too many users on it.
>=20
> i make checks and find with kernel 6.3.12-6.5.13 all is fine.
> on first machine that i have with kernel 6.4 and still work run kernel 6.=
4.2 and have problem.
>=20
> in my investigation problem is start after migration to kernel 6.4.x=20
>=20
> in 6.4 kernel is add rcuref :=20
>=20
> https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.4=20
>=20
> commit bc9d3a9f2afca189a6ae40225b6985e3c775375e
> Author: Thomas Gleixner <tglx@linutronix.de>
> Date: Thu Mar 23 21:55:32 2023 +0100
>=20
> net: dst: Switch to rcuref_t reference counting

Is it the culprit you look for? Had you done the bisection and it points
the culprit to that commit

>=20
> Under high contention dst_entry::__refcnt becomes a significant bottlenec=
k.
>=20
> atomic_inc_not_zero() is implemented with a cmpxchg() loop, which goes in=
to
> high retry rates on contention.
>=20
> Switch the reference count to rcuref_t which results in a significant
> performance gain. Rename the reference count member to __rcuref to reflect
> the change.
>=20
> The gain depends on the micro-architecture and the number of concurrent
> operations and has been measured in the range of +25% to +130% with a
> localhost memtier/memcached benchmark which amplifies the problem
> massively.
>=20
> Running the memtier/memcached benchmark over a real (1Gb) network
> connection the conversion on top of the false sharing fix for struct
> dst_entry::__refcnt results in a total gain in the 2%-5% range over the
> upstream baseline.
>=20
> Reported-by: Wangyang Guo <wangyang.guo@intel.com>
> Reported-by: Arjan Van De Ven <arjan.van.de.ven@intel.com>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Link: https://lore.kernel.org/r/20230307125538.989175656@linutronix.de
> Link: https://lore.kernel.org/r/20230323102800.215027837@linutronix.de
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
>=20
>=20
> and i think problem is here :=20
>=20
> --- a/net/core/dst.c
> +++ b/net/core/dst.c
> @@ -66,7 +66,7 @@ void dst_init(struct dst_entry *dst, str
> dst->tclassid =3D 0;
> #endif
> dst->lwtstate =3D NULL;
> - atomic_set(&dst->__refcnt, initial_ref);
> + rcuref_init(&dst->__refcnt, initial_ref);
> dst->__use =3D 0;
> dst->lastuse =3D jiffies;
> dst->flags =3D flags;
> @@ -162,31 +162,15 @@ EXPORT_SYMBOL(dst_dev_put);
>=20
> void dst_release(struct dst_entry *dst)
> {
> - if (dst) {
> - int newrefcnt;
> -
> - newrefcnt =3D atomic_dec_return(&dst->__refcnt);
> - if (WARN_ONCE(newrefcnt < 0, "dst_release underflow"))
> - net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
> - __func__, dst, newrefcnt);
> - if (!newrefcnt)
> - call_rcu_hurry(&dst->rcu_head, dst_destroy_rcu);
> - }
> + if (dst && rcuref_put(&dst->__refcnt))
> + call_rcu_hurry(&dst->rcu_head, dst_destroy_rcu);
> }
> EXPORT_SYMBOL(dst_release);
>=20
> void dst_release_immediate(struct dst_entry *dst)
> {
> - if (dst) {
> - int newrefcnt;
> -
> - newrefcnt =3D atomic_dec_return(&dst->__refcnt);
> - if (WARN_ONCE(newrefcnt < 0, "dst_release_immediate underflow"))
> - net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
> - __func__, dst, newrefcnt);
> - if (!newrefcnt)
> - dst_destroy(dst);
> - }
> + if (dst && rcuref_put(&dst->__refcnt))
> + dst_destroy(dst);
> }
> EXPORT_SYMBOL(dst_release_immediate);
>=20
>=20
> but this is my thinking
>=20

What do you think that above causes your regression?

Confused...

[To Thorsten: I'm unsure if the reporter do the bisection and suddenly he f=
ound
the culprit commit. Should I add it to regzbot? I had dealt with this repor=
ter
before when he reported nginx regression and he didn't respond with bisecti=
on
to the point that I had to mark it as inconclusive (see regzbot dashboard).
What advice can you provide to him?]

--=20
An old man doll... just what I always wanted! - Clara

--CmhnOXDG7xp31zXy
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZQ0EsgAKCRD2uYlJVVFO
o9QDAQDpvd9PEpZckbP7tZxkUL1QqIWYpzgiCXrgfcE/vJFnBwEA9JGq1g2jr+Z/
n37uKBx6W86b3MJGlDkQC4t+TrTnuAI=
=JJdN
-----END PGP SIGNATURE-----

--CmhnOXDG7xp31zXy--

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from wp530.webpack.hosteurope.de (wp530.webpack.hosteurope.de [80.237.130.52])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1ED01E534
	for <regressions@lists.linux.dev>; Fri, 22 Sep 2023 09:50:28 +0000 (UTC)
Received: from [2a02:8108:8980:2478:8cde:aa2c:f324:937e]; authenticated
	by wp530.webpack.hosteurope.de running ExIM with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128)
	id 1qjcnb-0000O5-6N; Fri, 22 Sep 2023 11:50:11 +0200
Message-ID: <bab8164c-f74f-44fe-ac3e-c078d30ab1e4@leemhuis.info>
Date: Fri, 22 Sep 2023 11:50:09 +0200
Precedence: bulk
X-Mailing-List: regressions@lists.linux.dev
List-Id: <regressions.lists.linux.dev>
List-Subscribe: <mailto:regressions+subscribe@lists.linux.dev>
List-Unsubscribe: <mailto:regressions+unsubscribe@lists.linux.dev>
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
Content-Language: en-US, de-DE
To: Bagas Sanjaya <bagasdotme@gmail.com>,
 Martin Zaharinov <micron10@gmail.com>
Cc: Eric Dumazet <edumazet@google.com>, Paolo Abeni <pabeni@redhat.com>,
 netdev <netdev@vger.kernel.org>, patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org,
 dsahern@gmail.com, Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>,
 Wangyang Guo <wangyang.guo@intel.com>,
 Arjan Van De Ven <arjan.van.de.ven@intel.com>,
 Thomas Gleixner <tglx@linutronix.de>,
 Linux Regressions <regressions@lists.linux.dev>
References: <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me>
 <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
 <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
 <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
 <85F1F301-BECA-4210-A81F-12CAEEC85FD7@gmail.com>
 <be58d429-90d1-42ff-a36b-da318db6ee68@gmail.com>
 <6A98504D-DB99-42A5-A829-B81739822CB2@gmail.com> <ZQ0EtyRxjzQTrPNd@debian.me>
From: "Linux regression tracking (Thorsten Leemhuis)"
 <regressions@leemhuis.info>
Reply-To: Linux regressions mailing list <regressions@lists.linux.dev>
In-Reply-To: <ZQ0EtyRxjzQTrPNd@debian.me>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-bounce-key: webpack.hosteurope.de;regressions@leemhuis.info;1695376229;4852bd26;
X-HE-SMSGID: 1qjcnb-0000O5-6N

On 22.09.23 05:06, Bagas Sanjaya wrote:
> On Thu, Sep 21, 2023 at 11:13:55AM +0300, Martin Zaharinov wrote:
>>
>> Its not easy to make this on production, have too many users on it.
>>
>> i make checks and find with kernel 6.3.12-6.5.13 all is fine.
>> on first machine that i have with kernel 6.4 and still work run kernel 6.4.2 and have problem.

This is confusing and hard to follow. You want to describe more
carefully which kernels worked (avoid ranges, as I doubt you have tested
everything between 6.3.12-6.5.13) and try to avoid complexity (you seem
to have two machines? if everything works on one, don't even bring it up
except maybe as a side note)

>> in my investigation problem is start after migration to kernel 6.4.x 
>>
>> in 6.4 kernel is add rcuref : 
>>
>> https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.4 
>>
>> commit bc9d3a9f2afca189a6ae40225b6985e3c775375e
>> Author: Thomas Gleixner <tglx@linutronix.de>
>> Date: Thu Mar 23 21:55:32 2023 +0100
>>
>> net: dst: Switch to rcuref_t reference counting
> 
> Is it the culprit you look for? Had you done the bisection and it points
> the culprit to that commit

Martin, if you suspect this to be the culprit try to revert it on top of
the latest kernel; if the problem then goes away it likely is the cause.

> [...]
>> but this is my thinking
> 
> What do you think that above causes your regression?
> 
> Confused...
> 
> [To Thorsten: I'm unsure if the reporter do the bisection and suddenly he found
> the culprit commit. Should I add it to regzbot?

For now: no, things are too confusing and without knowing the culprit I
guess nobody will look into this unless we are extremely lucky.

Ciao, Thorsten




From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA9154419
	for <regressions@lists.linux.dev>; Fri, 22 Sep 2023 11:09:08 +0000 (UTC)
Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-690bd8f89baso1777659b3a.2
        for <regressions@lists.linux.dev>; Fri, 22 Sep 2023 04:09:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695380948; x=1695985748; darn=lists.linux.dev;
        h=content-transfer-encoding:in-reply-to:from:references:cc:to
         :content-language:subject:user-agent:mime-version:date:message-id
         :from:to:cc:subject:date:message-id:reply-to;
        bh=SdhW5ZmgkkMXcQqSV8OZhLWTgesihtREKwvty/DX1/A=;
        b=U+HMXTdOY5bRRbYEjCmni+kWLyU8AVSqwqTL66za5ppbgVaQdS+utz0BulB+5NtA8Y
         9IWoItUqiHZH+0Xc2lc9ojPxmGZFkWaVHHt0d/yY5V9igScSY0FItKZktknw6pRkF5np
         IanPi0NQ6ny2PDZWOwczu+4ZGSu5FXzTefOIIokuFoNtIn+Z4rnxs/9kIsowyb8IHAla
         jHwYu5lh17J4J9RywOVdNfHBEfpc9+iX/7EqcGI3QbhaeLmN4FfDL7I5g45HELVuRbaD
         4Hjikxj/kU79uuYqItw37VaFRWFArCMrVi8soFXZxp7jZGr8Lv1zRbEvv/MMC1PTu9We
         Y2ug==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695380948; x=1695985748;
        h=content-transfer-encoding:in-reply-to:from:references:cc:to
         :content-language:subject:user-agent:mime-version:date:message-id
         :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
        bh=SdhW5ZmgkkMXcQqSV8OZhLWTgesihtREKwvty/DX1/A=;
        b=nTGavc+0mKcYeqC9fPleqHs93jQqSnrjwm+EaKEhomYxReSaF6+HYJ1+oBpYg3ggMz
         E86E42i3+EMzIKjR4sVI3KctAu5e+YpGTrNFKzT4W4ifpog/Lnj2txTCqDnEZJ17TSda
         njFlftQnFiqMYfexLJ3pGkf5z+CowMpFevurztTDe5cp9F34bwckzqqy3x9IsMTgys3Z
         mw++7Hj1DcSd7UEYZaeYmZWn7tonYGAIbIJMLmyTSOU4pmSBj4qx2EtkwmqTsUtiry//
         pi0ko2ylxoeNpeGPTXAIOJB9TL0AHv6PusiDudhwSy/hiVOByG19YepX93fD4+OsvbPc
         MyMg==
X-Gm-Message-State: AOJu0Yxm27w9Yt4dlRFqw7q8NUMsdHAwDasPU5yqtxr0J9l6zLhL/ecF
	mKYsKrI+QKv79xLaDdOlG0rWYrP6kmk=
X-Google-Smtp-Source: AGHT+IF9FbMEYzQFU0lHgOYWASfkHSHMN0ogg0RkFd7Q9Toczyt8nskYuzVLw8/uzWQIAxMrHK/iFQ==
X-Received: by 2002:a05:6a21:81a9:b0:15d:c274:2eb1 with SMTP id pd41-20020a056a2181a900b0015dc2742eb1mr1017324pzb.10.1695380947827;
        Fri, 22 Sep 2023 04:09:07 -0700 (PDT)
Received: from [192.168.0.106] ([103.124.138.83])
        by smtp.gmail.com with ESMTPSA id b1-20020a17090a800100b00276b60aa43bsm4634734pjn.17.2023.09.22.04.09.03
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Fri, 22 Sep 2023 04:09:07 -0700 (PDT)
Message-ID: <c0ea6ee9-92c2-4e9e-b896-d079b6150bbc@gmail.com>
Date: Fri, 22 Sep 2023 18:09:00 +0700
Precedence: bulk
X-Mailing-List: regressions@lists.linux.dev
List-Id: <regressions.lists.linux.dev>
List-Subscribe: <mailto:regressions+subscribe@lists.linux.dev>
List-Unsubscribe: <mailto:regressions+unsubscribe@lists.linux.dev>
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
Content-Language: en-US
To: Linux regressions mailing list <regressions@lists.linux.dev>,
 Martin Zaharinov <micron10@gmail.com>,
 Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Cc: Eric Dumazet <edumazet@google.com>, Paolo Abeni <pabeni@redhat.com>,
 netdev <netdev@vger.kernel.org>, patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org,
 dsahern@gmail.com, Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>,
 Wangyang Guo <wangyang.guo@intel.com>,
 Arjan Van De Ven <arjan.van.de.ven@intel.com>,
 Thomas Gleixner <tglx@linutronix.de>
References: <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me>
 <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
 <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
 <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
 <85F1F301-BECA-4210-A81F-12CAEEC85FD7@gmail.com>
 <be58d429-90d1-42ff-a36b-da318db6ee68@gmail.com>
 <6A98504D-DB99-42A5-A829-B81739822CB2@gmail.com> <ZQ0EtyRxjzQTrPNd@debian.me>
 <bab8164c-f74f-44fe-ac3e-c078d30ab1e4@leemhuis.info>
From: Bagas Sanjaya <bagasdotme@gmail.com>
In-Reply-To: <bab8164c-f74f-44fe-ac3e-c078d30ab1e4@leemhuis.info>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 22/09/2023 16:50, Linux regression tracking (Thorsten Leemhuis) wrote:
> On 22.09.23 05:06, Bagas Sanjaya wrote:
>> [To Thorsten: I'm unsure if the reporter do the bisection and suddenly he found
>> the culprit commit. Should I add it to regzbot?
> 
> For now: no, things are too confusing and without knowing the culprit I
> guess nobody will look into this unless we are extremely lucky.
> 

OK, thanks!

-- 
An old man doll... just what I always wanted! - Clara


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8A9C2591
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 04:05:22 +0000 (UTC)
Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72E13268E
	for <netdev@vger.kernel.org>; Thu, 14 Sep 2023 21:05:20 -0700 (PDT)
Received: by mail-lj1-x229.google.com with SMTP id 38308e7fff4ca-2bfb1167277so27241951fa.2
        for <netdev@vger.kernel.org>; Thu, 14 Sep 2023 21:05:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1694750718; x=1695355518; darn=vger.kernel.org;
        h=to:cc:date:message-id:subject:mime-version
         :content-transfer-encoding:from:from:to:cc:subject:date:message-id
         :reply-to;
        bh=YWcFcCaJYAkquxx4cCMFv1RMQ/hDHYGRQbHjVnWTsUM=;
        b=Zhug7PLF1CKbPbDoRepZwinTdiDsDTpYemYGiufX7NU2JFmAlHj8MkneqDWI/oZ/f6
         myOtRSFSGIOPSxnK3vzUoPqyLRIUFSCd/nCCF+I1e6UG2neeVPl+8qvFLeCK1zxoM8dA
         qxP6npvdxTOg7tzU0YQc93EWm6IIA+BsnVgbQov01w+EvM24GdKyzT73zUhzuvp9SxIp
         vhJYTRKwbvop1GTrK829FGRh7hherqyl+pxMeZxJQEYG36jV6FtDYDkJ2IHA23d97kJy
         7RsfHdfz2V8lharMhyydJbHW6kiqg9RJMt9zcRHCooR7xuMHnx108kdd6PPZYDB+bNql
         B74g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694750718; x=1695355518;
        h=to:cc:date:message-id:subject:mime-version
         :content-transfer-encoding:from:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=YWcFcCaJYAkquxx4cCMFv1RMQ/hDHYGRQbHjVnWTsUM=;
        b=BqcINY7n3dvtN7j8F80EQE02wFemdVhh5Z07IJKgVXLplX2loeT2csezczFY1GCmIz
         tWibQtBcG+4h8VvdnKECsqc0g3I0lMOpZhguRhpvcmuwK3zYxQP/OekiOaa+qvDNHp8m
         EpcU7yYzb/3zXW06e0ZxUir6DbLsiSdOCQwP57YSP7n1YQXNfHkrrX0gxDqHucOIrVyJ
         1eYxsUElHhqNkaL9JSseFCnfTn0EVdcjB5VK3UnDAnXcR/QGbUkNacFD0bNbxrSZSyOd
         yVOWUsD3PtAfZYK9K1FLAraE+fZ9pJ0IyNDkBLTU6LiJVsDfN75W4D0uV1xC+KtAtHYL
         Jz1g==
X-Gm-Message-State: AOJu0YxEl27cd8eKpnlpRqAhXrUEv0pTYM6TGh8s+TYMYgRbHqcdZzpA
	0BS1b6oD6p0ikUm+g11qHZPtKEcU3zw=
X-Google-Smtp-Source: AGHT+IEu2G9IYLF1eKV5qcmS3/2TlpG4N0ifikwu8WkL/swPKHA3si4uzsmvQ2rkpDHFU2NtYyImtA==
X-Received: by 2002:a2e:720a:0:b0:2bd:d34:f892 with SMTP id n10-20020a2e720a000000b002bd0d34f892mr546020ljc.3.1694750717832;
        Thu, 14 Sep 2023 21:05:17 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id vl3-20020a170907b60300b009adc77fe164sm732122ejc.66.2023.09.14.21.05.16
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Thu, 14 Sep 2023 21:05:17 -0700 (PDT)
From: Martin Zaharinov <micron10@gmail.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Urgent Bug Report Kernel crash 6.5.2 
Message-Id: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
Date: Fri, 15 Sep 2023 07:05:05 +0300
Cc: Eric Dumazet <edumazet@google.com>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com
To: netdev <netdev@vger.kernel.org>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi All=20
This is report from kernel 6.5.2 after 4 day up system hang and reboot =
after this error :



Sep 15 04:32:29 205.254.184.12 [399661.971344][   C31] kernel tried to =
execute NX-protected page - exploit attempt? (uid: 0)
Sep 15 04:32:29 205.254.184.12 [399661.971470][   C31] BUG: unable to =
handle page fault for address: ffffa10c52d43058
Sep 15 04:32:29 205.254.184.12 [399661.971586][   C31] #PF: supervisor =
instruction fetch in kernel mode
Sep 15 04:32:29 205.254.184.12 [399661.971680][   C31] #PF: =
error_code(0x0011) - permissions violation
Sep 15 04:32:29 205.254.184.12 [399661.971775][   C31] PGD 12601067 P4D =
12601067 PUD 80000002400001e3
Sep 15 04:32:29 205.254.184.12 [399661.971871][   C31] Oops: 0011 [#1] =
PREEMPT SMP
Sep 15 04:32:29 205.254.184.12 [399661.971963][   C31] CPU: 31 PID: 0 =
Comm: swapper/31 Tainted: G        W  O       6.5.2 #1
Sep 15 04:32:29 205.254.184.12 [399661.972079][   C31] Hardware name: =
Supermicro SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
Sep 15 04:32:29 205.254.184.12 [399661.972197][   C31] RIP: =
0010:0xffffa10c52d43058
Sep 15 04:32:29 205.254.184.12 [399661.972289][   C31] Code: 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 =
30 d4 52 0c a1 ff ff 00 00 00 00 00 00
Sep 15 04:32:29 205.254.184.12 [399661.972448][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
Sep 15 04:32:29 205.254.184.12 [399661.972543][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
Sep 15 04:32:29 205.254.184.12 [399661.972659][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
Sep 15 04:32:29 205.254.184.12 [399661.972774][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
Sep 15 04:32:29 205.254.184.12 [399661.972889][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
Sep 15 04:32:29 205.254.184.12 [399661.973005][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
Sep 15 04:32:29 205.254.184.12 [399661.973123][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
Sep 15 04:32:29 205.254.184.12 [399661.973244][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
Sep 15 04:32:29 205.254.184.12 [399661.973338][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
Sep 15 04:32:29 205.254.184.12 [399661.973454][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 15 04:32:29 205.254.184.12 [399661.973569][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Sep 15 04:32:29 205.254.184.12 [399661.973684][   C31] Call Trace:
Sep 15 04:32:29 205.254.184.12 [399661.973773][   C31]  <IRQ>
Sep 15 04:32:29 205.254.184.12 [399661.973859][   C31]  ? =
__die+0xe4/0xf0
Sep 15 04:32:29 205.254.184.12 [399661.973949][   C31]  ? =
page_fault_oops+0x144/0x3e0
Sep 15 04:32:29 205.254.184.12 [399661.974043][   C31]  ? =
exc_page_fault+0x92/0xa0
Sep 15 04:32:29 205.254.184.12 [399661.974136][   C31]  ? =
asm_exc_page_fault+0x22/0x30
Sep 15 04:32:29 205.254.184.12 [399661.974228][   C31]  ? =
kfree_skb_reason+0x33/0xf0
Sep 15 04:32:29 205.254.184.12 [399661.974321][   C31]  ? =
tcp_mtu_probe+0x3a6/0x7b0
Sep 15 04:32:29 205.254.184.12 [399661.974416][   C31]  ? =
tcp_write_xmit+0x7fa/0x1410
Sep 15 04:32:29 205.254.184.12 [399661.974509][   C31]  ? =
__tcp_push_pending_frames+0x2d/0xb0
Sep 15 04:32:29 205.254.184.12 [399661.974603][   C31]  ? =
tcp_rcv_established+0x381/0x610
Sep 15 04:32:29 205.254.184.12 [399661.974695][   C31]  ? =
sk_filter_trim_cap+0xc6/0x1c0
Sep 15 04:32:29 205.254.184.12 [399661.974787][   C31]  ? =
tcp_v4_do_rcv+0x11f/0x1f0
Sep 15 04:32:29 205.254.184.12 [399661.974877][   C31]  ? =
tcp_v4_rcv+0xfa1/0x1010
Sep 15 04:32:29 205.254.184.12 [399661.974968][   C31]  ? =
ip_protocol_deliver_rcu+0x1b/0x270
Sep 15 04:32:29 205.254.184.12 [399661.975062][   C31]  ? =
ip_local_deliver_finish+0x6d/0x90
Sep 15 04:32:29 205.254.184.12 [399661.976257][   C31]  ? =
process_backlog+0x10c/0x230
Sep 15 04:32:29 205.254.184.12 [399661.976352][   C31]  ? =
__napi_poll+0x20/0x180
Sep 15 04:32:29 205.254.184.12 [399661.976442][   C31]  ? =
net_rx_action+0x2a4/0x390
Sep 15 04:32:29 205.254.184.12 [399661.976534][   C31]  ? =
__do_softirq+0xd0/0x202
Sep 15 04:32:29 205.254.184.12 [399661.976626][   C31]  ? =
do_softirq+0x3a/0x50
Sep 15 04:32:29 205.254.184.12 [399661.976718][   C31]  </IRQ>
Sep 15 04:32:29 205.254.184.12 [399661.976805][   C31]  <TASK>
Sep 15 04:32:29 205.254.184.12 [399661.976890][   C31]  ? =
flush_smp_call_function_queue+0x3f/0x50
Sep 15 04:32:29 205.254.184.12 [399661.976988][   C31]  ? =
do_idle+0x14d/0x210
Sep 15 04:32:29 205.254.184.12 [399661.977078][   C31]  ? =
cpu_startup_entry+0x14/0x20
Sep 15 04:32:29 205.254.184.12 [399661.977168][   C31]  ? =
start_secondary+0xe1/0xf0
Sep 15 04:32:29 205.254.184.12 [399661.977262][   C31]  ? =
secondary_startup_64_no_verify+0x167/0x16b
Sep 15 04:32:29 205.254.184.12 [399661.977359][   C31]  </TASK>
Sep 15 04:32:29 205.254.184.12 [399661.977448][   C31] Modules linked =
in: nft_limit nf_conntrack_netlink  pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos=20
Sep 15 04:32:29 205.254.184.12 [399661.977720][   C31] CR2: =
ffffa10c52d43058
Sep 15 04:32:29 205.254.184.12 [399661.977809][   C31] ---[ end trace =
0000000000000000 ]---
Sep 15 04:32:29 205.254.184.12 [399661.977901][   C31] RIP: =
0010:0xffffa10c52d43058
Sep 15 04:32:29 205.254.184.12 [399661.977992][   C31] Code: 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 =
30 d4 52 0c a1 ff ff 00 00 00 00 00 00
Sep 15 04:32:29 205.254.184.12 [399661.978150][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
Sep 15 04:32:29 205.254.184.12 [399661.978243][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
Sep 15 04:32:29 205.254.184.12 [399661.978358][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
Sep 15 04:32:29 205.254.184.12 [399661.978472][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
Sep 15 04:32:29 205.254.184.12 [399661.978587][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
Sep 15 04:32:29 205.254.184.12 [399661.978702][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
Sep 15 04:32:29 205.254.184.12 [399661.978818][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
Sep 15 04:32:29 205.254.184.12 [399661.978940][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
Sep 15 04:32:29 205.254.184.12 [399661.979036][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
Sep 15 04:32:29 205.254.184.12 [399661.979150][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Sep 15 04:32:29 205.254.184.12 [399661.979265][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Sep 15 04:32:29 205.254.184.12 [399661.979381][   C31] Kernel panic - =
not syncing: Fatal exception in interrupt
Sep 15 04:32:29 205.254.184.12 [399662.084038][   C31] Kernel Offset: =
0x1e000000 from 0xffffffff81000000 (relocation range: =
0xffffffff80000000-0xffffffffbfffffff)
Sep 15 04:32:29 205.254.184.12 [399662.084162][   C31] Rebooting in 10 =
seconds..


Please if find fix update me .

m.=

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id DDEFF2591
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 06:45:22 +0000 (UTC)
Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 606A3CCD
	for <netdev@vger.kernel.org>; Thu, 14 Sep 2023 23:45:20 -0700 (PDT)
Received: by mail-qt1-x82f.google.com with SMTP id d75a77b69052e-415155b2796so147971cf.1
        for <netdev@vger.kernel.org>; Thu, 14 Sep 2023 23:45:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20230601; t=1694760319; x=1695365119; darn=vger.kernel.org;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=wvwiqqx7BcBIIqhMe2ijZb4+H4pqKVV4KLfWr6jeEvA=;
        b=b4C1bYPXUd2B5pzJ5gQJOUWdYVPmchNzIw17+Lc9OWbZ53Gh3r6DTqtP0Ztx5uKNwD
         8Nh/8nPWFu+0K5LGYEbDxNCvUfM9Vp2WHniZxJGleguUt30GDAJiU9NbvTvbk7Plqh0l
         iIrjxy5B7Zu1gLPNW9ahBGWgm8iWu+6m/A6kBWgewXvYyZMTl85JQNeTpK9GR3unRmHd
         jpNznckGuKi2r9tQncSk7IckibUvm9lwbRs/FN0Jy0SaxIeY87hXkRFr/P1/m7GsSfhX
         dZLvldFleWMtGWaiqZMs3tbLM7dpCpaq6nLCNyFYUgHq/cdUFfggHZEuN/+RB5UcR6uP
         F+rg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694760319; x=1695365119;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=wvwiqqx7BcBIIqhMe2ijZb4+H4pqKVV4KLfWr6jeEvA=;
        b=Q+PhJPnKUenGSMGA8c4iPbOuuzijV50rRP6jTmq+qrGdW+E9e6gQSmlIGcZBIet7+R
         fuhwRe5yxwdjmwRuEatl6P9WjJKs7EJzO3fhZYerWTitinGsaaJXNn4MMwCz5FK2NsQj
         4KvIKHd3/HryblpkdPwJSUgt6HmBN6gBQSARKixyF6Z7hlmbjsD7YjvKgiM/SzJhtBHT
         kvNvPiUI9KG/KVbl+NkJ8FJiJXjb2MZpqdZgO8CUkzajlRjvVd89jntHA2FbIqsrJJ+Q
         WErLAvK4994uhQ/ON28oLHw6evw7TGs9lg/zZOK2DEpCgfLtJuMxerLOXNQkT8hRM/S3
         3OUg==
X-Gm-Message-State: AOJu0YzLGCEL6p0rZsis7TDzGH8PQ/UdesJOQV0HbAhNXZjV5jJR2k+U
	KlyAs6iOayzWQFfdfIAsYrz55EZFImVaMeNR5l9cFg==
X-Google-Smtp-Source: AGHT+IE8dhLna9CPA/02i1/GKpBBcS0I/fWlPfIklbqb7uegAO76G+g1VGPsIznVndDoGYorOjRYU/ujbWeZaw/HzJ4=
X-Received: by 2002:ac8:5847:0:b0:3f5:2006:50f1 with SMTP id
 h7-20020ac85847000000b003f5200650f1mr203977qth.12.1694760319240; Thu, 14 Sep
 2023 23:45:19 -0700 (PDT)
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
In-Reply-To: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
From: Eric Dumazet <edumazet@google.com>
Date: Fri, 15 Sep 2023 08:45:07 +0200
Message-ID: <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
To: Martin Zaharinov <micron10@gmail.com>
Cc: netdev <netdev@vger.kernel.org>, Paolo Abeni <pabeni@redhat.com>, 
	patchwork-bot+netdevbpf@kernel.org, Jakub Kicinski <kuba@kernel.org>, 
	Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org, dsahern@gmail.com
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,
	USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On Fri, Sep 15, 2023 at 6:05=E2=80=AFAM Martin Zaharinov <micron10@gmail.co=
m> wrote:
>
> Hi All
> This is report from kernel 6.5.2 after 4 day up system hang and reboot af=
ter this error :
>
>
>
> Sep 15 04:32:29 205.254.184.12 [399661.971344][   C31] kernel tried to ex=
ecute NX-protected page - exploit attempt? (uid: 0)
> Sep 15 04:32:29 205.254.184.12 [399661.971470][   C31] BUG: unable to han=
dle page fault for address: ffffa10c52d43058
> Sep 15 04:32:29 205.254.184.12 [399661.971586][   C31] #PF: supervisor in=
struction fetch in kernel mode
> Sep 15 04:32:29 205.254.184.12 [399661.971680][   C31] #PF: error_code(0x=
0011) - permissions violation
> Sep 15 04:32:29 205.254.184.12 [399661.971775][   C31] PGD 12601067 P4D 1=
2601067 PUD 80000002400001e3
> Sep 15 04:32:29 205.254.184.12 [399661.971871][   C31] Oops: 0011 [#1] PR=
EEMPT SMP
> Sep 15 04:32:29 205.254.184.12 [399661.971963][   C31] CPU: 31 PID: 0 Com=
m: swapper/31 Tainted: G        W  O       6.5.2 #1
> Sep 15 04:32:29 205.254.184.12 [399661.972079][   C31] Hardware name: Sup=
ermicro SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
> Sep 15 04:32:29 205.254.184.12 [399661.972197][   C31] RIP: 0010:0xffffa1=
0c52d43058
> Sep 15 04:32:29 205.254.184.12 [399661.972289][   C31] Code: 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 30 d4 5=
2 0c a1 ff ff 00 00 00 00 00 00
> Sep 15 04:32:29 205.254.184.12 [399661.972448][   C31] RSP: 0018:ffffad0e=
0097ccc8 EFLAGS: 00010282
> Sep 15 04:32:29 205.254.184.12 [399661.972543][   C31] RAX: ffffa10c52d43=
058 RBX: ffffa10c52d43000 RCX: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.972659][   C31] RDX: 0000000000002=
712 RSI: 0000000000000246 RDI: ffffa10c52d43000
> Sep 15 04:32:29 205.254.184.12 [399661.972774][   C31] RBP: ffffa10c52d43=
000 R08: 0000000127a83c46 R09: 0000000000004d8c
> Sep 15 04:32:29 205.254.184.12 [399661.972889][   C31] R10: ffffe840ca0f7=
c00 R11: 0000000000000000 R12: ffffa10c8e764d80
> Sep 15 04:32:29 205.254.184.12 [399661.973005][   C31] R13: ffffa10c92b4c=
760 R14: 0000000000000058 R15: ffffa10c92b4c600
> Sep 15 04:32:29 205.254.184.12 [399661.973123][   C31] FS:  0000000000000=
000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.973244][   C31] CS:  0010 DS: 0000=
 ES: 0000 CR0: 0000000080050033
> Sep 15 04:32:29 205.254.184.12 [399661.973338][   C31] CR2: ffffa10c52d43=
058 CR3: 00000001059b8001 CR4: 00000000003706e0
> Sep 15 04:32:29 205.254.184.12 [399661.973454][   C31] DR0: 0000000000000=
000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.973569][   C31] DR3: 0000000000000=
000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Sep 15 04:32:29 205.254.184.12 [399661.973684][   C31] Call Trace:
> Sep 15 04:32:29 205.254.184.12 [399661.973773][   C31]  <IRQ>
> Sep 15 04:32:29 205.254.184.12 [399661.973859][   C31]  ? __die+0xe4/0xf0
> Sep 15 04:32:29 205.254.184.12 [399661.973949][   C31]  ? page_fault_oops=
+0x144/0x3e0
> Sep 15 04:32:29 205.254.184.12 [399661.974043][   C31]  ? exc_page_fault+=
0x92/0xa0
> Sep 15 04:32:29 205.254.184.12 [399661.974136][   C31]  ? asm_exc_page_fa=
ult+0x22/0x30
> Sep 15 04:32:29 205.254.184.12 [399661.974228][   C31]  ? kfree_skb_reaso=
n+0x33/0xf0
> Sep 15 04:32:29 205.254.184.12 [399661.974321][   C31]  ? tcp_mtu_probe+0=
x3a6/0x7b0
> Sep 15 04:32:29 205.254.184.12 [399661.974416][   C31]  ? tcp_write_xmit+=
0x7fa/0x1410
> Sep 15 04:32:29 205.254.184.12 [399661.974509][   C31]  ? __tcp_push_pend=
ing_frames+0x2d/0xb0
> Sep 15 04:32:29 205.254.184.12 [399661.974603][   C31]  ? tcp_rcv_establi=
shed+0x381/0x610
> Sep 15 04:32:29 205.254.184.12 [399661.974695][   C31]  ? sk_filter_trim_=
cap+0xc6/0x1c0
> Sep 15 04:32:29 205.254.184.12 [399661.974787][   C31]  ? tcp_v4_do_rcv+0=
x11f/0x1f0
> Sep 15 04:32:29 205.254.184.12 [399661.974877][   C31]  ? tcp_v4_rcv+0xfa=
1/0x1010

Your reports are not usable. Please make sure to include symbols next time.

Please read these parts (and possibly complete files)

Documentation/admin-guide/bug-hunting.rst:55:quality of the stack
trace by using file:`scripts/decode_stacktrace.sh`.

Documentation/admin-guide/reporting-issues.rst:978:
[user@something ~]$ sudo dmesg |
./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux
Documentation/admin-guide/reporting-issues.rst:985:
[user@something ~]$ sudo dmesg |
./linux-5.10.5/scripts/decode_stacktrace.sh \



> Sep 15 04:32:29 205.254.184.12 [399661.974968][   C31]  ? ip_protocol_del=
iver_rcu+0x1b/0x270
> Sep 15 04:32:29 205.254.184.12 [399661.975062][   C31]  ? ip_local_delive=
r_finish+0x6d/0x90
> Sep 15 04:32:29 205.254.184.12 [399661.976257][   C31]  ? process_backlog=
+0x10c/0x230
> Sep 15 04:32:29 205.254.184.12 [399661.976352][   C31]  ? __napi_poll+0x2=
0/0x180
> Sep 15 04:32:29 205.254.184.12 [399661.976442][   C31]  ? net_rx_action+0=
x2a4/0x390
> Sep 15 04:32:29 205.254.184.12 [399661.976534][   C31]  ? __do_softirq+0x=
d0/0x202
> Sep 15 04:32:29 205.254.184.12 [399661.976626][   C31]  ? do_softirq+0x3a=
/0x50
> Sep 15 04:32:29 205.254.184.12 [399661.976718][   C31]  </IRQ>
> Sep 15 04:32:29 205.254.184.12 [399661.976805][   C31]  <TASK>
> Sep 15 04:32:29 205.254.184.12 [399661.976890][   C31]  ? flush_smp_call_=
function_queue+0x3f/0x50
> Sep 15 04:32:29 205.254.184.12 [399661.976988][   C31]  ? do_idle+0x14d/0=
x210
> Sep 15 04:32:29 205.254.184.12 [399661.977078][   C31]  ? cpu_startup_ent=
ry+0x14/0x20
> Sep 15 04:32:29 205.254.184.12 [399661.977168][   C31]  ? start_secondary=
+0xe1/0xf0
> Sep 15 04:32:29 205.254.184.12 [399661.977262][   C31]  ? secondary_start=
up_64_no_verify+0x167/0x16b
> Sep 15 04:32:29 205.254.184.12 [399661.977359][   C31]  </TASK>
> Sep 15 04:32:29 205.254.184.12 [399661.977448][   C31] Modules linked in:=
 nft_limit nf_conntrack_netlink  pppoe pppox ppp_generic slhc nft_ct nft_na=
t nft_chain_nat nf_tables netconsole coretemp bonding i40e nf_nat_sip nf_co=
nntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_=
nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
> Sep 15 04:32:29 205.254.184.12 [399661.977720][   C31] CR2: ffffa10c52d43=
058
> Sep 15 04:32:29 205.254.184.12 [399661.977809][   C31] ---[ end trace 000=
0000000000000 ]---
> Sep 15 04:32:29 205.254.184.12 [399661.977901][   C31] RIP: 0010:0xffffa1=
0c52d43058
> Sep 15 04:32:29 205.254.184.12 [399661.977992][   C31] Code: 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 30 d4 5=
2 0c a1 ff ff 00 00 00 00 00 00
> Sep 15 04:32:29 205.254.184.12 [399661.978150][   C31] RSP: 0018:ffffad0e=
0097ccc8 EFLAGS: 00010282
> Sep 15 04:32:29 205.254.184.12 [399661.978243][   C31] RAX: ffffa10c52d43=
058 RBX: ffffa10c52d43000 RCX: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.978358][   C31] RDX: 0000000000002=
712 RSI: 0000000000000246 RDI: ffffa10c52d43000
> Sep 15 04:32:29 205.254.184.12 [399661.978472][   C31] RBP: ffffa10c52d43=
000 R08: 0000000127a83c46 R09: 0000000000004d8c
> Sep 15 04:32:29 205.254.184.12 [399661.978587][   C31] R10: ffffe840ca0f7=
c00 R11: 0000000000000000 R12: ffffa10c8e764d80
> Sep 15 04:32:29 205.254.184.12 [399661.978702][   C31] R13: ffffa10c92b4c=
760 R14: 0000000000000058 R15: ffffa10c92b4c600
> Sep 15 04:32:29 205.254.184.12 [399661.978818][   C31] FS:  0000000000000=
000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.978940][   C31] CS:  0010 DS: 0000=
 ES: 0000 CR0: 0000000080050033
> Sep 15 04:32:29 205.254.184.12 [399661.979036][   C31] CR2: ffffa10c52d43=
058 CR3: 00000001059b8001 CR4: 00000000003706e0
> Sep 15 04:32:29 205.254.184.12 [399661.979150][   C31] DR0: 0000000000000=
000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.979265][   C31] DR3: 0000000000000=
000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Sep 15 04:32:29 205.254.184.12 [399661.979381][   C31] Kernel panic - not=
 syncing: Fatal exception in interrupt
> Sep 15 04:32:29 205.254.184.12 [399662.084038][   C31] Kernel Offset: 0x1=
e000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xfff=
fffffbfffffff)
> Sep 15 04:32:29 205.254.184.12 [399662.084162][   C31] Rebooting in 10 se=
conds..
>
>
> Please if find fix update me .
>
> m.

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id A605915EAE
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 22:24:40 +0000 (UTC)
Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 601282736
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 15:23:36 -0700 (PDT)
Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2bf78950354so43080361fa.1
        for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 15:23:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1694816614; x=1695421414; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=uUU0eWW6K/1GiQwgXluBWiQNvBOWuUpRcJMir0I0Ku8=;
        b=Qxf5gZDVm+ZUN0525Xv5UsMgj8249zqyqvdhA4PYRJUqPYZ7mueyg9LV9X1BaK3dNq
         cJd3GnqlrpOUt/3Ryu9uuCH/nSjR2KDoskSZaQ3A5SOoA5HxcZELkqEEKyu86EkYtkj3
         A3x034qVrzgAGRofq4rRUEglEqP21P/xmMaOLfg1N3rUXdmZavHwy6CwnYvpI+Ae2JzA
         cZVhOyJhs3yALwFqR0np1ROTEtKkyZ3aiJTTAk9FGmQmeVfaS30UtpqCFBjQat/s7osH
         RE3iTEp1P2xiUxiFjE6EVw8VS1cVieDPckDEE+kPNplL+whkNi+Ys1mp6JyIi81OsNvV
         cAsA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694816614; x=1695421414;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=uUU0eWW6K/1GiQwgXluBWiQNvBOWuUpRcJMir0I0Ku8=;
        b=TYjK7ZTu+m/Nr3bRFa12EnKU2cRT09qW2hbBUbfm+oHMJFCkBjfdLoBbHRWZ6sQZSp
         j07xYcRP/OQhr4Xtgn9HaqRvdunUyg1kBltt3axoEsG/WTSCQEsSTdedH+mFW31cDkQH
         7sTIdDlCzhRRsvriuvXSvAPp2DomjQSYqa9VOVI2Er+sczImEh4r91mwZ4hUQJOQQgY7
         rdhix9xEpwJDW+y0UVMYnKYg42/7876raI3tTho4p9AiecKMfpvGNlxMQwdrczeQaAUs
         doi2Foez1hHBNStO3CHIR/EiT58aCDv9ussMnvit8afZVoowsDmK6awDoBDPBOwJuPOL
         QvMA==
X-Gm-Message-State: AOJu0YwyO18spV6MJdw/N7btaQWfmSQkZ3f1PSmdDa6REl2AqpYfBNVG
	Bwn4WjcVJXoRotNFgzARG1k=
X-Google-Smtp-Source: AGHT+IEtS1G+7SG7ac56a5vfMo8T5Cq9Pv/PjmDJIcUN4Dfqus2FFAc7bYe8JxshKcoQfZNgEl23Xg==
X-Received: by 2002:a2e:7c19:0:b0:2b9:ef0a:7d4b with SMTP id x25-20020a2e7c19000000b002b9ef0a7d4bmr2538491ljc.31.1694816614119;
        Fri, 15 Sep 2023 15:23:34 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id x4-20020a1709060a4400b0099bd6026f45sm2924097ejf.198.2023.09.15.15.23.33
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 15 Sep 2023 15:23:33 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
Date: Sat, 16 Sep 2023 01:23:22 +0300
Cc: netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <9E6AB656-0DE7-4837-A7AE-F8A0CCBC6F2D@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
To: Eric Dumazet <edumazet@google.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi Eric

run decode script =E2=80=A6 but i think miss function name =E2=80=A6 =
check log :=20


  15:	eb cf                	jmp    0xffffffffffffffe6
  17:	48 c7 c7 68 f6 e2 8e 	mov    $0xffffffff8ee2f668,%rdi
  1e:	c6 05 ac ae e6 00 01 	movb   $0x1,0xe6aeac(%rip)        # =
0xe6aed1
  25:	e8 11 71 c7 ff       	call   0xffffffffffc7713b
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb df                	jmp    0xd
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	cc                   	int3
  33:	cc                   	int3
  34:	cc                   	int3
  35:	cc                   	int3
  36:	cc                   	int3
  37:	cc                   	int3
  38:	cc                   	int3
  39:	cc                   	int3
  3a:	cc                   	int3
  3b:	48 89 fa             	mov    %rdi,%rdx
  3e:	83                   	.byte 0x83
  3f:	e2                   	.byte 0xe2

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	eb df                	jmp    0xffffffffffffffe3
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	cc                   	int3
   9:	cc                   	int3
   a:	cc                   	int3
   b:	cc                   	int3
   c:	cc                   	int3
   d:	cc                   	int3
   e:	cc                   	int3
   f:	cc                   	int3
  10:	cc                   	int3
  11:	48 89 fa             	mov    %rdi,%rdx
  14:	83                   	.byte 0x83
  15:	e2                   	.byte 0xe2
[40915.531389] RSP: 0018:ffffa62680318de8 EFLAGS: 00010296
[40915.531487] RAX: 0000000000000019 RBX: ffff982f02950c40 RCX: =
00000000fffbffff
[40915.531605] RDX: 00000000fffbffff RSI: 0000000000000001 RDI: =
00000000ffffffea
[40915.531721] RBP: ffff982e467d2000 R08: 0000000000000000 R09: =
00000000fffbffff
[40915.531839] R10: ffff98359d600000 R11: 0000000000000003 R12: =
ffff982f044e16c0
[40915.531956] R13: 0000000000000000 R14: 0000000000000258 R15: =
ffffa62680318f60
[40915.532075] FS:  0000000000000000(0000) GS:ffff98359fbc0000(0000) =
knlGS:0000000000000000
[40915.532195] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[40915.532291] CR2: 00005593eb3ff078 CR3: 0000000179f6e001 CR4: =
00000000003706e0
[40915.532409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[40915.532526] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[40915.532645] Call Trace:
[40915.532736]  <IRQ>
[40915.532824] ? __warn (??:?)
[40915.532918] ? report_bug (??:?)
[40915.533011] ? handle_bug (traps.c:?)
[40915.533104] ? exc_invalid_op (??:?)
[40915.533198] ? asm_exc_invalid_op (??:?)
[40915.533294] ? rcuref_put_slowpath (??:?)
[40915.533389] ? rcuref_put_slowpath (??:?)
[40915.533482] dst_release (??:?)
[40915.533576] __dev_queue_xmit (??:?)
[40915.533671] ? eth_header (??:?)
[40915.533766] ip_finish_output2 (ip_output.c:?)
[40915.533863] process_backlog (dev.c:?)
[40915.533958] __napi_poll (dev.c:?)
[40915.534050] net_rx_action (dev.c:?)
[40915.534140] __do_softirq (??:?)
[40915.534233] do_softirq (??:?)
[40915.534326]  </IRQ>
[40915.534413]  <TASK>
[40915.534503] flush_smp_call_function_queue (??:?)
[40915.534597] do_idle (build_policy.c:?)
[40915.534687] cpu_startup_entry (??:?)
[40915.534778] start_secondary (smpboot.c:?)
[40915.534871] secondary_startup_64_no_verify (??:?)
[40915.534968]  </TASK>
[40915.535057] ---[ end trace 0000000000000000 ]=E2=80=94



For me may be problem is in this part :=20

[40915.533863] process_backlog (dev.c:?)
[40915.533958] __napi_poll (dev.c:?)
[40915.534050] net_rx_action (dev.c:?)

this start after upgrade to kernel 6.3.x
with 6.2.x i dont have this problem.

m.

> On 15 Sep 2023, at 9:45, Eric Dumazet <edumazet@google.com> wrote:
>=20
> On Fri, Sep 15, 2023 at 6:05=E2=80=AFAM Martin Zaharinov =
<micron10@gmail.com> wrote:
>>=20
>> Hi All
>> This is report from kernel 6.5.2 after 4 day up system hang and =
reboot after this error :
>>=20
>>=20
>>=20
>> Sep 15 04:32:29 205.254.184.12 [399661.971344][   C31] kernel tried =
to execute NX-protected page - exploit attempt? (uid: 0)
>> Sep 15 04:32:29 205.254.184.12 [399661.971470][   C31] BUG: unable to =
handle page fault for address: ffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.971586][   C31] #PF: =
supervisor instruction fetch in kernel mode
>> Sep 15 04:32:29 205.254.184.12 [399661.971680][   C31] #PF: =
error_code(0x0011) - permissions violation
>> Sep 15 04:32:29 205.254.184.12 [399661.971775][   C31] PGD 12601067 =
P4D 12601067 PUD 80000002400001e3
>> Sep 15 04:32:29 205.254.184.12 [399661.971871][   C31] Oops: 0011 =
[#1] PREEMPT SMP
>> Sep 15 04:32:29 205.254.184.12 [399661.971963][   C31] CPU: 31 PID: 0 =
Comm: swapper/31 Tainted: G        W  O       6.5.2 #1
>> Sep 15 04:32:29 205.254.184.12 [399661.972079][   C31] Hardware name: =
Supermicro SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
>> Sep 15 04:32:29 205.254.184.12 [399661.972197][   C31] RIP: =
0010:0xffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.972289][   C31] Code: 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 =
58 30 d4 52 0c a1 ff ff 00 00 00 00 00 00
>> Sep 15 04:32:29 205.254.184.12 [399661.972448][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
>> Sep 15 04:32:29 205.254.184.12 [399661.972543][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.972659][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
>> Sep 15 04:32:29 205.254.184.12 [399661.972774][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
>> Sep 15 04:32:29 205.254.184.12 [399661.972889][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
>> Sep 15 04:32:29 205.254.184.12 [399661.973005][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
>> Sep 15 04:32:29 205.254.184.12 [399661.973123][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.973244][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
>> Sep 15 04:32:29 205.254.184.12 [399661.973338][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
>> Sep 15 04:32:29 205.254.184.12 [399661.973454][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.973569][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Sep 15 04:32:29 205.254.184.12 [399661.973684][   C31] Call Trace:
>> Sep 15 04:32:29 205.254.184.12 [399661.973773][   C31]  <IRQ>
>> Sep 15 04:32:29 205.254.184.12 [399661.973859][   C31]  ? =
__die+0xe4/0xf0
>> Sep 15 04:32:29 205.254.184.12 [399661.973949][   C31]  ? =
page_fault_oops+0x144/0x3e0
>> Sep 15 04:32:29 205.254.184.12 [399661.974043][   C31]  ? =
exc_page_fault+0x92/0xa0
>> Sep 15 04:32:29 205.254.184.12 [399661.974136][   C31]  ? =
asm_exc_page_fault+0x22/0x30
>> Sep 15 04:32:29 205.254.184.12 [399661.974228][   C31]  ? =
kfree_skb_reason+0x33/0xf0
>> Sep 15 04:32:29 205.254.184.12 [399661.974321][   C31]  ? =
tcp_mtu_probe+0x3a6/0x7b0
>> Sep 15 04:32:29 205.254.184.12 [399661.974416][   C31]  ? =
tcp_write_xmit+0x7fa/0x1410
>> Sep 15 04:32:29 205.254.184.12 [399661.974509][   C31]  ? =
__tcp_push_pending_frames+0x2d/0xb0
>> Sep 15 04:32:29 205.254.184.12 [399661.974603][   C31]  ? =
tcp_rcv_established+0x381/0x610
>> Sep 15 04:32:29 205.254.184.12 [399661.974695][   C31]  ? =
sk_filter_trim_cap+0xc6/0x1c0
>> Sep 15 04:32:29 205.254.184.12 [399661.974787][   C31]  ? =
tcp_v4_do_rcv+0x11f/0x1f0
>> Sep 15 04:32:29 205.254.184.12 [399661.974877][   C31]  ? =
tcp_v4_rcv+0xfa1/0x1010
>=20
> Your reports are not usable. Please make sure to include symbols next =
time.
>=20
> Please read these parts (and possibly complete files)
>=20
> Documentation/admin-guide/bug-hunting.rst:55:quality of the stack
> trace by using file:`scripts/decode_stacktrace.sh`.
>=20
> Documentation/admin-guide/reporting-issues.rst:978:
> [user@something ~]$ sudo dmesg |
> ./linux-5.10.5/scripts/decode_stacktrace.sh ./linux-5.10.5/vmlinux
> Documentation/admin-guide/reporting-issues.rst:985:
> [user@something ~]$ sudo dmesg |
> ./linux-5.10.5/scripts/decode_stacktrace.sh \
>=20
>=20
>=20
>> Sep 15 04:32:29 205.254.184.12 [399661.974968][   C31]  ? =
ip_protocol_deliver_rcu+0x1b/0x270
>> Sep 15 04:32:29 205.254.184.12 [399661.975062][   C31]  ? =
ip_local_deliver_finish+0x6d/0x90
>> Sep 15 04:32:29 205.254.184.12 [399661.976257][   C31]  ? =
process_backlog+0x10c/0x230
>> Sep 15 04:32:29 205.254.184.12 [399661.976352][   C31]  ? =
__napi_poll+0x20/0x180
>> Sep 15 04:32:29 205.254.184.12 [399661.976442][   C31]  ? =
net_rx_action+0x2a4/0x390
>> Sep 15 04:32:29 205.254.184.12 [399661.976534][   C31]  ? =
__do_softirq+0xd0/0x202
>> Sep 15 04:32:29 205.254.184.12 [399661.976626][   C31]  ? =
do_softirq+0x3a/0x50
>> Sep 15 04:32:29 205.254.184.12 [399661.976718][   C31]  </IRQ>
>> Sep 15 04:32:29 205.254.184.12 [399661.976805][   C31]  <TASK>
>> Sep 15 04:32:29 205.254.184.12 [399661.976890][   C31]  ? =
flush_smp_call_function_queue+0x3f/0x50
>> Sep 15 04:32:29 205.254.184.12 [399661.976988][   C31]  ? =
do_idle+0x14d/0x210
>> Sep 15 04:32:29 205.254.184.12 [399661.977078][   C31]  ? =
cpu_startup_entry+0x14/0x20
>> Sep 15 04:32:29 205.254.184.12 [399661.977168][   C31]  ? =
start_secondary+0xe1/0xf0
>> Sep 15 04:32:29 205.254.184.12 [399661.977262][   C31]  ? =
secondary_startup_64_no_verify+0x167/0x16b
>> Sep 15 04:32:29 205.254.184.12 [399661.977359][   C31]  </TASK>
>> Sep 15 04:32:29 205.254.184.12 [399661.977448][   C31] Modules linked =
in: nft_limit nf_conntrack_netlink  pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos
>> Sep 15 04:32:29 205.254.184.12 [399661.977720][   C31] CR2: =
ffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.977809][   C31] ---[ end trace =
0000000000000000 ]---
>> Sep 15 04:32:29 205.254.184.12 [399661.977901][   C31] RIP: =
0010:0xffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.977992][   C31] Code: 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 =
58 30 d4 52 0c a1 ff ff 00 00 00 00 00 00
>> Sep 15 04:32:29 205.254.184.12 [399661.978150][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
>> Sep 15 04:32:29 205.254.184.12 [399661.978243][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.978358][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
>> Sep 15 04:32:29 205.254.184.12 [399661.978472][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
>> Sep 15 04:32:29 205.254.184.12 [399661.978587][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
>> Sep 15 04:32:29 205.254.184.12 [399661.978702][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
>> Sep 15 04:32:29 205.254.184.12 [399661.978818][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.978940][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
>> Sep 15 04:32:29 205.254.184.12 [399661.979036][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
>> Sep 15 04:32:29 205.254.184.12 [399661.979150][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.979265][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Sep 15 04:32:29 205.254.184.12 [399661.979381][   C31] Kernel panic - =
not syncing: Fatal exception in interrupt
>> Sep 15 04:32:29 205.254.184.12 [399662.084038][   C31] Kernel Offset: =
0x1e000000 from 0xffffffff81000000 (relocation range: =
0xffffffff80000000-0xffffffffbfffffff)
>> Sep 15 04:32:29 205.254.184.12 [399662.084162][   C31] Rebooting in =
10 seconds..
>>=20
>>=20
>> Please if find fix update me .
>>=20
>> m.



From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 086C11B27C
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 23:00:38 +0000 (UTC)
Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73634AF
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 16:00:36 -0700 (PDT)
Received: by mail-lf1-x136.google.com with SMTP id 2adb3069b0e04-502b1bbe5c3so4390327e87.1
        for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 16:00:36 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1694818834; x=1695423634; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=V2/e2CxMtNmP7zEPWjzWz0aoQT2qM96e0VgIi6xCPFc=;
        b=b6K/nX+uqIVFY9A0vfkSHlN6qaJolewcKTNUoMogYK7++VCrznvCG7MHwhV1OboSEm
         n9Pf6hc/qCanKz08LmAI+ynRzjMfk4qH+xZYJ9Xf/dZmAypddXAXPOemtNSHX6beJis/
         y2cvrEwpqI0fR23b7QGkwXOvvKYMb/E5XtrE5xiuH3Bg/QMDXWiY2oBHZM39JAOuggNk
         nOvulvwWj92ZFMd3VjptdQA3npGM4CGvUrf4Rl2+GQ5LdJR6Ch1sCFid4JIOKhWXV1zp
         pPcz6PX4qkhj2N9G6smIV0XP4/1kdwZdZE50PeZN4z0r7r9r0TZjtgIuq/Cdtjm9/0Po
         3GvQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694818834; x=1695423634;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=V2/e2CxMtNmP7zEPWjzWz0aoQT2qM96e0VgIi6xCPFc=;
        b=j/43NvGKQnE+XtWtWc1jOO4tRo95QNPuo0kIK62NpWep2JN9kdaIY06Yhi/V4m0AFZ
         pVdO36Gq0ei7soCPMikR50qcTTYN7A4fDfU+Z9Nz03hj/8McqhEtw6X9zJ0dLgz16ZxY
         QtwJ/yUMClFdMw4XKAmIecersAsVj0ELI3Kw6tijVKKLSF9jtJmWtO//DN4IPDkbCSgM
         OHpS54jrIkcCqgnqjHF4ZPcvf0H39Oa0A3QTvMYtrGGBH+KHBQF0Yld44i/UMwVJln9R
         jtNU+lxzfie1ropGdraiPAHDduvtFBn2q97yQIZKjDdQ12JK7mrEserTZfC6drlU3vav
         oeqQ==
X-Gm-Message-State: AOJu0YyqWaRAcbowf+CEDuG1JfRhP7udi7OYg5bTpZBypkC2VC3e08sS
	JGvOeUhVrKk+1EQTdU0ENPFl4c1aQB8=
X-Google-Smtp-Source: AGHT+IG9r1nQNcBX/kxlwsSYoKmCBQJvfrpopuwi39XM7U+uZEhw8WPgnXE4UHVg5enOYJ4gSFZ/Pw==
X-Received: by 2002:a05:6512:2396:b0:4ff:a04c:8a5b with SMTP id c22-20020a056512239600b004ffa04c8a5bmr3126239lfv.47.1694818834113;
        Fri, 15 Sep 2023 16:00:34 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id va13-20020a17090711cd00b0097404f4a124sm2977549ejb.2.2023.09.15.16.00.33
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 15 Sep 2023 16:00:33 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
Date: Sat, 16 Sep 2023 02:00:22 +0300
Cc: Eric Dumazet <edumazet@google.com>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
To: netdev <netdev@vger.kernel.org>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Ok fix=20
one note this is kernel 6.5.3 =E2=80=A6


see log now :=20


[40915.530445] ------------[ cut here ]------------
[40915.530529] rcuref - imbalanced put()
[40915.530540] WARNING: CPU: 7 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
[40915.530698] Modules linked in: nf_conntrack_netlink nft_limit pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[40915.530899] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G           O      =
 6.5.3 #1
[40915.531018] Hardware name: Supermicro SYS-5038MR-H8TRF/X10SRD-F, BIOS =
3.3 10/28/2020
[40915.531137] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:267 =
(discriminator 1))
[40915.531230] Code: 31 c0 eb e2 80 3d c6 ae e6 00 00 74 0a c7 03 00 00 =
00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 8e c6 05 ac ae e6 00 01 e8 11 71 c7 =
ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	31 c0                	xor    %eax,%eax
   2:	eb e2                	jmp    0xffffffffffffffe6
   4:	80 3d c6 ae e6 00 00 	cmpb   $0x0,0xe6aec6(%rip)        # =
0xe6aed1
   b:	74 0a                	je     0x17
   d:	c7 03 00 00 00 e0    	movl   $0xe0000000,(%rbx)
  13:	31 c0                	xor    %eax,%eax
  15:	eb cf                	jmp    0xffffffffffffffe6
  17:	48 c7 c7 68 f6 e2 8e 	mov    $0xffffffff8ee2f668,%rdi
  1e:	c6 05 ac ae e6 00 01 	movb   $0x1,0xe6aeac(%rip)        # =
0xe6aed1
  25:	e8 11 71 c7 ff       	call   0xffffffffffc7713b
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb df                	jmp    0xd
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	cc                   	int3
  33:	cc                   	int3
  34:	cc                   	int3
  35:	cc                   	int3
  36:	cc                   	int3
  37:	cc                   	int3
  38:	cc                   	int3
  39:	cc                   	int3
  3a:	cc                   	int3
  3b:	48 89 fa             	mov    %rdi,%rdx
  3e:	83                   	.byte 0x83
  3f:	e2                   	.byte 0xe2

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	eb df                	jmp    0xffffffffffffffe3
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	cc                   	int3
   9:	cc                   	int3
   a:	cc                   	int3
   b:	cc                   	int3
   c:	cc                   	int3
   d:	cc                   	int3
   e:	cc                   	int3
   f:	cc                   	int3
  10:	cc                   	int3
  11:	48 89 fa             	mov    %rdi,%rdx
  14:	83                   	.byte 0x83
  15:	e2                   	.byte 0xe2
[40915.531389] RSP: 0018:ffffa62680318de8 EFLAGS: 00010296
[40915.531487] RAX: 0000000000000019 RBX: ffff982f02950c40 RCX: =
00000000fffbffff
[40915.531605] RDX: 00000000fffbffff RSI: 0000000000000001 RDI: =
00000000ffffffea
[40915.531721] RBP: ffff982e467d2000 R08: 0000000000000000 R09: =
00000000fffbffff
[40915.531839] R10: ffff98359d600000 R11: 0000000000000003 R12: =
ffff982f044e16c0
[40915.531956] R13: 0000000000000000 R14: 0000000000000258 R15: =
ffffa62680318f60
[40915.532075] FS:  0000000000000000(0000) GS:ffff98359fbc0000(0000) =
knlGS:0000000000000000
[40915.532195] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[40915.532291] CR2: 00005593eb3ff078 CR3: 0000000179f6e001 CR4: =
00000000003706e0
[40915.532409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[40915.532526] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[40915.532645] Call Trace:
[40915.532736]  <IRQ>
[40915.532824] ? __warn (kernel/panic.c:668)
[40915.532918] ? report_bug (lib/bug.c:223)
[40915.533011] ? handle_bug (arch/x86/kernel/traps.c:324)
[40915.533104] ? exc_invalid_op (arch/x86/kernel/traps.c:345 =
(discriminator 1))
[40915.533198] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[40915.533294] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[40915.533389] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[40915.533482] dst_release (./include/linux/rcuref.h:151 =
net/core/dst.c:166)
[40915.533576] __dev_queue_xmit (net/core/dev.c:4138)
[40915.533671] ? eth_header (net/ethernet/eth.c:83)
[40915.533766] ip_finish_output2 (./include/net/neighbour.h:544 =
net/ipv4/ip_output.c:230)
[40915.533863] process_backlog (net/core/dev.c:5451 net/core/dev.c:5566 =
net/core/dev.c:5895)
[40915.533958] __napi_poll+0x20/0x180
[40915.534050] net_rx_action (net/core/dev.c:5839 net/core/dev.c:5860 =
net/core/dev.c:6684)
[40915.534140] __do_softirq (./arch/x86/include/asm/bitops.h:319 =
kernel/softirq.c:550)
[40915.534233] do_softirq (kernel/softirq.c:463 (discriminator 32))
[40915.534326]  </IRQ>
[40915.534413]  <TASK>
[40915.534503] flush_smp_call_function_queue (kernel/smp.c:563 =
(discriminator 1))
[40915.534597] do_idle (kernel/sched/idle.c:295)
[40915.534687] cpu_startup_entry (kernel/sched/idle.c:379 (discriminator =
1))
[40915.534778] start_secondary (arch/x86/kernel/smpboot.c:326)
[40915.534871] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:441)
[40915.534968]  </TASK>
[40915.535057] ---[ end trace 0000000000000000 ]---

> On 15 Sep 2023, at 7:05, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi All=20
> This is report from kernel 6.5.2 after 4 day up system hang and reboot =
after this error :
>=20
>=20
>=20
> Sep 15 04:32:29 205.254.184.12 [399661.971344][   C31] kernel tried to =
execute NX-protected page - exploit attempt? (uid: 0)
> Sep 15 04:32:29 205.254.184.12 [399661.971470][   C31] BUG: unable to =
handle page fault for address: ffffa10c52d43058
> Sep 15 04:32:29 205.254.184.12 [399661.971586][   C31] #PF: supervisor =
instruction fetch in kernel mode
> Sep 15 04:32:29 205.254.184.12 [399661.971680][   C31] #PF: =
error_code(0x0011) - permissions violation
> Sep 15 04:32:29 205.254.184.12 [399661.971775][   C31] PGD 12601067 =
P4D 12601067 PUD 80000002400001e3
> Sep 15 04:32:29 205.254.184.12 [399661.971871][   C31] Oops: 0011 [#1] =
PREEMPT SMP
> Sep 15 04:32:29 205.254.184.12 [399661.971963][   C31] CPU: 31 PID: 0 =
Comm: swapper/31 Tainted: G        W  O       6.5.2 #1
> Sep 15 04:32:29 205.254.184.12 [399661.972079][   C31] Hardware name: =
Supermicro SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
> Sep 15 04:32:29 205.254.184.12 [399661.972197][   C31] RIP: =
0010:0xffffa10c52d43058
> Sep 15 04:32:29 205.254.184.12 [399661.972289][   C31] Code: 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 =
58 30 d4 52 0c a1 ff ff 00 00 00 00 00 00
> Sep 15 04:32:29 205.254.184.12 [399661.972448][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
> Sep 15 04:32:29 205.254.184.12 [399661.972543][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.972659][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
> Sep 15 04:32:29 205.254.184.12 [399661.972774][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
> Sep 15 04:32:29 205.254.184.12 [399661.972889][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
> Sep 15 04:32:29 205.254.184.12 [399661.973005][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
> Sep 15 04:32:29 205.254.184.12 [399661.973123][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.973244][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
> Sep 15 04:32:29 205.254.184.12 [399661.973338][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
> Sep 15 04:32:29 205.254.184.12 [399661.973454][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.973569][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Sep 15 04:32:29 205.254.184.12 [399661.973684][   C31] Call Trace:
> Sep 15 04:32:29 205.254.184.12 [399661.973773][   C31]  <IRQ>
> Sep 15 04:32:29 205.254.184.12 [399661.973859][   C31]  ? =
__die+0xe4/0xf0
> Sep 15 04:32:29 205.254.184.12 [399661.973949][   C31]  ? =
page_fault_oops+0x144/0x3e0
> Sep 15 04:32:29 205.254.184.12 [399661.974043][   C31]  ? =
exc_page_fault+0x92/0xa0
> Sep 15 04:32:29 205.254.184.12 [399661.974136][   C31]  ? =
asm_exc_page_fault+0x22/0x30
> Sep 15 04:32:29 205.254.184.12 [399661.974228][   C31]  ? =
kfree_skb_reason+0x33/0xf0
> Sep 15 04:32:29 205.254.184.12 [399661.974321][   C31]  ? =
tcp_mtu_probe+0x3a6/0x7b0
> Sep 15 04:32:29 205.254.184.12 [399661.974416][   C31]  ? =
tcp_write_xmit+0x7fa/0x1410
> Sep 15 04:32:29 205.254.184.12 [399661.974509][   C31]  ? =
__tcp_push_pending_frames+0x2d/0xb0
> Sep 15 04:32:29 205.254.184.12 [399661.974603][   C31]  ? =
tcp_rcv_established+0x381/0x610
> Sep 15 04:32:29 205.254.184.12 [399661.974695][   C31]  ? =
sk_filter_trim_cap+0xc6/0x1c0
> Sep 15 04:32:29 205.254.184.12 [399661.974787][   C31]  ? =
tcp_v4_do_rcv+0x11f/0x1f0
> Sep 15 04:32:29 205.254.184.12 [399661.974877][   C31]  ? =
tcp_v4_rcv+0xfa1/0x1010
> Sep 15 04:32:29 205.254.184.12 [399661.974968][   C31]  ? =
ip_protocol_deliver_rcu+0x1b/0x270
> Sep 15 04:32:29 205.254.184.12 [399661.975062][   C31]  ? =
ip_local_deliver_finish+0x6d/0x90
> Sep 15 04:32:29 205.254.184.12 [399661.976257][   C31]  ? =
process_backlog+0x10c/0x230
> Sep 15 04:32:29 205.254.184.12 [399661.976352][   C31]  ? =
__napi_poll+0x20/0x180
> Sep 15 04:32:29 205.254.184.12 [399661.976442][   C31]  ? =
net_rx_action+0x2a4/0x390
> Sep 15 04:32:29 205.254.184.12 [399661.976534][   C31]  ? =
__do_softirq+0xd0/0x202
> Sep 15 04:32:29 205.254.184.12 [399661.976626][   C31]  ? =
do_softirq+0x3a/0x50
> Sep 15 04:32:29 205.254.184.12 [399661.976718][   C31]  </IRQ>
> Sep 15 04:32:29 205.254.184.12 [399661.976805][   C31]  <TASK>
> Sep 15 04:32:29 205.254.184.12 [399661.976890][   C31]  ? =
flush_smp_call_function_queue+0x3f/0x50
> Sep 15 04:32:29 205.254.184.12 [399661.976988][   C31]  ? =
do_idle+0x14d/0x210
> Sep 15 04:32:29 205.254.184.12 [399661.977078][   C31]  ? =
cpu_startup_entry+0x14/0x20
> Sep 15 04:32:29 205.254.184.12 [399661.977168][   C31]  ? =
start_secondary+0xe1/0xf0
> Sep 15 04:32:29 205.254.184.12 [399661.977262][   C31]  ? =
secondary_startup_64_no_verify+0x167/0x16b
> Sep 15 04:32:29 205.254.184.12 [399661.977359][   C31]  </TASK>
> Sep 15 04:32:29 205.254.184.12 [399661.977448][   C31] Modules linked =
in: nft_limit nf_conntrack_netlink  pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos=20
> Sep 15 04:32:29 205.254.184.12 [399661.977720][   C31] CR2: =
ffffa10c52d43058
> Sep 15 04:32:29 205.254.184.12 [399661.977809][   C31] ---[ end trace =
0000000000000000 ]---
> Sep 15 04:32:29 205.254.184.12 [399661.977901][   C31] RIP: =
0010:0xffffa10c52d43058
> Sep 15 04:32:29 205.254.184.12 [399661.977992][   C31] Code: 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 =
58 30 d4 52 0c a1 ff ff 00 00 00 00 00 00
> Sep 15 04:32:29 205.254.184.12 [399661.978150][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
> Sep 15 04:32:29 205.254.184.12 [399661.978243][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.978358][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
> Sep 15 04:32:29 205.254.184.12 [399661.978472][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
> Sep 15 04:32:29 205.254.184.12 [399661.978587][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
> Sep 15 04:32:29 205.254.184.12 [399661.978702][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
> Sep 15 04:32:29 205.254.184.12 [399661.978818][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.978940][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
> Sep 15 04:32:29 205.254.184.12 [399661.979036][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
> Sep 15 04:32:29 205.254.184.12 [399661.979150][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 15 04:32:29 205.254.184.12 [399661.979265][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Sep 15 04:32:29 205.254.184.12 [399661.979381][   C31] Kernel panic - =
not syncing: Fatal exception in interrupt
> Sep 15 04:32:29 205.254.184.12 [399662.084038][   C31] Kernel Offset: =
0x1e000000 from 0xffffffff81000000 (relocation range: =
0xffffffff80000000-0xffffffffbfffffff)
> Sep 15 04:32:29 205.254.184.12 [399662.084162][   C31] Rebooting in 10 =
seconds..
>=20
>=20
> Please if find fix update me .
>=20
> m.



From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5D2D18E27
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 23:12:12 +0000 (UTC)
Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AAB3186
	for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 16:12:08 -0700 (PDT)
Received: by mail-wr1-x430.google.com with SMTP id ffacd0b85a97d-31f71b25a99so2524689f8f.2
        for <netdev@vger.kernel.org>; Fri, 15 Sep 2023 16:12:08 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1694819526; x=1695424326; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=oPbfqvYy13aoZ1ea+jFYAjstBU5hjGrzyMyws77GmUw=;
        b=KP0L+zrCboWwIdHdewwMahzo28B2GL2rNCss3UN3XtujR2jENtbPPDaov+xV5M/s0B
         sWvoRjVJWOE9j5VLxpglOcsPQV55V2ynvZc5gGtuqz7efwYKpbFNdvPr8EinuQbtzJgP
         VJSosOGIYVpQt0w/xI9t3cXhuTy7Mp7darJFyEtASC6fJsctKERDCFlUEqTlCxQ9R2zX
         tcUt9T8WBkPT2venqjNiRldRotQJa+oEjSqH0V4wiwTRfD6sVzxzXbx8wMaMG2P1v2gX
         ofaLumf2n+IwSgQ1jslEMUht94bSKuFbQOzSDEiXMF7ZptMlPjNbOcxq9oO1cfYFkbZ9
         p3Yg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694819526; x=1695424326;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=oPbfqvYy13aoZ1ea+jFYAjstBU5hjGrzyMyws77GmUw=;
        b=Ppd/Zclq4Lz2mYfR5bYDXLAx9FfQ/yFFMn4BUhyLxpKRc6QZo8k7ljDxdXmzTNgt/5
         ofw2vFH3DWnR3a956vI/Owp/TKZ9oMfU7HXlOKk4fVEhLOxY46ssSNVNu0q1BoaylqS9
         sVusXwjxfsnkbP/00B+zRuOJmk9sgPDIi7qyW/RsUCAWUK2QDiZn1UPOm5oCRrCUv2Ma
         44G6y98o9B3Qh53E6kqOVgQmVLibYNPMMGX27Q2hKl8tEYV81nx7bsO/JPG8On+5Heki
         iZYvE0f50TVxd2/3qRFnuteuB+WFSUEEI42H245t08kqrD+OlVcQHE1r5giVZGRFhlFu
         eZbQ==
X-Gm-Message-State: AOJu0YzwOf7Ad0j8AttX5sHj5wo1e/0IGvB+DWp/gyMxPf7r/OwsxXeE
	9taF701RhVW31sRKYo1RN437hwYcc+c=
X-Google-Smtp-Source: AGHT+IF4xufSQ4qvanp00plZZMMOGgVlD/DP0l23p9+nnXgNPXhEggL3zz3cBqXE2g45G3ZwyynAmQ==
X-Received: by 2002:adf:ec89:0:b0:315:8f4f:81b2 with SMTP id z9-20020adfec89000000b003158f4f81b2mr2594535wrn.64.1694819525447;
        Fri, 15 Sep 2023 16:12:05 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id z11-20020aa7d40b000000b005255f5735adsm2818904edq.24.2023.09.15.16.12.04
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 15 Sep 2023 16:12:05 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
Date: Sat, 16 Sep 2023 02:11:53 +0300
Cc: Eric Dumazet <edumazet@google.com>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
To: netdev <netdev@vger.kernel.org>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

one more log:

Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here =
]------------
Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at =
lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: nft_limit =
nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 nf_xnatlog(O) ipmi_si ipmi_devintf =
ipmi_msghandler rtc_cmos [last unloaded: BNGBOOT(O)]
Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: swapper/5 =
Tainted: G           O       6.5.2 #1
Sep 12 07:37:29  [151563.298975][    C5] Hardware name: Supermicro =
SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
Sep 12 07:37:29  [151563.299091][ C5] RIP: 0010:rcuref_put_slowpath =
(lib/rcuref.c:267 (discriminator 1))
Sep 12 07:37:29  [151563.299185][ C5] Code: 31 c0 eb e2 80 3d c7 b8 e6 =
00 00 74 0a c7 03 00 00 00 e0 31 c0 eb cf 48 c7 c7 9b f5 e2 9f c6 05 ad =
b8 e6 00 01 e8 01 7b c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc =
cc cc cc 48 89 fa 83 e2
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	31 c0                	xor    %eax,%eax
   2:	eb e2                	jmp    0xffffffffffffffe6
   4:	80 3d c7 b8 e6 00 00 	cmpb   $0x0,0xe6b8c7(%rip)        # =
0xe6b8d2
   b:	74 0a                	je     0x17
   d:	c7 03 00 00 00 e0    	movl   $0xe0000000,(%rbx)
  13:	31 c0                	xor    %eax,%eax
  15:	eb cf                	jmp    0xffffffffffffffe6
  17:	48 c7 c7 9b f5 e2 9f 	mov    $0xffffffff9fe2f59b,%rdi
  1e:	c6 05 ad b8 e6 00 01 	movb   $0x1,0xe6b8ad(%rip)        # =
0xe6b8d2
  25:	e8 01 7b c7 ff       	call   0xffffffffffc77b2b
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb df                	jmp    0xd
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	cc                   	int3
  33:	cc                   	int3
  34:	cc                   	int3
  35:	cc                   	int3
  36:	cc                   	int3
  37:	cc                   	int3
  38:	cc                   	int3
  39:	cc                   	int3
  3a:	cc                   	int3
  3b:	48 89 fa             	mov    %rdi,%rdx
  3e:	83                   	.byte 0x83
  3f:	e2                   	.byte 0xe2

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	eb df                	jmp    0xffffffffffffffe3
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	cc                   	int3
   9:	cc                   	int3
   a:	cc                   	int3
   b:	cc                   	int3
   c:	cc                   	int3
   d:	cc                   	int3
   e:	cc                   	int3
   f:	cc                   	int3
  10:	cc                   	int3
  11:	48 89 fa             	mov    %rdi,%rdx
  14:	83                   	.byte 0x83
  15:	e2                   	.byte 0xe2
Sep 12 07:37:29  [151563.299344][    C5] RSP: 0018:ffffad0e0033cde8 =
EFLAGS: 00010296
Sep 12 07:37:29  [151563.299440][    C5] RAX: 0000000000000019 RBX: =
ffffa10ba37ce100 RCX: 00000000fff7ffff
Sep 12 07:37:29  [151563.299558][    C5] RDX: 00000000fff7ffff RSI: =
0000000000000001 RDI: 00000000ffffffea
Sep 12 07:37:29  [151563.299677][    C5] RBP: ffffa10b05c76000 R08: =
0000000000000000 R09: 00000000fff7ffff
Sep 12 07:37:29  [151563.299796][    C5] R10: ffffa1125ae00000 R11: =
0000000000000003 R12: ffffa10b5f1a4ec0
Sep 12 07:37:29  [151563.299914][    C5] R13: 0000000000000000 R14: =
0000000000000258 R15: ffffad0e0033cf60
Sep 12 07:37:29  [151563.300030][    C5] FS:  0000000000000000(0000) =
GS:ffffa1125f740000(0000) knlGS:0000000000000000
Sep 12 07:37:29  [151563.300152][    C5] CS:  0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
Sep 12 07:37:29  [151563.300248][    C5] CR2: 00007fade7f56d40 CR3: =
000000010088e005 CR4: 00000000003706e0
Sep 12 07:37:29  [151563.300363][    C5] DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
Sep 12 07:37:29  [151563.300478][    C5] DR3: 0000000000000000 DR6: =
00000000fffe0ff0 DR7: 0000000000000400
Sep 12 07:37:29  [151563.300593][    C5] Call Trace:
Sep 12 07:37:29  [151563.300683][    C5]  <IRQ>
Sep 12 07:37:29  [151563.300769][ C5] ? __warn (kernel/panic.c:668)
Sep 12 07:37:29  [151563.300861][ C5] ? report_bug (lib/bug.c:223)
Sep 12 07:37:29  [151563.300952][ C5] ? handle_bug =
(arch/x86/kernel/traps.c:324)
Sep 12 07:37:29  [151563.301043][ C5] ? exc_invalid_op =
(arch/x86/kernel/traps.c:345 (discriminator 1))
Sep 12 07:37:29  [151563.301134][ C5] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
Sep 12 07:37:29  [151563.301225][ C5] ? rcuref_put_slowpath =
(lib/rcuref.c:267 (discriminator 1))
Sep 12 07:37:29  [151563.301319][ C5] ? rcuref_put_slowpath =
(lib/rcuref.c:267 (discriminator 1))
Sep 12 07:37:29  [151563.301412][ C5] dst_release =
(./include/linux/rcuref.h:151 net/core/dst.c:166)
Sep 12 07:37:29  [151563.301502][ C5] __dev_queue_xmit =
(net/core/dev.c:4138)
Sep 12 07:37:29  [151563.301595][ C5] ? eth_header =
(net/ethernet/eth.c:83)
Sep 12 07:37:29  [151563.301686][ C5] ip_finish_output2 =
(./include/net/neighbour.h:327 ./include/net/sock.h:2251 =
net/ipv4/ip_output.c:228)
Sep 12 07:37:29  [151563.301778][ C5] process_backlog =
(net/core/dev.c:5451 net/core/dev.c:5566 net/core/dev.c:5895)
Sep 12 07:37:29  [151563.301871][ C5] __napi_poll+0x20/0x180
Sep 12 07:37:29  [151563.301964][ C5] net_rx_action (net/core/dev.c:5839 =
net/core/dev.c:5860 net/core/dev.c:6684)
Sep 12 07:37:29  [151563.302057][ C5] __do_softirq =
(./arch/x86/include/asm/bitops.h:319 kernel/softirq.c:550)
Sep 12 07:37:29  [151563.302150][ C5] do_softirq (kernel/softirq.c:463 =
(discriminator 32))
Sep 12 07:37:29  [151563.302240][    C5]  </IRQ>
Sep 12 07:37:29  [151563.302326][    C5]  <TASK>
Sep 12 07:37:29  [151563.302416][ C5] flush_smp_call_function_queue =
(kernel/smp.c:563 (discriminator 1))
Sep 12 07:37:29  [151563.302518][ C5] do_idle (kernel/sched/idle.c:295)
Sep 12 07:37:29  [151563.302612][ C5] cpu_startup_entry =
(kernel/sched/idle.c:379 (discriminator 1))
Sep 12 07:37:29  [151563.302707][ C5] start_secondary =
(arch/x86/kernel/smpboot.c:326)
Sep 12 07:37:29  [151563.302805][ C5] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:441)
Sep 12 07:37:29  [151563.302900][    C5]  </TASK>
Sep 12 07:37:29  [151563.302986][    C5] ---[ end trace 0000000000000000 =
]---
Sep 15 04:32:29  [399661.971344][   C31] kernel tried to execute =
NX-protected page - exploit attempt? (uid: 0)
Sep 15 04:32:29  [399661.971470][   C31] BUG: unable to handle page =
fault for address: ffffa10c52d43058
Sep 15 04:32:29  [399661.971586][   C31] #PF: supervisor instruction =
fetch in kernel mode
Sep 15 04:32:29  [399661.971680][   C31] #PF: error_code(0x0011) - =
permissions violation
Sep 15 04:32:29  [399661.971775][   C31] PGD 12601067 P4D 12601067 PUD =
80000002400001e3
Sep 15 04:32:29  [399661.971871][   C31] Oops: 0011 [#1] PREEMPT SMP
Sep 15 04:32:29  [399661.971963][   C31] CPU: 31 PID: 0 Comm: swapper/31 =
Tainted: G        W  O       6.5.2 #1
Sep 15 04:32:29  [399661.972079][   C31] Hardware name: Supermicro =
SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
Sep 15 04:32:29  [399661.972197][   C31] RIP: 0010:0xffffa10c52d43058
Sep 15 04:32:29  [399661.972289][ C31] Code: 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 30 d4 52 0c a1 =
ff ff 00 00 00 00 00 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
	...
  30:	00 00                	add    %al,(%rax)
  32:	58                   	pop    %rax
  33:	30 d4                	xor    %dl,%ah
  35:*	52                   	push   %rdx		<-- trapping =
instruction
  36:	0c a1                	or     $0xa1,%al
  38:	ff                   	(bad)
  39:	ff 00                	incl   (%rax)
  3b:	00 00                	add    %al,(%rax)
  3d:	00 00                	add    %al,(%rax)
	...

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
	...
   8:	58                   	pop    %rax
   9:	30 d4                	xor    %dl,%ah
   b:	52                   	push   %rdx
   c:	0c a1                	or     $0xa1,%al
   e:	ff                   	(bad)
   f:	ff 00                	incl   (%rax)
  11:	00 00                	add    %al,(%rax)
  13:	00 00                	add    %al,(%rax)
	...
Sep 15 04:32:29  [399661.972448][   C31] RSP: 0018:ffffad0e0097ccc8 =
EFLAGS: 00010282
Sep 15 04:32:29  [399661.972543][   C31] RAX: ffffa10c52d43058 RBX: =
ffffa10c52d43000 RCX: 0000000000000000
Sep 15 04:32:29  [399661.972659][   C31] RDX: 0000000000002712 RSI: =
0000000000000246 RDI: ffffa10c52d43000
Sep 15 04:32:29  [399661.972774][   C31] RBP: ffffa10c52d43000 R08: =
0000000127a83c46 R09: 0000000000004d8c
Sep 15 04:32:29  [399661.972889][   C31] R10: ffffe840ca0f7c00 R11: =
0000000000000000 R12: ffffa10c8e764d80
Sep 15 04:32:29  [399661.973005][   C31] R13: ffffa10c92b4c760 R14: =
0000000000000058 R15: ffffa10c92b4c600
Sep 15 04:32:29  [399661.973123][   C31] FS:  0000000000000000(0000) =
GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
Sep 15 04:32:29  [399661.973244][   C31] CS:  0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
Sep 15 04:32:29  [399661.973338][   C31] CR2: ffffa10c52d43058 CR3: =
00000001059b8001 CR4: 00000000003706e0
Sep 15 04:32:29  [399661.973454][   C31] DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
Sep 15 04:32:29  [399661.973569][   C31] DR3: 0000000000000000 DR6: =
00000000fffe0ff0 DR7: 0000000000000400
Sep 15 04:32:29  [399661.973684][   C31] Call Trace:
Sep 15 04:32:29  [399661.973773][   C31]  <IRQ>
Sep 15 04:32:29  [399661.973859][ C31] ? __die =
(arch/x86/kernel/dumpstack.c:478 (discriminator 1) =
arch/x86/kernel/dumpstack.c:465 (discriminator 1) =
arch/x86/kernel/dumpstack.c:420 (discriminator 1) =
arch/x86/kernel/dumpstack.c:434 (discriminator 1))
Sep 15 04:32:29  [399661.973949][ C31] ? page_fault_oops =
(arch/x86/mm/fault.c:703)
Sep 15 04:32:29  [399661.974043][ C31] ? exc_page_fault =
(arch/x86/mm/fault.c:48 (discriminator 2) arch/x86/mm/fault.c:1479 =
(discriminator 2) arch/x86/mm/fault.c:1542 (discriminator 2))
Sep 15 04:32:29  [399661.974136][ C31] ? asm_exc_page_fault =
(./arch/x86/include/asm/idtentry.h:570)
Sep 15 04:32:29  [399661.974228][ C31] ? kfree_skb_reason =
(net/core/skbuff.c:1006 net/core/skbuff.c:1022 net/core/skbuff.c:1058)
Sep 15 04:32:29  [399661.974321][ C31] ? tcp_mtu_probe =
(./include/net/sock.h:1627 (discriminator 1) net/ipv4/tcp_output.c:2338 =
(discriminator 1) net/ipv4/tcp_output.c:2463 (discriminator 1))
Sep 15 04:32:29  [399661.974416][ C31] ? tcp_write_xmit =
(net/ipv4/tcp_output.c:2678)
Sep 15 04:32:29  [399661.974509][ C31] ? __tcp_push_pending_frames =
(net/ipv4/tcp_output.c:2940 (discriminator 1))
Sep 15 04:32:29  [399661.974603][ C31] ? tcp_rcv_established =
(net/ipv4/tcp_input.c:5626 net/ipv4/tcp_input.c:5620 =
net/ipv4/tcp_input.c:6066)
Sep 15 04:32:29  [399661.974695][ C31] ? sk_filter_trim_cap =
(./include/linux/rcupdate.h:781 net/core/filter.c:157)
Sep 15 04:32:29  [399661.974787][ C31] ? tcp_v4_do_rcv =
(net/ipv4/tcp_ipv4.c:1728)
Sep 15 04:32:29  [399661.974877][ C31] ? tcp_v4_rcv =
(./include/net/tcp.h:2342 (discriminator 1) net/ipv4/tcp_ipv4.c:2147 =
(discriminator 1))
Sep 15 04:32:29  [399661.974968][ C31] ? ip_protocol_deliver_rcu =
(net/ipv4/ip_input.c:205)
Sep 15 04:32:29  [399661.975062][ C31] ? ip_local_deliver_finish =
(net/ipv4/ip_input.c:233 (discriminator 1))
Sep 15 04:32:29  [399661.976257][ C31] ? process_backlog =
(net/core/dev.c:5451 net/core/dev.c:5566 net/core/dev.c:5895)
Sep 15 04:32:29  [399661.976352][ C31] ? __napi_poll+0x20/0x180
Sep 15 04:32:29  [399661.976442][ C31] ? net_rx_action =
(net/core/dev.c:5839 net/core/dev.c:5860 net/core/dev.c:6684)
Sep 15 04:32:29  [399661.976534][ C31] ? __do_softirq =
(./arch/x86/include/asm/bitops.h:319 kernel/softirq.c:550)
Sep 15 04:32:29  [399661.976626][ C31] ? do_softirq =
(kernel/softirq.c:463 (discriminator 32))
Sep 15 04:32:29  [399661.976718][   C31]  </IRQ>
Sep 15 04:32:29  [399661.976805][   C31]  <TASK>
Sep 15 04:32:29  [399661.976890][ C31] ? flush_smp_call_function_queue =
(kernel/smp.c:563 (discriminator 1))
Sep 15 04:32:29  [399661.976988][ C31] ? do_idle =
(kernel/sched/idle.c:295)
Sep 15 04:32:29  [399661.977078][ C31] ? cpu_startup_entry =
(kernel/sched/idle.c:379 (discriminator 1))
Sep 15 04:32:29  [399661.977168][ C31] ? start_secondary =
(arch/x86/kernel/smpboot.c:326)
Sep 15 04:32:29  [399661.977262][ C31] ? secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:441)
Sep 15 04:32:29  [399661.977359][   C31]  </TASK>
Sep 15 04:32:29  [399661.977448][   C31] Modules linked in: nft_limit =
nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 nf_xnatlog(O) ipmi_si ipmi_devintf =
ipmi_msghandler rtc_cmos [last unloaded: BNGBOOT(O)]
Sep 15 04:32:29  [399661.977720][   C31] CR2: ffffa10c52d43058
Sep 15 04:32:29  [399661.977809][   C31] ---[ end trace 0000000000000000 =
]---
Sep 15 04:32:29  [399661.977901][   C31] RIP: 0010:0xffffa10c52d43058
Sep 15 04:32:29  [399661.977992][ C31] Code: 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 30 d4 52 0c a1 =
ff ff 00 00 00 00 00 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
	...
  30:	00 00                	add    %al,(%rax)
  32:	58                   	pop    %rax
  33:	30 d4                	xor    %dl,%ah
  35:*	52                   	push   %rdx		<-- trapping =
instruction
  36:	0c a1                	or     $0xa1,%al
  38:	ff                   	(bad)
  39:	ff 00                	incl   (%rax)
  3b:	00 00                	add    %al,(%rax)
  3d:	00 00                	add    %al,(%rax)
	...

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
	...
   8:	58                   	pop    %rax
   9:	30 d4                	xor    %dl,%ah
   b:	52                   	push   %rdx
   c:	0c a1                	or     $0xa1,%al
   e:	ff                   	(bad)
   f:	ff 00                	incl   (%rax)
  11:	00 00                	add    %al,(%rax)
  13:	00 00                	add    %al,(%rax)
	...
Sep 15 04:32:29  [399661.978150][   C31] RSP: 0018:ffffad0e0097ccc8 =
EFLAGS: 00010282
Sep 15 04:32:29  [399661.978243][   C31] RAX: ffffa10c52d43058 RBX: =
ffffa10c52d43000 RCX: 0000000000000000
Sep 15 04:32:29  [399661.978358][   C31] RDX: 0000000000002712 RSI: =
0000000000000246 RDI: ffffa10c52d43000
Sep 15 04:32:29  [399661.978472][   C31] RBP: ffffa10c52d43000 R08: =
0000000127a83c46 R09: 0000000000004d8c
Sep 15 04:32:29  [399661.978587][   C31] R10: ffffe840ca0f7c00 R11: =
0000000000000000 R12: ffffa10c8e764d80
Sep 15 04:32:29  [399661.978702][   C31] R13: ffffa10c92b4c760 R14: =
0000000000000058 R15: ffffa10c92b4c600
Sep 15 04:32:29  [399661.978818][   C31] FS:  0000000000000000(0000) =
GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
Sep 15 04:32:29  [399661.978940][   C31] CS:  0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
Sep 15 04:32:29  [399661.979036][   C31] CR2: ffffa10c52d43058 CR3: =
00000001059b8001 CR4: 00000000003706e0
Sep 15 04:32:29  [399661.979150][   C31] DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
Sep 15 04:32:29  [399661.979265][   C31] DR3: 0000000000000000 DR6: =
00000000fffe0ff0 DR7: 0000000000000400
Sep 15 04:32:29  [399661.979381][   C31] Kernel panic - not syncing: =
Fatal exception in interrupt
Sep 15 04:32:29  [399662.084038][   C31] Kernel Offset: 0x1e000000 from =
0xffffffff81000000 (relocation range: =
0xffffffff80000000-0xffffffffbfffffff)
Sep 15 04:32:29  [399662.084162][   C31] Rebooting in 10 seconds..

> On 16 Sep 2023, at 2:00, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Ok fix=20
> one note this is kernel 6.5.3 =E2=80=A6
>=20
>=20
> see log now :=20
>=20
>=20
> [40915.530445] ------------[ cut here ]------------
> [40915.530529] rcuref - imbalanced put()
> [40915.530540] WARNING: CPU: 7 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
> [40915.530698] Modules linked in: nf_conntrack_netlink nft_limit pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
> [40915.530899] CPU: 7 PID: 0 Comm: swapper/7 Tainted: G           O    =
   6.5.3 #1
> [40915.531018] Hardware name: Supermicro SYS-5038MR-H8TRF/X10SRD-F, =
BIOS 3.3 10/28/2020
> [40915.531137] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:267 =
(discriminator 1))
> [40915.531230] Code: 31 c0 eb e2 80 3d c6 ae e6 00 00 74 0a c7 03 00 =
00 00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 8e c6 05 ac ae e6 00 01 e8 11 71 =
c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 =
e2
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 31 c0                 xor    %eax,%eax
>   2: eb e2                 jmp    0xffffffffffffffe6
>   4: 80 3d c6 ae e6 00 00 cmpb   $0x0,0xe6aec6(%rip)        # 0xe6aed1
>   b: 74 0a                 je     0x17
>   d: c7 03 00 00 00 e0     movl   $0xe0000000,(%rbx)
>  13: 31 c0                 xor    %eax,%eax
>  15: eb cf                 jmp    0xffffffffffffffe6
>  17: 48 c7 c7 68 f6 e2 8e mov    $0xffffffff8ee2f668,%rdi
>  1e: c6 05 ac ae e6 00 01 movb   $0x1,0xe6aeac(%rip)        # 0xe6aed1
>  25: e8 11 71 c7 ff        call   0xffffffffffc7713b
>  2a:* 0f 0b                 ud2     <-- trapping instruction
>  2c: eb df                 jmp    0xd
>  2e: cc                    int3
>  2f: cc                    int3
>  30: cc                    int3
>  31: cc                    int3
>  32: cc                    int3
>  33: cc                    int3
>  34: cc                    int3
>  35: cc                    int3
>  36: cc                    int3
>  37: cc                    int3
>  38: cc                    int3
>  39: cc                    int3
>  3a: cc                    int3
>  3b: 48 89 fa              mov    %rdi,%rdx
>  3e: 83                    .byte 0x83
>  3f: e2                    .byte 0xe2
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 0f 0b                 ud2
>   2: eb df                 jmp    0xffffffffffffffe3
>   4: cc                    int3
>   5: cc                    int3
>   6: cc                    int3
>   7: cc                    int3
>   8: cc                    int3
>   9: cc                    int3
>   a: cc                    int3
>   b: cc                    int3
>   c: cc                    int3
>   d: cc                    int3
>   e: cc                    int3
>   f: cc                    int3
>  10: cc                    int3
>  11: 48 89 fa              mov    %rdi,%rdx
>  14: 83                    .byte 0x83
>  15: e2                    .byte 0xe2
> [40915.531389] RSP: 0018:ffffa62680318de8 EFLAGS: 00010296
> [40915.531487] RAX: 0000000000000019 RBX: ffff982f02950c40 RCX: =
00000000fffbffff
> [40915.531605] RDX: 00000000fffbffff RSI: 0000000000000001 RDI: =
00000000ffffffea
> [40915.531721] RBP: ffff982e467d2000 R08: 0000000000000000 R09: =
00000000fffbffff
> [40915.531839] R10: ffff98359d600000 R11: 0000000000000003 R12: =
ffff982f044e16c0
> [40915.531956] R13: 0000000000000000 R14: 0000000000000258 R15: =
ffffa62680318f60
> [40915.532075] FS:  0000000000000000(0000) GS:ffff98359fbc0000(0000) =
knlGS:0000000000000000
> [40915.532195] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [40915.532291] CR2: 00005593eb3ff078 CR3: 0000000179f6e001 CR4: =
00000000003706e0
> [40915.532409] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
> [40915.532526] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
> [40915.532645] Call Trace:
> [40915.532736]  <IRQ>
> [40915.532824] ? __warn (kernel/panic.c:668)
> [40915.532918] ? report_bug (lib/bug.c:223)
> [40915.533011] ? handle_bug (arch/x86/kernel/traps.c:324)
> [40915.533104] ? exc_invalid_op (arch/x86/kernel/traps.c:345 =
(discriminator 1))
> [40915.533198] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
> [40915.533294] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
> [40915.533389] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
> [40915.533482] dst_release (./include/linux/rcuref.h:151 =
net/core/dst.c:166)
> [40915.533576] __dev_queue_xmit (net/core/dev.c:4138)
> [40915.533671] ? eth_header (net/ethernet/eth.c:83)
> [40915.533766] ip_finish_output2 (./include/net/neighbour.h:544 =
net/ipv4/ip_output.c:230)
> [40915.533863] process_backlog (net/core/dev.c:5451 =
net/core/dev.c:5566 net/core/dev.c:5895)
> [40915.533958] __napi_poll+0x20/0x180
> [40915.534050] net_rx_action (net/core/dev.c:5839 net/core/dev.c:5860 =
net/core/dev.c:6684)
> [40915.534140] __do_softirq (./arch/x86/include/asm/bitops.h:319 =
kernel/softirq.c:550)
> [40915.534233] do_softirq (kernel/softirq.c:463 (discriminator 32))
> [40915.534326]  </IRQ>
> [40915.534413]  <TASK>
> [40915.534503] flush_smp_call_function_queue (kernel/smp.c:563 =
(discriminator 1))
> [40915.534597] do_idle (kernel/sched/idle.c:295)
> [40915.534687] cpu_startup_entry (kernel/sched/idle.c:379 =
(discriminator 1))
> [40915.534778] start_secondary (arch/x86/kernel/smpboot.c:326)
> [40915.534871] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:441)
> [40915.534968]  </TASK>
> [40915.535057] ---[ end trace 0000000000000000 ]---
>=20
>> On 15 Sep 2023, at 7:05, Martin Zaharinov <micron10@gmail.com> wrote:
>>=20
>> Hi All=20
>> This is report from kernel 6.5.2 after 4 day up system hang and =
reboot after this error :
>>=20
>>=20
>>=20
>> Sep 15 04:32:29 205.254.184.12 [399661.971344][   C31] kernel tried =
to execute NX-protected page - exploit attempt? (uid: 0)
>> Sep 15 04:32:29 205.254.184.12 [399661.971470][   C31] BUG: unable to =
handle page fault for address: ffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.971586][   C31] #PF: =
supervisor instruction fetch in kernel mode
>> Sep 15 04:32:29 205.254.184.12 [399661.971680][   C31] #PF: =
error_code(0x0011) - permissions violation
>> Sep 15 04:32:29 205.254.184.12 [399661.971775][   C31] PGD 12601067 =
P4D 12601067 PUD 80000002400001e3
>> Sep 15 04:32:29 205.254.184.12 [399661.971871][   C31] Oops: 0011 =
[#1] PREEMPT SMP
>> Sep 15 04:32:29 205.254.184.12 [399661.971963][   C31] CPU: 31 PID: 0 =
Comm: swapper/31 Tainted: G        W  O       6.5.2 #1
>> Sep 15 04:32:29 205.254.184.12 [399661.972079][   C31] Hardware name: =
Supermicro SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
>> Sep 15 04:32:29 205.254.184.12 [399661.972197][   C31] RIP: =
0010:0xffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.972289][   C31] Code: 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 =
58 30 d4 52 0c a1 ff ff 00 00 00 00 00 00
>> Sep 15 04:32:29 205.254.184.12 [399661.972448][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
>> Sep 15 04:32:29 205.254.184.12 [399661.972543][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.972659][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
>> Sep 15 04:32:29 205.254.184.12 [399661.972774][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
>> Sep 15 04:32:29 205.254.184.12 [399661.972889][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
>> Sep 15 04:32:29 205.254.184.12 [399661.973005][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
>> Sep 15 04:32:29 205.254.184.12 [399661.973123][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.973244][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
>> Sep 15 04:32:29 205.254.184.12 [399661.973338][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
>> Sep 15 04:32:29 205.254.184.12 [399661.973454][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.973569][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Sep 15 04:32:29 205.254.184.12 [399661.973684][   C31] Call Trace:
>> Sep 15 04:32:29 205.254.184.12 [399661.973773][   C31]  <IRQ>
>> Sep 15 04:32:29 205.254.184.12 [399661.973859][   C31]  ? =
__die+0xe4/0xf0
>> Sep 15 04:32:29 205.254.184.12 [399661.973949][   C31]  ? =
page_fault_oops+0x144/0x3e0
>> Sep 15 04:32:29 205.254.184.12 [399661.974043][   C31]  ? =
exc_page_fault+0x92/0xa0
>> Sep 15 04:32:29 205.254.184.12 [399661.974136][   C31]  ? =
asm_exc_page_fault+0x22/0x30
>> Sep 15 04:32:29 205.254.184.12 [399661.974228][   C31]  ? =
kfree_skb_reason+0x33/0xf0
>> Sep 15 04:32:29 205.254.184.12 [399661.974321][   C31]  ? =
tcp_mtu_probe+0x3a6/0x7b0
>> Sep 15 04:32:29 205.254.184.12 [399661.974416][   C31]  ? =
tcp_write_xmit+0x7fa/0x1410
>> Sep 15 04:32:29 205.254.184.12 [399661.974509][   C31]  ? =
__tcp_push_pending_frames+0x2d/0xb0
>> Sep 15 04:32:29 205.254.184.12 [399661.974603][   C31]  ? =
tcp_rcv_established+0x381/0x610
>> Sep 15 04:32:29 205.254.184.12 [399661.974695][   C31]  ? =
sk_filter_trim_cap+0xc6/0x1c0
>> Sep 15 04:32:29 205.254.184.12 [399661.974787][   C31]  ? =
tcp_v4_do_rcv+0x11f/0x1f0
>> Sep 15 04:32:29 205.254.184.12 [399661.974877][   C31]  ? =
tcp_v4_rcv+0xfa1/0x1010
>> Sep 15 04:32:29 205.254.184.12 [399661.974968][   C31]  ? =
ip_protocol_deliver_rcu+0x1b/0x270
>> Sep 15 04:32:29 205.254.184.12 [399661.975062][   C31]  ? =
ip_local_deliver_finish+0x6d/0x90
>> Sep 15 04:32:29 205.254.184.12 [399661.976257][   C31]  ? =
process_backlog+0x10c/0x230
>> Sep 15 04:32:29 205.254.184.12 [399661.976352][   C31]  ? =
__napi_poll+0x20/0x180
>> Sep 15 04:32:29 205.254.184.12 [399661.976442][   C31]  ? =
net_rx_action+0x2a4/0x390
>> Sep 15 04:32:29 205.254.184.12 [399661.976534][   C31]  ? =
__do_softirq+0xd0/0x202
>> Sep 15 04:32:29 205.254.184.12 [399661.976626][   C31]  ? =
do_softirq+0x3a/0x50
>> Sep 15 04:32:29 205.254.184.12 [399661.976718][   C31]  </IRQ>
>> Sep 15 04:32:29 205.254.184.12 [399661.976805][   C31]  <TASK>
>> Sep 15 04:32:29 205.254.184.12 [399661.976890][   C31]  ? =
flush_smp_call_function_queue+0x3f/0x50
>> Sep 15 04:32:29 205.254.184.12 [399661.976988][   C31]  ? =
do_idle+0x14d/0x210
>> Sep 15 04:32:29 205.254.184.12 [399661.977078][   C31]  ? =
cpu_startup_entry+0x14/0x20
>> Sep 15 04:32:29 205.254.184.12 [399661.977168][   C31]  ? =
start_secondary+0xe1/0xf0
>> Sep 15 04:32:29 205.254.184.12 [399661.977262][   C31]  ? =
secondary_startup_64_no_verify+0x167/0x16b
>> Sep 15 04:32:29 205.254.184.12 [399661.977359][   C31]  </TASK>
>> Sep 15 04:32:29 205.254.184.12 [399661.977448][   C31] Modules linked =
in: nft_limit nf_conntrack_netlink  pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos=20
>> Sep 15 04:32:29 205.254.184.12 [399661.977720][   C31] CR2: =
ffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.977809][   C31] ---[ end trace =
0000000000000000 ]---
>> Sep 15 04:32:29 205.254.184.12 [399661.977901][   C31] RIP: =
0010:0xffffa10c52d43058
>> Sep 15 04:32:29 205.254.184.12 [399661.977992][   C31] Code: 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 =
58 30 d4 52 0c a1 ff ff 00 00 00 00 00 00
>> Sep 15 04:32:29 205.254.184.12 [399661.978150][   C31] RSP: =
0018:ffffad0e0097ccc8 EFLAGS: 00010282
>> Sep 15 04:32:29 205.254.184.12 [399661.978243][   C31] RAX: =
ffffa10c52d43058 RBX: ffffa10c52d43000 RCX: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.978358][   C31] RDX: =
0000000000002712 RSI: 0000000000000246 RDI: ffffa10c52d43000
>> Sep 15 04:32:29 205.254.184.12 [399661.978472][   C31] RBP: =
ffffa10c52d43000 R08: 0000000127a83c46 R09: 0000000000004d8c
>> Sep 15 04:32:29 205.254.184.12 [399661.978587][   C31] R10: =
ffffe840ca0f7c00 R11: 0000000000000000 R12: ffffa10c8e764d80
>> Sep 15 04:32:29 205.254.184.12 [399661.978702][   C31] R13: =
ffffa10c92b4c760 R14: 0000000000000058 R15: ffffa10c92b4c600
>> Sep 15 04:32:29 205.254.184.12 [399661.978818][   C31] FS:  =
0000000000000000(0000) GS:ffffa1125fdc0000(0000) knlGS:0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.978940][   C31] CS:  0010 DS: =
0000 ES: 0000 CR0: 0000000080050033
>> Sep 15 04:32:29 205.254.184.12 [399661.979036][   C31] CR2: =
ffffa10c52d43058 CR3: 00000001059b8001 CR4: 00000000003706e0
>> Sep 15 04:32:29 205.254.184.12 [399661.979150][   C31] DR0: =
0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Sep 15 04:32:29 205.254.184.12 [399661.979265][   C31] DR3: =
0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Sep 15 04:32:29 205.254.184.12 [399661.979381][   C31] Kernel panic - =
not syncing: Fatal exception in interrupt
>> Sep 15 04:32:29 205.254.184.12 [399662.084038][   C31] Kernel Offset: =
0x1e000000 from 0xffffffff81000000 (relocation range: =
0xffffffff80000000-0xffffffffbfffffff)
>> Sep 15 04:32:29 205.254.184.12 [399662.084162][   C31] Rebooting in =
10 seconds..
>>=20
>>=20
>> Please if find fix update me .
>>=20
>> m.
>=20
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB2461FA3
	for <netdev@vger.kernel.org>; Sat, 16 Sep 2023 08:27:21 +0000 (UTC)
Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124])
	by lindbergh.monkeyblade.net (Postfix) with ESMTP id DA79819AE
	for <netdev@vger.kernel.org>; Sat, 16 Sep 2023 01:27:20 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
	s=mimecast20190719; t=1694852840;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=xk07fBKmYtVdjfq/YbeUvoBCkb4c4JI3bYtw/NeUM+M=;
	b=b+KwQL+SUs2dYLPan8iQT6eIB063gvv+O15k8VgexuOJZ7AzHqApYdRrJ9lbChd4tHMn4g
	auCZJI8LvmxTIc7oTRjZ4Wx6eLszuLR7WsExF7oA0fgNhmMlYEWYPNFr7WP/s/sGPvcTqO
	knFW8Gj99z8R8bqrYDInJF4GdOALVAM=
Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com
 [209.85.218.69]) by relay.mimecast.com with ESMTP with STARTTLS
 (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id
 us-mta-477-9cbn1GZ5Oy2zDr0KLnX5QA-1; Sat, 16 Sep 2023 04:27:16 -0400
X-MC-Unique: 9cbn1GZ5Oy2zDr0KLnX5QA-1
Received: by mail-ej1-f69.google.com with SMTP id a640c23a62f3a-9adcb9ecc16so35060766b.0
        for <netdev@vger.kernel.org>; Sat, 16 Sep 2023 01:27:16 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694852835; x=1695457635;
        h=mime-version:user-agent:content-transfer-encoding:references
         :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state
         :from:to:cc:subject:date:message-id:reply-to;
        bh=xk07fBKmYtVdjfq/YbeUvoBCkb4c4JI3bYtw/NeUM+M=;
        b=a9xfl4g1FCdqDvortukjBHZ2QwZQ2c6r6EfjxPoRhqDRao2yqlbws5y81WtObq2/Al
         44b5Z9Q5RQEhZi8i6lOAhi+dvea9PsIG3389mUh4v+KPYlYQVQW5b3PjyMiOW4l2+p1j
         Cu3I1Kd5TEsodH5UDfOFXq1UE52y9FOdH/DfZSHYg5EQcn/pYrNcElJ4ZWGAI32nu1zw
         m9jjl+VZzaU7SsNVxvkfHKC4RXNbna52XxUoli7WjCFEQ9yhR9uFjjqNDS5a03V4nj7t
         kPTrXPx13VmGHH/NOjAEhwqfsdvalhovp5xXsqNPLw7I0YvkrHfOV6BWsFgNn1GpojYV
         Nb/w==
X-Gm-Message-State: AOJu0Yz7nu4mJ3burnMYlj96iVVFHOJYkRBj1W5LJX7kOZG3yEvOIy7v
	j2VgalVyCyEQTDt1MUgAWfEVEo41QSy7FaZS+sctRd+SPFOvAQma52bJIJ+PX9nSZblDJbRBMuQ
	DQPR3HVLjzkTVbQwr
X-Received: by 2002:a05:6402:40ca:b0:52f:bedf:8eea with SMTP id z10-20020a05640240ca00b0052fbedf8eeamr3139593edb.3.1694852835806;
        Sat, 16 Sep 2023 01:27:15 -0700 (PDT)
X-Google-Smtp-Source: AGHT+IHblJdE5s5EXInkymj4GKBO3QFfY17OBBsieQVpuYrD9NYx7HCuR+rTvZmmq6fLWdAEpF86Zw==
X-Received: by 2002:a05:6402:40ca:b0:52f:bedf:8eea with SMTP id z10-20020a05640240ca00b0052fbedf8eeamr3139585edb.3.1694852835508;
        Sat, 16 Sep 2023 01:27:15 -0700 (PDT)
Received: from gerbillo.redhat.com (146-241-247-149.dyn.eolo.it. [146.241.247.149])
        by smtp.gmail.com with ESMTPSA id x26-20020aa7dada000000b005224f840130sm3182970eds.60.2023.09.16.01.27.14
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Sat, 16 Sep 2023 01:27:15 -0700 (PDT)
Message-ID: <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Paolo Abeni <pabeni@redhat.com>
To: Martin Zaharinov <micron10@gmail.com>, netdev <netdev@vger.kernel.org>
Cc: Eric Dumazet <edumazet@google.com>, patchwork-bot+netdevbpf@kernel.org, 
	Jakub Kicinski
	 <kuba@kernel.org>, Stephen Hemminger <stephen@networkplumber.org>, 
	kuba+netdrv@kernel.org, dsahern@gmail.com
Date: Sat, 16 Sep 2023 10:27:13 +0200
In-Reply-To: <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
	 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
	 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
User-Agent: Evolution 3.46.4 (3.46.4-1.fc37) 
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,
	RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
> one more log:
>=20
> Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here ]--------=
----
> Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
> Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at lib/rcure=
f.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
> Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: nft_limit nf_=
conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc nft_ct nft_nat n=
ft_chain_nat nf_tables netconsole coretemp bonding i40e nf_nat_sip nf_connt=
rack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat=
_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_=
xnatlog(O) ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos [last unloaded: BN=
GBOOT(O)]
> Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: swapper/5 Ta=
inted: G           O       6.5.2 #1


You have out-of-tree modules taint in all the report you shared. Please
try to reproduce the issue with such taint, thanks!

Paolo


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76AB81877
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 11:35:32 +0000 (UTC)
Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 491D9113
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 04:35:29 -0700 (PDT)
Received: by mail-lf1-x12e.google.com with SMTP id 2adb3069b0e04-50317080342so109863e87.2
        for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 04:35:29 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1694950527; x=1695555327; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=Qfp5FTzbalVn97S6kK+aoO9oj6JX/14dKJTbxZ8JqII=;
        b=CFjAQN1ZoeWoQx05EqGVAuvpzC0SPTgGrkxaoZX6wLaXp2Rs4fXWWYxiobwdxHmHhm
         TxGH6aik8zwd47CJB6Jazo0+/UYXxB/q3h2KwJWNY8u6sH8LXo4JBf7AJPd6Lt00YEJJ
         rxh+PvL3o1rEVcPU00LEdzsvqSrM9maU40ZWPTvimJ5yOSpl8iStNiLRA89FCnr7yKCA
         MQMaf58Zt2RkcsRwkW2WQNFMWvicdOdcTp8qNwJUVZozB63sv5nzDML0nUNyfhLmaVUg
         MrOzxmkFQuNUkNxZc36xfMJEj0g4qdBbdQHI2ATtGVlkVPJoxA+UDHZApJTidJzODSNH
         N1lA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694950527; x=1695555327;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=Qfp5FTzbalVn97S6kK+aoO9oj6JX/14dKJTbxZ8JqII=;
        b=vT1zhdoSmuoZ9qYxHOy/RWKy+j2nopuVpsTiF5l4pgCKYutSMFuqE6BrlQa2Ak+3jv
         dvAs2HIKYSRqVwOVV71NMKNacKhbH1zDrQXbjPOII/8WHukd/WocvgX5sP3ebaBf4AtJ
         etxe+OsZz2SzWmiZxLSUefZtKS1JQXEWeVWBaJLHX1XV0hVOOIUvFMkF0OrlJsQtwUXu
         yrOFE0YV7n6E9I8xuRHKNPAiDuB79t8/N8YAfFWuwxlJMELNSQeEuEmUi6fxgW45d/c2
         HsTzjdyaKRW/X6Gf7bOEGxYjPq5Vk2eHa+ZXANbAke0BuEX/oT3p3OdRcCAD1EbhbZPP
         VuIw==
X-Gm-Message-State: AOJu0YwZ/U04UT6+KkxOdX8OVU9Mxq3lVw1PPx307+7GqSTJoCeZ5DlA
	lEOCDE92QWFBJl1/Z03Lg7g=
X-Google-Smtp-Source: AGHT+IEEG9KlPoqYLPVjYY9j7vHpCWDx3cfAVxGUDTVCMVJjEjt3c5FERGHtOKx3moYyKMQbXRsCIQ==
X-Received: by 2002:a05:6512:31cd:b0:503:1722:bf3a with SMTP id j13-20020a05651231cd00b005031722bf3amr236225lfe.1.1694950527069;
        Sun, 17 Sep 2023 04:35:27 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id y20-20020a17090668d400b009a5f7fb51d1sm4851710ejr.40.2023.09.17.04.35.25
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Sun, 17 Sep 2023 04:35:26 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
Date: Sun, 17 Sep 2023 14:35:15 +0300
Cc: netdev <netdev@vger.kernel.org>,
 Eric Dumazet <edumazet@google.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <B07167DB-6AAF-48DF-8627-9E977FD9F635@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
To: Paolo Abeni <pabeni@redhat.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi Paolo and Eric

See this is latest crash from kernel 6.5.3 without external moduls=E2=80=A6=
.


first is crash report , second is with decode:=20


Sep 17 11:43:11  [127675.391688][    C2] kernel tried to execute =
NX-protected page - exploit attempt? (uid: 0)
Sep 17 11:43:11  [127675.391780][    C2] BUG: unable to handle page =
fault for address: ffff9bd9ff20f858
Sep 17 11:43:11  [127675.391859][    C2] #PF: supervisor instruction =
fetch in kernel mode
Sep 17 11:43:11  [127675.391937][    C2] #PF: error_code(0x0011) - =
permissions violation
Sep 17 11:43:11  [127675.392014][    C2] PGD 1a601067 P4D 1a601067 PUD =
147b05063 PMD 800000023f2001e3
Sep 17 11:43:11  [127675.392099][    C2] Oops: 0011 [#1] PREEMPT SMP
Sep 17 11:43:11  [127675.392173][    C2] CPU: 2 PID: 0 Comm: swapper/2 =
Tainted: G           O       6.5.3 #1
Sep 17 11:43:11  [127675.392257][    C2] Hardware name: Supermicro =
SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
Sep 17 11:43:11  [127675.392338][    C2] RIP: 0010:0xffff9bd9ff20f858
Sep 17 11:43:11  [127675.392413][    C2] Code: 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 f8 20 ff d9 =
9b ff ff 00 00 00 00 00 00
Sep 17 11:43:11  [127675.392540][    C2] RSP: 0018:ffffadfe0007ccc8 =
EFLAGS: 00010282
Sep 17 11:43:11  [127675.392635][    C2] RAX: ffff9bd9ff20f858 RBX: =
ffff9bd9ff20f800 RCX: 0000000000000000
Sep 17 11:43:11  [127675.392753][    C2] RDX: 0000000000002711 RSI: =
0000000000000246 RDI: ffff9bd9ff20f800
Sep 17 11:43:11  [127675.392871][    C2] RBP: ffff9bd9ff20f800 R08: =
000000010ca6060f R09: 00000000000079f2
Sep 17 11:43:11  [127675.392988][    C2] R10: ffffd88b47077c00 R11: =
0000000000000000 R12: ffff9bd9bb6ca1c0
Sep 17 11:43:11  [127675.393107][    C2] R13: ffff9bd9b9013760 R14: =
0000000000000053 R15: ffff9bd9b9013600
Sep 17 11:43:11  [127675.393226][    C2] FS:  0000000000000000(0000) =
GS:ffff9be01f680000(0000) knlGS:0000000000000000
Sep 17 11:43:11  [127675.393347][    C2] CS:  0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
Sep 17 11:43:11  [127675.393445][    C2] CR2: ffff9bd9ff20f858 CR3: =
0000000234668001 CR4: 00000000003706e0
Sep 17 11:43:11  [127675.393562][    C2] DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
Sep 17 11:43:11  [127675.393677][    C2] DR3: 0000000000000000 DR6: =
00000000fffe0ff0 DR7: 0000000000000400
Sep 17 11:43:11  [127675.393794][    C2] Call Trace:
Sep 17 11:43:11  [127675.393886][    C2]  <IRQ>
Sep 17 11:43:11  [127675.393973][    C2]  ? __die+0xe4/0xf0
Sep 17 11:43:11  [127675.394065][    C2]  ? page_fault_oops+0x144/0x3e0
Sep 17 11:43:11  [127675.394157][    C2]  ? exc_page_fault+0x92/0xa0
Sep 17 11:43:11  [127675.394251][    C2]  ? asm_exc_page_fault+0x22/0x30
Sep 17 11:43:11  [127675.394347][    C2]  ? kfree_skb_reason+0x33/0xf0
Sep 17 11:43:11  [127675.394443][    C2]  ? tcp_mtu_probe+0x3a6/0x7b0
Sep 17 11:43:11  [127675.394539][    C2]  ? tcp_write_xmit+0x7fa/0x1410
Sep 17 11:43:11  [127675.394634][    C2]  ? =
__tcp_push_pending_frames+0x2d/0xb0
Sep 17 11:43:11  [127675.394727][    C2]  ? =
tcp_rcv_established+0x205/0x610
Sep 17 11:43:11  [127675.394822][    C2]  ? =
sk_filter_trim_cap+0xc6/0x1c0
Sep 17 11:43:11  [127675.394914][    C2]  ? tcp_v4_do_rcv+0x11f/0x1f0
Sep 17 11:43:11  [127675.395007][    C2]  ? tcp_v4_rcv+0xfa1/0x1010
Sep 17 11:43:11  [127675.395100][    C2]  ? =
ip_protocol_deliver_rcu+0x1b/0x270
Sep 17 11:43:11  [127675.395196][    C2]  ? =
ip_local_deliver_finish+0x6d/0x90
Sep 17 11:43:11  [127675.395289][    C2]  ? process_backlog+0x10c/0x230
Sep 17 11:43:11  [127675.395386][    C2]  ? __napi_poll+0x20/0x180
Sep 17 11:43:11  [127675.395478][    C2]  ? net_rx_action+0x2a4/0x390
Sep 17 11:43:11  [127675.395572][    C2]  ? __do_softirq+0xd0/0x202
Sep 17 11:43:11  [127675.395666][    C2]  ? do_softirq+0x3a/0x50
Sep 17 11:43:11  [127675.395760][    C2]  </IRQ>
Sep 17 11:43:11  [127675.395849][    C2]  <TASK>
Sep 17 11:43:11  [127675.395939][    C2]  ? =
flush_smp_call_function_queue+0x3f/0x50
Sep 17 11:43:11  [127675.396039][    C2]  ? do_idle+0x14d/0x210
Sep 17 11:43:11  [127675.396132][    C2]  ? cpu_startup_entry+0x14/0x20
Sep 17 11:43:11  [127675.396224][    C2]  ? start_secondary+0xe1/0xf0
Sep 17 11:43:11  [127675.396318][    C2]  ? =
secondary_startup_64_no_verify+0x167/0x16b
Sep 17 11:43:11  [127675.396417][    C2]  </TASK>
Sep 17 11:43:11  [127675.396504][    C2] Modules linked in: =
nf_conntrack_netlink nft_limit pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos
Sep 17 11:43:11  [127675.396775][    C2] CR2: ffff9bd9ff20f858
Sep 17 11:43:11  [127675.396868][    C2] ---[ end trace 0000000000000000 =
]---
Sep 17 11:43:11  [127675.396961][    C2] RIP: 0010:0xffff9bd9ff20f858
Sep 17 11:43:11  [127675.397052][    C2] Code: 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 f8 20 ff d9 =
9b ff ff 00 00 00 00 00 00
Sep 17 11:43:11  [127675.397211][    C2] RSP: 0018:ffffadfe0007ccc8 =
EFLAGS: 00010282
Sep 17 11:43:11  [127675.397305][    C2] RAX: ffff9bd9ff20f858 RBX: =
ffff9bd9ff20f800 RCX: 0000000000000000
Sep 17 11:43:11  [127675.397419][    C2] RDX: 0000000000002711 RSI: =
0000000000000246 RDI: ffff9bd9ff20f800
Sep 17 11:43:11  [127675.397535][    C2] RBP: ffff9bd9ff20f800 R08: =
000000010ca6060f R09: 00000000000079f2
Sep 17 11:43:11  [127675.397651][    C2] R10: ffffd88b47077c00 R11: =
0000000000000000 R12: ffff9bd9bb6ca1c0
Sep 17 11:43:11  [127675.397767][    C2] R13: ffff9bd9b9013760 R14: =
0000000000000053 R15: ffff9bd9b9013600
Sep 17 11:43:11  [127675.397886][    C2] FS:  0000000000000000(0000) =
GS:ffff9be01f680000(0000) knlGS:0000000000000000
Sep 17 11:43:11  [127675.398006][    C2] CS:  0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
Sep 17 11:43:11  [127675.398101][    C2] CR2: ffff9bd9ff20f858 CR3: =
0000000234668001 CR4: 00000000003706e0
Sep 17 11:43:11  [127675.398217][    C2] DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
Sep 17 11:43:11  [127675.398334][    C2] DR3: 0000000000000000 DR6: =
00000000fffe0ff0 DR7: 0000000000000400
Sep 17 11:43:11  [127675.398451][    C2] Kernel panic - not syncing: =
Fatal exception in interrupt
Sep 17 11:43:11  [127675.503611][    C2] Kernel Offset: 0x20000000 from =
0xffffffff81000000 (relocation range: =
0xffffffff80000000-0xffffffffbfffffff)
Sep 17 11:43:11  [127675.503734][    C2] Rebooting in 10 seconds..





Second with decode:=20



Sep 17 11:43:11  [127675.391688][    C2] kernel tried to execute =
NX-protected page - exploit attempt? (uid: 0)
Sep 17 11:43:11  [127675.391780][    C2] BUG: unable to handle page =
fault for address: ffff9bd9ff20f858
Sep 17 11:43:11  [127675.391859][    C2] #PF: supervisor instruction =
fetch in kernel mode
Sep 17 11:43:11  [127675.391937][    C2] #PF: error_code(0x0011) - =
permissions violation
Sep 17 11:43:11  [127675.392014][    C2] PGD 1a601067 P4D 1a601067 PUD =
147b05063 PMD 800000023f2001e3
Sep 17 11:43:11  [127675.392099][    C2] Oops: 0011 [#1] PREEMPT SMP
Sep 17 11:43:11  [127675.392173][    C2] CPU: 2 PID: 0 Comm: swapper/2 =
Tainted: G           O       6.5.3 #1
Sep 17 11:43:11  [127675.392257][    C2] Hardware name: Supermicro =
SYS-5038MR-H8TRF/X10SRD-F, BIOS 3.3 10/28/2020
Sep 17 11:43:11  [127675.392338][    C2] RIP: 0010:0xffff9bd9ff20f858
Sep 17 11:43:11 [127675.392413][ C2] Code: 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 f8 20 ff d9 9b ff =
ff 00 00 00 00 00 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
	...
  30:	00 00                	add    %al,(%rax)
  32:	58                   	pop    %rax
  33:*	f8                   	clc    		<-- trapping instruction
  34:	20 ff                	and    %bh,%bh
  36:	d9 9b ff ff 00 00    	fstps  0xffff(%rbx)
  3c:	00 00                	add    %al,(%rax)
	...

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
	...
   8:	58                   	pop    %rax
   9:	f8                   	clc
   a:	20 ff                	and    %bh,%bh
   c:	d9 9b ff ff 00 00    	fstps  0xffff(%rbx)
  12:	00 00                	add    %al,(%rax)
	...
Sep 17 11:43:11  [127675.392540][    C2] RSP: 0018:ffffadfe0007ccc8 =
EFLAGS: 00010282
Sep 17 11:43:11  [127675.392635][    C2] RAX: ffff9bd9ff20f858 RBX: =
ffff9bd9ff20f800 RCX: 0000000000000000
Sep 17 11:43:11  [127675.392753][    C2] RDX: 0000000000002711 RSI: =
0000000000000246 RDI: ffff9bd9ff20f800
Sep 17 11:43:11  [127675.392871][    C2] RBP: ffff9bd9ff20f800 R08: =
000000010ca6060f R09: 00000000000079f2
Sep 17 11:43:11  [127675.392988][    C2] R10: ffffd88b47077c00 R11: =
0000000000000000 R12: ffff9bd9bb6ca1c0
Sep 17 11:43:11  [127675.393107][    C2] R13: ffff9bd9b9013760 R14: =
0000000000000053 R15: ffff9bd9b9013600
Sep 17 11:43:11  [127675.393226][    C2] FS:  0000000000000000(0000) =
GS:ffff9be01f680000(0000) knlGS:0000000000000000
Sep 17 11:43:11  [127675.393347][    C2] CS:  0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
Sep 17 11:43:11  [127675.393445][    C2] CR2: ffff9bd9ff20f858 CR3: =
0000000234668001 CR4: 00000000003706e0
Sep 17 11:43:11  [127675.393562][    C2] DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
Sep 17 11:43:11  [127675.393677][    C2] DR3: 0000000000000000 DR6: =
00000000fffe0ff0 DR7: 0000000000000400
Sep 17 11:43:11  [127675.393794][    C2] Call Trace:
Sep 17 11:43:11  [127675.393886][    C2]  <IRQ>
Sep 17 11:43:11 [127675.393973][ C2] ? __die =
(arch/x86/kernel/dumpstack.c:478 (discriminator 1) =
arch/x86/kernel/dumpstack.c:465 (discriminator 1) =
arch/x86/kernel/dumpstack.c:420 (discriminator 1) =
arch/x86/kernel/dumpstack.c:434 (discriminator 1))
Sep 17 11:43:11 [127675.394065][ C2] ? page_fault_oops =
(arch/x86/mm/fault.c:703)
Sep 17 11:43:11 [127675.394157][ C2] ? exc_page_fault =
(arch/x86/mm/fault.c:48 (discriminator 2) arch/x86/mm/fault.c:1479 =
(discriminator 2) arch/x86/mm/fault.c:1542 (discriminator 2))
Sep 17 11:43:11 [127675.394251][ C2] ? asm_exc_page_fault =
(./arch/x86/include/asm/idtentry.h:570)
Sep 17 11:43:11 [127675.394347][ C2] ? kfree_skb_reason =
(net/core/skbuff.c:1006 net/core/skbuff.c:1022 net/core/skbuff.c:1058)
Sep 17 11:43:11 [127675.394443][ C2] ? tcp_mtu_probe =
(./include/net/sock.h:1627 (discriminator 1) net/ipv4/tcp_output.c:2338 =
(discriminator 1) net/ipv4/tcp_output.c:2463 (discriminator 1))
Sep 17 11:43:11 [127675.394539][ C2] ? tcp_write_xmit =
(net/ipv4/tcp_output.c:2678)
Sep 17 11:43:11 [127675.394634][ C2] ? __tcp_push_pending_frames =
(net/ipv4/tcp_output.c:2940 (discriminator 1))
Sep 17 11:43:11 [127675.394727][ C2] ? tcp_rcv_established =
(./include/net/tcp.h:2033 net/ipv4/tcp_input.c:5545 =
net/ipv4/tcp_input.c:6065)
Sep 17 11:43:11 [127675.394822][ C2] ? sk_filter_trim_cap =
(./include/linux/rcupdate.h:781 net/core/filter.c:157)
Sep 17 11:43:11 [127675.394914][ C2] ? tcp_v4_do_rcv =
(net/ipv4/tcp_ipv4.c:1728)
Sep 17 11:43:11 [127675.395007][ C2] ? tcp_v4_rcv =
(./include/net/tcp.h:2342 (discriminator 1) net/ipv4/tcp_ipv4.c:2147 =
(discriminator 1))
Sep 17 11:43:11 [127675.395100][ C2] ? ip_protocol_deliver_rcu =
(net/ipv4/ip_input.c:205)
Sep 17 11:43:11 [127675.395196][ C2] ? ip_local_deliver_finish =
(net/ipv4/ip_input.c:233 (discriminator 1))
Sep 17 11:43:11 [127675.395289][ C2] ? process_backlog =
(net/core/dev.c:5451 net/core/dev.c:5566 net/core/dev.c:5895)
Sep 17 11:43:11 [127675.395386][ C2] ? __napi_poll+0x20/0x180
Sep 17 11:43:11 [127675.395478][ C2] ? net_rx_action =
(net/core/dev.c:5839 net/core/dev.c:5860 net/core/dev.c:6684)
Sep 17 11:43:11 [127675.395572][ C2] ? __do_softirq =
(./arch/x86/include/asm/bitops.h:319 kernel/softirq.c:550)
Sep 17 11:43:11 [127675.395666][ C2] ? do_softirq (kernel/softirq.c:463 =
(discriminator 32))
Sep 17 11:43:11  [127675.395760][    C2]  </IRQ>
Sep 17 11:43:11  [127675.395849][    C2]  <TASK>
Sep 17 11:43:11 [127675.395939][ C2] ? flush_smp_call_function_queue =
(kernel/smp.c:563 (discriminator 1))
Sep 17 11:43:11 [127675.396039][ C2] ? do_idle (kernel/sched/idle.c:295)
Sep 17 11:43:11 [127675.396132][ C2] ? cpu_startup_entry =
(kernel/sched/idle.c:379 (discriminator 1))
Sep 17 11:43:11 [127675.396224][ C2] ? start_secondary =
(arch/x86/kernel/smpboot.c:326)
Sep 17 11:43:11 [127675.396318][ C2] ? secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:441)
Sep 17 11:43:11  [127675.396417][    C2]  </TASK>
Sep 17 11:43:11  [127675.396504][    C2] Modules linked in: =
nf_conntrack_netlink nft_limit pppoe pppox ppp_generic slhc nft_ct =
nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos
Sep 17 11:43:11  [127675.396775][    C2] CR2: ffff9bd9ff20f858
Sep 17 11:43:11  [127675.396868][    C2] ---[ end trace 0000000000000000 =
]---
Sep 17 11:43:11  [127675.396961][    C2] RIP: 0010:0xffff9bd9ff20f858
Sep 17 11:43:11 [127675.397052][ C2] Code: 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =
00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 58 f8 20 ff d9 9b ff =
ff 00 00 00 00 00 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
	...
  30:	00 00                	add    %al,(%rax)
  32:	58                   	pop    %rax
  33:*	f8                   	clc    		<-- trapping instruction
  34:	20 ff                	and    %bh,%bh
  36:	d9 9b ff ff 00 00    	fstps  0xffff(%rbx)
  3c:	00 00                	add    %al,(%rax)
	...

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
	...
   8:	58                   	pop    %rax
   9:	f8                   	clc
   a:	20 ff                	and    %bh,%bh
   c:	d9 9b ff ff 00 00    	fstps  0xffff(%rbx)
  12:	00 00                	add    %al,(%rax)
	...
Sep 17 11:43:11  [127675.397211][    C2] RSP: 0018:ffffadfe0007ccc8 =
EFLAGS: 00010282
Sep 17 11:43:11  [127675.397305][    C2] RAX: ffff9bd9ff20f858 RBX: =
ffff9bd9ff20f800 RCX: 0000000000000000
Sep 17 11:43:11  [127675.397419][    C2] RDX: 0000000000002711 RSI: =
0000000000000246 RDI: ffff9bd9ff20f800
Sep 17 11:43:11  [127675.397535][    C2] RBP: ffff9bd9ff20f800 R08: =
000000010ca6060f R09: 00000000000079f2
Sep 17 11:43:11  [127675.397651][    C2] R10: ffffd88b47077c00 R11: =
0000000000000000 R12: ffff9bd9bb6ca1c0
Sep 17 11:43:11  [127675.397767][    C2] R13: ffff9bd9b9013760 R14: =
0000000000000053 R15: ffff9bd9b9013600
Sep 17 11:43:11  [127675.397886][    C2] FS:  0000000000000000(0000) =
GS:ffff9be01f680000(0000) knlGS:0000000000000000
Sep 17 11:43:11  [127675.398006][    C2] CS:  0010 DS: 0000 ES: 0000 =
CR0: 0000000080050033
Sep 17 11:43:11  [127675.398101][    C2] CR2: ffff9bd9ff20f858 CR3: =
0000000234668001 CR4: 00000000003706e0
Sep 17 11:43:11  [127675.398217][    C2] DR0: 0000000000000000 DR1: =
0000000000000000 DR2: 0000000000000000
Sep 17 11:43:11  [127675.398334][    C2] DR3: 0000000000000000 DR6: =
00000000fffe0ff0 DR7: 0000000000000400
Sep 17 11:43:11  [127675.398451][    C2] Kernel panic - not syncing: =
Fatal exception in interrupt
Sep 17 11:43:11  [127675.503611][    C2] Kernel Offset: 0x20000000 from =
0xffffffff81000000 (relocation range: =
0xffffffff80000000-0xffffffffbfffffff)
Sep 17 11:43:11  [127675.503734][    C2] Rebooting in 10 seconds..



P.S.

upload kernel on 5 machine with diff hw and make same on every one .


Best regrads,
m.

> On 16 Sep 2023, at 12:04, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi Paolo
>=20
> in first report machine dont have out of tree module
>=20
> this bug is come after move from kernel 6.2 to 6.3
>=20
> m.
>=20
> On Sat, Sep 16, 2023, 11:27 Paolo Abeni <pabeni@redhat.com> wrote:
> On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
> > one more log:
> >=20
> > Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here =
]------------
> > Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
> > Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at =
lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
> > Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: =
nft_limit nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc =
nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 nf_xnatlog(O) ipmi_si ipmi_devintf =
ipmi_msghandler rtc_cmos [last unloaded: BNGBOOT(O)]
> > Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: =
swapper/5 Tainted: G           O       6.5.2 #1
>=20
>=20
> You have out-of-tree modules taint in all the report you shared. =
Please
> try to reproduce the issue with such taint, thanks!
>=20
> Paolo
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3D241877
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 11:40:54 +0000 (UTC)
Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 873D1120
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 04:40:51 -0700 (PDT)
Received: by mail-ed1-x536.google.com with SMTP id 4fb4d7f45d1cf-52683b68c2fso4296118a12.0
        for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 04:40:51 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1694950850; x=1695555650; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=vr/Pw2eP+hXtNULq43VQt5WuV8h9UDltzQ9I4y7geKE=;
        b=Ki14abauKgVDI3sqAbzp0y3QIlNvq9Hcep18tTmK3T8KoTlH11ca9PAnCjhn6JaDw1
         ijb9sqwQyguYbWxXyp8hzlyDaLO+M2ZsKROJG13BXGupSu3QqmXG8hCrJJOLdU0SyG84
         s1gSp4EVEoJgp3jPgXPgrFucagxwLL/Cu5OnNLJsvgZVMBScfMOtulWmXIidCpHp2hZS
         +u3+BegHJ0i2nSUR7QMClG55bKVJGg+95npTGzUh3FmXv/VjBweMhEMhUYtKVeRP4Uux
         vVmuZZIVh0l24KE1BV0pWxOStm2TaDN3aKZmjYOpfgVNIKq/tZEqrtNmM7tX4f6bI6ma
         c4dA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694950850; x=1695555650;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=vr/Pw2eP+hXtNULq43VQt5WuV8h9UDltzQ9I4y7geKE=;
        b=YHbN2Fgjod6YZ+crhEwg3baI4VPth4jcz9sgDdx+30THKKj2TNsQA4lV5Mw/MfbD6w
         vVw+PYguxJD848bJpxEQydYMYHXBO0bvBJL1A5CWWJMZc3C8xRbnv9VFfgfNpcULTEdY
         8ZfSmcergKLJXUfNhFL+FLVPT5hu6Xps/opgr23aFMzi+2XAVrN2VQhkeotKtMlcbzi6
         FDLuSyAY+e4xIqa1BFFpkLPm+mTHzSAZLPpB9pnJcjDHuDQFrUmOX41yn0mmDK2k9qye
         BOnoPjbwqZa5IjveY3dzyhp8Wzamd6pN9EeYXRdZfD2XG1RNbXnyq9a9OKkXS+P2NAaK
         2mZQ==
X-Gm-Message-State: AOJu0YyPTZXGONDrFkZxBmXZCj4MxGYFqyjcwZJfz7M8BGfg2nYj8xkJ
	mE5JmSCVfExZ3TGEKejf3O+G8Wea10s=
X-Google-Smtp-Source: AGHT+IE5WzMeN4WWSKHlNVBiEi86ST1QVHV4A10pTxXitOM+zh8CPcmppibnbnxDroLa0TNkYwby+w==
X-Received: by 2002:a05:6402:207a:b0:530:e090:1370 with SMTP id bd26-20020a056402207a00b00530e0901370mr1955676edb.35.1694950849678;
        Sun, 17 Sep 2023 04:40:49 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id w21-20020aa7cb55000000b0052ff7b17d38sm4540781edt.63.2023.09.17.04.40.48
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Sun, 17 Sep 2023 04:40:49 -0700 (PDT)
Content-Type: text/plain;
	charset=us-ascii
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
Date: Sun, 17 Sep 2023 14:40:38 +0300
Cc: netdev <netdev@vger.kernel.org>,
 Eric Dumazet <edumazet@google.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Florian Westphal <fw@strlen.de>,
 Paolo Abeni <pabeni@redhat.com>,
 Pablo Neira Ayuso <pablo@netfilter.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
To: Paolo Abeni <pabeni@redhat.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

One more in changelog for kernel 6.5 : =
https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.5

I see have many bug reports with :=20

Sep 17 11:43:11  [127675.395289][    C2]  ? process_backlog+0x10c/0x230
Sep 17 11:43:11  [127675.395386][    C2]  ? __napi_poll+0x20/0x180
Sep 17 11:43:11  [127675.395478][    C2]  ? net_rx_action+0x2a4/0x390


In all server have simple nftables rulls , ethernet card is intel xl710 =
or 82599. its a very simple config.

m.=09




> On 16 Sep 2023, at 12:04, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi Paolo
>=20
> in first report machine dont have out of tree module
>=20
> this bug is come after move from kernel 6.2 to 6.3
>=20
> m.
>=20
> On Sat, Sep 16, 2023, 11:27 Paolo Abeni <pabeni@redhat.com> wrote:
> On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
> > one more log:
> >=20
> > Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here =
]------------
> > Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
> > Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at =
lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
> > Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: =
nft_limit nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc =
nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 nf_xnatlog(O) ipmi_si ipmi_devintf =
ipmi_msghandler rtc_cmos [last unloaded: BNGBOOT(O)]
> > Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: =
swapper/5 Tainted: G           O       6.5.2 #1
>=20
>=20
> You have out-of-tree modules taint in all the report you shared. =
Please
> try to reproduce the issue with such taint, thanks!
>=20
> Paolo
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84383290B
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 11:55:40 +0000 (UTC)
Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1EF9A126
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 04:55:39 -0700 (PDT)
Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-9a645e54806so439913066b.0
        for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 04:55:39 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1694951737; x=1695556537; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=mGDVdE8MBVyhDYJiGJCU+gdHXhqBgu97DlCVywOnJKA=;
        b=kI8k3VJ63ijsbgb9Tkj64uKBKEd0dCvZZrLSo09nEiAxaVNITmdLWWd0c7AHNBAiXV
         h8w4pk1cXj7cwLBJW4oV9n1eICNFSz80PLsMMSNcV+ZmikXqlMilG0zQfrbNoshLzQgU
         QhoEdCX0UchXxCP8ErR3gLaZi4/KfdImSMpWVort+mxWpwgybTDxz5L9ghvb7GJ/ekLy
         /Ao6IofeLu42DVc0kWoBl4WlPxhXTIcgPfe2GkGtFOm4Mx+v2HNl52YN18YwovFvQc0y
         gdDa8h7zQJzTsP1x3KEg9aGrQgkkKMmR79vmQSfaCi4xhS+TUOGuqZwL6ijy0gt6q3kI
         YmwQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1694951737; x=1695556537;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=mGDVdE8MBVyhDYJiGJCU+gdHXhqBgu97DlCVywOnJKA=;
        b=MYeUDVGjLbYHn0/3fnv3gSyyYyx+/eyx0gGjTYgyO1V1pHHJ9gVeDRylMfn0IlhIHo
         fKrG+75e9FGW9oXd009e9FrP504g/UZxRmEhCtN+sVpsdRVFCV1OvSITYSf5g9pZHcxU
         L59ceRV6b+r7Bj9RMQbgrCjK0SylZHN0e27Gxni4Hm+NmOCGAN1qwPs7xwIy85PuNhgJ
         C8fFAxo723ayiq4YOHGaeCZ6wd1D5735MP7bJGEM3KXiVIGy5rZo4rZm86l7fTDw7jyv
         4Y3GfYN6tP1hg0jNtxWd+6n2PAQgLtcBWr5HYKofEsAoFTSSpDbAYRB+YFXXdTPyT1rD
         5FqQ==
X-Gm-Message-State: AOJu0YzRLDWEYgLoe+df4w3o4MT9QiQqy4RbIF+40i06c+ll2LhoRlmX
	JTmE/PW6gl+bq4m4/sUOQdNgqv6XHvg=
X-Google-Smtp-Source: AGHT+IHdJq8/FghWABbWDvAU0fJzeg+4D47ll+fRxGwRO0RFXspyVGAPqazd/3BT+2ECCtmNIARArA==
X-Received: by 2002:a17:906:3158:b0:9ad:c763:c3fd with SMTP id e24-20020a170906315800b009adc763c3fdmr5331685eje.28.1694951737475;
        Sun, 17 Sep 2023 04:55:37 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id y13-20020a170906470d00b0098884f86e41sm4831164ejq.123.2023.09.17.04.55.36
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Sun, 17 Sep 2023 04:55:37 -0700 (PDT)
Content-Type: text/plain;
	charset=us-ascii
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
Date: Sun, 17 Sep 2023 14:55:25 +0300
Cc: netdev <netdev@vger.kernel.org>,
 Eric Dumazet <edumazet@google.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
To: Paolo Abeni <pabeni@redhat.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi Eric
is it possible bug to come from this patch : =
https://patchwork.kernel.org/project/netdevbpf/cover/20230911170531.828100=
-1-edumazet@google.com/=20


m.

> On 17 Sep 2023, at 14:40, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> One more in changelog for kernel 6.5 : =
https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.5
>=20
> I see have many bug reports with :=20
>=20
> Sep 17 11:43:11  [127675.395289][    C2]  ? =
process_backlog+0x10c/0x230
> Sep 17 11:43:11  [127675.395386][    C2]  ? __napi_poll+0x20/0x180
> Sep 17 11:43:11  [127675.395478][    C2]  ? net_rx_action+0x2a4/0x390
>=20
>=20
> In all server have simple nftables rulls , ethernet card is intel =
xl710 or 82599. its a very simple config.
>=20
> m.=20
>=20
>=20
>=20
>=20
>> On 16 Sep 2023, at 12:04, Martin Zaharinov <micron10@gmail.com> =
wrote:
>>=20
>> Hi Paolo
>>=20
>> in first report machine dont have out of tree module
>>=20
>> this bug is come after move from kernel 6.2 to 6.3
>>=20
>> m.
>>=20
>> On Sat, Sep 16, 2023, 11:27 Paolo Abeni <pabeni@redhat.com> wrote:
>> On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
>>> one more log:
>>>=20
>>> Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here =
]------------
>>> Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
>>> Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at =
lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
>>> Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: =
nft_limit nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc =
nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 nf_xnatlog(O) ipmi_si ipmi_devintf =
ipmi_msghandler rtc_cmos [last unloaded: BNGBOOT(O)]
>>> Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: =
swapper/5 Tainted: G           O       6.5.2 #1
>>=20
>>=20
>> You have out-of-tree modules taint in all the report you shared. =
Please
>> try to reproduce the issue with such taint, thanks!
>>=20
>> Paolo
>>=20
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id B69EE2594
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 12:09:22 +0000 (UTC)
X-Greylist: delayed 302 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Sun, 17 Sep 2023 05:09:19 PDT
Received: from ciao.gmane.io (ciao.gmane.io [116.202.254.214])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DF8E12B
	for <netdev@vger.kernel.org>; Sun, 17 Sep 2023 05:09:18 -0700 (PDT)
Received: from list by ciao.gmane.io with local (Exim 4.92)
	(envelope-from <gl-netdev-2@m.gmane-mx.org>)
	id 1qhqVZ-0006L4-FK
	for netdev@vger.kernel.org; Sun, 17 Sep 2023 14:04:13 +0200
X-Injected-Via-Gmane: http://gmane.org/
To: netdev@vger.kernel.org
From: Holger =?iso-8859-1?q?Hoffst=E4tte?= <holger@applied-asynchrony.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
Date: Sun, 17 Sep 2023 12:04:08 -0000 (UTC)
Message-ID: <pan$7665b$7d10e741$bd96a442$c2861d9e@applied-asynchrony.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
	<51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
	<67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
	<8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
	<CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
	<43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
	<5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
User-Agent: Pan/0.151 (Butcha; a6f63273)
X-Spam-Status: No, score=0.2 required=5.0 tests=BAYES_40,
	HEADER_FROM_DIFFERENT_DOMAINS,SPF_HELO_NONE,SPF_PASS autolearn=no
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On Sun, 17 Sep 2023 14:55:25 +0300, Martin Zaharinov wrote:

> Hi Eric
> is it possible bug to come from this patch : https://patchwork.kernel.org/project/netdevbpf/cover/20230911170531.828100-1-edumazet@google.com/ 

No, because

1) those patches are not in any released kernel
2) they work fine

Holger


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0DE4749D
	for <netdev@vger.kernel.org>; Mon, 18 Sep 2023 08:09:55 +0000 (UTC)
Received: from mail-qt1-x832.google.com (mail-qt1-x832.google.com [IPv6:2607:f8b0:4864:20::832])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6F7C9F
	for <netdev@vger.kernel.org>; Mon, 18 Sep 2023 01:09:52 -0700 (PDT)
Received: by mail-qt1-x832.google.com with SMTP id d75a77b69052e-41513d2cca7so421841cf.0
        for <netdev@vger.kernel.org>; Mon, 18 Sep 2023 01:09:52 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20230601; t=1695024592; x=1695629392; darn=vger.kernel.org;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=38st7+CwNWQl13xaTkpZZQeswKXbWoJIBIN4bsM6Emo=;
        b=m3DsUHkI15ViQGFCMSBgYCDmylYRsWAZdL9HyTH1gFEjtNYeiWhUlYNG2wUKvzCctR
         c0SV4XyBSV2tFxH+3eSJLiSZ3qBA+CcQ0XiBv2Pp/GY8lEhGUEaZ+ob34SF1W6BVecd9
         Wud8BA72hymy8kXJrGkyyS5Z/xn5p1+es5253JUkzuh/lx2qBjiOrUIkmix1WaTJoDCn
         c0jry9kGpdb2W4vhQNFtsq3a+0NfVK4Uf4V+bkUOg2lJTeZB+FZBpgUnjvqXo7cScaJB
         szVLk0w5Ro1RV41J3a7vi1OvOYudt04gmhCv3lfHxL2qeF6g5JPrAS7+ogn2JEl+QchQ
         5wOQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695024592; x=1695629392;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=38st7+CwNWQl13xaTkpZZQeswKXbWoJIBIN4bsM6Emo=;
        b=ntu6E9R7zJBJiZcI8GaqPxYud3CDwcDV0In3myj4r9jHWw24iFuphnlbrhNysYtLhg
         IXgj0gffOQsgSA+e6nJXmy6pH70y9oO97EOfXYDcZZ/XycQgeIBLzRvR0jmLpaFo8SSl
         tToMghFg+ts47v/5H8aKPzpRho9xe8cmGQsuxQVcpN1wHlhYlZyVez2L0DkT1LpqDfEa
         0+yHuysg9WNwqXdQMUY9wHz+PlyXAmpBkWoTuP2l+MkT7MITm6zh990lWhlMf6uYGjiX
         VN042FwaCdz6/5rJjtirrbEkXIz0qSZtA5P+jEqVjphTFEWIiLQHh0pIclZDHVyqRfWl
         wJzA==
X-Gm-Message-State: AOJu0Yy4i+RcF3MmYy+/TLTRuDLlyOxQq3dT+vQa53aDBc6ELcT0Hfmx
	fRFkJ5wpQ/NHNY08SYsBNpbw2f7mneKVOPu/eRB0+Q==
X-Google-Smtp-Source: AGHT+IFlGfwR9VKfI97ywRNTNtaaOI6Ipg1gKde7gqFOeFzSJI/pU+LysoXQWS3rtxZdtS9fNlxrILghk8dzM2Hz8mg=
X-Received: by 2002:a05:622a:1341:b0:417:9238:5a30 with SMTP id
 w1-20020a05622a134100b0041792385a30mr384071qtk.29.1695024591626; Mon, 18 Sep
 2023 01:09:51 -0700 (PDT)
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com> <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com> <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
In-Reply-To: <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
From: Eric Dumazet <edumazet@google.com>
Date: Mon, 18 Sep 2023 10:09:40 +0200
Message-ID: <CANn89iLWi7yr184a52FRrO2QW=ffLEsy1DKj+f_Yz_nPuhaMPA@mail.gmail.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
To: Martin Zaharinov <micron10@gmail.com>
Cc: Paolo Abeni <pabeni@redhat.com>, netdev <netdev@vger.kernel.org>, 
	patchwork-bot+netdevbpf@kernel.org, Jakub Kicinski <kuba@kernel.org>, 
	Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org, dsahern@gmail.com, 
	Florian Westphal <fw@strlen.de>, Pablo Neira Ayuso <pablo@netfilter.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,
	USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On Sun, Sep 17, 2023 at 1:55=E2=80=AFPM Martin Zaharinov <micron10@gmail.co=
m> wrote:
>
> Hi Eric
> is it possible bug to come from this patch : https://patchwork.kernel.org=
/project/netdevbpf/cover/20230911170531.828100-1-edumazet@google.com/
>
>

Everything is possible, but this is not in 6.5 kernels.

I would suggest you start a bisection.

> m.
>
> > On 17 Sep 2023, at 14:40, Martin Zaharinov <micron10@gmail.com> wrote:
> >
> > One more in changelog for kernel 6.5 : https://cdn.kernel.org/pub/linux=
/kernel/v6.x/ChangeLog-6.5
> >
> > I see have many bug reports with :
> >
> > Sep 17 11:43:11  [127675.395289][    C2]  ? process_backlog+0x10c/0x230
> > Sep 17 11:43:11  [127675.395386][    C2]  ? __napi_poll+0x20/0x180
> > Sep 17 11:43:11  [127675.395478][    C2]  ? net_rx_action+0x2a4/0x390
> >
> >
> > In all server have simple nftables rulls , ethernet card is intel xl710=
 or 82599. its a very simple config.
> >
> > m.
> >
> >
> >
> >
> >> On 16 Sep 2023, at 12:04, Martin Zaharinov <micron10@gmail.com> wrote:
> >>
> >> Hi Paolo
> >>
> >> in first report machine dont have out of tree module
> >>
> >> this bug is come after move from kernel 6.2 to 6.3
> >>
> >> m.
> >>
> >> On Sat, Sep 16, 2023, 11:27 Paolo Abeni <pabeni@redhat.com> wrote:
> >> On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
> >>> one more log:
> >>>
> >>> Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here ]----=
--------
> >>> Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
> >>> Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at lib/r=
curef.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
> >>> Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: nft_limit=
 nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc nft_ct nft_n=
at nft_chain_nat nf_tables netconsole coretemp bonding i40e nf_nat_sip nf_c=
onntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf=
_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4=
 nf_xnatlog(O) ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos [last unloaded=
: BNGBOOT(O)]
> >>> Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: swapper/=
5 Tainted: G           O       6.5.2 #1
> >>
> >>
> >> You have out-of-tree modules taint in all the report you shared. Pleas=
e
> >> try to reproduce the issue with such taint, thanks!
> >>
> >> Paolo
> >>
> >
>

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE94C1863B
	for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 20:09:45 +0000 (UTC)
Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DDE5C0
	for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 13:09:43 -0700 (PDT)
Received: by mail-ed1-x52d.google.com with SMTP id 4fb4d7f45d1cf-530c9980556so4409662a12.2
        for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 13:09:43 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695154182; x=1695758982; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=VZkbljvZTAoRHRQIYlghSmHbIGwVd6aPE0rKs7jySvo=;
        b=ZkDLUlHz5Hsl1E8ePil91zzEkqbqHzfmctBWJn98Ucd7xC4p4m0zAQW/jcvTcj+kOf
         PmBAOr4hhm6moUstDzynMsZcZdbjuVjkpeiScdKZiupEqnLV7JsTMEnBMlTUhnYNgY0D
         wd+ZOx9VIaRf2vWEz9+ZWcko9ySYFeK7CQmODkrqPC8i1/FF7Od54vGsRyCVWZXgvsxh
         xS8jmdWe6Os8ZSTMli1fhnIRnncg3zdw4KZ+qUzFSx1o+dikJhW0Pt/PcfLF3qZIxQui
         ZmMtgMoXYjp/sZkxZLN1cT47vMca/yY88Vxgs57qEF5n9ruysn3FJPCizamLGYtSjJ+P
         xKNw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695154182; x=1695758982;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=VZkbljvZTAoRHRQIYlghSmHbIGwVd6aPE0rKs7jySvo=;
        b=tLsD7giPKAlU/YZ0e/sScKaa0XoT7b8k34KwM7XnwtAsFcqipH2yVUohVC9k7GLBeI
         KznEjYFkKHliQr3+MlL6x+y8DNHGpF7daJz831a1qtrpEBnWFfcY31QtY8VqT+ticR0o
         JLVyrqm1kKVe7Mox8R6ncLxHCDcQhmEUGX3ClwRExnuY8MMF0QgMCp+aV6/Zjnx3bjbg
         /yl1Jx+TrIBR74W2s+pPjPHbw4TiDgzD92dXolmEC6J7agJyRDJDM3PWRyY76zqbqmdT
         h6xJ0BmyYt1UlJw9i0bSNCy46GQXv7SyT089t4lpTq3JBWdh9dbmzxmDp+j32aJMdTCo
         N0Gg==
X-Gm-Message-State: AOJu0YzLcwy1340FqEg+2hcpMAs/g/VUZvTPgU7zdZl1XCtukT4sOcM2
	M+J6ZkboOFC9iVdg1MWDh00=
X-Google-Smtp-Source: AGHT+IF+xeaCz918viEAE+QzMNvIsNHA++GSHCovrJZVj1TaNhJhlkps6YKmfHCGY18ayzDnLnNpzA==
X-Received: by 2002:aa7:c38d:0:b0:522:c1b1:8cb0 with SMTP id k13-20020aa7c38d000000b00522c1b18cb0mr395996edq.33.1695154181716;
        Tue, 19 Sep 2023 13:09:41 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id n23-20020aa7c697000000b0052e2472f884sm7781943edq.21.2023.09.19.13.09.40
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Tue, 19 Sep 2023 13:09:40 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
Date: Tue, 19 Sep 2023 23:09:29 +0300
Cc: netdev <netdev@vger.kernel.org>,
 Eric Dumazet <edumazet@google.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
 <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
To: Paolo Abeni <pabeni@redhat.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi Eric

Yes this patch is not come in 6.5 kernel and queue for 6.6 i test but =
not ok for now.

One more i find same error have in old kernel 6.4.8  , update to kernel =
6.5.4 and same error is come .

Like this is hard to catch bug

see logs :


[1462610.861373] ------------[ cut here ]------------
[1462610.861480] rcuref - imbalanced put()
[1462610.861491] WARNING: CPU: 22 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath+0x5f/0x70
[1462610.861718] Modules linked in: nft_limit nf_conntrack_netlink  =
pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables =
netconsole coretemp bonding ixgbe mdio nf_nat_sip nf_conntrack_sip =
nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[1462610.862004] CPU: 22 PID: 0 Comm: swapper/22 Tainted: G           O  =
     6.4.8 #1
[1462610.863244] Hardware name: Supermicro Super Server/X10SRW-F, BIOS =
3.4 06/05/2021
[1462610.863368] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
[1462610.863469] Code: 31 c0 eb e2 80 3d 02 cd e6 00 00 74 0a c7 03 00 =
00 00 e0 31 c0 eb cf 48 c7 c7 7f 68 e5 a4 c6 05 e8 cc e6 00 01 e8 e1 ab =
c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 =
e2
[1462610.863637] RSP: 0018:ffffaee60070cc38 EFLAGS: 00010292
[1462610.863736] RAX: 0000000000000019 RBX: ffffa1cdc35e5780 RCX: =
00000000fff7ffff
[1462610.863857] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
[1462610.864129] RBP: ffffa1cf6aeb8de8 R08: 0000000000000000 R09: =
00000000fff7ffff
[1462610.864250] R10: ffffa1d51b000000 R11: 0000000000000003 R12: =
ffffa1cdc35e5740
[1462610.864370] R13: ffffa1cdc35e57a8 R14: ffffa1d51fda9008 R15: =
00000000ade2eb6e
[1462610.864489] FS:  0000000000000000(0000) GS:ffffa1d51fd80000(0000) =
knlGS:0000000000000000
[1462610.864615] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1462610.864713] CR2: 00007f057b8ad000 CR3: 0000000141881003 CR4: =
00000000001706e0
[1462610.864833] Call Trace:
[1462610.864928]  <IRQ>
[1462610.865021]  ? __warn+0x6c/0x130
[1462610.865124]  ? report_bug+0x1e4/0x260
[1462610.865223]  ? handle_bug+0x36/0x70
[1462610.865318]  ? exc_invalid_op+0x17/0x1a0
[1462610.865414]  ? asm_exc_invalid_op+0x16/0x20
[1462610.865517]  ? rcuref_put_slowpath+0x5f/0x70
[1462610.865618]  ? rcuref_put_slowpath+0x5f/0x70
[1462610.865719]  dst_release+0x2c/0x60
[1462610.865817]  rt_cache_route+0xbd/0xf0
[1462610.865913]  rt_set_nexthop.isra.0+0x1b6/0x440
[1462610.866008]  ip_route_input_slow+0x90e/0xc60
[1462610.866116]  ? nf_conntrack_udp_packet+0x16c/0x230 [nf_conntrack]
[1462610.866229]  ip_route_input_noref+0xed/0x100
[1462610.866328]  ip_rcv_finish_core.isra.0+0xb1/0x410
[1462610.866425]  ip_rcv+0xed/0x130
[1462610.866522]  ? ip_rcv_core.constprop.0+0x350/0x350
[1462610.866621]  process_backlog+0x10c/0x230
[1462610.866719]  __napi_poll+0x20/0x180
[1462610.866818]  net_rx_action+0x2a4/0x390
[1462610.866921]  __do_softirq+0xd0/0x202
[1462610.867020]  do_softirq+0x58/0x80
[1462610.867116]  </IRQ>
[1462610.867206]  <TASK>
[1462610.867298]  flush_smp_call_function_queue+0x3f/0x60
[1462610.867403]  do_idle+0x14d/0x210
[1462610.867500]  cpu_startup_entry+0x14/0x20
[1462610.867602]  start_secondary+0xec/0xf0
[1462610.867701]  secondary_startup_64_no_verify+0xf9/0xfb
[1462610.867799]  </TASK>
[1462610.867891] ---[ end trace 0000000000000000 ]=E2=80=94


And this si 6.5.4 :=20

[39651.441371] ------------[ cut here ]------------
[39651.441455] rcuref - imbalanced put()
[39651.441470] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath+0x5f/0x70
[39651.441633] Modules linked in: nft_limit pppoe pppox ppp_generic slhc =
nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp igb =
i2c_algo_bit i40e ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[39651.441805] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G           O    =
   6.5.3 #1
[39651.441911] Hardware name: To Be Filled By O.E.M. To Be Filled By =
O.E.M./EP2C612D8, BIOS P2.30 04/30/2018
[39651.442035] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
[39651.442131] Code: 31 c0 eb e2 80 3d 86 ae e6 00 00 74 0a c7 03 00 00 =
00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 9a c6 05 6c ae e6 00 01 e8 11 71 c7 =
ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
[39651.442294] RSP: 0018:ffffbb9a404b4de8 EFLAGS: 00010296
[39651.442390] RAX: 0000000000000019 RBX: ffffa13ac9a32640 RCX: =
00000000fff7ffff
[39651.442513] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
[39651.442630] RBP: ffffa13a44a04000 R08: 0000000000000000 R09: =
00000000fff7ffff
[39651.442748] R10: ffffa1419ae00000 R11: 0000000000000003 R12: =
ffffa13ab640bec0
[39651.442866] R13: 0000000000000000 R14: 0000000000000010 R15: =
ffffbb9a404b4f60
[39651.442985] FS:  0000000000000000(0000) GS:ffffa1419f900000(0000) =
knlGS:0000000000000000
[39651.443106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[39651.443201] CR2: 0000564f9e23f6e0 CR3: 000000010bcea002 CR4: =
00000000003706e0
[39651.443319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[39651.443438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[39651.443558] Call Trace:
[39651.443647]  <IRQ>
[39651.443736]  ? __warn+0x6c/0x130
[39651.443829]  ? report_bug+0x1e4/0x260
[39651.443924]  ? handle_bug+0x36/0x70
[39651.444016]  ? exc_invalid_op+0x17/0x1a0
[39651.444109]  ? asm_exc_invalid_op+0x16/0x20
[39651.444202]  ? rcuref_put_slowpath+0x5f/0x70
[39651.444297]  ? rcuref_put_slowpath+0x5f/0x70
[39651.444391]  dst_release+0x2c/0x60
[39651.444487]  __dev_queue_xmit+0x56c/0xbd0
[39651.444582]  ? nf_hook_slow+0x36/0xa0
[39651.444675]  ip_finish_output2+0x27b/0x520
[39651.444770]  process_backlog+0x10c/0x230
[39651.444866]  __napi_poll+0x20/0x180
[39651.444961]  net_rx_action+0x2a4/0x390
[39651.445055]  __do_softirq+0xd0/0x202
[39651.445148]  do_softirq+0x3a/0x50
[39651.445241]  </IRQ>
[39651.445329]  <TASK>
[39651.445416]  flush_smp_call_function_queue+0x3f/0x50
[39651.445516]  do_idle+0x14d/0x210
[39651.445609]  cpu_startup_entry+0x14/0x20
[39651.445702]  start_secondary+0xe1/0xf0
[39651.445797]  secondary_startup_64_no_verify+0x167/0x16b
[39651.445893]  </TASK>
[39651.445982] ---[ end trace 0000000000000000 ]=E2=80=94


best regards,
Martin

> On 17 Sep 2023, at 14:55, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi Eric
> is it possible bug to come from this patch : =
https://patchwork.kernel.org/project/netdevbpf/cover/20230911170531.828100=
-1-edumazet@google.com/=20
>=20
>=20
> m.
>=20
>> On 17 Sep 2023, at 14:40, Martin Zaharinov <micron10@gmail.com> =
wrote:
>>=20
>> One more in changelog for kernel 6.5 : =
https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.5
>>=20
>> I see have many bug reports with :=20
>>=20
>> Sep 17 11:43:11  [127675.395289][    C2]  ? =
process_backlog+0x10c/0x230
>> Sep 17 11:43:11  [127675.395386][    C2]  ? __napi_poll+0x20/0x180
>> Sep 17 11:43:11  [127675.395478][    C2]  ? net_rx_action+0x2a4/0x390
>>=20
>>=20
>> In all server have simple nftables rulls , ethernet card is intel =
xl710 or 82599. its a very simple config.
>>=20
>> m.=20
>>=20
>>=20
>>=20
>>=20
>>> On 16 Sep 2023, at 12:04, Martin Zaharinov <micron10@gmail.com> =
wrote:
>>>=20
>>> Hi Paolo
>>>=20
>>> in first report machine dont have out of tree module
>>>=20
>>> this bug is come after move from kernel 6.2 to 6.3
>>>=20
>>> m.
>>>=20
>>> On Sat, Sep 16, 2023, 11:27 Paolo Abeni <pabeni@redhat.com> wrote:
>>> On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
>>>> one more log:
>>>>=20
>>>> Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here =
]------------
>>>> Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
>>>> Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at =
lib/rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
>>>> Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: =
nft_limit nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc =
nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp bonding i40e =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 nf_xnatlog(O) ipmi_si ipmi_devintf =
ipmi_msghandler rtc_cmos [last unloaded: BNGBOOT(O)]
>>>> Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: =
swapper/5 Tainted: G           O       6.5.2 #1
>>>=20
>>>=20
>>> You have out-of-tree modules taint in all the report you shared. =
Please
>>> try to reproduce the issue with such taint, thanks!
>>>=20
>>> Paolo
>>>=20
>>=20
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id BBCCC1FC1
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 03:59:26 +0000 (UTC)
Received: from mail-qt1-x82e.google.com (mail-qt1-x82e.google.com [IPv6:2607:f8b0:4864:20::82e])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F584AC
	for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 20:59:24 -0700 (PDT)
Received: by mail-qt1-x82e.google.com with SMTP id d75a77b69052e-414ba610766so232711cf.0
        for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 20:59:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20230601; t=1695182363; x=1695787163; darn=vger.kernel.org;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=h9Ycw/9hA+zPDGYn+bxgsaVXjgOgiTsVBuBxIwxP8UU=;
        b=oF9fC093JQw2+7yZPj8Hps4QYiL6pg6MF6zyYLhjw5uHkfCCTTvd+N1WQ/LOvPuEVj
         sXzI0JCokbqz8jChvjs3B6e2Aoq+BZ8GG38jI546lbR3Kh2yar2Wk0y8dShsoY4/7qrn
         jivUHwe+0l6eFgauEpg5esOLAm2U6LRfDV/kJ3mRXmIhV9jKGtuTkytTUMol6dXeGaSn
         HihzheOZcWpkZ0jrVaiMKO1uSglVkQrc4H2b545CAQD2Sg+vP+eaM9IpBAKdm5cAVMBP
         mkg4qenWTWc0W6SwXXMZhhPZfRRpwWdYeldossdZDu5ND6u0XbwHDGj8dd62kk6vwtBp
         Yg+g==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695182363; x=1695787163;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=h9Ycw/9hA+zPDGYn+bxgsaVXjgOgiTsVBuBxIwxP8UU=;
        b=TeimB4/1rqIwDf40dBCIN70HqYGLVZvVrEZ9N2aNeTsDmCkMjZ1LyvOlA33wsjwHmw
         h5SOrXWsGp0gDlfpGUmdmY26+Vl2rFTxEoQhQaStoAEA0k0m5/Q1dRHGgkuWXsjg+sw+
         epteFUBSYMV02fs9anEQUsmWU/IdGp1apcXZQagvn7JHKUnsWH+kei59FM34hw/SO+7x
         uh4VHqS0kpUXAWIop79hBbjMhuEmw1SFWzZVx3KuWfz4qFm6PrwNf7Npv15e3iyol1Gf
         5OY2K9eJCAYHpUZC4cQ4Qe4495nEq3mPCoBbFuKU20FpyFNwQsUrhFcxbWOKwQGxHuEr
         7IrQ==
X-Gm-Message-State: AOJu0YzyAS9eu9TwIsp3Oif73c56yZeCZUU1jHQMMIx5c6eBn8dfGp7F
	nu/U6mW5SN7ZmAHUSPUa3UIFeVXMOnf94UwXlaiBCA==
X-Google-Smtp-Source: AGHT+IFayKogvSuHafvD9Y6iE7tOQYR0mmh4U6Xhi/Ate8rozJ8gc5ITjwotTr3UFnqiIBGHLMTQg9ikaSFr6bs9SUU=
X-Received: by 2002:a05:622a:211:b0:417:8dd5:df0d with SMTP id
 b17-20020a05622a021100b004178dd5df0dmr97137qtx.9.1695182362875; Tue, 19 Sep
 2023 20:59:22 -0700 (PDT)
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com> <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com> <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
In-Reply-To: <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
From: Eric Dumazet <edumazet@google.com>
Date: Wed, 20 Sep 2023 05:59:11 +0200
Message-ID: <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
To: Martin Zaharinov <micron10@gmail.com>
Cc: Paolo Abeni <pabeni@redhat.com>, netdev <netdev@vger.kernel.org>, 
	patchwork-bot+netdevbpf@kernel.org, Jakub Kicinski <kuba@kernel.org>, 
	Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org, dsahern@gmail.com, 
	Florian Westphal <fw@strlen.de>, Pablo Neira Ayuso <pablo@netfilter.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,
	USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On Tue, Sep 19, 2023 at 10:09=E2=80=AFPM Martin Zaharinov <micron10@gmail.c=
om> wrote:
>
> Hi Eric
>
> Yes this patch is not come in 6.5 kernel and queue for 6.6 i test but not=
 ok for now.

"not ok for now" ? What does this mean?
Pointing out patches that are not related to your issue is a waste of time.
If this was to bring my attention, this is a bad strategy, because I
will probably not read your future emails.

>
> One more i find same error have in old kernel 6.4.8  , update to kernel 6=
.5.4 and same error is come .
>
> Like this is hard to catch bug
>
> see logs :
>
>
> [1462610.861373] ------------[ cut here ]------------
> [1462610.861480] rcuref - imbalanced put()
> [1462610.861491] WARNING: CPU: 22 PID: 0 at lib/rcuref.c:267 rcuref_put_s=
lowpath+0x5f/0x70
> [1462610.861718] Modules linked in: nft_limit nf_conntrack_netlink  pppoe=
 pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole c=
oretemp bonding ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_connt=
rack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat =
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghan=
dler rtc_cmos
> [1462610.862004] CPU: 22 PID: 0 Comm: swapper/22 Tainted: G           O  =
     6.4.8 #1
> [1462610.863244] Hardware name: Supermicro Super Server/X10SRW-F, BIOS 3.=
4 06/05/2021
> [1462610.863368] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
> [1462610.863469] Code: 31 c0 eb e2 80 3d 02 cd e6 00 00 74 0a c7 03 00 00=
 00 e0 31 c0 eb cf 48 c7 c7 7f 68 e5 a4 c6 05 e8 cc e6 00 01 e8 e1 ab c7 ff=
 <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
> [1462610.863637] RSP: 0018:ffffaee60070cc38 EFLAGS: 00010292
> [1462610.863736] RAX: 0000000000000019 RBX: ffffa1cdc35e5780 RCX: 0000000=
0fff7ffff
> [1462610.863857] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: 0000000=
0ffffffea
> [1462610.864129] RBP: ffffa1cf6aeb8de8 R08: 0000000000000000 R09: 0000000=
0fff7ffff
> [1462610.864250] R10: ffffa1d51b000000 R11: 0000000000000003 R12: ffffa1c=
dc35e5740
> [1462610.864370] R13: ffffa1cdc35e57a8 R14: ffffa1d51fda9008 R15: 0000000=
0ade2eb6e
> [1462610.864489] FS:  0000000000000000(0000) GS:ffffa1d51fd80000(0000) kn=
lGS:0000000000000000
> [1462610.864615] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1462610.864713] CR2: 00007f057b8ad000 CR3: 0000000141881003 CR4: 0000000=
0001706e0
> [1462610.864833] Call Trace:
> [1462610.864928]  <IRQ>
> [1462610.865021]  ? __warn+0x6c/0x130
> [1462610.865124]  ? report_bug+0x1e4/0x260
> [1462610.865223]  ? handle_bug+0x36/0x70
> [1462610.865318]  ? exc_invalid_op+0x17/0x1a0
> [1462610.865414]  ? asm_exc_invalid_op+0x16/0x20
> [1462610.865517]  ? rcuref_put_slowpath+0x5f/0x70
> [1462610.865618]  ? rcuref_put_slowpath+0x5f/0x70
> [1462610.865719]  dst_release+0x2c/0x60
> [1462610.865817]  rt_cache_route+0xbd/0xf0
> [1462610.865913]  rt_set_nexthop.isra.0+0x1b6/0x440
> [1462610.866008]  ip_route_input_slow+0x90e/0xc60
> [1462610.866116]  ? nf_conntrack_udp_packet+0x16c/0x230 [nf_conntrack]
> [1462610.866229]  ip_route_input_noref+0xed/0x100
> [1462610.866328]  ip_rcv_finish_core.isra.0+0xb1/0x410
> [1462610.866425]  ip_rcv+0xed/0x130
> [1462610.866522]  ? ip_rcv_core.constprop.0+0x350/0x350
> [1462610.866621]  process_backlog+0x10c/0x230
> [1462610.866719]  __napi_poll+0x20/0x180
> [1462610.866818]  net_rx_action+0x2a4/0x390
> [1462610.866921]  __do_softirq+0xd0/0x202
> [1462610.867020]  do_softirq+0x58/0x80
> [1462610.867116]  </IRQ>
> [1462610.867206]  <TASK>
> [1462610.867298]  flush_smp_call_function_queue+0x3f/0x60
> [1462610.867403]  do_idle+0x14d/0x210
> [1462610.867500]  cpu_startup_entry+0x14/0x20
> [1462610.867602]  start_secondary+0xec/0xf0
> [1462610.867701]  secondary_startup_64_no_verify+0xf9/0xfb
> [1462610.867799]  </TASK>
> [1462610.867891] ---[ end trace 0000000000000000 ]=E2=80=94
>
>
> And this si 6.5.4 :
>
> [39651.441371] ------------[ cut here ]------------
> [39651.441455] rcuref - imbalanced put()
> [39651.441470] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:267 rcuref_put_slo=
wpath+0x5f/0x70
> [39651.441633] Modules linked in: nft_limit pppoe pppox ppp_generic slhc =
nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp igb i2c_algo_bit=
 i40e ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp =
nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntra=
ck nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler rtc_c=
mos
> [39651.441805] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G           O    =
   6.5.3 #1
> [39651.441911] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.=
M./EP2C612D8, BIOS P2.30 04/30/2018
> [39651.442035] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
> [39651.442131] Code: 31 c0 eb e2 80 3d 86 ae e6 00 00 74 0a c7 03 00 00 0=
0 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 9a c6 05 6c ae e6 00 01 e8 11 71 c7 ff <=
0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
> [39651.442294] RSP: 0018:ffffbb9a404b4de8 EFLAGS: 00010296
> [39651.442390] RAX: 0000000000000019 RBX: ffffa13ac9a32640 RCX: 00000000f=
ff7ffff
> [39651.442513] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: 00000000f=
fffffea
> [39651.442630] RBP: ffffa13a44a04000 R08: 0000000000000000 R09: 00000000f=
ff7ffff
> [39651.442748] R10: ffffa1419ae00000 R11: 0000000000000003 R12: ffffa13ab=
640bec0
> [39651.442866] R13: 0000000000000000 R14: 0000000000000010 R15: ffffbb9a4=
04b4f60
> [39651.442985] FS:  0000000000000000(0000) GS:ffffa1419f900000(0000) knlG=
S:0000000000000000
> [39651.443106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [39651.443201] CR2: 0000564f9e23f6e0 CR3: 000000010bcea002 CR4: 000000000=
03706e0
> [39651.443319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 000000000=
0000000
> [39651.443438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 000000000=
0000400
> [39651.443558] Call Trace:
> [39651.443647]  <IRQ>
> [39651.443736]  ? __warn+0x6c/0x130
> [39651.443829]  ? report_bug+0x1e4/0x260
> [39651.443924]  ? handle_bug+0x36/0x70
> [39651.444016]  ? exc_invalid_op+0x17/0x1a0
> [39651.444109]  ? asm_exc_invalid_op+0x16/0x20
> [39651.444202]  ? rcuref_put_slowpath+0x5f/0x70
> [39651.444297]  ? rcuref_put_slowpath+0x5f/0x70
> [39651.444391]  dst_release+0x2c/0x60
> [39651.444487]  __dev_queue_xmit+0x56c/0xbd0
> [39651.444582]  ? nf_hook_slow+0x36/0xa0
> [39651.444675]  ip_finish_output2+0x27b/0x520
> [39651.444770]  process_backlog+0x10c/0x230
> [39651.444866]  __napi_poll+0x20/0x180
> [39651.444961]  net_rx_action+0x2a4/0x390
> [39651.445055]  __do_softirq+0xd0/0x202
> [39651.445148]  do_softirq+0x3a/0x50
> [39651.445241]  </IRQ>
> [39651.445329]  <TASK>
> [39651.445416]  flush_smp_call_function_queue+0x3f/0x50
> [39651.445516]  do_idle+0x14d/0x210
> [39651.445609]  cpu_startup_entry+0x14/0x20
> [39651.445702]  start_secondary+0xe1/0xf0
> [39651.445797]  secondary_startup_64_no_verify+0x167/0x16b
> [39651.445893]  </TASK>
> [39651.445982] ---[ end trace 0000000000000000 ]=E2=80=94
>
>
> best regards,
> Martin

You keep sending traces without symbols, nobody here will even look at them=
.

Again, your best route is a bisection.

>
> > On 17 Sep 2023, at 14:55, Martin Zaharinov <micron10@gmail.com> wrote:
> >
> > Hi Eric
> > is it possible bug to come from this patch : https://patchwork.kernel.o=
rg/project/netdevbpf/cover/20230911170531.828100-1-edumazet@google.com/
> >
> >
> > m.
> >
> >> On 17 Sep 2023, at 14:40, Martin Zaharinov <micron10@gmail.com> wrote:
> >>
> >> One more in changelog for kernel 6.5 : https://cdn.kernel.org/pub/linu=
x/kernel/v6.x/ChangeLog-6.5
> >>
> >> I see have many bug reports with :
> >>
> >> Sep 17 11:43:11  [127675.395289][    C2]  ? process_backlog+0x10c/0x23=
0
> >> Sep 17 11:43:11  [127675.395386][    C2]  ? __napi_poll+0x20/0x180
> >> Sep 17 11:43:11  [127675.395478][    C2]  ? net_rx_action+0x2a4/0x390
> >>
> >>
> >> In all server have simple nftables rulls , ethernet card is intel xl71=
0 or 82599. its a very simple config.
> >>
> >> m.
> >>
> >>
> >>
> >>
> >>> On 16 Sep 2023, at 12:04, Martin Zaharinov <micron10@gmail.com> wrote=
:
> >>>
> >>> Hi Paolo
> >>>
> >>> in first report machine dont have out of tree module
> >>>
> >>> this bug is come after move from kernel 6.2 to 6.3
> >>>
> >>> m.
> >>>
> >>> On Sat, Sep 16, 2023, 11:27 Paolo Abeni <pabeni@redhat.com> wrote:
> >>> On Sat, 2023-09-16 at 02:11 +0300, Martin Zaharinov wrote:
> >>>> one more log:
> >>>>
> >>>> Sep 12 07:37:29  [151563.298466][    C5] ------------[ cut here ]---=
---------
> >>>> Sep 12 07:37:29  [151563.298550][    C5] rcuref - imbalanced put()
> >>>> Sep 12 07:37:29  [151563.298564][ C5] WARNING: CPU: 5 PID: 0 at lib/=
rcuref.c:267 rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
> >>>> Sep 12 07:37:29  [151563.298724][    C5] Modules linked in: nft_limi=
t nf_conntrack_netlink vlan_mon(O) pppoe pppox ppp_generic slhc nft_ct nft_=
nat nft_chain_nat nf_tables netconsole coretemp bonding i40e nf_nat_sip nf_=
conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp n=
f_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv=
4 nf_xnatlog(O) ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos [last unloade=
d: BNGBOOT(O)]
> >>>> Sep 12 07:37:29  [151563.298894][    C5] CPU: 5 PID: 0 Comm: swapper=
/5 Tainted: G           O       6.5.2 #1
> >>>
> >>>
> >>> You have out-of-tree modules taint in all the report you shared. Plea=
se
> >>> try to reproduce the issue with such taint, thanks!
> >>>
> >>> Paolo
> >>>
> >>
> >
>

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 81B0223C6
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 06:05:29 +0000 (UTC)
Received: from mail-lf1-x135.google.com (mail-lf1-x135.google.com [IPv6:2a00:1450:4864:20::135])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02DF6A1
	for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 23:05:25 -0700 (PDT)
Received: by mail-lf1-x135.google.com with SMTP id 2adb3069b0e04-502934c88b7so10671479e87.2
        for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 23:05:24 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695189923; x=1695794723; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=FXzQOtiYhDSAEw6FcMmFy5OEfBfW4gt2wAFNQBJirxY=;
        b=dJ85rta+pM8IZo3uZVE6VA5jiowVTJZ1txfZJ4aYdD9PWZifHnyaJlHHxzHC0Mh0gF
         GNCAWkGu/SbpxBLfduJCDo5QCokiYTHmqVQJTTRHhQGYPq34ElwHo0oLUbBU1kd5UPWa
         4L6X9RcR6xHqqsGJz72jqkOR4ZuGgCwzRnNiSJczm6BnTTTh56grTUolRgo6rwnOqvZ3
         sxmiQgUDQMeb2qB1Ra4ubFxYscrb3OCu3VDfWz98ESI4oSE2IKDOmf4aq9UOhhMZpPFw
         kR6zECBHGFGkfnnSjGMmTETs6BWZ2iIZ3zA5LGt2rqK41w0q8aExoGeC9e7mu6cLdsbh
         3eKg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695189923; x=1695794723;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=FXzQOtiYhDSAEw6FcMmFy5OEfBfW4gt2wAFNQBJirxY=;
        b=e25L8euBgkLZai1nDd3wh5Lypw51NHwjC49LaQqS2juX5XJWf6i8smUKlkOscQqM+S
         g+dB09q22E5Dnhw5L6A1t9jCAnP1KeH/fb2ifCjHjv/NW4EG2d+Cmxj0taYgIGsk76RL
         4RaetUXwn4qOrqAdqobXe5rAuT63rrcCtvQAjVBnEsWxEqj3xh1aUqyHad7KUSN/VH3R
         RGWpSyJ7VNd+JaJzPSbB2n4ufmWzpR3VIcIoCX7nK+bhpnwvsZFLQTc8tjf2C3R7AvCE
         QNg4yDwo4f8l15phW23ULpJeVKLNEEfjjaJz2K/Dz2K/PYhua991ULdSb40h4PKIouqG
         DUXg==
X-Gm-Message-State: AOJu0Ywvf8tKk4CzwRbtbevdnklkDY1mFN3RVtKFCWXdHOTG+YFOlseS
	T953oRSYjQj1mrg9IMcIaNFYfWgx7FU=
X-Google-Smtp-Source: AGHT+IF7Jck8OHIHin8Akr7Opp4kOHJA7jAyPhudPv0yMKml4lEn1RxgN+lmTXvG3/CLTnOQ66Iqfg==
X-Received: by 2002:a2e:a40f:0:b0:2bc:f4ee:ca57 with SMTP id p15-20020a2ea40f000000b002bcf4eeca57mr1158591ljn.48.1695189922564;
        Tue, 19 Sep 2023 23:05:22 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id oq14-20020a170906cc8e00b0099e12a49c8fsm8932558ejb.173.2023.09.19.23.05.21
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Tue, 19 Sep 2023 23:05:21 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
Date: Wed, 20 Sep 2023 09:05:10 +0300
Cc: Paolo Abeni <pabeni@redhat.com>,
 netdev <netdev@vger.kernel.org>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <BB129799-E196-428C-909D-721670DD5E21@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
 <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
To: Eric Dumazet <edumazet@google.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi Eric

> On 20 Sep 2023, at 6:59, Eric Dumazet <edumazet@google.com> wrote:
>=20
> On Tue, Sep 19, 2023 at 10:09=E2=80=AFPM Martin Zaharinov =
<micron10@gmail.com> wrote:
>>=20
>> Hi Eric
>>=20
>> Yes this patch is not come in 6.5 kernel and queue for 6.6 i test but =
not ok for now.
>=20
> "not ok for now" ? What does this mean?
> Pointing out patches that are not related to your issue is a waste of =
time.
> If this was to bring my attention, this is a bad strategy, because I
> will probably not read your future emails.
>=20

I'm sorry, I didn't speak correctly.
patch is very good but for kernel 6.6.
I enjoy your kernel improvements.=20
And thanks for that !!


>>=20
>> One more i find same error have in old kernel 6.4.8  , update to =
kernel 6.5.4 and same error is come .
>>=20
>> Like this is hard to catch bug
>>=20
>> see logs :
>>=20
>>=20
>> [1462610.861373] ------------[ cut here ]------------
>> [1462610.861480] rcuref - imbalanced put()
>> [1462610.861491] WARNING: CPU: 22 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath+0x5f/0x70
>> [1462610.861718] Modules linked in: nft_limit nf_conntrack_netlink  =
pppoe pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables =
netconsole coretemp bonding ixgbe mdio nf_nat_sip nf_conntrack_sip =
nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
>> [1462610.862004] CPU: 22 PID: 0 Comm: swapper/22 Tainted: G           =
O       6.4.8 #1
>> [1462610.863244] Hardware name: Supermicro Super Server/X10SRW-F, =
BIOS 3.4 06/05/2021
>> [1462610.863368] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
>> [1462610.863469] Code: 31 c0 eb e2 80 3d 02 cd e6 00 00 74 0a c7 03 =
00 00 00 e0 31 c0 eb cf 48 c7 c7 7f 68 e5 a4 c6 05 e8 cc e6 00 01 e8 e1 =
ab c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa =
83 e2
>> [1462610.863637] RSP: 0018:ffffaee60070cc38 EFLAGS: 00010292
>> [1462610.863736] RAX: 0000000000000019 RBX: ffffa1cdc35e5780 RCX: =
00000000fff7ffff
>> [1462610.863857] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
>> [1462610.864129] RBP: ffffa1cf6aeb8de8 R08: 0000000000000000 R09: =
00000000fff7ffff
>> [1462610.864250] R10: ffffa1d51b000000 R11: 0000000000000003 R12: =
ffffa1cdc35e5740
>> [1462610.864370] R13: ffffa1cdc35e57a8 R14: ffffa1d51fda9008 R15: =
00000000ade2eb6e
>> [1462610.864489] FS:  0000000000000000(0000) =
GS:ffffa1d51fd80000(0000) knlGS:0000000000000000
>> [1462610.864615] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [1462610.864713] CR2: 00007f057b8ad000 CR3: 0000000141881003 CR4: =
00000000001706e0
>> [1462610.864833] Call Trace:
>> [1462610.864928]  <IRQ>
>> [1462610.865021]  ? __warn+0x6c/0x130
>> [1462610.865124]  ? report_bug+0x1e4/0x260
>> [1462610.865223]  ? handle_bug+0x36/0x70
>> [1462610.865318]  ? exc_invalid_op+0x17/0x1a0
>> [1462610.865414]  ? asm_exc_invalid_op+0x16/0x20
>> [1462610.865517]  ? rcuref_put_slowpath+0x5f/0x70
>> [1462610.865618]  ? rcuref_put_slowpath+0x5f/0x70
>> [1462610.865719]  dst_release+0x2c/0x60
>> [1462610.865817]  rt_cache_route+0xbd/0xf0
>> [1462610.865913]  rt_set_nexthop.isra.0+0x1b6/0x440
>> [1462610.866008]  ip_route_input_slow+0x90e/0xc60
>> [1462610.866116]  ? nf_conntrack_udp_packet+0x16c/0x230 =
[nf_conntrack]
>> [1462610.866229]  ip_route_input_noref+0xed/0x100
>> [1462610.866328]  ip_rcv_finish_core.isra.0+0xb1/0x410
>> [1462610.866425]  ip_rcv+0xed/0x130
>> [1462610.866522]  ? ip_rcv_core.constprop.0+0x350/0x350
>> [1462610.866621]  process_backlog+0x10c/0x230
>> [1462610.866719]  __napi_poll+0x20/0x180
>> [1462610.866818]  net_rx_action+0x2a4/0x390
>> [1462610.866921]  __do_softirq+0xd0/0x202
>> [1462610.867020]  do_softirq+0x58/0x80
>> [1462610.867116]  </IRQ>
>> [1462610.867206]  <TASK>
>> [1462610.867298]  flush_smp_call_function_queue+0x3f/0x60
>> [1462610.867403]  do_idle+0x14d/0x210
>> [1462610.867500]  cpu_startup_entry+0x14/0x20
>> [1462610.867602]  start_secondary+0xec/0xf0
>> [1462610.867701]  secondary_startup_64_no_verify+0xf9/0xfb
>> [1462610.867799]  </TASK>
>> [1462610.867891] ---[ end trace 0000000000000000 ]=E2=80=94
>>=20
>>=20
>> And this si 6.5.4 :
>>=20
>> [39651.441371] ------------[ cut here ]------------
>> [39651.441455] rcuref - imbalanced put()
>> [39651.441470] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath+0x5f/0x70
>> [39651.441633] Modules linked in: nft_limit pppoe pppox ppp_generic =
slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp igb =
i2c_algo_bit i40e ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
>> [39651.441805] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G           O =
      6.5.3 #1
>> [39651.441911] Hardware name: To Be Filled By O.E.M. To Be Filled By =
O.E.M./EP2C612D8, BIOS P2.30 04/30/2018
>> [39651.442035] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
>> [39651.442131] Code: 31 c0 eb e2 80 3d 86 ae e6 00 00 74 0a c7 03 00 =
00 00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 9a c6 05 6c ae e6 00 01 e8 11 71 =
c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 =
e2
>> [39651.442294] RSP: 0018:ffffbb9a404b4de8 EFLAGS: 00010296
>> [39651.442390] RAX: 0000000000000019 RBX: ffffa13ac9a32640 RCX: =
00000000fff7ffff
>> [39651.442513] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
>> [39651.442630] RBP: ffffa13a44a04000 R08: 0000000000000000 R09: =
00000000fff7ffff
>> [39651.442748] R10: ffffa1419ae00000 R11: 0000000000000003 R12: =
ffffa13ab640bec0
>> [39651.442866] R13: 0000000000000000 R14: 0000000000000010 R15: =
ffffbb9a404b4f60
>> [39651.442985] FS:  0000000000000000(0000) GS:ffffa1419f900000(0000) =
knlGS:0000000000000000
>> [39651.443106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [39651.443201] CR2: 0000564f9e23f6e0 CR3: 000000010bcea002 CR4: =
00000000003706e0
>> [39651.443319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
>> [39651.443438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
>> [39651.443558] Call Trace:
>> [39651.443647]  <IRQ>
>> [39651.443736]  ? __warn+0x6c/0x130
>> [39651.443829]  ? report_bug+0x1e4/0x260
>> [39651.443924]  ? handle_bug+0x36/0x70
>> [39651.444016]  ? exc_invalid_op+0x17/0x1a0
>> [39651.444109]  ? asm_exc_invalid_op+0x16/0x20
>> [39651.444202]  ? rcuref_put_slowpath+0x5f/0x70
>> [39651.444297]  ? rcuref_put_slowpath+0x5f/0x70
>> [39651.444391]  dst_release+0x2c/0x60
>> [39651.444487]  __dev_queue_xmit+0x56c/0xbd0
>> [39651.444582]  ? nf_hook_slow+0x36/0xa0
>> [39651.444675]  ip_finish_output2+0x27b/0x520
>> [39651.444770]  process_backlog+0x10c/0x230
>> [39651.444866]  __napi_poll+0x20/0x180
>> [39651.444961]  net_rx_action+0x2a4/0x390
>> [39651.445055]  __do_softirq+0xd0/0x202
>> [39651.445148]  do_softirq+0x3a/0x50
>> [39651.445241]  </IRQ>
>> [39651.445329]  <TASK>
>> [39651.445416]  flush_smp_call_function_queue+0x3f/0x50
>> [39651.445516]  do_idle+0x14d/0x210
>> [39651.445609]  cpu_startup_entry+0x14/0x20
>> [39651.445702]  start_secondary+0xe1/0xf0
>> [39651.445797]  secondary_startup_64_no_verify+0x167/0x16b
>> [39651.445893]  </TASK>
>> [39651.445982] ---[ end trace 0000000000000000 ]=E2=80=94
>>=20
>>=20
>> best regards,
>> Martin
>=20
> You keep sending traces without symbols, nobody here will even look at =
them.
>=20


Here is trace with symbols :=20

[39651.441371] ------------[ cut here ]------------
[39651.441455] rcuref - imbalanced put()
[39651.441470] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
[39651.441633] Modules linked in: nft_limit pppoe pppox ppp_generic slhc =
nft_ct nft_nat nft_chain_nat nf_tables netconsole coretemp igb =
i2c_algo_bit i40e ixgbe mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[39651.441805] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G           O    =
   6.5.3 #1
[39651.441911] Hardware name: To Be Filled By O.E.M. To Be Filled By =
O.E.M./EP2C612D8, BIOS P2.30 04/30/2018
[39651.442035] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:267 =
(discriminator 1))
[39651.442131] Code: 31 c0 eb e2 80 3d 86 ae e6 00 00 74 0a c7 03 00 00 =
00 e0 31 c0 eb cf 48 c7 c7 68 f6 e2 9a c6 05 6c ae e6 00 01 e8 11 71 c7 =
ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	31 c0                	xor    %eax,%eax
   2:	eb e2                	jmp    0xffffffffffffffe6
   4:	80 3d 86 ae e6 00 00 	cmpb   $0x0,0xe6ae86(%rip)        # =
0xe6ae91
   b:	74 0a                	je     0x17
   d:	c7 03 00 00 00 e0    	movl   $0xe0000000,(%rbx)
  13:	31 c0                	xor    %eax,%eax
  15:	eb cf                	jmp    0xffffffffffffffe6
  17:	48 c7 c7 68 f6 e2 9a 	mov    $0xffffffff9ae2f668,%rdi
  1e:	c6 05 6c ae e6 00 01 	movb   $0x1,0xe6ae6c(%rip)        # =
0xe6ae91
  25:	e8 11 71 c7 ff       	call   0xffffffffffc7713b
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb df                	jmp    0xd
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	cc                   	int3
  33:	cc                   	int3
  34:	cc                   	int3
  35:	cc                   	int3
  36:	cc                   	int3
  37:	cc                   	int3
  38:	cc                   	int3
  39:	cc                   	int3
  3a:	cc                   	int3
  3b:	48 89 fa             	mov    %rdi,%rdx
  3e:	83                   	.byte 0x83
  3f:	e2                   	.byte 0xe2

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	eb df                	jmp    0xffffffffffffffe3
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	cc                   	int3
   9:	cc                   	int3
   a:	cc                   	int3
   b:	cc                   	int3
   c:	cc                   	int3
   d:	cc                   	int3
   e:	cc                   	int3
   f:	cc                   	int3
  10:	cc                   	int3
  11:	48 89 fa             	mov    %rdi,%rdx
  14:	83                   	.byte 0x83
  15:	e2                   	.byte 0xe2
[39651.442294] RSP: 0018:ffffbb9a404b4de8 EFLAGS: 00010296
[39651.442390] RAX: 0000000000000019 RBX: ffffa13ac9a32640 RCX: =
00000000fff7ffff
[39651.442513] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
[39651.442630] RBP: ffffa13a44a04000 R08: 0000000000000000 R09: =
00000000fff7ffff
[39651.442748] R10: ffffa1419ae00000 R11: 0000000000000003 R12: =
ffffa13ab640bec0
[39651.442866] R13: 0000000000000000 R14: 0000000000000010 R15: =
ffffbb9a404b4f60
[39651.442985] FS:  0000000000000000(0000) GS:ffffa1419f900000(0000) =
knlGS:0000000000000000
[39651.443106] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[39651.443201] CR2: 0000564f9e23f6e0 CR3: 000000010bcea002 CR4: =
00000000003706e0
[39651.443319] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[39651.443438] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[39651.443558] Call Trace:
[39651.443647]  <IRQ>
[39651.443736] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
[39651.443829] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[39651.443924] ? handle_bug (arch/x86/kernel/traps.c:324)
[39651.444016] ? exc_invalid_op (arch/x86/kernel/traps.c:345 =
(discriminator 1))
[39651.444109] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[39651.444202] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[39651.444297] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[39651.444391] dst_release (./arch/x86/include/asm/preempt.h:95 =
./include/linux/rcuref.h:151 net/core/dst.c:166)
[39651.444487] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4158)
[39651.444582] ? nf_hook_slow (./include/linux/netfilter.h:143 =
net/netfilter/core.c:626)
[39651.444675] ip_finish_output2 (./include/linux/netdevice.h:3088 =
./include/net/neighbour.h:528 ./include/net/neighbour.h:542 =
net/ipv4/ip_output.c:230)
[39651.444770] process_backlog (./include/linux/rcupdate.h:781 =
net/core/dev.c:5896)
[39651.444866] __napi_poll (net/core/dev.c:6461)
[39651.444961] net_rx_action (net/core/dev.c:6530 net/core/dev.c:6661)
[39651.445055] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
[39651.445148] do_softirq (kernel/softirq.c:463 (discriminator 32) =
kernel/softirq.c:450 (discriminator 32))
[39651.445241]  </IRQ>
[39651.445329]  <TASK>
[39651.445416] flush_smp_call_function_queue =
(./arch/x86/include/asm/irqflags.h:134 (discriminator 1) =
kernel/smp.c:570 (discriminator 1))
[39651.445516] do_idle (kernel/sched/idle.c:314)
[39651.445609] cpu_startup_entry (kernel/sched/idle.c:378)
[39651.445702] start_secondary (arch/x86/kernel/smpboot.c:326)
[39651.445797] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:441)
[39651.445893]  </TASK>
[39651.445982] ---[ end trace 0000000000000000 ]---



> Again, your best route is a bisection.

For now its not possible to make bisection , its hard to change kernel =
on running machine =E2=80=A6

is there another way to catch from where is come this bug message.

Best regards,
Martin=20





From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AAD83C24
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 06:16:12 +0000 (UTC)
Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBA079D
	for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 23:16:09 -0700 (PDT)
Received: by mail-pg1-x531.google.com with SMTP id 41be03b00d2f7-577fb90bb76so3585765a12.2
        for <netdev@vger.kernel.org>; Tue, 19 Sep 2023 23:16:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695190569; x=1695795369; darn=vger.kernel.org;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to;
        bh=uM+9IiWPEF3F1CPkHQIOgF++G8nvzeRdPlLs2FAEgto=;
        b=RHRooVprKmgR3BD7sK6BbnY8ni/6C56FRNm7eYYgx9SvkUtIF+DM4df1i0O4cQrjoP
         ArNSjK8W03DG8W/ADX0+2dENqoRSav8LZwgxsKD/PRwQwuD6Y45poRLnV3nS+HR9S0eg
         UF228RGkw8NgP4juWAZwdp+gb8XGHikrbWqyB5Ytmm2ozsnV9+//rFwh7/M51P/GR8e2
         KfSHiV2jmEQ30UpesVQDQNYdavRj6+fJ1BjjOYibaQ+xUTZqgCMM69lUSSpHswYimXfM
         2oCFq264YO2cq/0WEUwge49XSzqYgzhRJPd6gvY7djzzHKJlvaOzjwl53QIopozFjCpg
         nZwA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695190569; x=1695795369;
        h=in-reply-to:content-disposition:mime-version:references:message-id
         :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date
         :message-id:reply-to;
        bh=uM+9IiWPEF3F1CPkHQIOgF++G8nvzeRdPlLs2FAEgto=;
        b=u+QdpEYsXKki2Xksw6ZJcdY+9XV1JrPYmpGweRQRRwcFrfnLO+vn4w98zb8ICzSef3
         6srcvr9z9y5xnzA9mFOClcn8iDehbvViOYUB9bc9gArTFv9ndX7wrXPtb1cr4mQuMcIx
         CZsGXeYW6wn+xDD5tViTFo0pEtpmHCtLZuJjJYApEv16SE7VnZkTMk2P9eTn4geaoA3D
         qkNCgf8a8r7ShQtd85QEwqtk9He1D8+cwmqfThnvV/pJnMC348UZpSKtTPZKMN30VLsZ
         0FH3UjNP4gCBIq2SAvrx71rnB3306yJ9dof5UluL7ARTn5GLfq3Es7XymnDceOoK67+e
         RNUw==
X-Gm-Message-State: AOJu0YxHgFrfN22Bg1QaEzIfzVPLL2Y5rOE8vg/kW0KkNroOejEvYW8K
	UriO7ZYnPaSYMzPuGZXQJiY=
X-Google-Smtp-Source: AGHT+IFVbF1eOipnMLYiid4ujwUEYA6kn8aUf4HxYKWDYyGs+fm3mvOU7pWqYg/+INrNOpuQUp5WlA==
X-Received: by 2002:a05:6a20:5486:b0:14d:7130:7bb3 with SMTP id i6-20020a056a20548600b0014d71307bb3mr2167328pzk.13.1695190569140;
        Tue, 19 Sep 2023 23:16:09 -0700 (PDT)
Received: from debian.me ([103.124.138.83])
        by smtp.gmail.com with ESMTPSA id b1-20020aa78701000000b0068b1149ea4dsm2010522pfo.69.2023.09.19.23.16.08
        (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
        Tue, 19 Sep 2023 23:16:08 -0700 (PDT)
Received: by debian.me (Postfix, from userid 1000)
	id 0837096EFACB; Wed, 20 Sep 2023 13:16:04 +0700 (WIB)
Date: Wed, 20 Sep 2023 13:16:04 +0700
From: Bagas Sanjaya <bagasdotme@gmail.com>
To: Martin Zaharinov <micron10@gmail.com>,
	Eric Dumazet <edumazet@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>, netdev <netdev@vger.kernel.org>,
	patchwork-bot+netdevbpf@kernel.org,
	Jakub Kicinski <kuba@kernel.org>,
	Stephen Hemminger <stephen@networkplumber.org>,
	kuba+netdrv@kernel.org, dsahern@gmail.com,
	Florian Westphal <fw@strlen.de>,
	Pablo Neira Ayuso <pablo@netfilter.org>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
Message-ID: <ZQqOJOa_qTwz_k0V@debian.me>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
 <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com>
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="y/g/orgXWo/aLrJK"
Content-Disposition: inline
In-Reply-To: <BB129799-E196-428C-909D-721670DD5E21@gmail.com>
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,
	RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net


--y/g/orgXWo/aLrJK
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Sep 20, 2023 at 09:05:10AM +0300, Martin Zaharinov wrote:
> > On 20 Sep 2023, at 6:59, Eric Dumazet <edumazet@google.com> wrote:
> > Again, your best route is a bisection.
>=20
> For now its not possible to make bisection , its hard to change kernel on=
 running machine =E2=80=A6
>=20

You have to do bisection, unfortunately. There is many guides there on
Internet. Or you can read Documentation/admin-guide/bug-bisect.rst.

Bye!

--=20
An old man doll... just what I always wanted! - Clara

--y/g/orgXWo/aLrJK
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iHUEABYKAB0WIQSSYQ6Cy7oyFNCHrUH2uYlJVVFOowUCZQqOHgAKCRD2uYlJVVFO
o6H2AQDDfFl6X30Q86X2rOc98F9gU/lVu6FnDR845N53s4w9agEAlh5FsabMfC+K
2INhhkgWBUvMFztEauEKf0FPMC6bmwg=
=L2ji
-----END PGP SIGNATURE-----

--y/g/orgXWo/aLrJK--

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78321522F
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 07:04:07 +0000 (UTC)
Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FE8190
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:04:05 -0700 (PDT)
Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-9adca291f99so642709366b.2
        for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:04:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695193444; x=1695798244; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=NtvuKxMbgejKPBCSdN6koZbRZKuD3bJTK6qH37Jo/rA=;
        b=Li2lBTmk3qYWYXoQV3QDN5oOPgwKT4LcKx6QKXKxDWLqLwsxjW6r6hdf81OBS/ngvD
         ajzXqpdhGfwLPYCDDb0p6vdIk5dplP/0E+Rm4mQX/dE8epWcBLkx9/NHWyWg/J5RxoRc
         GI1K7RBr8wWP6if/pHm+emhgOsG8+uF93L4mOjgZInod7IhpFdgfnwHY850FaLcA2JaA
         tb5jN4L9OxWCwVyr1WIBXSJFna4QG9sxR29guQ6fI5iboFhVEWX5JJQ67D8tkWE3rxZC
         Ya4a8cRdjHshiMxtZSebR5nMsw9HANJGDwOhknncBx2Hlz8NcGJDLbsZyz6HB02geU9g
         V4ZQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695193444; x=1695798244;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=NtvuKxMbgejKPBCSdN6koZbRZKuD3bJTK6qH37Jo/rA=;
        b=ADG9Wg2fTZVJGk5pqCwV6YN0MlIX7BIj5fWvvctWePHDHkcUzMbvCR2CSp2qOwgNof
         9q7XzbqBsk4daHr8sj4OAGxByvcxgeSymZZ4oC+M8EPsRErcLH8bj6lMnNF/SovrQt5w
         zwnW81p5Kgtd2ohO13tXvBQZRA3UwN6kl+UbfBRDH0lCKCV8x52M6ha3Soq2obHXMUqj
         oZBK/xSzZmw0Qr9eshZ3lTHKLW+u+sVP3p+PuUhc5wDJoxASl6flQkOwBEtO+sRJo7q9
         KfzPEC3D+ZFyDAVsDd5++IM5AIDpbGGP/o1hEyVPD9l5Ol77wEotBX+8InyFQt3YPxGo
         CMVw==
X-Gm-Message-State: AOJu0YxoxwINOsxyGa/MNpGvoDlkTIfahKXaExtW8TcH26liW7mc9Be0
	F/mRq4lh+MZJLn7BISAZUMk=
X-Google-Smtp-Source: AGHT+IE+N80KjJYnEnssVoj2rN/N9PgJ8iUIr9k8fx0LxI/XSAgsGMTpd1y3t/UgbT+OZgQ8s12f6g==
X-Received: by 2002:a17:906:20d9:b0:99b:4ed4:5527 with SMTP id c25-20020a17090620d900b0099b4ed45527mr1417172ejc.25.1695193443578;
        Wed, 20 Sep 2023 00:04:03 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id p8-20020a1709061b4800b0099c53c44083sm8859768ejg.79.2023.09.20.00.04.02
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Wed, 20 Sep 2023 00:04:03 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <ZQqOJOa_qTwz_k0V@debian.me>
Date: Wed, 20 Sep 2023 10:03:51 +0300
Cc: Eric Dumazet <edumazet@google.com>,
 Paolo Abeni <pabeni@redhat.com>,
 netdev <netdev@vger.kernel.org>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
 <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me>
To: Bagas Sanjaya <bagasdotme@gmail.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi

Ok on first see all is look come after in kernel 6.4 add : atomics: =
Provide rcuref - scalable reference counting  ( =
https://www.spinics.net/lists/linux-tip-commits/msg62042.html )

I check all running machine with kernel 6.4.2 is minimal and have same =
bug report.

i have fell machine with kernel 6.3.9 and not see problems there .

and the problem may be is allocate in this part :=20

[39651.444202] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[39651.444297] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[39651.444391] dst_release (./arch/x86/include/asm/preempt.h:95 =
./include/linux/rcuref.h:151 net/core/dst.c:166)
[39651.444487] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4158)
[39651.444582] ? nf_hook_slow (./include/linux/netfilter.h:143 =
net/netfilter/core.c:626)

may be changes in dst.c make problem , I'm guessing at the moment.=20

but in real with kernel 6.3 all is fine for now.

dst.c changes 6.3.9 > 6.5.4 :

--- linux-6.3.9/net/core/dst.c	2023-06-21 14:02:19.000000000 +0000
+++ linux-6.5.4/net/core/dst.c	2023-09-19 10:30:30.000000000 +0000
@@ -66,7 +66,8 @@ void dst_init(struct dst_entry *dst, str
 	dst->tclassid =3D 0;
 #endif
 	dst->lwtstate =3D NULL;
-	atomic_set(&dst->__refcnt, initial_ref);
+	rcuref_init(&dst->__rcuref, initial_ref);
+	INIT_LIST_HEAD(&dst->rt_uncached);
 	dst->__use =3D 0;
 	dst->lastuse =3D jiffies;
 	dst->flags =3D flags;
@@ -162,31 +163,15 @@ EXPORT_SYMBOL(dst_dev_put);

 void dst_release(struct dst_entry *dst)
 {
-	if (dst) {
-		int newrefcnt;
-
-		newrefcnt =3D atomic_dec_return(&dst->__refcnt);
-		if (WARN_ONCE(newrefcnt < 0, "dst_release underflow"))
-			net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
-					     __func__, dst, newrefcnt);
-		if (!newrefcnt)
-			call_rcu_hurry(&dst->rcu_head, dst_destroy_rcu);
-	}
+	if (dst && rcuref_put(&dst->__rcuref))
+		call_rcu_hurry(&dst->rcu_head, dst_destroy_rcu);
 }
 EXPORT_SYMBOL(dst_release);

 void dst_release_immediate(struct dst_entry *dst)
 {
-	if (dst) {
-		int newrefcnt;
-
-		newrefcnt =3D atomic_dec_return(&dst->__refcnt);
-		if (WARN_ONCE(newrefcnt < 0, "dst_release_immediate =
underflow"))
-			net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
-					     __func__, dst, newrefcnt);
-		if (!newrefcnt)
-			dst_destroy(dst);
-	}
+	if (dst && rcuref_put(&dst->__rcuref))
+		dst_destroy(dst);
 }
 EXPORT_SYMBOL(dst_release_immediate);



> On 20 Sep 2023, at 9:16, Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>=20
> On Wed, Sep 20, 2023 at 09:05:10AM +0300, Martin Zaharinov wrote:
>>> On 20 Sep 2023, at 6:59, Eric Dumazet <edumazet@google.com> wrote:
>>> Again, your best route is a bisection.
>>=20
>> For now its not possible to make bisection , its hard to change =
kernel on running machine =E2=80=A6
>>=20
>=20
> You have to do bisection, unfortunately. There is many guides there on
> Internet. Or you can read Documentation/admin-guide/bug-bisect.rst.
>=20
> Bye!
>=20
> --=20
> An old man doll... just what I always wanted! - Clara


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9321F1C26
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 07:26:10 +0000 (UTC)
Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 661B793
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:26:07 -0700 (PDT)
Received: by mail-qt1-x82b.google.com with SMTP id d75a77b69052e-415155b2796so204251cf.1
        for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:26:07 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20230601; t=1695194766; x=1695799566; darn=vger.kernel.org;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=PEvUSZYn2QdyoynMg+EbUUA3yvQOOos33taxf2vHQIw=;
        b=VLld+sfgqSbrEaAj8WjEWCic+Znovc3Tb9jSTTK+WO7TAUrD7MllVKEkl7AhT2SS2e
         99N6255fb0Lz7a/wm43F20di86fcVFRWuoETyiJGFXgjlWc6O8wx8EnVrKuOnMklOlVT
         QH11i8JNzk4EHaFiWZLnaV89+P4cqpHCwLR9b1IqGkUIhosbC00ElQgYHWxD3d0wlFVi
         re7vhZrWpQKjuIzkv7udjcpDTG2bR6bV4K7vZ0NagYHqvmHIk6/nYCyJ3xwyG6LkPDNH
         IFhQHjfL6Q06T/Mdd7D7z2awJAuEfT1RA2N8E8JsKlzX81bVINlubI0Sjdj84DPcY8b1
         mvrA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695194766; x=1695799566;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=PEvUSZYn2QdyoynMg+EbUUA3yvQOOos33taxf2vHQIw=;
        b=NmMW3YVcDo8nSrOoRq0Cw3WcdnsP3Y98CYX9YGCFTPSNl3u2s54zW+2Tujt+pFZciD
         AuMOolwKsYDN39c3YKdfVTAYU8MNr8ui13hnTD/NdeTCu5/kLlyKHnbv2GaWmrcgjRIV
         6+3Z8/cz2hCpcHx9bf9L6nIT0ybrYNYxxd/Eb4RvaBOUfUWcnks6nthGfkczJjzq6z53
         iZ1iKElISpUlH6gs8lxgLSbK06R2O0XPBXrN8pgXxdp/pT4uwDnaDl2RmsTk6/CnU3OM
         fwED5E44cKCdLWMGxyi9sdrpzEd8+kUyOjb6qmJ5rE7PwG6N8YUljdcB1iN48SeGr9v4
         SZdA==
X-Gm-Message-State: AOJu0YxYZU1s60pgxH6nSv/HbHdaEyngjQ66S9yWBIjlLLRxeWdbGtJ6
	QUhlugb1eXjGrtzt1uaYYfEpvDrYYXgRaK8vjDchPg==
X-Google-Smtp-Source: AGHT+IGZe0+AHSCXUf+SgOKBwQLdbHtbQfJ7eNqn8GI+tHK505xeyQLT3Ll0cidm+HcXMILmlgW4mbwtt/1P/3/b14A=
X-Received: by 2002:a05:622a:3cd:b0:410:88dc:21b with SMTP id
 k13-20020a05622a03cd00b0041088dc021bmr215251qtx.26.1695194766220; Wed, 20 Sep
 2023 00:26:06 -0700 (PDT)
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com> <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com> <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com> <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me> <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
In-Reply-To: <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
From: Eric Dumazet <edumazet@google.com>
Date: Wed, 20 Sep 2023 09:25:54 +0200
Message-ID: <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
To: Martin Zaharinov <micron10@gmail.com>
Cc: Bagas Sanjaya <bagasdotme@gmail.com>, Paolo Abeni <pabeni@redhat.com>, 
	netdev <netdev@vger.kernel.org>, patchwork-bot+netdevbpf@kernel.org, 
	Jakub Kicinski <kuba@kernel.org>, Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org, 
	dsahern@gmail.com, Florian Westphal <fw@strlen.de>, Pablo Neira Ayuso <pablo@netfilter.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,
	USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On Wed, Sep 20, 2023 at 9:04=E2=80=AFAM Martin Zaharinov <micron10@gmail.co=
m> wrote:
>
> Hi
>
> Ok on first see all is look come after in kernel 6.4 add : atomics: Provi=
de rcuref - scalable reference counting  ( https://www.spinics.net/lists/li=
nux-tip-commits/msg62042.html )
>
> I check all running machine with kernel 6.4.2 is minimal and have same bu=
g report.
>
> i have fell machine with kernel 6.3.9 and not see problems there .
>
> and the problem may be is allocate in this part :
>
> [39651.444202] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
> [39651.444297] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
> [39651.444391] dst_release (./arch/x86/include/asm/preempt.h:95 ./include=
/linux/rcuref.h:151 net/core/dst.c:166)
> [39651.444487] __dev_queue_xmit (./include/net/dst.h:283 net/core/dev.c:4=
158)
> [39651.444582] ? nf_hook_slow (./include/linux/netfilter.h:143 net/netfil=
ter/core.c:626)
>
> may be changes in dst.c make problem , I'm guessing at the moment.
>
> but in real with kernel 6.3 all is fine for now.
>
> dst.c changes 6.3.9 > 6.5.4 :

Then start a real bisection. This is going to be the last time I say it.

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE0091C26
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 07:29:31 +0000 (UTC)
Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B8C7C9
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:29:30 -0700 (PDT)
Received: by mail-qt1-x82a.google.com with SMTP id d75a77b69052e-41761e9181eso200381cf.1
        for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:29:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=google.com; s=20230601; t=1695194969; x=1695799769; darn=vger.kernel.org;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=g8CBGO1I/tDVZ3US3iYHvNzW5AeogXVILN8LHzMcHOE=;
        b=36hQ0jWZSsQBeD6E5KlLx1TUvzI+bH03QnUqNj9P8b/E7/6HewdS3EXQWPKxn/8JaA
         KlxiAv/oXo1ZQp49QAO/ZBrP3k6wOoS+ATdx8c4dckY9muR/V0CGXPU85J0Z4aLnLgvT
         8RLVcXYsTbsUoJ/E5r9yHigFfAGAPoQ4DEH233re14mpK6ZABG5AoFH84BsHfiNryv1X
         GycrAH89FPisN/nCkTyqFtsNUToa4x9gzPqle7MjpW3nBaKKW9mmGXAoWG85KwRlMKm4
         WcCKJ5rg1gpSzslZKysbeDxFhT1GoUWRwEINgQjoIPCpFU2Kr0NcfyAk94wR7EBm9aEE
         GR0w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695194969; x=1695799769;
        h=content-transfer-encoding:cc:to:subject:message-id:date:from
         :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=g8CBGO1I/tDVZ3US3iYHvNzW5AeogXVILN8LHzMcHOE=;
        b=oYBgp67uA2O1i6A9icFaHTg8V8MuB1C7OVOkXdgFN7L0lxnBebIMF4JZ2KksyB3VRP
         5MyEV4J5p6AY8k3zoBtd1+BXWbPtdPB8mv282zESalcyG4H36tNtuZPJPUGUmrXRkeRq
         ZJk+7USAMfiIr63C0U/js6PDzVkyM+GbjvFe8DqK37j28am1iDVgR3S1cIX8AGjNDuCG
         lznSmSWccqeXgEGEtddu3QCElzLGvS2DhnpavY8g38vDn696Pen7fHcOTvMZZEi2nRvN
         qBSaK0ccoT2LAdGiTIEOmV0razsmOnxon+LA6zng6WbPzIL9awFx29WkQDF1jdkGoA/Y
         Vwtw==
X-Gm-Message-State: AOJu0YxRD45h2BAfxPfErhWwFT7O53DdS1SEGHvvhVOWpR3lR5OSl25W
	zhUsSFTbqi49GvDE1yewLAqwNUmf3a7PIEsJzmX7lQ==
X-Google-Smtp-Source: AGHT+IEmNp7iYwhpX0ipMJu8lXWl+bUUlXj6MEdDbPhow8vB4O+lJiSTl9l3uW61zvFNRVU9Q20UOWBKTvIEvkawyVo=
X-Received: by 2002:a05:622a:1d4:b0:416:6784:bd60 with SMTP id
 t20-20020a05622a01d400b004166784bd60mr123929qtw.21.1695194969239; Wed, 20 Sep
 2023 00:29:29 -0700 (PDT)
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com> <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com> <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com> <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me>
 <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com> <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
In-Reply-To: <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
From: Eric Dumazet <edumazet@google.com>
Date: Wed, 20 Sep 2023 09:29:18 +0200
Message-ID: <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
To: Martin Zaharinov <micron10@gmail.com>
Cc: Bagas Sanjaya <bagasdotme@gmail.com>, Paolo Abeni <pabeni@redhat.com>, 
	netdev <netdev@vger.kernel.org>, patchwork-bot+netdevbpf@kernel.org, 
	Jakub Kicinski <kuba@kernel.org>, Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org, 
	dsahern@gmail.com, Florian Westphal <fw@strlen.de>, Pablo Neira Ayuso <pablo@netfilter.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED,
	DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,
	ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,
	USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On Wed, Sep 20, 2023 at 9:25=E2=80=AFAM Eric Dumazet <edumazet@google.com> =
wrote:
>
> On Wed, Sep 20, 2023 at 9:04=E2=80=AFAM Martin Zaharinov <micron10@gmail.=
com> wrote:
> >
> > Hi
> >
> > Ok on first see all is look come after in kernel 6.4 add : atomics: Pro=
vide rcuref - scalable reference counting  ( https://www.spinics.net/lists/=
linux-tip-commits/msg62042.html )
> >
> > I check all running machine with kernel 6.4.2 is minimal and have same =
bug report.
> >
> > i have fell machine with kernel 6.3.9 and not see problems there .
> >
> > and the problem may be is allocate in this part :
> >
> > [39651.444202] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1=
))
> > [39651.444297] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1=
))
> > [39651.444391] dst_release (./arch/x86/include/asm/preempt.h:95 ./inclu=
de/linux/rcuref.h:151 net/core/dst.c:166)
> > [39651.444487] __dev_queue_xmit (./include/net/dst.h:283 net/core/dev.c=
:4158)
> > [39651.444582] ? nf_hook_slow (./include/linux/netfilter.h:143 net/netf=
ilter/core.c:626)
> >
> > may be changes in dst.c make problem , I'm guessing at the moment.
> >
> > but in real with kernel 6.3 all is fine for now.
> >
> > dst.c changes 6.3.9 > 6.5.4 :
>
> Then start a real bisection. This is going to be the last time I say it.

Or stick to an older kernel for your production, and wait for others
to find the issue.

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA78A5395
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 07:32:23 +0000 (UTC)
Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17554CA
	for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:32:22 -0700 (PDT)
Received: by mail-ed1-x535.google.com with SMTP id 4fb4d7f45d1cf-532c3b4b55eso1782388a12.0
        for <netdev@vger.kernel.org>; Wed, 20 Sep 2023 00:32:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695195140; x=1695799940; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=pORGOvhMLEtGSnXrx5AOPrF1BTFbPR5dAcAmBCxlUrc=;
        b=boWPKxqGjZtnHLfkKWJaEylniWfxUwUGjt5oNtDz1ZvNmzjdVenQlyl9u1br5dA1Pm
         ShL+285MHR/mGl7oJzMcTyWd7k0QtVUPRw5dzXqpVSt689/0X4cJ/Sof7pOi/Cdyo8+K
         nidQVq5wpEw2SMBbIR7AdySbF1uKY4CwD5hn+r+cB2h9uuETOtVzbtMhoSLCjng78w7l
         LwuYMUifKXyLrsxuAGtBh2fr5FVKfzr3P8EvMNHF5ekF6FkZvKLaFpeKDv9BFttEre7m
         bcODX2kCgReVm4zaeaf2G4vqiYFx4v9ZQpRXcLl1i16r4wWrWHJjvX5mkIun6gJCS9W8
         CcBg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695195140; x=1695799940;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=pORGOvhMLEtGSnXrx5AOPrF1BTFbPR5dAcAmBCxlUrc=;
        b=e1qQAGH6/aGw+qNkmg970lsQlZUsNODVKTS1qG8K5K/QscK8kfadVvXfEI9UzS+ucv
         kLYGS3JjrIQgd9fhtyALVORx91Cns+Llb9tbX3mdmnmC2TS1oJ6fdYoPFpk/rH42ePFb
         EPlj6NmiLUBNdG6qB6NTnFtsPI4sjPOHJD63udU5U1fHtCQg+LwNfwtO+OZPLgv/uNMw
         Wwn3Pma91ZkzgfjPRjIhiB67O76DCTZX8PpvxXbx0jih/IZkt2zt4v6lteN8pqmT7jE3
         gyowH8xqt+m3ZbcbzvFDS1FhscwoD8x8uDi0wlOTHvFWqcRSBw0lLrjQxsFG7ho7lAEY
         0n0w==
X-Gm-Message-State: AOJu0Yxy2C3qq4FW1317VcyusKTNMn3lZcxMQq5WtTcfU8ws9S/PLHCH
	BTOkCNQBDD5iFf7279D5Le8=
X-Google-Smtp-Source: AGHT+IHwgOFGVySNsijIj5w8sy9ifh8lnGO8HWM4FkIxlVPCEOQ5wn2XYuQ+pi029Zdttf2PjGKIlA==
X-Received: by 2002:a05:6402:14d6:b0:532:c6d7:b93c with SMTP id f22-20020a05640214d600b00532c6d7b93cmr1607086edx.5.1695195140380;
        Wed, 20 Sep 2023 00:32:20 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id eh16-20020a0564020f9000b005256aaa6e7asm3924058edb.78.2023.09.20.00.32.19
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Wed, 20 Sep 2023 00:32:19 -0700 (PDT)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
Date: Wed, 20 Sep 2023 10:32:08 +0300
Cc: Bagas Sanjaya <bagasdotme@gmail.com>,
 Paolo Abeni <pabeni@redhat.com>,
 netdev <netdev@vger.kernel.org>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <85F1F301-BECA-4210-A81F-12CAEEC85FD7@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
 <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me>
 <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
 <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
 <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
To: Eric Dumazet <edumazet@google.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS
	autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

I will make this yes .

And will wait if any find fix in future release.

Thanks for your time Eric

m.



> On 20 Sep 2023, at 10:29, Eric Dumazet <edumazet@google.com> wrote:
>=20
> On Wed, Sep 20, 2023 at 9:25=E2=80=AFAM Eric Dumazet =
<edumazet@google.com> wrote:
>>=20
>> On Wed, Sep 20, 2023 at 9:04=E2=80=AFAM Martin Zaharinov =
<micron10@gmail.com> wrote:
>>>=20
>>> Hi
>>>=20
>>> Ok on first see all is look come after in kernel 6.4 add : atomics: =
Provide rcuref - scalable reference counting  ( =
https://www.spinics.net/lists/linux-tip-commits/msg62042.html )
>>>=20
>>> I check all running machine with kernel 6.4.2 is minimal and have =
same bug report.
>>>=20
>>> i have fell machine with kernel 6.3.9 and not see problems there .
>>>=20
>>> and the problem may be is allocate in this part :
>>>=20
>>> [39651.444202] ? rcuref_put_slowpath (lib/rcuref.c:267 =
(discriminator 1))
>>> [39651.444297] ? rcuref_put_slowpath (lib/rcuref.c:267 =
(discriminator 1))
>>> [39651.444391] dst_release (./arch/x86/include/asm/preempt.h:95 =
./include/linux/rcuref.h:151 net/core/dst.c:166)
>>> [39651.444487] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4158)
>>> [39651.444582] ? nf_hook_slow (./include/linux/netfilter.h:143 =
net/netfilter/core.c:626)
>>>=20
>>> may be changes in dst.c make problem , I'm guessing at the moment.
>>>=20
>>> but in real with kernel 6.3 all is fine for now.
>>>=20
>>> dst.c changes 6.3.9 > 6.5.4 :
>>=20
>> Then start a real bisection. This is going to be the last time I say =
it.
>=20
> Or stick to an older kernel for your production, and wait for others
> to find the issue.


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 326A517996
	for <netdev@vger.kernel.org>; Thu, 21 Sep 2023 17:09:59 +0000 (UTC)
Received: from mail-ot1-x336.google.com (mail-ot1-x336.google.com [IPv6:2607:f8b0:4864:20::336])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFA412120
	for <netdev@vger.kernel.org>; Thu, 21 Sep 2023 10:09:27 -0700 (PDT)
Received: by mail-ot1-x336.google.com with SMTP id 46e09a7af769-6beff322a97so741270a34.3
        for <netdev@vger.kernel.org>; Thu, 21 Sep 2023 10:09:27 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695316167; x=1695920967; darn=vger.kernel.org;
        h=content-transfer-encoding:in-reply-to:from:content-language
         :references:cc:to:subject:user-agent:mime-version:date:message-id
         :from:to:cc:subject:date:message-id:reply-to;
        bh=AVcvvu/JFjnu+QrVzMYimtmbCiKo0Cmt5bIvcu2c+Og=;
        b=bD6jowcgW8QgQ2k2FEMeCG0rV8F6NAktmgtxg27XWVqvB8Cs2p52G46RMzIFpo2oTa
         yySMReiWn1AE/tkypIqFEJYbfHMPqbK31IRMjtSv2ewvGAwJ1vKmIZQkEe1Eke3e5G5r
         tjnr/jZh+yomx6gMm+gR7cuyDMXJYKyy006vvf4rsarh2vlS3vosWu3yTdIupQRC0/OL
         tWZ2Urw677gGoGSqXOq8J16BmjtSg0qUA8cFhf7Z56YNt3oxdOR6j4lU6i4LzrGzwh44
         wxcCd5j2JCLRDrk0d3P9yb1TeBt0SatctVVkAj3MwdqjcTWB57PeU4BKy0GYqQusFL+/
         ciug==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695316167; x=1695920967;
        h=content-transfer-encoding:in-reply-to:from:content-language
         :references:cc:to:subject:user-agent:mime-version:date:message-id
         :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to;
        bh=AVcvvu/JFjnu+QrVzMYimtmbCiKo0Cmt5bIvcu2c+Og=;
        b=HOvzdN0/2qVaeNHQN4Q/F4RQA1s+Wh5GOeqLe4FcYDgZpYxX2z59m3H6PMhYUtJZDu
         K63Fc8YDYnagC8aGUuETvoJpGbjD2A/7541M5mG7IVWJflh4LSw0DqIwZR+qeGQZOhFe
         VoseOn8847RodnZ8ZGzgASpExKpplmgoD8PXBT3N9MRK+yc3VUxe+Es8qlAylTgEEMP5
         dySg6bpgL1d7f5uMjYxiRTIUoHurClTpXhsCLXIOYiQe2FJpE/FuGkqO5K06oFvpgVmj
         hpWk+q67oQkKxjq3RSwxIJMv3Ko5b758X8Il11PAIabCzMB6j+tKUHhxWK8Lw+kDem8x
         h4rw==
X-Gm-Message-State: AOJu0YxjKAg4zgmuO1S6+iay2MUhpkYPewiy1yPl2xrtkfIg64zWn3oC
	HLespkt4MHUTyWLKmBqH4zkOAV3UOEs=
X-Google-Smtp-Source: AGHT+IG6oddQ51epnwGypAV5ylkotDRLWmvf1cAm59XtvXlGEBE+kRUDCVW24cX245cOZi/WF3b9zA==
X-Received: by 2002:a17:90a:4f4b:b0:26d:43f0:7ce4 with SMTP id w11-20020a17090a4f4b00b0026d43f07ce4mr5030413pjl.8.1695282654322;
        Thu, 21 Sep 2023 00:50:54 -0700 (PDT)
Received: from [192.168.0.106] ([103.124.138.83])
        by smtp.gmail.com with ESMTPSA id jx4-20020a17090b46c400b00274a43c3414sm2420784pjb.47.2023.09.21.00.50.50
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Thu, 21 Sep 2023 00:50:53 -0700 (PDT)
Message-ID: <be58d429-90d1-42ff-a36b-da318db6ee68@gmail.com>
Date: Thu, 21 Sep 2023 14:50:47 +0700
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
To: Martin Zaharinov <micron10@gmail.com>, Eric Dumazet <edumazet@google.com>
Cc: Paolo Abeni <pabeni@redhat.com>, netdev <netdev@vger.kernel.org>,
 patchwork-bot+netdevbpf@kernel.org, Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org,
 dsahern@gmail.com, Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
 <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me>
 <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
 <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
 <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
 <85F1F301-BECA-4210-A81F-12CAEEC85FD7@gmail.com>
Content-Language: en-US
From: Bagas Sanjaya <bagasdotme@gmail.com>
In-Reply-To: <85F1F301-BECA-4210-A81F-12CAEEC85FD7@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,
	RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

On 20/09/2023 14:32, Martin Zaharinov wrote:
> I will make this yes .
> 
> And will wait if any find fix in future release.
> 

Please don't top-post; reply inline with appropriate context instead.

Martin, what prevents you from doing bisection as Eric requested again?
If you only have production systems, why can't you afford to have
testing ones? Why not turning one of your prod machines to be testing
and bisect from there?

Sorry for inconvenience.

-- 
An old man doll... just what I always wanted! - Clara


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from lindbergh.monkeyblade.net (lindbergh.monkeyblade.net [23.128.96.19])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CC6A19BDD
	for <netdev@vger.kernel.org>; Thu, 21 Sep 2023 17:53:06 +0000 (UTC)
Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEFF4101AA
	for <netdev@vger.kernel.org>; Thu, 21 Sep 2023 10:52:30 -0700 (PDT)
Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-405361bb949so8345175e9.1
        for <netdev@vger.kernel.org>; Thu, 21 Sep 2023 10:52:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1695318748; x=1695923548; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=8GHbPt+uRiGazWwzPgy+ZoJBtbPuuy9FvR+oJREusTo=;
        b=H0aNzzYumxOiqq6nB58LAQuWcNLxF98CIBeFARjuyjl1Cr/YSd0oHCIqfhya9X9wOJ
         +boGagWWFYBBYVYfaxCw+o8YSI/TqdmJOiXaBa88cJ3Wtsbt8l9a38a2f9C4giDOmQqK
         D4pdAHV47mj/vLdMypVFf5IcNkgVVK9AsiPTaNtLnH40OYdVDvMOuuKJTAynF4S+jdtv
         hplVpJ6+IQldHKXiNbE82zW4uhclZXnmsc3WuKtbviUqkct5eWe5igjTp7G3VHLfPUF1
         gXMYaBTNjuglE1CpyJWrgCOYbZUkKPtfvhv5ryX5y1nP8b+a3v0nLHwfxbFWdN+iES48
         oFcw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1695318748; x=1695923548;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=8GHbPt+uRiGazWwzPgy+ZoJBtbPuuy9FvR+oJREusTo=;
        b=QW3xi2h/QaaMY1Ig3OPgC345hdMVlPriq+Jiswc28pUUaex1HCENeJx4exbxWq8ln0
         06gzSto+y+7ASCBLNo+bZ+x37g7WEOXbd01XPGWEpPTfgmZCys3LJ//Fj3ixDhQjJpxn
         U3kw4zerJDVb437s4zNrZVkXJi1GdbovUgViBzGM6VGxNBAWxIW5iTUyFPayScrWRFgm
         IUieFlW6VPMnXAeiRw5fmTElF2r7df/4JZ88nhSo1mUV7rtoGzsIveEQRVRJQAXheoBp
         V1q6ElmCvm9CSOcyk3WuPJkeyFp9czU/c+kIDwl1isSIZMRmRytZZ/h67cNEeBpB/a5d
         CvNg==
X-Gm-Message-State: AOJu0YydfWbIY1FPjay8pZO+zWBkiY+4DZ3qCOebLnPOKOrMSuedPBXn
	l/QjLdlmPJ7MA89L9RkwTCdCBUjWu94=
X-Google-Smtp-Source: AGHT+IHk6u2vH42kLc028jxZT7Xd8OecT2LXANAPrJi3C18LEbFJLa0bcZxvUBxJHM9dO3gn5lCZTQ==
X-Received: by 2002:a17:906:4d:b0:9aa:338:4a0f with SMTP id 13-20020a170906004d00b009aa03384a0fmr3877225ejg.72.1695284048245;
        Thu, 21 Sep 2023 01:14:08 -0700 (PDT)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id h13-20020a170906584d00b00992b8d56f3asm666963ejs.105.2023.09.21.01.14.07
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Thu, 21 Sep 2023 01:14:07 -0700 (PDT)
Content-Type: text/plain;
	charset=us-ascii
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <be58d429-90d1-42ff-a36b-da318db6ee68@gmail.com>
Date: Thu, 21 Sep 2023 11:13:55 +0300
Cc: Eric Dumazet <edumazet@google.com>,
 Paolo Abeni <pabeni@redhat.com>,
 netdev <netdev@vger.kernel.org>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Florian Westphal <fw@strlen.de>,
 Pablo Neira Ayuso <pablo@netfilter.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <6A98504D-DB99-42A5-A829-B81739822CB2@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <51294220-A244-46A9-A5B8-34819CE30CF4@gmail.com>
 <67303CFE-1938-4510-B9AE-5038BF98ABB7@gmail.com>
 <8a62f57a9454b0592ab82248fca5a21fc963995b.camel@redhat.com>
 <CALidq=UR=3rOHZczCnb1bEhbt9So60UZ5y60Cdh4aP41FkB5Tw@mail.gmail.com>
 <43ED0333-18AB-4C38-A615-7755E5BE9C3E@gmail.com>
 <5A853CC5-F15C-4F30-B845-D9E5B43EC039@gmail.com>
 <A416E134-BFAA-45FE-9061-9545F6DCC246@gmail.com>
 <CANn89iKXxyAQG-N+mdhNA8H+LEf=OK+goMFxYCV6yU1BpE=Xvw@mail.gmail.com>
 <BB129799-E196-428C-909D-721670DD5E21@gmail.com> <ZQqOJOa_qTwz_k0V@debian.me>
 <94BC75CD-A34A-4FED-A2EA-C18A28512230@gmail.com>
 <CANn89iKvv7F9G8AbYTEu7wca_SDHEp4GVTOEWk7_Yq0KFJrWgw@mail.gmail.com>
 <CANn89iJCJhJ=RWqPGkdbPoXhoa1W9ovi0s1t4242vsz-1=0WLw@mail.gmail.com>
 <85F1F301-BECA-4210-A81F-12CAEEC85FD7@gmail.com>
 <be58d429-90d1-42ff-a36b-da318db6ee68@gmail.com>
To: Bagas Sanjaya <bagasdotme@gmail.com>
X-Mailer: Apple Mail (2.3731.700.6)
X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT,
	FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=ham
	autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net

Hi Bagas,


Its not easy to make this on production, have too many users on it.

i make checks and find with kernel 6.3.12-6.5.13 all is fine.
on first machine that i have with kernel 6.4 and still work run kernel =
6.4.2 and have problem.

in my investigation problem is start after migration to kernel 6.4.x=20

in 6.4 kernel is add rcuref :=20

https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.4=20

commit bc9d3a9f2afca189a6ae40225b6985e3c775375e
Author: Thomas Gleixner <tglx@linutronix.de>
Date: Thu Mar 23 21:55:32 2023 +0100

net: dst: Switch to rcuref_t reference counting

Under high contention dst_entry::__refcnt becomes a significant =
bottleneck.

atomic_inc_not_zero() is implemented with a cmpxchg() loop, which goes =
into
high retry rates on contention.

Switch the reference count to rcuref_t which results in a significant
performance gain. Rename the reference count member to __rcuref to =
reflect
the change.

The gain depends on the micro-architecture and the number of concurrent
operations and has been measured in the range of +25% to +130% with a
localhost memtier/memcached benchmark which amplifies the problem
massively.

Running the memtier/memcached benchmark over a real (1Gb) network
connection the conversion on top of the false sharing fix for struct
dst_entry::__refcnt results in a total gain in the 2%-5% range over the
upstream baseline.

Reported-by: Wangyang Guo <wangyang.guo@intel.com>
Reported-by: Arjan Van De Ven <arjan.van.de.ven@intel.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Link: https://lore.kernel.org/r/20230307125538.989175656@linutronix.de
Link: https://lore.kernel.org/r/20230323102800.215027837@linutronix.de
Signed-off-by: Jakub Kicinski <kuba@kernel.org>


and i think problem is here :=20

--- a/net/core/dst.c
+++ b/net/core/dst.c
@@ -66,7 +66,7 @@ void dst_init(struct dst_entry *dst, str
dst->tclassid =3D 0;
#endif
dst->lwtstate =3D NULL;
- atomic_set(&dst->__refcnt, initial_ref);
+ rcuref_init(&dst->__refcnt, initial_ref);
dst->__use =3D 0;
dst->lastuse =3D jiffies;
dst->flags =3D flags;
@@ -162,31 +162,15 @@ EXPORT_SYMBOL(dst_dev_put);

void dst_release(struct dst_entry *dst)
{
- if (dst) {
- int newrefcnt;
-
- newrefcnt =3D atomic_dec_return(&dst->__refcnt);
- if (WARN_ONCE(newrefcnt < 0, "dst_release underflow"))
- net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
- __func__, dst, newrefcnt);
- if (!newrefcnt)
- call_rcu_hurry(&dst->rcu_head, dst_destroy_rcu);
- }
+ if (dst && rcuref_put(&dst->__refcnt))
+ call_rcu_hurry(&dst->rcu_head, dst_destroy_rcu);
}
EXPORT_SYMBOL(dst_release);

void dst_release_immediate(struct dst_entry *dst)
{
- if (dst) {
- int newrefcnt;
-
- newrefcnt =3D atomic_dec_return(&dst->__refcnt);
- if (WARN_ONCE(newrefcnt < 0, "dst_release_immediate underflow"))
- net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
- __func__, dst, newrefcnt);
- if (!newrefcnt)
- dst_destroy(dst);
- }
+ if (dst && rcuref_put(&dst->__refcnt))
+ dst_destroy(dst);
}
EXPORT_SYMBOL(dst_release_immediate);


but this is my thinking


Martin


> On 21 Sep 2023, at 10:50, Bagas Sanjaya <bagasdotme@gmail.com> wrote:
>=20
> On 20/09/2023 14:32, Martin Zaharinov wrote:
>> I will make this yes .
>>=20
>> And will wait if any find fix in future release.
>>=20
>=20
> Please don't top-post; reply inline with appropriate context instead.
>=20
> Martin, what prevents you from doing bisection as Eric requested =
again?
> If you only have production systems, why can't you afford to have
> testing ones? Why not turning one of your prod machines to be testing
> and bisect from there?
>=20
> Sorry for inconvenience.
>=20
> --=20
> An old man doll... just what I always wanted! - Clara
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XBfp7nKY"
Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9870698
	for <netdev@vger.kernel.org>; Thu, 16 Nov 2023 06:17:50 -0800 (PST)
Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-9f27af23441so122172166b.2
        for <netdev@vger.kernel.org>; Thu, 16 Nov 2023 06:17:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1700144269; x=1700749069; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=ZRCaV1gZrmP+/3CTX43FT7ldVw7K38kUlcOxcbp/9l8=;
        b=XBfp7nKY9Hrp3tchHkGoPpdrJWqvOakeYE212SpTrM+q4HtQa5Ttlw27onpdRivrda
         0UrdIPeOR//TKjzFdMcL+UAt8JycHDRsr1fXNoulqBhFbXeGqhREZiXXjYJtSYRLmgNX
         1hFfyhdw1X3efgxd4wzgxYQoH8cwBBQAv5HygpCFitfhVvHxRvUr24fN8Qt53kx+MU6F
         +9biFcqTwHU2W22bSnsT7FqmWXxqno4emfc2rwG3paWZflYRUfrDsE47PUzzdqtiAETU
         X0SwnHc6fSOXhUoFhekPZgFkh8TBf2wwik3taqpykABZy69e9zZ3kro+0eqL7mPuPJAh
         ViEA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1700144269; x=1700749069;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=ZRCaV1gZrmP+/3CTX43FT7ldVw7K38kUlcOxcbp/9l8=;
        b=tM97a7xqroNpucYRU5DLHgnsvL+EkEgg840Ya18m6FgGl6OEoXC3lgEzxl80HlzZkB
         jv545snhxw13SAcnBlYYAvS0gdBc4hrXtSBjTr0ceGHNJKiprL2ed14kSSpVk6WtcuBp
         S67veLEGFP9SJKEpxsonDdpNkcGAn3vzKvEAWTzSUyy8pNFXQ5nuiN1Egh7J7/GSv/uj
         0KKd/6HjuqDLCBAzGST3Ogb65a4yRZ1cSf/C4IxpHTAp4Sb5YRcx7VfYbwVC4Ej6sb06
         HXCTjxkL/s0eP6E3VuUr8GHWNtr0k9NwJBWANkqjyg7crVDaEph6YwjJqOVa5EL5yu3A
         BXVQ==
X-Gm-Message-State: AOJu0YzQVooXPGU8rlmwjCue2eMJf879TZmQc3w/dtBL5g+hNA7nONRv
	WmalJykO7y27Pm6SwWTIO+LayiB09HA=
X-Google-Smtp-Source: AGHT+IHQsbHwQWpG7l2hYRnIObjwYqCPGjH1FyTJkCX1NKtNBcLa6zMoPT/Bt+Z35n6h6E+yUIBeXw==
X-Received: by 2002:a17:906:1d0b:b0:9be:aebc:d480 with SMTP id n11-20020a1709061d0b00b009beaebcd480mr13647658ejh.24.1700144268774;
        Thu, 16 Nov 2023 06:17:48 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id ck20-20020a170906c45400b0099ce188be7fsm8448278ejb.3.2023.11.16.06.17.47
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Thu, 16 Nov 2023 06:17:48 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
Date: Thu, 16 Nov 2023 16:17:36 +0200
Cc: netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
To: Eric Dumazet <edumazet@google.com>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi All

report same problem with kernel 6.6.1 - i think problem is in rcu but =
=E2=80=A6 if have options to add people from RCU here.

See report :=20



[141229.505339] ------------[ cut here ]------------
[141229.505492] rcuref - imbalanced put()
[141229.505504] WARNING: CPU: 8 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
[141229.505821] Modules linked in: xsk_diag unix_diag iptable_filter =
xt_TCPMSS iptable_mangle xt_addrtype xt_nat xt_MASQUERADE iptable_nat =
ip_tables netconsole coretemp e1000 ixgbe mdio pppoe pppox sha1_ssse3 =
sha1_generic ppp_mppe libarc4 ppp_generic slhc nf_nat_sip =
nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4
[141229.506349] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G           O     =
  6.6.1 #1
[141229.506527] Hardware name: Persy Super Server/X11DDW-L, BIOS 4.0 =
07/11/2023
[141229.506701] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:267 =
(discriminator 1))
[141229.506843] Code: 31 c0 eb e2 80 3d ef 4e e6 00 00 74 0a c7 03 00 00 =
00 e0 31 c0 eb cf 48 c7 c7 07 99 e3 97 c6 05 d5 4e e6 00 01 e8 d1 1f c7 =
ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	31 c0                	xor    %eax,%eax
   2:	eb e2                	jmp    0xffffffffffffffe6
   4:	80 3d ef 4e e6 00 00 	cmpb   $0x0,0xe64eef(%rip)        # =
0xe64efa
   b:	74 0a                	je     0x17
   d:	c7 03 00 00 00 e0    	movl   $0xe0000000,(%rbx)
  13:	31 c0                	xor    %eax,%eax
  15:	eb cf                	jmp    0xffffffffffffffe6
  17:	48 c7 c7 07 99 e3 97 	mov    $0xffffffff97e39907,%rdi
  1e:	c6 05 d5 4e e6 00 01 	movb   $0x1,0xe64ed5(%rip)        # =
0xe64efa
  25:	e8 d1 1f c7 ff       	call   0xffffffffffc71ffb
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb df                	jmp    0xd
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	cc                   	int3
  33:	cc                   	int3
  34:	cc                   	int3
  35:	cc                   	int3
  36:	cc                   	int3
  37:	cc                   	int3
  38:	cc                   	int3
  39:	cc                   	int3
  3a:	cc                   	int3
  3b:	48 89 fa             	mov    %rdi,%rdx
  3e:	83                   	.byte 0x83
  3f:	e2                   	.byte 0xe2

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	eb df                	jmp    0xffffffffffffffe3
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	cc                   	int3
   9:	cc                   	int3
   a:	cc                   	int3
   b:	cc                   	int3
   c:	cc                   	int3
   d:	cc                   	int3
   e:	cc                   	int3
   f:	cc                   	int3
  10:	cc                   	int3
  11:	48 89 fa             	mov    %rdi,%rdx
  14:	83                   	.byte 0x83
  15:	e2                   	.byte 0xe2
[141229.507086] RSP: 0018:ffffa444449e0978 EFLAGS: 00010296
[141229.507229] RAX: 0000000000000019 RBX: ffff9b54866a4100 RCX: =
00000000fff7ffff
[141229.507404] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
[141229.507577] RBP: ffff9b53e57b1ec0 R08: 0000000000000000 R09: =
00000000fff7ffff
[141229.507751] R10: ffff9b62db200000 R11: 0000000000000003 R12: =
ffff9b5b0595c000
[141229.507929] R13: ffff9b5b09c32200 R14: ffff9b5b09e29a00 R15: =
ffff9b5b0557e080
[141229.508101] FS:  0000000000000000(0000) GS:ffff9b62dfa00000(0000) =
knlGS:0000000000000000
[141229.508279] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[141229.508425] CR2: 00007fbadced6a80 CR3: 000000096f014002 CR4: =
00000000003706e0
[141229.508599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[141229.508773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[141229.508947] Call Trace:
[141229.509079]  <IRQ>
[141229.509206] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
[141229.509342] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[141229.509482] ? handle_bug (arch/x86/kernel/traps.c:237)
[141229.509617] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
[141229.509751] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[141229.509892] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[141229.510028] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
[141229.510164] dst_release (./arch/x86/include/asm/preempt.h:95 =
./include/linux/rcuref.h:151 net/core/dst.c:166)
[141229.510302] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4324)
[141229.510441] vlan_dev_hard_start_xmit (net/8021q/vlan_dev.c:130)
[141229.510584] dev_hard_start_xmit (./include/linux/netdevice.h:4904 =
net/core/dev.c:3573 net/core/dev.c:3589)
[141229.510722] __dev_queue_xmit (./include/linux/netdevice.h:3278 =
(discriminator 25) net/core/dev.c:4370 (discriminator 25))
[141229.510862] ? eth_header (net/ethernet/eth.c:85)
[141229.510998] ip_finish_output2 (./include/net/neighbour.h:542 =
(discriminator 2) net/ipv4/ip_output.c:233 (discriminator 2))
[141229.511135] ip_sabotage_in (net/bridge/br_netfilter_hooks.c:881 =
net/bridge/br_netfilter_hooks.c:866)
[141229.511269] nf_hook_slow (./include/linux/netfilter.h:144 =
net/netfilter/core.c:626)
[141229.511406] ip_rcv (./include/linux/netfilter.h:259 =
./include/linux/netfilter.h:302 net/ipv4/ip_input.c:569)
[141229.511540] ? ip_rcv_core.constprop.0 (net/ipv4/ip_input.c:436)
[141229.511678] netif_receive_skb (net/core/dev.c:5552 =
net/core/dev.c:5666 net/core/dev.c:5752 net/core/dev.c:5811)
[141229.511814] br_handle_frame_finish (net/bridge/br_input.c:216)
[141229.511954] ? br_pass_frame_up (net/bridge/br_input.c:75)
[141229.512092] br_nf_hook_thresh (net/bridge/br_netfilter_hooks.c:1051)
[141229.512227] ? br_pass_frame_up (net/bridge/br_input.c:75)
[141229.512363] br_nf_pre_routing_finish =
(net/bridge/br_netfilter_hooks.c:427)
[141229.512501] ? br_pass_frame_up (net/bridge/br_input.c:75)
[141229.512644] ? nf_nat_ipv4_pre_routing =
(net/netfilter/nf_nat_proto.c:656) nf_nat
[141229.512792] br_nf_pre_routing (net/bridge/br_netfilter_hooks.c:538)
[141229.512928] ? br_nf_hook_thresh =
(net/bridge/br_netfilter_hooks.c:354)
[141229.513061] br_handle_frame (./include/linux/netfilter.h:144 =
net/bridge/br_input.c:272 net/bridge/br_input.c:417)
[141229.513196] ? br_pass_frame_up (net/bridge/br_input.c:75)
[141229.513333] __netif_receive_skb_core.constprop.0 =
(net/core/dev.c:5446 (discriminator 1))
[141229.513475] ? ip_finish_output2 (net/ipv4/ip_output.c:243)
[141229.513613] process_backlog (net/core/dev.c:5551 net/core/dev.c:5666 =
net/core/dev.c:5994)
[141229.513749] __napi_poll (net/core/dev.c:6556)
[141229.513887] net_rx_action (net/core/dev.c:6625 net/core/dev.c:6756)
[141229.514023] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
[141229.514158] do_softirq (kernel/softirq.c:463 (discriminator 32) =
kernel/softirq.c:450 (discriminator 32))
[141229.514292]  </IRQ>
[141229.514420]  <TASK>
[141229.514548] flush_smp_call_function_queue =
(./arch/x86/include/asm/irqflags.h:134 (discriminator 1) =
kernel/smp.c:579 (discriminator 1))
[141229.514688] do_idle (kernel/sched/idle.c:314)
[141229.514822] cpu_startup_entry (kernel/sched/idle.c:379)
[141229.516148] start_secondary (arch/x86/kernel/smpboot.c:326)
[141229.516291] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
[141229.516435]  </TASK>
[141229.516562] ---[ end trace 0000000000000000 ]=E2=80=94


Best regards,
Martin



> On 15 Sep 2023, at 9:45, Eric Dumazet <edumazet@google.com> wrote:
>=20
> scripts/decode_stacktrace.sh



From mboxrd@z Thu Jan  1 00:00:00 1970
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="a/47k6zS"
Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21CD6D44
	for <netdev@vger.kernel.org>; Wed,  6 Dec 2023 14:26:53 -0800 (PST)
Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-40c09f4814eso4295265e9.1
        for <netdev@vger.kernel.org>; Wed, 06 Dec 2023 14:26:53 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1701901611; x=1702506411; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=I8IQ04I+iRm3dx5k4NwzAHzUj1mzQpXP0awf5WuAlvw=;
        b=a/47k6zSaYoibI5r9r6TL+ycdpSXzCT6x0uS6Q7z34Ui9uRGJYU9hizcWwpGODrQNi
         GN5A4qAg2Ln0WspogN/1Hz3LKLUy87p7o3VMcwHAFsKGz0hnBTxqDiuOCpuAWifg9ano
         P5kMXiualcPg2Z2KLUp5s5WOtDU6Gi+/oAWfig61DlcTP0IbWRw097EONLS61Y2uQEFy
         g2Ecx28sg7nPeQqN/FPdYnOZfnAcVx/5cbyWy586w1GMaocltj39e6lS2uxqSJlZCOkj
         IJCpliBTstF8gMkL+3Ppq+qnTJs+tYD8LoFD8WpM2WDsDC4IqTgJwrJEcQPoSw9hGAdQ
         Wa0w==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1701901611; x=1702506411;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=I8IQ04I+iRm3dx5k4NwzAHzUj1mzQpXP0awf5WuAlvw=;
        b=WJDH+ik1QpsA6gq68vJrnKXS1r8Wpw74MyHcBpgWlvg7fX1AQm72jywqLpN/nsDDJC
         mZ1yB8NoFERjpO7X+N43MbA7l5xBCpGA0SapdcanlMCNatNIerdmnCeIGX42NOpVi/hR
         7BAhmLWXxZnA8d8Z52fC1QMkiSBLTue1shwYO+IG6EiL+KuT/yWCo3KLznmOqKkUPWqw
         TkSdU+bBRTZX5sgfc/Y8YippSRz6dLE4zw1vV5+aNr6iVjY+IFZp87CH/Wf8BgQqGiYY
         P9MczGKhdppp5+4D8c3bUXsprlt+nRi2e5Op6qqxad7SrCoyotM1Py3yfnU+f0ip1m0q
         FQHA==
X-Gm-Message-State: AOJu0YwL6/6Lb3t8dQ/50OQrrdjqWma0sgWU1xfHjdxai6i8pE38y8Sc
	tPWYBOwKHrNaL1UqPDHxOKA=
X-Google-Smtp-Source: AGHT+IEuX/V+ZCvsQTkr3A1Lus2EbnP8hbyVrWW4/pBHTdM6GywjeL0y8ar2ZxE677g724iHLpx/PA==
X-Received: by 2002:a05:600c:4f85:b0:40b:5e59:cc9d with SMTP id n5-20020a05600c4f8500b0040b5e59cc9dmr1129205wmq.126.1701901610966;
        Wed, 06 Dec 2023 14:26:50 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id g12-20020a05600c310c00b0040648217f4fsm966873wmo.39.2023.12.06.14.26.49
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Wed, 06 Dec 2023 14:26:49 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
Date: Thu, 7 Dec 2023 00:26:38 +0200
Cc: netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com
Content-Transfer-Encoding: quoted-printable
Message-Id: <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
To: Eric Dumazet <edumazet@google.com>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi all


its strange same problem is go on 6.6.4 same same debug log

diff hardware , users number and =E2=80=A6.

in debug log is same : lib/rcuref.c=20

in this line is :=20


        /*
         * If the reference count was already in the dead zone, then =
this
         * put() operation is imbalanced. Warn, put the reference count =
back to
         * DEAD and tell the caller to not deconstruct the object.
         */
        if (WARN_ONCE(cnt >=3D RCUREF_RELEASED, "rcuref - imbalanced =
put()")) {
                atomic_set(&ref->refcnt, RCUREF_DEAD);
                return false;
        }


[529520.875413] CPU: 13 PID: 0 Comm: swapper/13 Tainted: G           O   =
    6.6.3 #1
[529520.875533] Hardware name: Supermicro SYS-5038MR-H8TRF/X10SRD-F, =
BIOS 3.3 10/28/2020
[529520.875653] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
[529520.875748] Code: 31 c0 eb e2 80 3d 9e d1 e6 00 00 74 0a c7 03 00 00 =
00 e0 31 c0 eb cf 48 c7 c7 d9 96 e3 8f c6 05 84 d1 e6 00 01 e8 41 9d c7 =
ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 e2
[529520.875908] RSP: 0018:ffffa823c052cde8 EFLAGS: 00010296
[529520.876003] RAX: 0000000000000019 RBX: ffffa0f049053180 RCX: =
00000000fff7ffff
[529520.876122] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
[529520.876244] RBP: ffffa0f0a8fffec0 R08: 0000000000000000 R09: =
00000000fff7ffff
[529520.876364] R10: ffffa0f79ae00000 R11: 0000000000000003 R12: =
ffffa0f04655f000
[529520.876482] R13: 0000000000000258 R14: ffffa0f16ade1000 R15: =
ffffa0f79f964bd0
[529520.876601] FS:  0000000000000000(0000) GS:ffffa0f79f940000(0000) =
knlGS:0000000000000000
[529520.876723] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[529520.876822] CR2: 00007fa9bd56b3c8 CR3: 000000016e43e002 CR4: =
00000000003706e0
[529520.877043] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[529520.877164] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[529520.877287] Call Trace:
[529520.877382]  <IRQ>
[529520.877472]  ? __warn+0x6c/0x130
[529520.877566]  ? report_bug+0x1b8/0x200
[529520.877661]  ? handle_bug+0x36/0x70
[529520.877753]  ? exc_invalid_op+0x17/0x1a0
[529520.877849]  ? asm_exc_invalid_op+0x16/0x20
[529520.877947]  ? rcuref_put_slowpath+0x5f/0x70
[529520.878043]  ? rcuref_put_slowpath+0x5f/0x70
[529520.878136]  dst_release+0x1c/0x40
[529520.878229]  __dev_queue_xmit+0x594/0xcd0
[529520.878324]  ? eth_header+0x25/0xc0
[529520.878417]  ip_finish_output2+0x1a0/0x530
[529520.878514]  process_backlog+0x107/0x210
[529520.878610]  __napi_poll+0x20/0x180
[529520.878702]  net_rx_action+0x29f/0x380
[529520.878935]  __do_softirq+0xd0/0x202
[529520.879033]  do_softirq+0x3a/0x50
[529520.879127]  </IRQ>
[529520.879217]  <TASK>
[529520.879306]  flush_smp_call_function_queue+0x3f/0x50
[529520.879407]  do_idle+0x14d/0x210
[529520.879500]  cpu_startup_entry+0x21/0x30
[529520.879597]  start_secondary+0xe1/0xf0
[529520.879693]  secondary_startup_64_no_verify+0x166/0x16b
[529520.879793]  </TASK>
[529520.879884] ---[ end trace 0000000000000000 ]=E2=80=94


m.

> On 16 Nov 2023, at 16:17, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi All
>=20
> report same problem with kernel 6.6.1 - i think problem is in rcu but =
=E2=80=A6 if have options to add people from RCU here.
>=20
> See report :=20
>=20
>=20
>=20
> [141229.505339] ------------[ cut here ]------------
> [141229.505492] rcuref - imbalanced put()
> [141229.505504] WARNING: CPU: 8 PID: 0 at lib/rcuref.c:267 =
rcuref_put_slowpath (lib/rcuref.c:267 (discriminator 1))
> [141229.505821] Modules linked in: xsk_diag unix_diag iptable_filter =
xt_TCPMSS iptable_mangle xt_addrtype xt_nat xt_MASQUERADE iptable_nat =
ip_tables netconsole coretemp e1000 ixgbe mdio pppoe pppox sha1_ssse3 =
sha1_generic ppp_mppe libarc4 ppp_generic slhc nf_nat_sip =
nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4
> [141229.506349] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G           O   =
    6.6.1 #1
> [141229.506527] Hardware name: Persy Super Server/X11DDW-L, BIOS 4.0 =
07/11/2023
> [141229.506701] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:267 =
(discriminator 1))
> [141229.506843] Code: 31 c0 eb e2 80 3d ef 4e e6 00 00 74 0a c7 03 00 =
00 00 e0 31 c0 eb cf 48 c7 c7 07 99 e3 97 c6 05 d5 4e e6 00 01 e8 d1 1f =
c7 ff <0f> 0b eb df cc cc cc cc cc cc cc cc cc cc cc cc cc 48 89 fa 83 =
e2
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 31 c0                 xor    %eax,%eax
>   2: eb e2                 jmp    0xffffffffffffffe6
>   4: 80 3d ef 4e e6 00 00 cmpb   $0x0,0xe64eef(%rip)        # 0xe64efa
>   b: 74 0a                 je     0x17
>   d: c7 03 00 00 00 e0     movl   $0xe0000000,(%rbx)
>  13: 31 c0                 xor    %eax,%eax
>  15: eb cf                 jmp    0xffffffffffffffe6
>  17: 48 c7 c7 07 99 e3 97 mov    $0xffffffff97e39907,%rdi
>  1e: c6 05 d5 4e e6 00 01 movb   $0x1,0xe64ed5(%rip)        # 0xe64efa
>  25: e8 d1 1f c7 ff        call   0xffffffffffc71ffb
>  2a:* 0f 0b                 ud2     <-- trapping instruction
>  2c: eb df                 jmp    0xd
>  2e: cc                    int3
>  2f: cc                    int3
>  30: cc                    int3
>  31: cc                    int3
>  32: cc                    int3
>  33: cc                    int3
>  34: cc                    int3
>  35: cc                    int3
>  36: cc                    int3
>  37: cc                    int3
>  38: cc                    int3
>  39: cc                    int3
>  3a: cc                    int3
>  3b: 48 89 fa              mov    %rdi,%rdx
>  3e: 83                    .byte 0x83
>  3f: e2                    .byte 0xe2
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 0f 0b                 ud2
>   2: eb df                 jmp    0xffffffffffffffe3
>   4: cc                    int3
>   5: cc                    int3
>   6: cc                    int3
>   7: cc                    int3
>   8: cc                    int3
>   9: cc                    int3
>   a: cc                    int3
>   b: cc                    int3
>   c: cc                    int3
>   d: cc                    int3
>   e: cc                    int3
>   f: cc                    int3
>  10: cc                    int3
>  11: 48 89 fa              mov    %rdi,%rdx
>  14: 83                    .byte 0x83
>  15: e2                    .byte 0xe2
> [141229.507086] RSP: 0018:ffffa444449e0978 EFLAGS: 00010296
> [141229.507229] RAX: 0000000000000019 RBX: ffff9b54866a4100 RCX: =
00000000fff7ffff
> [141229.507404] RDX: 00000000fff7ffff RSI: 0000000000000001 RDI: =
00000000ffffffea
> [141229.507577] RBP: ffff9b53e57b1ec0 R08: 0000000000000000 R09: =
00000000fff7ffff
> [141229.507751] R10: ffff9b62db200000 R11: 0000000000000003 R12: =
ffff9b5b0595c000
> [141229.507929] R13: ffff9b5b09c32200 R14: ffff9b5b09e29a00 R15: =
ffff9b5b0557e080
> [141229.508101] FS:  0000000000000000(0000) GS:ffff9b62dfa00000(0000) =
knlGS:0000000000000000
> [141229.508279] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [141229.508425] CR2: 00007fbadced6a80 CR3: 000000096f014002 CR4: =
00000000003706e0
> [141229.508599] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
> [141229.508773] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
> [141229.508947] Call Trace:
> [141229.509079]  <IRQ>
> [141229.509206] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
> [141229.509342] ? report_bug (lib/bug.c:180 lib/bug.c:219)
> [141229.509482] ? handle_bug (arch/x86/kernel/traps.c:237)
> [141229.509617] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
> [141229.509751] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
> [141229.509892] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
> [141229.510028] ? rcuref_put_slowpath (lib/rcuref.c:267 (discriminator =
1))
> [141229.510164] dst_release (./arch/x86/include/asm/preempt.h:95 =
./include/linux/rcuref.h:151 net/core/dst.c:166)
> [141229.510302] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4324)
> [141229.510441] vlan_dev_hard_start_xmit (net/8021q/vlan_dev.c:130)
> [141229.510584] dev_hard_start_xmit (./include/linux/netdevice.h:4904 =
net/core/dev.c:3573 net/core/dev.c:3589)
> [141229.510722] __dev_queue_xmit (./include/linux/netdevice.h:3278 =
(discriminator 25) net/core/dev.c:4370 (discriminator 25))
> [141229.510862] ? eth_header (net/ethernet/eth.c:85)
> [141229.510998] ip_finish_output2 (./include/net/neighbour.h:542 =
(discriminator 2) net/ipv4/ip_output.c:233 (discriminator 2))
> [141229.511135] ip_sabotage_in (net/bridge/br_netfilter_hooks.c:881 =
net/bridge/br_netfilter_hooks.c:866)
> [141229.511269] nf_hook_slow (./include/linux/netfilter.h:144 =
net/netfilter/core.c:626)
> [141229.511406] ip_rcv (./include/linux/netfilter.h:259 =
./include/linux/netfilter.h:302 net/ipv4/ip_input.c:569)
> [141229.511540] ? ip_rcv_core.constprop.0 (net/ipv4/ip_input.c:436)
> [141229.511678] netif_receive_skb (net/core/dev.c:5552 =
net/core/dev.c:5666 net/core/dev.c:5752 net/core/dev.c:5811)
> [141229.511814] br_handle_frame_finish (net/bridge/br_input.c:216)
> [141229.511954] ? br_pass_frame_up (net/bridge/br_input.c:75)
> [141229.512092] br_nf_hook_thresh =
(net/bridge/br_netfilter_hooks.c:1051)
> [141229.512227] ? br_pass_frame_up (net/bridge/br_input.c:75)
> [141229.512363] br_nf_pre_routing_finish =
(net/bridge/br_netfilter_hooks.c:427)
> [141229.512501] ? br_pass_frame_up (net/bridge/br_input.c:75)
> [141229.512644] ? nf_nat_ipv4_pre_routing =
(net/netfilter/nf_nat_proto.c:656) nf_nat
> [141229.512792] br_nf_pre_routing =
(net/bridge/br_netfilter_hooks.c:538)
> [141229.512928] ? br_nf_hook_thresh =
(net/bridge/br_netfilter_hooks.c:354)
> [141229.513061] br_handle_frame (./include/linux/netfilter.h:144 =
net/bridge/br_input.c:272 net/bridge/br_input.c:417)
> [141229.513196] ? br_pass_frame_up (net/bridge/br_input.c:75)
> [141229.513333] __netif_receive_skb_core.constprop.0 =
(net/core/dev.c:5446 (discriminator 1))
> [141229.513475] ? ip_finish_output2 (net/ipv4/ip_output.c:243)
> [141229.513613] process_backlog (net/core/dev.c:5551 =
net/core/dev.c:5666 net/core/dev.c:5994)
> [141229.513749] __napi_poll (net/core/dev.c:6556)
> [141229.513887] net_rx_action (net/core/dev.c:6625 =
net/core/dev.c:6756)
> [141229.514023] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
> [141229.514158] do_softirq (kernel/softirq.c:463 (discriminator 32) =
kernel/softirq.c:450 (discriminator 32))
> [141229.514292]  </IRQ>
> [141229.514420]  <TASK>
> [141229.514548] flush_smp_call_function_queue =
(./arch/x86/include/asm/irqflags.h:134 (discriminator 1) =
kernel/smp.c:579 (discriminator 1))
> [141229.514688] do_idle (kernel/sched/idle.c:314)
> [141229.514822] cpu_startup_entry (kernel/sched/idle.c:379)
> [141229.516148] start_secondary (arch/x86/kernel/smpboot.c:326)
> [141229.516291] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
> [141229.516435]  </TASK>
> [141229.516562] ---[ end trace 0000000000000000 ]=E2=80=94
>=20
>=20
> Best regards,
> Martin
>=20
>=20
>=20
>> On 15 Sep 2023, at 9:45, Eric Dumazet <edumazet@google.com> wrote:
>>=20
>> scripts/decode_stacktrace.sh
>=20
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="uY4bAFKU";
	dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="RAVGDvtK"
Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A24310E0
	for <netdev@vger.kernel.org>; Fri,  8 Dec 2023 14:20:03 -0800 (PST)
From: Thomas Gleixner <tglx@linutronix.de>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de;
	s=2020; t=1702074001;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=e8jAcS2ppZ4k0tv8oAH166jXKDLK9pGQL2llNGvqKfA=;
	b=uY4bAFKUZFLWakieW/PAgNVjXvz3Uk23vueYEl5e8dxYVgJDdRYFPWENXeixdZ4Yh9aC3s
	9lKPhMlpQ5HhU4X+47KMnzTKqNzlSmNDRVerVKuVSa6ty6RZAhbSEzhFPFeq3/+S5MXZm+
	3mphO7JLeLP9YqZBXhPj0pO8F5cRsNm+YtAkuvuYqRDWcf38AuHuuPmQJztsxKJuTtziXP
	4/vGKp8Ao8x6+Biozw8jIdlS4UGnFIKhtvKpRPYELz4CCp4+TWCBKQQcZh+lhXWfPK9XGl
	2OoGY/CEaK6iHGAqw3F8Hrvb9WHCOePZHJtOV39cdMGT+EnzBojV3GtShBShTw==
DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de;
	s=2020e; t=1702074001;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=e8jAcS2ppZ4k0tv8oAH166jXKDLK9pGQL2llNGvqKfA=;
	b=RAVGDvtKiDiQr3HjNvacJRMMFY10zcWnfPQLkW/QPVR46ZPD57aQP/i0dljlMF+lyCnTj2
	RtVvrPWuSAmc+uCw==
To: Martin Zaharinov <micron10@gmail.com>, peterz@infradead.org
Cc: netdev <netdev@vger.kernel.org>, Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org, Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>, kuba+netdrv@kernel.org,
 dsahern@gmail.com, Eric Dumazet <edumazet@google.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
In-Reply-To: <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com>
Date: Fri, 08 Dec 2023 23:20:00 +0100
Message-ID: <87lea4qqun.ffs@tglx>
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Thu, Dec 07 2023 at 00:38, Martin Zaharinov wrote:
>> On 7 Dec 2023, at 0:26, Martin Zaharinov <micron10@gmail.com> wrote:
>>=20
>> in this line is :=20
>>=20
>>=20
>>        /*
>>         * If the reference count was already in the dead zone, then this
>>         * put() operation is imbalanced. Warn, put the reference count b=
ack to
>>         * DEAD and tell the caller to not deconstruct the object.
>>         */
>>        if (WARN_ONCE(cnt >=3D RCUREF_RELEASED, "rcuref - imbalanced put(=
)")) {
>>                atomic_set(&ref->refcnt, RCUREF_DEAD);
>>                return false;
>>        }

So a rcuref_put() operation triggers the warning because the reference
count is already dead, which means the rcuref_put() operation is
imbalanced.

>> [529520.875413] CPU: 13 PID: 0 Comm: swapper/13 Tainted: G           O  =
     6.6.3 #1

Can you reproduce this without the Out of Tree module?

>> [529520.875653] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
>> [529520.878136]  dst_release+0x1c/0x40
>> [529520.878229]  __dev_queue_xmit+0x594/0xcd0
>> [529520.878324]  ? eth_header+0x25/0xc0
>> [529520.878417]  ip_finish_output2+0x1a0/0x530
>> [529520.878514]  process_backlog+0x107/0x210
>> [529520.878610]  __napi_poll+0x20/0x180
>> [529520.878702]  net_rx_action+0x29f/0x380
>> [529520.878935]  __do_softirq+0xd0/0x202
>> [529520.879033]  do_softirq+0x3a/0x50

So this is one call chain triggering the issue...

>>> report same problem with kernel 6.6.1 - i think problem is in rcu
>>> but =E2=80=A6 if have options to add people from RCU here.

That's definitely not a RCU problem. It's a simple refcount fail.

Thanks,

        tglx


From mboxrd@z Thu Jan  1 00:00:00 1970
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eh50vvmS"
Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08EA810D2
	for <netdev@vger.kernel.org>; Fri,  8 Dec 2023 15:01:18 -0800 (PST)
Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-40c236624edso27208645e9.1
        for <netdev@vger.kernel.org>; Fri, 08 Dec 2023 15:01:17 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1702076476; x=1702681276; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=od2ApstYQJZccWNtkSJt62s+ojm3R/X5pGCZkNeuf6M=;
        b=eh50vvmSRgZaRYTWLibU+TdI8h+MRJhH9ROl+6S+Lgd2gNZBrbx/NWRmPAEU/N5wmJ
         5gSqpXgL75l2pGfqKRfX0BPwszubk+3l9dPz8sPxsOeKIFVI397uSOcrnXLDeUISRxFc
         t9RP1jK8mFXeU1aoDS1us/283f0IIysjLJk9+yI67KsVb/tejNiyURZh5zSGywX4Pp09
         vQ9+FpzzT1xEgDL3rf/G4jPppuku1G61C5rV9FCheIU2KvG94mbDk2qhVVniMVmnlKtE
         0ogs8AQ+NwWPUH5MpN/9xbNVX1m7Sg1A+vdOKMaw+r/FU4mhXNKI0PYGd7HEfSAWvRi1
         2jlw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1702076476; x=1702681276;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=od2ApstYQJZccWNtkSJt62s+ojm3R/X5pGCZkNeuf6M=;
        b=R+rce/R2+zYigLP86iIEpgK4F4lMZTb0DCk+ks8KgLNZ6hTgMFQvOukO1vukZw7aGh
         OMciWYB+u7pxV7RNesw1mxvm2lTLJAmKO/OGQ1O/IKZJQA93mFlQ7lRgvBDYEdDt9LVV
         1lQyUz+HiS9CALZH2F6XlRXl3MFzRYbz1vFn7xmLtLS9ed2yJokARuNzUUQcWygIjFWn
         Pn3GMzXy6eV7EriIRUqMl+2uqQbnpAiroQ2l1ZLNVVvUl22KLsM3llgfg9hJ/mNTTE0c
         3Uj09gTDIYOZS0H7lxqImHpjzmV/y4nJx5mCAesPMpGhCOT7BN6+R/W1TrDYRgQHAVPT
         w1oA==
X-Gm-Message-State: AOJu0YxLHHCLlz2qbLM1XI/qK6NApzolL+02LasMwYbj5p+aAGj+lRAB
	q9eew62WNV2hxUjymF8W8tQ=
X-Google-Smtp-Source: AGHT+IFCk962n+/NHp2aC9QcmNypY6ATu/gFtG9EKbO5RiKzOyElODpjnAEriWcJVS9hAa48MnQ/AQ==
X-Received: by 2002:a05:600c:181c:b0:40c:286e:d30e with SMTP id n28-20020a05600c181c00b0040c286ed30emr350270wmp.160.1702076476229;
        Fri, 08 Dec 2023 15:01:16 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id n10-20020a05600c500a00b004094e565e71sm4183060wmr.23.2023.12.08.15.01.15
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 08 Dec 2023 15:01:15 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <87lea4qqun.ffs@tglx>
Date: Sat, 9 Dec 2023 01:01:04 +0200
Cc: peterz@infradead.org,
 netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Eric Dumazet <edumazet@google.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
To: Thomas Gleixner <tglx@linutronix.de>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi Thomas,



> On 9 Dec 2023, at 0:20, Thomas Gleixner <tglx@linutronix.de> wrote:
>=20
> On Thu, Dec 07 2023 at 00:38, Martin Zaharinov wrote:
>>> On 7 Dec 2023, at 0:26, Martin Zaharinov <micron10@gmail.com> wrote:
>>>=20
>>> in this line is :=20
>>>=20
>>>=20
>>>       /*
>>>        * If the reference count was already in the dead zone, then =
this
>>>        * put() operation is imbalanced. Warn, put the reference =
count back to
>>>        * DEAD and tell the caller to not deconstruct the object.
>>>        */
>>>       if (WARN_ONCE(cnt >=3D RCUREF_RELEASED, "rcuref - imbalanced =
put()")) {
>>>               atomic_set(&ref->refcnt, RCUREF_DEAD);
>>>               return false;
>>>       }
>=20
> So a rcuref_put() operation triggers the warning because the reference
> count is already dead, which means the rcuref_put() operation is
> imbalanced.
>=20
>>> [529520.875413] CPU: 13 PID: 0 Comm: swapper/13 Tainted: G           =
O       6.6.3 #1
>=20
> Can you reproduce this without the Out of Tree module?
Same error without Out of Tree modules. i try many time from kernel =
6.5.x to now.

>=20
>>> [529520.875653] RIP: 0010:rcuref_put_slowpath+0x5f/0x70
>>> [529520.878136]  dst_release+0x1c/0x40
>>> [529520.878229]  __dev_queue_xmit+0x594/0xcd0
>>> [529520.878324]  ? eth_header+0x25/0xc0
>>> [529520.878417]  ip_finish_output2+0x1a0/0x530
>>> [529520.878514]  process_backlog+0x107/0x210
>>> [529520.878610]  __napi_poll+0x20/0x180
>>> [529520.878702]  net_rx_action+0x29f/0x380
>>> [529520.878935]  __do_softirq+0xd0/0x202
>>> [529520.879033]  do_softirq+0x3a/0x50
>=20
> So this is one call chain triggering the issue...
>=20
>>>> report same problem with kernel 6.6.1 - i think problem is in rcu
>>>> but =E2=80=A6 if have options to add people from RCU here.
>=20
> That's definitely not a RCU problem. It's a simple refcount fail.
>=20
> Thanks,
>=20
>        tglx
>=20

Is this a problem or only simple fail , and is it possible to catch what =
is a problem and fix this fail.

m.=

From mboxrd@z Thu Jan  1 00:00:00 1970
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="x/bDxY5x";
	dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="rqBOX7U3"
Received: from galois.linutronix.de (Galois.linutronix.de [IPv6:2a0a:51c0:0:12e:550::1])
	by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8DA40D0
	for <netdev@vger.kernel.org>; Tue, 12 Dec 2023 10:16:23 -0800 (PST)
From: Thomas Gleixner <tglx@linutronix.de>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de;
	s=2020; t=1702404981;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to:references:references;
	bh=vuaoZ4lMBkxsoPUY8HtPx04W3xi8KgehdLES/V07New=;
	b=x/bDxY5xPdBIf7Cg79rGjO45vZY0JbBbzscoURxAKST0KkpwZDVGtxS1QS0YUv6x3LRobM
	9scTt8/beU6Pix+zsWLOED5NTx6+MeCACaE1wDEIoe1RjXg9jMG9WImdrzJOpV/rSjs0pG
	m2+hT/Ys7qRBvZndaLiqVeyRw6EIt1RLg2YzbCNpS9EltekpQ5gPKXYGJMbAMuUfcsMeL2
	XZhuD9mW3tbCI/YcFKcvXtwXKYCeu3RuScWBNI8jS6CniOJATUNEZUZ8P+3OFy4YwLlDoo
	0Mj/fZvqssqOG/6I2oSIFeRS1oQFf61zNfN9QGqM3XSVJjM+TIMS73iBB3DKbQ==
DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de;
	s=2020e; t=1702404981;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 in-reply-to:in-reply-to:references:references;
	bh=vuaoZ4lMBkxsoPUY8HtPx04W3xi8KgehdLES/V07New=;
	b=rqBOX7U3LLFAi2SPwi0RjV2pLP/t8yqfloApcVJncMRRZMiWHBsVNfG9DkqrUOY6RySpO/
	oI7eo308rBuH43AQ==
To: Martin Zaharinov <micron10@gmail.com>
Cc: peterz@infradead.org, netdev <netdev@vger.kernel.org>, Paolo Abeni
 <pabeni@redhat.com>, patchwork-bot+netdevbpf@kernel.org, Jakub Kicinski
 <kuba@kernel.org>, Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org, dsahern@gmail.com, Eric Dumazet
 <edumazet@google.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
In-Reply-To: <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
 <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com>
Date: Tue, 12 Dec 2023 19:16:21 +0100
Message-ID: <87r0jrp9qi.ffs@tglx>
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain

Martin!

On Sat, Dec 09 2023 at 01:01, Martin Zaharinov wrote:
>> On 9 Dec 2023, at 0:20, Thomas Gleixner <tglx@linutronix.de> wrote:
>> That's definitely not a RCU problem. It's a simple refcount fail.
>> 
> Is this a problem or only simple fail , and is it possible to catch
> what is a problem and fix this fail.

Underaccounting a reference count is potentially Use After Free.

    if (rcuref_put(ref))
       call_rcu(ref....);

So after the grace period is over @ref will be freed. Depending on the
timing the context which does the extra put() might already operate on a
freed object.

How to catch that, that's a good question. There is no instrumentation
so far for this. Below is a straight forward trace_printk() based
tracking of rcurefs, which should help to narrow down the context.

Btw, how easy is this to reproduce?

Thanks,

        tglx
---
--- a/include/linux/rcuref.h
+++ b/include/linux/rcuref.h
@@ -64,8 +64,10 @@ static inline __must_check bool rcuref_g
 	 * Unconditionally increase the reference count. The saturation and
 	 * dead zones provide enough tolerance for this.
 	 */
-	if (likely(!atomic_add_negative_relaxed(1, &ref->refcnt)))
+	if (likely(!atomic_add_negative_relaxed(1, &ref->refcnt))) {
+		trace_printk("get(FASTPATH): %px\n", ref);
 		return true;
+	}
 
 	/* Handle the cases inside the saturation and dead zones */
 	return rcuref_get_slowpath(ref);
@@ -84,8 +86,10 @@ static __always_inline __must_check bool
 	 * Unconditionally decrease the reference count. The saturation and
 	 * dead zones provide enough tolerance for this.
 	 */
-	if (likely(!atomic_add_negative_release(-1, &ref->refcnt)))
+	if (likely(!atomic_add_negative_release(-1, &ref->refcnt))) {
+		trace_printk("put(FASTPATH): %px\n", ref);
 		return false;
+	}
 
 	/*
 	 * Handle the last reference drop and cases inside the saturation
--- a/lib/rcuref.c
+++ b/lib/rcuref.c
@@ -200,6 +200,7 @@ bool rcuref_get_slowpath(rcuref_t *ref)
 	 */
 	if (cnt >= RCUREF_RELEASED) {
 		atomic_set(&ref->refcnt, RCUREF_DEAD);
+		trace_printk("get(DEAD): %px %pS\n", ref, __builtin_return_address(0));
 		return false;
 	}
 
@@ -211,8 +212,15 @@ bool rcuref_get_slowpath(rcuref_t *ref)
 	 * object memory, but prevents the obvious reference count overflow
 	 * damage.
 	 */
-	if (WARN_ONCE(cnt > RCUREF_MAXREF, "rcuref saturated - leaking memory"))
+	if (cnt > RCUREF_MAXREF) {
+		trace_printk("get(SATURATED): %px %pS\n", ref, __builtin_return_address(0));
+		WARN_ONCE(1, "rcuref saturated - leaking memory");
 		atomic_set(&ref->refcnt, RCUREF_SATURATED);
+	} else {
+		trace_printk("get(UNDEFINED): %px %pS\n", ref, __builtin_return_address(0));
+		WARN_ON_ONCE(1);
+	}
+
 	return true;
 }
 EXPORT_SYMBOL_GPL(rcuref_get_slowpath);
@@ -248,9 +256,12 @@ bool rcuref_put_slowpath(rcuref_t *ref)
 		 * require a retry. If this fails the caller is not
 		 * allowed to deconstruct the object.
 		 */
-		if (!atomic_try_cmpxchg_release(&ref->refcnt, &cnt, RCUREF_DEAD))
+		if (!atomic_try_cmpxchg_release(&ref->refcnt, &cnt, RCUREF_DEAD)) {
+			trace_printk("put(NOTDEAD): %px %pS\n", ref, __builtin_return_address(0));
 			return false;
+		}
 
+		trace_printk("put(NOWDEAD): %px %pS\n", ref, __builtin_return_address(0));
 		/*
 		 * The caller can safely schedule the object for
 		 * deconstruction. Provide acquire ordering.
@@ -264,7 +275,9 @@ bool rcuref_put_slowpath(rcuref_t *ref)
 	 * put() operation is imbalanced. Warn, put the reference count back to
 	 * DEAD and tell the caller to not deconstruct the object.
 	 */
-	if (WARN_ONCE(cnt >= RCUREF_RELEASED, "rcuref - imbalanced put()")) {
+	if (cnt >= RCUREF_RELEASED) {
+		trace_printk("put(WASDEAD): %px %pS\n", ref, __builtin_return_address(0));
+		WARN_ONCE(1, "rcuref - imbalanced put()");
 		atomic_set(&ref->refcnt, RCUREF_DEAD);
 		return false;
 	}
@@ -274,8 +287,13 @@ bool rcuref_put_slowpath(rcuref_t *ref)
 	 * mean saturation value and tell the caller to not deconstruct the
 	 * object.
 	 */
-	if (cnt > RCUREF_MAXREF)
+	if (cnt > RCUREF_MAXREF) {
+		trace_printk("put(SATURATED): %px %pS\n", ref, __builtin_return_address(0));
 		atomic_set(&ref->refcnt, RCUREF_SATURATED);
+	} else {
+		trace_printk("put(UNDEFINED): %px %pS\n", ref, __builtin_return_address(0));
+		WARN_ON_ONCE(1);
+	}
 	return false;
 }
 EXPORT_SYMBOL_GPL(rcuref_put_slowpath);


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-ed1-f49.google.com (mail-ed1-f49.google.com [209.85.208.49])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1254C14274
	for <netdev@vger.kernel.org>; Tue, 19 Dec 2023 09:25:59 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="k8qRNa9H"
Received: by mail-ed1-f49.google.com with SMTP id 4fb4d7f45d1cf-553a45208ffso334721a12.3
        for <netdev@vger.kernel.org>; Tue, 19 Dec 2023 01:25:59 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1702977958; x=1703582758; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=/UmFXDzDjGX53ZCrVsZ8QohDcDki5KmmW0Faw7maqMs=;
        b=k8qRNa9HiqMj0ZERKy1gNBEuxhulkSWCTR8iyNTllWT5ioNYQhRfPCA7MHYgq3fVca
         j39pzLqd9iz7/eF8D9g8rI74c8/BVkukjorq7jnm/24hv35Dkt5xijs/uyc122as9p84
         JzqgMdzNz3MzPXtvqRdz7jrXAlqzLTosJvAmB59b2qiBxfU7LLHdxAvgXuOgKv6gKp47
         s7DnXFS/lqCGnzooDYhGf9aUjoQmjOoDF4ttmL6asmmDnHROv5lIG50ZNlY/L2GzdYZN
         2Qv6/arwfcA1VUXoNfY3Qyo7UodzGhnn1yF7kYY+2Lw4xffiKdymc6OSdIVvwXY1m1cO
         pQLQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1702977958; x=1703582758;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=/UmFXDzDjGX53ZCrVsZ8QohDcDki5KmmW0Faw7maqMs=;
        b=po1GzUIgCf7CYiMMqYf82WTO+o4tcy359Ci6PJIMZR6NkaTMQ/jM4LOSuKVEy9cHqR
         4+FKuFbDzzNuH+UuaNoN6GT/Fiihtww1LNSGsbF9fkVe0B9keDpoUctT7M4exvDqOdRi
         4sh4PSfJ/dl871RIPm9dEKg7ETRidiqRJ4B0G4HdVgVRT8ImFBJxJCNatErFxguv/ft/
         dwRl8Q3jLS2BsxHZZRJ982WtM+4DC/5hHQzCBERLX4HkUCP/3Btug1FlPp1atubT5ejZ
         xAONfAwiro8Gs+70StcDtAsE7zYX6SSrEVqB69wvNe6IG8SiWQl1fbXT3nmhXaPVvUyL
         Iu8g==
X-Gm-Message-State: AOJu0YwVFFnXDC0YvdMSk+nPo/amrr8tpHyUfQVdDuboo3nWVX3UWvXs
	HQ8pYIlhZjVCBQ45dcHCxjo=
X-Google-Smtp-Source: AGHT+IEhxmNdKHhKuxc2Gfo4ZgM30htzuVRT2i+X6+yQwwzJO0el7CvXvouTS79Gm/bKQ8V276hyMg==
X-Received: by 2002:a50:9313:0:b0:553:50d9:89e2 with SMTP id m19-20020a509313000000b0055350d989e2mr946969eda.19.1702977957963;
        Tue, 19 Dec 2023 01:25:57 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id c6-20020a0564021f8600b0055306f10c28sm3097478edc.28.2023.12.19.01.25.56
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Tue, 19 Dec 2023 01:25:57 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <87r0jrp9qi.ffs@tglx>
Date: Tue, 19 Dec 2023 11:25:46 +0200
Cc: peterz@infradead.org,
 netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Eric Dumazet <edumazet@google.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <6D816814-1334-4F22-AFF8-B5E42254038E@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
 <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com> <87r0jrp9qi.ffs@tglx>
To: Thomas Gleixner <tglx@linutronix.de>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi Thomas,
Thanks for your response!



> On 12 Dec 2023, at 20:16, Thomas Gleixner <tglx@linutronix.de> wrote:
>=20
> Martin!
>=20
> On Sat, Dec 09 2023 at 01:01, Martin Zaharinov wrote:
>>> On 9 Dec 2023, at 0:20, Thomas Gleixner <tglx@linutronix.de> wrote:
>>> That's definitely not a RCU problem. It's a simple refcount fail.
>>>=20
>> Is this a problem or only simple fail , and is it possible to catch
>> what is a problem and fix this fail.
>=20
> Underaccounting a reference count is potentially Use After Free.
>=20
>    if (rcuref_put(ref))
>       call_rcu(ref....);
>=20
> So after the grace period is over @ref will be freed. Depending on the
> timing the context which does the extra put() might already operate on =
a
> freed object.
>=20
> How to catch that, that's a good question. There is no instrumentation
> so far for this. Below is a straight forward trace_printk() based
> tracking of rcurefs, which should help to narrow down the context.
>=20
> Btw, how easy is this to reproduce?

Its not easy this report is generate on machine with 5-6k users , with =
traffic and one time is show on 1 day , other show after 4-5 days=E2=80=A6=





>=20
> Thanks,
>=20
>        tglx
> ---
> --- a/include/linux/rcuref.h
> +++ b/include/linux/rcuref.h
> @@ -64,8 +64,10 @@ static inline __must_check bool rcuref_g
> * Unconditionally increase the reference count. The saturation and
> * dead zones provide enough tolerance for this.
> */
> - if (likely(!atomic_add_negative_relaxed(1, &ref->refcnt)))
> + if (likely(!atomic_add_negative_relaxed(1, &ref->refcnt))) {
> + trace_printk("get(FASTPATH): %px\n", ref);
> return true;
> + }
>=20
> /* Handle the cases inside the saturation and dead zones */
> return rcuref_get_slowpath(ref);
> @@ -84,8 +86,10 @@ static __always_inline __must_check bool
> * Unconditionally decrease the reference count. The saturation and
> * dead zones provide enough tolerance for this.
> */
> - if (likely(!atomic_add_negative_release(-1, &ref->refcnt)))
> + if (likely(!atomic_add_negative_release(-1, &ref->refcnt))) {
> + trace_printk("put(FASTPATH): %px\n", ref);
> return false;
> + }
>=20
> /*
> * Handle the last reference drop and cases inside the saturation
> --- a/lib/rcuref.c
> +++ b/lib/rcuref.c
> @@ -200,6 +200,7 @@ bool rcuref_get_slowpath(rcuref_t *ref)
> */
> if (cnt >=3D RCUREF_RELEASED) {
> atomic_set(&ref->refcnt, RCUREF_DEAD);
> + trace_printk("get(DEAD): %px %pS\n", ref, =
__builtin_return_address(0));
> return false;
> }
>=20
> @@ -211,8 +212,15 @@ bool rcuref_get_slowpath(rcuref_t *ref)
> * object memory, but prevents the obvious reference count overflow
> * damage.
> */
> - if (WARN_ONCE(cnt > RCUREF_MAXREF, "rcuref saturated - leaking =
memory"))
> + if (cnt > RCUREF_MAXREF) {
> + trace_printk("get(SATURATED): %px %pS\n", ref, =
__builtin_return_address(0));
> + WARN_ONCE(1, "rcuref saturated - leaking memory");
> atomic_set(&ref->refcnt, RCUREF_SATURATED);
> + } else {
> + trace_printk("get(UNDEFINED): %px %pS\n", ref, =
__builtin_return_address(0));
> + WARN_ON_ONCE(1);
> + }
> +
> return true;
> }
> EXPORT_SYMBOL_GPL(rcuref_get_slowpath);
> @@ -248,9 +256,12 @@ bool rcuref_put_slowpath(rcuref_t *ref)
> * require a retry. If this fails the caller is not
> * allowed to deconstruct the object.
> */
> - if (!atomic_try_cmpxchg_release(&ref->refcnt, &cnt, RCUREF_DEAD))
> + if (!atomic_try_cmpxchg_release(&ref->refcnt, &cnt, RCUREF_DEAD)) {
> + trace_printk("put(NOTDEAD): %px %pS\n", ref, =
__builtin_return_address(0));
> return false;
> + }
>=20
> + trace_printk("put(NOWDEAD): %px %pS\n", ref, =
__builtin_return_address(0));
> /*
> * The caller can safely schedule the object for
> * deconstruction. Provide acquire ordering.
> @@ -264,7 +275,9 @@ bool rcuref_put_slowpath(rcuref_t *ref)
> * put() operation is imbalanced. Warn, put the reference count back to
> * DEAD and tell the caller to not deconstruct the object.
> */
> - if (WARN_ONCE(cnt >=3D RCUREF_RELEASED, "rcuref - imbalanced =
put()")) {
> + if (cnt >=3D RCUREF_RELEASED) {
> + trace_printk("put(WASDEAD): %px %pS\n", ref, =
__builtin_return_address(0));
> + WARN_ONCE(1, "rcuref - imbalanced put()");
> atomic_set(&ref->refcnt, RCUREF_DEAD);
> return false;
> }
> @@ -274,8 +287,13 @@ bool rcuref_put_slowpath(rcuref_t *ref)
> * mean saturation value and tell the caller to not deconstruct the
> * object.
> */
> - if (cnt > RCUREF_MAXREF)
> + if (cnt > RCUREF_MAXREF) {
> + trace_printk("put(SATURATED): %px %pS\n", ref, =
__builtin_return_address(0));
> atomic_set(&ref->refcnt, RCUREF_SATURATED);
> + } else {
> + trace_printk("put(UNDEFINED): %px %pS\n", ref, =
__builtin_return_address(0));
> + WARN_ON_ONCE(1);
> + }
> return false;
> }
> EXPORT_SYMBOL_GPL(rcuref_put_slowpath);

Apply this patch and will upload image on one machine as fast as =
possible and when get any reports will send you.

Best regards,
Martin


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCED71D525
	for <netdev@vger.kernel.org>; Tue, 19 Dec 2023 14:26:07 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linutronix.de
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linutronix.de
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="aIWyliap";
	dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="nQvuLWV0"
From: Thomas Gleixner <tglx@linutronix.de>
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de;
	s=2020; t=1702995965;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=7T+BVsISZ4M4K107mOguc7pWHjx5PSIHJSlTULmNLWI=;
	b=aIWyliapHoaRQBo0pQnSNU5xcdSrOQL4IezqOiISRVFLcb52AuSsXxTBxmXY/1SZ5GuV06
	UOo8OGcTWRjASfIuWj512S28KAAAUkeknrSIRwTQJwRiON0LGZxjDU8V3/V39Wz9YcCm8F
	bJHOuNEpI24b7+DthZAttPTe1t6hMESTBk2vdu6kCRcrn0wQMEwGobDE9sQZSJiQQA/jHH
	em5E33m8lJmzVrWwSq1KR1ayMvtUmdaBS8EPYtrs6/EkToSa1Q2RNINS2f8Etq2UazihTn
	KJWqrFO7StiMVDx/DWcfdROLtH2Ix7Q03Wblw9JDT8AbU14qlpexJVpCyMpSEA==
DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de;
	s=2020e; t=1702995965;
	h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
	 to:to:cc:cc:mime-version:mime-version:content-type:content-type:
	 content-transfer-encoding:content-transfer-encoding:
	 in-reply-to:in-reply-to:references:references;
	bh=7T+BVsISZ4M4K107mOguc7pWHjx5PSIHJSlTULmNLWI=;
	b=nQvuLWV04nLYxVFnZQ69kgSHL4M9M9msDAJJbUlSUuhYFnfPNskCyqV1b3zVPsa4FWkMW2
	MyDGThVm9Xg+QTBA==
To: Martin Zaharinov <micron10@gmail.com>
Cc: peterz@infradead.org, netdev <netdev@vger.kernel.org>, Paolo Abeni
 <pabeni@redhat.com>, patchwork-bot+netdevbpf@kernel.org, Jakub Kicinski
 <kuba@kernel.org>, Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org, dsahern@gmail.com, Eric Dumazet
 <edumazet@google.com>
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
In-Reply-To: <6D816814-1334-4F22-AFF8-B5E42254038E@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
 <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com> <87r0jrp9qi.ffs@tglx>
 <6D816814-1334-4F22-AFF8-B5E42254038E@gmail.com>
Date: Tue, 19 Dec 2023 15:26:04 +0100
Message-ID: <87v88ul14z.ffs@tglx>
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On Tue, Dec 19 2023 at 11:25, Martin Zaharinov wrote:
>> On 12 Dec 2023, at 20:16, Thomas Gleixner <tglx@linutronix.de> wrote:
>> Btw, how easy is this to reproduce?
>
> Its not easy this report is generate on machine with 5-6k users , with
> traffic and one time is show on 1 day , other show after 4-5 days=E2=80=A6

I love those bugs ...

> Apply this patch and will upload image on one machine as fast as
> possible and when get any reports will send you.

Let's see how that goes!

Thanks,

        tglx

From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8D9F28DBB
	for <netdev@vger.kernel.org>; Fri, 22 Dec 2023 17:26:56 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Vxo9x3uu"
Received: by mail-ej1-f48.google.com with SMTP id a640c23a62f3a-a235eb41251so253227366b.3
        for <netdev@vger.kernel.org>; Fri, 22 Dec 2023 09:26:56 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1703266015; x=1703870815; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=tBtABs6NA2NDnaUzXeX9xH2F54X7LgO8RIdh8vVHJdQ=;
        b=Vxo9x3uuE8UZVTRPZogOAUZ8BQZK2wd2qcSUKPV+rTQD0psL8LfZ9V/KiKPMe4Gztu
         eY7CRcnDE0nCU4FVBXIebo6nTZ8Va4fDuS3zlKqF0x/NZDWvu1j+gvw8x1zDJ486ts4q
         6liyDafO+k8Fn99Qv/Or7NHatUkrkre9P3OvQDB0TJQBCoa6OJcbj0BkHF1vM6+B7MsC
         G6k4gMypVvcxUjd9rlwI3JY/vpVGcpzBYCSQzNcacuhXyrGhBMjdiflwJrpRyeJzmAFc
         Ih507PcvfZBekbt7IPkJO6fnybl7Wv1Mpenzm6do5f5qXruqE6tcK9novy4WwUATiakX
         H4+A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1703266015; x=1703870815;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=tBtABs6NA2NDnaUzXeX9xH2F54X7LgO8RIdh8vVHJdQ=;
        b=iBuHNwjbf/2/d405WUl2l+tiC1tOl8y1PrpQiB4yWjx8wNLR5m4qgNVlo2WtNWeQTt
         bA+tZ2SrIgwlQ8gE6Y2ZsgTWGDyuk/XmousTM+Ban2GfJqIpFNhdK9Cr1IxGbNy0DMwx
         Huog/hcgCDWWGpDhVqKbIq8EMwnf4H9LjMuuzTXibggW2RYAoFR+zsqAEz3mucv4fhY5
         8R281m7idG2FE9AaK9Jk5OETk+txMC4UfAQZifN8Ysc0X+ptsiCjiCk89Tkk/O/602bP
         w2bsZMDhaZEMJzL3r1EOhuo/dVW/ojuSiXMsDugID4SsfmWzizaKcXv7pUvdOH7plLvp
         8r6g==
X-Gm-Message-State: AOJu0YzyHQHt6Zwi4qLcp//xmz4tZTfYmexOQ/Y2jb168uZmq8DjhH3y
	cqvfw/ibAKdRmVLPikkFrbc=
X-Google-Smtp-Source: AGHT+IH6NRmgpXmw2VKstw1pcjOM7IDHtIjW8bEPYKunP9wVvGQWxiSYa2K11yHtEQ8xJCjDUYm42g==
X-Received: by 2002:a17:906:225b:b0:a24:3546:489 with SMTP id 27-20020a170906225b00b00a2435460489mr581007ejr.127.1703266014773;
        Fri, 22 Dec 2023 09:26:54 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id mj25-20020a170906af9900b00a1ca6f5f189sm2267050ejb.179.2023.12.22.09.26.53
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 22 Dec 2023 09:26:54 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <87v88ul14z.ffs@tglx>
Date: Fri, 22 Dec 2023 19:26:42 +0200
Cc: peterz@infradead.org,
 netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Eric Dumazet <edumazet@google.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <FBF0EB18-E2B6-4076-968A-5F2ABE0F27E4@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
 <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com> <87r0jrp9qi.ffs@tglx>
 <6D816814-1334-4F22-AFF8-B5E42254038E@gmail.com> <87v88ul14z.ffs@tglx>
To: Thomas Gleixner <tglx@linutronix.de>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi Thomas,

 this is with applyed patch from you.
See logs


[43040.198064] ------------[ cut here ]------------
[43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
[43040.198685] Modules linked in: pppoe pppox ppp_generic slhc nft_limit =
nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb i2c_algo_bit =
e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe mdio =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler rtc_cmos
[43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           O    =
   6.6.8 #1
[43040.199660] Hardware name: VMware, Inc. VMware Virtual Platform/440BX =
Desktop Reference Platform, BIOS 6.00 11/12/2020
[43040.199886] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
[43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 f8 =
ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 78 =
06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 00
[43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
[43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
[43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
[43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
[43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
[43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
[43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
[43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
[43040.201994] Call Trace:
[43040.202095]  <IRQ>
[43040.202187]  ? __warn+0x6c/0x130
[43040.202301]  ? report_bug+0x1b8/0x200
[43040.202418]  ? handle_bug+0x36/0x70
[43040.202534]  ? exc_invalid_op+0x17/0x1a0
[43040.202652]  ? asm_exc_invalid_op+0x16/0x20
[43040.202781]  ? rcuref_put_slowpath+0x2f/0x70
[43040.202909]  dst_release+0x1c/0x40
[43040.203026]  rt_cache_route+0xbd/0xf0
[43040.203143]  rt_set_nexthop.isra.0+0x1b6/0x450
[43040.203272]  ip_route_input_slow+0x5d9/0xcc0
[43040.203401]  ? nf_conntrack_udp_packet+0x17c/0x240 [nf_conntrack]
[43040.203581]  ip_route_input_noref+0xe0/0xf0
[43040.203704]  ip_rcv_finish_core.isra.0+0xbb/0x440
[43040.203855]  ip_rcv+0xd5/0x110
[43040.203962]  ? ip_rcv_core+0x360/0x360
[43040.204079]  process_backlog+0x107/0x210
[43040.204201]  __napi_poll+0x20/0x180
[43040.204315]  net_rx_action+0x29f/0x380
[43040.204432]  __do_softirq+0xd0/0x202
[43040.204549]  irq_exit_rcu+0x82/0xa0
[43040.204667]  common_interrupt+0x7a/0xa0
[43040.204786]  </IRQ>
[43040.204876]  <TASK>
[43040.204965]  asm_common_interrupt+0x22/0x40
[43040.205090] RIP: 0010:acpi_safe_halt+0x1b/0x20
[43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 48 =
8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 fb =
f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb 9f
[43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
[43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
[43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
[43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
[43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
[43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
[43040.206593]  acpi_idle_enter+0x77/0xc0
[43040.206711]  cpuidle_enter_state+0x69/0x6a0
[43040.206835]  cpuidle_enter+0x24/0x40
[43040.206954]  do_idle+0x1a7/0x210
[43040.207066]  cpu_startup_entry+0x21/0x30
[43040.207188]  start_secondary+0xe1/0xf0
[43040.207310]  secondary_startup_64_no_verify+0x166/0x16b
[43040.207451]  </TASK>
[43040.207542] ---[ end trace 0000000000000000 ]---



[43040.198064] ------------[ cut here ]------------
[43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
[43040.198685] Modules linked in: pppoe pppox ppp_generic slhc nft_limit =
nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb i2c_algo_bit =
e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe mdio =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler rtc_cmos
[43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           O    =
   6.6.8 #1
[43040.199660] Hardware name: VMware, Inc. VMware Virtual Platform/440BX =
Desktop Reference Platform, BIOS 6.00 11/12/2020
[43040.199886] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
[43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 f8 =
ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 78 =
06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	07                   	(bad)
   1:	83 f8 ff             	cmp    $0xffffffff,%eax
   4:	75 19                	jne    0x1f
   6:	ba 00 00 00 e0       	mov    $0xe0000000,%edx
   b:	f0 0f b1 17          	lock cmpxchg %edx,(%rdi)
   f:	83 f8 ff             	cmp    $0xffffffff,%eax
  12:	74 04                	je     0x18
  14:	31 c0                	xor    %eax,%eax
  16:	5b                   	pop    %rbx
  17:	c3                   	ret
  18:	b8 01 00 00 00       	mov    $0x1,%eax
  1d:	5b                   	pop    %rbx
  1e:	c3                   	ret
  1f:	3d ff ff ff bf       	cmp    $0xbfffffff,%eax
  24:	77 14                	ja     0x3a
  26:	85 c0                	test   %eax,%eax
  28:	78 06                	js     0x30
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	31 c0                	xor    %eax,%eax
  2e:	eb e6                	jmp    0x16
  30:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
  36:	31 c0                	xor    %eax,%eax
  38:	eb dc                	jmp    0x16
  3a:	80                   	.byte 0x80
  3b:	3d e2 4e e3 00       	cmp    $0xe34ee2,%eax

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	31 c0                	xor    %eax,%eax
   4:	eb e6                	jmp    0xffffffffffffffec
   6:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
   c:	31 c0                	xor    %eax,%eax
   e:	eb dc                	jmp    0xffffffffffffffec
  10:	80                   	.byte 0x80
  11:	3d e2 4e e3 00       	cmp    $0xe34ee2,%eax
[43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
[43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
[43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
[43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
[43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
[43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
[43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
[43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
[43040.201994] Call Trace:
[43040.202095]  <IRQ>
[43040.202187] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
[43040.202301] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[43040.202418] ? handle_bug (arch/x86/kernel/traps.c:237)
[43040.202534] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
[43040.202652] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[43040.202781] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
[43040.202909] dst_release (net/core/dst.c:166 (discriminator 1))
[43040.203026] rt_cache_route (net/ipv4/route.c:1499)
[43040.203143] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
[43040.203272] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
[43040.203401] ? nf_conntrack_udp_packet =
(net/netfilter/nf_conntrack_proto_udp.c:124) nf_conntrack
[43040.203581] ip_route_input_noref (net/ipv4/route.c:2499)
[43040.203704] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
[43040.203855] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
[43040.203962] ? ip_rcv_core (net/ipv4/ip_input.c:436)
[43040.204079] process_backlog (net/core/dev.c:5997)
[43040.204201] __napi_poll (net/core/dev.c:6556)
[43040.204315] net_rx_action (net/core/dev.c:6625 net/core/dev.c:6756)
[43040.204432] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
[43040.204549] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
[43040.204667] common_interrupt (arch/x86/kernel/irq.c:247 =
(discriminator 47))
[43040.204786]  </IRQ>
[43040.204876]  <TASK>
[43040.204965] asm_common_interrupt =
(./arch/x86/include/asm/idtentry.h:640)
[43040.205090] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
[43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 48 =
8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 fb =
f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb 9f
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	ed                   	in     (%dx),%eax
   1:	c3                   	ret
   2:	66 66 2e 0f 1f 84 00 	data16 cs nopw 0x0(%rax,%rax,1)
   9:	00 00 00 00
   d:	66 90                	xchg   %ax,%ax
   f:	65 48 8b 04 25 40 32 	mov    %gs:0x23240,%rax
  16:	02 00
  18:	48 8b 00             	mov    (%rax),%rax
  1b:	a8 08                	test   $0x8,%al
  1d:	75 0c                	jne    0x2b
  1f:	eb 07                	jmp    0x28
  21:	0f 00 2d 57 0f 2c 00 	verw   0x2c0f57(%rip)        # 0x2c0f7f
  28:	fb                   	sti
  29:	f4                   	hlt
  2a:*	fa                   	cli    		<-- trapping instruction
  2b:	c3                   	ret
  2c:	0f 1f 00             	nopl   (%rax)
  2f:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
  33:	3c 01                	cmp    $0x1,%al
  35:	74 0b                	je     0x42
  37:	3c 02                	cmp    $0x2,%al
  39:	74 05                	je     0x40
  3b:	8b 7f 04             	mov    0x4(%rdi),%edi
  3e:	eb 9f                	jmp    0xffffffffffffffdf

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	fa                   	cli
   1:	c3                   	ret
   2:	0f 1f 00             	nopl   (%rax)
   5:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
   9:	3c 01                	cmp    $0x1,%al
   b:	74 0b                	je     0x18
   d:	3c 02                	cmp    $0x2,%al
   f:	74 05                	je     0x16
  11:	8b 7f 04             	mov    0x4(%rdi),%edi
  14:	eb 9f                	jmp    0xffffffffffffffb5
[43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
[43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
[43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
[43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
[43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
[43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
[43040.206593] acpi_idle_enter (drivers/acpi/processor_idle.c:709)
[43040.206711] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:267)
[43040.206835] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
[43040.206954] do_idle (kernel/sched/idle.c:134 kernel/sched/idle.c:215 =
kernel/sched/idle.c:282)
[43040.207066] cpu_startup_entry (kernel/sched/idle.c:379)
[43040.207188] start_secondary (arch/x86/kernel/smpboot.c:326)
[43040.207310] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
[43040.207451]  </TASK>
[43040.207542] ---[ end trace 0000000000000000 ]---

> On 19 Dec 2023, at 16:26, Thomas Gleixner <tglx@linutronix.de> wrote:
>=20
> On Tue, Dec 19 2023 at 11:25, Martin Zaharinov wrote:
>>> On 12 Dec 2023, at 20:16, Thomas Gleixner <tglx@linutronix.de> =
wrote:
>>> Btw, how easy is this to reproduce?
>>=20
>> Its not easy this report is generate on machine with 5-6k users , =
with
>> traffic and one time is show on 1 day , other show after 4-5 days=E2=80=
=A6
>=20
> I love those bugs ...
>=20
>> Apply this patch and will upload image on one machine as fast as
>> possible and when get any reports will send you.
>=20
> Let's see how that goes!
>=20
> Thanks,
>=20
>        tglx


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-ej1-f53.google.com (mail-ej1-f53.google.com [209.85.218.53])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1FD511187
	for <netdev@vger.kernel.org>; Fri, 29 Dec 2023 12:00:19 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="g8h4uXkt"
Received: by mail-ej1-f53.google.com with SMTP id a640c23a62f3a-a2776ce12bcso110816766b.3
        for <netdev@vger.kernel.org>; Fri, 29 Dec 2023 04:00:19 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1703851218; x=1704456018; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=vUxYdm+ICQTUWj1mgfyxHWMeFMsMTaTLmbNLT+pIsBc=;
        b=g8h4uXktfg8R//+wdiYkzutMfFb6Qo6K8/BVGH9o4QnPQm8k9LrWZyKZitCBVH9GRC
         hZH3mo9Rdain3dl6ZDohS+cqirE+/xOq3WbgnJE9uVbuzAU85qsDtwY9/RwSBHy5ESri
         OKTwxZIoCMz9bAC+u6llcNqupjHYheqdV9JNoGqpNUNqJJyq4gVq1HhwTEIbUrBD9jDE
         5jMK0k/KuELuRDcoyzPd6YCfwUDPzOyhpiCc2VKzUQSaEECn1YlJtF9eQJd0miMEJYYC
         d+EGaVk5oxSuBWDYFGIux7pERMKEituiibAkxRmjxApclDOLtdaWLWlpfXsPHl0Jh3GV
         bI6Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1703851218; x=1704456018;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=vUxYdm+ICQTUWj1mgfyxHWMeFMsMTaTLmbNLT+pIsBc=;
        b=e6/NG1yjGrOy+37El9I+YcXOZm3Dp70u8hAqnKgJuvRQZbMAXu29Ze5kta/DehBtKn
         6bw4AvCIgoumIs758CjBAZkXWW0yOYfIifya390Wr6VxxRcjTSCh6NpxLTVbYlpNn+J1
         2dy0ZPcfA/S3tHQtisFOJdHDrPtoqtyYkp+msIXKnVCdGiug8tCRDeYdtROF88W3CEve
         V7jdnnu4mOGTwPQNBbfYQqJXmpigCYO3Zyb/CdGl06YiVYWs3INiz0t3x5ZalUClCfSR
         j6SMNr0hBkQX24LlzrwPc9i6xEEgL38JAH8uxjoQIOFmvPQ55e6bsDH0J2j+63NxUugV
         hnyQ==
X-Gm-Message-State: AOJu0YzyaiN3MqNSzHLoWavQkqUuKbqIJNjXcFwpJzQMgfQoxkwviBSt
	iCdjHy+UC0KkR/oMFHlvl2k=
X-Google-Smtp-Source: AGHT+IEiC5d0mAEYHXXlecqy6cwAlc/nOlq1YNQLfJm/aDe/OHhThKldCneSDu4txrOn144s5N2Rew==
X-Received: by 2002:a17:906:386:b0:a27:4bbe:5ddc with SMTP id b6-20020a170906038600b00a274bbe5ddcmr1880003eja.19.1703851217654;
        Fri, 29 Dec 2023 04:00:17 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id mf6-20020a1709071a4600b00a26aa8f3372sm7687678ejc.27.2023.12.29.04.00.16
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 29 Dec 2023 04:00:17 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <FBF0EB18-E2B6-4076-968A-5F2ABE0F27E4@gmail.com>
Date: Fri, 29 Dec 2023 14:00:05 +0200
Cc: peterz@infradead.org,
 netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Eric Dumazet <edumazet@google.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <7D08EC48-F22C-4CC8-839F-2A9677E93DF0@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
 <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com> <87r0jrp9qi.ffs@tglx>
 <6D816814-1334-4F22-AFF8-B5E42254038E@gmail.com> <87v88ul14z.ffs@tglx>
 <FBF0EB18-E2B6-4076-968A-5F2ABE0F27E4@gmail.com>
To: Thomas Gleixner <tglx@linutronix.de>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi Thomas,

One more report from second machine:

[21299.954952] ------------[ cut here ]------------
[21299.955047] WARNING: CPU: 15 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
[21299.955153] Modules linked in: nft_limit nft_ct nft_nat nft_chain_nat =
nf_tables netconsole coretemp virtio_net net_failover failover =
virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio =
virtio_ring e1000e e1000 vmxnet3 i40e ixgbe mdio bnxt_en nf_nat_sip =
nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 rtc_cmos
[21299.955378] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G           O    =
   6.6.8 #1
[21299.955475] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 =
Gen10, BIOS U30 02/09/2023
[21299.955575] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
[21299.955662] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 f8 =
ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 78 =
06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	07                   	(bad)
   1:	83 f8 ff             	cmp    $0xffffffff,%eax
   4:	75 19                	jne    0x1f
   6:	ba 00 00 00 e0       	mov    $0xe0000000,%edx
   b:	f0 0f b1 17          	lock cmpxchg %edx,(%rdi)
   f:	83 f8 ff             	cmp    $0xffffffff,%eax
  12:	74 04                	je     0x18
  14:	31 c0                	xor    %eax,%eax
  16:	5b                   	pop    %rbx
  17:	c3                   	ret
  18:	b8 01 00 00 00       	mov    $0x1,%eax
  1d:	5b                   	pop    %rbx
  1e:	c3                   	ret
  1f:	3d ff ff ff bf       	cmp    $0xbfffffff,%eax
  24:	77 14                	ja     0x3a
  26:	85 c0                	test   %eax,%eax
  28:	78 06                	js     0x30
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	31 c0                	xor    %eax,%eax
  2e:	eb e6                	jmp    0x16
  30:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
  36:	31 c0                	xor    %eax,%eax
  38:	eb dc                	jmp    0x16
  3a:	80                   	.byte 0x80
  3b:	3d e2 4e e3 00       	cmp    $0xe34ee2,%eax

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	31 c0                	xor    %eax,%eax
   4:	eb e6                	jmp    0xffffffffffffffec
   6:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
   c:	31 c0                	xor    %eax,%eax
   e:	eb dc                	jmp    0xffffffffffffffec
  10:	80                   	.byte 0x80
  11:	3d e2 4e e3 00       	cmp    $0xe34ee2,%eax
[21299.955793] RSP: 0018:ffff96a7c0578c30 EFLAGS: 00010246
[21299.955879] RAX: 0000000000000000 RBX: ffff8b75d1e49a80 RCX: =
ffff8b75c6667c80
[21299.955974] RDX: ffff8b84bfbe4f08 RSI: 00000000fffffe01 RDI: =
ffff8b75d1e49a80
[21299.956070] RBP: ffff8b84bfbe4f08 R08: ffff8b84bfbe4f08 R09: =
0000000000000001
[21299.956167] R10: 0000000000028530 R11: 0000000000000001 R12: =
ffff8b75d1e49a40
[21299.956261] R13: ffff8b75d1e49aa8 R14: ffff8b84bfbe4f08 R15: =
00000000c26ab667
[21299.956358] FS:  0000000000000000(0000) GS:ffff8b84bfbc0000(0000) =
knlGS:0000000000000000
[21299.956457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[21299.956540] CR2: 00007f2e185c73c8 CR3: 0000000950014003 CR4: =
00000000003706e0
[21299.956635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[21299.956730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[21299.956826] Call Trace:
[21299.956905]  <IRQ>
[21299.956983] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
[21299.957065] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[21299.957147] ? handle_bug (arch/x86/kernel/traps.c:237)
[21299.957228] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
[21299.957308] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[21299.957393] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
[21299.957476] dst_release (net/core/dst.c:166 (discriminator 1))
[21299.957559] rt_cache_route (net/ipv4/route.c:1499)
[21299.957641] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
[21299.957722] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
[21299.957804] ? free_unref_page (./include/linux/list.h:150 =
(discriminator 1) ./include/linux/list.h:169 (discriminator 1) =
mm/page_alloc.c:2377 (discriminator 1) mm/page_alloc.c:2428 =
(discriminator 1))
[21299.957889] ip_route_input_noref (net/ipv4/route.c:2499)
[21299.957972] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
[21299.958058] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
[21299.958139] ? ip_rcv_core (net/ipv4/ip_input.c:436)
[21299.958220] process_backlog (net/core/dev.c:5997)
[21299.958302] __napi_poll (net/core/dev.c:6556)
[21299.958384] net_rx_action (net/core/dev.c:6625 net/core/dev.c:6756)
[21299.958466] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
[21299.958549] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
[21299.958631] sysvec_call_function_single (arch/x86/kernel/smp.c:262 =
(discriminator 47))
[21299.958714]  </IRQ>
[21299.958792]  <TASK>
[21299.958869] asm_sysvec_call_function_single =
(./arch/x86/include/asm/idtentry.h:656)
[21299.958953] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
[21299.959038] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 48 =
8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 fb =
f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb 9f
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	ed                   	in     (%dx),%eax
   1:	c3                   	ret
   2:	66 66 2e 0f 1f 84 00 	data16 cs nopw 0x0(%rax,%rax,1)
   9:	00 00 00 00
   d:	66 90                	xchg   %ax,%ax
   f:	65 48 8b 04 25 40 32 	mov    %gs:0x23240,%rax
  16:	02 00
  18:	48 8b 00             	mov    (%rax),%rax
  1b:	a8 08                	test   $0x8,%al
  1d:	75 0c                	jne    0x2b
  1f:	eb 07                	jmp    0x28
  21:	0f 00 2d 57 0f 2c 00 	verw   0x2c0f57(%rip)        # 0x2c0f7f
  28:	fb                   	sti
  29:	f4                   	hlt
  2a:*	fa                   	cli    		<-- trapping instruction
  2b:	c3                   	ret
  2c:	0f 1f 00             	nopl   (%rax)
  2f:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
  33:	3c 01                	cmp    $0x1,%al
  35:	74 0b                	je     0x42
  37:	3c 02                	cmp    $0x2,%al
  39:	74 05                	je     0x40
  3b:	8b 7f 04             	mov    0x4(%rdi),%edi
  3e:	eb 9f                	jmp    0xffffffffffffffdf

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	fa                   	cli
   1:	c3                   	ret
   2:	0f 1f 00             	nopl   (%rax)
   5:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
   9:	3c 01                	cmp    $0x1,%al
   b:	74 0b                	je     0x18
   d:	3c 02                	cmp    $0x2,%al
   f:	74 05                	je     0x16
  11:	8b 7f 04             	mov    0x4(%rdi),%edi
  14:	eb 9f                	jmp    0xffffffffffffffb5
[21299.959162] RSP: 0018:ffff96a7c015be80 EFLAGS: 00000246
[21299.959247] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
[21299.959343] RDX: ffff8b84bfbc0000 RSI: ffff8b75c76ba000 RDI: =
ffff8b75c76ba064
[21299.959437] RBP: ffffffffae216ea0 R08: ffffffffae216ea0 R09: =
0000000000000003
[21299.959533] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
[21299.959630] R13: ffffffffae216f08 R14: ffffffffae216f20 R15: =
0000000000000000
[21299.959725] acpi_idle_enter (drivers/acpi/processor_idle.c:709)
[21299.959807] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:267)
[21299.959890] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
[21299.959975] do_idle (kernel/sched/idle.c:134 kernel/sched/idle.c:215 =
kernel/sched/idle.c:282)
[21299.960058] cpu_startup_entry (kernel/sched/idle.c:379)
[21299.960140] start_secondary (arch/x86/kernel/smpboot.c:326)
[21299.960223] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
[21299.960306]  </TASK>
[21299.960384] ---[ end trace 0000000000000000 ]---

> On 22 Dec 2023, at 19:26, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi Thomas,
>=20
> this is with applyed patch from you.
> See logs
>=20
>=20
> [43040.198064] ------------[ cut here ]------------
> [43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
> [43040.198685] Modules linked in: pppoe pppox ppp_generic slhc =
nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb =
i2c_algo_bit e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe =
mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp =
nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat =
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler =
rtc_cmos
> [43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           O  =
     6.6.8 #1
> [43040.199660] Hardware name: VMware, Inc. VMware Virtual =
Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
> [43040.199886] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
> [43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
> [43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
> [43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
> [43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
> [43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
> [43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
> [43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
> [43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
> [43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
> [43040.201994] Call Trace:
> [43040.202095]  <IRQ>
> [43040.202187]  ? __warn+0x6c/0x130
> [43040.202301]  ? report_bug+0x1b8/0x200
> [43040.202418]  ? handle_bug+0x36/0x70
> [43040.202534]  ? exc_invalid_op+0x17/0x1a0
> [43040.202652]  ? asm_exc_invalid_op+0x16/0x20
> [43040.202781]  ? rcuref_put_slowpath+0x2f/0x70
> [43040.202909]  dst_release+0x1c/0x40
> [43040.203026]  rt_cache_route+0xbd/0xf0
> [43040.203143]  rt_set_nexthop.isra.0+0x1b6/0x450
> [43040.203272]  ip_route_input_slow+0x5d9/0xcc0
> [43040.203401]  ? nf_conntrack_udp_packet+0x17c/0x240 [nf_conntrack]
> [43040.203581]  ip_route_input_noref+0xe0/0xf0
> [43040.203704]  ip_rcv_finish_core.isra.0+0xbb/0x440
> [43040.203855]  ip_rcv+0xd5/0x110
> [43040.203962]  ? ip_rcv_core+0x360/0x360
> [43040.204079]  process_backlog+0x107/0x210
> [43040.204201]  __napi_poll+0x20/0x180
> [43040.204315]  net_rx_action+0x29f/0x380
> [43040.204432]  __do_softirq+0xd0/0x202
> [43040.204549]  irq_exit_rcu+0x82/0xa0
> [43040.204667]  common_interrupt+0x7a/0xa0
> [43040.204786]  </IRQ>
> [43040.204876]  <TASK>
> [43040.204965]  asm_common_interrupt+0x22/0x40
> [43040.205090] RIP: 0010:acpi_safe_halt+0x1b/0x20
> [43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
> [43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
> [43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
> [43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
> [43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
> [43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
> [43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
> [43040.206593]  acpi_idle_enter+0x77/0xc0
> [43040.206711]  cpuidle_enter_state+0x69/0x6a0
> [43040.206835]  cpuidle_enter+0x24/0x40
> [43040.206954]  do_idle+0x1a7/0x210
> [43040.207066]  cpu_startup_entry+0x21/0x30
> [43040.207188]  start_secondary+0xe1/0xf0
> [43040.207310]  secondary_startup_64_no_verify+0x166/0x16b
> [43040.207451]  </TASK>
> [43040.207542] ---[ end trace 0000000000000000 ]---
>=20
>=20
>=20
> [43040.198064] ------------[ cut here ]------------
> [43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
> [43040.198685] Modules linked in: pppoe pppox ppp_generic slhc =
nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb =
i2c_algo_bit e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe =
mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp =
nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat =
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler =
rtc_cmos
> [43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           O  =
     6.6.8 #1
> [43040.199660] Hardware name: VMware, Inc. VMware Virtual =
Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
> [43040.199886] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
> [43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 07                    (bad)
>   1: 83 f8 ff              cmp    $0xffffffff,%eax
>   4: 75 19                 jne    0x1f
>   6: ba 00 00 00 e0        mov    $0xe0000000,%edx
>   b: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
>   f: 83 f8 ff              cmp    $0xffffffff,%eax
>  12: 74 04                 je     0x18
>  14: 31 c0                 xor    %eax,%eax
>  16: 5b                    pop    %rbx
>  17: c3                    ret
>  18: b8 01 00 00 00        mov    $0x1,%eax
>  1d: 5b                    pop    %rbx
>  1e: c3                    ret
>  1f: 3d ff ff ff bf        cmp    $0xbfffffff,%eax
>  24: 77 14                 ja     0x3a
>  26: 85 c0                 test   %eax,%eax
>  28: 78 06                 js     0x30
>  2a:* 0f 0b                 ud2     <-- trapping instruction
>  2c: 31 c0                 xor    %eax,%eax
>  2e: eb e6                 jmp    0x16
>  30: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>  36: 31 c0                 xor    %eax,%eax
>  38: eb dc                 jmp    0x16
>  3a: 80                    .byte 0x80
>  3b: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 0f 0b                 ud2
>   2: 31 c0                 xor    %eax,%eax
>   4: eb e6                 jmp    0xffffffffffffffec
>   6: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>   c: 31 c0                 xor    %eax,%eax
>   e: eb dc                 jmp    0xffffffffffffffec
>  10: 80                    .byte 0x80
>  11: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
> [43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
> [43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
> [43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
> [43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
> [43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
> [43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
> [43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
> [43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
> [43040.201994] Call Trace:
> [43040.202095]  <IRQ>
> [43040.202187] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
> [43040.202301] ? report_bug (lib/bug.c:180 lib/bug.c:219)
> [43040.202418] ? handle_bug (arch/x86/kernel/traps.c:237)
> [43040.202534] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
> [43040.202652] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
> [43040.202781] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
> [43040.202909] dst_release (net/core/dst.c:166 (discriminator 1))
> [43040.203026] rt_cache_route (net/ipv4/route.c:1499)
> [43040.203143] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
> [43040.203272] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
> [43040.203401] ? nf_conntrack_udp_packet =
(net/netfilter/nf_conntrack_proto_udp.c:124) nf_conntrack
> [43040.203581] ip_route_input_noref (net/ipv4/route.c:2499)
> [43040.203704] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
> [43040.203855] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
> [43040.203962] ? ip_rcv_core (net/ipv4/ip_input.c:436)
> [43040.204079] process_backlog (net/core/dev.c:5997)
> [43040.204201] __napi_poll (net/core/dev.c:6556)
> [43040.204315] net_rx_action (net/core/dev.c:6625 net/core/dev.c:6756)
> [43040.204432] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
> [43040.204549] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
> [43040.204667] common_interrupt (arch/x86/kernel/irq.c:247 =
(discriminator 47))
> [43040.204786]  </IRQ>
> [43040.204876]  <TASK>
> [43040.204965] asm_common_interrupt =
(./arch/x86/include/asm/idtentry.h:640)
> [43040.205090] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
> [43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: ed                    in     (%dx),%eax
>   1: c3                    ret
>   2: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
>   9: 00 00 00 00
>   d: 66 90                 xchg   %ax,%ax
>   f: 65 48 8b 04 25 40 32 mov    %gs:0x23240,%rax
>  16: 02 00
>  18: 48 8b 00              mov    (%rax),%rax
>  1b: a8 08                 test   $0x8,%al
>  1d: 75 0c                 jne    0x2b
>  1f: eb 07                 jmp    0x28
>  21: 0f 00 2d 57 0f 2c 00 verw   0x2c0f57(%rip)        # 0x2c0f7f
>  28: fb                    sti
>  29: f4                    hlt
>  2a:* fa                    cli     <-- trapping instruction
>  2b: c3                    ret
>  2c: 0f 1f 00              nopl   (%rax)
>  2f: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>  33: 3c 01                 cmp    $0x1,%al
>  35: 74 0b                 je     0x42
>  37: 3c 02                 cmp    $0x2,%al
>  39: 74 05                 je     0x40
>  3b: 8b 7f 04              mov    0x4(%rdi),%edi
>  3e: eb 9f                 jmp    0xffffffffffffffdf
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: fa                    cli
>   1: c3                    ret
>   2: 0f 1f 00              nopl   (%rax)
>   5: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>   9: 3c 01                 cmp    $0x1,%al
>   b: 74 0b                 je     0x18
>   d: 3c 02                 cmp    $0x2,%al
>   f: 74 05                 je     0x16
>  11: 8b 7f 04              mov    0x4(%rdi),%edi
>  14: eb 9f                 jmp    0xffffffffffffffb5
> [43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
> [43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
> [43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
> [43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
> [43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
> [43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
> [43040.206593] acpi_idle_enter (drivers/acpi/processor_idle.c:709)
> [43040.206711] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:267)
> [43040.206835] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
> [43040.206954] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
> [43040.207066] cpu_startup_entry (kernel/sched/idle.c:379)
> [43040.207188] start_secondary (arch/x86/kernel/smpboot.c:326)
> [43040.207310] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
> [43040.207451]  </TASK>
> [43040.207542] ---[ end trace 0000000000000000 ]---
>=20
>> On 19 Dec 2023, at 16:26, Thomas Gleixner <tglx@linutronix.de> wrote:
>>=20
>> On Tue, Dec 19 2023 at 11:25, Martin Zaharinov wrote:
>>>> On 12 Dec 2023, at 20:16, Thomas Gleixner <tglx@linutronix.de> =
wrote:
>>>> Btw, how easy is this to reproduce?
>>>=20
>>> Its not easy this report is generate on machine with 5-6k users , =
with
>>> traffic and one time is show on 1 day , other show after 4-5 days=E2=80=
=A6
>>=20
>> I love those bugs ...
>>=20
>>> Apply this patch and will upload image on one machine as fast as
>>> possible and when get any reports will send you.
>>=20
>> Let's see how that goes!
>>=20
>> Thanks,
>>=20
>>       tglx
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 544C31E523
	for <netdev@vger.kernel.org>; Thu,  4 Jan 2024 20:51:27 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bAniLtL9"
Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-3368abe1093so679487f8f.2
        for <netdev@vger.kernel.org>; Thu, 04 Jan 2024 12:51:26 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1704401485; x=1705006285; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=pKTBO+/w9JChel3rVYWNf/F0r+bEzNH2LNbP/Kbh4Bg=;
        b=bAniLtL9jkSZllpngNHTIAJ5QZAcdB54tS7JqnS6XFSnlOqTD561ViYt48RmzT/E5X
         fyCSBmGaA6K6Rhtfnw8DRvlJqkUiNkBmvBHKODIeitm+k7lRyCEw1MMVvQ10ZJ5d+k4w
         kn0TL3Wg862igCE18rWTerMZXPeb4HcOA/bRhh59KieqAfkf1fcolSLYcNb8hRDGe5OM
         lCAkZHDCELUywZdgB/PzXkIrr+cOp5Sl8KF49pm5NapNT1OAUeiPB5zzsSOWAEBO4cUr
         y2nZYgYce4GOEGFTn4+Z2TCHGHMmAFAr4SLbm2s3vE51GOjzF5MD+sVNWPSgFJ2ej9p1
         SUgw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1704401485; x=1705006285;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=pKTBO+/w9JChel3rVYWNf/F0r+bEzNH2LNbP/Kbh4Bg=;
        b=t4tiBuMmq7EOLz1nZNozOsMvZcNoi5jz909C8hxLoP54KmWCdg52naCoURWzEA+39Y
         rxh+E2aSdvGNUDGyJy2e/Mym33suG7Mah5qrtf/Q0ClU8Rx0RtRkl6QE2M2YgGNM8XFc
         rNqNwZePmxjtLSgje1sLAEKQO4q3hAh29mUONEyyaZom1D6PNBXemWxQqZK2WEm+iDK7
         u5p2A09Wlv+nW7yZBTeOh+856nR4bGCtLA3W8Xa+I/Jkv5hX3K+Z97j8exlsUh8Jk/MG
         IQBAp12BzrM1OtyLLWin6PYEetKFYtmAOZu2VpcKzDjO0DNdIxe3CGGwkEy+TAUCtv2m
         rUaw==
X-Gm-Message-State: AOJu0Yy8uKcqpW1jKeMNu+jcpGo6M7jSOA4qgguAxCdmD8qYEmu9l47H
	IniMoLG9+grXW7w2QTddFQM=
X-Google-Smtp-Source: AGHT+IEDNv3yB+kt/6jhzdCDsLqJAcaHuiNCU+qEM96DyhjhqNfoMY0/aa05VuwdnvD8DVPeYLmmKw==
X-Received: by 2002:a5d:4808:0:b0:336:6db9:6a7e with SMTP id l8-20020a5d4808000000b003366db96a7emr606029wrq.8.1704401484883;
        Thu, 04 Jan 2024 12:51:24 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id r3-20020a5d4e43000000b003375009accesm83742wrt.50.2024.01.04.12.51.23
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Thu, 04 Jan 2024 12:51:24 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <7D08EC48-F22C-4CC8-839F-2A9677E93DF0@gmail.com>
Date: Thu, 4 Jan 2024 22:51:12 +0200
Cc: peterz@infradead.org,
 netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Eric Dumazet <edumazet@google.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <E6A168E5-5BB6-47AD-B8D6-D6BE2F2C528F@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
 <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com> <87r0jrp9qi.ffs@tglx>
 <6D816814-1334-4F22-AFF8-B5E42254038E@gmail.com> <87v88ul14z.ffs@tglx>
 <FBF0EB18-E2B6-4076-968A-5F2ABE0F27E4@gmail.com>
 <7D08EC48-F22C-4CC8-839F-2A9677E93DF0@gmail.com>
To: Thomas Gleixner <tglx@linutronix.de>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi Thomas ,

Happy New Year!

here is two debugs from two new installed machins with kernel 6.6.9:

dmesg1 :

[ 2257.449125] ------------[ cut here ]------------
[ 2257.449245] WARNING: CPU: 1 PID: 40622 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
[ 2257.449373] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[ 2257.449642] CPU: 1 PID: 40622 Comm: nc Tainted: G           O       =
6.6.9 #1
[ 2257.449761] Hardware name: Supermicro PIO-5038MR-H8TRF-NODE/X10SRD-F, =
BIOS 3.3 10/28/2020
[ 2257.449883] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
[ 2257.449977] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 f8 =
ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 78 =
06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 00
[ 2257.450135] RSP: 0000:ffffb455cef83b78 EFLAGS: 00010246
[ 2257.450227] RAX: 0000000000000000 RBX: ffff94873bb77dc0 RCX: =
ffff9486c0d46b80
[ 2257.450341] RDX: ffff948736578428 RSI: 00000000fffffe01 RDI: =
ffff94873bb77dc0
[ 2257.450456] RBP: ffff948736578428 R08: ffff948e1fa64f08 R09: =
0000000000000001
[ 2257.450570] R10: 0000000000028530 R11: 0000000000000001 R12: =
ffff94873bb77d80
[ 2257.450685] R13: ffff94873bb77de8 R14: ffff948e1fa64f08 R15: =
000000000266f59d
[ 2257.450802] FS:  00007f0cdbc73800(0000) GS:ffff948e1fa40000(0000) =
knlGS:0000000000000000
[ 2257.450918] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2257.451012] CR2: 00007f0cdc3f5c30 CR3: 0000000178ea0002 CR4: =
00000000003706e0
[ 2257.451127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[ 2257.451240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[ 2257.451353] Call Trace:
[ 2257.451441]  <TASK>
[ 2257.451526]  ? __warn+0x6c/0x130
[ 2257.451616]  ? report_bug+0x1b8/0x200
[ 2257.451707]  ? handle_bug+0x36/0x70
[ 2257.451797]  ? exc_invalid_op+0x17/0x1a0
[ 2257.451886]  ? asm_exc_invalid_op+0x16/0x20
[ 2257.452038]  ? rcuref_put_slowpath+0x2f/0x70
[ 2257.452129]  dst_release+0x1c/0x40
[ 2257.452222]  rt_cache_route+0xbd/0xf0
[ 2257.452313]  ? kmem_cache_alloc+0x31/0x390
[ 2257.452404]  rt_set_nexthop.isra.0+0x1b6/0x450
[ 2257.452495]  ip_route_input_slow+0x5d9/0xcc0
[ 2257.452586]  ? nft_nat_do_chain+0x7f/0xd0 [nft_chain_nat]
[ 2257.452681]  ? nf_conntrack_udp_packet+0xcf/0x240 [nf_conntrack]
[ 2257.452784]  ? nf_nat_inet_fn+0x36f/0x3f0 [nf_nat]
[ 2257.452880]  ip_route_input_noref+0xe0/0xf0
[ 2257.452970]  ip_rcv_finish_core.isra.0+0xbb/0x440
[ 2257.453064]  ip_rcv+0xd5/0x110
[ 2257.453151]  ? ip_rcv_core+0x360/0x360
[ 2257.453240]  process_backlog+0x107/0x210
[ 2257.453330]  __napi_poll+0x20/0x180
[ 2257.453420]  net_rx_action+0x29f/0x380
[ 2257.453510]  __do_softirq+0xd0/0x202
[ 2257.453599]  irq_exit_rcu+0x82/0xa0
[ 2257.453689]  sysvec_call_function_single+0x32/0x80
[ 2257.453781]  asm_sysvec_call_function_single+0x16/0x20
[ 2257.453874] RIP: 0033:0x7f0cdc5928b2
[ 2257.453963] Code: 06 00 00 4c 89 65 88 49 83 fd 08 0f 84 f7 06 00 00 =
49 83 fd 26 0f 84 05 07 00 00 4d 85 ed 0f 84 5f 01 00 00 41 0f b6 44 24 =
04 <89> c6 40 c0 ee 04 0f 84 72 06 00 00 41 0f b6 54 24 05 83 e2 03 ff
[ 2257.454121] RSP: 002b:00007ffc04d3e890 EFLAGS: 00000206
[ 2257.454215] RAX: 0000000000000012 RBX: 00007f0cdc444db8 RCX: =
00007f0cdc4e6e60
[ 2257.454329] RDX: 0000000000000009 RSI: 00007f0cdc57ef30 RDI: =
00007f0cdc42c808
[ 2257.454442] RBP: 00007ffc04d3e9b0 R08: 00007f0cdc445028 R09: =
00007ffc04d3e940
[ 2257.454555] R10: 00007f0cdbf00be8 R11: 0000000000000000 R12: =
00007f0cdc42c898
[ 2257.454670] R13: 0000000000000006 R14: 0000000600000006 R15: =
00007f0cdc581000
[ 2257.454784]  </TASK>
[ 2257.454869] ---[ end trace 0000000000000000 ]=E2=80=94


[ 2257.449125] ------------[ cut here ]------------
[ 2257.449245] WARNING: CPU: 1 PID: 40622 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
[ 2257.449373] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[ 2257.449642] CPU: 1 PID: 40622 Comm: nc Tainted: G           O       =
6.6.9 #1
[ 2257.449761] Hardware name: Supermicro PIO-5038MR-H8TRF-NODE/X10SRD-F, =
BIOS 3.3 10/28/2020
[ 2257.449883] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
[ 2257.449977] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 f8 =
ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 78 =
06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	07                   	(bad)
   1:	83 f8 ff             	cmp    $0xffffffff,%eax
   4:	75 19                	jne    0x1f
   6:	ba 00 00 00 e0       	mov    $0xe0000000,%edx
   b:	f0 0f b1 17          	lock cmpxchg %edx,(%rdi)
   f:	83 f8 ff             	cmp    $0xffffffff,%eax
  12:	74 04                	je     0x18
  14:	31 c0                	xor    %eax,%eax
  16:	5b                   	pop    %rbx
  17:	c3                   	ret
  18:	b8 01 00 00 00       	mov    $0x1,%eax
  1d:	5b                   	pop    %rbx
  1e:	c3                   	ret
  1f:	3d ff ff ff bf       	cmp    $0xbfffffff,%eax
  24:	77 14                	ja     0x3a
  26:	85 c0                	test   %eax,%eax
  28:	78 06                	js     0x30
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	31 c0                	xor    %eax,%eax
  2e:	eb e6                	jmp    0x16
  30:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
  36:	31 c0                	xor    %eax,%eax
  38:	eb dc                	jmp    0x16
  3a:	80                   	.byte 0x80
  3b:	3d e2 4c e3 00       	cmp    $0xe34ce2,%eax

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	31 c0                	xor    %eax,%eax
   4:	eb e6                	jmp    0xffffffffffffffec
   6:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
   c:	31 c0                	xor    %eax,%eax
   e:	eb dc                	jmp    0xffffffffffffffec
  10:	80                   	.byte 0x80
  11:	3d e2 4c e3 00       	cmp    $0xe34ce2,%eax
[ 2257.450135] RSP: 0000:ffffb455cef83b78 EFLAGS: 00010246
[ 2257.450227] RAX: 0000000000000000 RBX: ffff94873bb77dc0 RCX: =
ffff9486c0d46b80
[ 2257.450341] RDX: ffff948736578428 RSI: 00000000fffffe01 RDI: =
ffff94873bb77dc0
[ 2257.450456] RBP: ffff948736578428 R08: ffff948e1fa64f08 R09: =
0000000000000001
[ 2257.450570] R10: 0000000000028530 R11: 0000000000000001 R12: =
ffff94873bb77d80
[ 2257.450685] R13: ffff94873bb77de8 R14: ffff948e1fa64f08 R15: =
000000000266f59d
[ 2257.450802] FS:  00007f0cdbc73800(0000) GS:ffff948e1fa40000(0000) =
knlGS:0000000000000000
[ 2257.450918] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2257.451012] CR2: 00007f0cdc3f5c30 CR3: 0000000178ea0002 CR4: =
00000000003706e0
[ 2257.451127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[ 2257.451240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[ 2257.451353] Call Trace:
[ 2257.451441]  <TASK>
[ 2257.451526] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
[ 2257.451616] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[ 2257.451707] ? handle_bug (arch/x86/kernel/traps.c:237)
[ 2257.451797] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
[ 2257.451886] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[ 2257.452038] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
[ 2257.452129] dst_release (net/core/dst.c:166 (discriminator 1))
[ 2257.452222] rt_cache_route (net/ipv4/route.c:1499)
[ 2257.452313] ? kmem_cache_alloc (mm/slab.h:711 (discriminator 1) =
mm/slub.c:3461 (discriminator 1) mm/slub.c:3487 (discriminator 1) =
mm/slub.c:3494 (discriminator 1) mm/slub.c:3503 (discriminator 1))
[ 2257.452404] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
[ 2257.452495] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
[ 2257.452586] ? nft_nat_do_chain (net/netfilter/nft_chain_nat.c:33) =
nft_chain_nat
[ 2257.452681] ? nf_conntrack_udp_packet =
(net/netfilter/nf_conntrack_proto_udp.c:130) nf_conntrack
[ 2257.452784] ? nf_nat_inet_fn (net/netfilter/nf_nat_core.c:844) nf_nat
[ 2257.452880] ip_route_input_noref (net/ipv4/route.c:2499)
[ 2257.452970] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
[ 2257.453064] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
[ 2257.453151] ? ip_rcv_core (net/ipv4/ip_input.c:436)
[ 2257.453240] process_backlog (net/core/dev.c:6000)
[ 2257.453330] __napi_poll (net/core/dev.c:6559)
[ 2257.453420] net_rx_action (net/core/dev.c:6628 net/core/dev.c:6759)
[ 2257.453510] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
[ 2257.453599] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
[ 2257.453689] sysvec_call_function_single (arch/x86/kernel/smp.c:262 =
(discriminator 69))
[ 2257.453781] asm_sysvec_call_function_single =
(./arch/x86/include/asm/idtentry.h:656)
[ 2257.453874] RIP: 0033:0x7f0cdc5928b2
[ 2257.453963] Code: 06 00 00 4c 89 65 88 49 83 fd 08 0f 84 f7 06 00 00 =
49 83 fd 26 0f 84 05 07 00 00 4d 85 ed 0f 84 5f 01 00 00 41 0f b6 44 24 =
04 <89> c6 40 c0 ee 04 0f 84 72 06 00 00 41 0f b6 54 24 05 83 e2 03 ff
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	06                   	(bad)
   1:	00 00                	add    %al,(%rax)
   3:	4c 89 65 88          	mov    %r12,-0x78(%rbp)
   7:	49 83 fd 08          	cmp    $0x8,%r13
   b:	0f 84 f7 06 00 00    	je     0x708
  11:	49 83 fd 26          	cmp    $0x26,%r13
  15:	0f 84 05 07 00 00    	je     0x720
  1b:	4d 85 ed             	test   %r13,%r13
  1e:	0f 84 5f 01 00 00    	je     0x183
  24:	41 0f b6 44 24 04    	movzbl 0x4(%r12),%eax
  2a:*	89 c6                	mov    %eax,%esi		<-- =
trapping instruction
  2c:	40 c0 ee 04          	shr    $0x4,%sil
  30:	0f 84 72 06 00 00    	je     0x6a8
  36:	41 0f b6 54 24 05    	movzbl 0x5(%r12),%edx
  3c:	83 e2 03             	and    $0x3,%edx
  3f:	ff                   	.byte 0xff

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	89 c6                	mov    %eax,%esi
   2:	40 c0 ee 04          	shr    $0x4,%sil
   6:	0f 84 72 06 00 00    	je     0x67e
   c:	41 0f b6 54 24 05    	movzbl 0x5(%r12),%edx
  12:	83 e2 03             	and    $0x3,%edx
  15:	ff                   	.byte 0xff
[ 2257.454121] RSP: 002b:00007ffc04d3e890 EFLAGS: 00000206
[ 2257.454215] RAX: 0000000000000012 RBX: 00007f0cdc444db8 RCX: =
00007f0cdc4e6e60
[ 2257.454329] RDX: 0000000000000009 RSI: 00007f0cdc57ef30 RDI: =
00007f0cdc42c808
[ 2257.454442] RBP: 00007ffc04d3e9b0 R08: 00007f0cdc445028 R09: =
00007ffc04d3e940
[ 2257.454555] R10: 00007f0cdbf00be8 R11: 0000000000000000 R12: =
00007f0cdc42c898
[ 2257.454670] R13: 0000000000000006 R14: 0000000600000006 R15: =
00007f0cdc581000
[ 2257.454784]  </TASK>
[ 2257.454869] ---[ end trace 0000000000000000 ]=E2=80=94


dmesg2 :=20

[ 2567.167952] ------------[ cut here ]------------
[ 2567.168053] WARNING: CPU: 11 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
[ 2567.168175] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[ 2567.168445] CPU: 11 PID: 0 Comm: swapper/11 Tainted: G           O    =
   6.6.9 #1
[ 2567.168561] Hardware name: Supermicro X10SRD-F/X10SRD-F, BIOS 3.4 =
06/05/2021
[ 2567.168675] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
[ 2567.168767] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 f8 =
ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 78 =
06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 00
[ 2567.168924] RSP: 0018:ffffaeaf80418d00 EFLAGS: 00010246
[ 2567.169017] RAX: 0000000000000000 RBX: ffff9fef84d6a940 RCX: =
0000000000000074
[ 2567.169132] RDX: ffff9fefe2e30000 RSI: 0000000000000000 RDI: =
ffff9fef84d6a940
[ 2567.169246] RBP: ffff9fefe2e306c0 R08: 0000000000000000 R09: =
0000000000029300
[ 2567.169359] R10: 0000000000029300 R11: ffffaeaf80418d90 R12: =
ffff9fef8aebe000
[ 2567.169473] R13: ffff9fef80896800 R14: ffff9fef85335200 R15: =
ffff9fef8ae07080
[ 2567.169586] FS:  0000000000000000(0000) GS:ffff9ff6dfcc0000(0000) =
knlGS:0000000000000000
[ 2567.169702] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2567.169795] CR2: 00007f4eaa7e6650 CR3: 0000000156dcd006 CR4: =
00000000003706e0
[ 2567.169908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[ 2567.170022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[ 2567.170137] Call Trace:
[ 2567.170224]  <IRQ>
[ 2567.170309]  ? __warn+0x6c/0x130
[ 2567.170399]  ? report_bug+0x1b8/0x200
[ 2567.170488]  ? handle_bug+0x36/0x70
[ 2567.170577]  ? exc_invalid_op+0x17/0x1a0
[ 2567.170667]  ? asm_exc_invalid_op+0x16/0x20
[ 2567.170758]  ? rcuref_put_slowpath+0x2f/0x70
[ 2567.170850]  dst_release+0x1c/0x40
[ 2567.170939]  __dev_queue_xmit+0x598/0xce0
[ 2567.171029]  vlan_dev_hard_start_xmit+0x82/0xc0
[ 2567.171122]  dev_hard_start_xmit+0x95/0xe0
[ 2567.171216]  __dev_queue_xmit+0x863/0xce0
[ 2567.171305]  ? eth_header+0x25/0xc0
[ 2567.171394]  ip_finish_output2+0x1a0/0x530
[ 2567.171485]  process_backlog+0x107/0x210
[ 2567.171575]  __napi_poll+0x20/0x180
[ 2567.171663]  net_rx_action+0x29f/0x380
[ 2567.171752]  ? rebalance_domains+0x14c/0x300
[ 2567.171843]  __do_softirq+0xd0/0x202
[ 2567.171932]  irq_exit_rcu+0x82/0xa0
[ 2567.172022]  common_interrupt+0x7a/0xa0
[ 2567.172111]  </IRQ>
[ 2567.172198]  <TASK>
[ 2567.172283]  asm_common_interrupt+0x22/0x40
[ 2567.172374] RIP: 0010:cpuidle_enter_state+0xa3/0x6a0
[ 2567.172467] Code: 46 40 40 0f 84 02 01 00 00 e8 c9 a0 70 ff e8 d4 f6 =
ff ff 31 ff 49 89 c6 e8 0a b9 6f ff 45 84 ff 0f 85 d9 00 00 00 fb 45 85 =
ed <0f> 88 b8 00 00 00 49 63 cd 48 8b 04 24 48 6b f1 68 49 29 c6 48 8d
[ 2567.172623] RSP: 0018:ffffaeaf80177e98 EFLAGS: 00000202
[ 2567.172715] RAX: ffff9ff6dfce3a80 RBX: ffff9fef81338000 RCX: =
000000000000001f
[ 2567.172828] RDX: 00000255b721ed84 RSI: 00000000238e3b7a RDI: =
0000000000000000
[ 2567.172942] RBP: ffffffffba216ea0 R08: 0000000000000004 R09: =
ffff9ff6dfcdef00
[ 2567.173055] R10: ffff9ff6dfcdef00 R11: 0000000000000007 R12: =
0000000000000001
[ 2567.173168] R13: 0000000000000001 R14: 00000255b721ed84 R15: =
0000000000000000
[ 2567.173283]  ? cpuidle_enter_state+0x96/0x6a0
[ 2567.173374]  cpuidle_enter+0x24/0x40
[ 2567.173464]  do_idle+0x1a7/0x210
[ 2567.173552]  cpu_startup_entry+0x21/0x30
[ 2567.173642]  start_secondary+0xe1/0xf0
[ 2567.173732]  secondary_startup_64_no_verify+0x178/0x17b
[ 2567.173825]  </TASK>
[ 2567.173910] ---[ end trace 0000000000000000 ]=E2=80=94


[ 2567.167952] ------------[ cut here ]------------
[ 2567.168053] WARNING: CPU: 11 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
[ 2567.168175] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
[ 2567.168445] CPU: 11 PID: 0 Comm: swapper/11 Tainted: G           O    =
   6.6.9 #1
[ 2567.168561] Hardware name: Supermicro X10SRD-F/X10SRD-F, BIOS 3.4 =
06/05/2021
[ 2567.168675] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
[ 2567.168767] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 f8 =
ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 78 =
06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	07                   	(bad)
   1:	83 f8 ff             	cmp    $0xffffffff,%eax
   4:	75 19                	jne    0x1f
   6:	ba 00 00 00 e0       	mov    $0xe0000000,%edx
   b:	f0 0f b1 17          	lock cmpxchg %edx,(%rdi)
   f:	83 f8 ff             	cmp    $0xffffffff,%eax
  12:	74 04                	je     0x18
  14:	31 c0                	xor    %eax,%eax
  16:	5b                   	pop    %rbx
  17:	c3                   	ret
  18:	b8 01 00 00 00       	mov    $0x1,%eax
  1d:	5b                   	pop    %rbx
  1e:	c3                   	ret
  1f:	3d ff ff ff bf       	cmp    $0xbfffffff,%eax
  24:	77 14                	ja     0x3a
  26:	85 c0                	test   %eax,%eax
  28:	78 06                	js     0x30
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	31 c0                	xor    %eax,%eax
  2e:	eb e6                	jmp    0x16
  30:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
  36:	31 c0                	xor    %eax,%eax
  38:	eb dc                	jmp    0x16
  3a:	80                   	.byte 0x80
  3b:	3d e2 4c e3 00       	cmp    $0xe34ce2,%eax

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	31 c0                	xor    %eax,%eax
   4:	eb e6                	jmp    0xffffffffffffffec
   6:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
   c:	31 c0                	xor    %eax,%eax
   e:	eb dc                	jmp    0xffffffffffffffec
  10:	80                   	.byte 0x80
  11:	3d e2 4c e3 00       	cmp    $0xe34ce2,%eax
[ 2567.168924] RSP: 0018:ffffaeaf80418d00 EFLAGS: 00010246
[ 2567.169017] RAX: 0000000000000000 RBX: ffff9fef84d6a940 RCX: =
0000000000000074
[ 2567.169132] RDX: ffff9fefe2e30000 RSI: 0000000000000000 RDI: =
ffff9fef84d6a940
[ 2567.169246] RBP: ffff9fefe2e306c0 R08: 0000000000000000 R09: =
0000000000029300
[ 2567.169359] R10: 0000000000029300 R11: ffffaeaf80418d90 R12: =
ffff9fef8aebe000
[ 2567.169473] R13: ffff9fef80896800 R14: ffff9fef85335200 R15: =
ffff9fef8ae07080
[ 2567.169586] FS:  0000000000000000(0000) GS:ffff9ff6dfcc0000(0000) =
knlGS:0000000000000000
[ 2567.169702] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2567.169795] CR2: 00007f4eaa7e6650 CR3: 0000000156dcd006 CR4: =
00000000003706e0
[ 2567.169908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
[ 2567.170022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
[ 2567.170137] Call Trace:
[ 2567.170224]  <IRQ>
[ 2567.170309] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
[ 2567.170399] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[ 2567.170488] ? handle_bug (arch/x86/kernel/traps.c:237)
[ 2567.170577] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
[ 2567.170667] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[ 2567.170758] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
[ 2567.170850] dst_release (net/core/dst.c:166 (discriminator 1))
[ 2567.170939] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4327)
[ 2567.171029] vlan_dev_hard_start_xmit (net/8021q/vlan_dev.c:130)
[ 2567.171122] dev_hard_start_xmit (./include/linux/netdevice.h:4926 =
net/core/dev.c:3576 net/core/dev.c:3592)
[ 2567.171216] __dev_queue_xmit (./include/linux/netdevice.h:3300 =
(discriminator 25) net/core/dev.c:4373 (discriminator 25))
[ 2567.171305] ? eth_header (net/ethernet/eth.c:85)
[ 2567.171394] ip_finish_output2 (./include/net/neighbour.h:542 =
(discriminator 2) net/ipv4/ip_output.c:233 (discriminator 2))
[ 2567.171485] process_backlog (net/core/dev.c:6000)
[ 2567.171575] __napi_poll (net/core/dev.c:6559)
[ 2567.171663] net_rx_action (net/core/dev.c:6628 net/core/dev.c:6759)
[ 2567.171752] ? rebalance_domains (kernel/sched/fair.c:11719 =
kernel/sched/fair.c:11895)
[ 2567.171843] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
[ 2567.171932] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
[ 2567.172022] common_interrupt (arch/x86/kernel/irq.c:247 =
(discriminator 47))
[ 2567.172111]  </IRQ>
[ 2567.172198]  <TASK>
[ 2567.172283] asm_common_interrupt =
(./arch/x86/include/asm/idtentry.h:640)
[ 2567.172374] RIP: 0010:cpuidle_enter_state =
(drivers/cpuidle/cpuidle.c:291)
[ 2567.172467] Code: 46 40 40 0f 84 02 01 00 00 e8 c9 a0 70 ff e8 d4 f6 =
ff ff 31 ff 49 89 c6 e8 0a b9 6f ff 45 84 ff 0f 85 d9 00 00 00 fb 45 85 =
ed <0f> 88 b8 00 00 00 49 63 cd 48 8b 04 24 48 6b f1 68 49 29 c6 48 8d
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	46                   	rex.RX
   1:	40                   	rex
   2:	40 0f 84 02 01 00 00 	rex je 0x10b
   9:	e8 c9 a0 70 ff       	call   0xffffffffff70a0d7
   e:	e8 d4 f6 ff ff       	call   0xfffffffffffff6e7
  13:	31 ff                	xor    %edi,%edi
  15:	49 89 c6             	mov    %rax,%r14
  18:	e8 0a b9 6f ff       	call   0xffffffffff6fb927
  1d:	45 84 ff             	test   %r15b,%r15b
  20:	0f 85 d9 00 00 00    	jne    0xff
  26:	fb                   	sti
  27:	45 85 ed             	test   %r13d,%r13d
  2a:*	0f 88 b8 00 00 00    	js     0xe8		<-- trapping =
instruction
  30:	49 63 cd             	movslq %r13d,%rcx
  33:	48 8b 04 24          	mov    (%rsp),%rax
  37:	48 6b f1 68          	imul   $0x68,%rcx,%rsi
  3b:	49 29 c6             	sub    %rax,%r14
  3e:	48                   	rex.W
  3f:	8d                   	.byte 0x8d

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 88 b8 00 00 00    	js     0xbe
   6:	49 63 cd             	movslq %r13d,%rcx
   9:	48 8b 04 24          	mov    (%rsp),%rax
   d:	48 6b f1 68          	imul   $0x68,%rcx,%rsi
  11:	49 29 c6             	sub    %rax,%r14
  14:	48                   	rex.W
  15:	8d                   	.byte 0x8d
[ 2567.172623] RSP: 0018:ffffaeaf80177e98 EFLAGS: 00000202
[ 2567.172715] RAX: ffff9ff6dfce3a80 RBX: ffff9fef81338000 RCX: =
000000000000001f
[ 2567.172828] RDX: 00000255b721ed84 RSI: 00000000238e3b7a RDI: =
0000000000000000
[ 2567.172942] RBP: ffffffffba216ea0 R08: 0000000000000004 R09: =
ffff9ff6dfcdef00
[ 2567.173055] R10: ffff9ff6dfcdef00 R11: 0000000000000007 R12: =
0000000000000001
[ 2567.173168] R13: 0000000000000001 R14: 00000255b721ed84 R15: =
0000000000000000
[ 2567.173283] ? cpuidle_enter_state (drivers/cpuidle/cpuidle.c:285)
[ 2567.173374] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
[ 2567.173464] do_idle (kernel/sched/idle.c:134 kernel/sched/idle.c:215 =
kernel/sched/idle.c:282)
[ 2567.173552] cpu_startup_entry (kernel/sched/idle.c:379)
[ 2567.173642] start_secondary (arch/x86/kernel/smpboot.c:326)
[ 2567.173732] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:449)
[ 2567.173825]  </TASK>
[ 2567.173910] ---[ end trace 0000000000000000 ]=E2=80=94

best regards,
Martin


> On 29 Dec 2023, at 14:00, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi Thomas,
>=20
> One more report from second machine:
>=20
> [21299.954952] ------------[ cut here ]------------
> [21299.955047] WARNING: CPU: 15 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
> [21299.955153] Modules linked in: nft_limit nft_ct nft_nat =
nft_chain_nat nf_tables netconsole coretemp virtio_net net_failover =
failover virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio =
virtio_ring e1000e e1000 vmxnet3 i40e ixgbe mdio bnxt_en nf_nat_sip =
nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 rtc_cmos
> [21299.955378] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G           O  =
     6.6.8 #1
> [21299.955475] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 =
Gen10, BIOS U30 02/09/2023
> [21299.955575] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
> [21299.955662] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 07                    (bad)
>   1: 83 f8 ff              cmp    $0xffffffff,%eax
>   4: 75 19                 jne    0x1f
>   6: ba 00 00 00 e0        mov    $0xe0000000,%edx
>   b: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
>   f: 83 f8 ff              cmp    $0xffffffff,%eax
>  12: 74 04                 je     0x18
>  14: 31 c0                 xor    %eax,%eax
>  16: 5b                    pop    %rbx
>  17: c3                    ret
>  18: b8 01 00 00 00        mov    $0x1,%eax
>  1d: 5b                    pop    %rbx
>  1e: c3                    ret
>  1f: 3d ff ff ff bf        cmp    $0xbfffffff,%eax
>  24: 77 14                 ja     0x3a
>  26: 85 c0                 test   %eax,%eax
>  28: 78 06                 js     0x30
>  2a:* 0f 0b                 ud2     <-- trapping instruction
>  2c: 31 c0                 xor    %eax,%eax
>  2e: eb e6                 jmp    0x16
>  30: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>  36: 31 c0                 xor    %eax,%eax
>  38: eb dc                 jmp    0x16
>  3a: 80                    .byte 0x80
>  3b: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 0f 0b                 ud2
>   2: 31 c0                 xor    %eax,%eax
>   4: eb e6                 jmp    0xffffffffffffffec
>   6: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>   c: 31 c0                 xor    %eax,%eax
>   e: eb dc                 jmp    0xffffffffffffffec
>  10: 80                    .byte 0x80
>  11: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
> [21299.955793] RSP: 0018:ffff96a7c0578c30 EFLAGS: 00010246
> [21299.955879] RAX: 0000000000000000 RBX: ffff8b75d1e49a80 RCX: =
ffff8b75c6667c80
> [21299.955974] RDX: ffff8b84bfbe4f08 RSI: 00000000fffffe01 RDI: =
ffff8b75d1e49a80
> [21299.956070] RBP: ffff8b84bfbe4f08 R08: ffff8b84bfbe4f08 R09: =
0000000000000001
> [21299.956167] R10: 0000000000028530 R11: 0000000000000001 R12: =
ffff8b75d1e49a40
> [21299.956261] R13: ffff8b75d1e49aa8 R14: ffff8b84bfbe4f08 R15: =
00000000c26ab667
> [21299.956358] FS:  0000000000000000(0000) GS:ffff8b84bfbc0000(0000) =
knlGS:0000000000000000
> [21299.956457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [21299.956540] CR2: 00007f2e185c73c8 CR3: 0000000950014003 CR4: =
00000000003706e0
> [21299.956635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
> [21299.956730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
> [21299.956826] Call Trace:
> [21299.956905]  <IRQ>
> [21299.956983] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
> [21299.957065] ? report_bug (lib/bug.c:180 lib/bug.c:219)
> [21299.957147] ? handle_bug (arch/x86/kernel/traps.c:237)
> [21299.957228] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
> [21299.957308] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
> [21299.957393] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
> [21299.957476] dst_release (net/core/dst.c:166 (discriminator 1))
> [21299.957559] rt_cache_route (net/ipv4/route.c:1499)
> [21299.957641] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
> [21299.957722] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
> [21299.957804] ? free_unref_page (./include/linux/list.h:150 =
(discriminator 1) ./include/linux/list.h:169 (discriminator 1) =
mm/page_alloc.c:2377 (discriminator 1) mm/page_alloc.c:2428 =
(discriminator 1))
> [21299.957889] ip_route_input_noref (net/ipv4/route.c:2499)
> [21299.957972] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
> [21299.958058] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
> [21299.958139] ? ip_rcv_core (net/ipv4/ip_input.c:436)
> [21299.958220] process_backlog (net/core/dev.c:5997)
> [21299.958302] __napi_poll (net/core/dev.c:6556)
> [21299.958384] net_rx_action (net/core/dev.c:6625 net/core/dev.c:6756)
> [21299.958466] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
> [21299.958549] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
> [21299.958631] sysvec_call_function_single (arch/x86/kernel/smp.c:262 =
(discriminator 47))
> [21299.958714]  </IRQ>
> [21299.958792]  <TASK>
> [21299.958869] asm_sysvec_call_function_single =
(./arch/x86/include/asm/idtentry.h:656)
> [21299.958953] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
> [21299.959038] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: ed                    in     (%dx),%eax
>   1: c3                    ret
>   2: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
>   9: 00 00 00 00
>   d: 66 90                 xchg   %ax,%ax
>   f: 65 48 8b 04 25 40 32 mov    %gs:0x23240,%rax
>  16: 02 00
>  18: 48 8b 00              mov    (%rax),%rax
>  1b: a8 08                 test   $0x8,%al
>  1d: 75 0c                 jne    0x2b
>  1f: eb 07                 jmp    0x28
>  21: 0f 00 2d 57 0f 2c 00 verw   0x2c0f57(%rip)        # 0x2c0f7f
>  28: fb                    sti
>  29: f4                    hlt
>  2a:* fa                    cli     <-- trapping instruction
>  2b: c3                    ret
>  2c: 0f 1f 00              nopl   (%rax)
>  2f: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>  33: 3c 01                 cmp    $0x1,%al
>  35: 74 0b                 je     0x42
>  37: 3c 02                 cmp    $0x2,%al
>  39: 74 05                 je     0x40
>  3b: 8b 7f 04              mov    0x4(%rdi),%edi
>  3e: eb 9f                 jmp    0xffffffffffffffdf
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: fa                    cli
>   1: c3                    ret
>   2: 0f 1f 00              nopl   (%rax)
>   5: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>   9: 3c 01                 cmp    $0x1,%al
>   b: 74 0b                 je     0x18
>   d: 3c 02                 cmp    $0x2,%al
>   f: 74 05                 je     0x16
>  11: 8b 7f 04              mov    0x4(%rdi),%edi
>  14: eb 9f                 jmp    0xffffffffffffffb5
> [21299.959162] RSP: 0018:ffff96a7c015be80 EFLAGS: 00000246
> [21299.959247] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
> [21299.959343] RDX: ffff8b84bfbc0000 RSI: ffff8b75c76ba000 RDI: =
ffff8b75c76ba064
> [21299.959437] RBP: ffffffffae216ea0 R08: ffffffffae216ea0 R09: =
0000000000000003
> [21299.959533] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
> [21299.959630] R13: ffffffffae216f08 R14: ffffffffae216f20 R15: =
0000000000000000
> [21299.959725] acpi_idle_enter (drivers/acpi/processor_idle.c:709)
> [21299.959807] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:267)
> [21299.959890] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
> [21299.959975] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
> [21299.960058] cpu_startup_entry (kernel/sched/idle.c:379)
> [21299.960140] start_secondary (arch/x86/kernel/smpboot.c:326)
> [21299.960223] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
> [21299.960306]  </TASK>
> [21299.960384] ---[ end trace 0000000000000000 ]---
>=20
>> On 22 Dec 2023, at 19:26, Martin Zaharinov <micron10@gmail.com> =
wrote:
>>=20
>> Hi Thomas,
>>=20
>> this is with applyed patch from you.
>> See logs
>>=20
>>=20
>> [43040.198064] ------------[ cut here ]------------
>> [43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
>> [43040.198685] Modules linked in: pppoe pppox ppp_generic slhc =
nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb =
i2c_algo_bit e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe =
mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp =
nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat =
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler =
rtc_cmos
>> [43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           O =
      6.6.8 #1
>> [43040.199660] Hardware name: VMware, Inc. VMware Virtual =
Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
>> [43040.199886] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
>> [43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
>> [43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
>> [43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
>> [43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
>> [43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
>> [43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
>> [43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
>> [43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
>> [43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
>> [43040.201994] Call Trace:
>> [43040.202095]  <IRQ>
>> [43040.202187]  ? __warn+0x6c/0x130
>> [43040.202301]  ? report_bug+0x1b8/0x200
>> [43040.202418]  ? handle_bug+0x36/0x70
>> [43040.202534]  ? exc_invalid_op+0x17/0x1a0
>> [43040.202652]  ? asm_exc_invalid_op+0x16/0x20
>> [43040.202781]  ? rcuref_put_slowpath+0x2f/0x70
>> [43040.202909]  dst_release+0x1c/0x40
>> [43040.203026]  rt_cache_route+0xbd/0xf0
>> [43040.203143]  rt_set_nexthop.isra.0+0x1b6/0x450
>> [43040.203272]  ip_route_input_slow+0x5d9/0xcc0
>> [43040.203401]  ? nf_conntrack_udp_packet+0x17c/0x240 [nf_conntrack]
>> [43040.203581]  ip_route_input_noref+0xe0/0xf0
>> [43040.203704]  ip_rcv_finish_core.isra.0+0xbb/0x440
>> [43040.203855]  ip_rcv+0xd5/0x110
>> [43040.203962]  ? ip_rcv_core+0x360/0x360
>> [43040.204079]  process_backlog+0x107/0x210
>> [43040.204201]  __napi_poll+0x20/0x180
>> [43040.204315]  net_rx_action+0x29f/0x380
>> [43040.204432]  __do_softirq+0xd0/0x202
>> [43040.204549]  irq_exit_rcu+0x82/0xa0
>> [43040.204667]  common_interrupt+0x7a/0xa0
>> [43040.204786]  </IRQ>
>> [43040.204876]  <TASK>
>> [43040.204965]  asm_common_interrupt+0x22/0x40
>> [43040.205090] RIP: 0010:acpi_safe_halt+0x1b/0x20
>> [43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
>> [43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
>> [43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
>> [43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
>> [43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
>> [43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
>> [43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
>> [43040.206593]  acpi_idle_enter+0x77/0xc0
>> [43040.206711]  cpuidle_enter_state+0x69/0x6a0
>> [43040.206835]  cpuidle_enter+0x24/0x40
>> [43040.206954]  do_idle+0x1a7/0x210
>> [43040.207066]  cpu_startup_entry+0x21/0x30
>> [43040.207188]  start_secondary+0xe1/0xf0
>> [43040.207310]  secondary_startup_64_no_verify+0x166/0x16b
>> [43040.207451]  </TASK>
>> [43040.207542] ---[ end trace 0000000000000000 ]---
>>=20
>>=20
>>=20
>> [43040.198064] ------------[ cut here ]------------
>> [43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
>> [43040.198685] Modules linked in: pppoe pppox ppp_generic slhc =
nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb =
i2c_algo_bit e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe =
mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp =
nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat =
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler =
rtc_cmos
>> [43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           O =
      6.6.8 #1
>> [43040.199660] Hardware name: VMware, Inc. VMware Virtual =
Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
>> [43040.199886] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
>> [43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
>> All code
>> =3D=3D=3D=3D=3D=3D=3D=3D
>>  0: 07                    (bad)
>>  1: 83 f8 ff              cmp    $0xffffffff,%eax
>>  4: 75 19                 jne    0x1f
>>  6: ba 00 00 00 e0        mov    $0xe0000000,%edx
>>  b: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
>>  f: 83 f8 ff              cmp    $0xffffffff,%eax
>> 12: 74 04                 je     0x18
>> 14: 31 c0                 xor    %eax,%eax
>> 16: 5b                    pop    %rbx
>> 17: c3                    ret
>> 18: b8 01 00 00 00        mov    $0x1,%eax
>> 1d: 5b                    pop    %rbx
>> 1e: c3                    ret
>> 1f: 3d ff ff ff bf        cmp    $0xbfffffff,%eax
>> 24: 77 14                 ja     0x3a
>> 26: 85 c0                 test   %eax,%eax
>> 28: 78 06                 js     0x30
>> 2a:* 0f 0b                 ud2     <-- trapping instruction
>> 2c: 31 c0                 xor    %eax,%eax
>> 2e: eb e6                 jmp    0x16
>> 30: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>> 36: 31 c0                 xor    %eax,%eax
>> 38: eb dc                 jmp    0x16
>> 3a: 80                    .byte 0x80
>> 3b: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>>=20
>> Code starting with the faulting instruction
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>  0: 0f 0b                 ud2
>>  2: 31 c0                 xor    %eax,%eax
>>  4: eb e6                 jmp    0xffffffffffffffec
>>  6: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>>  c: 31 c0                 xor    %eax,%eax
>>  e: eb dc                 jmp    0xffffffffffffffec
>> 10: 80                    .byte 0x80
>> 11: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>> [43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
>> [43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
>> [43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
>> [43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
>> [43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
>> [43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
>> [43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
>> [43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
>> [43040.201994] Call Trace:
>> [43040.202095]  <IRQ>
>> [43040.202187] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
>> [43040.202301] ? report_bug (lib/bug.c:180 lib/bug.c:219)
>> [43040.202418] ? handle_bug (arch/x86/kernel/traps.c:237)
>> [43040.202534] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
>> [43040.202652] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
>> [43040.202781] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
>> [43040.202909] dst_release (net/core/dst.c:166 (discriminator 1))
>> [43040.203026] rt_cache_route (net/ipv4/route.c:1499)
>> [43040.203143] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
>> [43040.203272] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
>> [43040.203401] ? nf_conntrack_udp_packet =
(net/netfilter/nf_conntrack_proto_udp.c:124) nf_conntrack
>> [43040.203581] ip_route_input_noref (net/ipv4/route.c:2499)
>> [43040.203704] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
>> [43040.203855] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
>> [43040.203962] ? ip_rcv_core (net/ipv4/ip_input.c:436)
>> [43040.204079] process_backlog (net/core/dev.c:5997)
>> [43040.204201] __napi_poll (net/core/dev.c:6556)
>> [43040.204315] net_rx_action (net/core/dev.c:6625 =
net/core/dev.c:6756)
>> [43040.204432] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
>> [43040.204549] irq_exit_rcu (kernel/softirq.c:436 =
kernel/softirq.c:641 kernel/softirq.c:653)
>> [43040.204667] common_interrupt (arch/x86/kernel/irq.c:247 =
(discriminator 47))
>> [43040.204786]  </IRQ>
>> [43040.204876]  <TASK>
>> [43040.204965] asm_common_interrupt =
(./arch/x86/include/asm/idtentry.h:640)
>> [43040.205090] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
>> [43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
>> All code
>> =3D=3D=3D=3D=3D=3D=3D=3D
>>  0: ed                    in     (%dx),%eax
>>  1: c3                    ret
>>  2: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
>>  9: 00 00 00 00
>>  d: 66 90                 xchg   %ax,%ax
>>  f: 65 48 8b 04 25 40 32 mov    %gs:0x23240,%rax
>> 16: 02 00
>> 18: 48 8b 00              mov    (%rax),%rax
>> 1b: a8 08                 test   $0x8,%al
>> 1d: 75 0c                 jne    0x2b
>> 1f: eb 07                 jmp    0x28
>> 21: 0f 00 2d 57 0f 2c 00 verw   0x2c0f57(%rip)        # 0x2c0f7f
>> 28: fb                    sti
>> 29: f4                    hlt
>> 2a:* fa                    cli     <-- trapping instruction
>> 2b: c3                    ret
>> 2c: 0f 1f 00              nopl   (%rax)
>> 2f: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>> 33: 3c 01                 cmp    $0x1,%al
>> 35: 74 0b                 je     0x42
>> 37: 3c 02                 cmp    $0x2,%al
>> 39: 74 05                 je     0x40
>> 3b: 8b 7f 04              mov    0x4(%rdi),%edi
>> 3e: eb 9f                 jmp    0xffffffffffffffdf
>>=20
>> Code starting with the faulting instruction
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>  0: fa                    cli
>>  1: c3                    ret
>>  2: 0f 1f 00              nopl   (%rax)
>>  5: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>>  9: 3c 01                 cmp    $0x1,%al
>>  b: 74 0b                 je     0x18
>>  d: 3c 02                 cmp    $0x2,%al
>>  f: 74 05                 je     0x16
>> 11: 8b 7f 04              mov    0x4(%rdi),%edi
>> 14: eb 9f                 jmp    0xffffffffffffffb5
>> [43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
>> [43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
>> [43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
>> [43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
>> [43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
>> [43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
>> [43040.206593] acpi_idle_enter (drivers/acpi/processor_idle.c:709)
>> [43040.206711] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:267)
>> [43040.206835] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
>> [43040.206954] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
>> [43040.207066] cpu_startup_entry (kernel/sched/idle.c:379)
>> [43040.207188] start_secondary (arch/x86/kernel/smpboot.c:326)
>> [43040.207310] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
>> [43040.207451]  </TASK>
>> [43040.207542] ---[ end trace 0000000000000000 ]---
>>=20
>>> On 19 Dec 2023, at 16:26, Thomas Gleixner <tglx@linutronix.de> =
wrote:
>>>=20
>>> On Tue, Dec 19 2023 at 11:25, Martin Zaharinov wrote:
>>>>> On 12 Dec 2023, at 20:16, Thomas Gleixner <tglx@linutronix.de> =
wrote:
>>>>> Btw, how easy is this to reproduce?
>>>>=20
>>>> Its not easy this report is generate on machine with 5-6k users , =
with
>>>> traffic and one time is show on 1 day , other show after 4-5 =
days=E2=80=A6
>>>=20
>>> I love those bugs ...
>>>=20
>>>> Apply this patch and will upload image on one machine as fast as
>>>> possible and when get any reports will send you.
>>>=20
>>> Let's see how that goes!
>>>=20
>>> Thanks,
>>>=20
>>>      tglx
>>=20
>=20


From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from mail-lf1-f42.google.com (mail-lf1-f42.google.com [209.85.167.42])
	(using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits))
	(No client certificate requested)
	by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8819612E40
	for <netdev@vger.kernel.org>; Sun,  7 Jan 2024 11:03:42 +0000 (UTC)
Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com
Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com
Authentication-Results: smtp.subspace.kernel.org;
	dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c2FfSGlr"
Received: by mail-lf1-f42.google.com with SMTP id 2adb3069b0e04-50e67e37661so1123425e87.0
        for <netdev@vger.kernel.org>; Sun, 07 Jan 2024 03:03:42 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20230601; t=1704625420; x=1705230220; darn=vger.kernel.org;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:from:to:cc:subject:date
         :message-id:reply-to;
        bh=jhpIoPfcltdksIbdLXaQuqx8cV/sX4/Brd7fg6fTS7c=;
        b=c2FfSGlrDJKDyEeHct38b1ccBDW3d5gQECnQbQ/LjwsmGzgGwJLSz/EzUERZJDiUS1
         GsPmbvC+ipEplpXWolfxir0US30WcAnXLzx9UPqRgB9iPf8Zw3P6n1aezej3yw7ewaAr
         YHfJMNtw2ui1H+2Tb3W/pavkqV9z+GWc+3S884Lcb2Mfs9uOZh9ghnJ5KfafVpf6VeFb
         oj+N9lqzocX9SuMdYK+hQaIQ+yGFZLWvrIi2D73nioJGNKiWMa0iu9sCDFfnOcYMwfp+
         js9DVli3gfXe+E9xvBva2h4YznA60BpMMQmkUWEpv37EXzKPss/NUOoYbhRSDwBineBi
         5fEA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20230601; t=1704625420; x=1705230220;
        h=to:references:message-id:content-transfer-encoding:cc:date
         :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc
         :subject:date:message-id:reply-to;
        bh=jhpIoPfcltdksIbdLXaQuqx8cV/sX4/Brd7fg6fTS7c=;
        b=WnMC96YeztnJJFEK2FeTD0QdTo8+08s3WJui+O38756JRmOUaO4QiEGPe9z3pRckn5
         8VHQyhEuh7fdCgtvH7I+8Yi2RxQ2USbuijw0ZpYL9/pEE6wTI0PA7oFHr6RuSCXttV0r
         PuKLZ11BV8ssS4OmuG+MRhJk4mvwi8m4hNG7RINXg8pcX+C0fAOjWVtfw8I2+dFRFq1U
         b5WDxIzhCn6sBy/AxAgbpvyHcTN02/LNpVLXpN/qUgwmVHnWdpavNP2Yvw+odwRnO6w8
         xKrlpuuqoArhBymZahpT3xoii2fyxlAww+VtoeoVA9GkNsPn+1bTh99U494WPkNQpYvz
         3W0g==
X-Gm-Message-State: AOJu0YyN7g312LTP36nZFQN0P6LPBt5A9l2O6hwkIMHN7NC+Tl4SPEff
	BypuIdWyrzlSHyHJMnB/NX0=
X-Google-Smtp-Source: AGHT+IEAebcXio2m4DRdr8xHmxE7NN1B2wLXycs55xv5oCEuwHDEbxbb4k+RCOYSqf9njINnwfP4bw==
X-Received: by 2002:a19:ca04:0:b0:50e:3e3a:3d6e with SMTP id a4-20020a19ca04000000b0050e3e3a3d6emr750816lfg.110.1704625419618;
        Sun, 07 Jan 2024 03:03:39 -0800 (PST)
Received: from smtpclient.apple ([178.254.237.20])
        by smtp.gmail.com with ESMTPSA id b1-20020a170906490100b00a26b36311ecsm2966535ejq.146.2024.01.07.03.03.38
        (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
        Sun, 07 Jan 2024 03:03:38 -0800 (PST)
Content-Type: text/plain;
	charset=utf-8
Precedence: bulk
X-Mailing-List: netdev@vger.kernel.org
List-Id: <netdev.vger.kernel.org>
List-Subscribe: <mailto:netdev+subscribe@vger.kernel.org>
List-Unsubscribe: <mailto:netdev+unsubscribe@vger.kernel.org>
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.200.91.1.1\))
Subject: Re: Urgent Bug Report Kernel crash 6.5.2
From: Martin Zaharinov <micron10@gmail.com>
In-Reply-To: <E6A168E5-5BB6-47AD-B8D6-D6BE2F2C528F@gmail.com>
Date: Sun, 7 Jan 2024 13:03:27 +0200
Cc: peterz@infradead.org,
 netdev <netdev@vger.kernel.org>,
 Paolo Abeni <pabeni@redhat.com>,
 patchwork-bot+netdevbpf@kernel.org,
 Jakub Kicinski <kuba@kernel.org>,
 Stephen Hemminger <stephen@networkplumber.org>,
 kuba+netdrv@kernel.org,
 dsahern@gmail.com,
 Eric Dumazet <edumazet@google.com>
Content-Transfer-Encoding: quoted-printable
Message-Id: <1CD1762C-C6C0-4840-A4BD-3A77F40CA02A@gmail.com>
References: <64CCB695-BA43-48F5-912A-AFD5B9C103A7@gmail.com>
 <CANn89iL9Twf+Rzm9v_dwsH_iG4YkW3fAc2Hnx2jypN_Qf9oojw@mail.gmail.com>
 <D773F198-BCE3-4D43-9C27-2C2CA34062AC@gmail.com>
 <8E92BAA8-0FC6-4D29-BB4D-B6B60047A1D2@gmail.com>
 <5E63894D-913B-416C-B901-F628BB6C00E0@gmail.com> <87lea4qqun.ffs@tglx>
 <2B5C19AE-C125-45A3-8C6F-CA6BBC01A6D9@gmail.com> <87r0jrp9qi.ffs@tglx>
 <6D816814-1334-4F22-AFF8-B5E42254038E@gmail.com> <87v88ul14z.ffs@tglx>
 <FBF0EB18-E2B6-4076-968A-5F2ABE0F27E4@gmail.com>
 <7D08EC48-F22C-4CC8-839F-2A9677E93DF0@gmail.com>
 <E6A168E5-5BB6-47AD-B8D6-D6BE2F2C528F@gmail.com>
To: Thomas Gleixner <tglx@linutronix.de>
X-Mailer: Apple Mail (2.3774.200.91.1.1)

Hi Thomas=20

this is one more report from one machine=20

Here you will see have to bug report in same day:


[Sat Jan  6 07:37:23 2024] ------------[ cut here ]------------
[Sat Jan 6 07:37:23 2024] WARNING: CPU: 12 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
[Sat Jan  6 07:37:23 2024] Modules linked in:  pppoe pppox ppp_generic =
slhc nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe mdio =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos megaraid_sas
[Sat Jan  6 07:37:23 2024] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G    =
       O       6.6.10 #1
[Sat Jan  6 07:37:23 2024] Hardware name: Huawei 2288H V5/BC11SPSCB0, =
BIOS 7.80 10/28/2020
[Sat Jan 6 07:37:23 2024] RIP: 0010:rcuref_put_slowpath =
(lib/rcuref.c:294 (discriminator 1))
[Sat Jan 6 07:37:23 2024] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f =
b1 17 83 f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 =
14 85 c0 78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d =
b2 4a e3 00
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	07                   	(bad)
   1:	83 f8 ff             	cmp    $0xffffffff,%eax
   4:	75 19                	jne    0x1f
   6:	ba 00 00 00 e0       	mov    $0xe0000000,%edx
   b:	f0 0f b1 17          	lock cmpxchg %edx,(%rdi)
   f:	83 f8 ff             	cmp    $0xffffffff,%eax
  12:	74 04                	je     0x18
  14:	31 c0                	xor    %eax,%eax
  16:	5b                   	pop    %rbx
  17:	c3                   	ret
  18:	b8 01 00 00 00       	mov    $0x1,%eax
  1d:	5b                   	pop    %rbx
  1e:	c3                   	ret
  1f:	3d ff ff ff bf       	cmp    $0xbfffffff,%eax
  24:	77 14                	ja     0x3a
  26:	85 c0                	test   %eax,%eax
  28:	78 06                	js     0x30
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	31 c0                	xor    %eax,%eax
  2e:	eb e6                	jmp    0x16
  30:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
  36:	31 c0                	xor    %eax,%eax
  38:	eb dc                	jmp    0x16
  3a:	80                   	.byte 0x80
  3b:	3d b2 4a e3 00       	cmp    $0xe34ab2,%eax

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	31 c0                	xor    %eax,%eax
   4:	eb e6                	jmp    0xffffffffffffffec
   6:	c7 07 00 00 00 a0    	movl   $0xa0000000,(%rdi)
   c:	31 c0                	xor    %eax,%eax
   e:	eb dc                	jmp    0xffffffffffffffec
  10:	80                   	.byte 0x80
  11:	3d b2 4a e3 00       	cmp    $0xe34ab2,%eax
[Sat Jan  6 07:37:23 2024] RSP: 0018:ffffa773091ccdd8 EFLAGS: 00010246
[Sat Jan  6 07:37:23 2024] RAX: 0000000000000000 RBX: ffff8f458c192d00 =
RCX: 0000000000000042
[Sat Jan  6 07:37:23 2024] RDX: ffff8f455ad71800 RSI: 0000000000000000 =
RDI: ffff8f458c192d00
[Sat Jan  6 07:37:23 2024] RBP: ffff8f455ad71ec0 R08: 0000000000000000 =
R09: 0000000000000000
[Sat Jan  6 07:37:23 2024] R10: 0000000000000002 R11: ffffa773091ccd90 =
R12: ffff8f25c68df800
[Sat Jan  6 07:37:23 2024] R13: 000000000000000e R14: 0000000000000010 =
R15: ffff8f64bf8a4d10
[Sat Jan  6 07:37:23 2024] FS:  0000000000000000(0000) =
GS:ffff8f64bf880000(0000) knlGS:0000000000000000
[Sat Jan  6 07:37:23 2024] CS:  0010 DS: 0000 ES: 0000 CR0: =
0000000080050033
[Sat Jan  6 07:37:23 2024] CR2: 00007fbd91318650 CR3: 000000177e014005 =
CR4: 00000000003706e0
[Sat Jan  6 07:37:23 2024] DR0: 0000000000000000 DR1: 0000000000000000 =
DR2: 0000000000000000
[Sat Jan  6 07:37:23 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 =
DR7: 0000000000000400
[Sat Jan  6 07:37:23 2024] Call Trace:
[Sat Jan  6 07:37:23 2024]  <IRQ>
[Sat Jan 6 07:37:23 2024] ? __warn (kernel/panic.c:235 =
kernel/panic.c:673)
[Sat Jan 6 07:37:23 2024] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[Sat Jan 6 07:37:23 2024] ? handle_bug (arch/x86/kernel/traps.c:237)
[Sat Jan 6 07:37:23 2024] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
[Sat Jan 6 07:37:23 2024] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[Sat Jan 6 07:37:23 2024] ? rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
[Sat Jan 6 07:37:23 2024] dst_release (net/core/dst.c:166 (discriminator =
1))
[Sat Jan 6 07:37:23 2024] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4327)
[Sat Jan 6 07:37:23 2024] ? nf_hook_slow =
(./include/linux/netfilter.h:144 net/netfilter/core.c:626)
[Sat Jan 6 07:37:23 2024] ip_finish_output2 =
(./include/net/neighbour.h:526 ./include/net/neighbour.h:540 =
net/ipv4/ip_output.c:233)
[Sat Jan 6 07:37:23 2024] process_backlog (net/core/dev.c:6000)
[Sat Jan 6 07:37:23 2024] __napi_poll (net/core/dev.c:6559)
[Sat Jan 6 07:37:23 2024] net_rx_action (net/core/dev.c:6628 =
net/core/dev.c:6759)
[Sat Jan 6 07:37:23 2024] __do_softirq =
(./arch/x86/include/asm/preempt.h:27 kernel/softirq.c:564)
[Sat Jan 6 07:37:23 2024] irq_exit_rcu (kernel/softirq.c:436 =
kernel/softirq.c:641 kernel/softirq.c:653)
[Sat Jan 6 07:37:23 2024] sysvec_call_function_single =
(arch/x86/kernel/smp.c:262 (discriminator 47))
[Sat Jan  6 07:37:23 2024]  </IRQ>
[Sat Jan  6 07:37:23 2024]  <TASK>
[Sat Jan 6 07:37:23 2024] asm_sysvec_call_function_single =
(./arch/x86/include/asm/idtentry.h:656)
[Sat Jan 6 07:37:23 2024] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
[Sat Jan 6 07:37:23 2024] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 =
66 90 65 48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d c7 =
0c 2c 00 fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b =
7f 04 eb 9f
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	ed                   	in     (%dx),%eax
   1:	c3                   	ret
   2:	66 66 2e 0f 1f 84 00 	data16 cs nopw 0x0(%rax,%rax,1)
   9:	00 00 00 00
   d:	66 90                	xchg   %ax,%ax
   f:	65 48 8b 04 25 40 32 	mov    %gs:0x23240,%rax
  16:	02 00
  18:	48 8b 00             	mov    (%rax),%rax
  1b:	a8 08                	test   $0x8,%al
  1d:	75 0c                	jne    0x2b
  1f:	eb 07                	jmp    0x28
  21:	0f 00 2d c7 0c 2c 00 	verw   0x2c0cc7(%rip)        # 0x2c0cef
  28:	fb                   	sti
  29:	f4                   	hlt
  2a:*	fa                   	cli    		<-- trapping instruction
  2b:	c3                   	ret
  2c:	0f 1f 00             	nopl   (%rax)
  2f:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
  33:	3c 01                	cmp    $0x1,%al
  35:	74 0b                	je     0x42
  37:	3c 02                	cmp    $0x2,%al
  39:	74 05                	je     0x40
  3b:	8b 7f 04             	mov    0x4(%rdi),%edi
  3e:	eb 9f                	jmp    0xffffffffffffffdf

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	fa                   	cli
   1:	c3                   	ret
   2:	0f 1f 00             	nopl   (%rax)
   5:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
   9:	3c 01                	cmp    $0x1,%al
   b:	74 0b                	je     0x18
   d:	3c 02                	cmp    $0x2,%al
   f:	74 05                	je     0x16
  11:	8b 7f 04             	mov    0x4(%rdi),%edi
  14:	eb 9f                	jmp    0xffffffffffffffb5
[Sat Jan  6 07:37:23 2024] RSP: 0018:ffffa773007fbe80 EFLAGS: 00000246
[Sat Jan  6 07:37:23 2024] RAX: 0000000000004000 RBX: 0000000000000001 =
RCX: 000000000000001f
[Sat Jan  6 07:37:23 2024] RDX: ffff8f64bf880000 RSI: ffff8f454c6f6800 =
RDI: ffff8f454c6f6864
[Sat Jan  6 07:37:23 2024] RBP: ffffffffaa216ea0 R08: ffffffffaa216ea0 =
R09: 0000000000000003
[Sat Jan  6 07:37:23 2024] R10: 0000000000000002 R11: 0000000000000007 =
R12: 0000000000000001
[Sat Jan  6 07:37:23 2024] R13: ffffffffaa216f08 R14: ffffffffaa216f20 =
R15: 0000000000000000
[Sat Jan 6 07:37:23 2024] acpi_idle_enter =
(drivers/acpi/processor_idle.c:709)
[Sat Jan 6 07:37:23 2024] cpuidle_enter_state =
(drivers/cpuidle/cpuidle.c:267)
[Sat Jan 6 07:37:23 2024] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
[Sat Jan 6 07:37:23 2024] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
[Sat Jan 6 07:37:23 2024] cpu_startup_entry (kernel/sched/idle.c:379)
[Sat Jan 6 07:37:23 2024] start_secondary =
(arch/x86/kernel/smpboot.c:326)
[Sat Jan 6 07:37:23 2024] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:449)
[Sat Jan  6 07:37:23 2024]  </TASK>
[Sat Jan  6 07:37:23 2024] ---[ end trace 0000000000000000 ]---
[Sat Jan  6 21:33:28 2024] ------------[ cut here ]------------
[Sat Jan  6 21:33:28 2024] rcuref - imbalanced put()
[Sat Jan 6 21:33:28 2024] WARNING: CPU: 26 PID: 0 at lib/rcuref.c:279 =
rcuref_put_slowpath (lib/rcuref.c:279 (discriminator 1))
[Sat Jan  6 21:33:28 2024] Modules linked in: pppoe pppox ppp_generic =
slhc nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe mdio =
nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 ipmi_si ipmi_devintf ipmi_msghandler =
rtc_cmos megaraid_sas
[Sat Jan  6 21:33:28 2024] CPU: 26 PID: 0 Comm: swapper/26 Tainted: G    =
    W  O       6.6.10 #1
[Sat Jan  6 21:33:28 2024] Hardware name: Huawei 2288H V5/BC11SPSCB0, =
BIOS 7.80 10/28/2020
[Sat Jan 6 21:33:28 2024] RIP: 0010:rcuref_put_slowpath =
(lib/rcuref.c:279 (discriminator 1))
[Sat Jan 6 21:33:28 2024] Code: 31 c0 eb dc 80 3d b2 4a e3 00 00 74 0a =
c7 03 00 00 00 e0 31 c0 eb c9 48 c7 c7 54 29 e4 a9 c6 05 98 4a e3 00 01 =
e8 db 7c c3 ff <0f> 0b eb df cc cc cc cc cc cc cc 48 89 fa 83 e2 07 48 =
85 f6 74 7f
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	31 c0                	xor    %eax,%eax
   2:	eb dc                	jmp    0xffffffffffffffe0
   4:	80 3d b2 4a e3 00 00 	cmpb   $0x0,0xe34ab2(%rip)        # =
0xe34abd
   b:	74 0a                	je     0x17
   d:	c7 03 00 00 00 e0    	movl   $0xe0000000,(%rbx)
  13:	31 c0                	xor    %eax,%eax
  15:	eb c9                	jmp    0xffffffffffffffe0
  17:	48 c7 c7 54 29 e4 a9 	mov    $0xffffffffa9e42954,%rdi
  1e:	c6 05 98 4a e3 00 01 	movb   $0x1,0xe34a98(%rip)        # =
0xe34abd
  25:	e8 db 7c c3 ff       	call   0xffffffffffc37d05
  2a:*	0f 0b                	ud2    		<-- trapping instruction
  2c:	eb df                	jmp    0xd
  2e:	cc                   	int3
  2f:	cc                   	int3
  30:	cc                   	int3
  31:	cc                   	int3
  32:	cc                   	int3
  33:	cc                   	int3
  34:	cc                   	int3
  35:	48 89 fa             	mov    %rdi,%rdx
  38:	83 e2 07             	and    $0x7,%edx
  3b:	48 85 f6             	test   %rsi,%rsi
  3e:	74 7f                	je     0xbf

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	0f 0b                	ud2
   2:	eb df                	jmp    0xffffffffffffffe3
   4:	cc                   	int3
   5:	cc                   	int3
   6:	cc                   	int3
   7:	cc                   	int3
   8:	cc                   	int3
   9:	cc                   	int3
   a:	cc                   	int3
   b:	48 89 fa             	mov    %rdi,%rdx
   e:	83 e2 07             	and    $0x7,%edx
  11:	48 85 f6             	test   %rsi,%rsi
  14:	74 7f                	je     0x95
[Sat Jan  6 21:33:28 2024] RSP: 0018:ffffa7730d528dd8 EFLAGS: 00010292
[Sat Jan  6 21:33:28 2024] RAX: 0000000000000019 RBX: ffff8f4573f7d000 =
RCX: 00000000ffefffff
[Sat Jan  6 21:33:28 2024] RDX: 00000000ffefffff RSI: 0000000000000001 =
RDI: 00000000ffffffea
[Sat Jan  6 21:33:28 2024] RBP: ffff8f265f02d6c0 R08: 0000000000000000 =
R09: 00000000ffefffff
[Sat Jan  6 21:33:28 2024] R10: ffff8f64b6800000 R11: 0000000000000003 =
R12: ffff8f25c68df800
[Sat Jan  6 21:33:28 2024] R13: 000000000000000e R14: 0000000000000010 =
R15: ffff8f44c0024d10
[Sat Jan  6 21:33:28 2024] FS:  0000000000000000(0000) =
GS:ffff8f44c0000000(0000) knlGS:0000000000000000
[Sat Jan  6 21:33:28 2024] CS:  0010 DS: 0000 ES: 0000 CR0: =
0000000080050033
[Sat Jan  6 21:33:28 2024] CR2: 00007fd79aca5000 CR3: 000000015d226005 =
CR4: 00000000003706e0
[Sat Jan  6 21:33:28 2024] DR0: 0000000000000000 DR1: 0000000000000000 =
DR2: 0000000000000000
[Sat Jan  6 21:33:28 2024] DR3: 0000000000000000 DR6: 00000000fffe0ff0 =
DR7: 0000000000000400
[Sat Jan  6 21:33:28 2024] Call Trace:
[Sat Jan  6 21:33:28 2024]  <IRQ>
[Sat Jan 6 21:33:28 2024] ? __warn (kernel/panic.c:235 =
kernel/panic.c:673)
[Sat Jan 6 21:33:28 2024] ? report_bug (lib/bug.c:180 lib/bug.c:219)
[Sat Jan 6 21:33:28 2024] ? handle_bug (arch/x86/kernel/traps.c:237)
[Sat Jan 6 21:33:28 2024] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
[Sat Jan 6 21:33:28 2024] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
[Sat Jan 6 21:33:28 2024] ? rcuref_put_slowpath (lib/rcuref.c:279 =
(discriminator 1))
[Sat Jan 6 21:33:28 2024] ? rcuref_put_slowpath (lib/rcuref.c:279 =
(discriminator 1))
[Sat Jan 6 21:33:28 2024] dst_release (net/core/dst.c:166 (discriminator =
1))
[Sat Jan 6 21:33:28 2024] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4327)
[Sat Jan 6 21:33:28 2024] ? nf_hook_slow =
(./include/linux/netfilter.h:144 net/netfilter/core.c:626)
[Sat Jan 6 21:33:28 2024] ip_finish_output2 =
(./include/net/neighbour.h:526 ./include/net/neighbour.h:540 =
net/ipv4/ip_output.c:233)
[Sat Jan 6 21:33:28 2024] process_backlog (net/core/dev.c:6000)
[Sat Jan 6 21:33:28 2024] __napi_poll (net/core/dev.c:6559)
[Sat Jan 6 21:33:28 2024] net_rx_action (net/core/dev.c:6628 =
net/core/dev.c:6759)
[Sat Jan 6 21:33:28 2024] __do_softirq =
(./arch/x86/include/asm/preempt.h:27 kernel/softirq.c:564)
[Sat Jan 6 21:33:28 2024] irq_exit_rcu (kernel/softirq.c:436 =
kernel/softirq.c:641 kernel/softirq.c:653)
[Sat Jan 6 21:33:28 2024] sysvec_call_function_single =
(arch/x86/kernel/smp.c:262 (discriminator 47))
[Sat Jan  6 21:33:28 2024]  </IRQ>
[Sat Jan  6 21:33:28 2024]  <TASK>
[Sat Jan 6 21:33:28 2024] asm_sysvec_call_function_single =
(./arch/x86/include/asm/idtentry.h:656)
[Sat Jan 6 21:33:28 2024] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
[Sat Jan 6 21:33:28 2024] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 =
66 90 65 48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d c7 =
0c 2c 00 fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b =
7f 04 eb 9f
All code
=3D=3D=3D=3D=3D=3D=3D=3D
   0:	ed                   	in     (%dx),%eax
   1:	c3                   	ret
   2:	66 66 2e 0f 1f 84 00 	data16 cs nopw 0x0(%rax,%rax,1)
   9:	00 00 00 00
   d:	66 90                	xchg   %ax,%ax
   f:	65 48 8b 04 25 40 32 	mov    %gs:0x23240,%rax
  16:	02 00
  18:	48 8b 00             	mov    (%rax),%rax
  1b:	a8 08                	test   $0x8,%al
  1d:	75 0c                	jne    0x2b
  1f:	eb 07                	jmp    0x28
  21:	0f 00 2d c7 0c 2c 00 	verw   0x2c0cc7(%rip)        # 0x2c0cef
  28:	fb                   	sti
  29:	f4                   	hlt
  2a:*	fa                   	cli    		<-- trapping instruction
  2b:	c3                   	ret
  2c:	0f 1f 00             	nopl   (%rax)
  2f:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
  33:	3c 01                	cmp    $0x1,%al
  35:	74 0b                	je     0x42
  37:	3c 02                	cmp    $0x2,%al
  39:	74 05                	je     0x40
  3b:	8b 7f 04             	mov    0x4(%rdi),%edi
  3e:	eb 9f                	jmp    0xffffffffffffffdf

Code starting with the faulting instruction
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
   0:	fa                   	cli
   1:	c3                   	ret
   2:	0f 1f 00             	nopl   (%rax)
   5:	0f b6 47 08          	movzbl 0x8(%rdi),%eax
   9:	3c 01                	cmp    $0x1,%al
   b:	74 0b                	je     0x18
   d:	3c 02                	cmp    $0x2,%al
   f:	74 05                	je     0x16
  11:	8b 7f 04             	mov    0x4(%rdi),%edi
  14:	eb 9f                	jmp    0xffffffffffffffb5
[Sat Jan  6 21:33:28 2024] RSP: 0018:ffffa77300e7be80 EFLAGS: 00000246
[Sat Jan  6 21:33:28 2024] RAX: 0000000000004000 RBX: 0000000000000001 =
RCX: 000000000000001f
[Sat Jan  6 21:33:28 2024] RDX: ffff8f44c0000000 RSI: ffff8f454c6fc800 =
RDI: ffff8f454c6fc864
[Sat Jan  6 21:33:28 2024] RBP: ffffffffaa216ea0 R08: ffffffffaa216ea0 =
R09: 00003fd7b44cb0a0
[Sat Jan  6 21:33:28 2024] R10: 0000000000000002 R11: 0000000000000007 =
R12: 0000000000000001
[Sat Jan  6 21:33:28 2024] R13: ffffffffaa216f08 R14: ffffffffaa216f20 =
R15: 0000000000000000
[Sat Jan 6 21:33:28 2024] acpi_idle_enter =
(drivers/acpi/processor_idle.c:709)
[Sat Jan 6 21:33:28 2024] cpuidle_enter_state =
(drivers/cpuidle/cpuidle.c:267)
[Sat Jan 6 21:33:28 2024] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
[Sat Jan 6 21:33:28 2024] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
[Sat Jan 6 21:33:28 2024] cpu_startup_entry (kernel/sched/idle.c:379)
[Sat Jan 6 21:33:28 2024] start_secondary =
(arch/x86/kernel/smpboot.c:326)
[Sat Jan 6 21:33:28 2024] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:449)
[Sat Jan  6 21:33:28 2024]  </TASK>
[Sat Jan  6 21:33:28 2024] ---[ end trace 0000000000000000 ]---

> On 4 Jan 2024, at 22:51, Martin Zaharinov <micron10@gmail.com> wrote:
>=20
> Hi Thomas ,
>=20
> Happy New Year!
>=20
> here is two debugs from two new installed machins with kernel 6.6.9:
>=20
> dmesg1 :
>=20
> [ 2257.449125] ------------[ cut here ]------------
> [ 2257.449245] WARNING: CPU: 1 PID: 40622 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
> [ 2257.449373] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
> [ 2257.449642] CPU: 1 PID: 40622 Comm: nc Tainted: G           O       =
6.6.9 #1
> [ 2257.449761] Hardware name: Supermicro =
PIO-5038MR-H8TRF-NODE/X10SRD-F, BIOS 3.3 10/28/2020
> [ 2257.449883] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
> [ 2257.449977] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 =
00
> [ 2257.450135] RSP: 0000:ffffb455cef83b78 EFLAGS: 00010246
> [ 2257.450227] RAX: 0000000000000000 RBX: ffff94873bb77dc0 RCX: =
ffff9486c0d46b80
> [ 2257.450341] RDX: ffff948736578428 RSI: 00000000fffffe01 RDI: =
ffff94873bb77dc0
> [ 2257.450456] RBP: ffff948736578428 R08: ffff948e1fa64f08 R09: =
0000000000000001
> [ 2257.450570] R10: 0000000000028530 R11: 0000000000000001 R12: =
ffff94873bb77d80
> [ 2257.450685] R13: ffff94873bb77de8 R14: ffff948e1fa64f08 R15: =
000000000266f59d
> [ 2257.450802] FS:  00007f0cdbc73800(0000) GS:ffff948e1fa40000(0000) =
knlGS:0000000000000000
> [ 2257.450918] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2257.451012] CR2: 00007f0cdc3f5c30 CR3: 0000000178ea0002 CR4: =
00000000003706e0
> [ 2257.451127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
> [ 2257.451240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
> [ 2257.451353] Call Trace:
> [ 2257.451441]  <TASK>
> [ 2257.451526]  ? __warn+0x6c/0x130
> [ 2257.451616]  ? report_bug+0x1b8/0x200
> [ 2257.451707]  ? handle_bug+0x36/0x70
> [ 2257.451797]  ? exc_invalid_op+0x17/0x1a0
> [ 2257.451886]  ? asm_exc_invalid_op+0x16/0x20
> [ 2257.452038]  ? rcuref_put_slowpath+0x2f/0x70
> [ 2257.452129]  dst_release+0x1c/0x40
> [ 2257.452222]  rt_cache_route+0xbd/0xf0
> [ 2257.452313]  ? kmem_cache_alloc+0x31/0x390
> [ 2257.452404]  rt_set_nexthop.isra.0+0x1b6/0x450
> [ 2257.452495]  ip_route_input_slow+0x5d9/0xcc0
> [ 2257.452586]  ? nft_nat_do_chain+0x7f/0xd0 [nft_chain_nat]
> [ 2257.452681]  ? nf_conntrack_udp_packet+0xcf/0x240 [nf_conntrack]
> [ 2257.452784]  ? nf_nat_inet_fn+0x36f/0x3f0 [nf_nat]
> [ 2257.452880]  ip_route_input_noref+0xe0/0xf0
> [ 2257.452970]  ip_rcv_finish_core.isra.0+0xbb/0x440
> [ 2257.453064]  ip_rcv+0xd5/0x110
> [ 2257.453151]  ? ip_rcv_core+0x360/0x360
> [ 2257.453240]  process_backlog+0x107/0x210
> [ 2257.453330]  __napi_poll+0x20/0x180
> [ 2257.453420]  net_rx_action+0x29f/0x380
> [ 2257.453510]  __do_softirq+0xd0/0x202
> [ 2257.453599]  irq_exit_rcu+0x82/0xa0
> [ 2257.453689]  sysvec_call_function_single+0x32/0x80
> [ 2257.453781]  asm_sysvec_call_function_single+0x16/0x20
> [ 2257.453874] RIP: 0033:0x7f0cdc5928b2
> [ 2257.453963] Code: 06 00 00 4c 89 65 88 49 83 fd 08 0f 84 f7 06 00 =
00 49 83 fd 26 0f 84 05 07 00 00 4d 85 ed 0f 84 5f 01 00 00 41 0f b6 44 =
24 04 <89> c6 40 c0 ee 04 0f 84 72 06 00 00 41 0f b6 54 24 05 83 e2 03 =
ff
> [ 2257.454121] RSP: 002b:00007ffc04d3e890 EFLAGS: 00000206
> [ 2257.454215] RAX: 0000000000000012 RBX: 00007f0cdc444db8 RCX: =
00007f0cdc4e6e60
> [ 2257.454329] RDX: 0000000000000009 RSI: 00007f0cdc57ef30 RDI: =
00007f0cdc42c808
> [ 2257.454442] RBP: 00007ffc04d3e9b0 R08: 00007f0cdc445028 R09: =
00007ffc04d3e940
> [ 2257.454555] R10: 00007f0cdbf00be8 R11: 0000000000000000 R12: =
00007f0cdc42c898
> [ 2257.454670] R13: 0000000000000006 R14: 0000000600000006 R15: =
00007f0cdc581000
> [ 2257.454784]  </TASK>
> [ 2257.454869] ---[ end trace 0000000000000000 ]=E2=80=94
>=20
>=20
> [ 2257.449125] ------------[ cut here ]------------
> [ 2257.449245] WARNING: CPU: 1 PID: 40622 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
> [ 2257.449373] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
> [ 2257.449642] CPU: 1 PID: 40622 Comm: nc Tainted: G           O       =
6.6.9 #1
> [ 2257.449761] Hardware name: Supermicro =
PIO-5038MR-H8TRF-NODE/X10SRD-F, BIOS 3.3 10/28/2020
> [ 2257.449883] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
> [ 2257.449977] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 =
00
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 07                    (bad)
>   1: 83 f8 ff              cmp    $0xffffffff,%eax
>   4: 75 19                 jne    0x1f
>   6: ba 00 00 00 e0        mov    $0xe0000000,%edx
>   b: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
>   f: 83 f8 ff              cmp    $0xffffffff,%eax
>  12: 74 04                 je     0x18
>  14: 31 c0                 xor    %eax,%eax
>  16: 5b                    pop    %rbx
>  17: c3                    ret
>  18: b8 01 00 00 00        mov    $0x1,%eax
>  1d: 5b                    pop    %rbx
>  1e: c3                    ret
>  1f: 3d ff ff ff bf        cmp    $0xbfffffff,%eax
>  24: 77 14                 ja     0x3a
>  26: 85 c0                 test   %eax,%eax
>  28: 78 06                 js     0x30
>  2a:* 0f 0b                 ud2     <-- trapping instruction
>  2c: 31 c0                 xor    %eax,%eax
>  2e: eb e6                 jmp    0x16
>  30: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>  36: 31 c0                 xor    %eax,%eax
>  38: eb dc                 jmp    0x16
>  3a: 80                    .byte 0x80
>  3b: 3d e2 4c e3 00        cmp    $0xe34ce2,%eax
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 0f 0b                 ud2
>   2: 31 c0                 xor    %eax,%eax
>   4: eb e6                 jmp    0xffffffffffffffec
>   6: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>   c: 31 c0                 xor    %eax,%eax
>   e: eb dc                 jmp    0xffffffffffffffec
>  10: 80                    .byte 0x80
>  11: 3d e2 4c e3 00        cmp    $0xe34ce2,%eax
> [ 2257.450135] RSP: 0000:ffffb455cef83b78 EFLAGS: 00010246
> [ 2257.450227] RAX: 0000000000000000 RBX: ffff94873bb77dc0 RCX: =
ffff9486c0d46b80
> [ 2257.450341] RDX: ffff948736578428 RSI: 00000000fffffe01 RDI: =
ffff94873bb77dc0
> [ 2257.450456] RBP: ffff948736578428 R08: ffff948e1fa64f08 R09: =
0000000000000001
> [ 2257.450570] R10: 0000000000028530 R11: 0000000000000001 R12: =
ffff94873bb77d80
> [ 2257.450685] R13: ffff94873bb77de8 R14: ffff948e1fa64f08 R15: =
000000000266f59d
> [ 2257.450802] FS:  00007f0cdbc73800(0000) GS:ffff948e1fa40000(0000) =
knlGS:0000000000000000
> [ 2257.450918] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2257.451012] CR2: 00007f0cdc3f5c30 CR3: 0000000178ea0002 CR4: =
00000000003706e0
> [ 2257.451127] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
> [ 2257.451240] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
> [ 2257.451353] Call Trace:
> [ 2257.451441]  <TASK>
> [ 2257.451526] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
> [ 2257.451616] ? report_bug (lib/bug.c:180 lib/bug.c:219)
> [ 2257.451707] ? handle_bug (arch/x86/kernel/traps.c:237)
> [ 2257.451797] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
> [ 2257.451886] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
> [ 2257.452038] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
> [ 2257.452129] dst_release (net/core/dst.c:166 (discriminator 1))
> [ 2257.452222] rt_cache_route (net/ipv4/route.c:1499)
> [ 2257.452313] ? kmem_cache_alloc (mm/slab.h:711 (discriminator 1) =
mm/slub.c:3461 (discriminator 1) mm/slub.c:3487 (discriminator 1) =
mm/slub.c:3494 (discriminator 1) mm/slub.c:3503 (discriminator 1))
> [ 2257.452404] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
> [ 2257.452495] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
> [ 2257.452586] ? nft_nat_do_chain (net/netfilter/nft_chain_nat.c:33) =
nft_chain_nat
> [ 2257.452681] ? nf_conntrack_udp_packet =
(net/netfilter/nf_conntrack_proto_udp.c:130) nf_conntrack
> [ 2257.452784] ? nf_nat_inet_fn (net/netfilter/nf_nat_core.c:844) =
nf_nat
> [ 2257.452880] ip_route_input_noref (net/ipv4/route.c:2499)
> [ 2257.452970] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
> [ 2257.453064] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
> [ 2257.453151] ? ip_rcv_core (net/ipv4/ip_input.c:436)
> [ 2257.453240] process_backlog (net/core/dev.c:6000)
> [ 2257.453330] __napi_poll (net/core/dev.c:6559)
> [ 2257.453420] net_rx_action (net/core/dev.c:6628 net/core/dev.c:6759)
> [ 2257.453510] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
> [ 2257.453599] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
> [ 2257.453689] sysvec_call_function_single (arch/x86/kernel/smp.c:262 =
(discriminator 69))
> [ 2257.453781] asm_sysvec_call_function_single =
(./arch/x86/include/asm/idtentry.h:656)
> [ 2257.453874] RIP: 0033:0x7f0cdc5928b2
> [ 2257.453963] Code: 06 00 00 4c 89 65 88 49 83 fd 08 0f 84 f7 06 00 =
00 49 83 fd 26 0f 84 05 07 00 00 4d 85 ed 0f 84 5f 01 00 00 41 0f b6 44 =
24 04 <89> c6 40 c0 ee 04 0f 84 72 06 00 00 41 0f b6 54 24 05 83 e2 03 =
ff
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 06                    (bad)
>   1: 00 00                 add    %al,(%rax)
>   3: 4c 89 65 88           mov    %r12,-0x78(%rbp)
>   7: 49 83 fd 08           cmp    $0x8,%r13
>   b: 0f 84 f7 06 00 00     je     0x708
>  11: 49 83 fd 26           cmp    $0x26,%r13
>  15: 0f 84 05 07 00 00     je     0x720
>  1b: 4d 85 ed              test   %r13,%r13
>  1e: 0f 84 5f 01 00 00     je     0x183
>  24: 41 0f b6 44 24 04     movzbl 0x4(%r12),%eax
>  2a:* 89 c6                 mov    %eax,%esi <-- trapping instruction
>  2c: 40 c0 ee 04           shr    $0x4,%sil
>  30: 0f 84 72 06 00 00     je     0x6a8
>  36: 41 0f b6 54 24 05     movzbl 0x5(%r12),%edx
>  3c: 83 e2 03              and    $0x3,%edx
>  3f: ff                    .byte 0xff
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 89 c6                 mov    %eax,%esi
>   2: 40 c0 ee 04           shr    $0x4,%sil
>   6: 0f 84 72 06 00 00     je     0x67e
>   c: 41 0f b6 54 24 05     movzbl 0x5(%r12),%edx
>  12: 83 e2 03              and    $0x3,%edx
>  15: ff                    .byte 0xff
> [ 2257.454121] RSP: 002b:00007ffc04d3e890 EFLAGS: 00000206
> [ 2257.454215] RAX: 0000000000000012 RBX: 00007f0cdc444db8 RCX: =
00007f0cdc4e6e60
> [ 2257.454329] RDX: 0000000000000009 RSI: 00007f0cdc57ef30 RDI: =
00007f0cdc42c808
> [ 2257.454442] RBP: 00007ffc04d3e9b0 R08: 00007f0cdc445028 R09: =
00007ffc04d3e940
> [ 2257.454555] R10: 00007f0cdbf00be8 R11: 0000000000000000 R12: =
00007f0cdc42c898
> [ 2257.454670] R13: 0000000000000006 R14: 0000000600000006 R15: =
00007f0cdc581000
> [ 2257.454784]  </TASK>
> [ 2257.454869] ---[ end trace 0000000000000000 ]=E2=80=94
>=20
>=20
> dmesg2 :=20
>=20
> [ 2567.167952] ------------[ cut here ]------------
> [ 2567.168053] WARNING: CPU: 11 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
> [ 2567.168175] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
> [ 2567.168445] CPU: 11 PID: 0 Comm: swapper/11 Tainted: G           O  =
     6.6.9 #1
> [ 2567.168561] Hardware name: Supermicro X10SRD-F/X10SRD-F, BIOS 3.4 =
06/05/2021
> [ 2567.168675] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
> [ 2567.168767] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 =
00
> [ 2567.168924] RSP: 0018:ffffaeaf80418d00 EFLAGS: 00010246
> [ 2567.169017] RAX: 0000000000000000 RBX: ffff9fef84d6a940 RCX: =
0000000000000074
> [ 2567.169132] RDX: ffff9fefe2e30000 RSI: 0000000000000000 RDI: =
ffff9fef84d6a940
> [ 2567.169246] RBP: ffff9fefe2e306c0 R08: 0000000000000000 R09: =
0000000000029300
> [ 2567.169359] R10: 0000000000029300 R11: ffffaeaf80418d90 R12: =
ffff9fef8aebe000
> [ 2567.169473] R13: ffff9fef80896800 R14: ffff9fef85335200 R15: =
ffff9fef8ae07080
> [ 2567.169586] FS:  0000000000000000(0000) GS:ffff9ff6dfcc0000(0000) =
knlGS:0000000000000000
> [ 2567.169702] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2567.169795] CR2: 00007f4eaa7e6650 CR3: 0000000156dcd006 CR4: =
00000000003706e0
> [ 2567.169908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
> [ 2567.170022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
> [ 2567.170137] Call Trace:
> [ 2567.170224]  <IRQ>
> [ 2567.170309]  ? __warn+0x6c/0x130
> [ 2567.170399]  ? report_bug+0x1b8/0x200
> [ 2567.170488]  ? handle_bug+0x36/0x70
> [ 2567.170577]  ? exc_invalid_op+0x17/0x1a0
> [ 2567.170667]  ? asm_exc_invalid_op+0x16/0x20
> [ 2567.170758]  ? rcuref_put_slowpath+0x2f/0x70
> [ 2567.170850]  dst_release+0x1c/0x40
> [ 2567.170939]  __dev_queue_xmit+0x598/0xce0
> [ 2567.171029]  vlan_dev_hard_start_xmit+0x82/0xc0
> [ 2567.171122]  dev_hard_start_xmit+0x95/0xe0
> [ 2567.171216]  __dev_queue_xmit+0x863/0xce0
> [ 2567.171305]  ? eth_header+0x25/0xc0
> [ 2567.171394]  ip_finish_output2+0x1a0/0x530
> [ 2567.171485]  process_backlog+0x107/0x210
> [ 2567.171575]  __napi_poll+0x20/0x180
> [ 2567.171663]  net_rx_action+0x29f/0x380
> [ 2567.171752]  ? rebalance_domains+0x14c/0x300
> [ 2567.171843]  __do_softirq+0xd0/0x202
> [ 2567.171932]  irq_exit_rcu+0x82/0xa0
> [ 2567.172022]  common_interrupt+0x7a/0xa0
> [ 2567.172111]  </IRQ>
> [ 2567.172198]  <TASK>
> [ 2567.172283]  asm_common_interrupt+0x22/0x40
> [ 2567.172374] RIP: 0010:cpuidle_enter_state+0xa3/0x6a0
> [ 2567.172467] Code: 46 40 40 0f 84 02 01 00 00 e8 c9 a0 70 ff e8 d4 =
f6 ff ff 31 ff 49 89 c6 e8 0a b9 6f ff 45 84 ff 0f 85 d9 00 00 00 fb 45 =
85 ed <0f> 88 b8 00 00 00 49 63 cd 48 8b 04 24 48 6b f1 68 49 29 c6 48 =
8d
> [ 2567.172623] RSP: 0018:ffffaeaf80177e98 EFLAGS: 00000202
> [ 2567.172715] RAX: ffff9ff6dfce3a80 RBX: ffff9fef81338000 RCX: =
000000000000001f
> [ 2567.172828] RDX: 00000255b721ed84 RSI: 00000000238e3b7a RDI: =
0000000000000000
> [ 2567.172942] RBP: ffffffffba216ea0 R08: 0000000000000004 R09: =
ffff9ff6dfcdef00
> [ 2567.173055] R10: ffff9ff6dfcdef00 R11: 0000000000000007 R12: =
0000000000000001
> [ 2567.173168] R13: 0000000000000001 R14: 00000255b721ed84 R15: =
0000000000000000
> [ 2567.173283]  ? cpuidle_enter_state+0x96/0x6a0
> [ 2567.173374]  cpuidle_enter+0x24/0x40
> [ 2567.173464]  do_idle+0x1a7/0x210
> [ 2567.173552]  cpu_startup_entry+0x21/0x30
> [ 2567.173642]  start_secondary+0xe1/0xf0
> [ 2567.173732]  secondary_startup_64_no_verify+0x178/0x17b
> [ 2567.173825]  </TASK>
> [ 2567.173910] ---[ end trace 0000000000000000 ]=E2=80=94
>=20
>=20
> [ 2567.167952] ------------[ cut here ]------------
> [ 2567.168053] WARNING: CPU: 11 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
> [ 2567.168175] Modules linked in: nft_limit nf_conntrack_netlink pppoe =
pppox ppp_generic slhc nft_ct nft_nat nft_chain_nat nf_tables netconsole =
coretemp bonding ixgbe mdio i40e nf_nat_sip nf_conntrack_sip nf_nat_pptp =
nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_ftp =
nf_conntrack_ftp nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 =
ipmi_si ipmi_devintf ipmi_msghandler rtc_cmos
> [ 2567.168445] CPU: 11 PID: 0 Comm: swapper/11 Tainted: G           O  =
     6.6.9 #1
> [ 2567.168561] Hardware name: Supermicro X10SRD-F/X10SRD-F, BIOS 3.4 =
06/05/2021
> [ 2567.168675] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
> [ 2567.168767] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4c e3 =
00
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 07                    (bad)
>   1: 83 f8 ff              cmp    $0xffffffff,%eax
>   4: 75 19                 jne    0x1f
>   6: ba 00 00 00 e0        mov    $0xe0000000,%edx
>   b: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
>   f: 83 f8 ff              cmp    $0xffffffff,%eax
>  12: 74 04                 je     0x18
>  14: 31 c0                 xor    %eax,%eax
>  16: 5b                    pop    %rbx
>  17: c3                    ret
>  18: b8 01 00 00 00        mov    $0x1,%eax
>  1d: 5b                    pop    %rbx
>  1e: c3                    ret
>  1f: 3d ff ff ff bf        cmp    $0xbfffffff,%eax
>  24: 77 14                 ja     0x3a
>  26: 85 c0                 test   %eax,%eax
>  28: 78 06                 js     0x30
>  2a:* 0f 0b                 ud2     <-- trapping instruction
>  2c: 31 c0                 xor    %eax,%eax
>  2e: eb e6                 jmp    0x16
>  30: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>  36: 31 c0                 xor    %eax,%eax
>  38: eb dc                 jmp    0x16
>  3a: 80                    .byte 0x80
>  3b: 3d e2 4c e3 00        cmp    $0xe34ce2,%eax
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 0f 0b                 ud2
>   2: 31 c0                 xor    %eax,%eax
>   4: eb e6                 jmp    0xffffffffffffffec
>   6: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>   c: 31 c0                 xor    %eax,%eax
>   e: eb dc                 jmp    0xffffffffffffffec
>  10: 80                    .byte 0x80
>  11: 3d e2 4c e3 00        cmp    $0xe34ce2,%eax
> [ 2567.168924] RSP: 0018:ffffaeaf80418d00 EFLAGS: 00010246
> [ 2567.169017] RAX: 0000000000000000 RBX: ffff9fef84d6a940 RCX: =
0000000000000074
> [ 2567.169132] RDX: ffff9fefe2e30000 RSI: 0000000000000000 RDI: =
ffff9fef84d6a940
> [ 2567.169246] RBP: ffff9fefe2e306c0 R08: 0000000000000000 R09: =
0000000000029300
> [ 2567.169359] R10: 0000000000029300 R11: ffffaeaf80418d90 R12: =
ffff9fef8aebe000
> [ 2567.169473] R13: ffff9fef80896800 R14: ffff9fef85335200 R15: =
ffff9fef8ae07080
> [ 2567.169586] FS:  0000000000000000(0000) GS:ffff9ff6dfcc0000(0000) =
knlGS:0000000000000000
> [ 2567.169702] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2567.169795] CR2: 00007f4eaa7e6650 CR3: 0000000156dcd006 CR4: =
00000000003706e0
> [ 2567.169908] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
> [ 2567.170022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
> [ 2567.170137] Call Trace:
> [ 2567.170224]  <IRQ>
> [ 2567.170309] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
> [ 2567.170399] ? report_bug (lib/bug.c:180 lib/bug.c:219)
> [ 2567.170488] ? handle_bug (arch/x86/kernel/traps.c:237)
> [ 2567.170577] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
> [ 2567.170667] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
> [ 2567.170758] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
> [ 2567.170850] dst_release (net/core/dst.c:166 (discriminator 1))
> [ 2567.170939] __dev_queue_xmit (./include/net/dst.h:283 =
net/core/dev.c:4327)
> [ 2567.171029] vlan_dev_hard_start_xmit (net/8021q/vlan_dev.c:130)
> [ 2567.171122] dev_hard_start_xmit (./include/linux/netdevice.h:4926 =
net/core/dev.c:3576 net/core/dev.c:3592)
> [ 2567.171216] __dev_queue_xmit (./include/linux/netdevice.h:3300 =
(discriminator 25) net/core/dev.c:4373 (discriminator 25))
> [ 2567.171305] ? eth_header (net/ethernet/eth.c:85)
> [ 2567.171394] ip_finish_output2 (./include/net/neighbour.h:542 =
(discriminator 2) net/ipv4/ip_output.c:233 (discriminator 2))
> [ 2567.171485] process_backlog (net/core/dev.c:6000)
> [ 2567.171575] __napi_poll (net/core/dev.c:6559)
> [ 2567.171663] net_rx_action (net/core/dev.c:6628 net/core/dev.c:6759)
> [ 2567.171752] ? rebalance_domains (kernel/sched/fair.c:11719 =
kernel/sched/fair.c:11895)
> [ 2567.171843] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
> [ 2567.171932] irq_exit_rcu (kernel/softirq.c:436 kernel/softirq.c:641 =
kernel/softirq.c:653)
> [ 2567.172022] common_interrupt (arch/x86/kernel/irq.c:247 =
(discriminator 47))
> [ 2567.172111]  </IRQ>
> [ 2567.172198]  <TASK>
> [ 2567.172283] asm_common_interrupt =
(./arch/x86/include/asm/idtentry.h:640)
> [ 2567.172374] RIP: 0010:cpuidle_enter_state =
(drivers/cpuidle/cpuidle.c:291)
> [ 2567.172467] Code: 46 40 40 0f 84 02 01 00 00 e8 c9 a0 70 ff e8 d4 =
f6 ff ff 31 ff 49 89 c6 e8 0a b9 6f ff 45 84 ff 0f 85 d9 00 00 00 fb 45 =
85 ed <0f> 88 b8 00 00 00 49 63 cd 48 8b 04 24 48 6b f1 68 49 29 c6 48 =
8d
> All code
> =3D=3D=3D=3D=3D=3D=3D=3D
>   0: 46                    rex.RX
>   1: 40                    rex
>   2: 40 0f 84 02 01 00 00 rex je 0x10b
>   9: e8 c9 a0 70 ff        call   0xffffffffff70a0d7
>   e: e8 d4 f6 ff ff        call   0xfffffffffffff6e7
>  13: 31 ff                 xor    %edi,%edi
>  15: 49 89 c6              mov    %rax,%r14
>  18: e8 0a b9 6f ff        call   0xffffffffff6fb927
>  1d: 45 84 ff              test   %r15b,%r15b
>  20: 0f 85 d9 00 00 00     jne    0xff
>  26: fb                    sti
>  27: 45 85 ed              test   %r13d,%r13d
>  2a:* 0f 88 b8 00 00 00     js     0xe8 <-- trapping instruction
>  30: 49 63 cd              movslq %r13d,%rcx
>  33: 48 8b 04 24           mov    (%rsp),%rax
>  37: 48 6b f1 68           imul   $0x68,%rcx,%rsi
>  3b: 49 29 c6              sub    %rax,%r14
>  3e: 48                    rex.W
>  3f: 8d                    .byte 0x8d
>=20
> Code starting with the faulting instruction
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>   0: 0f 88 b8 00 00 00     js     0xbe
>   6: 49 63 cd              movslq %r13d,%rcx
>   9: 48 8b 04 24           mov    (%rsp),%rax
>   d: 48 6b f1 68           imul   $0x68,%rcx,%rsi
>  11: 49 29 c6              sub    %rax,%r14
>  14: 48                    rex.W
>  15: 8d                    .byte 0x8d
> [ 2567.172623] RSP: 0018:ffffaeaf80177e98 EFLAGS: 00000202
> [ 2567.172715] RAX: ffff9ff6dfce3a80 RBX: ffff9fef81338000 RCX: =
000000000000001f
> [ 2567.172828] RDX: 00000255b721ed84 RSI: 00000000238e3b7a RDI: =
0000000000000000
> [ 2567.172942] RBP: ffffffffba216ea0 R08: 0000000000000004 R09: =
ffff9ff6dfcdef00
> [ 2567.173055] R10: ffff9ff6dfcdef00 R11: 0000000000000007 R12: =
0000000000000001
> [ 2567.173168] R13: 0000000000000001 R14: 00000255b721ed84 R15: =
0000000000000000
> [ 2567.173283] ? cpuidle_enter_state (drivers/cpuidle/cpuidle.c:285)
> [ 2567.173374] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
> [ 2567.173464] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
> [ 2567.173552] cpu_startup_entry (kernel/sched/idle.c:379)
> [ 2567.173642] start_secondary (arch/x86/kernel/smpboot.c:326)
> [ 2567.173732] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:449)
> [ 2567.173825]  </TASK>
> [ 2567.173910] ---[ end trace 0000000000000000 ]=E2=80=94
>=20
> best regards,
> Martin
>=20
>=20
>> On 29 Dec 2023, at 14:00, Martin Zaharinov <micron10@gmail.com> =
wrote:
>>=20
>> Hi Thomas,
>>=20
>> One more report from second machine:
>>=20
>> [21299.954952] ------------[ cut here ]------------
>> [21299.955047] WARNING: CPU: 15 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
>> [21299.955153] Modules linked in: nft_limit nft_ct nft_nat =
nft_chain_nat nf_tables netconsole coretemp virtio_net net_failover =
failover virtio_pci virtio_pci_legacy_dev virtio_pci_modern_dev virtio =
virtio_ring e1000e e1000 vmxnet3 i40e ixgbe mdio bnxt_en nf_nat_sip =
nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp nf_nat_tftp =
nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat nf_conntrack =
nf_defrag_ipv6 nf_defrag_ipv4 rtc_cmos
>> [21299.955378] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G           O =
      6.6.8 #1
>> [21299.955475] Hardware name: HPE ProLiant DL380 Gen10/ProLiant DL380 =
Gen10, BIOS U30 02/09/2023
>> [21299.955575] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
>> [21299.955662] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
>> All code
>> =3D=3D=3D=3D=3D=3D=3D=3D
>>  0: 07                    (bad)
>>  1: 83 f8 ff              cmp    $0xffffffff,%eax
>>  4: 75 19                 jne    0x1f
>>  6: ba 00 00 00 e0        mov    $0xe0000000,%edx
>>  b: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
>>  f: 83 f8 ff              cmp    $0xffffffff,%eax
>> 12: 74 04                 je     0x18
>> 14: 31 c0                 xor    %eax,%eax
>> 16: 5b                    pop    %rbx
>> 17: c3                    ret
>> 18: b8 01 00 00 00        mov    $0x1,%eax
>> 1d: 5b                    pop    %rbx
>> 1e: c3                    ret
>> 1f: 3d ff ff ff bf        cmp    $0xbfffffff,%eax
>> 24: 77 14                 ja     0x3a
>> 26: 85 c0                 test   %eax,%eax
>> 28: 78 06                 js     0x30
>> 2a:* 0f 0b                 ud2     <-- trapping instruction
>> 2c: 31 c0                 xor    %eax,%eax
>> 2e: eb e6                 jmp    0x16
>> 30: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>> 36: 31 c0                 xor    %eax,%eax
>> 38: eb dc                 jmp    0x16
>> 3a: 80                    .byte 0x80
>> 3b: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>>=20
>> Code starting with the faulting instruction
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>  0: 0f 0b                 ud2
>>  2: 31 c0                 xor    %eax,%eax
>>  4: eb e6                 jmp    0xffffffffffffffec
>>  6: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>>  c: 31 c0                 xor    %eax,%eax
>>  e: eb dc                 jmp    0xffffffffffffffec
>> 10: 80                    .byte 0x80
>> 11: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>> [21299.955793] RSP: 0018:ffff96a7c0578c30 EFLAGS: 00010246
>> [21299.955879] RAX: 0000000000000000 RBX: ffff8b75d1e49a80 RCX: =
ffff8b75c6667c80
>> [21299.955974] RDX: ffff8b84bfbe4f08 RSI: 00000000fffffe01 RDI: =
ffff8b75d1e49a80
>> [21299.956070] RBP: ffff8b84bfbe4f08 R08: ffff8b84bfbe4f08 R09: =
0000000000000001
>> [21299.956167] R10: 0000000000028530 R11: 0000000000000001 R12: =
ffff8b75d1e49a40
>> [21299.956261] R13: ffff8b75d1e49aa8 R14: ffff8b84bfbe4f08 R15: =
00000000c26ab667
>> [21299.956358] FS:  0000000000000000(0000) GS:ffff8b84bfbc0000(0000) =
knlGS:0000000000000000
>> [21299.956457] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [21299.956540] CR2: 00007f2e185c73c8 CR3: 0000000950014003 CR4: =
00000000003706e0
>> [21299.956635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: =
0000000000000000
>> [21299.956730] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: =
0000000000000400
>> [21299.956826] Call Trace:
>> [21299.956905]  <IRQ>
>> [21299.956983] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
>> [21299.957065] ? report_bug (lib/bug.c:180 lib/bug.c:219)
>> [21299.957147] ? handle_bug (arch/x86/kernel/traps.c:237)
>> [21299.957228] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
>> [21299.957308] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
>> [21299.957393] ? rcuref_put_slowpath (lib/rcuref.c:294 (discriminator =
1))
>> [21299.957476] dst_release (net/core/dst.c:166 (discriminator 1))
>> [21299.957559] rt_cache_route (net/ipv4/route.c:1499)
>> [21299.957641] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
>> [21299.957722] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
>> [21299.957804] ? free_unref_page (./include/linux/list.h:150 =
(discriminator 1) ./include/linux/list.h:169 (discriminator 1) =
mm/page_alloc.c:2377 (discriminator 1) mm/page_alloc.c:2428 =
(discriminator 1))
>> [21299.957889] ip_route_input_noref (net/ipv4/route.c:2499)
>> [21299.957972] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
>> [21299.958058] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
>> [21299.958139] ? ip_rcv_core (net/ipv4/ip_input.c:436)
>> [21299.958220] process_backlog (net/core/dev.c:5997)
>> [21299.958302] __napi_poll (net/core/dev.c:6556)
>> [21299.958384] net_rx_action (net/core/dev.c:6625 =
net/core/dev.c:6756)
>> [21299.958466] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
>> [21299.958549] irq_exit_rcu (kernel/softirq.c:436 =
kernel/softirq.c:641 kernel/softirq.c:653)
>> [21299.958631] sysvec_call_function_single (arch/x86/kernel/smp.c:262 =
(discriminator 47))
>> [21299.958714]  </IRQ>
>> [21299.958792]  <TASK>
>> [21299.958869] asm_sysvec_call_function_single =
(./arch/x86/include/asm/idtentry.h:656)
>> [21299.958953] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
>> [21299.959038] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
>> All code
>> =3D=3D=3D=3D=3D=3D=3D=3D
>>  0: ed                    in     (%dx),%eax
>>  1: c3                    ret
>>  2: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
>>  9: 00 00 00 00
>>  d: 66 90                 xchg   %ax,%ax
>>  f: 65 48 8b 04 25 40 32 mov    %gs:0x23240,%rax
>> 16: 02 00
>> 18: 48 8b 00              mov    (%rax),%rax
>> 1b: a8 08                 test   $0x8,%al
>> 1d: 75 0c                 jne    0x2b
>> 1f: eb 07                 jmp    0x28
>> 21: 0f 00 2d 57 0f 2c 00 verw   0x2c0f57(%rip)        # 0x2c0f7f
>> 28: fb                    sti
>> 29: f4                    hlt
>> 2a:* fa                    cli     <-- trapping instruction
>> 2b: c3                    ret
>> 2c: 0f 1f 00              nopl   (%rax)
>> 2f: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>> 33: 3c 01                 cmp    $0x1,%al
>> 35: 74 0b                 je     0x42
>> 37: 3c 02                 cmp    $0x2,%al
>> 39: 74 05                 je     0x40
>> 3b: 8b 7f 04              mov    0x4(%rdi),%edi
>> 3e: eb 9f                 jmp    0xffffffffffffffdf
>>=20
>> Code starting with the faulting instruction
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>  0: fa                    cli
>>  1: c3                    ret
>>  2: 0f 1f 00              nopl   (%rax)
>>  5: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>>  9: 3c 01                 cmp    $0x1,%al
>>  b: 74 0b                 je     0x18
>>  d: 3c 02                 cmp    $0x2,%al
>>  f: 74 05                 je     0x16
>> 11: 8b 7f 04              mov    0x4(%rdi),%edi
>> 14: eb 9f                 jmp    0xffffffffffffffb5
>> [21299.959162] RSP: 0018:ffff96a7c015be80 EFLAGS: 00000246
>> [21299.959247] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
>> [21299.959343] RDX: ffff8b84bfbc0000 RSI: ffff8b75c76ba000 RDI: =
ffff8b75c76ba064
>> [21299.959437] RBP: ffffffffae216ea0 R08: ffffffffae216ea0 R09: =
0000000000000003
>> [21299.959533] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
>> [21299.959630] R13: ffffffffae216f08 R14: ffffffffae216f20 R15: =
0000000000000000
>> [21299.959725] acpi_idle_enter (drivers/acpi/processor_idle.c:709)
>> [21299.959807] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:267)
>> [21299.959890] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
>> [21299.959975] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
>> [21299.960058] cpu_startup_entry (kernel/sched/idle.c:379)
>> [21299.960140] start_secondary (arch/x86/kernel/smpboot.c:326)
>> [21299.960223] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
>> [21299.960306]  </TASK>
>> [21299.960384] ---[ end trace 0000000000000000 ]---
>>=20
>>> On 22 Dec 2023, at 19:26, Martin Zaharinov <micron10@gmail.com> =
wrote:
>>>=20
>>> Hi Thomas,
>>>=20
>>> this is with applyed patch from you.
>>> See logs
>>>=20
>>>=20
>>> [43040.198064] ------------[ cut here ]------------
>>> [43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath+0x2f/0x70
>>> [43040.198685] Modules linked in: pppoe pppox ppp_generic slhc =
nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb =
i2c_algo_bit e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe =
mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp =
nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat =
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler =
rtc_cmos
>>> [43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           =
O       6.6.8 #1
>>> [43040.199660] Hardware name: VMware, Inc. VMware Virtual =
Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
>>> [43040.199886] RIP: 0010:rcuref_put_slowpath+0x2f/0x70
>>> [43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
>>> [43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
>>> [43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
>>> [43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
>>> [43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
>>> [43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
>>> [43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
>>> [43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
>>> [43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
>>> [43040.201994] Call Trace:
>>> [43040.202095]  <IRQ>
>>> [43040.202187]  ? __warn+0x6c/0x130
>>> [43040.202301]  ? report_bug+0x1b8/0x200
>>> [43040.202418]  ? handle_bug+0x36/0x70
>>> [43040.202534]  ? exc_invalid_op+0x17/0x1a0
>>> [43040.202652]  ? asm_exc_invalid_op+0x16/0x20
>>> [43040.202781]  ? rcuref_put_slowpath+0x2f/0x70
>>> [43040.202909]  dst_release+0x1c/0x40
>>> [43040.203026]  rt_cache_route+0xbd/0xf0
>>> [43040.203143]  rt_set_nexthop.isra.0+0x1b6/0x450
>>> [43040.203272]  ip_route_input_slow+0x5d9/0xcc0
>>> [43040.203401]  ? nf_conntrack_udp_packet+0x17c/0x240 [nf_conntrack]
>>> [43040.203581]  ip_route_input_noref+0xe0/0xf0
>>> [43040.203704]  ip_rcv_finish_core.isra.0+0xbb/0x440
>>> [43040.203855]  ip_rcv+0xd5/0x110
>>> [43040.203962]  ? ip_rcv_core+0x360/0x360
>>> [43040.204079]  process_backlog+0x107/0x210
>>> [43040.204201]  __napi_poll+0x20/0x180
>>> [43040.204315]  net_rx_action+0x29f/0x380
>>> [43040.204432]  __do_softirq+0xd0/0x202
>>> [43040.204549]  irq_exit_rcu+0x82/0xa0
>>> [43040.204667]  common_interrupt+0x7a/0xa0
>>> [43040.204786]  </IRQ>
>>> [43040.204876]  <TASK>
>>> [43040.204965]  asm_common_interrupt+0x22/0x40
>>> [43040.205090] RIP: 0010:acpi_safe_halt+0x1b/0x20
>>> [43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
>>> [43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
>>> [43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
>>> [43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
>>> [43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
>>> [43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
>>> [43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
>>> [43040.206593]  acpi_idle_enter+0x77/0xc0
>>> [43040.206711]  cpuidle_enter_state+0x69/0x6a0
>>> [43040.206835]  cpuidle_enter+0x24/0x40
>>> [43040.206954]  do_idle+0x1a7/0x210
>>> [43040.207066]  cpu_startup_entry+0x21/0x30
>>> [43040.207188]  start_secondary+0xe1/0xf0
>>> [43040.207310]  secondary_startup_64_no_verify+0x166/0x16b
>>> [43040.207451]  </TASK>
>>> [43040.207542] ---[ end trace 0000000000000000 ]---
>>>=20
>>>=20
>>>=20
>>> [43040.198064] ------------[ cut here ]------------
>>> [43040.198407] WARNING: CPU: 47 PID: 0 at lib/rcuref.c:294 =
rcuref_put_slowpath (lib/rcuref.c:294 (discriminator 1))
>>> [43040.198685] Modules linked in: pppoe pppox ppp_generic slhc =
nft_limit nft_ct nft_nat nft_chain_nat nf_tables netconsole tg3 igb =
i2c_algo_bit e1000e bnxt_en mlx5_core mlxfw mlx4_en mlx4_core i40e ixgbe =
mdio nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp =
nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat =
nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipmi_devintf ipmi_msghandler =
rtc_cmos
>>> [43040.199478] CPU: 47 PID: 0 Comm: swapper/47 Tainted: G           =
O       6.6.8 #1
>>> [43040.199660] Hardware name: VMware, Inc. VMware Virtual =
Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
>>> [43040.199886] RIP: 0010:rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
>>> [43040.200028] Code: 07 83 f8 ff 75 19 ba 00 00 00 e0 f0 0f b1 17 83 =
f8 ff 74 04 31 c0 5b c3 b8 01 00 00 00 5b c3 3d ff ff ff bf 77 14 85 c0 =
78 06 <0f> 0b 31 c0 eb e6 c7 07 00 00 00 a0 31 c0 eb dc 80 3d e2 4e e3 =
00
>>> All code
>>> =3D=3D=3D=3D=3D=3D=3D=3D
>>> 0: 07                    (bad)
>>> 1: 83 f8 ff              cmp    $0xffffffff,%eax
>>> 4: 75 19                 jne    0x1f
>>> 6: ba 00 00 00 e0        mov    $0xe0000000,%edx
>>> b: f0 0f b1 17           lock cmpxchg %edx,(%rdi)
>>> f: 83 f8 ff              cmp    $0xffffffff,%eax
>>> 12: 74 04                 je     0x18
>>> 14: 31 c0                 xor    %eax,%eax
>>> 16: 5b                    pop    %rbx
>>> 17: c3                    ret
>>> 18: b8 01 00 00 00        mov    $0x1,%eax
>>> 1d: 5b                    pop    %rbx
>>> 1e: c3                    ret
>>> 1f: 3d ff ff ff bf        cmp    $0xbfffffff,%eax
>>> 24: 77 14                 ja     0x3a
>>> 26: 85 c0                 test   %eax,%eax
>>> 28: 78 06                 js     0x30
>>> 2a:* 0f 0b                 ud2     <-- trapping instruction
>>> 2c: 31 c0                 xor    %eax,%eax
>>> 2e: eb e6                 jmp    0x16
>>> 30: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>>> 36: 31 c0                 xor    %eax,%eax
>>> 38: eb dc                 jmp    0x16
>>> 3a: 80                    .byte 0x80
>>> 3b: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>>>=20
>>> Code starting with the faulting instruction
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> 0: 0f 0b                 ud2
>>> 2: 31 c0                 xor    %eax,%eax
>>> 4: eb e6                 jmp    0xffffffffffffffec
>>> 6: c7 07 00 00 00 a0     movl   $0xa0000000,(%rdi)
>>> c: 31 c0                 xor    %eax,%eax
>>> e: eb dc                 jmp    0xffffffffffffffec
>>> 10: 80                    .byte 0x80
>>> 11: 3d e2 4e e3 00        cmp    $0xe34ee2,%eax
>>> [43040.200387] RSP: 0018:ffffa39d83e88c30 EFLAGS: 00010246
>>> [43040.200528] RAX: 0000000000000000 RBX: ffff9c58e966b840 RCX: =
ffff9c5bc4e35680
>>> [43040.200700] RDX: ffff9c5fafde4f08 RSI: 00000000fffffe01 RDI: =
ffff9c58e966b840
>>> [43040.200871] RBP: ffff9c5fafde4f08 R08: ffff9c5fafde4f08 R09: =
0000000000000001
>>> [43040.201044] R10: 00000000000286e0 R11: 0000000000000001 R12: =
ffff9c58e966b800
>>> [43040.201255] R13: ffff9c58e966b868 R14: ffff9c5fafde4f08 R15: =
000000008f5de42b
>>> [43040.201439] FS:  0000000000000000(0000) GS:ffff9c5fafdc0000(0000) =
knlGS:0000000000000000
>>> [43040.201642] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> [43040.201799] CR2: 00007f1401217714 CR3: 0000000464b94003 CR4: =
00000000001706e0
>>> [43040.201994] Call Trace:
>>> [43040.202095]  <IRQ>
>>> [43040.202187] ? __warn (kernel/panic.c:235 kernel/panic.c:673)
>>> [43040.202301] ? report_bug (lib/bug.c:180 lib/bug.c:219)
>>> [43040.202418] ? handle_bug (arch/x86/kernel/traps.c:237)
>>> [43040.202534] ? exc_invalid_op (arch/x86/kernel/traps.c:258 =
(discriminator 1))
>>> [43040.202652] ? asm_exc_invalid_op =
(./arch/x86/include/asm/idtentry.h:568)
>>> [43040.202781] ? rcuref_put_slowpath (lib/rcuref.c:294 =
(discriminator 1))
>>> [43040.202909] dst_release (net/core/dst.c:166 (discriminator 1))
>>> [43040.203026] rt_cache_route (net/ipv4/route.c:1499)
>>> [43040.203143] rt_set_nexthop.isra.0 (net/ipv4/route.c:1606 =
(discriminator 1))
>>> [43040.203272] ip_route_input_slow (./include/net/lwtunnel.h:140 =
net/ipv4/route.c:1875 net/ipv4/route.c:2154 net/ipv4/route.c:2337)
>>> [43040.203401] ? nf_conntrack_udp_packet =
(net/netfilter/nf_conntrack_proto_udp.c:124) nf_conntrack
>>> [43040.203581] ip_route_input_noref (net/ipv4/route.c:2499)
>>> [43040.203704] ip_rcv_finish_core.isra.0 (net/ipv4/ip_input.c:367 =
(discriminator 1))
>>> [43040.203855] ip_rcv (net/ipv4/ip_input.c:448 =
./include/linux/netfilter.h:304 ./include/linux/netfilter.h:298 =
net/ipv4/ip_input.c:569)
>>> [43040.203962] ? ip_rcv_core (net/ipv4/ip_input.c:436)
>>> [43040.204079] process_backlog (net/core/dev.c:5997)
>>> [43040.204201] __napi_poll (net/core/dev.c:6556)
>>> [43040.204315] net_rx_action (net/core/dev.c:6625 =
net/core/dev.c:6756)
>>> [43040.204432] __do_softirq (./arch/x86/include/asm/preempt.h:27 =
kernel/softirq.c:564)
>>> [43040.204549] irq_exit_rcu (kernel/softirq.c:436 =
kernel/softirq.c:641 kernel/softirq.c:653)
>>> [43040.204667] common_interrupt (arch/x86/kernel/irq.c:247 =
(discriminator 47))
>>> [43040.204786]  </IRQ>
>>> [43040.204876]  <TASK>
>>> [43040.204965] asm_common_interrupt =
(./arch/x86/include/asm/idtentry.h:640)
>>> [43040.205090] RIP: 0010:acpi_safe_halt =
(./arch/x86/include/asm/irqflags.h:37 =
./arch/x86/include/asm/irqflags.h:72 drivers/acpi/processor_idle.c:113)
>>> [43040.205220] Code: ed c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 65 =
48 8b 04 25 40 32 02 00 48 8b 00 a8 08 75 0c eb 07 0f 00 2d 57 0f 2c 00 =
fb f4 <fa> c3 0f 1f 00 0f b6 47 08 3c 01 74 0b 3c 02 74 05 8b 7f 04 eb =
9f
>>> All code
>>> =3D=3D=3D=3D=3D=3D=3D=3D
>>> 0: ed                    in     (%dx),%eax
>>> 1: c3                    ret
>>> 2: 66 66 2e 0f 1f 84 00 data16 cs nopw 0x0(%rax,%rax,1)
>>> 9: 00 00 00 00
>>> d: 66 90                 xchg   %ax,%ax
>>> f: 65 48 8b 04 25 40 32 mov    %gs:0x23240,%rax
>>> 16: 02 00
>>> 18: 48 8b 00              mov    (%rax),%rax
>>> 1b: a8 08                 test   $0x8,%al
>>> 1d: 75 0c                 jne    0x2b
>>> 1f: eb 07                 jmp    0x28
>>> 21: 0f 00 2d 57 0f 2c 00 verw   0x2c0f57(%rip)        # 0x2c0f7f
>>> 28: fb                    sti
>>> 29: f4                    hlt
>>> 2a:* fa                    cli     <-- trapping instruction
>>> 2b: c3                    ret
>>> 2c: 0f 1f 00              nopl   (%rax)
>>> 2f: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>>> 33: 3c 01                 cmp    $0x1,%al
>>> 35: 74 0b                 je     0x42
>>> 37: 3c 02                 cmp    $0x2,%al
>>> 39: 74 05                 je     0x40
>>> 3b: 8b 7f 04              mov    0x4(%rdi),%edi
>>> 3e: eb 9f                 jmp    0xffffffffffffffdf
>>>=20
>>> Code starting with the faulting instruction
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> 0: fa                    cli
>>> 1: c3                    ret
>>> 2: 0f 1f 00              nopl   (%rax)
>>> 5: 0f b6 47 08           movzbl 0x8(%rdi),%eax
>>> 9: 3c 01                 cmp    $0x1,%al
>>> b: 74 0b                 je     0x18
>>> d: 3c 02                 cmp    $0x2,%al
>>> f: 74 05                 je     0x16
>>> 11: 8b 7f 04              mov    0x4(%rdi),%edi
>>> 14: eb 9f                 jmp    0xffffffffffffffb5
>>> [43040.205578] RSP: 0018:ffffa39d8234fe80 EFLAGS: 00000246
>>> [43040.205718] RAX: 0000000000004000 RBX: 0000000000000001 RCX: =
000000000000001f
>>> [43040.205890] RDX: ffff9c5fafdc0000 RSI: ffff9c5882e95800 RDI: =
ffff9c5882e95864
>>> [43040.206063] RBP: ffffffffa9216ea0 R08: ffffffffa9216ea0 R09: =
0000000000000003
>>> [43040.206246] R10: 0000000000000002 R11: 0000000000000008 R12: =
0000000000000001
>>> [43040.206419] R13: ffffffffa9216f08 R14: ffffffffa9216f20 R15: =
0000000000000000
>>> [43040.206593] acpi_idle_enter (drivers/acpi/processor_idle.c:709)
>>> [43040.206711] cpuidle_enter_state (drivers/cpuidle/cpuidle.c:267)
>>> [43040.206835] cpuidle_enter (drivers/cpuidle/cpuidle.c:390 =
(discriminator 2))
>>> [43040.206954] do_idle (kernel/sched/idle.c:134 =
kernel/sched/idle.c:215 kernel/sched/idle.c:282)
>>> [43040.207066] cpu_startup_entry (kernel/sched/idle.c:379)
>>> [43040.207188] start_secondary (arch/x86/kernel/smpboot.c:326)
>>> [43040.207310] secondary_startup_64_no_verify =
(arch/x86/kernel/head_64.S:433)
>>> [43040.207451]  </TASK>
>>> [43040.207542] ---[ end trace 0000000000000000 ]---
>>>=20
>>>> On 19 Dec 2023, at 16:26, Thomas Gleixner <tglx@linutronix.de> =
wrote:
>>>>=20
>>>> On Tue, Dec 19 2023 at 11:25, Martin Zaharinov wrote:
>>>>>> On 12 Dec 2023, at 20:16, Thomas Gleixner <tglx@linutronix.de> =
wrote:
>>>>>> Btw, how easy is this to reproduce?
>>>>>=20
>>>>> Its not easy this report is generate on machine with 5-6k users , =
with
>>>>> traffic and one time is show on 1 day , other show after 4-5 =
days=E2=80=A6
>>>>=20
>>>> I love those bugs ...
>>>>=20
>>>>> Apply this patch and will upload image on one machine as fast as
>>>>> possible and when get any reports will send you.
>>>>=20
>>>> Let's see how that goes!
>>>>=20
>>>> Thanks,
>>>>=20
>>>>     tglx
>>>=20
>>=20
>=20


