From: John Ogness <john.ogness@linutronix.de>
To: Petr Mladek <pmladek@suse.com>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>,
Steven Rostedt <rostedt@goodmis.org>,
Thomas Gleixner <tglx@linutronix.de>,
linux-kernel@vger.kernel.org,
Francesco Dolcini <francesco@dolcini.it>,
kernel test robot <oliver.sang@intel.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Subject: Re: [PATCH printk v3 02/14] printk: Adjust mapping for 32bit seq macros
Date: Mon, 15 Jan 2024 11:58:39 +0106 [thread overview]
Message-ID: <87a5p6g96w.fsf@jogness.linutronix.de> (raw)
In-Reply-To: <ZaGBlDTuQ0hYBpeJ@alley>
On 2024-01-12, Petr Mladek <pmladek@suse.com> wrote:
>> > Reported-by: Francesco Dolcini <francesco@dolcini.it>
>> > Reported-by: kernel test robot <oliver.sang@intel.com>
>> > Closes: https://lore.kernel.org/oe-lkp/202311171611.78d41dbe-oliver.sang@intel.com
>> > Reported-by: kernel test robot <oliver.sang@intel.com>
>> > Closes: https://lore.kernel.org/oe-lkp/202311161555.3ee16fc9-oliver.sang@intel.com
>> > Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
>> > Signed-off-by: John Ogness <john.ogness@linutronix.de>
>>
>> Great catch! It must have been complicated to debug this.
>>
>> Reviewed-by: Petr Mladek <pmladek@suse.com>
>
> That said, I am a bit nervous that a bug like this might cause
> workqueue stall and panic() the kernel.
>
> At least, this is how I read
> https://lore.kernel.org/oe-lkp/202311171611.78d41dbe-oliver.sang@intel.com/
Note that the bug is reported for code that mainline will never
have. This patch "fixes" the problem before it is a problem. From the
perspective of mainline, the problem never existed. Perhaps it is
inappropriate to list these with the Closes tag.
> It looks like it caused some loop and refcout overlow or so.
> But I might be wrong.
>
> I would like to better understand this and check if we could prevent
> it somehow.
The code in question can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/tree/kernel/printk/printk_ringbuffer.c?id=768c33033e41ad195a9304ebb183afb730d5ae98#n2031
The URL above points to prb_next_reserve_seq(), which is in an infinite
loop.
desc_last_finalized_seq() is returning a huge garbage value due to the
32bit sequence value 1 being incorrectly mapped to 0xffffffff00000001
(which is 18446744069414584321). When desc_read_finalized_seq() is
called (line 2082), it obviously returns -EINVAL because there is no
record with this huge sequence number.
The code interrupts -EINVAL to mean that the record has been
overwritten, and so it tries again with an updated last finalized
sequence number => infinite loop.
With this patch applied, the 32bit value 1 is correctly mapped to
sequence number 1 and all is OK.
The problem was that desc_last_finalized_seq() returned a sequence
number that was higher than the highest record. That was the bug. As
long as desc_last_finalized_seq() always returns a sequence number that
exists or used to exist, the retry loop is fine. And since the tail
record is always in the finalized state, there will always be a
finalized record available.
John
next prev parent reply other threads:[~2024-01-15 10:52 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-14 21:41 [PATCH printk v3 00/14] fix console flushing John Ogness
2023-12-14 21:41 ` [PATCH printk v3 01/14] printk: nbcon: Relocate 32bit seq macros John Ogness
2024-01-12 10:14 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 02/14] printk: Adjust mapping for " John Ogness
2023-12-15 9:55 ` Sebastian Andrzej Siewior
2023-12-15 10:10 ` John Ogness
2023-12-15 10:58 ` Sebastian Andrzej Siewior
2024-01-12 10:28 ` Petr Mladek
2024-01-12 18:14 ` Petr Mladek
2024-01-15 8:51 ` Sebastian Andrzej Siewior
2024-01-15 10:52 ` John Ogness [this message]
2024-01-15 16:17 ` Petr Mladek
2024-01-15 17:08 ` John Ogness
2023-12-14 21:41 ` [PATCH printk v3 03/14] printk: Use prb_first_seq() as base " John Ogness
2024-01-12 16:19 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 04/14] printk: ringbuffer: Do not skip non-finalized records with prb_next_seq() John Ogness
2024-01-12 18:05 ` Petr Mladek
2024-01-15 11:55 ` John Ogness
2024-01-15 17:00 ` Petr Mladek
2024-02-05 11:33 ` John Ogness
2024-02-06 17:27 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 05/14] printk: ringbuffer: Clarify special lpos values John Ogness
2024-01-30 13:12 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 06/14] printk: For @suppress_panic_printk check for other CPU in panic John Ogness
2023-12-14 21:41 ` [PATCH printk v3 07/14] printk: Add this_cpu_in_panic() John Ogness
2023-12-14 21:41 ` [PATCH printk v3 08/14] printk: ringbuffer: Cleanup reader terminology John Ogness
2024-01-30 14:36 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 09/14] printk: Wait for all reserved records with pr_flush() John Ogness
2024-01-31 11:36 ` Petr Mladek
2024-02-05 13:33 ` John Ogness
2024-02-07 9:20 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 10/14] printk: ringbuffer: Skip non-finalized records in panic John Ogness
2024-02-01 16:56 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 11/14] printk: ringbuffer: Consider committed as finalized " John Ogness
2024-02-01 18:00 ` Petr Mladek
2024-02-05 14:08 ` John Ogness
2024-02-07 10:11 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 12/14] printk: Disable passing console lock owner completely during panic() John Ogness
2023-12-14 21:42 ` [PATCH printk v3 13/14] printk: Avoid non-panic CPUs writing to ringbuffer John Ogness
2024-02-02 9:26 ` Petr Mladek
2023-12-14 21:42 ` [PATCH printk v3 14/14] panic: Flush kernel log buffer at the end John Ogness
2024-02-02 9:30 ` Petr Mladek
2024-02-02 9:38 ` [PATCH printk v3 00/14] fix console flushing Petr Mladek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87a5p6g96w.fsf@jogness.linutronix.de \
--to=john.ogness@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=francesco@dolcini.it \
--cc=linux-kernel@vger.kernel.org \
--cc=oliver.sang@intel.com \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=senozhatsky@chromium.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.