Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

Greetings fellas,

I have encountered a critical bug in the Linux vanilla kernel that
leads to a kernel panic during the shutdown or reboot process. The
issue arises after all services, including `journald`, have been
stopped. As a result, the machine fails to complete the shutdown or
reboot procedure, effectively causing the system to hang and not shut
down or reboot.

Here are the details of the issue:

- Affected Versions: Before kernel version 6.8.10, the bug caused a
quick display of a kernel trace dump before the shutdown/reboot
completed. Starting from version 6.8.10 and continuing into version
6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
preventing the shutdown or reboot from completing and leaving the
machine stuck.

- Symptoms:
  - In normal shutdown/reboot scenarios, the kernel trace dump briefly
appears as the last message on the screen.
  - In rescue mode, the kernel panic message is displayed. Normally it
is not shown.

Since `journald` is stopped before this issue occurs, no textual logs
are available. However, I have captured two pictures illustrating
these related issues, which I am attaching to this email for your
reference. Also added my custom kernel config.

Thank you for your attention to this matter. Please let me know if any
additional information is required to assist in diagnosing and
resolving this bug.

Best regards,

Ilkka Naulapää

[-- Attachment #2: kernel-6.9.1-reboot-panic.config.zst --]
[-- Type: application/octet-stream, Size: 68900 bytes --]

[-- Attachment #3: 6.9.1_panic.heic --]
[-- Type: image/heic, Size: 2949625 bytes --]

[-- Attachment #4: 6.8.9_trace.heic --]
[-- Type: image/heic, Size: 2349596 bytes --]

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Linux regression tracking (Thorsten Leemhuis)]

[CCing a few people]

On 24.05.24 12:31, Ilkka Naulapää wrote:
> 
> I have encountered a critical bug in the Linux vanilla kernel that
> leads to a kernel panic during the shutdown or reboot process. The
> issue arises after all services, including `journald`, have been
> stopped. As a result, the machine fails to complete the shutdown or
> reboot procedure, effectively causing the system to hang and not shut
> down or reboot.

Thx for the report. Not my area of expertise, so take this with a gain
of salt. But given the versions your mention in your report and the
screenshot that mentioned tracefs_free_inode I suspect this is caused by
baa23a8d4360d ("tracefs: Reset permissions on remount if permissions are
options"). A few fixes for it will soon hit mainline and are meant to be
backported to affected stable trees:

https://lore.kernel.org/all/20240523212406.254317554@goodmis.org/
https://lore.kernel.org/all/20240523174419.1e5885a5@gandalf.local.home/

You might want to try them – or recheck once they hit the stable trees
you are about. If they don't work, please report back.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

> Here are the details of the issue:
> 
> - Affected Versions: Before kernel version 6.8.10, the bug caused a
> quick display of a kernel trace dump before the shutdown/reboot
> completed. Starting from version 6.8.10 and continuing into version
> 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> preventing the shutdown or reboot from completing and leaving the
> machine stuck.
> 
> - Symptoms:
>   - In normal shutdown/reboot scenarios, the kernel trace dump briefly
> appears as the last message on the screen.
>   - In rescue mode, the kernel panic message is displayed. Normally it
> is not shown.
> 
> Since `journald` is stopped before this issue occurs, no textual logs
> are available. However, I have captured two pictures illustrating
> these related issues, which I am attaching to this email for your
> reference. Also added my custom kernel config.
> 
> Thank you for your attention to this matter. Please let me know if any
> additional information is required to assist in diagnosing and
> resolving this bug.
> 
> Best regards,
> 
> Ilkka Naulapää

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Fri, 24 May 2024 12:50:08 +0200
"Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:

> [CCing a few people]
> 

Thanks for the Cc.

> On 24.05.24 12:31, Ilkka Naulapää wrote:
> > 
> > I have encountered a critical bug in the Linux vanilla kernel that
> > leads to a kernel panic during the shutdown or reboot process. The
> > issue arises after all services, including `journald`, have been
> > stopped. As a result, the machine fails to complete the shutdown or
> > reboot procedure, effectively causing the system to hang and not shut
> > down or reboot.  

To understand this, did you do anything with tracing? Before shutting down,
is there anything in /sys/kernel/tracing/instances directory?
Were any of the files/directories permissions in /sys/kernel/tracing changed?

> 
> Thx for the report. Not my area of expertise, so take this with a gain
> of salt. But given the versions your mention in your report and the
> screenshot that mentioned tracefs_free_inode I suspect this is caused by
> baa23a8d4360d ("tracefs: Reset permissions on remount if permissions are
> options"). A few fixes for it will soon hit mainline and are meant to be
> backported to affected stable trees:
> 
> https://lore.kernel.org/all/20240523212406.254317554@goodmis.org/
> https://lore.kernel.org/all/20240523174419.1e5885a5@gandalf.local.home/
> 
> You might want to try them – or recheck once they hit the stable trees
> you are about. If they don't work, please report back.

There's been quite a bit of updates in this code, but this looks new to me.
I have more fixes that were just pulled by Linus today.

  https://git.kernel.org/torvalds/c/0eb03c7e8e2a4cc3653eb5eeb2d2001182071215

I'm not sure how relevant that is for this. But if you can reproduce it
with that commit, then this is a new bug.

-- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Fri, 24 May 2024 12:50:08 +0200
"Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:

> > - Affected Versions: Before kernel version 6.8.10, the bug caused a
> > quick display of a kernel trace dump before the shutdown/reboot
> > completed. Starting from version 6.8.10 and continuing into version
> > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> > preventing the shutdown or reboot from completing and leaving the
> > machine stuck.

Ah, I bet it was this commit: baa23a8d4360d ("tracefs: Reset permissions on
remount if permissions are options"), which added a "iput" callback to the
dentry without calling iput, leaving stale inodes around.

This is fixed with:

  0bcfd9aa4dafa ("tracefs: Clear EVENT_INODE flag in tracefs_drop_inode()")

Try adding just that patch. It will at least make it go back to what was
happening before 6.8.10 (I hope!).

-- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

[-- Attachment #1: Type: text/plain, Size: 1639 bytes --]

hi,

I took 6.9.2 and applied that 0bcfd9aa4dafa to it. Now the kernel is
serving me both problems; the trace and the panic as the pic shows.

> To understand this, did you do anything with tracing? Before shutting down,
> is there anything in /sys/kernel/tracing/instances directory?
> Were any of the files/directories permissions in /sys/kernel/tracing changed?

And to answer your question, I did not do any tracing or so and the
/sys/kernel/tracing is empty.
Just plain boot-up, no matter if in full desktop or in bare rescue
mode, ends up the same way.

--Ilkka

On Fri, May 24, 2024 at 8:19 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 24 May 2024 12:50:08 +0200
> "Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:
>
> > > - Affected Versions: Before kernel version 6.8.10, the bug caused a
> > > quick display of a kernel trace dump before the shutdown/reboot
> > > completed. Starting from version 6.8.10 and continuing into version
> > > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> > > preventing the shutdown or reboot from completing and leaving the
> > > machine stuck.
>
> Ah, I bet it was this commit: baa23a8d4360d ("tracefs: Reset permissions on
> remount if permissions are options"), which added a "iput" callback to the
> dentry without calling iput, leaving stale inodes around.
>
> This is fixed with:
>
>   0bcfd9aa4dafa ("tracefs: Clear EVENT_INODE flag in tracefs_drop_inode()")
>
> Try adding just that patch. It will at least make it go back to what was
> happening before 6.8.10 (I hope!).
>
> -- Steve

[-- Attachment #2: 6.9.2_trace_panic.heic --]
[-- Type: image/heif, Size: 2576274 bytes --]

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

Hi Steven,

I took some time and bisected the 6.8.9 - 6.8.10 and git gave the
panic inducing commit:

414fb08628143 (tracefs: Reset permissions on remount if permissions are options)

I reverted that commit to 6.9.2 and now it only serves the trace but
the panic is gone. But I can live with it.

--Ilkka

On Sun, May 26, 2024 at 8:42 PM Ilkka Naulapää <digirigawa@gmail.com> wrote:
>
> hi,
>
> I took 6.9.2 and applied that 0bcfd9aa4dafa to it. Now the kernel is
> serving me both problems; the trace and the panic as the pic shows.
>
> > To understand this, did you do anything with tracing? Before shutting down,
> > is there anything in /sys/kernel/tracing/instances directory?
> > Were any of the files/directories permissions in /sys/kernel/tracing changed?
>
> And to answer your question, I did not do any tracing or so and the
> /sys/kernel/tracing is empty.
> Just plain boot-up, no matter if in full desktop or in bare rescue
> mode, ends up the same way.
>
> --Ilkka
>
> On Fri, May 24, 2024 at 8:19 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> >
> > On Fri, 24 May 2024 12:50:08 +0200
> > "Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:
> >
> > > > - Affected Versions: Before kernel version 6.8.10, the bug caused a
> > > > quick display of a kernel trace dump before the shutdown/reboot
> > > > completed. Starting from version 6.8.10 and continuing into version
> > > > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> > > > preventing the shutdown or reboot from completing and leaving the
> > > > machine stuck.
> >
> > Ah, I bet it was this commit: baa23a8d4360d ("tracefs: Reset permissions on
> > remount if permissions are options"), which added a "iput" callback to the
> > dentry without calling iput, leaving stale inodes around.
> >
> > This is fixed with:
> >
> >   0bcfd9aa4dafa ("tracefs: Clear EVENT_INODE flag in tracefs_drop_inode()")
> >
> > Try adding just that patch. It will at least make it go back to what was
> > happening before 6.8.10 (I hope!).
> >
> > -- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Greg KH]

On Mon, May 27, 2024 at 07:40:21PM +0300, Ilkka Naulapää wrote:
> Hi Steven,
> 
> I took some time and bisected the 6.8.9 - 6.8.10 and git gave the
> panic inducing commit:
> 
> 414fb08628143 (tracefs: Reset permissions on remount if permissions are options)
> 
> I reverted that commit to 6.9.2 and now it only serves the trace but
> the panic is gone. But I can live with it.

Steven, should we revert that?

Or is there some other change that we should take to resolve this?

thanks,

greg k-h

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Mon, 27 May 2024 20:14:42 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> On Mon, May 27, 2024 at 07:40:21PM +0300, Ilkka Naulapää wrote:
> > Hi Steven,
> > 
> > I took some time and bisected the 6.8.9 - 6.8.10 and git gave the
> > panic inducing commit:
> > 
> > 414fb08628143 (tracefs: Reset permissions on remount if permissions are options)
> > 
> > I reverted that commit to 6.9.2 and now it only serves the trace but
> > the panic is gone. But I can live with it.  
> 
> Steven, should we revert that?
> 
> Or is there some other change that we should take to resolve this?
> 

Before we revert it (as it may be a bug in mainline), Ilkka, can you
test v6.10-rc1?  If it exists there, it will let me know whether or not
I missed something.

Thanks,

-- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

I tried 6.10-rc1 and it still ends up to panic

--Ilkka


On Tue, May 28, 2024 at 12:44 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Mon, 27 May 2024 20:14:42 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
>
> > On Mon, May 27, 2024 at 07:40:21PM +0300, Ilkka Naulapää wrote:
> > > Hi Steven,
> > >
> > > I took some time and bisected the 6.8.9 - 6.8.10 and git gave the
> > > panic inducing commit:
> > >
> > > 414fb08628143 (tracefs: Reset permissions on remount if permissions are options)
> > >
> > > I reverted that commit to 6.9.2 and now it only serves the trace but
> > > the panic is gone. But I can live with it.
> >
> > Steven, should we revert that?
> >
> > Or is there some other change that we should take to resolve this?
> >
>
> Before we revert it (as it may be a bug in mainline), Ilkka, can you
> test v6.10-rc1?  If it exists there, it will let me know whether or not
> I missed something.
>
> Thanks,
>
> -- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Fri, 24 May 2024 12:50:08 +0200
"Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:

> > - Affected Versions: Before kernel version 6.8.10, the bug caused a
> > quick display of a kernel trace dump before the shutdown/reboot
> > completed. Starting from version 6.8.10 and continuing into version
> > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> > preventing the shutdown or reboot from completing and leaving the
> > machine stuck.

You state "Before kernel version 6.8.10, the bug caused ...". Does that
mean that a bug was happening before v6.8.10? But did not cause a panic?

I just noticed your second screen shot from your report, and it has:

 "cache_from_obj: Wrong slab cache, tracefs_inode_cache but object is from inode_cache"

So somehow an tracefs_inode was allocated from the inode_cache and is
now being freed by the tracefs_inode logic? Did this happen before
6.8.10? If so, this code could just be triggering an issue from an
unrelated bug.

Thanks,

-- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

yeah, the cache_from_obj tracing bug (without panic) has been
displayed quite some time now - maybe even since 6.7.x or so. I could
try checking a few versions back for this and try bisecting it if I
can find when this started.

--Ilkka

On Tue, May 28, 2024 at 1:31 AM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 24 May 2024 12:50:08 +0200
> "Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:
>
> > > - Affected Versions: Before kernel version 6.8.10, the bug caused a
> > > quick display of a kernel trace dump before the shutdown/reboot
> > > completed. Starting from version 6.8.10 and continuing into version
> > > 6.9.0 and 6.9.1, this issue has escalated to a kernel panic,
> > > preventing the shutdown or reboot from completing and leaving the
> > > machine stuck.
>
> You state "Before kernel version 6.8.10, the bug caused ...". Does that
> mean that a bug was happening before v6.8.10? But did not cause a panic?
>
> I just noticed your second screen shot from your report, and it has:
>
>  "cache_from_obj: Wrong slab cache, tracefs_inode_cache but object is from inode_cache"
>
> So somehow an tracefs_inode was allocated from the inode_cache and is
> now being freed by the tracefs_inode logic? Did this happen before
> 6.8.10? If so, this code could just be triggering an issue from an
> unrelated bug.
>
> Thanks,
>
> -- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Tue, 28 May 2024 07:51:30 +0300
Ilkka Naulapää <digirigawa@gmail.com> wrote:

> yeah, the cache_from_obj tracing bug (without panic) has been
> displayed quite some time now - maybe even since 6.7.x or so. I could
> try checking a few versions back for this and try bisecting it if I
> can find when this started.
> 

OK, so I don't think the commit your last bisect hit is the cause of
the bug. It added a delay (via RCU) and is causing the real bug to blow
up more.

Can you add this patch to v6.9.2 and hopefully it crashes in a better
location that we can find where the mixup happened.

You may need to add the other commit (too if this doesn't trigger.

Thanks,

-- Steve

diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
index 417c840e6403..7af3f696696d 100644
--- a/fs/tracefs/inode.c
+++ b/fs/tracefs/inode.c
@@ -50,6 +50,7 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
 	list_add_rcu(&ti->list, &tracefs_inodes);
 	spin_unlock_irqrestore(&tracefs_inode_lock, flags);
 
+	ti->magic = 20240823;
 	return &ti->vfs_inode;
 }
 
@@ -66,6 +67,7 @@ static void tracefs_free_inode(struct inode *inode)
 	struct tracefs_inode *ti = get_tracefs(inode);
 	unsigned long flags;
 
+	BUG_ON(ti->magic != 20240823);
 	spin_lock_irqsave(&tracefs_inode_lock, flags);
 	list_del_rcu(&ti->list);
 	spin_unlock_irqrestore(&tracefs_inode_lock, flags);
@@ -271,16 +273,6 @@ static const struct inode_operations tracefs_file_inode_operations = {
 	.setattr	= tracefs_setattr,
 };
 
-struct inode *tracefs_get_inode(struct super_block *sb)
-{
-	struct inode *inode = new_inode(sb);
-	if (inode) {
-		inode->i_ino = get_next_ino();
-		simple_inode_init_ts(inode);
-	}
-	return inode;
-}
-
 struct tracefs_mount_opts {
 	kuid_t uid;
 	kgid_t gid;
@@ -448,6 +440,17 @@ static const struct super_operations tracefs_super_operations = {
 	.show_options	= tracefs_show_options,
 };
 
+struct inode *tracefs_get_inode(struct super_block *sb)
+{
+	struct inode *inode = new_inode(sb);
+	BUG_ON(sb->s_op != &tracefs_super_operations);
+	if (inode) {
+		inode->i_ino = get_next_ino();
+		simple_inode_init_ts(inode);
+	}
+	return inode;
+}
+
 /*
  * It would be cleaner if eventfs had its own dentry ops.
  *
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index f704d8348357..dda7d2708e30 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -16,6 +16,7 @@ struct tracefs_inode {
 	};
 	/* The below gets initialized with memset_after(ti, 0, vfs_inode) */
 	struct list_head	list;
+	unsigned long		magic;
 	unsigned long           flags;
 	void                    *private;
 };

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

[-- Attachment #1: Type: text/plain, Size: 3434 bytes --]

applied your patch without others, so trace and panic there.
Screenshot attached. Also tested kernels backward and found out that
this trace bug first triggered on 6.6-rc1.

Let me know if you need more assistance with this.

--Ilkka

On Tue, May 28, 2024 at 9:47 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Tue, 28 May 2024 07:51:30 +0300
> Ilkka Naulapää <digirigawa@gmail.com> wrote:
>
> > yeah, the cache_from_obj tracing bug (without panic) has been
> > displayed quite some time now - maybe even since 6.7.x or so. I could
> > try checking a few versions back for this and try bisecting it if I
> > can find when this started.
> >
>
> OK, so I don't think the commit your last bisect hit is the cause of
> the bug. It added a delay (via RCU) and is causing the real bug to blow
> up more.
>
> Can you add this patch to v6.9.2 and hopefully it crashes in a better
> location that we can find where the mixup happened.
>
> You may need to add the other commit (too if this doesn't trigger.
>
> Thanks,
>
> -- Steve
>
> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> index 417c840e6403..7af3f696696d 100644
> --- a/fs/tracefs/inode.c
> +++ b/fs/tracefs/inode.c
> @@ -50,6 +50,7 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
>         list_add_rcu(&ti->list, &tracefs_inodes);
>         spin_unlock_irqrestore(&tracefs_inode_lock, flags);
>
> +       ti->magic = 20240823;
>         return &ti->vfs_inode;
>  }
>
> @@ -66,6 +67,7 @@ static void tracefs_free_inode(struct inode *inode)
>         struct tracefs_inode *ti = get_tracefs(inode);
>         unsigned long flags;
>
> +       BUG_ON(ti->magic != 20240823);
>         spin_lock_irqsave(&tracefs_inode_lock, flags);
>         list_del_rcu(&ti->list);
>         spin_unlock_irqrestore(&tracefs_inode_lock, flags);
> @@ -271,16 +273,6 @@ static const struct inode_operations tracefs_file_inode_operations = {
>         .setattr        = tracefs_setattr,
>  };
>
> -struct inode *tracefs_get_inode(struct super_block *sb)
> -{
> -       struct inode *inode = new_inode(sb);
> -       if (inode) {
> -               inode->i_ino = get_next_ino();
> -               simple_inode_init_ts(inode);
> -       }
> -       return inode;
> -}
> -
>  struct tracefs_mount_opts {
>         kuid_t uid;
>         kgid_t gid;
> @@ -448,6 +440,17 @@ static const struct super_operations tracefs_super_operations = {
>         .show_options   = tracefs_show_options,
>  };
>
> +struct inode *tracefs_get_inode(struct super_block *sb)
> +{
> +       struct inode *inode = new_inode(sb);
> +       BUG_ON(sb->s_op != &tracefs_super_operations);
> +       if (inode) {
> +               inode->i_ino = get_next_ino();
> +               simple_inode_init_ts(inode);
> +       }
> +       return inode;
> +}
> +
>  /*
>   * It would be cleaner if eventfs had its own dentry ops.
>   *
> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> index f704d8348357..dda7d2708e30 100644
> --- a/fs/tracefs/internal.h
> +++ b/fs/tracefs/internal.h
> @@ -16,6 +16,7 @@ struct tracefs_inode {
>         };
>         /* The below gets initialized with memset_after(ti, 0, vfs_inode) */
>         struct list_head        list;
> +       unsigned long           magic;
>         unsigned long           flags;
>         void                    *private;
>  };

[-- Attachment #2: 6.9.2-steve-patch.jpeg --]
[-- Type: image/jpeg, Size: 2230490 bytes --]

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Wed, 29 May 2024 21:36:08 +0300
Ilkka Naulapää <digirigawa@gmail.com> wrote:

> applied your patch without others, so trace and panic there.
> Screenshot attached. Also tested kernels backward and found out that

Bah, it's still in an RCU callback, which doesn't tell us why a
normal inode is being sent to the trace inode free list.

> this trace bug first triggered on 6.6-rc1.

Hmm, that's when eventfs was added.

> 
> Let me know if you need more assistance with this.

Let me make a debug patch (that crashes on this issue) for that kernel,
and perhaps you could bisect it?

Thanks!

-- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Wed, 29 May 2024 14:47:57 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> Let me make a debug patch (that crashes on this issue) for that kernel,
> and perhaps you could bisect it?

Can you try this on 6.6-rc1 and see if it gives you any other splats?

Hmm, you can switch it to WARN_ON and that way it may not crash the
machine, and you can use dmesg to get the output.

Thanks,

-- Steve


diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
index de5b72216b1a..a090495e78c9 100644
--- a/fs/tracefs/inode.c
+++ b/fs/tracefs/inode.c
@@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
 		return NULL;
 
 	ti->flags = 0;
+	ti->magic = 20240823;
 
 	return &ti->vfs_inode;
 }
 
 static void tracefs_free_inode(struct inode *inode)
 {
-	kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
+	struct tracefs_inode *ti = get_tracefs(inode);
+
+	BUG_ON(ti->magic != 20240823);
+	kmem_cache_free(tracefs_inode_cachep, ti);
 }
 
 static ssize_t default_read_file(struct file *file, char __user *buf,
@@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
 	.rmdir		= tracefs_syscall_rmdir,
 };
 
-struct inode *tracefs_get_inode(struct super_block *sb)
-{
-	struct inode *inode = new_inode(sb);
-	if (inode) {
-		inode->i_ino = get_next_ino();
-		inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
-	}
-	return inode;
-}
-
 struct tracefs_mount_opts {
 	kuid_t uid;
 	kgid_t gid;
@@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
 		return;
 
 	ti = get_tracefs(inode);
+	BUG_ON(ti->magic != 20240823);
 	if (ti && ti->flags & TRACEFS_EVENT_INODE)
 		eventfs_set_ef_status_free(dentry);
 	iput(inode);
@@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
 	return dentry;
 }
 
+struct inode *tracefs_get_inode(struct super_block *sb)
+{
+	struct inode *inode = new_inode(sb);
+
+	BUG_ON(sb->s_op != &tracefs_super_operations);
+	if (inode) {
+		inode->i_ino = get_next_ino();
+		inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
+	}
+	return inode;
+}
+
 /**
  * tracefs_create_file - create a file in the tracefs filesystem
  * @name: a pointer to a string containing the name of the file to create.
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index 69c2b1d87c46..9f6f303a9e58 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -9,6 +9,7 @@ enum {
 struct tracefs_inode {
 	unsigned long           flags;
 	void                    *private;
+	unsigned long		magic;
 	struct inode            vfs_inode;
 };
 

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

[-- Attachment #1: Type: text/plain, Size: 3297 bytes --]

applied your patch and here's the output.

--Ilkka

On Wed, May 29, 2024 at 10:48 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 29 May 2024 14:47:57 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
>
> > Let me make a debug patch (that crashes on this issue) for that kernel,
> > and perhaps you could bisect it?
>
> Can you try this on 6.6-rc1 and see if it gives you any other splats?
>
> Hmm, you can switch it to WARN_ON and that way it may not crash the
> machine, and you can use dmesg to get the output.
>
> Thanks,
>
> -- Steve
>
>
> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> index de5b72216b1a..a090495e78c9 100644
> --- a/fs/tracefs/inode.c
> +++ b/fs/tracefs/inode.c
> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
>                 return NULL;
>
>         ti->flags = 0;
> +       ti->magic = 20240823;
>
>         return &ti->vfs_inode;
>  }
>
>  static void tracefs_free_inode(struct inode *inode)
>  {
> -       kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
> +       struct tracefs_inode *ti = get_tracefs(inode);
> +
> +       BUG_ON(ti->magic != 20240823);
> +       kmem_cache_free(tracefs_inode_cachep, ti);
>  }
>
>  static ssize_t default_read_file(struct file *file, char __user *buf,
> @@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
>         .rmdir          = tracefs_syscall_rmdir,
>  };
>
> -struct inode *tracefs_get_inode(struct super_block *sb)
> -{
> -       struct inode *inode = new_inode(sb);
> -       if (inode) {
> -               inode->i_ino = get_next_ino();
> -               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> -       }
> -       return inode;
> -}
> -
>  struct tracefs_mount_opts {
>         kuid_t uid;
>         kgid_t gid;
> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
>                 return;
>
>         ti = get_tracefs(inode);
> +       BUG_ON(ti->magic != 20240823);
>         if (ti && ti->flags & TRACEFS_EVENT_INODE)
>                 eventfs_set_ef_status_free(dentry);
>         iput(inode);
> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
>         return dentry;
>  }
>
> +struct inode *tracefs_get_inode(struct super_block *sb)
> +{
> +       struct inode *inode = new_inode(sb);
> +
> +       BUG_ON(sb->s_op != &tracefs_super_operations);
> +       if (inode) {
> +               inode->i_ino = get_next_ino();
> +               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> +       }
> +       return inode;
> +}
> +
>  /**
>   * tracefs_create_file - create a file in the tracefs filesystem
>   * @name: a pointer to a string containing the name of the file to create.
> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> index 69c2b1d87c46..9f6f303a9e58 100644
> --- a/fs/tracefs/internal.h
> +++ b/fs/tracefs/internal.h
> @@ -9,6 +9,7 @@ enum {
>  struct tracefs_inode {
>         unsigned long           flags;
>         void                    *private;
> +       unsigned long           magic;
>         struct inode            vfs_inode;
>  };
>

[-- Attachment #2: 6.6-rc1-steve-patch.jpeg --]
[-- Type: image/jpeg, Size: 1702955 bytes --]

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Thu, 30 May 2024 16:02:37 +0300
Ilkka Naulapää <digirigawa@gmail.com> wrote:

> applied your patch and here's the output.
> 

Unfortunately, it doesn't give me any new information. I added one more
BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/

-- Steve

diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
index de5b72216b1a..a090495e78c9 100644
--- a/fs/tracefs/inode.c
+++ b/fs/tracefs/inode.c
@@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
 		return NULL;
 
 	ti->flags = 0;
+	ti->magic = 20240823;
 
 	return &ti->vfs_inode;
 }
 
 static void tracefs_free_inode(struct inode *inode)
 {
-	kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
+	struct tracefs_inode *ti = get_tracefs(inode);
+
+	BUG_ON(ti->magic != 20240823);
+	kmem_cache_free(tracefs_inode_cachep, ti);
 }
 
 static ssize_t default_read_file(struct file *file, char __user *buf,
@@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
 	.rmdir		= tracefs_syscall_rmdir,
 };
 
-struct inode *tracefs_get_inode(struct super_block *sb)
-{
-	struct inode *inode = new_inode(sb);
-	if (inode) {
-		inode->i_ino = get_next_ino();
-		inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
-	}
-	return inode;
-}
-
 struct tracefs_mount_opts {
 	kuid_t uid;
 	kgid_t gid;
@@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
 		return;
 
 	ti = get_tracefs(inode);
+	BUG_ON(ti->magic != 20240823);
 	if (ti && ti->flags & TRACEFS_EVENT_INODE)
 		eventfs_set_ef_status_free(dentry);
 	iput(inode);
@@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
 	return dentry;
 }
 
+struct inode *tracefs_get_inode(struct super_block *sb)
+{
+	struct inode *inode = new_inode(sb);
+
+	BUG_ON(sb->s_op != &tracefs_super_operations);
+	if (inode) {
+		inode->i_ino = get_next_ino();
+		inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
+	}
+	return inode;
+}
+
 /**
  * tracefs_create_file - create a file in the tracefs filesystem
  * @name: a pointer to a string containing the name of the file to create.
diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
index 69c2b1d87c46..9059b8b11bb6 100644
--- a/fs/tracefs/internal.h
+++ b/fs/tracefs/internal.h
@@ -9,12 +9,15 @@ enum {
 struct tracefs_inode {
 	unsigned long           flags;
 	void                    *private;
+	unsigned long		magic;
 	struct inode            vfs_inode;
 };
 
 static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
 {
-	return container_of(inode, struct tracefs_inode, vfs_inode);
+	struct tracefs_inode *ti = container_of(inode, struct tracefs_inode, vfs_inode);
+	BUG_ON(ti->magic != 20240823);
+	return ti;
 }
 
 struct dentry *tracefs_start_creating(const char *name, struct dentry *parent);

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

[-- Attachment #1: Type: text/plain, Size: 3707 bytes --]

sorry longer delay, been a bit busy but here is the result from that
new patch. Only applied this patch so if the previous one is needed
also, let me know and I'll rerun it.

--Ilkka

On Thu, May 30, 2024 at 5:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 30 May 2024 16:02:37 +0300
> Ilkka Naulapää <digirigawa@gmail.com> wrote:
>
> > applied your patch and here's the output.
> >
>
> Unfortunately, it doesn't give me any new information. I added one more
> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
>
> -- Steve
>
> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> index de5b72216b1a..a090495e78c9 100644
> --- a/fs/tracefs/inode.c
> +++ b/fs/tracefs/inode.c
> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
>                 return NULL;
>
>         ti->flags = 0;
> +       ti->magic = 20240823;
>
>         return &ti->vfs_inode;
>  }
>
>  static void tracefs_free_inode(struct inode *inode)
>  {
> -       kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
> +       struct tracefs_inode *ti = get_tracefs(inode);
> +
> +       BUG_ON(ti->magic != 20240823);
> +       kmem_cache_free(tracefs_inode_cachep, ti);
>  }
>
>  static ssize_t default_read_file(struct file *file, char __user *buf,
> @@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
>         .rmdir          = tracefs_syscall_rmdir,
>  };
>
> -struct inode *tracefs_get_inode(struct super_block *sb)
> -{
> -       struct inode *inode = new_inode(sb);
> -       if (inode) {
> -               inode->i_ino = get_next_ino();
> -               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> -       }
> -       return inode;
> -}
> -
>  struct tracefs_mount_opts {
>         kuid_t uid;
>         kgid_t gid;
> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
>                 return;
>
>         ti = get_tracefs(inode);
> +       BUG_ON(ti->magic != 20240823);
>         if (ti && ti->flags & TRACEFS_EVENT_INODE)
>                 eventfs_set_ef_status_free(dentry);
>         iput(inode);
> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
>         return dentry;
>  }
>
> +struct inode *tracefs_get_inode(struct super_block *sb)
> +{
> +       struct inode *inode = new_inode(sb);
> +
> +       BUG_ON(sb->s_op != &tracefs_super_operations);
> +       if (inode) {
> +               inode->i_ino = get_next_ino();
> +               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> +       }
> +       return inode;
> +}
> +
>  /**
>   * tracefs_create_file - create a file in the tracefs filesystem
>   * @name: a pointer to a string containing the name of the file to create.
> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> index 69c2b1d87c46..9059b8b11bb6 100644
> --- a/fs/tracefs/internal.h
> +++ b/fs/tracefs/internal.h
> @@ -9,12 +9,15 @@ enum {
>  struct tracefs_inode {
>         unsigned long           flags;
>         void                    *private;
> +       unsigned long           magic;
>         struct inode            vfs_inode;
>  };
>
>  static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
>  {
> -       return container_of(inode, struct tracefs_inode, vfs_inode);
> +       struct tracefs_inode *ti = container_of(inode, struct tracefs_inode, vfs_inode);
> +       BUG_ON(ti->magic != 20240823);
> +       return ti;
>  }
>
>  struct dentry *tracefs_start_creating(const char *name, struct dentry *parent);

[-- Attachment #2: 6.6-rc1-steve-patch-2.jpeg --]
[-- Type: image/jpeg, Size: 2087477 bytes --]

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Linux regression tracking (Thorsten Leemhuis)]

Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
for once, to make this easily accessible to everyone.

Ilkka or Steven, what happened to this? This thread looks stalled. I
also was unsuccessful when looking for other threads related to this
report or the culprit. Did it fall through the cracks or am I missing
something here?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

On 02.06.24 09:32, Ilkka Naulapää wrote:
> sorry longer delay, been a bit busy but here is the result from that
> new patch. Only applied this patch so if the previous one is needed
> also, let me know and I'll rerun it.
> 
> --Ilkka
> 
> On Thu, May 30, 2024 at 5:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>>
>> On Thu, 30 May 2024 16:02:37 +0300
>> Ilkka Naulapää <digirigawa@gmail.com> wrote:
>>
>>> applied your patch and here's the output.
>>>
>>
>> Unfortunately, it doesn't give me any new information. I added one more
>> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
>>
>> -- Steve
>>
>> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
>> index de5b72216b1a..a090495e78c9 100644
>> --- a/fs/tracefs/inode.c
>> +++ b/fs/tracefs/inode.c
>> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
>>                 return NULL;
>>
>>         ti->flags = 0;
>> +       ti->magic = 20240823;
>>
>>         return &ti->vfs_inode;
>>  }
>>
>>  static void tracefs_free_inode(struct inode *inode)
>>  {
>> -       kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
>> +       struct tracefs_inode *ti = get_tracefs(inode);
>> +
>> +       BUG_ON(ti->magic != 20240823);
>> +       kmem_cache_free(tracefs_inode_cachep, ti);
>>  }
>>
>>  static ssize_t default_read_file(struct file *file, char __user *buf,
>> @@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
>>         .rmdir          = tracefs_syscall_rmdir,
>>  };
>>
>> -struct inode *tracefs_get_inode(struct super_block *sb)
>> -{
>> -       struct inode *inode = new_inode(sb);
>> -       if (inode) {
>> -               inode->i_ino = get_next_ino();
>> -               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
>> -       }
>> -       return inode;
>> -}
>> -
>>  struct tracefs_mount_opts {
>>         kuid_t uid;
>>         kgid_t gid;
>> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
>>                 return;
>>
>>         ti = get_tracefs(inode);
>> +       BUG_ON(ti->magic != 20240823);
>>         if (ti && ti->flags & TRACEFS_EVENT_INODE)
>>                 eventfs_set_ef_status_free(dentry);
>>         iput(inode);
>> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
>>         return dentry;
>>  }
>>
>> +struct inode *tracefs_get_inode(struct super_block *sb)
>> +{
>> +       struct inode *inode = new_inode(sb);
>> +
>> +       BUG_ON(sb->s_op != &tracefs_super_operations);
>> +       if (inode) {
>> +               inode->i_ino = get_next_ino();
>> +               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
>> +       }
>> +       return inode;
>> +}
>> +
>>  /**
>>   * tracefs_create_file - create a file in the tracefs filesystem
>>   * @name: a pointer to a string containing the name of the file to create.
>> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
>> index 69c2b1d87c46..9059b8b11bb6 100644
>> --- a/fs/tracefs/internal.h
>> +++ b/fs/tracefs/internal.h
>> @@ -9,12 +9,15 @@ enum {
>>  struct tracefs_inode {
>>         unsigned long           flags;
>>         void                    *private;
>> +       unsigned long           magic;
>>         struct inode            vfs_inode;
>>  };
>>
>>  static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
>>  {
>> -       return container_of(inode, struct tracefs_inode, vfs_inode);
>> +       struct tracefs_inode *ti = container_of(inode, struct tracefs_inode, vfs_inode);
>> +       BUG_ON(ti->magic != 20240823);
>> +       return ti;
>>  }
>>
>>  struct dentry *tracefs_start_creating(const char *name, struct dentry *parent);

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Wed, 12 Jun 2024 15:36:22 +0200
"Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:

> Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
> for once, to make this easily accessible to everyone.
> 
> Ilkka or Steven, what happened to this? This thread looks stalled. I
> also was unsuccessful when looking for other threads related to this
> report or the culprit. Did it fall through the cracks or am I missing
> something here?

Honesty, I have no idea where the bug is. I can't reproduce it. These
patches I sent would check all the places that add to the list to make
sure the proper trace_inode was being added, and the output shows that
they are all correct. Then suddenly, something that came from the
inode cache is calling the tracefs inode cache to free it, and that's
where the bug is happening.

This really looks like another bug that the recent changes have made
more predominate.

-- Steve


> 
> Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> --
> Everything you wanna know about Linux kernel regression tracking:
> https://linux-regtracking.leemhuis.info/about/#tldr
> If I did something stupid, please tell me, as explained on that page.
> 
> #regzbot poke
> 
> On 02.06.24 09:32, Ilkka Naulapää wrote:
> > sorry longer delay, been a bit busy but here is the result from that
> > new patch. Only applied this patch so if the previous one is needed
> > also, let me know and I'll rerun it.
> > 
> > --Ilkka
> > 
> > On Thu, May 30, 2024 at 5:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:  
> >>
> >> On Thu, 30 May 2024 16:02:37 +0300
> >> Ilkka Naulapää <digirigawa@gmail.com> wrote:
> >>  
> >>> applied your patch and here's the output.
> >>>  
> >>
> >> Unfortunately, it doesn't give me any new information. I added one more
> >> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
> >>
> >> -- Steve
> >>
> >> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> >> index de5b72216b1a..a090495e78c9 100644
> >> --- a/fs/tracefs/inode.c
> >> +++ b/fs/tracefs/inode.c
> >> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
> >>                 return NULL;
> >>
> >>         ti->flags = 0;
> >> +       ti->magic = 20240823;
> >>
> >>         return &ti->vfs_inode;
> >>  }
> >>
> >>  static void tracefs_free_inode(struct inode *inode)
> >>  {
> >> -       kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
> >> +       struct tracefs_inode *ti = get_tracefs(inode);
> >> +
> >> +       BUG_ON(ti->magic != 20240823);
> >> +       kmem_cache_free(tracefs_inode_cachep, ti);
> >>  }
> >>
> >>  static ssize_t default_read_file(struct file *file, char __user *buf,
> >> @@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
> >>         .rmdir          = tracefs_syscall_rmdir,
> >>  };
> >>
> >> -struct inode *tracefs_get_inode(struct super_block *sb)
> >> -{
> >> -       struct inode *inode = new_inode(sb);
> >> -       if (inode) {
> >> -               inode->i_ino = get_next_ino();
> >> -               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> >> -       }
> >> -       return inode;
> >> -}
> >> -
> >>  struct tracefs_mount_opts {
> >>         kuid_t uid;
> >>         kgid_t gid;
> >> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
> >>                 return;
> >>
> >>         ti = get_tracefs(inode);
> >> +       BUG_ON(ti->magic != 20240823);
> >>         if (ti && ti->flags & TRACEFS_EVENT_INODE)
> >>                 eventfs_set_ef_status_free(dentry);
> >>         iput(inode);
> >> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
> >>         return dentry;
> >>  }
> >>
> >> +struct inode *tracefs_get_inode(struct super_block *sb)
> >> +{
> >> +       struct inode *inode = new_inode(sb);
> >> +
> >> +       BUG_ON(sb->s_op != &tracefs_super_operations);
> >> +       if (inode) {
> >> +               inode->i_ino = get_next_ino();
> >> +               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> >> +       }
> >> +       return inode;
> >> +}
> >> +
> >>  /**
> >>   * tracefs_create_file - create a file in the tracefs filesystem
> >>   * @name: a pointer to a string containing the name of the file to create.
> >> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> >> index 69c2b1d87c46..9059b8b11bb6 100644
> >> --- a/fs/tracefs/internal.h
> >> +++ b/fs/tracefs/internal.h
> >> @@ -9,12 +9,15 @@ enum {
> >>  struct tracefs_inode {
> >>         unsigned long           flags;
> >>         void                    *private;
> >> +       unsigned long           magic;
> >>         struct inode            vfs_inode;
> >>  };
> >>
> >>  static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
> >>  {
> >> -       return container_of(inode, struct tracefs_inode, vfs_inode);
> >> +       struct tracefs_inode *ti = container_of(inode, struct tracefs_inode, vfs_inode);
> >> +       BUG_ON(ti->magic != 20240823);
> >> +       return ti;
> >>  }
> >>
> >>  struct dentry *tracefs_start_creating(const char *name, struct dentry *parent);  

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

ok, so if you don't have any idea where this bug is after those debug
patches, I'll try to find some time to bisect it as a last resort.
Stay tuned.

--Ilkka

On Wed, Jun 12, 2024 at 6:56 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Wed, 12 Jun 2024 15:36:22 +0200
> "Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:
>
> > Hi, Thorsten here, the Linux kernel's regression tracker. Top-posting
> > for once, to make this easily accessible to everyone.
> >
> > Ilkka or Steven, what happened to this? This thread looks stalled. I
> > also was unsuccessful when looking for other threads related to this
> > report or the culprit. Did it fall through the cracks or am I missing
> > something here?
>
> Honesty, I have no idea where the bug is. I can't reproduce it. These
> patches I sent would check all the places that add to the list to make
> sure the proper trace_inode was being added, and the output shows that
> they are all correct. Then suddenly, something that came from the
> inode cache is calling the tracefs inode cache to free it, and that's
> where the bug is happening.
>
> This really looks like another bug that the recent changes have made
> more predominate.
>
> -- Steve
>
>
> >
> > Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
> > --
> > Everything you wanna know about Linux kernel regression tracking:
> > https://linux-regtracking.leemhuis.info/about/#tldr
> > If I did something stupid, please tell me, as explained on that page.
> >
> > #regzbot poke
> >
> > On 02.06.24 09:32, Ilkka Naulapää wrote:
> > > sorry longer delay, been a bit busy but here is the result from that
> > > new patch. Only applied this patch so if the previous one is needed
> > > also, let me know and I'll rerun it.
> > >
> > > --Ilkka
> > >
> > > On Thu, May 30, 2024 at 5:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:
> > >>
> > >> On Thu, 30 May 2024 16:02:37 +0300
> > >> Ilkka Naulapää <digirigawa@gmail.com> wrote:
> > >>
> > >>> applied your patch and here's the output.
> > >>>
> > >>
> > >> Unfortunately, it doesn't give me any new information. I added one more
> > >> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
> > >>
> > >> -- Steve
> > >>
> > >> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
> > >> index de5b72216b1a..a090495e78c9 100644
> > >> --- a/fs/tracefs/inode.c
> > >> +++ b/fs/tracefs/inode.c
> > >> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
> > >>                 return NULL;
> > >>
> > >>         ti->flags = 0;
> > >> +       ti->magic = 20240823;
> > >>
> > >>         return &ti->vfs_inode;
> > >>  }
> > >>
> > >>  static void tracefs_free_inode(struct inode *inode)
> > >>  {
> > >> -       kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
> > >> +       struct tracefs_inode *ti = get_tracefs(inode);
> > >> +
> > >> +       BUG_ON(ti->magic != 20240823);
> > >> +       kmem_cache_free(tracefs_inode_cachep, ti);
> > >>  }
> > >>
> > >>  static ssize_t default_read_file(struct file *file, char __user *buf,
> > >> @@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
> > >>         .rmdir          = tracefs_syscall_rmdir,
> > >>  };
> > >>
> > >> -struct inode *tracefs_get_inode(struct super_block *sb)
> > >> -{
> > >> -       struct inode *inode = new_inode(sb);
> > >> -       if (inode) {
> > >> -               inode->i_ino = get_next_ino();
> > >> -               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> > >> -       }
> > >> -       return inode;
> > >> -}
> > >> -
> > >>  struct tracefs_mount_opts {
> > >>         kuid_t uid;
> > >>         kgid_t gid;
> > >> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
> > >>                 return;
> > >>
> > >>         ti = get_tracefs(inode);
> > >> +       BUG_ON(ti->magic != 20240823);
> > >>         if (ti && ti->flags & TRACEFS_EVENT_INODE)
> > >>                 eventfs_set_ef_status_free(dentry);
> > >>         iput(inode);
> > >> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
> > >>         return dentry;
> > >>  }
> > >>
> > >> +struct inode *tracefs_get_inode(struct super_block *sb)
> > >> +{
> > >> +       struct inode *inode = new_inode(sb);
> > >> +
> > >> +       BUG_ON(sb->s_op != &tracefs_super_operations);
> > >> +       if (inode) {
> > >> +               inode->i_ino = get_next_ino();
> > >> +               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
> > >> +       }
> > >> +       return inode;
> > >> +}
> > >> +
> > >>  /**
> > >>   * tracefs_create_file - create a file in the tracefs filesystem
> > >>   * @name: a pointer to a string containing the name of the file to create.
> > >> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
> > >> index 69c2b1d87c46..9059b8b11bb6 100644
> > >> --- a/fs/tracefs/internal.h
> > >> +++ b/fs/tracefs/internal.h
> > >> @@ -9,12 +9,15 @@ enum {
> > >>  struct tracefs_inode {
> > >>         unsigned long           flags;
> > >>         void                    *private;
> > >> +       unsigned long           magic;
> > >>         struct inode            vfs_inode;
> > >>  };
> > >>
> > >>  static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
> > >>  {
> > >> -       return container_of(inode, struct tracefs_inode, vfs_inode);
> > >> +       struct tracefs_inode *ti = container_of(inode, struct tracefs_inode, vfs_inode);
> > >> +       BUG_ON(ti->magic != 20240823);
> > >> +       return ti;
> > >>  }
> > >>
> > >>  struct dentry *tracefs_start_creating(const char *name, struct dentry *parent);
>

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Linux regression tracking (Thorsten Leemhuis)]

On 13.06.24 09:32, Ilkka Naulapää wrote:
> On Wed, Jun 12, 2024 at 6:56 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>> On Wed, 12 Jun 2024 15:36:22 +0200
>> "Linux regression tracking (Thorsten Leemhuis)" <regressions@leemhuis.info> wrote:
>>>
>>> Ilkka or Steven, what happened to this? This thread looks stalled. I
>>> also was unsuccessful when looking for other threads related to this
>>> report or the culprit. Did it fall through the cracks or am I missing
>>> something here?
>
>> Honesty, I have no idea where the bug is. I can't reproduce it. [...]

Steven, thx for the update! And yeah, that's how it sometimes is. Given
that we haven't seen similar reports (at least afaics) it's nothing I
worry much about.

> ok, so if you don't have any idea where this bug is after those debug
> patches, I'll try to find some time to bisect it as a last resort.
> Stay tuned.

Yeah, that would be great help. Thank you, too!

Ciao, Thorsten

>>> On 02.06.24 09:32, Ilkka Naulapää wrote:
>>>> sorry longer delay, been a bit busy but here is the result from that
>>>> new patch. Only applied this patch so if the previous one is needed
>>>> also, let me know and I'll rerun it.
>>>>
>>>> --Ilkka
>>>>
>>>> On Thu, May 30, 2024 at 5:00 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>>>>>
>>>>> On Thu, 30 May 2024 16:02:37 +0300
>>>>> Ilkka Naulapää <digirigawa@gmail.com> wrote:
>>>>>
>>>>>> applied your patch and here's the output.
>>>>>>
>>>>>
>>>>> Unfortunately, it doesn't give me any new information. I added one more
>>>>> BUG on, want to try this? Otherwise, I'm pretty much at a lost. :-/
>>>>>
>>>>> -- Steve
>>>>>
>>>>> diff --git a/fs/tracefs/inode.c b/fs/tracefs/inode.c
>>>>> index de5b72216b1a..a090495e78c9 100644
>>>>> --- a/fs/tracefs/inode.c
>>>>> +++ b/fs/tracefs/inode.c
>>>>> @@ -39,13 +39,17 @@ static struct inode *tracefs_alloc_inode(struct super_block *sb)
>>>>>                 return NULL;
>>>>>
>>>>>         ti->flags = 0;
>>>>> +       ti->magic = 20240823;
>>>>>
>>>>>         return &ti->vfs_inode;
>>>>>  }
>>>>>
>>>>>  static void tracefs_free_inode(struct inode *inode)
>>>>>  {
>>>>> -       kmem_cache_free(tracefs_inode_cachep, get_tracefs(inode));
>>>>> +       struct tracefs_inode *ti = get_tracefs(inode);
>>>>> +
>>>>> +       BUG_ON(ti->magic != 20240823);
>>>>> +       kmem_cache_free(tracefs_inode_cachep, ti);
>>>>>  }
>>>>>
>>>>>  static ssize_t default_read_file(struct file *file, char __user *buf,
>>>>> @@ -147,16 +151,6 @@ static const struct inode_operations tracefs_dir_inode_operations = {
>>>>>         .rmdir          = tracefs_syscall_rmdir,
>>>>>  };
>>>>>
>>>>> -struct inode *tracefs_get_inode(struct super_block *sb)
>>>>> -{
>>>>> -       struct inode *inode = new_inode(sb);
>>>>> -       if (inode) {
>>>>> -               inode->i_ino = get_next_ino();
>>>>> -               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
>>>>> -       }
>>>>> -       return inode;
>>>>> -}
>>>>> -
>>>>>  struct tracefs_mount_opts {
>>>>>         kuid_t uid;
>>>>>         kgid_t gid;
>>>>> @@ -384,6 +378,7 @@ static void tracefs_dentry_iput(struct dentry *dentry, struct inode *inode)
>>>>>                 return;
>>>>>
>>>>>         ti = get_tracefs(inode);
>>>>> +       BUG_ON(ti->magic != 20240823);
>>>>>         if (ti && ti->flags & TRACEFS_EVENT_INODE)
>>>>>                 eventfs_set_ef_status_free(dentry);
>>>>>         iput(inode);
>>>>> @@ -568,6 +563,18 @@ struct dentry *eventfs_end_creating(struct dentry *dentry)
>>>>>         return dentry;
>>>>>  }
>>>>>
>>>>> +struct inode *tracefs_get_inode(struct super_block *sb)
>>>>> +{
>>>>> +       struct inode *inode = new_inode(sb);
>>>>> +
>>>>> +       BUG_ON(sb->s_op != &tracefs_super_operations);
>>>>> +       if (inode) {
>>>>> +               inode->i_ino = get_next_ino();
>>>>> +               inode->i_atime = inode->i_mtime = inode_set_ctime_current(inode);
>>>>> +       }
>>>>> +       return inode;
>>>>> +}
>>>>> +
>>>>>  /**
>>>>>   * tracefs_create_file - create a file in the tracefs filesystem
>>>>>   * @name: a pointer to a string containing the name of the file to create.
>>>>> diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h
>>>>> index 69c2b1d87c46..9059b8b11bb6 100644
>>>>> --- a/fs/tracefs/internal.h
>>>>> +++ b/fs/tracefs/internal.h
>>>>> @@ -9,12 +9,15 @@ enum {
>>>>>  struct tracefs_inode {
>>>>>         unsigned long           flags;
>>>>>         void                    *private;
>>>>> +       unsigned long           magic;
>>>>>         struct inode            vfs_inode;
>>>>>  };
>>>>>
>>>>>  static inline struct tracefs_inode *get_tracefs(const struct inode *inode)
>>>>>  {
>>>>> -       return container_of(inode, struct tracefs_inode, vfs_inode);
>>>>> +       struct tracefs_inode *ti = container_of(inode, struct tracefs_inode, vfs_inode);
>>>>> +       BUG_ON(ti->magic != 20240823);
>>>>> +       return ti;
>>>>>  }
>>>>>
>>>>>  struct dentry *tracefs_start_creating(const char *name, struct dentry *parent);
>>
> 
> 

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Steven Rostedt]

On Thu, 13 Jun 2024 10:32:24 +0300
Ilkka Naulapää <digirigawa@gmail.com> wrote:

> ok, so if you don't have any idea where this bug is after those debug
> patches, I'll try to find some time to bisect it as a last resort.
> Stay tuned.

FYI,

I just debugged a strange crash that was caused by my config having
something leftover from your config. Specifically, that was:

CONFIG_FORCE_NR_CPUS

Do you get any warning about nr cpus not matching at boot up?

Regardless, can you disable that and see if you still get the same
crash.

Thanks,

-- Steve

Re: Bug in Kernel 6.8.x, 6.9.x Causing Trace/Panic During Shutdown/Reboot

[Ilkka Naulapää]

disabled  CONFIG_FORCE_NR_CPUS option for 6.9.5 but the trace + panic
still exists. So that one didn't help. I've also been bisecting the
trace but have not finished it yet as the last half dozen builds
produced non-bootable kernels. Anyway, I will continue it soon(ish)
when I have a bit more free time.

--Ilkka

On Tue, Jun 18, 2024 at 5:52 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Thu, 13 Jun 2024 10:32:24 +0300
> Ilkka Naulapää <digirigawa@gmail.com> wrote:
>
> > ok, so if you don't have any idea where this bug is after those debug
> > patches, I'll try to find some time to bisect it as a last resort.
> > Stay tuned.
>
> FYI,
>
> I just debugged a strange crash that was caused by my config having
> something leftover from your config. Specifically, that was:
>
> CONFIG_FORCE_NR_CPUS
>
> Do you get any warning about nr cpus not matching at boot up?
>
> Regardless, can you disable that and see if you still get the same
> crash.
>
> Thanks,
>
> -- Steve