Backport of "xfs: open code ioend needs workqueue helper" to 5.10?

Backport of "xfs: open code ioend needs workqueue helper" to 5.10?

[Christian Theune]

Hi,

afaict this was fixed in 5.13 but hasn’t been backported. I’ve seen one of our VMs running 5.10.169 crash with this.

Anybody willing to backport this? It’s only triggered a single time so far and we are rolling out 5.15 anyways, but maybe this was an oversight … ?

commit 7adb8f14e134d5f885d47c4ccd620836235f0b7f
Author: Brian Foster <bfoster@redhat.com>
Date:   Fri Apr 9 10:27:55 2021 -0700

    xfs: open code ioend needs workqueue helper
    
    Open code xfs_ioend_needs_workqueue() into the only remaining
    caller.
    
    Signed-off-by: Brian Foster <bfoster@redhat.com>
    Reviewed-by: Christoph Hellwig <hch@lst.de>
    Reviewed-by: Darrick J. Wong <djwong@kernel.org>
    Signed-off-by: Darrick J. Wong <djwong@kernel.org>


Kind regards,
Christian

-- 
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

Re: Backport of "xfs: open code ioend needs workqueue helper" to 5.10?

[Amir Goldstein]

On Wed, Apr 12, 2023 at 5:06 PM Christian Theune <ct@flyingcircus.io> wrote:
>
> Hi,
>
> afaict this was fixed in 5.13 but hasn’t been backported. I’ve seen one of our VMs running 5.10.169 crash with this.
>
> Anybody willing to backport this? It’s only triggered a single time so far and we are rolling out 5.15 anyways, but maybe this was an oversight … ?

What do you mean by "crash with this"?

There is no logic change in the commit mentioned below.

Thanks,
Amir.

>
> commit 7adb8f14e134d5f885d47c4ccd620836235f0b7f
> Author: Brian Foster <bfoster@redhat.com>
> Date:   Fri Apr 9 10:27:55 2021 -0700
>
>     xfs: open code ioend needs workqueue helper
>
>     Open code xfs_ioend_needs_workqueue() into the only remaining
>     caller.
>
>     Signed-off-by: Brian Foster <bfoster@redhat.com>
>     Reviewed-by: Christoph Hellwig <hch@lst.de>
>     Reviewed-by: Darrick J. Wong <djwong@kernel.org>
>     Signed-off-by: Darrick J. Wong <djwong@kernel.org>
>
>
> Kind regards,
> Christian
>
> --
> Christian Theune · ct@flyingcircus.io · +49 345 219401 0
> Flying Circus Internet Operations GmbH · https://flyingcircus.io
> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
>

Re: Backport of "xfs: open code ioend needs workqueue helper" to 5.10?

[Christian Theune]

Hi,

ugh. Sorry, looks like I jumped the gun. Mea culpa.

We experienced a hang like this:

Apr 05 11:51:27 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 05 11:51:27 kernel: task:xfs-conv/vdc1   state:D stack:    0 pid:  606 ppid:     2 flags:0x00004080
Apr 05 11:51:27 kernel: Workqueue: xfs-conv/vdc1 xfs_end_io [xfs]
Apr 05 11:51:27 kernel: Call Trace:
Apr 05 11:51:27 kernel:  __schedule+0x274/0x870
Apr 05 11:51:27 kernel:  schedule+0x46/0xb0
Apr 05 11:51:27 kernel:  xlog_grant_head_wait+0xc5/0x1d0 [xfs]
Apr 05 11:51:27 kernel:  xlog_grant_head_check+0xde/0x100 [xfs]
Apr 05 11:51:27 kernel:  xfs_log_reserve+0xbe/0x1b0 [xfs]
Apr 05 11:51:27 kernel:  xfs_trans_reserve+0x143/0x180 [xfs]
Apr 05 11:51:27 kernel:  xfs_trans_alloc+0xee/0x1a0 [xfs]
Apr 05 11:51:27 kernel:  xfs_iomap_write_unwritten+0x120/0x2e0 [xfs]
Apr 05 11:51:27 kernel:  ? record_times+0x15/0x90
Apr 05 11:51:27 kernel:  xfs_end_ioend+0xd8/0x140 [xfs]
Apr 05 11:51:27 kernel:  xfs_end_io+0xb8/0xf0 [xfs]
Apr 05 11:51:27 kernel:  process_one_work+0x1b6/0x350
Apr 05 11:51:27 kernel:  rescuer_thread+0x1d1/0x3a0
Apr 05 11:51:27 kernel:  ? worker_thread+0x3e0/0x3e0
Apr 05 11:51:27 kernel:  kthread+0x11b/0x140
Apr 05 11:51:27 kernel:  ? kthread_associate_blkcg+0xb0/0xb0
Apr 05 11:51:27 kernel:  ret_from_fork+0x22/0x30

Which seems to be similar to this:
https://bugs.launchpad.net/bugs/1996269

I followed their patchset here:
https://review.opendev.org/c/starlingx/kernel/+/864257

And I was under the impression that I picked the right one to ask for backporting, but it seems that was incorrect. I went through the list again and I think the following patches are the ones missing from 5.10:

8182ec00803085354761bbadf0287cad7eac0e2f - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0035-xfs-drop-submit-side-trans-alloc-for-append-ioends.patch
edbf1eb9032b84631031d9b43570e262f3461c24 - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0036-xfs-open-code-ioend-needs-workqueue-helper.patch
170e31793806ce5e5a9647b6340954536244518e - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0037-xfs-drop-unused-ioend-private-merge-and-setfilesize-.patch
2fd609b6c90a88630a50fb317473b210759b3873 - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0038-xfs-drop-unnecessary-setfilesize-helper.patch

The first one in the series was applied in 5.10.129 afaict and landed in mainstream in 5.12 already.

Cheers,
Christian

> On 12. Apr 2023, at 16:12, Amir Goldstein <amir73il@gmail.com> wrote:
> 
> On Wed, Apr 12, 2023 at 5:06 PM Christian Theune <ct@flyingcircus.io> wrote:
>> 
>> Hi,
>> 
>> afaict this was fixed in 5.13 but hasn’t been backported. I’ve seen one of our VMs running 5.10.169 crash with this.
>> 
>> Anybody willing to backport this? It’s only triggered a single time so far and we are rolling out 5.15 anyways, but maybe this was an oversight … ?
> 
> What do you mean by "crash with this"?
> 
> There is no logic change in the commit mentioned below.
> 
> Thanks,
> Amir.
> 
>> 
>> commit 7adb8f14e134d5f885d47c4ccd620836235f0b7f
>> Author: Brian Foster <bfoster@redhat.com>
>> Date:   Fri Apr 9 10:27:55 2021 -0700
>> 
>>    xfs: open code ioend needs workqueue helper
>> 
>>    Open code xfs_ioend_needs_workqueue() into the only remaining
>>    caller.
>> 
>>    Signed-off-by: Brian Foster <bfoster@redhat.com>
>>    Reviewed-by: Christoph Hellwig <hch@lst.de>
>>    Reviewed-by: Darrick J. Wong <djwong@kernel.org>
>>    Signed-off-by: Darrick J. Wong <djwong@kernel.org>
>> 
>> 
>> Kind regards,
>> Christian
>> 
>> --
>> Christian Theune · ct@flyingcircus.io · +49 345 219401 0
>> Flying Circus Internet Operations GmbH · https://flyingcircus.io
>> Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
>> HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick
>> 

Liebe Grüße,
Christian Theune

-- 
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

Re: Backport of "xfs: open code ioend needs workqueue helper" to 5.10?

[Amir Goldstein]

On Wed, Apr 12, 2023 at 6:58 PM Christian Theune <ct@flyingcircus.io> wrote:
>
> Hi,
>
> ugh. Sorry, looks like I jumped the gun. Mea culpa.
>
> We experienced a hang like this:
>
> Apr 05 11:51:27 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 05 11:51:27 kernel: task:xfs-conv/vdc1   state:D stack:    0 pid:  606 ppid:     2 flags:0x00004080
> Apr 05 11:51:27 kernel: Workqueue: xfs-conv/vdc1 xfs_end_io [xfs]
> Apr 05 11:51:27 kernel: Call Trace:
> Apr 05 11:51:27 kernel:  __schedule+0x274/0x870
> Apr 05 11:51:27 kernel:  schedule+0x46/0xb0
> Apr 05 11:51:27 kernel:  xlog_grant_head_wait+0xc5/0x1d0 [xfs]
> Apr 05 11:51:27 kernel:  xlog_grant_head_check+0xde/0x100 [xfs]
> Apr 05 11:51:27 kernel:  xfs_log_reserve+0xbe/0x1b0 [xfs]
> Apr 05 11:51:27 kernel:  xfs_trans_reserve+0x143/0x180 [xfs]
> Apr 05 11:51:27 kernel:  xfs_trans_alloc+0xee/0x1a0 [xfs]
> Apr 05 11:51:27 kernel:  xfs_iomap_write_unwritten+0x120/0x2e0 [xfs]
> Apr 05 11:51:27 kernel:  ? record_times+0x15/0x90
> Apr 05 11:51:27 kernel:  xfs_end_ioend+0xd8/0x140 [xfs]
> Apr 05 11:51:27 kernel:  xfs_end_io+0xb8/0xf0 [xfs]
> Apr 05 11:51:27 kernel:  process_one_work+0x1b6/0x350
> Apr 05 11:51:27 kernel:  rescuer_thread+0x1d1/0x3a0
> Apr 05 11:51:27 kernel:  ? worker_thread+0x3e0/0x3e0
> Apr 05 11:51:27 kernel:  kthread+0x11b/0x140
> Apr 05 11:51:27 kernel:  ? kthread_associate_blkcg+0xb0/0xb0
> Apr 05 11:51:27 kernel:  ret_from_fork+0x22/0x30
>
> Which seems to be similar to this:
> https://bugs.launchpad.net/bugs/1996269
>
> I followed their patchset here:
> https://review.opendev.org/c/starlingx/kernel/+/864257
>
> And I was under the impression that I picked the right one to ask for backporting, but it seems that was incorrect. I went through the list again and I think the following patches are the ones missing from 5.10:
>
> 8182ec00803085354761bbadf0287cad7eac0e2f - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0035-xfs-drop-submit-side-trans-alloc-for-append-ioends.patch
> edbf1eb9032b84631031d9b43570e262f3461c24 - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0036-xfs-open-code-ioend-needs-workqueue-helper.patch
> 170e31793806ce5e5a9647b6340954536244518e - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0037-xfs-drop-unused-ioend-private-merge-and-setfilesize-.patch
> 2fd609b6c90a88630a50fb317473b210759b3873 - https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0038-xfs-drop-unnecessary-setfilesize-helper.patch
>

The only commit that fixes the bug is:
7cd3099f4925 xfs: drop submit side trans alloc for append ioends

The rest are just code cleanups.

That fix was missed in my original backports from v5.13 because of a tool error,
so thank you for pointing it out.

I have added it to my test branch and will follow up with posting to
stable later on.

Chandan,

Please make sure you include this fix when you get to considering
fixes from v5.13 to 5.4.y.

I will wait with posting this fix to 5.10.y until I get the v5.13
backports wish list from you.

Thanks,
Amir.

Re: Backport of "xfs: open code ioend needs workqueue helper" to 5.10?

[Chandan Babu R]

On Thu, Apr 13, 2023 at 10:44:43 AM +0300, Amir Goldstein wrote:
> On Wed, Apr 12, 2023 at 6:58 PM Christian Theune <ct@flyingcircus.io> wrote:
>>
>> Hi,
>>
>> ugh. Sorry, looks like I jumped the gun. Mea culpa.
>>
>> We experienced a hang like this:
>>
>> Apr 05 11:51:27 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Apr 05 11:51:27 kernel: task:xfs-conv/vdc1   state:D stack:    0 pid:  606 ppid:     2 flags:0x00004080
>> Apr 05 11:51:27 kernel: Workqueue: xfs-conv/vdc1 xfs_end_io [xfs]
>> Apr 05 11:51:27 kernel: Call Trace:
>> Apr 05 11:51:27 kernel:  __schedule+0x274/0x870
>> Apr 05 11:51:27 kernel:  schedule+0x46/0xb0
>> Apr 05 11:51:27 kernel:  xlog_grant_head_wait+0xc5/0x1d0 [xfs]
>> Apr 05 11:51:27 kernel:  xlog_grant_head_check+0xde/0x100 [xfs]
>> Apr 05 11:51:27 kernel:  xfs_log_reserve+0xbe/0x1b0 [xfs]
>> Apr 05 11:51:27 kernel:  xfs_trans_reserve+0x143/0x180 [xfs]
>> Apr 05 11:51:27 kernel:  xfs_trans_alloc+0xee/0x1a0 [xfs]
>> Apr 05 11:51:27 kernel:  xfs_iomap_write_unwritten+0x120/0x2e0 [xfs]
>> Apr 05 11:51:27 kernel:  ? record_times+0x15/0x90
>> Apr 05 11:51:27 kernel:  xfs_end_ioend+0xd8/0x140 [xfs]
>> Apr 05 11:51:27 kernel:  xfs_end_io+0xb8/0xf0 [xfs]
>> Apr 05 11:51:27 kernel:  process_one_work+0x1b6/0x350
>> Apr 05 11:51:27 kernel:  rescuer_thread+0x1d1/0x3a0
>> Apr 05 11:51:27 kernel:  ? worker_thread+0x3e0/0x3e0
>> Apr 05 11:51:27 kernel:  kthread+0x11b/0x140
>> Apr 05 11:51:27 kernel:  ? kthread_associate_blkcg+0xb0/0xb0
>> Apr 05 11:51:27 kernel:  ret_from_fork+0x22/0x30
>>
>> Which seems to be similar to this:
>> https://bugs.launchpad.net/bugs/1996269
>>
>> I followed their patchset here:
>> https://review.opendev.org/c/starlingx/kernel/+/864257
>>
>> And I was under the impression that I picked the right one to ask
>> for backporting, but it seems that was incorrect. I went through the
>> list again and I think the following patches are the ones missing
>> from 5.10:
>>
>> 8182ec00803085354761bbadf0287cad7eac0e2f -
>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0035-xfs-drop-submit-side-trans-alloc-for-append-ioends.patch
>> edbf1eb9032b84631031d9b43570e262f3461c24 -
>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0036-xfs-open-code-ioend-needs-workqueue-helper.patch
>> 170e31793806ce5e5a9647b6340954536244518e -
>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0037-xfs-drop-unused-ioend-private-merge-and-setfilesize-.patch
>> 2fd609b6c90a88630a50fb317473b210759b3873 -
>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0038-xfs-drop-unnecessary-setfilesize-helper.patch
>>
>
> The only commit that fixes the bug is:
> 7cd3099f4925 xfs: drop submit side trans alloc for append ioends
>
> The rest are just code cleanups.
>
> That fix was missed in my original backports from v5.13 because of a tool error,
> so thank you for pointing it out.
>
> I have added it to my test branch and will follow up with posting to
> stable later on.
>
> Chandan,
>
> Please make sure you include this fix when you get to considering
> fixes from v5.13 to 5.4.y.
>

Sure, I will do that. However ...

> I will wait with posting this fix to 5.10.y until I get the v5.13
> backports wish list from you.
>

Since I am working on another XFS work item there will be some delay before I
share the list of patches to be backported from v5.13 to 5.4.y.

-- 
chandan

Re: Backport of "xfs: open code ioend needs workqueue helper" to 5.10?

[Christian Theune]

Hi Amir and Chandan,

much appreciated! I’m building up a history of being a truffle pig WRT forgotten fixes. :)

Christian

> On 13. Apr 2023, at 12:31, Chandan Babu R <chandan.babu@oracle.com> wrote:
> 
> On Thu, Apr 13, 2023 at 10:44:43 AM +0300, Amir Goldstein wrote:
>> On Wed, Apr 12, 2023 at 6:58 PM Christian Theune <ct@flyingcircus.io> wrote:
>>> 
>>> Hi,
>>> 
>>> ugh. Sorry, looks like I jumped the gun. Mea culpa.
>>> 
>>> We experienced a hang like this:
>>> 
>>> Apr 05 11:51:27 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Apr 05 11:51:27 kernel: task:xfs-conv/vdc1   state:D stack:    0 pid:  606 ppid:     2 flags:0x00004080
>>> Apr 05 11:51:27 kernel: Workqueue: xfs-conv/vdc1 xfs_end_io [xfs]
>>> Apr 05 11:51:27 kernel: Call Trace:
>>> Apr 05 11:51:27 kernel:  __schedule+0x274/0x870
>>> Apr 05 11:51:27 kernel:  schedule+0x46/0xb0
>>> Apr 05 11:51:27 kernel:  xlog_grant_head_wait+0xc5/0x1d0 [xfs]
>>> Apr 05 11:51:27 kernel:  xlog_grant_head_check+0xde/0x100 [xfs]
>>> Apr 05 11:51:27 kernel:  xfs_log_reserve+0xbe/0x1b0 [xfs]
>>> Apr 05 11:51:27 kernel:  xfs_trans_reserve+0x143/0x180 [xfs]
>>> Apr 05 11:51:27 kernel:  xfs_trans_alloc+0xee/0x1a0 [xfs]
>>> Apr 05 11:51:27 kernel:  xfs_iomap_write_unwritten+0x120/0x2e0 [xfs]
>>> Apr 05 11:51:27 kernel:  ? record_times+0x15/0x90
>>> Apr 05 11:51:27 kernel:  xfs_end_ioend+0xd8/0x140 [xfs]
>>> Apr 05 11:51:27 kernel:  xfs_end_io+0xb8/0xf0 [xfs]
>>> Apr 05 11:51:27 kernel:  process_one_work+0x1b6/0x350
>>> Apr 05 11:51:27 kernel:  rescuer_thread+0x1d1/0x3a0
>>> Apr 05 11:51:27 kernel:  ? worker_thread+0x3e0/0x3e0
>>> Apr 05 11:51:27 kernel:  kthread+0x11b/0x140
>>> Apr 05 11:51:27 kernel:  ? kthread_associate_blkcg+0xb0/0xb0
>>> Apr 05 11:51:27 kernel:  ret_from_fork+0x22/0x30
>>> 
>>> Which seems to be similar to this:
>>> https://bugs.launchpad.net/bugs/1996269
>>> 
>>> I followed their patchset here:
>>> https://review.opendev.org/c/starlingx/kernel/+/864257
>>> 
>>> And I was under the impression that I picked the right one to ask
>>> for backporting, but it seems that was incorrect. I went through the
>>> list again and I think the following patches are the ones missing
>>> from 5.10:
>>> 
>>> 8182ec00803085354761bbadf0287cad7eac0e2f -
>>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0035-xfs-drop-submit-side-trans-alloc-for-append-ioends.patch
>>> edbf1eb9032b84631031d9b43570e262f3461c24 -
>>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0036-xfs-open-code-ioend-needs-workqueue-helper.patch
>>> 170e31793806ce5e5a9647b6340954536244518e -
>>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0037-xfs-drop-unused-ioend-private-merge-and-setfilesize-.patch
>>> 2fd609b6c90a88630a50fb317473b210759b3873 -
>>> https://review.opendev.org/c/starlingx/kernel/+/864257/5/kernel-std/centos/patches/0038-xfs-drop-unnecessary-setfilesize-helper.patch
>>> 
>> 
>> The only commit that fixes the bug is:
>> 7cd3099f4925 xfs: drop submit side trans alloc for append ioends
>> 
>> The rest are just code cleanups.
>> 
>> That fix was missed in my original backports from v5.13 because of a tool error,
>> so thank you for pointing it out.
>> 
>> I have added it to my test branch and will follow up with posting to
>> stable later on.
>> 
>> Chandan,
>> 
>> Please make sure you include this fix when you get to considering
>> fixes from v5.13 to 5.4.y.
>> 
> 
> Sure, I will do that. However ...
> 
>> I will wait with posting this fix to 5.10.y until I get the v5.13
>> backports wish list from you.
>> 
> 
> Since I am working on another XFS work item there will be some delay before I
> share the list of patches to be backported from v5.13 to 5.4.y.
> 
> -- 
> chandan


Liebe Grüße,
Christian Theune

-- 
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick