jbd2: fix deadlock while checkpoint thread waits commit thread to finish (backport to 4.14)

jbd2: fix deadlock while checkpoint thread waits commit thread to finish (backport to 4.14)

[Ivan Zahariev]

Hello,

We're running Linux kernel 4.14.x and our systems occasionally suffer a 
bug which is already fixed: 
https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e

This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch 
applies cleanly. The two files "fs/jbd2/checkpoint.c" and 
"fs/jbd2/journal.c" seem pretty identical in the affected sections 
compared to kernel 5.4 where we have this bugfix already applied.

Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe 
that we backport it manually in our kernel 4.14 builds? Or is the "ext4" 
system in 4.14 and 5.4 fundamentally different and this would lead to 
data loss or other problems?

Thank you.

Best regards.
--Ivan

Re: jbd2: fix deadlock while checkpoint thread waits commit thread to finish (backport to 4.14)

[Theodore Ts'o]

On Wed, Jul 07, 2021 at 09:42:25PM +0300, Ivan Zahariev wrote:
> Hello,
> 
> We're running Linux kernel 4.14.x and our systems occasionally suffer a bug
> which is already fixed: https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
> 
> This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
> applies cleanly. The two files "fs/jbd2/checkpoint.c" and
> "fs/jbd2/journal.c" seem pretty identical in the affected sections compared
> to kernel 5.4 where we have this bugfix already applied.
> 
> Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
> that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
> system in 4.14 and 5.4 fundamentally different and this would lead to data
> loss or other problems?

The commit was over two years ago, so my memory is not going to be
perfect.  However, Jan had made a comment suggesting the approach in
this commit because it should be easier to backport into older stble
kernels[1].

   "Since proper locking change is going to be a bit more involved, can you
    perhaps fix this deadlock by just dropping j_checkpoint_mutex in
    log_do_checkpoint() when we are going to wait for transaction commit. I've
    checked and that should be fine and that is going to be much easier change
    to backport into stable kernels..."

[1] https://marc.info/?l=linux-ext4&m=154212553014669&w=2

So I suspect it was just that I failed to remember to add a "Cc:
stable@kernel.org" and so it was never automatically backported into
4.14 or 4.19.

Do you have a reliable reproduction which is triggering the deadlock
on your kernels?  If so, have you tried applying the patch and does it
make the problem go away for you?

Cheers,

						- Ted

Re: jbd2: fix deadlock while checkpoint thread waits commit thread to finish (backport to 4.14)

[Ivan Zahariev]

Out of thousand machines, one would trigger the problem about every 1 to 
10 days. Some machines trigger the problem much often than others. So I 
can say that we have a way to verify quickly if applying the patch will 
fix this for us.

The most important question is: Is it safe to apply the patch on 
production machines with kernel 4.14?

We can't risk data loss. And I lack the expertise to asses what risks 
this small patch brings.

Best regards.
--Ivan

On 8.7.2021 г. 2:52, Theodore Ts'o wrote:
> On Wed, Jul 07, 2021 at 09:42:25PM +0300, Ivan Zahariev wrote:
>> Hello,
>>
>> We're running Linux kernel 4.14.x and our systems occasionally suffer a bug
>> which is already fixed: https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
>>
>> This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
>> applies cleanly. The two files "fs/jbd2/checkpoint.c" and
>> "fs/jbd2/journal.c" seem pretty identical in the affected sections compared
>> to kernel 5.4 where we have this bugfix already applied.
>>
>> Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
>> that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
>> system in 4.14 and 5.4 fundamentally different and this would lead to data
>> loss or other problems?
> The commit was over two years ago, so my memory is not going to be
> perfect.  However, Jan had made a comment suggesting the approach in
> this commit because it should be easier to backport into older stble
> kernels[1].
>
>     "Since proper locking change is going to be a bit more involved, can you
>      perhaps fix this deadlock by just dropping j_checkpoint_mutex in
>      log_do_checkpoint() when we are going to wait for transaction commit. I've
>      checked and that should be fine and that is going to be much easier change
>      to backport into stable kernels..."
>
> [1] https://marc.info/?l=linux-ext4&m=154212553014669&w=2
>
> So I suspect it was just that I failed to remember to add a "Cc:
> stable@kernel.org" and so it was never automatically backported into
> 4.14 or 4.19.
>
> Do you have a reliable reproduction which is triggering the deadlock
> on your kernels?  If so, have you tried applying the patch and does it
> make the problem go away for you?
>
> Cheers,
>
> 						- Ted

Re: jbd2: fix deadlock while checkpoint thread waits commit thread to finish (backport to 4.14)

[Jan Kara]

On Thu 08-07-21 06:45:35, Ivan Zahariev wrote:
> Out of thousand machines, one would trigger the problem about every 1 to 10
> days. Some machines trigger the problem much often than others. So I can say
> that we have a way to verify quickly if applying the patch will fix this for
> us.
> 
> The most important question is: Is it safe to apply the patch on production
> machines with kernel 4.14?
> 
> We can't risk data loss. And I lack the expertise to asses what risks this
> small patch brings.

The fix should work correctly even for older kernels. I'm not aware of any
changes in this area in the past that could conflict...

								Honza

> On 8.7.2021 г. 2:52, Theodore Ts'o wrote:
> > On Wed, Jul 07, 2021 at 09:42:25PM +0300, Ivan Zahariev wrote:
> > > Hello,
> > > 
> > > We're running Linux kernel 4.14.x and our systems occasionally suffer a bug
> > > which is already fixed: https://github.com/torvalds/linux/commit/53cf978457325d8fb2cdecd7981b31a8229e446e
> > > 
> > > This bugfix hasn't been ported to Linux kernels 4.14 or 4.19. The patch
> > > applies cleanly. The two files "fs/jbd2/checkpoint.c" and
> > > "fs/jbd2/journal.c" seem pretty identical in the affected sections compared
> > > to kernel 5.4 where we have this bugfix already applied.
> > > 
> > > Is it on purpose that this bugfix hasn't been ported to 4.14? Is it safe
> > > that we backport it manually in our kernel 4.14 builds? Or is the "ext4"
> > > system in 4.14 and 5.4 fundamentally different and this would lead to data
> > > loss or other problems?
> > The commit was over two years ago, so my memory is not going to be
> > perfect.  However, Jan had made a comment suggesting the approach in
> > this commit because it should be easier to backport into older stble
> > kernels[1].
> > 
> >     "Since proper locking change is going to be a bit more involved, can you
> >      perhaps fix this deadlock by just dropping j_checkpoint_mutex in
> >      log_do_checkpoint() when we are going to wait for transaction commit. I've
> >      checked and that should be fine and that is going to be much easier change
> >      to backport into stable kernels..."
> > 
> > [1] https://marc.info/?l=linux-ext4&m=154212553014669&w=2
> > 
> > So I suspect it was just that I failed to remember to add a "Cc:
> > stable@kernel.org" and so it was never automatically backported into
> > 4.14 or 4.19.
> > 
> > Do you have a reliable reproduction which is triggering the deadlock
> > on your kernels?  If so, have you tried applying the patch and does it
> > make the problem go away for you?
> > 
> > Cheers,
> > 
> > 						- Ted
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR