* [PATCH -next] jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction
@ 2022-03-17 1:27 Ye Bin
2022-03-17 10:00 ` Jan Kara
0 siblings, 1 reply; 3+ messages in thread
From: Ye Bin @ 2022-03-17 1:27 UTC (permalink / raw)
To: tytso, adilger.kernel, linux-ext4; +Cc: linux-kernel, jack, lczerner, Ye Bin
we got issue as follows:
[ 72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal
[ 72.826847] EXT4-fs (sda): Remounting filesystem read-only
fallocate: fallocate failed: Read-only file system
[ 74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay
[ 74.793597] ------------[ cut here ]------------
[ 74.794203] kernel BUG at fs/jbd2/transaction.c:2063!
[ 74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[ 74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150
[ 74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60
[ 74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202
[ 74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90
[ 74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20
[ 74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90
[ 74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00
[ 74.807301] FS: 0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000
[ 74.808338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0
[ 74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 74.811897] Call Trace:
[ 74.812241] <TASK>
[ 74.812566] __jbd2_journal_refile_buffer+0x12f/0x180
[ 74.813246] jbd2_journal_refile_buffer+0x4c/0xa0
[ 74.813869] jbd2_journal_commit_transaction.cold+0xa1/0x148
[ 74.817550] kjournald2+0xf8/0x3e0
[ 74.819056] kthread+0x153/0x1c0
[ 74.819963] ret_from_fork+0x22/0x30
Above issue may happen as follows:
write truncate kjournald2
generic_perform_write
ext4_write_begin
ext4_walk_page_buffers
do_journal_get_write_access ->add BJ_Reserved list
ext4_journalled_write_end
ext4_walk_page_buffers
write_end_fn
ext4_handle_dirty_metadata
***************JBD2 ABORT**************
jbd2_journal_dirty_metadata
-> return -EROFS, jh in reserved_list
jbd2_journal_commit_transaction
while (commit_transaction->t_reserved_list)
jh = commit_transaction->t_reserved_list;
truncate_pagecache_range
do_invalidatepage
ext4_journalled_invalidatepage
jbd2_journal_invalidatepage
journal_unmap_buffer
__dispose_buffer
__jbd2_journal_unfile_buffer
jbd2_journal_put_journal_head ->put last ref_count
__journal_remove_journal_head
bh->b_private = NULL;
jh->b_bh = NULL;
jbd2_journal_refile_buffer(journal, jh);
bh = jh2bh(jh);
->bh is NULL, later will trigger null-ptr-deref
journal_free_journal_head(jh);
As after 96f1e0974575 commit, handle reserved list will not hold "journal->j_state_lock"
when kjournald2 commit transaction. So journal_unmap_buffer maybe free
journal_head when handle reserved list. And lead to null-ptr-deref or some
strange errors.
As reserved list almost time is empty. Use "journal->j_state_lock" to protect
handle reserved list can simply solve above issue.
Fixes: 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction")
Signed-off-by: Ye Bin <yebin10@huawei.com>
---
fs/jbd2/commit.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
index 5b9408e3b370..2b737b928d26 100644
--- a/fs/jbd2/commit.c
+++ b/fs/jbd2/commit.c
@@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal)
jbd2_journal_wait_updates(journal);
commit_transaction->t_state = T_SWITCH;
- write_unlock(&journal->j_state_lock);
J_ASSERT (atomic_read(&commit_transaction->t_outstanding_credits) <=
journal->j_max_transaction_buffers);
@@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
jbd2_journal_refile_buffer(journal, jh);
}
+ write_unlock(&journal->j_state_lock);
/*
* Now try to drop any written-back buffers from the journal's
* checkpoint lists. We do this *before* commit because it potentially
--
2.31.1
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [PATCH -next] jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction 2022-03-17 1:27 [PATCH -next] jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction Ye Bin @ 2022-03-17 10:00 ` Jan Kara [not found] ` <62331D5C.2010808@huawei.com> 0 siblings, 1 reply; 3+ messages in thread From: Jan Kara @ 2022-03-17 10:00 UTC (permalink / raw) To: Ye Bin; +Cc: tytso, adilger.kernel, linux-ext4, linux-kernel, jack, lczerner On Thu 17-03-22 09:27:55, Ye Bin wrote: > we got issue as follows: > [ 72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal > [ 72.826847] EXT4-fs (sda): Remounting filesystem read-only > fallocate: fallocate failed: Read-only file system > [ 74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay > [ 74.793597] ------------[ cut here ]------------ > [ 74.794203] kernel BUG at fs/jbd2/transaction.c:2063! > [ 74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI > [ 74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150 > [ 74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60 > [ 74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202 > [ 74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 > [ 74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90 > [ 74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20 > [ 74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90 > [ 74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00 > [ 74.807301] FS: 0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000 > [ 74.808338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0 > [ 74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > [ 74.811897] Call Trace: > [ 74.812241] <TASK> > [ 74.812566] __jbd2_journal_refile_buffer+0x12f/0x180 > [ 74.813246] jbd2_journal_refile_buffer+0x4c/0xa0 > [ 74.813869] jbd2_journal_commit_transaction.cold+0xa1/0x148 > [ 74.817550] kjournald2+0xf8/0x3e0 > [ 74.819056] kthread+0x153/0x1c0 > [ 74.819963] ret_from_fork+0x22/0x30 > > Above issue may happen as follows: > write truncate kjournald2 > generic_perform_write > ext4_write_begin > ext4_walk_page_buffers > do_journal_get_write_access ->add BJ_Reserved list > ext4_journalled_write_end > ext4_walk_page_buffers > write_end_fn > ext4_handle_dirty_metadata > ***************JBD2 ABORT************** > jbd2_journal_dirty_metadata > -> return -EROFS, jh in reserved_list > jbd2_journal_commit_transaction > while (commit_transaction->t_reserved_list) > jh = commit_transaction->t_reserved_list; > truncate_pagecache_range > do_invalidatepage > ext4_journalled_invalidatepage > jbd2_journal_invalidatepage > journal_unmap_buffer > __dispose_buffer > __jbd2_journal_unfile_buffer > jbd2_journal_put_journal_head ->put last ref_count > __journal_remove_journal_head > bh->b_private = NULL; > jh->b_bh = NULL; > jbd2_journal_refile_buffer(journal, jh); > bh = jh2bh(jh); > ->bh is NULL, later will trigger null-ptr-deref > journal_free_journal_head(jh); > > As after 96f1e0974575 commit, handle reserved list will not hold "journal->j_state_lock" > when kjournald2 commit transaction. So journal_unmap_buffer maybe free > journal_head when handle reserved list. And lead to null-ptr-deref or some > strange errors. > As reserved list almost time is empty. Use "journal->j_state_lock" to protect > handle reserved list can simply solve above issue. > > Fixes: 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction") > Signed-off-by: Ye Bin <yebin10@huawei.com> Good spotting! Thanks for the analysis and the patch! > diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c > index 5b9408e3b370..2b737b928d26 100644 > --- a/fs/jbd2/commit.c > +++ b/fs/jbd2/commit.c > @@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal) > jbd2_journal_wait_updates(journal); > > commit_transaction->t_state = T_SWITCH; > - write_unlock(&journal->j_state_lock); > > J_ASSERT (atomic_read(&commit_transaction->t_outstanding_credits) <= > journal->j_max_transaction_buffers); > @@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal) > jbd2_journal_refile_buffer(journal, jh); > } > > + write_unlock(&journal->j_state_lock); > /* > * Now try to drop any written-back buffers from the journal's > * checkpoint lists. We do this *before* commit because it potentially Honestly, using j_state_lock seems like unnecessarily big hammer for this. I'd rather use journal->j_list_lock for this which is much more natural for list processing. And we grab it anyway from jbd2_journal_refile_buffer() so it is not like there will be any additional overhead anyway. Honza -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 3+ messages in thread
[parent not found: <62331D5C.2010808@huawei.com>]
* Re: [PATCH -next] jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction [not found] ` <62331D5C.2010808@huawei.com> @ 2022-03-17 13:30 ` Jan Kara 0 siblings, 0 replies; 3+ messages in thread From: Jan Kara @ 2022-03-17 13:30 UTC (permalink / raw) To: yebin; +Cc: Jan Kara, tytso, adilger.kernel, linux-ext4, linux-kernel, lczerner On Thu 17-03-22 19:37:00, yebin wrote: > > > On 2022/3/17 18:00, Jan Kara wrote: > > On Thu 17-03-22 09:27:55, Ye Bin wrote: > > > we got issue as follows: > > > [ 72.796117] EXT4-fs error (device sda): ext4_journal_check_start:83: comm fallocate: Detected aborted journal > > > [ 72.826847] EXT4-fs (sda): Remounting filesystem read-only > > > fallocate: fallocate failed: Read-only file system > > > [ 74.791830] jbd2_journal_commit_transaction: jh=0xffff9cfefe725d90 bh=0x0000000000000000 end delay > > > [ 74.793597] ------------[ cut here ]------------ > > > [ 74.794203] kernel BUG at fs/jbd2/transaction.c:2063! > > > [ 74.794886] invalid opcode: 0000 [#1] PREEMPT SMP PTI > > > [ 74.795533] CPU: 4 PID: 2260 Comm: jbd2/sda-8 Not tainted 5.17.0-rc8-next-20220315-dirty #150 > > > [ 74.798327] RIP: 0010:__jbd2_journal_unfile_buffer+0x3e/0x60 > > > [ 74.801971] RSP: 0018:ffffa828c24a3cb8 EFLAGS: 00010202 > > > [ 74.802694] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 > > > [ 74.803601] RDX: 0000000000000001 RSI: ffff9cfefe725d90 RDI: ffff9cfefe725d90 > > > [ 74.804554] RBP: ffff9cfefe725d90 R08: 0000000000000000 R09: ffffa828c24a3b20 > > > [ 74.805471] R10: 0000000000000001 R11: 0000000000000001 R12: ffff9cfefe725d90 > > > [ 74.806385] R13: ffff9cfefe725d98 R14: 0000000000000000 R15: ffff9cfe833a4d00 > > > [ 74.807301] FS: 0000000000000000(0000) GS:ffff9d01afb00000(0000) knlGS:0000000000000000 > > > [ 74.808338] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > > [ 74.809084] CR2: 00007f2b81bf4000 CR3: 0000000100056000 CR4: 00000000000006e0 > > > [ 74.810047] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > [ 74.810981] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 > > > [ 74.811897] Call Trace: > > > [ 74.812241] <TASK> > > > [ 74.812566] __jbd2_journal_refile_buffer+0x12f/0x180 > > > [ 74.813246] jbd2_journal_refile_buffer+0x4c/0xa0 > > > [ 74.813869] jbd2_journal_commit_transaction.cold+0xa1/0x148 > > > [ 74.817550] kjournald2+0xf8/0x3e0 > > > [ 74.819056] kthread+0x153/0x1c0 > > > [ 74.819963] ret_from_fork+0x22/0x30 > > > > > > Above issue may happen as follows: > > > write truncate kjournald2 > > > generic_perform_write > > > ext4_write_begin > > > ext4_walk_page_buffers > > > do_journal_get_write_access ->add BJ_Reserved list > > > ext4_journalled_write_end > > > ext4_walk_page_buffers > > > write_end_fn > > > ext4_handle_dirty_metadata > > > ***************JBD2 ABORT************** > > > jbd2_journal_dirty_metadata > > > -> return -EROFS, jh in reserved_list > > > jbd2_journal_commit_transaction > > > while (commit_transaction->t_reserved_list) > > > jh = commit_transaction->t_reserved_list; > > > truncate_pagecache_range > > > do_invalidatepage > > > ext4_journalled_invalidatepage > > > jbd2_journal_invalidatepage > > > journal_unmap_buffer > > > __dispose_buffer > > > __jbd2_journal_unfile_buffer > > > jbd2_journal_put_journal_head ->put last ref_count > > > __journal_remove_journal_head > > > bh->b_private = NULL; > > > jh->b_bh = NULL; > > > jbd2_journal_refile_buffer(journal, jh); > > > bh = jh2bh(jh); > > > ->bh is NULL, later will trigger null-ptr-deref > > > journal_free_journal_head(jh); > > > > > > As after 96f1e0974575 commit, handle reserved list will not hold "journal->j_state_lock" > > > when kjournald2 commit transaction. So journal_unmap_buffer maybe free > > > journal_head when handle reserved list. And lead to null-ptr-deref or some > > > strange errors. > > > As reserved list almost time is empty. Use "journal->j_state_lock" to protect > > > handle reserved list can simply solve above issue. > > > > > > Fixes: 96f1e0974575("jbd2: avoid long hold times of j_state_lock while committing a transaction") > > > Signed-off-by: Ye Bin <yebin10@huawei.com> > > Good spotting! Thanks for the analysis and the patch! > > > > > diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c > > > index 5b9408e3b370..2b737b928d26 100644 > > > --- a/fs/jbd2/commit.c > > > +++ b/fs/jbd2/commit.c > > > @@ -488,7 +488,6 @@ void jbd2_journal_commit_transaction(journal_t *journal) > > > jbd2_journal_wait_updates(journal); > > > commit_transaction->t_state = T_SWITCH; > > > - write_unlock(&journal->j_state_lock); > > > J_ASSERT (atomic_read(&commit_transaction->t_outstanding_credits) <= > > > journal->j_max_transaction_buffers); > > > @@ -527,6 +526,7 @@ void jbd2_journal_commit_transaction(journal_t *journal) > > > jbd2_journal_refile_buffer(journal, jh); > > > } > > > + write_unlock(&journal->j_state_lock); > > > /* > > > * Now try to drop any written-back buffers from the journal's > > > * checkpoint lists. We do this *before* commit because it potentially > > Honestly, using j_state_lock seems like unnecessarily big hammer for this. > > I'd rather use journal->j_list_lock for this which is much more natural for > > list processing. And we grab it anyway from jbd2_journal_refile_buffer() so > > it is not like there will be any additional overhead anyway. > > > > Honza > If use journal->j_list_lock for reserved list, we must change order of > jh->b_state_lock and > journal->j_list_lock. > <../cgi-bin/global.cgi?pattern=j_list_lock&type=symbol> > I have another idea as follows. Do you think it is feasible? Oh right, I forgot the lock ordering is not very convenient for these two locks. Ok, then I probably prefer your original solution to the one below. But please add a comment like: We use journal->j_state_lock here to serialize processing of t_reserved_list with eviction of buffers from journal_unmap_buffer(). Thanks! Honza > diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c > index 73ed02f061e1..00c51a92a5cc 100644 > --- a/fs/jbd2/transaction.c > +++ b/fs/jbd2/transaction.c > @@ -2282,6 +2282,7 @@ static int journal_unmap_buffer(journal_t *journal, > struct buffer_head *bh, > if (!jh) > goto zap_buffer_unlocked; > > +repeat: > /* OK, we have data buffer in journaled mode */ > write_lock(&journal->j_state_lock); > spin_lock(&jh->b_state_lock); > @@ -2357,6 +2358,11 @@ static int journal_unmap_buffer(journal_t *journal, > struct buffer_head *bh, > goto zap_buffer; > } > } > + } else if (transaction->t_state == T_SWITCH) { > + spin_unlock(&journal->j_list_lock); > + spin_unlock(&jh->b_state_lock); > + wait_transaction_switching(journal); > + goto repeat; > } else if (transaction == journal->j_committing_transaction) { > JBUFFER_TRACE(jh, "on committing transaction"); > /* > -- > > -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-03-17 13:30 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-03-17 1:27 [PATCH -next] jbd2: Fix null-ptr-deref when process reserved list in jbd2_journal_commit_transaction Ye Bin
2022-03-17 10:00 ` Jan Kara
[not found] ` <62331D5C.2010808@huawei.com>
2022-03-17 13:30 ` Jan Kara
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox