From: David Howells <dhowells@redhat.com>
To: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
Cc: dhowells@redhat.com, "idryomov@gmail.com" <idryomov@gmail.com>,
Alex Markuze <amarkuze@redhat.com>,
"slava@dubeyko.com" <slava@dubeyko.com>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: [PATCH v2] ceph: Fix kernel crash in generic/397 test
Date: Tue, 28 Jan 2025 20:01:09 +0000 [thread overview]
Message-ID: <3532744.1738094469@warthog.procyon.org.uk> (raw)
In-Reply-To: <3469649.1738083455@warthog.procyon.org.uk>
I added some tracing to fs/ceph/addr.c and this highlights the bug causing the
hang that I'm seeing.
So what I see is ceph_writepages_start() being entered and getting a
collection of folios from filemap_get_folios_tag():
netfs_ceph_writepages: i=10000004f52 ix=0
netfs_ceph_wp_get_folios: i=10000004f52 oix=0 ix=8000000000000 nr=6
Then we get out the first dirty folio from the batch and attempt to lock it:
netfs_folio: i=10000004f52 ix=00003-00003 ceph-wb-lock
which succeeds. We then pass through a number of lines:
netfs_ceph_wp_track: i=10000004f52 line=1218
which is the "/* shift unused page to beginning of fbatch */" comment, then:
netfs_ceph_wp_track: i=10000004f52 line=1238
which is followed by "offset = ceph_fscrypt_page_offset(pages[0]);", then:
netfs_ceph_wp_track: i=10000004f52 line=1264
which is the error handling path of:
if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
rc = -EIO;
goto release_folios;
}
and then:
netfs_ceph_wp_track: i=10000004f52 line=1389
which is "release_folios:".
We then reenter ceph_writepages_start(), get the same batch of dirty folios
and try to lock them again:
netfs_ceph_writepages: i=10000004f52 ix=0
netfs_ceph_wp_get_folios: i=10000004f52 oix=0 ix=8000000000000 nr=6
netfs_folio: i=10000004f52 ix=00003-00003 ceph-wb-lock
and that's where we hang.
I think the problem is that the error handling here:
if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
rc = -EIO;
goto release_folios;
}
is insufficient. The folios are locked and can't just be released.
Why ceph_inc_osd_stopping_blocker() fails is also something that needs looking
at.
David
next prev parent reply other threads:[~2025-01-28 20:01 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-01-17 3:50 [PATCH v2] ceph: Fix kernel crash in generic/397 test Viacheslav Dubeyko
2025-01-17 12:06 ` Ilya Dryomov
2025-01-17 19:18 ` Viacheslav Dubeyko
2025-01-17 21:32 ` David Howells
2025-01-17 22:08 ` Viacheslav Dubeyko
2025-01-20 9:33 ` David Howells
2025-01-20 9:47 ` Alex Markuze
2025-01-27 18:40 ` Viacheslav Dubeyko
2025-01-28 16:13 ` David Howells
2025-01-28 16:57 ` David Howells
2025-01-28 20:01 ` David Howells [this message]
2025-01-28 20:16 ` Viacheslav Dubeyko
2025-01-28 22:34 ` David Howells
2025-01-28 22:37 ` Viacheslav Dubeyko
2025-01-29 10:39 ` Alex Markuze
2025-01-29 13:42 ` David Howells
2025-01-29 13:54 ` Alex Markuze
2025-01-29 13:41 ` David Howells
2025-01-29 19:16 ` Viacheslav Dubeyko
2025-02-14 20:29 ` David Howells
2025-02-14 20:54 ` Viacheslav Dubeyko
2025-04-15 17:59 ` Viacheslav Dubeyko
2025-06-01 16:23 ` Ilya Dryomov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3532744.1738094469@warthog.procyon.org.uk \
--to=dhowells@redhat.com \
--cc=Slava.Dubeyko@ibm.com \
--cc=amarkuze@redhat.com \
--cc=ceph-devel@vger.kernel.org \
--cc=idryomov@gmail.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=slava@dubeyko.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).