From: David Howells <dhowells@redhat.com>
To: Viacheslav Dubeyko <Slava.Dubeyko@ibm.com>
Cc: dhowells@redhat.com, "idryomov@gmail.com" <idryomov@gmail.com>,
Alex Markuze <amarkuze@redhat.com>,
"slava@dubeyko.com" <slava@dubeyko.com>,
"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
"ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: [PATCH v2] ceph: Fix kernel crash in generic/397 test
Date: Tue, 28 Jan 2025 20:01:09 +0000 [thread overview]
Message-ID: <3532744.1738094469@warthog.procyon.org.uk> (raw)
In-Reply-To: <3469649.1738083455@warthog.procyon.org.uk>
I added some tracing to fs/ceph/addr.c and this highlights the bug causing the
hang that I'm seeing.
So what I see is ceph_writepages_start() being entered and getting a
collection of folios from filemap_get_folios_tag():
netfs_ceph_writepages: i=10000004f52 ix=0
netfs_ceph_wp_get_folios: i=10000004f52 oix=0 ix=8000000000000 nr=6
Then we get out the first dirty folio from the batch and attempt to lock it:
netfs_folio: i=10000004f52 ix=00003-00003 ceph-wb-lock
which succeeds. We then pass through a number of lines:
netfs_ceph_wp_track: i=10000004f52 line=1218
which is the "/* shift unused page to beginning of fbatch */" comment, then:
netfs_ceph_wp_track: i=10000004f52 line=1238
which is followed by "offset = ceph_fscrypt_page_offset(pages[0]);", then:
netfs_ceph_wp_track: i=10000004f52 line=1264
which is the error handling path of:
if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
rc = -EIO;
goto release_folios;
}
and then:
netfs_ceph_wp_track: i=10000004f52 line=1389
which is "release_folios:".
We then reenter ceph_writepages_start(), get the same batch of dirty folios
and try to lock them again:
netfs_ceph_writepages: i=10000004f52 ix=0
netfs_ceph_wp_get_folios: i=10000004f52 oix=0 ix=8000000000000 nr=6
netfs_folio: i=10000004f52 ix=00003-00003 ceph-wb-lock
and that's where we hang.
I think the problem is that the error handling here:
if (!ceph_inc_osd_stopping_blocker(fsc->mdsc)) {
rc = -EIO;
goto release_folios;
}
is insufficient. The folios are locked and can't just be released.
Why ceph_inc_osd_stopping_blocker() fails is also something that needs looking
at.
David
next prev parent reply other threads:[~2025-01-28 20:01 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-01-17 3:50 [PATCH v2] ceph: Fix kernel crash in generic/397 test Viacheslav Dubeyko
2025-01-17 12:06 ` Ilya Dryomov
2025-01-17 19:18 ` Viacheslav Dubeyko
2025-01-17 21:32 ` David Howells
2025-01-17 22:08 ` Viacheslav Dubeyko
2025-01-20 9:33 ` David Howells
2025-01-20 9:47 ` Alex Markuze
2025-01-27 18:40 ` Viacheslav Dubeyko
2025-01-28 16:13 ` David Howells
2025-01-28 16:57 ` David Howells
2025-01-28 20:01 ` David Howells [this message]
2025-01-28 20:16 ` Viacheslav Dubeyko
2025-01-28 22:34 ` David Howells
2025-01-28 22:37 ` Viacheslav Dubeyko
2025-01-29 10:39 ` Alex Markuze
2025-01-29 13:42 ` David Howells
2025-01-29 13:54 ` Alex Markuze
2025-01-29 13:41 ` David Howells
2025-01-29 19:16 ` Viacheslav Dubeyko
2025-02-14 20:29 ` David Howells
2025-02-14 20:54 ` Viacheslav Dubeyko
2025-04-15 17:59 ` Viacheslav Dubeyko
2025-06-01 16:23 ` Ilya Dryomov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=3532744.1738094469@warthog.procyon.org.uk \
--to=dhowells@redhat.com \
--cc=Slava.Dubeyko@ibm.com \
--cc=amarkuze@redhat.com \
--cc=ceph-devel@vger.kernel.org \
--cc=idryomov@gmail.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=slava@dubeyko.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.