From: Bastian Blank <bastian.blank@credativ.de>
To: Jeff Layton <jlayton@kernel.org>, Ilya Dryomov <idryomov@gmail.com>
Cc: ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: PROBLEM: SLAB use-after-free with ceph(fs)
Date: Tue, 4 Jan 2022 10:49:04 +0100 [thread overview]
Message-ID: <YdQYEF9WCTWTAzOp@softhammer.credativ.lan> (raw)
Moin
A customer reported panics inside memory management. Before all
occurances there are reports about SLAB missmatch in the log. The
"crash" tool shows freelist corruption in the memory dump. This makes
this problem a use-after-free somewhere inside the ceph module.
The crashs happen during high load situations, while copying data
between two cephfs.
| [152791.777454] ceph: dropping dirty+flushing - state for 00000000c039d4cc 1099526092092
| [152791.777457] ------------[ cut here ]------------
| [152791.777458] cache_from_obj: Wrong slab cache. jbd2_journal_handle but object is from kmalloc-256
| [152791.777473] WARNING: CPU: 76 PID: 2676615 at mm/slab.h:521 kmem_cache_free+0x260/0x2b0
[…]
| [152791.777530] CPU: 76 PID: 2676615 Comm: kworker/76:2 Kdump: loaded Not tainted 5.4.0-81-generic #91-Ubuntu
| [152791.777531] Hardware name: HPE ProLiant DL385 Gen10/ProLiant DL385 Gen10, BIOS A40 10/28/2021
| [152791.777540] Workqueue: ceph-msgr ceph_con_workfn [libceph]
| [152791.777542] RIP: 0010:kmem_cache_free+0x260/0x2b0
[…]
| [152791.777550] Call Trace:
| [152791.777562] ceph_free_cap_flush+0x1d/0x20 [ceph]
| [152791.777568] remove_session_caps_cb+0xcf/0x4b0 [ceph]
| [152791.777573] ceph_iterate_session_caps+0xc8/0x2a0 [ceph]
| [152791.777578] ? wake_up_session_cb+0xe0/0xe0 [ceph]
| [152791.777583] remove_session_caps+0x55/0x190 [ceph]
| [152791.777587] ? cleanup_session_requests+0x104/0x130 [ceph]
| [152791.777592] handle_session+0x4c7/0x5e0 [ceph]
| [152791.777597] dispatch+0x279/0x610 [ceph]
| [152791.777602] try_read+0x566/0x8c0 [libceph]
They reported the same in all tested kernels since 5.4, up to 5.15.5 or
so. Sadly I have no tests with newer builds available.
Any ideas how I can debug this further?
Regards,
Bastian
--
Bastian Blank
Berater
Telefon: +49 2166 9901-194
E-Mail: bastian.blank@credativ.de
credativ GmbH, HRB Mönchengladbach 12080, USt-ID-Nummer: DE204566209
Trompeterallee 108, 41189 Mönchengladbach
Geschäftsführung: Dr. Michael Meskes, Geoff Richardson, Peter Lilley
Unser Umgang mit personenbezogenen Daten unterliegt
folgenden Bestimmungen: https://www.credativ.de/datenschutz
next reply other threads:[~2022-01-04 9:57 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-04 9:49 Bastian Blank [this message]
2022-01-04 12:00 ` PROBLEM: SLAB use-after-free with ceph(fs) Jeff Layton
2022-01-04 12:20 ` Bastian Blank
2022-01-04 12:29 ` Jeff Layton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YdQYEF9WCTWTAzOp@softhammer.credativ.lan \
--to=bastian.blank@credativ.de \
--cc=ceph-devel@vger.kernel.org \
--cc=idryomov@gmail.com \
--cc=jlayton@kernel.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox