From: Dave Chinner <david@fromorbit.com>
To: linux-xfs@vger.kernel.org
Subject: [PATCH 0/4] xfs: fix CIL shutdown UAF and shutdown hang
Date: Tue, 22 Jun 2021 14:06:00 +1000 [thread overview]
Message-ID: <20210622040604.1290539-1-david@fromorbit.com> (raw)
Hi folks,
The following patches implement an initial fix for the UAF that can
occur in the CIL push code when a racing shutdown occurs. This was a
zero-day bug in the delayed logging code, and only recently
uncovered by the CIL pipelining changes that addresses a different
zero-day bug in the delayed logging code. This UAF exists regardless
in all kernels that support delayed logging (i.e. since 2.6.36), but
is extremely unlikely that anyone has hit it as it requires a
shutdown with extremely tight timing tolerances to trigger a UAF.
This is more of a problem for the current for-next tree, though,
because there is now a call to xlog_wait_on_iclog() in the UAF
window. While we don't reference the CIL context after the wait,
this will soon be needed to fix the /other/ zero-day problems found
by the CIL pipelining changes.
The encapsulation of the entire CIL commit iclog processing epilogue
in the icloglock effectively serialises this code against shutdown
races and allows us to error out before attaching the context to the
iclog if a shutdown has already occurred. Callbacks used to be under
the icloglock, but were split out in 2008 because of icloglock
contention causing log scalability problems (sound familiar? :).
Delayed logging fixed those icloglock scalability issues by moving
it out of the hot transaction commit path, so we can move the
callbacks back under the icloglock without re-introducing ancient
problems and solve the initial UAF problem this way.
With that problem solved, we can then fix the call to
xlog_wait_on_iclog() in the CIL push code by ensuring that it only
waits on older iclogs via LSN checks. As the wait drops the icloglock and
potentially re-opens us to the above UAF on shutdown, we have to be
careful not to reference the CIL context after the wait returns.
Hence the patches don't really fix the underlying cause of the
shutdown UAF here - this is intended as a low impact, easily
backportable solution to the problem. Work to fix the underlying
shutdown brokenness to remove the need to hold the icloglock from
callback attachment to xlog_state_release_iclog() is needed
(underway) before we can then apply start record ordering fixes and
re-introduce the CIL pipelining fixes and the rest of the CIL
scalabilty work....
Cheers,
Dave.
next reply other threads:[~2021-06-22 4:06 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-22 4:06 Dave Chinner [this message]
2021-06-22 4:06 ` [PATCH 1/4] xfs: don't nest icloglock inside ic_callback_lock Dave Chinner
2021-06-22 12:38 ` Brian Foster
2021-06-22 22:42 ` Dave Chinner
2021-06-23 10:18 ` Brian Foster
2021-06-25 20:52 ` Darrick J. Wong
2021-06-22 4:06 ` [PATCH 2/4] xfs: remove callback dequeue loop from xlog_state_do_iclog_callbacks Dave Chinner
2021-06-22 12:39 ` Brian Foster
2021-06-22 22:56 ` Dave Chinner
2021-06-25 20:57 ` Darrick J. Wong
2021-06-22 4:06 ` [PATCH 3/4] xfs: Fix a CIL UAF by getting get rid of the iclog callback lock Dave Chinner
2021-06-22 12:41 ` Brian Foster
2021-06-25 21:02 ` Darrick J. Wong
2021-06-22 4:06 ` [PATCH 4/4] xfs: don't wait on future iclogs when pushing the CIL Dave Chinner
2021-06-22 12:41 ` Brian Foster
2021-06-25 21:02 ` Darrick J. Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20210622040604.1290539-1-david@fromorbit.com \
--to=david@fromorbit.com \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox