Re: Backporting of series xfs/iomap: fix data corruption due to stale cached iomap

["Darrick J. Wong" <djwong@kernel.org>]

On Fri, Jun 30, 2023 at 04:05:36PM +0300, Amir Goldstein wrote:
> On Fri, Jun 30, 2023 at 3:30 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
> >
> > On Fri, Jun 30, 2023 at 11:39 AM Amir Goldstein <amir73il@gmail.com> wrote:
> > >
> > > On Thu, Jun 29, 2023 at 10:31 PM Ignat Korchagin <ignat@cloudflare.com> wrote:
> > > >
> > > > On Thu, Jun 29, 2023 at 7:14 PM Darrick J. Wong <djwong@kernel.org> wrote:
> > > > >
> > > > > [add the xfs lts maintainers]
> > > > >
> > > > > On Thu, Jun 29, 2023 at 05:34:00PM +0100, Matthew Wilcox wrote:
> > > > > > On Thu, Jun 29, 2023 at 05:09:41PM +0100, Daniel Dao wrote:
> > > > > > > Hi Dave and Derrick,
> > > > > > >
> > > > > > > We are tracking down some corruptions on xfs for our rocksdb workload,
> > > > > > > running on kernel 6.1.25. The corruptions were
> > > > > > > detected by rocksdb block checksum. The workload seems to share some
> > > > > > > similarities
> > > > > > > with the multi-threaded write workload described in
> > > > > > > https://lore.kernel.org/linux-fsdevel/20221129001632.GX3600936@dread.disaster.area/
> > > > > > >
> > > > > > > Can we backport the patch series to stable since it seemed to fix data
> > > > > > > corruptions ?
> > > > > >
> > > > > > For clarity, are you asking for permission or advice about doing this
> > > > > > yourself, or are you asking somebody else to do the backport for you?
> > > > >
> > > > > Nobody's officially committed to backporting and testing patches for
> > > > > 6.1; are you (Cloudflare) volunteering?
> > > >
> > > > Yes, we have applied them on top of 6.1.36, will be gradually
> > > > releasing to our servers and will report back if we see the issues go
> > > > away
> > > >
> > >
> > > Getting feedback back from Cloudflare production servers is awesome
> > > but it's not enough.
> > >
> > > The standard for getting xfs LTS backports approved is:
> > > 1. Test the backports against regressions with several rounds of fstests
> > >     check -g auto on selected xfs configurations [1]
> > > 2. Post the backport series to xfs list and get an ACK from upstream
> > >     xfs maintainers
> > >
> > > We have volunteers doing this work for 5.4.y, 5.10.y and 5.15.y.
> > > We do not yet have a volunteer to do that work for 6.1.y.
> > >
> > > The question is whether you (or your team) are volunteering to
> > > do that work for 6.1.y xfs backports to help share the load?
> >
> > We are not a big team and apart from other internal project work our
> > efforts are focused on fixing this issue in production, because it
> > affects many teams and workloads. If we confirm that these patches fix
> > the issue in production, we will definitely consider dedicating some
> > work to ensure they are officially backported. But if not - we would
> > be required to search for a fix first before we can commit to any
> > work.
> >
> > So, IOW - can we come back to you a bit later on this after we get the
> > feedback from production?
> >
> 
> Of course.
> The volunteering question for 6.1.y is independent.
> 
> When you decide that you have a series of backports
> that proves to fix a real bug in production,
> a way to test the series will be worked out.

/me notes that xfs/558 and xfs/559 (in fstests) are the functional tests
for these patches that you're backporting; it would be useful to have a
third party (i.e. not just the reporter and the author) confirm that the
two fstests pass when real workloads are fixed.

--D

> Thanks,
> Amir.