[ANNOUNCE] work tree for untorn filesystem writes

[ANNOUNCE] work tree for untorn filesystem writes

[Darrick J. Wong]

Hi everyone,

Nobody else has stepped up to do this, so I've created a work branch for
the fs side of untorn writes:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=fs-atomic_2024-11-04

Can you all check this to make sure that I merged it correctly?  And
maybe go test this on your storage hardware? :)

If all goes well then I think the next step is to ask brauner very
nicely if he'd consider adding this to the vfs trees for 6.13.  If not
then I guess we can submit it ourselves, though we probably ought to ask
rothwell to add the branch to for-next asap.

PS: We're now past -rc6 so please reply quickly so that this doesn't
slip yet another cycle.

Catherine: John's on vacation all week, could you please send me the
latest versions of the xfs_io pwrite-atomic patch and the fstest for it?

--D

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Carlos Maiolino]

On Mon, Nov 04, 2024 at 04:43:41PM -0800, Darrick J. Wong wrote:
> Hi everyone,
> 
> Nobody else has stepped up to do this, so I've created a work branch for
> the fs side of untorn writes:
> https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=fs-atomic_2024-11-04
> 
> Can you all check this to make sure that I merged it correctly?  And
> maybe go test this on your storage hardware? :)
> 
> If all goes well then I think the next step is to ask brauner very
> nicely if he'd consider adding this to the vfs trees for 6.13.  If not
> then I guess we can submit it ourselves, though we probably ought to ask
> rothwell to add the branch to for-next asap.
> 
> PS: We're now past -rc6 so please reply quickly so that this doesn't
> slip yet another cycle.
> 
> Catherine: John's on vacation all week, could you please send me the
> latest versions of the xfs_io pwrite-atomic patch and the fstest for it?

I am kind confused here now. IIRC Jens pulled the first three patches from
John's series into his tree, and John asked me to pull the other ones. I'm much
happier to see a single person pulling the whole series instead of splitting it
into different maintainers though.

Giving how spread the series is, I'd say going through vfs tree would be the
best place, but I'm not opposed to pull them myself.

Carlos

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Jens Axboe]

On 11/5/24 4:19 AM, Carlos Maiolino wrote:
> On Mon, Nov 04, 2024 at 04:43:41PM -0800, Darrick J. Wong wrote:
>> Hi everyone,
>>
>> Nobody else has stepped up to do this, so I've created a work branch for
>> the fs side of untorn writes:
>> https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=fs-atomic_2024-11-04
>>
>> Can you all check this to make sure that I merged it correctly?  And
>> maybe go test this on your storage hardware? :)
>>
>> If all goes well then I think the next step is to ask brauner very
>> nicely if he'd consider adding this to the vfs trees for 6.13.  If not
>> then I guess we can submit it ourselves, though we probably ought to ask
>> rothwell to add the branch to for-next asap.
>>
>> PS: We're now past -rc6 so please reply quickly so that this doesn't
>> slip yet another cycle.
>>
>> Catherine: John's on vacation all week, could you please send me the
>> latest versions of the xfs_io pwrite-atomic patch and the fstest for it?
> 
> I am kind confused here now. IIRC Jens pulled the first three patches
> from John's series into his tree, and John asked me to pull the other
> ones. I'm much happier to see a single person pulling the whole series
> instead of splitting it into different maintainers though.
> 
> Giving how spread the series is, I'd say going through vfs tree would
> be the best place, but I'm not opposed to pull them myself.

Guys, not sure why this is so difficult to grasp. I already pulled the
initial bits weeks ago, into an immutable branch:

https://git.kernel.dk/cgit/linux/log/?h=for-6.13/block-atomic

which was subsequently also pulled into for-6.13/block. Whoever wants
to stage the xfs bits must simply:

1) Pull the above for-6.13/block-atomic branch
2) Apply XFS bits on top

Why is this so difficult to grasp? It's a pretty common method for cross
subsystem work - it avoids introducing conflicts when later work goes
into each subsystem, and freedom of either side to send a PR before the
other.

So please don't start committing the patches again, it'll just cause
duplicate (and empty) commits in Linus's tree.

-- 
Jens Axboe

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Theodore Ts'o]

On Tue, Nov 05, 2024 at 05:52:05AM -0700, Jens Axboe wrote:
> 
> Why is this so difficult to grasp? It's a pretty common method for
> cross subsystem work - it avoids introducing conflicts when later
> work goes into each subsystem, and freedom of either side to send a
> PR before the other.
> 
> So please don't start committing the patches again, it'll just cause
> duplicate (and empty) commits in Linus's tree.

Jens, what's going on is that in order to test untorn (aka "atomic"
although that's a bit of a misnomer) writes, changes are needed in the
block, vfs, and ext4 or xfs git trees.  So we are aware that you had
taken the block-related patches into the block tree.  What Darrick has
done is to apply the the vfs patches on top of the block commits, and
then applied the ext4 and xfs patches on top of that.

I'm willing to allow the ext4 patches to flow to Linus's tree without
it personally going through the ext4 tree.  If all Maintainers
required that patches which touched their trees had to go through
their respective trees, it would require multiple (strictly ordered)
pull requests during the merge window, or multiple merge windows, to
land these series.  Since you insisted on the block changes had to go
through the block tree, we're trying to accomodate you; and also (a)
we don't want to have duplicate commits in Linus's tree; and at the
same time, (b) but these patches have been waiting to land for almost
two years, and we're also trying to make things land a bit more
expeditiously.

Cheers,

					- Ted

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Jens Axboe]

On 11/5/24 8:08 AM, Theodore Ts'o wrote:
> On Tue, Nov 05, 2024 at 05:52:05AM -0700, Jens Axboe wrote:
>>
>> Why is this so difficult to grasp? It's a pretty common method for
>> cross subsystem work - it avoids introducing conflicts when later
>> work goes into each subsystem, and freedom of either side to send a
>> PR before the other.
>>
>> So please don't start committing the patches again, it'll just cause
>> duplicate (and empty) commits in Linus's tree.
> 
> Jens, what's going on is that in order to test untorn (aka "atomic"
> although that's a bit of a misnomer) writes, changes are needed in the
> block, vfs, and ext4 or xfs git trees.  So we are aware that you had
> taken the block-related patches into the block tree.  What Darrick has
> done is to apply the the vfs patches on top of the block commits, and
> then applied the ext4 and xfs patches on top of that.

And what I'm saying is that is _wrong_. Darrick should be pulling the
branch that you cut from my email:

for-6.13/block-atomic

rather than re-applying patches. At least if the intent is to send that
branch to Linus. But even if it's just for testing, pretty silly to have
branches with duplicate commits out there when the originally applied
patches can just be pulled in.

> I'm willing to allow the ext4 patches to flow to Linus's tree without
> it personally going through the ext4 tree.  If all Maintainers
> required that patches which touched their trees had to go through
> their respective trees, it would require multiple (strictly ordered)
> pull requests during the merge window, or multiple merge windows, to

That is simply not true. There's ZERO ordering required here. Like I
also mentioned in my reply, and that you also snipped out, is that no
ordering is implied here - either tree can send their PR at any time.

> land these series.  Since you insisted on the block changes had to go
> through the block tree, we're trying to accomodate you; and also (a)
> we don't want to have duplicate commits in Linus's tree; and at the
> same time, (b) but these patches have been waiting to land for almost
> two years, and we're also trying to make things land a bit more
> expeditiously.

Just pull the branch that was created for it... There's zero other
things in there outside of the 3 commits.

-- 
Jens Axboe

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Darrick J. Wong]

On Tue, Nov 05, 2024 at 08:11:52AM -0700, Jens Axboe wrote:
> On 11/5/24 8:08 AM, Theodore Ts'o wrote:
> > On Tue, Nov 05, 2024 at 05:52:05AM -0700, Jens Axboe wrote:
> >>
> >> Why is this so difficult to grasp? It's a pretty common method for
> >> cross subsystem work - it avoids introducing conflicts when later
> >> work goes into each subsystem, and freedom of either side to send a
> >> PR before the other.
> >>
> >> So please don't start committing the patches again, it'll just cause
> >> duplicate (and empty) commits in Linus's tree.
> > 
> > Jens, what's going on is that in order to test untorn (aka "atomic"
> > although that's a bit of a misnomer) writes, changes are needed in the
> > block, vfs, and ext4 or xfs git trees.  So we are aware that you had
> > taken the block-related patches into the block tree.  What Darrick has
> > done is to apply the the vfs patches on top of the block commits, and
> > then applied the ext4 and xfs patches on top of that.
> 
> And what I'm saying is that is _wrong_. Darrick should be pulling the
> branch that you cut from my email:
> 
> for-6.13/block-atomic
> 
> rather than re-applying patches. At least if the intent is to send that
> branch to Linus. But even if it's just for testing, pretty silly to have
> branches with duplicate commits out there when the originally applied
> patches can just be pulled in.

I *did* start my branch at the end of your block-atomic branch.

Notice how the commits I added yesterday have a parent commitid of
1eadb157947163ca72ba8963b915fdc099ce6cca, which is the head of your
for-6.13/block-atomic branch?

But, it's my fault for not explicitly stating that I did that.  One of
the lessons I apparently keep needing to learn is that senior developers
here don't actually pull and examine the branches I link to in my emails
before hitting Reply All to scold.  You obviously didn't.

Maybe the lesson I really need to learn here is that none of this
constant pointless aggravation in my life is worth it.

--D

> > I'm willing to allow the ext4 patches to flow to Linus's tree without
> > it personally going through the ext4 tree.  If all Maintainers
> > required that patches which touched their trees had to go through
> > their respective trees, it would require multiple (strictly ordered)
> > pull requests during the merge window, or multiple merge windows, to
> 
> That is simply not true. There's ZERO ordering required here. Like I
> also mentioned in my reply, and that you also snipped out, is that no
> ordering is implied here - either tree can send their PR at any time.
> 
> > land these series.  Since you insisted on the block changes had to go
> > through the block tree, we're trying to accomodate you; and also (a)
> > we don't want to have duplicate commits in Linus's tree; and at the
> > same time, (b) but these patches have been waiting to land for almost
> > two years, and we're also trying to make things land a bit more
> > expeditiously.
> 
> Just pull the branch that was created for it... There's zero other
> things in there outside of the 3 commits.
> 
> -- 
> Jens Axboe

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Jens Axboe]

On 11/5/24 8:40 AM, Darrick J. Wong wrote:
> On Tue, Nov 05, 2024 at 08:11:52AM -0700, Jens Axboe wrote:
>> On 11/5/24 8:08 AM, Theodore Ts'o wrote:
>>> On Tue, Nov 05, 2024 at 05:52:05AM -0700, Jens Axboe wrote:
>>>>
>>>> Why is this so difficult to grasp? It's a pretty common method for
>>>> cross subsystem work - it avoids introducing conflicts when later
>>>> work goes into each subsystem, and freedom of either side to send a
>>>> PR before the other.
>>>>
>>>> So please don't start committing the patches again, it'll just cause
>>>> duplicate (and empty) commits in Linus's tree.
>>>
>>> Jens, what's going on is that in order to test untorn (aka "atomic"
>>> although that's a bit of a misnomer) writes, changes are needed in the
>>> block, vfs, and ext4 or xfs git trees.  So we are aware that you had
>>> taken the block-related patches into the block tree.  What Darrick has
>>> done is to apply the the vfs patches on top of the block commits, and
>>> then applied the ext4 and xfs patches on top of that.
>>
>> And what I'm saying is that is _wrong_. Darrick should be pulling the
>> branch that you cut from my email:
>>
>> for-6.13/block-atomic
>>
>> rather than re-applying patches. At least if the intent is to send that
>> branch to Linus. But even if it's just for testing, pretty silly to have
>> branches with duplicate commits out there when the originally applied
>> patches can just be pulled in.
> 
> I *did* start my branch at the end of your block-atomic branch.
> 
> Notice how the commits I added yesterday have a parent commitid of
> 1eadb157947163ca72ba8963b915fdc099ce6cca, which is the head of your
> for-6.13/block-atomic branch?

Ah that's my bad, I didn't see a merge commit, so assumed it was just
applied on top. Checking now, yeah it does look like it's done right!
Would've been nicer on top of current -rc and with a proper merge
commit, but that's really more of a style preference. Though -rc1 is
pretty early...

> But, it's my fault for not explicitly stating that I did that.  One of
> the lessons I apparently keep needing to learn is that senior developers
> here don't actually pull and examine the branches I link to in my emails
> before hitting Reply All to scold.  You obviously didn't.

I did click the link, in my defense it was on the phone this morning.
And this wasn't meant as a scolding, nor do I think my wording really
implies any scolding. My frustration was that I had explained this
previously, and this seemed like another time to do the exact same. So
my apologies if it came off like that, was not the intent.

-- 
Jens Axboe

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Christian Brauner]

On Tue, Nov 05, 2024 at 08:54:40AM -0700, Jens Axboe wrote:
> On 11/5/24 8:40 AM, Darrick J. Wong wrote:
> > On Tue, Nov 05, 2024 at 08:11:52AM -0700, Jens Axboe wrote:
> >> On 11/5/24 8:08 AM, Theodore Ts'o wrote:
> >>> On Tue, Nov 05, 2024 at 05:52:05AM -0700, Jens Axboe wrote:
> >>>>
> >>>> Why is this so difficult to grasp? It's a pretty common method for
> >>>> cross subsystem work - it avoids introducing conflicts when later
> >>>> work goes into each subsystem, and freedom of either side to send a
> >>>> PR before the other.
> >>>>
> >>>> So please don't start committing the patches again, it'll just cause
> >>>> duplicate (and empty) commits in Linus's tree.
> >>>
> >>> Jens, what's going on is that in order to test untorn (aka "atomic"
> >>> although that's a bit of a misnomer) writes, changes are needed in the
> >>> block, vfs, and ext4 or xfs git trees.  So we are aware that you had
> >>> taken the block-related patches into the block tree.  What Darrick has
> >>> done is to apply the the vfs patches on top of the block commits, and
> >>> then applied the ext4 and xfs patches on top of that.
> >>
> >> And what I'm saying is that is _wrong_. Darrick should be pulling the
> >> branch that you cut from my email:
> >>
> >> for-6.13/block-atomic
> >>
> >> rather than re-applying patches. At least if the intent is to send that
> >> branch to Linus. But even if it's just for testing, pretty silly to have
> >> branches with duplicate commits out there when the originally applied
> >> patches can just be pulled in.
> > 
> > I *did* start my branch at the end of your block-atomic branch.
> > 
> > Notice how the commits I added yesterday have a parent commitid of
> > 1eadb157947163ca72ba8963b915fdc099ce6cca, which is the head of your
> > for-6.13/block-atomic branch?
> 
> Ah that's my bad, I didn't see a merge commit, so assumed it was just
> applied on top. Checking now, yeah it does look like it's done right!
> Would've been nicer on top of current -rc and with a proper merge
> commit, but that's really more of a style preference. Though -rc1 is
> pretty early...
> 
> > But, it's my fault for not explicitly stating that I did that.  One of
> > the lessons I apparently keep needing to learn is that senior developers
> > here don't actually pull and examine the branches I link to in my emails
> > before hitting Reply All to scold.  You obviously didn't.
> 
> I did click the link, in my defense it was on the phone this morning.
> And this wasn't meant as a scolding, nor do I think my wording really
> implies any scolding. My frustration was that I had explained this
> previously, and this seemed like another time to do the exact same. So
> my apologies if it came off like that, was not the intent.

Fwiw, I pulled the branch that Darrick provided into vfs.untorn.writes
and it all looks sane to me.

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Carlos Maiolino]

On Wed, Nov 06, 2024 at 11:40:00AM +0100, Christian Brauner wrote:
> On Tue, Nov 05, 2024 at 08:54:40AM -0700, Jens Axboe wrote:
> > On 11/5/24 8:40 AM, Darrick J. Wong wrote:
> > > On Tue, Nov 05, 2024 at 08:11:52AM -0700, Jens Axboe wrote:
> > >> On 11/5/24 8:08 AM, Theodore Ts'o wrote:
> > >>> On Tue, Nov 05, 2024 at 05:52:05AM -0700, Jens Axboe wrote:
> > >>>>
> > >>>> Why is this so difficult to grasp? It's a pretty common method for
> > >>>> cross subsystem work - it avoids introducing conflicts when later
> > >>>> work goes into each subsystem, and freedom of either side to send a
> > >>>> PR before the other.
> > >>>>
> > >>>> So please don't start committing the patches again, it'll just cause
> > >>>> duplicate (and empty) commits in Linus's tree.
> > >>>
> > >>> Jens, what's going on is that in order to test untorn (aka "atomic"
> > >>> although that's a bit of a misnomer) writes, changes are needed in the
> > >>> block, vfs, and ext4 or xfs git trees.  So we are aware that you had
> > >>> taken the block-related patches into the block tree.  What Darrick has
> > >>> done is to apply the the vfs patches on top of the block commits, and
> > >>> then applied the ext4 and xfs patches on top of that.
> > >>
> > >> And what I'm saying is that is _wrong_. Darrick should be pulling the
> > >> branch that you cut from my email:
> > >>
> > >> for-6.13/block-atomic
> > >>
> > >> rather than re-applying patches. At least if the intent is to send that
> > >> branch to Linus. But even if it's just for testing, pretty silly to have
> > >> branches with duplicate commits out there when the originally applied
> > >> patches can just be pulled in.
> > > 
> > > I *did* start my branch at the end of your block-atomic branch.
> > > 
> > > Notice how the commits I added yesterday have a parent commitid of
> > > 1eadb157947163ca72ba8963b915fdc099ce6cca, which is the head of your
> > > for-6.13/block-atomic branch?
> > 
> > Ah that's my bad, I didn't see a merge commit, so assumed it was just
> > applied on top. Checking now, yeah it does look like it's done right!
> > Would've been nicer on top of current -rc and with a proper merge
> > commit, but that's really more of a style preference. Though -rc1 is
> > pretty early...
> > 
> > > But, it's my fault for not explicitly stating that I did that.  One of
> > > the lessons I apparently keep needing to learn is that senior developers
> > > here don't actually pull and examine the branches I link to in my emails
> > > before hitting Reply All to scold.  You obviously didn't.
> > 
> > I did click the link, in my defense it was on the phone this morning.
> > And this wasn't meant as a scolding, nor do I think my wording really
> > implies any scolding. My frustration was that I had explained this
> > previously, and this seemed like another time to do the exact same. So
> > my apologies if it came off like that, was not the intent.
> 
> Fwiw, I pulled the branch that Darrick provided into vfs.untorn.writes
> and it all looks sane to me.
> 

Sounds good, will you submit a pull-request from it or shall I still submit the
remaining ones to Linus?

Carlos

Re: [ANNOUNCE] work tree for untorn filesystem writes

[Ritesh Harjani]

"Darrick J. Wong" <djwong@kernel.org> writes:

> Hi everyone,
>
> Nobody else has stepped up to do this, so I've created a work branch for
> the fs side of untorn writes:
> https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git/log/?h=fs-atomic_2024-11-04
>
> Can you all check this to make sure that I merged it correctly?

Sorry, I couldn't reply earlier(I am currently on travel). Yes, the ext4
merge looks correct to me. You have taken the latest v4 of the ext4
atomic write series [1]. 

[1]: https://lore.kernel.org/linux-ext4/cover.1730437365.git.ritesh.list@gmail.com/

> And maybe go test this on your storage hardware? :)

Due to limited connectivity during my travel, I don't have the access to
the hardware. But as I mentioned the merge looks correct to me and I had
tested those patches earlier on Power and x86.
But I will in general re-test the mentioned fs branch for both XFS and
ext4 once I reach back but I don't think we need to wait for that as the
merge looks good to me.

Also, I noticed that we might have missed to add a Tested-by from
Ojaswin for XFS series here [2]. Although Ojaswin mentioned that he
might also re-test the mentioned FS atomic write branch for both XFS and
EXT4.

[2]: https://lore.kernel.org/linux-xfs/Zxnp8bma2KrMDg5m@li-bb2b2a4c-3307-11b2-a85c-8fa5c3a69313.ibm.com/


-ritesh

> If all goes well then I think the next step is to ask brauner very
> nicely if he'd consider adding this to the vfs trees for 6.13.  If not
> then I guess we can submit it ourselves, though we probably ought to ask
> rothwell to add the branch to for-next asap.
>
> PS: We're now past -rc6 so please reply quickly so that this doesn't
> slip yet another cycle.
>
> Catherine: John's on vacation all week, could you please send me the
> latest versions of the xfs_io pwrite-atomic patch and the fstest for it?
>
> --D