Changes to XFS patch integration process

Changes to XFS patch integration process

[Carlos Maiolino]

Hello,

Following recent events, I decided to improve (hopefully it will be an
imporvement), how XFS integration flows.

For a while, XFS integration follows a process where we stack patches on
for-next for a few days (in a good moment a couple weeks), and let them cook
there before sending them to Linus.

This works but has a big down side. Patches for the next merge window are
kept in my trees locally, and although I test them from time to time, they
only hit linux-next late in the cycle, also, it's hard for others to know what
is queued or not (and to develop work on top of that).

The main change I'm going to do, is in for-next branch.

Currently I push patches there only when I've selected which patches
I'm going to send to Linus on the next batch, whether it's a -RC or
the next merge window.

I'll change it, by pushing to for-next everything that has been reviewed
and smoke tested, whether it's for a -RC or next merge window. This will
enable xfs patches to stay in linux-next as much as possible before they
get sent to Linus.

The biggest change here is that for-next will likely need to be rebased
more often than today. But also patches will spend more time under testings
in linux-next and everybody will have a more updated tree to work on.

Also, I'm still thinking how to handle pull requests I receive. I try
hard to not change the commit hashes from the PRs, so I'm still not sure
how feasible it will be to keep the same hash ids from PRs giving more often
than not I'll need to rebase the next merge tree on the top of fixes for the
current -RC and in some cases, on top of other trees with dependencies.

I'm sending this information now as if you have any concern, please let me know.

By now, I'm creating a new branch named xfs-6.15-merge to aggregate everything
I already have for 6.15, and later this week I'll be merging it into for-next

Hopefully, for the majority of xfs tree users, this should be mostly transparent.
Only those who watch the development really close may notice differences.

Cheers,
Carlos

Re: Changes to XFS patch integration process

[Christoph Hellwig]

On Mon, Mar 03, 2025 at 11:42:12AM +0100, Carlos Maiolino wrote:
> The biggest change here is that for-next will likely need to be rebased
> more often than today. But also patches will spend more time under testings
> in linux-next and everybody will have a more updated tree to work on.

FYI, what other trees do is to keep separate branches for the current
and next release, i.e. right now: for-6.14 and for-6.15 and merge those
into the for-next or have both of them in linux-next (e.g. for-linus and
for-next).  In that case most of the time you don't need to rebase at
all.  Instead you might occasionally need to merge the current into the
next tree to resolve conflicts, and Linus is fine with that if you
document the reason for that merge.

> 
> Also, I'm still thinking how to handle pull requests I receive. I try
> hard to not change the commit hashes from the PRs, so I'm still not sure
> how feasible it will be to keep the same hash ids from PRs giving more often
> than not I'll need to rebase the next merge tree on the top of fixes for the
> current -RC and in some cases, on top of other trees with dependencies.

With the above you just keep the pull requests as-is.

Re: Changes to XFS patch integration process

[Carlos Maiolino]

On Mon, Mar 03, 2025 at 03:05:47PM +0100, Christoph Hellwig wrote:
> On Mon, Mar 03, 2025 at 11:42:12AM +0100, Carlos Maiolino wrote:
> > The biggest change here is that for-next will likely need to be rebased
> > more often than today. But also patches will spend more time under testings
> > in linux-next and everybody will have a more updated tree to work on.
> 
> FYI, what other trees do is to keep separate branches for the current
> and next release, i.e. right now: for-6.14 and for-6.15 and merge those
> into the for-next or have both of them in linux-next (e.g. for-linus and
> for-next).  In that case most of the time you don't need to rebase at
> all.  Instead you might occasionally need to merge the current into the
> next tree to resolve conflicts, and Linus is fine with that if you
> document the reason for that merge.

This is pretty much aligned with my intentions, I haven't looked close yet how
other subsystems deals with it, but by a few releases now, I keep a
xfs-fixes-$ver branch which I collect patches for the current version, so adding
a new branch for the next merge window is what I aimed to do with
xfs-6.15-merge.

The question for me now lies exactly on how to synchronize both. You partially
answered my question, although merging the current into next sounds weird to me.

If I merge current into next, and send Linus a PR for each (let's say for -rc7
and in sequence for the next merge window), Linus will receive two PRs with
possibly the same patches, and yet, on the merge window PR, there will also be a
merge commit from -current, is this what you're describing?

Thanks for the input.

> 
> >
> > Also, I'm still thinking how to handle pull requests I receive. I try
> > hard to not change the commit hashes from the PRs, so I'm still not sure
> > how feasible it will be to keep the same hash ids from PRs giving more often
> > than not I'll need to rebase the next merge tree on the top of fixes for the
> > current -RC and in some cases, on top of other trees with dependencies.
> 
> With the above you just keep the pull requests as-is.
> 
> 

Sounds reasonable

Re: Changes to XFS patch integration process

[Christoph Hellwig]

On Mon, Mar 03, 2025 at 04:00:39PM +0100, Carlos Maiolino wrote:
> This is pretty much aligned with my intentions, I haven't looked close yet how
> other subsystems deals with it, but by a few releases now, I keep a
> xfs-fixes-$ver branch which I collect patches for the current version, so adding
> a new branch for the next merge window is what I aimed to do with
> xfs-6.15-merge.
> 
> The question for me now lies exactly on how to synchronize both. You partially
> answered my question, although merging the current into next sounds weird to me.

for-next really is the convention for what goes into linux-next.
For dma-mapping I kept a separate for-linus instead of merging the
current one into for-next which also works, but you still might have
to merge them occasionally.

> 
> If I merge current into next, and send Linus a PR for each (let's say for -rc7
> and in sequence for the next merge window), Linus will receive two PRs with
> possibly the same patches, and yet, on the merge window PR, there will also be a
> merge commit from -current, is this what you're describing?

Yes, but only if you really need the merge because the code for the
next merge window requires it to base new patches onto it.  If the
merge is a no-op you don't need to merge, and if it is not required
for code added Linus prefers to to the merge himself, possible with
a little bit of guidance.

Re: Changes to XFS patch integration process

[Darrick J. Wong]

On Mon, Mar 03, 2025 at 04:00:39PM +0100, Carlos Maiolino wrote:
> On Mon, Mar 03, 2025 at 03:05:47PM +0100, Christoph Hellwig wrote:
> > On Mon, Mar 03, 2025 at 11:42:12AM +0100, Carlos Maiolino wrote:
> > > The biggest change here is that for-next will likely need to be rebased
> > > more often than today. But also patches will spend more time under testings
> > > in linux-next and everybody will have a more updated tree to work on.
> > 
> > FYI, what other trees do is to keep separate branches for the current
> > and next release, i.e. right now: for-6.14 and for-6.15 and merge those
> > into the for-next or have both of them in linux-next (e.g. for-linus and
> > for-next).  In that case most of the time you don't need to rebase at
> > all.  Instead you might occasionally need to merge the current into the
> > next tree to resolve conflicts, and Linus is fine with that if you
> > document the reason for that merge.

Separate branches for 6.14 and 6.15 that then get merged into a for-next
is what I did when I had separate trains running at the same time.  Most
of the time I just rolled the post-rc6 fixes into the next release, so I
usually only dealt with one at a time.

(to some grumbling)

> This is pretty much aligned with my intentions, I haven't looked close yet how
> other subsystems deals with it, but by a few releases now, I keep a
> xfs-fixes-$ver branch which I collect patches for the current version, so adding
> a new branch for the next merge window is what I aimed to do with
> xfs-6.15-merge.
> 
> The question for me now lies exactly on how to synchronize both. You partially
> answered my question, although merging the current into next sounds weird to me.
> 
> If I merge current into next, and send Linus a PR for each (let's say for -rc7
> and in sequence for the next merge window), Linus will receive two PRs with
> possibly the same patches, and yet, on the merge window PR, there will also be a
> merge commit from -current, is this what you're describing?

If I had a for-6.14 and a for-6.15 branch, I'd base the PRs off of those
branches, not the for-next branch itself.

> Thanks for the input.
> 
> > 
> > >
> > > Also, I'm still thinking how to handle pull requests I receive. I try
> > > hard to not change the commit hashes from the PRs, so I'm still not sure
> > > how feasible it will be to keep the same hash ids from PRs giving more often
> > > than not I'll need to rebase the next merge tree on the top of fixes for the
> > > current -RC and in some cases, on top of other trees with dependencies.
> > 
> > With the above you just keep the pull requests as-is.
> > 
> > 
> 
> Sounds reasonable

Or you can ask the PR submitter to rebase off latest for-6.15 and handle
the merge themselves.

--D

Re: Changes to XFS patch integration process

[Carlos Maiolino]

On Tue, Mar 04, 2025 at 12:20:59PM -0800, Darrick J. Wong wrote:
> On Mon, Mar 03, 2025 at 04:00:39PM +0100, Carlos Maiolino wrote:
> > On Mon, Mar 03, 2025 at 03:05:47PM +0100, Christoph Hellwig wrote:
> > > On Mon, Mar 03, 2025 at 11:42:12AM +0100, Carlos Maiolino wrote:
> > > > The biggest change here is that for-next will likely need to be rebased
> > > > more often than today. But also patches will spend more time under testings
> > > > in linux-next and everybody will have a more updated tree to work on.
> > >
> > > FYI, what other trees do is to keep separate branches for the current
> > > and next release, i.e. right now: for-6.14 and for-6.15 and merge those
> > > into the for-next or have both of them in linux-next (e.g. for-linus and
> > > for-next).  In that case most of the time you don't need to rebase at
> > > all.  Instead you might occasionally need to merge the current into the
> > > next tree to resolve conflicts, and Linus is fine with that if you
> > > document the reason for that merge.
> 
> Separate branches for 6.14 and 6.15 that then get merged into a for-next
> is what I did when I had separate trains running at the same time.  Most
> of the time I just rolled the post-rc6 fixes into the next release, so I
> usually only dealt with one at a time.
> 
> (to some grumbling)
> 
> > This is pretty much aligned with my intentions, I haven't looked close yet how
> > other subsystems deals with it, but by a few releases now, I keep a
> > xfs-fixes-$ver branch which I collect patches for the current version, so adding
> > a new branch for the next merge window is what I aimed to do with
> > xfs-6.15-merge.
> >
> > The question for me now lies exactly on how to synchronize both. You partially
> > answered my question, although merging the current into next sounds weird to me.
> >
> > If I merge current into next, and send Linus a PR for each (let's say for -rc7
> > and in sequence for the next merge window), Linus will receive two PRs with
> > possibly the same patches, and yet, on the merge window PR, there will also be a
> > merge commit from -current, is this what you're describing?
> 
> If I had a for-6.14 and a for-6.15 branch, I'd base the PRs off of those
> branches, not the for-next branch itself.

I see what you mean, but from another POV, you'd be basing a PR on top of one
series of patches, not on top of everything.

Today, what we have, is a relatively stable for-next branch, where we just
really rebase when something goes wrong, so, usually, when I push things into
for-next, I've had it tested for a big while.

Per my conversations off-list (specially with hch), is that this shouldn't be
the purpose at all of for-next, but a testing branch where (almost anything) can
go wrong, within reason of course. Please correct me if I'm wrong here.

At the same time, I wish we have a branch that everybody can work with, which
contains 'everything' staged, ready to go, so I'd do all the merge between
current and next release myself into such branch. I think having a branch ready
for people to work with is a maintainer's job, and people shouldn't be bothered
by trying to figure out which branch they should use to base their patches on
top.

I'm hoping to use the master's branch for that if nobody has any objection.

Carlos

> 
> > Thanks for the input.
> >
> > >
> > > >
> > > > Also, I'm still thinking how to handle pull requests I receive. I try
> > > > hard to not change the commit hashes from the PRs, so I'm still not sure
> > > > how feasible it will be to keep the same hash ids from PRs giving more often
> > > > than not I'll need to rebase the next merge tree on the top of fixes for the
> > > > current -RC and in some cases, on top of other trees with dependencies.
> > >
> > > With the above you just keep the pull requests as-is.
> > >
> > >
> >
> > Sounds reasonable
> 
> Or you can ask the PR submitter to rebase off latest for-6.15 and handle
> the merge themselves.
> 
> --D

Re: Changes to XFS patch integration process

[Christoph Hellwig]

On Thu, Mar 06, 2025 at 08:50:51AM +0100, Carlos Maiolino wrote:
> > If I had a for-6.14 and a for-6.15 branch, I'd base the PRs off of those
> > branches, not the for-next branch itself.
> 
> I see what you mean, but from another POV, you'd be basing a PR on top of one
> series of patches, not on top of everything.

Well, while some subsystems have tons of topic branches that turns into
a mess really quickly.  So what Darrick said makes the most sense in
general.  There might be occasional corner cases where you'd want to
be more fine grained, but they should be very rare.

> 
> Today, what we have, is a relatively stable for-next branch, where we just
> really rebase when something goes wrong, so, usually, when I push things into
> for-next, I've had it tested for a big while.
> 
> Per my conversations off-list (specially with hch), is that this shouldn't be
> the purpose at all of for-next, but a testing branch where (almost anything) can
> go wrong, within reason of course. Please correct me if I'm wrong here.

I would expect for-next to have some amount of sanity testing.  But the
idea is indeed to have the code integrated with other kernel changes
rather sooner than later.

> 
> At the same time, I wish we have a branch that everybody can work with, which
> contains 'everything' staged, ready to go, so I'd do all the merge between
> current and next release myself into such branch. I think having a branch ready
> for people to work with is a maintainer's job, and people shouldn't be bothered
> by trying to figure out which branch they should use to base their patches on
> top.
> 
> I'm hoping to use the master's branch for that if nobody has any objection.

Using master is really confusing.

As I said earlier and Darrick also said the most usual thing is
to have one branch for $CURRELEASE fixes and one for $NEXTRELEASE
development work.  for-next is then a temporary merge of those two.
If you need $CURRELEASE changes in $NEXTRELEASE to avoid a mess, you
either rebase $NEXTRELEASE (usually earlier in the merge window) or
pull the $CURRELEASE into $NEXTRELEASE with a well-documented
merge commit message documenting why it had to be done.

Re: Changes to XFS patch integration process

[Darrick J. Wong]

On Thu, Mar 06, 2025 at 06:08:41PM +0100, Christoph Hellwig wrote:
> On Thu, Mar 06, 2025 at 08:50:51AM +0100, Carlos Maiolino wrote:
> > > If I had a for-6.14 and a for-6.15 branch, I'd base the PRs off of those
> > > branches, not the for-next branch itself.
> > 
> > I see what you mean, but from another POV, you'd be basing a PR on top of one
> > series of patches, not on top of everything.
> 
> Well, while some subsystems have tons of topic branches that turns into
> a mess really quickly.  So what Darrick said makes the most sense in
> general.  There might be occasional corner cases where you'd want to
> be more fine grained, but they should be very rare.
> 
> > 
> > Today, what we have, is a relatively stable for-next branch, where we just
> > really rebase when something goes wrong, so, usually, when I push things into
> > for-next, I've had it tested for a big while.
> > 
> > Per my conversations off-list (specially with hch), is that this shouldn't be
> > the purpose at all of for-next, but a testing branch where (almost anything) can
> > go wrong, within reason of course. Please correct me if I'm wrong here.
> 
> I would expect for-next to have some amount of sanity testing.  But the
> idea is indeed to have the code integrated with other kernel changes
> rather sooner than later.
> 
> > 
> > At the same time, I wish we have a branch that everybody can work with, which
> > contains 'everything' staged, ready to go, so I'd do all the merge between
> > current and next release myself into such branch. I think having a branch ready
> > for people to work with is a maintainer's job, and people shouldn't be bothered
> > by trying to figure out which branch they should use to base their patches on
> > top.
> > 
> > I'm hoping to use the master's branch for that if nobody has any objection.
> 
> Using master is really confusing.
> 
> As I said earlier and Darrick also said the most usual thing is
> to have one branch for $CURRELEASE fixes and one for $NEXTRELEASE
> development work.  for-next is then a temporary merge of those two.
> If you need $CURRELEASE changes in $NEXTRELEASE to avoid a mess, you
> either rebase $NEXTRELEASE (usually earlier in the merge window) or
> pull the $CURRELEASE into $NEXTRELEASE with a well-documented
> merge commit message documenting why it had to be done.

Eliminating the possibility of such messes is also why I avoided doing
bugfixes and merge window prep whenever I could. ;)

--D

Re: Changes to XFS patch integration process

[Christoph Hellwig]

On Thu, Mar 06, 2025 at 09:40:12AM -0800, Darrick J. Wong wrote:
> Eliminating the possibility of such messes is also why I avoided doing
> bugfixes and merge window prep whenever I could. ;)

Thats a good idea in general, but xfs took it a bit too far.  We really
should hae had major changes in linux-next for a ew weeks and not in the
last minute.