[MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[James Bottomley]

I think the only point of agreement on this topic will be that how
bcachefs was handled wasn't correct at many levels.  I think this shows
we need more formality around feature inclusion, including a possible
probationary period and even things like mentorship and we definitely
need a formal process that extends beyond Linus for deciding we can no
longer work with someone any more.  I don't think anyone has the right
answer on this so (shock, horror), this is a genuine discussion topic
and one that would probably extend beyond the maintainer summit.  The
problem is that while the bcachefs saga did stray into CoC territory,
the fundamental issues were technical and community not around conduct
and I think a large part of the solution will involve discussing how
you mentor someone in community building and how you objectively
measure when they're failing to the extend that they're damaging
surrounding communities.

We probably also all have somewhat of an evolution of our positions on
this as well so I can start the ball rolling by detailing mine.  It's
public that I thought bcachefs shouldn't have gone in in the first
place:

https://lore.kernel.org/all/?q=f:bottomley%20s:bcachefs

for most of the problems it eventually caused.  However after mature
reflection, I think this was wrong: ab initio exclusion, even with
valid and evidence based reasons, will make us into a narrow minded and
ossified club.  I still think there should be discussion of the ab
initio problems but they should form part of the probation and
development plan for the feature, so everyone knows they always have a
chance to prove that they can do better than others thought at the
time.  It is probable that this probation and development plan can be
evolved at the time over email (I don't think one size fits all is ever
going to work for this) but a key point will be having at least one and
possibly more existing maintainers being responsible for executing it
(finding these people is going to be a challenge, I know).  The second
part is even more problematic: how do you measure forward progress
during the probationary period and judge whether the training wheels
should come off or the feature should be ejected?  If there's a clear
plan, then assessing progress against that solves some of the problem,
but not all and if the final decision is no instead of yes, there needs
to be a written down set of reasons for why this is (and possibly a
post mortem discussion of how everyone could do better next time
around).

However, I'm sure others will have different ideas.

Regards,

James

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Steven Rostedt]

On Thu, 21 Aug 2025 09:56:15 +0100
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

What exactly do you mean by "feature inclusion"?

Something that requires a new maintainer? As with the bcachefs, the issue
was with how the new maintainer worked with the current workflow.

Maybe you mean "maintainer inclusion and ejection"?

> However, I'm sure others will have different ideas.

The thing is, I believe there's a lot of features and maintainers that are
added. Most go unnoticed as the feature is a niche (much like bcachefs was).

Perhaps we should have a maintainer mentorship program. I try to work with
others to help them become a new maintainer. I was doing that with Daniel
Bristot, and I've done it for Masami Hiramatsu and I'm currently helping
others to become maintainers for the trace and verification tooling.

I share my scripts and explain how to do a pull request. How to use
linux-next and what to and more importantly, what not to send during during
the -rc releases.

I'm sure others have helped developers become maintainers as well. Perhaps
we should get together and come up with a formal way to become a maintainer?
Because honestly, it's currently done by trial and error. I think that
should change.

-- Steve

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[James Bottomley]

On Thu, 2025-08-21 at 12:27 -0400, Steven Rostedt wrote:
> On Thu, 21 Aug 2025 09:56:15 +0100
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> What exactly do you mean by "feature inclusion"?

It's deliberately vague. I am aware we include many new features (like
new SCSI or nvme drivers) every year that don't lead to this type of
conflict.  The last one in SCSI I recall was the aic94xx driver writer
demanding the SCSI subsystem change for his driver rather than vice
versa.
> 
> Something that requires a new maintainer? As with the bcachefs, the
> issue was with how the new maintainer worked with the current
> workflow.

That's not really it either: new drivers tend to get an entry in
MAINTAINERS as well.

> Maybe you mean "maintainer inclusion and ejection"?
> 
> > However, I'm sure others will have different ideas.

I think this is technical not personal, so I'd like to keep it at
features.

> The thing is, I believe there's a lot of features and maintainers
> that are added. Most go unnoticed as the feature is a niche (much
> like bcachefs was).

Yes, except the likely problems with bcachefs were pointed out ahead of
time so we had warning there were likely to be problems.

> Perhaps we should have a maintainer mentorship program. I try to work
> with others to help them become a new maintainer. I was doing that
> with Daniel Bristot, and I've done it for Masami Hiramatsu and I'm
> currently helping others to become maintainers for the trace and
> verification tooling.
> 
> I share my scripts and explain how to do a pull request. How to use
> linux-next and what to and more importantly, what not to send during
> during the -rc releases.

I'm not sure that covers it.  As I read the situation it was more about
how you work with others when there are things in the kernel you'd like
to introduce or change to support your feature.  Hence it's really
about working with rather than against the community.

> I'm sure others have helped developers become maintainers as well.
> Perhaps we should get together and come up with a formal way to
> become a maintainer? Because honestly, it's currently done by trial
> and error. I think that should change.

That wouldn't hurt, but that problem that I see is that some fairly
drastic action has been taken on what can be characterised as a whim,
so I think we need some formality around how and when this happens.

Regards,

James

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Steven Rostedt]

On Thu, 21 Aug 2025 18:44:07 +0100
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> > I share my scripts and explain how to do a pull request. How to use
> > linux-next and what to and more importantly, what not to send during
> > during the -rc releases.  
> 
> I'm not sure that covers it.  As I read the situation it was more about
> how you work with others when there are things in the kernel you'd like
> to introduce or change to support your feature.  Hence it's really
> about working with rather than against the community.

What I'm suggesting is to have a program to help newcomers that are taking
on a maintainer role. This program can not only teach what needs to be done
to be a maintainer, but also vet the people that are coming into our
ecosystem. If there's a lot of push back from the individual on how to
interact with the community, then that individual can be denied becoming a
maintainer.


> 
> > I'm sure others have helped developers become maintainers as well.
> > Perhaps we should get together and come up with a formal way to
> > become a maintainer? Because honestly, it's currently done by trial
> > and error. I think that should change.  
> 
> That wouldn't hurt, but that problem that I see is that some fairly
> drastic action has been taken on what can be characterised as a whim,
> so I think we need some formality around how and when this happens.

If it was policy for Kent to work with a mentor before he could send
patches directly to Linus, would this have uncovered the issues before they
became as large as they had become?

-- Steve

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[James Bottomley]

On Thu, 2025-08-21 at 15:15 -0400, Steven Rostedt wrote:
> On Thu, 21 Aug 2025 18:44:07 +0100
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > > I share my scripts and explain how to do a pull request. How to
> > > use linux-next and what to and more importantly, what not to send
> > > during during the -rc releases.  
> > 
> > I'm not sure that covers it.  As I read the situation it was more
> > about how you work with others when there are things in the kernel
> > you'd like to introduce or change to support your feature.  Hence
> > it's really about working with rather than against the community.
> 
> What I'm suggesting is to have a program to help newcomers that are
> taking on a maintainer role. This program can not only teach what
> needs to be done to be a maintainer, but also vet the people that are
> coming into our ecosystem. If there's a lot of push back from the
> individual on how to interact with the community, then that
> individual can be denied becoming a maintainer.

As I said, I think this is a good idea, it just wouldn't have solved
the problems we saw.  The initial pull request has a huge thread which,
when summarized, pretty much predicted the issues that were seen. 
Although he went into MAINTAINERS with an R tag, Brian Foster did
attempt to act as a buffer, something which I don't think we've thanked
Brian for enough, but something which ultimately failed probably due to
lack of empowerment.

> > > I'm sure others have helped developers become maintainers as
> > > well. Perhaps we should get together and come up with a formal
> > > way to become a maintainer? Because honestly, it's currently done
> > > by trial and error. I think that should change.  
> > 
> > That wouldn't hurt, but that problem that I see is that some fairly
> > drastic action has been taken on what can be characterised as a
> > whim, so I think we need some formality around how and when this
> > happens.
> 
> If it was policy for Kent to work with a mentor before he could send
> patches directly to Linus, would this have uncovered the issues
> before they became as large as they had become?

Well, no, the thread I pointed to as part of this proposal pretty much
predicted what actually happened.  So the problems were known ahead of
time and didn't need to be discovered, we just needed a better
mitigation mechanism (which a supervision program could form part of).

Regards,

James

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Paul Moore]

On Thu, Aug 21, 2025 at 12:35 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> Perhaps we should have a maintainer mentorship program. I try to work with
> others to help them become a new maintainer. I was doing that with Daniel
> Bristot, and I've done it for Masami Hiramatsu and I'm currently helping
> others to become maintainers for the trace and verification tooling.

I realize this wasn't the original focus of James' mail, my apologies
for continuing on the tangent, but I do think some form of a
maintainer/reviewer/developer/etc. mentorship program is a good idea.
Like Steven, and surely many others (staging tree?), I've done similar
things in the security space, and even in the most informal
arrangements I believe it has helped people get up to speed with our
somewhat unusual development practices and not-always-documented
processes.

I would expect the program to be fairly informal, especially at first,
with perhaps an hour every week or two where an existing maintainer
could work with a mentee off-list to answer questions, explain
process, code, or anything else relevant to kernel
development/maintenance.  Time zones would be a challenge for any
interactive discussions, but that's a common problem for community
development these days, and finding ways to resolve that would be an
important part of the mentorship.

--
paul-moore.com

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Mauro Carvalho Chehab]

Em Thu, 21 Aug 2025 12:27:50 -0400
Steven Rostedt <rostedt@goodmis.org> escreveu:

> On Thu, 21 Aug 2025 09:56:15 +0100
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> What exactly do you mean by "feature inclusion"?
> 
> Something that requires a new maintainer? As with the bcachefs, the issue
> was with how the new maintainer worked with the current workflow.
> 
> Maybe you mean "maintainer inclusion and ejection"?
> 
> > However, I'm sure others will have different ideas.  
> 
> The thing is, I believe there's a lot of features and maintainers that are
> added. Most go unnoticed as the feature is a niche (much like bcachefs was).

On a side note: I never used myself bcachefs, and I'm not aware of its
current status and how much it depends on the current maintainer.

Yet, IMO, I don't like the idea that, if a maintainer leaves the
project for whatever reason (including misbehavior), features would
be excluded - even if they're experimental.

So, I'd say that, except if we would be willing to face legal issues, 
or the feature is really bad, the best would be to give at least one
or two kernel cycles to see if someone else steps up - and if the
feature is experimental(*), perhaps move it to staging while nobody
steps up.


(*) where IMHO it should be sitting in the first place when it got
    merged, being an experimental feature.

> 
> Perhaps we should have a maintainer mentorship program. I try to work with
> others to help them become a new maintainer. I was doing that with Daniel
> Bristot, and I've done it for Masami Hiramatsu and I'm currently helping
> others to become maintainers for the trace and verification tooling.
> 
> I share my scripts and explain how to do a pull request. How to use
> linux-next and what to and more importantly, what not to send during during
> the -rc releases.
> 
> I'm sure others have helped developers become maintainers as well. Perhaps
> we should get together and come up with a formal way to become a maintainer?
> Because honestly, it's currently done by trial and error. I think that
> should change.

Agreed with training: this can help getting things right.

Thanks,
Mauro

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Matthew Wilcox]

On Fri, Aug 22, 2025 at 01:39:35PM +0200, Mauro Carvalho Chehab wrote:
> On a side note: I never used myself bcachefs, and I'm not aware of its
> current status and how much it depends on the current maintainer.
> 
> Yet, IMO, I don't like the idea that, if a maintainer leaves the
> project for whatever reason (including misbehavior), features would
> be excluded - even if they're experimental.
> 
> So, I'd say that, except if we would be willing to face legal issues, 
> or the feature is really bad, the best would be to give at least one
> or two kernel cycles to see if someone else steps up - and if the
> feature is experimental(*), perhaps move it to staging while nobody
> steps up.

Kent is bcachefs.  There's no team who might be able to step up, and
while the code is certainly clear enough, anyone who takes it over will
have to deal with Kent, an army of internet trolls and having to learn
an incredibly complex codebase.  I wouldn't wish that on anyone.

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Theodore Ts'o]

On Thu, Aug 21, 2025 at 09:56:15AM +0100, James Bottomley wrote:
> I think the only point of agreement on this topic will be that how
> bcachefs was handled wasn't correct at many levels.  I think this shows
> we need more formality around feature inclusion, including a possible
> probationary period and even things like mentorship and we definitely
> need a formal process that extends beyond Linus for deciding we can no
> longer work with someone any more.

I think we are conflating three different things in this discussion
thread, and it would be helpful if we separated them out.

  1.  What is the process by which a particular feature be included
         or ejected?
  2.  What is the process by which a developer should be excluded
         from the deevlopment community?  And this goes beyond
	 Code of Conduct violations, but in the case of a maintainer,
	 when that person has displayed toxic tendencies which are
	 sufficiently bad that other deevlopersa and maintainers refuse to
	 work with the individual, and when that person has been accused of
	 promoting a toxic environmet which is harming the entire
	 community?
  3.  The question of maintainer mentorship, which is very different
         from (2) as there are a large set of skills which a much broader
	 front including avoiding maintainer burnout, the product management
	 side of being a maintainer (e.g. working with companies to
	 motivate them to invest in a featrue which benefits not only the
	 companies' business interest, but the community as a whole),
	 managing volunteer, etc.

(2) is a very hard problem, and so there is a tendency to focus on
solving problems (1) and (2).  However, using bcachefs and its
maintainera as a motivating case for solutions to address (1) and (3)
very likely going to result in skewing the discussion around the best
ways of addressing (1) and (3).

As far as (2), our baseline way of handling things is quite ad hoc.
At the moment, individual developers will simply refuse to work
someone who is accused of being toxic, by doing things such as:

   (a) using mail filters to redirect e-mail from that person
       to /dev/null,
   (b) telling a higher-level maintainer that because of (a) they
       would appreciate it if any pull requests from that individual
       include changes to their subsystem or sub-subsysttem,
       that those commits should be presumed to be auto-NACK'ed,
       and requesting that the PULL request should be rejected,
   (c) if the behaviour of said person exasperates a higher-level
       maintainer to such an extent that the higher-level maintainer
       refuse to accept patches or pull requests from said
       individual, and
   (d) informing program committees of invite-only workshops and/or
       conferences that if that individual attends, they will refuse
       to attend because of that individual's toxicity.

I will note that (b) and (c) can be appealed to someone higher up on
the maintainer hierarchy, unless that higher-level maintainer is
Linus, at which point there is no higher level authority to take that
appeal, and that (b), (c), and (d) are effectivly a way that
developers and maintainers are effectively saying, "it's either him or
me!", and as someone who has to manage volunteers, if a sufficiently
large number of volunteers are sufficiently p*ssed off that they are
threatening to withdraw, the wise maintainer (or program committee)
should take heed.

Now, the above is inherently very messy.  But fortunately, it's only
happened once in thirty five years, and before we propose to put some
kind of mechanism in place, we need to make sure that the side effects
of that mechanism don't end up making things worse off.

There is the saying that "bad facts make bad law", and the specifics
of this most recent controversy are especially challenging.  I would
urge caution before trying to create a complex set of policies and
mechanim when we've only had one such corner case in over 35 years.

Cheers,

						- Ted

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Theodore Ts'o]

On Thu, Aug 21, 2025 at 04:34:07PM -0400, Theodore Ts'o wrote:
> (2) is a very hard problem, and so there is a tendency to focus on
> solving problems (1) and (2).

 "solving problems (1) and (3) instead."

Apologies for the typo.

						- Ted

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Matthew Wilcox]

On Thu, Aug 21, 2025 at 04:34:07PM -0400, Theodore Ts'o wrote:
> There is the saying that "bad facts make bad law", and the specifics
> of this most recent controversy are especially challenging.  I would
> urge caution before trying to create a complex set of policies and
> mechanim when we've only had one such corner case in over 35 years.

Well. we may have dodged a few bullets before now.  Just in filesystems,
I can think of Hans Reiser, Jeff Merkey, Boaz Harrosh, Daniel Phillips
(no, i'm not saying any of the others did anything as heinous as Hans,
but they were all pretty disastrous in their own ways).

I don't think we can necessarily generalise from these examples to,
say, Lustre.  That has its own unique challenges, and I don't think that
making them do more paperwork will be helpful.

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Mauro Carvalho Chehab]

Em Thu, 21 Aug 2025 23:21:38 +0100
Matthew Wilcox <willy@infradead.org> escreveu:

> On Thu, Aug 21, 2025 at 04:34:07PM -0400, Theodore Ts'o wrote:
> > There is the saying that "bad facts make bad law", and the specifics
> > of this most recent controversy are especially challenging.  I would
> > urge caution before trying to create a complex set of policies and
> > mechanim when we've only had one such corner case in over 35 years.  
> 
> Well. we may have dodged a few bullets before now.  Just in filesystems,
> I can think of Hans Reiser, Jeff Merkey, Boaz Harrosh, Daniel Phillips
> (no, i'm not saying any of the others did anything as heinous as Hans,
> but they were all pretty disastrous in their own ways).

There are other cases as well: there was a media driver maintainer that
did pretty bad things, including physical threats against other
maintainers. I even got a report that he did threat to life another
maintainer who complained he would be violating GPL copyrights.

> I don't think we can necessarily generalise from these examples to,
> say, Lustre.  That has its own unique challenges, and I don't think that
> making them do more paperwork will be helpful.

Agreed. I don't think those few examples have much in common:
each had different types of issues. So, I don't think any text
would be enough to cover such cases, as they're punctual.

Probably the only thing that could be more effective would be to have
an e-signed CLA for the ones which become maintainers.


Thanks,
Mauro

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[James Bottomley]

On Thu, 2025-08-21 at 16:34 -0400, Theodore Ts'o wrote:
> On Thu, Aug 21, 2025 at 09:56:15AM +0100, James Bottomley wrote:
> > I think the only point of agreement on this topic will be that how
> > bcachefs was handled wasn't correct at many levels.  I think this
> > shows
> > we need more formality around feature inclusion, including a
> > possible
> > probationary period and even things like mentorship and we
> > definitely
> > need a formal process that extends beyond Linus for deciding we can
> > no
> > longer work with someone any more.
> 
> I think we are conflating three different things in this discussion
> thread, and it would be helpful if we separated them out.
> 
>   1.  What is the process by which a particular feature be included
>          or ejected?
>   2.  What is the process by which a developer should be excluded
>          from the deevlopment community?  And this goes beyond
> Code of Conduct violations, but in the case of a maintainer,
> when that person has displayed toxic tendencies which are
> sufficiently bad that other deevlopersa and maintainers refuse to
> work with the individual, and when that person has been accused of
> promoting a toxic environmet which is harming the entire
> community?
>   3.  The question of maintainer mentorship, which is very different
>          from (2) as there are a large set of skills which a much
> broader
> front including avoiding maintainer burnout, the product management
> side of being a maintainer (e.g. working with companies to
> motivate them to invest in a featrue which benefits not only the
> companies' business interest, but the community as a whole),
> managing volunteer, etc.
> 
> (2) is a very hard problem, and so there is a tendency to focus on
> solving problems (1) and (2).  However, using bcachefs and its
> maintainera as a motivating case for solutions to address (1) and (3)
> very likely going to result in skewing the discussion around the best
> ways of addressing (1) and (3).

I agree that 1 (at least for inclusion) and 3 are pretty much in hand:
1 runs by itself naturally and Steve wants to do 3.

> As far as (2), our baseline way of handling things is quite ad hoc.
> At the moment, individual developers will simply refuse to work
> someone who is accused of being toxic, by doing things such as:
> 
>    (a) using mail filters to redirect e-mail from that person
>        to /dev/null,
>    (b) telling a higher-level maintainer that because of (a) they
>        would appreciate it if any pull requests from that individual
>        include changes to their subsystem or sub-subsysttem,
>        that those commits should be presumed to be auto-NACK'ed,
>        and requesting that the PULL request should be rejected,
>    (c) if the behaviour of said person exasperates a higher-level
>        maintainer to such an extent that the higher-level maintainer
>        refuse to accept patches or pull requests from said
>        individual, and
>    (d) informing program committees of invite-only workshops and/or
>        conferences that if that individual attends, they will refuse
>        to attend because of that individual's toxicity.
> 
> I will note that (b) and (c) can be appealed to someone higher up on
> the maintainer hierarchy, unless that higher-level maintainer is
> Linus, at which point there is no higher level authority to take that
> appeal, and that (b), (c), and (d) are effectivly a way that
> developers and maintainers are effectively saying, "it's either him
> or me!",

So what I saw is that as developers exercised this and effectively
disengaged unless directly attacked, it pretty much became all on Linus
because no-one was left in the chain. This is precisely where I think
we could do with an alternative mechanism.

> and as someone who has to manage volunteers, if a sufficiently
> large number of volunteers are sufficiently p*ssed off that they are
> threatening to withdraw, the wise maintainer (or program committee)
> should take heed.
> 
> Now, the above is inherently very messy.  But fortunately, it's only
> happened once in thirty five years, and before we propose to put some
> kind of mechanism in place, we need to make sure that the side
> effects of that mechanism don't end up making things worse off.

Well, what we ended up with is one person in the chain (Linus), no
actual decision except a failed pull request and nothing actually said
which has lead to a raft of internet speculation.

I agree whatever gets put in place will be messy, I just think it could
be a bit less messy and a bit more definitive.

> There is the saying that "bad facts make bad law", and the specifics
> of this most recent controversy are especially challenging.  I would
> urge caution before trying to create a complex set of policies and
> mechanim when we've only had one such corner case in over 35 years.

The banks were very keen on not being stress tested before 2008 and we
all saw where that lead. The crisis forced them to confront the issue,
which does argue that when something like this comes along you take the
opportunity to improve. I'm also not arguing for a labyrinthine
process. I think I'd be happy if we sort out two things

   1. That the decision be taken by more than one person rather than
      abdicating to last man standing
   2. The outcome be documented clearly.

Regards,

James

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Greg KH]

On Fri, Aug 22, 2025 at 09:09:04AM +0100, James Bottomley wrote:
> So what I saw is that as developers exercised this and effectively
> disengaged unless directly attacked, it pretty much became all on Linus
> because no-one was left in the chain. This is precisely where I think
> we could do with an alternative mechanism.

You are implying here that we all just "ran away" and left Linus to hold
the bag here, which is NOT the case at all.  This specific issue has
been discussed to death in a lot of different threads, public and
private with lots of people involved and none of that would have been
any different had we had some sort of "process document" ahead of time.

So I don't think that attempting to codify the very rare occurances like
this is going to really help out much, given that they are all unique to
their time/place/subsystem based on our past history like this.

> > Now, the above is inherently very messy.  But fortunately, it's only
> > happened once in thirty five years, and before we propose to put some
> > kind of mechanism in place, we need to make sure that the side
> > effects of that mechanism don't end up making things worse off.
> 
> Well, what we ended up with is one person in the chain (Linus), no
> actual decision except a failed pull request and nothing actually said
> which has lead to a raft of internet speculation.

It's not our job to quell "internet speculation", sorry.  Just because
we normally work in public for almost everything, doesn't mean that some
things can't be done in private as well.  And again, just because you
haven't seen a public decision doesn't mean that there hasn't been one
made :)

sorry,

greg k-h

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Rafael J. Wysocki]

On Fri, Aug 22, 2025 at 2:03 PM Greg KH <greg@kroah.com> wrote:
>
> On Fri, Aug 22, 2025 at 09:09:04AM +0100, James Bottomley wrote:
> > So what I saw is that as developers exercised this and effectively
> > disengaged unless directly attacked, it pretty much became all on Linus
> > because no-one was left in the chain. This is precisely where I think
> > we could do with an alternative mechanism.
>
> You are implying here that we all just "ran away" and left Linus to hold
> the bag here, which is NOT the case at all.  This specific issue has
> been discussed to death in a lot of different threads, public and
> private with lots of people involved and none of that would have been
> any different had we had some sort of "process document" ahead of time.
>
> So I don't think that attempting to codify the very rare occurrences like
> this is going to really help out much, given that they are all unique to
> their time/place/subsystem based on our past history like this.

I agree.

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Theodore Tso]

On Fri, Aug 22, 2025 at 02:03:13PM +0200, Greg KH wrote:
> 
> It's not our job to quell "internet speculation", sorry.  Just because
> we normally work in public for almost everything, doesn't mean that some
> things can't be done in private as well.  And again, just because you
> haven't seen a public decision doesn't mean that there hasn't been one
> made :)

The other thing I'll add here is that the best analogy I can think of
here is that this is a HR / Personnel issue.  These sorts of things,
whether they are a matter of someone not working well with the team
(at which point the manager needs to figure out how to resolve the
issue, and will often need to engage in private mediation /
interventions), or being caught on camera at a Coldplay concert, will
always have private conversations that will never be made part of the
public record --- as it should be, as much as content creators looking
for clickbait might wish otherwise.

Now, if what James is trying to say is that we could have avoided the
whole situation by refusing to allow bcachefs to be included in the
first place, I'm going to have to respectfully disagree with that
proposal as a way to avoid problems in the future.

I'm not sure that the fact that various developer-to-developer
relationships would have degraded to the point that it had by the end
of this whole saga could have been predicted at the point when we were
making the "to include or not to include bcachefs in Linux mainline".
I don't think we could have predicted whether or not a perspective
future maintainer would utterly refuse private offers of coaching from
the beginning.  And I don't think we should proactively refuse to
accept a feature just because someone's inter-personal relationships
are not perfect.

The current baseline is that the media subsystem, networking, or BPF
maintainer's decide what features to accept and who they will accept
pull requests from.  The same us true all the way up the hierarchy
maintainer tree up to Linus.  What is the alternative that we could
use?  That some democratic voting procedure, or some kind of core team
would stick their oar into making this decision?  I'm not sure that
would be an improvement; in fact, IMHO, it will very likely be
significantly worse.

I'm sure that as a result of this whole sitution, maintainers may very
well be more careful before accepting a new feature from a perspective
submaintainer who might be challenged in the teamwork department.  But
I'm not sure trying to codify this would be helpful --- because I
fundamentally disagree with the premise that we can accurately predict
how future stories will end.  Hindsight, after all, is 20/20.

If someone wants to suggest a concrete proposal, perhaps that's
something we can discuss.  But my personal opinion is having an
open-ended discussion of how we could have avoided the messiness of
bcachefs would probably not be a good use of time at the Maintainer's
Summit.

						- Ted

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[James Bottomley]

On Fri, 2025-08-22 at 08:24 -0400, Theodore Tso wrote:
> On Fri, Aug 22, 2025 at 02:03:13PM +0200, Greg KH wrote:
> > 
> > It's not our job to quell "internet speculation", sorry.  Just
> > because we normally work in public for almost everything, doesn't
> > mean that some things can't be done in private as well.  And again,
> > just because you haven't seen a public decision doesn't mean that
> > there hasn't been one made :)
> 
> The other thing I'll add here is that the best analogy I can think of
> here is that this is a HR / Personnel issue.  These sorts of things,
> whether they are a matter of someone not working well with the team
> (at which point the manager needs to figure out how to resolve the
> issue, and will often need to engage in private mediation /
> interventions), or being caught on camera at a Coldplay concert, will
> always have private conversations that will never be made part of the
> public record --- as it should be, as much as content creators
> looking for clickbait might wish otherwise.
> 
> Now, if what James is trying to say is that we could have avoided the
> whole situation by refusing to allow bcachefs to be included in the
> first place, I'm going to have to respectfully disagree with that
> proposal as a way to avoid problems in the future.

I did tackle that in the original proposal:

https://lore.kernel.org/all/fc0994de40776609928e8e438355a24a54f1ad10.camel@HansenPartnership.com/

But for those who didn't read it the precis is I said on reflection I
wouldn't ban anything at the inclusion stage even though that's what I
initially argued for

> I'm not sure that the fact that various developer-to-developer
> relationships would have degraded to the point that it had by the end
> of this whole saga could have been predicted at the point when we
> were making the "to include or not to include bcachefs in Linux
> mainline". I don't think we could have predicted whether or not a
> perspective future maintainer would utterly refuse private offers of
> coaching from the beginning.  And I don't think we should proactively
> refuse to accept a feature just because someone's inter-personal
> relationships are not perfect.

I think one takeaway is that there were a bunch of predictions which
ultimately turned out to be true and to make a success of it we should
have had a plan for coping with the foreseen issues.

> The current baseline is that the media subsystem, networking, or BPF
> maintainer's decide what features to accept and who they will accept
> pull requests from.  The same us true all the way up the hierarchy
> maintainer tree up to Linus.  What is the alternative that we could
> use?  That some democratic voting procedure, or some kind of core
> team would stick their oar into making this decision?  I'm not sure
> that would be an improvement; in fact, IMHO, it will very likely be
> significantly worse.

When did this become about how our current maintainer pull system
works?  I'm certainly not advocating changing it.

> I'm sure that as a result of this whole sitution, maintainers may
> very well be more careful before accepting a new feature from a
> perspective submaintainer who might be challenged in the teamwork
> department.  But I'm not sure trying to codify this would be helpful
> --- because I fundamentally disagree with the premise that we can
> accurately predict how future stories will end.  Hindsight, after
> all, is 20/20.

I don't think anyone made that premise.  However, when problems are
foreseen its not unreasonable to plan to try to overcome them and be
successful.

> If someone wants to suggest a concrete proposal, perhaps that's
> something we can discuss.  But my personal opinion is having an
> open-ended discussion of how we could have avoided the messiness of
> bcachefs would probably not be a good use of time at the Maintainer's
> Summit.

Well I did ask for two concrete things, but I can certainly repeat:

On Fri, 2025-08-22 at 09:09 +0100, James Bottomley wrote:
>  I think I'd be happy if we sort out two things
> 
>    1. That the decision be taken by more than one person rather than
>       abdicating to last man standing
>    2. The outcome be documented clearly.
> 

Regards,

James

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Theodore Tso]

On Fri, Aug 22, 2025 at 04:31:49PM +0100, James Bottomley wrote:
> 
> When did this become about how our current maintainer pull system
> works?  I'm certainly not advocating changing it.

  ...

> Well I did ask for two concrete things, but I can certainly repeat:
> 
> On Fri, 2025-08-22 at 09:09 +0100, James Bottomley wrote:
> >  I think I'd be happy if we sort out two things
> > 
> >    1. That the decision be taken by more than one person rather than
> >       abdicating to last man standing

Well, if we keep the current maintainer pull system, then the natural
consequence is that it *is* up to the higher-level maintainer to make
the decision.  In the case where we don't have maintainer teams, then
it will be a single person.

So I don't know how to square your request with your assertion that
you don't want to advocate changing how our current maintainer
hierarchy works.  Could you clarify your thinking here?

Thanks,

							- Ted

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Mauro Carvalho Chehab]

Em Fri, 22 Aug 2025 16:31:49 +0100
James Bottomley <James.Bottomley@HansenPartnership.com> escreveu:

> Well I did ask for two concrete things, but I can certainly repeat:
> 
> On Fri, 2025-08-22 at 09:09 +0100, James Bottomley wrote:
> >  I think I'd be happy if we sort out two things
> > 
> >    1. That the decision be taken by more than one person rather than
> >       abdicating to last man standing
> >    2. The outcome be documented clearly.

There are some aspects here:

- Who will communicate the decision? 

The way I see, the best would be if this would be done by the subsystem 
maintainers who accepted/acked the feature addition.

- Who will be involved on such discussion?

I'd say the subsystem core maintainers and developers plus the top 
maintainer and eventually TAB. Feature removal may cause troubles 
to distro maintainers, as some may have it enabled as well. So,
better having more people know in advance.

- How this will be documented?

Depending on the reasons why a feature is dropped, e.g. if it
involves personal data, I don't think the entire process can be
transparent, but surely a sanitized summary should be documented.

IMHO, the best way to document it is at the patch dropping such
feature, which will explain why the feature is removed. IMO, the
best would be to have such patch containing SOB from multiple
people:

- core subsystem developers and maintainers;
- Ack or SOB by the top level maintainers, if pertinent.

Thanks,
Mauro

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[Mauro Carvalho Chehab]

Em Fri, 22 Aug 2025 08:24:24 -0400
"Theodore Tso" <tytso@mit.edu> escreveu:

> On Fri, Aug 22, 2025 at 02:03:13PM +0200, Greg KH wrote:

> The current baseline is that the media subsystem, networking, or BPF
> maintainer's decide what features to accept and who they will accept
> pull requests from.  

On media, we typically place things that deserve more discussion under
staging. We did that for stateful decoders and encoders, for instance.
The same was done for stateless codecs. It means that any drivers written
to use such features also go to staging. Not always possible, but
something like that IMO serves to signalize to users, distro-maintainers
and the maintainers of such feature that, while we're ok to have it
being tested, we're yet seeing issues that need more discussions and/or
fixes.


> The same us true all the way up the hierarchy
> maintainer tree up to Linus.  What is the alternative that we could
> use?  That some democratic voting procedure, or some kind of core team
> would stick their oar into making this decision?  I'm not sure that
> would be an improvement; in fact, IMHO, it will very likely be
> significantly worse.

Agreed. It is really hard to see when problems will arise, specially
in cases like this.

Thanks,
Mauro

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[James Bottomley]

On Fri, 2025-08-22 at 14:03 +0200, Greg KH wrote:
> On Fri, Aug 22, 2025 at 09:09:04AM +0100, James Bottomley wrote:
> > So what I saw is that as developers exercised this and effectively
> > disengaged unless directly attacked, it pretty much became all on
> > Linus because no-one was left in the chain. This is precisely where
> > I think we could do with an alternative mechanism.
> 
> You are implying here that we all just "ran away" and left Linus to
> hold the bag here, which is NOT the case at all.  This specific issue
> has been discussed to death in a lot of different threads, public and
> private with lots of people involved and none of that would have been
> any different had we had some sort of "process document" ahead of
> time.

I didn't ask for a process document.  I was clear about what I was
asking for in the part of the email you cut in your reply.

> So I don't think that attempting to codify the very rare occurances
> like this is going to really help out much, given that they are all
> unique to their time/place/subsystem based on our past history like
> this.
> 
> > > Now, the above is inherently very messy.  But fortunately, it's
> > > only happened once in thirty five years, and before we propose to
> > > put some kind of mechanism in place, we need to make sure that
> > > the side effects of that mechanism don't end up making things
> > > worse off.
> > 
> > Well, what we ended up with is one person in the chain (Linus), no
> > actual decision except a failed pull request and nothing actually
> > said which has lead to a raft of internet speculation.
> 
> It's not our job to quell "internet speculation", sorry.

I didn't say it was.

>   Just because we normally work in public for almost everything,
> doesn't mean that some things can't be done in private as well.  And
> again, just because you haven't seen a public decision doesn't mean
> that there hasn't been one made :)

I get that in the current political climate transparency is taking a
back seat.  However, it does lie at the heart of the open in open
source so I think we should be making a bit more effort to be better.

Being transparent would have controlled (not quelled because there's
always conspiracy theorists) the internet speculation not because it
would make it someone's job but because it's simply a natural
consequence of doing the right thing.

Regards,

James

Re: [MAINTAINER SUMMIT] Adding more formality around feature inclusion and ejection

[dan.j.williams]

James Bottomley wrote:
> I think the only point of agreement on this topic will be that how
> bcachefs was handled wasn't correct at many levels.  I think this shows
> we need more formality around feature inclusion, including a possible
> probationary period and even things like mentorship and we definitely
> need a formal process that extends beyond Linus for deciding we can no
> longer work with someone any more.

A different perspective, the informal, albeit messy, process eventually
arrived at an outcome that does put project health first. There is a
risk here of over-indexing on a proactive formal process for what is an
infrequent, latent, and emergent problem. Look, sometimes it is not
clear that an individual will continually fail to respect personal and
community boundaries until they repeatedly fail to respect personal and
community boundaries.

The change I hope that comes from this is indeed more maturity and
courage around boundary setting. It reinforces a lesson it took me a
while to learn in my career: technical correctness and brilliant ideas
are necessary but insufficient for moving Linux forward. This community
does not lack for talent and ideas. Maintaining trust and collaboration,
that is the hard work of Linux.

This anecdote from Pat rang in my ears this past week:

"You need to be right less and effective more!" [1]

[1]: https://www.linkedin.com/posts/patgelsinger_sometimes-its-not-about-being-right-its-activity-7361475388390191104-rFVb