Re: [MAINTAINERS/KERNEL SUMMIT] Trust and maintenance of file systems

[Kent Overstreet <kent.overstreet@linux.dev>]

On Sun, Sep 10, 2023 at 03:51:42PM -0400, James Bottomley wrote:
> OK, so now we've strayed into the causes of maintainer burnout.  Syzbot
> is undoubtedly a stressor, but one way of coping with a stressor is to
> put it into perspective: Syzbot is really a latter day coverity and
> everyone was much happier when developers ignored coverity reports and
> they went into a dedicated pile that was looked over by a team of
> people trying to sort the serious issues from the wrong but not
> exploitable ones.  I'd also have to say that anyone who allows older
> filesystems into customer facing infrastructure is really signing up
> themselves for the risk they're running, so I'd personally be happy if
> older fs teams simply ignored all the syzbot reports.

The problem with syzbot, and fuzzing in general is that reports come out
at random which makes it impossible to pick a think and work on it, i.e.
focus on the task at hand.

To be able to work productively, it's critical that we be able to find
out if our code is broken /when we're still working on it/, which means
getting quick testing feedback. Failing that, if we have to go back and
fix up old code, we really want to be able to look at a file/module/some
reasonably sized chunk, load it up into our brains, fix up what's wrong,
and move onto the next thing.

Syzbot is the absolute worst for developer productivity.

I've been talking about code coverage analysis as a partial replacement
for fuzz testing because you can look at the report for a file, figure
out what tests are missing, and do the work all at once. We'll never
catch all the bugs fuzz testing will find that way, but anything that
reduces our reliance on it would be a good thing.

The real long term solution is, of course, to start rewriting stuff in
Rust.

> > Burnout amongst fs maintainers is a real problem.  I have no idea how
> > to solve it.
> 
> I already suggested we should share coping strategies:
> 
> https://lore.kernel.org/ksummit/ab9cfd857e32635f626a906410ad95877a22f0db.camel@HansenPartnership.com/
> 
> The sources of stress aren't really going to decrease, but how people
> react to them could change.  Syzbot (and bugs in general) are a case in
> point.  We used not to treat seriously untriaged bug reports, but now
> lots of people feel they can't ignore any fuzzer report.  We've tipped
> to far into "everything's a crisis" mode and we really need to come
> back and think that not every bug is actually exploitable or even
> important.

Yeah, burnout is a symptom of too many impossible to meet priorities;
the solution is to figure out what our priorities actually are.

As the codebases I've written have grown, my own mentality has
shifted... when I was younger, every bug was something that had to be
fixed. Now I have to think more in terms of "how much time am I spending
fixing bugs, which bugs am I fixing, and how do I balance that against
my long term priorities".

in particular, the stuff that shows up in a dashboard may be the
/easiest/ to work on - it's in a nice todo list! - but if I spent all my
time on that I wouldn't get to the bugs and issues users are reporting.

Of course, if users are reporting too many bugs that means test coverage
is missing or the automated tests aren't being looked at enough, so it's
a real balancing act.

The other big change in my thinking has been going from trying to fix
every bug when I first see it (and at times going through real heroics
to do so) - to now trying to focus more on making debugging easy; if I
can't figure out a bug right away I'll often add more assertions/debug
output and wait for it to pop next time. That kind of thing has a real
long term impact; the thing I strive for is a codebase where when
something goes wrong it tells you /everything/ about what went wrong.