Re: Regressions in stable releases

[Greg Kroah-Hartman <gregkh@linuxfoundation.org>]

On Thu, Aug 05, 2021 at 12:44:54PM -0700, Guenter Roeck wrote:
> On Thu, Aug 05, 2021 at 07:43:20PM +0200, Greg Kroah-Hartman wrote:
> > On Thu, Aug 05, 2021 at 10:39:22AM -0700, Guenter Roeck wrote:
> > > On Thu, Aug 05, 2021 at 06:19:15PM +0200, Greg Kroah-Hartman wrote:
> > > > On Thu, Aug 05, 2021 at 09:11:02AM -0700, Guenter Roeck wrote:
> > > > > Hi folks,
> > > > > 
> > > > > we have (at least) two severe regressions in stable releases right now.
> > > > > 
> > > > > [SHAs are from linux-5.10.y]
> > > > > 
> > > > > 2435dcfd16ac spi: mediatek: fix fifo rx mode
> > > > > 	Breaks SPI access on all Mediatek devices for small transactions
> > > > > 	(including all Mediatek based Chromebooks since they use small SPI
> > > > > 	 transactions for EC communication)
> > > > > 
> > > > > 60789afc02f5 Bluetooth: Shutdown controller after workqueues are flushed or cancelled
> > > > > 	Breaks Bluetooth on various devices (Mediatek and possibly others)
> > > > > 	Discussion: https://lkml.org/lkml/2021/7/28/569
> > > > 
> > > > Are either of these being tracked on the regressions list?  I have not
> > > > noticed them being reported there, or on the stable list :(
> > > > 
> > > 
> > > I wasn't aware of regressions@lists.linux.dev. Clueless me. And this is the
> > > report on the stable list, or at least that was the idea. Should I send separate
> > > emails to regressions@ with details ?
> > 
> > For regressions in Linus's tree, yes please do.  I have seen many stable
> > regressions also sent there as they mirror regressions in Linus's tree
> > (right now there is at least one ACPI regression that hopefully will
> > show up in Linus's tree soon that has hit stable as well..)
> > 
> > > > > Unfortunately, it appears that all our testing doesn't cover SPI and Bluetooth.
> > > > > 
> > > > > I understand that upstream is just as broken until fixes are applied there.
> > > > > Still, it shows that our test coverage is far from where it needs to be,
> > > > > and/or that we may be too aggressive with backporting patches to stable
> > > > > releases.
> > > > > 
> > > > > If you have an idea how to improve the situation, please let me know.
> > > > 
> > > > We need to get tests running in kernelci on real hardware, that's going
> > > > to be much more helpful here.
> > > > 
> > > 
> > > Yes, I know. Of course it didn't help that our internal testing didn't catch
> > > the problem until after the fact either.
> > 
> > There will always be issues that can only be caught on real hardware, we
> > are all just human.  The goal is to handle them quickly when they are
> > caught.
> > 
> > I'll go revert the above commits now, thanks.
> 
> If you do that, it needs to be done all the way to v4.19.y.

4.9.y for the first one, 4.4.y for the second.

thanks,

greg k-h