Next LTS release

Next LTS release

[Mason]

Hello,

I read the kernel summit summary.
https://lwn.net/Articles/662966/

I was wondering if the 4.4 release set a "precedent" in that the
next few LTS releases going forward would be aimed for early Q1?

The reason I'm asking is that my company plans to release an SDK
in Q4, and I was hoping to bundle a recent kernel.

I currently provide 4.4 but most of my port was accepted in later
versions. I was hoping I wouldn't need to backport all that code.

What do the tea leaves say?

Regards.

Re: Next LTS release

[Greg KH]

On Mon, Jun 06, 2016 at 09:36:47AM +0200, Mason wrote:
> Hello,
> 
> I read the kernel summit summary.
> https://lwn.net/Articles/662966/
> 
> I was wondering if the 4.4 release set a "precedent" in that the
> next few LTS releases going forward would be aimed for early Q1?

Probably.

> The reason I'm asking is that my company plans to release an SDK
> in Q4, and I was hoping to bundle a recent kernel.

Nothing preventing you from always just advancing your kernel to the
latest LTS one when it comes out, right?  You shouldn't have anything
tied to a specific kernel version if all is working well :)

> I currently provide 4.4 but most of my port was accepted in later
> versions. I was hoping I wouldn't need to backport all that code.
> 
> What do the tea leaves say?

Something in Q1, no idea what number specifically, but if all of your
code is already merged upstream, it doesn't really matter what you pick,
anything will end up working for you.

thanks,

greg k-h

Re: Next LTS release

[Mason]

On 08/06/2016 03:56, Greg KH wrote:
> On Mon, Jun 06, 2016 at 09:36:47AM +0200, Mason wrote:
>> Hello,
>>
>> I read the kernel summit summary.
>> https://lwn.net/Articles/662966/
>>
>> I was wondering if the 4.4 release set a "precedent" in that the
>> next few LTS releases going forward would be aimed for early Q1?
> 
> Probably.

Bummer.

>> The reason I'm asking is that my company plans to release an SDK
>> in Q4, and I was hoping to bundle a recent kernel.
> 
> Nothing preventing you from always just advancing your kernel to the
> latest LTS one when it comes out, right?

Some customers, once they have something working, they don't want
to change anything. If I release a v4.4-based SDK, and then such
a customer comes along, I'll be stuck having to support v4.4 until
my beard is grey.
(I was told we are still expected to support several 2.6 kernels.)

>> I currently provide 4.4 but most of my port was accepted in later
>> versions. I was hoping I wouldn't need to backport all that code.
>>
>> What do the tea leaves say?
> 
> Something in Q1, no idea what number specifically, but if all of your
> code is already merged upstream, it doesn't really matter what you pick,
> anything will end up working for you.

The platform code got merged in 4.5 and 4.6 but I'm still missing
several important (but non-essential) drivers. Thus 4.4 isn't really
helpful to me, I'd have to port everything.

By the way, looking at the LTS releases, it seems some (older) releases
are supported longer than others (3.16, 3.12, 3.10, 3.4, 3.2). Is that
because some are supported by other groups/distros? (IIUC, many Android
devices run 3.4 and 3.10)

https://www.kernel.org/category/releases.html

Regards.

Re: Next LTS release

[Greg KH]

On Wed, Jun 08, 2016 at 02:59:33PM +0200, Mason wrote:
> On 08/06/2016 03:56, Greg KH wrote:
> > On Mon, Jun 06, 2016 at 09:36:47AM +0200, Mason wrote:
> >> Hello,
> >>
> >> I read the kernel summit summary.
> >> https://lwn.net/Articles/662966/
> >>
> >> I was wondering if the 4.4 release set a "precedent" in that the
> >> next few LTS releases going forward would be aimed for early Q1?
> > 
> > Probably.
> 
> Bummer.

Can't please everyone, sorry :)

> >> The reason I'm asking is that my company plans to release an SDK
> >> in Q4, and I was hoping to bundle a recent kernel.
> > 
> > Nothing preventing you from always just advancing your kernel to the
> > latest LTS one when it comes out, right?
> 
> Some customers, once they have something working, they don't want
> to change anything. If I release a v4.4-based SDK, and then such
> a customer comes along, I'll be stuck having to support v4.4 until
> my beard is grey.
> (I was told we are still expected to support several 2.6 kernels.)

You can tell them that they are running insecure kernels that are
trivial to break into, and provide them with the latest kernel release
to resolve that.

In fact, your contracts might require that, given that everyone knows we
are fixing about 10 things every day in newer stable kernels, and 2-3
things every day in very old long-term kernels.

> >> I currently provide 4.4 but most of my port was accepted in later
> >> versions. I was hoping I wouldn't need to backport all that code.
> >>
> >> What do the tea leaves say?
> > 
> > Something in Q1, no idea what number specifically, but if all of your
> > code is already merged upstream, it doesn't really matter what you pick,
> > anything will end up working for you.
> 
> The platform code got merged in 4.5 and 4.6 but I'm still missing
> several important (but non-essential) drivers. Thus 4.4 isn't really
> helpful to me, I'd have to port everything.

Then pick 4.6 and support it and then give them 4.7, and 4.8, and so on.
It's not hard, and everything will still work just fine, that's a
guarantee we made a long time ago and it's worked.

> By the way, looking at the LTS releases, it seems some (older) releases
> are supported longer than others (3.16, 3.12, 3.10, 3.4, 3.2). Is that
> because some are supported by other groups/distros? (IIUC, many Android
> devices run 3.4 and 3.10)

Those 3.4 and 3.10 Android releases are not getting updates, they just
happened to start out with the LTS kernel, so really, it didn't matter
what kernel they picked :(

Some are supported longer than others because we have developers willing
to do that support, for various different reasons (usually because it
makes their day-job easier).  There's no real specific reason to justify
any of the time lengths for them, sorry.

hope this helps,

greg k-h

Re: Next LTS release

[Willy Tarreau]

Just chiming in based on my experience with old kernels.

On Wed, Jun 08, 2016 at 08:22:38AM -0700, Greg KH wrote:
> > >> The reason I'm asking is that my company plans to release an SDK
> > >> in Q4, and I was hoping to bundle a recent kernel.
> > > 
> > > Nothing preventing you from always just advancing your kernel to the
> > > latest LTS one when it comes out, right?
> > 
> > Some customers, once they have something working, they don't want
> > to change anything. If I release a v4.4-based SDK, and then such
> > a customer comes along, I'll be stuck having to support v4.4 until
> > my beard is grey.
> > (I was told we are still expected to support several 2.6 kernels.)
> 
> You can tell them that they are running insecure kernels that are
> trivial to break into, and provide them with the latest kernel release
> to resolve that.

FWIW I just checked, and since we dropped 2.6.32.y 3 months ago, at least
2-3 null pointer dereferences affect it, that can be used either just to
crash the system, or even to gain privileges under certain conditions.

There is a good rule of thumb : if your kernel doesn't appear anymore
on https://www.kernel.org/category/releases.html, YOU'RE IN VERY SERIOUS
TROUBLE!

> > The platform code got merged in 4.5 and 4.6 but I'm still missing
> > several important (but non-essential) drivers. Thus 4.4 isn't really
> > helpful to me, I'd have to port everything.
> 
> Then pick 4.6 and support it and then give them 4.7, and 4.8, and so on.
> It's not hard, and everything will still work just fine, that's a
> guarantee we made a long time ago and it's worked.

And even to limit the number of upgrades, first jump from one version to
another until one is declared LTS and you can stick to it for as long as
it is supported. Customers will prefer to run on a well-known LTS version
than one you maintain yourself anyway, so they'll be willing to accept
these small update steps for a few versions at the beginning. And they'll
benefit from improvements.

> Some are supported longer than others because we have developers willing
> to do that support, for various different reasons (usually because it
> makes their day-job easier).  There's no real specific reason to justify
> any of the time lengths for them, sorry.

I personally take over maintenance of the kernels we ship with our
products, and we ensure to pick LTS kernels to save me from doing
useless work. In the past we used to have quite a number of backports,
and by catching up with more recent LTS versions we could get rid of a
lot of patches and simplify our job. It also means that I have less
extended LTS work to do, which is good considering that the difficulty
grows with time. I definitely think that most users who absolutely need
LTS kernels should proceed similarly, and that those who don't absolutely
need LTS should use a recent enough kernel that is still actively maintained.

I'd also like to mention something regarding very long time maintenance
for having worked on quite old kernels for some time (including 2.4).
There's a point where the quality starts to decrease again due to bugs
introduced in backports, just because the code is so much different that
it's hard to get a backport right. I can't say how long it takes for a
kernel to be screwed, but I definitely screwed 2.4 a few times, and would
have screwed 2.6.32 many times if some careful reviewers had not spotted
my mistakes in time. Reviewers will not participate forever to an antique
release, and there will even be a moment where they will make mistakes as
well. Just for this it's important to force customers to upgrade. Establish
a 3-4 years plan if that works for you, but you need to enforce a deadline.

Just my two cents,
Willy

Re: Next LTS release

[Mason]

On 08/06/2016 20:14, Willy Tarreau wrote:

> On Wed, Jun 08, 2016 at 08:22:38AM -0700, Greg KH wrote:
>
>> You can tell them that they are running insecure kernels that are
>> trivial to break into, and provide them with the latest kernel release
>> to resolve that.
> 
> FWIW I just checked, and since we dropped 2.6.32.y 3 months ago, at least
> 2-3 null pointer dereferences affect it, that can be used either just to
> crash the system, or even to gain privileges under certain conditions.

Would you believe me if I told you that we provide kernel version
3.4.39 because "applying security fixes breaks compatibility with
binary kernel modules" ?

What's worse, some customers agree with that "logic".

Regards.

Re: Next LTS release

[Willy Tarreau]

On Wed, Jun 08, 2016 at 11:39:34PM +0200, Mason wrote:
> On 08/06/2016 20:14, Willy Tarreau wrote:
> 
> > On Wed, Jun 08, 2016 at 08:22:38AM -0700, Greg KH wrote:
> >
> >> You can tell them that they are running insecure kernels that are
> >> trivial to break into, and provide them with the latest kernel release
> >> to resolve that.
> > 
> > FWIW I just checked, and since we dropped 2.6.32.y 3 months ago, at least
> > 2-3 null pointer dereferences affect it, that can be used either just to
> > crash the system, or even to gain privileges under certain conditions.
> 
> Would you believe me if I told you that we provide kernel version
> 3.4.39 because "applying security fixes breaks compatibility with
> binary kernel modules" ?

Oh I totally believe you, don't worry.

> What's worse, some customers agree with that "logic".

Yes there are plenty of such customers hosting botnets and spam relays
who are not aware of it. And when they sell products based on such kernels,
it's end users who are exposed. And generally these are the same who want
all the features they believe they'll need so you can't even cross fingers
for their kernel not to enable the dangerous features.

Sometimes you just need to throw the towel and work for another company
where you won't see these stupid customers anymore. When developers
willing to do this job will become rare, either they'll get paid a lot
for a really boring job or customers will start to think how to cut costs
by using less dangerous components that are more easily maintained.

Willy