stable-security kernel updates

stable-security kernel updates

[Sasha Levin]

[-- Attachment #1.1: Type: text/plain, Size: 391 bytes --]

Hi all,

Updates for stable-security kernels have been released:

	- v3.12.58-security
	- v3.14.67-security
	- v3.18.31-security
	- v4.1.22-security
	- v4.4.8-security
	- v4.5.2-security

They are available at:

	https://git.kernel.org/cgit/linux/kernel/git/sashal/linux-stable-security.git/

Or:

	https://github.com/sashalevin/linux-stable-security


Thanks,
Sasha


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Jiri Slaby]

[-- Attachment #1.1: Type: text/plain, Size: 696 bytes --]

On 04/20/2016, 09:50 PM, Sasha Levin wrote:
> Updates for stable-security kernels have been released:
> 
> 	- v3.12.58-security

I suggest nobody uses that kernel.

That tree does not make much sense to me. For example, what's the
purpose of "kernel: Provide READ_ONCE and ASSIGN_ONCE" (commit
230fa253df6352af12ad0a16128760b5cb3f92df upstream) without actually
using the added macros (this commit was only a prerequisite)?

Ok, not that bad, it is only unused code, but why are *not* these in the
security tree?
ipr: Fix out-of-bounds null overwrite
Input: powermate - fix oops with malicious USB descriptors
rapidio/rionet: fix deadlock on SMP

thanks,
-- 
js
suse labs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Willy Tarreau]

Hi Jiri,

On Thu, Apr 21, 2016 at 08:43:55AM +0200, Jiri Slaby wrote:
> On 04/20/2016, 09:50 PM, Sasha Levin wrote:
> > Updates for stable-security kernels have been released:
> > 
> > 	- v3.12.58-security
> 
> I suggest nobody uses that kernel.
> 
> That tree does not make much sense to me. For example, what's the
> purpose of "kernel: Provide READ_ONCE and ASSIGN_ONCE" (commit
> 230fa253df6352af12ad0a16128760b5cb3f92df upstream) without actually
> using the added macros (this commit was only a prerequisite)?
> 
> Ok, not that bad, it is only unused code, but why are *not* these in the
> security tree?
> ipr: Fix out-of-bounds null overwrite
> Input: powermate - fix oops with malicious USB descriptors
> rapidio/rionet: fix deadlock on SMP

This illustrates exactly what I suspected would happen because that's the
same trouble we all face when picking backports for our respective trees
except that since the selection barrier is much higher here, lots of
important ones will be missing.

Cheers,
Willy

Re: stable-security kernel updates

[Sasha Levin]

Hey Willy,

On 04/21/2016 03:11 AM, Willy Tarreau wrote:
> This illustrates exactly what I suspected would happen because that's the
> same trouble we all face when picking backports for our respective trees
> except that since the selection barrier is much higher here, lots of
> important ones will be missing

Right. I fully agree that there will be important security commits that'll
get missed, whether because they were missed in the stable selection or
the stable-security selection.

I'd like to point out again that updating the entire stable tree is the
preferable way to patch against security (and non-security) issues. The
stable-security tree is a best-effort solution to provide a stop-gap in
between said stable tree updates.


Thanks,
Sasha

Re: stable-security kernel updates

[Greg KH]

On Thu, Apr 21, 2016 at 07:27:39AM -0400, Sasha Levin wrote:
> Hey Willy,
> 
> On 04/21/2016 03:11 AM, Willy Tarreau wrote:
> > This illustrates exactly what I suspected would happen because that's the
> > same trouble we all face when picking backports for our respective trees
> > except that since the selection barrier is much higher here, lots of
> > important ones will be missing
> 
> Right. I fully agree that there will be important security commits that'll
> get missed, whether because they were missed in the stable selection or
> the stable-security selection.
> 
> I'd like to point out again that updating the entire stable tree is the
> preferable way to patch against security (and non-security) issues.

s/preferable/only/ :)

> The
> stable-security tree is a best-effort solution to provide a stop-gap in
> between said stable tree updates.

What are you "stop-gapping" then?  The 7-10 days between stable
releases?

confused,

greg k-h

Re: stable-security kernel updates

[Sasha Levin]

On 04/21/2016 08:36 AM, Greg KH wrote:
> On Thu, Apr 21, 2016 at 07:27:39AM -0400, Sasha Levin wrote:
>> Hey Willy,
>>
>> On 04/21/2016 03:11 AM, Willy Tarreau wrote:
>>> This illustrates exactly what I suspected would happen because that's the
>>> same trouble we all face when picking backports for our respective trees
>>> except that since the selection barrier is much higher here, lots of
>>> important ones will be missing
>>
>> Right. I fully agree that there will be important security commits that'll
>> get missed, whether because they were missed in the stable selection or
>> the stable-security selection.
>>
>> I'd like to point out again that updating the entire stable tree is the
>> preferable way to patch against security (and non-security) issues.
> 
> s/preferable/only/ :)

Really? Even though as I showed updating your stable tree religiously would
still leave you vulnerable to "ancient" privesc exploits?

If anything, the *only* way is updating the entire kernel tree.

>> The
>> stable-security tree is a best-effort solution to provide a stop-gap in
>> between said stable tree updates.
> 
> What are you "stop-gapping" then?  The 7-10 days between stable
> releases?

In a perfect world where everyone has a team of kernel hackers on hand
reviewing stable commits, verifying the resulting kernel doesn't regress
their product, and fixes existing regressions for their product it might
be 7-10 days.

In the real world, this process takes much longer.

Doing a full rebase of the kernel tree is a much more costly process than
cherry picking a handful of security commits.


Thanks,
Sasha

Re: stable-security kernel updates

[Willy Tarreau]

On Thu, Apr 21, 2016 at 10:01:29AM -0400, Sasha Levin wrote:
> > What are you "stop-gapping" then?  The 7-10 days between stable
> > releases?
> 
> In a perfect world where everyone has a team of kernel hackers on hand
> reviewing stable commits, verifying the resulting kernel doesn't regress
> their product, and fixes existing regressions for their product it might
> be 7-10 days.
> 
> In the real world, this process takes much longer.
> 
> Doing a full rebase of the kernel tree is a much more costly process than
> cherry picking a handful of security commits.

Usually what is being done is mostly to check the intersection areas
between local patches and the updated parts from the next kernel. I'm
not saying it doesn't take some time, I mean for most products, only
certain areas are being considered since you usually have lots of
"CONFIG_* is not set" in a product. It's totally different for a distro
however.

Regards,
Willy

Re: stable-security kernel updates

[Sasha Levin]

[-- Attachment #1.1: Type: text/plain, Size: 2237 bytes --]

On 04/21/2016 02:43 AM, Jiri Slaby wrote:
> On 04/20/2016, 09:50 PM, Sasha Levin wrote:
>> Updates for stable-security kernels have been released:
>>
>> 	- v3.12.58-security
> 
> I suggest nobody uses that kernel.
> 
> That tree does not make much sense to me. For example, what's the
> purpose of "kernel: Provide READ_ONCE and ASSIGN_ONCE" (commit
> 230fa253df6352af12ad0a16128760b5cb3f92df upstream) without actually
> using the added macros (this commit was only a prerequisite)?

Looking at this, I believe that my scripts failed to merge the
follow up commit, and I missed that. I'll improve this so it won't
happen in the future. Thank you for this report.

> Ok, not that bad, it is only unused code, but why are *not* these in the
> security tree?
> ipr: Fix out-of-bounds null overwrite

Is there a particular way to exploit this that I'm missing?

> Input: powermate - fix oops with malicious USB descriptors

This requires physical access to the machine.

> rapidio/rionet: fix deadlock on SMP

Seemed a bit borderline I suppose. There's nothing specific the
user can do to actually trigger this?


Another thing to note here is that security patch selection database
is shared between versions, so if a given commit gets marked as security
later on (someone figured out it's a CVE or something similar), it'll
get added to the stable-security tree even if it was initially skipped.


So I've also ended up auditing the 3.12 for missing CVE fixes and these
ones ended up being at the top of the list. Could you explain why they
are not in the 3.12 stable tree (and as a result can't get to users of
the corresponding stable-security tree)?

(CVE-2015-7513) 0185604 KVM: x86: Reload pit counters for all channels when restoring state
(CVE-2015-8539) 096fe9e KEYS: Fix handling of stored error in a negatively instantiated user key
(CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons

So while the stable-security tree might be missing commits that might
or might not have security impact, it seems the 3.12 tree itself is
missing fixes for privilege escalation CVEs from last year. Should I
be recommending that no one uses 3.12?


Thanks,
Sasha



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Jiri Slaby]

[-- Attachment #1.1: Type: text/plain, Size: 3344 bytes --]

On 04/21/2016, 01:11 PM, Sasha Levin wrote:
>> Ok, not that bad, it is only unused code, but why are *not* these in the
>> security tree?
>> ipr: Fix out-of-bounds null overwrite
> 
> Is there a particular way to exploit this that I'm missing?

Any (write > 100) to "/sys/.../fw_update" writes '0' out of bounds, I
suppose.

But the point is different: I don't even need to care if there is one.
And more, I don't even want to wait for one to appear.

>> Input: powermate - fix oops with malicious USB descriptors
> 
> This requires physical access to the machine.

This is no relevant argument. There are plenty of studying rooms with
computers and I don't want users to crash a machine by a buggy driver.
OK, in this particular case, a broken cable, buggy bus or FW bug or
whatever would be needed on the top of that. But I am not a god to know
the circumstances before they occur, so better be safe now as it's
clearly a bugfix.

>> rapidio/rionet: fix deadlock on SMP
> 
> Seemed a bit borderline I suppose. There's nothing specific the
> user can do to actually trigger this?

Given my experience with fuzzers and bug hunting, how is not just heavy
loading the machine sufficient?

Pardom my ignorance, how can you actually be sure?

> Another thing to note here is that security patch selection database
> is shared between versions, so if a given commit gets marked as security
> later on (someone figured out it's a CVE or something similar), it'll
> get added to the stable-security tree even if it was initially skipped.

But that's too late. You then have to force people update immediately
while you actually would not need to.

> So I've also ended up auditing the 3.12 for missing CVE fixes and these
> ones ended up being at the top of the list. Could you explain why they
> are not in the 3.12 stable tree (and as a result can't get to users of
> the corresponding stable-security tree)?

Sure. They didn't apply or were not marked as stable. In both cases it
is the code maintainer responsibility to take care of those. At least by
pinging the stable list with SHAs.

On the top of that, I monitor SLE12 changes and:

> (CVE-2015-7513) 0185604 KVM: x86: Reload pit counters for all channels when restoring state

This was not evaluated for SLE12 yet.

> (CVE-2015-8539) 096fe9e KEYS: Fix handling of stored error in a negatively instantiated user key

Backported now. Thanks for noting.

> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons

Does not exist in the CVE database/is not confirmed yet AFAICS.

> So while the stable-security tree might be missing commits that might
> or might not have security impact, it seems the 3.12 tree itself is
> missing fixes for privilege escalation CVEs from last year. Should I
> be recommending that no one uses 3.12?

First, I am not deliberately filtering commits on an invalid basis.

Second, every fart can have a CVE number today. CVE number should be by
no means used as a decision.

Third, whatever is missing and is applicable, I am putting in.

Fourth, naturally, there is a lot of patches missing in the net flowing
in the large sea of patches. But given your count of patches, you have ~
2 times higher chance to miss something important.

thanks,
-- 
js
suse labs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Jiri Slaby]

[-- Attachment #1.1: Type: text/plain, Size: 374 bytes --]

On 04/21/2016, 01:59 PM, Jiri Slaby wrote:
>> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
> 
> Does not exist in the CVE database/is not confirmed yet AFAICS.

And now I am looking at the patch and I remember why I threw it away.
crypto_memneq is not in 3.12 yet and I was not keen enough to backport  it.

thanks,
-- 
js
suse labs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Greg KH]

On Thu, Apr 21, 2016 at 02:05:41PM +0200, Jiri Slaby wrote:
> On 04/21/2016, 01:59 PM, Jiri Slaby wrote:
> >> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
> > 
> > Does not exist in the CVE database/is not confirmed yet AFAICS.
> 
> And now I am looking at the patch and I remember why I threw it away.
> crypto_memneq is not in 3.12 yet and I was not keen enough to backport  it.

Which brings up the question, Sasha, why did you think these CVEs were
relevant for 3.12?  What were you basing that list on?

thanks,

greg k-h

Re: stable-security kernel updates

[Willy Tarreau]

On Thu, Apr 21, 2016 at 09:39:18PM +0900, Greg KH wrote:
> On Thu, Apr 21, 2016 at 02:05:41PM +0200, Jiri Slaby wrote:
> > On 04/21/2016, 01:59 PM, Jiri Slaby wrote:
> > >> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
> > > 
> > > Does not exist in the CVE database/is not confirmed yet AFAICS.
> > 
> > And now I am looking at the patch and I remember why I threw it away.
> > crypto_memneq is not in 3.12 yet and I was not keen enough to backport  it.
> 
> Which brings up the question, Sasha, why did you think these CVEs were
> relevant for 3.12?  What were you basing that list on?

Yep same question here because in fact checking what is *missing* is
harder than checking what should not have been there. I'm pretty sure
I missed a lot of things in 2.6.32 (though Ben and Moritz helped a lot)
but precisely the fact that they provided me fixes I wasn't aware of is
a sign that I can miss things.
 
Any reliable process to check for missing fixes is welcome of course. For
now the best way I found is to pick from more recent stable versions, which
also ensures people upgrading from and older branch to a newer branch will
not find a bug they used to see fixed.

Cheers,
Willy

Re: stable-security kernel updates

[Sasha Levin]

On 04/21/2016 08:39 AM, Greg KH wrote:
> On Thu, Apr 21, 2016 at 02:05:41PM +0200, Jiri Slaby wrote:
>> > On 04/21/2016, 01:59 PM, Jiri Slaby wrote:
>>>> > >> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
>>> > > 
>>> > > Does not exist in the CVE database/is not confirmed yet AFAICS.
>> > 
>> > And now I am looking at the patch and I remember why I threw it away.
>> > crypto_memneq is not in 3.12 yet and I was not keen enough to backport  it.
> Which brings up the question, Sasha, why did you think these CVEs were
> relevant for 3.12?  What were you basing that list on?

The EVM one? Because there exists a vulnerability in the 3.12 EVM code which
allows an attacker to essentially circumvent integrity checks, and the reason
it wasn't fixed was because a memory comparison helper function wasn't backported?

For the other CVEs I've listed? I looked at what went in to 3.14 but not 3.12,
and audited the resulting list to confirm that the vulnerability existed on 3.12.


Thanks,
Sasha

Re: stable-security kernel updates

[Jiri Slaby]

On 04/21/2016, 03:54 PM, Sasha Levin wrote:
> On 04/21/2016 08:39 AM, Greg KH wrote:
>> On Thu, Apr 21, 2016 at 02:05:41PM +0200, Jiri Slaby wrote:
>>>> On 04/21/2016, 01:59 PM, Jiri Slaby wrote:
>>>>>>>> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
>>>>>>
>>>>>> Does not exist in the CVE database/is not confirmed yet AFAICS.
>>>>
>>>> And now I am looking at the patch and I remember why I threw it away.
>>>> crypto_memneq is not in 3.12 yet and I was not keen enough to backport  it.
>> Which brings up the question, Sasha, why did you think these CVEs were
>> relevant for 3.12?  What were you basing that list on?
> 
> The EVM one? Because there exists a vulnerability in the 3.12 EVM code which
> allows an attacker to essentially circumvent integrity checks, and the reason
> it wasn't fixed was because a memory comparison helper function wasn't backported?

Because sometimes the breakage risk is much higher than fixing a bug.
This one was evaluated for 3.12.55 and not included at that time for
that very reason.

Now, given it it upstream for much longer, I reevaluated that and put
that into the 3.12 tree.

> For the other CVEs I've listed? I looked at what went in to 3.14 but not 3.12,
> and audited the resulting list to confirm that the vulnerability existed on 3.12.

Where exactly is 0185604 and 096fe9e contained in 3.14? I actually don't
see them in any of Greg's stable tree.

thanks,
-- 
js
suse labs

Re: stable-security kernel updates

[Willy Tarreau]

On Thu, Apr 21, 2016 at 04:13:07PM +0200, Jiri Slaby wrote:
> On 04/21/2016, 03:54 PM, Sasha Levin wrote:
> > On 04/21/2016 08:39 AM, Greg KH wrote:
> >> On Thu, Apr 21, 2016 at 02:05:41PM +0200, Jiri Slaby wrote:
> >>>> On 04/21/2016, 01:59 PM, Jiri Slaby wrote:
> >>>>>>>> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
> >>>>>>
> >>>>>> Does not exist in the CVE database/is not confirmed yet AFAICS.
> >>>>
> >>>> And now I am looking at the patch and I remember why I threw it away.
> >>>> crypto_memneq is not in 3.12 yet and I was not keen enough to backport  it.
> >> Which brings up the question, Sasha, why did you think these CVEs were
> >> relevant for 3.12?  What were you basing that list on?
> > 
> > The EVM one? Because there exists a vulnerability in the 3.12 EVM code which
> > allows an attacker to essentially circumvent integrity checks, and the reason
> > it wasn't fixed was because a memory comparison helper function wasn't backported?
> 
> Because sometimes the breakage risk is much higher than fixing a bug.
> This one was evaluated for 3.12.55 and not included at that time for
> that very reason.
> 
> Now, given it it upstream for much longer, I reevaluated that and put
> that into the 3.12 tree.
> 
> > For the other CVEs I've listed? I looked at what went in to 3.14 but not 3.12,
> > and audited the resulting list to confirm that the vulnerability existed on 3.12.
> 
> Where exactly is 0185604 and 096fe9e contained in 3.14? I actually don't
> see them in any of Greg's stable tree.

Indeed, the first one was brought into 3.2 and 3.18 (so it's missing from
3.4 to 3.14), and the second one is in 3.18.

Willy

Re: stable-security kernel updates

[Sasha Levin]

On 04/21/2016 10:13 AM, Jiri Slaby wrote:
> On 04/21/2016, 03:54 PM, Sasha Levin wrote:
>> On 04/21/2016 08:39 AM, Greg KH wrote:
>>> On Thu, Apr 21, 2016 at 02:05:41PM +0200, Jiri Slaby wrote:
>>>>> On 04/21/2016, 01:59 PM, Jiri Slaby wrote:
>>>>>>>>> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
>>>>>>>
>>>>>>> Does not exist in the CVE database/is not confirmed yet AFAICS.
>>>>>
>>>>> And now I am looking at the patch and I remember why I threw it away.
>>>>> crypto_memneq is not in 3.12 yet and I was not keen enough to backport  it.
>>> Which brings up the question, Sasha, why did you think these CVEs were
>>> relevant for 3.12?  What were you basing that list on?
>>
>> The EVM one? Because there exists a vulnerability in the 3.12 EVM code which
>> allows an attacker to essentially circumvent integrity checks, and the reason
>> it wasn't fixed was because a memory comparison helper function wasn't backported?
> 
> Because sometimes the breakage risk is much higher than fixing a bug.
> This one was evaluated for 3.12.55 and not included at that time for
> that very reason.
> 
> Now, given it it upstream for much longer, I reevaluated that and put
> that into the 3.12 tree.

Okay, fair enough.

>> For the other CVEs I've listed? I looked at what went in to 3.14 but not 3.12,
>> and audited the resulting list to confirm that the vulnerability existed on 3.12.
> 
> Where exactly is 0185604 and 096fe9e contained in 3.14? I actually don't
> see them in any of Greg's stable tree.

You're right, I looked at the 3.18 tree rather than 3.14, where there are:

8dc1d26 KVM: x86: Reload pit counters for all channels when restoring state
3fee639 KEYS: Fix handling of stored error in a negatively instantiated user key

This means that missing CVE fixes are quite common with stable trees?


Thanks,
Sasha

Re: stable-security kernel updates

[Willy Tarreau]

On Thu, Apr 21, 2016 at 10:27:46AM -0400, Sasha Levin wrote:
> This means that missing CVE fixes are quite common with stable trees?

Until someone reports they are missing :-)

Willy

Re: stable-security kernel updates

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

On Thu, 2016-04-21 at 16:33 +0200, Willy Tarreau wrote:
> On Thu, Apr 21, 2016 at 10:27:46AM -0400, Sasha Levin wrote:
> > 
> > This means that missing CVE fixes are quite common with stable
> > trees?
> Until someone reports they are missing :-)

Or they are unfixed upstream (there are a good few of those).

Debian has a public list of all unembargoed kernel security issues that
have CVEs (and a few that don't), with references to any upstream
commits and fixed stable versions - but only for the stable branches
that our stable releases follow.

The mapping of CVE IDs to commits may be useful to other stable
maintainers, even if the rest isn't.

svn co svn://scm.alioth.debian.org/svn/kernel-sec/

Ben.

-- 
Ben Hutchings
The generation of random numbers is too important to be left to chance.
                                                            - Robert Coveyou

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Willy Tarreau]

On Tue, Apr 26, 2016 at 01:14:13AM +0200, Ben Hutchings wrote:
> On Thu, 2016-04-21 at 16:33 +0200, Willy Tarreau wrote:
> > On Thu, Apr 21, 2016 at 10:27:46AM -0400, Sasha Levin wrote:
> > > 
> > > This means that missing CVE fixes are quite common with stable
> > > trees?
> > Until someone reports they are missing :-)
> 
> Or they are unfixed upstream (there are a good few of those).
> 
> Debian has a public list of all unembargoed kernel security issues that
> have CVEs (and a few that don't), with references to any upstream
> commits and fixed stable versions - but only for the stable branches
> that our stable releases follow.
> 
> The mapping of CVE IDs to commits may be useful to other stable
> maintainers, even if the rest isn't.
> 
> svn co�svn://scm.alioth.debian.org/svn/kernel-sec/

Thanks for sharing this Ben, it can indeed be helpful sometimes and it's
well organized!

Willy

Re: stable-security kernel updates

[Sasha Levin]

[-- Attachment #1.1: Type: text/plain, Size: 5746 bytes --]

On 04/21/2016 07:59 AM, Jiri Slaby wrote:
> On 04/21/2016, 01:11 PM, Sasha Levin wrote:
>>> Ok, not that bad, it is only unused code, but why are *not* these in the
>>> security tree?
>>> ipr: Fix out-of-bounds null overwrite
>>
>> Is there a particular way to exploit this that I'm missing?
> 
> Any (write > 100) to "/sys/.../fw_update" writes '0' out of bounds, I
> suppose.
> 
> But the point is different: I don't even need to care if there is one.
> And more, I don't even want to wait for one to appear.
> 
>>> Input: powermate - fix oops with malicious USB descriptors
>>
>> This requires physical access to the machine.
> 
> This is no relevant argument. There are plenty of studying rooms with
> computers and I don't want users to crash a machine by a buggy driver.
> OK, in this particular case, a broken cable, buggy bus or FW bug or
> whatever would be needed on the top of that. But I am not a god to know
> the circumstances before they occur, so better be safe now as it's
> clearly a bugfix.
> 
>>> rapidio/rionet: fix deadlock on SMP
>>
>> Seemed a bit borderline I suppose. There's nothing specific the
>> user can do to actually trigger this?
> 
> Given my experience with fuzzers and bug hunting, how is not just heavy
> loading the machine sufficient?
> 
> Pardom my ignorance, how can you actually be sure?

I'm not, same way you can't be sure about your stable patch selection either.

A commit that may not look to you like stable material might turn out to be one,
so how is this different for stable-security?

>> Another thing to note here is that security patch selection database
>> is shared between versions, so if a given commit gets marked as security
>> later on (someone figured out it's a CVE or something similar), it'll
>> get added to the stable-security tree even if it was initially skipped.
> 
> But that's too late. You then have to force people update immediately
> while you actually would not need to.

"immediately" is a strong word. Right now they won't update at all, so even
if they take a month to update that's better than the current state.

I'm not trying to replace the stable trees, I'm trying to help users who don't
update the stable tree that often to at least receive critical fixes in between
those updates.

>> So I've also ended up auditing the 3.12 for missing CVE fixes and these
>> ones ended up being at the top of the list. Could you explain why they
>> are not in the 3.12 stable tree (and as a result can't get to users of
>> the corresponding stable-security tree)?
> 
> Sure. They didn't apply or were not marked as stable. In both cases it
> is the code maintainer responsibility to take care of those. At least by
> pinging the stable list with SHAs.

Given this, how can you tell people they should be using your stable tree
rather than updating the kernel as a whole? The 3.12 tree is missing a big
chunk of commits that are stable material even though they weren't tagged
as such.

Those commits fix real bugs and by not shipping them users are given a false
sense of security by just updating the stable tree rather than the entire
kernel tree.

> On the top of that, I monitor SLE12 changes and:
> 
>> (CVE-2015-7513) 0185604 KVM: x86: Reload pit counters for all channels when restoring state
> 
> This was not evaluated for SLE12 yet.

This is an almost half a year old vulnerability that allows guests to crash
KVM hosts; something I'd consider quite critical these days.

This sort of thing is something that might encourage users not to follow the
stable tree at all. If updating the stable tree won't fix these sort of critical
issues, then why bother at all?

>> (CVE-2015-8539) 096fe9e KEYS: Fix handling of stored error in a negatively instantiated user key
> 
> Backported now. Thanks for noting.
> 
>> (CVE-2016-2085) 613317b EVM: Use crypto_memneq() for digest comparisons
> 
> Does not exist in the CVE database/is not confirmed yet AFAICS.

I saw it being shipped in Ubuntu kernels at the beginning of the month
with that CVE tag. Not sure about how that works though.

Ignoring the CVE stuff, this is a very serious issue and has a fix
that was supposed to make it to users of the stable tree. Why didn't it?

>> So while the stable-security tree might be missing commits that might
>> or might not have security impact, it seems the 3.12 tree itself is
>> missing fixes for privilege escalation CVEs from last year. Should I
>> be recommending that no one uses 3.12?
> 
> First, I am not deliberately filtering commits on an invalid basis.

I'd be happy to set clearer guidelines for what I consider a security
fix if that's the concern here?

> Second, every fart can have a CVE number today. CVE number should be by
> no means used as a decision.

Agreed, but it's safe to assume that anything with a CVE tag is worth
looking at. I don't think I've listed any "farts" above.

> Third, whatever is missing and is applicable, I am putting in.

Same for the stable-security tree. If you see anything I've missed please
let me know and I'll include it.

> Fourth, naturally, there is a lot of patches missing in the net flowing
> in the large sea of patches. But given your count of patches, you have ~
> 2 times higher chance to miss something important.

I do? Per the definition of stable-security, I only have to select them
from the ones you already selected. So while I have to look through ~50
commits per version you need to look at hundreds.

You're way more likely than me to miss a commit that was supposed to be
in stable, than me missing something that had to go into stable-security.


Thanks,
Sasha


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Jiri Slaby]

[-- Attachment #1.1: Type: text/plain, Size: 4627 bytes --]

On 04/21/2016, 03:53 PM, Sasha Levin wrote:
>> Pardom my ignorance, how can you actually be sure?
> 
> I'm not, same way you can't be sure about your stable patch selection either.

I repeat I am not doing any selection.

Patches are not included iff they do not apply and I am not confident
enough to backport them. In that case, a FAILED message is sent to the
authors. And when the authors don't care about that, the patch is not
included.

> A commit that may not look to you like stable material might turn out to be one,
> so how is this different for stable-security?

I do not select.

> "immediately" is a strong word. Right now they won't update at all, so even
> if they take a month to update that's better than the current state.
> 
> I'm not trying to replace the stable trees, I'm trying to help users who don't
> update the stable tree that often to at least receive critical fixes in between
> those updates.

And that's the point. I doubt you can separate patches to critical fixes
and the others.

>>> So I've also ended up auditing the 3.12 for missing CVE fixes and these
>>> ones ended up being at the top of the list. Could you explain why they
>>> are not in the 3.12 stable tree (and as a result can't get to users of
>>> the corresponding stable-security tree)?
>>
>> Sure. They didn't apply or were not marked as stable. In both cases it
>> is the code maintainer responsibility to take care of those. At least by
>> pinging the stable list with SHAs.
> 
> Given this, how can you tell people they should be using your stable tree
> rather than updating the kernel as a whole? The 3.12 tree is missing a big
> chunk of commits that are stable material even though they weren't tagged
> as such.

That's why more than one stable tree is released for every kernel. It's
growing. If you know about any more issues, share with us, they will be
fixed.

> Those commits fix real bugs and by not shipping them users are given a false
> sense of security by just updating the stable tree rather than the entire
> kernel tree.

Updating versions is painful, because new features and code refactoring
introduce bugs. We do not backport those. That's the difference.

>>> (CVE-2015-7513) 0185604 KVM: x86: Reload pit counters for all channels when restoring state
>>
>> This was not evaluated for SLE12 yet.
> 
> This is an almost half a year old vulnerability that allows guests to crash
> KVM hosts; something I'd consider quite critical these days.
> 
> This sort of thing is something that might encourage users not to follow the
> stable tree at all. If updating the stable tree won't fix these sort of critical
> issues, then why bother at all?

Well, if that's so critical, why is there no trace on the stable list
about it to be applied to all trees?

I indeed committed it into 3.12 already.

> I'd be happy to set clearer guidelines for what I consider a security
> fix if that's the concern here?

Yes, please.

>> Fourth, naturally, there is a lot of patches missing in the net flowing
>> in the large sea of patches. But given your count of patches, you have ~
>> 2 times higher chance to miss something important.
> 
> I do? Per the definition of stable-security, I only have to select them
> from the ones you already selected. So while I have to look through ~50
> commits per version you need to look at hundreds.

But yet, you missed some important fixes in a single release.

> You're way more likely than me to miss a commit that was supposed to be
> in stable,

Naturally, see above. I am not doing filtering though. I include
everything which fixes a bug and I am aware of.

> than me missing something that had to go into stable-security.

Doing any filtering on patches that somebody proposed to be in stable is
anything but -security.

If people feel that's too much updates, they should run some sort of BSD
or something. Anyway, it's nonsense to review all the patches and
pretend to understand the code well enough to decide whether the patches
are OK or not. It's bullshit. That's the very reason why any selection
won't ever work.

And provided we run stable trees on all our products with no dedicated
people to stable patches (except me collecting 3.12 and randomly
applying stable patches) and we saw only little bugs introduced by
stable over the past decade, I do not believe they chose the right
approach. Stable really pays off as it stays even though the process
could be improved in many ways as was discussed many times, of course.

thanks,
-- 
js
suse labs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Sasha Levin]

[-- Attachment #1.1: Type: text/plain, Size: 1319 bytes --]

[Sorry I'm cutting out lots of stuff here, I just want to understand the point
below first]

On 04/21/2016 10:54 AM, Jiri Slaby wrote:
> On 04/21/2016, 03:53 PM, Sasha Levin wrote:
>>> Pardom my ignorance, how can you actually be sure?
>>
>> I'm not, same way you can't be sure about your stable patch selection either.
> 
> I repeat I am not doing any selection.
> 
> Patches are not included iff they do not apply and I am not confident
> enough to backport them. In that case, a FAILED message is sent to the
> authors. And when the authors don't care about that, the patch is not
> included.
> 
>> A commit that may not look to you like stable material might turn out to be one,
>> so how is this different for stable-security?
> 
> I do not select.

So how exactly do commits go into your stable tree?

When I'm doing my stable work, I go through a range of mainline commits, see which
ones are stable material and follow the stable rules, and cherry pick them to my tree.

So at least in my case, there is patch selection every time I work on a stable tree.

I may end up thinking a given commit is not stable material, and people may point out
that it is, so I may end up going back and cherry picking it later. Have it never
happened with any of your trees?


Thanks,
Sasha


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Sasha Levin]

[-- Attachment #1.1: Type: text/plain, Size: 3523 bytes --]

On 04/21/2016 10:54 AM, Jiri Slaby wrote:
> On 04/21/2016, 03:53 PM, Sasha Levin wrote:
>> I'm not trying to replace the stable trees, I'm trying to help users who don't
>> update the stable tree that often to at least receive critical fixes in between
>> those updates.
> 
> And that's the point. I doubt you can separate patches to critical fixes
> and the others.

Build fixes? new device IDs? quirks? We can agree that it's not security.

We can also agree that there are commits that clearly pose no security risk, right?

From the latest 3.18 release, stuff like:

07508eb iw_cxgb3: Fix incorrectly returning error on success
983cabe iio: pressure: mpl115: fix temperature offset sign
c8d69b6 sched: Fix crash in sched_init_numa()
etc'

Obviously don't pose a security risk.

If we agree on that, then we've left with ~30% of the original tree, where the
security status is questionable. As I've mentioned, I'd be happy to discuss
the criteria for what we consider a "security" fix. Maybe it'll be all 30%, maybe less.

>> Given this, how can you tell people they should be using your stable tree
>> rather than updating the kernel as a whole? The 3.12 tree is missing a big
>> chunk of commits that are stable material even though they weren't tagged
>> as such.
> 
> That's why more than one stable tree is released for every kernel. It's
> growing. If you know about any more issues, share with us, they will be
> fixed.

If this is what you expect me to do, why was your first response on this
stable-security release was "I suggest nobody uses that kernel."?

When I started working on the 3.18 stable tree, it was easy to see that there is
a problem maintaining these type of trees. Some commits were never added, some
backported incorrectly, some reverted on one tree but not the others, and so on.

Instead of bashing Greg and other maintainers that the stable tree is a dumb idea
that can never work ("how can you decide what's a real fix and what's not?") I went
to write a set of tools that was supposed to help maintainers build correct stable
trees and validate their correctness versus other stable trees (available here, for
quite a while: https://git.kernel.org/cgit/linux/kernel/git/sashal/stable-tools.git/).

I've never received a comment on it, or heard of anyone using it, even though it
could have easily caught so many issues with each tree (such as the CVEs I've
copied earlier - this is how I dug them up).


The stable tree idea was born as a result of real need by users/customers/projects.
It's implementation isn't perfect, but we're all working together on making it better:
catching bugs, improving automation and reviewing each others work.

The stable-security idea was also a result of the same need. I didn't make it up
just so I'll have more stuff to maintain, I started it because users simply weren't
following the stable tree after a certain point. I can wave my finger at them all
I want, I can even ask Greg to wave his finger at them as well, but it's not going
to change them; they still will be running months old kernel in production, being
exposed to same nasty bugs.

If you have a better way of solving this I'm all ears. I'd happily change stable-security
if you have a plan of making it work better somehow. However, if thats not the case,
can we work together or improving it? Or at least not be sending each other inflammatory
mails about missing or not missing commits?


Thanks,
Sasha




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: stable-security kernel updates

[Bjørn Mork]

Sasha Levin <sasha.levin@oracle.com> writes:
> On 04/21/2016 02:43 AM, Jiri Slaby wrote:
>
>> Input: powermate - fix oops with malicious USB descriptors
>
> This requires physical access to the machine.

You wish.

Say you have some internal USB connected device with replacable
firmware.  LTE modem, fingerprint reader, webcam - you name it. How do
you know that this cannot be abused to impersonate some other USB
device?  Yes, changing the firmware of those devices should of course
require admin privileges.  But is that always so?  Writing firmware to a
modem, for example, is typically done over a serial device similar to
the one used for normal modem operations. Privileges necessary to manage
the modem will also include changing the firmware. Physical access is
not necessary.

Do you trust the firmware protection of all your non-removable USB
devices?


Bjørn

Re: stable-security kernel updates

[Willy Tarreau]

On Wed, Apr 20, 2016 at 03:50:34PM -0400, Sasha Levin wrote:
> Hi all,
> 
> Updates for stable-security kernels have been released:
> 
> 	- v3.12.58-security
> 	- v3.14.67-security
> 	- v3.18.31-security
> 	- v4.1.22-security
> 	- v4.4.8-security
> 	- v4.5.2-security

Sasha, regardless the rest of the discussion in this thread, I find
myself confused by the naming above. For me, 3.12.58-something means
"something" on top of 3.12.58. Many (most? all?) forks work like this,
but here it seems that instead it's 3.12 + your selection of fixes
from kernels up to 3.12.58. I guess it would be much less confusing to
call it something like 3.12.0-security58 or something like this (or
maybe simply 3.12.0.58). Some people might be tempted to upgrade from
3.12.40 to 3.12.58-security and will possibly discover some breakage
due to bugs that were fixed between 3.12 and 3.12.40 and which are not
fixed in 3.12.58-security.

Thanks,
willy

Re: stable-security kernel updates

[Sasha Levin]

On 04/21/2016 08:56 AM, Willy Tarreau wrote:
> On Wed, Apr 20, 2016 at 03:50:34PM -0400, Sasha Levin wrote:
>> Hi all,
>>
>> Updates for stable-security kernels have been released:
>>
>> 	- v3.12.58-security
>> 	- v3.14.67-security
>> 	- v3.18.31-security
>> 	- v4.1.22-security
>> 	- v4.4.8-security
>> 	- v4.5.2-security
> 
> Sasha, regardless the rest of the discussion in this thread, I find
> myself confused by the naming above. For me, 3.12.58-something means
> "something" on top of 3.12.58. Many (most? all?) forks work like this,
> but here it seems that instead it's 3.12 + your selection of fixes
> from kernels up to 3.12.58. I guess it would be much less confusing to
> call it something like 3.12.0-security58 or something like this (or
> maybe simply 3.12.0.58). Some people might be tempted to upgrade from
> 3.12.40 to 3.12.58-security and will possibly discover some breakage
> due to bugs that were fixed between 3.12 and 3.12.40 and which are not
> fixed in 3.12.58-security.

That makes sense. I'll change that.


Thanks,
Sasha