Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d" [resend]

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d" [resend]

[Paolo Bonzini]

Resending with LKML in Cc, since posting to the linux-cve-announce
mailing list is restricted to moderators.

On 2/20/24 19:34, Greg Kroah-Hartman wrote:
> Description
> ===========
>
> In the Linux kernel, the following vulnerability has been resolved:
>
> Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"
>
> This reverts commit 5e2cf333b7bd5d3e62595a44d598a254c697cd74.
>
> That commit introduced the following race and can cause system hung.
>
>   md_write_start:             raid5d:
>   // mddev->in_sync == 1
>   set "MD_SB_CHANGE_PENDING"
>                              // running before md_write_start wakeup it
>                               waiting "MD_SB_CHANGE_PENDING" cleared
>                               >>>>>>>>> hung
>   wakeup mddev->thread
>   ...
>   waiting "MD_SB_CHANGE_PENDING" cleared
>   >>>> hung, raid5d should clear this flag
>   but get hung by same flag.
>
> The issue reverted commit fixing is fixed by last patch in a new way.

Sometimes less than optimal descriptions end up even in Linux commit
messages, and I understand that you're "not going to be adding anything
additional to the report" other than the git commit message. [1]  But
this description is not just "suboptimal" English, it also makes zero
sense since it refers to a "last patch" that does not exist.

There are dozens of distros, both commercial and non-commercial, whose
users need a *real* description of what is being fixed.  By writing CVE
descriptions that make no sense, you're creating more work for everyone
involved, without putting in place a process to clarify these things
except through "the maintainers of the relevant subsystem
affected"---who are not CC'd to these messages and therefore might not
even know that the CVE announcement exists.

My suggestion is to CC the author of the fix and the maintainer, and if
possible even go through a pre-verification phase similar to what's done
for AUTOSEL patches.  If some commit messages are irredeemable, or some
situations are just too complex, and no one is willing to put the work
that's required to do the work properly, the maintainer should have the
possibility to NACK the creation of an unusable CVE entry like this one.

(Somewhat related to this, how are you going to handle patch
dependencies?  Sasha's GSD updates has a separate entry for each commit,
the result being "vulnerabilities" with "no functional change" in their
description.  Are they instead going to be rolled into a single entry
like this one now that you're actually creating CVEs?)

I am cautiously optimistic that this can be worked out and I agree with
you that lots of bug fixes going into stable have potential security
impact.  But as this example shows, there's still more than a few kinks
to be ironed out.

> The Linux kernel CVE team has assigned CVE-2023-52437 to this issue.
>
>
> Affected and fixed versions
> ===========================
>
> 	Issue introduced in 5.15.75 with commit 9e86dffd0b02 and fixed in 5.15.148 with commit 84c39986fe6d
> 	Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.1.74 with commit bed0acf330b2
> 	Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.1.75 with commit cfa468382858
> 	Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.6.13 with commit e16a0bbdb7e5
> 	Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.6.14 with commit aab69ef76970
> 	Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.7.1 with commit 0de40f76d567
> 	Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.7.2 with commit 87165c64fe1a

So which one is it of these 6.{1,6,7}.y releases that fixed the issue?

> The Linux kernel CVE team recommends that you update to the latest
> stable kernel version for this, and many other bugfixes.  Individual
> changes are never tested alone, but rather are part of a larger kernel
> release.  Cherry-picking individual commits is not recommended or
> supported by the Linux kernel community at all.  If however, updating to
> the latest release is impossible, the individual changes to resolve this
> issue can be found at these commits:
> 	https://git.kernel.org/stable/c/84c39986fe6dd77aa15f08712339f5d4eb7dbe27
> 	https://git.kernel.org/stable/c/bed0acf330b2c50c688f6d9cfbcac2aa57a8e613
> 	https://git.kernel.org/stable/c/cfa46838285814c3a27faacf7357f0a65bb5d152
> 	https://git.kernel.org/stable/c/e16a0bbdb7e590a6607b0d82915add738c03c069
> 	https://git.kernel.org/stable/c/aab69ef769707ad987ff905d79e0bd6591812580
> 	https://git.kernel.org/stable/c/0de40f76d567133b871cd6ad46bb87afbce46983
> 	https://git.kernel.org/stable/c/87165c64fe1a98bbab7280c58df3c83be2c98478
> 	https://git.kernel.org/stable/c/bed9e27baf52a09b7ba2a3714f1e24e17ced386d

Half of these are reverting the revert.  I understand that
"cherry-picking individual commits is not recommended" but it looks like
this is a bug in whatever scripts you are using.  Are they public, so
that fixes can be developed in the open?

Also, commit 87165c64fe1a9 (the revert of the revert) was marked 5.19+
but 5.15.148 does have the original revert.  Does that mean that 5.15.148
still has the "issue with raid5 with journal device" (another hang, see
https://lore.kernel.org/linux-raid/20240123005700.9302-1-dan@danm.net/)
mentioned in the commit message for 87165c64fe1a9?  If so, that
contradicts the fact that updating to the latest release of a given LTS
branch is the best course of action, since for some users 5.15.147 might
be better than 5.15.148.

Paolo

[1] https://lwn.net/ml/linux-kernel/2024021518-stature-frightful-e7fc@gregkh/
[2] https://lwn.net/ml/linux-kernel/2024021430-blanching-spotter-c7c8@gregkh/

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Sasha Levin]

On Wed, Feb 21, 2024 at 02:15:12PM +0100, Paolo Bonzini wrote:
>>On 2/21/24 14:10, Sasha Levin wrote:
>>>On Wed, Feb 21, 2024 at 01:58:04PM +0100, Paolo Bonzini wrote:
>>>>On 2/21/24 13:34, Sasha Levin wrote:
>>>>>[snip]
>
>This conversation needs to be public.

I've cc'ed lkml.

>Please send your original answer as a reply to the message with 
>"[resend]" in the subject, which has LKML in the Cc list, and we'll 
>take it from there.  I'm sorry about the confusion, but I didn't 
>expect linux-cve-announce to be moderator-only, as that is a deviation 
>from the common practice of vger.kernel.org.

You'll find that the convention for *-announce mailing list is that
they, as the name suggests, are used only for announces and are
moderated.

So for example, linux-kernel@ is the lkml we all know and love, while
linux-kernel-announce@ is an announcement-only mailing list used to
announce releases.

But thank you for pointing out this "deviation".

-- 
Thanks,
Sasha

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Paolo Bonzini]

On 2/21/24 15:35, Sasha Levin wrote:
> On Wed, Feb 21, 2024 at 02:15:12PM +0100, Paolo Bonzini wrote:
>>> On 2/21/24 14:10, Sasha Levin wrote:
>>>> On Wed, Feb 21, 2024 at 01:58:04PM +0100, Paolo Bonzini wrote:
>>>>> On 2/21/24 13:34, Sasha Levin wrote:
>>>>>> [snip]
>>
>> This conversation needs to be public.
> 
> I've cc'ed lkml.

This is an empty message, so it's not going to help making public the 
part of the discussion that was CC'd to the moderator-only list.  I take 
it that I can quote any part of the private thread?  I'd like to have 
your explicit permission for that, if you are not going to resend your 
answer.

> You'll find that the convention for *-announce mailing list is that
> they, as the name suggests, are used only for announces and are
> moderated.

Indeed, all two of them.  ¯\_(ツ)_/¯

Paolo

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Sasha Levin]

On Wed, Feb 21, 2024 at 04:02:45PM +0100, Paolo Bonzini wrote:
>On 2/21/24 15:35, Sasha Levin wrote:
>>On Wed, Feb 21, 2024 at 02:15:12PM +0100, Paolo Bonzini wrote:
>>>>On 2/21/24 14:10, Sasha Levin wrote:
>>>>>On Wed, Feb 21, 2024 at 01:58:04PM +0100, Paolo Bonzini wrote:
>>>>>>On 2/21/24 13:34, Sasha Levin wrote:
>>>>>>>[snip]
>>>
>>>This conversation needs to be public.
>>
>>I've cc'ed lkml.
>
>This is an empty message, so it's not going to help making public the 
>part of the discussion that was CC'd to the moderator-only list.  I 
>take it that I can quote any part of the private thread?  I'd like to 
>have your explicit permission for that, if you are not going to resend 
>your answer.

Please do. Sorry - trying to do this before my morning coffee...

-- 
Thanks,
Sasha

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Paolo Bonzini]

On 2/21/24 16:11, Sasha Levin wrote:
> Please do. Sorry - trying to do this before my morning coffee...

Thanks.

On 2/21/24 14:10, Sasha Levin wrote:
> On Wed, Feb 21, 2024 at 10:01:26AM +0100, Paolo Bonzini wrote:
>> On 2/21/24 13:34, Sasha Levin wrote:
>>> On Wed, Feb 21, 2024 at 2/21/24 13:34, Sasha Levin wrote:
>>>> There are dozens of distros, both commercial and non-commercial, whose
>>>> users need a *real* description of what is being fixed.  By writing
>>>> CVE descriptions that make no sense, you're creating more work for
>>>> everyone involved, without putting in place a process to clarify these
>>>> things except through "the maintainers of the relevant subsystem
>>>> affected"---who are not CC'd to these messages and therefore might not
>>>> even know that the CVE announcement exists. [...] My suggestion is
>>>> to CC the author of the fix and the maintainer
>>>
>>> This whole process didn't exist up until this point, and distros as well
>>> as individuals were creating CVEs without a useful messages nor without
>>> acks from the relevant authors/maintainers. What changed now?
>>
>> First, that _when_ a CVE was created, there was a human in the loop
>> deciding who to write.  Here we have scripts that clearly got
>> confused, pulling patches that were not included in the description.
>
> Commit messages make the most sense when the need is to relay technical
> information about an issue.

No doubt, but here only one commit message was included and it's hiding
information about the bug it introduced.

To recap:

- the CVE description comes from was upstream commit bed9e27baf52

- neither the CVE mitigation section nor the mentioned kernel releases
fix the bug mentioned in the upstream commit, because the mitigation
section also includes commits that _revert_ commit bed9e27baf52

- this second revert is not mentioned anywhere, so the CVE description
is at best misleading; or perhaps more accurately described as
"completely f***ed up".

I'm sure it's just a bug in the scripts, but it's worrisome that you
don't acknowledge this.

>> Second, that there is going to be thousands more occurrences of this
>> work overall.
>>
>> Third, that the relevant maintainers were not on the hook as the sole
>> "authority to dispute or modify an assigned CVE".  Maintainers were
>> given this additional work without, as far as I know, anybody telling
>> us, and I'd like to be able to do it efficiently and without adding
>> more bureaucracy to an already-thankless work.
>
> So now you're asking us to drop this additional work on them by
> reviewing CVE requests?

It doesn't have to be mandatory.  But for people that _do_ want to do
the work, they might as well do it before the CVE is publicly announced,
rather than after.  At least give us the possibility of doing it without
bureaucracy.

>>> Looking at CVE-2024-0562, which we've previously discussed as an
>>> example, were the maintainers of the relevant subsystem consulted before
>>> the CVE was assigned?
>>
>> CVE-2024-0562 matches perfectly what you're doing: create a CVE only
>> after the bug is fixed, to improve communication on how to update.  Of
>> course the timing was off by a year, but hey "a bug is a bug" no?
>>
>> Early communication of these issues is indeed an improvement that can
>> be made by having the kernel CNA.  But if everything is automated and
>> on day one you have already such garbage, it's not going to go well.
>
> The timing is off, but my point is that the maintainers were never
> consulted whether this issue is really a security issue or a dud.
> Is it really a security issue?

It is a use-after-free so, going by the recent announcements on
linux-cve-announce, it deserves a CVE.

>>> I'd also note that the if a certain user/distro requires massaged
>>> explanations to be "user readable" and dumbed down to layman terms, then
>>> it is on that user/distro to provide it to them (heck, I was doing this
>>> exact work while making Ksplice updates years ago).
>>
>> It's not "a certain user/distro", it's all of them---whether
>> commercial or not.  And again, by removing any human vetting of what
>> goes into CVEs, you're pushing work on N distros thousands of times
>> per year instead of having it done only once upstream.
>
> So I've worked in quite a few places in the past decade or so (Oracle,
> Microsoft, Google, ...) and let me tell you from experience that all
> those companies deeply care about security and CVEs, and the best CVE
> description was one that simply pointed to the upstream commit.

I agree, but you're talking about commit references that were already
validated by a human.  Something like this CVE, which says "the last
patch will fix the bug", or a variable rename with "no functional change
intended", certainly isn't a great CVE description.

> So yes, I disagree with your "all of them" statement. For that matter,
> I'd argue that the number of users who need massaged messages are the
> vast minority.

They don't need massaged messages.  They need correct and complete ones.
   You're completely removing the human part of the work and expecting
the result to be of comparable quality.  That's not going to happen, and
this CVE is an example of this.

All of them will read "The issue reverted commit fixing is fixed by last
patch in a new way" in the description and think what crap is this.

Some of them will see

         Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.7.1
with commit 0de40f76d567
         Issue introduced in 6.1 with commit 5e2cf333b7bd and fixed in 6.7.2
with commit 87165c64fe1a

and will wonder which release actually fixes the bug.

A few of them will also notice that the suggested mitigation is to
absolutely do nothing, and will check if it's already April 1st.

This CVE is an order of magnitude worse than CVE-2024-0562 in terms of
confusion and wasted time for everyone involved.

>> I'll also note that you're not answering any of my questions:
>>
>> 1) as KVM maintainer (who is not allowed to post linux-cve-announce)
>> how can I be aware of all CVEs that are created and affect my subsystem?
>
> How were you aware until a few weeks ago where CVE assignments were
> handled by different entities?

They more often than not CC'd me before the patch was committed and
provided me the CVE id, and I was able to provide input or dispute the
assignment beforehand.  This is exactly what I'm suggesting that the
kernel should do.

I don't recall anything like CVE-2024-0562 happening for KVM, but it's a
small subsystem.

>> 2) how are you going to handle patch dependencies?  Are they going to
>> be rolled into a single entry or split into multiple announcements?
>
> Likely multiple different entries.

So, looking at
https://git.kernel.org/pub/scm/linux/security/vulns.git/tree/cve/review/6.7.proposed
I see

aeb686a98a9e usb: gadget: uvc: Allocate uvc_requests one at a time
da324ffce34c usb: gadget: uvc: Fix use-after-free for inflight usb_requests

What vulnerability is the first one fixing?  Looking at your GSD
entries, will there even be CVEs purporting that renaming a variable
from "foo" to "bar" is fixing a vulnerability?

>> 3) are the scripts used to generate the CVEs public?  Can fixes to the
>> scripts be developed in the open?
>
> Yup - https://git.kernel.org/pub/scm/linux/security/vulns.git/ .

What about the detection part?

I like to assume the best of people, so I'll assume that this is just
naïveté rather than an intentional attempt at burning everything down.
But please, let's take a step back and understand what the proposed
workflow fixes and breaks for everyone (especially maintainers and
distros).  Then make a proper solution.  In the meanwhile you can keep
sending test announcements to linux-cve-announce, and those can be used
to debug the process and the scripts.

In fact it would be nice if bippy included at the end the command line
that was used, to aid the reproduction and fixing of bugs.

Again: I'm optimistic that this process can be an improvement over what
various CNAs were doing before.  But right now you're pushing out crap
that simply doesn't live up to the standards of how Linux operates, so
please stop.

Paolo

>> 4) what happened with 5.15 having only the revert but not the
>> revert-of-revert?  Does this mean that it has the bug fixed by commit
>> 87165c64fe1a9, and does this contradict the fact that updating to the
>> latest release of a given LTS branch is the best course of action?
>
> It is always best to run the latest released kernel.

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Sasha Levin]

On Wed, Feb 21, 2024 at 04:56:31PM +0100, Paolo Bonzini wrote:
>To recap:
>
>- the CVE description comes from was upstream commit bed9e27baf52
>
>- neither the CVE mitigation section nor the mentioned kernel releases
>fix the bug mentioned in the upstream commit, because the mitigation
>section also includes commits that _revert_ commit bed9e27baf52
>
>- this second revert is not mentioned anywhere, so the CVE description
>is at best misleading; or perhaps more accurately described as
>"completely f***ed up".
>
>I'm sure it's just a bug in the scripts, but it's worrisome that you
>don't acknowledge this.

No objections that commit messages aren't perfect, they're just better
than the some unhelpful text like:

"""
In the Linux kernel before 6.5.9, there is a NULL pointer dereference in send_acknowledge in net/nfc/nci/spi.c.
"""

My choice was to actually just call out the offending commit and be done
with it, without additional description.

>>So now you're asking us to drop this additional work on them by
>>reviewing CVE requests?
>
>It doesn't have to be mandatory.  But for people that _do_ want to do
>the work, they might as well do it before the CVE is publicly announced,
>rather than after.  At least give us the possibility of doing it without
>bureaucracy.

[...]

>> How were you aware until a few weeks ago where CVE assignments were
>> handled by different entities?
>
>They more often than not CC'd me before the patch was committed and
>provided me the CVE id, and I was able to provide input or dispute the
>assignment beforehand.  This is exactly what I'm suggesting that the
>kernel should do.

This is fascinating to know, because when multiple members of the
community asked to review CVEs before they are assigned, certain few
CNAs blatantly ignored such requests.

Would you want to expand on why you got the courtesy of being able to
review these assignments, while the rest of us had to jump through the
hoops of becoming our own CNA just to stop from this crap from happening
to us?

[ snip ]

>>So yes, I disagree with your "all of them" statement. For that matter,
>>I'd argue that the number of users who need massaged messages are the
>>vast minority.
>
>They don't need massaged messages.  They need correct and complete ones.
>  You're completely removing the human part of the work and expecting
>the result to be of comparable quality.  That's not going to happen, and
>this CVE is an example of this.

I definitely agree that it would be nice to have better messages in
CVEs, and I'd welcome interested parties to follow the process that was
in place up until two weeks ago, and request an amendment to a CVE with
a better description (or dispute an invalid one) with the CNA.

This is exactly the same process that most of us had to follow in the
past to address crappy CVE descriptions or bogus CVEs.

>>>2) how are you going to handle patch dependencies?  Are they going to
>>>be rolled into a single entry or split into multiple announcements?
>>
>>Likely multiple different entries.
>
>So, looking at
>https://git.kernel.org/pub/scm/linux/security/vulns.git/tree/cve/review/6.7.proposed
>I see
>
>aeb686a98a9e usb: gadget: uvc: Allocate uvc_requests one at a time
>da324ffce34c usb: gadget: uvc: Fix use-after-free for inflight usb_requests
>
>What vulnerability is the first one fixing?  Looking at your GSD

Well, if you look at the first patch, it says "This patch is 1 of 2
patches addressing the use-after-free issue.".

>entries, will there even be CVEs purporting that renaming a variable
>from "foo" to "bar" is fixing a vulnerability?

Maybe? Hopefully not if it's not a real security issue.

>>>3) are the scripts used to generate the CVEs public?  Can fixes to the
>>>scripts be developed in the open?
>>
>>Yup - https://git.kernel.org/pub/scm/linux/security/vulns.git/ .
>
>What about the detection part?

Manual at this point, we're looking into converging workflows but it's
one of those things we'd need to address after we got the basics in
place.

>I like to assume the best of people, so I'll assume that this is just
>naïveté rather than an intentional attempt at burning everything down.
>But please, let's take a step back and understand what the proposed
>workflow fixes and breaks for everyone (especially maintainers and
>distros).  Then make a proper solution.  In the meanwhile you can keep
>sending test announcements to linux-cve-announce, and those can be used
>to debug the process and the scripts.

No objections on future improvements, right now we're still trying to
get the basics working which is where the strong pushback on "feature
requests" is coming from.

>In fact it would be nice if bippy included at the end the command line
>that was used, to aid the reproduction and fixing of bugs.

Bippy just maps commits to trees and formats the json, or did I
misunderstand what you meant?

-- 
Thanks,
Sasha

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Paolo Bonzini]

On 2/21/24 17:22, Sasha Levin wrote:
> On Wed, Feb 21, 2024 at 04:56:31PM +0100, Paolo Bonzini wrote:
>>> So now you're asking us to drop this additional work on them by
>>> reviewing CVE requests?
>>
>> It doesn't have to be mandatory.  But for people that _do_ want to do
>> the work, they might as well do it before the CVE is publicly announced,
>> rather than after.  At least give us the possibility of doing it without
>> bureaucracy.
> 
> [...]
> 
>>> How were you aware until a few weeks ago where CVE assignments were
>>> handled by different entities?
>>
>> They more often than not CC'd me before the patch was committed and
>> provided me the CVE id, and I was able to provide input or dispute the
>> assignment beforehand.  This is exactly what I'm suggesting that the
>> kernel should do.
> 
> This is fascinating to know, because when multiple members of the
> community asked to review CVEs before they are assigned, certain few
> CNAs blatantly ignored such requests.
>
> Would you want to expand on why you got the courtesy of being able to
> review these assignments, while the rest of us had to jump through the
> hoops of becoming our own CNA just to stop from this crap from happening
> to us?

Probably because I generally get CVEs myself ahead of time for 
corruption or use-after-free bugs (in arch-independent or x86 KVM code). 
  So when somebody comes with a CVE it's usually Project Zero stuff and 
not something like CVE-2024-0562, and we work together.

If you can tell me (even privately) what CNAs these were, I can check. 
But probably the answer is simply that either they have never touched 
KVM, or the issues were under some kind of embargo.

>>> So yes, I disagree with your "all of them" statement. For that matter,
>>> I'd argue that the number of users who need massaged messages are the
>>> vast minority.
>>
>> They don't need massaged messages.  They need correct and complete ones.
>> You're completely removing the human part of the work and expecting
>> the result to be of comparable quality.  That's not going to happen, and
>> this CVE is an example of this.
> 
> I definitely agree that it would be nice to have better messages in
> CVEs, and I'd welcome interested parties to follow the process that was
> in place up until two weeks ago, and request an amendment to a CVE with
> a better description (or dispute an invalid one) with the CNA.
> This is exactly the same process that most of us had to follow in the
> past to address crappy CVE descriptions or bogus CVEs.

And what I am saying is that we need to do better.  Give the maintainer 
an opportunity to say at least yes or no.

>>>> 2) how are you going to handle patch dependencies?  Are they going to
>>>> be rolled into a single entry or split into multiple announcements?
>>>
>>> Likely multiple different entries.
>>
>> So, looking at
>> https://git.kernel.org/pub/scm/linux/security/vulns.git/tree/cve/review/6.7.proposed
>> I see
>>
>> aeb686a98a9e usb: gadget: uvc: Allocate uvc_requests one at a time
>> da324ffce34c usb: gadget: uvc: Fix use-after-free for inflight 
>> usb_requests
>>
>> What vulnerability is the first one fixing?  Looking at your GSD
> 
> Well, if you look at the first patch, it says "This patch is 1 of 2
> patches addressing the use-after-free issue.".

Right, it's *the* issue.  It's one CVE, not two.  The first patch is a 
dependency of the actual fix, it's not its own vulnerability.

>> entries, will there even be CVEs purporting that renaming a variable
>> from "foo" to "bar" is fixing a vulnerability?
> 
> Maybe? Hopefully not if it's not a real security issue.

What if it's a dependency of the actual issue/fix?

>> I like to assume the best of people, so I'll assume that this is just
>> naïveté rather than an intentional attempt at burning everything down.
>> But please, let's take a step back and understand what the proposed
>> workflow fixes and breaks for everyone (especially maintainers and
>> distros).  Then make a proper solution.  In the meanwhile you can keep
>> sending test announcements to linux-cve-announce, and those can be used
>> to debug the process and the scripts.
> 
> No objections on future improvements, right now we're still trying to
> get the basics working which is where the strong pushback on "feature
> requests" is coming from.

Got it.  But multiple commits per CVE is not a feature request, it's 
basic acknowledgement of how Linux is developed.

>> In fact it would be nice if bippy included at the end the command line
>> that was used, to aid the reproduction and fixing of bugs.
> 
> Bippy just maps commits to trees and formats the json, or did I
> misunderstand what you meant?

I would like to run bippy myself to understand how the commits in 
CVE-2023-52437 were chosen.

Paolo

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Greg Kroah-Hartman]

On Wed, Feb 21, 2024 at 04:56:31PM +0100, Paolo Bonzini wrote:
> To recap:
> 
> - the CVE description comes from was upstream commit bed9e27baf52
> 
> - neither the CVE mitigation section nor the mentioned kernel releases
> fix the bug mentioned in the upstream commit, because the mitigation
> section also includes commits that _revert_ commit bed9e27baf52
> 
> - this second revert is not mentioned anywhere, so the CVE description
> is at best misleading; or perhaps more accurately described as
> "completely f***ed up".
> 
> I'm sure it's just a bug in the scripts, but it's worrisome that you
> don't acknowledge this.

Yes, this is a bug in the scripts, but it wasn't obvious what you were
objecting to here honestly.  Reverts were not anything I tested the
scripts with before now, and I'm sure there are going to be more cases
that fail in odd ways too.  We'll fix them when they show up, that's the
best we can do.

I'll look at it tomorrow and try to figure it out, if nothing else, I'll
just manually update the json record and push the update to cve.org as
that's the "canonical" record here.  The json files will be updated over
time as new releases happen and patches flow backwards, so they will be
updated, but for now, sending out new email messages all the time would
be a mess.

However in this case, I'll fix it up and send out a new announcement as
obviously it's wrong in places.

If you want to replace the wording in the description here with anything
else better, PLEASE let us know and we will be glad to do so.

That's the benifit of being a CNA, we can ACTUALLY MODIFY the CVE
records, previously it was almost impossible to ever do so.

thanks,

greg k-h

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Paolo Bonzini]

On 2/21/24 19:21, Greg Kroah-Hartman wrote:
> On Wed, Feb 21, 2024 at 04:56:31PM +0100, Paolo Bonzini wrote:
>> To recap:
>>
>> - the CVE description comes from was upstream commit bed9e27baf52
>>
>> - neither the CVE mitigation section nor the mentioned kernel releases
>> fix the bug mentioned in the upstream commit, because the mitigation
>> section also includes commits that _revert_ commit bed9e27baf52
>>
>> - this second revert is not mentioned anywhere, so the CVE description
>> is at best misleading; or perhaps more accurately described as
>> "completely f***ed up".
>>
>> I'm sure it's just a bug in the scripts, but it's worrisome that you
>> don't acknowledge this.
> 
> Yes, this is a bug in the scripts, but it wasn't obvious what you were
> objecting to here honestly.  Reverts were not anything I tested the
> scripts with before now, and I'm sure there are going to be more cases
> that fail in odd ways too.  We'll fix them when they show up, that's the
> best we can do. [...]
>
> If you want to replace the wording in the description here with anything
> else better, PLEASE let us know and we will be glad to do so.

But there's not even a documented way to do it.

All that the document says is "the authority to dispute or modify an 
assigned CVE for a specific kernel change lies solely with the 
maintainers of the relevant subsystem affected".  But it doesn't say:

* how the maintainer would ask for such a modification or dispute

* if and how anyone else could propose them

* whether the CVE team can also do them unilaterally

Perhaps since there's no archive for cve@kernel.org, there should be a 
public discussion mailing list (e.g. linux-cve@vger) that maintainers 
can reply to?  The private cve@kernel.org alias would then be just for 
the request of embargoed CVEs.

It would be great if modifications or disputes could simply be sent as 
patches to the vulns.git repo.  You guys can have push hooks or 
something like that that take care of sending messages to 
linux-cve-announce etc.

Another underspecified part is the early request of CVEs.  Some 
questions I have:

* what information is needed

* is there a limit on embargo length similar to security@kernel.org

* should it be acked by the subsystem maintainer

More in general, I think you're underestimating the extra work for the 
"listeners" of CVEs, that will come from bugs in the script or other 
not-so-well-defined aspects of the process.  I really think it would be 
a good idea to behave "as if" you were already creating CVE, but for now 
just send out the announcements and publish the JSON in a git repo.

As we run the experiment for a while, we can get input from interested 
maintainers and third parties.  I am sure I can find someone from the 
Red Hat product security team to explain their desires, clarify how they 
consume CVE announcements, and what simplifies/complicates their job. 
Their needs are probably not that unique.

In the meanwhile, you already have the benefit of coordinating the 
creation of CVEs, avoiding surprises like CVE-2024-0562 and allowing the 
modifications.

> That's the benifit of being a CNA, we can ACTUALLY MODIFY the CVE
> records, previously it was almost impossible to ever do so.

Agreed.  There is potential to do much better than before.

Paolo

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Greg Kroah-Hartman]

On Thu, Feb 22, 2024 at 10:58:22AM +0100, Paolo Bonzini wrote:
> On 2/21/24 19:21, Greg Kroah-Hartman wrote:
> > On Wed, Feb 21, 2024 at 04:56:31PM +0100, Paolo Bonzini wrote:
> > > To recap:
> > > 
> > > - the CVE description comes from was upstream commit bed9e27baf52
> > > 
> > > - neither the CVE mitigation section nor the mentioned kernel releases
> > > fix the bug mentioned in the upstream commit, because the mitigation
> > > section also includes commits that _revert_ commit bed9e27baf52
> > > 
> > > - this second revert is not mentioned anywhere, so the CVE description
> > > is at best misleading; or perhaps more accurately described as
> > > "completely f***ed up".
> > > 
> > > I'm sure it's just a bug in the scripts, but it's worrisome that you
> > > don't acknowledge this.
> > 
> > Yes, this is a bug in the scripts, but it wasn't obvious what you were
> > objecting to here honestly.  Reverts were not anything I tested the
> > scripts with before now, and I'm sure there are going to be more cases
> > that fail in odd ways too.  We'll fix them when they show up, that's the
> > best we can do. [...]
> > 
> > If you want to replace the wording in the description here with anything
> > else better, PLEASE let us know and we will be glad to do so.
> 
> But there's not even a documented way to do it.
> 
> All that the document says is "the authority to dispute or modify an
> assigned CVE for a specific kernel change lies solely with the maintainers
> of the relevant subsystem affected".  But it doesn't say:
> 
> * how the maintainer would ask for such a modification or dispute

Email.

> * if and how anyone else could propose them

Email.

> * whether the CVE team can also do them unilaterally

Yup, through email :)

> Perhaps since there's no archive for cve@kernel.org, there should be a
> public discussion mailing list (e.g. linux-cve@vger) that maintainers can
> reply to?  The private cve@kernel.org alias would then be just for the
> request of embargoed CVEs.

What's wrong with lkml for discussion?  I'll add a "reply-to" to point
there as well so that it will properly redirect if you respond to a
message sent to the -announce list.

> It would be great if modifications or disputes could simply be sent as
> patches to the vulns.git repo.  You guys can have push hooks or something
> like that that take care of sending messages to linux-cve-announce etc.

Yes, I'll gladly take that as well!

> Another underspecified part is the early request of CVEs.  Some questions I
> have:
> 
> * what information is needed

What ever information you feel is necessary.  What would you do if you
had to make up your mind on this?

At the minimum, a git id :)

> * is there a limit on embargo length similar to security@kernel.org

We have no embargos here, you should NOT be emailing this alias about
any unfixed security things, the document explicitly states this.

Funnily enough, one major Linux vendor has already done just that, and
exposed the fact that they were getting leaked information sent to them
and possibly an embargo is now broken.

The fallout of this is going to be "interesting".

> * should it be acked by the subsystem maintainer

Not needed, but sure, if you want to.

> More in general, I think you're underestimating the extra work for the
> "listeners" of CVEs, that will come from bugs in the script or other
> not-so-well-defined aspects of the process.  I really think it would be a
> good idea to behave "as if" you were already creating CVE, but for now just
> send out the announcements and publish the JSON in a git repo.

How is it any different from the existing "listeners" of CVEs today?  in
fact, it's a world better as the meta-data we are now providing in the
json files is so much better than all of the crud that all of the other
CNAs were putting in there, it's not even a fair comparison.

So far some vendors have said "this is great!"  No one has said "this
sucks", but if you know of any, let us know, and we will be glad to
adjust the information published as needed.

> As we run the experiment for a while, we can get input from interested
> maintainers and third parties.  I am sure I can find someone from the Red
> Hat product security team to explain their desires, clarify how they consume
> CVE announcements, and what simplifies/complicates their job. Their needs
> are probably not that unique.

I've already had soo many threads from the "Red Hat product security
team" including flow charts and other fun things to last me quite a
while already, and that's just in the past few days.

Given that the "Red Hat product security team" was the #1 offender in
creating CVEs against the kernel that caused us so much work and
headaches that it pushed me over the edge to go through all of the extra
work to actually become a CNA, I think it's worth considering what you
are asking for here.

In other words, they know how to contact us, before, we could never
contact them.  This is up to them to decide how to interact, not us.
And remember, the number of RH systems that that team affects is pretty
much in the rounding error of the other Linux systems out there, not
that I'm counting or anything, just saying :)

> > That's the benifit of being a CNA, we can ACTUALLY MODIFY the CVE
> > records, previously it was almost impossible to ever do so.
> 
> Agreed.  There is potential to do much better than before.

Totally agreed, thanks for the questions!

greg k-h

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Paolo Bonzini]

On 2/22/24 13:55, Greg Kroah-Hartman wrote:
>> All that the document says is "the authority to dispute or modify an
>> assigned CVE for a specific kernel change lies solely with the maintainers
>> of the relevant subsystem affected".  But it doesn't say:
>>
>> * how the maintainer would ask for such a modification or dispute
> 
> Email.
> 
>> * if and how anyone else could propose them
> 
> Email.
> 
>> * whether the CVE team can also do them unilaterally
> 
> Yup, through email :)

Now we're talking. :)

>> Perhaps since there's no archive for cve@kernel.org, there should be a
>> public discussion mailing list (e.g. linux-cve@vger) that maintainers can
>> reply to?  The private cve@kernel.org alias would then be just for the
>> request of embargoed CVEs.
> 
> What's wrong with lkml for discussion?  I'll add a "reply-to" to point
> there as well so that it will properly redirect if you respond to a
> message sent to the -announce list.

Well, LKML is not the most searchable archive and subscribing to it puts 
measurable stress on the kernel.org servers (mostly due to gmail 
shenanigans, but still).

Plus (relatively) fine grained mailing lists are really cheap to 
subscribe to lore/NNTP.  So if the reply-to points to LKML + the 
subsystem mailing list for the maintainers + a new ML for the security 
folks (and these three are also CC'd on the announcements, at least the 
last two), that would be nice to have.  I can work on patches to 
vulns.git, for example to integrate with get_maintainer.pl, if you ack 
the idea.

The linux-cve@ mailing list would also be a natural place to send 
patches to vulns.git.

>> It would be great if modifications or disputes could simply be sent as
>> patches to the vulns.git repo.  You guys can have push hooks or something
>> like that that take care of sending messages to linux-cve-announce etc.
> 
> Yes, I'll gladly take that as well!

Would be nice to have that documented.  Again, I can put everything in 
words once we have some agreement.

>> Another underspecified part is the early request of CVEs.  Some questions I
>> have:
>>
>> * what information is needed
> 
> What ever information you feel is necessary.  What would you do if you
> had to make up your mind on this?
> 
> At the minimum, a git id :)
> 
>> * is there a limit on embargo length similar to security@kernel.org
> 
> We have no embargos here, you should NOT be emailing this alias about
> any unfixed security things, the document explicitly states this.

Wait that's not what the docs say:

+If anyone wishes to
+have a CVE assigned before an issue is resolved with a commit, please
+contact the kernel CVE assignment team at <cve@kernel.org> to get an
+identifier assigned from their batch of reserved identifiers.

>> * should it be acked by the subsystem maintainer
> 
> Not needed, but sure, if you want to.

Ok, then we should add something to that end in the docs as well, 
suggesting Cc of the maintainers.

>> More in general, I think you're underestimating the extra work for the
>> "listeners" of CVEs, that will come from bugs in the script or other
>> not-so-well-defined aspects of the process.  I really think it would be a
>> good idea to behave "as if" you were already creating CVE, but for now just
>> send out the announcements and publish the JSON in a git repo.
> 
> How is it any different from the existing "listeners" of CVEs today?  in
> fact, it's a world better as the meta-data we are now providing in the
> json files is so much better than all of the crud that all of the other
> CNAs were putting in there, it's not even a fair comparison.

Well, it's different in that it will at least double the CVEs processed 
every year (potentially more---if we go by the number of GSD entries, 
it's more like 5x), and the delta will be coming from one CNA only.

So it's just that it's a lot of new work.  So far the only thing for 
which I can say "this sucks" is having one CVE per commit in a 
multi-patch fix.  That's somewhat ingrained in the operation of the 
bippy script, but not unfixable (and BTW I wouldn't mind rewriting bippy 
in Python, but that's a different story).

> I've already had soo many threads from the "Red Hat product security
> team" including flow charts and other fun things to last me quite a
> while already, and that's just in the past few days.

Lol yeah I've seen some of those too (but only this morning---I'm not 
writing on their behalf, in case that was unclear).

Paolo

> Given that the "Red Hat product security team" was the #1 offender in
> creating CVEs against the kernel that caused us so much work and
> headaches that it pushed me over the edge to go through all of the extra
> work to actually become a CNA, I think it's worth considering what you
> are asking for here.
> 
> In other words, they know how to contact us, before, we could never
> contact them.  This is up to them to decide how to interact, not us.
> And remember, the number of RH systems that that team affects is pretty
> much in the rounding error of the other Linux systems out there, not
> that I'm counting or anything, just saying :)
>
>>> That's the benifit of being a CNA, we can ACTUALLY MODIFY the CVE
>>> records, previously it was almost impossible to ever do so.
>>
>> Agreed.  There is potential to do much better than before.
> 
> Totally agreed, thanks for the questions!
> 
> greg k-h
> 

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Greg Kroah-Hartman]

Sorry for the delay in response, been a busy week...

On Thu, Feb 22, 2024 at 02:31:06PM +0100, Paolo Bonzini wrote:
> > > Perhaps since there's no archive for cve@kernel.org, there should be a
> > > public discussion mailing list (e.g. linux-cve@vger) that maintainers can
> > > reply to?  The private cve@kernel.org alias would then be just for the
> > > request of embargoed CVEs.
> > 
> > What's wrong with lkml for discussion?  I'll add a "reply-to" to point
> > there as well so that it will properly redirect if you respond to a
> > message sent to the -announce list.
> 
> Well, LKML is not the most searchable archive and subscribing to it puts
> measurable stress on the kernel.org servers (mostly due to gmail
> shenanigans, but still).

lore is a great search tool for lkml, and you can subscribe to feeds
from it very easily.

> Plus (relatively) fine grained mailing lists are really cheap to subscribe
> to lore/NNTP.

Lore picks out stuff from lkml just as easily.

> So if the reply-to points to LKML + the subsystem mailing
> list for the maintainers + a new ML for the security folks (and these three
> are also CC'd on the announcements, at least the last two), that would be
> nice to have.  I can work on patches to vulns.git, for example to integrate
> with get_maintainer.pl, if you ack the idea.

That might be a bit noisy, for some commits, but sure, I can see the
value in being notified about a CVE for my subsystem.  If you have a
specific 'get_maintainer.pl' command line invocation you think would be
good, I can easily add it to the scripts.

> The linux-cve@ mailing list would also be a natural place to send patches to
> vulns.git.

Proliferation of mailing lists are a pain, I'd like to resist that for
now if possible.  Just use lkml and cc: cve@kernel.org if you want to
send us patches.

> > > * is there a limit on embargo length similar to security@kernel.org
> > 
> > We have no embargos here, you should NOT be emailing this alias about
> > any unfixed security things, the document explicitly states this.
> 
> Wait that's not what the docs say:
> 
> +If anyone wishes to
> +have a CVE assigned before an issue is resolved with a commit, please
> +contact the kernel CVE assignment team at <cve@kernel.org> to get an
> +identifier assigned from their batch of reserved identifiers.

The document also says:
	If you feel you have found an unfixed security issue, please
	follow the :doc:`normal Linux kernel security bug reporting
	process<../process/security-bugs>`.

So _IFF_ you have an unfixed security issue that security@k.o knows
about, and you MUST have a CVE id at this point in time (i.e. because it
needs to be referenced somewhere), then you can email cve@k.o to get
one, but the number of times I forsee that being the case based on the
security@k.o workload is going to be very small.

So we can take those on a case-by-case basis please, that's not going to
be a normal occurrence.

> So it's just that it's a lot of new work.  So far the only thing for which I
> can say "this sucks" is having one CVE per commit in a multi-patch fix.
> That's somewhat ingrained in the operation of the bippy script, but not
> unfixable (and BTW I wouldn't mind rewriting bippy in Python, but that's a
> different story).

Yes, the "multi-patch" issue is going to be real, and we will handle it
when it happens.  The current scripts don't handle that well, but they
DO allow us to hand-write entries, which is probably what is going to
happen for those types of rare issues.

And yes, bippy in Python would be nice, but my python skills are bad
(and you didn't want me writing it in perl), so I went with what I could
do in the amount of time I had to get it up and running.

> > I've already had soo many threads from the "Red Hat product security
> > team" including flow charts and other fun things to last me quite a
> > while already, and that's just in the past few days.
> 
> Lol yeah I've seen some of those too (but only this morning---I'm not
> writing on their behalf, in case that was unclear).

I met with someone from Red Hat earlier this week and hopefully most of
this will be resolved soon, I'm waiting to hear back from them for the
issues we discussed.

thanks,

greg k-h

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Greg Kroah-Hartman]

On Thu, Feb 29, 2024 at 06:32:03AM +0100, Greg Kroah-Hartman wrote:
> On Thu, Feb 22, 2024 at 02:31:06PM +0100, Paolo Bonzini wrote:
> > So if the reply-to points to LKML + the subsystem mailing
> > list for the maintainers + a new ML for the security folks (and these three
> > are also CC'd on the announcements, at least the last two), that would be
> > nice to have.  I can work on patches to vulns.git, for example to integrate
> > with get_maintainer.pl, if you ack the idea.
> 
> That might be a bit noisy, for some commits, but sure, I can see the
> value in being notified about a CVE for my subsystem.  If you have a
> specific 'get_maintainer.pl' command line invocation you think would be
> good, I can easily add it to the scripts.

Would:
	--no-keywords --no-git --no-git-fallback --norolestats --nol

be a good pattern to follow?

thanks,

greg k-h

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Paolo Bonzini]

On Thu, Feb 29, 2024 at 7:05 AM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
>
> On Thu, Feb 29, 2024 at 06:32:03AM +0100, Greg Kroah-Hartman wrote:
> > On Thu, Feb 22, 2024 at 02:31:06PM +0100, Paolo Bonzini wrote:
> > > So if the reply-to points to LKML + the subsystem mailing
> > > list for the maintainers + a new ML for the security folks (and these three
> > > are also CC'd on the announcements, at least the last two), that would be
> > > nice to have.  I can work on patches to vulns.git, for example to integrate
> > > with get_maintainer.pl, if you ack the idea.
> >
> > That might be a bit noisy, for some commits, but sure, I can see the
> > value in being notified about a CVE for my subsystem.  If you have a
> > specific 'get_maintainer.pl' command line invocation you think would be
> > good, I can easily add it to the scripts.
>
> Would:
>         --no-keywords --no-git --no-git-fallback --norolestats --nol
>
> be a good pattern to follow?

I would include lists as well. it would be nice to exclude
reviewed-bys but that's not easy to do in get_maintainer.pI

Paolo

Re: CVE-2023-52437: Revert "md/raid5: Wait for MD_SB_CHANGE_PENDING in raid5d"

[Greg Kroah-Hartman]

On Thu, Feb 29, 2024 at 11:53:56AM +0100, Paolo Bonzini wrote:
> On Thu, Feb 29, 2024 at 7:05 AM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > On Thu, Feb 29, 2024 at 06:32:03AM +0100, Greg Kroah-Hartman wrote:
> > > On Thu, Feb 22, 2024 at 02:31:06PM +0100, Paolo Bonzini wrote:
> > > > So if the reply-to points to LKML + the subsystem mailing
> > > > list for the maintainers + a new ML for the security folks (and these three
> > > > are also CC'd on the announcements, at least the last two), that would be
> > > > nice to have.  I can work on patches to vulns.git, for example to integrate
> > > > with get_maintainer.pl, if you ack the idea.
> > >
> > > That might be a bit noisy, for some commits, but sure, I can see the
> > > value in being notified about a CVE for my subsystem.  If you have a
> > > specific 'get_maintainer.pl' command line invocation you think would be
> > > good, I can easily add it to the scripts.
> >
> > Would:
> >         --no-keywords --no-git --no-git-fallback --norolestats --nol
> >
> > be a good pattern to follow?
> 
> I would include lists as well. it would be nice to exclude
> reviewed-bys but that's not easy to do in get_maintainer.pI

That would hit lkml for everything, which is probably not a good idea :(

I think maintainers for now would be a good start, let me see about
adding that the next chance I get.

thanks,

greg k-h