[Qemu-devel] QEMU CII Best Practices record

[Qemu-devel] QEMU CII Best Practices record

[Daniel P. Berrange]

Many projects these days are recording progress wrt CII best practices
for FLOOS projects. I filled out a record for QEMU:

  https://bestpractices.coreinfrastructure.org/projects/1309

I only looked at the 'Passing' criteria, not considered the 'Silver' and
'Gold' criteria. So if anyone else wants to contribute, register an
account there and tell me the username whereupon I can add you as a
collaborator.

Two items I don't think QEMU achieves for the basic "Passing" criteria

 -  The release notes MUST identify every publicly known vulnerability
    that is fixed in each new release.

    I don't see a list of CVEs mentioned in our release Changelogs or
    indeed a historic list of CVEs anywhere even outside the release
    notes ?

 - It is SUGGESTED that if the software produced by the project includes
   software written using a memory-unsafe language (e.g., C or C++), then
   at least one dynamic tool (e.g., a fuzzer or web application scanner)
   be routinely used in combination with a mechanism to detect memory
   safety problems such as buffer overwrites.

   NB this is not 'coverity' which falls under the 'static anlaysis'
   group. I'm unclear if anyone in the community does regular fuzzing
   or analysis with ASAN & equiv ?

If i'm wrong just say....

There's many questions under Silver/Gold level we likely don't meet and
some of them start to get quiet opinionated about the way a project
should be run, so IMHO its not unreasonable to say we're not going to aim
for perfection in this respect.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QEMU CII Best Practices record

[Stefan Hajnoczi]

On Fri, Oct 13, 2017 at 02:25:07PM +0100, Daniel P. Berrange wrote:
> Many projects these days are recording progress wrt CII best practices
> for FLOOS projects. I filled out a record for QEMU:
> 
>   https://bestpractices.coreinfrastructure.org/projects/1309
> 
> I only looked at the 'Passing' criteria, not considered the 'Silver' and
> 'Gold' criteria. So if anyone else wants to contribute, register an
> account there and tell me the username whereupon I can add you as a
> collaborator.
> 
> Two items I don't think QEMU achieves for the basic "Passing" criteria
> 
>  -  The release notes MUST identify every publicly known vulnerability
>     that is fixed in each new release.
> 
>     I don't see a list of CVEs mentioned in our release Changelogs or
>     indeed a historic list of CVEs anywhere even outside the release
>     notes ?
> 
>  - It is SUGGESTED that if the software produced by the project includes
>    software written using a memory-unsafe language (e.g., C or C++), then
>    at least one dynamic tool (e.g., a fuzzer or web application scanner)
>    be routinely used in combination with a mechanism to detect memory
>    safety problems such as buffer overwrites.
> 
>    NB this is not 'coverity' which falls under the 'static anlaysis'
>    group. I'm unclear if anyone in the community does regular fuzzing
>    or analysis with ASAN & equiv ?

I'm not aware of automated ASAN or Valgrind runs although developers
tend to run them in ad-hoc fashion during development.

Stefan

Re: [Qemu-devel] QEMU CII Best Practices record

[Peter Maydell]

On 13 October 2017 at 14:25, Daniel P. Berrange <berrange@redhat.com> wrote:
> Many projects these days are recording progress wrt CII best practices
> for FLOOS projects. I filled out a record for QEMU:
>
>   https://bestpractices.coreinfrastructure.org/projects/1309
>
> I only looked at the 'Passing' criteria, not considered the 'Silver' and
> 'Gold' criteria. So if anyone else wants to contribute, register an
> account there and tell me the username whereupon I can add you as a
> collaborator.

For the questions about "50% of bug reports must be acknowledged"
and ditto enhancement requests, did you mine the launchpad data
or are you just guessing? :-) Similarly for vulnerability report
response time.

I think you're fudging the test-policy questions in our favour a bit.

>  -  The release notes MUST identify every publicly known vulnerability
>     that is fixed in each new release.
>
>     I don't see a list of CVEs mentioned in our release Changelogs or
>     indeed a historic list of CVEs anywhere even outside the release
>     notes ?

Indeed I don't think we do this. I would say that as a project we
essentially push the job of rolling new releases for CVEs, informing
users about CVE fixes, etc, to our downstream distributors.

I suspect we only pass the "no vulns unpatched for more than 60 days"
if you allow "patched in bleeding edge master and in distros
but not in any upstream release" to count.

thanks
-- PMM

Re: [Qemu-devel] QEMU CII Best Practices record

[Daniel P. Berrange]

On Mon, Oct 23, 2017 at 06:55:44PM +0100, Peter Maydell wrote:
> On 13 October 2017 at 14:25, Daniel P. Berrange <berrange@redhat.com> wrote:
> > Many projects these days are recording progress wrt CII best practices
> > for FLOOS projects. I filled out a record for QEMU:
> >
> >   https://bestpractices.coreinfrastructure.org/projects/1309
> >
> > I only looked at the 'Passing' criteria, not considered the 'Silver' and
> > 'Gold' criteria. So if anyone else wants to contribute, register an
> > account there and tell me the username whereupon I can add you as a
> > collaborator.
> 
> For the questions about "50% of bug reports must be acknowledged"
> and ditto enhancement requests, did you mine the launchpad data
> or are you just guessing? :-) Similarly for vulnerability report
> response time.

I didn't measure it, just used gut feeling. I see people like Thomas Huth
and David Gilbert in particular responding to many bugs which come in and
triaging existing bugs. So I think we're in the ballpark give or take 10%.

For vulnerability reports I think we get good response, between QEMU's secalert
team, and the distros security teams, we've got good coverage & response.

> I think you're fudging the test-policy questions in our favour a bit.

IMHO the way the CII website is setup with everyone self-certifying,
means it is largely a game. I view it is a way of identifying notable
gaps where we might consider improving our working practice, and as a
rough guide to outsides to understand our project, rather than a 100%
accurate reflection of what we do.

But if people think I've got something that is grossly inaccurate
please do point it out.



> >  -  The release notes MUST identify every publicly known vulnerability
> >     that is fixed in each new release.
> >
> >     I don't see a list of CVEs mentioned in our release Changelogs or
> >     indeed a historic list of CVEs anywhere even outside the release
> >     notes ?
> 
> Indeed I don't think we do this. I would say that as a project we
> essentially push the job of rolling new releases for CVEs, informing
> users about CVE fixes, etc, to our downstream distributors.

> I suspect we only pass the "no vulns unpatched for more than 60 days"
> if you allow "patched in bleeding edge master and in distros
> but not in any upstream release" to count.

I think patched in git master is sufficient to consider it a pass on the
criteria - they don't mention any specifics about having to maintain
multiple stable branches and backport.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QEMU CII Best Practices record

[Daniel P. Berrange]

On Tue, Oct 24, 2017 at 08:42:03AM +0100, Daniel P. Berrange wrote:
> On Mon, Oct 23, 2017 at 06:55:44PM +0100, Peter Maydell wrote:
> > On 13 October 2017 at 14:25, Daniel P. Berrange <berrange@redhat.com> wrote:
> > > Many projects these days are recording progress wrt CII best practices
> > > for FLOOS projects. I filled out a record for QEMU:
> > >
> > >   https://bestpractices.coreinfrastructure.org/projects/1309
> > >
> > > I only looked at the 'Passing' criteria, not considered the 'Silver' and
> > > 'Gold' criteria. So if anyone else wants to contribute, register an
> > > account there and tell me the username whereupon I can add you as a
> > > collaborator.
> > 
> > For the questions about "50% of bug reports must be acknowledged"
> > and ditto enhancement requests, did you mine the launchpad data
> > or are you just guessing? :-) Similarly for vulnerability report
> > response time.
> 
> I didn't measure it, just used gut feeling. I see people like Thomas Huth
> and David Gilbert in particular responding to many bugs which come in and
> triaging existing bugs. So I think we're in the ballpark give or take 10%.
> 
> For vulnerability reports I think we get good response, between QEMU's secalert
> team, and the distros security teams, we've got good coverage & response.
> 
> > I think you're fudging the test-policy questions in our favour a bit.
> 
> IMHO the way the CII website is setup with everyone self-certifying,
> means it is largely a game. I view it is a way of identifying notable
> gaps where we might consider improving our working practice, and as a
> rough guide to outsides to understand our project, rather than a 100%
> accurate reflection of what we do.
> 
> But if people think I've got something that is grossly inaccurate
> please do point it out.
> 
> 
> 
> > >  -  The release notes MUST identify every publicly known vulnerability
> > >     that is fixed in each new release.
> > >
> > >     I don't see a list of CVEs mentioned in our release Changelogs or
> > >     indeed a historic list of CVEs anywhere even outside the release
> > >     notes ?
> > 
> > Indeed I don't think we do this. I would say that as a project we
> > essentially push the job of rolling new releases for CVEs, informing
> > users about CVE fixes, etc, to our downstream distributors.
> 
> > I suspect we only pass the "no vulns unpatched for more than 60 days"
> > if you allow "patched in bleeding edge master and in distros
> > but not in any upstream release" to count.
> 
> I think patched in git master is sufficient to consider it a pass on the
> criteria - they don't mention any specifics about having to maintain
> multiple stable branches and backport.

That said, I wonder if we should put 'security response handling' on the
agenda for the QEMU mini summit tomorrow. In particular I think it is
pretty bad that we don't publish any list of what CVEs affect QEMU and
the GIT hash of the corresponding GIT master fix, nor mention them in
the release notes for each major release. 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QEMU CII Best Practices record

[Peter Maydell]

On 24 October 2017 at 08:46, Daniel P. Berrange <berrange@redhat.com> wrote:
> That said, I wonder if we should put 'security response handling' on the
> agenda for the QEMU mini summit tomorrow. In particular I think it is
> pretty bad that we don't publish any list of what CVEs affect QEMU and
> the GIT hash of the corresponding GIT master fix, nor mention them in
> the release notes for each major release.

Yep, happy to talk about that. Personally I'd like to see us doing
better here, but since I don't have the time to do it myself I can
understand if nobody else has the time to do it either :-)

thanks
-- PMM