Re: [PATCH v3 1/3] cve-report: add scripts to generate CVE reports

[Grygorii Tertychnyi <gtertych@cisco.com>]

On Mon Oct29 2018 @ 23:29, akuster808 <akuster808@gmail.com> 
wrote:

> Grygorii,
>
> I was good to see you at OEDeM.

Thanks Armin.

> I have some feedback.
>
> On 10/10/18 9:25 AM, grygorii tertychnyi via Openembedded-core 
> wrote:
>> cvert-foss - generate CVE report for the list of packages.
>>   Analyze the whole image manifest to align with the complex
>>   CPE configurations.
>>
>> cvert-update - update NVD feeds and store CVE structues dump.
>>   CVE dump is a pickled representation of the cve_struct 
>>   dictionary.
>>
>> cvert.py - python library used by cvert-* scripts.
>>   NVD JSON Vulnerability Feeds 
>>   https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
>>
>> Usage examples:
>>
>>   o Download CVE feeds to "nvdfeed" directory
>>     % cvert-update nvdfeed
>>   o Update CVE feeds and store a dump in a file
>>     % cvert-update --store cvedump nvdfeed
>>   o Generate a CVE report
>>     % cvert-foss --feed-dir nvdfeed --output report-foss.txt 
>>     cve-manifest
>>   o (faster) Use dump file to generate a CVE report
>>     % cvert-foss --restore cvedump --output report-foss.txt 
>>     cve-manifest
>>   o Generate a full report
>>     % cvert-foss --restore cvedump --show-description 
>>     --show-reference \
>>                  --output report-foss-full.txt cve-manifest 
>>                  report-foss-full.txt

Oh, there is a typo. "cvert-foss" script requires only one 
positional argument - "cve-manifest".
So, the "Generate a full report" command should look like this:

       % cvert-foss --restore cvedump --show-description 
       --show-reference \
                    --output report-foss-full.txt cve-manifest

Is not a big deal. Just to be on the same page.

>
> The cve-manifest, I could not figure out a way to create this 
> from the
> above steps. I had your patches included when I did an image 
> build then
> I got the "report-foss-full.txt" created. I am guessing I missed 
> some
> steps somewhere. Can you clarify?

I try to explain it this way:

There are three patches:
  o 1st is "cve-report: add scripts to generate CVE reports"
    "cvert-foss" takes the "cve-manifest" (input), and generates
    the CVE report (output). So, you are right, these scripts
    do not produce "cve-manifest" files.

  o 2nd is "cvert-kernel - generate CVE report for the Linux 
  kernel"
    "cvert-kernel" analyzes kernel git dir and generate the kernel 
    CVE report

  o 3rd is "cve-report.bbclass: add class"
    it generates "cve-manifest" file for the given <image> and 
    then
    calls "cvert-foss" to produce the CVE report.

So, if you applied all three patches, modified "local.conf" and 
run "bitbake -c report_cve <image>" the "cve-manifest" is 
generated as part of the "generate_report_handler" function of 
"cve-report.bbclass".

So, back to you question "how to generate cve-manifest", the 
easiest way is:

echo 'INHERIT += "cve-report"' >> conf/local.conf
echo 'CVE_REPORT_MODE[packageonly] = "1"' >> conf/local.conf
bitbake -c report_cve <image>

It is described in the 3rd patch. Here, in the 1st patch I 
described only the "cve-manifest" format. Same format is used by 
"cve-check-tool", it is well-known and simply enough, so, no need 
to change it.

Being a part of scripts directory it can be used outside of any 
bbclass. You can create simple manifests and generate CVE reports 
right on command line, simple example:

 % echo "python,3.5.5,CVE-2017-17522 CVE-2018-1061" > 
 cve-manifest.txt
 % ./cvert-foss --debug --feed-dir cve-feeds --output 
 report-foss.txt cve-manifest.txt
# 2018-10-29 19:27:48,673 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.meta
# 2018-10-29 19:27:50,250 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2002.json.gz
# 2018-10-29 19:27:53,537 % CVERT % DEBUG    % parsing year 2002
# 2018-10-29 19:27:53,575 % CVERT % DEBUG    % cve records: 6667
# 2018-10-29 19:27:53,605 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2003.meta
# 2018-10-29 19:27:55,121 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2003.json.gz
# 2018-10-29 19:27:57,360 % CVERT % DEBUG    % parsing year 2003
# 2018-10-29 19:27:57,369 % CVERT % DEBUG    % cve records: 8167
# 2018-10-29 19:27:57,382 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2004.meta
# 2018-10-29 19:27:58,825 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2004.json.gz
# 2018-10-29 19:28:01,595 % CVERT % DEBUG    % parsing year 2004
# 2018-10-29 19:28:01,613 % CVERT % DEBUG    % cve records: 10810
# 2018-10-29 19:28:01,636 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2005.meta
# 2018-10-29 19:28:03,008 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2005.json.gz
# 2018-10-29 19:28:06,088 % CVERT % DEBUG    % parsing year 2005
# 2018-10-29 19:28:06,119 % CVERT % DEBUG    % cve records: 15424
# 2018-10-29 19:28:06,152 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2006.meta
# 2018-10-29 19:28:07,627 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2006.json.gz
# 2018-10-29 19:28:11,392 % CVERT % DEBUG    % parsing year 2006
# 2018-10-29 19:28:11,439 % CVERT % DEBUG    % cve records: 22408
# 2018-10-29 19:28:11,489 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.meta
# 2018-10-29 19:28:13,163 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2007.json.gz
# 2018-10-29 19:28:17,432 % CVERT % DEBUG    % parsing year 2007
# 2018-10-29 19:28:17,478 % CVERT % DEBUG    % cve records: 28850
# 2018-10-29 19:28:17,530 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2008.meta
# 2018-10-29 19:28:19,195 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2008.json.gz
# 2018-10-29 19:28:22,937 % CVERT % DEBUG    % parsing year 2008
# 2018-10-29 19:28:23,489 % CVERT % DEBUG    % cve records: 35840
# 2018-10-29 19:28:23,556 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2009.meta
# 2018-10-29 19:28:25,127 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2009.json.gz
# 2018-10-29 19:28:29,013 % CVERT % DEBUG    % parsing year 2009
# 2018-10-29 19:28:29,049 % CVERT % DEBUG    % cve records: 40705
# 2018-10-29 19:28:29,104 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2010.meta
# 2018-10-29 19:28:31,082 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2010.json.gz
# 2018-10-29 19:28:35,571 % CVERT % DEBUG    % parsing year 2010
# 2018-10-29 19:28:35,609 % CVERT % DEBUG    % cve records: 45638
# 2018-10-29 19:28:35,677 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2011.meta
# 2018-10-29 19:28:37,263 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2011.json.gz
# 2018-10-29 19:28:43,598 % CVERT % DEBUG    % parsing year 2011
# 2018-10-29 19:28:43,630 % CVERT % DEBUG    % cve records: 50043
# 2018-10-29 19:28:43,739 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2012.meta
# 2018-10-29 19:28:45,264 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2012.json.gz
# 2018-10-29 19:28:49,718 % CVERT % DEBUG    % parsing year 2012
# 2018-10-29 19:28:49,755 % CVERT % DEBUG    % cve records: 55215
# 2018-10-29 19:28:49,830 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2013.meta
# 2018-10-29 19:28:51,455 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2013.json.gz
# 2018-10-29 19:28:56,101 % CVERT % DEBUG    % parsing year 2013
# 2018-10-29 19:28:56,142 % CVERT % DEBUG    % cve records: 60882
# 2018-10-29 19:28:56,210 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2014.meta
# 2018-10-29 19:28:57,977 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2014.json.gz
# 2018-10-29 19:29:02,677 % CVERT % DEBUG    % parsing year 2014
# 2018-10-29 19:29:03,693 % CVERT % DEBUG    % cve records: 68824
# 2018-10-29 19:29:03,783 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2015.meta
# 2018-10-29 19:29:05,820 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2015.json.gz
# 2018-10-29 19:29:09,577 % CVERT % DEBUG    % parsing year 2015
# 2018-10-29 19:29:09,632 % CVERT % DEBUG    % cve records: 76251
# 2018-10-29 19:29:09,720 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2016.meta
# 2018-10-29 19:29:11,362 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2016.json.gz
# 2018-10-29 19:29:17,063 % CVERT % DEBUG    % parsing year 2016
# 2018-10-29 19:29:17,128 % CVERT % DEBUG    % cve records: 84957
# 2018-10-29 19:29:17,238 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2017.meta
# 2018-10-29 19:29:19,102 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2017.json.gz
# 2018-10-29 19:29:25,282 % CVERT % DEBUG    % parsing year 2017
# 2018-10-29 19:29:25,377 % CVERT % DEBUG    % cve records: 98334
# 2018-10-29 19:29:25,564 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2018.meta
# 2018-10-29 19:29:27,195 % CVERT % DEBUG    % downloading 
  https://static.nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-2018.json.gz
# 2018-10-29 19:29:32,262 % CVERT % DEBUG    % parsing year 2018
# 2018-10-29 19:29:32,323 % CVERT % DEBUG    % cve records: 106349
# 2018-10-29 19:29:32,912 % CVERT % DEBUG    % match python 3.5.5: 
  cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,265 % CVERT % DEBUG    % ignore "*" in 
  cpe:2.3:a:python_software_foundation:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,271 % CVERT % DEBUG    % match python 3.5.5: 
  cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,271 % CVERT % DEBUG    % unpatched 
  CVE-2017-18207 python,3.5.5
# 2018-10-29 19:29:33,486 % CVERT % DEBUG    % match python 3.5.5: 
  cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:33,487 % CVERT % DEBUG    %   patched 
  CVE-2018-1061 python,3.5.5
# 2018-10-29 19:29:33,772 % CVERT % DEBUG    % ignore "-" in 
  cpe:2.3:a:python:python:-:*:*:*:*:*:*:*
# 2018-10-29 19:29:34,585 % CVERT % DEBUG    % match python 3.5.5: 
  cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:34,585 % CVERT % DEBUG    % unpatched 
  CVE-2018-1060 python,3.5.5
# 2018-10-29 19:29:36,668 % CVERT % DEBUG    % match python 3.5.5: 
  cpe:2.3:a:python:python:*:*:*:*:*:*:*:*
# 2018-10-29 19:29:36,668 % CVERT % DEBUG    %   patched 
  CVE-2017-17522 python,3.5.5
 % cat report-foss.txt
  patched |  7.5 | CVE-2018-1061      | python | 3.5.5
  patched |  8.8 | CVE-2017-17522     | python | 3.5.5
unpatched |  6.5 | CVE-2017-18207     | python | 3.5.5
unpatched |  7.5 | CVE-2018-1060      | python | 3.5.5

Of course, it is better to use first "cvert-update --store ..." 
and then multiple times "cvert-foss --restore ..." - much faster 
on multiple runs.

> Also, is there a way to version the report file by image or 
> standard
> buildID so that I don't accidentally overwrite the txt 
> file. Maybe a
> future enhancement is to group the packages by layer. I ran this 
> with
> the meta-security layer and I now need to look up what packages 
> belong
> to what layer to go work on them.

I think it is possible, but again as part of bbclass 
enhancement. If I understood Richard correctly on OEDeM, the 
bbclass should be modified anyway ("do not copy 
functionality"). But these cvert scripts do not depend on it. And 
can be seen independently.

> Overall, I like this implementation and seems to be fast.
>
> Thanks for the patches.

Thanks. It is inspiring.

> kind regards,
>
> Armin
>
>>
>> Manifest example:
>>
>>   bash,4.2,CVE-2014-7187
>>   python,2.7.35,
>>   python,3.5.5,CVE-2017-17522 CVE-2018-1061
>>
>> Report example:
>>
>>     patched |  7.5 | CVE-2018-1061      | python | 3.5.5
>>     patched | 10.0 | CVE-2014-7187      | bash | 4.2
>>     patched |  8.8 | CVE-2017-17522     | python | 3.5.5
>>   unpatched | 10.0 | CVE-2014-6271      | bash | 4.2
>>   unpatched | 10.0 | CVE-2014-6277      | bash | 4.2
>>   unpatched | 10.0 | CVE-2014-6278      | bash | 4.2
>>   unpatched | 10.0 | CVE-2014-7169      | bash | 4.2
>>   unpatched | 10.0 | CVE-2014-7186      | bash | 4.2
>>   unpatched |  4.6 | CVE-2012-3410      | bash | 4.2
>>   unpatched |  8.4 | CVE-2016-7543      | bash | 4.2
>>   unpatched |  5.0 | CVE-2010-3492      | python | 2.7.35
>>   unpatched |  5.3 | CVE-2016-1494      | python | 2.7.35
>>   unpatched |  6.5 | CVE-2017-18207     | python | 3.5.5
>>   unpatched |  6.5 | CVE-2017-18207     | python | 2.7.35
>>   unpatched |  7.1 | CVE-2013-7338      | python | 2.7.35
>>   unpatched |  7.5 | CVE-2018-1060      | python | 3.5.5
>>   unpatched |  8.8 | CVE-2017-17522     | python | 2.7.35
>>
>> Signed-off-by: grygorii tertychnyi <gtertych@cisco.com>
>> ---
>>
>> Changes in v3:
>>  o better logging: cvert.py lib log messages are controlled by 
>>  cvert-* scripts
>>  o add more examples
>>  o add short params ("-o" "--output", "-f", "--feed-dir", etc)
>>  o fix double entries in manifest
>>  o fix pylint warnings
>>
>>  scripts/cvert-foss   | 151 ++++++++++++++++
>>  scripts/cvert-update |  79 +++++++++
>>  scripts/cvert.py     | 473 
>>  +++++++++++++++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 703 insertions(+)
>>  create mode 100755 scripts/cvert-foss
>>  create mode 100755 scripts/cvert-update
>>  create mode 100644 scripts/cvert.py
>>
>> diff --git a/scripts/cvert-foss b/scripts/cvert-foss
>> new file mode 100755
>> index 000000000000..00fbf2c0687b
>> --- /dev/null
>> +++ b/scripts/cvert-foss
>> @@ -0,0 +1,151 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (c) 2018 by Cisco Systems, Inc.
>> +#
>> +# This program is free software; you can redistribute it 
>> and/or modify
>> +# it under the terms of the GNU General Public License version 
>> 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be 
>> useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty 
>> of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See 
>> the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public 
>> License along
>> +# with this program; if not, write to the Free Software 
>> Foundation, Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +
>> +""" Generate CVE report for the given CVE manifest
>> +"""
>> +
>> +import sys
>> +import textwrap
>> +import argparse
>> +import logging
>> +import logging.config
>> +import cvert
>> +
>> +def report_foss():
>> +    """Generate CVE report"""
>> +
>> +    parser = argparse.ArgumentParser(
>> +        formatter_class=argparse.RawDescriptionHelpFormatter,
>> +        description=textwrap.dedent("""
>> +        Generate CVE report for the given CVE manifest.
>> +        """),
>> +        epilog=textwrap.dedent("""
>> +        @ run examples:
>> +
>> +        # Download (update) NVD feeds in "nvdfeed" directory
>> +        # and prepare the report for the "cve-manifest" file
>> +        %% %(prog)s --feed-dir nvdfeed --output 
>> report-foss.txt cve-manifest
>> +
>> +        # Use existed NVD feeds in "nvdfeed" directory
>> +        # and prepare the report for the "cve-manifest" file
>> +        %% %(prog)s --offline --feed-dir nvdfeed --output 
>> report-foss.txt cve-manifest
>> +
>> +        # (faster) Restore CVE dump from "cvedump" (must 
>> exist)
>> +        # and prepare the report for the "cve-manifest" file
>> +        %% %(prog)s --restore cvedump --output report-foss.txt 
>> cve-manifest
>> +
>> +        # Restore CVE dump from "cvedump" (must exist)
>> +        # and prepare the extended report for the 
>> "cve-manifest" file
>> +        %% %(prog)s --restore cvedump --show-description 
>> --show-reference --output report-foss.txt cve-manifest
>> +
>> +        @ manifest example:
>> +
>> +        bash,4.2,CVE-2014-7187
>> +        python,2.7.35,
>> +        python,3.5.5,CVE-2017-17522 CVE-2018-1061
>> +
>> +        @ report example output:
>> +
>> +        . patched | 10.0 | CVE-2014-7187      | bash | 4.2
>> +        . patched |  7.5 | CVE-2018-1061      | python | 3.5.5
>> +        . patched |  8.8 | CVE-2017-17522     | python | 3.5.5
>> +        unpatched | 10.0 | CVE-2014-6271      | bash | 4.2
>> +        unpatched | 10.0 | CVE-2014-6277      | bash | 4.2
>> +        unpatched | 10.0 | CVE-2014-6278      | bash | 4.2
>> +        unpatched | 10.0 | CVE-2014-7169      | bash | 4.2
>> +        unpatched | 10.0 | CVE-2014-7186      | bash | 4.2
>> +        unpatched |  4.6 | CVE-2012-3410      | bash | 4.2
>> +        unpatched |  8.4 | CVE-2016-7543      | bash | 4.2
>> +        unpatched |  5.0 | CVE-2010-3492      | python | 
>> 2.7.35
>> +        unpatched |  5.3 | CVE-2016-1494      | python | 
>> 2.7.35
>> +        unpatched |  6.5 | CVE-2017-18207     | python | 3.5.5
>> +        unpatched |  6.5 | CVE-2017-18207     | python | 
>> 2.7.35
>> +        unpatched |  7.1 | CVE-2013-7338      | python | 
>> 2.7.35
>> +        unpatched |  7.5 | CVE-2018-1060      | python | 3.5.5
>> +        unpatched |  8.8 | CVE-2017-17522     | python | 
>> 2.7.35
>> +        """))
>> +
>> +    group = parser.add_mutually_exclusive_group(required=True)
>> +    group.add_argument("-f", "--feed-dir", help="feeds 
>> directory")
>> +    group.add_argument("-d", "--restore", help="load CVE data 
>> structures from file",
>> +                       metavar="FILENAME")
>> +    parser.add_argument("--offline", help="do not update from 
>> NVD site",
>> +                        action="store_true")
>> +    parser.add_argument("-o", "--output", help="save report to 
>> the file")
>> +    parser.add_argument("--show-description", help='show 
>> "Description" in the report',
>> +                        action="store_true")
>> +    parser.add_argument("--show-reference", help='show 
>> "Reference" in the report',
>> +                        action="store_true")
>> +    parser.add_argument("--debug", help="print debug 
>> messages",
>> +                        action="store_true")
>> +
>> +    parser.add_argument("cve_manifest", help="file with a list 
>> of packages, "
>> +                        "each line contains three comma 
>> separated values: name, "
>> +                        "version and a space separated list of 
>> patched CVEs, "
>> +                        "e.g.: python,3.5.5,CVE-2017-17522 
>> CVE-2018-1061",
>> +                        metavar="cve-manifest")
>> +
>> +    args = parser.parse_args()
>> +
>> +    logging.config.dictConfig(cvert.logconfig(args.debug))
>> +
>> +    cve_manifest = {}
>> +
>> +    with open(args.cve_manifest, "r") as fil:
>> +        for lin in fil:
>> +            lin = lin.rstrip()
>> +
>> +            # skip empty lines
>> +            if not lin:
>> +                continue
>> +
>> +            product, version, patched = lin.split(",", 
>> maxsplit=3)
>> +
>> +            if product in cve_manifest:
>> +                cve_manifest[product][version] = 
>> patched.split()
>> +            else:
>> +                cve_manifest[product] = {
>> +                    version: patched.split()
>> +                }
>> +
>> +    if args.restore:
>> +        cve_struct = cvert.load_cve(args.restore)
>> +    elif args.feed_dir:
>> +        cve_struct = cvert.update_feeds(args.feed_dir, 
>> args.offline)
>> +
>> +    if not cve_struct and args.offline:
>> +        parser.error("No CVEs found. Try to turn off offline 
>> mode or use other file to restore.")
>> +
>> +    if args.output:
>> +        output = open(args.output, "w")
>> +    else:
>> +        output = sys.stdout
>> +
>> +    report = cvert.generate_report(cve_manifest, cve_struct)
>> +
>> +    cvert.print_report(report,
>> +                       show_description=args.show_description,
>> +                       show_reference=args.show_reference,
>> +                       output=output)
>> +
>> +    if args.output:
>> +        output.close()
>> +
>> +
>> +if __name__ == "__main__":
>> +    report_foss()
>> diff --git a/scripts/cvert-update b/scripts/cvert-update
>> new file mode 100755
>> index 000000000000..3b3f5572a83c
>> --- /dev/null
>> +++ b/scripts/cvert-update
>> @@ -0,0 +1,79 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (c) 2018 by Cisco Systems, Inc.
>> +#
>> +# This program is free software; you can redistribute it 
>> and/or modify
>> +# it under the terms of the GNU General Public License version 
>> 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be 
>> useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty 
>> of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See 
>> the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public 
>> License along
>> +# with this program; if not, write to the Free Software 
>> Foundation, Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +
>> +""" Update NVD feeds and store CVE blob locally
>> +"""
>> +
>> +
>> +import textwrap
>> +import argparse
>> +import logging
>> +import logging.config
>> +import cvert
>> +
>> +
>> +def update_cvert():
>> +    """Update CVE storage"""
>> +
>> +    parser = argparse.ArgumentParser(
>> +        formatter_class=argparse.RawDescriptionHelpFormatter,
>> +        description=textwrap.dedent("""
>> +        Update NVD feeds and store CVE blob locally.
>> +        """),
>> +        epilog=textwrap.dedent("""
>> +        examples:
>> +
>> +        # Download NVD feeds to  "nvdfeed" directory.
>> +        # If there are meta files in the directory, they will 
>> be updated
>> +        # and only fresh archives will be downloaded
>> +        %% %(prog)s nvdfeed
>> +
>> +        # Inspect NVD feeds in "nvdfeed" directory
>> +        # and prepare a CVE dump python blob "cvedump".
>> +        # Use it later as input for cvert-* scripts (for 
>> speeding up)
>> +        %% %(prog)s --offline --store cvedump nvdfeed
>> +
>> +        # Download (update) NVD feeds and prepare the CVE dump
>> +        %% %(prog)s --store cvedump nvdfeed
>> +        """))
>> +
>> +    parser.add_argument("-d", "--store", help="save CVE data 
>> structures in file",
>> +                        metavar="FILENAME")
>> +    parser.add_argument("--offline", help="do not update from 
>> NVD site",
>> +                        action="store_true")
>> +    parser.add_argument("--debug", help="print debug 
>> messages",
>> +                        action="store_true")
>> +
>> +    parser.add_argument("feed_dir", help="feeds directory",
>> +                        metavar="feed-dir")
>> +
>> +    args = parser.parse_args()
>> +
>> +    logging.config.dictConfig(cvert.logconfig(args.debug))
>> +
>> +    cve_struct = cvert.update_feeds(args.feed_dir, 
>> args.offline)
>> +
>> +    if not cve_struct and args.offline:
>> +        parser.error("No CVEs found in {0}. Try turn off 
>> offline mode.".format(args.feed_dir))
>> +
>> +    if args.store:
>> +        cvert.save_cve(args.store, cve_struct)
>> +
>> +
>> +if __name__ == "__main__":
>> +    update_cvert()
>> diff --git a/scripts/cvert.py b/scripts/cvert.py
>> new file mode 100644
>> index 000000000000..f93b95c84965
>> --- /dev/null
>> +++ b/scripts/cvert.py
>> @@ -0,0 +1,473 @@
>> +#!/usr/bin/env python3
>> +#
>> +# Copyright (c) 2018 by Cisco Systems, Inc.
>> +#
>> +# This program is free software; you can redistribute it 
>> and/or modify
>> +# it under the terms of the GNU General Public License version 
>> 2 as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it will be 
>> useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty 
>> of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See 
>> the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public 
>> License along
>> +# with this program; if not, write to the Free Software 
>> Foundation, Inc.,
>> +# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
>> +#
>> +
>> +""" CVERT library: set of functions for CVE reports
>> +"""
>> +
>> +
>> +import os
>> +import re
>> +import sys
>> +import json
>> +import gzip
>> +import pickle
>> +import logging
>> +import hashlib
>> +import datetime
>> +import textwrap
>> +import urllib.request
>> +import distutils.version
>> +
>> +
>> +logging.getLogger(__name__).addHandler(logging.NullHandler())
>> +
>> +
>> +def generate_report(manifest, cve_struct):
>> +    """Generate CVE report"""
>> +
>> +    report = []
>> +
>> +    for cve in cve_struct:
>> +        affected = set()
>> +
>> +        for conf in cve_struct[cve]["nodes"]:
>> +            affected = 
>> affected.union(process_configuration(manifest, conf))
>> +
>> +        for key in affected:
>> +            product, version = key.split(",")
>> +            patched = manifest[product][version]
>> +
>> +            if cve in patched:
>> +                cve_item = {"status": "patched"}
>> +            else:
>> +                cve_item = {"status": "unpatched"}
>> +
>> +            cve_item["CVSS"] = 
>> "{0:.1f}".format(cve_struct[cve]["score"])
>> +            cve_item["CVE"] = cve
>> +            cve_item["product"] = product
>> +            cve_item["version"] = version
>> +            cve_item["description"] = 
>> cve_struct[cve]["description"]
>> +            cve_item["reference"] = [x["url"] for x in 
>> cve_struct[cve]["reference"]]
>> +
>> +            logging.debug("%9s %s %s,%s",
>> +                          cve_item["status"], cve_item["CVE"],
>> +                          cve_item["product"], 
>> cve_item["version"])
>> +
>> +            report.append(cve_item)
>> +
>> +    return sorted(report, key=lambda x: (x["status"], 
>> x["product"], x["CVSS"], x["CVE"]))
>> +
>> +
>> +def process_configuration(manifest, conf):
>> +    """Recursive call to process all CVE configurations"""
>> +
>> +    operator = conf["operator"]
>> +
>> +    if operator not in ["OR", "AND"]:
>> +        raise ValueError("operator {} is not 
>> supported".format(operator))
>> +
>> +    operator = True if operator == "AND" else False
>> +    match = False
>> +    affected = set()
>> +
>> +    if "cpe" in conf:
>> +        match = process_cpe(manifest, conf["cpe"][0], 
>> affected)
>> +
>> +        for cpe in conf["cpe"][1:]:
>> +            package_match = process_cpe(manifest, cpe, 
>> affected)
>> +
>> +            # match = match <operator> package_match
>> +            match = operator ^ ((operator ^ match) or 
>> (operator ^ package_match))
>> +    elif "children" in conf:
>> +        product_set = process_configuration(manifest, 
>> conf["children"][0])
>> +
>> +        if product_set:
>> +            match = True
>> +            affected = affected.union(product_set)
>> +
>> +        for child in conf["children"][1:]:
>> +            product_set = process_configuration(manifest, 
>> child)
>> +            package_match = True if product_set else False
>> +
>> +            # match = match OP package_match
>> +            match = operator ^ ((operator ^ match) or 
>> (operator ^ package_match))
>> +
>> +            if package_match:
>> +                affected = affected.union(product_set)
>> +
>> +    if match:
>> +        return affected
>> +
>> +    return ()
>> +
>> +
>> +def process_cpe(manifest, cpe, affected):
>> +    """Match CPE with all manifest packages"""
>> +
>> +    if not cpe["vulnerable"]:
>> +        # ignore non vulnerable part
>> +        return False
>> +
>> +    version_range = {}
>> +
>> +    for flag in ["versionStartIncluding",
>> +                 "versionStartExcluding",
>> +                 "versionEndIncluding",
>> +                 "versionEndExcluding"]:
>> +        if flag in cpe:
>> +            version_range[flag] = cpe[flag]
>> +
>> +    # take only "product" and "version"
>> +    product, version = cpe["cpe23Uri"].split(":")[4:6]
>> +
>> +    if product not in manifest:
>> +        return False
>> +
>> +    if not version_range:
>> +        if version == "*":
>> +            # ignore CVEs that touches all versions of 
>> package,
>> +            # can not fix it anyway
>> +            logging.debug('ignore "*" in %s', cpe["cpe23Uri"])
>> +            return False
>> +        elif version == "-":
>> +            # "-" means NA
>> +            #
>> +            # NA (i.e. "not applicable/not used"). The logical 
>> value NA
>> +            # SHOULD be assigned when there is no legal or 
>> meaningful
>> +            # value for that attribute, or when that attribute 
>> is not
>> +            # used as part of the description.
>> +            # This includes the situation in which an 
>> attribute has
>> +            # an obtainable value that is null
>> +            #
>> +            # Ignores CVEs if version is not set
>> +            logging.debug('ignore "-" in %s', cpe["cpe23Uri"])
>> +            return False
>> +        else:
>> +            version_range["versionExactMatch"] = version
>> +
>> +    result = False
>> +
>> +    for version in manifest[product]:
>> +        try:
>> +            if match_version(version,
>> +                             version_range):
>> +                logging.debug("match %s %s: %s", product, 
>> version, cpe["cpe23Uri"])
>> +                affected.add("{},{}".format(product, version))
>> +
>> +                result = True
>> +        except TypeError:
>> +            # version comparison is a very tricky
>> +            # sometimes provider changes product version in a 
>> strange manner
>> +            # and the above comparison just failed
>> +            # so here we try to make version string "more 
>> standard"
>> +
>> +            if match_version(twik_version(version),
>> +                             [twik_version(v) for v in 
>> version_range]):
>> +                logging.debug("match %s %s (twiked): %s", 
>> product, twik_version(version),
>> +                              cpe["cpe23Uri"])
>> +                affected.add("{},{}".format(product, version))
>> +
>> +                result = True
>> +
>> +    return result
>> +
>> +
>> +def match_version(version, vrange):
>> +    """Match version with the version range"""
>> +
>> +    result = False
>> +    version = util_version(version)
>> +
>> +    if "versionExactMatch" in vrange:
>> +        if version == 
>> util_version(vrange["versionExactMatch"]):
>> +            result = True
>> +    else:
>> +        result = True
>> +
>> +        if "versionStartIncluding" in vrange:
>> +            result = result and version >= 
>> util_version(vrange["versionStartIncluding"])
>> +
>> +        if "versionStartExcluding" in vrange:
>> +            result = result and version > 
>> util_version(vrange["versionStartExcluding"])
>> +
>> +        if "versionEndIncluding" in vrange:
>> +            result = result and version <= 
>> util_version(vrange["versionEndIncluding"])
>> +
>> +        if "versionEndExcluding" in vrange:
>> +            result = result and version < 
>> util_version(vrange["versionEndExcluding"])
>> +
>> +    return result
>> +
>> +
>> +def util_version(version):
>> +    """Simplify package version"""
>> +    return 
>> distutils.version.LooseVersion(version.split("+git")[0])
>> +
>> +
>> +def twik_version(version):
>> +    """Return "standard" version for complex cases"""
>> +    return "v1" + re.sub(r"^[a-zA-Z]+", "", version)
>> +
>> +
>> +def print_report(report, width=70, show_description=False, 
>> show_reference=False, output=sys.stdout):
>> +    """Print out final report"""
>> +
>> +    for cve in report:
>> +        print("{0:>9s} | {1:>4s} | {2:18s} | {3} | 
>> {4}".format(cve["status"], cve["CVSS"],
>> + 
>> cve["CVE"], cve["product"],
>> + 
>> cve["version"]),
>> +              file=output)
>> +
>> +        if show_description:
>> +            print("{0:>9s} + {1}".format(" ", "Description"), 
>> file=output)
>> +
>> +            for lin in textwrap.wrap(cve["description"], 
>> width=width):
>> +                print("{0:>9s}   {1}".format(" ", lin), 
>> file=output)
>> +
>> +        if show_reference:
>> +            print("{0:>9s} + {1}".format(" ", "Reference"), 
>> file=output)
>> +
>> +            for url in cve["reference"]:
>> +                print("{0:>9s}   {1}".format(" ", url), 
>> file=output)
>> +
>> +
>> +def update_feeds(feed_dir, offline=False, start=2002):
>> +    """Update all JSON feeds"""
>> +
>> +    feed_dir = os.path.realpath(feed_dir)
>> +    year_now = datetime.datetime.now().year
>> +    cve_struct = {}
>> +
>> +    for year in range(start, year_now + 1):
>> +        update_year(cve_struct, year, feed_dir, offline)
>> +
>> +    return cve_struct
>> +
>> +
>> +def update_year(cve_struct, year, feed_dir, offline):
>> +    """Update one JSON feed for the particular year"""
>> +
>> +    url_prefix = 
>> "https://static.nvd.nist.gov/feeds/json/cve/1.0"
>> +    file_prefix = "nvdcve-1.0-{0}".format(year)
>> +
>> +    meta = {
>> +        "url": "{0}/{1}.meta".format(url_prefix, file_prefix),
>> +        "file": os.path.join(feed_dir, 
>> "{0}.meta".format(file_prefix))
>> +    }
>> +
>> +    feed = {
>> +        "url": "{0}/{1}.json.gz".format(url_prefix, 
>> file_prefix),
>> +        "file": os.path.join(feed_dir, 
>> "{0}.json.gz".format(file_prefix))
>> +    }
>> +
>> +    ctx = {}
>> +
>> +    if not offline:
>> +        ctx = download_feed(meta, feed)
>> +
>> +        if not "meta" in ctx or not "feed" in ctx:
>> +            return
>> +
>> +    if not os.path.isfile(meta["file"]):
>> +        return
>> +
>> +    if not os.path.isfile(feed["file"]):
>> +        return
>> +
>> +    if not "meta" in ctx:
>> +        ctx["meta"] = ctx_meta(meta["file"])
>> +
>> +    if not "sha256" in ctx["meta"]:
>> +        return
>> +
>> +    if not "feed" in ctx:
>> +        ctx["feed"] = ctx_gzip(feed["file"], 
>> ctx["meta"]["sha256"])
>> +
>> +    if not ctx["feed"]:
>> +        return
>> +
>> +    logging.debug("parsing year %s", year)
>> +
>> +    for cve_item in ctx["feed"]["CVE_Items"]:
>> +        iden, cve = parse_item(cve_item)
>> +
>> +        if not iden:
>> +            continue
>> +
>> +        if not cve:
>> +            logging.error("%s parse error", iden)
>> +            break
>> +
>> +        if iden in cve_struct:
>> +            logging.error("%s duplicated", iden)
>> +            break
>> +
>> +        cve_struct[iden] = cve
>> +
>> +    logging.debug("cve records: %d", len(cve_struct))
>> +
>> +
>> +def ctx_meta(filename):
>> +    """Parse feed meta file"""
>> +
>> +    if not os.path.isfile(filename):
>> +        return {}
>> +
>> +    ctx = {}
>> +
>> +    with open(filename) as fil:
>> +        for lin in fil:
>> +            pair = lin.split(":", maxsplit=1)
>> +            ctx[pair[0]] = pair[1].rstrip()
>> +
>> +    return ctx
>> +
>> +
>> +def ctx_gzip(filename, checksum=""):
>> +    """Parse feed archive file"""
>> +
>> +    if not os.path.isfile(filename):
>> +        return {}
>> +
>> +    with gzip.open(filename) as fil:
>> +        try:
>> +            ctx = fil.read()
>> +        except (EOFError, OSError):
>> +            logging.error("failed to process gz archive %s", 
>> filename, exc_info=True)
>> +            return {}
>> +
>> +    if checksum and checksum.upper() != 
>> hashlib.sha256(ctx).hexdigest().upper():
>> +        return {}
>> +
>> +    return json.loads(ctx.decode())
>> +
>> +
>> +def parse_item(cve_item):
>> +    """Parse one JSON CVE entry"""
>> +
>> +    cve_id = cve_item["cve"]["CVE_data_meta"]["ID"][:]
>> +    impact = cve_item["impact"]
>> +
>> +    if not impact:
>> +        # REJECTed CVE
>> +        return None, None
>> +
>> +    if "baseMetricV3" in impact:
>> +        score = impact["baseMetricV3"]["cvssV3"]["baseScore"]
>> +    elif "baseMetricV2" in impact:
>> +        score = impact["baseMetricV2"]["cvssV2"]["baseScore"]
>> +    else:
>> +        return cve_id, None
>> +
>> +    return cve_id, {
>> +        "score": score,
>> +        "nodes": cve_item["configurations"]["nodes"][:],
>> +        "reference": 
>> cve_item["cve"]["references"]["reference_data"][:],
>> +        "description": 
>> cve_item["cve"]["description"]["description_data"][0]["value"]
>> +    }
>> +
>> +
>> +def download_feed(meta, feed):
>> +    """Download and parse feed"""
>> +
>> +    ctx = {}
>> +
>> +    if not retrieve_url(meta["url"], meta["file"]):
>> +        return {}
>> +
>> +    ctx["meta"] = ctx_meta(meta["file"])
>> +
>> +    if not "sha256" in ctx["meta"]:
>> +        return {}
>> +
>> +    ctx["feed"] = ctx_gzip(feed["file"], 
>> ctx["meta"]["sha256"])
>> +
>> +    if not ctx["feed"]:
>> +        if not retrieve_url(feed["url"], feed["file"]):
>> +            return {}
>> +
>> +        ctx["feed"] = ctx_gzip(feed["file"], 
>> ctx["meta"]["sha256"])
>> +
>> +    return ctx
>> +
>> +
>> +def retrieve_url(url, filename=None):
>> +    """Download file by URL"""
>> +
>> +    if filename:
>> +        os.makedirs(os.path.dirname(filename), exist_ok=True)
>> +
>> +    logging.debug("downloading %s", url)
>> +
>> +    try:
>> +        urllib.request.urlretrieve(url, filename=filename)
>> +    except urllib.error.HTTPError:
>> +        logging.error("failed to download URL %s", url, 
>> exc_info=True)
>> +        return False
>> +
>> +    return True
>> +
>> +
>> +def logconfig(debug_flag=False):
>> +    """Return default log config"""
>> +
>> +    return {
>> +        "version": 1,
>> +        "formatters": {
>> +            "f": {
>> +                "format": "# %(asctime)s %% CVERT %% 
>> %(levelname)-8s %% %(message)s"
>> +            }
>> +        },
>> +        "handlers": {
>> +            "h": {
>> +                "class": "logging.StreamHandler",
>> +                "formatter": "f",
>> +                "level": logging.DEBUG if debug_flag else 
>> logging.INFO
>> +            }
>> +        },
>> +        "root": {
>> +            "handlers": ["h"],
>> +            "level": logging.DEBUG if debug_flag else 
>> logging.INFO
>> +        },
>> +    }
>> +
>> +
>> +def save_cve(filename, cve_struct):
>> +    """Save CVE structure in the file"""
>> +
>> +    filename = os.path.realpath(filename)
>> +
>> +    logging.debug("saving %d CVE records to %s", 
>> len(cve_struct), filename)
>> +
>> +    with open(filename, "wb") as fil:
>> +        pickle.dump(cve_struct, fil)
>> +
>> +
>> +def load_cve(filename):
>> +    """Load CVE structure from the file"""
>> +
>> +    filename = os.path.realpath(filename)
>> +
>> +    logging.debug("loading from %s", filename)
>> +
>> +    with open(filename, "rb") as fil:
>> +        cve_struct = pickle.load(fil)
>> +
>> +    logging.debug("cve records: %d", len(cve_struct))
>> +
>> +    return cve_struct