[Buildroot] [PATCH V2 1/2] scripts/cve: Avoid to do a complete clone of cve git repository

[Buildroot] [PATCH V2 1/2] scripts/cve: Avoid to do a complete clone of cve git repository

[Michael Trimarchi]

Just a simple clone and pull with --depth 1 should be enough to parse the
cve and generate the pkg-stats report.

From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB.
The download size does change: from 983.55MiB down to 270.78MiB.
it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).

Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
---
V1->V2:
    - Add statistics from Yann E. Morin
    - Use git pull --depth 1 for update the repo

---
 support/scripts/cve.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index e25825581e..dcb3a63925 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -72,7 +72,7 @@ class CVE:
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
             subprocess.check_call(
-                ["git", "pull"],
+                ["git", "pull", "--depth", "1"],
                 cwd=nvd_git_dir,
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
@@ -82,7 +82,7 @@ class CVE:
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
             subprocess.check_call(
-                ["git", "clone", NVD_BASE_URL, nvd_git_dir],
+                ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir],
                 stdout=subprocess.DEVNULL,
                 stderr=subprocess.DEVNULL,
             )
-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH V2 2/2] scripts/cve: Restart the clone if the pull generate an exception

[Michael Trimarchi]

If we are not able to pull from the directory, restart from a clean
clone. This can happen for corrupt repository or unfinished download

Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
---
V1->V2: Adjust the commit message

---
 support/scripts/cve.py | 21 ++++++++++++++-------
 1 file changed, 14 insertions(+), 7 deletions(-)

diff --git a/support/scripts/cve.py b/support/scripts/cve.py
index dcb3a63925..6cd9aab963 100755
--- a/support/scripts/cve.py
+++ b/support/scripts/cve.py
@@ -21,6 +21,7 @@ import datetime
 import os
 import distutils.version
 import json
+import shutil
 import subprocess
 import sys
 import operator
@@ -69,15 +70,21 @@ class CVE:
 
     @staticmethod
     def download_nvd(nvd_git_dir):
+        done = False
         print(f"Updating from {NVD_BASE_URL}")
         if os.path.exists(nvd_git_dir):
-            subprocess.check_call(
-                ["git", "pull", "--depth", "1"],
-                cwd=nvd_git_dir,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-            )
-        else:
+            try:
+                subprocess.check_call(
+                    ["git", "pull", "--depth", "1"],
+                    cwd=nvd_git_dir,
+                    stdout=subprocess.DEVNULL,
+                    stderr=subprocess.DEVNULL,
+                )
+                done = True
+            except:
+                shutil.rmtree(nvd_git_dir)
+
+        if (not done):
             # Create the directory and its parents; git
             # happily clones into an empty directory.
             os.makedirs(nvd_git_dir)
-- 
2.43.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2 1/2] scripts/cve: Avoid to do a complete clone of cve git repository

[Thomas Petazzoni via buildroot]

On Tue,  3 Sep 2024 10:57:44 +0200
Michael Trimarchi <michael@amarulasolutions.com> wrote:

> Just a simple clone and pull with --depth 1 should be enough to parse the
> cve and generate the pkg-stats report.
> 
> From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB.
> The download size does change: from 983.55MiB down to 270.78MiB.
> it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
> 
> Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
> ---
> V1->V2:
>     - Add statistics from Yann E. Morin
>     - Use git pull --depth 1 for update the repo

Applied to next, thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2 2/2] scripts/cve: Restart the clone if the pull generate an exception

[Thomas Petazzoni via buildroot]

On Tue,  3 Sep 2024 10:57:45 +0200
Michael Trimarchi <michael@amarulasolutions.com> wrote:

> diff --git a/support/scripts/cve.py b/support/scripts/cve.py
> index dcb3a63925..6cd9aab963 100755
> --- a/support/scripts/cve.py
> +++ b/support/scripts/cve.py
> @@ -21,6 +21,7 @@ import datetime
>  import os
>  import distutils.version
>  import json
> +import shutil
>  import subprocess
>  import sys
>  import operator
> @@ -69,15 +70,21 @@ class CVE:
>  
>      @staticmethod
>      def download_nvd(nvd_git_dir):
> +        done = False
>          print(f"Updating from {NVD_BASE_URL}")
>          if os.path.exists(nvd_git_dir):
> -            subprocess.check_call(
> -                ["git", "pull", "--depth", "1"],
> -                cwd=nvd_git_dir,
> -                stdout=subprocess.DEVNULL,
> -                stderr=subprocess.DEVNULL,
> -            )
> -        else:
> +            try:
> +                subprocess.check_call(
> +                    ["git", "pull", "--depth", "1"],
> +                    cwd=nvd_git_dir,
> +                    stdout=subprocess.DEVNULL,
> +                    stderr=subprocess.DEVNULL,
> +                )
> +                done = True
> +            except:
> +                shutil.rmtree(nvd_git_dir)

The thing I'm worried about is that you can also get a failure of "git
pull" for example due to a network timeout or something like that,
which doesn't need a full git clone, but just a "try again" later...
and now we're going to wipe out the entire local clone, and try to
clone everything again. Is that a good idea?

Also, perhaps we need to show an error message if the "git pull"
failed, and saying we're falling back to a full clone, or something?

> +        if (not done):

	   if not done:


is sufficient, we are not writing C code here :-)

Thanks!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2 2/2] scripts/cve: Restart the clone if the pull generate an exception

[Michael Nazzareno Trimarchi]

Hi Thomas

On Tue, Sep 3, 2024 at 8:52 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> On Tue,  3 Sep 2024 10:57:45 +0200
> Michael Trimarchi <michael@amarulasolutions.com> wrote:
>
> > diff --git a/support/scripts/cve.py b/support/scripts/cve.py
> > index dcb3a63925..6cd9aab963 100755
> > --- a/support/scripts/cve.py
> > +++ b/support/scripts/cve.py
> > @@ -21,6 +21,7 @@ import datetime
> >  import os
> >  import distutils.version
> >  import json
> > +import shutil
> >  import subprocess
> >  import sys
> >  import operator
> > @@ -69,15 +70,21 @@ class CVE:
> >
> >      @staticmethod
> >      def download_nvd(nvd_git_dir):
> > +        done = False
> >          print(f"Updating from {NVD_BASE_URL}")
> >          if os.path.exists(nvd_git_dir):
> > -            subprocess.check_call(
> > -                ["git", "pull", "--depth", "1"],
> > -                cwd=nvd_git_dir,
> > -                stdout=subprocess.DEVNULL,
> > -                stderr=subprocess.DEVNULL,
> > -            )
> > -        else:
> > +            try:
> > +                subprocess.check_call(
> > +                    ["git", "pull", "--depth", "1"],
> > +                    cwd=nvd_git_dir,
> > +                    stdout=subprocess.DEVNULL,
> > +                    stderr=subprocess.DEVNULL,
> > +                )
> > +                done = True
> > +            except:
> > +                shutil.rmtree(nvd_git_dir)
>
> The thing I'm worried about is that you can also get a failure of "git
> pull" for example due to a network timeout or something like that,
> which doesn't need a full git clone, but just a "try again" later...
> and now we're going to wipe out the entire local clone, and try to
> clone everything again. Is that a good idea?
>
> Also, perhaps we need to show an error message if the "git pull"
> failed, and saying we're falling back to a full clone, or something?
>
> > +        if (not done):
>
>            if not done:
>
>
> is sufficient, we are not writing C code here :-)

Sorry, I will ask my colleague to write python and send a better
strategy. I totally agree with you.

Anyway I have a plan to add more information on cve reporting, hope
that you like the idea.
Mostly I'm playing with parser for jenkins and love to add buildroot
too but I don't have enough
information to show

https://github.com/jenkinsci/analysis-model/pull/1085

Michael


>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com



-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2 1/2] scripts/cve: Avoid to do a complete clone of cve git repository

[Yann E. MORIN]

Michael, All,

On 2024-09-03 20:50 +0200, Thomas Petazzoni via buildroot spake thusly:
> On Tue,  3 Sep 2024 10:57:44 +0200
> Michael Trimarchi <michael@amarulasolutions.com> wrote:
> 
> > Just a simple clone and pull with --depth 1 should be enough to parse the
> > cve and generate the pkg-stats report.
> > 
> > From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB.
> > The download size does change: from 983.55MiB down to 270.78MiB.
> > it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
> > 
> > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
> > ---
> > V1->V2:
> >     - Add statistics from Yann E. Morin
> >     - Use git pull --depth 1 for update the repo
> 
> Applied to next, thanks!

I don't understand: Michael said in the first iteration that we should
drop the package:
    https://lore.kernel.org/buildroot/CAOf5uw=m4LOk97OT1dTP=2-uP6QZ1WQyHfKYSQmWCirDaxXvgQ@mail.gmail.com/

Only the first clone is slow, the following calls will just pull (mostly
nothing most of the time), so the optimisation is not really worth it.

Also, in the download backend for git, we stopped doing shallow clone
because they were causing issues (but might not be applicable here).

Anyway, too late, that's been applied...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2 1/2] scripts/cve: Avoid to do a complete clone of cve git repository

[Michael Nazzareno Trimarchi]

Hi Yann

On Tue, Sep 3, 2024 at 9:24 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Michael, All,
>
> On 2024-09-03 20:50 +0200, Thomas Petazzoni via buildroot spake thusly:
> > On Tue,  3 Sep 2024 10:57:44 +0200
> > Michael Trimarchi <michael@amarulasolutions.com> wrote:
> >
> > > Just a simple clone and pull with --depth 1 should be enough to parse the
> > > cve and generate the pkg-stats report.
> > >
> > > From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB.
> > > The download size does change: from 983.55MiB down to 270.78MiB.
> > > it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
> > >
> > > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
> > > ---
> > > V1->V2:
> > >     - Add statistics from Yann E. Morin
> > >     - Use git pull --depth 1 for update the repo
> >
> > Applied to next, thanks!
>
> I don't understand: Michael said in the first iteration that we should
> drop the package:
>     https://lore.kernel.org/buildroot/CAOf5uw=m4LOk97OT1dTP=2-uP6QZ1WQyHfKYSQmWCirDaxXvgQ@mail.gmail.com/
>
> Only the first clone is slow, the following calls will just pull (mostly
> nothing most of the time), so the optimisation is not really worth it.
>
> Also, in the download backend for git, we stopped doing shallow clone
> because they were causing issues (but might not be applicable here).
>

I have reposted it, fixing the pull. Yann if you really don't like it,
people can drop, revert

Michael

> Anyway, too late, that's been applied...
>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'



-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2 1/2] scripts/cve: Avoid to do a complete clone of cve git repository

[Thomas Petazzoni via buildroot]

Hello Michael,

On Tue,  3 Sep 2024 10:57:44 +0200
Michael Trimarchi <michael@amarulasolutions.com> wrote:

> Just a simple clone and pull with --depth 1 should be enough to parse the
> cve and generate the pkg-stats report.
> 
> From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB.
> The download size does change: from 983.55MiB down to 270.78MiB.
> it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
> 
> Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
> ---
> V1->V2:
>     - Add statistics from Yann E. Morin
>     - Use git pull --depth 1 for update the repo

I am sorry, but I had to revert this commit... as it doesn't work:

Updating from https://github.com/fkie-cad/nvd-json-data-feeds/
Traceback (most recent call last):
  File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1346, in <module>
    __main__()
  File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1335, in __main__
    check_package_cves(args.nvd_path, packages)
  File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 660, in check_package_cves
    for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
  File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 105, in read_nvd_dir
    CVE.download_nvd(nvd_git_dir)
  File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 74, in download_nvd
    subprocess.check_call(
  File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'pull', '--depth', '1']' returned non-zero exit status 128.

if I go to the machine in question:

buildroot@buildroot:~/nvd/git$ git pull --depth 1
remote: Enumerating objects: 164, done.
remote: Counting objects: 100% (140/140), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 63 (delta 54), reused 44 (delta 36), pack-reused 0 (from 0)
Unpacking objects: 100% (63/63), 18.15 KiB | 26.00 KiB/s, done.
From https://github.com/fkie-cad/nvd-json-data-feeds
 + 51b091a6a4...7b69235976 main       -> origin/main  (forced update)
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint: 
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint: 
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

I had already done a git reset --hard, and see if it was just a
one-time issue, but nope. So there's something that doesn't work here.

In case that matters, this machine has:

$ git --version
git version 2.39.2

Best regards,

Thomas Petazzoni
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2 1/2] scripts/cve: Avoid to do a complete clone of cve git repository

[Michael Nazzareno Trimarchi]

Hi Thomas

On Thu, Sep 12, 2024 at 12:44 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello Michael,
>
> On Tue,  3 Sep 2024 10:57:44 +0200
> Michael Trimarchi <michael@amarulasolutions.com> wrote:
>
> > Just a simple clone and pull with --depth 1 should be enough to parse the
> > cve and generate the pkg-stats report.
> >
> > From a full clone and a depth-1 clone, and the size delta is 2.9GiB vs. 2.2GiB.
> > The download size does change: from 983.55MiB down to 270.78MiB.
> > it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link).
> >
> > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com>
> > ---
> > V1->V2:
> >     - Add statistics from Yann E. Morin
> >     - Use git pull --depth 1 for update the repo
>
> I am sorry, but I had to revert this commit... as it doesn't work:
>
> Updating from https://github.com/fkie-cad/nvd-json-data-feeds/
> Traceback (most recent call last):
>   File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1346, in <module>
>     __main__()
>   File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 1335, in __main__
>     check_package_cves(args.nvd_path, packages)
>   File "/home/buildroot/buildroot-stats/./support/scripts/pkg-stats", line 660, in check_package_cves
>     for cve in cvecheck.CVE.read_nvd_dir(nvd_path):
>   File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 105, in read_nvd_dir
>     CVE.download_nvd(nvd_git_dir)
>   File "/home/buildroot/buildroot-stats/support/scripts/cve.py", line 74, in download_nvd
>     subprocess.check_call(
>   File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
>     raise CalledProcessError(retcode, cmd)
> subprocess.CalledProcessError: Command '['git', 'pull', '--depth', '1']' returned non-zero exit status 128.
>
> if I go to the machine in question:
>
> buildroot@buildroot:~/nvd/git$ git pull --depth 1
> remote: Enumerating objects: 164, done.
> remote: Counting objects: 100% (140/140), done.
> remote: Compressing objects: 100% (27/27), done.
> remote: Total 63 (delta 54), reused 44 (delta 36), pack-reused 0 (from 0)
> Unpacking objects: 100% (63/63), 18.15 KiB | 26.00 KiB/s, done.
> From https://github.com/fkie-cad/nvd-json-data-feeds
>  + 51b091a6a4...7b69235976 main       -> origin/main  (forced update)
> hint: You have divergent branches and need to specify how to reconcile them.
> hint: You can do so by running one of the following commands sometime before
> hint: your next pull:
> hint:
> hint:   git config pull.rebase false  # merge
> hint:   git config pull.rebase true   # rebase
> hint:   git config pull.ff only       # fast-forward only
> hint:
> hint: You can replace "git config" with "git config --global" to set a default
> hint: preference for all repositories. You can also pass --rebase, --no-rebase,
> hint: or --ff-only on the command line to override the configured default per
> hint: invocation.
> fatal: Need to specify how to reconcile divergent branches.
>
> I had already done a git reset --hard, and see if it was just a
> one-time issue, but nope. So there's something that doesn't work here.
>
> In case that matters, this machine has:
>
> $ git --version
> git version 2.39.2
>

Yes sorry, I always use tools that are the latest,  git version 2.43.0

Make sense. I will create a better setup for me


Michael

> Best regards,
>
> Thomas Petazzoni
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com



-- 
Michael Nazzareno Trimarchi
Co-Founder & Chief Executive Officer
M. +39 347 913 2170
michael@amarulasolutions.com
__________________________________

Amarula Solutions BV
Joop Geesinkweg 125, 1114 AB, Amsterdam, NL
T. +31 (0)85 111 9172
info@amarulasolutions.com
www.amarulasolutions.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot