* [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository @ 2024-09-02 7:41 Michael Trimarchi 2024-09-02 7:41 ` [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception Michael Trimarchi 2024-09-02 16:39 ` [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository Yann E. MORIN 0 siblings, 2 replies; 7+ messages in thread From: Michael Trimarchi @ 2024-09-02 7:41 UTC (permalink / raw) To: buildroot; +Cc: Michael Trimarchi Just a simple clone --depth 1 should be enough to parse the cve and generate the pkg-stats report. The space now does not take in account of the full history and only use 2.2Gb Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com> --- support/scripts/cve.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/support/scripts/cve.py b/support/scripts/cve.py index e25825581e..7d115a28b5 100755 --- a/support/scripts/cve.py +++ b/support/scripts/cve.py @@ -82,7 +82,7 @@ class CVE: # happily clones into an empty directory. os.makedirs(nvd_git_dir) subprocess.check_call( - ["git", "clone", NVD_BASE_URL, nvd_git_dir], + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, ) -- 2.43.0 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 7+ messages in thread
* [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception 2024-09-02 7:41 [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository Michael Trimarchi @ 2024-09-02 7:41 ` Michael Trimarchi 2024-09-02 16:41 ` Yann E. MORIN 2024-09-02 16:39 ` [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository Yann E. MORIN 1 sibling, 1 reply; 7+ messages in thread From: Michael Trimarchi @ 2024-09-02 7:41 UTC (permalink / raw) To: buildroot; +Cc: Michael Trimarchi If we are not able to pull from the directory, restart from a clean clone Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com> --- support/scripts/cve.py | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/support/scripts/cve.py b/support/scripts/cve.py index 7d115a28b5..b205b72e65 100755 --- a/support/scripts/cve.py +++ b/support/scripts/cve.py @@ -21,6 +21,7 @@ import datetime import os import distutils.version import json +import shutil import subprocess import sys import operator @@ -69,15 +70,21 @@ class CVE: @staticmethod def download_nvd(nvd_git_dir): + done = False print(f"Updating from {NVD_BASE_URL}") if os.path.exists(nvd_git_dir): - subprocess.check_call( - ["git", "pull"], - cwd=nvd_git_dir, - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) - else: + try: + subprocess.check_call( + ["git", "pull"], + cwd=nvd_git_dir, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + done = True + except: + shutil.rmtree(nvd_git_dir) + + if (not done): # Create the directory and its parents; git # happily clones into an empty directory. os.makedirs(nvd_git_dir) -- 2.43.0 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception 2024-09-02 7:41 ` [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception Michael Trimarchi @ 2024-09-02 16:41 ` Yann E. MORIN 2024-09-02 16:44 ` Michael Nazzareno Trimarchi 0 siblings, 1 reply; 7+ messages in thread From: Yann E. MORIN @ 2024-09-02 16:41 UTC (permalink / raw) To: Michael Trimarchi; +Cc: buildroot Michael, All, On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly: > If we are not able to pull from the directory, restart from a clean > clone In what situations would such a problem arise? Is it due to a shallow clone as proposed in the first patch? If so, then that's a good reason not to do a shallow clone. Regards, Yann E. MORIN. > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com> > --- > support/scripts/cve.py | 21 ++++++++++++++------- > 1 file changed, 14 insertions(+), 7 deletions(-) > > diff --git a/support/scripts/cve.py b/support/scripts/cve.py > index 7d115a28b5..b205b72e65 100755 > --- a/support/scripts/cve.py > +++ b/support/scripts/cve.py > @@ -21,6 +21,7 @@ import datetime > import os > import distutils.version > import json > +import shutil > import subprocess > import sys > import operator > @@ -69,15 +70,21 @@ class CVE: > > @staticmethod > def download_nvd(nvd_git_dir): > + done = False > print(f"Updating from {NVD_BASE_URL}") > if os.path.exists(nvd_git_dir): > - subprocess.check_call( > - ["git", "pull"], > - cwd=nvd_git_dir, > - stdout=subprocess.DEVNULL, > - stderr=subprocess.DEVNULL, > - ) > - else: > + try: > + subprocess.check_call( > + ["git", "pull"], > + cwd=nvd_git_dir, > + stdout=subprocess.DEVNULL, > + stderr=subprocess.DEVNULL, > + ) > + done = True > + except: > + shutil.rmtree(nvd_git_dir) > + > + if (not done): > # Create the directory and its parents; git > # happily clones into an empty directory. > os.makedirs(nvd_git_dir) > -- > 2.43.0 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception 2024-09-02 16:41 ` Yann E. MORIN @ 2024-09-02 16:44 ` Michael Nazzareno Trimarchi 0 siblings, 0 replies; 7+ messages in thread From: Michael Nazzareno Trimarchi @ 2024-09-02 16:44 UTC (permalink / raw) To: Yann E. MORIN; +Cc: buildroot Hi Yann On Mon, Sep 2, 2024 at 6:41 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Michael, All, > > On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly: > > If we are not able to pull from the directory, restart from a clean > > clone > > In what situations would such a problem arise? > > Is it due to a shallow clone as proposed in the first patch? If so, then > that's a good reason not to do a shallow clone. > You test if the dir exists but if you cancel the git clone without terminating it, you can not continue. It's happen always and does not depend on the previous patch Michael > Regards, > Yann E. MORIN. > > > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com> > > --- > > support/scripts/cve.py | 21 ++++++++++++++------- > > 1 file changed, 14 insertions(+), 7 deletions(-) > > > > diff --git a/support/scripts/cve.py b/support/scripts/cve.py > > index 7d115a28b5..b205b72e65 100755 > > --- a/support/scripts/cve.py > > +++ b/support/scripts/cve.py > > @@ -21,6 +21,7 @@ import datetime > > import os > > import distutils.version > > import json > > +import shutil > > import subprocess > > import sys > > import operator > > @@ -69,15 +70,21 @@ class CVE: > > > > @staticmethod > > def download_nvd(nvd_git_dir): > > + done = False > > print(f"Updating from {NVD_BASE_URL}") > > if os.path.exists(nvd_git_dir): > > - subprocess.check_call( > > - ["git", "pull"], > > - cwd=nvd_git_dir, > > - stdout=subprocess.DEVNULL, > > - stderr=subprocess.DEVNULL, > > - ) > > - else: > > + try: > > + subprocess.check_call( > > + ["git", "pull"], > > + cwd=nvd_git_dir, > > + stdout=subprocess.DEVNULL, > > + stderr=subprocess.DEVNULL, > > + ) > > + done = True > > + except: > > + shutil.rmtree(nvd_git_dir) > > + > > + if (not done): > > # Create the directory and its parents; git > > # happily clones into an empty directory. > > os.makedirs(nvd_git_dir) > > -- > > 2.43.0 > > > > _______________________________________________ > > buildroot mailing list > > buildroot@buildroot.org > > https://lists.buildroot.org/mailman/listinfo/buildroot > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' -- Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 michael@amarulasolutions.com __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 info@amarulasolutions.com www.amarulasolutions.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository 2024-09-02 7:41 [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository Michael Trimarchi 2024-09-02 7:41 ` [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception Michael Trimarchi @ 2024-09-02 16:39 ` Yann E. MORIN 2024-09-02 16:43 ` Michael Nazzareno Trimarchi 1 sibling, 1 reply; 7+ messages in thread From: Yann E. MORIN @ 2024-09-02 16:39 UTC (permalink / raw) To: Michael Trimarchi; +Cc: buildroot Michael, All, On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly: > Just a simple clone --depth 1 should be enough to parse the > cve and generate the pkg-stats report. The space now does > not take in account of the full history and only use 2.2Gb I just did both a full clone and a depth-1 clone, and the size delta is not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league. Still, the download size does change: from 983.55MiB down to 270.78MiB; the remote has to compress objects specifically for the depth-1 clone, which takes a bit of time, but there less deltas to resolve locally, so it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link). Is it eventually worth it? I like things to be simple rather than have complex heuristics, and storage nowadays is cheap, and bandwidth is only really paid on the first clone, since the following calls would only pull what's missing, which will eventually cause the local clone to grow to a size that would not differ by much if the first clone had been a full one. So, I 'm a bit comnficted about that one, but would tend to say no... Leaving pending for other maintainers to chime in. Regards, Yann E. MORIN. > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com> > --- > support/scripts/cve.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/support/scripts/cve.py b/support/scripts/cve.py > index e25825581e..7d115a28b5 100755 > --- a/support/scripts/cve.py > +++ b/support/scripts/cve.py > @@ -82,7 +82,7 @@ class CVE: > # happily clones into an empty directory. > os.makedirs(nvd_git_dir) > subprocess.check_call( > - ["git", "clone", NVD_BASE_URL, nvd_git_dir], > + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], > stdout=subprocess.DEVNULL, > stderr=subprocess.DEVNULL, > ) > -- > 2.43.0 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository 2024-09-02 16:39 ` [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository Yann E. MORIN @ 2024-09-02 16:43 ` Michael Nazzareno Trimarchi 2024-09-02 16:53 ` Michael Nazzareno Trimarchi 0 siblings, 1 reply; 7+ messages in thread From: Michael Nazzareno Trimarchi @ 2024-09-02 16:43 UTC (permalink / raw) To: Yann E. MORIN; +Cc: buildroot HI Yann On Mon, Sep 2, 2024 at 6:39 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > Michael, All, > > On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly: > > Just a simple clone --depth 1 should be enough to parse the > > cve and generate the pkg-stats report. The space now does > > not take in account of the full history and only use 2.2Gb > > I just did both a full clone and a depth-1 clone, and the size delta is > not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league. > > Still, the download size does change: from 983.55MiB down to 270.78MiB; > the remote has to compress objects specifically for the depth-1 clone, > which takes a bit of time, but there less deltas to resolve locally, so > it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link). > If you have a dsl and download link of 10Mbps this makes really sense and this just tend to increase in size. You 1 minutes can be ten 10 minutes on network congestion Michael > Is it eventually worth it? I like things to be simple rather than have > complex heuristics, and storage nowadays is cheap, and bandwidth is only > really paid on the first clone, since the following calls would only > pull what's missing, which will eventually cause the local clone to grow > to a size that would not differ by much if the first clone had been a > full one. > > So, I 'm a bit comnficted about that one, but would tend to say no... > Leaving pending for other maintainers to chime in. > > Regards, > Yann E. MORIN. > > > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com> > > --- > > support/scripts/cve.py | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/support/scripts/cve.py b/support/scripts/cve.py > > index e25825581e..7d115a28b5 100755 > > --- a/support/scripts/cve.py > > +++ b/support/scripts/cve.py > > @@ -82,7 +82,7 @@ class CVE: > > # happily clones into an empty directory. > > os.makedirs(nvd_git_dir) > > subprocess.check_call( > > - ["git", "clone", NVD_BASE_URL, nvd_git_dir], > > + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], > > stdout=subprocess.DEVNULL, > > stderr=subprocess.DEVNULL, > > ) > > -- > > 2.43.0 > > > > _______________________________________________ > > buildroot mailing list > > buildroot@buildroot.org > > https://lists.buildroot.org/mailman/listinfo/buildroot > > -- > .-----------------.--------------------.------------------.--------------------. > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > '------------------------------^-------^------------------^--------------------' -- Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 michael@amarulasolutions.com __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 info@amarulasolutions.com www.amarulasolutions.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository 2024-09-02 16:43 ` Michael Nazzareno Trimarchi @ 2024-09-02 16:53 ` Michael Nazzareno Trimarchi 0 siblings, 0 replies; 7+ messages in thread From: Michael Nazzareno Trimarchi @ 2024-09-02 16:53 UTC (permalink / raw) To: Yann E. MORIN; +Cc: buildroot Hi Yann On Mon, Sep 2, 2024 at 6:43 PM Michael Nazzareno Trimarchi <michael@amarulasolutions.com> wrote: > > HI Yann > > On Mon, Sep 2, 2024 at 6:39 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote: > > > > Michael, All, > > > > On 2024-09-02 09:41 +0200, Michael Trimarchi spake thusly: > > > Just a simple clone --depth 1 should be enough to parse the > > > cve and generate the pkg-stats report. The space now does > > > not take in account of the full history and only use 2.2Gb > > > > I just did both a full clone and a depth-1 clone, and the size delta is > > not that much: 2.9GiB vs. 2.2GiB, so we're still in the same league. > > > > Still, the download size does change: from 983.55MiB down to 270.78MiB; > > the remote has to compress objects specifically for the depth-1 clone, > > which takes a bit of time, but there less deltas to resolve locally, so > > it's a net time win too: 2m17s vs 1min7s (on a 100Mbps link). > > > > If you have a dsl and download link of 10Mbps this makes really sense > and this just tend > to increase in size. You 1 minutes can be ten 10 minutes on network congestion > I think that the real reason could that the next pull will be problematic. Yes, you can drop this patch Anyway the second one does not depend on it Michael > Michael > > > Is it eventually worth it? I like things to be simple rather than have > > complex heuristics, and storage nowadays is cheap, and bandwidth is only > > really paid on the first clone, since the following calls would only > > pull what's missing, which will eventually cause the local clone to grow > > to a size that would not differ by much if the first clone had been a > > full one. > > > > So, I 'm a bit comnficted about that one, but would tend to say no... > > Leaving pending for other maintainers to chime in. > > > > Regards, > > Yann E. MORIN. > > > > > Signed-off-by: Michael Trimarchi <michael@amarulasolutions.com> > > > --- > > > support/scripts/cve.py | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/support/scripts/cve.py b/support/scripts/cve.py > > > index e25825581e..7d115a28b5 100755 > > > --- a/support/scripts/cve.py > > > +++ b/support/scripts/cve.py > > > @@ -82,7 +82,7 @@ class CVE: > > > # happily clones into an empty directory. > > > os.makedirs(nvd_git_dir) > > > subprocess.check_call( > > > - ["git", "clone", NVD_BASE_URL, nvd_git_dir], > > > + ["git", "clone", "--depth", "1", NVD_BASE_URL, nvd_git_dir], > > > stdout=subprocess.DEVNULL, > > > stderr=subprocess.DEVNULL, > > > ) > > > -- > > > 2.43.0 > > > > > > _______________________________________________ > > > buildroot mailing list > > > buildroot@buildroot.org > > > https://lists.buildroot.org/mailman/listinfo/buildroot > > > > -- > > .-----------------.--------------------.------------------.--------------------. > > | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | > > | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | > > | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | > > | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | > > '------------------------------^-------^------------------^--------------------' > > > > -- > Michael Nazzareno Trimarchi > Co-Founder & Chief Executive Officer > M. +39 347 913 2170 > michael@amarulasolutions.com > __________________________________ > > Amarula Solutions BV > Joop Geesinkweg 125, 1114 AB, Amsterdam, NL > T. +31 (0)85 111 9172 > info@amarulasolutions.com > www.amarulasolutions.com -- Michael Nazzareno Trimarchi Co-Founder & Chief Executive Officer M. +39 347 913 2170 michael@amarulasolutions.com __________________________________ Amarula Solutions BV Joop Geesinkweg 125, 1114 AB, Amsterdam, NL T. +31 (0)85 111 9172 info@amarulasolutions.com www.amarulasolutions.com _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-09-02 16:53 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-09-02 7:41 [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository Michael Trimarchi 2024-09-02 7:41 ` [Buildroot] [PATCH 2/2] scripts/cve: Restart the clone if the pull generate an exception Michael Trimarchi 2024-09-02 16:41 ` Yann E. MORIN 2024-09-02 16:44 ` Michael Nazzareno Trimarchi 2024-09-02 16:39 ` [Buildroot] [PATCH 1/2] scripts/cve: Avoid to do a complete clone of cve git repository Yann E. MORIN 2024-09-02 16:43 ` Michael Nazzareno Trimarchi 2024-09-02 16:53 ` Michael Nazzareno Trimarchi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox