* [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful user-facing tools
2017-06-18 14:01 [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Yann E. MORIN
@ 2017-06-18 14:01 ` Yann E. MORIN
2017-06-19 1:11 ` Ricardo Martincoski
2017-06-18 14:01 ` [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts Yann E. MORIN
` (3 subsequent siblings)
4 siblings, 1 reply; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-18 14:01 UTC (permalink / raw)
To: buildroot
Add a tools directory to store useful little scripts, tools, etc...
Users may add that directory in their PATH if they want to, but this
is not mandatory.
Add the first tool: brmake, a small script that redirects the build
output log to a file, keeping just Buildroot's own messages, with the
date+time added at the start of the line.
We need to unbuffer the output of make so that, when the script is
interrupted (SIGINT, ^C), there is no lingering output not yet digested
by the logger loop.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
Changes v1 -> v2:
- rename contrib -> tools (Arnout)
- detect missing unbuffer (Arnout)
Notes:
- Arnout suggested using 'stdbuf' instead of 'unbuffer', but that has
some shortcomings in weird corner-cases because it uses LD_PRELOAD
to call a constructor that forces settings on stdout/stderr/stdin
(it cost me three days to investigate a build failure because of
that). So, I stuck with 'unbuffer'...
- I also tried to use bash' coproc, but it seems there was a
regression in later bash versions that causes an incorrect
redirect. So, I stuck with 'unbuffer'...
---
docs/manual/writing-rules.txt | 5 +++--
tools/00-README.txt | 9 +++++++++
tools/brmake | 37 +++++++++++++++++++++++++++++++++++++
3 files changed, 49 insertions(+), 2 deletions(-)
create mode 100644 tools/00-README.txt
create mode 100755 tools/brmake
diff --git a/docs/manual/writing-rules.txt b/docs/manual/writing-rules.txt
index 3a2c8dcb37..c144874a0d 100644
--- a/docs/manual/writing-rules.txt
+++ b/docs/manual/writing-rules.txt
@@ -146,5 +146,6 @@ syntax, refer to http://www.methods.co.nz/asciidoc/userguide.html[].
=== Support scripts
-Some scripts in the +support/+ directory are written in Python and should follow
-the https://www.python.org/dev/peps/pep-0008/[PEP8 Style Guide for Python Code].
+Some scripts in the +support/+ and +tools/+ directories are written in
+Python and should follow the
+https://www.python.org/dev/peps/pep-0008/[PEP8 Style Guide for Python Code].
diff --git a/tools/00-README.txt b/tools/00-README.txt
new file mode 100644
index 0000000000..ffc0fbf5ef
--- /dev/null
+++ b/tools/00-README.txt
@@ -0,0 +1,9 @@
+This directory contains various useful scripts and tools for working
+with Buildroot. You need not add this directory in your PATH to use
+any of those tools, but you may do so if you want.
+
+brmake
+ a script that can be run instead of make, that prepends the date in
+ front of each line, redirects all of the build output to a file
+ ("'br.log' in the current directory), and just ouputs the Buildroot
+ messages (those lines starting with >>>) on stdout.
diff --git a/tools/brmake b/tools/brmake
new file mode 100755
index 0000000000..9f7b4bc65f
--- /dev/null
+++ b/tools/brmake
@@ -0,0 +1,37 @@
+#!/bin/bash
+# (C) 2016, "Yann E. MORIN" <yann.morin.1998@free.fr>
+# License: WTFPL, https://spdx.org/licenses/WTFPL.html
+
+main() {
+ local ret start d h m mf fs
+
+ if ! which unbuffer >/dev/null 2>&1; then
+ printf "you need to install 'unbuffer' (from package expect or expect-dev)\n" >&2
+ exit 1
+ fi
+
+ start=${SECONDS}
+
+ ( exec 2>&1; unbuffer make "${@}"; ) \
+ > >( while read line; do
+ printf "%(%Y-%m-%dT%H:%M:%S)T %s\n" -1 "${line}"
+ done \
+ |tee -a br.log \
+ |grep --colour=never -E '>>>'
+ )
+ ret=${?}
+
+ d=$((SECONDS-start))
+ printf "Done in "
+ h=$((d/3600))
+ d=$((d%3600))
+ [ ${h} -eq 0 ] || { printf "%dh " ${h}; mf="02"; }
+ m=$((d/60))
+ d=$((d%60))
+ [ ${m} -eq 0 ] || { printf "%${mf}dmin " ${m}; sf="02"; }
+ printf "%${sf}ds\n" ${d}
+
+ return ${ret}
+}
+
+main "${@}"
--
2.11.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful user-facing tools
2017-06-18 14:01 ` [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful " Yann E. MORIN
@ 2017-06-19 1:11 ` Ricardo Martincoski
2017-06-19 15:42 ` Yann E. MORIN
0 siblings, 1 reply; 25+ messages in thread
From: Ricardo Martincoski @ 2017-06-19 1:11 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote:
> Add a tools directory to store useful little scripts, tools, etc...
> Users may add that directory in their PATH if they want to, but this
> is not mandatory.
>
> Add the first tool: brmake, a small script that redirects the build
> output log to a file, keeping just Buildroot's own messages, with the
> date+time added at the start of the line.
>
> We need to unbuffer the output of make so that, when the script is
> interrupted (SIGINT, ^C), there is no lingering output not yet digested
> by the logger loop.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
I tested the script and it works very well in general.
- I successfully built the qemu_arm_versatile_defconfig using the script,
pressing CTRL+C a few times and resuming by calling it again.
- Also, by inserting an error to a package I got the expected behaviour: the
error message is in the log file and the script returned the error code.
- Tested using 'GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)'
BUT if it is called without a .config file, it hangs forever waiting for input,
and the menuconfig is sent to the log file.
Maybe we can bailout if no .config is present.
Also, 2 small nits below.
> +++ b/tools/00-README.txt
> @@ -0,0 +1,9 @@
> +This directory contains various useful scripts and tools for working
> +with Buildroot. You need not add this directory in your PATH to use
> +any of those tools, but you may do so if you want.
> +
> +brmake
> + a script that can be run instead of make, that prepends the date in
> + front of each line, redirects all of the build output to a file
> + ("'br.log' in the current directory), and just ouputs the Buildroot
Maybe this file will be dropped as discussed in
http://patchwork.ozlabs.org/patch/777446/
If not, there is a typo
ouputs -> outputs
> + messages (those lines starting with >>>) on stdout.
> diff --git a/tools/brmake b/tools/brmake
> new file mode 100755
> index 0000000000..9f7b4bc65f
> --- /dev/null
> +++ b/tools/brmake
> @@ -0,0 +1,37 @@
> +#!/bin/bash
> +# (C) 2016, "Yann E. MORIN" <yann.morin.1998@free.fr>
> +# License: WTFPL, https://spdx.org/licenses/WTFPL.html
> +
> +main() {
> + local ret start d h m mf fs
fs is unused.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful user-facing tools
2017-06-19 1:11 ` Ricardo Martincoski
@ 2017-06-19 15:42 ` Yann E. MORIN
2017-06-21 2:06 ` Ricardo Martincoski
0 siblings, 1 reply; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-19 15:42 UTC (permalink / raw)
To: buildroot
Ricardo, All,
On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly:
> On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote:
> > Add a tools directory to store useful little scripts, tools, etc...
> > Users may add that directory in their PATH if they want to, but this
> > is not mandatory.
> >
> > Add the first tool: brmake, a small script that redirects the build
> > output log to a file, keeping just Buildroot's own messages, with the
> > date+time added at the start of the line.
> >
> > We need to unbuffer the output of make so that, when the script is
> > interrupted (SIGINT, ^C), there is no lingering output not yet digested
> > by the logger loop.
> >
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> > Cc: Arnout Vandecappelle <arnout@mind.be>
>
> I tested the script and it works very well in general.
>
> - I successfully built the qemu_arm_versatile_defconfig using the script,
> pressing CTRL+C a few times and resuming by calling it again.
> - Also, by inserting an error to a package I got the expected behaviour: the
> error message is in the log file and the script returned the error code.
> - Tested using 'GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)'
>
> BUT if it is called without a .config file, it hangs forever waiting for input,
> and the menuconfig is sent to the log file.
Indeed, my use-case is to only use it in an already-configured tree.
> Maybe we can bailout if no .config is present.
Hmmm... It's not trivial do detect, because the .config file is not
necessarily in the current directory, for example:
brmake -C /pat/to/buil-dir
and I don't want to write a make option interpeter either...
But I'll see what I can do.
> Also, 2 small nits below.
>
> > +++ b/tools/00-README.txt
> > @@ -0,0 +1,9 @@
> > +This directory contains various useful scripts and tools for working
> > +with Buildroot. You need not add this directory in your PATH to use
> > +any of those tools, but you may do so if you want.
> > +
> > +brmake
> > + a script that can be run instead of make, that prepends the date in
> > + front of each line, redirects all of the build output to a file
> > + ("'br.log' in the current directory), and just ouputs the Buildroot
>
> Maybe this file will be dropped as discussed in
> http://patchwork.ozlabs.org/patch/777446/
> If not, there is a typo
>
> ouputs -> outputs
Yup, I already dropped the file locally.
> > + messages (those lines starting with >>>) on stdout.
> > diff --git a/tools/brmake b/tools/brmake
> > new file mode 100755
> > index 0000000000..9f7b4bc65f
> > --- /dev/null
> > +++ b/tools/brmake
> > @@ -0,0 +1,37 @@
> > +#!/bin/bash
> > +# (C) 2016, "Yann E. MORIN" <yann.morin.1998@free.fr>
> > +# License: WTFPL, https://spdx.org/licenses/WTFPL.html
> > +
> > +main() {
> > + local ret start d h m mf fs
>
> fs is unused.
OK. Thanks for the review and tests! :-)
Regards,
Yann E. MORIN.
> Regards,
> Ricardo
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful user-facing tools
2017-06-19 15:42 ` Yann E. MORIN
@ 2017-06-21 2:06 ` Ricardo Martincoski
2017-06-21 5:21 ` Yann E. MORIN
0 siblings, 1 reply; 25+ messages in thread
From: Ricardo Martincoski @ 2017-06-21 2:06 UTC (permalink / raw)
To: buildroot
Hello,
On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote:
> On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly:
[snip]
>> BUT if it is called without a .config file, it hangs forever waiting for input,
>> and the menuconfig is sent to the log file.
>
> Indeed, my use-case is to only use it in an already-configured tree.
>
>> Maybe we can bailout if no .config is present.
>
> Hmmm... It's not trivial do detect, because the .config file is not
> necessarily in the current directory, for example:
>
> brmake -C /pat/to/buil-dir
>
> and I don't want to write a make option interpeter either...
Oh, I see now it is not trivial.
>
> But I'll see what I can do.
Only on the first use the user can get confused.
I think adding a usage example in the file header is enough, saying the script
must be run inside or pointing to (-C from make) an output directory already
configured, or something like that.
[snip]
>> > + a script that can be run instead of make, that prepends the date in
>> > + front of each line, redirects all of the build output to a file
>> > + ("'br.log' in the current directory), and just ouputs the Buildroot
>>
>> Maybe this file will be dropped as discussed in
>> http://patchwork.ozlabs.org/patch/777446/
>> If not, there is a typo
>>
>> ouputs -> outputs
>
> Yup, I already dropped the file locally.
[snip]
The purpose of the script (initially in 00-README.txt, with the typo fixed)
could be placed in the script header as well.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful user-facing tools
2017-06-21 2:06 ` Ricardo Martincoski
@ 2017-06-21 5:21 ` Yann E. MORIN
2017-06-21 21:44 ` Arnout Vandecappelle
0 siblings, 1 reply; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-21 5:21 UTC (permalink / raw)
To: buildroot
Ricardo, All,
On 2017-06-20 23:06 -0300, Ricardo Martincoski spake thusly:
> On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote:
> > On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly:
> [snip]
> >> BUT if it is called without a .config file, it hangs forever waiting for input,
> >> and the menuconfig is sent to the log file.
> >
> > Indeed, my use-case is to only use it in an already-configured tree.
> >
> >> Maybe we can bailout if no .config is present.
> >
> > Hmmm... It's not trivial do detect, because the .config file is not
> > necessarily in the current directory, for example:
> >
> > brmake -C /pat/to/buil-dir
> >
> > and I don't want to write a make option interpeter either...
>
> Oh, I see now it is not trivial.
Yet I think I was able to do it in a minimalist way that should cover
99.999999% of the cases (or so I hope, famous last words):
https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/contrib&id=221c72431bb12e0dde4051d9317d9fee2bee6eb2
I'll resubmit soon (after a bit more testing...)
> Only on the first use the user can get confused.
> I think adding a usage example in the file header is enough, saying the script
> must be run inside or pointing to (-C from make) an output directory already
> configured, or something like that.
>
> [snip]
> >> > + a script that can be run instead of make, that prepends the date in
> >> > + front of each line, redirects all of the build output to a file
> >> > + ("'br.log' in the current directory), and just ouputs the Buildroot
> >>
> >> Maybe this file will be dropped as discussed in
> >> http://patchwork.ozlabs.org/patch/777446/
> >> If not, there is a typo
> >>
> >> ouputs -> outputs
> >
> > Yup, I already dropped the file locally.
> [snip]
>
> The purpose of the script (initially in 00-README.txt, with the typo fixed)
> could be placed in the script header as well.
Yep, good idea. I'll do that.
Thanks!
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful user-facing tools
2017-06-21 5:21 ` Yann E. MORIN
@ 2017-06-21 21:44 ` Arnout Vandecappelle
2017-06-22 19:18 ` Yann E. MORIN
0 siblings, 1 reply; 25+ messages in thread
From: Arnout Vandecappelle @ 2017-06-21 21:44 UTC (permalink / raw)
To: buildroot
On 21-06-17 07:21, Yann E. MORIN wrote:
> Ricardo, All,
>
> On 2017-06-20 23:06 -0300, Ricardo Martincoski spake thusly:
>> On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote:
>>> On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly:
>> [snip]
>>>> BUT if it is called without a .config file, it hangs forever waiting for input,
>>>> and the menuconfig is sent to the log file.
>>>
>>> Indeed, my use-case is to only use it in an already-configured tree.
>>>
>>>> Maybe we can bailout if no .config is present.
>>>
>>> Hmmm... It's not trivial do detect, because the .config file is not
>>> necessarily in the current directory, for example:
>>>
>>> brmake -C /pat/to/buil-dir
>>>
>>> and I don't want to write a make option interpeter either...
>>
>> Oh, I see now it is not trivial.
How about instead make the script do
make -C ${0%/*}/.. O=${PWD} "${@}"
if $PWD is not the Buildroot dir. This still allows overriding either option on
the command line.
Not that this solves the issue...
> Yet I think I was able to do it in a minimalist way that should cover
> 99.999999% of the cases (or so I hope, famous last words):
> https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/contrib&id=221c72431bb12e0dde4051d9317d9fee2bee6eb2
I haven't tried, but doesn't it do the wrong thing with
make O=... -C ...
?
But it still doesn't completely solve the issue, because "brmake menuconfig"
still does the wrong thing... Unfortunately I don't know how to solve that
(other than detecting menuconfig as an argument, obviously).
On a related note, perhaps it's better to redirect stdin in case there is
something interactive during the build (e.g. Qt license approval).
Regards,
Arnout
> I'll resubmit soon (after a bit more testing...)
>
>> Only on the first use the user can get confused.
>> I think adding a usage example in the file header is enough, saying the script
>> must be run inside or pointing to (-C from make) an output directory already
>> configured, or something like that.
>>
>> [snip]
>>>>> + a script that can be run instead of make, that prepends the date in
>>>>> + front of each line, redirects all of the build output to a file
>>>>> + ("'br.log' in the current directory), and just ouputs the Buildroot
>>>>
>>>> Maybe this file will be dropped as discussed in
>>>> http://patchwork.ozlabs.org/patch/777446/
>>>> If not, there is a typo
>>>>
>>>> ouputs -> outputs
>>>
>>> Yup, I already dropped the file locally.
>> [snip]
>>
>> The purpose of the script (initially in 00-README.txt, with the typo fixed)
>> could be placed in the script header as well.
>
> Yep, good idea. I'll do that.
>
> Thanks!
>
> Regards,
> Yann E. MORIN.
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful user-facing tools
2017-06-21 21:44 ` Arnout Vandecappelle
@ 2017-06-22 19:18 ` Yann E. MORIN
0 siblings, 0 replies; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-22 19:18 UTC (permalink / raw)
To: buildroot
Arnout, All,
On 2017-06-21 23:44 +0200, Arnout Vandecappelle spake thusly:
> On 21-06-17 07:21, Yann E. MORIN wrote:
> > Ricardo, All,
> >
> > On 2017-06-20 23:06 -0300, Ricardo Martincoski spake thusly:
> >> On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote:
> >>> On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly:
> >> [snip]
> >>>> BUT if it is called without a .config file, it hangs forever waiting for input,
> >>>> and the menuconfig is sent to the log file.
> >>>
> >>> Indeed, my use-case is to only use it in an already-configured tree.
> >>>
> >>>> Maybe we can bailout if no .config is present.
> >>>
> >>> Hmmm... It's not trivial do detect, because the .config file is not
> >>> necessarily in the current directory, for example:
> >>>
> >>> brmake -C /pat/to/buil-dir
> >>>
> >>> and I don't want to write a make option interpeter either...
> >>
> >> Oh, I see now it is not trivial.
>
> How about instead make the script do
>
> make -C ${0%/*}/.. O=${PWD} "${@}"
What if one calls it with: brmake -C /path/to/my/O-dir
But as much as we try to 'fix' it, we always uncover corner cases,
becausze the above would consider ${PWD} to be the output dir if
/path/to/my/O-dir is not already configured... :-/
Corner cases after corner cases, I think I'd just try to catch the
obvious and trivial, and leave out the weird cases.
So, either there is a .config in the CWD, or we abort. Deal? ;-)
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts
2017-06-18 14:01 [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Yann E. MORIN
2017-06-18 14:01 ` [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful " Yann E. MORIN
@ 2017-06-18 14:01 ` Yann E. MORIN
2017-06-18 14:12 ` Thomas Petazzoni
2017-06-18 14:01 ` [Buildroot] [PATCH 3/4 v2] tools: move get-developers out of support/scripts/ Yann E. MORIN
` (2 subsequent siblings)
4 siblings, 1 reply; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-18 14:01 UTC (permalink / raw)
To: buildroot
Move it to the top-level tools/ directory, so that it is easier to
find for users.
Add a legacy symlink for those users who already used them, so as
not to break their habits.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
docs/manual/adding-packages-tips.txt | 12 +--
support/scripts/test-pkg | 198 +----------------------------------
tools/00-README.txt | 5 +
tools/test-pkg | 197 ++++++++++++++++++++++++++++++++++
4 files changed, 209 insertions(+), 203 deletions(-)
mode change 100755 => 120000 support/scripts/test-pkg
create mode 100755 tools/test-pkg
diff --git a/docs/manual/adding-packages-tips.txt b/docs/manual/adding-packages-tips.txt
index d1eab2b440..0047dfd14f 100644
--- a/docs/manual/adding-packages-tips.txt
+++ b/docs/manual/adding-packages-tips.txt
@@ -45,9 +45,9 @@ continuously test random configurations. However, these only build the
`master` branch of the git tree, and your new fancy package is not yet
there.
-Buildroot provides a script in +support/scripts/test-pkg+ that uses the
-same base configurations as used by the autobuilders so you can test
-your package in the same conditions.
+Buildroot provides a script in +tools/test-pkg+ that uses the same base
+configurations as used by the autobuilders so you can test your package
+in the same conditions.
First, create a config snippet that contains all the necessary options
needed to enable your package, but without any architecture or toolchain
@@ -74,7 +74,7 @@ Then run the +test-pkg+ script, by telling it what config snippet to use
and what package to test:
----
-$ ./support/scripts/test-pkg -c libcurl.config -p libcurl
+$ ./tools/test-pkg -c libcurl.config -p libcurl
----
This will try to build your package against all the toolchains used
@@ -83,7 +83,7 @@ too long to do so). The output lists all toolchains and the corresponding
result (excerpt, results are fake):
----
-$ ./support/scripts/test-pkg -c libcurl.config -p libcurl
+$ ./tools/test-pkg -c libcurl.config -p libcurl
armv5-ctng-linux-gnueabi [ 1/11]: OK
armv7-ctng-linux-gnueabihf [ 2/11]: OK
br-aarch64-glibc [ 3/11]: SKIPPED
@@ -124,7 +124,7 @@ The +test-pkg+ script accepts a few options, for which you can get some
help by running:
----
-$ ./support/scripts/test-pkg -h
+$ ./tools/test-pkg -h
----
[[github-download-url]]
diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
deleted file mode 100755
index 7633a21e53..0000000000
--- a/support/scripts/test-pkg
+++ /dev/null
@@ -1,197 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-TOOLCHAINS_URL='http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv'
-
-main() {
- local o O opts
- local cfg dir pkg random toolchain
- local ret nb nb_skip nb_fail nb_legal nb_tc build_dir
- local -a toolchains
-
- o='hc:d:p:r:'
- O='help,config-snippet:build-dir:package:,random:'
- opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")"
- eval set -- "${opts}"
-
- random=0
- while [ ${#} -gt 0 ]; do
- case "${1}" in
- (-h|--help)
- help; exit 0
- ;;
- (-c|--config-snippet)
- cfg="${2}"; shift 2
- ;;
- (-d|--build-dir)
- dir="${2}"; shift 2
- ;;
- (-p|--package)
- pkg="${2}"; shift 2
- ;;
- (-r|--random)
- random="${2}"; shift 2
- ;;
- (--)
- shift; break
- ;;
- esac
- done
- if [ -z "${cfg}" ]; then
- printf "error: no config snippet specified\n" >&2; exit 1
- fi
- if [ ! -e "${cfg}" ]; then
- printf "error: %s: no such file\n" "${cfg}" >&2; exit 1
- fi
- if [ -z "${dir}" ]; then
- dir="${HOME}/br-test-pkg"
- fi
-
- # Extract the URLs of the toolchains; drop internal toolchains
- # E.g.: http://server/path/to/name.config,arch,libc
- # --> http://server/path/to/name.config
- toolchains=($(curl -s "${TOOLCHAINS_URL}" \
- |sed -r -e 's/,.*//; /internal/d;' \
- |if [ ${random} -gt 0 ]; then \
- sort -R |head -n ${random}
- else
- cat
- fi |sort
- )
- )
-
- nb_tc="${#toolchains[@]}"
- if [ ${nb_tc} -eq 0 ]; then
- printf "error: no toolchain found (networking issue?)\n" >&2; exit 1
- fi
-
- nb=0
- nb_skip=0
- nb_fail=0
- nb_legal=0
- for toolchainconfig in "${toolchains[@]}"; do
- : $((nb++))
- # Using basename(1) on a URL works nicely
- toolchain="$(basename "${toolchainconfig}" .config)"
- build_dir="${dir}/${toolchain}"
- printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc}
- build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" && ret=0 || ret=${?}
- case ${ret} in
- (0) printf "OK\n";;
- (1) : $((nb_skip++)); printf "SKIPPED\n";;
- (2) : $((nb_fail++)); printf "FAILED\n";;
- (3) : $((nb_legal++)); printf "FAILED\n";;
- esac
- done
-
- printf "%d builds, %d skipped, %d build failed, %d legal-info failed\n" \
- ${nb} ${nb_skip} ${nb_fail} ${nb_legal}
-}
-
-build_one() {
- local dir="${1}"
- local url="${2}"
- local cfg="${3}"
- local pkg="${4}"
-
- mkdir -p "${dir}"
-
- if ! curl -s "${url}" >"${dir}/.config"; then
- return 2
- fi
-
- cat >>"${dir}/.config" <<-_EOF_
- BR2_INIT_NONE=y
- BR2_SYSTEM_BIN_SH_NONE=y
- # BR2_PACKAGE_BUSYBOX is not set
- # BR2_TARGET_ROOTFS_TAR is not set
- _EOF_
- cat "${cfg}" >>"${dir}/.config"
-
- if ! make O="${dir}" olddefconfig > "${dir}/logfile" 2>&1; then
- return 2
- fi
- # We want all the options from the snippet to be present as-is (set
- # or not set) in the actual .config; if one of them is not, it means
- # some dependency from the toolchain or arch is not available, in
- # which case this config is untestable and we skip it.
- # We don't care about the locale to sort in, as long as both sort are
- # done in the same locale.
- comm -23 <(sort "${cfg}") <(sort "${dir}/.config") >"${dir}/missing.config"
- if [ -s "${dir}/missing.config" ]; then
- return 1
- fi
- # Remove file, it's empty anyway.
- rm -f "${dir}/missing.config"
-
- if [ -n "${pkg}" ]; then
- if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then
- return 2
- fi
- fi
-
- # shellcheck disable=SC2086
- if ! make O="${dir}" ${pkg} >> "${dir}/logfile" 2>&1; then
- return 2
- fi
-
- # legal-info done systematically, because some packages have different
- # sources depending on the configuration (e.g. lua-5.2 vs. lua-5.3)
- if [ -n "${pkg}" ]; then
- if ! make O="${dir}" "${pkg}-legal-info" >> "${dir}/logfile" 2>&1; then
- return 3
- fi
- fi
-}
-
-help() {
- cat <<_EOF_
-test-pkg: test-build a package against various toolchains and architectures
-
-The supplied config snippet is appended to each toolchain config, the
-resulting configuration is checked to ensure it still contains all options
-specified in the snippet; if any is missing, the build is skipped, on the
-assumption that the package under test requires a toolchain or architecture
-feature that is missing.
-
-In case failures are noticed, you can fix the package and just re-run the
-same command again; it will re-run the test where it failed. If you did
-specify a package (with -p), the package build dir will be removed first.
-
-The list of toolchains is retrieved from the Buildroot autobuilders, available
-at ${TOOLCHAINS_URL}.
-
-Options:
-
- -h, --help
- Print this help.
-
- -c CFG, --config-snippet CFG
- Use the CFG file as the source for the config snippet. This file
- should contain all the config options required to build a package.
-
- -d DIR, --build-dir DIR
- Do the builds in directory DIR, one sub-dir per toolchain.
-
- -p PKG, --package PKG
- Test-build the package PKG, by running 'make PKG'; if not specified,
- just runs 'make'.
-
- -r N, --random N
- Limit the tests to the N randomly selected toolchains, instead of
- building with all toolchains.
-
-Example:
-
- Testing libcec would require a config snippet that contains:
- BR2_PACKAGE_LIBCEC=y
-
- Testing libcurl with openSSL support would require a snippet such as:
- BR2_PACKAGE_OPENSSL=y
- BR2_PACKAGE_LIBCURL=y
-
-_EOF_
-}
-
-my_name="${0##*/}"
-main "${@}"
diff --git a/support/scripts/test-pkg b/support/scripts/test-pkg
new file mode 120000
index 0000000000..2acaa87478
--- /dev/null
+++ b/support/scripts/test-pkg
@@ -0,0 +1 @@
+../../tools/test-pkg
\ No newline@end of file
diff --git a/tools/00-README.txt b/tools/00-README.txt
index ffc0fbf5ef..092e0eac8e 100644
--- a/tools/00-README.txt
+++ b/tools/00-README.txt
@@ -7,3 +7,8 @@ brmake
front of each line, redirects all of the build output to a file
("'br.log' in the current directory), and just ouputs the Buildroot
messages (those lines starting with >>>) on stdout.
+
+test-pkg
+ a script that tests a specific package against a set of various
+ toolchains, with the goal to detect toolchain-related dependencies
+ (wchar, threads...)
diff --git a/tools/test-pkg b/tools/test-pkg
new file mode 100755
index 0000000000..7633a21e53
--- /dev/null
+++ b/tools/test-pkg
@@ -0,0 +1,197 @@
+#!/usr/bin/env bash
+set -e
+
+TOOLCHAINS_URL='http://autobuild.buildroot.org/toolchains/configs/toolchain-configs.csv'
+
+main() {
+ local o O opts
+ local cfg dir pkg random toolchain
+ local ret nb nb_skip nb_fail nb_legal nb_tc build_dir
+ local -a toolchains
+
+ o='hc:d:p:r:'
+ O='help,config-snippet:build-dir:package:,random:'
+ opts="$(getopt -n "${my_name}" -o "${o}" -l "${O}" -- "${@}")"
+ eval set -- "${opts}"
+
+ random=0
+ while [ ${#} -gt 0 ]; do
+ case "${1}" in
+ (-h|--help)
+ help; exit 0
+ ;;
+ (-c|--config-snippet)
+ cfg="${2}"; shift 2
+ ;;
+ (-d|--build-dir)
+ dir="${2}"; shift 2
+ ;;
+ (-p|--package)
+ pkg="${2}"; shift 2
+ ;;
+ (-r|--random)
+ random="${2}"; shift 2
+ ;;
+ (--)
+ shift; break
+ ;;
+ esac
+ done
+ if [ -z "${cfg}" ]; then
+ printf "error: no config snippet specified\n" >&2; exit 1
+ fi
+ if [ ! -e "${cfg}" ]; then
+ printf "error: %s: no such file\n" "${cfg}" >&2; exit 1
+ fi
+ if [ -z "${dir}" ]; then
+ dir="${HOME}/br-test-pkg"
+ fi
+
+ # Extract the URLs of the toolchains; drop internal toolchains
+ # E.g.: http://server/path/to/name.config,arch,libc
+ # --> http://server/path/to/name.config
+ toolchains=($(curl -s "${TOOLCHAINS_URL}" \
+ |sed -r -e 's/,.*//; /internal/d;' \
+ |if [ ${random} -gt 0 ]; then \
+ sort -R |head -n ${random}
+ else
+ cat
+ fi |sort
+ )
+ )
+
+ nb_tc="${#toolchains[@]}"
+ if [ ${nb_tc} -eq 0 ]; then
+ printf "error: no toolchain found (networking issue?)\n" >&2; exit 1
+ fi
+
+ nb=0
+ nb_skip=0
+ nb_fail=0
+ nb_legal=0
+ for toolchainconfig in "${toolchains[@]}"; do
+ : $((nb++))
+ # Using basename(1) on a URL works nicely
+ toolchain="$(basename "${toolchainconfig}" .config)"
+ build_dir="${dir}/${toolchain}"
+ printf "%40s [%*d/%d]: " "${toolchain}" ${#nb_tc} ${nb} ${nb_tc}
+ build_one "${build_dir}" "${toolchainconfig}" "${cfg}" "${pkg}" && ret=0 || ret=${?}
+ case ${ret} in
+ (0) printf "OK\n";;
+ (1) : $((nb_skip++)); printf "SKIPPED\n";;
+ (2) : $((nb_fail++)); printf "FAILED\n";;
+ (3) : $((nb_legal++)); printf "FAILED\n";;
+ esac
+ done
+
+ printf "%d builds, %d skipped, %d build failed, %d legal-info failed\n" \
+ ${nb} ${nb_skip} ${nb_fail} ${nb_legal}
+}
+
+build_one() {
+ local dir="${1}"
+ local url="${2}"
+ local cfg="${3}"
+ local pkg="${4}"
+
+ mkdir -p "${dir}"
+
+ if ! curl -s "${url}" >"${dir}/.config"; then
+ return 2
+ fi
+
+ cat >>"${dir}/.config" <<-_EOF_
+ BR2_INIT_NONE=y
+ BR2_SYSTEM_BIN_SH_NONE=y
+ # BR2_PACKAGE_BUSYBOX is not set
+ # BR2_TARGET_ROOTFS_TAR is not set
+ _EOF_
+ cat "${cfg}" >>"${dir}/.config"
+
+ if ! make O="${dir}" olddefconfig > "${dir}/logfile" 2>&1; then
+ return 2
+ fi
+ # We want all the options from the snippet to be present as-is (set
+ # or not set) in the actual .config; if one of them is not, it means
+ # some dependency from the toolchain or arch is not available, in
+ # which case this config is untestable and we skip it.
+ # We don't care about the locale to sort in, as long as both sort are
+ # done in the same locale.
+ comm -23 <(sort "${cfg}") <(sort "${dir}/.config") >"${dir}/missing.config"
+ if [ -s "${dir}/missing.config" ]; then
+ return 1
+ fi
+ # Remove file, it's empty anyway.
+ rm -f "${dir}/missing.config"
+
+ if [ -n "${pkg}" ]; then
+ if ! make O="${dir}" "${pkg}-dirclean" >> "${dir}/logfile" 2>&1; then
+ return 2
+ fi
+ fi
+
+ # shellcheck disable=SC2086
+ if ! make O="${dir}" ${pkg} >> "${dir}/logfile" 2>&1; then
+ return 2
+ fi
+
+ # legal-info done systematically, because some packages have different
+ # sources depending on the configuration (e.g. lua-5.2 vs. lua-5.3)
+ if [ -n "${pkg}" ]; then
+ if ! make O="${dir}" "${pkg}-legal-info" >> "${dir}/logfile" 2>&1; then
+ return 3
+ fi
+ fi
+}
+
+help() {
+ cat <<_EOF_
+test-pkg: test-build a package against various toolchains and architectures
+
+The supplied config snippet is appended to each toolchain config, the
+resulting configuration is checked to ensure it still contains all options
+specified in the snippet; if any is missing, the build is skipped, on the
+assumption that the package under test requires a toolchain or architecture
+feature that is missing.
+
+In case failures are noticed, you can fix the package and just re-run the
+same command again; it will re-run the test where it failed. If you did
+specify a package (with -p), the package build dir will be removed first.
+
+The list of toolchains is retrieved from the Buildroot autobuilders, available
+at ${TOOLCHAINS_URL}.
+
+Options:
+
+ -h, --help
+ Print this help.
+
+ -c CFG, --config-snippet CFG
+ Use the CFG file as the source for the config snippet. This file
+ should contain all the config options required to build a package.
+
+ -d DIR, --build-dir DIR
+ Do the builds in directory DIR, one sub-dir per toolchain.
+
+ -p PKG, --package PKG
+ Test-build the package PKG, by running 'make PKG'; if not specified,
+ just runs 'make'.
+
+ -r N, --random N
+ Limit the tests to the N randomly selected toolchains, instead of
+ building with all toolchains.
+
+Example:
+
+ Testing libcec would require a config snippet that contains:
+ BR2_PACKAGE_LIBCEC=y
+
+ Testing libcurl with openSSL support would require a snippet such as:
+ BR2_PACKAGE_OPENSSL=y
+ BR2_PACKAGE_LIBCURL=y
+
+_EOF_
+}
+
+my_name="${0##*/}"
+main "${@}"
--
2.11.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts
2017-06-18 14:01 ` [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts Yann E. MORIN
@ 2017-06-18 14:12 ` Thomas Petazzoni
2017-06-18 14:16 ` Yann E. MORIN
2017-06-22 7:57 ` Arnout Vandecappelle
0 siblings, 2 replies; 25+ messages in thread
From: Thomas Petazzoni @ 2017-06-18 14:12 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, 18 Jun 2017 16:01:43 +0200, Yann E. MORIN wrote:
> docs/manual/adding-packages-tips.txt | 12 +--
> support/scripts/test-pkg | 198 +----------------------------------
> tools/00-README.txt | 5 +
> tools/test-pkg | 197 ++++++++++++++++++++++++++++++++++
> 4 files changed, 209 insertions(+), 203 deletions(-)
> mode change 100755 => 120000 support/scripts/test-pkg
> create mode 100755 tools/test-pkg
No strong opinion on whether such a move is desirable, but at the very
list I believe we should not add this 00-README.txt file. Nowhere else
in the tree we have files named like this, and I don't think we should
start adding these.
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts
2017-06-18 14:12 ` Thomas Petazzoni
@ 2017-06-18 14:16 ` Yann E. MORIN
2017-06-22 7:57 ` Arnout Vandecappelle
1 sibling, 0 replies; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-18 14:16 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2017-06-18 16:12 +0200, Thomas Petazzoni spake thusly:
> On Sun, 18 Jun 2017 16:01:43 +0200, Yann E. MORIN wrote:
> > docs/manual/adding-packages-tips.txt | 12 +--
> > support/scripts/test-pkg | 198 +----------------------------------
> > tools/00-README.txt | 5 +
> > tools/test-pkg | 197 ++++++++++++++++++++++++++++++++++
> > 4 files changed, 209 insertions(+), 203 deletions(-)
> > mode change 100755 => 120000 support/scripts/test-pkg
> > create mode 100755 tools/test-pkg
>
> No strong opinion on whether such a move is desirable,
That was a proposal by Arnout:
http://lists.busybox.net/pipermail/buildroot/2016-October/174290.html
> but at the very
> list I believe we should not add this 00-README.txt file. Nowhere else
> in the tree we have files named like this, and I don't think we should
> start adding these.
I'm OK for dropping it.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts
2017-06-18 14:12 ` Thomas Petazzoni
2017-06-18 14:16 ` Yann E. MORIN
@ 2017-06-22 7:57 ` Arnout Vandecappelle
2017-06-22 8:13 ` Thomas Petazzoni
1 sibling, 1 reply; 25+ messages in thread
From: Arnout Vandecappelle @ 2017-06-22 7:57 UTC (permalink / raw)
To: buildroot
On 18-06-17 16:12, Thomas Petazzoni wrote:
> Hello,
>
> On Sun, 18 Jun 2017 16:01:43 +0200, Yann E. MORIN wrote:
>
>> docs/manual/adding-packages-tips.txt | 12 +--
>> support/scripts/test-pkg | 198 +----------------------------------
>> tools/00-README.txt | 5 +
>> tools/test-pkg | 197 ++++++++++++++++++++++++++++++++++
>> 4 files changed, 209 insertions(+), 203 deletions(-)
>> mode change 100755 => 120000 support/scripts/test-pkg
>> create mode 100755 tools/test-pkg
>
> No strong opinion on whether such a move is desirable, but at the very
> list I believe we should not add this 00-README.txt file. Nowhere else
> in the tree we have files named like this, and I don't think we should
> start adding these.
Well, we have readme.txt in boards/, we have a README at the top level, we have
a README generated by legal-info, we have a readme.txt in checkpackagelib... We
indeed don't have a 00-README.txt though. IMO all of these files should be
called README.md (or readme.md) so they are nicely formatted when viewed online.
For this particular instance, I do like having some kind of README there. We
need an overview of the existing tools somewhere, and I think a README file is a
more appropriate place than the manual because it's easier to keep consistent.
From the manual you can point to the directory.
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts
2017-06-22 7:57 ` Arnout Vandecappelle
@ 2017-06-22 8:13 ` Thomas Petazzoni
2017-06-22 8:30 ` Arnout Vandecappelle
0 siblings, 1 reply; 25+ messages in thread
From: Thomas Petazzoni @ 2017-06-22 8:13 UTC (permalink / raw)
To: buildroot
Hello,
On Thu, 22 Jun 2017 09:57:36 +0200, Arnout Vandecappelle wrote:
> > No strong opinion on whether such a move is desirable, but at the very
> > list I believe we should not add this 00-README.txt file. Nowhere else
> > in the tree we have files named like this, and I don't think we should
> > start adding these.
>
> Well, we have readme.txt in boards/, we have a README at the top level, we have
> a README generated by legal-info, we have a readme.txt in checkpackagelib... We
> indeed don't have a 00-README.txt though. IMO all of these files should be
> called README.md (or readme.md) so they are nicely formatted when viewed online.
>
> For this particular instance, I do like having some kind of README there. We
> need an overview of the existing tools somewhere, and I think a README file is a
> more appropriate place than the manual because it's easier to keep consistent.
> From the manual you can point to the directory.
ACK, but then, named "readme.txt", to match what we're doing elsewhere:
$ find . -name 'readme.txt' | wc -l
99
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts
2017-06-22 8:13 ` Thomas Petazzoni
@ 2017-06-22 8:30 ` Arnout Vandecappelle
2017-06-22 19:20 ` Yann E. MORIN
0 siblings, 1 reply; 25+ messages in thread
From: Arnout Vandecappelle @ 2017-06-22 8:30 UTC (permalink / raw)
To: buildroot
On 22-06-17 10:13, Thomas Petazzoni wrote:
> Hello,
>
> On Thu, 22 Jun 2017 09:57:36 +0200, Arnout Vandecappelle wrote:
>
>>> No strong opinion on whether such a move is desirable, but at the very
>>> list I believe we should not add this 00-README.txt file. Nowhere else
>>> in the tree we have files named like this, and I don't think we should
>>> start adding these.
>>
>> Well, we have readme.txt in boards/, we have a README at the top level, we have
>> a README generated by legal-info, we have a readme.txt in checkpackagelib... We
>> indeed don't have a 00-README.txt though. IMO all of these files should be
>> called README.md (or readme.md) so they are nicely formatted when viewed online.
>>
>> For this particular instance, I do like having some kind of README there. We
>> need an overview of the existing tools somewhere, and I think a README file is a
>> more appropriate place than the manual because it's easier to keep consistent.
>> From the manual you can point to the directory.
>
> ACK, but then, named "readme.txt", to match what we're doing elsewhere:
>
> $ find . -name 'readme.txt' | wc -l
> 99
Well, the point I tried to make in the paragraph above is that readme.txt is
not a great name, readme.md would be better so it gets nicely formatted when
viewed online on github, gitlab, etc and also in some editors. README.md is even
better because then it is sorted first - which for the tools directory is very
relevant IMO.
Note that 96 of those 99 readmes are in board/. But they still should be called
readme.md :-)
Regards,
Arnout
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts
2017-06-22 8:30 ` Arnout Vandecappelle
@ 2017-06-22 19:20 ` Yann E. MORIN
0 siblings, 0 replies; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-22 19:20 UTC (permalink / raw)
To: buildroot
Arnout, Thomas, All,
On 2017-06-22 10:30 +0200, Arnout Vandecappelle spake thusly:
> On 22-06-17 10:13, Thomas Petazzoni wrote:
> > Hello,
> >
> > On Thu, 22 Jun 2017 09:57:36 +0200, Arnout Vandecappelle wrote:
> >
> >>> No strong opinion on whether such a move is desirable, but at the very
> >>> list I believe we should not add this 00-README.txt file. Nowhere else
> >>> in the tree we have files named like this, and I don't think we should
> >>> start adding these.
> >>
> >> Well, we have readme.txt in boards/, we have a README at the top level, we have
> >> a README generated by legal-info, we have a readme.txt in checkpackagelib... We
> >> indeed don't have a 00-README.txt though. IMO all of these files should be
> >> called README.md (or readme.md) so they are nicely formatted when viewed online.
> >>
> >> For this particular instance, I do like having some kind of README there. We
> >> need an overview of the existing tools somewhere, and I think a README file is a
> >> more appropriate place than the manual because it's easier to keep consistent.
> >> From the manual you can point to the directory.
> >
> > ACK, but then, named "readme.txt", to match what we're doing elsewhere:
> >
> > $ find . -name 'readme.txt' | wc -l
> > 99
OK, I'll re-instate the readme, and will change its name to readme.txt.
> Well, the point I tried to make in the paragraph above is that readme.txt is
> not a great name, readme.md would be better so it gets nicely formatted when
> viewed online on github, gitlab, etc and also in some editors. README.md is even
> better because then it is sorted first - which for the tools directory is very
> relevant IMO.
>
> Note that 96 of those 99 readmes are in board/. But they still should be called
> readme.md :-)
IIUC, github et al. expect README.md to contain Markdown, not plain
text. A simple README (or maybe readme as well) would still be
displayed, but no rednering would be done.
In any case, this is a topic for a completely different series.
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 3/4 v2] tools: move get-developers out of support/scripts/
2017-06-18 14:01 [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Yann E. MORIN
2017-06-18 14:01 ` [Buildroot] [PATCH 1/4 v2] tools: add a directory to store some useful " Yann E. MORIN
2017-06-18 14:01 ` [Buildroot] [PATCH 2/4 v2] tools: move test-pkg out of support/scripts Yann E. MORIN
@ 2017-06-18 14:01 ` Yann E. MORIN
2017-06-18 14:01 ` [Buildroot] [PATCH 4/4 v2] tools: move check-package " Yann E. MORIN
2017-06-18 14:15 ` [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Thomas Petazzoni
4 siblings, 0 replies; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-18 14:01 UTC (permalink / raw)
To: buildroot
Move it to the top-level tools/ directory, so that it is easier to
find for users.
Add a legacy symlink for those users who already used them, so as
not to break their habits.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
docs/manual/contribute.txt | 2 +-
support/scripts/get-developers | 84 +--------------------------
tools/00-README.txt | 5 ++
tools/get-developers | 83 ++++++++++++++++++++++++++
{support/scripts => tools}/getdeveloperlib.py | 0
5 files changed, 90 insertions(+), 84 deletions(-)
mode change 100755 => 120000 support/scripts/get-developers
create mode 100755 tools/get-developers
rename {support/scripts => tools}/getdeveloperlib.py (100%)
diff --git a/docs/manual/contribute.txt b/docs/manual/contribute.txt
index 719032bcc4..c10587726c 100644
--- a/docs/manual/contribute.txt
+++ b/docs/manual/contribute.txt
@@ -295,7 +295,7 @@ information). This tool reads your patches and outputs the appropriate
+git send-email+ command to use:
---------------------
-$ ./support/scripts/get-developers outgoing/*
+$ ./tools/get-developers outgoing/*
---------------------
Use the output of +get-developers+ to send your patches:
diff --git a/support/scripts/get-developers b/support/scripts/get-developers
deleted file mode 100755
index 40ed08ffe1..0000000000
--- a/support/scripts/get-developers
+++ /dev/null
@@ -1,83 +0,0 @@
-#!/usr/bin/env python
-
-import argparse
-import getdeveloperlib
-
-def parse_args():
- parser = argparse.ArgumentParser()
- parser.add_argument('patches', metavar='P', type=argparse.FileType('r'), nargs='*',
- help='list of patches (use - to read patches from stdin)')
- parser.add_argument('-a', dest='architecture', action='store',
- help='find developers in charge of this architecture')
- parser.add_argument('-p', dest='package', action='store',
- help='find developers in charge of this package')
- parser.add_argument('-c', dest='check', action='store_const',
- const=True, help='list files not handled by any developer')
- return parser.parse_args()
-
-def __main__():
- devs = getdeveloperlib.parse_developers()
- if devs is None:
- sys.exit(1)
- args = parse_args()
-
- # Check that only one action is given
- action = 0
- if args.architecture is not None:
- action += 1
- if args.package is not None:
- action += 1
- if args.check:
- action += 1
- if len(args.patches) != 0:
- action += 1
- if action > 1:
- print("Cannot do more than one action")
- return
- if action == 0:
- print("No action specified")
- return
-
- # Handle the check action
- if args.check:
- files = getdeveloperlib.check_developers(devs)
- for f in files:
- print(f)
-
- # Handle the architecture action
- if args.architecture is not None:
- for dev in devs:
- if args.architecture in dev.architectures:
- print(dev.name)
- return
-
- # Handle the package action
- if args.package is not None:
- for dev in devs:
- if args.package in dev.packages:
- print(dev.name)
- return
-
- # Handle the patches action
- if len(args.patches) != 0:
- (files, infras) = getdeveloperlib.analyze_patches(args.patches)
- matching_devs = set()
- for dev in devs:
- # See if we have developers matching by package name
- for f in files:
- if dev.hasfile(f):
- matching_devs.add(dev.name)
- # See if we have developers matching by package infra
- for i in infras:
- if i in dev.infras:
- matching_devs.add(dev.name)
-
- result = "--to buildroot at buildroot.org"
- for dev in matching_devs:
- result += " --cc \"%s\"" % dev
-
- if result != "":
- print("git send-email %s" % result)
-
-__main__()
-
diff --git a/support/scripts/get-developers b/support/scripts/get-developers
new file mode 120000
index 0000000000..7209567e15
--- /dev/null
+++ b/support/scripts/get-developers
@@ -0,0 +1 @@
+../../tools/get-developers
\ No newline at end of file
diff --git a/tools/00-README.txt b/tools/00-README.txt
index 092e0eac8e..2740b97e5b 100644
--- a/tools/00-README.txt
+++ b/tools/00-README.txt
@@ -8,6 +8,11 @@ brmake
("'br.log' in the current directory), and just ouputs the Buildroot
messages (those lines starting with >>>) on stdout.
+get-developpers
+ a script to return the list of people interested in a specific part
+ of Buildroot, so they can be Cc:ed on a mail. Accepts a patch as
+ input, a package name or and architecture name.
+
test-pkg
a script that tests a specific package against a set of various
toolchains, with the goal to detect toolchain-related dependencies
diff --git a/tools/get-developers b/tools/get-developers
new file mode 100755
index 0000000000..40ed08ffe1
--- /dev/null
+++ b/tools/get-developers
@@ -0,0 +1,83 @@
+#!/usr/bin/env python
+
+import argparse
+import getdeveloperlib
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('patches', metavar='P', type=argparse.FileType('r'), nargs='*',
+ help='list of patches (use - to read patches from stdin)')
+ parser.add_argument('-a', dest='architecture', action='store',
+ help='find developers in charge of this architecture')
+ parser.add_argument('-p', dest='package', action='store',
+ help='find developers in charge of this package')
+ parser.add_argument('-c', dest='check', action='store_const',
+ const=True, help='list files not handled by any developer')
+ return parser.parse_args()
+
+def __main__():
+ devs = getdeveloperlib.parse_developers()
+ if devs is None:
+ sys.exit(1)
+ args = parse_args()
+
+ # Check that only one action is given
+ action = 0
+ if args.architecture is not None:
+ action += 1
+ if args.package is not None:
+ action += 1
+ if args.check:
+ action += 1
+ if len(args.patches) != 0:
+ action += 1
+ if action > 1:
+ print("Cannot do more than one action")
+ return
+ if action == 0:
+ print("No action specified")
+ return
+
+ # Handle the check action
+ if args.check:
+ files = getdeveloperlib.check_developers(devs)
+ for f in files:
+ print(f)
+
+ # Handle the architecture action
+ if args.architecture is not None:
+ for dev in devs:
+ if args.architecture in dev.architectures:
+ print(dev.name)
+ return
+
+ # Handle the package action
+ if args.package is not None:
+ for dev in devs:
+ if args.package in dev.packages:
+ print(dev.name)
+ return
+
+ # Handle the patches action
+ if len(args.patches) != 0:
+ (files, infras) = getdeveloperlib.analyze_patches(args.patches)
+ matching_devs = set()
+ for dev in devs:
+ # See if we have developers matching by package name
+ for f in files:
+ if dev.hasfile(f):
+ matching_devs.add(dev.name)
+ # See if we have developers matching by package infra
+ for i in infras:
+ if i in dev.infras:
+ matching_devs.add(dev.name)
+
+ result = "--to buildroot at buildroot.org"
+ for dev in matching_devs:
+ result += " --cc \"%s\"" % dev
+
+ if result != "":
+ print("git send-email %s" % result)
+
+__main__()
+
diff --git a/support/scripts/getdeveloperlib.py b/tools/getdeveloperlib.py
similarity index 100%
rename from support/scripts/getdeveloperlib.py
rename to tools/getdeveloperlib.py
--
2.11.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 4/4 v2] tools: move check-package out of support/scripts/
2017-06-18 14:01 [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Yann E. MORIN
` (2 preceding siblings ...)
2017-06-18 14:01 ` [Buildroot] [PATCH 3/4 v2] tools: move get-developers out of support/scripts/ Yann E. MORIN
@ 2017-06-18 14:01 ` Yann E. MORIN
2017-06-19 1:13 ` Ricardo Martincoski
2017-06-19 3:22 ` Ricardo Martincoski
2017-06-18 14:15 ` [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Thomas Petazzoni
4 siblings, 2 replies; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-18 14:01 UTC (permalink / raw)
To: buildroot
Move it to the top-level tools/ directory, so that it is easier to
find for users.
Add a legacy symlink for those users who already used them, so as
not to break their habits.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
---
support/scripts/check-package | 145 +--------------------
support/scripts/pkg-stats | 2 +-
tools/00-README.txt | 4 +
tools/check-package | 144 ++++++++++++++++++++
.../scripts => tools}/checkpackagelib/__init__.py | 0
{support/scripts => tools}/checkpackagelib/base.py | 0
{support/scripts => tools}/checkpackagelib/lib.py | 0
.../checkpackagelib/lib_config.py | 0
.../scripts => tools}/checkpackagelib/lib_hash.py | 0
.../scripts => tools}/checkpackagelib/lib_mk.py | 0
.../scripts => tools}/checkpackagelib/lib_patch.py | 0
.../scripts => tools}/checkpackagelib/readme.txt | 10 +-
12 files changed, 155 insertions(+), 150 deletions(-)
mode change 100755 => 120000 support/scripts/check-package
create mode 100755 tools/check-package
rename {support/scripts => tools}/checkpackagelib/__init__.py (100%)
rename {support/scripts => tools}/checkpackagelib/base.py (100%)
rename {support/scripts => tools}/checkpackagelib/lib.py (100%)
rename {support/scripts => tools}/checkpackagelib/lib_config.py (100%)
rename {support/scripts => tools}/checkpackagelib/lib_hash.py (100%)
rename {support/scripts => tools}/checkpackagelib/lib_mk.py (100%)
rename {support/scripts => tools}/checkpackagelib/lib_patch.py (100%)
rename {support/scripts => tools}/checkpackagelib/readme.txt (91%)
diff --git a/support/scripts/check-package b/support/scripts/check-package
deleted file mode 100755
index 74ea46f819..0000000000
--- a/support/scripts/check-package
+++ /dev/null
@@ -1,144 +0,0 @@
-#!/usr/bin/env python
-# See support/scripts/checkpackagelib/readme.txt before editing this file.
-
-from __future__ import print_function
-import argparse
-import inspect
-import re
-import sys
-
-import checkpackagelib.lib_config
-import checkpackagelib.lib_hash
-import checkpackagelib.lib_mk
-import checkpackagelib.lib_patch
-
-VERBOSE_LEVEL_TO_SHOW_IGNORED_FILES = 3
-flags = None # Command line arguments.
-
-
-def parse_args():
- parser = argparse.ArgumentParser()
-
- # Do not use argparse.FileType("r") here because only files with known
- # format will be open based on the filename.
- parser.add_argument("files", metavar="F", type=str, nargs="*",
- help="list of files")
-
- parser.add_argument("--manual-url", action="store",
- default="http://nightly.buildroot.org/",
- help="default: %(default)s")
- parser.add_argument("--verbose", "-v", action="count", default=0)
-
- # Now the debug options in the order they are processed.
- parser.add_argument("--include-only", dest="include_list", action="append",
- help="run only the specified functions (debug)")
- parser.add_argument("--exclude", dest="exclude_list", action="append",
- help="do not run the specified functions (debug)")
- parser.add_argument("--dry-run", action="store_true", help="print the "
- "functions that would be called for each file (debug)")
-
- return parser.parse_args()
-
-
-CONFIG_IN_FILENAME = re.compile("/Config\.\S*$")
-FILE_IS_FROM_A_PACKAGE = re.compile("package/[^/]*/")
-
-
-def get_lib_from_filename(fname):
- if FILE_IS_FROM_A_PACKAGE.search(fname) is None:
- return None
- if CONFIG_IN_FILENAME.search(fname):
- return checkpackagelib.lib_config
- if fname.endswith(".hash"):
- return checkpackagelib.lib_hash
- if fname.endswith(".mk"):
- return checkpackagelib.lib_mk
- if fname.endswith(".patch"):
- return checkpackagelib.lib_patch
- return None
-
-
-def is_a_check_function(m):
- if not inspect.isclass(m):
- return False
- # do not call the base class
- if m.__name__.startswith("_"):
- return False
- if flags.include_list and m.__name__ not in flags.include_list:
- return False
- if flags.exclude_list and m.__name__ in flags.exclude_list:
- return False
- return True
-
-
-def print_warnings(warnings):
- # Avoid the need to use 'return []' at the end of every check function.
- if warnings is None:
- return 0 # No warning generated.
-
- for level, message in enumerate(warnings):
- if flags.verbose >= level:
- print(message.replace("\t", "< tab >").rstrip())
- return 1 # One more warning to count.
-
-
-def check_file_using_lib(fname):
- # Count number of warnings generated and lines processed.
- nwarnings = 0
- nlines = 0
-
- lib = get_lib_from_filename(fname)
- if not lib:
- if flags.verbose >= VERBOSE_LEVEL_TO_SHOW_IGNORED_FILES:
- print("{}: ignored".format(fname))
- return nwarnings, nlines
- classes = inspect.getmembers(lib, is_a_check_function)
-
- if flags.dry_run:
- functions_to_run = [c[0] for c in classes]
- print("{}: would run: {}".format(fname, functions_to_run))
- return nwarnings, nlines
-
- objects = [c[1](fname, flags.manual_url) for c in classes]
-
- for cf in objects:
- nwarnings += print_warnings(cf.before())
- for lineno, text in enumerate(open(fname, "r").readlines()):
- nlines += 1
- for cf in objects:
- nwarnings += print_warnings(cf.check_line(lineno + 1, text))
- for cf in objects:
- nwarnings += print_warnings(cf.after())
-
- return nwarnings, nlines
-
-
-def __main__():
- global flags
- flags = parse_args()
-
- if len(flags.files) == 0:
- print("No files to check style")
- sys.exit(1)
-
- # Accumulate number of warnings generated and lines processed.
- total_warnings = 0
- total_lines = 0
-
- for fname in flags.files:
- nwarnings, nlines = check_file_using_lib(fname)
- total_warnings += nwarnings
- total_lines += nlines
-
- # The warning messages are printed to stdout and can be post-processed
- # (e.g. counted by 'wc'), so for stats use stderr. Wait all warnings are
- # printed, for the case there are many of them, before printing stats.
- sys.stdout.flush()
- print("{} lines processed".format(total_lines), file=sys.stderr)
- print("{} warnings generated".format(total_warnings), file=sys.stderr)
-
- if total_warnings > 0:
- sys.exit(1)
-
-
-__main__()
diff --git a/support/scripts/check-package b/support/scripts/check-package
new file mode 120000
index 0000000000..16a6051e3a
--- /dev/null
+++ b/support/scripts/check-package
@@ -0,0 +1 @@
+../../tools/check-package
\ No newline at end of file
diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats
index 4cf1f82518..f827877052 100755
--- a/support/scripts/pkg-stats
+++ b/support/scripts/pkg-stats
@@ -337,7 +337,7 @@ for i in $(find boot/ linux/ package/ toolchain/ -name '*.mk' | sort) ; do
fi
file_list=$(find ${package_dir} -name '*.mk' -o -name '*.in*' -o -name '*.hash')
- nwarnings=$(./support/scripts/check-package ${file_list} 2>&1 | sed '/\([0-9]*\) warnings generated/!d; s//\1/')
+ nwarnings=$(./tools/check-package ${file_list} 2>&1 | sed '/\([0-9]*\) warnings generated/!d; s//\1/')
if [ ${nwarnings} -eq 0 ] ; then
echo "<td class=\"centered correct\">${nwarnings}</td>"
else
diff --git a/tools/00-README.txt b/tools/00-README.txt
index 2740b97e5b..5e563cc59a 100644
--- a/tools/00-README.txt
+++ b/tools/00-README.txt
@@ -8,6 +8,10 @@ brmake
("'br.log' in the current directory), and just ouputs the Buildroot
messages (those lines starting with >>>) on stdout.
+check-package
+ a script that checks the coding style of a package's Config.in and
+ .mk files, and also tests them for various types of typoes.
+
get-developpers
a script to return the list of people interested in a specific part
of Buildroot, so they can be Cc:ed on a mail. Accepts a patch as
diff --git a/tools/check-package b/tools/check-package
new file mode 100755
index 0000000000..cdbac94929
--- /dev/null
+++ b/tools/check-package
@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+# See tools/checkpackagelib/readme.txt before editing this file.
+
+from __future__ import print_function
+import argparse
+import inspect
+import re
+import sys
+
+import checkpackagelib.lib_config
+import checkpackagelib.lib_hash
+import checkpackagelib.lib_mk
+import checkpackagelib.lib_patch
+
+VERBOSE_LEVEL_TO_SHOW_IGNORED_FILES = 3
+flags = None # Command line arguments.
+
+
+def parse_args():
+ parser = argparse.ArgumentParser()
+
+ # Do not use argparse.FileType("r") here because only files with known
+ # format will be open based on the filename.
+ parser.add_argument("files", metavar="F", type=str, nargs="*",
+ help="list of files")
+
+ parser.add_argument("--manual-url", action="store",
+ default="http://nightly.buildroot.org/",
+ help="default: %(default)s")
+ parser.add_argument("--verbose", "-v", action="count", default=0)
+
+ # Now the debug options in the order they are processed.
+ parser.add_argument("--include-only", dest="include_list", action="append",
+ help="run only the specified functions (debug)")
+ parser.add_argument("--exclude", dest="exclude_list", action="append",
+ help="do not run the specified functions (debug)")
+ parser.add_argument("--dry-run", action="store_true", help="print the "
+ "functions that would be called for each file (debug)")
+
+ return parser.parse_args()
+
+
+CONFIG_IN_FILENAME = re.compile("/Config\.\S*$")
+FILE_IS_FROM_A_PACKAGE = re.compile("package/[^/]*/")
+
+
+def get_lib_from_filename(fname):
+ if FILE_IS_FROM_A_PACKAGE.search(fname) is None:
+ return None
+ if CONFIG_IN_FILENAME.search(fname):
+ return checkpackagelib.lib_config
+ if fname.endswith(".hash"):
+ return checkpackagelib.lib_hash
+ if fname.endswith(".mk"):
+ return checkpackagelib.lib_mk
+ if fname.endswith(".patch"):
+ return checkpackagelib.lib_patch
+ return None
+
+
+def is_a_check_function(m):
+ if not inspect.isclass(m):
+ return False
+ # do not call the base class
+ if m.__name__.startswith("_"):
+ return False
+ if flags.include_list and m.__name__ not in flags.include_list:
+ return False
+ if flags.exclude_list and m.__name__ in flags.exclude_list:
+ return False
+ return True
+
+
+def print_warnings(warnings):
+ # Avoid the need to use 'return []' at the end of every check function.
+ if warnings is None:
+ return 0 # No warning generated.
+
+ for level, message in enumerate(warnings):
+ if flags.verbose >= level:
+ print(message.replace("\t", "< tab >").rstrip())
+ return 1 # One more warning to count.
+
+
+def check_file_using_lib(fname):
+ # Count number of warnings generated and lines processed.
+ nwarnings = 0
+ nlines = 0
+
+ lib = get_lib_from_filename(fname)
+ if not lib:
+ if flags.verbose >= VERBOSE_LEVEL_TO_SHOW_IGNORED_FILES:
+ print("{}: ignored".format(fname))
+ return nwarnings, nlines
+ classes = inspect.getmembers(lib, is_a_check_function)
+
+ if flags.dry_run:
+ functions_to_run = [c[0] for c in classes]
+ print("{}: would run: {}".format(fname, functions_to_run))
+ return nwarnings, nlines
+
+ objects = [c[1](fname, flags.manual_url) for c in classes]
+
+ for cf in objects:
+ nwarnings += print_warnings(cf.before())
+ for lineno, text in enumerate(open(fname, "r").readlines()):
+ nlines += 1
+ for cf in objects:
+ nwarnings += print_warnings(cf.check_line(lineno + 1, text))
+ for cf in objects:
+ nwarnings += print_warnings(cf.after())
+
+ return nwarnings, nlines
+
+
+def __main__():
+ global flags
+ flags = parse_args()
+
+ if len(flags.files) == 0:
+ print("No files to check style")
+ sys.exit(1)
+
+ # Accumulate number of warnings generated and lines processed.
+ total_warnings = 0
+ total_lines = 0
+
+ for fname in flags.files:
+ nwarnings, nlines = check_file_using_lib(fname)
+ total_warnings += nwarnings
+ total_lines += nlines
+
+ # The warning messages are printed to stdout and can be post-processed
+ # (e.g. counted by 'wc'), so for stats use stderr. Wait all warnings are
+ # printed, for the case there are many of them, before printing stats.
+ sys.stdout.flush()
+ print("{} lines processed".format(total_lines), file=sys.stderr)
+ print("{} warnings generated".format(total_warnings), file=sys.stderr)
+
+ if total_warnings > 0:
+ sys.exit(1)
+
+
+__main__()
diff --git a/support/scripts/checkpackagelib/__init__.py b/tools/checkpackagelib/__init__.py
similarity index 100%
rename from support/scripts/checkpackagelib/__init__.py
rename to tools/checkpackagelib/__init__.py
diff --git a/support/scripts/checkpackagelib/base.py b/tools/checkpackagelib/base.py
similarity index 100%
rename from support/scripts/checkpackagelib/base.py
rename to tools/checkpackagelib/base.py
diff --git a/support/scripts/checkpackagelib/lib.py b/tools/checkpackagelib/lib.py
similarity index 100%
rename from support/scripts/checkpackagelib/lib.py
rename to tools/checkpackagelib/lib.py
diff --git a/support/scripts/checkpackagelib/lib_config.py b/tools/checkpackagelib/lib_config.py
similarity index 100%
rename from support/scripts/checkpackagelib/lib_config.py
rename to tools/checkpackagelib/lib_config.py
diff --git a/support/scripts/checkpackagelib/lib_hash.py b/tools/checkpackagelib/lib_hash.py
similarity index 100%
rename from support/scripts/checkpackagelib/lib_hash.py
rename to tools/checkpackagelib/lib_hash.py
diff --git a/support/scripts/checkpackagelib/lib_mk.py b/tools/checkpackagelib/lib_mk.py
similarity index 100%
rename from support/scripts/checkpackagelib/lib_mk.py
rename to tools/checkpackagelib/lib_mk.py
diff --git a/support/scripts/checkpackagelib/lib_patch.py b/tools/checkpackagelib/lib_patch.py
similarity index 100%
rename from support/scripts/checkpackagelib/lib_patch.py
rename to tools/checkpackagelib/lib_patch.py
diff --git a/support/scripts/checkpackagelib/readme.txt b/tools/checkpackagelib/readme.txt
similarity index 91%
rename from support/scripts/checkpackagelib/readme.txt
rename to tools/checkpackagelib/readme.txt
index 0444d17992..8012c72e39 100644
--- a/support/scripts/checkpackagelib/readme.txt
+++ b/tools/checkpackagelib/readme.txt
@@ -57,19 +57,19 @@ Some hints when changing this code:
Usage examples:
- to get a list of check functions that would be called without actually
calling them you can use the --dry-run option:
-$ support/scripts/check-package --dry-run package/yourfavorite/*
+$ tools/check-package --dry-run package/yourfavorite/*
- when you just added a new check function, e.g. Something, check how it behaves
for all current packages:
-$ support/scripts/check-package --include-only Something $(find package -type f)
+$ tools/check-package --include-only Something $(find package -type f)
- the effective processing time (when the .pyc were already generated and all
files to be processed are cached in the RAM) should stay in the order of few
seconds:
-$ support/scripts/check-package $(find package -type f) >/dev/null ; \
- time support/scripts/check-package $(find package -type f) >/dev/null
+$ tools/check-package $(find package -type f) >/dev/null ; \
+ time tools/check-package $(find package -type f) >/dev/null
- vim users can navigate the warnings (most editors probably have similar
function) since warnings are generated in the form 'path/file:line: warning':
$ find package/ -name 'Config.*' > filelist && vim -c \
- 'set makeprg=support/scripts/check-package\ $(cat\ filelist)' -c make -c copen
+ 'set makeprg=tools/check-package\ $(cat\ filelist)' -c make -c copen
--
2.11.0
^ permalink raw reply related [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 4/4 v2] tools: move check-package out of support/scripts/
2017-06-18 14:01 ` [Buildroot] [PATCH 4/4 v2] tools: move check-package " Yann E. MORIN
@ 2017-06-19 1:13 ` Ricardo Martincoski
2017-06-19 18:46 ` Yann E. MORIN
2017-06-19 3:22 ` Ricardo Martincoski
1 sibling, 1 reply; 25+ messages in thread
From: Ricardo Martincoski @ 2017-06-19 1:13 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote:
[snip]
> rename {support/scripts => tools}/checkpackagelib/base.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_config.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_hash.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_mk.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_patch.py (100%)
> rename {support/scripts => tools}/checkpackagelib/readme.txt (91%)
nit: there are 6 more comments using the old path.
-# See support/scripts/checkpackagelib/readme.txt before editing this file.
+# See tools/checkpackagelib/readme.txt before editing this file.
It could be fixed when applying using this sed:
sed -e 's,support/scripts/,tools/,g' -i tools/checkpackagelib/*.py
Another option is to remove the 6 comments if you think they are too repetitive.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 4/4 v2] tools: move check-package out of support/scripts/
2017-06-19 1:13 ` Ricardo Martincoski
@ 2017-06-19 18:46 ` Yann E. MORIN
0 siblings, 0 replies; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-19 18:46 UTC (permalink / raw)
To: buildroot
Ricardo, All,
On 2017-06-18 22:13 -0300, Ricardo Martincoski spake thusly:
> On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote:
> [snip]
> > rename {support/scripts => tools}/checkpackagelib/base.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_config.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_hash.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_mk.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_patch.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/readme.txt (91%)
>
> nit: there are 6 more comments using the old path.
Damn, right.
> -# See support/scripts/checkpackagelib/readme.txt before editing this file.
> +# See tools/checkpackagelib/readme.txt before editing this file.
>
> It could be fixed when applying using this sed:
> sed -e 's,support/scripts/,tools/,g' -i tools/checkpackagelib/*.py
>
> Another option is to remove the 6 comments if you think they are too repetitive.
Fixed.
Thanks for the review! :-)
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 4/4 v2] tools: move check-package out of support/scripts/
2017-06-18 14:01 ` [Buildroot] [PATCH 4/4 v2] tools: move check-package " Yann E. MORIN
2017-06-19 1:13 ` Ricardo Martincoski
@ 2017-06-19 3:22 ` Ricardo Martincoski
2017-06-19 18:48 ` Yann E. MORIN
1 sibling, 1 reply; 25+ messages in thread
From: Ricardo Martincoski @ 2017-06-19 3:22 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote:
> ---
> support/scripts/check-package | 145 +--------------------
> support/scripts/pkg-stats | 2 +-
> tools/00-README.txt | 4 +
> tools/check-package | 144 ++++++++++++++++++++
> .../scripts => tools}/checkpackagelib/__init__.py | 0
> {support/scripts => tools}/checkpackagelib/base.py | 0
> {support/scripts => tools}/checkpackagelib/lib.py | 0
> .../checkpackagelib/lib_config.py | 0
> .../scripts => tools}/checkpackagelib/lib_hash.py | 0
> .../scripts => tools}/checkpackagelib/lib_mk.py | 0
> .../scripts => tools}/checkpackagelib/lib_patch.py | 0
> .../scripts => tools}/checkpackagelib/readme.txt | 10 +-
> 12 files changed, 155 insertions(+), 150 deletions(-)
> mode change 100755 => 120000 support/scripts/check-package
> create mode 100755 tools/check-package
> rename {support/scripts => tools}/checkpackagelib/__init__.py (100%)
> rename {support/scripts => tools}/checkpackagelib/base.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_config.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_hash.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_mk.py (100%)
> rename {support/scripts => tools}/checkpackagelib/lib_patch.py (100%)
> rename {support/scripts => tools}/checkpackagelib/readme.txt (91%)
I forgot earlier...
Also the entry in the DEVELOPERS file must be updated
F: support/scripts/check*package*
Probably if I didn't used '*' it would be easier to find.
I didn't tested but I assume we could use
F: tools/check-package
F: tools/checkpackagelib/
Regards,
Ricardo
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 4/4 v2] tools: move check-package out of support/scripts/
2017-06-19 3:22 ` Ricardo Martincoski
@ 2017-06-19 18:48 ` Yann E. MORIN
2017-06-19 20:27 ` Ricardo Martincoski
0 siblings, 1 reply; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-19 18:48 UTC (permalink / raw)
To: buildroot
Ricardo, All,
On 2017-06-19 00:22 -0300, Ricardo Martincoski spake thusly:
> On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote:
>
> > ---
> > support/scripts/check-package | 145 +--------------------
> > support/scripts/pkg-stats | 2 +-
> > tools/00-README.txt | 4 +
> > tools/check-package | 144 ++++++++++++++++++++
> > .../scripts => tools}/checkpackagelib/__init__.py | 0
> > {support/scripts => tools}/checkpackagelib/base.py | 0
> > {support/scripts => tools}/checkpackagelib/lib.py | 0
> > .../checkpackagelib/lib_config.py | 0
> > .../scripts => tools}/checkpackagelib/lib_hash.py | 0
> > .../scripts => tools}/checkpackagelib/lib_mk.py | 0
> > .../scripts => tools}/checkpackagelib/lib_patch.py | 0
> > .../scripts => tools}/checkpackagelib/readme.txt | 10 +-
> > 12 files changed, 155 insertions(+), 150 deletions(-)
> > mode change 100755 => 120000 support/scripts/check-package
> > create mode 100755 tools/check-package
> > rename {support/scripts => tools}/checkpackagelib/__init__.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/base.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_config.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_hash.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_mk.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/lib_patch.py (100%)
> > rename {support/scripts => tools}/checkpackagelib/readme.txt (91%)
>
> I forgot earlier...
>
> Also the entry in the DEVELOPERS file must be updated
> F: support/scripts/check*package*
>
> Probably if I didn't used '*' it would be easier to find.
> I didn't tested but I assume we could use
> F: tools/check-package
> F: tools/checkpackagelib/
I'll fix as you suggest, to drop the midle wildcard.
Also, you happen to have two entries in the DEVELOPPERS file, with two
different emails:
1368 N:? Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
1369 F:? package/atop/
and
1374 N:? Ricardo Martincoski <ricardo.martincoski@gmail.com>
1375 F:? support/scripts/check*package*
Is that expected?
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 4/4 v2] tools: move check-package out of support/scripts/
2017-06-19 18:48 ` Yann E. MORIN
@ 2017-06-19 20:27 ` Ricardo Martincoski
0 siblings, 0 replies; 25+ messages in thread
From: Ricardo Martincoski @ 2017-06-19 20:27 UTC (permalink / raw)
To: buildroot
Hello,
>> On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote:
> Also, you happen to have two entries in the DEVELOPPERS file, with two
> different emails:
>
> 1368 N:? Ricardo Martincoski <ricardo.martincoski@datacom.ind.br>
> 1369 F:? package/atop/
>
> and
>
> 1374 N:? Ricardo Martincoski <ricardo.martincoski@gmail.com>
> 1375 F:? support/scripts/check*package*
>
> Is that expected?
Yes, professional and personal entries.
Regards,
Ricardo
^ permalink raw reply [flat|nested] 25+ messages in thread
* [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools
2017-06-18 14:01 [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Yann E. MORIN
` (3 preceding siblings ...)
2017-06-18 14:01 ` [Buildroot] [PATCH 4/4 v2] tools: move check-package " Yann E. MORIN
@ 2017-06-18 14:15 ` Thomas Petazzoni
2017-06-18 14:23 ` Yann E. MORIN
4 siblings, 1 reply; 25+ messages in thread
From: Thomas Petazzoni @ 2017-06-18 14:15 UTC (permalink / raw)
To: buildroot
Hello,
On Sun, 18 Jun 2017 16:01:40 +0200, Yann E. MORIN wrote:
> Currently, we provide a few user-facing utilities, like get-developers,
> in support/scripts/ . But this directory also contains internal scripts
> that a user should not be directly concerned with. Besides, it is
> two-level deep in the hierarchy, which is not really nice.
>
> So, we introduce tools/ as a top-level directory, with the goal to store
> and expose all user-facing utilities, while keeping our internal scripts
> in support/scripts/ .
>
> The first patch in the series introduces a new utility, brmake, a
> wrapper around make that stores all the output to a log file and
> displays only the >>> lines.
>
> The second and subsequent patches each move a few user-facing utilities
> out of support/scripts/ and into tools/ .
I am wondering if there is a very clear cut boundary between
user-facing tools and non-user facing tools.
For example, graph-build-time or graph-depends are internally used by
the Buildroot Makefile, but may also be used directly. The size-stats
tool is also used internally by the Buildroot Makefile, but
size-stats-compare is meant to be used directly.
What about support/scripts/genimage.sh ?
Best regards,
Thomas
--
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools
2017-06-18 14:15 ` [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools Thomas Petazzoni
@ 2017-06-18 14:23 ` Yann E. MORIN
2017-06-21 22:04 ` Arnout Vandecappelle
0 siblings, 1 reply; 25+ messages in thread
From: Yann E. MORIN @ 2017-06-18 14:23 UTC (permalink / raw)
To: buildroot
Thomas, All,
On 2017-06-18 16:15 +0200, Thomas Petazzoni spake thusly:
> On Sun, 18 Jun 2017 16:01:40 +0200, Yann E. MORIN wrote:
> > Currently, we provide a few user-facing utilities, like get-developers,
> > in support/scripts/ . But this directory also contains internal scripts
> > that a user should not be directly concerned with. Besides, it is
> > two-level deep in the hierarchy, which is not really nice.
> >
> > So, we introduce tools/ as a top-level directory, with the goal to store
> > and expose all user-facing utilities, while keeping our internal scripts
> > in support/scripts/ .
> >
> > The first patch in the series introduces a new utility, brmake, a
> > wrapper around make that stores all the output to a log file and
> > displays only the >>> lines.
> >
> > The second and subsequent patches each move a few user-facing utilities
> > out of support/scripts/ and into tools/ .
>
> I am wondering if there is a very clear cut boundary between
> user-facing tools and non-user facing tools.
Yes, the frontier is not trivial.
I'd say that if a tool is meant to be called by the user, then it
belongs to tools/. If it is not meant to be alled by a user (even if it
could be so) then it belongs to support/scripts/.
> For example, graph-build-time or graph-depends are internally used by
> the Buildroot Makefile, but may also be used directly.
The primary use for those scripts is still an internal use. They are
supposed to be called from our infra.
> The size-stats
> tool is also used internally by the Buildroot Makefile, but
> size-stats-compare is meant to be used directly.
Then the former stays in support/scripts/ while the latter should move
to tools/.
> What about support/scripts/genimage.sh ?
I would leave it to support/scripts/. It is an internal helper./
Besides, it needs a few of our variables (TARGET_DIR and BINARIES_DIR),
so I see it as internal.
Of course, I might not have moved all that can/should be moved. This
series serves as a base for further discussion.
See the initial submission:
[PATCH] contrib: add a directory to store "contrib" stuff
http://lists.busybox.net/pipermail/buildroot/2016-October/174238.html
Regards,
Yann E. MORIN.
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 223 225 172 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
^ permalink raw reply [flat|nested] 25+ messages in thread* [Buildroot] [PATCH 0/4 v2] tools: add a directory to contain usefull user-facing tools
2017-06-18 14:23 ` Yann E. MORIN
@ 2017-06-21 22:04 ` Arnout Vandecappelle
0 siblings, 0 replies; 25+ messages in thread
From: Arnout Vandecappelle @ 2017-06-21 22:04 UTC (permalink / raw)
To: buildroot
On 18-06-17 16:23, Yann E. MORIN wrote:
> Thomas, All,
>
> On 2017-06-18 16:15 +0200, Thomas Petazzoni spake thusly:
>> On Sun, 18 Jun 2017 16:01:40 +0200, Yann E. MORIN wrote:
>>> Currently, we provide a few user-facing utilities, like get-developers,
>>> in support/scripts/ . But this directory also contains internal scripts
>>> that a user should not be directly concerned with. Besides, it is
>>> two-level deep in the hierarchy, which is not really nice.
>>>
>>> So, we introduce tools/ as a top-level directory, with the goal to store
>>> and expose all user-facing utilities, while keeping our internal scripts
>>> in support/scripts/ .
>>>
>>> The first patch in the series introduces a new utility, brmake, a
>>> wrapper around make that stores all the output to a log file and
>>> displays only the >>> lines.
>>>
>>> The second and subsequent patches each move a few user-facing utilities
>>> out of support/scripts/ and into tools/ .
>>
>> I am wondering if there is a very clear cut boundary between
>> user-facing tools and non-user facing tools.
>
> Yes, the frontier is not trivial.
>
> I'd say that if a tool is meant to be called by the user, then it
> belongs to tools/. If it is not meant to be alled by a user (even if it
> could be so) then it belongs to support/scripts/.
Sounds good to me!
Just to be clear: the main reason why I liked the idea of this directory is
because it makes it easier for users to discover these scripts.
>> For example, graph-build-time or graph-depends are internally used by
>> the Buildroot Makefile, but may also be used directly.
>
> The primary use for those scripts is still an internal use. They are
> supposed to be called from our infra.
>
>> The size-stats
>> tool is also used internally by the Buildroot Makefile, but
>> size-stats-compare is meant to be used directly.
>
> Then the former stays in support/scripts/ while the latter should move
> to tools/.
>
>> What about support/scripts/genimage.sh ?
>
> I would leave it to support/scripts/. It is an internal helper./
> Besides, it needs a few of our variables (TARGET_DIR and BINARIES_DIR),
> so I see it as internal.
For me, that is a definite indication that it should stay in support/scripts.
Regards,
Arnout
>
> Of course, I might not have moved all that can/should be moved. This
> series serves as a base for further discussion.
>
> See the initial submission:
>
> [PATCH] contrib: add a directory to store "contrib" stuff
> http://lists.busybox.net/pipermail/buildroot/2016-October/174238.html
>
> Regards,
> Yann E. MORIN.
>
>> Best regards,
>>
>> Thomas
>> --
>> Thomas Petazzoni, CTO, Free Electrons
>> Embedded Linux and Kernel engineering
>> http://free-electrons.com
>
--
Arnout Vandecappelle arnout at mind be
Senior Embedded Software Architect +32-16-286500
Essensium/Mind http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
^ permalink raw reply [flat|nested] 25+ messages in thread