* [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file @ 2016-10-16 17:17 Masahiro Yamada 2016-10-16 17:17 ` [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order Masahiro Yamada 2016-10-16 20:15 ` [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file Tom Rini 0 siblings, 2 replies; 10+ messages in thread From: Masahiro Yamada @ 2016-10-16 17:17 UTC (permalink / raw) To: u-boot Supporting shell-style wildcards for the --defconfigs option will be useful to run the moveconfig tool against a specific platform. For example, "uniphier*" in the file passed by --defconfigs option will be expanded to defconfig files that start with "uniphier". This is easier than listing out all the UniPhier defconfig files. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> --- tools/moveconfig.py | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 87e2bb2..67cf5f0 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -127,7 +127,8 @@ Available options standard commit message is used which may need to be edited. -d, --defconfigs - Specify a file containing a list of defconfigs to move + Specify a file containing a list of defconfigs to move. The defconfig + files can be given with shell-style wildcards. -n, --dry-run Perform a trial run that does not make any changes. It is useful to @@ -180,6 +181,7 @@ import copy import difflib import filecmp import fnmatch +import glob import multiprocessing import optparse import os @@ -284,6 +286,17 @@ def get_make_cmd(): sys.exit('GNU Make not found') return ret[0].rstrip() +def get_matched_defconfigs(defconfigs_file): + """Get all the defconfig files that match the patterns in a file.""" + # use a set rather than a list in order to avoid double-counting + defconfigs = set() + for line in open(defconfigs_file): + line = os.path.join('configs', line.strip()) + defconfigs |= set(glob.glob(line)) + defconfigs |= set(glob.glob(line + '_defconfig')) + + return [ defconfig[len('configs') + 1:] for defconfig in defconfigs ] + def get_all_defconfigs(): """Get all the defconfig files under the configs/ directory.""" defconfigs = [] @@ -1204,13 +1217,7 @@ def move_config(configs, options): reference_src_dir = None if options.defconfigs: - defconfigs = [line.strip() for line in open(options.defconfigs)] - for i, defconfig in enumerate(defconfigs): - if not defconfig.endswith('_defconfig'): - defconfigs[i] = defconfig + '_defconfig' - if not os.path.exists(os.path.join('configs', defconfigs[i])): - sys.exit('%s - defconfig does not exist. Stopping.' % - defconfigs[i]) + defconfigs = get_matched_defconfigs(options.defconfigs) else: defconfigs = get_all_defconfigs() -- 1.9.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order 2016-10-16 17:17 [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file Masahiro Yamada @ 2016-10-16 17:17 ` Masahiro Yamada 2016-10-16 20:12 ` Tom Rini 2016-10-16 20:15 ` [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file Tom Rini 1 sibling, 1 reply; 10+ messages in thread From: Masahiro Yamada @ 2016-10-16 17:17 UTC (permalink / raw) To: u-boot This will make the log more readable. The log is not deterministic in case of parallel processing, though. Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> --- tools/moveconfig.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tools/moveconfig.py b/tools/moveconfig.py index 67cf5f0..49cb1e2 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -1221,6 +1221,8 @@ def move_config(configs, options): else: defconfigs = get_all_defconfigs() + defconfigs.sort() + progress = Progress(len(defconfigs)) slots = Slots(configs, options, progress, reference_src_dir) -- 1.9.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order 2016-10-16 17:17 ` [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order Masahiro Yamada @ 2016-10-16 20:12 ` Tom Rini 2016-10-18 16:23 ` Masahiro Yamada 0 siblings, 1 reply; 10+ messages in thread From: Tom Rini @ 2016-10-16 20:12 UTC (permalink / raw) To: u-boot On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote: > This will make the log more readable. The log is not deterministic > in case of parallel processing, though. > > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> I'm honestly ambivalent about this since single threaded is basically never the case, right? -- Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161016/d380d513/attachment.sig> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order 2016-10-16 20:12 ` Tom Rini @ 2016-10-18 16:23 ` Masahiro Yamada 2016-10-18 19:25 ` Tom Rini 0 siblings, 1 reply; 10+ messages in thread From: Masahiro Yamada @ 2016-10-18 16:23 UTC (permalink / raw) To: u-boot Hi Tom, 2016-10-17 5:12 GMT+09:00 Tom Rini <trini@konsulko.com>: > On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote: >> This will make the log more readable. The log is not deterministic >> in case of parallel processing, though. >> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > > I'm honestly ambivalent about this since single threaded is basically > never the case, right? Right. If you want to diff two logs, you need to pass -j1 option. But, my main motivation is not that. You will be able to find the log of boards you are interested in. For example, when some boards are listed in "suspicious results" at the end of conversion, you may want to check the log to see what happened. If the log is sorted *almost* alphabetically, it will be easier to jump there. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order 2016-10-18 16:23 ` Masahiro Yamada @ 2016-10-18 19:25 ` Tom Rini 2016-10-19 5:40 ` Masahiro Yamada 0 siblings, 1 reply; 10+ messages in thread From: Tom Rini @ 2016-10-18 19:25 UTC (permalink / raw) To: u-boot On Wed, Oct 19, 2016 at 01:23:24AM +0900, Masahiro Yamada wrote: > Hi Tom, > > 2016-10-17 5:12 GMT+09:00 Tom Rini <trini@konsulko.com>: > > On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote: > >> This will make the log more readable. The log is not deterministic > >> in case of parallel processing, though. > >> > >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > > > > I'm honestly ambivalent about this since single threaded is basically > > never the case, right? > > > Right. > If you want to diff two logs, you need to pass -j1 option. > > But, my main motivation is not that. > You will be able to find the log of boards you are interested in. > > For example, when some boards are listed in "suspicious results" > at the end of conversion, you may want to check the log > to see what happened. If the log is sorted *almost* alphabetically, > it will be easier to jump there. I'm still ambivalent. I 'less' the log and search doesn't care about order. And I try and fix one broken board at a time since it's often the case that N boards will have the same form of thinko. -- Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161018/c0110126/attachment.sig> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order 2016-10-18 19:25 ` Tom Rini @ 2016-10-19 5:40 ` Masahiro Yamada 0 siblings, 0 replies; 10+ messages in thread From: Masahiro Yamada @ 2016-10-19 5:40 UTC (permalink / raw) To: u-boot 2016-10-19 4:25 GMT+09:00 Tom Rini <trini@konsulko.com>: > On Wed, Oct 19, 2016 at 01:23:24AM +0900, Masahiro Yamada wrote: >> Hi Tom, >> >> 2016-10-17 5:12 GMT+09:00 Tom Rini <trini@konsulko.com>: >> > On Mon, Oct 17, 2016 at 02:17:23AM +0900, Masahiro Yamada wrote: >> >> This will make the log more readable. The log is not deterministic >> >> in case of parallel processing, though. >> >> >> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> >> > >> > I'm honestly ambivalent about this since single threaded is basically >> > never the case, right? >> >> >> Right. >> If you want to diff two logs, you need to pass -j1 option. >> >> But, my main motivation is not that. >> You will be able to find the log of boards you are interested in. >> >> For example, when some boards are listed in "suspicious results" >> at the end of conversion, you may want to check the log >> to see what happened. If the log is sorted *almost* alphabetically, >> it will be easier to jump there. > > I'm still ambivalent. I 'less' the log and search doesn't care about > order. And I try and fix one broken board at a time since it's often > the case that N boards will have the same form of thinko. OK, fine. Please drop this patch. I sent v2 only for 1/2. -- Best Regards Masahiro Yamada ^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file 2016-10-16 17:17 [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file Masahiro Yamada 2016-10-16 17:17 ` [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order Masahiro Yamada @ 2016-10-16 20:15 ` Tom Rini 2016-10-18 16:14 ` Masahiro Yamada 1 sibling, 1 reply; 10+ messages in thread From: Tom Rini @ 2016-10-16 20:15 UTC (permalink / raw) To: u-boot On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote: > Supporting shell-style wildcards for the --defconfigs option will be > useful to run the moveconfig tool against a specific platform. For > example, "uniphier*" in the file passed by --defconfigs option will > be expanded to defconfig files that start with "uniphier". This is > easier than listing out all the UniPhier defconfig files. > > Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > --- > > tools/moveconfig.py | 23 +++++++++++++++-------- > 1 file changed, 15 insertions(+), 8 deletions(-) > > diff --git a/tools/moveconfig.py b/tools/moveconfig.py > index 87e2bb2..67cf5f0 100755 > --- a/tools/moveconfig.py > +++ b/tools/moveconfig.py > @@ -127,7 +127,8 @@ Available options > standard commit message is used which may need to be edited. > > -d, --defconfigs > - Specify a file containing a list of defconfigs to move > + Specify a file containing a list of defconfigs to move. The defconfig > + files can be given with shell-style wildcards. > > -n, --dry-run > Perform a trial run that does not make any changes. It is useful to > @@ -180,6 +181,7 @@ import copy > import difflib > import filecmp > import fnmatch > +import glob > import multiprocessing > import optparse > import os > @@ -284,6 +286,17 @@ def get_make_cmd(): > sys.exit('GNU Make not found') > return ret[0].rstrip() > > +def get_matched_defconfigs(defconfigs_file): > + """Get all the defconfig files that match the patterns in a file.""" > + # use a set rather than a list in order to avoid double-counting > + defconfigs = set() > + for line in open(defconfigs_file): > + line = os.path.join('configs', line.strip()) > + defconfigs |= set(glob.glob(line)) > + defconfigs |= set(glob.glob(line + '_defconfig')) > + > + return [ defconfig[len('configs') + 1:] for defconfig in defconfigs ] > + > def get_all_defconfigs(): > """Get all the defconfig files under the configs/ directory.""" > defconfigs = [] > @@ -1204,13 +1217,7 @@ def move_config(configs, options): > reference_src_dir = None > > if options.defconfigs: > - defconfigs = [line.strip() for line in open(options.defconfigs)] > - for i, defconfig in enumerate(defconfigs): > - if not defconfig.endswith('_defconfig'): > - defconfigs[i] = defconfig + '_defconfig' > - if not os.path.exists(os.path.join('configs', defconfigs[i])): > - sys.exit('%s - defconfig does not exist. Stopping.' % > - defconfigs[i]) > + defconfigs = get_matched_defconfigs(options.defconfigs) I don't see error handling in the case of files (and now globs) not existing now, is this just handled by other parts of the code? Thanks! -- Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161016/c0865ed5/attachment.sig> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file 2016-10-16 20:15 ` [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file Tom Rini @ 2016-10-18 16:14 ` Masahiro Yamada 2016-10-18 19:23 ` Tom Rini 0 siblings, 1 reply; 10+ messages in thread From: Masahiro Yamada @ 2016-10-18 16:14 UTC (permalink / raw) To: u-boot 2016-10-17 5:15 GMT+09:00 Tom Rini <trini@konsulko.com>: > On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote: > >> Supporting shell-style wildcards for the --defconfigs option will be >> useful to run the moveconfig tool against a specific platform. For >> example, "uniphier*" in the file passed by --defconfigs option will >> be expanded to defconfig files that start with "uniphier". This is >> easier than listing out all the UniPhier defconfig files. >> >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> >> --- >> >> tools/moveconfig.py | 23 +++++++++++++++-------- >> 1 file changed, 15 insertions(+), 8 deletions(-) >> >> diff --git a/tools/moveconfig.py b/tools/moveconfig.py >> index 87e2bb2..67cf5f0 100755 >> --- a/tools/moveconfig.py >> +++ b/tools/moveconfig.py >> @@ -127,7 +127,8 @@ Available options >> standard commit message is used which may need to be edited. >> >> -d, --defconfigs >> - Specify a file containing a list of defconfigs to move >> + Specify a file containing a list of defconfigs to move. The defconfig >> + files can be given with shell-style wildcards. >> >> -n, --dry-run >> Perform a trial run that does not make any changes. It is useful to >> @@ -180,6 +181,7 @@ import copy >> import difflib >> import filecmp >> import fnmatch >> +import glob >> import multiprocessing >> import optparse >> import os >> @@ -284,6 +286,17 @@ def get_make_cmd(): >> sys.exit('GNU Make not found') >> return ret[0].rstrip() >> >> +def get_matched_defconfigs(defconfigs_file): >> + """Get all the defconfig files that match the patterns in a file.""" >> + # use a set rather than a list in order to avoid double-counting >> + defconfigs = set() >> + for line in open(defconfigs_file): >> + line = os.path.join('configs', line.strip()) >> + defconfigs |= set(glob.glob(line)) >> + defconfigs |= set(glob.glob(line + '_defconfig')) >> + >> + return [ defconfig[len('configs') + 1:] for defconfig in defconfigs ] >> + >> def get_all_defconfigs(): >> """Get all the defconfig files under the configs/ directory.""" >> defconfigs = [] >> @@ -1204,13 +1217,7 @@ def move_config(configs, options): >> reference_src_dir = None >> >> if options.defconfigs: >> - defconfigs = [line.strip() for line in open(options.defconfigs)] >> - for i, defconfig in enumerate(defconfigs): >> - if not defconfig.endswith('_defconfig'): >> - defconfigs[i] = defconfig + '_defconfig' >> - if not os.path.exists(os.path.join('configs', defconfigs[i])): >> - sys.exit('%s - defconfig does not exist. Stopping.' % >> - defconfigs[i]) >> + defconfigs = get_matched_defconfigs(options.defconfigs) > > I don't see error handling in the case of files (and now globs) not > existing now, is this just handled by other parts of the code? Thanks! No, it will be removed. With this patch, each line will be a pattern that may match multiple defconfigs, or that may match nothing. Is it important to check users' possible typos? ^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file 2016-10-18 16:14 ` Masahiro Yamada @ 2016-10-18 19:23 ` Tom Rini 2016-10-18 19:25 ` Joe Hershberger 0 siblings, 1 reply; 10+ messages in thread From: Tom Rini @ 2016-10-18 19:23 UTC (permalink / raw) To: u-boot On Wed, Oct 19, 2016 at 01:14:58AM +0900, Masahiro Yamada wrote: > 2016-10-17 5:15 GMT+09:00 Tom Rini <trini@konsulko.com>: > > On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote: > > > >> Supporting shell-style wildcards for the --defconfigs option will be > >> useful to run the moveconfig tool against a specific platform. For > >> example, "uniphier*" in the file passed by --defconfigs option will > >> be expanded to defconfig files that start with "uniphier". This is > >> easier than listing out all the UniPhier defconfig files. > >> > >> Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> > >> --- > >> > >> tools/moveconfig.py | 23 +++++++++++++++-------- > >> 1 file changed, 15 insertions(+), 8 deletions(-) > >> > >> diff --git a/tools/moveconfig.py b/tools/moveconfig.py > >> index 87e2bb2..67cf5f0 100755 > >> --- a/tools/moveconfig.py > >> +++ b/tools/moveconfig.py > >> @@ -127,7 +127,8 @@ Available options > >> standard commit message is used which may need to be edited. > >> > >> -d, --defconfigs > >> - Specify a file containing a list of defconfigs to move > >> + Specify a file containing a list of defconfigs to move. The defconfig > >> + files can be given with shell-style wildcards. > >> > >> -n, --dry-run > >> Perform a trial run that does not make any changes. It is useful to > >> @@ -180,6 +181,7 @@ import copy > >> import difflib > >> import filecmp > >> import fnmatch > >> +import glob > >> import multiprocessing > >> import optparse > >> import os > >> @@ -284,6 +286,17 @@ def get_make_cmd(): > >> sys.exit('GNU Make not found') > >> return ret[0].rstrip() > >> > >> +def get_matched_defconfigs(defconfigs_file): > >> + """Get all the defconfig files that match the patterns in a file.""" > >> + # use a set rather than a list in order to avoid double-counting > >> + defconfigs = set() > >> + for line in open(defconfigs_file): > >> + line = os.path.join('configs', line.strip()) > >> + defconfigs |= set(glob.glob(line)) > >> + defconfigs |= set(glob.glob(line + '_defconfig')) > >> + > >> + return [ defconfig[len('configs') + 1:] for defconfig in defconfigs ] > >> + > >> def get_all_defconfigs(): > >> """Get all the defconfig files under the configs/ directory.""" > >> defconfigs = [] > >> @@ -1204,13 +1217,7 @@ def move_config(configs, options): > >> reference_src_dir = None > >> > >> if options.defconfigs: > >> - defconfigs = [line.strip() for line in open(options.defconfigs)] > >> - for i, defconfig in enumerate(defconfigs): > >> - if not defconfig.endswith('_defconfig'): > >> - defconfigs[i] = defconfig + '_defconfig' > >> - if not os.path.exists(os.path.join('configs', defconfigs[i])): > >> - sys.exit('%s - defconfig does not exist. Stopping.' % > >> - defconfigs[i]) > >> + defconfigs = get_matched_defconfigs(options.defconfigs) > > > > I don't see error handling in the case of files (and now globs) not > > existing now, is this just handled by other parts of the code? Thanks! > > > No, it will be removed. > > With this patch, each line will be a pattern > that may match multiple defconfigs, > or that may match nothing. Ah, OK. > Is it important to check users' possible typos? Well, will a list of invalid config names say something useful, or fail silently or fail in unexpected ways? -- Tom -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: Digital signature URL: <http://lists.denx.de/pipermail/u-boot/attachments/20161018/ad00a201/attachment.sig> ^ permalink raw reply [flat|nested] 10+ messages in thread
* [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file 2016-10-18 19:23 ` Tom Rini @ 2016-10-18 19:25 ` Joe Hershberger 0 siblings, 0 replies; 10+ messages in thread From: Joe Hershberger @ 2016-10-18 19:25 UTC (permalink / raw) To: u-boot On Tue, Oct 18, 2016 at 2:23 PM, Tom Rini <trini@konsulko.com> wrote: > On Wed, Oct 19, 2016 at 01:14:58AM +0900, Masahiro Yamada wrote: >> 2016-10-17 5:15 GMT+09:00 Tom Rini <trini@konsulko.com>: >> > On Mon, Oct 17, 2016 at 02:17:22AM +0900, Masahiro Yamada wrote: >> > I don't see error handling in the case of files (and now globs) not >> > existing now, is this just handled by other parts of the code? Thanks! >> >> >> No, it will be removed. >> >> With this patch, each line will be a pattern >> that may match multiple defconfigs, >> or that may match nothing. > > Ah, OK. > >> Is it important to check users' possible typos? > > Well, will a list of invalid config names say something useful, or fail > silently or fail in unexpected ways? It seems like it would be good to warn if a glob (or otherwise) results in an empty set, because that it probably not what was intended. -Joe ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2016-10-19 5:40 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2016-10-16 17:17 [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file Masahiro Yamada 2016-10-16 17:17 ` [U-Boot] [PATCH 2/2] tools: moveconfig: process defconfig files in alphabetical order Masahiro Yamada 2016-10-16 20:12 ` Tom Rini 2016-10-18 16:23 ` Masahiro Yamada 2016-10-18 19:25 ` Tom Rini 2016-10-19 5:40 ` Masahiro Yamada 2016-10-16 20:15 ` [U-Boot] [PATCH 1/2] tools: moveconfig: support wildcards in --defconfigs file Tom Rini 2016-10-18 16:14 ` Masahiro Yamada 2016-10-18 19:23 ` Tom Rini 2016-10-18 19:25 ` Joe Hershberger
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox