* [PATCH 2/5] moveconfig: Read the database in a separate function
2021-12-18 15:09 [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature Simon Glass
@ 2021-12-18 15:09 ` Simon Glass
2022-01-25 15:57 ` Tom Rini
2021-12-18 15:09 ` [PATCH 3/5] moveconfig: Sort the options Simon Glass
` (3 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: Simon Glass @ 2021-12-18 15:09 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Simon Glass, Trevor Woerner
Move this code out into a function so it can be used elsewhere.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/moveconfig.py | 78 ++++++++++++++++++++++++++++-----------------
1 file changed, 49 insertions(+), 29 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 8d059be039d..4ad892e2c0c 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1339,6 +1339,54 @@ IMPLY_FLAGS = {
'Allow Kconfig options outside arch/ and /board/ to imply'],
};
+
+def read_database():
+ """Read in the config database
+
+ Returns:
+ tuple:
+ set of all config options seen (each a str)
+ set of all defconfigs seen (each a str)
+ dict of configs for each defconfig:
+ key: defconfig name, e.g. "MPC8548CDS_legacy_defconfig"
+ value: dict:
+ key: CONFIG option
+ value: Value of option
+ dict of defconfigs for each config:
+ key: CONFIG option
+ value: set of boards using that option
+
+ """
+ configs = {}
+
+ # key is defconfig name, value is dict of (CONFIG_xxx, value)
+ config_db = {}
+
+ # Set of all config options we have seen
+ all_configs = set()
+
+ # Set of all defconfigs we have seen
+ all_defconfigs = set()
+
+ defconfig_db = collections.defaultdict(set)
+ with open(CONFIG_DATABASE) as fd:
+ for line in fd.readlines():
+ line = line.rstrip()
+ if not line: # Separator between defconfigs
+ config_db[defconfig] = configs
+ all_defconfigs.add(defconfig)
+ configs = {}
+ elif line[0] == ' ': # CONFIG line
+ config, value = line.strip().split('=', 1)
+ configs[config] = value
+ defconfig_db[config].add(defconfig)
+ all_configs.add(config)
+ else: # New defconfig
+ defconfig = line
+
+ return all_configs, all_defconfigs, config_db, defconfig_db
+
+
def do_imply_config(config_list, add_imply, imply_flags, skip_added,
check_kconfig=True, find_superset=False):
"""Find CONFIG options which imply those in the list
@@ -1385,35 +1433,7 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
if add_imply and add_imply != 'all':
add_imply = add_imply.split(',')
- # key is defconfig name, value is dict of (CONFIG_xxx, value)
- config_db = {}
-
- # Holds a dict containing the set of defconfigs that contain each config
- # key is config, value is set of defconfigs using that config
- defconfig_db = collections.defaultdict(set)
-
- # Set of all config options we have seen
- all_configs = set()
-
- # Set of all defconfigs we have seen
- all_defconfigs = set()
-
- # Read in the database
- configs = {}
- with open(CONFIG_DATABASE) as fd:
- for line in fd.readlines():
- line = line.rstrip()
- if not line: # Separator between defconfigs
- config_db[defconfig] = configs
- all_defconfigs.add(defconfig)
- configs = {}
- elif line[0] == ' ': # CONFIG line
- config, value = line.strip().split('=', 1)
- configs[config] = value
- defconfig_db[config].add(defconfig)
- all_configs.add(config)
- else: # New defconfig
- defconfig = line
+ all_configs, all_defconfigs, config_db, defconfig_db = read_database()
# Work through each target config option in turn, independently
for config in config_list:
--
2.34.1.173.g76aa8bc2d0-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 3/5] moveconfig: Sort the options
2021-12-18 15:09 [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature Simon Glass
2021-12-18 15:09 ` [PATCH 2/5] moveconfig: Read the database in a separate function Simon Glass
@ 2021-12-18 15:09 ` Simon Glass
2022-01-25 15:57 ` Tom Rini
2021-12-18 15:09 ` [PATCH 4/5] moveconfig: Allow adding unit tests Simon Glass
` (2 subsequent siblings)
4 siblings, 1 reply; 10+ messages in thread
From: Simon Glass @ 2021-12-18 15:09 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Simon Glass, Trevor Woerner
Put the options in sorted order by their short name so it is easier to
find an option.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/moveconfig.py | 22 ++++++++++++----------
1 file changed, 12 insertions(+), 10 deletions(-)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 4ad892e2c0c..369589727cc 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1591,26 +1591,28 @@ def main():
help='a file containing a list of defconfigs to move, '
"one per line (for example 'snow_defconfig') "
"or '-' to read from stdin")
- parser.add_option('-i', '--imply', action='store_true', default=False,
- help='find options which imply others')
- parser.add_option('-I', '--imply-flags', type='string', default='',
- help="control the -i option ('help' for help")
- parser.add_option('-n', '--dry-run', action='store_true', default=False,
- help='perform a trial run (show log with no changes)')
parser.add_option('-e', '--exit-on-error', action='store_true',
default=False,
help='exit immediately on any error')
- parser.add_option('-s', '--force-sync', action='store_true', default=False,
- help='force sync by savedefconfig')
- parser.add_option('-S', '--spl', action='store_true', default=False,
- help='parse config options defined for SPL build')
parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
action='store_true', default=False,
help='only cleanup the headers')
+ parser.add_option('-i', '--imply', action='store_true', default=False,
+ help='find options which imply others')
+ parser.add_option('-I', '--imply-flags', type='string', default='',
+ help="control the -i option ('help' for help")
parser.add_option('-j', '--jobs', type='int', default=cpu_count,
help='the number of jobs to run simultaneously')
+ parser.add_option('-n', '--dry-run', action='store_true', default=False,
+ help='perform a trial run (show log with no changes)')
parser.add_option('-r', '--git-ref', type='string',
help='the git ref to clone for building the autoconf.mk')
+ parser.add_option('-s', '--force-sync', action='store_true', default=False,
+ help='force sync by savedefconfig')
+ parser.add_option('-S', '--spl', action='store_true', default=False,
+ help='parse config options defined for SPL build')
+ parser.add_option('-t', '--test', action='store_true', default=False,
+ help='run unit tests')
parser.add_option('-y', '--yes', action='store_true', default=False,
help="respond 'yes' to any prompts")
parser.add_option('-v', '--verbose', action='store_true', default=False,
--
2.34.1.173.g76aa8bc2d0-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 4/5] moveconfig: Allow adding unit tests
2021-12-18 15:09 [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature Simon Glass
2021-12-18 15:09 ` [PATCH 2/5] moveconfig: Read the database in a separate function Simon Glass
2021-12-18 15:09 ` [PATCH 3/5] moveconfig: Sort the options Simon Glass
@ 2021-12-18 15:09 ` Simon Glass
2022-01-25 15:57 ` Tom Rini
2021-12-18 15:09 ` [PATCH 5/5] moveconfig: Allow querying board configuration Simon Glass
2022-01-25 15:57 ` [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature Tom Rini
4 siblings, 1 reply; 10+ messages in thread
From: Simon Glass @ 2021-12-18 15:09 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Simon Glass, Trevor Woerner
Add a -t option to run unit tests in this program. So far, there is none.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
tools/moveconfig.py | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 369589727cc..71a7736ca63 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -14,6 +14,7 @@ import asteval
import collections
import copy
import difflib
+import doctest
import filecmp
import fnmatch
import glob
@@ -28,6 +29,7 @@ import sys
import tempfile
import threading
import time
+import unittest
from buildman import bsettings
from buildman import kconfiglib
@@ -1621,6 +1623,13 @@ def main():
(options, configs) = parser.parse_args()
+ if options.test:
+ sys.argv = [sys.argv[0]]
+ fail, count = doctest.testmod()
+ if fail:
+ return 1
+ unittest.main()
+
if len(configs) == 0 and not any((options.force_sync, options.build_db,
options.imply)):
parser.print_usage()
--
2.34.1.173.g76aa8bc2d0-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH 5/5] moveconfig: Allow querying board configuration
2021-12-18 15:09 [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature Simon Glass
` (2 preceding siblings ...)
2021-12-18 15:09 ` [PATCH 4/5] moveconfig: Allow adding unit tests Simon Glass
@ 2021-12-18 15:09 ` Simon Glass
2022-01-25 15:57 ` Tom Rini
2022-01-25 15:57 ` [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature Tom Rini
4 siblings, 1 reply; 10+ messages in thread
From: Simon Glass @ 2021-12-18 15:09 UTC (permalink / raw)
To: U-Boot Mailing List; +Cc: Simon Glass, Trevor Woerner
It is useful to be able to find out which boards define a particular
option, or combination of options. This is not as easy as grepping the
defconfig files since many options are implied by others.
Add a -f option to the moveconfig tool to permit this. Update the
documentation to cover this, including a better title for the doc page.
Signed-off-by: Simon Glass <sjg@chromium.org>
---
doc/develop/moveconfig.rst | 25 ++++++++++-
tools/moveconfig.py | 86 ++++++++++++++++++++++++++++++++++++--
2 files changed, 105 insertions(+), 6 deletions(-)
diff --git a/doc/develop/moveconfig.rst b/doc/develop/moveconfig.rst
index dcd4d927e40..2f53ea52b71 100644
--- a/doc/develop/moveconfig.rst
+++ b/doc/develop/moveconfig.rst
@@ -1,7 +1,7 @@
.. SPDX-License-Identifier: GPL-2.0+
-moveconfig
-==========
+moveconfig - Migrating and querying CONFIG options
+==================================================
Since Kconfig was introduced to U-Boot, we have worked on moving
config options from headers to Kconfig (defconfig).
@@ -129,6 +129,24 @@ To process CONFIG_CMD_FPGAD only for a subset of configs based on path match::
./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d -
+Finding boards with particular CONFIG combinations
+--------------------------------------------------
+
+You can use `moveconfig.py` to figure out which boards have a CONFIG enabled, or
+which do not. To use it, first build a database::
+
+ ./tools/moveconfig.py -b
+
+Then you can run queries using the `-f` flag followed by a list of CONFIG terms.
+Each term is CONFIG name, with or without a tilde (~) prefix. The tool searches
+for boards which match the CONFIG name, or do not match if tilde is used. For
+example, to find boards which enabled CONFIG_SCSI but not CONFIG_BLK::
+
+ tools/moveconfig.py -f SCSI ~BLK
+ 3 matches
+ pg_wcom_seli8_defconfig highbank_defconfig pg_wcom_expu1_defconfig
+
+
Finding implied CONFIGs
-----------------------
@@ -235,6 +253,9 @@ Available options
Specify a file containing a list of defconfigs to move. The defconfig
files can be given with shell-style wildcards. Use '-' to read from stdin.
+ -f, --find
+ Find boards with a given config combination
+
-n, --dry-run
Perform a trial run that does not make any changes. It is useful to
see what is going to happen before one actually runs it.
diff --git a/tools/moveconfig.py b/tools/moveconfig.py
index 71a7736ca63..a86c07caa6e 100755
--- a/tools/moveconfig.py
+++ b/tools/moveconfig.py
@@ -1569,6 +1569,79 @@ def do_imply_config(config_list, add_imply, imply_flags, skip_added,
add_imply_rule(config[CONFIG_LEN:], fname, linenum)
+def do_find_config(config_list):
+ """Find boards with a given combination of CONFIGs
+
+ Params:
+ config_list: List of CONFIG options to check (each a string consisting
+ of a config option, with or without a CONFIG_ prefix. If an option
+ is preceded by a tilde (~) then it must be false, otherwise it must
+ be true)
+ """
+ all_configs, all_defconfigs, config_db, defconfig_db = read_database()
+
+ # Get the whitelist
+ with open('scripts/config_whitelist.txt') as inf:
+ adhoc_configs = set(inf.read().splitlines())
+
+ # Start with all defconfigs
+ out = all_defconfigs
+
+ # Work through each config in turn
+ adhoc = []
+ for item in config_list:
+ # Get the real config name and whether we want this config or not
+ cfg = item
+ want = True
+ if cfg[0] == '~':
+ want = False
+ cfg = cfg[1:]
+
+ if cfg in adhoc_configs:
+ adhoc.append(cfg)
+ continue
+
+ # Search everything that is still in the running. If it has a config
+ # that we want, or doesn't have one that we don't, add it into the
+ # running for the next stage
+ in_list = out
+ out = set()
+ for defc in in_list:
+ has_cfg = cfg in config_db[defc]
+ if has_cfg == want:
+ out.add(defc)
+ if adhoc:
+ print(f"Error: Not in Kconfig: %s" % ' '.join(adhoc))
+ else:
+ print(f'{len(out)} matches')
+ print(' '.join(out))
+
+
+def prefix_config(cfg):
+ """Prefix a config with CONFIG_ if needed
+
+ This handles ~ operator, which indicates that the CONFIG should be disabled
+
+ >>> prefix_config('FRED')
+ 'CONFIG_FRED'
+ >>> prefix_config('CONFIG_FRED')
+ 'CONFIG_FRED'
+ >>> prefix_config('~FRED')
+ '~CONFIG_FRED'
+ >>> prefix_config('~CONFIG_FRED')
+ '~CONFIG_FRED'
+ >>> prefix_config('A123')
+ 'CONFIG_A123'
+ """
+ op = ''
+ if cfg[0] == '~':
+ op = cfg[0]
+ cfg = cfg[1:]
+ if not cfg.startswith('CONFIG_'):
+ cfg = 'CONFIG_' + cfg
+ return op + cfg
+
+
def main():
try:
cpu_count = multiprocessing.cpu_count()
@@ -1596,6 +1669,8 @@ def main():
parser.add_option('-e', '--exit-on-error', action='store_true',
default=False,
help='exit immediately on any error')
+ parser.add_option('-f', '--find', action='store_true', default=False,
+ help='Find boards with a given config combination')
parser.add_option('-H', '--headers-only', dest='cleanup_headers_only',
action='store_true', default=False,
help='only cleanup the headers')
@@ -1631,13 +1706,12 @@ def main():
unittest.main()
if len(configs) == 0 and not any((options.force_sync, options.build_db,
- options.imply)):
+ options.imply, options.find)):
parser.print_usage()
sys.exit(1)
# prefix the option name with CONFIG_ if missing
- configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config
- for config in configs ]
+ configs = [prefix_config(cfg) for cfg in configs]
check_top_directory()
@@ -1663,6 +1737,10 @@ def main():
options.skip_added)
return
+ if options.find:
+ do_find_config(configs)
+ return
+
config_db = {}
db_queue = queue.Queue()
t = DatabaseThread(config_db, db_queue)
@@ -1705,4 +1783,4 @@ def main():
fd.write('\n')
if __name__ == '__main__':
- main()
+ sys.exit(main())
--
2.34.1.173.g76aa8bc2d0-goog
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH 5/5] moveconfig: Allow querying board configuration
2021-12-18 15:09 ` [PATCH 5/5] moveconfig: Allow querying board configuration Simon Glass
@ 2022-01-25 15:57 ` Tom Rini
0 siblings, 0 replies; 10+ messages in thread
From: Tom Rini @ 2022-01-25 15:57 UTC (permalink / raw)
To: Simon Glass; +Cc: U-Boot Mailing List, Trevor Woerner
[-- Attachment #1: Type: text/plain, Size: 521 bytes --]
On Sat, Dec 18, 2021 at 08:09:46AM -0700, Simon Glass wrote:
> It is useful to be able to find out which boards define a particular
> option, or combination of options. This is not as easy as grepping the
> defconfig files since many options are implied by others.
>
> Add a -f option to the moveconfig tool to permit this. Update the
> documentation to cover this, including a better title for the doc page.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
Applied to u-boot/master, thanks!
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature
2021-12-18 15:09 [PATCH 1/5] moveconfig: Correct operation of the 'imply' feature Simon Glass
` (3 preceding siblings ...)
2021-12-18 15:09 ` [PATCH 5/5] moveconfig: Allow querying board configuration Simon Glass
@ 2022-01-25 15:57 ` Tom Rini
4 siblings, 0 replies; 10+ messages in thread
From: Tom Rini @ 2022-01-25 15:57 UTC (permalink / raw)
To: Simon Glass; +Cc: U-Boot Mailing List, Trevor Woerner
[-- Attachment #1: Type: text/plain, Size: 279 bytes --]
On Sat, Dec 18, 2021 at 08:09:42AM -0700, Simon Glass wrote:
> This doesn't work anymore, since the Kconfig update. The script has no
> tests so we did not notice. Fix it.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
Applied to u-boot/master, thanks!
--
Tom
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread