Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Arnaud Lacombe]

Hi,

On Tue, Feb 1, 2011 at 6:27 PM, Ulf Magnusson <ulfalizer.lkml@gmail.com> wrote:
> Hi,
>
> This is the initial release of Kconfiglib: a Python library for
> scripting, debugging, and extracting information from Kconfig-based
> configuration systems.  It can be used to programmatically generate a
> .config when the '*conf' tools are too inflexible, to quickly find out
> interesting information about a Kconfig configuration such as dependency
> relations between symbols and where undefined symbols are referenced,
> and in applications that need to parse and extract information from
> Kconfig files.
>
Does that mean that when a kconfig-language change will happen, one
will not only must have to be Lex/YaCC and C fluent, but also, perl
and python ? That seem to be a lot of duplication to me.

I'd rather see the backend be changed so that it could be used with
SWIG to generate language bindings. I had WIP in this area, but
changes needed are rather intrusive. That said, the testsuite might be
interesting, I've been thinking about that for quite some time.

 - Arnaud

> For a much longer introduction including multiple examples, see
> arch/kconfig/kconfiglib.py.
>
> Have fun!
>
> Signed-off-by: Ulf Magnusson <ulfalizer.lkml@gmail.com>
> ---
> Convenience links:
>
> Documentation, generated from kconfiglib.py with pydoc -w:
> http://dl.dropbox.com/u/10406197/kconfiglib.html
>
> Examples as separate files:
> http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
>
>
> The patch should be preferably be applied to a recent kernel, i.e. Linus's
> (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
> kconfigtest.py test suite - which compares output character-for-character -
> will indicate failure on older (a few months old) kernels versions even though
> the outputs are functionally equivalent.
>
>  Documentation/kbuild/kconfig-language.txt |    5 +
>  Documentation/kbuild/kconfig.txt          |    8 +
>  README                                    |   13 +
>  scripts/kconfig/Makefile                  |   26 +-
>  scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
>  scripts/kconfig/kconfigtest.py            |  396 +++
>  6 files changed, 4365 insertions(+), 1 deletions(-)
>  create mode 100644 scripts/kconfig/kconfiglib.py
>  create mode 100644 scripts/kconfig/kconfigtest.py
> [...]

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Tue, Feb 01, 2011 at 07:47:14PM -0500, Arnaud Lacombe wrote:
> Hi,
> 
> On Tue, Feb 1, 2011 at 6:27 PM, Ulf Magnusson <ulfalizer.lkml@gmail.com> wrote:
> > Hi,
> >
> > This is the initial release of Kconfiglib: a Python library for
> > scripting, debugging, and extracting information from Kconfig-based
> > configuration systems.  It can be used to programmatically generate a
> > .config when the '*conf' tools are too inflexible, to quickly find out
> > interesting information about a Kconfig configuration such as dependency
> > relations between symbols and where undefined symbols are referenced,
> > and in applications that need to parse and extract information from
> > Kconfig files.
> >
> Does that mean that when a kconfig-language change will happen, one
> will not only must have to be Lex/YaCC and C fluent, but also, perl
> and python ? That seem to be a lot of duplication to me.
> 
> I'd rather see the backend be changed so that it could be used with
> SWIG to generate language bindings. I had WIP in this area, but
> changes needed are rather intrusive. That said, the testsuite might be
> interesting, I've been thinking about that for quite some time.
> 
>  - Arnaud
> 
I originally experimented with patching the back end, but as you say the
changes would probably have to be very invasive/obfuscating in order to
extract all the information you can get out of Kconfiglib.  So I instead
went the minimally invasive route with a completely stand-alone library
together with a test suite.

I'm willing to update Kconfiglib when future additions/modifications are
made to the Kconfig language so that all tests in the compatibility test
suite keep passing.  I've added lots of comments and documentation to
internal methods and tried to keep the code clean and straightforward,
so it should be pretty easy for others to get into the code as well.

If by Perl you're referring to streamline_config.pl, I suspect it could
be rewritten as a smallish Kconfiglib script, though I haven't looked at
it closely.  That version might also be much better at respecting symbol
dependencies.
> > For a much longer introduction including multiple examples, see
> > arch/kconfig/kconfiglib.py.
> >
> > Have fun!
> >
> > Signed-off-by: Ulf Magnusson <ulfalizer.lkml@gmail.com>
> > ---
> > Convenience links:
> >
> > Documentation, generated from kconfiglib.py with pydoc -w:
> > http://dl.dropbox.com/u/10406197/kconfiglib.html
> >
> > Examples as separate files:
> > http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
> >
> >
> > The patch should be preferably be applied to a recent kernel, i.e. Linus's
> > (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
> > kconfigtest.py test suite - which compares output character-for-character -
> > will indicate failure on older (a few months old) kernels versions even though
> > the outputs are functionally equivalent.
> >
> >  Documentation/kbuild/kconfig-language.txt |    5 +
> >  Documentation/kbuild/kconfig.txt          |    8 +
> >  README                                    |   13 +
> >  scripts/kconfig/Makefile                  |   26 +-
> >  scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
> >  scripts/kconfig/kconfigtest.py            |  396 +++
> >  6 files changed, 4365 insertions(+), 1 deletions(-)
> >  create mode 100644 scripts/kconfig/kconfiglib.py
> >  create mode 100644 scripts/kconfig/kconfigtest.py
> > [...]

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Wed, Feb 02, 2011 at 12:27:52AM +0100, Ulf Magnusson wrote:
> Hi,
> 
> This is the initial release of Kconfiglib: a Python library for
> scripting, debugging, and extracting information from Kconfig-based
> configuration systems.  It can be used to programmatically generate a
> .config when the '*conf' tools are too inflexible, to quickly find out
> interesting information about a Kconfig configuration such as dependency
> relations between symbols and where undefined symbols are referenced,
> and in applications that need to parse and extract information from
> Kconfig files.
> 
> For a much longer introduction including multiple examples, see
> arch/kconfig/kconfiglib.py.
> 
> Have fun!
> 
> Signed-off-by: Ulf Magnusson <ulfalizer.lkml@gmail.com>
> ---
> Convenience links:
> 
> Documentation, generated from kconfiglib.py with pydoc -w:
> http://dl.dropbox.com/u/10406197/kconfiglib.html
> 
> Examples as separate files:
> http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
> 
> 
> The patch should be preferably be applied to a recent kernel, i.e. Linus's
> (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
> kconfigtest.py test suite - which compares output character-for-character -
> will indicate failure on older (a few months old) kernels versions even though
> the outputs are functionally equivalent.
> 
> [...]

Seems linux-kbuild and linux-doc won't accept the patch (too large?), so
here's a link to the message on linux-kernel:
https://lkml.org/lkml/2011/2/1/439

/Ulf Magnusson

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Thu, Feb 03, 2011 at 10:58:55PM +0100, Ulf Magnusson wrote:
> On Wed, Feb 02, 2011 at 12:27:52AM +0100, Ulf Magnusson wrote:
> > Hi,
> > 
> > This is the initial release of Kconfiglib: a Python library for
> > scripting, debugging, and extracting information from Kconfig-based
> > configuration systems.  It can be used to programmatically generate a
> > .config when the '*conf' tools are too inflexible, to quickly find out
> > interesting information about a Kconfig configuration such as dependency
> > relations between symbols and where undefined symbols are referenced,
> > and in applications that need to parse and extract information from
> > Kconfig files.
> > 
> > For a much longer introduction including multiple examples, see
> > arch/kconfig/kconfiglib.py.
> > 
> > Have fun!
> > 
> > Signed-off-by: Ulf Magnusson <ulfalizer.lkml@gmail.com>
> > ---
> > Convenience links:
> > 
> > Documentation, generated from kconfiglib.py with pydoc -w:
> > http://dl.dropbox.com/u/10406197/kconfiglib.html
> > 
> > Examples as separate files:
> > http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
> > 
> > 
> > The patch should be preferably be applied to a recent kernel, i.e. Linus's
> > (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
> > kconfigtest.py test suite - which compares output character-for-character -
> > will indicate failure on older (a few months old) kernels versions even though
> > the outputs are functionally equivalent.
> > 
> > [...]
> 
> Seems linux-kbuild and linux-doc won't accept the patch (too large?), so
> here's a link to the message on linux-kernel:
> https://lkml.org/lkml/2011/2/1/439
> 
> /Ulf Magnusson

Oh, and here's the patch message. Apply with 'git am'.
http://dl.dropbox.com/u/10406197/kconfiglib

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Randy Dunlap]

On Wed, 2 Feb 2011 00:27:55 +0100 Ulf Magnusson wrote:

> Hi,
> 
> This is the initial release of Kconfiglib: a Python library for
> scripting, debugging, and extracting information from Kconfig-based
> configuration systems.  It can be used to programmatically generate a
> .config when the '*conf' tools are too inflexible, to quickly find out
> interesting information about a Kconfig configuration such as dependency
> relations between symbols and where undefined symbols are referenced,
> and in applications that need to parse and extract information from
> Kconfig files.
> 
> For a much longer introduction including multiple examples, see
> arch/kconfig/kconfiglib.py.
> 
> Have fun!
> 
> Signed-off-by: Ulf Magnusson <ulfalizer.lkml@gmail.com>
> ---
> Convenience links:
> 
> Documentation, generated from kconfiglib.py with pydoc -w:
> http://dl.dropbox.com/u/10406197/kconfiglib.html
> 
> Examples as separate files:
> http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
> 
> 
> The patch should be preferably be applied to a recent kernel, i.e. Linus's
> (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
> kconfigtest.py test suite - which compares output character-for-character -
> will indicate failure on older (a few months old) kernels versions even though
> the outputs are functionally equivalent.
> 
>  Documentation/kbuild/kconfig-language.txt |    5 +
>  Documentation/kbuild/kconfig.txt          |    8 +
>  README                                    |   13 +
>  scripts/kconfig/Makefile                  |   26 +-
>  scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
>  scripts/kconfig/kconfigtest.py            |  396 +++
>  6 files changed, 4365 insertions(+), 1 deletions(-)
>  create mode 100644 scripts/kconfig/kconfiglib.py
>  create mode 100644 scripts/kconfig/kconfigtest.py
> 

> diff --git a/README b/README
> index 1b81d28..bb5e68f 100644
> --- a/README
> +++ b/README
> @@ -196,6 +196,19 @@ CONFIGURING the kernel:
>  			   values to 'n' as much as possible.
>  	"make randconfig"  Create a ./.config file by setting symbol
>  			   values to random values.
> +	"make scriptconfig SCRIPT=<path to script>" Run a Kconfiglib
> +			   script (see scripts/kconfig/kconfiglib.py).  This
> +			   can be used to programatically generate a
> +			   ./.config, and for applications that need to
> +			   extract information from Kconfig files.
> +	"make iscriptconfig" Launch an interactive Python shell
> +			   for running Kconfiglib on the architecture's
> +			   Kconfig configuration.  The kconfiglib and sys
> +			   (for sys.argv[1] - the base Kconfig file) modules
> +			   will be imported automatically, and a Config
> +			   instance 'c' will be created for the architecture
> +			   (using c = kconfiglib.Config(sys.argv[1])).
> +
>  
>     You can find more information on using the Linux kernel config tools
>     in Documentation/kbuild/kconfig.txt.

Hi Ulf,

This is interesting.  I just wish I could read it.  ;)
I'll get over it.


1.  It would be really Good to have "make scriptconfig SCRIPT=<path to script>"
and "make iscriptconfig" (similar to above, but shortened) in "make help" output.

2.  My first test (using your ex1.py script) failed because I used O=xx64 (build
directory):

rddunlap@chimera:lnx-2638-rc3> make O=xx64 scriptconfig SCRIPT=~/pkg/kconfiglib/ex1.py
  GEN     /lnx/src/lnx-2638-rc3/xx64/Makefile
Traceback (most recent call last):
  File "/home/rddunlap/pkg/kconfiglib/ex1.py", line 1, in <module>
    import kconfiglib
ImportError: No module named kconfiglib
make[2]: *** [scriptconfig] Error 1
make[1]: *** [scriptconfig] Error 2
make: *** [sub-make] Error 2

Does kconfiglib support O=builddir generally?  I can't tell that it does (yet)
since all ex[1-7].py fail in this manner.
It needs to support/allow O=builddir.

Thanks.  I'll keep looking...

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Rob Landley]

On 02/04/2011 04:35 PM, Randy Dunlap wrote:
>> The patch should be preferably be applied to a recent kernel, i.e. Linus's
>> (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
>> kconfigtest.py test suite - which compares output character-for-character -
>> will indicate failure on older (a few months old) kernels versions even though
>> the outputs are functionally equivalent.
>>
>>  Documentation/kbuild/kconfig-language.txt |    5 +
>>  Documentation/kbuild/kconfig.txt          |    8 +
>>  README                                    |   13 +
>>  scripts/kconfig/Makefile                  |   26 +-
>>  scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
>>  scripts/kconfig/kconfigtest.py            |  396 +++
>>  6 files changed, 4365 insertions(+), 1 deletions(-)
>>  create mode 100644 scripts/kconfig/kconfiglib.py
>>  create mode 100644 scripts/kconfig/kconfigtest.py

Wait, I thought this was an extra standalone library.  Are you saying
you want to make it so Linux will no longer compile on a build machine
that doesn't have Python installed?

If this is merely an extra developer tool ala bloat-o-meter and
checkpatch.pl then it's merely uninteresting to me.  (I myself wrote a
quick and dirty http://kernel.org/doc/make/menuconfig2html.py to
generate http://kernel.org/doc/menuconfig/x86.html and friends years
ago, and I still run it to update that once in a while.  There's not
much to it.)

But adding new prerequities to a build machine would be really annoying
for my use cases.

Rob

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Fri, Feb 04, 2011 at 02:35:29PM -0800, Randy Dunlap wrote:
> On Wed, 2 Feb 2011 00:27:55 +0100 Ulf Magnusson wrote:
> 
> > Hi,
> > 
> > This is the initial release of Kconfiglib: a Python library for
> > scripting, debugging, and extracting information from Kconfig-based
> > configuration systems.  It can be used to programmatically generate a
> > .config when the '*conf' tools are too inflexible, to quickly find out
> > interesting information about a Kconfig configuration such as dependency
> > relations between symbols and where undefined symbols are referenced,
> > and in applications that need to parse and extract information from
> > Kconfig files.
> > 
> > For a much longer introduction including multiple examples, see
> > arch/kconfig/kconfiglib.py.
> > 
> > Have fun!
> > 
> > Signed-off-by: Ulf Magnusson <ulfalizer.lkml@gmail.com>
> > ---
> > Convenience links:
> > 
> > Documentation, generated from kconfiglib.py with pydoc -w:
> > http://dl.dropbox.com/u/10406197/kconfiglib.html
> > 
> > Examples as separate files:
> > http://dl.dropbox.com/u/10406197/kconfiglib-examples.tar.gz
> > 
> > 
> > The patch should be preferably be applied to a recent kernel, i.e. Linus's
> > (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
> > kconfigtest.py test suite - which compares output character-for-character -
> > will indicate failure on older (a few months old) kernels versions even though
> > the outputs are functionally equivalent.
> > 
> >  Documentation/kbuild/kconfig-language.txt |    5 +
> >  Documentation/kbuild/kconfig.txt          |    8 +
> >  README                                    |   13 +
> >  scripts/kconfig/Makefile                  |   26 +-
> >  scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
> >  scripts/kconfig/kconfigtest.py            |  396 +++
> >  6 files changed, 4365 insertions(+), 1 deletions(-)
> >  create mode 100644 scripts/kconfig/kconfiglib.py
> >  create mode 100644 scripts/kconfig/kconfigtest.py
> > 
> 
> > diff --git a/README b/README
> > index 1b81d28..bb5e68f 100644
> > --- a/README
> > +++ b/README
> > @@ -196,6 +196,19 @@ CONFIGURING the kernel:
> >  			   values to 'n' as much as possible.
> >  	"make randconfig"  Create a ./.config file by setting symbol
> >  			   values to random values.
> > +	"make scriptconfig SCRIPT=<path to script>" Run a Kconfiglib
> > +			   script (see scripts/kconfig/kconfiglib.py).  This
> > +			   can be used to programatically generate a
> > +			   ./.config, and for applications that need to
> > +			   extract information from Kconfig files.
> > +	"make iscriptconfig" Launch an interactive Python shell
> > +			   for running Kconfiglib on the architecture's
> > +			   Kconfig configuration.  The kconfiglib and sys
> > +			   (for sys.argv[1] - the base Kconfig file) modules
> > +			   will be imported automatically, and a Config
> > +			   instance 'c' will be created for the architecture
> > +			   (using c = kconfiglib.Config(sys.argv[1])).
> > +
> >  
> >     You can find more information on using the Linux kernel config tools
> >     in Documentation/kbuild/kconfig.txt.
> 
> Hi Ulf,
> 
> This is interesting.  I just wish I could read it.  ;)
> I'll get over it.
> 
> 
> 1.  It would be really Good to have "make scriptconfig SCRIPT=<path to script>"
> and "make iscriptconfig" (similar to above, but shortened) in "make help" output.

I have added short descriptions to "make help", mostly referring people
to the script itself. Do you think I should explain it more fully? I
felt bad about adding big blobs of text when the "make help"
descriptions for all the other *config targets are short oneliners :)

> 
> 2.  My first test (using your ex1.py script) failed because I used O=xx64 (build
> directory):
> 
> rddunlap@chimera:lnx-2638-rc3> make O=xx64 scriptconfig SCRIPT=~/pkg/kconfiglib/ex1.py
>   GEN     /lnx/src/lnx-2638-rc3/xx64/Makefile
> Traceback (most recent call last):
>   File "/home/rddunlap/pkg/kconfiglib/ex1.py", line 1, in <module>
>     import kconfiglib
> ImportError: No module named kconfiglib
> make[2]: *** [scriptconfig] Error 1
> make[1]: *** [scriptconfig] Error 2
> make: *** [sub-make] Error 2
> 
> Does kconfiglib support O=builddir generally?  I can't tell that it does (yet)
> since all ex[1-7].py fail in this manner.
> It needs to support/allow O=builddir.

I hadn't considered O. I'll see if I can cook up a patch.

/Ulf

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Filip Honckiewicz]

Hello Ulf!

I'm working on pretty same thing also in python... Now I'm sad,
because you made it and I don't, and also my code looks like dung in
comparison to yours. How long did you make this?

BTW. "SLOW_WORK" symbol used in drivers/gpu/drm/Kconfig in "menucofnig
DRM" as one of selects isn't an unnecessary symbol? I can't find any
SLOW_WORK entry.

Filip

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Fri, Feb 04, 2011 at 05:42:50PM -0600, Rob Landley wrote:
> On 02/04/2011 04:35 PM, Randy Dunlap wrote:
> >> The patch should be preferably be applied to a recent kernel, i.e. Linus's
> >> (2.6.38-rc3 at the time of writing).  Due to recent Kconfig changes, the
> >> kconfigtest.py test suite - which compares output character-for-character -
> >> will indicate failure on older (a few months old) kernels versions even though
> >> the outputs are functionally equivalent.
> >>
> >>  Documentation/kbuild/kconfig-language.txt |    5 +
> >>  Documentation/kbuild/kconfig.txt          |    8 +
> >>  README                                    |   13 +
> >>  scripts/kconfig/Makefile                  |   26 +-
> >>  scripts/kconfig/kconfiglib.py             | 3918 +++++++++++++++++++++++++++++
> >>  scripts/kconfig/kconfigtest.py            |  396 +++
> >>  6 files changed, 4365 insertions(+), 1 deletions(-)
> >>  create mode 100644 scripts/kconfig/kconfiglib.py
> >>  create mode 100644 scripts/kconfig/kconfigtest.py
> 
> Wait, I thought this was an extra standalone library.  Are you saying
> you want to make it so Linux will no longer compile on a build machine
> that doesn't have Python installed?

No - it's completely standalone, and should have no effect on things
that do not use Kconfiglib. The only changes besides adding the script
itself is to add two (well, three - there's one that's only used by
kconfigtest.py) targets to scripts/kconfig/Makefile that make it easier
to use the library by ensuring the environment is set up correctly.

> 
> If this is merely an extra developer tool ala bloat-o-meter and
> checkpatch.pl then it's merely uninteresting to me.  (I myself wrote a
> quick and dirty http://kernel.org/doc/make/menuconfig2html.py to
> generate http://kernel.org/doc/menuconfig/x86.html and friends years
> ago, and I still run it to update that once in a while.  There's not
> much to it.)
> 
> But adding new prerequities to a build machine would be really annoying
> for my use cases.
> 
> Rob

You would only need to have Python installed if you are going to use
Kconfiglib directly (hard to get around with a Python library).

/Ulf

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Sat, Feb 05, 2011 at 01:28:50AM +0100, Filip Honckiewicz wrote:
> Hello Ulf!
> 
> I'm working on pretty same thing also in python... Now I'm sad,
> because you made it and I don't, and also my code looks like dung in
> comparison to yours. How long did you make this?

You didn't see the early versions ;)

I was a bit afraid that someone would come out with something similar
before it was done. Now you've given me a bad conscience :(

I've been working on it on and off in my spare time for six months or so.
Originally it was a project for automatically generating a minimal kernel for a
given system by automatically turning off options one by one and testing the
resulting kernel in an emulator (yeah, the minimal kernel you get out of that
process is broken in many ways, but you get some idea of what's needed and
what's not at least). That then grew into a general-purpose library.

> 
> BTW. "SLOW_WORK" symbol used in drivers/gpu/drm/Kconfig in "menucofnig
> DRM" as one of selects isn't an unnecessary symbol? I can't find any
> SLOW_WORK entry.

iscriptconfig says

A Config instance 'c' for the architecture (i386) has been created.
>>> c["SLOW_WORK"].is_defined()
False

so yes, it it's undefined on i386 at least. Grepping through the kernel shows
it only appears in drivers/gpu/drm/Kconfig, so it's in fact undefined on all
arches.

/Ulf

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Michal Marek]

On 2.2.2011 00:27, Ulf Magnusson wrote:
> Hi,
> 
> This is the initial release of Kconfiglib: a Python library for
> scripting, debugging, and extracting information from Kconfig-based
> configuration systems.  It can be used to programmatically generate a
> .config when the '*conf' tools are too inflexible, to quickly find out
> interesting information about a Kconfig configuration such as dependency
> relations between symbols and where undefined symbols are referenced,
> and in applications that need to parse and extract information from
> Kconfig files.
> 
> For a much longer introduction including multiple examples, see
> arch/kconfig/kconfiglib.py.

Hi,

this looks like a very powerful tool, but I have a similar concern like
Arnaud had - being completely standalone, it reimplements most of the C
kconfig code. One option to reduce this duplication would be a swig
wrapper, another one would be to let the C code parse the Kconfig files
and write the required information in some digested form, that would be
easier to parse by scripts. Something like:
$ scripts/kconfig/conf_inspect --kconfig=Kconfig --eval='FOO || BAR'
y
$ scripts/kconfig/conf_inspect ... --dump-symbols
config FOO
type: bool
valule: m
visible: y
prompt: "zzz"
depends: X & Y
select: Z
...
$ scripts/kconfig/conf_inspect ... --dump-symbols \
    --fields='depends,select,value'
config FOO
depends: X & Y
select: Z
value: m

config BAR
...

etc. The idea is that for instance instead of parsing the Kconfig files,
the Python code could fill it's data structures by reading the flat dump
provided by the C kconfig. There would be still lot to do in Python,
e.g. parsing and evaluating expressions, but it would be a small step
forward already. And people wanting to write quick&dirty scripts in
bash/awk/perl would make use of the C code as well.

Michal

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Fri, Feb 11, 2011 at 12:14:36AM +0100, Michal Marek wrote:
> On 2.2.2011 00:27, Ulf Magnusson wrote:
> > Hi,
> > 
> > This is the initial release of Kconfiglib: a Python library for
> > scripting, debugging, and extracting information from Kconfig-based
> > configuration systems.  It can be used to programmatically generate a
> > .config when the '*conf' tools are too inflexible, to quickly find out
> > interesting information about a Kconfig configuration such as dependency
> > relations between symbols and where undefined symbols are referenced,
> > and in applications that need to parse and extract information from
> > Kconfig files.
> > 
> > For a much longer introduction including multiple examples, see
> > arch/kconfig/kconfiglib.py.
> 
> Hi,
> 
> this looks like a very powerful tool, but I have a similar concern like
> Arnaud had - being completely standalone, it reimplements most of the C
> kconfig code. One option to reduce this duplication would be a swig
> wrapper, another one would be to let the C code parse the Kconfig files
> and write the required information in some digested form, that would be
> easier to parse by scripts. Something like:
> $ scripts/kconfig/conf_inspect --kconfig=Kconfig --eval='FOO || BAR'
> y
> $ scripts/kconfig/conf_inspect ... --dump-symbols
> config FOO
> type: bool
> valule: m
> visible: y
> prompt: "zzz"
> depends: X & Y
> select: Z
> ...
> $ scripts/kconfig/conf_inspect ... --dump-symbols \
>     --fields='depends,select,value'
> config FOO
> depends: X & Y
> select: Z
> value: m
> 
> config BAR
> ...
> 
> etc. The idea is that for instance instead of parsing the Kconfig files,
> the Python code could fill it's data structures by reading the flat dump
> provided by the C kconfig. There would be still lot to do in Python,
> e.g. parsing and evaluating expressions, but it would be a small step
> forward already. And people wanting to write quick&dirty scripts in
> bash/awk/perl would make use of the C code as well.
>
Offloading some of the parsing to a new tool would probably be doable,
though some Kconfig subleties might complicate things. For example,
symbols that appear within choices aren't seen as real selectable choice
items if they depend on previous symbols in specific ways (see
_determine_actual_items() in kconfiglib.py), and capturing this
information in the output while at the same preserving the ordering of
the symbols could be messy. I guess you could sort this out within
Kconfiglib instead, though other tools might then have to reimplement
the same (messy) rules.

I'm not so sure about offloading anything else this way. For dynamically
updating symbol values in a script you would need to somehow preserve
state between invocations (or pass huge lists of assignments back and
forth), which could lead to an unwieldy protocol. You might need to go
the SWIG route instead.

/Ulf

Re: [PATCH] [ANNOUNCE] kconfig: Kconfiglib: a flexible Python Kconfig parser

[Ulf Magnusson]

On Wed, Feb 16, 2011 at 05:48:32AM +0100, Ulf Magnusson wrote:
> On Fri, Feb 11, 2011 at 12:14:36AM +0100, Michal Marek wrote:
> > On 2.2.2011 00:27, Ulf Magnusson wrote:
> > > Hi,
> > > 
> > > This is the initial release of Kconfiglib: a Python library for
> > > scripting, debugging, and extracting information from Kconfig-based
> > > configuration systems.  It can be used to programmatically generate a
> > > .config when the '*conf' tools are too inflexible, to quickly find out
> > > interesting information about a Kconfig configuration such as dependency
> > > relations between symbols and where undefined symbols are referenced,
> > > and in applications that need to parse and extract information from
> > > Kconfig files.
> > > 
> > > For a much longer introduction including multiple examples, see
> > > arch/kconfig/kconfiglib.py.
> > 
> > Hi,
> > 
> > this looks like a very powerful tool, but I have a similar concern like
> > Arnaud had - being completely standalone, it reimplements most of the C
> > kconfig code. One option to reduce this duplication would be a swig
> > wrapper, another one would be to let the C code parse the Kconfig files
> > and write the required information in some digested form, that would be
> > easier to parse by scripts. Something like:
> > $ scripts/kconfig/conf_inspect --kconfig=Kconfig --eval='FOO || BAR'
> > y
> > $ scripts/kconfig/conf_inspect ... --dump-symbols
> > config FOO
> > type: bool
> > valule: m
> > visible: y
> > prompt: "zzz"
> > depends: X & Y
> > select: Z
> > ...
> > $ scripts/kconfig/conf_inspect ... --dump-symbols \
> >     --fields='depends,select,value'
> > config FOO
> > depends: X & Y
> > select: Z
> > value: m
> > 
> > config BAR
> > ...
> > 
> > etc. The idea is that for instance instead of parsing the Kconfig files,
> > the Python code could fill it's data structures by reading the flat dump
> > provided by the C kconfig. There would be still lot to do in Python,
> > e.g. parsing and evaluating expressions, but it would be a small step
> > forward already. And people wanting to write quick&dirty scripts in
> > bash/awk/perl would make use of the C code as well.
> >
> Offloading some of the parsing to a new tool would probably be doable,
> though some Kconfig subleties might complicate things. For example,
> symbols that appear within choices aren't seen as real selectable choice
> items if they depend on previous symbols in specific ways (see
> _determine_actual_items() in kconfiglib.py), and capturing this
> information in the output while at the same preserving the ordering of
> the symbols could be messy. I guess you could sort this out within
> Kconfiglib instead, though other tools might then have to reimplement
> the same (messy) rules.
> 
Then again, ordering information is retained (for example, assignments
in .config files are written out in the order symbols appear within the
Kconfig files), so you could probably just iterate over all items in
Kconfig order after parsing + implicit submenu creation (which causes
the weird choice behavior) to capture such stuff as well.

> I'm not so sure about offloading anything else this way. For dynamically
> updating symbol values in a script you would need to somehow preserve
> state between invocations (or pass huge lists of assignments back and
> forth), which could lead to an unwieldy protocol. You might need to go
> the SWIG route instead.

/Ulf