What about make mergeconfig ?

What about make mergeconfig ?

[Benjamin Herrenschmidt]

Hi folks !

I'm not good enough at make and friends to do that myself without
spending a lot more time than I have at hand, but I figured it might be
something doable in a blink for whoever knows Kconfig guts :)

What about something like:

make mergeconfig <path_to_file>

That would merge all entries in the specified file with the
current .config. By mergeing, that basically means that rule:

N + N = N
m + N = m
Y + N = Y
m + Y = Y

(that is, we basically take for each entry max(.config, merge file)

The idea here is that on archs like powerpc, we have the ability to
build kernels that can boot several platforms. However, the defconfigs
we ship (g5_defconfig, pseries_defconfig, maple_defconfig, cell... ) are
tailored for one platform.

Now, if (for testing typically) I want to build a kernel that boots (and
has all the necessary drivers) for both a g5 and a cell, I need to start
from one of the defconfigs (the g5 one) and basically add in manually
all the bits necessary from the other one (the cell one).

Thus it might be useful to have a mecanism to automate that...

Thanks in advance for whichever guru wants to have a look at this :)

Cheers,
Ben.

Re: What about make mergeconfig ?

[David Woodhouse]

On Wed, 2006-10-25 at 15:46 +1000, Benjamin Herrenschmidt wrote:
> 
> make mergeconfig <path_to_file>
> 
> That would merge all entries in the specified file with the
> current .config. By mergeing, that basically means that rule:
> 
> N + N = N
> m + N = m
> Y + N = Y
> m + Y = Y  

We have something vaguely similar in the Fedora package (I think it's
only in the CVS tree rather than in the SRPM itself, but the CVS tree is
public too).

http://cvs.fedora.redhat.com/viewcvs/*checkout*/rpms/kernel/devel/merge.pl?rev=1.9

It doesn't do quite what you asked for -- it works with 'incremental'
configuration. So any options specified in _any_ form in the second
config are overridden in the output. It lets us start with a generic
config, then apply options (turning stuff both on and off) for PowerPC
in general, and then for specific builds on top of that.

-- 
dwmw2

Re: What about make mergeconfig ?

[Jesper Juhl]

On 25/10/06, Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> Hi folks !
>
> I'm not good enough at make and friends to do that myself without
> spending a lot more time than I have at hand, but I figured it might be
> something doable in a blink for whoever knows Kconfig guts :)
>
> What about something like:
>
> make mergeconfig <path_to_file>
>
> That would merge all entries in the specified file with the
> current .config. By mergeing, that basically means that rule:
>
> N + N = N
> m + N = m
> Y + N = Y
> m + Y = Y
>
> (that is, we basically take for each entry max(.config, merge file)
>
> The idea here is that on archs like powerpc, we have the ability to
> build kernels that can boot several platforms. However, the defconfigs
> we ship (g5_defconfig, pseries_defconfig, maple_defconfig, cell... ) are
> tailored for one platform.
>
> Now, if (for testing typically) I want to build a kernel that boots (and
> has all the necessary drivers) for both a g5 and a cell, I need to start
> from one of the defconfigs (the g5 one) and basically add in manually
> all the bits necessary from the other one (the cell one).
>
> Thus it might be useful to have a mecanism to automate that...
>
Hmm, wouldn't you only have to build that config once and then in the
future just use "make oldconfig" to keep it up-to-date with newer
kernels ?

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: What about make mergeconfig ?

[Benjamin Herrenschmidt]

> Hmm, wouldn't you only have to build that config once and then in the
> future just use "make oldconfig" to keep it up-to-date with newer
> kernels ?

Well, it's handy to be able to pick up a fresh new upstream tree and
build a config for pmac+pseries for example.

Ben.

Re: What about make mergeconfig ?

[Dick Streefland]

Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
| I'm not good enough at make and friends to do that myself without
| spending a lot more time than I have at hand, but I figured it might be
| something doable in a blink for whoever knows Kconfig guts :)
| 
| What about something like:
| 
| make mergeconfig <path_to_file>
| 
| That would merge all entries in the specified file with the
| current .config. By mergeing, that basically means that rule:
| 
| N + N = N
| m + N = m
| Y + N = Y
| m + Y = Y
| 
| (that is, we basically take for each entry max(.config, merge file)

Can't you do that with just a sort command?

  sort .config other.config > new.config

-- 
Dick Streefland                      ////                      Altium BV
dick.streefland@altium.nl           (@ @)          http://www.altium.com
--------------------------------oOO--(_)--OOo---------------------------

Re: What about make mergeconfig ?

[Jan Engelhardt]

>Can't you do that with just a sort command?
>
>  sort .config other.config > new.config

That does not work where .config and other.config have the same symbol 
listed, kconfig will bark and use the first value encountered. Because I 
do have exactly that problem with my patch series (changes some Ys to 
Ms), I am in need of the following patch to Kconfig TDTRT.

This is probably also what the OP is looking for, except that it does 
not require a special 'mergeconfig', but works with all 'oldconfig', 
'menuconfig', xconfig, etc.


kconfig_override.diff
Signed-off-by: Jan Engelhardt <jengelh@gmx.de>

Index: linux-2.6.18_rc4/scripts/kconfig/confdata.c
===================================================================
--- linux-2.6.18_rc4.orig/scripts/kconfig/confdata.c
+++ linux-2.6.18_rc4/scripts/kconfig/confdata.c
@@ -170,8 +170,7 @@ load:
 					sym->type = S_BOOLEAN;
 			}
 			if (sym->flags & def_flags) {
-				conf_warning("trying to reassign symbol %s", sym->name);
-				break;
+				conf_warning("override: reassigning to symbol %s", sym->name);
 			}
 			switch (sym->type) {
 			case S_BOOLEAN:
@@ -207,8 +206,7 @@ load:
 					sym->type = S_OTHER;
 			}
 			if (sym->flags & def_flags) {
-				conf_warning("trying to reassign symbol %s", sym->name);
-				break;
+				conf_warning("override: reassigning to symbol %s", sym->name);
 			}
 			switch (sym->type) {
 			case S_TRISTATE:
@@ -284,11 +282,9 @@ load:
 				}
 				break;
 			case yes:
-				if (cs->def[def].tri != no) {
-					conf_warning("%s creates inconsistent choice state", sym->name);
-					cs->flags &= ~def_flags;
-				} else
-					cs->def[def].val = sym;
+				if(cs->def[def].tri != no)
+					conf_warning("override: %s turns state choice", sym->name);
+				cs->def[def].val = sym;
 				break;
 			}
 			cs->def[def].tri = E_OR(cs->def[def].tri, sym->def[def].tri);
#<EOF>


	-`J'
-- 

Re: What about make mergeconfig ?

[Dick Streefland]

Jan Engelhardt <jengelh@linux01.gwdg.de> wrote:
| >Can't you do that with just a sort command?
| >
| >  sort .config other.config > new.config
| 
| That does not work where .config and other.config have the same symbol 
| listed, kconfig will bark and use the first value encountered. Because I 
| do have exactly that problem with my patch series (changes some Ys to 
| Ms), I am in need of the following patch to Kconfig TDTRT.

Or you can use the following hack:

  (sort .config other.config; echo set) | sh | grep ^CONFIG_ > new.config

-- 
Dick Streefland                      ////                      Altium BV
dick.streefland@altium.nl           (@ @)          http://www.altium.com
--------------------------------oOO--(_)--OOo---------------------------

Re: What about make mergeconfig ?

[Jan Engelhardt]

>| >Can't you do that with just a sort command?
>| >
>| >  sort .config other.config > new.config
>| 
>| That does not work where .config and other.config have the same symbol 
>| listed, kconfig will bark and use the first value encountered. Because I 
>| do have exactly that problem with my patch series (changes some Ys to 
>| Ms), I am in need of the following patch to Kconfig TDTRT.
>
>Or you can use the following hack:
>
>  (sort .config other.config; echo set) | sh | grep ^CONFIG_ > new.config

That does not properly deal with "# CONFIG_XYZ is not set" lines in 
other.config.


	-`J'
-- 

Re: What about make mergeconfig ?

[Geert Uytterhoeven]

On Thu, 26 Oct 2006, Jan Engelhardt wrote:
> >| >Can't you do that with just a sort command?
> >| >
> >| >  sort .config other.config > new.config
> >| 
> >| That does not work where .config and other.config have the same symbol 
> >| listed, kconfig will bark and use the first value encountered. Because I 
> >| do have exactly that problem with my patch series (changes some Ys to 
> >| Ms), I am in need of the following patch to Kconfig TDTRT.
> >
> >Or you can use the following hack:
> >
> >  (sort .config other.config; echo set) | sh | grep ^CONFIG_ > new.config
> 
> That does not properly deal with "# CONFIG_XYZ is not set" lines in 
> other.config.

What about RCS merge?

merge -p other.config .config.old .config > other.config.new

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: What about make mergeconfig ?

[Jan Engelhardt]

>> >Or you can use the following hack:
>> >
>> >  (sort .config other.config; echo set) | sh | grep ^CONFIG_ > new.config
>> 
>> That does not properly deal with "# CONFIG_XYZ is not set" lines in 
>> other.config.
>
>What about RCS merge?

I take it we do not want to depend on too many tools (remember the 
kconfig implementation language debate).

>merge -p other.config .config.old .config > other.config.new

This also does not seem conflict-safe.


	-`J'
-- 

Re: What about make mergeconfig ?

[Geert Uytterhoeven]

On Thu, 26 Oct 2006, Jan Engelhardt wrote:
> >> >Or you can use the following hack:
> >> >
> >> >  (sort .config other.config; echo set) | sh | grep ^CONFIG_ > new.config
> >> 
> >> That does not properly deal with "# CONFIG_XYZ is not set" lines in 
> >> other.config.
> >
> >What about RCS merge?
> 
> I take it we do not want to depend on too many tools (remember the 
> kconfig implementation language debate).

If you have CVS installed, you have RCS merge.

> >merge -p other.config .config.old .config > other.config.new
> 
> This also does not seem conflict-safe.

Indeed, you can still have conflicts, which you have to resolve manually.

But it depends on what you want to achieve: do you want to set each config
option in the destination config to max(config1.option, config2.option), or do
you want to apply the recent changes for one config (which may include
disabling options) to another config?

For the latter, merge should work fine.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: What about make mergeconfig ?

[Jan Engelhardt]

>> >What about RCS merge?
>> 
>> I take it we do not want to depend on too many tools (remember the 
>> kconfig implementation language debate).
>
>If you have CVS installed, you have RCS merge.

11:54 ichi:~ > rpm -q cvs rcs
cvs-1.12.12-19
package rcs is not installed

11:54 ichi:~ > gzip -cd /ARCHIVES.gz | grep "/merge$"
./CD1/suse/i586/rcs-5.7-879.i586.rpm:
    -rwxr-xr-x    1 root    root 45252 May  2 09:42 /usr/bin/merge

CVS does not need RCS.

>>>merge -p other.config .config.old .config > other.config.new
>> 
>> This also does not seem conflict-safe.
>
>Indeed, you can still have conflicts, which you have to resolve manually.
>
>But it depends on what you want to achieve: do you want to set each config
>option in the destination config to max(config1.option, config2.option), or do
>you want to apply the recent changes for one config (which may include
>disabling options) to another config?

In my case, the latter.

>For the latter, merge should work fine.

Is merge a lot different from what `patch` is doing?


	-`J'
-- 

Re: What about make mergeconfig ?

[Geert Uytterhoeven]

On Fri, 27 Oct 2006, Jan Engelhardt wrote:
> >> >What about RCS merge?
> >> 
> >> I take it we do not want to depend on too many tools (remember the 
> >> kconfig implementation language debate).
> >
> >If you have CVS installed, you have RCS merge.
> 
> 11:54 ichi:~ > rpm -q cvs rcs
> cvs-1.12.12-19
> package rcs is not installed
> 
> 11:54 ichi:~ > gzip -cd /ARCHIVES.gz | grep "/merge$"
> ./CD1/suse/i586/rcs-5.7-879.i586.rpm:
>     -rwxr-xr-x    1 root    root 45252 May  2 09:42 /usr/bin/merge
> 
> CVS does not need RCS.

OK, so CVS assimilated RCS merge internally.

> >>>merge -p other.config .config.old .config > other.config.new
> >> 
> >> This also does not seem conflict-safe.
> >
> >Indeed, you can still have conflicts, which you have to resolve manually.
> >
> >But it depends on what you want to achieve: do you want to set each config
> >option in the destination config to max(config1.option, config2.option), or do
> >you want to apply the recent changes for one config (which may include
> >disabling options) to another config?
> 
> In my case, the latter.
> 
> >For the latter, merge should work fine.
> 
> Is merge a lot different from what `patch` is doing?

Yes, it does 3-way merges (merge `my' version with `your' version using a
common ancestor). I.e. it's better in resolving conflicts than patch.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: What about make mergeconfig ?

[Christoph Lameter]

On Wed, 25 Oct 2006, Benjamin Herrenschmidt wrote:

> The idea here is that on archs like powerpc, we have the ability to
> build kernels that can boot several platforms. However, the defconfigs
> we ship (g5_defconfig, pseries_defconfig, maple_defconfig, cell... ) are
> tailored for one platform.

Same issue on IA64. Would be good to have.

Re: What about make mergeconfig ?

[Samuel Tardieu]

>>>>> "Benjamin" == Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

Benjamin> That would merge all entries in the specified file with the
Benjamin> current .config.

You don't need it to be a Makefile target, it can be an external
script. Would this one (untested!) do what you want?

  Sam
-- 
Samuel Tardieu -- sam@rfc1149.net -- http://www.rfc1149.net/

#! /usr/bin/python
#
# (c) 2006 Samuel Tardieu <sam@rfc1149.net>
#
# This software may be used and distributed according to the terms
# of the GNU General Public License, incorporated herein by reference.
#
# Usage: mergeconfig config1 config2 ... > newconfig
#
# To get the formatting back, it is advised to run "make oldconfig" on the
# result.
#
# Be careful in not using the same file as input and output

import sre, sys

_not_set = sre.compile('# (CONFIG_\S+) is not set')
_set = sre.compile('(CONFIG_[^=]+)=(.*)')

def read_config(fn):
    """Read a kernel configuration file and return a dictionary with
    all the options present in the file. If an option is commented out,
    set its value as None."""
    d = {}
    for l in file(fn):
        l = l.rstrip('\r\n')
        x = _not_set.match(l)
        if x: d[x.group(1)] = None
        x = _set.match(l)
        if x: d[x.group(1)] = x.group(2)
    return d

def merge_option(o, v1, v2):
    """Merge option value v1 and v2."""
    if 'y' in [v1, v2]: return 'y'
    if 'm' in [v1, v2]: return 'm'
    if v1 != v2:
        sys.stderr.write('Option %s has two incompatible values: %s and %s' %
                         (o, v1, v2))
        sys.exit(1)
    return v1

def merge_config_into(a, b):
    """Merge configuration dictionary a into configuration dictionary b.
    In addition, this function returns b after the merge."""
    for k, v in a.items():
        try:
            b[v] = merge_option(k, v, b[v])
        except KeyError:
            b[k] = v
    return b

def output_config(d):
    for k, v in d.items():
        if v is None: print '# %s is not set' % k
        else: print '%s=%s' % (k, v)

if __name__ == '__main__':
    output_config (reduce(lambda x, y: merge_config_into(x, read_config(y)),
                          sys.argv[1:], {}))