compat-drivers wishlist

compat-drivers wishlist

[Johannes Berg]

Hi,

I've long been fed up with the state of compat's build system, so I
figured I'd start by soliciting wishlist items.

FWIW, I think that the whole compat-drivers/scripts/ dir needs to be
rewritten for this ... :-)

My personal wishlist is below. I realize that some of these wishes cause
contradictions/problems in their implementation, but nonetheless I have
them :-)

1) Ability to copy only some drivers.
   We're interested in using compat-drivers to ship our driver to
customers for
   older kernels. We're not at all interested in shipping DRM, Atheros
drivers,
   etc.

2) To make that easier: split all patches per *file* in addition to per
change.
   That way, it would be possible to apply only patches for those files
that are
   actually in the tree. Right now, I heavily patch out those patches
that aren't
   really need.
   I suppose the best way would be to have directories for each change
and change
   the script that applies them to parse the patch header first. Or just
ignore
   patches that fail with "no file", but I don't know if that's easy to
determine
   from the patch exit status code.

3) Separate the output from the compat-drivers tree.
   There's no reason to ship patches/, most of scripts/ etc. in the
resulting
   tarball, in fact it's just confusing. Having the output in a separate
   directory, copying there only the few things that are needed, would
be very
   helpful.

4) Kconfig.
   This is a bit conflicting with 1) because a valid Kconfig structure
requires
   all the sub-Kconfig files exist. Maybe ship all Kconfig files, or
automate a
   way to remove "source" statements for files that don't exist
(probably easy)

I'll leave it at that for now, and follow up with some concrete
implementation ideas :-)

Any other thoughts?

johannes

Re: compat-drivers wishlist

[Luis R. Rodriguez]

On Thu, Mar 28, 2013 at 2:49 PM, Johannes Berg
<johannes@sipsolutions.net> wrote:
> Hi,
>
> I've long been fed up with the state of compat's build system, so I
> figured I'd start by soliciting wishlist items.
>
> FWIW, I think that the whole compat-drivers/scripts/ dir needs to be
> rewritten for this ... :-)
>
> My personal wishlist is below. I realize that some of these wishes cause
> contradictions/problems in their implementation, but nonetheless I have
> them :-)
>
> 1) Ability to copy only some drivers.
>    We're interested in using compat-drivers to ship our driver to
> customers for
>    older kernels. We're not at all interested in shipping DRM, Atheros
> drivers,
>    etc.

Seems reasonable. Maybe admin-update can read your own .config

> 2) To make that easier: split all patches per *file* in addition to per
> change.
>    That way, it would be possible to apply only patches for those files
> that are
>    actually in the tree. Right now, I heavily patch out those patches
> that aren't
>    really need.
>    I suppose the best way would be to have directories for each change
> and change
>    the script that applies them to parse the patch header first. Or just
> ignore
>    patches that fail with "no file", but I don't know if that's easy to
> determine
>    from the patch exit status code.

GNU patch could use some love. See commit 4a1a94f93 on compat-drivers.
Perhaps this is an area we'd be interested in extending functionality
as well. Having collateral evolutions split up each into their own
directory is a fine approach as well, but lets think long term here,
given that we're seeing what short term script/hack solutions can cost
us.

Maybe we need a context patch parser that given a .config will only
apply patches for the desired files. For this to work we'd need a
mapping of CONFIG_FOO symbols to respective driver path files regexps.
The Linux kernel MAINTAINERS file comes to mind as something close to
get us there, but that does not yet use CONFIG_ symbols, the title is
a bit loose.

Kconfig however gives us the config to let us build a target, and the
Makefile should let us extract its target dependencies, so I wonder if
we can use this to dynamically build kconfig to target maps.

> 3) Separate the output from the compat-drivers tree.
>    There's no reason to ship patches/, most of scripts/ etc. in the
> resulting
>    tarball, in fact it's just confusing. Having the output in a separate
>    directory, copying there only the few things that are needed, would
> be very
>    helpful.

Sure, makes sense.

> 4) Kconfig.
>    This is a bit conflicting with 1) because a valid Kconfig structure
> requires
>    all the sub-Kconfig files exist. Maybe ship all Kconfig files, or
> automate a
>    way to remove "source" statements for files that don't exist
> (probably easy)

I think its possible to remove not desired source code based on
reading .config. This can be done as a pass if the Kconfig to Makefile
thing is figured out. Then as a secondary step I think we can remove
directories that have just a Kconfig / Makefile, but that's
theoretical, in practice this may be harder, the biggest issue I think
would be to ensure removal of the calling 'source foo/Kconfig' entry,
or we extend Kconfig to support Kconfig imports that don't exist, sort
of with Makefile and the -include foo.mk trick.

> I'll leave it at that for now, and follow up with some concrete
> implementation ideas :-)
>
> Any other thoughts?

There, some feedback, thanks for taking the time to dumping your ideas.

  Luis

Re: compat-drivers wishlist

[Luis R. Rodriguez]

On Thu, Mar 28, 2013 at 3:15 PM, Luis R. Rodriguez
<mcgrof@do-not-panic.com> wrote:
> On Thu, Mar 28, 2013 at 2:49 PM, Johannes Berg
> <johannes@sipsolutions.net> wrote:
>> 4) Kconfig.
>>    This is a bit conflicting with 1) because a valid Kconfig structure
>> requires
>>    all the sub-Kconfig files exist. Maybe ship all Kconfig files, or
>> automate a
>>    way to remove "source" statements for files that don't exist
>> (probably easy)
>
> I think its possible to remove not desired source code based on
> reading .config. This can be done as a pass if the Kconfig to Makefile
> thing is figured out. Then as a secondary step I think we can remove
> directories that have just a Kconfig / Makefile, but that's
> theoretical, in practice this may be harder, the biggest issue I think
> would be to ensure removal of the calling 'source foo/Kconfig' entry,
> or we extend Kconfig to support Kconfig imports that don't exist, sort
> of with Makefile and the -include foo.mk trick.

Another thing I should mention here about Kconfig.

'make localmodconfig' is great and IMHO something which we should
strive for as well for end users. The issue comes in when you have to
build something for remote systems. For this I wrote a script a while
ago to let us extract only the necessary components from the kernel,
scp to target, and extract back our .config [0]. This is hacky but
something to keep in mind.

[0] https://lkml.org/lkml/2009/11/12/350

  Luis

Re: compat-drivers wishlist

[Johannes Berg]

On Thu, 2013-03-28 at 15:24 -0700, Luis R. Rodriguez wrote:

> Another thing I should mention here about Kconfig.
> 
> 'make localmodconfig' is great and IMHO something which we should
> strive for as well for end users. The issue comes in when you have to
> build something for remote systems. For this I wrote a script a while
> ago to let us extract only the necessary components from the kernel,
> scp to target, and extract back our .config [0]. This is hacky but
> something to keep in mind.

Yes, that's another interesting thing to do. I myself tend to think it's
less important as even building all of compat shouldn't take all that
much time. OTOH, it's clearly a lot more than what you usually want.

The tricky part here is that for those users that have most reason to
use compat, those who need it for new driver modules, it won't work
because they can't already be loaded on the system.

Anyway I don't see why we couldn't just copy the streamline config perl
script, with the Kconfig tree we build it would probably work just fine.

johannes

Re: compat-drivers wishlist

[Johannes Berg]

On Thu, 2013-03-28 at 15:15 -0700, Luis R. Rodriguez wrote:

> > 1) Ability to copy only some drivers.

> Seems reasonable. Maybe admin-update can read your own .config

This is a tricky proposition, see below & my other mail on Kconfig
thoughts.

> GNU patch could use some love. 

I'd hate to have to rely on a modified version of that :-)

FWIW, I think there's a python library to apply (unified) patches, if we
want to get fancy.

> See commit 4a1a94f93 on compat-drivers.

Ah, what's that for? Refreshing patches? I've never seen dependencies in
the compat patches so mostly I refresh manually, but yeah, the refresh
part is something I'd have to think about. Though, to be honest, when
you can just blow away your output directory and try again I'm not too
worried about this. Trying to apply the patches could just leave it in
the state you're in, and then you modify them until it works?

> Maybe we need a context patch parser that given a .config will only
> apply patches for the desired files. For this to work we'd need a
> mapping of CONFIG_FOO symbols to respective driver path files regexps.

Even assuming we do have such a mapping, I think it's the wrong approach
and causes a chicken-and-egg problem.

As I wrote in my other mail, you don't have a .config file before you
know the target kernel. This is because, for example, there's no way to
know if PCI is enabled in the target kernel, and thus sometimes you get
PCI drivers and sometimes not. Also I think maintenance and driver split
is not usually per Kconfig symbol but per directory. There are many more
Kconfig symbols, after all.

Also above you said you could do the source copying (or pruning) based
on the .config file. This has the exact same issue, I'd rather have the
copying or pruning be based on directories.

Basically I'm saying the .config file only exists on the target build
system, while the selective copying/pruning needs to be done on the
tarball creation system.

> > 4) Kconfig.

> I think its possible to remove not desired source code based on
> reading .config.

See above.

Oh, another issue I haven't addressed is default configurations, but I
think that's easy to do by providing something like the kernel's
defconfig.

johannes

Re: compat-drivers wishlist

[Hauke Mehrtens]

On 03/28/2013 11:27 PM, Johannes Berg wrote:
> On Thu, 2013-03-28 at 15:15 -0700, Luis R. Rodriguez wrote:
> 
>>> 1) Ability to copy only some drivers.
> 
>> Seems reasonable. Maybe admin-update can read your own .config
> 
> This is a tricky proposition, see below & my other mail on Kconfig
> thoughts.
> 
>> GNU patch could use some love. 
> 
> I'd hate to have to rely on a modified version of that :-)
> 
> FWIW, I think there's a python library to apply (unified) patches, if we
> want to get fancy.
> 
>> See commit 4a1a94f93 on compat-drivers.
> 
> Ah, what's that for? Refreshing patches? I've never seen dependencies in
> the compat patches so mostly I refresh manually, but yeah, the refresh
> part is something I'd have to think about. Though, to be honest, when
> you can just blow away your output directory and try again I'm not too
> worried about this. Trying to apply the patches could just leave it in
> the state you're in, and then you modify them until it works?

Refreshing patches should not be a problem. Normally I use
"./scripts/admin-refresh.sh -n -p -c -u refresh" to refresh the patches
and this should work with some minor changes with the proposal you made.

>> Maybe we need a context patch parser that given a .config will only
>> apply patches for the desired files. For this to work we'd need a
>> mapping of CONFIG_FOO symbols to respective driver path files regexps.
> 
> Even assuming we do have such a mapping, I think it's the wrong approach
> and causes a chicken-and-egg problem.
> 
> As I wrote in my other mail, you don't have a .config file before you
> know the target kernel. This is because, for example, there's no way to
> know if PCI is enabled in the target kernel, and thus sometimes you get
> PCI drivers and sometimes not. Also I think maintenance and driver split
> is not usually per Kconfig symbol but per directory. There are many more
> Kconfig symbols, after all.
> 
> Also above you said you could do the source copying (or pruning) based
> on the .config file. This has the exact same issue, I'd rather have the
> copying or pruning be based on directories.
> 
> Basically I'm saying the .config file only exists on the target build
> system, while the selective copying/pruning needs to be done on the
> tarball creation system.

How do you want to make the tar generation being configurable? The build
time Kconfig options should be generated from the Kconfig files provided
by the kernel and not by some own files.
I think some manual edited config file with the directories or files to
be included sounds good. This function would not be used by the normal
user so that should be good enough.

>>> 4) Kconfig.
> 
>> I think its possible to remove not desired source code based on
>> reading .config.
> 
> See above.
> 
> Oh, another issue I haven't addressed is default configurations, but I
> think that's easy to do by providing something like the kernel's
> defconfig.
> 
> johannes

Re: compat-drivers wishlist

[Johannes Berg]

On Thu, 2013-03-28 at 23:54 +0100, Hauke Mehrtens wrote:

> > Ah, what's that for? Refreshing patches? I've never seen dependencies in
> > the compat patches so mostly I refresh manually, but yeah, the refresh
> > part is something I'd have to think about. Though, to be honest, when
> > you can just blow away your output directory and try again I'm not too
> > worried about this. Trying to apply the patches could just leave it in
> > the state you're in, and then you modify them until it works?
> 
> Refreshing patches should not be a problem. Normally I use
> "./scripts/admin-refresh.sh -n -p -c -u refresh" to refresh the patches
> and this should work with some minor changes with the proposal you made.

Ok. I have no idea what that does, since most of the time my
"refreshing" isn't really refreshing but making them apply again, and I
do that manually.

> > Basically I'm saying the .config file only exists on the target build
> > system, while the selective copying/pruning needs to be done on the
> > tarball creation system.
> 
> How do you want to make the tar generation being configurable? The build
> time Kconfig options should be generated from the Kconfig files provided
> by the kernel and not by some own files.

Right, the build time Kconfig should just be copied as other source
files.

> I think some manual edited config file with the directories or files to
> be included sounds good. This function would not be used by the normal
> user so that should be good enough.

Indeed, that's what I was thinking. Sorry for not making that clear.

johannes

Re: compat-drivers wishlist

[Johannes Berg]

> 1) Ability to copy only some drivers.
> 2) To make that easier: split all patches per *file* in addition to per change.

You could think that an alternative to these would be for me to
post-process the sources and remove everything I don't care about.
However, that gets tricky because my compat package is created from my
own tree, and there the patches don't always apply. Therefore these
wishes.

I think for 2), I'd propose the following structure:

patches/collateral-evolutions/network/0001-netdev_ops/
	-> INFO
	-> rndis.patch
	-> usbnet.patch
	-> ath6kl.patch

Since the filename doesn't matter all that much, an automated conversion
could just split the patch per file and store it in a filename obtained
as follows:
	drivers/net/usb/rndis_host.c
->	drivers_net_usb_rndis_host.c

Easy :-)

The script to apply them gets a little more difficult, but I think we
should start using python for the tree-creation-time scripting which
will make this easy enough. Just walk the tree, etc.

> 3) Separate the output from the compat-drivers tree.

Should be easy, though it probably requires rewriting admin-update.
OTOH, admin-refresh etc. all pretty much goes away since you can just rm
-rf the output directory.

> 4) Kconfig.

So my wishes 1-3 apply at "tree creation" time, which is when the compat
source tree (and later tarball) is created and patched for the
backports. I've already suggested to rewrite that scripting in python,
and I'm willing to invest time in that if it stands a chance of getting
accepted :-)

This one is a bit more difficult. On the one hand we need to have all
the symbols from the kernel, but we don't want to leak them into the
compat namespace. Right now, for example, we do things like renaming
CONFIG_USB_NET_RNDIS_HOST to CONFIG_USB_NET_COMPAT_RNDIS_HOST which we
really should do for everything -- we get into trouble occasionally
because somebody enables something in the base kernel and then tries to
disable it in compat, which doesn't work.

I've been thinking a while about this, and I think I pretty much found a
solution.

First of all, it's easy to write a small Kconfig program (using the
existing Kconfig C code of course) that generates a list of all symbols
present in a given Kconfig tree, even disabled ones. Here's the program,
you just have to link it with zconf.tab.c and pass a root Kconfig file
as the only command line argument:
http://p.sipsolutions.net/3300bc91340ae747.txt

Equipped with a list of config symbols, we can go through the sources
and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also
CONFIG_*_MODULE). We can't just replace CONFIG_ with CPTCFG_ because
then we get kernel symbols wrong. We could rename all CONFIG_ to CPTCFG_
and import the kernel symbols, which we have to do for Kconfig anyway,
but I feel that doing that would make things more brittle. (*)

Now obviously if we have a local Kconfig tree, it's pretty useless. The
reason is that we'll have things like

config MYDRIVER_PCI
	tristate "my good PCI driver"
	depends on PCI && HAS_IOMEM
config MYDRIVER_USB
	tristate "my good USB driver"
	depends on USB

or something like that, and the Kconfig tree we have clearly doesn't
know about PCI, HAS_IOMEM or USB. We'll have to teach it, which is
surprisingly easy:

path/to/syms-program Kconfig > local-symbol-list
sed -i 's/$/=/' local-symbol-list
grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel

dotconf-to-kconf.py is this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
(Clearly it has to be rewritten in a different language since this has
to run at build time, not just at tarball creation time, but that's easy
to do)

Now we can "source Kconfig.kernel" into our local Kconfig tree, and the
"depends on USB" can magically work the right way. For the symbol walk
(syms program above) we'll have to include an empty file, but that's not
terribly difficult either.

Oh, whenever we execute menuconfig/oldconfig/... we set the environment
variable CONFIG_ to CPTCFG_ -- this will make the tool output CPTCFG_
instead of CONFIG_ as the prefix for all the output variables.

This is about as far as I got with the implementation. There are a few
issues left. 

1) All of the kernel symbols like USB get put into the .config file,
e.g. CPTCFG_USB. This doesn't actually matter all that much because
they're prefixed CPTCFG_ and the code we compile still looks for CONFIG_
for the kernel symbols -- remember that we only renamed the ones we
found in our local Kconfig tree.

2) I have no idea how to go from a .config file to a working
Makefile/autoconf.h system.

3) If KLIB_BUILD dir or the kernel config changes, this has to be
regenerated and oldconfig must be run. For instance, if USB was now
disabled in the base kernel then we shouldn't attempt to build USB
drivers, etc. This is a bit tricky because we also don't want to redo
everything for each "make". I'm not sure how much we care about this,
but we can probably solve it by storing a checksum of
$KLIB_BUILD/.config and doing the preparation steps only if that
changes, or so.

4) This assumes KLIB_BUILD is configured externally, it's not possible
to configure that through menuconfig since the Kconfig tree needs to be
built first. I don't think this is a problem, but it looks like Luis
wanted to put that into Kconfig so I'm saying it won't really work.

Thoughts?

johannes

(*) post-scriptum: this can and should actually happen at tree creation
time, not at build time

Re: compat-drivers wishlist

[Hauke Mehrtens]

On 03/28/2013 11:20 PM, Johannes Berg wrote:
> 
>> 1) Ability to copy only some drivers.
>> 2) To make that easier: split all patches per *file* in addition to per change.
> 
> You could think that an alternative to these would be for me to
> post-process the sources and remove everything I don't care about.
> However, that gets tricky because my compat package is created from my
> own tree, and there the patches don't always apply. Therefore these
> wishes.
> 
> I think for 2), I'd propose the following structure:
> 
> patches/collateral-evolutions/network/0001-netdev_ops/
> 	-> INFO
> 	-> rndis.patch
> 	-> usbnet.patch
> 	-> ath6kl.patch
> 
> Since the filename doesn't matter all that much, an automated conversion
> could just split the patch per file and store it in a filename obtained
> as follows:
> 	drivers/net/usb/rndis_host.c
> ->	drivers_net_usb_rndis_host.c
> 
> Easy :-)
> 
> The script to apply them gets a little more difficult, but I think we
> should start using python for the tree-creation-time scripting which
> will make this easy enough. Just walk the tree, etc.
> 
>> 3) Separate the output from the compat-drivers tree.
> 
> Should be easy, though it probably requires rewriting admin-update.
> OTOH, admin-refresh etc. all pretty much goes away since you can just rm
> -rf the output directory.
> 
>> 4) Kconfig.
> 
> So my wishes 1-3 apply at "tree creation" time, which is when the compat
> source tree (and later tarball) is created and patched for the
> backports. I've already suggested to rewrite that scripting in python,
> and I'm willing to invest time in that if it stands a chance of getting
> accepted :-)
> 
> This one is a bit more difficult. On the one hand we need to have all
> the symbols from the kernel, but we don't want to leak them into the
> compat namespace. Right now, for example, we do things like renaming
> CONFIG_USB_NET_RNDIS_HOST to CONFIG_USB_NET_COMPAT_RNDIS_HOST which we
> really should do for everything -- we get into trouble occasionally
> because somebody enables something in the base kernel and then tries to
> disable it in compat, which doesn't work.
> 
> I've been thinking a while about this, and I think I pretty much found a
> solution.
> 
> First of all, it's easy to write a small Kconfig program (using the
> existing Kconfig C code of course) that generates a list of all symbols
> present in a given Kconfig tree, even disabled ones. Here's the program,
> you just have to link it with zconf.tab.c and pass a root Kconfig file
> as the only command line argument:
> http://p.sipsolutions.net/3300bc91340ae747.txt
> 
> Equipped with a list of config symbols, we can go through the sources
> and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also
> CONFIG_*_MODULE). We can't just replace CONFIG_ with CPTCFG_ because
> then we get kernel symbols wrong. We could rename all CONFIG_ to CPTCFG_
> and import the kernel symbols, which we have to do for Kconfig anyway,
> but I feel that doing that would make things more brittle. (*)
> 
> Now obviously if we have a local Kconfig tree, it's pretty useless. The
> reason is that we'll have things like
> 
> config MYDRIVER_PCI
> 	tristate "my good PCI driver"
> 	depends on PCI && HAS_IOMEM
> config MYDRIVER_USB
> 	tristate "my good USB driver"
> 	depends on USB
> 
> or something like that, and the Kconfig tree we have clearly doesn't
> know about PCI, HAS_IOMEM or USB. We'll have to teach it, which is
> surprisingly easy:
> 
> path/to/syms-program Kconfig > local-symbol-list
> sed -i 's/$/=/' local-symbol-list
> grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel
> 
> dotconf-to-kconf.py is this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
> (Clearly it has to be rewritten in a different language since this has
> to run at build time, not just at tarball creation time, but that's easy
> to do)
> 
> Now we can "source Kconfig.kernel" into our local Kconfig tree, and the
> "depends on USB" can magically work the right way. For the symbol walk
> (syms program above) we'll have to include an empty file, but that's not
> terribly difficult either.

We also have to change all the select SOME_KERNEL_OPTION into a depends
SOME_KERNEL_OPTION, because the options of the used kernel can not be
changed.

> Oh, whenever we execute menuconfig/oldconfig/... we set the environment
> variable CONFIG_ to CPTCFG_ -- this will make the tool output CPTCFG_
> instead of CONFIG_ as the prefix for all the output variables.
> 
> This is about as far as I got with the implementation. There are a few
> issues left. 
> 
> 1) All of the kernel symbols like USB get put into the .config file,
> e.g. CPTCFG_USB. This doesn't actually matter all that much because
> they're prefixed CPTCFG_ and the code we compile still looks for CONFIG_
> for the kernel symbols -- remember that we only renamed the ones we
> found in our local Kconfig tree.
> 
> 2) I have no idea how to go from a .config file to a working
> Makefile/autoconf.h system.
> 
> 3) If KLIB_BUILD dir or the kernel config changes, this has to be
> regenerated and oldconfig must be run. For instance, if USB was now
> disabled in the base kernel then we shouldn't attempt to build USB
> drivers, etc. This is a bit tricky because we also don't want to redo
> everything for each "make". I'm not sure how much we care about this,
> but we can probably solve it by storing a checksum of
> $KLIB_BUILD/.config and doing the preparation steps only if that
> changes, or so.

compat should be changed in this way to not always backport any e.g. all
USB stuff and depend on USB stuff if the USB subsystem was not
activated. I think this should be modularized to have one *.ko for the
USB stuff, one for the PCMCIA stuff and so on.
Currently compat.ko depends on USB for some kernel versions.

This could also be done later.

> 4) This assumes KLIB_BUILD is configured externally, it's not possible
> to configure that through menuconfig since the Kconfig tree needs to be
> built first. I don't think this is a problem, but it looks like Luis
> wanted to put that into Kconfig so I'm saying it won't really work.
> 
> Thoughts?
> 
> johannes
> 
> (*) post-scriptum: this can and should actually happen at tree creation
> time, not at build time
> 

Re: compat-drivers wishlist

[Johannes Berg]

On Thu, 2013-03-28 at 23:40 +0100, Hauke Mehrtens wrote:
> > Now we can "source Kconfig.kernel" into our local Kconfig tree, and the
> > "depends on USB" can magically work the right way. For the symbol walk
> > (syms program above) we'll have to include an empty file, but that's not
> > terribly difficult either.
> 
> We also have to change all the select SOME_KERNEL_OPTION into a depends
> SOME_KERNEL_OPTION, because the options of the used kernel can not be
> changed.

Huh, good point. I didn't think about that, and it probably doesn't even
exist? But yeah, I agree, we need to do this. It's not terribly
difficult though -- after we have a list of local symbols we can go
through and rewrite "select NONLOCAL" to "depends on NONLOCAL".

> compat should be changed in this way to not always backport any e.g. all
> USB stuff and depend on USB stuff if the USB subsystem was not
> activated. I think this should be modularized to have one *.ko for the
> USB stuff, one for the PCMCIA stuff and so on.
> Currently compat.ko depends on USB for some kernel versions.

I don't think compat backports USB/PCMCIA/...? I was under the
impression that it was necessary to be from the base kernel anyway, but
I don't really know. In theory I agree, anything that's backported in
compat needs to be included into the Kconfig somehow, particularly
things like this.

Actually, I guess we do have things like that now, just not USB/PCMCIA:

CONFIG_COMPAT_FIRMWARE_CLASS
CONFIG_COMPAT_NET_SCH_CODEL
CONFIG_COMPAT_NET_SCH_FQ_CODEL
CONFIG_COMPAT_KFIFO
CONFIG_COMPAT_CORDIC
CONFIG_COMPAT_CRC8

and something about CONFIG_GENERIC_ATOMIC64 on non-64 bit platforms.

This is interesting! It probably means we need to generate a Kconfig
file for the kernel versions as well, at build time:

config COMPAT_KERNEL_2_6_26
	def_bool y

Obviously then compat relies more on compat-drivers to actually build?
It could have its own Kconfig file, and probably best if the creation of
this Kconfig.compat file is done through a script there.

Anyway this is special. So we can treat it specially, and do something
like

config COMPAT_CORDIC
	bool
	default !CORDIC

which should work just fine since we imported the symbols from the base
kernel. We'll then get a CPTCFG_COMPAT_CORDIC symbol that we can use to
build (or not) the cordic code.

For something that depends on a specific kernel version it's just as
simple:

config COMPAT_FIRMWARE_CLASS
	tristate
	default m if COMPAT_KERNEL_2_6_33 && !COMPAT_RHEL_6_1

or so.

johannes

Re: compat-drivers wishlist

[Luis R. Rodriguez]

On Thu, Mar 28, 2013 at 3:20 PM, Johannes Berg
<johannes@sipsolutions.net> wrote:
>
>> 1) Ability to copy only some drivers.
>> 2) To make that easier: split all patches per *file* in addition to per change.
>
> You could think that an alternative to these would be for me to
> post-process the sources and remove everything I don't care about.
> However, that gets tricky because my compat package is created from my
> own tree, and there the patches don't always apply. Therefore these
> wishes.
>
> I think for 2), I'd propose the following structure:
>
> patches/collateral-evolutions/network/0001-netdev_ops/
>         -> INFO
>         -> rndis.patch
>         -> usbnet.patch
>         -> ath6kl.patch

That's fine, and in fact with the other long term things I'm looking
at, such as SmPL to exrpress these collateral evolutions, is to
categorize the collateral evolutions into different types and see if
we can get *all* of them implemented in compat alone. That is, strive
to get rid of all patches required.

This requires a study of the different types of collateral evolutions
we have. compat already addresses the ones we can implement without
requiring changing the upstream drivers. I'm now starting to address
-- what is upstream willing to do to help with backports. An example
is the static inlines.

https://lkml.org/lkml/2013/3/28/187

If the above patch is accepted I'd then try to send upstream the
netdev_attach_ops() one, and that would nuke that entire patch from
compat-drivers.

> Since the filename doesn't matter all that much, an automated conversion
> could just split the patch per file and store it in a filename obtained
> as follows:
>         drivers/net/usb/rndis_host.c
> ->      drivers_net_usb_rndis_host.c
>
> Easy :-)

True. I welcome this change if it helps you.

> The script to apply them gets a little more difficult, but I think we
> should start using python for the tree-creation-time scripting which
> will make this easy enough. Just walk the tree, etc.

ACK.

>> 3) Separate the output from the compat-drivers tree.
>
> Should be easy, though it probably requires rewriting admin-update.
> OTOH, admin-refresh etc. all pretty much goes away since you can just rm
> -rf the output directory.
>
>> 4) Kconfig.
>
> So my wishes 1-3 apply at "tree creation" time, which is when the compat
> source tree (and later tarball) is created and patched for the
> backports. I've already suggested to rewrite that scripting in python,
> and I'm willing to invest time in that if it stands a chance of getting
> accepted :-)

I've long said bring it. I'm on the Python bandwagon now, my first
rewrite was ckmake.

> This one is a bit more difficult. On the one hand we need to have all
> the symbols from the kernel, but we don't want to leak them into the
> compat namespace. Right now, for example, we do things like renaming
> CONFIG_USB_NET_RNDIS_HOST to CONFIG_USB_NET_COMPAT_RNDIS_HOST which we
> really should do for everything -- we get into trouble occasionally
> because somebody enables something in the base kernel and then tries to
> disable it in compat, which doesn't work.

True.

> I've been thinking a while about this, and I think I pretty much found a
> solution.

Yay.

> First of all, it's easy to write a small Kconfig program (using the
> existing Kconfig C code of course) that generates a list of all symbols
> present in a given Kconfig tree, even disabled ones. Here's the program,
> you just have to link it with zconf.tab.c and pass a root Kconfig file
> as the only command line argument:
> http://p.sipsolutions.net/3300bc91340ae747.txt
>
> Equipped with a list of config symbols, we can go through the sources
> and Makefiles and replace CONFIG_* with (e.g.) CPTCFG_* (and also
> CONFIG_*_MODULE). We can't just replace CONFIG_ with CPTCFG_ because
> then we get kernel symbols wrong. We could rename all CONFIG_ to CPTCFG_
> and import the kernel symbols, which we have to do for Kconfig anyway,
> but I feel that doing that would make things more brittle. (*)
>
> Now obviously if we have a local Kconfig tree, it's pretty useless. The
> reason is that we'll have things like
>
> config MYDRIVER_PCI
>         tristate "my good PCI driver"
>         depends on PCI && HAS_IOMEM
> config MYDRIVER_USB
>         tristate "my good USB driver"
>         depends on USB
>
> or something like that, and the Kconfig tree we have clearly doesn't
> know about PCI, HAS_IOMEM or USB. We'll have to teach it, which is
> surprisingly easy:
>
> path/to/syms-program Kconfig > local-symbol-list
> sed -i 's/$/=/' local-symbol-list
> grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel

Neat trick! This doesn't handle the mapping of depdendencies but
certainly already makes good assumptions given that you *have* these
things built. It does not however take into consideration for example
that a user may say -- let me build cfg80211 alone, build it, install
it and now have their 802.11 driver also selected (if they didn't).

> dotconf-to-kconf.py is this little tool: http://p.sipsolutions.net/986bfbd397ad51c5.txt
> (Clearly it has to be rewritten in a different language since this has
> to run at build time, not just at tarball creation time, but that's easy
> to do)

I actually like the simplicity of it in Python and prefer it if we can
live with it, but given that we follow the kernel build system I'd
rather see the kernel first take the bullet on swallowing Python...
not sure, anyway doesn't matter much if its simple.

> Now we can "source Kconfig.kernel" into our local Kconfig tree, and the
> "depends on USB" can magically work the right way. For the symbol walk
> (syms program above) we'll have to include an empty file, but that's not
> terribly difficult either.
>
> Oh, whenever we execute menuconfig/oldconfig/... we set the environment
> variable CONFIG_ to CPTCFG_ -- this will make the tool output CPTCFG_
> instead of CONFIG_ as the prefix for all the output variables.
>
> This is about as far as I got with the implementation. There are a few
> issues left.

Like Kconfig renames that went upstream and mapping them to their
older name on the users's older kernels? This likely could be
addressed by having these maps laid out for each kernel, but would
require manual maintenance. It shouldn't be so hard if we do it
*moving forward*.

The other bigger thing I think is the dependency map output not
existing so it'd let users break a build / existing dep.

> 1) All of the kernel symbols like USB get put into the .config file,
> e.g. CPTCFG_USB. This doesn't actually matter all that much because
> they're prefixed CPTCFG_ and the code we compile still looks for CONFIG_
> for the kernel symbols -- remember that we only renamed the ones we
> found in our local Kconfig tree.
>
> 2) I have no idea how to go from a .config file to a working
> Makefile/autoconf.h system.

Hm, we'd already have the Makefile for each target no? Are you trying
to avoid having to ship each Makefile for each pulled driver?
For autoconf we have our own thingy so far.
Maybe I'm not getting what you mean here.

> 3) If KLIB_BUILD dir or the kernel config changes, this has to be
> regenerated and oldconfig must be run. For instance, if USB was now
> disabled in the base kernel then we shouldn't attempt to build USB
> drivers, etc. This is a bit tricky because we also don't want to redo
> everything for each "make". I'm not sure how much we care about this,
> but we can probably solve it by storing a checksum of
> $KLIB_BUILD/.config and doing the preparation steps only if that
> changes, or so.

Sure.

> 4) This assumes KLIB_BUILD is configured externally, it's not possible
> to configure that through menuconfig since the Kconfig tree needs to be
> built first. I don't think this is a problem, but it looks like Luis
> wanted to put that into Kconfig so I'm saying it won't really work.

I didn't want to put it into Kconfig actually, I was just throwing the
variables on the menuconfig display to ensure the user building is
aware of the sources of everything.

  Luis

Re: compat-drivers wishlist

[Johannes Berg]

On Thu, 2013-03-28 at 16:23 -0700, Luis R. Rodriguez wrote:

> This requires a study of the different types of collateral evolutions
> we have. compat already addresses the ones we can implement without
> requiring changing the upstream drivers. I'm now starting to address
> -- what is upstream willing to do to help with backports. An example
> is the static inlines.
> 
> https://lkml.org/lkml/2013/3/28/187
> 
> If the above patch is accepted I'd then try to send upstream the
> netdev_attach_ops() one, and that would nuke that entire patch from
> compat-drivers.

Right, this is a goal in itself, but I don't think we'll get rid of all
patches. Consider things like the completely different API for multicast
lists, for example.

> > Since the filename doesn't matter all that much, an automated conversion
> > could just split the patch per file and store it in a filename obtained
> > as follows:
> >         drivers/net/usb/rndis_host.c
> > ->      drivers_net_usb_rndis_host.c
> >
> > Easy :-)
> 
> True. I welcome this change if it helps you.

It would help me quite a bit. I suppose I could actually send you
patches for this soon. OTOH, without addressing my wishlist item 1) I
probably won't update to a current compat-drivers base ... I heavily
patch admin-update.sh now.

> > path/to/syms-program Kconfig > local-symbol-list
> > sed -i 's/$/=/' local-symbol-list
> > grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel
> 
> Neat trick! This doesn't handle the mapping of depdendencies but
> certainly already makes good assumptions given that you *have* these
> things built.

Are you referring to the "select" issue Hauke pointed out? Or what other
handling of dependencies are you thinking of? This entire trick exists
to allow compat's Kconfig to know about the base kernel's Kconfig so the
user can select the drivers that are actually possible given the base
kernel. For example, if building for kernel without PCI, you wouldn't be
able to select PCI drivers, etc.

> It does not however take into consideration for example
> that a user may say -- let me build cfg80211 alone, build it, install
> it and now have their 802.11 driver also selected (if they didn't).

That ... no I don't think you can do that. Nor do I think you even want
to? You can't build cfg80211 from one compat and the driver from
another, they won't be compatible anyway. So building cfg80211 first and
then installing it would be useless unless they build their driver from
the same tree later, but then it doesn't really matter that they still
have to select cfg80211? Or maybe I'm misunderstanding?

> I actually like the simplicity of it in Python and prefer it if we can
> live with it, but given that we follow the kernel build system I'd
> rather see the kernel first take the bullet on swallowing Python...
> not sure, anyway doesn't matter much if its simple.

It won't be more difficult even in bash, I'm just faster at writing it
in python :-)

> > This is about as far as I got with the implementation. There are a few
> > issues left.
> 
> Like Kconfig renames that went upstream and mapping them to their
> older name on the users's older kernels? This likely could be
> addressed by having these maps laid out for each kernel, but would
> require manual maintenance. It shouldn't be so hard if we do it
> *moving forward*.

Hmm, I never thought about that. Does that happen frequently? Do you
have any examples? Presumably, an example of this would be if you have
e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new
one, and all drivers that exist now have "depends on USB" while on an
old kernel USB won't exist, so you wouldn't be able to select such a
driver?

Anyway it seems like we must already maintain such a mapping in a way,
maybe a bit more implicitly? The way here you'd do it is add something
like this:

config USB
	bool
	default y
	depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5

or something like that?

> The other bigger thing I think is the dependency map output not
> existing so it'd let users break a build / existing dep.

I think it wouldn't break, worst case you wouldn't be able to select a
driver that you should be able to select?

> > 2) I have no idea how to go from a .config file to a working
> > Makefile/autoconf.h system.
> 
> Hm, we'd already have the Makefile for each target no? Are you trying
> to avoid having to ship each Makefile for each pulled driver?

Yes, we do have the makefiles of course, as copied from the kernel
sources. We want to keep them, obviously. I'm not trying to avoid that.
I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_*
symbols into make when it reads those makefiles.

> For autoconf we have our own thingy so far.

Yeah, but how do we get the symbols into make? That part isn't clear to
me at all, even for the regular kernel.

> > 4) This assumes KLIB_BUILD is configured externally, it's not possible
> > to configure that through menuconfig since the Kconfig tree needs to be
> > built first. I don't think this is a problem, but it looks like Luis
> > wanted to put that into Kconfig so I'm saying it won't really work.
> 
> I didn't want to put it into Kconfig actually, I was just throwing the
> variables on the menuconfig display to ensure the user building is
> aware of the sources of everything.

Ok :-)

Here's another issue I haven't really solved yet: All of compat must be
modules, so selecting Y for any symbol, e.g. CFG80211, must be
disallowed. This seems a little difficult, maybe the best choice would
be to simply hack menuconfig/... tool to disallow setting tristate
options to Y.

johannes

Re: compat-drivers wishlist

[Luis R. Rodriguez]

On Thu, Mar 28, 2013 at 4:40 PM, Johannes Berg
<johannes@sipsolutions.net> wrote:
> On Thu, 2013-03-28 at 16:23 -0700, Luis R. Rodriguez wrote:
>
>> This requires a study of the different types of collateral evolutions
>> we have. compat already addresses the ones we can implement without
>> requiring changing the upstream drivers. I'm now starting to address
>> -- what is upstream willing to do to help with backports. An example
>> is the static inlines.
>>
>> https://lkml.org/lkml/2013/3/28/187
>>
>> If the above patch is accepted I'd then try to send upstream the
>> netdev_attach_ops() one, and that would nuke that entire patch from
>> compat-drivers.
>
> Right, this is a goal in itself, but I don't think we'll get rid of all
> patches. Consider things like the completely different API for multicast
> lists, for example.

I am willing to accept perhaps some changes may not be possible and we
may need to live with patches but.. if we do at the very least we can
use SmPL grammar to express it. Whether or not we can express this in
SmPL that's a separate question but to help with that question we need
to start categorizing these types of collateral evolutions.

Julia in case you are interested in exploring the one Johannes
mentioned you can see commit 6dc94f7b9 on compat-drivers or just look
at the file:

patches/collateral-evolutions/network/25-multicast-list_head.patch

>> > Since the filename doesn't matter all that much, an automated conversion
>> > could just split the patch per file and store it in a filename obtained
>> > as follows:
>> >         drivers/net/usb/rndis_host.c
>> > ->      drivers_net_usb_rndis_host.c
>> >
>> > Easy :-)
>>
>> True. I welcome this change if it helps you.
>
> It would help me quite a bit. I suppose I could actually send you
> patches for this soon. OTOH, without addressing my wishlist item 1) I
> probably won't update to a current compat-drivers base ... I heavily
> patch admin-update.sh now.

OK.

>> > path/to/syms-program Kconfig > local-symbol-list
>> > sed -i 's/$/=/' local-symbol-list
>> > grep -v -f local-symbol-list $KLIB_BUILD/.config | dotconf-to-kconf.py > Kconfig.kernel
>>
>> Neat trick! This doesn't handle the mapping of depdendencies but
>> certainly already makes good assumptions given that you *have* these
>> things built.
>
> Are you referring to the "select" issue Hauke pointed out? Or what other
> handling of dependencies are you thinking of? This entire trick exists
> to allow compat's Kconfig to know about the base kernel's Kconfig so the
> user can select the drivers that are actually possible given the base
> kernel. For example, if building for kernel without PCI, you wouldn't be
> able to select PCI drivers, etc.

No I'm concerned about the dependency within compat-drivers itself on
allowing users to build an updated driver that breaks existing stock
drivers that they didn't enable on compat-drivers. One example: say a
user decides to build ath9k, but also has a USB dongle and ath9k_htc
but does not build the ath9k_htc module, only ath9k. Loading ath9k_htc
will fail after he installs that build.

>> It does not however take into consideration for example
>> that a user may say -- let me build cfg80211 alone, build it, install
>> it and now have their 802.11 driver also selected (if they didn't).
>
> That ... no I don't think you can do that. Nor do I think you even want
> to? You can't build cfg80211 from one compat and the driver from
> another, they won't be compatible anyway.

Well of course, I'm just saying -- it'd be possible to do this. But I
guess this is possible to day anyway :)

> So building cfg80211 first and
> then installing it would be useless unless they build their driver from
> the same tree later, but then it doesn't really matter that they still
> have to select cfg80211? Or maybe I'm misunderstanding?

No its fine, its a general build dependency issue but that issue
exists today, we just don't allow for huge different set of
configurations. We allow:

  * build everything
  * driver-select

And these are tested (or at least the ones I maintain are).

Kconfig opens the door for users doing complex configurations and
these could end up being reported as "issues", flooding the lists, and
time on people.

>> I actually like the simplicity of it in Python and prefer it if we can
>> live with it, but given that we follow the kernel build system I'd
>> rather see the kernel first take the bullet on swallowing Python...
>> not sure, anyway doesn't matter much if its simple.
>
> It won't be more difficult even in bash, I'm just faster at writing it
> in python :-)

OK ;)

>> > This is about as far as I got with the implementation. There are a few
>> > issues left.
>>
>> Like Kconfig renames that went upstream and mapping them to their
>> older name on the users's older kernels? This likely could be
>> addressed by having these maps laid out for each kernel, but would
>> require manual maintenance. It shouldn't be so hard if we do it
>> *moving forward*.
>
> Hmm, I never thought about that. Does that happen frequently?

Not *that* frequently but handling that is a fucking bitch. At times I
wish the maintainer would send me a patch but so far only the
contributors to compat have addressed that.

This is not a 'collateral evolution' per se, but certainly something
to consider if we can extract logic from git to address it for us.

> Do you
> have any examples? Presumably, an example of this would be if you have
> e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new
> one, and all drivers that exist now have "depends on USB" while on an
> old kernel USB won't exist, so you wouldn't be able to select such a
> driver?

Typically its not core stuff. Its all driver poo. At times its simply
renames from upstream to downstream mapping issues. compat-drivers
examples:

c5cfdf2d138ba20a648990d9615c44942eef5fd3
8c8113109f6916c4926413aa3416cefffaa5d9b8

> Anyway it seems like we must already maintain such a mapping in a way,
> maybe a bit more implicitly? The way here you'd do it is add something
> like this:
>
> config USB
>         bool
>         default y
>         depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5
>
> or something like that?

Sure. I think you're trying to automate this though?

BTW I'm trying to move things to BACKPORT_ prefix for poo given that
the kernel does use compat_ already for some poo. Not a big issue,
whatever.

>> The other bigger thing I think is the dependency map output not
>> existing so it'd let users break a build / existing dep.
>
> I think it wouldn't break, worst case you wouldn't be able to select a
> driver that you should be able to select?

Yes perhaps, its wroth for us to try it, likely better than the
current unscalable situation.

>> > 2) I have no idea how to go from a .config file to a working
>> > Makefile/autoconf.h system.
>>
>> Hm, we'd already have the Makefile for each target no? Are you trying
>> to avoid having to ship each Makefile for each pulled driver?
>
> Yes, we do have the makefiles of course, as copied from the kernel
> sources. We want to keep them, obviously. I'm not trying to avoid that.
> I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_*
> symbols into make when it reads those makefiles.

Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?

>> For autoconf we have our own thingy so far.
>
> Yeah, but how do we get the symbols into make? That part isn't clear to
> me at all, even for the regular kernel.

Yeah, never figured that poo out myself... The Makefile somehow has to
end up parsing the .config as a Makefile for variables, how that's
done no sure. For compat-drivers we just used assignments on
config.mk.

>> > 4) This assumes KLIB_BUILD is configured externally, it's not possible
>> > to configure that through menuconfig since the Kconfig tree needs to be
>> > built first. I don't think this is a problem, but it looks like Luis
>> > wanted to put that into Kconfig so I'm saying it won't really work.
>>
>> I didn't want to put it into Kconfig actually, I was just throwing the
>> variables on the menuconfig display to ensure the user building is
>> aware of the sources of everything.
>
> Ok :-)
>
> Here's another issue I haven't really solved yet: All of compat must be
> modules, so selecting Y for any symbol, e.g. CFG80211, must be
> disallowed. This seems a little difficult, maybe the best choice would
> be to simply hack menuconfig/... tool to disallow setting tristate
> options to Y.

If that's 100% true, sure! But say you want to disable a feature, or
enable a new one you didn't have ?

  Luis

Re: compat-drivers wishlist

[Johannes Berg]

On Thu, 2013-03-28 at 17:08 -0700, Luis R. Rodriguez wrote:

> > Are you referring to the "select" issue Hauke pointed out? Or what other
> > handling of dependencies are you thinking of? This entire trick exists
> > to allow compat's Kconfig to know about the base kernel's Kconfig so the
> > user can select the drivers that are actually possible given the base
> > kernel. For example, if building for kernel without PCI, you wouldn't be
> > able to select PCI drivers, etc.
> 
> No I'm concerned about the dependency within compat-drivers itself on
> allowing users to build an updated driver that breaks existing stock
> drivers that they didn't enable on compat-drivers. One example: say a
> user decides to build ath9k, but also has a USB dongle and ath9k_htc
> but does not build the ath9k_htc module, only ath9k. Loading ath9k_htc
> will fail after he installs that build.

Yes. That's unfortunate, but I don't really think we should _force_
users to enable ath9k_htc? And we can't know what kind of USB devices
they're going to plug in, so even "make localmodconfig" won't help all
that much. We could default to all the drivers that the user has
_installed_, but that'd mean maintaining a Kconfig <-> module mapping
(though maybe the localmodconfig script has that?) and it wouldn't be
what many users want? Why build USB drivers just because your distro did
if you don't own the device, after all...

IOW, I'm not sure what we can do about this.

> No its fine, its a general build dependency issue but that issue
> exists today, we just don't allow for huge different set of
> configurations. We allow:
> 
>   * build everything
>   * driver-select
> 
> And these are tested (or at least the ones I maintain are).
> 
> Kconfig opens the door for users doing complex configurations and
> these could end up being reported as "issues", flooding the lists, and
> time on people.

I'm not all that worried about that, to be honest. We should probably
default to running something like "allmodconfig", just like we do now.
Also, the kernel's configuration is much more complex and did you see
people complain there? :-)
In addition, we get the advantage that then people are unable to make
things that don't build. I for one have in fact gotten plenty of bug
reports because of wrong or missing ifdefs in config.mk (granted,
through my own fault) -- e.g. config.mk enabling debugfs when the base
kernel didn't allow for it, etc. So many issues are also solved by this.

> >> Like Kconfig renames that went upstream and mapping them to their
> >> older name on the users's older kernels? This likely could be
> >> addressed by having these maps laid out for each kernel, but would
> >> require manual maintenance. It shouldn't be so hard if we do it
> >> *moving forward*.
> >
> > Hmm, I never thought about that. Does that happen frequently?
> 
> Not *that* frequently but handling that is a fucking bitch. At times I
> wish the maintainer would send me a patch but so far only the
> contributors to compat have addressed that.
> 
> This is not a 'collateral evolution' per se, but certainly something
> to consider if we can extract logic from git to address it for us.

I don't really see what we could extract from git. The way I see it, the
only way to really address it is to add this to compat/Kconfig:

config NEW_SYMBOL_NAME
	tristate
	default OLD_SYMBOL_NAME

(or bool, and maybe with "depends on BACKPORT_KERNEL_X_Y_Z" added in)

> > Do you
> > have any examples? Presumably, an example of this would be if you have
> > e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new
> > one, and all drivers that exist now have "depends on USB" while on an
> > old kernel USB won't exist, so you wouldn't be able to select such a
> > driver?
> 
> Typically its not core stuff. Its all driver poo. At times its simply
> renames from upstream to downstream mapping issues. compat-drivers
> examples:
> 
> c5cfdf2d138ba20a648990d9615c44942eef5fd3

Heh, beating me with my own driver, always a good plan :-)

However this is entirely within the backports package so it's not an
issue at all?

> 8c8113109f6916c4926413aa3416cefffaa5d9b8

Ditto here. In fact both of these patches just go away with my Kconfig
scheme since I prefix CPTCFG_ instead of CONFIG_ and thus kernel symbols
aren't used any more by the Makefile etc. stuff in backports.

> > config USB
> >         bool
> >         default y
> >         depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5
> >
> > or something like that?

Heh, almost the same I wrote above...

> Sure. I think you're trying to automate this though?

But no, I don't think I can automate this part.

What I can automate is for drivers that are part of the backports
package. For *dependencies* those drivers have, like my hypothetical USB
example, we'll have to maintain it manually. It shouldn't be too
difficult though.

> BTW I'm trying to move things to BACKPORT_ prefix for poo given that
> the kernel does use compat_ already for some poo. Not a big issue,
> whatever.

Good plan. I'll make the COMPAT_KERNEL_X_Y_Z symbols
BACKPORT_KERNEL_X_Y_Z instead, and also the other symbols with COMPAT in
the name. For the automatic ones I picked CPTCFG_ which is hopefully
unique enough and has the advantage of having the same length as CONFIG_
(which is selfish of me, but it helps me maintain the code since we work
inside a backport package much of the time)

> >> > 2) I have no idea how to go from a .config file to a working
> >> > Makefile/autoconf.h system.
> >>
> >> Hm, we'd already have the Makefile for each target no? Are you trying
> >> to avoid having to ship each Makefile for each pulled driver?
> >
> > Yes, we do have the makefiles of course, as copied from the kernel
> > sources. We want to keep them, obviously. I'm not trying to avoid that.
> > I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_*
> > symbols into make when it reads those makefiles.
> 
> Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?

Yes, we can mostly do that, and get rid of all the patches that modify
symbols.

> >> For autoconf we have our own thingy so far.
> >
> > Yeah, but how do we get the symbols into make? That part isn't clear to
> > me at all, even for the regular kernel.
> 
> Yeah, never figured that poo out myself... The Makefile somehow has to
> end up parsing the .config as a Makefile for variables, how that's
> done no sure. For compat-drivers we just used assignments on
> config.mk.

Yeah ... I think the makefile just has to include the .config, I'll have
to check it out. Shouldn't be too difficult. I think the difficulty with
the backport system is that it has to call the kernel's Makefile, and
has to export everything for that ... anyway I'll figure out something,
can't be too hard.

> > Here's another issue I haven't really solved yet: All of compat must be
> > modules, so selecting Y for any symbol, e.g. CFG80211, must be
> > disallowed. This seems a little difficult, maybe the best choice would
> > be to simply hack menuconfig/... tool to disallow setting tristate
> > options to Y.
> 
> If that's 100% true, sure! But say you want to disable a feature, or
> enable a new one you didn't have ?

Well everything inside backports that's allowed to be tristate in the
kernel, must be N or M in backports, selecting Y for a tristate (module)
doesn't make sense. I'm already doing this now.

johannes

Re: compat-drivers wishlist

[Luis R. Rodriguez]

On Sat, Mar 30, 2013 at 11:30 AM, Johannes Berg
<johannes@sipsolutions.net> wrote:
> On Thu, 2013-03-28 at 17:08 -0700, Luis R. Rodriguez wrote:
>
>> > Are you referring to the "select" issue Hauke pointed out? Or what other
>> > handling of dependencies are you thinking of? This entire trick exists
>> > to allow compat's Kconfig to know about the base kernel's Kconfig so the
>> > user can select the drivers that are actually possible given the base
>> > kernel. For example, if building for kernel without PCI, you wouldn't be
>> > able to select PCI drivers, etc.
>>
>> No I'm concerned about the dependency within compat-drivers itself on
>> allowing users to build an updated driver that breaks existing stock
>> drivers that they didn't enable on compat-drivers. One example: say a
>> user decides to build ath9k, but also has a USB dongle and ath9k_htc
>> but does not build the ath9k_htc module, only ath9k. Loading ath9k_htc
>> will fail after he installs that build.
>
> Yes. That's unfortunate, but I don't really think we should _force_
> users to enable ath9k_htc? And we can't know what kind of USB devices
> they're going to plug in, so even "make localmodconfig" won't help all
> that much. We could default to all the drivers that the user has
> _installed_, but that'd mean maintaining a Kconfig <-> module mapping
> (though maybe the localmodconfig script has that?) and it wouldn't be
> what many users want? Why build USB drivers just because your distro did
> if you don't own the device, after all...
>
> IOW, I'm not sure what we can do about this.

See commit 8ae9673098 on compat, specifically the reference to Andi
Kleen's module symbol namespace. This would have solved that issue but
also allow, for example, iwlwifi to use say your stock mac80211 while
perhaps ath9k does need the backported wireless. Some distributions
have expressed interest in this as a feature (not specifically this
example, just saying any driver). The usage of the updates/ directory
would still enable us to let the backport driver supersede the older
stock kernel one.

Right now we don't make cfg80211, mac80211 use a prefix for their own
symbols but perhaps we should, that would address this. It would not
incur cost on RAM given that you'd only load the old module if a
device required it, and there is no extra storage cost given that
you'd still install the backport drivers as we do today.

Thoughts?

>> No its fine, its a general build dependency issue but that issue
>> exists today, we just don't allow for huge different set of
>> configurations. We allow:
>>
>>   * build everything
>>   * driver-select
>>
>> And these are tested (or at least the ones I maintain are).
>>
>> Kconfig opens the door for users doing complex configurations and
>> these could end up being reported as "issues", flooding the lists, and
>> time on people.
>
> I'm not all that worried about that, to be honest. We should probably
> default to running something like "allmodconfig", just like we do now.
> Also, the kernel's configuration is much more complex and did you see
> people complain there? :-)

Actually I do, I believe randconfig is used to test builds by some
folks, is how some obscure Kconfig bugs have been found, that is, a
driver developer not testing building under certain Kconfig settings.

Asking users to build everything resolves this in part, and having
users use menuconfig should then be something we have maintainers for
those options. This lets users peg those maintainers on the bug
reports found on those kconfig options. Thoughts?

> In addition, we get the advantage that then people are unable to make
> things that don't build. I for one have in fact gotten plenty of bug
> reports because of wrong or missing ifdefs in config.mk (granted,
> through my own fault) -- e.g. config.mk enabling debugfs when the base
> kernel didn't allow for it, etc. So many issues are also solved by this.

That's indeed awesome, I can't wait!

>> >> Like Kconfig renames that went upstream and mapping them to their
>> >> older name on the users's older kernels? This likely could be
>> >> addressed by having these maps laid out for each kernel, but would
>> >> require manual maintenance. It shouldn't be so hard if we do it
>> >> *moving forward*.
>> >
>> > Hmm, I never thought about that. Does that happen frequently?
>>
>> Not *that* frequently but handling that is a fucking bitch. At times I
>> wish the maintainer would send me a patch but so far only the
>> contributors to compat have addressed that.
>>
>> This is not a 'collateral evolution' per se, but certainly something
>> to consider if we can extract logic from git to address it for us.
>
> I don't really see what we could extract from git. The way I see it, the
> only way to really address it is to add this to compat/Kconfig:
>
> config NEW_SYMBOL_NAME
>         tristate
>         default OLD_SYMBOL_NAME
>
> (or bool, and maybe with "depends on BACKPORT_KERNEL_X_Y_Z" added in)

Well I like the idea :D

It does mean manual work though and upon integration we'd discover
this as a build failure, then we'd fix it.

>> > Do you
>> > have any examples? Presumably, an example of this would be if you have
>> > e.g. CONFIG_USB_DEVICES in an old kernel, but it's CONFIG_USB in the new
>> > one, and all drivers that exist now have "depends on USB" while on an
>> > old kernel USB won't exist, so you wouldn't be able to select such a
>> > driver?
>>
>> Typically its not core stuff. Its all driver poo. At times its simply
>> renames from upstream to downstream mapping issues. compat-drivers
>> examples:
>>
>> c5cfdf2d138ba20a648990d9615c44942eef5fd3
>
> Heh, beating me with my own driver, always a good plan :-)
>
> However this is entirely within the backports package so it's not an
> issue at all?

Sure, in this case, yes.

>> 8c8113109f6916c4926413aa3416cefffaa5d9b8
>
> Ditto here. In fact both of these patches just go away with my Kconfig
> scheme since I prefix CPTCFG_ instead of CONFIG_ and thus kernel symbols
> aren't used any more by the Makefile etc. stuff in backports.

OK neat.

>> > config USB
>> >         bool
>> >         default y
>> >         depends on USB_DEVICES && COMPAT_KERNEL_2_3_4_5
>> >
>> > or something like that?
>
> Heh, almost the same I wrote above...
>
>> Sure. I think you're trying to automate this though?
>
> But no, I don't think I can automate this part.
>
> What I can automate is for drivers that are part of the backports
> package. For *dependencies* those drivers have, like my hypothetical USB
> example, we'll have to maintain it manually. It shouldn't be too
> difficult though.

OK. I think this could be automated if we could script the question:

What dependencies did this module have in the original kernel? depmod
to the rescue.

/lib/modules/$(uname -r)/modules.dep

But that does the inverse, module to symbol lookup (BTW lets be clear
on language, symbol: export symbol, config: CONFIG_ poo), we need
something to query the dependencies based on a given CONFIG_ and I
believe we can get this, just gotta write it up. This can be done
later though for sure :)

>> BTW I'm trying to move things to BACKPORT_ prefix for poo given that
>> the kernel does use compat_ already for some poo. Not a big issue,
>> whatever.
>
> Good plan. I'll make the COMPAT_KERNEL_X_Y_Z symbols
> BACKPORT_KERNEL_X_Y_Z instead, and also the other symbols with COMPAT in
> the name. For the automatic ones I picked CPTCFG_ which is hopefully
> unique enough and has the advantage of having the same length as CONFIG_
> (which is selfish of me, but it helps me maintain the code since we work
> inside a backport package much of the time)

Sure, thanks!

>> >> > 2) I have no idea how to go from a .config file to a working
>> >> > Makefile/autoconf.h system.
>> >>
>> >> Hm, we'd already have the Makefile for each target no? Are you trying
>> >> to avoid having to ship each Makefile for each pulled driver?
>> >
>> > Yes, we do have the makefiles of course, as copied from the kernel
>> > sources. We want to keep them, obviously. I'm not trying to avoid that.
>> > I'm saying I have no idea how to get the CONFIG_* or rather CPTCFG_*
>> > symbols into make when it reads those makefiles.
>>
>> Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?
>
> Yes, we can mostly do that, and get rid of all the patches that modify
> symbols.

:D

perl -pi -e 's|CONFIG_|CONFIG_BACKPORT_|g' some/Makefile

But this would also end up renaming dependencies the Makefile might
have on stuff not provided by compat, an example:

ifeq ($(CONFIG_COMPAT),y)
  videodev-objs += v4l2-compat-ioctl32.o
endif

Would end up:

ifeq ($(CONFIG_BACKPORT_COMPAT),y)
  videodev-objs += v4l2-compat-ioctl32.o
endif

Maybe we need to script this to not do the mangle for config options
that are not present and assigned on the existing config.mk or .config
(not sure what these map to exactly on your tree yet).

>> >> For autoconf we have our own thingy so far.
>> >
>> > Yeah, but how do we get the symbols into make? That part isn't clear to
>> > me at all, even for the regular kernel.
>>
>> Yeah, never figured that poo out myself... The Makefile somehow has to
>> end up parsing the .config as a Makefile for variables, how that's
>> done no sure. For compat-drivers we just used assignments on
>> config.mk.
>
> Yeah ... I think the makefile just has to include the .config,

I reached the same conclusion I just never saw it being done, but I
also noticed I had to actually call export on these too.

> I'll have
> to check it out. Shouldn't be too difficult. I think the difficulty with
> the backport system is that it has to call the kernel's Makefile, and
> has to export everything for that ... anyway I'll figure out something,
> can't be too hard.

OK cool.

>> > Here's another issue I haven't really solved yet: All of compat must be
>> > modules, so selecting Y for any symbol, e.g. CFG80211, must be
>> > disallowed. This seems a little difficult, maybe the best choice would
>> > be to simply hack menuconfig/... tool to disallow setting tristate
>> > options to Y.
>>
>> If that's 100% true, sure! But say you want to disable a feature, or
>> enable a new one you didn't have ?
>
> Well everything inside backports that's allowed to be tristate in the
> kernel, must be N or M in backports, selecting Y for a tristate (module)
> doesn't make sense. I'm already doing this now.

Ah tristate, yeah that make sense and would be a good idea to help avoid issues.

 Luis

Re: compat-drivers wishlist

[Johannes Berg]

On Sat, 2013-03-30 at 14:35 -0700, Luis R. Rodriguez wrote:

> See commit 8ae9673098 on compat, specifically the reference to Andi
> Kleen's module symbol namespace. This would have solved that issue but
> also allow, for example, iwlwifi to use say your stock mac80211 while
> perhaps ath9k does need the backported wireless. Some distributions
> have expressed interest in this as a feature (not specifically this
> example, just saying any driver). The usage of the updates/ directory
> would still enable us to let the backport driver supersede the older
> stock kernel one.
> 
> Right now we don't make cfg80211, mac80211 use a prefix for their own
> symbols but perhaps we should, that would address this. It would not
> incur cost on RAM given that you'd only load the old module if a
> device required it, and there is no extra storage cost given that
> you'd still install the backport drivers as we do today.
> 
> Thoughts?

Ok yes, this is still an interesting thing that we might consider doing.
I consider it orthogonal to my current work though.

Tackling this would be tricky though, you don't want to patch all the
different uses of a symbol, so you'd have to do something like module
symbol namespaces, and teach the build environment about it, or
something like that. Maybe it could be done with the linker's symbol
versioning.

Anyway -- this is interesting but not my (primary) goal right now.

> Actually I do, I believe randconfig is used to test builds by some
> folks, is how some obscure Kconfig bugs have been found, that is, a
> driver developer not testing building under certain Kconfig settings.

Yes, of course, this happens all the time, but it's just a bug.

> Asking users to build everything resolves this in part, and having
> users use menuconfig should then be something we have maintainers for
> those options. This lets users peg those maintainers on the bug
> reports found on those kconfig options. Thoughts?

Users can attach their backport .config file to the bug report, so this
should be fairly easy to identify in the bug. Also, this is really not
much different from getting a "build failed with these options" bug on
the kernel. It's different, of course, but many such issues would
probably replicate in the kernel as well.

> > config NEW_SYMBOL_NAME
> >         tristate
> >         default OLD_SYMBOL_NAME
> >
> > (or bool, and maybe with "depends on BACKPORT_KERNEL_X_Y_Z" added in)
> 
> Well I like the idea :D
> 
> It does mean manual work though and upon integration we'd discover
> this as a build failure, then we'd fix it.

Yes. We don't get to magically avoid porting work :-)

> > What I can automate is for drivers that are part of the backports
> > package. For *dependencies* those drivers have, like my hypothetical USB
> > example, we'll have to maintain it manually. It shouldn't be too
> > difficult though.
> 
> OK. I think this could be automated if we could script the question:
> 
> What dependencies did this module have in the original kernel? depmod
> to the rescue.
> 
> /lib/modules/$(uname -r)/modules.dep
> 
> But that does the inverse, module to symbol lookup (BTW lets be clear
> on language, symbol: export symbol, config: CONFIG_ poo), we need
> something to query the dependencies based on a given CONFIG_ and I
> believe we can get this, just gotta write it up. This can be done
> later though for sure :)

Yeah, agree, later.

> >> Ah yes. Can't we assume a direct mapping CONFIG_FOO --> CPTCFG_FOO ?
> >
> > Yes, we can mostly do that, and get rid of all the patches that modify
> > symbols.
> 
> :D
> 
> perl -pi -e 's|CONFIG_|CONFIG_BACKPORT_|g' some/Makefile
[...]
> Maybe we need to script this to not do the mangle for config options
> that are not present and assigned on the existing config.mk or .config
> (not sure what these map to exactly on your tree yet).

Yes. I did pretty much exactly that. There's no config.mk, I only have
a .config file and that's generated at build time via the kconfig
mechanism.

> > Yeah ... I think the makefile just has to include the .config,
> 
> I reached the same conclusion I just never saw it being done, but I
> also noticed I had to actually call export on these too.

include .config
export

works for me. Just "export" means export all.

johannes

Re: compat-drivers wishlist

[Luis R. Rodriguez]

OK sweet, so just to keep track of pending work then:

https://backports.wiki.kernel.org/index.php/Todo

  Luis

Re: compat-drivers wishlist

[Johannes Berg]

On Thu, 2013-03-28 at 23:20 +0100, Johannes Berg wrote:
> > 1) Ability to copy only some drivers.
> > 2) To make that easier: split all patches per *file* in addition to per change.
> > 3) Separate the output from the compat-drivers tree.

done.

> > 4) Kconfig.

pretty much done -- right now users can get into unbuildable situations,
I obviously want to avoid.


I also decided to merge compat and compat-drivers -- right now I see no
value in them being combined. I therefore created a new backports.git
tree.

Note that this is very very raw. It doesn't even have the ability to
*build* the drivers yet, which is kinda the whole point ... But I had to
start somewhere, and obviously that was at step 1: building the
sources :-)

It's all here, if anyone wants to risk a look.
http://git.sipsolutions.net/?p=backports.git;a=summary

I'll continue next week, most likely.

johannes