[ANN] new backports project

[ANN] new backports project

[Johannes Berg]

Hey,

I guess just a few days later it's time to announce it ... I've been
working on a merge of compat/compat-drivers into a new backports tree.
That by itself would be boring, so here's what I also did:

 * made driver copying variable
   As discussed a few days ago, I wanted this to be able to port only
some
   drivers. To really achieve this without breaking all patches, I also
had to
   - split patches and make applying them smarter
   - remove Makefile patches and mangle the Makefiles in a scripted way
 * added Kconfig support
   This allows users to better select which drivers they want, with
their
   dependencies etc. To achieve this, it was necessary to also mangle
the Kconfig
   files in some ways, but that's all scripted as well
 * separate output directory
   My new script writes in a new output directory, which helps separate
the
   compat code creation step from the build step, and end users don't
get
   burdened with the scripting needed for the creation.

To be fair, I should say what I broke. I think the system should be
generic enough to allow adding it back easily though.
 * graphics drivers -- I couldn't even get the patches to apply
 * ethernet drivers -- I just wasn't sure which ones were needed etc.
 * "crap"/staging/... patches
 * plain 'make' -- you now need to do 'make allyesconfig' or so
first :-)
 * Redhat kernel backport support -- this should be fairly easy

I'm hoping that the improvements I have are sufficient to get everyone
to help out, and have the old compat/compat-drivers die ... :-)

The code is available here (for now):
http://git.sipsolutions.net/?p=backports.git;a=summary

And in case you're wondering ... I got ckmake to build an
"allyesconfig" (except for CFG80211_INTERNAL_REGDB which I forced off)
against all kernels on Luis's build machine. This took forever, but I
got this:

1   2.6.24              [  OK  ]
2   2.6.25              [  OK  ]
3   2.6.26              [  OK  ]
4   2.6.27              [  OK  ]
5   2.6.28              [  OK  ]
6   2.6.29              [  OK  ]
7   2.6.30              [  OK  ]
8   2.6.31              [  OK  ]
9   2.6.32              [  OK  ]
10  2.6.33              [  OK  ]
11  2.6.34              [  OK  ]
12  2.6.35              [  OK  ]
13  2.6.36              [  OK  ]
14  2.6.37              [  OK  ]
15  2.6.38              [  OK  ]
16  2.6.39              [  OK  ]
17  3.0.65              [  OK  ]
18  3.1.10              [  OK  ]
19  3.2.38              [  OK  ]
20  3.3.8               [  OK  ]
21  3.4.32              [  OK  ]
22  3.5.7               [  OK  ]
23  3.6.11              [  OK  ]
24  3.7.9               [  OK  ]
25  3.8.0               [  OK  ]
26  3.9-rc1             [  OK  ]

johannes

Re: [ANN] new backports project

[Norman Shulman]

On Tue, Apr 2, 2013 at 9:19 AM, Johannes Berg <johannes@sipsolutions.net> wrote:

> To be fair, I should say what I broke. I think the system should be
> generic enough to allow adding it back easily though.
>  * graphics drivers -- I couldn't even get the patches to apply
>  * ethernet drivers -- I just wasn't sure which ones were needed etc.

alx please.

Norm

--
Norman Shulman
Sr. Developer/Architect
N-Dimension Solutions Inc.
9030 Leslie St, Unit 300
Richmond Hill, ON L4B 1G2
Canada

Tel: 905 707-8884 x 226
Fax: 905 707-0886

This email and any files transmitted with it are solely intended for
the use of the named recipient(s) and may contain information that is
privileged and confidential. If you receive this email in error,
please immediately notify the sender and delete this message in all
its forms.

Re: [ANN] new backports project

[Johannes Berg]

>  * Redhat kernel backport support -- this should be fairly easy

I addressed this, but I'm unsure of it and don't have a way to test it,
so any testing would be appreciated (as it is in general, of course)

johannes

Re: [ANN] new backports project

[Luis R. Rodriguez]

On Tue, Apr 2, 2013 at 10:20 AM, Johannes Berg
<johannes@sipsolutions.net> wrote:
>
>>  * Redhat kernel backport support -- this should be fairly easy
>
> I addressed this, but I'm unsure of it and don't have a way to test it,
> so any testing would be appreciated (as it is in general, of course)

FWIW the LINUX_BACKPORT() stuff replaced RHEL specific changes, that
is, LINUX_BACKPORT() stuff extended the work that folks used to add
RHEL support making it more generic. This helps try to avoid clashes
with *any* Linux distribution trying to cherry pick features from the
kernel, so long as they don't use the same prefix for exported
symbols. I wonder if there's any more need for RHEL / SUSE specific
kernel checks now after LINUX_BACKPORT() stuff.

  Luis

Re: [ANN] new backports project

[Phil Perry]

On 02/04/13 18:20, Johannes Berg wrote:
>
>>   * Redhat kernel backport support -- this should be fairly easy
>

I too am very interested in support for RHEL.

We had support up to date for compat-wireless-3.5 under RHEL-6.3.

Unsurprisingly the release of RHEL-6.4 broke a whole load of stuff given 
the amount of backporting RH did into their kernel (I believe they 
backported the wireless stack from ~ kernel-3.5 into their kernel).

I (and a colleague) played with compat-drivers-2013-03-07-u.tar.bz2 to 
get alx to build under RHEL-6.4 but I never got as far as looking at 
building everything. The seemingly endless cycle of breakage on every 
RHEL release and having to fix it kind of left me losing the will to 
keep fixing it, especially as for the moment the RHEL kernel has a 
relatively recent wireless driver stack. Any solution that might ease 
this break and fix cycle would be very welcome here.

> I addressed this, but I'm unsure of it and don't have a way to test it,
> so any testing would be appreciated (as it is in general, of course)
>

I've not looked closely at what you've done, but I have an interest in 
building/packaging compat-drivers for RHEL and can possibly help test. 
If you have a snapshot that you think should build on RHEL or that I 
could use as a starting point?

Phil

Re: [ANN] new backports project

[Johannes Berg]

Hi,

> I too am very interested in support for RHEL.

:-)
I should say that I have no idea what RHEL does etc.

> We had support up to date for compat-wireless-3.5 under RHEL-6.3.
> 
> Unsurprisingly the release of RHEL-6.4 broke a whole load of stuff given 
> the amount of backporting RH did into their kernel (I believe they 
> backported the wireless stack from ~ kernel-3.5 into their kernel).

Makes sense.

> I (and a colleague) played with compat-drivers-2013-03-07-u.tar.bz2 to 
> get alx to build under RHEL-6.4 but I never got as far as looking at 
> building everything. The seemingly endless cycle of breakage on every 
> RHEL release and having to fix it kind of left me losing the will to 
> keep fixing it, especially as for the moment the RHEL kernel has a 
> relatively recent wireless driver stack. Any solution that might ease 
> this break and fix cycle would be very welcome here.
> 
> > I addressed this, but I'm unsure of it and don't have a way to test it,
> > so any testing would be appreciated (as it is in general, of course)
> >
> 
> I've not looked closely at what you've done, but I have an interest in 
> building/packaging compat-drivers for RHEL and can possibly help test. 
> If you have a snapshot that you think should build on RHEL or that I 
> could use as a starting point?

I only addressed what compat-drivers-... had a few days ago.

However, I simplified the system a bit. I don't have any snapshot, but I
did announce the git tree. Grab that, run ./gentree.py against
linux-next and you should get something. Then you will have to hack up
the Kconfig (backport/compat/Kconfig) and include files
(backport/include/*), presumably.

If you're interested in alx, you'd have to add that -- look at the
copy-list.alx file for that.

I hope that soon enough Luis might cut some testing releases from this
though.

johannes

Re: [ANN] new backports project

[Luis R. Rodriguez]

On Tue, Apr 2, 2013 at 2:23 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
> I only addressed what compat-drivers-... had a few days ago.
>
> However, I simplified the system a bit. I don't have any snapshot, but I
> did announce the git tree. Grab that, run ./gentree.py against
> linux-next and you should get something. Then you will have to hack up
> the Kconfig (backport/compat/Kconfig) and include files
> (backport/include/*), presumably.

Or you can wait until we merge. At this point I'm freezing development
on compat and compat-drivers to merge all this work onto Johannes'
tree.

> If you're interested in alx, you'd have to add that -- look at the
> copy-list.alx file for that.

Johannes already committed a few things here, I'll be sure to test it as well.

> I hope that soon enough Luis might cut some testing releases from this
> though.

Absolutely, full steam ahead, just give us a few days.

I should note that RHEL / SUSE / Debian all had the same issues --
running into symbols we are exporting with the same upstream name. The
move to use LINUX_BACKPORT() should help cure these issues in general
moving forward. Additionally if you really want to help in this area
please review commit 8ae96730 on compat.git specifically Andi Kleen's
work on module namespace. If we can figure out to support module
namespace *ourselves*, ie, without Linus having to carry this crap
around, then we'd solve all these issues easily, I believe we wouldn't
have to rely on LINUX_BACKPORT() and in fact that would also mean
renaming our *driver* symbols. This means for instance you can run one
drivers with your stock mac80211, another with the backported version,
should you ever have to support such crazy environement.

  Luis

Re: [ANN] new backports project

[Andy Gospodarek]

On Tue, Apr 02, 2013 at 10:11:48PM +0100, Phil Perry wrote:
> On 02/04/13 18:20, Johannes Berg wrote:
> >
> >>  * Redhat kernel backport support -- this should be fairly easy
> >
> 
> I too am very interested in support for RHEL.
> 
> We had support up to date for compat-wireless-3.5 under RHEL-6.3.
> 
> Unsurprisingly the release of RHEL-6.4 broke a whole load of stuff
> given the amount of backporting RH did into their kernel (I believe
> they backported the wireless stack from ~ kernel-3.5 into their
> kernel).
> 
> I (and a colleague) played with compat-drivers-2013-03-07-u.tar.bz2
> to get alx to build under RHEL-6.4 but I never got as far as looking
> at building everything. The seemingly endless cycle of breakage on
> every RHEL release and having to fix it kind of left me losing the
> will to keep fixing it, especially as for the moment the RHEL kernel
> has a relatively recent wireless driver stack. Any solution that
> might ease this break and fix cycle would be very welcome here.

I'm glad to hear the use of compat-wireless is at least moderately
useful on RHEL6.  A few of us have an interest in keeping compat going
on RHEL6, but none had the cycles to get compat.git fixed-up before 6.4
was released.  I'm actually in the process of posting patches for 6.3.z
as someone had created an upstream bugzilla for this, so I would
encourage anyone who wants it working on 6.4 to do the same.

> >I addressed this, but I'm unsure of it and don't have a way to test it,
> >so any testing would be appreciated (as it is in general, of course)
> >
> 
> I've not looked closely at what you've done, but I have an interest
> in building/packaging compat-drivers for RHEL and can possibly help
> test. If you have a snapshot that you think should build on RHEL or
> that I could use as a starting point?
> 

I would also like to see this -- though I guess I can probably just
clone the git repo and check it out.

Re: [ANN] new backports project

[Andy Gospodarek]

On Tue, Apr 02, 2013 at 02:42:56PM -0700, Luis R. Rodriguez wrote:
> On Tue, Apr 2, 2013 at 2:23 PM, Johannes Berg <johannes@sipsolutions.net> wrote:
> > I only addressed what compat-drivers-... had a few days ago.
> >
> > However, I simplified the system a bit. I don't have any snapshot, but I
> > did announce the git tree. Grab that, run ./gentree.py against
> > linux-next and you should get something. Then you will have to hack up
> > the Kconfig (backport/compat/Kconfig) and include files
> > (backport/include/*), presumably.
> 
> Or you can wait until we merge. At this point I'm freezing development
> on compat and compat-drivers to merge all this work onto Johannes'
> tree.
> 
> > If you're interested in alx, you'd have to add that -- look at the
> > copy-list.alx file for that.
> 
> Johannes already committed a few things here, I'll be sure to test it as well.
> 
> > I hope that soon enough Luis might cut some testing releases from this
> > though.
> 
> Absolutely, full steam ahead, just give us a few days.
> 
> I should note that RHEL / SUSE / Debian all had the same issues --
> running into symbols we are exporting with the same upstream name. The
> move to use LINUX_BACKPORT() should help cure these issues in general
> moving forward. Additionally if you really want to help in this area
> please review commit 8ae96730 on compat.git specifically Andi Kleen's
> work on module namespace. If we can figure out to support module
> namespace *ourselves*, ie, without Linus having to carry this crap
> around, then we'd solve all these issues easily, I believe we wouldn't
> have to rely on LINUX_BACKPORT() and in fact that would also mean
> renaming our *driver* symbols. This means for instance you can run one
> drivers with your stock mac80211, another with the backported version,
> should you ever have to support such crazy environement.
> 

As I was watching the patches for LINUX_BACKPORT fly past over the last
few weeks and making changes to allow RHEL to work with compat, I
wondered, "Why not add the LINUX_BACKPORT macro to each one of the
symbols backported?"

Of course that seems extremely tedious, but is there a compelling reason
why something like this should not be done -- other than possibly doing
as Luis suggests and finding a way to resurrect Andi Kleen's proposal
for module symbol namespaces -- then I think the option to begin a more
liberal use of the LINUX_BACKPORT macro is worth considering.

Re: [ANN] new backports project

[Luis R. Rodriguez]

On Thu, Apr 4, 2013 at 7:12 PM, Andy Gospodarek <andy@greyhouse.net> wrote:
> On Tue, Apr 02, 2013 at 02:42:56PM -0700, Luis R. Rodriguez wrote:
>> I should note that RHEL / SUSE / Debian all had the same issues --
>> running into symbols we are exporting with the same upstream name. The
>> move to use LINUX_BACKPORT() should help cure these issues in general
>> moving forward. Additionally if you really want to help in this area
>> please review commit 8ae96730 on compat.git specifically Andi Kleen's
>> work on module namespace. If we can figure out to support module
>> namespace *ourselves*, ie, without Linus having to carry this crap
>> around, then we'd solve all these issues easily, I believe we wouldn't
>> have to rely on LINUX_BACKPORT() and in fact that would also mean
>> renaming our *driver* symbols. This means for instance you can run one
>> drivers with your stock mac80211, another with the backported version,
>> should you ever have to support such crazy environement.
>>
>
> As I was watching the patches for LINUX_BACKPORT fly past over the last
> few weeks and making changes to allow RHEL to work with compat, I
> wondered, "Why not add the LINUX_BACKPORT macro to each one of the
> symbols backported?"

Its sort of already done and committed!

> Of course that seems extremely tedious, but is there a compelling reason
> why something like this should not be done -- other than possibly doing
> as Luis suggests and finding a way to resurrect Andi Kleen's proposal
> for module symbol namespaces -- then I think the option to begin a more
> liberal use of the LINUX_BACKPORT macro is worth considering.

Johannes some idea to help with the tedious work of doing this moving
forward for new code. I'll let him dive in when he awakens.

  Luis

Re: [ANN] new backports project

[Johannes Berg]

On Thu, 2013-04-04 at 19:29 -0700, Luis R. Rodriguez wrote:

> > Of course that seems extremely tedious, but is there a compelling reason
> > why something like this should not be done -- other than possibly doing
> > as Luis suggests and finding a way to resurrect Andi Kleen's proposal
> > for module symbol namespaces -- then I think the option to begin a more
> > liberal use of the LINUX_BACKPORT macro is worth considering.
> 
> Johannes some idea to help with the tedious work of doing this moving
> forward for new code. I'll let him dive in when he awakens.

Well my idea here was more related to the fact that right now we copy a
lot of code into compat, and it seems a bit tedious to maintain this
manually. Also, if using the 'partial backport' where you only put a
single driver into the copy-list file, we still carry everything needed
to backport any driver etc.

My idea revolves around a few observations I made:
 a) we typically copy an entire subsystem (usually a C and H file) from
the
    kernel and then add LINUX_BACKPORT() to the symbols
 b) we already need to add some configuration for this, in Kconfig (in
the new
    system anyway)

So take for example the recent addition of HDMI. Luis did basically
three things:

 1) add it to Makefile/Kconfig
 2) copy kernel/drivers/video/hdmi.c to backport/compat/hdmi.c
    and kernel/include/linux/hdmi.h to backport/include/linux/hdmi.h
 3) modify the hdmi.h file like this:

+#ifndef CPTCFG_BACKPORT_BUILD_HDMI
+#include_next <linux/hdmi.h>
+#else
...
+#define hdmi_spd_infoframe_init LINUX_BACKPORT(hdmi_spd_infoframe_init)
...
+#define hdmi_audio_infoframe_init
LINUX_BACKPORT(hdmi_audio_infoframe_init)
(etc)
#endif

I'll note that he forgot to add LINUX_BACKPORT() to
hdmi_spd_infoframe_init and hdmi_spd_infoframe_pack :-)


So with these observations, why not do the following instead:

 1) add it to Makefile/Kconfig (manually), and include the information
about the
    C/H file(s) into e.g. the Kconfig file, or a separate configuration
file. I
    wouldn't use the copy-list since we need to do some transformation
steps.

 2) At backport generation time, use that information to copy the C/H
files just
    as Luis did manually for HDMI, however, copy
kernel/include/linux/hdmi.h to
    backport/include/linux/backport_hdmi.h instead.

 3) At backport generation time, create include/linux/hdmi.h like this:

#ifndef CPTCFG_BACKPORT_BUILD_HDMI
#include_next <linux/hdmi.h>
#else
#ifndef __BACKPORT_HDMI
#define __BACKPORT_HDMI
#define hdmi_avi_infoframe_init LINUX_BACKPORT(hdmi_avi_infoframe_init)
#define hdmi_avi_infoframe_pack LINUX_BACKPORT(hdmi_avi_infoframe_pack)
...
#include <linux/backport_hdmi.h>
#endif /* __BACKPORT_HDMI */
#endif /* CPTCFG_BACKPORT_BUILD_HDMI */

The biggest 'issue' here is how to find the symbols. Luis even missed
some, as I noticed, because there's an easy algorithm for this:

sed 's/^EXPORT_SYMBOL\(_GPL\|\)(\([^)]*\)/#define \2 LINUX_BACKPORT(\2)/;t;d' < C file(s)


Basically that was my idea. Maybe we should attempt to do this instead
of backporting HDMI? It seems for HDMI this would yield a more correct
version that Luis's current patch :-D

If there are subsystems with multiple header files (not counting uapi/,
those don't declare symbols and can probably just copied as is) then we
might want to stick #undef in there before each #define.


Extending it a bit further, we could do this _after_ we did
Makefile/Kconfig "impossible" analysis. This way, HDMI would only be
copied if the copy-list included a driver that requires it, otherwise it
would be left out.

One way to configure it would be to put this into the Kconfig (as
structured comments), maybe like this:

config BACKPORT_BUILD_HDMI
	...
	#backport-C-file drivers/video/hdmi.c
	#backport-H-file include/linux/hdmi.h

That'd be easy to read and easy to see almost everything (except the
Makefile snippet) at once. Heck, we could even generate the Makefile
line automatically by just writing
	obj-$(CPTCFG_BACKPORT_BUILD_HDMI) += hdmi.o
to the end of backport/compat/Makefile ...

Thoughts?

johannes

Re: [ANN] new backports project

[Johannes Berg]

On Fri, 2013-04-05 at 12:12 +0200, Johannes Berg wrote:

> config BACKPORT_BUILD_HDMI
> 	...
> 	#backport-C-file drivers/video/hdmi.c
> 	#backport-H-file include/linux/hdmi.h

If we generate the makefile line for this automatically, we'll have to
be careful, some things should be built-in:

compat-$(CPTCFG_BACKPORT_BUILD_HDMI) += hdmi.o

while others should be modules

obj-$(CPTCFG_BACKPORT_BUILD_AVERAGE) += average.o


I think the way to distinguish this would be to check whether the
BACKPORT_BUILD_* option is "bool" (built-in or not present) or tristate
(will be made modular only)

johannes