Re: Backporting other subsystems on Linux

Re: Backporting other subsystems on Linux

[Luis R. Rodriguez]

Increasing audience.

On Wed, Feb 02, 2011 at 01:27:10PM -0800, Zhao, Shanyu wrote:
> Hi Luis,
> 
> Thank you for the quick response. When I say USB, I didn't mean USB network
> drivers, I'm talking about USB host/device controller drivers or gadget
> drivers. For example, we can backport USB3 support to, say, 2.6.28.
> 
> From what I understand, I can add any driver to your compat-wireless tree and
> build it the same way as other wireless drivers. Am I right?

That's right.

> Then your tree should really be called compat-drivers. :) I'm not sure if
> you're welcoming non-network related drivers.

I see the value in providing backport support for not just 802.11 but
other drivers as well. Today compat-wireless backports: 802.11, Bluetooth,
Ethernet, and because some of these drivers require larger components like
the Sonic Silicon Backplane (ssb) for b44 an b43, it means it also backports
a bit more.

I would be hesitant to allow for more drivers to go into compat-wireless if
it was not for the fact that I am actually getting a good slew of contributions
from other members and this helps me maintain this thing. So -- if are willing
to upkeep backport support for other non 802.11/Bluetooth/Ethernet drivers
in compat-wireless, I gladly welcome the changes.

> What's the routine of maintaining a driver in compat-xxx?

I use linux-next.git to suck out drivers and we apply backport stuff to them
during a kernel development cycle. Generic kernel backport crap goes into the
compat.git tree: new request_firmware API got added recently, so we have a
compat_firmware_class, and respective udev changes. Things that are specific to
the stuff you are backport get stuffed into the compat-wireless.git tree.
Typically this involves patches which apply onto the linux-next.git code we
suck out which we cannot backport through compat.git. Preference is to always
push for compat.git changes over some nasty patch with ifdefs, and only leave
as a last resort uses patches on compat-wireless. The patches in
compat-wireless patches/ directory are sorted by the API change that requires
patching the kernel. This helps add support for new drivers that we want to add
to compat-wireless. The offsets for patches are adjusted regularly through some
Quilt magic that Hauke Mehrtens implemented (./scripts/admin-update refresh).
When crap fails to compile we either remove the driver or fix it. PCMCIA for
example was deemed as pointless to backport on our last Linux wireless summit
so there is some desire to remove all that crap but so far no one has so we go
on with it.

> Do you periodically
> run compat-xxx along with compat and latest kernel (mainstream) and fix any
> patch that no longer applies cleanly? How about stable kernel like 2.6.35.3?
> How do you align compat, compat-wireless and mainstream kernel?

Apart from bleeding edge code from linux-next.git, I also make branches for
both compat-wireless.git and compat.git for each stable kernel release. This
follows the model H. Peter Anvin has implemented in the maintenance of the
linux-2.6-allstable.git tree. When a branch there say on linux-2.6.36.y gets a new
extraversion and I know it has some juicy stuff I update my linux-2.6.36.y trees
as well and make a release through the stable compat-wireless page:

http://wireless.kernel.org/en/users/Download/stable/

It used to be that we only made bleeding edge releases based on linux-next.git but
that proved too unstable for some users.

Check out the ChangeLog for the latest stable compat-wireless, I have automated
the ChangeLog release to include the git shortlog of all components involved
between each stable kernel release.

The next question you have which you have not asked me yet is how do you address
cherry picks or patches not upstream. For that I have added three directories,
they are self described by their names with a URL to their respective README:

  * linux-next-cherry-picks/ http://bit.ly/h76wrL 
  * linux-next-pending/      http://bit.ly/eY4aCY 
  * pending-stable/          http://bit.ly/dOsi7J 

What I really need to get to at some point is to use implement a
menuconfig thingy, we already drag the Kconfigs in but use config.mk for
makefile/dependency mapping. Patches welcomed.

  Luis

Re: Backporting other subsystems on Linux

[Luis R. Rodriguez]

On Wed, Feb 02, 2011 at 02:09:26PM -0800, Luis Rodriguez wrote:
> Increasing audience.
> 
> On Wed, Feb 02, 2011 at 01:27:10PM -0800, Zhao, Shanyu wrote:
> > Hi Luis,
> > 
> > Thank you for the quick response. When I say USB, I didn't mean USB network
> > drivers, I'm talking about USB host/device controller drivers or gadget
> > drivers. For example, we can backport USB3 support to, say, 2.6.28.
> > 
> > From what I understand, I can add any driver to your compat-wireless tree and
> > build it the same way as other wireless drivers. Am I right?
> 
> That's right.
> 
> > Then your tree should really be called compat-drivers. :) I'm not sure if
> > you're welcoming non-network related drivers.
> 
> I see the value in providing backport support for not just 802.11 but
> other drivers as well. Today compat-wireless backports: 802.11, Bluetooth,
> Ethernet, and because some of these drivers require larger components like
> the Sonic Silicon Backplane (ssb) for b44 an b43, it means it also backports
> a bit more.
> 
> I would be hesitant to allow for more drivers to go into compat-wireless if
> it was not for the fact that I am actually getting a good slew of contributions
> from other members and this helps me maintain this thing. So -- if are willing
> to upkeep backport support for other non 802.11/Bluetooth/Ethernet drivers
> in compat-wireless, I gladly welcome the changes.
> 
> > What's the routine of maintaining a driver in compat-xxx?
> 
> I use linux-next.git to suck out drivers and we apply backport stuff to them
> during a kernel development cycle. Generic kernel backport crap goes into the
> compat.git tree: new request_firmware API got added recently, so we have a
> compat_firmware_class, and respective udev changes. Things that are specific to
> the stuff you are backport get stuffed into the compat-wireless.git tree.
> Typically this involves patches which apply onto the linux-next.git code we
> suck out which we cannot backport through compat.git. Preference is to always
> push for compat.git changes over some nasty patch with ifdefs, and only leave
> as a last resort uses patches on compat-wireless. The patches in
> compat-wireless patches/ directory are sorted by the API change that requires
> patching the kernel. This helps add support for new drivers that we want to add
> to compat-wireless. The offsets for patches are adjusted regularly through some
> Quilt magic that Hauke Mehrtens implemented (./scripts/admin-update refresh).
> When crap fails to compile we either remove the driver or fix it. PCMCIA for
> example was deemed as pointless to backport on our last Linux wireless summit
> so there is some desire to remove all that crap but so far no one has so we go
> on with it.
> 
> > Do you periodically
> > run compat-xxx along with compat and latest kernel (mainstream) and fix any
> > patch that no longer applies cleanly? How about stable kernel like 2.6.35.3?
> > How do you align compat, compat-wireless and mainstream kernel?
> 
> Apart from bleeding edge code from linux-next.git, I also make branches for
> both compat-wireless.git and compat.git for each stable kernel release. This
> follows the model H. Peter Anvin has implemented in the maintenance of the
> linux-2.6-allstable.git tree. When a branch there say on linux-2.6.36.y gets a new
> extraversion and I know it has some juicy stuff I update my linux-2.6.36.y trees
> as well and make a release through the stable compat-wireless page:
> 
> http://wireless.kernel.org/en/users/Download/stable/
> 
> It used to be that we only made bleeding edge releases based on linux-next.git but
> that proved too unstable for some users.
> 
> Check out the ChangeLog for the latest stable compat-wireless, I have automated
> the ChangeLog release to include the git shortlog of all components involved
> between each stable kernel release.
> 
> The next question you have which you have not asked me yet is how do you address
> cherry picks or patches not upstream. For that I have added three directories,
> they are self described by their names with a URL to their respective README:
> 
>   * linux-next-cherry-picks/ http://bit.ly/h76wrL 
>   * linux-next-pending/      http://bit.ly/eY4aCY 
>   * pending-stable/          http://bit.ly/dOsi7J 
> 
> What I really need to get to at some point is to use implement a
> menuconfig thingy, we already drag the Kconfigs in but use config.mk for
> makefile/dependency mapping. Patches welcomed.

BTW one idea I have as of recent is to consider using Hudson for automatic
build testing for compat-wireless but haven't yet mucked with it. Eventually
this will be more and more important as time goes by and we support older
kernels. The goal is to at the *very least* always support down to the last
stable 2.6 kernel listed on kernel.org, today that is 2.6.27 and I think its
a safe promise to commit to backporting *at least* to that kernel. We go
beyond 2.6.27 though today.

[1] http://en.wikipedia.org/wiki/Hudson_(software)

  Luis

RE: Backporting other subsystems on Linux

[Zhao, Shanyu]

Hi Luis,

Thank you for the detailed explanation! 

I just started to explore potential options of backporting USB subsystems. Currently I'm not at the point where I'm ready to backport drivers. I'll work on that later.

>> The next question you have which you have not asked me yet is how do you
>address
>> cherry picks or patches not upstream. For that I have added three
>directories,
>> they are self described by their names with a URL to their respective
>README:
>>
>>   * linux-next-cherry-picks/ http://bit.ly/h76wrL
>>   * linux-next-pending/      http://bit.ly/eY4aCY
>>   * pending-stable/          http://bit.ly/dOsi7J
>>
>> What I really need to get to at some point is to use implement a
>> menuconfig thingy, we already drag the Kconfigs in but use config.mk for
>> makefile/dependency mapping. Patches welcomed.
Well, I actually didn't have that question in mind. :) I read the README files in these folders and I totally agree that there should not be patches under these 3 folders. I think compat-wireless should just fulfill the function of backporting a driver from a (stable) kernel to an old kernel. Not adding additional patches. Driver developers should strive to make important patches into the stable releases. 

>
>BTW one idea I have as of recent is to consider using Hudson for automatic
>build testing for compat-wireless but haven't yet mucked with it. Eventually
>this will be more and more important as time goes by and we support older
>kernels. The goal is to at the *very least* always support down to the last
>stable 2.6 kernel listed on kernel.org, today that is 2.6.27 and I think its
>a safe promise to commit to backporting *at least* to that kernel. We go
>beyond 2.6.27 though today.
This is a good idea!

Shanyu

Re: Backporting other subsystems on Linux

[Hauke Mehrtens]

On 02/02/2011 11:26 PM, Luis R. Rodriguez wrote:
> On Wed, Feb 02, 2011 at 02:09:26PM -0800, Luis Rodriguez wrote:
>> Increasing audience.
>>
>> On Wed, Feb 02, 2011 at 01:27:10PM -0800, Zhao, Shanyu wrote:
>>> Hi Luis,
>>>
>>> Thank you for the quick response. When I say USB, I didn't mean USB network
>>> drivers, I'm talking about USB host/device controller drivers or gadget
>>> drivers. For example, we can backport USB3 support to, say, 2.6.28.
>>>
>>> From what I understand, I can add any driver to your compat-wireless tree and
>>> build it the same way as other wireless drivers. Am I right?
>>
>> That's right.
>>
>>> Then your tree should really be called compat-drivers. :) I'm not sure if
>>> you're welcoming non-network related drivers.

When adding more and more stuff we should probably rename
compat-wireless to something else. ;-)
I am fine with adding USB3 support, if Intel will help maintaining the
stuff in compat-wireless after they added it.

>> What I really need to get to at some point is to use implement a
>> menuconfig thingy, we already drag the Kconfigs in but use config.mk for
>> makefile/dependency mapping. Patches welcomed.
> 
> BTW one idea I have as of recent is to consider using Hudson for automatic
> build testing for compat-wireless but haven't yet mucked with it. Eventually
> this will be more and more important as time goes by and we support older
> kernels. The goal is to at the *very least* always support down to the last
> stable 2.6 kernel listed on kernel.org, today that is 2.6.27 and I think its
> a safe promise to commit to backporting *at least* to that kernel. We go
> beyond 2.6.27 though today.

Automatic compile testing is really needed for compat-wireless. There
are often some special kernel version where it does not compile, like
now. ;-)

Some time ago I started writing a small bash script which tries to build
compat-wireless for various kernel versions and sends a email if
something went wrong. This was more or less hacked together from the
compat-wireless release script and some other scripts I have here. I
stopped working on this after I recognized that my server is way to slow
for running this script (compat-wireless build for one kernel toke more
than 30 minutes).

Using hudson or buildbot [2] is the better choice, but it should just
take 1 to 2 hours to get this hacked script working. Luis do you have a
server with hudson installed or a server capable for running hudson and
building compat-wireless? My server has a Intel Celeron with 2.4 GHz and
512MB Ram. I have to clean up my script before releasing it to someone
else, but if someone is interest I have no problem with making it relay
for use.

[1] http://en.wikipedia.org/wiki/Hudson_(software)
[2] http://en.wikipedia.org/wiki/BuildBot

Hauke

Re: Backporting other subsystems on Linux

[Luis R. Rodriguez]

On Thu, Feb 3, 2011 at 1:15 PM, Hauke Mehrtens <hauke@hauke-m.de> wrote:
> On 02/02/2011 11:26 PM, Luis R. Rodriguez wrote:
>> On Wed, Feb 02, 2011 at 02:09:26PM -0800, Luis Rodriguez wrote:
>>> Increasing audience.
>>>
>>> On Wed, Feb 02, 2011 at 01:27:10PM -0800, Zhao, Shanyu wrote:
>>>> Hi Luis,
>>>>
>>>> Thank you for the quick response. When I say USB, I didn't mean USB network
>>>> drivers, I'm talking about USB host/device controller drivers or gadget
>>>> drivers. For example, we can backport USB3 support to, say, 2.6.28.
>>>>
>>>> From what I understand, I can add any driver to your compat-wireless tree and
>>>> build it the same way as other wireless drivers. Am I right?
>>>
>>> That's right.
>>>
>>>> Then your tree should really be called compat-drivers. :) I'm not sure if
>>>> you're welcoming non-network related drivers.
>
> When adding more and more stuff we should probably rename
> compat-wireless to something else. ;-)

:) sure. compat-kernel ? Only reason I stopped adding more compat
stuff was I found ALSA folks already had their own backport effort and
that even goes down to 2.4. The style is a bit different though too. I
did try to do my own ALSA backport ala compat-wireless and I do have
that git tree still up on git.kernel.org. I then considered trying
video but Ubuntu folks did the whole in-kernel DRM backport so I lost
motivation as they had that taken care of. If we do want more kernel
stuff backported all together we essentially will just need people
taking ownership of helping upkeep other subsystems. I will welcome
patches :)

> I am fine with adding USB3 support, if Intel will help maintaining the
> stuff in compat-wireless after they added it.

Agreed.

>>> What I really need to get to at some point is to use implement a
>>> menuconfig thingy, we already drag the Kconfigs in but use config.mk for
>>> makefile/dependency mapping. Patches welcomed.
>>
>> BTW one idea I have as of recent is to consider using Hudson for automatic
>> build testing for compat-wireless but haven't yet mucked with it. Eventually
>> this will be more and more important as time goes by and we support older
>> kernels. The goal is to at the *very least* always support down to the last
>> stable 2.6 kernel listed on kernel.org, today that is 2.6.27 and I think its
>> a safe promise to commit to backporting *at least* to that kernel. We go
>> beyond 2.6.27 though today.
>
> Automatic compile testing is really needed for compat-wireless. There
> are often some special kernel version where it does not compile, like
> now. ;-)
>
> Some time ago I started writing a small bash script which tries to build
> compat-wireless for various kernel versions and sends a email if
> something went wrong. This was more or less hacked together from the
> compat-wireless release script and some other scripts I have here. I
> stopped working on this after I recognized that my server is way to slow
> for running this script (compat-wireless build for one kernel toke more
> than 30 minutes).
>
> Using hudson or buildbot [2] is the better choice, but it should just
> take 1 to 2 hours to get this hacked script working. Luis do you have a
> server with hudson installed or a server capable for running hudson and
> building compat-wireless? My server has a Intel Celeron with 2.4 GHz and
> 512MB Ram. I have to clean up my script before releasing it to someone
> else, but if someone is interest I have no problem with making it relay
> for use.

Yes, I do. I also have a Hudson demo to attend tomorrow that Lance is
putting together so I can also likely pick his brain. Let me try to
set that up.

  Luis