Staging, place holder for better company/community development model

Staging, place holder for better company/community development model

[Luis R. Rodriguez]

A lot of people really hate the staging tree. I don't but let me tell
you why and I'd like to see if you concur with these particular
concrete use cases and ideas on how to further use it.

The ath9k driver came to many as a big surprise -- and since it was a
surprise we had to do the work ourselves as a team at Atheros in
closed doors, without the community's involvement until we got
something standing up and not smelling as bad. Our own change in
direction to work on things upstream can be seen later as well by the
release of the 11n Otus driver and documentation provided to
interested developers to port it to mac80211 (not to mention similar
type of work for ath5k) -- Johannes quickly then ported it and created
the ar9170 11n USB driver which is its replacement for otus and
targeted for wireless-testing. Otus is currently part of the staging
tree. While ar9170 has no 802.11n support users wishing to test 11n
USB with an open driver can use the "vendor" driver. The idea is to
minimize as time goes by the "port" effort and get things out to the
community faster.

With future devices we may want to create a better path for
integration into upstream drivers. But I also want users to get
support for the devices as soon as those devices hit shelves in the
market, maybe even before. So I would like to think of staging not
only as a place for people to put drivers which a company has no
resources to do the right job but also perhaps to _do_ the actual port
work _with_ the community together. By doing so we get devices
supported with whatever ugly piece of code makes the device run (as
long as its open, upstreamble, etc) but we also can engage with the
community on the actual engineering and future of the actual driver we
do want to support in Linux.

As time goes by hopefully staging will not be necessary as companies
(like ours) will have an immediate well defined structure for their
drivers to easily add support for further devices.

If we should take this approach -- should we send patches for wireless
staging to John, or Greg? It would still be "crap" so I don't expect
John to accept to help maintain crap but what if its crap with a clear
defined path to un-crap land?

  Luis

Re: Staging, place holder for better company/community development model

[Greg KH]

On Mon, Mar 02, 2009 at 11:14:56PM -0800, Luis R. Rodriguez wrote:
> A lot of people really hate the staging tree.

They do?  I've not gotten any complaints that I can remember about it.

> I don't but let me tell you why and I'd like to see if you concur with
> these particular concrete use cases and ideas on how to further use
> it.
> 
> The ath9k driver came to many as a big surprise -- and since it was a
> surprise we had to do the work ourselves as a team at Atheros in
> closed doors, without the community's involvement until we got
> something standing up and not smelling as bad. Our own change in
> direction to work on things upstream can be seen later as well by the
> release of the 11n Otus driver and documentation provided to
> interested developers to port it to mac80211 (not to mention similar
> type of work for ath5k) -- Johannes quickly then ported it and created
> the ar9170 11n USB driver which is its replacement for otus and
> targeted for wireless-testing. Otus is currently part of the staging
> tree. While ar9170 has no 802.11n support users wishing to test 11n
> USB with an open driver can use the "vendor" driver. The idea is to
> minimize as time goes by the "port" effort and get things out to the
> community faster.
> 
> With future devices we may want to create a better path for
> integration into upstream drivers. But I also want users to get
> support for the devices as soon as those devices hit shelves in the
> market, maybe even before. So I would like to think of staging not
> only as a place for people to put drivers which a company has no
> resources to do the right job but also perhaps to _do_ the actual port
> work _with_ the community together.

This is already happening today, with at least two different network
drivers, so I have no objection to this at all.

> By doing so we get devices supported with whatever ugly piece of code
> makes the device run (as long as its open, upstreamble, etc) but we
> also can engage with the community on the actual engineering and
> future of the actual driver we do want to support in Linux.
> 
> As time goes by hopefully staging will not be necessary as companies
> (like ours) will have an immediate well defined structure for their
> drivers to easily add support for further devices.

Sounds fine to me.

> If we should take this approach -- should we send patches for wireless
> staging to John, or Greg? It would still be "crap" so I don't expect
> John to accept to help maintain crap but what if its crap with a clear
> defined path to un-crap land?

I'll gladly take them, no objections from me.

thanks,

greg k-h

Re: Staging, place holder for better company/community development model

[Luis R. Rodriguez]

On Mon, Mar 2, 2009 at 11:30 PM, Greg KH <greg@kroah.com> wrote:
> On Mon, Mar 02, 2009 at 11:14:56PM -0800, Luis R. Rodriguez wrote:
>> A lot of people really hate the staging tree.
>
> They do?  I've not gotten any complaints that I can remember about it.

Heh.......

>> I don't but let me tell you why and I'd like to see if you concur with
>> these particular concrete use cases and ideas on how to further use
>> it.
>>
>> The ath9k driver came to many as a big surprise -- and since it was a
>> surprise we had to do the work ourselves as a team at Atheros in
>> closed doors, without the community's involvement until we got
>> something standing up and not smelling as bad. Our own change in
>> direction to work on things upstream can be seen later as well by the
>> release of the 11n Otus driver and documentation provided to
>> interested developers to port it to mac80211 (not to mention similar
>> type of work for ath5k) -- Johannes quickly then ported it and created
>> the ar9170 11n USB driver which is its replacement for otus and
>> targeted for wireless-testing. Otus is currently part of the staging
>> tree. While ar9170 has no 802.11n support users wishing to test 11n
>> USB with an open driver can use the "vendor" driver. The idea is to
>> minimize as time goes by the "port" effort and get things out to the
>> community faster.
>>
>> With future devices we may want to create a better path for
>> integration into upstream drivers. But I also want users to get
>> support for the devices as soon as those devices hit shelves in the
>> market, maybe even before. So I would like to think of staging not
>> only as a place for people to put drivers which a company has no
>> resources to do the right job but also perhaps to _do_ the actual port
>> work _with_ the community together.
>
> This is already happening today, with at least two different network
> drivers, so I have no objection to this at all.

Oh nice, which ones and what companies if you can say.

>> By doing so we get devices supported with whatever ugly piece of code
>> makes the device run (as long as its open, upstreamble, etc) but we
>> also can engage with the community on the actual engineering and
>> future of the actual driver we do want to support in Linux.
>>
>> As time goes by hopefully staging will not be necessary as companies
>> (like ours) will have an immediate well defined structure for their
>> drivers to easily add support for further devices.
>
> Sounds fine to me.
>
>> If we should take this approach -- should we send patches for wireless
>> staging to John, or Greg? It would still be "crap" so I don't expect
>> John to accept to help maintain crap but what if its crap with a clear
>> defined path to un-crap land?
>
> I'll gladly take them, no objections from me.

Alright nice, will bounce this internally now.

   Luis

Re: Staging, place holder for better company/community development model

[Greg KH]

On Mon, Mar 02, 2009 at 11:45:14PM -0800, Luis R. Rodriguez wrote:
> On Mon, Mar 2, 2009 at 11:30 PM, Greg KH <greg@kroah.com> wrote:
> > This is already happening today, with at least two different network
> > drivers, so I have no objection to this at all.
> 
> Oh nice, which ones and what companies if you can say.

I would suggest that you take a look at the -staging patches currently
in the linux-next tree to see the patches and signed-off-by markings if
you are curious :)

thanks,

greg k-h

Re: Staging, place holder for better company/community development model

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

On Mon, 2009-03-02 at 23:14 -0800, Luis R. Rodriguez wrote:

> If we should take this approach -- should we send patches for wireless
> staging to John, or Greg? It would still be "crap" so I don't expect
> John to accept to help maintain crap but what if its crap with a clear
> defined path to un-crap land?

FWIW, I would MUCH prefer if the staging crap never passed any official
mailing lists.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Staging, place holder for better company/community development model

[Greg KH]

On Sun, Mar 08, 2009 at 01:07:45PM +0100, Johannes Berg wrote:
> On Mon, 2009-03-02 at 23:14 -0800, Luis R. Rodriguez wrote:
> 
> > If we should take this approach -- should we send patches for wireless
> > staging to John, or Greg? It would still be "crap" so I don't expect
> > John to accept to help maintain crap but what if its crap with a clear
> > defined path to un-crap land?
> 
> FWIW, I would MUCH prefer if the staging crap never passed any official
> mailing lists.

What do you mean by this?  You don't ever want to see any staging
patches on any mailing lists?  That doesn't sound helpful.

I can provide a simple procmail rule if you really don't like this kind
of thing, but that seems pretty extreme...

Re: Staging, place holder for better company/community development model

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

On Sun, 2009-03-08 at 15:33 -0700, Greg KH wrote:

> > FWIW, I would MUCH prefer if the staging crap never passed any official
> > mailing lists.
> 
> What do you mean by this?  You don't ever want to see any staging
> patches on any mailing lists?  That doesn't sound helpful.

Staging drivers will typically be an extremely huge amount of crap that
just floods mailing lists. I don't care about those I don't read, but I
would prefer to not have wireless staging drivers cross the wireless
list which makes it seem like somebody is actually interested in those
drivers.

> I can provide a simple procmail rule if you really don't like this kind
> of thing, but that seems pretty extreme...

I can very well filter on my own, thank you; I'm more worried about the
noise for random third parties who then infer that this is something
that somebody is working on and that might be useful to work on for
them. Neither of that is true for any wireless staging driver, they all
require complete rewrites that can, imo, use the old code only for
reference, not for implementation.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Staging, place holder for better company/community development model

[Greg KH]

On Mon, Mar 09, 2009 at 08:21:22AM +0100, Johannes Berg wrote:
> On Sun, 2009-03-08 at 15:33 -0700, Greg KH wrote:
> 
> > > FWIW, I would MUCH prefer if the staging crap never passed any official
> > > mailing lists.
> > 
> > What do you mean by this?  You don't ever want to see any staging
> > patches on any mailing lists?  That doesn't sound helpful.
> 
> Staging drivers will typically be an extremely huge amount of crap that
> just floods mailing lists. I don't care about those I don't read, but I
> would prefer to not have wireless staging drivers cross the wireless
> list which makes it seem like somebody is actually interested in those
> drivers.

I was asked by the wireless maintainer and some of the wireless
developers to send such patches to the list, so that everyone knows
exactly what is going on in the staging tree.

If you don't like this, then just ignore them.

> > I can provide a simple procmail rule if you really don't like this kind
> > of thing, but that seems pretty extreme...
> 
> I can very well filter on my own, thank you; I'm more worried about the
> noise for random third parties who then infer that this is something
> that somebody is working on and that might be useful to work on for
> them.  Neither of that is true for any wireless staging driver, they
> all require complete rewrites that can, imo, use the old code only for
> reference, not for implementation.

That's not true, some of them are being reworked as you type, to be a
"real" wireless driver eventually.

Incremental development over time is sometimes a good thing, instead of
total rewrites :)

thanks,

greg k-h

Re: Staging, place holder for better company/community development model

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

On Mon, 2009-03-09 at 12:41 -0700, Greg KH wrote:

> > Staging drivers will typically be an extremely huge amount of crap that
> > just floods mailing lists. I don't care about those I don't read, but I
> > would prefer to not have wireless staging drivers cross the wireless
> > list which makes it seem like somebody is actually interested in those
> > drivers.
> 
> I was asked by the wireless maintainer and some of the wireless
> developers to send such patches to the list, so that everyone knows
> exactly what is going on in the staging tree.

This is a misrepresentation, a number of people have asked for this on
specific drivers they are actively working on and that already are in
usable shape.

> That's not true, some of them are being reworked as you type, to be a
> "real" wireless driver eventually.
> 
> Incremental development over time is sometimes a good thing, instead of
> total rewrites :)

Except it doesn't work for most of the wireless drivers you've sucked in
without asking any wireless developers whether that makes any sense or
not.

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Staging, place holder for better company/community development model

[Greg KH]

On Mon, Mar 09, 2009 at 10:20:33PM +0100, Johannes Berg wrote:
> On Mon, 2009-03-09 at 12:41 -0700, Greg KH wrote:
> 
> > > Staging drivers will typically be an extremely huge amount of crap that
> > > just floods mailing lists. I don't care about those I don't read, but I
> > > would prefer to not have wireless staging drivers cross the wireless
> > > list which makes it seem like somebody is actually interested in those
> > > drivers.
> > 
> > I was asked by the wireless maintainer and some of the wireless
> > developers to send such patches to the list, so that everyone knows
> > exactly what is going on in the staging tree.
> 
> This is a misrepresentation, a number of people have asked for this on
> specific drivers they are actively working on and that already are in
> usable shape.

Ok, so I'll be glad to not cc: the linux-wireless list if the maintainer
asks me to.

> > That's not true, some of them are being reworked as you type, to be a
> > "real" wireless driver eventually.
> > 
> > Incremental development over time is sometimes a good thing, instead of
> > total rewrites :)
> 
> Except it doesn't work for most of the wireless drivers you've sucked in
> without asking any wireless developers whether that makes any sense or
> not.

Any specific examples?

thanks,

greg k-h

Re: Staging, place holder for better company/community development model

[Johannes Berg]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

On Tue, 2009-03-10 at 22:00 -0700, Greg KH wrote:

> > Except it doesn't work for most of the wireless drivers you've sucked in
> > without asking any wireless developers whether that makes any sense or
> > not.
> 
> Any specific examples?

wlan-ng is the largest example, it's a load of crap, much of it being a
driver for hardware we already have drivers for, and the remaining
hardware being mostly unavailable these days. The driver served it's
purpose at a time, but the authors had years and years of time to bring
it in and never bothered. It needs to be incorporated into a
rearchitectured orinoco driver, or so.

I also remember objecting to rt2860/rt2870, and the only good thing that
has come out of adding those is a spatch that might otherwise not have
been applied to those drivers. Afaict, no new people have helped with
rt2800, the rt2x00 based driver for this hardware, because they've come
in contact with rt2860/rt2870.

I don't remember any discussion about rtl8187se either.

All of those bring their own 802.11 stack, and changing to the Linux one
will /necessarily/ require an entire rewrite of the drivers because the
stacks operate /completely/ differently. Even the clean, in-kernel
bcm43xx driver was rewritten to b43 for mac80211, and rtl8187se ships
the old ieee80211_softmac code that I and a few others wrote...

johannes

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Staging, place holder for better company/community development model

[Greg KH]

On Wed, Mar 11, 2009 at 10:08:11AM +0100, Johannes Berg wrote:
> On Tue, 2009-03-10 at 22:00 -0700, Greg KH wrote:
> 
> > > Except it doesn't work for most of the wireless drivers you've sucked in
> > > without asking any wireless developers whether that makes any sense or
> > > not.
> > 
> > Any specific examples?
> 
> wlan-ng is the largest example, it's a load of crap, much of it being a
> driver for hardware we already have drivers for, and the remaining
> hardware being mostly unavailable these days. The driver served it's
> purpose at a time, but the authors had years and years of time to bring
> it in and never bothered. It needs to be incorporated into a
> rearchitectured orinoco driver, or so.

So you are objecting to the fact that some people are spending their own
time cleaning up this driver, which is only enabled for the devices that
are not supported by any other Linux wireless driver, because the
original developers never took the time to do the cleaning up
themselves?

Why do you want to tell others how to spend their time, when they are
trying to get devices to work properly with Linux that otherwise would
not?

> I also remember objecting to rt2860/rt2870, and the only good thing that
> has come out of adding those is a spatch that might otherwise not have
> been applied to those drivers. Afaict, no new people have helped with
> rt2800, the rt2x00 based driver for this hardware, because they've come
> in contact with rt2860/rt2870.

So you think that the staging tree is pulling away developer help for
the other "proper" kernel drivers?

> I don't remember any discussion about rtl8187se either.

Was there supposed to be some?

I sure discussed it, as I have hardware here that needs it, yet trying
to get it working in the in-kernel driver was pretty much impossible.
So I added the staging driver and lots of users are now happy that Linux
works on their hardware and they don't have to go download some random
driver to get it to work.

> All of those bring their own 802.11 stack, and changing to the Linux one
> will /necessarily/ require an entire rewrite of the drivers because the
> stacks operate /completely/ differently.

I agree, and don't see how this is a problem.

> Even the clean, in-kernel bcm43xx driver was rewritten to b43 for
> mac80211, and rtl8187se ships the old ieee80211_softmac code that I
> and a few others wrote...

Yes, these drivers are going to take a lot of work to get into "proper"
mergable shape.  But that's the whole point of the staging tree!  To do
this work, in the kernel, with lots of other developers, all the while
letting real people use their hardware with Linux today.

thanks,

greg k-h

Re: Staging, place holder for better company/community development model

[J.R. Mauro]

On Wed, Mar 11, 2009 at 12:02 PM, Greg KH <greg@kroah.com> wrote:
> On Wed, Mar 11, 2009 at 10:08:11AM +0100, Johannes Berg wrote:
>> On Tue, 2009-03-10 at 22:00 -0700, Greg KH wrote:
>>
>> > > Except it doesn't work for most of the wireless drivers you've sucked in
>> > > without asking any wireless developers whether that makes any sense or
>> > > not.

That doesn't mean staging is useless, nor does it imply that no one
wants the drivers or wants to work on them. A lot of the drivers in
staging are getting the attention they need, it's not like things are
just getting dropped in there and abandoned. Staging has only been
around for a few months now, so it's really too early to say whether
it will work or not... and hey, maybe it won't work out, but I think
it has a lot of promise, and has already had real results. It's just
lacking momentum and autonomy because it's a new thing.

It looks like the solution to your particular problem would be a
mailing list devoted to staging. I know I'd like to see one just
because I miss a lot of patches that go in until I pull from Greg, and
it would be nice for the discussion to be centralized. You could steer
clear of such a list, and we could work on all the drivers until we
think they're ready, and only then pass it by the relevant
maintainers.

Greg, does a special list sound like a good idea? The driverdev list
seems really low-traffic, maybe we could use that?

>> >
>> > Any specific examples?
>>
>> wlan-ng is the largest example, it's a load of crap, much of it being a
>> driver for hardware we already have drivers for, and the remaining
>> hardware being mostly unavailable these days. The driver served it's
>> purpose at a time, but the authors had years and years of time to bring
>> it in and never bothered. It needs to be incorporated into a
>> rearchitectured orinoco driver, or so.
>
> So you are objecting to the fact that some people are spending their own
> time cleaning up this driver, which is only enabled for the devices that
> are not supported by any other Linux wireless driver, because the
> original developers never took the time to do the cleaning up
> themselves?
>
> Why do you want to tell others how to spend their time, when they are
> trying to get devices to work properly with Linux that otherwise would
> not?
>
>> I also remember objecting to rt2860/rt2870, and the only good thing that
>> has come out of adding those is a spatch that might otherwise not have
>> been applied to those drivers. Afaict, no new people have helped with
>> rt2800, the rt2x00 based driver for this hardware, because they've come
>> in contact with rt2860/rt2870.
>
> So you think that the staging tree is pulling away developer help for
> the other "proper" kernel drivers?
>
>> I don't remember any discussion about rtl8187se either.
>
> Was there supposed to be some?
>
> I sure discussed it, as I have hardware here that needs it, yet trying
> to get it working in the in-kernel driver was pretty much impossible.
> So I added the staging driver and lots of users are now happy that Linux
> works on their hardware and they don't have to go download some random
> driver to get it to work.
>
>> All of those bring their own 802.11 stack, and changing to the Linux one
>> will /necessarily/ require an entire rewrite of the drivers because the
>> stacks operate /completely/ differently.
>
> I agree, and don't see how this is a problem.
>
>> Even the clean, in-kernel bcm43xx driver was rewritten to b43 for
>> mac80211, and rtl8187se ships the old ieee80211_softmac code that I
>> and a few others wrote...
>
> Yes, these drivers are going to take a lot of work to get into "proper"
> mergable shape.  But that's the whole point of the staging tree!  To do
> this work, in the kernel, with lots of other developers, all the while
> letting real people use their hardware with Linux today.

And it /is/ working to attract people both to help and to encourage
people to get their drivers in. Staging would be really nice for
bringing in new drivers so that we can prevent things like wonky 80211
stacks being used and not ported simply because outside driver project
don't have the time and manpower for it. It's entirely too early to
see some of the benefits that staging hasn't yet brought to the table
that I think it will. The two problems I see are that a lot of people
don't know about it yet, and we just obviously need to sort out the
logistics to avoid impacting people who don't want to be impacted.

Hopefully making staging a little more autonomous with its own mailing
list would help; people who want to pitch in know where to find it,
people who don't want to look at code until its ready don't even have
to know it exists. In the meantime, we can get old, crappy drivers
into shape (eventually), and prevent new drivers from going down the
wrong path or dying out. And in the meantime, people don't have to
scrounge for support for their devices.

>
> thanks,
>
> greg k-h
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Staging, place holder for better company/community development model

[Greg KH]

On Wed, Mar 11, 2009 at 03:33:21PM -0400, J.R. Mauro wrote:
> Greg, does a special list sound like a good idea? The driverdev list
> seems really low-traffic, maybe we could use that?

Yes, we should use that, I'll work on that this week.

thanks,

greg k-h

Re: Staging, place holder for better company/community development model

[Leon Woestenberg]

Hello,

On Tue, Mar 3, 2009 at 8:14 AM, Luis R. Rodriguez <mcgrof@gmail.com> wrote:
>
> With future devices we may want to create a better path for
> integration into upstream drivers. But I also want users to get
> support for the devices as soon as those devices hit shelves in the
> market, maybe even before.
>
The situation prior to the staging/ tree, was that niche drivers often
lived outside of the kernel tree, without much exposure and without
API consistency.

With staging/ available, there should be no reason for either a
company or individual(s) to develop a Linux driver outside the kernel
tree.

A driver can live there until enough momentum (devices in the field,
in Linux developer hands) exists to have the driver reworked for main
tree clearance.

At least, with this in the back of my head I submitted my driver to
staging which would otherwise live in a secret place. I have contacted
the hardware vendor and they are (not yet) interested in Linux
support. I am counting the days before they will.

Regards,
-- 
Leon