Idea: Automatic binary driver compiling system

Idea: Automatic binary driver compiling system

[Benjamin Bach]

Hi all,

First off: I'm new on this list. I'm also new at starting projects. 
Anyways, I've decided on this idea for my first year computer science 
project. I have three months to set it off and definitely the whole 
thing would be released into the world afterwards.

Second: I don't want opinions on the issue of making it easier for 
companies to create binary (pre-compiled) drivers. Yes, we all want more 
open source drivers. Of course.

OK, so here goes: According to Distrowatch there's currently 377 
different distributions out. We have multiple architectures and fresh 
kernel patches every week. If we multiply these numbers, we'd find that 
a company wanting to release a closed-source driver module for Linux 
would face - say - 5,000 compilations a month. Impossible. So most 
companies just release largely incompatible binary drivers in a 
frustratingly limited variety.

I've been looking around for material about the topic of binary Linux 
drivers and creating such in a broad-scale batch mode. But I haven't 
found any. Oh yes, except for Documentation/stable_api_nonsens.txt which 
I found very helpful. But the issue is not having a stable interface - 
rather it is about having a good debugging and building tool that 
handles very very large amounts of kernel versions and patches.

I would be very grateful for pointers to any helpful resource and also 
thoughts on what problems I'm facing. Also I'm aware that this is 
probably not the first time "batch module compiling" has been mentioned... ?

Sincerely,
Benjamin Bach

Re: Idea: Automatic binary driver compiling system

[Christoph Lameter]

On Fri, 17 Mar 2006, Benjamin Bach wrote:

> I would be very grateful for pointers to any helpful resource and also
> thoughts on what problems I'm facing. Also I'm aware that this is probably not
> the first time "batch module compiling" has been mentioned... ?

"make modules" build all modules. You can batch it with nohup?

Re: Idea: Automatic binary driver compiling system

[Lee Revell]

On Fri, 2006-03-17 at 19:00 +0100, Benjamin Bach wrote:
> Hi all,
> 
> First off: I'm new on this list. I'm also new at starting projects. 
> Anyways, I've decided on this idea for my first year computer science 
> project. I have three months to set it off and definitely the whole 
> thing would be released into the world afterwards.
> 
> Second: I don't want opinions on the issue of making it easier for 
> companies to create binary (pre-compiled) drivers. Yes, we all want more 
> open source drivers. Of course.

Then let's end the thread here because this is the crux of ther issue -
no one WANTS it to be easier for vendors to ship binary drivers.

Linux will not change to accomodate vendors desire for binary drivers.
The vendors must adapt to Linux.  End of story.

Lee

Re: Idea: Automatic binary driver compiling system

[Benjamin Bach]

Lee Revell wrote:
> Then let's end the thread here because this is the crux of ther issue -
> no one WANTS it to be easier for vendors to ship binary drivers.
>
> Linux will not change to accomodate vendors desire for binary drivers.
> The vendors must adapt to Linux.  End of story.
>   
I see two pictures:

1) Open source does not accommodate binary drivers. In that way hardware 
support is weakened but an increasing number of companies should adopt 
open source drivers because demand increases even without super-duper 
hardware support.

2) Open source does in fact accommodate binary drivers because some 
idiot (me) just took the liberty of starting this project just to see 
what happens. It's exactly the same as when some people go and fork a 
project and everybody says it'll weaken the movement. But something 
could happen: It could be beneficial! Linux get's better hardware 
support, more people adopt it and business begins to realize that maybe 
they should try to co-operate a bit more software-wise. That's what we 
all want them to do, yes?

As of now Linux won't force vendors into creating open source drivers. 
That'd be like a small dog barking.

I here every word when people talk about the benefits of open source 
drivers. But really, think: If all these benefits why isn't there 
anymore open source drivers? And the answer is not "because companies 
are stupid". If we can't make it attractive enough to release driver 
source code then maybe the world just isn't matured yet. And we'll wait. 
But while waiting I want my 3d card to perform better and my mp3-player 
to work.

...

Bugga, here we went and turned it into opinions =)



/ Benjamin

Re: Idea: Automatic binary driver compiling system

[Arjan van de Ven]

> As of now Linux won't force vendors into creating open source drivers. 

it has happened several times already. So your claim is false.

> But really, think: If all these benefits why isn't there 
> anymore open source drivers? 

there are over a thousand open source drivers, and at most a handful
binary ones. Please go do your math.

Re: Idea: Automatic binary driver compiling system

[Benjamin Bach]

Arjan van de Ven wrote:
> there are over a thousand open source drivers, and at most a handful
> binary ones. Please go do your math.
>   
You're doing the wrong comparison. How many drivers are missing or
lacking in ability? And if you add to your handful of binary drivers
those thousands that exist for xp...
well, numbers do change. Also, most open source drivers aren't made by
the vendors themselves.

We're doing subjective math here. It doesn't change the fact that Linux
would be better off with improved hardware support, right?


/ Benjamin

Re: Idea: Automatic binary driver compiling system

[Arjan van de Ven]

On Sat, 2006-03-18 at 16:53 +0100, Benjamin Bach wrote:
> Arjan van de Ven wrote:
> > there are over a thousand open source drivers, and at most a handful
> > binary ones. Please go do your math.
> >   
> You're doing the wrong comparison. How many drivers are missing

not too many. This is largely because hardware interfaces are getting
increasingly standardized (it's cheaper for the hw vendors to not have
to create a new driver for Windows XP)

>  or
> lacking in ability?

some. But the vast majority is "good enough" by any standard.

>  And if you add to your handful of binary drivers
> those thousands that exist for xp...

then it's clear that linux is better off ;)
(and yes while XP has more drivers, in linux a driver would generally
drive the hardware that in the windows world uses 10 to 20  drivers)

> well, numbers do change. Also, most open source drivers aren't made by
> the vendors themselves.

and? For standard interfaces... no big deal.
And for non-standard interfaces.. it's increasingly done with the vendor
help

> 
> We're doing subjective math here. It doesn't change the fact that Linux
> would be better off with improved hardware support, right?

yes. But "more binary drivers" is absolutely not "better off"; but
that's going towards the usual bimonthly troll topic so lets not go
there and stop here.

Re: Idea: Automatic binary driver compiling system

[Benjamin Bach]

Wow, kudos to your work with the consequences of binary drivers. I 
certainly do not wish to add any redundant remarks of trolling sentences 
to this discussion. I've read +50 posts about binary drivers on this 
mailing list and in conclusion to that, I'd like to only add the following:

My idea was not to compromise the structure of the kernel. Nothing 
should be changed here. I also see a very notable resistance to binary 
drivers from distributions. Looking at the way ATI and NVidia drivers 
are treated by Fedora, SUSE and Ubuntu, I actually think they too have 
an agenda on this matter, and somehow it resembles their agenda on 
av-codecs. It's a sneaky-sneaky thing - if the user doesn't know a 
binary driver exists, we won't tell him. FC5 recently released made this 
huge "oops... we banned non-GPL modules in the kernel".

Anyways, I'm very happy with the combination of intelligence and 
idealism on this list, and suddenly I feel more attracted to writing a 
driver instead. For my Rio Karma mp3 player. It's a USB thing.. should 
be do-able in 3 months even though I've never written a driver.


Cheers everybody, and thanks for sharing! =)

/ Benjamin


Arjan van de Ven wrote:
> On Sat, 2006-03-18 at 16:53 +0100, Benjamin Bach wrote:
>   
>> Arjan van de Ven wrote:
>>     
>>> there are over a thousand open source drivers, and at most a handful
>>> binary ones. Please go do your math.
>>>   
>>>       
>> You're doing the wrong comparison. How many drivers are missing
>>     
>
> not too many. This is largely because hardware interfaces are getting
> increasingly standardized (it's cheaper for the hw vendors to not have
> to create a new driver for Windows XP)
>
>   
>>  or
>> lacking in ability?
>>     
>
> some. But the vast majority is "good enough" by any standard.
>
>   
>>  And if you add to your handful of binary drivers
>> those thousands that exist for xp...
>>     
>
> then it's clear that linux is better off ;)
> (and yes while XP has more drivers, in linux a driver would generally
> drive the hardware that in the windows world uses 10 to 20  drivers)
>
>   
>> well, numbers do change. Also, most open source drivers aren't made by
>> the vendors themselves.
>>     
>
> and? For standard interfaces... no big deal.
> And for non-standard interfaces.. it's increasingly done with the vendor
> help
>
>   
>> We're doing subjective math here. It doesn't change the fact that Linux
>> would be better off with improved hardware support, right?
>>     
>
> yes. But "more binary drivers" is absolutely not "better off"; but
> that's going towards the usual bimonthly troll topic so lets not go
> there and stop here.
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>   

Re: Idea: Automatic binary driver compiling system

[Jesper Juhl]

On 3/19/06, Benjamin Bach <benjamin@overtag.dk> wrote:
[snip]
>
> Anyways, I'm very happy with the combination of intelligence and
> idealism on this list, and suddenly I feel more attracted to writing a
> driver instead. For my Rio Karma mp3 player. It's a USB thing.. should
> be do-able in 3 months even though I've never written a driver.
>

Writing a GPL'ed driver sounds like a great project.

For a USB device I'd suggest you take a look at libusb for writing a
userspace driver as opposed to a kernelspace one. Userspace drivers
for USB devices are often preferable.

Also take a look at a few documents in the Documentation/ dir in the
kernel source:
  HOWTO
  SubmittingDrivers
  CodingStyle

If you can get docs for the device (or are able to reverse engineer
it) and get a driver done, more power to you. :-)

>
> Cheers everybody, and thanks for sharing! =)
>
> / Benjamin
>
>
> Arjan van de Ven wrote:
[snip]

Top-posting is generally frawned upon on the list as it's annoying and
makes threads hard to read. Please consider changing that habbit.


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Idea: Automatic binary driver compiling system

[Bob Copeland]

> Anyways, I'm very happy with the combination of intelligence and
> idealism on this list, and suddenly I feel more attracted to writing a
> driver instead. For my Rio Karma mp3 player. It's a USB thing.. should
> be do-able in 3 months even though I've never written a driver.
>

I've already done this and it is in 2.6.16.  There's still some work
to be done on the filesystem; check out
http://linux-karma.sourceforge.net to help out.

-Bob

Re: Idea: Automatic binary driver compiling system

[Benjamin Bach]

Bob Copeland wrote:
>> Anyways, I'm very happy with the combination of intelligence and
>> idealism on this list, and suddenly I feel more attracted to writing a
>> driver instead. For my Rio Karma mp3 player. It's a USB thing.. should
>> be do-able in 3 months even though I've never written a driver.
>>
>>     
>
> I've already done this and it is in 2.6.16.  There's still some work
> to be done on the filesystem; check out
> http://linux-karma.sourceforge.net to help out.
>
>   
Yeah, I immediately found out =) Some very good explanation on your site 
(http://bobcopeland.com/karma/). I see that you've implemented most of 
the work, and I can't really figure out any isolated development 
suitable for a school project. But please, if you have an idea, say so.

Otherwise I'll probably dig up something. Just needs to be a small 
kernel-whatever project.

Is there someone maintaining a list of non-implemented ideas for kernel 
features/drivers?

/ Benjamin

Re: Idea: Automatic binary driver compiling system

[Denis Vlasenko]

On Sunday 19 March 2006 18:12, Benjamin Bach wrote:
> Bob Copeland wrote:
> >> Anyways, I'm very happy with the combination of intelligence and
> >> idealism on this list, and suddenly I feel more attracted to writing a
> >> driver instead. For my Rio Karma mp3 player. It's a USB thing.. should
> >> be do-able in 3 months even though I've never written a driver.
> >>
> >>     
> >
> > I've already done this and it is in 2.6.16.  There's still some work
> > to be done on the filesystem; check out
> > http://linux-karma.sourceforge.net to help out.
> >
> >   
> Yeah, I immediately found out =) Some very good explanation on your site 
> (http://bobcopeland.com/karma/). I see that you've implemented most of 
> the work, and I can't really figure out any isolated development 
> suitable for a school project. But please, if you have an idea, say so.
> 
> Otherwise I'll probably dig up something. Just needs to be a small 
> kernel-whatever project.
> 
> Is there someone maintaining a list of non-implemented ideas for kernel 
> features/drivers?

Wireless stack and drivers.

http://bcm43xx.berlios.de/
http://195.66.192.167/linux/acx_patches/current_sm/README

(you will also have to checkout wireless-git tree, above README
explains how to do it)
--
vda

Re: Idea: Automatic binary driver compiling system

[Bob Copeland]

On 3/19/06, Benjamin Bach <benjamin@overtag.dk> wrote:
> Otherwise I'll probably dig up something. Just needs to be a small
> kernel-whatever project.
>
> Is there someone maintaining a list of non-implemented ideas for kernel
> features/drivers?

Although neither of these are easy and you very well might not get
anything done in three months, a couple of bits of hardware that I
have for which there are incomplete/no drivers, and where the
manufacturer refuses to give out specs are:

- Ricoh MMC/SD controllers.  The project to figure those out is at:
http://mmc.drzeus.cx/wiki/Controllers/Ricoh/Frontreport

- 3D for NVidia.  I know many people would take an open but basic 3D
driver over the fully featured binary one - many people already use
the 2D 'nv' driver for that reason.  Rudolf Cornelissen has reverse
engineered various bits of it (though it may apply only to the
geforce-1 era cards) over here:
http://web.inter.nl.net/users/be-hold/BeOS/NVdriver/3dnews.html

You will find it's a whole lot easier to write drivers when you have
specs though, and the resulting drivers will also be better.  But
depending on the scope of your project, you could definitely learn
something either way.

Another thing that would be a lot easier to accomplish in 3 months
would be to write a userspace filesystem using FUSE for something that
isn't ordinarily accessed by filesystems; for example currently you
can mount remote machines over ssh, cameras that can talk to gphoto,
tar archives, gmail, etc.

Re: Idea: Automatic binary driver compiling system

[Lee Revell]

On Mon, 2006-03-20 at 10:46 -0500, Bob Copeland wrote:
> On 3/19/06, Benjamin Bach <benjamin@overtag.dk> wrote:
> > Otherwise I'll probably dig up something. Just needs to be a small
> > kernel-whatever project.
> >
> > Is there someone maintaining a list of non-implemented ideas for kernel
> > features/drivers?
> 
> Although neither of these are easy and you very well might not get
> anything done in three months, a couple of bits of hardware that I
> have for which there are incomplete/no drivers, and where the
> manufacturer refuses to give out specs are:
> 
> - Ricoh MMC/SD controllers.  The project to figure those out is at:
> http://mmc.drzeus.cx/wiki/Controllers/Ricoh/Frontreport
> 
> - 3D for NVidia.  I know many people would take an open but basic 3D
> driver over the fully featured binary one - many people already use
> the 2D 'nv' driver for that reason.  Rudolf Cornelissen has reverse
> engineered various bits of it (though it may apply only to the
> geforce-1 era cards) over here:
> http://web.inter.nl.net/users/be-hold/BeOS/NVdriver/3dnews.html
> 

Lots of people don't even need 3D but have to run nvidia's driver to get
multihead support, it seems this would be much easier than full 3D
implementation.

> You will find it's a whole lot easier to write drivers when you have
> specs though, and the resulting drivers will also be better.  But
> depending on the scope of your project, you could definitely learn
> something either way.
> 
> Another thing that would be a lot easier to accomplish in 3 months
> would be to write a userspace filesystem using FUSE for something that
> isn't ordinarily accessed by filesystems; for example currently you
> can mount remote machines over ssh, cameras that can talk to gphoto,
> tar archives, gmail, etc.

Another easy project if you have old sound cards lying around is to port
some of the old OSS drivers to ALSA (a list was posted on LKML a while
back).  This probably will take from a weekend to a few weeks.

Lee

Re: Idea: Automatic binary driver compiling system

[Pavel Machek]

Hi!

> > Otherwise I'll probably dig up something. Just needs to be a small
> > kernel-whatever project.
> >
> > Is there someone maintaining a list of non-implemented ideas for kernel
> > features/drivers?
> 
> Although neither of these are easy and you very well might not get
> anything done in three months, a couple of bits of hardware that I
> have for which there are incomplete/no drivers, and where the
> manufacturer refuses to give out specs are:
> 
> - Ricoh MMC/SD controllers.  The project to figure those out is at:
> http://mmc.drzeus.cx/wiki/Controllers/Ricoh/Frontreport
> 
> - 3D for NVidia.  I know many people would take an open but basic 3D
> driver over the fully featured binary one - many people already use
> the 2D 'nv' driver for that reason.  Rudolf Cornelissen has reverse
> engineered various bits of it (though it may apply only to the
> geforce-1 era cards) over here:
> http://web.inter.nl.net/users/be-hold/BeOS/NVdriver/3dnews.html

Or help openzaurus project. I'd certainly need help with collie :-).

								Pavel
-- 
Picture of sleeping (Linux) penguin wanted...

Re: Idea: Automatic binary driver compiling system

[Jesper Juhl]

On 3/18/06, Benjamin Bach <benjamin@overtag.dk> wrote:
> Arjan van de Ven wrote:
> > there are over a thousand open source drivers, and at most a handful
> > binary ones. Please go do your math.
> >
> You're doing the wrong comparison. How many drivers are missing or
> lacking in ability? And if you add to your handful of binary drivers
> those thousands that exist for xp...
> well, numbers do change. Also, most open source drivers aren't made by
> the vendors themselves.
>
> We're doing subjective math here. It doesn't change the fact that Linux
> would be better off with improved hardware support, right?
>

I strongly disagree.

Linux will be better off with improved hardware support by Open Source
drivers - yes.

With improved hardware support by closed source, binary only drivers,
Linux could very well end up being a *lot* worse off.

Once companies have the abillity to easily create closed drivers it
seems very likely that more of them will start withholding specs,
making it impossible to create open drivers.
This in turn will lead to less and less hardware supported by open
drivers. The result of this will be more and more useless bugreports
on LKML making it increasingly hard to maintain and improve the
kernel.  Users will report a bug that may or may not be in an open
part of the kernel, but it will be impossible to tell due to all the
binary only drivers for the users hardware she has loaded.
Binary drivers also put the stability and security of the users system
at the mercy of a commercial entity outside the users control.  A
binary blob loaded into the kernel could do *anything* to compromise
the system or cause it to become unstable and noone would be able to
tell if it's the driver or something else - Linux stability will
suffer and users will in many cases blame Linux, not the driver
vendors.

There's also the issue of companies going out of business, taking
their closed drivers with them. Now the hardware that used to be
supported by Linux is no longer useful since noone can keep the driver
updated.  One could then opt to keep the binary interfaces static to
keep old vendor drivers working, but then we'd quickly accumulate a
bunch of backwards compatibility cruft that would bloat the kernel and
turn it into an unmaintainable mess.

For users there will be a short term bennefit of having more new and
shiny toys work on Linux, but in the longer term it could easily
destroy the usefulness of Linux as a free Operating System.

So please, let's *not* do work to make binary only closed source
drivers easier to make or maintain, it's counterproductive in the long
run.


It's a lot like peeing in your pants to try and stay warm in a
blizzard. It may feel nice initially, but a little later on when your
legs freeze up you'll be a lot worse off than you were initially.


--
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: Idea: Automatic binary driver compiling system

[Adrian Bunk]

On Sat, Mar 18, 2006 at 04:53:26PM +0100, Benjamin Bach wrote:
>...
> We're doing subjective math here. It doesn't change the fact that Linux
> would be better off with improved hardware support, right?

This is only true for the short-term future.

Whether Linux is better off with improved hardware support through 
binary modules for the long-term future is a more difficult question. 
Arjan recently posted an extreme example of the dangers of hardware 
support through binary modules [1].

You might agree or disagree with such opinions. But it has already been 
discussed to death on this list, and unless you are a troll you should 
read the past flamewars^Wdiscussions regarding this topic in the mailing 
list archives instead of restarting the same old discussion.

If you can accept the fact that less people consider your project useful 
than you might have hoped, there is no reason for you to not implement 
your project.

> / Benjamin

cu
Adrian

[1] http://lwn.net/Articles/162686/

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: Idea: Automatic binary driver compiling system

[Sander]

Benjamin Bach wrote (ao):
> Linux get's better hardware support

With the help of binary drivers? Not likely. The smart people on this
list can't help for example.

> As of now Linux won't force vendors into creating open source drivers.
> That'd be like a small dog barking.

Well, as a customer you can put your money where your mouth is, and
simply buy other hardware. That is a way of forcing, or at least
stimulating the market.

As a kernel hacker you can choose to reverse engineer.

Btw, read this thread and tell me if you still agree that binary drivers
and depending on a company are a good thing:

http://forum.matrox.com/mga/viewtopic.php?t=19208&sid=724d14d42714800f9d13ee6286f37f0b

	Kind regards, Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: Idea: Automatic binary driver compiling system

[Nix]

On 17 Mar 2006, Lee Revell murmured woefully:
> On Fri, 2006-03-17 at 19:00 +0100, Benjamin Bach wrote:
>> First off: I'm new on this list. I'm also new at starting projects. 
>> Anyways, I've decided on this idea for my first year computer science 
>> project. I have three months to set it off and definitely the whole 
>> thing would be released into the world afterwards.
>> 
>> Second: I don't want opinions on the issue of making it easier for 
>> companies to create binary (pre-compiled) drivers. Yes, we all want more 
>> open source drivers. Of course.
> 
> Then let's end the thread here because this is the crux of ther issue -
> no one WANTS it to be easier for vendors to ship binary drivers.

Well, this could be useful for distributors, too, except that distributors
already have mechanisms for autobuilding kernels and modules.

-- 
`Come now, you should know that whenever you plan the duration of your
 unplanned downtime, you should add in padding for random management
 freakouts.'