[RFC] binary firmware and modules

[RFC] binary firmware and modules

[Jon Masters]

[-- Attachment #1: Type: text/plain, Size: 1073 bytes --]

Hi folks,

Right now, various kernel modules are being migrated over to use
request_firmware in order to pull in binary firmware blobs from userland
when the module is loaded. This makes sense.

However, there is right now little mechanism in place to automatically
determine which binary firmware blobs must be included with a kernel in
order to satisfy the prerequisites of these drivers. This affects
vendors, but also regular users to a certain extent too.

The attached patch introduces MODULE_FIRMWARE as one way of advertising
that a particular firmware file is to be loaded - it will then show up
via modinfo and could be used e.g. when packaging a kernel. I've also
given an example via the QLogic gla2xxx driver.

I expect that there are better ways to do this, so I'd like comments
from those closer to the ground on this problem. In particular, it would
be better if drivers didn't have to list the firmware twice - once in
request_firmware and then via MODULE_FIRMWARE. Maybe API change time?

Jon.

P.S. Until I fix evolution again, I'm using an attachment.


[-- Attachment #2: module_firmware.patch --]
[-- Type: text/x-patch, Size: 1205 bytes --]

diff -urN linux-2.6.16.2_orig/drivers/scsi/qla2xxx/qla_os.c linux-2.6.16.2_dev/drivers/scsi/qla2xxx/qla_os.c
--- linux-2.6.16.2_orig/drivers/scsi/qla2xxx/qla_os.c   2006-04-07 17:56:47.000000000 +0100
+++ linux-2.6.16.2_dev/drivers/scsi/qla2xxx/qla_os.c    2006-04-12 20:22:15.000000000 +0100
@@ -2765,3 +2765,9 @@
 MODULE_DESCRIPTION("QLogic Fibre Channel HBA Driver");
 MODULE_LICENSE("GPL");
 MODULE_VERSION(QLA2XXX_VERSION);
+MODULE_FIRMWARE("ql2100_fw.bin");
+MODULE_FIRMWARE("ql2200_fw.bin");
+MODULE_FIRMWARE("ql2300_fw.bin");
+MODULE_FIRMWARE("ql2322_fw.bin");
+MODULE_FIRMWARE("ql6312_fw.bin");
+MODULE_FIRMWARE("ql2400_fw.bin");
diff -urN linux-2.6.16.2_orig/include/linux/module.h linux-2.6.16.2_dev/include/linux/module.h
--- linux-2.6.16.2_orig/include/linux/module.h  2006-04-07 17:56:47.000000000 +0100
+++ linux-2.6.16.2_dev/include/linux/module.h   2006-04-12 13:51:56.000000000 +0100
@@ -155,6 +155,8 @@
 */
 #define MODULE_VERSION(_version) MODULE_INFO(version, _version)

+#define MODULE_FIRMWARE(_firmware) MODULE_INFO(firmware, _firmware)
+
 /* Given an address, look for it in the exception tables */
 const struct exception_table_entry *search_exception_tables(unsigned long add);

Re: [RFC] binary firmware and modules

[Oliver Neukum]

Am Samstag, 15. April 2006 10:10 schrieb Jon Masters:
> The attached patch introduces MODULE_FIRMWARE as one way of advertising
> that a particular firmware file is to be loaded - it will then show up
> via modinfo and could be used e.g. when packaging a kernel. I've also
> given an example via the QLogic gla2xxx driver.

Strictly speaking, what is the connection with modules? Statically
compiled drivers need their firmware, too. Secondly, do all drivers
know at compile time which firmware they'll need?

	Regards
		Oliver

Re: [RFC] binary firmware and modules

[John W. Linville]

On Sat, Apr 15, 2006 at 11:54:22AM +0200, Oliver Neukum wrote:
> Am Samstag, 15. April 2006 10:10 schrieb Jon Masters:
> > The attached patch introduces MODULE_FIRMWARE as one way of advertising
 
> Strictly speaking, what is the connection with modules? Statically

The same as MODULE_AUTHOR, MODULE_LICENSE, etc.  The divide is more
logical than physical.

> compiled drivers need their firmware, too. Secondly, do all drivers
> know at compile time which firmware they'll need?

They have to know what they will request, do they not?

John
-- 
John W. Linville
linville@tuxdriver.com

Re: [RFC] binary firmware and modules

[Arjan van de Ven]

On Mon, 2006-04-17 at 10:22 -0400, John W. Linville wrote:
> On Sat, Apr 15, 2006 at 11:54:22AM +0200, Oliver Neukum wrote:
> > Am Samstag, 15. April 2006 10:10 schrieb Jon Masters:
> > > The attached patch introduces MODULE_FIRMWARE as one way of advertising
>  
> > Strictly speaking, what is the connection with modules? Statically
> 
> The same as MODULE_AUTHOR, MODULE_LICENSE, etc.  The divide is more
> logical than physical.
> 
> > compiled drivers need their firmware, too. Secondly, do all drivers
> > know at compile time which firmware they'll need?
> 
> They have to know what they will request, do they not?


in order to not fall in the naming-policy trap: do we need a translation
layer here? eg the module asks for firmware-<modulename>
and userspace then somehow maps that to a full filename via a lookup
table?

Re: [RFC] binary firmware and modules

[Marcel Holtmann]

Hi Arjan,

> > > compiled drivers need their firmware, too. Secondly, do all drivers
> > > know at compile time which firmware they'll need?
> > 
> > They have to know what they will request, do they not?
> 
> in order to not fall in the naming-policy trap: do we need a translation
> layer here? eg the module asks for firmware-<modulename>
> and userspace then somehow maps that to a full filename via a lookup
> table?

why do we need that? Currently it is not needed and I don't see a reason
to make it more complicated.

The important thing is to export the used firmware names to the
userspace, because this piece of information is only stored inside the
kernel source code right now.

Regards

Marcel

Re: [RFC] binary firmware and modules

[Duncan Sands]

On Monday 17 April 2006 16:29, Arjan van de Ven wrote:
> On Mon, 2006-04-17 at 10:22 -0400, John W. Linville wrote:
> > On Sat, Apr 15, 2006 at 11:54:22AM +0200, Oliver Neukum wrote:
> > > Am Samstag, 15. April 2006 10:10 schrieb Jon Masters:
> > > > The attached patch introduces MODULE_FIRMWARE as one way of advertising
> >  
> > > Strictly speaking, what is the connection with modules? Statically
> > 
> > The same as MODULE_AUTHOR, MODULE_LICENSE, etc.  The divide is more
> > logical than physical.
> > 
> > > compiled drivers need their firmware, too. Secondly, do all drivers
> > > know at compile time which firmware they'll need?
> > 
> > They have to know what they will request, do they not?
> 
> 
> in order to not fall in the naming-policy trap: do we need a translation
> layer here? eg the module asks for firmware-<modulename>
> and userspace then somehow maps that to a full filename via a lookup
> table?

Hi Arjan, since the same module can handle different devices which need
different firmware, firmware-<modulename> is not sufficient.  An example
is the speedtouch USB modems, which need different firmware depending on
the modem revision.  In fact each modem needs two blobs uploaded, a small
one (stage 1 blob) followed by a large one (stage 2 blob).  These blobs
are fairly independant, for example a given stage 1 blob can work for a
wider range of modem revisions than a given stage 2 blob.  Also, the
manufacturer made a bunch of exotic variations on the modem (all with
the same product ids, but different revisions); I don't have a complete
list, and I don't know which ones require special firmware.  That means
that the driver does not have a complete list of different firmware names.
It could always use the same names "stage1" and "stage2", regardless of
the revision, and expect the user to place the right firmware there; but
then what about people who have multiple modems with different revisions
(like me)?  As a result of all this, the driver needs to export the following
to userspace, to let user-space find the right firmware:

(0) module name
(1) stage 1 or stage 2
(2) modem major revision number
(3) modem minor revision number (I don't know if this is really needed,
but I heard a rumour that it was: ISDN modems needing special firmware
but only varying in their minor revision from non-ISDN modems IIRC).

This is all exported in the hotplug environment.  However, current hotplug
scripts don't have any cleverness in them for handling this kind of thing
(I don't know about udev).  So the driver uses the following rather nasty
but effective scheme: first it looks for the following file:

	speedtch-<stage>.bin.<major_revision>.<minor_revision>

for example speedtch-1.bin.4.0

If that is not found, it looks for

	speedtch-<stage>.bin.<major_revision>

If that is not found, it looks for

	speedtch-<stage>.bin

If that is not found, then it gives up.

This allows people with only one modem to put their firmware in
speedtch-1.bin and speedtch-2.bin and not have to think about
revisions.  But at the same time it allows distributions to
distribute multiple firmware blobs, for example a blob for
general revision 4 modems, but with a special blob for revision 4.1
modems (this could be the ISDN kind; I made up the minor revision 1),
simply by providing files speedtouch-2.bin.4 and speedtouch-2.bin.4.1
It's got to be said that no distribution that I know of actually does
this.

All the best,

Duncan.

Re: [RFC] binary firmware and modules

[Marcel Holtmann]

Hi Duncan,

> > in order to not fall in the naming-policy trap: do we need a translation
> > layer here? eg the module asks for firmware-<modulename>
> > and userspace then somehow maps that to a full filename via a lookup
> > table?
> 
> since the same module can handle different devices which need
> different firmware, firmware-<modulename> is not sufficient.  An example
> is the speedtouch USB modems, which need different firmware depending on
> the modem revision.  In fact each modem needs two blobs uploaded, a small
> one (stage 1 blob) followed by a large one (stage 2 blob).  These blobs
> are fairly independant, for example a given stage 1 blob can work for a
> wider range of modem revisions than a given stage 2 blob.  Also, the
> manufacturer made a bunch of exotic variations on the modem (all with
> the same product ids, but different revisions); I don't have a complete
> list, and I don't know which ones require special firmware.  That means
> that the driver does not have a complete list of different firmware names.
> It could always use the same names "stage1" and "stage2", regardless of
> the revision, and expect the user to place the right firmware there; but
> then what about people who have multiple modems with different revisions
> (like me)?  As a result of all this, the driver needs to export the following
> to userspace, to let user-space find the right firmware:
> 
> (0) module name
> (1) stage 1 or stage 2
> (2) modem major revision number
> (3) modem minor revision number (I don't know if this is really needed,
> but I heard a rumour that it was: ISDN modems needing special firmware
> but only varying in their minor revision from non-ISDN modems IIRC).
> 
> This is all exported in the hotplug environment.  However, current hotplug
> scripts don't have any cleverness in them for handling this kind of thing
> (I don't know about udev).

we don't really have hotplug anymore. Now it is all handled by udev or
the udev helper programs.

For my drivers, I don't have this problem of different firmware images
for different versions and I personally don't really like abusing the
firmware name for some kind of revision control. However initially the
idea behind request_firmware() was that the userspace is totally dumb
and its only job is to copy a binary image from the filesystem into the
kernel memory, so it can be used by the driver. Another idea was that we
can reuse the firmware filenames from the Windows driver if they are
separate from the driver. This allowed us in most cases to tell the end
users to simply extract that file and copy it to /lib/firmware/ and
everything was working.

So for the fist step, I think a MODULE_FIRMWARE() extension (with maybe
and additional comment per firmware name) will be a good think to feed
back the information of firmware files to the userspace. After that we
might need to look into messed up devices and drivers. I also expect
that some WiFi drivers might have the same problem.

Regards

Marcel

Re: [RFC] binary firmware and modules

[Jon Masters]

On 4/15/06, Jon Masters <jcm@redhat.com> wrote:

> The attached patch introduces MODULE_FIRMWARE as one way of advertising
> that a particular firmware file is to be loaded - it will then show up
> via modinfo and could be used e.g. when packaging a kernel. I've also
> given an example via the QLogic gla2xxx driver.

Ok. If nobody shouts today I'm going to suggest this go into 2.6.17. I
think more ellaborate schemes will come up later, but we also need
something usable out there now.

As others have pointed out, cunning schemes to hack how
request_firmware et al work are all very well and good, but often we
just don't know what firmware we'll need until runtime. Unless or
until there's a good way to address that, I think modules will need to
advertise every firmware and distros will have to package all possible
firmwares, just in case.

Jon.

Re: [RFC] binary firmware and modules

[Duncan Sands]

On Tuesday 18 April 2006 15:16, Jon Masters wrote:
> On 4/15/06, Jon Masters <jcm@redhat.com> wrote:
> 
> > The attached patch introduces MODULE_FIRMWARE as one way of advertising
> > that a particular firmware file is to be loaded - it will then show up
> > via modinfo and could be used e.g. when packaging a kernel. I've also
> > given an example via the QLogic gla2xxx driver.
> 
> Ok. If nobody shouts today I'm going to suggest this go into 2.6.17. I
> think more ellaborate schemes will come up later, but we also need
> something usable out there now.
> 
> As others have pointed out, cunning schemes to hack how
> request_firmware et al work are all very well and good, but often we
> just don't know what firmware we'll need until runtime. Unless or
> until there's a good way to address that, I think modules will need to
> advertise every firmware and distros will have to package all possible
> firmwares, just in case.

Hi Jon, this approach seems mistaken to me.  If I understand it right, 
your mental model is that the driver has a list of file names for firmware
files, and calls user-space with the right file-name for the device in
question.  Given that model, having drivers tell the world about their
firmware file list is reasonable; but I think the model is a bad one.
Much better would be to have drivers work at a higher level of abstraction,
and have user-space map the driver's abstract firmware request to a
particular firmware file.  The kernel should say: I am the x driver,
I want firmware for the y device, my version is v, the device version
is r, etc etc.  Userspace should work out that the appropriate file
is ql2322_fw.bin and upload it.  This is similar to the way the kernel
handles other requests for services, such as loading modules (the kernel
can ask for a particular module, but it can also ask for a module which
provides a certain functionality).  Of course this requires making udev
firmware handling more intelligent.

I gave the example of the speedtouch driver to show how complicated
things can be.  I didn't mean to suggest that the scheme it uses is
a good one - it is a bad one, in that the real solution is to make
userspace smarter.  In any case, I don't see how your suggested patch
could reasonably work with the speedtouch driver - after all, the
driver doesn't have a list of possible firmware files, and can't
have one because no-one knows exactly which files exist or may be
needed; and even if I had a decent list, I think it would be a mistake
to hardwire it into the kernel.

Ciao,

D.

Re: [RFC] binary firmware and modules

[Jon Masters]

On 4/18/06, Duncan Sands <duncan.sands@math.u-psud.fr> wrote:
> On Tuesday 18 April 2006 15:16, Jon Masters wrote:
> > On 4/15/06, Jon Masters <jcm@redhat.com> wrote:
> >
> > > The attached patch introduces MODULE_FIRMWARE as one way of advertising
> > > that a particular firmware file is to be loaded - it will then show up
> > > via modinfo and could be used e.g. when packaging a kernel. I've also
> > > given an example via the QLogic gla2xxx driver.
> >
> > Ok. If nobody shouts today I'm going to suggest this go into 2.6.17. I
> > think more ellaborate schemes will come up later, but we also need
> > something usable out there now.
> >
> > As others have pointed out, cunning schemes to hack how
> > request_firmware et al work are all very well and good, but often we
> > just don't know what firmware we'll need until runtime. Unless or
> > until there's a good way to address that, I think modules will need to
> > advertise every firmware and distros will have to package all possible
> > firmwares, just in case.

> Hi Jon, this approach seems mistaken to me.  If I understand it right,
> your mental model is that the driver has a list of file names for firmware
> files, and calls user-space with the right file-name for the device in
> question.

Yes, as is the way it is done right now. There's room for userspace to
massage the request a little but right now it /really shouldn't/
because that's not co-ordinated with the driver.

> Given that model, having drivers tell the world about their
> firmware file list is reasonable; but I think the model is a bad one.

Yes, perhaps it is, but that's how it is now. The point of my mail was
that right now we have zero way to package up a kernel when the
firmware is out-of-tree and this is rapidly becoming reality so we
need a solution right away.

> Much better would be to have drivers work at a higher level of abstraction

Yes, but remember that you then have to embed lots of driver logic in
userspace. Since it's unlikely that driver writers are going to be
able to invest lots of time in keeping what they're doing in sync with
special handling in udev, I think that approach is also a mistake. Do
you really expect to push updated logic into udev every time you
update your driver for a quick hardware change? really?

> I gave the example of the speedtouch driver to show how complicated
> things can be.  I didn't mean to suggest that the scheme it uses is
> a good one - it is a bad one, in that the real solution is to make
> userspace smarter.

I think it's a mistake to take that logic out of the driver proper. To
say "we'll just have userspace figure it out" is asking for weird
undebugable situations. I'm not in favor of bloat, but this is hardly
bloating the kernel - most drivers are going to request firmware by
filename right now, so let's just have a way to export that filename
for the moment at least.

> In any case, I don't see how your suggested patch
> could reasonably work with the speedtouch driver

I own a speedtouch here myself. I had to extract the firmware by hand
and install it. Unless something has changed, this means that we're
not going to get into a situation where that firmware is being shipped
out due to the licensing on it. I get your point (sort of) but I think
you're overcomplicating this for the sake of drivers like speedtouch
where none of this logic works anyway.

I'm not saying this is a perfect approach. It's /an/ approach, it's
simple and it works right now for many possible drivers. Packagers can
trivially rip out info from modinfo and use it as a list of files to
look for in a package (for example). It's also hardly difficult to
switch to a grand magical solution sometime in the future.

Jon.

Re: [RFC] binary firmware and modules

[Duncan Sands]

...
> > Hi Jon, this approach seems mistaken to me.  If I understand it right,
> > your mental model is that the driver has a list of file names for firmware
> > files, and calls user-space with the right file-name for the device in
> > question.
> 
> Yes, as is the way it is done right now. There's room for userspace to
> massage the request a little but right now it /really shouldn't/
> because that's not co-ordinated with the driver.
> 
> > Given that model, having drivers tell the world about their
> > firmware file list is reasonable; but I think the model is a bad one.
> 
> Yes, perhaps it is, but that's how it is now. The point of my mail was
> that right now we have zero way to package up a kernel when the
> firmware is out-of-tree and this is rapidly becoming reality so we
> need a solution right away.

This is a problem for speedtouch users too - I get regular reports from users
that their modem failed to find the firmware, usually because it wasn't in the
initrd.

> > Much better would be to have drivers work at a higher level of abstraction
> 
> Yes, but remember that you then have to embed lots of driver logic in
> userspace. Since it's unlikely that driver writers are going to be
> able to invest lots of time in keeping what they're doing in sync with
> special handling in udev, I think that approach is also a mistake. Do
> you really expect to push updated logic into udev every time you
> update your driver for a quick hardware change? really?

I don't understand what you're saying here - what kind of situation are
you thinking of when you talk about "update your driver for a quick hardware
change"?

> > I gave the example of the speedtouch driver to show how complicated
> > things can be.  I didn't mean to suggest that the scheme it uses is
> > a good one - it is a bad one, in that the real solution is to make
> > userspace smarter.
> 
> I think it's a mistake to take that logic out of the driver proper. To
> say "we'll just have userspace figure it out" is asking for weird
> undebugable situations. I'm not in favor of bloat, but this is hardly
> bloating the kernel - most drivers are going to request firmware by
> filename right now, so let's just have a way to export that filename
> for the moment at least.

You're creating a new kernel API here, so it needs to be good from the
word go: "for the moment at least" could easily become "needs to be
supported forever".  It's a pity if the scheme can't handle even mildly
complicated situations.

> 
> > In any case, I don't see how your suggested patch
> > could reasonably work with the speedtouch driver
> 
> I own a speedtouch here myself. I had to extract the firmware by hand
> and install it. Unless something has changed, this means that we're
> not going to get into a situation where that firmware is being shipped
> out due to the licensing on it. 

There's no reason to think that Thomson is dead set against having
distributions distribute firmware.  I bet the problem is simply that
no distribution has yet tried to sort the licensing problem out.
Maybe one day distributions will ship with speedtouch firmware.  If
so, it would be sad to then discover that the kernel+tools are too
unsophisticated to handle the situation.

> I get your point (sort of) but I think 
> you're overcomplicating this for the sake of drivers like speedtouch
> where none of this logic works anyway.
> 
> I'm not saying this is a perfect approach. It's /an/ approach, it's
> simple and it works right now for many possible drivers. Packagers can
> trivially rip out info from modinfo and use it as a list of files to
> look for in a package (for example). It's also hardly difficult to
> switch to a grand magical solution sometime in the future.

Putting in infrastructure that is adequate for most drivers but not all,
will simply make it much harder to move one day to an infrastructure that
can handle the requirements of all drivers, because most people won't be
motivated.

As I remarked in another email, MODULE_FIRMWARE could be made to work
with the speedtouch driver as long as it is possible to specify patterns,
or at least initial parts of filenames.

Ciao,

Duncan.

Re: [RFC] binary firmware and modules

[Jon Masters]

On 4/18/06, Duncan Sands <duncan.sands@math.u-psud.fr> wrote:

> > > Given that model, having drivers tell the world about their
> > > firmware file list is reasonable; but I think the model is a bad one.

jcm>> Yes, perhaps it is, but that's how it is now. The point of my mail was
jcm>> that right now we have zero way to package up a kernel when the
jcm>> firmware is out-of-tree and this is rapidly becoming reality so we
jcm>> need a solution right away.

> This is a problem for speedtouch users too - I get regular reports from users
> that their modem failed to find the firmware, usually because it wasn't in the
> initrd.

Yes. As we've already said, it's unlikely that's going to change any
time soon for those shipping kernels - licensing on the speedtouch
firmware isn't conducive AFAIK.

> > > Much better would be to have drivers work at a higher level of abstraction

jcm>> Do you really expect to push updated logic into udev every time you
jcm>> update your driver for a quick hardware change? really?

> I don't understand what you're saying here - what kind of situation are
> you thinking of when you talk about "update your driver for a quick hardware
> change"?

Piece of hardware you always thought worked one way, but doesn't.
Quick hacks for new devices/fixes for old ones, etc. etc. There are
lots of reasons you might want to change firmware logic and the idea
of doing it in two places seems a bad one to me.

jcm>> so let's just have a way to export that filename
jcm>> for the moment at least.

> You're creating a new kernel API here, so it needs to be good from the
> word go: "for the moment at least" could easily become "needs to be
> supported forever".  It's a pity if the scheme can't handle even mildly
> complicated situations.

I'm creating a new internal API, yes. One that I think solves the
problem for most drivers and can stick around for a while - when I say
"for the moment at least" I mean that in the same sense that any part
of the kernel stays around until someone comes along with something
new that everyone jumps at. But I think it's a useful one line patch
:-)

> > > In any case, I don't see how your suggested patch
> > > could reasonably work with the speedtouch driver

jcm>> I own a speedtouch here myself. I had to extract the firmware by hand
jcm>> and install it. Unless something has changed, this means that we're
jcm>> not going to get into a situation where that firmware is being shipped
jcm>> out due to the licensing on it.

> There's no reason to think that Thomson is dead set against having
> distributions distribute firmware.

But there's no reason to think any different either - and I don't
think it's likely vendors are going to go out of their way for every
driver to have lengthy legal debates over licensing. If these folks
want their firmware to be distributed, then they need to facilitate
that. To be fair to QLogic, their firmware has a specific license on
it and they seem to be a player.

> Maybe one day distributions will ship with speedtouch firmware.  If
> so, it would be sad to then discover that the kernel+tools are too
> unsophisticated to handle the situation.

What's the problem? Really? It's just a file list of firmwares - if
you really can't have a list then you just have to special case the
speedtouch driver. Meanwhile, other folks are able to happily use
MODULE_FIRMWARE to advertise their firmware requirements :-)

> As I remarked in another email, MODULE_FIRMWARE could be made to work
> with the speedtouch driver as long as it is possible to specify patterns,
> or at least initial parts of filenames.

You can specify whatever filename you like - it's down to whatever
parses the modinfo sections to rip this out in the right way at
packaging time. I strongly prefer file names.

Jon.

Re: [RFC] binary firmware and modules

[Marcel Holtmann]

Hi Duncan,

> > > The attached patch introduces MODULE_FIRMWARE as one way of advertising
> > > that a particular firmware file is to be loaded - it will then show up
> > > via modinfo and could be used e.g. when packaging a kernel. I've also
> > > given an example via the QLogic gla2xxx driver.
> > 
> > Ok. If nobody shouts today I'm going to suggest this go into 2.6.17. I
> > think more ellaborate schemes will come up later, but we also need
> > something usable out there now.
> > 
> > As others have pointed out, cunning schemes to hack how
> > request_firmware et al work are all very well and good, but often we
> > just don't know what firmware we'll need until runtime. Unless or
> > until there's a good way to address that, I think modules will need to
> > advertise every firmware and distros will have to package all possible
> > firmwares, just in case.
> 
> this approach seems mistaken to me.  If I understand it right, 
> your mental model is that the driver has a list of file names for firmware
> files, and calls user-space with the right file-name for the device in
> question.  Given that model, having drivers tell the world about their
> firmware file list is reasonable; but I think the model is a bad one.
> Much better would be to have drivers work at a higher level of abstraction,
> and have user-space map the driver's abstract firmware request to a
> particular firmware file.  The kernel should say: I am the x driver,
> I want firmware for the y device, my version is v, the device version
> is r, etc etc.  Userspace should work out that the appropriate file
> is ql2322_fw.bin and upload it.  This is similar to the way the kernel
> handles other requests for services, such as loading modules (the kernel
> can ask for a particular module, but it can also ask for a module which
> provides a certain functionality).  Of course this requires making udev
> firmware handling more intelligent.
> 
> I gave the example of the speedtouch driver to show how complicated
> things can be.  I didn't mean to suggest that the scheme it uses is
> a good one - it is a bad one, in that the real solution is to make
> userspace smarter.  In any case, I don't see how your suggested patch
> could reasonably work with the speedtouch driver - after all, the
> driver doesn't have a list of possible firmware files, and can't
> have one because no-one knows exactly which files exist or may be
> needed; and even if I had a decent list, I think it would be a mistake
> to hardwire it into the kernel.

we have two kind of devices that need firmware download. The easy and
clean ones which need one or two files and these basically change not
that often. In most cases these are the network or storage devices and
for exactly these we need the MODULE_FIRMWARE() support to know which
files have to be put into initrd.

The messed up devices like the Speedtouch and maybe even some WiFi
dongles are another story.

Regards

Marcel

Re: [RFC] binary firmware and modules

[Duncan Sands]

Hi Marcel,

> we have two kind of devices that need firmware download. The easy and
> clean ones which need one or two files and these basically change not
> that often. In most cases these are the network or storage devices and
> for exactly these we need the MODULE_FIRMWARE() support to know which
> files have to be put into initrd.

> The messed up devices like the Speedtouch and maybe even some WiFi
> dongles are another story.

I don't know why you consider the speedtouch to be messed up.  What's
messed up is not the modems themselves, but the fact that we don't know
what modems exist, and how they differ in their firmware requirements.

Anyway, speedtouch users also need their firmware to end up in any initrd.
Since the driver expects all firmware files to start with "speedtch",
the MODULE_FIRMWARE scheme would work for the speedtouch driver too as
long as it allows the driver to specify just the initial part of a file
name.  You could go all the way to regular expressions, but that seems
a bit ridiculous.

Ciao,

Duncan.

Re: [RFC] binary firmware and modules

[Marcel Holtmann]

Hi Duncan,

> > we have two kind of devices that need firmware download. The easy and
> > clean ones which need one or two files and these basically change not
> > that often. In most cases these are the network or storage devices and
> > for exactly these we need the MODULE_FIRMWARE() support to know which
> > files have to be put into initrd.
> 
> > The messed up devices like the Speedtouch and maybe even some WiFi
> > dongles are another story.
> 
> I don't know why you consider the speedtouch to be messed up.  What's
> messed up is not the modems themselves, but the fact that we don't know
> what modems exist, and how they differ in their firmware requirements.

if you don't know the firmware requirements, then this is what I call
messed up. I now that this is basically the fault of the manufacturer or
missing specifications, but wild guessing on the firmware doesn't really
help. A kernel driver should know which firmware it needs.

> Anyway, speedtouch users also need their firmware to end up in any initrd.
> Since the driver expects all firmware files to start with "speedtch",
> the MODULE_FIRMWARE scheme would work for the speedtouch driver too as
> long as it allows the driver to specify just the initial part of a file
> name.  You could go all the way to regular expressions, but that seems
> a bit ridiculous.

I personally prefer full firmware names. This makes the dependency easy
and even an end user can call modinfo and see what firmware is expected
by a certain driver (without looking at the source code).

Regards

Marcel

Re: [RFC] binary firmware and modules

[Mark Lord]

Marcel Holtmann wrote:
..
> I personally prefer full firmware names. This makes the dependency easy
> and even an end user can call modinfo and see what firmware is expected
> by a certain driver (without looking at the source code).

How does one handle the case of updated firmware from the manufacturer,
which requires *no* driver changes?  If the driver has all of the previously
known names/versions hardcoded, then would it refuse to use the new stuff?

I'm probably misunderstanding something here.

Re: [RFC] binary firmware and modules

[Marcel Holtmann]

Hi Mark,

> > I personally prefer full firmware names. This makes the dependency easy
> > and even an end user can call modinfo and see what firmware is expected
> > by a certain driver (without looking at the source code).
> 
> How does one handle the case of updated firmware from the manufacturer,
> which requires *no* driver changes?  If the driver has all of the previously
> known names/versions hardcoded, then would it refuse to use the new stuff?

if no driver change is needed, then you simply can replace the firmware
file and reload the driver. With the BlueFRITZ! USB driver we did this a
bunch of times and it worked out perfectly. The firmware name in this
case is only a placeholder.

Regards

Marcel

Re: [RFC] binary firmware and modules

[Jon Masters]

On 4/19/06, Marcel Holtmann <marcel@holtmann.org> wrote:
> Hi Mark,

mlord>> How does one handle the case of updated firmware from the manufacturer,
mlord>> which requires *no* driver changes?  If the driver has all of
the previously
mlord>> known names/versions hardcoded, then would it refuse to use
the new stuff?

I guess Mark is saying "what if the filename changes". The thing is,
as part of some other work I'm doing right now, I've talked to a few
folks about naming conventions for firmware, etc. In an ideal world,
we'd have some LANNA process for firmware naming that was flexible and
yet allowed us to handle this all cleanly, and we'd perhaps be able to
request_firmware with a minimum version number or whatever. Right now,
the kernel only supports pulling in a binary blob - determining
whether it's the right one is somewhat vague.

> if no driver change is needed, then you simply can replace the firmware
> file and reload the driver. With the BlueFRITZ! USB driver we did this a
> bunch of times and it worked out perfectly. The firmware name in this
> case is only a placeholder.

That's really going to have to continue for the moment I think, until
we can get better infrastructure for handling firmware updates in
userspace and managing versions. The point of my original patch is
that it gives us a starting point to be able to think about this more.

Jon.

Re: [RFC] binary firmware and modules

[Marcel Holtmann]

Hi Jon,

> > The attached patch introduces MODULE_FIRMWARE as one way of advertising
> > that a particular firmware file is to be loaded - it will then show up
> > via modinfo and could be used e.g. when packaging a kernel. I've also
> > given an example via the QLogic gla2xxx driver.
> 
> Ok. If nobody shouts today I'm going to suggest this go into 2.6.17. I
> think more ellaborate schemes will come up later, but we also need
> something usable out there now.

I like to see this in 2.6.17, too. I also think that it would be a good
idea if we can add a description to each firmware file. Like we do with
the module parameters.

> As others have pointed out, cunning schemes to hack how
> request_firmware et al work are all very well and good, but often we
> just don't know what firmware we'll need until runtime. Unless or
> until there's a good way to address that, I think modules will need to
> advertise every firmware and distros will have to package all possible
> firmwares, just in case.

And remember that request_firmware() is also used for overriding CIS
tables for PCMCIA cards.

Regards

Marcel