Roadmap to restoring working usb module autoloading?

Roadmap to restoring working usb module autoloading?

[Adam J. Richter]

	I am concerned that the situation with usb module autoloading
seems to be deadlocked and so I want to propose a roadmap for
getting to the point where usb module autoloading works when a
user installs the latest versions of everything and does not
make any additional modifications.

	Here is my proposal:

	0. We agree that the endpoint of roadmap is not necessarily the
final destination that we are going to settle on, but rather a point
where everything works again and one that is closer to where we want
to be than before.

	1. I have just posted changes to usbmodules to recognize
both the old and new versions of modules.usbmap.  If the patches
work for everyone, then I would appreciate it if Thomas would make
a new release of usbutils that includes these changes.  There should
be no compatability problem with this.

	2. As I understand the situation, Linus and Alan have not
integrated Keith's module table version number patches after a number
of 2.4.1-pre releases.  The big advantage of using version numbers
for each ID structure instead of having depmod decide based on the
kernel version string is that you can update different components
of the kernel if yours is somewhat customized (e.g., imagine a 2.5.x
USB backport to 2.4).  However, this is currently defeated by an
error check in depmod which compalins "Modules have a mixture of
version ___ and version ___".   Also, integer version numbers that
increase by one create an ID allocation problem if somebody wants
to ship a modified device ID structure and a modutils that will
recognize it.  These are issues that have occurred to me while
wondering why the changes have not been integrated yet.  They
are just second guessing, though.  Someone should corner one of
them at LinuxWorld about it in any case.  I'm not sure what the other
advantages of the structure version numbers are supposed to be, but
I realize I'm probably not doing it justice in this paragraph.

	Although my understanding of the situtation is obviously
incomplete, I think the idea behind these version numbers was to
avoid version skew ("you have to upgrade the kernel if you upgrade
modutils").  However, that is preferable to nonfunction ("if you
upgrade the kernel, your modules won't work").  I'm also beginning
to think that there may be...maybe...some method to the madness of
not going with version numbers for the device ID tables, at least
enough so that I'm not sufficiently confident that Linus is wrong
that would I want to support an intransagence contest between
the standard kernels and the standard userland facilities for USB hot
plugging.  So, I propose that, for the meantime that modutils be changed
to accomodate Linus's kernels for now (I am willing to write these patches):

	2a. Back out the device ID structure version number code for now
and instead have a new modutils that supports only the new format for
usb_device_id.  module_device_id tables are not in 2.2 and the old
format was used only in 2.4.0-pre 2.4.0-test kernels, so everyone
is supposed to upgrade anyhow, especially if they're upgrading other
components on their system.

	2b. depmod wants usb_device_id.match_flags toward the end of
the structure and Linus's kernels put it toward the beginning.  I know
we had talked about rearranging the usb_device_id structure this way
to improve compatability.  Again, I propose having depmod accomodate
Linus's kernels.

	3. Likewise, the rest of USB userland support scripts should
be updated.  I do not use these.  Perhaps david-b could comment on this.

	If there are people in the midst of negotiating some other fix
with Linus and those who have his ear on this matter, I don't mean to
preempt their efforts.  But, I don't think that is what is going on.
I think everybody is waiting for some other event to break this impasse
and, in the meantime, USB hot plugging remains out of commission and
Linux will make a bad impression in this area with people who are trying
Linux because of the 2.4 hoopla.

	This does not mean that device ID version number are wrong
or should not get into Linus's kernels.  I just want the software
to continue to work with the current kernels in the meantime.

	Comments?  (Flames?)

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Roadmap to restoring working usb module autoloading?

[David Brownell]

> From: Adam J. Richter <adam@yggdrasil.com>
> Sent: Saturday, January 20, 2001 5:32 AM
>
> I am concerned that the situation with usb module autoloading
> seems to be deadlocked

With regards to getting kernel and modutils talking again, in
production releases of both.


> Here is my proposal:
> 
> 0. We agree that the endpoint of roadmap is not necessarily the
> final destination that we are going to settle on, but rather a point
> where everything works again and one that is closer to where we want
> to be than before.

Yep.  Milestones of note include "works as well as the prerelease",
and "bugs noted in prerelease are now fixed".


> 1. I have just posted changes to usbmodules to recognize
> both the old and new versions of modules.usbmap.  If the patches
> work for everyone, then I would appreciate it if Thomas would make
> a new release of usbutils that includes these changes.  There should
> be no compatability problem with this.

Thanks.  This should get rid of the "doesn't work with usbutils"
set of problems.


> 2. As I understand the situation, Linus and Alan have not
> integrated Keith's module table version number patches after a number
> of 2.4.1-pre releases.  The big advantage of using version numbers
> for each ID structure instead of having depmod decide based on the
> kernel version string

It was using structure size before, which was clearly not going to
work in cases like this (same size, fields rearranged).  I don't know
that the (optional?) kernel version string was being considered.


>     is that you can update different components
> of the kernel if yours is somewhat customized (e.g., imagine a 2.5.x
> USB backport to 2.4).  However, this is currently defeated by an
> error check in depmod which compalins "Modules have a mixture of
> version ___ and version ___".   Also, integer version numbers that
> increase by one create an ID allocation problem if somebody wants
> to ship a modified device ID structure and a modutils that will
> recognize it. 

I'm not sure we'd want to encourage or support that.  It'd
effectively be a kernel fork.


>     These are issues that have occurred to me while
> wondering why the changes have not been integrated yet.  They
> are just second guessing, though.

I'm concerned that's most of what we have to go by.

Another thought is that it was a recognition that the failure
modes related to the netdevice stuff (see Andrew's recent post)
were a bit too severe to push on other parts of hotplug without
having that fixed first.   Complete system hangs are not polite
in "production" kernels.


>      Someone should corner one of
> them at LinuxWorld about it in any case.  I'm not sure what the other
> advantages of the structure version numbers are supposed to be, but
> I realize I'm probably not doing it justice in this paragraph.
> 
> Although my understanding of the situtation is obviously
> incomplete, I think the idea behind these version numbers was to
> avoid version skew ("you have to upgrade the kernel if you upgrade
> modutils").  However, that is preferable to nonfunction ("if you
> upgrade the kernel, your modules won't work").  I'm also beginning
> to think that there may be...maybe...some method to the madness of
> not going with version numbers for the device ID tables, at least
> enough so that I'm not sufficiently confident that Linus is wrong
> that would I want to support an intransagence contest between
> the standard kernels and the standard userland facilities for USB hot
> plugging.  So, I propose that, for the meantime that modutils be changed
> to accomodate Linus's kernels for now (I am willing to write these patches):

I'm really concerned about that second-guessing issue.  That's not
the way to achieve consistent forward motion, at least in my book,
and likely contributed to the original 2.4.0 problem.


> 3. Likewise, the rest of USB userland support scripts should
> be updated.  I do not use these.  Perhaps david-b could comment on this.

They've been pretty much up-to-date for a few weeks, and won't care
so long as the "match_flags" shows up in the first field of the
text output.

I will note those scripts currently ignore the match_flags
field, since bitfields are just nasty to deal with in shell
scripts (fixable).  And they're expecting only "new style"
output from modutils (also fixable).  Anyone wanting to hotplug
with prerelease or 2.4.0-test kernels can use "usbmodules", or
older scripts ... anyone wanting a different USB agent (in C?),
feel free to provide one.


> If there are people in the midst of negotiating some other fix
> with Linus and those who have his ear on this matter, I don't mean to
> preempt their efforts.  But, I don't think that is what is going on.
> I think everybody is waiting for some other event to break this impasse

For my part, I'm wanting to hear something from Linus.  We're
now in decent shape (matching functionality in the prerelease,
unfortunately including that netdevice problem) if he'd merge that
patch from Keith.

- Dave



> and, in the meantime, USB hot plugging remains out of commission and
> Linux will make a bad impression in this area with people who are trying
> Linux because of the 2.4 hoopla.
> 
> This does not mean that device ID version number are wrong
> or should not get into Linus's kernels.  I just want the software
> to continue to work with the current kernels in the meantime.
>
> Comments?  (Flames?)
> 
> Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
> adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
> +1 408 261-6630         | g g d r a s i l   United States of America
> fax +1 408 261-6631      "Free Software For The Rest Of Us."
> 



_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Roadmap to restoring working usb module autoloading?

[Adam J. Richter]

>> = Adam Richter
>  = David Brownell

>> [...] The big advantage of using version numbers
>> for each ID structure instead of having depmod decide based on the
>> kernel version string
>>    is that you can update different components
>> of the kernel if yours is somewhat customized (e.g., imagine a 2.5.x
>> USB backport to 2.4).  However, this is currently defeated by an
>> error check in depmod which compalins "Modules have a mixture of
>> version ___ and version ___".   Also, integer version numbers that
>> increase by one create an ID allocation problem if somebody wants
>> to ship a modified device ID structure and a modutils that will
>> recognize it. 

>I'm not sure we'd want to encourage or support that.  It'd
>effectively be a kernel fork.

	It's free software, so "forks" that work tend to weave back
into the standard versions and are in integral part of the process.
In practice, I believe just about every Linux distribution has had
to modify their kernels, and this often helps get those changes into
Linus's kernels.  I believe at least one Linux distribution shipped
the 2.4 "back port" of the USB code in their 2.2 kernel, which, was
probably a net benefit to the world.

	Of course, it's also easy enough to stip out the code from
depmod that complains about the device ID version number differences
without changing anything else if device ID version numbers do get
integrated.

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Roadmap to restoring working usb module autoloading?

[Keith Owens]

On Sat, 20 Jan 2001 18:25:23 -0800, 
"Adam J. Richter" <adam@yggdrasil.com> wrote:
>	Of course, it's also easy enough to stip out the code from
>depmod that complains about the device ID version number differences
>without changing anything else if device ID version numbers do get
>integrated.

And break everything downstream from depmod.  Most (all?) of the
utilities that read the depmod generated maps rely on the header line
to tell them which format of table they are reading.  One header line
-> one supported table format in any map -> all input device ids must
be the same for a map.

If you want a generated map format that can cope with different input
formats in the same map then each line needs a tag that says which
fields it contains.  Before you even think about that approach, be
absolutely certain that the utilities which read the text can cope with
missing or extra fields.  Under all circumstances.  Without ever making
any errors.  Assumptions about the side effects of missing or new
fields must be valid for all fields, in all tables, forever.


_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Roadmap to restoring working usb module autoloading?

[Adam J. Richter]

>> = Adam Richter
>  = Keith Owens

>"Adam J. Richter" <adam@yggdrasil.com> wrote:
>>	Of course, it's also easy enough to stip out the code from
>>depmod that complains about the device ID version number differences
>>without changing anything else if device ID version numbers do get
>>integrated.

>And break everything downstream from depmod.  Most (all?) of the
>utilities that read the depmod generated maps rely on the header line
>to tell them which format of table they are reading.  [...]

	Whoa!  I did not say strip out the header lines.  I said allow
device ID structure version numbers to be different.  For example,
I might have version 2 usb_device_id's and version 1 parport_device_id's.

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Roadmap to restoring working usb module autoloading?

[Keith Owens]

On Sat, 20 Jan 2001 19:17:02 -0800, 
"Adam J. Richter" <adam@yggdrasil.com> wrote:
>	Whoa!  I did not say strip out the header lines.  I said allow
>device ID structure version numbers to be different.  For example,
>I might have version 2 usb_device_id's and version 1 parport_device_id's.

depmod already allows that.  The table version number is per table
type, it is not a global kernel -> depmod version number.  With my
kernel patch we already have usb at version 2 and everything else at
version 1.  If that does not work then it is a depmod bug.


_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel