Re: hotplug (was devfs)

Re: hotplug (was devfs)

[Oliver.Neukum]

On Wed, 13 Nov 2002, Nick Craig-Wood wrote:

> On Tue, Nov 12, 2002 at 01:51:08PM +0100, Oliver Neukum wrote:
> > > Actually, here's a question: are /sbin/hotplug upcalls serialized in
> > > some fashion?  I'd hate to online a thousand devices in my disk array
> > > and have the machine forkbomb itself.
> >
> > Nope, no serialisation. You don't have any guarantee even that
> > addition will be delivered before removal.
>
> And that is why (we finally discovered) we were getting
> non-deterministic device numbering of USB nodes.
>
> We have machines with 6 x 4 port USB <-> serial converters attached.
> These would get randomly assigned usb device ids and hence random
> /dev/ttyUSB nodes.  Not very useful when there is a load of different
> things attached to the 24 serial ports!

Please clarify. Did this happen if you connected a hub these gadgets
sit on? Do you use the same type of device six times ?

> Sometimes we also found that one of the devices wouldn't get
> initialised properly.
>
> We fixed these problems by removing hotplug and loading the relevant
> kernel modules in the correct order and voila a perfectly

Modules ? Plural?

> deterministic order for the /dev/ttyUSBs with all devices initialised.
> Plugging in our USB bus with 24 devices on it does indeed produce a
> mini-forkbomb effect ;-) (Especially since these Keyspan devices are
> initialised twice - once without firmware and once with firmware.)

There's a further problem.

> So - perhaps hotplug ought to be serialised?

Definitely, but how far?

	Regards
		Oliver




-------------------------------------------------------
This sf.net email is sponsored by: Are you worried about 
your web server security? Click here for a FREE Thawte 
Apache SSL Guide and answer your Apache SSL security 
needs: http://www.gothawte.com/rd523.html
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: hotplug (was devfs)

[David Brownell]

Oliver.Neukum@lrz.uni-muenchen.de wrote:
> On Wed, 13 Nov 2002, Nick Craig-Wood wrote:
> 
>>We have machines with 6 x 4 port USB <-> serial converters attached.
>>These would get randomly assigned usb device ids and hence random
>>/dev/ttyUSB nodes.  Not very useful when there is a load of different
>>things attached to the 24 serial ports!

Right, this is why "physical" paths are needed as well as
sequentially assigned /dev/...N nodes.


>>Sometimes we also found that one of the devices wouldn't get
>>initialised properly.
>>
>>We fixed these problems by removing hotplug and loading the relevant
>>kernel modules in the correct order and voila a perfectly
> 
> Modules ? Plural?

Sounds like a better fix might have been to teach hotplug
about your additional modules... or to fix the modules
so they initialized correctly!


>>So - perhaps hotplug ought to be serialised?
> 
> 
> Definitely, but how far?

Until about the last week before 2.4.0final shipped, hotplug
was serialized.  The serialization got removed at the same
time keventd got added.

Why was it removed?  Because serializing meant that kernel
locks had to be held until a user mode program returned.
That caused deadlock when, for example, network hotplug
had to grab that same lock ... it basically wasn't possible
to get away from that problem in at least that case, and by
extension likely other cases.

So I'm not sure I see hotplug ever getting serialized.
But then, I don't see any reason it would ever fix the
problem of needing to map /dev/... nodes to the physical
device they represent, either.

- Dave





-------------------------------------------------------
This sf.net email is sponsored by: Are you worried about 
your web server security? Click here for a FREE Thawte 
Apache SSL Guide and answer your Apache SSL security 
needs: http://www.gothawte.com/rd523.html
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: hotplug (was devfs)

[Nick Craig-Wood]

On Wed, Nov 13, 2002 at 03:15:01PM +0100, Oliver.Neukum@lrz.uni-muenchen.de wrote:
> On Wed, 13 Nov 2002, Nick Craig-Wood wrote:
> > On Tue, Nov 12, 2002 at 01:51:08PM +0100, Oliver Neukum wrote:
> > > > Actually, here's a question: are /sbin/hotplug upcalls serialized in
> > > > some fashion?  I'd hate to online a thousand devices in my disk array
> > > > and have the machine forkbomb itself.
> > >
> > > Nope, no serialisation. You don't have any guarantee even that
> > > addition will be delivered before removal.
> >
> > And that is why (we finally discovered) we were getting
> > non-deterministic device numbering of USB nodes.
> >
> > We have machines with 6 x 4 port USB <-> serial converters attached.
> > These would get randomly assigned usb device ids and hence random
> > /dev/ttyUSB nodes.  Not very useful when there is a load of different
> > things attached to the 24 serial ports!
> 
> Please clarify. Did this happen if you connected a hub these gadgets
> sit on?

It happens on bootup when effectively the USB subsystem becomes aware
of all the devices at once and also when you disconnect and reconnect
the hub the devices sit on.

> Do you use the same type of device six times ?

Yes.

> > Sometimes we also found that one of the devices wouldn't get
> > initialised properly.
> >
> > We fixed these problems by removing hotplug and loading the relevant
> > kernel modules in the correct order and voila a perfectly
> 
> Modules ? Plural?

We found we had to load the keyspan module then the usb-uhci module
for ideal startup without hotplug.

> > deterministic order for the /dev/ttyUSBs with all devices initialised.
> > Plugging in our USB bus with 24 devices on it does indeed produce a
> > mini-forkbomb effect ;-) (Especially since these Keyspan devices are
> > initialised twice - once without firmware and once with firmware.)
> 
> There's a further problem.

Do you mean the double initialisation?  That is the way the keyspan
devices work.  They enumerate first time as a dumb device, get sent
their firmware and then enumerate again.

> > So - perhaps hotplug ought to be serialised?
> 
> Definitely, but how far?

Indeed!

-- 
Nick Craig-Wood
ncw1@axis.demon.co.uk


-------------------------------------------------------
This sf.net email is sponsored by: Are you worried about 
your web server security? Click here for a FREE Thawte 
Apache SSL Guide and answer your Apache SSL security 
needs: http://www.gothawte.com/rd523.html
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

RE: hotplug (was devfs)

[Ed Vance]

On Wed, November 13, 2002 at 10:45 AM, Greg KH wrote:
> On Wed, Nov 13, 2002 at 10:45:43AM -0800, Ed Vance wrote:
> > On Wed, November 13, 2002 at 10:05 AM, Greg KH wrote: 
> > > On Wed, Nov 13, 2002 at 06:06:06PM +0000, Nick Craig-Wood wrote:
> > > > 
> > > > So I'll be able to say usb bus1/1/4/1 port 3 should be 
> /dev/ttyUSB15
> > > > and it will always be that port?  That would be perfect.
> > > 
> > > Yes, that is the goal.
> > > 
> > 
> > Do you expect that goal to eventually be applied to 
> CompactPCI Hot-Swap
> > bus/slot port 3?
> 
> Yes, in a round-about way.  If the device that is in that 
> specific slot
> with that specific port, registers with, for example, the network
> subsystem, and we have decided that anything in that slot should be
> called "eth42", then we can do that based on the topology of 
> the device.
> 
> It really depends on the device that is existing in a 
> specific location
> (network, scsi, etc.) and not so much as the specific location will
> always be a network card called "ethX", as you have to look 
> at the type
> of device too.
> 
> Does that make sense?

Yes.
> 
> I can tell it's getting to be the time to start writing all 
> of this down
> for people to hash out... :)
> 

Very cool. 

Thanks,
Ed

---------------------------------------------------------------- 
Ed Vance              edv (at) macrolink (dot) com
Macrolink, Inc.       1500 N. Kellogg Dr  Anaheim, CA  92807
----------------------------------------------------------------

RE: hotplug (was devfs)

[Ed Vance]

On Wed, November 13, 2002 at 10:05 AM, Greg KH wrote: 
> 
> On Wed, Nov 13, 2002 at 06:06:06PM +0000, Nick Craig-Wood wrote:
> > 
> > So I'll be able to say usb bus1/1/4/1 port 3 should be /dev/ttyUSB15
> > and it will always be that port?  That would be perfect.
> 
> Yes, that is the goal.
> 

Do you expect that goal to eventually be applied to CompactPCI Hot-Swap
bus/slot port 3?

Thanks,
-Ed

---------------------------------------------------------------- 
Ed Vance              edv (at) macrolink (dot) com
Macrolink, Inc.       1500 N. Kellogg Dr  Anaheim, CA  92807
----------------------------------------------------------------

Re: hotplug (was devfs)

[Greg KH]

On Wed, Nov 13, 2002 at 10:45:43AM -0800, Ed Vance wrote:
> On Wed, November 13, 2002 at 10:05 AM, Greg KH wrote: 
> > 
> > On Wed, Nov 13, 2002 at 06:06:06PM +0000, Nick Craig-Wood wrote:
> > > 
> > > So I'll be able to say usb bus1/1/4/1 port 3 should be /dev/ttyUSB15
> > > and it will always be that port?  That would be perfect.
> > 
> > Yes, that is the goal.
> > 
> 
> Do you expect that goal to eventually be applied to CompactPCI Hot-Swap
> bus/slot port 3?

Yes, in a round-about way.  If the device that is in that specific slot
with that specific port, registers with, for example, the network
subsystem, and we have decided that anything in that slot should be
called "eth42", then we can do that based on the topology of the device.

It really depends on the device that is existing in a specific location
(network, scsi, etc.) and not so much as the specific location will
always be a network card called "ethX", as you have to look at the type
of device too.

Does that make sense?

I can tell it's getting to be the time to start writing all of this down
for people to hash out... :)

thanks,

greg k-h

devfs

[Ian Molton]

On 11 Nov 2002 20:50:39 -0500
Robert Love <rml@tech9.net> wrote:

> On Mon, 2002-11-11 at 20:32, Ryan Anderson wrote:
> 
> > From an outsider point of view (and because nobody else responded),
> > I think the big question here would be: Would you use it after this
> > cleanup?
> > 
> > If you say yes, I'd say that's a good sign in its favor.
> 
> Good heuristic, except Al would not use devfs in either case :)

Personally I love devfs. I've not looked at its internals, but its never
failed me yet, and I like the way /dev only contains stuff that actually
exists.

And if Al doesnt like the code... hes free to reimplement it...

Re: devfs

[john slee]

On Tue, Nov 12, 2002 at 10:43:41AM +0100, Xavier Bestel wrote:
> I'm wondering if a totally userspace solution could replace devs ?
> Something using hotplug + sysfs and creating directories/nodes as they
> appear on the system. This way, the policy (how do I name what) could be
> moved out of the kernel.

curious!  you mean similar to (and a logical extension of) the 'disks'
command in solaris?  at least i think thats what its called...

j.

-- 
toyota power: http://indigoid.net/

Re: devfs

[Sean Neakums]

commence  john slee quotation:

> On Tue, Nov 12, 2002 at 10:43:41AM +0100, Xavier Bestel wrote:
>> I'm wondering if a totally userspace solution could replace devs ?
>> Something using hotplug + sysfs and creating directories/nodes as
>> they appear on the system. This way, the policy (how do I name
>> what) could be moved out of the kernel.
>
> curious!  you mean similar to (and a logical extension of) the
> 'disks' command in solaris?  at least i think thats what its
> called...

Except that a /sbin/hotplug er, "solution", would be dynamic.  I had
always assumed in the back of my mind that such a solution would make
devfs go away some day, although I don't think I actually read about
it anywhere.

Actually, here's a question: are /sbin/hotplug upcalls serialized in
some fashion?  I'd hate to online a thousand devices in my disk array
and have the machine forkbomb itself.

-- 
 /                          |
[|] Sean Neakums            |  Questions are a burden to others;
[|] <sneakums@zork.net>     |      answers a prison for oneself.
 \                          |

Re: devfs

[Oliver Neukum]

> Actually, here's a question: are /sbin/hotplug upcalls serialized in
> some fashion?  I'd hate to online a thousand devices in my disk array
> and have the machine forkbomb itself.

Nope, no serialisation. You don't have any guarantee even that
addition will be delivered before removal.

	Regards
		Oliver

Re: hotplug (was devfs)

[Nick Craig-Wood]

On Tue, Nov 12, 2002 at 01:51:08PM +0100, Oliver Neukum wrote:
> > Actually, here's a question: are /sbin/hotplug upcalls serialized in
> > some fashion?  I'd hate to online a thousand devices in my disk array
> > and have the machine forkbomb itself.
> 
> Nope, no serialisation. You don't have any guarantee even that
> addition will be delivered before removal.

And that is why (we finally discovered) we were getting
non-deterministic device numbering of USB nodes.

We have machines with 6 x 4 port USB <-> serial converters attached.
These would get randomly assigned usb device ids and hence random
/dev/ttyUSB nodes.  Not very useful when there is a load of different
things attached to the 24 serial ports!

Sometimes we also found that one of the devices wouldn't get
initialised properly.

We fixed these problems by removing hotplug and loading the relevant
kernel modules in the correct order and voila a perfectly
deterministic order for the /dev/ttyUSBs with all devices initialised.

Plugging in our USB bus with 24 devices on it does indeed produce a
mini-forkbomb effect ;-) (Especially since these Keyspan devices are
initialised twice - once without firmware and once with firmware.)

So - perhaps hotplug ought to be serialised?

-- 
Nick Craig-Wood
ncw1@axis.demon.co.uk

Re: hotplug (was devfs)

[Greg KH]

On Wed, Nov 13, 2002 at 10:48:09AM +0000, Nick Craig-Wood wrote:
> 
> We fixed these problems by removing hotplug and loading the relevant
> kernel modules in the correct order and voila a perfectly
> deterministic order for the /dev/ttyUSBs with all devices initialised.

deterministic for you :)

What hotplug will do is allow you to assign a /dev entry to a specific
device, so that you can go off of the topology, and not just the order
in which the devices are found.  That is how this problem will be
solved properly.

> Plugging in our USB bus with 24 devices on it does indeed produce a
> mini-forkbomb effect ;-) (Especially since these Keyspan devices are
> initialised twice - once without firmware and once with firmware.)
> 
> So - perhaps hotplug ought to be serialised?

No, it's not needed for this problem.  There has been talk of
serializing stuff in userspace, which is the proper way to handle some
of the remove before add was seen problems.

thanks,

greg k-h

Re: hotplug (was devfs)

[Nick Craig-Wood]

On Wed, Nov 13, 2002 at 09:02:04AM -0800, Greg KH wrote:
> On Wed, Nov 13, 2002 at 10:48:09AM +0000, Nick Craig-Wood wrote:
> > 
> > We fixed these problems by removing hotplug and loading the relevant
> > kernel modules in the correct order and voila a perfectly
> > deterministic order for the /dev/ttyUSBs with all devices initialised.
> 
> deterministic for you :)

Indeed!  It was deterministic in the sense of

1) I booted the machine 10 times and the devices came up in the same
order ;-)

2) the order of the devices was related to the usb topology, like this
(these are usb bus positions as noted by hub.c)

  1/1     <- hub1
  1/1/1   <- keyspan 1 /dev/ttyUSB0..3
  1/1/2   <- keyspan 2 /dev/ttyUSB4..7
  1/1/3   <- keyspan 3 /dev/ttyUSB8..11
  1/1/4   <- hub2
  1/1/4/1 <- keyspan 4 /dev/ttyUSB12..15
  1/1/4/2 <- keyspan 5 /dev/ttyUSB16..19
  1/1/4/3 <- keyspan 6 /dev/ttyUSB20..23

That seemed like a sensible order to me!

> What hotplug will do is allow you to assign a /dev entry to a specific
> device, so that you can go off of the topology, and not just the order
> in which the devices are found.  That is how this problem will be
> solved properly.

So I'll be able to say usb bus1/1/4/1 port 3 should be /dev/ttyUSB15
and it will always be that port?  That would be perfect.

> > Plugging in our USB bus with 24 devices on it does indeed produce a
> > mini-forkbomb effect ;-) (Especially since these Keyspan devices are
> > initialised twice - once without firmware and once with firmware.)
> > 
> > So - perhaps hotplug ought to be serialised?
> 
> No, it's not needed for this problem.  There has been talk of
> serializing stuff in userspace, which is the proper way to handle some
> of the remove before add was seen problems.

Userspace serialisation would have solved this problem for us too I
think without the extra mapping mechanism.

-- 
Nick Craig-Wood
ncw1@axis.demon.co.uk

Re: hotplug (was devfs)

[Greg KH]

On Wed, Nov 13, 2002 at 06:06:06PM +0000, Nick Craig-Wood wrote:
> 
> So I'll be able to say usb bus1/1/4/1 port 3 should be /dev/ttyUSB15
> and it will always be that port?  That would be perfect.

Yes, that is the goal.

thanks,

greg k-h

Re: hotplug (was devfs)

[Oliver Neukum]

> > So - perhaps hotplug ought to be serialised?
>
> No, it's not needed for this problem.  There has been talk of
> serializing stuff in userspace, which is the proper way to handle some
> of the remove before add was seen problems.

We may need some kind of load control.
The thought of firing up hundreds of hotplug scripts
simultaneously is not pretty.

	Regards
		Oliver