Re: waiting for an unknown set of udev /dev entries to complete

[Kay Sievers <kay.sievers@vrfy.org>]

On Sun, Nov 20, 2005 at 09:25:29AM -0800, Patrick Mansfield wrote:
> On Sun, Nov 20, 2005 at 07:03:57AM +0100, Kay Sievers wrote:
> > On Fri, Nov 18, 2005 at 02:30:45PM -0800, Patrick Mansfield wrote:
> 
> > > That is, if I load a module or otherwise generate an unknown number of
> > > hotplug events for a single action or event - like a modprobe of a scsi
> > > HBA or even modprobe sd_mod - how can I tell when all hotplug events have
> > > been handled and /dev entries created?
> > 
> > You should be able to look for /dev/.udev/queue/*block*, and see if
> > there are events still running.
> 
> OK, (I think) I need this for current fc devel, udev 075, so I'll use the
> method to wait for all /dev/sd's to show up.

It will hit Fedora devel pretty soon after the release, I expect.

> If udev logging is off, I might not even hit this problem. For slow
> devices (perhaps iSCSI for a congested network and a slow responding
> target/disk), it could still be a problem.

Hmm, we should better handle this.

> > > 1) (I'm looking for scsi block devices) Waiting for an equal number of
> > > /dev/sd* nodes to show up as is found under /sys/block/sd* and
> > > /sys/block/sd*/sd*. Is specific, and failure modes are kind of ugly (a
> > > timeout could be used, but if hotplug is really slow and is making
> > > progress you don't want to stop waiting).
> > > 
> > > 2) Waiting for all hotplug processes to finish.  This is more general
> > > purpose (not scsi or block specific, but doesn't handle children of
> > > kernel.hotplug that are not named the same (at least my script below has
> > > this problem).
> > 
> > You may not detach from the event process for short living programs,
> > that will leave the event in the queue. If you need to do really complicated
> > things, you may need to synchronize the events yourself at a higher level.
> 
> You mean you can't wait for a particular process to finish? 
> 
> Otherwise, I don't understand how the "event queue" relates to the
> problem. I see udev has a list of msg's. It forks udev_event_process, it
> will exec multiple run programs. When udev_event_process, reap_sigchilds()
> calls udev_done() and it removes the msg.

Yes, but if you try to resolve complex dependencies with udev event
processes itself, like let the event look at the queue, you have to be
very careful no to run into weird locking situations. It may work,
but I never tried this.
Today, only events for the parent devices or the underlying physical
devices are serialized by the udev daemon.

It may be better to hook asynchronously in the events for the actual
device events which have been created by the module load, instead of
just to continue with the setup script after loading the modules.
Depends on your setup, if that is feasible.

> > There is no "kernel.hotplug" that runs anything. Usually it's "udevsend",
> > which just passes the event to the udev daemon. On SUSE it's disabled
> > completely for a long time.
> 
> OK, I should have remembered that (even though I'm not running SUSE) ...

It is this way on Fedora since HAL depends on udev to cover the sysfs timing
issues.

> > The udev daemon listens to netlink and /sbin/hotplug does almost nothing
> > on all recent systems. It is only used for the "input" system which lacked
> > driver core support until 2.6.15.
> 
> > > Module loads should check this during boot and install, but I haven't
> > > found such checks (so far, I see some stuff in suse network code but that
> > > is for networks). Most cases will just work for a small number of disks
> > > and a sufficient interval between module load and mounting (for boot) or
> > > use (for an install) of the intended device.
> > 
> > What do you mean with: "Module loads should check this" in this case?
> 
> After you load a module, like a SCSI HBA driver or even sd_mod, you should
> really wait for the devices to show up before you try and use them.  For
> initrd, this means you should wait for the root /dev to show up (if it has
> a /sys/block/XX entry); for installs, this means wait for all /dev entries
> to show up.
> 
> People have asked about this for general hotplug (plugging in a device,
> loading a module, then immediately trying to access the /dev/XXX node),
> but I haven't seen reports of problem for this on normal boot (initrd root
> mounting) or for installs, maybe iSCSI makes hitting this more likely.

Yes, if you know what you are waiting for, like the root device, you
just may spin until it arrives, for all other cases, watching the event queue
should work pretty well.

> > > With current FC rawhide/devel, I am working on iscsi install, and had udev
> > > "error" logging enabled on a medium speed dual processor system. So the
> > > scripts are very slow, and the partition setup code is being run right
> > > after iscsi scan (really discovery/login and then scan) and it does not
> > > see all the devices (and I probably have some bugs still). 
> > 
> > Hmm, care to explain that better, I don't really understand...
> 
> Just that udev /dev creation is slow with udev_log="err", so after a
> login to an iSCSI target completes (iSCSI tools issue a scan of the target
> via user space), the /dev entries are not available for about 60 seconds.
> 
> So if immediately after the iSCSI login, if we look for or try to use /dev
> block install targets, we can have problems.
> 
> > > It takes over a second per scsi sd, and I have about 60 of them.
> > 
> > But these are not serialized, are they?
> 
> Right. I can see multiple processes running. I don't know why they are
> _so_ slow. With udev_log=info, they run fairly fast (i.e. I don't think
> there is a problem), but with udev_log=err, it's really slow, so I guess
> all that the logging is the bottleneck (maybe some fsync in syslog?).

Yes, prefix the logfile in syslog.conf with a "-", which is not the default
on Fedora, and this will go away.

> How can I change the udev_log level without rebooting?

You can change the udev daemons internal setting with:
  udevcontrol log_priority=<level/number>

And you may want to add:
  RUN+="socket:/org/kernel/udev/monitor"
as one of the last rules to your udev rule set, I think Fedora does not
have this by default. Then run:
  udevmonitor
and watch the exact event timing - from the kernel event, until the actual
udev event was handled. That is a lot easier than reading syslog.

Kay


-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
Register for a JBoss Training Course.  Free Certification Exam
for All Training Attendees Through End of 2005. For more info visit:
http://ads.osdn.com/?ad_idv28&alloc_id\x16845&op=click
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel