Update on hot plugging serialization without kernel event queuing

Update on hot plugging serialization without kernel event queuing

[Adam J. Richter]

	A couple of weeks ago on this list, I described an algorithm
to support hotplug event synchronization without an event queue.  Oliver
Neukum and I got into a long discussion about it, which I took to private
email, promising to post the conclusions to this list.  So, here goes.

	The algorithm that I originally posted would theoretically work,
but the way most shell scripts and network daemons are written would
introduce race conditions that would be solvable only by modifying
any daemons and helper program do things like take a device ID on the
command line and abort if the device's ID no longer matches (because
somebody swapped the card in the interim).

	Fortunately, there is an approach which would keep the kernel
and user level code quite simple.  You can avoid having race conditions
and avoid adding event queues in the kernel by following one simple rule:
once hot plugging is enabled, the mechanism to enable access to a device
that has been inserted should be activated from user level, not from
the kernel.  This allows the user level agent to acquire a lock
corresponding to the inserted device _before_ it activates the slot.

	Presumably, activation of the inserted device would be done by
some ioctl, which could be same for all bus types, so the core hot plugging
user level support code could be written not to care about what kind
of hotplug bus it was handling an event for, and, indeed, would work
unmodified for new types of busses.

	The advantages of this approach are:

		o Requires only minor kernel changes.

		o Events only block on each other when necessary.
		  (E.g., processing slotA events to do not block the
		  processing of slotB events).

		o Locking done in userland enables more flexibility.
		  E.g., embedded systems that know there devices are
		  not truely removable could delete the locking code;
		  user interface events like beeps could occur without
		  waiting on the locks.

		o Safe transition when hotplugging is being activated.
		  Activate hotplugging and then call /sbin/hotplug on
		  each already existing slot.  The worst that will happen
		  is a fake remove and insert event if a device is inserted
		  during the transition.

	Anyhow, here an appendix that shows pseudo-code implementing
this approach.


The kernel code would look something like this:

insertion_event(slot) {
	regiseter_in_proc_bus(slot);
	// The device now appears in /proc, but all operations will
	// return -EIO, execpt open() and the ioctl to activate the slot.
	// This way, if there is initialization for a device previously
	// occupying this slot, that initialization will fail out.

	if (hotplugging_enabled)
		userland_helper("/sbin/hotplug", slot, 0);
		// Do not wait for hotplug to return.
	else
		ioctl_enable_slot(slot);
		 // We could delete this branch if we do not want to
		 // support an initial root on a hotpluggable device.
}

ioctl_enable_slot(slot) {
	if (!slot->device_present)	// Device has since been removed
		return -ENODEV;
	if (!slot->enabled) {
		slot->enabled = 1; 	// Allow IO operations.
		bind_existing_kernel_drivers(slot);
		// ^^ Binding drivers could (should?) be a separeate ioctl.
	}
}

ioctl_enable_hotplugging(void) {
	hotplugging_enabled = 1;
}


/sbin/hotplug looks something like this:

userland_hotplug_handler(slot)
{
	acquire_lock(slot); // Could be implemented with flock on a file,
			    // SysV sempahore, or some other way.

	slotfd = open(slot, O_RDWR);

	// In some super-rare where a user inserts and removes a card
	// a bunch of times very quickly, we could optimize away
	// some of the insert and remove events by having an ioctl
	// to check the value slot->enabled, but that basically never
	// happens, so it's not worth implementing that optimization.

	sprintf(remove_hanlder, "/var/run/remove/%s", slot);
	if (access(remove_handler, X_OK))
		system(remove_handler);
	unlink(remove_handler);
	if (ioctl(slotfd, ENABLE_SLOT, NULL) = 0) {
		handle_insert_event(slot);
		// ^^ This is where the device's ID information is read.
		// Device-specific drivers that this routine calls
		// will create remove_handler script if necessary.
	}
	close(slotfd);
        release_lock(slot);
}


Note to Oliver Neukum and Miles Lane: the was_plugged_in[] persistent
variable described in my previous examples is replaced in the pseudo-code
here with the "/var/run/remove/%s" file.

Many thanks to Miles Lane for reminding me to post the results of
this discussion already.

Adam J. Richter     __     ______________   4880 Stevens Creek Blvd, Suite 104
adam@yggdrasil.com     \ /                  San Jose, California 95129-1034
+1 408 261-6630         | g g d r a s i l   United States of America
fax +1 408 261-6631      "Free Software For The Rest Of Us."

_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: Update on hot plugging serialization without kernel event queuing

[David Brownell]

----- Original Message ----- 
From: "Adam J. Richter" <adam@yggdrasil.com>
To: <linux-hotplug-devel@lists.sourceforge.net>
Sent: Saturday, February 24, 2001 8:25 PM
Subject: Update on hot plugging serialization without kernel event queuing

>    ... deletia ...
>
> Fortunately, there is an approach which would keep the kernel
> and user level code quite simple.  You can avoid having race conditions
> and avoid adding event queues in the kernel by following one simple rule:
> once hot plugging is enabled, the mechanism to enable access to a device
> that has been inserted should be activated from user level, not from
> the kernel.  This allows the user level agent to acquire a lock
> corresponding to the inserted device _before_ it activates the slot.

That's a good basic strategy that should certainly work.  Though "slot"
sounds rather specific to one kind of hardware device ... and at another
level that's what "mount" does for disk-like storage, so I wonder which
subsystems aren't now supporting this strategy.  (And PCI seems to
be factored in terms of "function" ...  unlike USB right now.)

It'd be good if the "configure through sysadmin tool" cases and hotplug
alert handling used some common "enable now-connected device"
back-end code.  It sounds to me like that's what you're suggesting
be created.  The "CompactPCI" and "Hotplug PCI" hotplugging
will certainly need a tool to issue their bus-specific ioctls.  (I hope the
user interface differs only in use of blue vs red leds!)  We know we
need such a facility for PCMCIA ("cardctl").

I'm not sure of how the details would work though.  How would
what you're saying affect some of the hotplugging work that's gotten
discussed so far, like SCSI, Cardbus, PCI, PCMCIA, and USB?

- Dave



_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
http://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel