Re: [RFC] New Driver Model for 2.5

[Jeff Garzik <jgarzik@mandrakesoft.com>]

Patrick Mochel wrote:
> 
> One July afternoon, while hacking on the pm_dev layer for the purpose of
> system-wide power management support, I decided that I was quite tired of
> trying to make this layer look like a tree and feel like a tree, but not
> have any real integration with the actual device drivers..
> 
> I had read the accounts of what the goals were for 2.5. And, after some
> conversations with Linus and the (gasp) ACPI guys, I realized that I had a
> good chunk of the infrastructural code written; it was a matter of working
> out a few crucial details and massaging it in nicely.
> 
> I have had the chance this week (after moving and vacationing) to update
> the (read: write some) documentation for it. I will not go into details,
> and will let the document speak for itself.
> 
> With all luck, this should go into the early stages of 2.5, and allow a
> significant cleanup of many drivers. Such a model will also allow for neat
> tricks like full device power management support, and Plug N Play
> capabilities.
> 
> In order to support the new driver model, I have written a small in-memory
> filesystem, called ddfs, to export a unified interface to userland. It is
> mentioned in the doc, and is pretty self-explanatory. More information
> will be available soon.
> 
> There is code available for the model and ddfs at:
> 
> http://kernel.org/pub/linux/kernel/people/mochel/device/
> 
> but there are some fairly large caveats concerning it.
> 
> First, I feel comfortable with the device layer code and the ddfs
> code. Though, the PCI code is still work in progress. I am still working
> out some of the finer details concerning it.
> 
> Next is the environment under which I developed it all. It was on an ia32
> box, with only PCI support, and using ACPI. The latter didn't have too
> much of an effect on the development, but there are a few items explicitly
> inspired by it..
> 
> I am hoping both the PCI code, and the structure and in general can be
> further improved based on the input of the driver maintainers.
> 
> This model is not final, and may be way off from what most people actually
> want. It has gotten tentative blessing from all those that have seen it,
> though they number but a few. It's definitely not the only solution...
> 
> That said, enjoy; and have at it.
> 
>         -pat
> 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> The (New) Linux Kernel Driver Model
> 
> Version 0.01
> 
> 17 October 2001
> 
> Overview
> ~~~~~~~~
> 
> This driver model is a unification of all the current, disparate driver models
> that are currently in the kernel. It is intended is to augment the
> bus-specific drivers for bridges and devices by consolidating a set of data
> and operations into globally accessible data structures.
> 
> Current driver models implement some sort of tree-like structure (sometimes
> just a list) for the devices they control. But, there is no linkage between
> the different bus types.
> 
> A common data structure can provide this linkage with little overhead: when a
> bus driver discovers a particular device, it can insert it into the global
> tree as well as its local tree. In fact, the local tree becomes just a subset
> of the global tree.
> 
> Common data fields can also be moved out of the local bus models into the
> global model. Some of the manipulation of these fields can also be
> consolidated. Most likely, manipulation functions will become a set
> of helper functions, which the bus drivers wrap around to include any
> bus-specific items.
> 
> The common device and bridge interface currently reflects the goals of the
> modern PC: namely the ability to do seamless Plug and Play, power management,
> and hot plug. (The model dictated by Intel and Microsoft (read: ACPI) ensures
> us that any device in the system may fit any of these criteria.)
> 
> In reality, not every bus will be able to support such operations. But, most
> buses will support a majority of those operations, and all future buses will.
> In other words, a bus that doesn't support an operation is the exception,
> instead of the other way around.
> 
> Drivers
> ~~~~~~~
> 
> The callbacks for bridges and devices are intended to be singular for a
> particular type of bus. For each type of bus that has support compiled in the
> kernel, there should be one statically allocated structure with the
> appropriate callbacks that each device (or bridge) of that type share.
> 
> Each bus layer should implement the callbacks for these drivers. It then
> forwards the calls on to the device-specific callbacks. This means that
> device-specific drivers must still implement callbacks for each operation.
> But, they are not called from the top level driver layer.
> 
> This does add another layer of indirection for calling one of these functions,
> but there are benefits that are believed to outweigh this slowdown.
> 
> First, it prevents device-specific drivers from having to know about the
> global device layer. This speeds up integration time incredibly. It also
> allows drivers to be more portable across kernel versions. Note that the
> former was intentional, the latter is an added bonus.
> 
> Second, this added indirection allows the bus to perform any additional logic
> necessary for its child devices. A bus layer may add additional information to
> the call, or translate it into something meaningful for its children.
> 
> This could be done in the driver, but if it happens for every object of a
> particular type, it is best done at a higher level.
> 
> Recap
> ~~~~~
> 
> Instances of devices and bridges are allocated dynamically as the system
> discovers their existence. Their fields describe the individual object.
> Drivers - in the global sense - are statically allocated and singular for a
> particular type of bus. They describe a set of operations that every type of
> bus could implement, the implementation following the bus's semantics.
> 
> Downstream Access
> ~~~~~~~~~~~~~~~~~
> 
> Common data fields have been moved out of individual bus layers into a common
> data structure. But, these fields must still be accessed by the bus layers,
> and
> sometimes by the device-specific drivers.
> 
> Other bus layers are encouraged to do what has been done for the PCI layer.
> struct pci_dev now looks like this:
> 
> struct pci_dev {
>         ...
> 
>         struct device device;
> };
> 
> Note first that it is statically allocated. This means only one allocation on
> device discovery. Note also that it is at the _end_ of struct pci_dev. This is
> to make people think about what they're doing when switching between the bus
> driver and the global driver; and to prevent against mindless casts between
> the two.
> 
> The PCI bus layer freely accesses the fields of struct device. It knows about
> the structure of struct pci_dev, and it should know the structure of struct
> device. PCI devices that have been converted generally do not touch the fields
> of struct device. More precisely, device-specific drivers should not touch
> fields of struct device unless there is a strong compelling reason to do so.
> 
> This abstraction is prevention of unnecessary pain during transitional phases.
> If the name of the field changes or is removed, then every downstream driver
> will break. On the other hand, if only the bus layer (and not the device
> layer) accesses struct device, it is only those that need to change.
> 
> User Interface
> ~~~~~~~~~~~~~~
> 
> By virtue of having a complete hierarchical view of all the devices in the
> system, exporting a complete hierarchical view to userspace becomes relatively
> easy. Whenever a device is inserted into the tree, a file or directory can be
> created for it.
> 
> In this model, a directory is created for each bridge and each device. When it
> is created, it is populated with a set of default files, first at the global
> layer, then at the bus layer. The device layer may then add its own files.
> 
> These files export data about the driver and can be used to modify behavior of
> the driver or even device.
> 
> For example, at the global layer, a file named 'status' is created for each
> device. When read, it reports to the user the name of the device, its bus ID,
> its current power state, and the name of the driver its using.
> 
> By writing to this file, you can have control over the device. By writing
> "suspend 3" to this file, one could place the device into power state "3".
> Basically, by writing to this file, the user has access to the operations
> defined in struct device_driver.
> 
> The PCI layer also adds default files. For devices, it adds a "resource" file
> and a "wake" file. The former reports the BAR information for the device; the
> latter reports the wake capabilities of the device.
> 
> The device layer could also add files for device-specific data reporting and
> control.
> 
> The dentry to the device's directory is kept in struct device. It also keeps a
> linked list of all the files in the directory, with pointers to their read and
> write callbacks. This allows the driver layer to maintain full control of its
> destiny. If it desired to override the default behavior of a file, or simply
> remove it, it could easily do so. (It is assumed that the files added upstream
> will always be a known quantity.)
> 
> These features were initially implemented using procfs. However, after one
> conversation with Linus, a new filesystem - ddfs - was created to implement
> these features. It is an in-memory filesystem, based heavily off of ramfs,
> though it uses procfs as inspiration for its callback functionality.
> 
> Device Structures
> ~~~~~~~~~~~~~~~~~
> 
> struct device {
>         struct list_head        bus_list;
>         struct io_bus           *parent;
>         struct io_bus           *subordinate;
> 
>         char                    name[DEVICE_NAME_SIZE];
>         char                    bus_id[BUS_ID_SIZE];
> 
>         struct dentry           *dentry;
>         struct list_head        files;
> 
>         struct  semaphore       lock;
> 
>         struct device_driver    *driver;
>         void                    *driver_data;
>         void                    *platform_data;
> 
>         u32                     current_state;
>         unsigned char           *saved_state;
> };
> 
> bus_list:
>         List of all devices on a particular bus; i.e. the device's siblings
> 
> parent:
>         The parent bridge for the device.
> 
> subordinate:
>         If the device is a bridge itself, this points to the struct io_bus that is
>         created for it.
> 
> name:
>         Human readable (descriptive) name of device. E.g. "Intel EEPro 100"
> 
> bus_id:
>         Parsable (yet ASCII) bus id. E.g. "00:04.00" (PCI Bus 0, Device 4, Function
>         0). It is necessary to have a searchable bus id for each device; making it
>         ASCII allows us to use it for its directory name without translating it.
> 
> dentry:
>         Pointer to driver's ddfs directory.
> 
> files:
>         Linked list of all the files that a driver has in its ddfs directory.
> 
> lock:
>         Driver specific lock.
> 
> driver:
>         Pointer to a struct device_driver, the common operations for each device. See
>         next section.
> 
> driver_data:
>         Private data for the driver.
>         Much like the PCI implementation of this field, this allows device-specific
>         drivers to keep a pointer to a device-specific data.
> 
> platform_data:
>         Data that the platform (firmware) provides about the device.
>         For example, the ACPI BIOS or EFI may have additional information about the
>         device that is not directly mappable to any existing kernel data structure.
>         It also allows the platform driver (e.g. ACPI) to a driver without the driver
>         having to have explicit knowledge of (atrocities like) ACPI.
> 
> current_state:
>         Current power state of the device. For PCI and other modern devices, this is
>         0-3, though it's not necessarily limited to those values.
> 
> saved_state:
>         Pointer to driver-specific set of saved state.
>         Having it here allows modules to be unloaded on system suspend and reloaded
>         on resume and maintain state across transitions.
>         It also allows generic drivers to maintain state across system state
>         transitions.
>         (I've implemented a generic PCI driver for devices that don't have a
>         device-specific driver. Instead of managing some vector of saved state
>         for each device the generic driver supports, it can simply store it here.)
> 
> struct device_driver {
>         int     (*probe)        (struct device *dev);
>         int     (*remove)       (struct device *dev);
> 
>         int     (*init)         (struct device *dev);
>         int     (*shutdown)     (struct device *dev);
> 
>         int     (*save_state)   (struct device *dev, u32 state);
>         int     (*restore_state)(struct device *dev);
> 
>         int     (*suspend)      (struct device *dev, u32 state);
>         int     (*resume)       (struct device *dev);
> }
> 
> probe:
>         Check for device existence and associate driver with it.
> 
> remove:
>         Dissociate driver with device. Releases device so that it could be used by
>         another driver. Also, if it is a hotplug device (hotplug PCI, Cardbus), an
>         ejection event could take place here.
> 
> init:
>         Initialise the device - allocate resources, irqs, etc.
> 
> shutdown:
>         "De-initialise" the device - release resources, free memory, etc.
> 
> save_state:
>         Save current device state before entering suspend state.
> 
> restore_state:
>         Restore device state, after coming back from suspend state.
> 
> suspend:
>         Physically enter suspend state.
> 
> resume:
>         Physically leave suspend state and re-initialise hardware.
> 
> Initially, the probe/remove sequence followed the PCI semantics exactly, but
> have since been broken up into a four-stage process: probe(), remove(),
> init(), and shutdown().
> 
> While it's not entirely necessary in all environments, breaking them up so
> each routine does only one thing makes sense.
> 
> Hot-pluggable devices may also benefit from this model, especially ones that
> can be subjected to suprise removals - only the remove function would be
> called, and the driver could easily know if the there was still hardware there
> to shutdown.
> 
> Drivers that are controlling failing, or buggy, hardware, by allowing the user
> to trigger a removal of the driver from userspace, without trying to shutdown
> down the device.
> 
> In each case that remove() is called without a shutdown(), it's important to
> note that resources will still need to be freed; it's only the hardware that
> cannot be assumed to be present.

So, remove() might be called without a shutdown(), and then asked to
perform the duties normally performed by shutdown()?  That sounds like
API dain bramage.  :)

Your proposal sounds ok, my one objection is separating probe/remove
further into init/shutdown.  Can you give real-life cases where this
will be useful?  I don't see it causing much except headache.

The preferred way of doing things (IMHO) is to do some simply sanity
checking of the h/w device at probe time, and then perform lots of
initialization and such at device/interface open time.  You ideally want
a device driver lifecycle to look like

probe:
	register interface
	sanity check h/w to make sure it's there and alive
	stop DMA/interrupts/etc., just in case
	start timer to powerdown h/w in N seconds

dev_open:
	wake up device, if necessary
	init device

dev_close:
	stop DMA/interrupts/etc.
	start timer to powerdown h/w in N seconds

With that in mind, init -really- happens at device open, and in
additional is driven more through normal user interaction via standard
APIs, than the PCI and PM subsystems.

-- 
Jeff Garzik      | "Mind if I drive?" -Sam
Building 1024    | "Not if you don't mind me clawing at the dash
MandrakeSoft     |  and shrieking like a cheerleader." -Max