Question about multiple modules talking to one adapter

Question about multiple modules talking to one adapter

[Timur Tabi]

Hi,

We have a PCI adapter that supports three different APIs / interfaces.  The adapter is an
OpenIB device, a regular 10GB ethernet Netdev device, and there's also a proprietary
interface called CCIL that we invented.  The CCIL interface is really just a bunch of
custom IOCtls.

The problem we have is that our adapter appears as just one PCI device, so it has one
memory buffer and one IRQ.  All three interfaces need access to adapter memory and need an
ISR.

We're going to have a separate module for each interface.  For simplicity sake, I'll call
these openib.ko, netdev.ko, and ccil.ko.  openib.ko will be in the drivers/inifiniband
directory. netdev.ko will be in the drivers/network directory.  I don't know yet where
we'll put ccil.ko.

The way I see it, there are four different ways to implement these drivers, and I would
like to know which method the Linux community would prefer:

1) Each driver registers its own ISR and has its own mapping the adapter memory.  This is
the simplest approach, but the problem is that every time an ISR is called, it does a read
across the PCI bus to determine whether the interrupt is for it.  Believe it or not, this
is actually pretty expensive in terms of performance.  If anyone has an idea on how to
cleanly solve that particular problem, then this is the approach we'll take.  Otherwise,
we'd rather do implement one of the other two methods.

2) Create a single driver which does nothing but register an ISR and map the kernel
memory.  Let's call this the CRM driver.  The other three drivers can then use XXXXXX to
provide callbacks for the ISR and obtain the address of the mapping.  The ISR will then
query the adapter and call the appropriate callback.

3) A variation on #2: Instead of having a separate driver with the CRM code, one of the
three modules will have that code.  The other two modules will then have a dependency on
that first module.  The problem is that I don't know which of the three modules should
have the CRM.  Plus, this creates an artificial dependency.

4) Another variation on #2: Each module has a copy of the CRM code, but only the CRM in
the first module that's loaded is used.  As each module loads, it tries to find one of the
other two modules.  If it does find one of the other modules, it uses that other module's
CRM instead of its own.  Otherwise, it exports its own CRM entry points.

Thanks.

-- 
Timur Tabi
Staff Software Engineer
timur.tabi@ammasso.com

One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
      -- Ed Smylie, NASA engineer for Apollo 13

--
Kernelnewbies: Help each other learn about the Linux kernel.
Archive:       http://mail.nl.linux.org/kernelnewbies/
FAQ:           http://kernelnewbies.org/faq/

Re: Question about multiple modules talking to one adapter

[Roland Dreier]

    Timur> 1) Each driver registers its own ISR and has its own
    Timur> mapping the adapter memory.

This is silly and leads to all sorts of horrible code.  For example
you'll have to deal with the fact that only one driver can do
request_mem_region() on the PCI adapter, and no matter what you come
up with, it's going to be ugly.

    Timur> 2) Create a single driver which does nothing but register
    Timur> an ISR and map the kernel memory.  Let's call this the CRM
    Timur> driver.  The other three drivers can then use XXXXXX to
    Timur> provide callbacks for the ISR and obtain the address of the
    Timur> mapping.  The ISR will then query the adapter and call the
    Timur> appropriate callback.

This is the best solution.  I don't see any disadvantage to this.

 - R.