Trying to reset a PCIe device and scratching my head...

Trying to reset a PCIe device and scratching my head...

[Jimmie Mayfield]

Hello.  I'm in the process of writing a device driver for a PCIe
device where the bulk of the logic, including the PCIe interface, is 
implemented on a couple of large FPGAs.  Current target platform is x86 
Linux.

Having the PCIe interface implemented inside FPGA 'A' makes upgrading
that particular FPGA rather troublesome.  In a perfect world, one would
be able to upgrade the FPGA without having to reboot the machine.  The
hardware guys have designed the card to reload that FPGA image upon a
slot reset...either fundamental or hot.  (If I could guarantee the
card would always be plugged into a hotplug-capable slot, I suppose I could
forge a hotplug event and toggle power to the slot.  But I can't impose
that requirement on customers.)

So I'd like to be able to send either a fundamental or hot reset to the
device but so far I've had no success.  Is there a recommended way of doing 
this on x86 Linux?  

Jimmie

Re: Trying to reset a PCIe device and scratching my head...

[Clemens Ladisch]

Jimmie Mayfield wrote:
> Having the PCIe interface implemented inside FPGA 'A' makes upgrading
> that particular FPGA rather troublesome.  In a perfect world, one would
> be able to upgrade the FPGA without having to reboot the machine.  The
> hardware guys have designed the card to reload that FPGA image upon a
> slot reset...either fundamental or hot.
> 
> So I'd like to be able to send either a fundamental or hot reset to the
> device but so far I've had no success.

The PCIe AER driver (drivers/pci/pcie/aer/) sends a hot reset when
it has received a fatal error.  As far as I can tell, it just sets the
PCI_BRIDGE_CTL_BUS_RESET bit of the upstream bridge; everything else
is just infrastructure to handle error reporting and to notify the
device driver about the reset.


Regards,
Clemens

Re: Trying to reset a PCIe device and scratching my head...

[Jesse Barnes]

On Tue, 28 Sep 2010 09:28:14 +0200
Clemens Ladisch <clemens@ladisch.de> wrote:

> Jimmie Mayfield wrote:
> > Having the PCIe interface implemented inside FPGA 'A' makes upgrading
> > that particular FPGA rather troublesome.  In a perfect world, one would
> > be able to upgrade the FPGA without having to reboot the machine.  The
> > hardware guys have designed the card to reload that FPGA image upon a
> > slot reset...either fundamental or hot.
> > 
> > So I'd like to be able to send either a fundamental or hot reset to the
> > device but so far I've had no success.
> 
> The PCIe AER driver (drivers/pci/pcie/aer/) sends a hot reset when
> it has received a fatal error.  As far as I can tell, it just sets the
> PCI_BRIDGE_CTL_BUS_RESET bit of the upstream bridge; everything else
> is just infrastructure to handle error reporting and to notify the
> device driver about the reset.

Right, if your system supports this, it may be the easiest way to go.
But devices often have proprietary ways of resetting themselves too;
maybe you could change your device to reset if a specific bit in config
or MMIO space was flipped.  You'd likely need some sort of delay before
accessing the device again, but it should be bounded and fairly fixed,
so probably not a big deal for the driver to handle.

-- 
Jesse Barnes, Intel Open Source Technology Center

Re: Trying to reset a PCIe device and scratching my head...

[Jimmie Mayfield]

On Tue, Sep 28, 2010 at 09:06:55AM -0700, Jesse Barnes wrote:
> Right, if your system supports this, it may be the easiest way to go.

I was under the impression that PCIe requires support for at least
fundamental reset and maybe hot reset (secondary bus reset) though I'd
have to go back to the spec to verify the latter.  Are you saying that
not all PCIe chipsets support this?  Sorry if this seems like a dumb 
question...I'm by no means a PCIe guru.

> But devices often have proprietary ways of resetting themselves too;
> maybe you could change your device to reset if a specific bit in config
> or MMIO space was flipped.  You'd likely need some sort of delay before
> accessing the device again, but it should be bounded and fairly fixed,
> so probably not a big deal for the driver to handle.

Indeed, the device does support a proprietary reset via MMIO but 
that reset only affects those components that lie behind the card's PCIe 
interface.  So it's nearly a full device reset but not quite.  The FPGA 
containing the PCIe interface does not get reloaded.  Unfortunately, that 
behavior is likely set in stone.  

Jimmie