Re: [PATCH v8 2/4] fpga manager: add sysfs interface document

[Jason Gunthorpe <jgunthorpe@obsidianresearch.com>]

On Thu, Jan 15, 2015 at 08:45:02PM +0000, One Thousand Gnomes wrote:

> > - Hand over to a DT overlay (how does this work?) Lock transfers
> >   from FD to kernel
> 
> That bit isn't stateful so I would actually have expected something in
> the kernel ABI along the lines of 
> 
>            request_fpga(blah)
> 
> which does
> 
>              if in use by user
>                     EBUSY
>              lock it (all user opens will fail)
> 
> and
> 
>             release_fpga(blah)

I am imagining two ways to start a kernel FPGA, the in-kernel method
triggered by a DT node:

  fpga = request_and_lock_fpga(of_get_fpga_controller(dev->of_node))
  fw = request_firmat(of_get_fpga_firmware(dev->of_node));
  fpga_program_fw(fpga, fw);

  for_each_child_of_node(dev->of_node, child)
    .. of_platform_bus_probe(child ... ) ..

  .. somehow fpga and its lock transfer to dev->of_node ..

The problem with this is it assumes the FPGA is ready to go
immediately after fpga_program_fw. There are a few platforms that can
manage this, but many others require at least some kind of startup
sequence - eg wait for clocking PLLs to lock, do low level setup, etc.

For very common cases (like Zynq can have a pretty common setup scheme
for the PL side) the DT binding can guide the kernel to run the right
code, and the code can live in the kernel because it is simple and
broadly useful.

For wild cases I'd like to just punt the setup code to user space. It
is safer and simpler to run that complexity in a user process than in
the kernel.

Maybe there is a better way to handle this, but I have been under the
impression that these days it is frowned on for the kernel to wait for
userspace?

So my idea is to use the user space method to also load a 'kernel'
fpga, the process follows the kernel flow:

   fd = open("/dev/fpga0"); // request_and_lock_fpga

   ioctl(fd,START_PROGRAMMING); // fpga_program_fw
   write(fd,fw,fw_size);
   
   // Arbitary complexity here
   userspace_setup_fpga();

   icotl(fd,BIND_DT_OVERLAY, .. ?? ..) // for_each_child_of_node

This is what the state is about, if the setup fails I expect the
kernel to go and unlock and deprogram the FPGA if fd is closed. Only a
success on BIND_DT_OVERLAY will make the FPGA permanent beyond closing
fd.

Pantelis: I think this explains why a fd and ioctls. configfs, sysfs,
etc lack the FD context to do the cleanup. 

Essentially, I think a running FPGA should always be attached to some
context that is the user - either the user is in-kernel (a DT of_node)
or the user is in userspace (the fd).

The invarient is - if there is no user then the kernel automatically
makes the FPGA deprogrammed.

Having a device that is potentially attached to the CPU bus running
'rouge' is dangerous. We can't realistically deprogram it safely if we
don't know what it is doing. Eg deprogramming in the middle of a DMA
operation between the CPU and FPGA will often crash the entire system.

The only way to provide reasonable safe guards is to have a clear
chain of ownership. When the kernel takes ownership of the FPGA it
also takes ownership of any cleanup required prior to deprogram.

Jason