FPGA manager user space interface

FPGA manager user space interface

[Florian Fainelli]

Hi all,

I just wrote a FPGA manager driver for the TS-7300 board which features
an Altera Cyclone II on board. While the finished solution is my case
might be something like the MFD driver for the FPGA devices requesting
the bitstream load and registering the different devices it exposes,
during development it is nice to exercise the FPGA manager driver to
load something:

- a quick way is to add a pair of sysfs attributes to define the
bitstream filename and to trigger the load

- offer a more consistent and robust interface through e.g; a character
device that you can write/read/poll to see the loading progress about

- have the driver exposing FPGA peripherals be exposing an user space
interface to trigger a (re)load/(re)/configuration, although that's
really something that belongs in the FPGA manager it seems

I am mostly curious if these were taken into account during the initial
design and it is agreed upon that yes these are some of options and that
userspace loading is just anecdotal we do not need any userspace
interface, or if this is just missing and we want one? Either way, I
don't mind submitting what I came up with for the TS-7300 board.

Thanks!
--
Florian

FPGA manager user space interface

[Michal Simek]

Hi Florian,

I didn't follow all stuff regarding FPGA manager and I can't remember
the reasons against user interfaces.  Alan probably knows.

Thanks,
Michal

On 10.7.2016 23:28, Florian Fainelli wrote:
> Hi all,
> 
> I just wrote a FPGA manager driver for the TS-7300 board which features
> an Altera Cyclone II on board. While the finished solution is my case
> might be something like the MFD driver for the FPGA devices requesting
> the bitstream load and registering the different devices it exposes,
> during development it is nice to exercise the FPGA manager driver to
> load something:
> 
> - a quick way is to add a pair of sysfs attributes to define the
> bitstream filename and to trigger the load
> 
> - offer a more consistent and robust interface through e.g; a character
> device that you can write/read/poll to see the loading progress about
> 
> - have the driver exposing FPGA peripherals be exposing an user space
> interface to trigger a (re)load/(re)/configuration, although that's
> really something that belongs in the FPGA manager it seems
> 
> I am mostly curious if these were taken into account during the initial
> design and it is agreed upon that yes these are some of options and that
> userspace loading is just anecdotal we do not need any userspace
> interface, or if this is just missing and we want one? Either way, I
> don't mind submitting what I came up with for the TS-7300 board.
> 
> Thanks!
> --
> Florian
> 

FPGA manager user space interface

[atull]

On Sun, 10 Jul 2016, Florian Fainelli wrote:

Hi Florian,

I'm in the process of upstreaming FPGA Regions support which
allows reprogramming by applying Device Tree Overlays.  If you
want to see a tree which has the latest patches for that, I have
all that in our github tree.

https://github.com/altera-opensource/linux-socfpga/tree/socfpga-4.6

The documentation for the DT overlays is in
 Documentation/devicetree/bindings/fpga/fpga-region.txt

Documentation for the DT configfs interface is in
 Documentation/devicetree/configfs-overlays.txt

The idea here is explained at legnth in the doc, but basically
the user can use the configfs interface to apply a DT
overlay that causes the FPGA to be reprogrammed and child
devices to get added and probed.

I like DT overlays a lot, but I do not expect that (or anything
else) will be a good interface for every use case.  The FPGA
Manager Framework went through a legnthy discussion on the
mailing list where several people wanted the interface to match
what they were already using.  Interfaces similar to what you
suggest below were proposed and shot down.  My conclusion from
all that was to implement the kernel API functions such that
people could add whatever interfaces they needed for their use
case.


> Hi all,
> 
> I just wrote a FPGA manager driver for the TS-7300 board which features
> an Altera Cyclone II on board. While the finished solution is my case
> might be something like the MFD driver for the FPGA devices requesting
> the bitstream load and registering the different devices it exposes,
> during development it is nice to exercise the FPGA manager driver to
> load something:

It sounds like your goal is to use the FPGA Manager API inside
the kernel, but it would be helpful to have an easy userspace
interface during development.  Is that right?

> 
> - a quick way is to add a pair of sysfs attributes to define the
> bitstream filename and to trigger the load
> 
> - offer a more consistent and robust interface through e.g; a character
> device that you can write/read/poll to see the loading progress about

Sysfs and char driver interfaces have been proposed and shot down
a few times already.

The first version of the FPGA Manager Framework was a char
driver.  To program the FPGA, the user had to do 'cat
bitstream-file > /dev/fpga0'.  Bridge support was a separate
thing that was also controlled by sysfs.  It was messy and if
userspace could easily do the wrong thing and crash the system.

> 
> - have the driver exposing FPGA peripherals be exposing an user space
> interface to trigger a (re)load/(re)/configuration, although that's
> really something that belongs in the FPGA manager it seems

If you use the device tree configfs interface, you can do that.
It might not be exactly what you are thinking of.  And there is
a learning curve in getting the overlays right.

> 
> I am mostly curious if these were taken into account during the initial
> design and it is agreed upon that yes these are some of options and that
> userspace loading is just anecdotal we do not need any userspace
> interface, or if this is just missing and we want one?

There will be use cases that will need various userspace interfaces.
It's been hard to get people who are already using FPGAs to agree
on using any one interface because they would have to change the
code they already have running.

> Either way, I
> don't mind submitting what I came up with for the TS-7300 board.

Yes, that would be great.

> 
> Thanks!
> --
> Florian
> 

FPGA manager user space interface

[Florian Fainelli]

Hi Alan,

On 07/11/2016 09:59 AM, atull wrote:
> On Sun, 10 Jul 2016, Florian Fainelli wrote:
> 
> Hi Florian,
> 
> I'm in the process of upstreaming FPGA Regions support which
> allows reprogramming by applying Device Tree Overlays.  If you
> want to see a tree which has the latest patches for that, I have
> all that in our github tree.
> 
> https://github.com/altera-opensource/linux-socfpga/tree/socfpga-4.6
> 
> The documentation for the DT overlays is in
>  Documentation/devicetree/bindings/fpga/fpga-region.txt
> 
> Documentation for the DT configfs interface is in
>  Documentation/devicetree/configfs-overlays.txt
> 
> The idea here is explained at legnth in the doc, but basically
> the user can use the configfs interface to apply a DT
> overlay that causes the FPGA to be reprogrammed and child
> devices to get added and probed.
> 
> I like DT overlays a lot, but I do not expect that (or anything
> else) will be a good interface for every use case.  The FPGA
> Manager Framework went through a legnthy discussion on the
> mailing list where several people wanted the interface to match
> what they were already using.  Interfaces similar to what you
> suggest below were proposed and shot down.  My conclusion from
> all that was to implement the kernel API functions such that
> people could add whatever interfaces they needed for their use
> case.

While I agree that the current state of the FPGA manager is completely
sane, and people making products including a FPGA will want something
(kernel driver or user-space scripts) that load a given set of bistreams
(under version control), it still feels like there is a small bit of
debugging and or use cases where it is desireable to have a way, from
user-space to load an arbitrary FPGA bitstream which is not set in stone
from e.g: a Linux kernel driver. More than being able to change
something on the filesystem, it's about being able to reload the
bitstream from user-space that seems to be critically missing imho.

Do you have links to the original discussion so I could get an idea of
what were the different points back then?

Thanks!

> 
> 
>> Hi all,
>>
>> I just wrote a FPGA manager driver for the TS-7300 board which features
>> an Altera Cyclone II on board. While the finished solution is my case
>> might be something like the MFD driver for the FPGA devices requesting
>> the bitstream load and registering the different devices it exposes,
>> during development it is nice to exercise the FPGA manager driver to
>> load something:
> 
> It sounds like your goal is to use the FPGA Manager API inside
> the kernel, but it would be helpful to have an easy userspace
> interface during development.  Is that right?
> 
>>
>> - a quick way is to add a pair of sysfs attributes to define the
>> bitstream filename and to trigger the load
>>
>> - offer a more consistent and robust interface through e.g; a character
>> device that you can write/read/poll to see the loading progress about
> 
> Sysfs and char driver interfaces have been proposed and shot down
> a few times already.
> 
> The first version of the FPGA Manager Framework was a char
> driver.  To program the FPGA, the user had to do 'cat
> bitstream-file > /dev/fpga0'.  Bridge support was a separate
> thing that was also controlled by sysfs.  It was messy and if
> userspace could easily do the wrong thing and crash the system.
> 
>>
>> - have the driver exposing FPGA peripherals be exposing an user space
>> interface to trigger a (re)load/(re)/configuration, although that's
>> really something that belongs in the FPGA manager it seems
> 
> If you use the device tree configfs interface, you can do that.
> It might not be exactly what you are thinking of.  And there is
> a learning curve in getting the overlays right.
> 
>>
>> I am mostly curious if these were taken into account during the initial
>> design and it is agreed upon that yes these are some of options and that
>> userspace loading is just anecdotal we do not need any userspace
>> interface, or if this is just missing and we want one?
> 
> There will be use cases that will need various userspace interfaces.
> It's been hard to get people who are already using FPGAs to agree
> on using any one interface because they would have to change the
> code they already have running.
> 
>> Either way, I
>> don't mind submitting what I came up with for the TS-7300 board.
> 
> Yes, that would be great.
> 
>>
>> Thanks!
>> --
>> Florian
>>


-- 
Florian

FPGA manager user space interface

[atull]

On Mon, 11 Jul 2016, Florian Fainelli wrote:

> Hi Alan,
> 
> On 07/11/2016 09:59 AM, atull wrote:
> > On Sun, 10 Jul 2016, Florian Fainelli wrote:
> > 
> > Hi Florian,
> > 
> > I'm in the process of upstreaming FPGA Regions support which
> > allows reprogramming by applying Device Tree Overlays.  If you
> > want to see a tree which has the latest patches for that, I have
> > all that in our github tree.
> > 
> > https://github.com/altera-opensource/linux-socfpga/tree/socfpga-4.6
> > 
> > The documentation for the DT overlays is in
> >  Documentation/devicetree/bindings/fpga/fpga-region.txt
> > 
> > Documentation for the DT configfs interface is in
> >  Documentation/devicetree/configfs-overlays.txt
> > 
> > The idea here is explained at legnth in the doc, but basically
> > the user can use the configfs interface to apply a DT
> > overlay that causes the FPGA to be reprogrammed and child
> > devices to get added and probed.
> > 
> > I like DT overlays a lot, but I do not expect that (or anything
> > else) will be a good interface for every use case.  The FPGA
> > Manager Framework went through a legnthy discussion on the
> > mailing list where several people wanted the interface to match
> > what they were already using.  Interfaces similar to what you
> > suggest below were proposed and shot down.  My conclusion from
> > all that was to implement the kernel API functions such that
> > people could add whatever interfaces they needed for their use
> > case.
> 
> While I agree that the current state of the FPGA manager is completely
> sane, and people making products including a FPGA will want something
> (kernel driver or user-space scripts) that load a given set of bistreams
> (under version control), it still feels like there is a small bit of
> debugging and or use cases where it is desireable to have a way, from
> user-space to load an arbitrary FPGA bitstream which is not set in stone
> from e.g: a Linux kernel driver. More than being able to change
> something on the filesystem, it's about being able to reload the
> bitstream from user-space that seems to be critically missing imho.
> 

Did you look into DT overlays?  DT overlays can be applied from
userspace using the configfs interface.  The FPGA Region code
will handle bridges and loading the nodes for the child devices.
You can also remove the overlay, the child devices will be
unloaded and the bridges disabled so you can iterate during your
development.

> Do you have links to the original discussion so I could get an idea of
> what were the different points back then?
> 
> Thanks!

Exposing reprogramming in sysfs or as a char driver was rejected
as:
* it makes it easy to do the wrong thing and crash your system
* it didn't give us a way of showing the heirarchy of what a
system with multiple fpgas and bridges could have.
* It makes it difficult to know which fpga you are reprogramming
if you are doing 'cat image > /dev/fpga0' since your system may
have several fpgas.
* Your system may have bridges between the cpu and the fpga.
Those will need to be mapped not only to the fpga they are
connected to, but more finely than that.  Some fpga images
need different bridges to be enabled/disabled.

When dynamic device trees started to become a reality in
the kernel, that was a natural solution to a lot of this.

Feel free to google the lkml archives you want more details.  The
reason I am opposed to it now is it that is begging for the
system to be crashed by doing a simple operation from userspace.
And it makes it impossible to export to userspace the kind
of control that the user needs except perhaps in only
the most simple system.

Here's a few easy ways to crash your system if you can reprogram
from userspace:
1. Reprogram the FPGA without shutting down the bridges first.
Spurious data goes out onto a processor bus and things get hosed.
2. Program the FGPA, load some drivers to support the hw in
the FPGA, then reprogram the FPGA.
3. Program the FPGA using DT overlays and then use sysfs
as a back door to program it again.
4. A system with multiple FPGAs will need to be brought up
or reconfigured in a sensible order.  DT overlays can be
applied on top of dt overlays, bringing up the system in
order.

Alan

> 
> > 
> > 
> >> Hi all,
> >>
> >> I just wrote a FPGA manager driver for the TS-7300 board which features
> >> an Altera Cyclone II on board. While the finished solution is my case
> >> might be something like the MFD driver for the FPGA devices requesting
> >> the bitstream load and registering the different devices it exposes,
> >> during development it is nice to exercise the FPGA manager driver to
> >> load something:
> > 
> > It sounds like your goal is to use the FPGA Manager API inside
> > the kernel, but it would be helpful to have an easy userspace
> > interface during development.  Is that right?
> > 
> >>
> >> - a quick way is to add a pair of sysfs attributes to define the
> >> bitstream filename and to trigger the load
> >>
> >> - offer a more consistent and robust interface through e.g; a character
> >> device that you can write/read/poll to see the loading progress about
> > 
> > Sysfs and char driver interfaces have been proposed and shot down
> > a few times already.
> > 
> > The first version of the FPGA Manager Framework was a char
> > driver.  To program the FPGA, the user had to do 'cat
> > bitstream-file > /dev/fpga0'.  Bridge support was a separate
> > thing that was also controlled by sysfs.  It was messy and if
> > userspace could easily do the wrong thing and crash the system.
> > 
> >>
> >> - have the driver exposing FPGA peripherals be exposing an user space
> >> interface to trigger a (re)load/(re)/configuration, although that's
> >> really something that belongs in the FPGA manager it seems
> > 
> > If you use the device tree configfs interface, you can do that.
> > It might not be exactly what you are thinking of.  And there is
> > a learning curve in getting the overlays right.
> > 
> >>
> >> I am mostly curious if these were taken into account during the initial
> >> design and it is agreed upon that yes these are some of options and that
> >> userspace loading is just anecdotal we do not need any userspace
> >> interface, or if this is just missing and we want one?
> > 
> > There will be use cases that will need various userspace interfaces.
> > It's been hard to get people who are already using FPGAs to agree
> > on using any one interface because they would have to change the
> > code they already have running.
> > 
> >> Either way, I
> >> don't mind submitting what I came up with for the TS-7300 board.
> > 
> > Yes, that would be great.
> > 
> >>
> >> Thanks!
> >> --
> >> Florian
> >>
> 
> 
> -- 
> Florian
> 

FPGA manager user space interface

[Florian Fainelli]

On 07/12/2016 08:15 AM, atull wrote:
> On Mon, 11 Jul 2016, Florian Fainelli wrote:
> 
>> Hi Alan,
>>
>> On 07/11/2016 09:59 AM, atull wrote:
>>> On Sun, 10 Jul 2016, Florian Fainelli wrote:
>>>
>>> Hi Florian,
>>>
>>> I'm in the process of upstreaming FPGA Regions support which
>>> allows reprogramming by applying Device Tree Overlays.  If you
>>> want to see a tree which has the latest patches for that, I have
>>> all that in our github tree.
>>>
>>> https://github.com/altera-opensource/linux-socfpga/tree/socfpga-4.6
>>>
>>> The documentation for the DT overlays is in
>>>  Documentation/devicetree/bindings/fpga/fpga-region.txt
>>>
>>> Documentation for the DT configfs interface is in
>>>  Documentation/devicetree/configfs-overlays.txt
>>>
>>> The idea here is explained at legnth in the doc, but basically
>>> the user can use the configfs interface to apply a DT
>>> overlay that causes the FPGA to be reprogrammed and child
>>> devices to get added and probed.
>>>
>>> I like DT overlays a lot, but I do not expect that (or anything
>>> else) will be a good interface for every use case.  The FPGA
>>> Manager Framework went through a legnthy discussion on the
>>> mailing list where several people wanted the interface to match
>>> what they were already using.  Interfaces similar to what you
>>> suggest below were proposed and shot down.  My conclusion from
>>> all that was to implement the kernel API functions such that
>>> people could add whatever interfaces they needed for their use
>>> case.
>>
>> While I agree that the current state of the FPGA manager is completely
>> sane, and people making products including a FPGA will want something
>> (kernel driver or user-space scripts) that load a given set of bistreams
>> (under version control), it still feels like there is a small bit of
>> debugging and or use cases where it is desireable to have a way, from
>> user-space to load an arbitrary FPGA bitstream which is not set in stone
>> from e.g: a Linux kernel driver. More than being able to change
>> something on the filesystem, it's about being able to reload the
>> bitstream from user-space that seems to be critically missing imho.
>>
> 
> Did you look into DT overlays?  DT overlays can be applied from
> userspace using the configfs interface.  The FPGA Region code
> will handle bridges and loading the nodes for the child devices.
> You can also remove the overlay, the child devices will be
> unloaded and the bridges disabled so you can iterate during your
> development.

I did, and this is solving one aspect of working with FPGAs which is how
to represent the different devices they offer, but this is not quite
what my question nor concerns are about, those were more about the
debugability and working with

> 
>> Do you have links to the original discussion so I could get an idea of
>> what were the different points back then?
>>
>> Thanks!
> 
> Exposing reprogramming in sysfs or as a char driver was rejected
> as:
> * it makes it easy to do the wrong thing and crash your system
> * it didn't give us a way of showing the heirarchy of what a
> system with multiple fpgas and bridges could have.

That is true, and it is also pretty much true with any Linux device
driver unless you have special code that parses the hierarchy in /sys
(which should happen regardless if you need to care bout hierarchy).

> * It makes it difficult to know which fpga you are reprogramming
> if you are doing 'cat image > /dev/fpga0' since your system may
> have several fpgas.

My out of tree code creates on /dev/fpgaX where X matches what the FPGA
manger ida returns, so there is a strict 1:1 mapping between the two,
and, the character device is bound to the backing device of the FPGA
manager, so that does not seem like an actual concern.

> * Your system may have bridges between the cpu and the fpga.
> Those will need to be mapped not only to the fpga they are
> connected to, but more finely than that.  Some fpga images
> need different bridges to be enabled/disabled.
> 
> When dynamic device trees started to become a reality in
> the kernel, that was a natural solution to a lot of this.

Keep in mind that DT overlays are a tool to solve one very specific
problem: representing accurarely what a FPGA device provides and making
sure that Linux's device driver model adheres to that, but it does not
solve the other part of my question which is, given an existing
framework, how do I load arbitraty bistreams because my workflow
involves working on the bistream more than the FPGA manager driver which
is just the host side of the development tools.

> 
> Feel free to google the lkml archives you want more details.  The
> reason I am opposed to it now is it that is begging for the
> system to be crashed by doing a simple operation from userspace.
> And it makes it impossible to export to userspace the kind
> of control that the user needs except perhaps in only
> the most simple system.

That part is absolutely fair in a multi-FPGA system with complex
bridging and or daisy chaining, although I don't really see anything
that could not be solvable by having proper parenting between devices
and therefore knowing the topology, and that should already exist thanks
to DT overlays and/or careful parenting (e.g: through MFD).

> 
> Here's a few easy ways to crash your system if you can reprogram
> from userspace:
> 1. Reprogram the FPGA without shutting down the bridges first.
> Spurious data goes out onto a processor bus and things get hosed.

This seems to be a similar, if not yet identical problem to doing system
suspend/resume, and the device driver model should help you figure out
the parenting of peripherals and allow shutting them down before you can
reconfigure the FPGA with a new bistream and set of peripherals.

> 2. Program the FGPA, load some drivers to support the hw in
> the FPGA, then reprogram the FPGA.
> 3. Program the FPGA using DT overlays and then use sysfs
> as a back door to program it again.
> 4. A system with multiple FPGAs will need to be brought up
> or reconfigured in a sensible order.  DT overlays can be
> applied on top of dt overlays, bringing up the system in
> order.

Same thing here, the device driver model helps you, and to get an
accurate device driver model, DT is one option, but not the only one.

Thanks for sharing your thoughts, I am not going to argue we need a
character device for debugging/programming, and I am okay with keeping
this code out of tree for my own development purposes.

> 
> Alan
> 
>>
>>>
>>>
>>>> Hi all,
>>>>
>>>> I just wrote a FPGA manager driver for the TS-7300 board which features
>>>> an Altera Cyclone II on board. While the finished solution is my case
>>>> might be something like the MFD driver for the FPGA devices requesting
>>>> the bitstream load and registering the different devices it exposes,
>>>> during development it is nice to exercise the FPGA manager driver to
>>>> load something:
>>>
>>> It sounds like your goal is to use the FPGA Manager API inside
>>> the kernel, but it would be helpful to have an easy userspace
>>> interface during development.  Is that right?
>>>
>>>>
>>>> - a quick way is to add a pair of sysfs attributes to define the
>>>> bitstream filename and to trigger the load
>>>>
>>>> - offer a more consistent and robust interface through e.g; a character
>>>> device that you can write/read/poll to see the loading progress about
>>>
>>> Sysfs and char driver interfaces have been proposed and shot down
>>> a few times already.
>>>
>>> The first version of the FPGA Manager Framework was a char
>>> driver.  To program the FPGA, the user had to do 'cat
>>> bitstream-file > /dev/fpga0'.  Bridge support was a separate
>>> thing that was also controlled by sysfs.  It was messy and if
>>> userspace could easily do the wrong thing and crash the system.
>>>
>>>>
>>>> - have the driver exposing FPGA peripherals be exposing an user space
>>>> interface to trigger a (re)load/(re)/configuration, although that's
>>>> really something that belongs in the FPGA manager it seems
>>>
>>> If you use the device tree configfs interface, you can do that.
>>> It might not be exactly what you are thinking of.  And there is
>>> a learning curve in getting the overlays right.
>>>
>>>>
>>>> I am mostly curious if these were taken into account during the initial
>>>> design and it is agreed upon that yes these are some of options and that
>>>> userspace loading is just anecdotal we do not need any userspace
>>>> interface, or if this is just missing and we want one?
>>>
>>> There will be use cases that will need various userspace interfaces.
>>> It's been hard to get people who are already using FPGAs to agree
>>> on using any one interface because they would have to change the
>>> code they already have running.
>>>
>>>> Either way, I
>>>> don't mind submitting what I came up with for the TS-7300 board.
>>>
>>> Yes, that would be great.
>>>
>>>>
>>>> Thanks!
>>>> --
>>>> Florian
>>>>
>>
>>
>> -- 
>> Florian
>>


-- 
Florian