Device Description for FPGA Components on x86 system

Device Description for FPGA Components on x86 system

[Federico Vaga]

Hello,

I'm looking for guidance

What I have:
* Intel x86_64 computer
* PCIe card with FPGA on it

What I want to achieve:
* load an FPGA bitstream on the card
* load a device-tree like description for the FPGA devices contained in the 
bitstream

This is achievable on ARM with DeviceTree, overlay-dt, fpga-mgr; but I'm 
puzzled about the x86_64 use-case. I'm not able to find recent and clear 
information.

Does anyone know if this is doable? Perhaps with ACPI SSDTs overlay? Or with 
the DT?

thanks 

Re: Device Description for FPGA Components on x86 system

[Federico Vaga]

Hello,

sorry to push for an answer but I do not want to take the risk of designing 
something useless. I do not know how should I interpret a no-answer.

If the solution really does not exist today, then I would like to collect 
opinions/arguments/requirements on the topic so that I can write something 
useful not only for CERN but for the entire community.

Thank you

On Wednesday, March 27, 2019 6:17:18 PM CEST Federico Vaga wrote:
> Hello,
> 
> I'm looking for guidance
> 
> What I have:
> * Intel x86_64 computer
> * PCIe card with FPGA on it
> 
> What I want to achieve:
> * load an FPGA bitstream on the card
> * load a device-tree like description for the FPGA devices contained in the
> bitstream
> 
> This is achievable on ARM with DeviceTree, overlay-dt, fpga-mgr; but I'm
> puzzled about the x86_64 use-case. I'm not able to find recent and clear
> information.
> 
> Does anyone know if this is doable? Perhaps with ACPI SSDTs overlay? Or with
> the DT?
> 
> thanks

Re: Device Description for FPGA Components on x86 system

[Eric Schwarz]

Hi,

everything you want is already available and on the way to mainline 
concerning support for various FPGA loading modes or available for 
checkout from a git repository.
All that has already been discussed on the mailing list.

FPGA loading interface is available here [1].
Patchset missing for FPGA loading has been sent to the mailing list from 
Anatolij Gustschin for various Linux kernel versions. Link to the most 
recent patchset version [2].
FPGA Manager mailing list archive link [3] - Please read up the story 
here around those patches and also the replies of the others.

Cheers
Eric

[1] https://github.com/vdsao/fpga-cfg
[2] https://marc.info/?l=linux-fpga&m=155078072107199&w=2
[3] https://marc.info/?l=linux-fpga

Am 10.04.2019 12:01, schrieb Federico Vaga:

> Hello,
> 
> sorry to push for an answer but I do not want to take the risk of 
> designing
> something useless. I do not know how should I interpret a no-answer.
> 
> If the solution really does not exist today, then I would like to 
> collect
> opinions/arguments/requirements on the topic so that I can write 
> something
> useful not only for CERN but for the entire community.
> 
> Thank you
> 
> On Wednesday, March 27, 2019 6:17:18 PM CEST Federico Vaga wrote:
> 
>> Hello,
>> 
>> I'm looking for guidance
>> 
>> What I have:
>> * Intel x86_64 computer
>> * PCIe card with FPGA on it
>> 
>> What I want to achieve:
>> * load an FPGA bitstream on the card
>> * load a device-tree like description for the FPGA devices contained 
>> in the
>> bitstream
>> 
>> This is achievable on ARM with DeviceTree, overlay-dt, fpga-mgr; but 
>> I'm
>> puzzled about the x86_64 use-case. I'm not able to find recent and 
>> clear
>> information.
>> 
>> Does anyone know if this is doable? Perhaps with ACPI SSDTs overlay? 
>> Or with
>> the DT?
>> 
>> thanks

Re: Device Description for FPGA Components on x86 system

[Federico Vaga]

Hi,

P.S. sorry if I'm too verbose, hopefully it is useful

thanks for the answer

On Wednesday, April 10, 2019 12:30:14 PM CEST Eric Schwarz wrote:
> Hi,
> 
> everything you want is already available and on the way to mainline
> concerning support for various FPGA loading modes or available for
> checkout from a git repository.
> All that has already been discussed on the mailing list.
> 
> FPGA loading interface is available here [1].
> Patchset missing for FPGA loading has been sent to the mailing list from
> Anatolij Gustschin for various Linux kernel versions. Link to the most
> recent patchset version [2].
> FPGA Manager mailing list archive link [3] - Please read up the story
> here around those patches and also the replies of the others.

This does not answer the problem, which perhaps need to be clarified.

Loading FPGA is **not** the problem, I listed it in the things I want to 
achieve because it is a pre-requirement for the real problem and because the 
two processes are linked (or could be).

I continue by commenting myself below, trying to make the use case clearer.

> 
> Cheers
> Eric
> 
> [1] https://github.com/vdsao/fpga-cfg
> [2] https://marc.info/?l=linux-fpga&m=155078072107199&w=2
> [3] https://marc.info/?l=linux-fpga
> 
> Am 10.04.2019 12:01, schrieb Federico Vaga:
> > Hello,
> > 
> > sorry to push for an answer but I do not want to take the risk of
> > designing
> > something useless. I do not know how should I interpret a no-answer.
> > 
> > If the solution really does not exist today, then I would like to
> > collect
> > opinions/arguments/requirements on the topic so that I can write
> > something
> > useful not only for CERN but for the entire community.
> > 
> > Thank you
> > 
> > On Wednesday, March 27, 2019 6:17:18 PM CEST Federico Vaga wrote:
> >> Hello,
> >> 
> >> I'm looking for guidance
> >> 
> >> What I have:
> >> * Intel x86_64 computer
> >> * PCIe card with FPGA on it
> >> 
> >> What I want to achieve:
> >> * load an FPGA bitstream on the card
> >> * load a device-tree like description for the FPGA devices contained
> >> in the bitstream

Let me first elaborate on my knowledge to avoid misunderstandings.

On ARM, nowadays, we boot with a device tree. Later we program an FPGA in 
which there are other devices described by a device tree overlay. This can be 
done easily.

A typical PC (x86/x86_64) does not boot with DeviceTree (it is possible, but 
it is not common and probably not even suggested, not sure), instead it uses 
ACPI.

The FPGA Manager has support only for DeviceTree (there are patching floating 
around to load a bitstream with configfs, debugfs or a chardevice (guilty))

Most drivers foresee a DeviceTree loading but not an ACPI one (my feeling, I 
did not extract exact numbers from the sources)

DeviceTree overlay requires that the system boots with DeviceTree.

DeviceTree and ACPI do not work together

So, this is the state of art that I am aware of. Correct me if, and where, I 
am wrong.


Restarting from this point. I have a PC (x86_64) with a PCIe FPGA card (e.g. 
sis8160, spec, links below). How to load the FPGA bitstream (not really a 
problem as you correctly pointed out) **and** load all the IP-core instances 
in that FPGA bitstream so that drivers will start running?

- Is there a recommendation for such use case?
- ACPI SSDT overlay? 
- DT overlay?
- is there a standard way to load FPGA IP-core devices which is architecture 
independent?

A simple and practical example. The i2c-ocore.c is a platform_driver for an 
HDL I2C Master from open cores. I synthesize it and then load it on the FPGA. 
How to create the Linux platform_device instance to driver that IP-core? How 
to do that when you have also IRQ controller(s), DMA engine(s), EEPROM(s) and 
other devices?

The fastest solution is to do what was common on ARM systems: having all 
platform devices declared (hard coded) in a file and load them. Which is not a 
good solution, for the same reasons why arm stuff moved to devicetree.

Is it clearer?

I do not know if it important to highlight but those cards are extensible, 
potentially any FMC module could be plugged and this needs a different FPGA, 
with different FPGA devices etc. So, It is not possible to hardcode the 
description of all possible FPGA code (infinite) that can enable the usage of 
all possible FMC module (not infinite, but definitively grater than 1)


https://www.struck.de/sis8160.html
https://ohwr.org/project/spec/wikis/home


> >> 
> >> This is achievable on ARM with DeviceTree, overlay-dt, fpga-mgr; but
> >> I'm
> >> puzzled about the x86_64 use-case. I'm not able to find recent and
> >> clear
> >> information.
> >> 
> >> Does anyone know if this is doable? Perhaps with ACPI SSDTs overlay?
> >> Or with
> >> the DT?
> >> 
> >> thanks

Re: Device Description for FPGA Components on x86 system

[Alan Tull]

On Wed, Apr 10, 2019 at 7:50 AM Federico Vaga <federico.vaga@cern.ch> wrote:

Hi Federico,

I wish I could point you to a complete solution, but there is a lot of
work to be done in this area.  Most of what is in the kernel is a low
level in-kernel API [4].  As you correctly state, the hardest part of
this is doing the enumerating if you are on x86 and aren't using
devicetree.

>
> Hi,
>
> P.S. sorry if I'm too verbose, hopefully it is useful
>
> thanks for the answer
>
> On Wednesday, April 10, 2019 12:30:14 PM CEST Eric Schwarz wrote:
> > Hi,
> >
> > everything you want is already available and on the way to mainline
> > concerning support for various FPGA loading modes or available for
> > checkout from a git repository.
> > All that has already been discussed on the mailing list.
> >
> > FPGA loading interface is available here [1].
> > Patchset missing for FPGA loading has been sent to the mailing list from
> > Anatolij Gustschin for various Linux kernel versions. Link to the most
> > recent patchset version [2].
> > FPGA Manager mailing list archive link [3] - Please read up the story
> > here around those patches and also the replies of the others.
>
> This does not answer the problem, which perhaps need to be clarified.
>
> Loading FPGA is **not** the problem, I listed it in the things I want to
> achieve because it is a pre-requirement for the real problem and because the
> two processes are linked (or could be).
>
> I continue by commenting myself below, trying to make the use case clearer.
>
> >
> > Cheers
> > Eric
> >
> > [1] https://github.com/vdsao/fpga-cfg
> > [2] https://marc.info/?l=linux-fpga&m=155078072107199&w=2
> > [3] https://marc.info/?l=linux-fpga
> >
> > Am 10.04.2019 12:01, schrieb Federico Vaga:
> > > Hello,
> > >
> > > sorry to push for an answer but I do not want to take the risk of
> > > designing
> > > something useless. I do not know how should I interpret a no-answer.
> > >
> > > If the solution really does not exist today, then I would like to
> > > collect
> > > opinions/arguments/requirements on the topic so that I can write
> > > something
> > > useful not only for CERN but for the entire community.
> > >
> > > Thank you
> > >
> > > On Wednesday, March 27, 2019 6:17:18 PM CEST Federico Vaga wrote:
> > >> Hello,
> > >>
> > >> I'm looking for guidance
> > >>
> > >> What I have:
> > >> * Intel x86_64 computer
> > >> * PCIe card with FPGA on it
> > >>
> > >> What I want to achieve:
> > >> * load an FPGA bitstream on the card
> > >> * load a device-tree like description for the FPGA devices contained
> > >> in the bitstream
>
> Let me first elaborate on my knowledge to avoid misunderstandings.
>
> On ARM, nowadays, we boot with a device tree. Later we program an FPGA in
> which there are other devices described by a device tree overlay. This can be
> done easily.
>
> A typical PC (x86/x86_64) does not boot with DeviceTree (it is possible, but
> it is not common and probably not even suggested, not sure), instead it uses
> ACPI.

I have heard it suggested that we work on using DT overlays on x86*.
It's clear there's work to be done to make that work.  I don't know if
anybody has really tried.  It seems impractical to map or translate a
x86 systems ACPI into a DT and go from there.  One suggestion a few
years ago was adding a partial DT that had nodes that could serve as
overlay targets and have that running in parallel with ACPI.

>
> The FPGA Manager has support only for DeviceTree (there are patching floating
> around to load a bitstream with configfs, debugfs or a chardevice (guilty))

There's one other interface in the kernel upstream. The DFL (device
feature list) framework built on the FPGA manager/bridge/region stuff
[5].   It's probably not what you specifically are looking for, I'm
mentioning as it exists in upstream.  It has a limited type of
enumeration and appears to mostly be geared for acceleration rather
than adding and enumerating any random hardware block.  Also it
requires specific bitstreams as the feature list is in fpga fabric.

>
> Most drivers foresee a DeviceTree loading but not an ACPI one (my feeling, I
> did not extract exact numbers from the sources)
>
> DeviceTree overlay requires that the system boots with DeviceTree.
>
> DeviceTree and ACPI do not work together
>
> So, this is the state of art that I am aware of. Correct me if, and where, I
> am wrong.
>
>
> Restarting from this point. I have a PC (x86_64) with a PCIe FPGA card (e.g.
> sis8160, spec, links below). How to load the FPGA bitstream (not really a
> problem as you correctly pointed out) **and** load all the IP-core instances
> in that FPGA bitstream so that drivers will start running?
>
> - Is there a recommendation for such use case?
> - ACPI SSDT overlay?
> - DT overlay?
> - is there a standard way to load FPGA IP-core devices which is architecture
> independent?
>
> A simple and practical example. The i2c-ocore.c is a platform_driver for an
> HDL I2C Master from open cores. I synthesize it and then load it on the FPGA.
> How to create the Linux platform_device instance to driver that IP-core? How
> to do that when you have also IRQ controller(s), DMA engine(s), EEPROM(s) and
> other devices?
>
> The fastest solution is to do what was common on ARM systems: having all
> platform devices declared (hard coded) in a file and load them. Which is not a
> good solution, for the same reasons why arm stuff moved to devicetree.
>
> Is it clearer?
>
> I do not know if it important to highlight but those cards are extensible,
> potentially any FMC module could be plugged and this needs a different FPGA,
> with different FPGA devices etc. So, It is not possible to hardcode the
> description of all possible FPGA code (infinite) that can enable the usage of
> all possible FMC module (not infinite, but definitively grater than 1)
>
>
> https://www.struck.de/sis8160.html
> https://ohwr.org/project/spec/wikis/home
>
>
> > >>
> > >> This is achievable on ARM with DeviceTree, overlay-dt, fpga-mgr; but
> > >> I'm
> > >> puzzled about the x86_64 use-case. I'm not able to find recent and
> > >> clear
> > >> information.
> > >>
> > >> Does anyone know if this is doable? Perhaps with ACPI SSDTs overlay?
> > >> Or with
> > >> the DT?
> > >>
> > >> thanks

Thanks,
Alan

[4] https://www.kernel.org/doc/html/latest/driver-api/fpga/index.html
[5] https://github.com/torvalds/linux/blob/master/Documentation/fpga/dfl.txt

Re: Device Description for FPGA Components on x86 system

[Federico Vaga]

Hi Alan,

thanks for your answer

On Wednesday, April 10, 2019 4:21:09 PM CEST Alan Tull wrote:
> On Wed, Apr 10, 2019 at 7:50 AM Federico Vaga <federico.vaga@cern.ch> wrote:
> 
> Hi Federico,
> 
> I wish I could point you to a complete solution, but there is a lot of
> work to be done in this area.  Most of what is in the kernel is a low
> level in-kernel API [4].  As you correctly state, the hardest part of
> this is doing the enumerating if you are on x86 and aren't using
> devicetree.
> 
> > Hi,
> > 
> > P.S. sorry if I'm too verbose, hopefully it is useful
> > 
> > thanks for the answer
> > 
> > On Wednesday, April 10, 2019 12:30:14 PM CEST Eric Schwarz wrote:
> > > Hi,
> > > 
> > > everything you want is already available and on the way to mainline
> > > concerning support for various FPGA loading modes or available for
> > > checkout from a git repository.
> > > All that has already been discussed on the mailing list.
> > > 
> > > FPGA loading interface is available here [1].
> > > Patchset missing for FPGA loading has been sent to the mailing list from
> > > Anatolij Gustschin for various Linux kernel versions. Link to the most
> > > recent patchset version [2].
> > > FPGA Manager mailing list archive link [3] - Please read up the story
> > > here around those patches and also the replies of the others.
> > 
> > This does not answer the problem, which perhaps need to be clarified.
> > 
> > Loading FPGA is **not** the problem, I listed it in the things I want to
> > achieve because it is a pre-requirement for the real problem and because
> > the two processes are linked (or could be).
> > 
> > I continue by commenting myself below, trying to make the use case
> > clearer.
> > 
> > > Cheers
> > > Eric
> > > 
> > > [1] https://github.com/vdsao/fpga-cfg
> > > [2] https://marc.info/?l=linux-fpga&m=155078072107199&w=2
> > > [3] https://marc.info/?l=linux-fpga
> > > 
> > > Am 10.04.2019 12:01, schrieb Federico Vaga:
> > > > Hello,
> > > > 
> > > > sorry to push for an answer but I do not want to take the risk of
> > > > designing
> > > > something useless. I do not know how should I interpret a no-answer.
> > > > 
> > > > If the solution really does not exist today, then I would like to
> > > > collect
> > > > opinions/arguments/requirements on the topic so that I can write
> > > > something
> > > > useful not only for CERN but for the entire community.
> > > > 
> > > > Thank you
> > > > 
> > > > On Wednesday, March 27, 2019 6:17:18 PM CEST Federico Vaga wrote:
> > > >> Hello,
> > > >> 
> > > >> I'm looking for guidance
> > > >> 
> > > >> What I have:
> > > >> * Intel x86_64 computer
> > > >> * PCIe card with FPGA on it
> > > >> 
> > > >> What I want to achieve:
> > > >> * load an FPGA bitstream on the card
> > > >> * load a device-tree like description for the FPGA devices contained
> > > >> in the bitstream
> > 
> > Let me first elaborate on my knowledge to avoid misunderstandings.
> > 
> > On ARM, nowadays, we boot with a device tree. Later we program an FPGA in
> > which there are other devices described by a device tree overlay. This can
> > be done easily.
> > 
> > A typical PC (x86/x86_64) does not boot with DeviceTree (it is possible,
> > but it is not common and probably not even suggested, not sure), instead
> > it uses ACPI.
> 
> I have heard it suggested that we work on using DT overlays on x86*.
> It's clear there's work to be done to make that work.  I don't know if
> anybody has really tried.  It seems impractical to map or translate a
> x86 systems ACPI into a DT and go from there.  


> One suggestion a few
> years ago was adding a partial DT that had nodes that could serve as
> overlay targets and have that running in parallel with ACPI.

This is also one of my ideas for a solution to our problems. Are you able to 
easily retrieve that conversation? Perhaps this is something on which I can 
work on.

> > The FPGA Manager has support only for DeviceTree (there are patching
> > floating around to load a bitstream with configfs, debugfs or a
> > chardevice (guilty))
> There's one other interface in the kernel upstream. The DFL (device
> feature list) framework built on the FPGA manager/bridge/region stuff
> [5].   It's probably not what you specifically are looking for, I'm
> mentioning as it exists in upstream.  It has a limited type of
> enumeration and appears to mostly be geared for acceleration rather
> than adding and enumerating any random hardware block.  Also it
> requires specific bitstreams as the feature list is in fpga fabric.
> 
> > Most drivers foresee a DeviceTree loading but not an ACPI one (my feeling,
> > I did not extract exact numbers from the sources)
> > 
> > DeviceTree overlay requires that the system boots with DeviceTree.
> > 
> > DeviceTree and ACPI do not work together
> > 
> > So, this is the state of art that I am aware of. Correct me if, and where,
> > I am wrong.
> > 
> > 
> > Restarting from this point. I have a PC (x86_64) with a PCIe FPGA card
> > (e.g. sis8160, spec, links below). How to load the FPGA bitstream (not
> > really a problem as you correctly pointed out) **and** load all the
> > IP-core instances in that FPGA bitstream so that drivers will start
> > running?
> > 
> > - Is there a recommendation for such use case?
> > - ACPI SSDT overlay?
> > - DT overlay?
> > - is there a standard way to load FPGA IP-core devices which is
> > architecture independent?
> > 
> > A simple and practical example. The i2c-ocore.c is a platform_driver for
> > an
> > HDL I2C Master from open cores. I synthesize it and then load it on the
> > FPGA. How to create the Linux platform_device instance to driver that
> > IP-core? How to do that when you have also IRQ controller(s), DMA
> > engine(s), EEPROM(s) and other devices?
> > 
> > The fastest solution is to do what was common on ARM systems: having all
> > platform devices declared (hard coded) in a file and load them. Which is
> > not a good solution, for the same reasons why arm stuff moved to
> > devicetree.
> > 
> > Is it clearer?
> > 
> > I do not know if it important to highlight but those cards are extensible,
> > potentially any FMC module could be plugged and this needs a different
> > FPGA, with different FPGA devices etc. So, It is not possible to hardcode
> > the description of all possible FPGA code (infinite) that can enable the
> > usage of all possible FMC module (not infinite, but definitively grater
> > than 1)
> > 
> > 
> > https://www.struck.de/sis8160.html
> > https://ohwr.org/project/spec/wikis/home
> > 
> > > >> This is achievable on ARM with DeviceTree, overlay-dt, fpga-mgr; but
> > > >> I'm
> > > >> puzzled about the x86_64 use-case. I'm not able to find recent and
> > > >> clear
> > > >> information.
> > > >> 
> > > >> Does anyone know if this is doable? Perhaps with ACPI SSDTs overlay?
> > > >> Or with
> > > >> the DT?
> > > >> 
> > > >> thanks
> 
> Thanks,
> Alan
> 
> [4] https://www.kernel.org/doc/html/latest/driver-api/fpga/index.html
> [5] https://github.com/torvalds/linux/blob/master/Documentation/fpga/dfl.txt

Re: Device Description for FPGA Components on x86 system

[Enrico Weigelt, metux IT consult]

On 10.04.19 16:21, Alan Tull wrote:

> I wish I could point you to a complete solution, but there is a lot of> work to be done in this area.  Most of what is in the kernel is a low>
level in-kernel API [4].  As you correctly state, the hardest part of>
this is doing the enumerating if you are on x86 and aren't using>
devicetree.
ACK. I believe, introducing oftree the right way to go. I've got that
on my 2do list, too, but yet too busy w/ other things. So, if you'd
like to work on that, I'd highly appreciate that.

I'd like to also use oftree overlays for some platform drivers that
essentially just initialize/wire up generic drivers to perform some
board specific function (eg. drivers/platform/x86/pcengines-apuv2.c).


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287