* Re: [Question] rk3399 vfio-pci/sr-iov support
@ 2019-12-09 14:07 ` Robin Murphy
0 siblings, 0 replies; 12+ messages in thread
From: Robin Murphy @ 2019-12-09 14:07 UTC (permalink / raw)
To: Peter Geis, Heiko Stuebner
Cc: open list:ARM/Rockchip SoC..., linux-pci-u79uwXL29TY76Z2rM5mHXA
On 09/12/2019 1:28 pm, Peter Geis wrote:
> Good Morning,
>
> I'm back with more pcie fun on the rk3399.
> I'm trying to get pcie passthrough working for a vm on the rk3399, and
> have encountered some roadblocks.
>
> First, vfio-pci doesn't work on the rk3399, as the pcie controller
> doesn't bind explicitly to a iommu.
> [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
>
> # find /sys/kernel/iommu_groups/ -type l
> /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> /sys/kernel/iommu_groups/2/devices/ff900000.vop
>
> # virsh start openwrt
> error: Failed to start domain openwrt
> error: internal error: Process exited prior to exec: libvirt: error :
> internal error: Invalid device 0000:01:00.0 iommu_group file
> /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink
That much I can help with somewhat: the major impediment is that RK3399
doesn't have an IOMMU in front of PCIe. As far as I'm aware your only
option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode
(which I don't know an awful lot about beyond that it's a thing).
Robin.
> Second, sr-iov support is broken.
> root@rockpro64:/sys/bus/pci/devices/0000:01:00.0# echo 1 > sriov_numvfs
> bash: echo: write error: Input/output error
> [37352.907558] pci 0000:01:10.0: [8086:1520] type 7f class 0xffffff
> [37352.907578] pci 0000:01:10.0: unknown header type 7f, ignoring device
>
> Do any of y'all have some insight into these issues?
>
> _______________________________________________
> Linux-rockchip mailing list
> Linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org
> http://lists.infradead.org/mailman/listinfo/linux-rockchip
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Question] rk3399 vfio-pci/sr-iov support
@ 2019-12-09 17:05 ` Alex Williamson
0 siblings, 0 replies; 12+ messages in thread
From: Alex Williamson @ 2019-12-09 17:05 UTC (permalink / raw)
To: Robin Murphy
Cc: Peter Geis, Heiko Stuebner, open list:ARM/Rockchip SoC...,
linux-pci
On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy@arm.com> wrote:
> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > Good Morning,
> >
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > have encountered some roadblocks.
> >
> > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > doesn't bind explicitly to a iommu.
> > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> >
> > # find /sys/kernel/iommu_groups/ -type l
> > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> >
> > # virsh start openwrt
> > error: Failed to start domain openwrt
> > error: internal error: Process exited prior to exec: libvirt: error :
> > internal error: Invalid device 0000:01:00.0 iommu_group file
> > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink
>
> That much I can help with somewhat: the major impediment is that RK3399
> doesn't have an IOMMU in front of PCIe. As far as I'm aware your only
> option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode
> (which I don't know an awful lot about beyond that it's a thing).
And it's a thing that's really only useful if your motivation is to run
something like DPDK in the host and you're not concerned about
isolation between userspace drivers and the host kernel, and you don't
mind tainting the kernel. It's not useful for things like assigning a
device to a VM as we can't readily do that without an IOMMU for
translation. Thanks,
Alex
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Question] rk3399 vfio-pci/sr-iov support
@ 2019-12-09 17:05 ` Alex Williamson
0 siblings, 0 replies; 12+ messages in thread
From: Alex Williamson @ 2019-12-09 17:05 UTC (permalink / raw)
To: Robin Murphy
Cc: open list:ARM/Rockchip SoC..., Heiko Stuebner, Peter Geis,
linux-pci-u79uwXL29TY76Z2rM5mHXA
On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org> wrote:
> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > Good Morning,
> >
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > have encountered some roadblocks.
> >
> > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > doesn't bind explicitly to a iommu.
> > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> >
> > # find /sys/kernel/iommu_groups/ -type l
> > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> >
> > # virsh start openwrt
> > error: Failed to start domain openwrt
> > error: internal error: Process exited prior to exec: libvirt: error :
> > internal error: Invalid device 0000:01:00.0 iommu_group file
> > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink
>
> That much I can help with somewhat: the major impediment is that RK3399
> doesn't have an IOMMU in front of PCIe. As far as I'm aware your only
> option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode
> (which I don't know an awful lot about beyond that it's a thing).
And it's a thing that's really only useful if your motivation is to run
something like DPDK in the host and you're not concerned about
isolation between userspace drivers and the host kernel, and you don't
mind tainting the kernel. It's not useful for things like assigning a
device to a VM as we can't readily do that without an IOMMU for
translation. Thanks,
Alex
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Question] rk3399 vfio-pci/sr-iov support
@ 2019-12-10 1:23 ` Peter Geis
0 siblings, 0 replies; 12+ messages in thread
From: Peter Geis @ 2019-12-10 1:23 UTC (permalink / raw)
To: Alex Williamson
Cc: Robin Murphy, Heiko Stuebner, open list:ARM/Rockchip SoC...,
linux-pci
On Mon, Dec 9, 2019 at 12:05 PM Alex Williamson
<alex.williamson@redhat.com> wrote:
>
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy@arm.com> wrote:
>
> > On 09/12/2019 1:28 pm, Peter Geis wrote:
> > > Good Morning,
> > >
> > > I'm back with more pcie fun on the rk3399.
> > > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > > have encountered some roadblocks.
> > >
> > > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > > doesn't bind explicitly to a iommu.
> > > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> > >
> > > # find /sys/kernel/iommu_groups/ -type l
> > > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> > >
> > > # virsh start openwrt
> > > error: Failed to start domain openwrt
> > > error: internal error: Process exited prior to exec: libvirt: error :
> > > internal error: Invalid device 0000:01:00.0 iommu_group file
> > > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink
> >
> > That much I can help with somewhat: the major impediment is that RK3399
> > doesn't have an IOMMU in front of PCIe. As far as I'm aware your only
> > option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode
> > (which I don't know an awful lot about beyond that it's a thing).
>
> And it's a thing that's really only useful if your motivation is to run
> something like DPDK in the host and you're not concerned about
> isolation between userspace drivers and the host kernel, and you don't
> mind tainting the kernel. It's not useful for things like assigning a
> device to a VM as we can't readily do that without an IOMMU for
> translation. Thanks,
>
> Alex
Thanks Robin,
That is an unfortunate limitation, especially considering the device
is supposed to support sr-iov.
I did try getting VFIO_NOIOMMU to work, but qemu does not like it, at
least for pci devices.
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Question] rk3399 vfio-pci/sr-iov support
@ 2019-12-10 1:23 ` Peter Geis
0 siblings, 0 replies; 12+ messages in thread
From: Peter Geis @ 2019-12-10 1:23 UTC (permalink / raw)
To: Alex Williamson
Cc: open list:ARM/Rockchip SoC..., Robin Murphy, Heiko Stuebner,
linux-pci-u79uwXL29TY76Z2rM5mHXA
On Mon, Dec 9, 2019 at 12:05 PM Alex Williamson
<alex.williamson-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
>
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy-5wv7dgnIgG8@public.gmane.org> wrote:
>
> > On 09/12/2019 1:28 pm, Peter Geis wrote:
> > > Good Morning,
> > >
> > > I'm back with more pcie fun on the rk3399.
> > > I'm trying to get pcie passthrough working for a vm on the rk3399, and
> > > have encountered some roadblocks.
> > >
> > > First, vfio-pci doesn't work on the rk3399, as the pcie controller
> > > doesn't bind explicitly to a iommu.
> > > [37528.138212] vfio-pci 0000:01:00.0: assign IRQ: got 226
> > > [37528.138254] vfio-pci: probe of 0000:01:00.0 failed with error -22
> > >
> > > # find /sys/kernel/iommu_groups/ -type l
> > > /sys/kernel/iommu_groups/1/devices/ff8f0000.vop
> > > /sys/kernel/iommu_groups/2/devices/ff900000.vop
> > >
> > > # virsh start openwrt
> > > error: Failed to start domain openwrt
> > > error: internal error: Process exited prior to exec: libvirt: error :
> > > internal error: Invalid device 0000:01:00.0 iommu_group file
> > > /sys/bus/pci/devices/0000:01:00.0/iommu_group is not a symlink
> >
> > That much I can help with somewhat: the major impediment is that RK3399
> > doesn't have an IOMMU in front of PCIe. As far as I'm aware your only
> > option is to resort to the "here be dragons" CONFIG_VFIO_NOIOMMU mode
> > (which I don't know an awful lot about beyond that it's a thing).
>
> And it's a thing that's really only useful if your motivation is to run
> something like DPDK in the host and you're not concerned about
> isolation between userspace drivers and the host kernel, and you don't
> mind tainting the kernel. It's not useful for things like assigning a
> device to a VM as we can't readily do that without an IOMMU for
> translation. Thanks,
>
> Alex
Thanks Robin,
That is an unfortunate limitation, especially considering the device
is supposed to support sr-iov.
I did try getting VFIO_NOIOMMU to work, but qemu does not like it, at
least for pci devices.
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [Question] rk3399 vfio-pci/sr-iov support
2019-12-09 14:07 ` Robin Murphy
@ 2021-11-15 19:50 ` Adam Joseph
-1 siblings, 0 replies; 12+ messages in thread
From: Adam Joseph @ 2021-11-15 19:50 UTC (permalink / raw)
To: Robin Murphy
Cc: Peter Geis, Heiko Stuebner, open list:ARM/Rockchip SoC...,
linux-pci
On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy@arm.com> wrote:
> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399,
> > and have encountered some roadblocks.
>
> That much I can help with somewhat: the major impediment is that
> RK3399 doesn't have an IOMMU in front of PCIe.
For the more limited case of defending against attacks from
hostile/buggy firmware on PCIe devices: is it possible to use the
RK3399 PCIe "inbound address translation" support instead of an IOMMU?
The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
configure address translation (including a base/bounds check) for
inbound-to-SoC memory writes, but details are quite sparse.
Linux appears to not use this functionality; from
drivers/pci/controller/pcie-rockchip-host.c we can see that it
disables the base/bounds (sets them to the entire 32-bit space, which
is all RAM on RK3399 since it supports only 4GB) and passes all 32
bits of incoming memory writes:
static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
int region_no, u8 num_pass_bits,
u32 lower_addr, u32 upper_addr)
...
static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
{
...
err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);
Is this a dead end? If not I might pursue it, if I can get the
necessary documentation. I couldn't find any mention of ATS in the
RK3399 manual; if the PCIe RC allows that then all bets are off anyways.
- a
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [Question] rk3399 vfio-pci/sr-iov support
@ 2021-11-15 19:50 ` Adam Joseph
0 siblings, 0 replies; 12+ messages in thread
From: Adam Joseph @ 2021-11-15 19:50 UTC (permalink / raw)
To: Robin Murphy
Cc: Peter Geis, Heiko Stuebner, open list:ARM/Rockchip SoC...,
linux-pci
On Mon, 9 Dec 2019 14:07:02 +0000
Robin Murphy <robin.murphy@arm.com> wrote:
> On 09/12/2019 1:28 pm, Peter Geis wrote:
> > I'm back with more pcie fun on the rk3399.
> > I'm trying to get pcie passthrough working for a vm on the rk3399,
> > and have encountered some roadblocks.
>
> That much I can help with somewhat: the major impediment is that
> RK3399 doesn't have an IOMMU in front of PCIe.
For the more limited case of defending against attacks from
hostile/buggy firmware on PCIe devices: is it possible to use the
RK3399 PCIe "inbound address translation" support instead of an IOMMU?
The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
configure address translation (including a base/bounds check) for
inbound-to-SoC memory writes, but details are quite sparse.
Linux appears to not use this functionality; from
drivers/pci/controller/pcie-rockchip-host.c we can see that it
disables the base/bounds (sets them to the entire 32-bit space, which
is all RAM on RK3399 since it supports only 4GB) and passes all 32
bits of incoming memory writes:
static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
int region_no, u8 num_pass_bits,
u32 lower_addr, u32 upper_addr)
...
static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
{
...
err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);
Is this a dead end? If not I might pursue it, if I can get the
necessary documentation. I couldn't find any mention of ATS in the
RK3399 manual; if the PCIe RC allows that then all bets are off anyways.
- a
_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [Question] rk3399 vfio-pci/sr-iov support
2021-11-15 19:50 ` Adam Joseph
@ 2021-11-16 10:28 ` Robin Murphy
-1 siblings, 0 replies; 12+ messages in thread
From: Robin Murphy @ 2021-11-16 10:28 UTC (permalink / raw)
To: Adam Joseph
Cc: Peter Geis, Heiko Stuebner, open list:ARM/Rockchip SoC...,
linux-pci
On 2021-11-15 19:50, Adam Joseph wrote:
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy@arm.com> wrote:
>> On 09/12/2019 1:28 pm, Peter Geis wrote:
>>> I'm back with more pcie fun on the rk3399.
>>> I'm trying to get pcie passthrough working for a vm on the rk3399,
>>> and have encountered some roadblocks.
>>
>> That much I can help with somewhat: the major impediment is that
>> RK3399 doesn't have an IOMMU in front of PCIe.
>
> For the more limited case of defending against attacks from
> hostile/buggy firmware on PCIe devices: is it possible to use the
> RK3399 PCIe "inbound address translation" support instead of an IOMMU?
I guess it might be possible to reserve some memory with the new
"restricted-dma-pool" semantics then tweak the PCIe setup to match that.
It will mostly depend on what the RC does with transactions that don't
match any ATU entry. These kinds of registers typically only decode a
small number of naturally-aligned power-of-two-sized address ranges so
would not be able to support the kind of arbitrary translation/page
protection you'd need for something like VFIO.
> The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
> configure address translation (including a base/bounds check) for
> inbound-to-SoC memory writes, but details are quite sparse.
>
> Linux appears to not use this functionality; from
> drivers/pci/controller/pcie-rockchip-host.c we can see that it
> disables the base/bounds (sets them to the entire 32-bit space, which
> is all RAM on RK3399 since it supports only 4GB) and passes all 32
> bits of incoming memory writes:
>
> static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
> int region_no, u8 num_pass_bits,
> u32 lower_addr, u32 upper_addr)
> ...
>
> static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
> {
> ...
> err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);
>
> Is this a dead end? If not I might pursue it, if I can get the
> necessary documentation. I couldn't find any mention of ATS in the
> RK3399 manual; if the PCIe RC allows that then all bets are off anyways.
Knowing this RC, if it ever did see an ATS translated transaction header
it would probably SError the system, like it does for such outlandish
things as trying to probe config space behind bridges :)
Robin.
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: [Question] rk3399 vfio-pci/sr-iov support
@ 2021-11-16 10:28 ` Robin Murphy
0 siblings, 0 replies; 12+ messages in thread
From: Robin Murphy @ 2021-11-16 10:28 UTC (permalink / raw)
To: Adam Joseph
Cc: Peter Geis, Heiko Stuebner, open list:ARM/Rockchip SoC...,
linux-pci
On 2021-11-15 19:50, Adam Joseph wrote:
> On Mon, 9 Dec 2019 14:07:02 +0000
> Robin Murphy <robin.murphy@arm.com> wrote:
>> On 09/12/2019 1:28 pm, Peter Geis wrote:
>>> I'm back with more pcie fun on the rk3399.
>>> I'm trying to get pcie passthrough working for a vm on the rk3399,
>>> and have encountered some roadblocks.
>>
>> That much I can help with somewhat: the major impediment is that
>> RK3399 doesn't have an IOMMU in front of PCIe.
>
> For the more limited case of defending against attacks from
> hostile/buggy firmware on PCIe devices: is it possible to use the
> RK3399 PCIe "inbound address translation" support instead of an IOMMU?
I guess it might be possible to reserve some memory with the new
"restricted-dma-pool" semantics then tweak the PCIe setup to match that.
It will mostly depend on what the RC does with transactions that don't
match any ATU entry. These kinds of registers typically only decode a
small number of naturally-aligned power-of-two-sized address ranges so
would not be able to support the kind of arbitrary translation/page
protection you'd need for something like VFIO.
> The RK3399 TRM, v1.3 "Part 2", section 17.5.5.2.1 explains how to
> configure address translation (including a base/bounds check) for
> inbound-to-SoC memory writes, but details are quite sparse.
>
> Linux appears to not use this functionality; from
> drivers/pci/controller/pcie-rockchip-host.c we can see that it
> disables the base/bounds (sets them to the entire 32-bit space, which
> is all RAM on RK3399 since it supports only 4GB) and passes all 32
> bits of incoming memory writes:
>
> static int rockchip_pcie_prog_ib_atu(struct rockchip_pcie *rockchip,
> int region_no, u8 num_pass_bits,
> u32 lower_addr, u32 upper_addr)
> ...
>
> static int rockchip_pcie_cfg_atu(struct rockchip_pcie *rockchip)
> {
> ...
> err = rockchip_pcie_prog_ib_atu(rockchip, 2, 32 - 1, 0x0, 0);
>
> Is this a dead end? If not I might pursue it, if I can get the
> necessary documentation. I couldn't find any mention of ATS in the
> RK3399 manual; if the PCIe RC allows that then all bets are off anyways.
Knowing this RC, if it ever did see an ATS translated transaction header
it would probably SError the system, like it does for such outlandish
things as trying to probe config space behind bridges :)
Robin.
_______________________________________________
Linux-rockchip mailing list
Linux-rockchip@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-rockchip
^ permalink raw reply [flat|nested] 12+ messages in thread