Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson <ydirson@free.fr>]

Christian wrote:
> Am 19.12.21 um 17:00 schrieb Yann Dirson:
> > Alex wrote:
> >> Thinking about this more, I think the problem might be related to
> >> CPU
> >> access to "VRAM".  APUs don't have dedicated VRAM, they use a
> >> reserved
> >> carve out region at the top of system memory.  For CPU access to
> >> this
> >> memory, we kmap the physical address of the carve out region of
> >> system
> >> memory.  You'll need to make sure that region is accessible to the
> >> guest.
> > So basically, the non-virt flow is is: (video?) BIOS reserves
> > memory, marks it
> > as reserved in e820, stores the physaddr somewhere, which the GPU
> > driver gets.
> > Since I suppose this includes the framebuffer, this probably has to
> > occur around
> > the moment the driver calls
> > drm_aperture_remove_conflicting_pci_framebuffers()
> > (which happens before this hw init step), right ?
> 
> Well, that partially correct. The efifb is using the PCIe resources
> to
> access the framebuffer and as far as I know we use that one to kick
> it out.
> 
> The stolen memory we get over e820/registers is separate to that.
> 
> > ... which brings me to a point that's been puzzling me for some
> > time, which is
> > that as the hw init fails, the efifb driver is still using the
> > framebuffer.
> 
> No, it isn't. You are probably just still seeing the same screen.
> 
> The issue is most likely that while efi was kicked out nobody
> re-programmed the display hardware to show something different.
> 
> > Am I right in suspecting that efifb should get stripped of its
> > ownership of the
> > fb aperture first, and that if I don't get a black screen on
> > hw_init failure
> > that issue should be the first focus point ?
> 
> You assumption with the black screen is incorrect. Since the hardware
> works independent even if you kick out efi you still have the same
> screen content, you just can't update it anymore.

It's not only that the screen keeps its contents, it's that the dom0
happily continues updating it.

> But putting efi asside what Alex pointed out pretty much breaks your
> neck trying to forward the device. You maybe could try to hack the
> driver to use the PCIe BAR for framebuffer access, but that might be
> quite a bit slower.
> 
> Regards,
> Christian.
> 
> >
> >> Alex
> >>
> >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher
> >> <alexdeucher@gmail.com>
> >> wrote:
> >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
> >>> wrote:
> >>>> Alex wrote:
> >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr>
> >>>>> wrote:
> >>>>>> Hi Alex,
> >>>>>>
> >>>>>>> We have not validated virtualization of our integrated
> >>>>>>> GPUs.  I
> >>>>>>> don't
> >>>>>>> know that it will work at all.  We had done a bit of
> >>>>>>> testing but
> >>>>>>> ran
> >>>>>>> into the same issues with the PSP, but never had a chance
> >>>>>>> to
> >>>>>>> debug
> >>>>>>> further because this feature is not productized.
> >>>>>> ...
> >>>>>>> You need a functional PSP to get the GPU driver up and
> >>>>>>> running.
> >>>>>> Ah, thanks for the hint :)
> >>>>>>
> >>>>>> I guess that if I want to have any chance to get the PSP
> >>>>>> working
> >>>>>> I'm
> >>>>>> going to need more details on it.  A quick search some time
> >>>>>> ago
> >>>>>> mostly
> >>>>>> brought reverse-engineering work, rather than official AMD
> >>>>>> doc.
> >>>>>>   Are
> >>>>>> there some AMD resources I missed ?
> >>>>> The driver code is pretty much it.
> >>>> Let's try to shed some more light on how things work, taking as
> >>>> excuse
> >>>> psp_v12_0_ring_create().
> >>>>
> >>>> First, register access through [RW]REG32_SOC15() is implemented
> >>>> in
> >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> >>>> [RW]REG32(),
> >>>> except it has to be more complex in the SR-IOV case.
> >>>> Has the RLC anything to do with SR-IOV ?
> >>> When running the driver on a SR-IOV virtual function (VF), some
> >>> registers are not available directly via the VF's MMIO aperture
> >>> so
> >>> they need to go through the RLC.  For bare metal or passthrough
> >>> this
> >>> is not relevant.
> >>>
> >>>> It accesses registers in the MMIO range of the MP0 IP, and the
> >>>> "MP0"
> >>>> name correlates highly with MMIO accesses in PSP-handling code.
> >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
> >>>> version
> >>> Yes.
> >>>
> >>>> reported at v11.0.3 by discovery seems to contradict the use of
> >>>> v12.0
> >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> >>>> something ?
> >>> Typo in the ip discovery table on renoir.
> >>>
> >>>> More generally (and mostly out of curiosity while we're at it),
> >>>> do we
> >>>> have a way to match IPs listed at discovery time with the ones
> >>>> used
> >>>> in the driver ?
> >>> In general, barring typos, the code is shared at the major
> >>> version
> >>> level.  The actual code may or may not need changes to handle
> >>> minor
> >>> revision changes in an IP.  The driver maps the IP versions from
> >>> the
> >>> ip discovery table to the code contained in the driver.
> >>>
> >>>> ---
> >>>>
> >>>> As for the register names, maybe we could have a short
> >>>> explanation of
> >>>> how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems
> >>>> to
> >>>> be
> >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not
> >>>> sure
> >>>> of the "SMN" part -- that could refer to the "System Management
> >>>> Network",
> >>>> described in [0] as an internal bus.  Are we accessing this
> >>>> register
> >>>> through this SMN ?
> >>> These registers are just mailboxes for the PSP firmware.  All of
> >>> the
> >>> C2PMSG registers functionality is defined by the PSP firmware.
> >>>   They
> >>> are basically scratch registers used to communicate between the
> >>> driver
> >>> and the PSP firmware.
> >>>
> >>>>
> >>>>>   On APUs, the PSP is shared with
> >>>>> the CPU and the rest of the platform.  The GPU driver just
> >>>>> interacts
> >>>>> with it for a few specific tasks:
> >>>>> 1. Loading Trusted Applications (e.g., trusted firmware
> >>>>> applications
> >>>>> that run on the PSP for specific functionality, e.g., HDCP and
> >>>>> content
> >>>>> protection, etc.)
> >>>>> 2. Validating and loading firmware for other engines on the
> >>>>> SoC.
> >>>>>   This
> >>>>> is required to use those engines.
> >>>> Trying to understand in more details how we start the PSP up, I
> >>>> noticed
> >>>> that psp_v12_0 has support for loading a sOS firmware, but never
> >>>> calls
> >>>> init_sos_microcode() - and anyway there is no sos firmware for
> >>>> renoir
> >>>> and green_sardine, which seem to be the only ASICs with this PSP
> >>>> version.
> >>>> Is it something that's just not been completely wired up yet ?
> >>> On APUs, the PSP is shared with the CPU so the PSP firmware is
> >>> part
> >>> of
> >>> the sbios image.  The driver doesn't load it.  We only load it on
> >>> dGPUs where the driver is responsible for the chip
> >>> initialization.
> >>>
> >>>> That also rings a bell, that we have nothing about Secure OS in
> >>>> the doc
> >>>> yet (not even the acronym in the glossary).
> >>>>
> >>>>
> >>>>> I'm not too familiar with the PSP's path to memory from the GPU
> >>>>> perspective.  IIRC, most memory used by the PSP goes through
> >>>>> carve
> >>>>> out
> >>>>> "vram" on APUs so it should work, but I would double check if
> >>>>> there
> >>>>> are any system memory allocations that used to interact with
> >>>>> the PSP
> >>>>> and see if changing them to vram helps.  It does work with the
> >>>>> IOMMU
> >>>>> enabled on bare metal, so it should work in passthrough as well
> >>>>> in
> >>>>> theory.
> >>>> I can see a single case in the PSP code where GTT is used
> >>>> instead
> >>>> of
> >>>> vram: to create fw_pri_bo when SR-IOV is not used (and there has
> >>>> to be a reason, since the SR-IOV code path does use vram).
> >>>> Changing it to vram does not make a difference, but then the
> >>>> only bo that seems to be used at that point is the one for the
> >>>> psp ring,
> >>>> which is allocated in vram, so I'm not too much surprised.
> >>>>
> >>>> Maybe I should double-check bo_create calls to hunt for more ?
> >>> We looked into this a bit ourselves and ran into the same issues.
> >>> We'd probably need to debug this with the PSP team to make
> >>> further
> >>> progress, but this was not productized so neither team had the
> >>> resources to delve further.
> >>>
> >>> Alex
> >>>
> >>>>
> >>>> [0]
> >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
> 
>