Re: Various problems trying to vga-passthrough a Renoir iGPU to a xen/qubes-os hvm

[Yann Dirson <ydirson@free.fr>]

Alex wrote:
> Thinking about this more, I think the problem might be related to CPU
> access to "VRAM".  APUs don't have dedicated VRAM, they use a
> reserved
> carve out region at the top of system memory.  For CPU access to this
> memory, we kmap the physical address of the carve out region of
> system
> memory.  You'll need to make sure that region is accessible to the
> guest.

So basically, the non-virt flow is is: (video?) BIOS reserves memory, marks it
as reserved in e820, stores the physaddr somewhere, which the GPU driver gets.
Since I suppose this includes the framebuffer, this probably has to occur around
the moment the driver calls drm_aperture_remove_conflicting_pci_framebuffers()
(which happens before this hw init step), right ?

... which brings me to a point that's been puzzling me for some time, which is
that as the hw init fails, the efifb driver is still using the framebuffer.

Am I right in suspecting that efifb should get stripped of its ownership of the
fb aperture first, and that if I don't get a black screen on hw_init failure
that issue should be the first focus point ?

> 
> Alex
> 
> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher <alexdeucher@gmail.com>
> wrote:
> >
> > On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@free.fr>
> > wrote:
> > >
> > > Alex wrote:
> > > > On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@free.fr>
> > > > wrote:
> > > > >
> > > > > Hi Alex,
> > > > >
> > > > > > We have not validated virtualization of our integrated
> > > > > > GPUs.  I
> > > > > > don't
> > > > > > know that it will work at all.  We had done a bit of
> > > > > > testing but
> > > > > > ran
> > > > > > into the same issues with the PSP, but never had a chance
> > > > > > to
> > > > > > debug
> > > > > > further because this feature is not productized.
> > > > > ...
> > > > > > You need a functional PSP to get the GPU driver up and
> > > > > > running.
> > > > >
> > > > > Ah, thanks for the hint :)
> > > > >
> > > > > I guess that if I want to have any chance to get the PSP
> > > > > working
> > > > > I'm
> > > > > going to need more details on it.  A quick search some time
> > > > > ago
> > > > > mostly
> > > > > brought reverse-engineering work, rather than official AMD
> > > > > doc.
> > > > >  Are
> > > > > there some AMD resources I missed ?
> > > >
> > > > The driver code is pretty much it.
> > >
> > > Let's try to shed some more light on how things work, taking as
> > > excuse
> > > psp_v12_0_ring_create().
> > >
> > > First, register access through [RW]REG32_SOC15() is implemented
> > > in
> > > terms of __[RW]REG32_SOC15_RLC__(), which is basically a
> > > [RW]REG32(),
> > > except it has to be more complex in the SR-IOV case.
> > > Has the RLC anything to do with SR-IOV ?
> >
> > When running the driver on a SR-IOV virtual function (VF), some
> > registers are not available directly via the VF's MMIO aperture so
> > they need to go through the RLC.  For bare metal or passthrough
> > this
> > is not relevant.
> >
> > >
> > > It accesses registers in the MMIO range of the MP0 IP, and the
> > > "MP0"
> > > name correlates highly with MMIO accesses in PSP-handling code.
> > > Is "MP0" another name for PSP (and "MP1" for SMU) ?  The MP0
> > > version
> >
> > Yes.
> >
> > > reported at v11.0.3 by discovery seems to contradict the use of
> > > v12.0
> > > for RENOIR as set by soc15_set_ip_blocks(), or do I miss
> > > something ?
> >
> > Typo in the ip discovery table on renoir.
> >
> > >
> > > More generally (and mostly out of curiosity while we're at it),
> > > do we
> > > have a way to match IPs listed at discovery time with the ones
> > > used
> > > in the driver ?
> >
> > In general, barring typos, the code is shared at the major version
> > level.  The actual code may or may not need changes to handle minor
> > revision changes in an IP.  The driver maps the IP versions from
> > the
> > ip discovery table to the code contained in the driver.
> >
> > >
> > > ---
> > >
> > > As for the register names, maybe we could have a short
> > > explanation of
> > > how they are structured ?  Eg. mmMP0_SMN_C2PMSG_69: that seems to
> > > be
> > > a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not
> > > sure
> > > of the "SMN" part -- that could refer to the "System Management
> > > Network",
> > > described in [0] as an internal bus.  Are we accessing this
> > > register
> > > through this SMN ?
> >
> > These registers are just mailboxes for the PSP firmware.  All of
> > the
> > C2PMSG registers functionality is defined by the PSP firmware.
> >  They
> > are basically scratch registers used to communicate between the
> > driver
> > and the PSP firmware.
> >
> > >
> > >
> > > >  On APUs, the PSP is shared with
> > > > the CPU and the rest of the platform.  The GPU driver just
> > > > interacts
> > > > with it for a few specific tasks:
> > > > 1. Loading Trusted Applications (e.g., trusted firmware
> > > > applications
> > > > that run on the PSP for specific functionality, e.g., HDCP and
> > > > content
> > > > protection, etc.)
> > > > 2. Validating and loading firmware for other engines on the
> > > > SoC.
> > > >  This
> > > > is required to use those engines.
> > >
> > > Trying to understand in more details how we start the PSP up, I
> > > noticed
> > > that psp_v12_0 has support for loading a sOS firmware, but never
> > > calls
> > > init_sos_microcode() - and anyway there is no sos firmware for
> > > renoir
> > > and green_sardine, which seem to be the only ASICs with this PSP
> > > version.
> > > Is it something that's just not been completely wired up yet ?
> >
> > On APUs, the PSP is shared with the CPU so the PSP firmware is part
> > of
> > the sbios image.  The driver doesn't load it.  We only load it on
> > dGPUs where the driver is responsible for the chip initialization.
> >
> > >
> > > That also rings a bell, that we have nothing about Secure OS in
> > > the doc
> > > yet (not even the acronym in the glossary).
> > >
> > >
> > > > I'm not too familiar with the PSP's path to memory from the GPU
> > > > perspective.  IIRC, most memory used by the PSP goes through
> > > > carve
> > > > out
> > > > "vram" on APUs so it should work, but I would double check if
> > > > there
> > > > are any system memory allocations that used to interact with
> > > > the PSP
> > > > and see if changing them to vram helps.  It does work with the
> > > > IOMMU
> > > > enabled on bare metal, so it should work in passthrough as well
> > > > in
> > > > theory.
> > >
> > > I can see a single case in the PSP code where GTT is used instead
> > > of
> > > vram: to create fw_pri_bo when SR-IOV is not used (and there has
> > > to be a reason, since the SR-IOV code path does use vram).
> > > Changing it to vram does not make a difference, but then the
> > > only bo that seems to be used at that point is the one for the
> > > psp ring,
> > > which is allocated in vram, so I'm not too much surprised.
> > >
> > > Maybe I should double-check bo_create calls to hunt for more ?
> >
> > We looked into this a bit ourselves and ran into the same issues.
> > We'd probably need to debug this with the PSP team to make further
> > progress, but this was not productized so neither team had the
> > resources to delve further.
> >
> > Alex
> >
> > >
> > >
> > > [0]
> > > https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
>