Re: Re: [Patch RFC] ttm: nouveau accelerated on Xen pv-ops kernel

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

> > >>>         vma->vm_private_data = bo;
> > >>> -       vma->vm_flags |= VM_RESERVED | VM_IO | VM_MIXEDMAP |
> > >>> VM_DONTEXPAND;
> > >>> +       vma->vm_flags |= VM_RESERVED | VM_MIXEDMAP |
> > >>> VM_DONTEXPAND;
> > >>> +       if (!((bo->mem.placement & TTM_PL_MASK_MEM) &
> > >>> TTM_PL_FLAG_TT))
> > >>> +               vma->vm_flags |= VM_IO;
> > >>> +       vma->vm_page_prot = vma_get_vm_prot(vma->vm_flags);
> > >>>         return 0;
> > >>>  out_unref:
> > >>>         ttm_bo_unref(&bo);
> > >>>
.. snip.
> > >>> nVidia GeForce 9400 GT.

..snip..
> 
> Hmm... I just verified that this patch fixes KMS/Nouveau issues in Xen on 
> my two primary test boxes (GeForce 6200, GeForce 7300).  However, on my 
> really old machines (AGP GeForce2 MX200), this causes a new crash.  These 
> old boxes were actually working fine in Xen prior to this patch, just 
> w/out 3d acceleration.  Now I get the following messages in dmesg:
> 
> [  129.637319] [drm] nouveau 0000:01:00.0: Allocating FIFO number 1
> [  129.638853] [drm] nouveau 0000:01:00.0: nouveau_channel_alloc: 
> initialised FIFO 1
> [  129.643791] X: Corrupted page table at address 40412000
> [  129.643815] *pdpt = 0000000015216001 *pde = 0000000000000000 
> [  129.643856] Bad pagetable: 000f [#1] SMP 
.. snip..
> [  129.652216] [drm] nouveau 0000:01:00.0: nouveau_channel_free: freeing 
> fifo 1
> 
> 
> And my X log ends abruptly after this line:
> (II) NOUVEAU(0): Opened GPU Channel 1


So, I've spent the last two weeks trying to get this to work.

What I found out was that the majority of the problems were with the Linux AGP code
(drivers/char/agp/*). Now I can get Kernel Modesetting 
working under Radeon HD3200 (PCI-e), NVidia GeForce FX5200 (AGP), and I think
other ones too (hadn't tested yet). However, I am still failing at the same
spot as Michael: the dreaded Opened GPU Channel 1...

Anyhow.. if you want the patches:

git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git devel/kms-fixes

(and the "devel/drm.backport" which is a drop of drm/* from 2.6.34-rc7 in
2.6.32 universe).

The explanation:

1). Newer boxes with PCI-e cards with more than 4GB allocated to dom0.
Those cards have their own IOMMU built in on the cards and use the PCI
DMA. The problem is that some of those cards set the DMA mask to 32-bit,
meaning they are only comfortable allocating pages under the 4GB mark.
The Xen-SWIOTLB kicks in and happily gives a bounce buffer for pages
that are above the 32-bit mark, which are then programmed to the IOMMU.
Except that the graphic drivers do not sync the bounce buffer so the
result ends up stored in the bounce buffer and not in the buffer the
graphic drivers expect (oops).  That however should not have happend
as any pages allocated from the TTM library use the 'alloc_page' with
 _GFP_DMA32 flag, which means that all of those pages WILL be under
4GB under baremetal. However under Xen that flag is useless and the page
can be very well allocated above the 4GB mark (and is if you are using
the debug Xen hypervisor). The patch titled
 "ttm: When TTM_PAGE_FLAG_DMA32 allocate pages under" fixes that by
replacing the PFNs with ones under the 4GB.

It makes my Radeon HD3200 work, and I believe other PCI-e cards too.

2). i915 cards:
The earlier cards (before PCI-e) depend on the AGP to the virtual to
physical address translations.  The old AGP code is safe for specific 
controllers: say i915 - and only if you have CONFIG_DMAR enabled. So
there is a patch if you don't have that enabled:

      intel-agp: Warn when !USE_PCI_DMA_API and inserting bogus bus addresses.

3) i915 and before.
For older ones, such as the ICH5 (which has an i860 chipset), the Linux
code blindly uses the virt_to_phys and page_to_phys which are not
Xen-safe. For this fancy NV30 card (which is AGP) I've gotten to the
point of the framebuffer displaying properly. The patches that fix this
are:
  intel-agp: Use Xen back-door to get page's physical address for i8[1,3]0
      agp-backend: Use Xen back-door to get bus address for scratch page.
      agp: If _GFP_DMA_32 flag is set enforce pages to be under 4GB under Xen.
      agp: Program the GART with the real physical address under Xen.
      agp: Use Xen back-door to get bus address for legacy code.


4). I did check out the AGP code for the other chipsets, so added
experimental support for AMD64 chipset. Haven't tested it at all.

Summary:
Most of those patches are not upstream materials. Right now I am
just trying to get those bugs fixed and then later on revisit them with
a more comprehensive patch.