IOMMU support for xen in BIOS of AMD based HP DL385g7 and kernel 3.0.0-rc7

IOMMU support for xen in BIOS of AMD based HP DL385g7 and kernel 3.0.0-rc7

[M. Schneider]

Hello Jörg,

Please allow me one question about IOMMU support in AMDS's 890FX chipset 
and 6176SE CPUs used in HP DL385g7 servers. These chips support IOMMU as 
far as I know. BIOS of HP DL385g7 doesn't currently offer an IOMMU flag.

I have contacted already HP support due to this missing BIOS setting 
options but less or more my request has been ignored (no solution and no 
plans for solving of this important issue  yet) so I am looking for a 
workaround as IOMMU support is required for passing thoruhg of PCI 
devices to domU domains (xen 4.0 / 4.1).

I have created xen debian live binary-hybrid iso images with kernel 
2.6.39-2 and 3.0.0-rc7 (aufs patch) for easy testing (inside the linke 
below there is information where you can download these images).
http://www.it-infrastrukturen.com/fileadmin/linux/debian-live-xen/README.xen

In the kernel 3.0.0-rc7 of 
xen41-wheezy-kernel3-amd64-live-gnome-binary-hybrid.iso I have enalbed 
IOMMU-debugging (s. the link to the image below)
rsync -avP 
rsync://www.it-infrastrukturen.ch/ftp/xen41-wheezy-kernel3-amd64-live-gnome-binary-hybrid.iso 
.

In the boot log are messages as follows:
# ---

(XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) ERST table is invalid
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2300.118 MHz processor.
(XEN) Initing memory sharing.
(XEN) AMD-Vi: IOMMU not found!
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  ->  Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 64 KiB.
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN)  - Nested Page Tables (NPT)
(XEN)  - Last Branch Record (LBR) Virtualisation
(XEN)  - Next-RIP Saved on #VMEXIT
(XEN)  - Pause-Intercept Filter
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Brought up 24 CPUs
(XEN) Xenoprofile: AMD IBS detected (0x0000001f)
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 ->  0x1937000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   000000023c000000->000000023e000000 (376344 pages to be allocated)
(XEN)  Init. ramdisk: 000000043dc18000->000000043fdff600

# ---

There is no log entry for an IVRS table (thank you to Wei Wang at AMD 
for his kind hints)

Please take a lok at full log files if you need more details:
http://www.it-infrastrukturen.com/fileadmin/linux/paste/xm_dmesg.xen41_on_DL385g7_NOK.txt
http://www.it-infrastrukturen.com/fileadmin/linux/paste/dmesg.xen41_on_DL385g7_NOK.txt

Thank you in advance for any answer/hints

Best regards,
Mark Schneider

-- 
IT-Infrastrukturen Schneider

Wieslergasse 6
CH-8049 Zürich

Phone:  +41 43 818 4508
Mobile: +41 76 522 2923

ms@it-infrastrukturen.com
http://www.it-infrastrukturen.com

Re: IOMMU support for xen in BIOS of AMD based HP DL385g7 and kernel 3.0.0-rc7

[Roedel, Joerg]

Hi Mark,

On Sun, Jul 17, 2011 at 06:34:33AM -0400, M. Schneider wrote:
> Please allow me one question about IOMMU support in AMDS's 890FX chipset 
> and 6176SE CPUs used in HP DL385g7 servers. These chips support IOMMU as 
> far as I know. BIOS of HP DL385g7 doesn't currently offer an IOMMU flag.
> 
> I have contacted already HP support due to this missing BIOS setting 
> options but less or more my request has been ignored (no solution and no 
> plans for solving of this important issue  yet) so I am looking for a 
> workaround as IOMMU support is required for passing thoruhg of PCI 
> devices to domU domains (xen 4.0 / 4.1).

I am sorry but if there is no IVRS table there is not much we can do.
The IVRS table contains the information which device uses which
request-ids. Without this information the IOMMU can not work reliably.

Regards,

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632

Re: IOMMU support for xen in BIOS of AMD based HP DL385g7 and kernel 3.0.0-rc7

[M. Schneider]

Am 20.07.2011 14:45, schrieb Roedel, Joerg:
> On Sun, Jul 17, 2011 at 06:34:33AM -0400, M. Schneider wrote:
>    
>> Please allow me one question about IOMMU support in AMDS's 890FX chipset
>> and 6176SE CPUs used in HP DL385g7 servers. These chips support IOMMU as
>> far as I know. BIOS of HP DL385g7 doesn't currently offer an IOMMU flag.
>>
>> I have contacted already HP support due to this missing BIOS setting
>> options but less or more my request has been ignored (no solution and no
>> plans for solving of this important issue  yet) so I am looking for a
>> workaround as IOMMU support is required for passing thoruhg of PCI
>> devices to domU domains (xen 4.0 / 4.1).
>>      
> I am sorry but if there is no IVRS table there is not much we can do.
> The IVRS table contains the information which device uses which
> request-ids. Without this information the IOMMU can not work reliably.
>    

Thank you very much fo your answer Jörg.

Some BIOSes (like BIOS of HD DL385g7 server with AMD CPUs) don't have 
the IOMMU option.
I have to correct my prevoius email as HP DL385g7 servers have the AMD 
SR5690/SP5100 chipset (supporting IOMMU).

Is it possible to write a workaround for xen 4.1 or 4.2 to create such 
IVRS table in code (in case it has static entries for specific server 
type) and add a boot option like "iommu=force-dl385g7" ?

Thank you in advance for any hints.

Best regards,
Mark

-- 
IT-Infrastrukturen Schneider

Wieslergasse 6
CH-8049 Zürich

Phone:  +41 43 818 4508
Mobile: +41 76 522 2923

ms@it-infrastrukturen.com
http://www.it-infrastrukturen.com

Re: IOMMU support for xen in BIOS of AMD based HP DL385g7 and kernel 3.0.0-rc7

[M. Schneider]

[-- Attachment #1: Type: text/plain, Size: 2688 bytes --]

Am 25.07.2011 11:03, schrieb Roedel, Joerg:
> On Thu, Jul 21, 2011 at 11:53:50AM -0400, M. Schneider wrote:
>    
>> Am 20.07.2011 14:45, schrieb Roedel, Joerg:
>>      
>>> On Sun, Jul 17, 2011 at 06:34:33AM -0400, M. Schneider wrote:
>>>
>>>        
>>>> Please allow me one question about IOMMU support in AMDS's 890FX chipset
>>>> and 6176SE CPUs used in HP DL385g7 servers. These chips support IOMMU as
>>>> far as I know. BIOS of HP DL385g7 doesn't currently offer an IOMMU flag.
>>>>
>>>> I have contacted already HP support due to this missing BIOS setting
>>>> options but less or more my request has been ignored (no solution and no
>>>> plans for solving of this important issue  yet) so I am looking for a
>>>> workaround as IOMMU support is required for passing thoruhg of PCI
>>>> devices to domU domains (xen 4.0 / 4.1).
>>>>
>>>>          
>>> I am sorry but if there is no IVRS table there is not much we can do.
>>> The IVRS table contains the information which device uses which
>>> request-ids. Without this information the IOMMU can not work reliably.
>>>
>>>        
>> Thank you very much fo your answer Jörg.
>>
>> Some BIOSes (like BIOS of HD DL385g7 server with AMD CPUs) don't have
>> the IOMMU option.
>> I have to correct my prevoius email as HP DL385g7 servers have the AMD
>> SR5690/SP5100 chipset (supporting IOMMU).
>>
>> Is it possible to write a workaround for xen 4.1 or 4.2 to create such
>> IVRS table in code (in case it has static entries for specific server
>> type) and add a boot option like "iommu=force-dl385g7" ?
>>
>> Thank you in advance for any hints.
>>      
> This is basically possible. The Xen code needs to do what the BIOS fails
> to do. But for Xen this is a lot more difficult to get right because Xen
> can not make the same assumptions about the hardware like a BIOS can.  I
> can easily imagine that this will end in a maintenance nighmare if
> people start using this. I personally wouldn't do that for Linux, but it
> is up to the Xen IOMMU maintainer to decide that for the Xen side. I put
> Wei Wang on Cc, he maintains AMD IOMMU for Xen.	
>
> Regards,
>
> 	Joerg

Thank you very much Joerg,

Does the output of HP diagnostic contain all information required for 
creating such IVRS table? (s. attachment)

It the IVRS table has only "static" entries such boot option would 
generally solve buggy BIOS IOMMU-issues.

Thank you in advance for any hints.

Best regards,
Mark

-- 
IT-Infrastrukturen Schneider

Wieslergasse 6
CH-8049 Zürich

Phone:  +41 43 818 4508
Mobile: +41 76 522 2923

ms@it-infrastrukturen.com
http://www.it-infrastrukturen.com


[-- Attachment #2: survey-ext-all-DL385g7.html --]
[-- Type: text/html, Size: 490727 bytes --]

[-- Attachment #3: survey-ext-variousDL385g7.html --]
[-- Type: text/html, Size: 101005 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: IOMMU support for xen in BIOS of AMD based HP DL385g7 and kernel 3.0.0-rc7

[Roedel, Joerg]

On Thu, Jul 21, 2011 at 11:53:50AM -0400, M. Schneider wrote:
> Am 20.07.2011 14:45, schrieb Roedel, Joerg:
> > On Sun, Jul 17, 2011 at 06:34:33AM -0400, M. Schneider wrote:
> >    
> >> Please allow me one question about IOMMU support in AMDS's 890FX chipset
> >> and 6176SE CPUs used in HP DL385g7 servers. These chips support IOMMU as
> >> far as I know. BIOS of HP DL385g7 doesn't currently offer an IOMMU flag.
> >>
> >> I have contacted already HP support due to this missing BIOS setting
> >> options but less or more my request has been ignored (no solution and no
> >> plans for solving of this important issue  yet) so I am looking for a
> >> workaround as IOMMU support is required for passing thoruhg of PCI
> >> devices to domU domains (xen 4.0 / 4.1).
> >>      
> > I am sorry but if there is no IVRS table there is not much we can do.
> > The IVRS table contains the information which device uses which
> > request-ids. Without this information the IOMMU can not work reliably.
> >    
> 
> Thank you very much fo your answer Jörg.
> 
> Some BIOSes (like BIOS of HD DL385g7 server with AMD CPUs) don't have 
> the IOMMU option.
> I have to correct my prevoius email as HP DL385g7 servers have the AMD 
> SR5690/SP5100 chipset (supporting IOMMU).
> 
> Is it possible to write a workaround for xen 4.1 or 4.2 to create such 
> IVRS table in code (in case it has static entries for specific server 
> type) and add a boot option like "iommu=force-dl385g7" ?
> 
> Thank you in advance for any hints.

This is basically possible. The Xen code needs to do what the BIOS fails
to do. But for Xen this is a lot more difficult to get right because Xen
can not make the same assumptions about the hardware like a BIOS can.  I
can easily imagine that this will end in a maintenance nighmare if
people start using this. I personally wouldn't do that for Linux, but it
is up to the Xen IOMMU maintainer to decide that for the Xen side. I put
Wei Wang on Cc, he maintains AMD IOMMU for Xen.	

Regards,

	Joerg

-- 
AMD Operating System Research Center

Advanced Micro Devices GmbH Einsteinring 24 85609 Dornach
General Managers: Alberto Bozzo, Andrew Bowd
Registration: Dornach, Landkr. Muenchen; Registerger. Muenchen, HRB Nr. 43632

Re: IOMMU support for xen in BIOS of AMD based HP DL385g7 and kernel 3.0.0-rc7

[Wei Wang2]

On Monday 25 July 2011 11:03:35 Roedel, Joerg wrote:
> On Thu, Jul 21, 2011 at 11:53:50AM -0400, M. Schneider wrote:
> > Am 20.07.2011 14:45, schrieb Roedel, Joerg:
> > > On Sun, Jul 17, 2011 at 06:34:33AM -0400, M. Schneider wrote:
> > >> Please allow me one question about IOMMU support in AMDS's 890FX
> > >> chipset and 6176SE CPUs used in HP DL385g7 servers. These chips
> > >> support IOMMU as far as I know. BIOS of HP DL385g7 doesn't currently
> > >> offer an IOMMU flag.
> > >>
> > >> I have contacted already HP support due to this missing BIOS setting
> > >> options but less or more my request has been ignored (no solution and
> > >> no plans for solving of this important issue  yet) so I am looking for
> > >> a workaround as IOMMU support is required for passing thoruhg of PCI
> > >> devices to domU domains (xen 4.0 / 4.1).
> > >
> > > I am sorry but if there is no IVRS table there is not much we can do.
> > > The IVRS table contains the information which device uses which
> > > request-ids. Without this information the IOMMU can not work reliably.
> >
> > Thank you very much fo your answer Jörg.
> >
> > Some BIOSes (like BIOS of HD DL385g7 server with AMD CPUs) don't have
> > the IOMMU option.
> > I have to correct my prevoius email as HP DL385g7 servers have the AMD
> > SR5690/SP5100 chipset (supporting IOMMU).
> >
> > Is it possible to write a workaround for xen 4.1 or 4.2 to create such
> > IVRS table in code (in case it has static entries for specific server
> > type) and add a boot option like "iommu=force-dl385g7" ?
> >
> > Thank you in advance for any hints.
>
> This is basically possible. The Xen code needs to do what the BIOS fails
> to do. But for Xen this is a lot more difficult to get right because Xen
> can not make the same assumptions about the hardware like a BIOS can.  I
> can easily imagine that this will end in a maintenance nighmare if
> people start using this. I personally wouldn't do that for Linux, but it
> is up to the Xen IOMMU maintainer to decide that for the Xen side. I put
> Wei Wang on Cc, he maintains AMD IOMMU for Xen.

Xen also relies on BIOS to setup IVRS table for iommu device. You could also 
look at Xen 3.2, there were some old bits that enables iommu directly using 
pci configure space. But just as Joerg said, this was not an optimal and 
reliable way to enable iommu, we do not maintain these codes any more.
Thanks,
Wei

> Regards,
>
> 	Joerg