* Re: PCI passthrough for domU allocated with more than 4G memory
@ 2012-09-03 16:25 Daniel Di Giacomo
2012-09-04 17:15 ` Pasi Kärkkäinen
0 siblings, 1 reply; 4+ messages in thread
From: Daniel Di Giacomo @ 2012-09-03 16:25 UTC (permalink / raw)
To: xen-devel
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Same problem here.
I'm using gentoo with kernel 3.5.3 (gentoo-sources) and XEN 4.1.2 as dom0.
DomU is also gentoo with the same kernel version.
When I assign more than 4G memory to this domU, the memory for the
passed-through Intel-gigabit-nic cannot be allocated and the
initialization of the device fails.
With 3968M memory assigned everything works fine.
Assigning 8GB RAM to domU works without any problems, when I use the
old kernel-version 2.6.34 (xen-sources) in domU.
Regards
Daniel
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iQIcBAEBAgAGBQJQRNn9AAoJEO8IGUBgh06fhtwQAJ0eocxHWeiEJSROM5SObIAm
V/ZulEaLOg/XmavL0KWGeNgHA3N/JtAvKMvpzLxjsGszFFD3h9ndtX0eRHnQPYC3
38tMNjIGllUzEiHl5ESCnaChrekI6/hH7NYqoMmXndbJkexUmlaJyRd6Rgy4SIhn
k6x2nwYG9gIg6wMVbQadbosYxOPKdKRosL1mglBuzK/J/c5hdqzxK4j6DfMVHonx
aXw1B1/Ga7zxwG9651bheCrDbXi8yVqadGoXjXZWhp2vQSxwh4k2QgOaTY5vgOjJ
UTkuhtlrAI3ZdyCHpLq8dd8oS6QUyQmjJdbQe5kTMufA5GGl4b//zU4pDieOqkFp
YCAPghCDk+l691Lg/UkvVgj4jgbTDsvcYtF3uM8h+dOs4L1R36WgSsuAqvwT+LjH
DWc5g7h6VsML4J5gcXsG0IvLycHvWhpiDbJD8gp0qzcXAamMRcBkAJhVal4JnwmI
2f4WMPyVEGjkRUgV22AOgNTjflxG9Rla9CbnPamYj4yVh47lsRix+hlZbrPUl3MW
V9UsKUpt+E6dCl56dq7YziZkZj9iLABG+j4rKNk93DTJMgPbwXck4BCXIrztm+s+
Y3QAh+5k0aM/Om3nUftcYrF3z8TDHWTCCXA/JRQ6G/PAo+K19pom6KpjanZQ41IC
G0t3eBJdeJpCNNvgtXiM
=QheK
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PCI passthrough for domU allocated with more than 4G memory
2012-09-03 16:25 PCI passthrough for domU allocated with more than 4G memory Daniel Di Giacomo
@ 2012-09-04 17:15 ` Pasi Kärkkäinen
0 siblings, 0 replies; 4+ messages in thread
From: Pasi Kärkkäinen @ 2012-09-04 17:15 UTC (permalink / raw)
To: Daniel Di Giacomo; +Cc: xen-devel
On Mon, Sep 03, 2012 at 06:25:37PM +0200, Daniel Di Giacomo wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Same problem here.
> I'm using gentoo with kernel 3.5.3 (gentoo-sources) and XEN 4.1.2 as dom0.
> DomU is also gentoo with the same kernel version.
> When I assign more than 4G memory to this domU, the memory for the
> passed-through Intel-gigabit-nic cannot be allocated and the
> initialization of the device fails.
> With 3968M memory assigned everything works fine.
> Assigning 8GB RAM to domU works without any problems, when I use the
> old kernel-version 2.6.34 (xen-sources) in domU.
>
I think you need Xen 4.2 and e820_host=1 option.
-- Pasi
^ permalink raw reply [flat|nested] 4+ messages in thread
* PCI passthrough for domU allocated with more than 4G memory
@ 2012-08-01 5:57 Fan, Huaxiang
2012-08-04 11:12 ` Konrad Rzeszutek Wilk
0 siblings, 1 reply; 4+ messages in thread
From: Fan, Huaxiang @ 2012-08-01 5:57 UTC (permalink / raw)
To: xen-users@lists.xen.org
Cc: Konrad Rzeszutek Wilk (konrad.wilk@oracle.com),
xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 6106 bytes --]
Hi,
I have encountered some strange problems when I was trying to PCI passthrough Broadcom 5709/5716 NICs to domUs allocated with more than 4G memory. Please see below for details.
My environment is:
Hardware Platform: DELL R210 with 2 Broadcom 5709 NICs and 2 Broadcom 5716 NICs
Xen: xen 4.2 unstable (64bits for hypervisor and 32bit for tools)
Kernel for both dom0 and domUs: xenified kernel 2.6.32.57 (32bit)
OS: CentOS 6.2 (32bit)
The general info regarding to xen can be get via below command
# xl info
host : 7.8
release : 2.6.32.57
version : #1 SMP Fri Jul 6 18:44:16 CST 2012
machine : i686
nr_cpus : 8
max_cpu_id : 31
nr_nodes : 1
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2660
hw_caps : bfebfbff:28100800:00000000:00003b40:0098e3fd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 8182
free_memory : 7046
sharing_freed_memory : 0
sharing_used_memory : 0
free_cpus : 0
xen_major : 4
xen_minor : 2
xen_extra : -unstable
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xff400000
xen_changeset : unavailable
xen_commandline : dom0_mem=1024M dom0_max_vcpus=2 dom0_vcpus_pin
cc_compiler : gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)
cc_compile_by : root
cc_compile_domain :
cc_compile_date : Thu Jul 12 11:20:56 CST 2012
xend_config_format : 4
case 1> I specified PV-domU config as below
memory=3072
maxmem=6144
name="bs"
vif=['ip=169.254.254.1,script=vif-nat',]
disk=['file:/root/bs.img,xvda1,w']
kernel='/root/vmlinuz'
extra="iommu=soft console=hvc0"
ramdisk='/root/initrd.img'
root="/dev/xvda1 ro"
pci=['01:00.0','01:00.1']
and then start domU. After that I executed below command
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 2 r----- 88.7
bs 2 3072 1 -b---- 1.1
It seemed normal to me. But when I logon bs domU, and executed below command
# cat /proc/meminfo | head
MemTotal: 6158940 kB
MemFree: 2944776 kB
Buffers: 5108 kB
Cached: 32292 kB
SwapCached: 0 kB
Active: 21456 kB
Inactive: 22936 kB
Active(anon): 7000 kB
Inactive(anon): 108 kB
Active(file): 14456 kB
It indicated the total memory was 6G, why?
When I back to dom0, I executed below command
# xl mem-set bs 6144
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 1024 2 r----- 93.5
bs 2 6144 1 -b---- 10.5
It seemed normal to me. But when I logon bs domU again and executed below command
# cat /proc/meminfo | head
MemTotal: 9304668 kB
MemFree: 6087540 kB
Buffers: 5168 kB
Cached: 32464 kB
SwapCached: 0 kB
Active: 22300 kB
Inactive: 22408 kB
Active(anon): 7080 kB
Inactive(anon): 108 kB
Active(file): 15220 kB
It indicated total memory was 9G (6G + 3G). It was wired. Any idea about this?
Case 2> I specified PV-domU config as below
memory=6144
maxmem=6144
name="bs"
vif=['ip=169.254.254.1,script=vif-nat',]
disk=['file:/root/bs.img,xvda1,w']
kernel='/root/vmlinuz'
extra="iommu=soft console=hvc0"
ramdisk='/root/initrd.img'
root="/dev/xvda1 ro"
pci=['01:00.0','01:00.1']
and then start domU. After that I executed below command
# xl list
Name ID Mem VCPUs State Time(s)
Domain-0 0 648 2 r----- 120.5
bs 3 3360 1 -b---- 7.0
the output was very confusing. Why dom0 memory had been shrank to 648M and only 3360M assigned to bs domU?
My own analysis:
I extracted the bios e820 memory map on bs domU as below
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
[ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
[ 0.000000] Xen: 0000000000100000 - 00000000bf699000 (usable)
[ 0.000000] Xen: 00000000bf699000 - 00000000bf6af000 (reserved)
[ 0.000000] Xen: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
[ 0.000000] Xen: 00000000bf6ce000 - 00000000c0000000 (reserved)
[ 0.000000] Xen: 00000000e0000000 - 00000000f0000000 (reserved)
[ 0.000000] Xen: 00000000fe000000 - 0000000100000000 (reserved)
[ 0.000000] Xen: 0000000180000000 - 00000001c33ec000 (usable)
I think the root cause might be related to the holes between c0000000 and e0000000 and between f0000000 and fe000000 and between 100000000 and 180000000. And I think the e820_host option set according to my tracking.
Thanks in advance
HUAXIANG FAN
Software Engineer II
WEBSENSE NETWORK SECURITY TECHNOLOGY R&D (BEIJING) CO. LTD.
ph: +8610.5884.4327
fax: +8610.5884.4727
www.websense.cn<http://www.websense.cn>
Websense TRITON(tm)
For Essential Information Protection(tm)
Web Security<http://www.websense.com/content/Regional/SCH/WebSecurityOverview.aspx> | Data Security<http://www.websense.com/content/Regional/SCH/DataSecurity.aspx> | Email Security<http://www.websense.com/content/Regional/SCH/MessagingSecurity.aspx>
Protected by Websense Hosted Email Security -- www.websense.com
[-- Attachment #1.2: Type: text/html, Size: 19602 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: PCI passthrough for domU allocated with more than 4G memory
2012-08-01 5:57 Fan, Huaxiang
@ 2012-08-04 11:12 ` Konrad Rzeszutek Wilk
0 siblings, 0 replies; 4+ messages in thread
From: Konrad Rzeszutek Wilk @ 2012-08-04 11:12 UTC (permalink / raw)
To: Fan, Huaxiang
Cc: xen-users@lists.xen.org, xen-devel@lists.xen.org,
Konrad Rzeszutek Wilk (konrad.wilk@oracle.com)
On Wed, Aug 01, 2012 at 05:57:13AM +0000, Fan, Huaxiang wrote:
> Hi,
>
> I have encountered some strange problems when I was trying to PCI passthrough Broadcom 5709/5716 NICs to domUs allocated with more than 4G memory. Please see below for details.
>
> My environment is:
> Hardware Platform: DELL R210 with 2 Broadcom 5709 NICs and 2 Broadcom 5716 NICs
> Xen: xen 4.2 unstable (64bits for hypervisor and 32bit for tools)
> Kernel for both dom0 and domUs: xenified kernel 2.6.32.57 (32bit)
> OS: CentOS 6.2 (32bit)
>
> The general info regarding to xen can be get via below command
>
> # xl info
>
> host : 7.8
>
> release : 2.6.32.57
>
> version : #1 SMP Fri Jul 6 18:44:16 CST 2012
>
> machine : i686
>
> nr_cpus : 8
>
> max_cpu_id : 31
>
> nr_nodes : 1
>
> cores_per_socket : 4
>
> threads_per_core : 2
>
> cpu_mhz : 2660
>
> hw_caps : bfebfbff:28100800:00000000:00003b40:0098e3fd:00000000:00000001:00000000
>
> virt_caps : hvm hvm_directio
>
> total_memory : 8182
>
> free_memory : 7046
>
> sharing_freed_memory : 0
>
> sharing_used_memory : 0
>
> free_cpus : 0
>
> xen_major : 4
>
> xen_minor : 2
>
> xen_extra : -unstable
>
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
>
> xen_scheduler : credit
>
> xen_pagesize : 4096
>
> platform_params : virt_start=0xff400000
>
> xen_changeset : unavailable
>
> xen_commandline : dom0_mem=1024M dom0_max_vcpus=2 dom0_vcpus_pin
>
> cc_compiler : gcc version 4.4.6 20110731 (Red Hat 4.4.6-3) (GCC)
>
> cc_compile_by : root
>
> cc_compile_domain :
>
> cc_compile_date : Thu Jul 12 11:20:56 CST 2012
>
> xend_config_format : 4
>
> case 1> I specified PV-domU config as below
>
> memory=3072
>
> maxmem=6144
>
> name="bs"
>
> vif=['ip=169.254.254.1,script=vif-nat',]
>
> disk=['file:/root/bs.img,xvda1,w']
>
> kernel='/root/vmlinuz'
>
> extra="iommu=soft console=hvc0"
>
> ramdisk='/root/initrd.img'
>
> root="/dev/xvda1 ro"
>
> pci=['01:00.0','01:00.1']
>
> and then start domU. After that I executed below command
>
> # xl list
>
> Name ID Mem VCPUs State Time(s)
>
> Domain-0 0 1024 2 r----- 88.7
>
> bs 2 3072 1 -b---- 1.1
>
> It seemed normal to me. But when I logon bs domU, and executed below command
>
> # cat /proc/meminfo | head
>
> MemTotal: 6158940 kB
>
> MemFree: 2944776 kB
So that is the right amount. It has around 3GB of free kernel space.
>
> Buffers: 5108 kB
>
> Cached: 32292 kB
>
> SwapCached: 0 kB
>
> Active: 21456 kB
>
> Inactive: 22936 kB
>
> Active(anon): 7000 kB
>
> Inactive(anon): 108 kB
>
> Active(file): 14456 kB
>
> It indicated the total memory was 6G, why?
I wish you also included the DirectMap number. Irregardless of
what /proc/meminfo says, what did your dmesg say in 'Memory' section?
> When I back to dom0, I executed below command
>
> # xl mem-set bs 6144
>
> # xl list
>
> Name ID Mem VCPUs State Time(s)
>
> Domain-0 0 1024 2 r----- 93.5
>
> bs 2 6144 1 -b---- 10.5
> It seemed normal to me. But when I logon bs domU again and executed below command
>
> # cat /proc/meminfo | head
>
> MemTotal: 9304668 kB
>
> MemFree: 6087540 kB
So that is right. 6GB of free space.
>
> Buffers: 5168 kB
>
> Cached: 32464 kB
>
> SwapCached: 0 kB
>
> Active: 22300 kB
>
> Inactive: 22408 kB
>
> Active(anon): 7080 kB
>
> Inactive(anon): 108 kB
>
> Active(file): 15220 kB
>
>
>
> It indicated total memory was 9G (6G + 3G). It was wired. Any idea about this?
Presumarily b/c '3G' of it is the E820_UNUSUABLE or the big gap in the
E820. But irregardless of that - do you have 6GB of ram in your guest?
> Case 2> I specified PV-domU config as below
>
> memory=6144
>
> maxmem=6144
>
> name="bs"
>
> vif=['ip=169.254.254.1,script=vif-nat',]
>
> disk=['file:/root/bs.img,xvda1,w']
>
> kernel='/root/vmlinuz'
>
> extra="iommu=soft console=hvc0"
>
> ramdisk='/root/initrd.img'
>
> root="/dev/xvda1 ro"
>
> pci=['01:00.0','01:00.1']
>
> and then start domU. After that I executed below command
>
> # xl list
>
> Name ID Mem VCPUs State Time(s)
>
> Domain-0 0 648 2 r----- 120.5
>
> bs 3 3360 1 -b---- 7.0
>
> the output was very confusing. Why dom0 memory had been shrank to 648M and only 3360M assigned to bs domU?
I think you are seeing a bug in the xl with the autoballoon. Smoebody
mentioned this on the xen-devel.
>
> My own analysis:
> I extracted the bios e820 memory map on bs domU as below
>
> [ 0.000000] BIOS-provided physical RAM map:
>
> [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable)
>
> [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved)
>
> [ 0.000000] Xen: 0000000000100000 - 00000000bf699000 (usable)
>
> [ 0.000000] Xen: 00000000bf699000 - 00000000bf6af000 (reserved)
>
> [ 0.000000] Xen: 00000000bf6af000 - 00000000bf6ce000 (ACPI data)
>
> [ 0.000000] Xen: 00000000bf6ce000 - 00000000c0000000 (reserved)
>
> [ 0.000000] Xen: 00000000e0000000 - 00000000f0000000 (reserved)
>
> [ 0.000000] Xen: 00000000fe000000 - 0000000100000000 (reserved)
>
> [ 0.000000] Xen: 0000000180000000 - 00000001c33ec000 (usable)
>
> I think the root cause might be related to the holes between c0000000 and e0000000 and between f0000000 and fe000000 and between 100000000 and 180000000. And I think the e820_host option set according to my tracking.
Not sure what you mean by 'my tracking'.
The holes are there to provide the PCI subsystem the space to stick the
MMIO BARs of your PCI device. The memory that was "taken" out of those
areas is then appended to the next E820_RAM region. It should have
gone to 0x1000000, not to 0x18000000 though.
Ohh, you are using the ancient 2.6.32 domU! Well, that is ancient -
and it probably has bugs. Have you considered using something more
recent?
>
> Thanks in advance
> HUAXIANG FAN
> Software Engineer II
>
> WEBSENSE NETWORK SECURITY TECHNOLOGY R&D (BEIJING) CO. LTD.
> ph: +8610.5884.4327
> fax: +8610.5884.4727
> www.websense.cn<http://www.websense.cn>
>
> Websense TRITON(tm)
> For Essential Information Protection(tm)
> Web Security<http://www.websense.com/content/Regional/SCH/WebSecurityOverview.aspx> | Data Security<http://www.websense.com/content/Regional/SCH/DataSecurity.aspx> | Email Security<http://www.websense.com/content/Regional/SCH/MessagingSecurity.aspx>
>
>
>
> Protected by Websense Hosted Email Security -- www.websense.com
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-09-04 17:15 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-03 16:25 PCI passthrough for domU allocated with more than 4G memory Daniel Di Giacomo
2012-09-04 17:15 ` Pasi Kärkkäinen
-- strict thread matches above, loose matches on Subject: below --
2012-08-01 5:57 Fan, Huaxiang
2012-08-04 11:12 ` Konrad Rzeszutek Wilk
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.