From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anthony Wright Subject: Re: Memory allocation going seriously wonky on 4.1.1 Date: Thu, 06 Oct 2011 15:59:32 +0100 Message-ID: <4E8DC254.7060001@overnetdata.com> References: <4E85DB1A.5060606@overnetdata.com> <20111001120706.GZ12984@reaktio.net> <4E89BC80.4030105@overnetdata.com> <20111003190917.GF15608@phenom.oracle.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------000906080401070707060407" Return-path: In-Reply-To: <20111003190917.GF15608@phenom.oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------000906080401070707060407 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit On 03/10/2011 20:09, Konrad Rzeszutek Wilk wrote: > On Mon, Oct 03, 2011 at 02:45:36PM +0100, Anthony Wright wrote: >> On 01/10/2011 13:07, Pasi Kärkkäinen wrote: >>> On Fri, Sep 30, 2011 at 04:07:06PM +0100, Anthony Wright wrote: >>>> I have a machine with 16GB of RAM running 32 bit xen 4.1.1 with Dom0 >>>> running a 3.0.4 linux kernel and 23 paravirtualized DomUs. >>>> >>> Hey, >>> >>> Why 32bit Xen? I think all (most?) hypervisor testing happens with 64bit Xen. >>> Note that you can still have 32bit dom0 kernel with 64b hypervisor. >> I'm going to investigate 64 bit xen on a 32 bit Dom0 soon, but as to the >> 32 xen, I work quite a lot with older hardware and Xen since it does >> paravirtualisation, and it's a really good way to re-use old hardware. >>>> As the free ram gets to 2391 MB free the system behaves as if it's got >>>> no more ram, and starts grabbing ram from Dom0, before getting even more >>>> confused and failing to start DomUs and eventually crashing the whole >>>> machine. I have attached a tarball with the output of xl dmesg, xl info >>>> & xl list at various points. >>>> >>> - Do you use dom0_mem= parameter for Xen? >>> - Did you disable dom0 ballooning? >>> >>> http://wiki.xen.org/xenwiki/XenBestPractices >> I was already using the dom0_mem= parameter on the xen hypervisor, but I >> disabled dom0 ballooning (enable-dom0-ballooning no), and just for good >> measure I also set (dom0-min-mem 2000). This had no effect, when I >> increase the 10-4 memory requirements, on restart the Dom0 memory is >> reduced to 1713MB and xl info reports 2391MB free. >>>> Stage 1 >>>> Everything is fine, Dom0 has 2000MB according to xl list, and 'xl info' >>>> says there is 2405MB free. >>> - How much memory did dom0 initially have? >>> - Did you use mem= parameter for dom0 kernel (vmlinuz) ? >>> >>> See: http://wiki.xen.org/xenwiki/Linux_30_bugs >> dom0 starts with 2000MB. >> I didn't use mem= on the linux kernel - I thought it wasn't neccessary. > It is if your kernel does not have these two patches: > > xen: use maximum reservation to limit amount of usable RAM > xen/e820: if there is no dom0_mem=, don't tweak extra_pages. > > .. oh and you also need to update the hypervisor to include: > " > For dom0, this requires at patch to Xen ('x86: use 'dom0_mem' to limit > the number of pages for dom0') (c/s 23790) > " > > Grrr.. Let me update the 3.0 bug-list >>From reading around some of the bug reports, I thought this was a problem caused by the fact that I have 16GB of ram. As a result I reduced the amount of physical ram in the machine down to 8GB, but still got the same effect (with the two kernel patches, but not with the xen patch as I can't find it). I've attached a log of xl info & xl list as the domU that trips the machine up is started (10-5). Dom0 is supposed to have 700MB of ram (dom0_mem=500M), dom0-min-mem is 700 and enable-dom0-ballooning is no. --------------000906080401070707060407 Content-Type: text/plain; name="dom-mem.log" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="dom-mem.log" AET-Box-4:~ # date ; xl info ; xl list Thu Oct 6 14:53:19 UTC 2011 host : AET-Box-4 release : 3.0.4 version : #1 SMP Mon Oct 3 05:22:53 GMT 2011 machine : i686 nr_cpus : 16 nr_nodes : 2 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2533 hw_caps : bfebfbff:2c100000:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 8181 free_memory : 1491 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .1 xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable xen_commandline : dom0_mem=700M sched=credit cc_compiler : gcc version 4.4.3 (GCC) cc_compile_by : cc_compile_domain : [unknown] cc_compile_date : Wed Sep 21 08:25:36 GMT 2011 xend_config_format : 4 Name ID Mem VCPUs State Time(s) Domain-0 0 700 8 r----- 64.3 1-15 1 1300 8 -b---- 9.3 22-5 2 400 2 -b---- 8.9 7-5 3 400 2 -b---- 8.5 8-4 4 200 1 -b---- 3.3 16-5 5 400 2 -b---- 6.2 12-2 6 200 1 -b---- 4.3 21-2 7 200 1 -b---- 4.1 4-5 8 400 2 -b---- 6.0 15-2 9 200 1 -b---- 4.3 5-4 10 200 1 -b---- 2.7 19-5 11 400 2 -b---- 4.3 17-4 12 200 1 -b---- 2.5 13-5 13 400 2 -b---- 1.4 20-4 14 200 1 -b---- 2.8 11-4 15 200 1 -b---- 2.5 14-4 16 200 1 -b---- 1.3 23-6 17 200 1 -b---- 1.1 6-2 18 200 1 --p--- 0.0 AET-Box-4:~ # date ; xl info ; xl list Thu Oct 6 14:53:23 UTC 2011 host : AET-Box-4 release : 3.0.4 version : #1 SMP Mon Oct 3 05:22:53 GMT 2011 machine : i686 nr_cpus : 16 nr_nodes : 2 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2533 hw_caps : bfebfbff:2c100000:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 8181 free_memory : 1635 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .1 xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable xen_commandline : dom0_mem=700M sched=credit cc_compiler : gcc version 4.4.3 (GCC) cc_compile_by : cc_compile_domain : [unknown] cc_compile_date : Wed Sep 21 08:25:36 GMT 2011 xend_config_format : 4 Name ID Mem VCPUs State Time(s) Domain-0 0 555 8 r----- 65.3 1-15 1 1300 8 -b---- 9.5 22-5 2 400 2 -b---- 8.9 7-5 3 400 2 -b---- 8.5 8-4 4 200 1 -b---- 3.4 16-5 5 400 2 -b---- 6.5 12-2 6 200 1 -b---- 4.3 21-2 7 200 1 -b---- 4.2 4-5 8 400 2 -b---- 6.1 15-2 9 200 1 -b---- 4.3 5-4 10 200 1 -b---- 2.8 19-5 11 400 2 -b---- 4.8 17-4 12 200 1 -b---- 2.8 13-5 13 400 2 -b---- 1.4 20-4 14 200 1 -b---- 2.8 11-4 15 200 1 -b---- 2.6 14-4 16 200 1 -b---- 1.5 23-6 17 200 1 -b---- 1.2 6-2 18 200 1 -b---- 0.8 (null) 19 0 0 --p--- 0.0 AET-Box-4:~ # date ; xl info ; xl list Thu Oct 6 14:53:28 UTC 2011 host : AET-Box-4 release : 3.0.4 version : #1 SMP Mon Oct 3 05:22:53 GMT 2011 machine : i686 nr_cpus : 16 nr_nodes : 2 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2533 hw_caps : bfebfbff:2c100000:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 8181 free_memory : 1635 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .1 xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable xen_commandline : dom0_mem=700M sched=credit cc_compiler : gcc version 4.4.3 (GCC) cc_compile_by : cc_compile_domain : [unknown] cc_compile_date : Wed Sep 21 08:25:36 GMT 2011 xend_config_format : 4 Name ID Mem VCPUs State Time(s) Domain-0 0 555 8 r----- 66.4 1-15 1 1300 8 -b---- 9.6 22-5 2 400 2 -b---- 8.9 7-5 3 400 2 -b---- 8.5 8-4 4 200 1 -b---- 3.4 16-5 5 400 2 -b---- 7.0 12-2 6 200 1 -b---- 4.4 21-2 7 200 1 -b---- 4.2 4-5 8 400 2 -b---- 6.5 15-2 9 200 1 -b---- 4.3 5-4 10 200 1 -b---- 2.8 19-5 11 400 2 -b---- 5.5 17-4 12 200 1 -b---- 2.8 13-5 13 400 2 -b---- 1.5 20-4 14 200 1 -b---- 3.0 11-4 15 200 1 -b---- 2.7 14-4 16 200 1 -b---- 2.2 23-6 17 200 1 -b---- 2.2 6-2 18 200 1 -b---- 0.8 10-5 19 0 0 --p--- 0.0 AET-Box-4:~ # date ; xl info ; xl list Thu Oct 6 14:53:30 UTC 2011 host : AET-Box-4 release : 3.0.4 version : #1 SMP Mon Oct 3 05:22:53 GMT 2011 machine : i686 nr_cpus : 16 nr_nodes : 2 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2533 hw_caps : bfebfbff:2c100000:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 8181 free_memory : 1235 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .1 xen_caps : xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable xen_commandline : dom0_mem=700M sched=credit cc_compiler : gcc version 4.4.3 (GCC) cc_compile_by : cc_compile_domain : [unknown] cc_compile_date : Wed Sep 21 08:25:36 GMT 2011 xend_config_format : 4 Name ID Mem VCPUs State Time(s) Domain-0 0 555 8 r----- 67.2 1-15 1 1300 8 -b---- 9.7 22-5 2 400 2 -b---- 8.9 7-5 3 400 2 -b---- 8.5 8-4 4 200 1 -b---- 3.4 16-5 5 400 2 r----- 7.5 12-2 6 200 1 -b---- 4.4 21-2 7 200 1 -b---- 4.2 4-5 8 400 2 r----- 6.9 15-2 9 200 1 -b---- 4.3 5-4 10 200 1 -b---- 2.8 19-5 11 400 2 -b---- 5.7 17-4 12 200 1 -b---- 2.8 13-5 13 400 2 -b---- 1.5 20-4 14 200 1 -b---- 3.0 11-4 15 200 1 -b---- 2.8 14-4 16 200 1 -b---- 2.7 23-6 17 200 1 -b---- 2.7 6-2 18 200 1 -b---- 0.9 10-5 19 400 1 --p--- 0.0 --------------000906080401070707060407 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------000906080401070707060407--