Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Dec 19, 2012 at 10:55:49AM +0000, James Dingwall wrote:
> On 2012-12-19 08:47, James Dingwall wrote:
> >Hi,
> >
> >I have encountered an apparently benign error on two systems where
> >the dom0 kernel log is flooded with messages like:
> >
> >[52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff]
> >cannot be added
> >[52482.163860] xen_balloon: reserve_additional_memory:
> >add_memory() failed: -17
> >
> >The first line is from drivers/xen/xen-balloon.c, the second from
> >mm/memory_hotplug.c
> >
> >The trigger for the messages seems to be the first occasion that a
> >Xen guest is shutdown.  I have noted this in a vanilla 3.6.7 and
> >kernel 3.5.0-18 built from Ubuntu sources.  Xen version is 4.2.0.  It
> >is not clear why the dom0 is kernel is trying to balloon up as the
> >Xen
> >command line is specifies a fixed dom0 memory allocation and
> >noselfballooning is specified for the kernel and ballooning is also
> >disabled in the xend-config.sxp / xl.conf (one system using xm,
> >another xl)
> >
> >xen command line:
> >placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1
> >dom0_mem=max:6144m
> >
> >kernel command line:
> >root=/dev/loop0 ro console=tty1 console=hvc0 earlyprintk=xen
> >nomodeset noselfballooning
> >
> >Examining /proc/iomem does show that the dom0 memory allocation is
> >actually 64kb short of 6144Mb:
> >
> >cat /proc/iomem | grep System\ RAM
> >00010000-0009bfff : System RAM      [573440 bytes]
> >00100000-cb2dffff : System RAM      [3407740928 bytes]
> >100000000-1b4d83fff : System RAM    [3034071040 bytes]
> >
> >Total system ram: 6442385408 - 6x2^30 = 65536
> >
> >The memory range indicated in the log message is "Unusable memory" in
> >/proc/iomem:
> >1b4d84000-82fffffff : Unusable memory
> >
> >Another point of interest is that we have multiple "identical"
> >hardware platforms (Dell T320) for the system running the 3.5.0-18
> >kernel but only see this error on a slightly more recent system.
> >Older systems show in /proc/iomem that all memory is System RAM.
> >
> >100000000-82fffffff : System RAM  [older system BIOS 1.0]

Wow. That is a lot of memory :-)

> >
> >100000000-1b4d83fff : System RAM  [newer system BIOS 1.3]
> >1b4d84000-82fffffff : Unusable memory
> >
> >The BIOS revision between the old and new has changed so I was
> >wondering if it is possible that there is a white list which affects
> >the impact of the kernel option:
> >CONFIG_X86_RESERVE_LOW=64
> >This is only a guess since the amount of memory reserved is
> >equivalent to the short fall calculated above.  If this is the right
> >area perhaps the dom0 calculation for its memory entitlement needs to
> >be taught to not to try and hotplug the missing 64k when it has been
> >reserved.
> >
> >If any other information would be useful then please let me know.
> 
> With some further investigation we have determined that the
> different BIOS version does not seem to be a factor and the key
> point is actually the Xen command line.  The reason that we had max:
> specified is that without it we could not boot the kernel/xen
> combination on an AMD platform.  I will do some further testing to

What happend? Did you report it in another email on this mailing list?

> see what the result of dom0_mem=6144m,max:6144m as suggested
> http://wiki.xen.org/wiki/Xen_Best_Practices gets us.
> 
> placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1
> dom0_mem=max:6144m

I think you don't need xsave=0 anymore? It was only needed with
a Fedora stock kernel (and the underlaying issue I believe is
now fixed).

> results in /proc/iomem having an unusable range and top reports
> 6083900k of memory in dom0
> 
> placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1
> dom0_mem=6144m
> no unusable range, top reports 5605976k of memory in dom0 and no log
> messages

Hmm..

If you were to run this with 'loglevel=8 debug' with the
dom0_mem=6144m and dom0_mem=max:6144m could you send both
'xl dmesg' and 'dmesg' outputs?
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>