From mboxrd@z Thu Jan 1 00:00:00 1970 From: George Dunlap Subject: Re: Xen balloon driver discuss Date: Mon, 29 Nov 2010 10:12:25 +0000 Message-ID: <4CF37C89.5080406@eu.citrix.com> References: > < <0df9d6cb-eeaf-4661-a324-ca3eba4c3376@default 003501cb8dff$ffb96c70$ff2c4550$@com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Dan Magenheimer Cc: cloudroot , tinnycloud , xen devel List-Id: xen-devel@lists.xenproject.org FYI, the balloon driver in 2.6.18 was meant to be working at some point.=20 The xen tree has some drivers which will compile for 2.6.18 externally=20 and will run in HVM mode. More modern kernels need Stefano's pv-on-hvm=20 patch series to be able to access xenstore (which is a requisite for a=20 working balloon driver). -George On 28/11/10 02:36, Dan Magenheimer wrote: > Am I understanding correctly that you are running each linux-2.6.18 as > HVM (not PV)? I didn=92t think that the linux-2.6.18 balloon driver wor= ked > at all in an HVM guest. > > You also didn=92t say what version of Xen you are using. If you are > running xen-unstable, you should also provide the changeset number. > > In any case, any load of HVM guests should never crash Xen itself, but > if you are running HVM guests, I probably can=92t help much as I almost > never run HVM guests. > > *From:* cloudroot [mailto:cloudroot@sina.com] > *Sent:* Friday, November 26, 2010 11:55 PM > *To:* tinnycloud; Dan Magenheimer; xen devel > *Cc:* george.dunlap@eu.citrix.com > *Subject:* re: Xen balloon driver discuss > > Hi Dan: > > I have set the benchmark to test balloon driver, but unfortunately the > Xen crashed on memory Panic. > > Before I attach the details output from serial port(which takes time on > next run), I am afraid of I might miss something on test environment. > > My dom0 kernel is 2.6.31, pvops. > > Well currently there is no driver/xen/balloon.c on this kernel source > tree, so I build the xen-balloon.ko, Xen-platform-pci.ko form > > linux-2.6.18.x86_64, and installed in Dom U, which is redhat 5.4. > > What I did is put a C program in the each Dom U(total 24 HVM), the > program will allocate the memory and fill it with random string repeatl= y. > > And in dom0, a phthon monitor will collect the meminfo from xenstore an= d > calculate the target to balloon from Committed_AS. > > The panic happens when the program is running in just one Dom. > > I am writing to ask whether my balloon driver is out of date, or where > can I get the latest source code, > > I=92ve googled a lot, but still have a lot of confusion on those source= tree. > > Many thanks. > > *From:* tinnycloud [mailto:tinnycloud@hotmail.com] > *Date:* 2010.11.23 22:58 > *TO:* 'Dan Magenheimer'; 'xen devel' > *CC:* 'george.dunlap@eu.citrix.com' > *Subject:* re: Xen balloon driver discuss > > HI Dan: > > Appreciate for your presentation in summarizing the memory overcommit, > really vivid and in great help. > > Well, I guess recently days the strategy in my mind will fall into the > solution Set C in pdf. > > The tmem solution your worked out for memory overcommit is both > efficient and effective. > > I guess I will have a try on Linux Guest. > > The real situation I have is most of the running VMs on host are > windows. So I had to come up those policies to balance the memory. > > Although policies are all workload dependent. Good news is host workloa= d > is configurable, and not very heavy > > So I will try to figure out some favorable policy. The policies referre= d > in pdf are good start for me. > > Today, instead of trying to implement =93/proc/meminfo=94 with shared p= ages, > I hacked the balloon driver to have another > > workqueue periodically write meminfo into xenstore through xenbus, whic= h > solve the problem of xenstrore high CPU > > utilization problem. > > Later I will try to google more on how Citrix does. > > Thanks for your help, or do you have any better idea for windows guest? > > *Sent:* Dan Magenheimer [mailto:dan.magenheimer@oracle.com] > *Date:* 2010.11.23 1:47 > *To:* MaoXiaoyun; xen devel > *CC:* george.dunlap@eu.citrix.com > *Subject:* RE: Xen balloon driver discuss > > Xenstore IS slow and you could improve xenballoond performance by only > sending the single CommittedAS value from xenballoond in domU to dom0 > instead of all of /proc/meminfo. But you are making an assumption that > getting memory utilization information from domU to dom0 FASTER (e.g. > with a shared page) will provide better ballooning results. I have not > found this to be the case, which is what led to my investigation into > self-ballooning, which led to Transcendent Memory. See the 2010 Xen > Summit for more information. > > In your last paragraph below =93Regards balloon strategy=94, the proble= m is > it is not easy to define =93enough memory=94 and =93shortage of memory=94= within > any guest and almost impossible to define it and effectively load > balance across many guests. See my Linux Plumber=92s Conference > presentation (with complete speaker notes) here: > > http://oss.oracle.com/projects/tmem/dist/documentation/presentations/Me= mMgmtVirtEnv-LPC2010-Final.pdf > > http://oss.oracle.com/projects/tmem/dist/documentation/presentations/Me= mMgmtVirtEnv-LPC2010-SpkNotes.pdf > > *From:* MaoXiaoyun [mailto:tinnycloud@hotmail.com] > *Sent:* Sunday, November 21, 2010 9:33 PM > *To:* xen devel > *Cc:* Dan Magenheimer; george.dunlap@eu.citrix.com > *Subject:* RE: Xen balloon driver discuss > > > Since currently /cpu/meminfo is sent to domain 0 via xenstore, which in > my opinoin is slow. > What I want to do is: there is a shared page between domU and dom0, and > domU periodically > update the meminfo into the page, while on the other side dom0 retrive > the updated data for > caculating the target, which is used by guest for balloning. > > The problem I met is, currently I don't know how to implement a shared > page between > dom0 and domU. > Would it like dom 0 alloc a unbound event and wait guest to connect, an= d > transfer date through > grant table? > Or someone has more efficient way? > many thanks. > >> From: tinnycloud@hotmail.com >> To: xen-devel@lists.xensource.com >> CC: dan.magenheimer@oracle.com; George.Dunlap@eu.citrix.com >> Subject: Xen balloon driver discuss >> Date: Sun, 21 Nov 2010 14:26:01 +0800 >> >> Hi: >> Greeting first. >> >> I was trying to run about 24 HVMS (currently only Linux, later will >> involve Windows) on one physical server with 24GB memory, 16CPUs. >> Each VM is configured with 2GB memory, and I reserved 8GB memory for >> dom0. >> For safety reason, only domain U's memory is allowed to balloon. >> >> Inside domain U, I used xenballooned provide by xensource, >> periodically write /proc/meminfo into xenstore in dom >> 0(/local/domain/did/memory/meminfo). >> And in domain 0, I wrote a python script to read the meminfo, like >> xen provided strategy, use Committed_AS to calculate the domain U bal= loon >> target. >> The time interval is ! 1 seconds. >> >> Inside each VM, I setup a apache server for test. Well, I'd >> like to say the result is not so good. >> It appears that too much read/write on xenstore, when I give some of >> the stress(by using ab) to guest domains, >> the CPU usage of xenstore is up to 100%. Thus the monitor running in >> dom0 also response quite slowly. >> Also, in ab test, the Committed_AS grows very fast, reach to maxmem >> in short time, but in fact the only a small amount >> of memory guest really need, so I guess there should be some more to >> be taken into consideration for ballooning. >> >> For xenstore issue, I first plan to wrote a C program inside domain >> U to replace xenballoond to see whether the situation >> will be refined. If not, how about set up event channel directly for >> domU and dom0, would it be faster? >> >> Regards balloon strategy, I would do like this, when there ! are >> enough memory , just fulfill the guest balloon request, and when shor= tage >> of memory, distribute memory evenly on the guests those request >> inflation. >> >> Does anyone have better suggestion, thanks in advance. >> >