[Qemu-devel] Running a large number of qemu instances

[Qemu-devel] Running a large number of qemu instances

[Michael Sallaway]

Hi,

I've posted the below (edited) question to the qemu forum, but I'm not 
sure if that's more a support forum -- and I figured this question is 
about half-half. :-)  But if this is incorrect, please let me know! :-)

(even if it's just pointing me at the location in the qemu source where 
the scheduling/cpu time allocation stuff is done, any help would be great!)

---

I'm trying to use qemu to emulate a large number of networking devices 
that we have, for testing purposes. The idea is to get as many as 
possible going (hopefully >1000), but at the moment I'm having trouble 
even getting past 100.

It's running on a dell 1950 (basically, 8 cores, and 16GB of ram), 
running ubuntu 8.04 (64-bit). Getting them all up and running is fine, 
and I can have 100 machines going pretty easily (system load average of 
about 4 or 5), with all the boxes nicely responsive: but as soon as it 
gets to 150 or 200 or so, the machine just slows to a crawl, load 
average hits 100+, and the machines are extremely unresponsive.

 From the look of it, I would gather that it's because the server just 
spends so much time context switching. The virtual devices aren't under 
any load at all (they're running a custom linux build (for an x86 
target), so they're mostly idling, doing nothing. What I'm wondering, is 
if there's any way to configure either qemu or the server so that it 
doesn't do as much context switching, or so that each virtual machine 
doesn't need as large a cpu-time slice as it currently gets. I figured 
most of the CPU time (at the moment) is spent running qemu, which runs 
the virtual device, which sits in a linux kernel, busy looping waiting 
for something to happen. At a guess, I'd say I could either (a) reduce 
the amount of CPU time needed/requested by qemu (is that done by qemu, 
or by the host machine's scheduler?), or (b) have the guest virtual 
machine "return" more quickly, if that's even possible: I think I might 
be saying the same thing, in a roundabout way. :-) I've had a look at 
the qemu source and haven't had any luck figuring out where I might 
start looking for something like that.... thoughts? ideas?

Ideally, I'd like to have hundreds and hundreds (even 1000?) virtual 
machines, that are all idling, doing not very much at all, except for 
the one time every 5 or 10 minutes when it needs to send a few packets 
back or forward.

Thoughts? Ideas? Comments?

Thanks for your time!

Cheers,
Michael

Re: [Qemu-devel] Running a large number of qemu instances

[Glauber Costa]

On Mon, Sep 29, 2008 at 5:12 AM, Michael Sallaway
<michael-mailinglist@sallaway.com> wrote:
> Hi,
>
> I've posted the below (edited) question to the qemu forum, but I'm not sure
> if that's more a support forum -- and I figured this question is about
> half-half. :-)  But if this is incorrect, please let me know! :-)
>
> (even if it's just pointing me at the location in the qemu source where the
> scheduling/cpu time allocation stuff is done, any help would be great!)
>
> ---
>
> I'm trying to use qemu to emulate a large number of networking devices that
> we have, for testing purposes. The idea is to get as many as possible going
> (hopefully >1000), but at the moment I'm having trouble even getting past
> 100.
>
> It's running on a dell 1950 (basically, 8 cores, and 16GB of ram), running
> ubuntu 8.04 (64-bit). Getting them all up and running is fine, and I can
> have 100 machines going pretty easily (system load average of about 4 or 5),
> with all the boxes nicely responsive: but as soon as it gets to 150 or 200
> or so, the machine just slows to a crawl, load average hits 100+, and the
> machines are extremely unresponsive.
>
> From the look of it, I would gather that it's because the server just spends
> so much time context switching. The virtual devices aren't under any load at
> all (they're running a custom linux build (for an x86 target), so they're
> mostly idling, doing nothing. What I'm wondering, is if there's any way to
> configure either qemu or the server so that it doesn't do as much context
> switching, or so that each virtual machine doesn't need as large a cpu-time
> slice as it currently gets. I figured most of the CPU time (at the moment)
> is spent running qemu, which runs the virtual device, which sits in a linux
> kernel, busy looping waiting for something to happen. At a guess, I'd say I
> could either (a) reduce the amount of CPU time needed/requested by qemu (is
> that done by qemu, or by the host machine's scheduler?), or (b) have the
> guest virtual machine "return" more quickly, if that's even possible: I
> think I might be saying the same thing, in a roundabout way. :-) I've had a
> look at the qemu source and haven't had any luck figuring out where I might
> start looking for something like that.... thoughts? ideas?
>
> Ideally, I'd like to have hundreds and hundreds (even 1000?) virtual
> machines, that are all idling, doing not very much at all, except for the
> one time every 5 or 10 minutes when it needs to send a few packets back or
> forward.
As a quick hunch, I believe they are probably doing so much work
because of the delivery of timer interrupts, Not so much to do with
context switches.
as soon as we have hpet emulation, it should help (at least for
dyntick capable guests)

>
> Thoughts? Ideas? Comments?
>
> Thanks for your time!
>
> Cheers,
> Michael
>
>
>



-- 
Glauber Costa.
"Free as in Freedom"
http://glommer.net

"The less confident you are, the more serious you have to act."