[Qemu-devel] Re: request : qemu-smp as target

[Qemu-devel] Re: request : qemu-smp as target

[Blue Swirl]

Hi,

The architecture used in sparc target (sun4m) supports SMP up to a maximum 
of 16 CPUs. At hardware emulation level (hw/*, target-sparc/*), it would be 
easy to add the missing interprocessor interrupts, per-CPU counters and 
atomic instructions. It would also be simple to add the prom functions for 
starting/stopping CPUs to Proll. Maybe some days' work in total.

Higher level (vl.c, cpu-exec.c) could need more work. Maybe Fabrice can 
enlighten us?

For some reason, Sparc performance is low (1/10 of native x86 nbench) 
compared to x86 (2/3). Simulating SMP on a uniprocessor would only decrease 
performance.

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar - get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

Re: [Qemu-devel] Re: request : qemu-smp as target

[Paul Brook]

On Saturday 14 May 2005 10:37, Blue Swirl wrote:
> Hi,
>
> The architecture used in sparc target (sun4m) supports SMP up to a maximum
> of 16 CPUs. At hardware emulation level (hw/*, target-sparc/*), it would be
> easy to add the missing interprocessor interrupts, per-CPU counters and
> atomic instructions. It would also be simple to add the prom functions for
> starting/stopping CPUs to Proll. Maybe some days' work in total.
>
> Higher level (vl.c, cpu-exec.c) could need more work. Maybe Fabrice can
> enlighten us?

I guess you'd really want to simulate multiple CPUs with multiple host 
threads. One of the additional problems could then be memory/cache coherency. 
I'm not sure how much of a problem this would be in practice. If both host 
and guest require the same (or no) explicit SMP memory barriert it's not a 
problem. It the guest has stronger coherency requirements than the host we 
have a problem.

> For some reason, Sparc performance is low (1/10 of native x86 nbench)
> compared to x86 (2/3). Simulating SMP on a uniprocessor would only decrease
> performance.

It think x86-on-x86 user-mode uses code-copying by default. ie. it runs a lot 
of the the code unmodified. In my experience i386-softmmu is generally 10-15x 
slower than native, and arm-user is 5-10x slower.

Paul

[Qemu-devel] Re: request : qemu-smp as target

[Fabrice Bellard]

Blue Swirl wrote:
> Hi,
> 
> The architecture used in sparc target (sun4m) supports SMP up to a 
> maximum of 16 CPUs. At hardware emulation level (hw/*, target-sparc/*), 
> it would be easy to add the missing interprocessor interrupts, per-CPU 
> counters and atomic instructions. It would also be simple to add the 
> prom functions for starting/stopping CPUs to Proll. Maybe some days' 
> work in total.
> 
> Higher level (vl.c, cpu-exec.c) could need more work. Maybe Fabrice can 
> enlighten us?

SMP est definitely possible in QEMU - a few days of work are necessary 
to add the missing generic support and an x86 implementation... but 
currently I prefer to work an other topics.

Just for your information, some choices need to be made:

1) Do the CPUs share the same translation cache ?

2) The first implementation would use a cycle counter to schedule 
between CPUs. Is it interesting to go further and to use a host thread 
for each guest CPU at the expense of more locking overhead ?

Fabrice.

Re: [Qemu-devel] Re: request : qemu-smp as target

[Jonas Maebe]

On 14 May 2005, at 14:16, Fabrice Bellard wrote:

> 1) Do the CPUs share the same translation cache ?
>
> 2) The first implementation would use a cycle counter to schedule  
> between CPUs. Is it interesting to go further and to use a host  
> thread for each guest CPU at the expense of more locking overhead ?

These two choices are closely related: if you use a host thread for  
each guest cpu, you almost need different translation caches, because  
otherwise it becomes very difficult to flush the translation cache.


Jonas

[Qemu-devel] Re: request : qemu-smp as target

[Blue Swirl]

>SMP est definitely possible in QEMU - a few days of work are necessary to 
>add the missing generic support and an x86 implementation... but currently 
>I prefer to work an other topics.
>
>Just for your information, some choices need to be made:
>
>1) Do the CPUs share the same translation cache ?

This could be very useful, but wouldn't the cache need to be indexed by 
physical addresses, not virtual?

>2) The first implementation would use a cycle counter to schedule between 
>CPUs. Is it interesting to go further and to use a host thread for each 
>guest CPU at the expense of more locking overhead ?

I'd skip the cycle counter at first iteration and schedule using a host 
timer and CPU idling. What are the things that need locking, by the way?

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: [Qemu-devel] Re: request : qemu-smp as target

[Blue Swirl]

>I guess you'd really want to simulate multiple CPUs with multiple host
>threads. One of the additional problems could then be memory/cache 
>coherency.
>I'm not sure how much of a problem this would be in practice. If both host
>and guest require the same (or no) explicit SMP memory barriert it's not a
>problem. It the guest has stronger coherency requirements than the host we
>have a problem.

Sparc32 architecure requires flushes and atomic instructions in critical 
code in Total Store Ordering mode. There is also higher performance mode 
requiring memory barriers called Partial Store Ordering, but I think Linux 
doesn't enable it.

> > For some reason, Sparc performance is low (1/10 of native x86 nbench)
> > compared to x86 (2/3). Simulating SMP on a uniprocessor would only 
>decrease
> > performance.
>
>It think x86-on-x86 user-mode uses code-copying by default. ie. it runs a 
>lot
>of the the code unmodified. In my experience i386-softmmu is generally 
>10-15x
>slower than native, and arm-user is 5-10x slower.

Good point. With -no-code-copy I get about 1/6 of native nbench in x86-x86, 
comparable to Sparc figure.

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: [Qemu-devel] Re: request : qemu-smp as target

[Joe Batt]

On Sat, 2005-05-14 at 14:16 +0200, Fabrice Bellard wrote:
...
> 2) The first implementation would use a cycle counter to schedule 
> between CPUs. Is it interesting to go further and to use a host thread 
> for each guest CPU at the expense of more locking overhead ?

What inter processor synchronization issues are there?  Could you take
this a step further and use processes on different machines for each
processor? (There are many shared memory implementations to choose
from.)  I have ignorantly implemented an SH2 emulator, but have zero
understanding of an SMP system.  Are there so many resources shared
between the CPUs to make this a ridiculous proposition?

It could make for a interesting distributed single image system.

-- 
Joe Batt <Joe@soliddesign.net>

[Qemu-devel] Re: request : qemu-smp as target

[octane indice]

Quoting Fabrice Bellard <fabrice@bellard.org> : 
> SMP est definitely possible in QEMU - a few days of work are
> necessary to add the missing generic support and an x86
> implementation...

ok.

> but  currently I prefer to work an other topics.
> 
ok. 
So, perhaps in the next releases?


---------------------------------------------
Protect your mails from viruses thanks to Alinto Premium services http://www.alinto.com

Re: [Qemu-devel] Re: request : qemu-smp as target

[Paul Brook]

> What inter processor synchronization issues are there?  Could you take
> this a step further and use processes on different machines for each
> processor? (There are many shared memory implementations to choose
> from.)  Are there so many resources shared  
> between the CPUs to make this a ridiculous proposition?

Baically most SMP/shared memory systems assume very low latency communication 
between CPUs and memory. For example on opteron systems remote memory latency 
is of the order of 200 cpu cycles. Typical ethernet latency is several 
million cycles.

To do single-system image over a high latency interconnect (eg. ethernet) you 
need OS and preferably use application support to avoid high-contention 
memory areas. Simulating a SMP system over multiple separate nodes is 
theoretically possible, but performance would probaby be abysmal.

The only solution I can imagine being even vaguely worthwhile is a running 
user-mode qemu on top of a native openmozix system.

> I have ignorantly implemented an SH2 emulator, 

Cool. Any chance you're going to make these changes publicly available?

Paul

Re: [Qemu-devel] Re: request : qemu-smp as target

[Joe Batt]

On Tue, 2005-05-17 at 21:21 +0100, Paul Brook wrote:
> > What inter processor synchronization issues are there?  Could you take
> > this a step further and use processes on different machines for each
> > processor? (There are many shared memory implementations to choose
> > from.)  Are there so many resources shared  
> > between the CPUs to make this a ridiculous proposition?
> 
> Baically most SMP/shared memory systems assume very low latency communication 
> between CPUs and memory. For example on opteron systems remote memory latency 
> is of the order of 200 cpu cycles. Typical ethernet latency is several 
> million cycles.

But how often will the virtual CPUs need the same page and is there any
other shared resource other than memory?  I don't know how independent
each CPU is.  Though in side discussions, everyone agrees with you, I
haven't seen numbers to convince my gut.  If page only needs to be
faulted back and forth every couple million cycles, then it might work.

> The only solution I can imagine being even vaguely worthwhile is a running 
> user-mode qemu on top of a native openmozix system.
OpenMosix is very interesting, but is a pain to setup.  How about this:

  ssh -f host1 qemu -cpu-server $KEY
  ssh -f host2 qemu -cpu-server $KEY
  qemu -cpu-client host1:$KEY \
       -cpu-client host2:$KEY \
       -hda server.image

> > I have ignorantly implemented an SH2 emulator, 
> 
> Cool. Any chance you're going to make these changes publicly available?
It was a Java implementation for a customer.  Not my property and not
integrated with any free software.

> Paul
-- 
Joe Batt <Joe@soliddesign.net>

Re: [Qemu-devel] Re: request : qemu-smp as target

[Paul Brook]

On Tuesday 17 May 2005 21:41, Joe Batt wrote:
> On Tue, 2005-05-17 at 21:21 +0100, Paul Brook wrote:
> > > What inter processor synchronization issues are there?  Could you take
> > > this a step further and use processes on different machines for each
> > > processor? (There are many shared memory implementations to choose
> > > from.)  Are there so many resources shared
> > > between the CPUs to make this a ridiculous proposition?
> >
> > Baically most SMP/shared memory systems assume very low latency
> > communication between CPUs and memory. For example on opteron systems
> > remote memory latency is of the order of 200 cpu cycles. Typical ethernet
> > latency is several million cycles.
>
> But how often will the virtual CPUs need the same page and is there any
> other shared resource other than memory?  I don't know how independent
> each CPU is.  Though in side discussions, everyone agrees with you, I
> haven't seen numbers to convince my gut.  If page only needs to be
> faulted back and forth every couple million cycles, then it might work.

Everything but the cpu (and possibly the APIC) is shared. This is why big SMP 
systems cost orders of magnitude more than a similar size cluster.

One of the biggest sources of contention is probably going to be kernel 
spinlocks. Every time the kernel needs to acquire a spinlock that page will 
need to be faulted across to that CPU.

> > Cool. Any chance you're going to make these changes publicly available?
>
> It was a Java implementation for a customer.  Not my property and not
> integrated with any free software.

Oh! I thought you'd implemented SH support for qemu :-)

Paul

Re: [Qemu-devel] Re: request : qemu-smp as target

[Mark Williamson]

> The only solution I can imagine being even vaguely worthwhile is a running
> user-mode qemu on top of a native openmozix system.

Probably if you want to run a distributed SMP-style sytem using QEmu, the most 
effective approach is going to be running OpenMosix *in* QEmu, on multiple 
hosts.

Sadly, this is much less transparent than QEmu / the host doing the 
distribution for you :-( but I can probably think of situations where it 
might be useful... (how about running a virtual OM cluster on a bunch of 
mostly-idle Windows boxes?)

Cheers,
Mark

>
> > I have ignorantly implemented an SH2 emulator,
>
> Cool. Any chance you're going to make these changes publicly available?
>
> Paul
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

Re: [Qemu-devel] Re: request : qemu-smp as target

[Mark Williamson]

> But how often will the virtual CPUs need the same page and is there any
> other shared resource other than memory?  I don't know how independent
> each CPU is.  Though in side discussions, everyone agrees with you, I
> haven't seen numbers to convince my gut.  If page only needs to be
> faulted back and forth every couple million cycles, then it might work.

In the applications, probably very independent.  In the kernel, highly 
dependent: different CPUs may access shared data structures *and* protect 
them with spinlocks.  As Paul said in a separate mail, spinlocks are going to 
be way more expensive in this sort of distributed environment.

All that being said, a company called "Virtual Iron" has got a 
fully-virtualising solution that presents an SMP to the guest OS but actually 
distributes computation across a cluster.  I have yet to see the product 
itself - no idea when it'll be released.  It also sounds *really* difficult 
to make go fast but at least suggests this sort of thing can perform 
reasonably for some workloads.

Cheers,
Mark

> > The only solution I can imagine being even vaguely worthwhile is a
> > running user-mode qemu on top of a native openmozix system.
>
> OpenMosix is very interesting, but is a pain to setup.  How about this:
>
>   ssh -f host1 qemu -cpu-server $KEY
>   ssh -f host2 qemu -cpu-server $KEY
>   qemu -cpu-client host1:$KEY \
>        -cpu-client host2:$KEY \
>        -hda server.image
>
> > > I have ignorantly implemented an SH2 emulator,
> >
> > Cool. Any chance you're going to make these changes publicly available?
>
> It was a Java implementation for a customer.  Not my property and not
> integrated with any free software.
>
> > Paul

Re[2]: [Qemu-devel] Re: request : qemu-smp as target

[Igor Shmukler]

Hi,

I mostly agree with everything said, but I'd like to add some thoughts tp the pot.

I think it's important to understand that that a product that coverts clusters to 
virtual MP could be designed with different requirements in mind.

We are working on a research project that represents cluster as a NUMA machine. This 
is enough for a NUMA aware OS and performace is not bad.

If we were to make a regular SMP, it would probably not work as well.

I am not a Mosix guy, but I would think that installing QEMU on OM will solve 
nothing. It will only shift problems to be addressed at a different layer. Maybe I 
am wrong

Sincerely,

Igor.

> In the applications, probably very independent.  In the kernel, highly 
> dependent: different CPUs may access shared data structures *and* protect 
> them with spinlocks.  As Paul said in a separate mail, spinlocks are going to 
> be way more expensive in this sort of distributed environment.
> 
> All that being said, a company called "Virtual Iron" has got a 
> fully-virtualising solution that presents an SMP to the guest OS but actually 
> distributes computation across a cluster.  I have yet to see the product 
> itself - no idea when it'll be released.  It also sounds *really* difficult 
> to make go fast but at least suggests this sort of thing can perform 
> reasonably for some workloads.
> 
> Cheers,
> Mark