plan 9 status

plan 9 status

[Ronald G. Minnich]

well, well. Plan 9 is up and running and mounting a root file system over 
9p to a linux machine running a 9p server (Russ cox's u9fs). 

I've still got a glitch in there, but boy this is fun. 

It's a memory corruption, doubtless some other mistake on my part :-)

I'm off on other stuff this week but hope to nail it next week.

ron


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Mark A. Williamson]

> well, well. Plan 9 is up and running and mounting a root file system over
> 9p to a linux machine running a 9p server (Russ cox's u9fs).

Cool!  I for one can't wait to try out Plan 9 on my hardware.

I'm curious as to how your source for this port will be distributed...  In 
fact, I don't know how the p9 source is managed - is it possible for you to 
get the port checked in to the main tree, or will it be available as a 
separate patch?

> I'm off on other stuff this week but hope to nail it next week.

Cool!  We're really close to Xen 2.0 release and it's doubly nice to have new 
OS ports working as well :-)

Cheers,
Mark


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Ronald G. Minnich]

On Wed, 13 Oct 2004, Mark A. Williamson wrote:

> I'm curious as to how your source for this port will be distributed...  
> In fact, I don't know how the p9 source is managed - is it possible for
> you to get the port checked in to the main tree, or will it be available
> as a separate patch?

I have been told I can get it in. There will be issues as the Xen stuff is 
-- I guess -- GPL and Plna 9 is not, so we're going to get into that 
damned mess.

So I don't know how it will go.

ron


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Kip Macy]

>
> I have been told I can get it in. There will be issues as the Xen stuff is
> -- I guess -- GPL and Plna 9 is not, so we're going to get into that
> damned mess.

So long as the source is not intermingled it shouldn't be an issue.


			-Kip


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Keir Fraser]

> 
> 
> On Wed, 13 Oct 2004, Mark A. Williamson wrote:
> 
> > I'm curious as to how your source for this port will be distributed...  
> > In fact, I don't know how the p9 source is managed - is it possible for
> > you to get the port checked in to the main tree, or will it be available
> > as a separate patch?
> 
> I have been told I can get it in. There will be issues as the Xen stuff is 
> -- I guess -- GPL and Plna 9 is not, so we're going to get into that 
> damned mess.
> 
> So I don't know how it will go.

Personally, I'm happy to relicense our 'unprivileged guest' drivers
under a BSD-style license. This would include at least the following
files that contain no Linux code:

linux-2.6.8.1-xen-sparse:
 drivers/xen/netfront/*
 drivers/xen/blkfront/*
 drivers/xen/evtchn/*
 drivers/console/*
 arch/xen/i386/mm/hypervisor.c
 arch/xen/kernel/ctrl_if.c
 arch/xen/kernel/evtchn.c
 include/asm-xen/ctrl_if.h
 include/asm-xen/evtchn.h
 include/asm-xen/hypervisor.h
 include/asm-xen/multicall.h
 include/asm-xen/suspend.h

I consider this the least we can do given the lack of documentation
that would allow OS porters to roll their own. :-)

There is the issue that there are bits and pieces of other people's
code in there as well --- e.g., Christian Limpach, Mike Wray, Mark
Williamson. But I don't foresee any problems.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Wesley Parish]

I would've thought a (legally) efficient way of doing that would be to have a 
library-set available under the LGPL, that call the GPL code and are called 
by the non-GPLed guest OSes.

I'm not sure about the actual efficiency in terms of processor cycles, of such 
an approach, but if it collocates the lawyers in the dogbox, which is where 
they should be, instead of breathing down your neck, that should be an 
acceptable trade.

Just my $0.02 (and it's very highly inflated of course!)

Wesley Parish

On Thu, 14 Oct 2004 21:10, Keir Fraser wrote:
> > On Wed, 13 Oct 2004, Mark A. Williamson wrote:
> > > I'm curious as to how your source for this port will be distributed...
> > > In fact, I don't know how the p9 source is managed - is it possible for
> > > you to get the port checked in to the main tree, or will it be
> > > available as a separate patch?
> >
> > I have been told I can get it in. There will be issues as the Xen stuff
> > is -- I guess -- GPL and Plna 9 is not, so we're going to get into that
> > damned mess.
> >
> > So I don't know how it will go.
>
> Personally, I'm happy to relicense our 'unprivileged guest' drivers
> under a BSD-style license. This would include at least the following
> files that contain no Linux code:
>
> linux-2.6.8.1-xen-sparse:
>  drivers/xen/netfront/*
>  drivers/xen/blkfront/*
>  drivers/xen/evtchn/*
>  drivers/console/*
>  arch/xen/i386/mm/hypervisor.c
>  arch/xen/kernel/ctrl_if.c
>  arch/xen/kernel/evtchn.c
>  include/asm-xen/ctrl_if.h
>  include/asm-xen/evtchn.h
>  include/asm-xen/hypervisor.h
>  include/asm-xen/multicall.h
>  include/asm-xen/suspend.h
>
> I consider this the least we can do given the lack of documentation
> that would allow OS porters to roll their own. :-)
>
> There is the issue that there are bits and pieces of other people's
> code in there as well --- e.g., Christian Limpach, Mike Wray, Mark
> Williamson. But I don't foresee any problems.
>
>  -- Keir
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel

-- 
Wesley Parish
* * *
Clinersterton beademung - in all of love.  RIP James Blish
* * *
Mau e ki, "He aha te mea nui?"
You ask, "What is the most important thing?"
Maku e ki, "He tangata, he tangata, he tangata."
I reply, "It is people, it is people, it is people."


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Keir Fraser]

No need for LGPL trickery as we are talking about separately licensing
files that contain only our code. We can do that as long as we take no
GPL code that is copyrighted by someone else.

The method you describe below for circumventing the GPL by adding an
'interface barrier' of LGPL code is certainly popular (I could
cite a number of examples). However, I don't believe it is permitted
by the GPL. :-)  Linux permits it for binary-only drivers only because
Linus is prepared to suffer it.

 -- Keir

> I would've thought a (legally) efficient way of doing that would be to have a 
> library-set available under the LGPL, that call the GPL code and are called 
> by the non-GPLed guest OSes.
> 
> I'm not sure about the actual efficiency in terms of processor cycles, of such 
> an approach, but if it collocates the lawyers in the dogbox, which is where 
> they should be, instead of breathing down your neck, that should be an 
> acceptable trade.
> 
> Just my $0.02 (and it's very highly inflated of course!)
> 
> Wesley Parish


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Ronald G. Minnich]

On Thu, 14 Oct 2004, Keir Fraser wrote:

> Personally, I'm happy to relicense our 'unprivileged guest' drivers
> under a BSD-style license. This would include at least the following
> files that contain no Linux code:
> 
> linux-2.6.8.1-xen-sparse:
>  drivers/xen/netfront/*
>  drivers/xen/blkfront/*
>  drivers/xen/evtchn/*
>  drivers/console/*
>  arch/xen/i386/mm/hypervisor.c
>  arch/xen/kernel/ctrl_if.c
>  arch/xen/kernel/evtchn.c
>  include/asm-xen/ctrl_if.h
>  include/asm-xen/evtchn.h
>  include/asm-xen/hypervisor.h
>  include/asm-xen/multicall.h
>  include/asm-xen/suspend.h

That would help.

I think it's a good idea.

thanks

ron


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Keir Fraser]

I'll look into cleaning up / adding licensing notices to the
repository next week.

 -- Keir
 
> 
> 
> On Thu, 14 Oct 2004, Keir Fraser wrote:
> 
> > Personally, I'm happy to relicense our 'unprivileged guest' drivers
> > under a BSD-style license. This would include at least the following
> > files that contain no Linux code:
> > 
> > linux-2.6.8.1-xen-sparse:
> >  drivers/xen/netfront/*
> >  drivers/xen/blkfront/*
> >  drivers/xen/evtchn/*
> >  drivers/console/*
> >  arch/xen/i386/mm/hypervisor.c
> >  arch/xen/kernel/ctrl_if.c
> >  arch/xen/kernel/evtchn.c
> >  include/asm-xen/ctrl_if.h
> >  include/asm-xen/evtchn.h
> >  include/asm-xen/hypervisor.h
> >  include/asm-xen/multicall.h
> >  include/asm-xen/suspend.h
> 
> That would help.
> 
> I think it's a good idea.
> 
> thanks
> 
> ron



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Keir Fraser]

I have now amended the licensing terms for the files below, as noted
at the top of each file concerned. I also added a clarifying COPYING
file to the root of the repository.

Any files that you have derived solely from any version of the files
below are also subject to the new license (this also includes versions
you may have taken from within a different version of Linux, including
v2.4). I'd appreciate you adding the license text to the top of such
files though. 

 -- Keir

> > Personally, I'm happy to relicense our 'unprivileged guest' drivers
> > under a BSD-style license. This would include at least the following
> > files that contain no Linux code:
> > 
> > linux-2.6.8.1-xen-sparse:
> >  drivers/xen/netfront/*
> >  drivers/xen/blkfront/*
> >  drivers/xen/evtchn/*
> >  drivers/console/*
> >  arch/xen/i386/mm/hypervisor.c
> >  arch/xen/kernel/ctrl_if.c
> >  arch/xen/kernel/evtchn.c
> >  include/asm-xen/ctrl_if.h
> >  include/asm-xen/evtchn.h
> >  include/asm-xen/hypervisor.h
> >  include/asm-xen/multicall.h
> >  include/asm-xen/suspend.h
> 
> That would help.
> 
> I think it's a good idea.
> 
> thanks
> 
> ron



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: plan 9 status

[Mark A. Williamson]

> There is the issue that there are bits and pieces of other people's
> code in there as well --- e.g., Christian Limpach, Mike Wray, Mark
> Williamson. But I don't foresee any problems.

Fine by me.  The code I wrote really belongs to Intel but they BSD license 
stuff anyhow, so it shouldn't be a problem.

Mark


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Xen & I/O in clusters - problems!

[Rune Johan Andresen]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

Hi, we are benchmarking Xen in a cluster, and got some bad results. We 
might do something wrong, and wonder if anyone
have similar problems.

When we benchmark troughput from native Linux to native Linux (two 
physical nodes in the cluster) we get 786.034 MByte/s
When we benchmark from a virtual domain  (running on Xen on a physical 
node) to an another virtual domain (on another physical node) we get
56.480 MByte/s (1:16)

The difference is huge, and we wonder if the bottleneck could be the 
fact that we are using software routing (We use this in order to route 
from the physical node to the virtual OSs), or if this is just a 
downside of Xen?

I would guess it IS the SW routing, so is there any good alternatives 
to make virtual domains communicate on a cluster without sw routing?

cheers,
Rune J.A


[-- Attachment #2: Type: text/enriched, Size: 1010 bytes --]

Hi, we are benchmarking Xen in a cluster, and got some bad results. We
might do something wrong, and wonder if anyone

have similar problems. 


When we benchmark troughput from native Linux to native Linux (two
physical nodes in the cluster) we get
<fontfamily><param>Geneva</param><bigger>786.034 MByte/s

</bigger></fontfamily>When we benchmark from a virtual domain 
(running on Xen on a physical node) to an another virtual domain (on
another physical node) we get

<fontfamily><param>Geneva</param><bigger>56.480 MByte/s (1:16) 


</bigger></fontfamily>The difference is huge, and we wonder if the
bottleneck could be the fact that we are using software routing (We
use this in order to route from the physical node to the virtual OSs),
or if this is just a downside of Xen? 


I would guess it IS the SW routing, so is there any good alternatives
to make virtual domains communicate on a cluster without sw routing? 

<fontfamily><param>Geneva</param><bigger>

</bigger></fontfamily>cheers,

Rune J.A

Re: Xen & I/O in clusters - problems!

[Ian Pratt]

> Hi, we are benchmarking Xen in a cluster, and got some bad results. We
> might do something wrong, and wonder if anyone
> have similar problems.
> 
> When we benchmark troughput from native Linux to native Linux (two
> physical nodes in the cluster) we get 786.034 MByte/s
> When we benchmark from a virtual domain (running on Xen on a physical
> node) to an another virtual domain (on another physical node) we get
> 56.480 MByte/s (1:16)

(Presumably you mean MBits rather than Mbytes)

The numbers you're getting are terrible compared to what we see.
Running between virtual domains on a cluster we measure
throughput as high as 897Mb/s (same as Linux native).

Our results were recorded with dual 2.4GHz Xeons with tg3 NICs
and a 128KB socket buffer, measured using ttcp. With the virtual
domain running on the other physical CPU from domain 0 we get
897Mb/s. We get similar results running the virtual domain on the
other hyperthread of the same physical CPU. We observe a
performance reduction if we run the virtual domain on the same
(logical) CPU as domain 0, down to 660Mb/s [843Mb/s on a dual
3GHz machine, so we appear to be CPU limited in this case].

> The difference is huge, and we wonder if the bottleneck could be the
> fact that we are using software routing (We use this in order to route
> from the physical node to the virtual OSs), or if this is just a
> downside of Xen?

Our results were recorded using the dom0 linux bridge code rather
than using routing.

One thing to check is that you have don't have
CONFIG_IP_NF_CONNTRACK set to 'y' -- this slays performance.

Also, if you're running multiple domains on the same CPU you may
be running into CPU scheduling issues. Some tweaks to scheduler
parameters may fix this.

> I would guess it IS the SW routing, so is there any good alternatives
> to make virtual domains communicate on a cluster without sw routing?

The Xen 2.0 architecture is not as slick as the
monolithic-hypervisor approach of Xen 1.2, but we get better
hardware support and a lot more flexibility. However, we do burn
more CPU to achieve the same IO rate. We just have to wait for
Moore's law to catch up ;-)

Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Xen & I/O in clusters - problems!

[Mark A. Williamson]

> When we benchmark from a virtual domain  (running on Xen on a physical
> node) to an another virtual domain (on another physical node) we get
> 56.480 MByte/s (1:16)

Ouch.  How are you benchmarking this?  (what tool, what parameters, etc.).  
It'll help me reproduce this on our test systems.  Then we'll know if it's 
your config or if there's something to track down.

We did see some weird performance for small packets at one stage and I'm not 
sure if that was ever resolved.  If it's the same problem, I can do a binary 
chop search of changesets in order to locate it.

Cheers,
Mark


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Xen & I/O in clusters - problems!

[Rune Johan Andresen]

Hi, we benchmark with Scali MPI. We are in the first stage with an easy 
ping program which
first send 1 byte to measure the latency, then we tested with 10^5, 
10^7 and 10^8 size packages. We also
get the same results when sending from domain0 to domain0 in the 
cluster. We are now testing if the routing table
is the bottleneck, I let you know the results. Thank you :)

Cheers,
Rune J.A

On Oct 15, 2004, at 1:24 AM, Mark A. Williamson wrote:

>> When we benchmark from a virtual domain  (running on Xen on a physical
>> node) to an another virtual domain (on another physical node) we get
>> 56.480 MByte/s (1:16)
>
> Ouch.  How are you benchmarking this?  (what tool, what parameters, 
> etc.).
> It'll help me reproduce this on our test systems.  Then we'll know if 
> it's
> your config or if there's something to track down.
>
> We did see some weird performance for small packets at one stage and 
> I'm not
> sure if that was ever resolved.  If it's the same problem, I can do a 
> binary
> chop search of changesets in order to locate it.
>
> Cheers,
> Mark



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Xen & I/O in clusters - problems! New Information

[Rune Johan Andresen]

Hi, we have some additional information about our problem benchmarking 
Xen in clusters:

Between two Xen dom0 domains (between two physical computers in the 
cluster) we got these strange results:
(We use ttcp socketbuffsize, 10^4 -> 10^6)

Kernel 2.4:
Xen Dom0 -> Xen Dom0: ca. 65 000 KB/s

Kernel 2.6:
Xen Dom0 -> Xen Dom0: ca. 80 000 KB/s (?)

Native Linux:
Native Linux -> Native Linux: ca. 114 000 KB/s

What is new and strage is that Xen Dom0 use about 60% of the CPU when 
transfering or receiving, while
Native Linux ony use 6-7%(!) It seems like we have a problem with the 
DMA here(?). We use Xen 2.0,
Gigabit ethernet.

I tried 'mv /lib/tls /lib/tls.disabled' on each node without success

Under is the dmesg for the Xen node:

Linux version 2.6.8.1-xen0 (root@comp-pvfs-0-17.local) (gcc version 
3.2.3 20030502 (Red Hat Linux 3.2.3-34)) #1 Tue Oct 12 14:10:47 GMT 
2004
BIOS-provided physical RAM map:
  Xen: 0000000000000000 - 0000000008000000 (usable)
128MB LOWMEM available.
On node 0 totalpages: 32768
   DMA zone: 4096 pages, LIFO batch:1
   Normal zone: 28672 pages, LIFO batch:7
   HighMem zone: 0 pages, LIFO batch:1
DMI not present.
Built 1 zonelists
Kernel command line: root=/dev/sda1 ro console=tty0
Initializing CPU#0
PID hash table entries: 1024 (order 10: 8192 bytes)
Xen reported: 3400.171 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 125260k/131072k available (2636k kernel code, 5624k reserved, 
834k data, 396k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor 
mode... Ok.
Calibrating delay loop... 6789.52 BogoMIPS
Mount-cache hash table entries: 512 (order: 0, 4096 bytes)
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000
CPU: After vendor identify, caps:  bfebfbff 00000000 00000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: After all inits, caps:        beebcbe1 00000000 00000000 00000080
CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz stepping 09
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
NET: Registered protocol family 16
PCI: Using configuration type Xen
SCSI subsystem initialized
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Probing PCI hardware (bus 01)
PCI: Probing PCI hardware (bus 02)
PCI: Probing PCI hardware (bus 03)
PCI: Probing PCI hardware
Initializing Cryptographic API
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
Using anticipatory io scheduler
nbd: registered device at major 43
Intel(R) PRO/1000 Network Driver - version 5.2.52-k4
Copyright (c) 1999-2004 Intel Corporation.
PCI: Obtained IRQ 18 for device 0000:01:01.0
PCI: Setting latency timer of device 0000:01:01.0 to 64
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
PCI: Obtained IRQ 21 for device 0000:03:02.0
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
pcnet32.c:v1.30i 06.28.2004 tsbogend@alpha.franken.de
e100: Intel(R) PRO/100 Network Driver, 3.0.18
e100: Copyright(c) 1999-2004 Intel Corporation
Xen virtual console successfully installed as ttyS
Event-channel device installed.
Initialising Xen netif backend
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with 
idebus=xx
hda: SAMSUNG CD-ROM SN-124, ATAPI CD/DVD-ROM drive
ide1: I/O resource 0x170-0x177 not free.
ide1: ports already in use, skipping probe
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: ATAPI 24X CD-ROM drive, 128kB Cache
Uniform CD-ROM driver Revision: 3.20
PCI: Obtained IRQ 24 for device 0000:02:01.0
PCI: Obtained IRQ 25 for device 0000:02:01.1
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
         <Adaptec 3960D Ultra160 SCSI adapter>
         aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs

(scsi0:A:0): 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
   Vendor: SEAGATE   Model: ST336607LW        Rev: DS09
   Type:   Direct-Access                      ANSI SCSI revision: 03
scsi0:A:0:0: Tagged Queuing enabled.  Depth 32
scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
         <Adaptec 3960D Ultra160 SCSI adapter>
         aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs

Red Hat/Adaptec aacraid driver (1.1.2-lk2 Oct 12 2004)
3ware Storage Controller device driver for Linux v1.26.00.039.
3w-xxxx: No cards found.
libata version 1.02 loaded.
ata_piix version 1.02
ata_piix: combined mode detected
PCI: Obtained IRQ 17 for device 0000:00:1f.2
ata: 0x1f0 IDE port busy
PCI: Setting latency timer of device 0000:00:1f.2 to 64
ata1: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xFEA8 irq 15
ata1: SATA port has no device.
scsi2 : ata_piix
SCSI device sda: 71132959 512-byte hdwr sectors (36420 MB)
SCSI device sda: drive cache: write through
  sda: sda1 sda2 sda3 sda4 < sda5 >
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
mice: PS/2 mouse device common for all mice
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
md: raid0 personality registered as nr 2
md: raid1 personality registered as nr 3
md: raid5 personality registered as nr 4
raid5: automatically using best checksumming function: pIII_sse
    pIII_sse  :   440.400 MB/sec
raid5: using function: pIII_sse (440.400 MB/sec)
md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
device-mapper: 4.1.0-ioctl (2003-12-10) initialised: dm@uk.sistina.com
NET: Registered protocol family 2
IP: routing cache hash table of 1024 buckets, 8Kbytes
TCP: Hash tables configured (established 8192 bind 16384)
NET: Registered protocol family 1
NET: Registered protocol family 17
Bridge firewalling registered
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: sda1: orphan cleanup on readonly fs
ext3_orphan_cleanup: deleting unreferenced inode 4718
EXT3-fs: sda1: 1 orphan inode deleted
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 396k freed

   ***************************************************************
   ***************************************************************
   ** WARNING: Currently emulating unsupported memory accesses  **
   **          in /lib/tls libraries. The emulation is very     **
   **          slow, and may not work correctly with all        **
   **          programs (e.g., some may 'Segmentation fault').  **
   **          TO ENSURE FULL PERFORMANCE AND CORRECT FUNCTION, **
   **          YOU MUST EXECUTE THE FOLLOWING AS ROOT:          **
   **          mv /lib/tls /lib/tls.disabled                    **
   ***************************************************************
   ***************************************************************

Pausing... 5\b\b\b\b\b\b\b\b\b\b\b\bPausing... 4\b\b\b\b\b\b\b\b\b\b\b\bPausing... 
3\b\b\b\b\b\b\b\b\b\b\b\bPausing... 2\b\b\b\b\b\b\b\b\b\b\b\bPausing... 
1\b\b\b\b\b\b\b\b\b\b\b\bContinuing...

EXT3 FS on sda1, internal journal
Adding 1020116k swap on /dev/sda3.  Priority:-1 extents:1
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda5, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
process `syslogd' is using obsolete setsockopt SO_BSDCOMPAT
process `snmpd' is using obsolete setsockopt SO_BSDCOMPAT

Cheers,
Rune




On Oct 15, 2004, at 1:24 AM, Mark A. Williamson wrote:

>> When we benchmark from a virtual domain  (running on Xen on a physical
>> node) to an another virtual domain (on another physical node) we get
>> 56.480 MByte/s (1:16)
>
> Ouch.  How are you benchmarking this?  (what tool, what parameters, 
> etc.).
> It'll help me reproduce this on our test systems.  Then we'll know if 
> it's
> your config or if there's something to track down.
>
> We did see some weird performance for small packets at one stage and 
> I'm not
> sure if that was ever resolved.  If it's the same problem, I can do a 
> binary
> chop search of changesets in order to locate it.
>
> Cheers,
> Mark
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on 
> ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give 
> us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out 
> more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Xen & I/O in clusters - problems! New Information

[Ian Pratt]

> Between two Xen dom0 domains (between two physical computers in the 
> cluster) we got these strange results:
> (We use ttcp socketbuffsize, 10^4 -> 10^6)
> 
> Kernel 2.6:
> Xen Dom0 -> Xen Dom0: ca. 80 000 KB/s (?)
> 
> Native Linux:
> Native Linux -> Native Linux: ca. 114 000 KB/s
> 
> What is new and strage is that Xen Dom0 use about 60% of the CPU when 
> transfering or receiving, while
> Native Linux ony use 6-7%(!) It seems like we have a problem with the 
> DMA here(?). We use Xen 2.0,
> Gigabit ethernet.

dom0 to dom0 performance really shouldn't be any difference from
native. It certainly isn't on any of our machines.

The only thing I can think of is that something stupid might be
happening with interrupts on your machines. Can you compare the
rate that the relevant interrupts are going up in
/proc/interrupts between xenLinux and native.

There's no interrupt sharing or anything daft like that going on?
Are you using NAPI on the native e1000 linux driver?


Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Xen & I/O in clusters - problems! New Information

[Rune Johan Andresen]

We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the samre 
results between
two xen dom0 nodes. I am not sure if these interrupts tells anything:

Native Linux:


  CPU0
   0:   87290435          XT-PIC  timer
   1:          2          XT-PIC  keyboard
   2:          0          XT-PIC  cascade
   3:    7668994          XT-PIC  eth0
   7:          0          XT-PIC  ehci-hcd
   8:          1          XT-PIC  rtc
  10:          3          XT-PIC  usb-uhci
  11:     332088          XT-PIC  aic7xxx, aic7xxx, usb-uhci
  14:          1          XT-PIC  ide0
  15:          0          XT-PIC  libata
NMI:          0
ERR:          0


Xen Dom0:

           CPU0
   1:          2        Phys-irq  keyboard
  14:          3        Phys-irq  ide0
  18:     954304        Phys-irq  eth0
  24:       7313        Phys-irq  aic7xxx
  25:         30        Phys-irq  aic7xxx
128:          1     Dynamic-irq  misdirect
129:          0     Dynamic-irq  ctrl-if
130:     241914     Dynamic-irq  timer
131:          0     Dynamic-irq  timer_dbg, net-be-dbg
132:          0     Dynamic-irq  console
NMI:          0
ERR:          0

If you can see anything which is not normal behavior of xen please
tell us :)

Cheers,
Rune


On Oct 21, 2004, at 10:24 PM, Ian Pratt wrote:

>
>> Between two Xen dom0 domains (between two physical computers in the
>> cluster) we got these strange results:
>> (We use ttcp socketbuffsize, 10^4 -> 10^6)
>>
>> Kernel 2.6:
>> Xen Dom0 -> Xen Dom0: ca. 80 000 KB/s (?)
>>
>> Native Linux:
>> Native Linux -> Native Linux: ca. 114 000 KB/s
>>
>> What is new and strage is that Xen Dom0 use about 60% of the CPU when
>> transfering or receiving, while
>> Native Linux ony use 6-7%(!) It seems like we have a problem with the
>> DMA here(?). We use Xen 2.0,
>> Gigabit ethernet.
>
> dom0 to dom0 performance really shouldn't be any difference from
> native. It certainly isn't on any of our machines.
>
> The only thing I can think of is that something stupid might be
> happening with interrupts on your machines. Can you compare the
> rate that the relevant interrupts are going up in
> /proc/interrupts between xenLinux and native.
>
> There's no interrupt sharing or anything daft like that going on?
> Are you using NAPI on the native e1000 linux driver?
>
>
> Ian



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Xen & I/O in clusters - problems! New Information

[Ian Pratt]

> 
> We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the samre 
> results between
> two xen dom0 nodes. I am not sure if these interrupts tells anything:

It's the different rate at which the eth0 interrupt counts go up
during your bandwidth tests that is interesting. e.g. poll it
once a second during the test.

It looks like your native Linux is not using legacy PIC mode
rather than using the ioapic. Have you tried an SMP native
kernel to see if it gets the same interrupt layout as Xen?
 
Ian

> Native Linux:
> 
> 
>   CPU0
>    0:   87290435          XT-PIC  timer
>    1:          2          XT-PIC  keyboard
>    2:          0          XT-PIC  cascade
>    3:    7668994          XT-PIC  eth0
>    7:          0          XT-PIC  ehci-hcd
>    8:          1          XT-PIC  rtc
>   10:          3          XT-PIC  usb-uhci
>   11:     332088          XT-PIC  aic7xxx, aic7xxx, usb-uhci
>   14:          1          XT-PIC  ide0
>   15:          0          XT-PIC  libata
> NMI:          0
> ERR:          0
> 
> 
> Xen Dom0:
> 
>            CPU0
>    1:          2        Phys-irq  keyboard
>   14:          3        Phys-irq  ide0
>   18:     954304        Phys-irq  eth0
>   24:       7313        Phys-irq  aic7xxx
>   25:         30        Phys-irq  aic7xxx
> 128:          1     Dynamic-irq  misdirect
> 129:          0     Dynamic-irq  ctrl-if
> 130:     241914     Dynamic-irq  timer
> 131:          0     Dynamic-irq  timer_dbg, net-be-dbg
> 132:          0     Dynamic-irq  console
> NMI:          0
> ERR:          0
> 
> If you can see anything which is not normal behavior of xen please
> tell us :)
> 
> Cheers,
> Rune


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: Xen & I/O in clusters - problems! New Information

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2917 bytes --]

On Fri, Oct 22, 2004 at 06:47:35PM +0100, Ian Pratt wrote:
> > 
> > We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the samre 
> > results between
> > two xen dom0 nodes. I am not sure if these interrupts tells anything:
> 
> It's the different rate at which the eth0 interrupt counts go up
> during your bandwidth tests that is interesting. e.g. poll it
> once a second during the test.
> 

We tried sending 1 MB and measured:
non-SMP native Linux:
	~ 130k interrupts
	114 kB/s
native Linux with compiled-in SMP support, single CPU:
	~ 140k interrupts
	114 kB/s
Xen0:
	~ 180k interrupts
	80 kB/s

> It looks like your native Linux is not using legacy PIC mode
> rather than using the ioapic. Have you tried an SMP native
> kernel to see if it gets the same interrupt layout as Xen?
>  

Layout in native linux w/o SMP:

           CPU0       
  0:  130710161          XT-PIC  timer
  1:          2          XT-PIC  keyboard
  2:          0          XT-PIC  cascade
  3:   12872557          XT-PIC  eth0
  7:          0          XT-PIC  ehci-hcd
  8:          1          XT-PIC  rtc
 10:          3          XT-PIC  usb-uhci
 11:     445263          XT-PIC  aic7xxx, aic7xxx, usb-uhci
 14:          1          XT-PIC  ide0
 15:          0          XT-PIC  libata
NMI:          0 
ERR:          0


Layout in native linux compiled with SMP:

           CPU0       
  0:     121362    IO-APIC-edge  timer
  1:          2    IO-APIC-edge  keyboard
  2:          0          XT-PIC  cascade
 14:          5    IO-APIC-edge  ide0
 16:          0   IO-APIC-level  usb-uhci
 18:     177583   IO-APIC-level  eth0
 19:          0   IO-APIC-level  usb-uhci
 24:       7587   IO-APIC-level  aic7xxx
 25:         30   IO-APIC-level  aic7xxx
NMI:          0 
LOC:     121307 
ERR:          0
MIS:          0


Layout in Xen0:

           CPU0       
  1:          2        Phys-irq  keyboard
 14:          3        Phys-irq  ide0
 18:     267274        Phys-irq  eth0
 24:      15910        Phys-irq  aic7xxx
 25:         30        Phys-irq  aic7xxx
128:          1     Dynamic-irq  misdirect
129:          0     Dynamic-irq  ctrl-if
130:     771917     Dynamic-irq  timer
131:          0     Dynamic-irq  timer_dbg, net-be-dbg
132:          0     Dynamic-irq  console
NMI:          0 
ERR:          0


Questions:
Do you think interrupt sharing is the problem?
Or the use of IO-APIC?
Is this an inherent problem with Xen?
Is it possible to change the interrupt scheme in Xen in order to achieve the same performance as in native linux?


Cheers,
Håvard


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click

Re: Xen & I/O in clusters - problems! New Information

[Ian Pratt]

> On Fri, Oct 22, 2004 at 06:47:35PM +0100, Ian Pratt wrote:
> > > 
> > > We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the samre 
> > > results between
> > > two xen dom0 nodes. I am not sure if these interrupts tells anything:
> > 
> > It's the different rate at which the eth0 interrupt counts go up
> > during your bandwidth tests that is interesting. e.g. poll it
> > once a second during the test.
> > 
> 
> We tried sending 1 MB and measured:

1MB isn't really very much with a 128KB socket buffer. Do you get
the same results with larger transfers?

> non-SMP native Linux:
> 	~ 130k interrupts
> 	114 kB/s
> native Linux with compiled-in SMP support, single CPU:
> 	~ 140k interrupts
> 	114 kB/s
> Xen0:
> 	~ 180k interrupts
> 	80 kB/s

It's pretty odd that Xen's taking more interrupts. Are you using
the same native kernel version as you are for Xen?

Also, what happens if you boot xen with 'nosmp' on the Xen
command line.

We've got Xen tcp performance results from a bunch of machines,
and dom0 to dom0 performance has always been almost identical to
native for 1500 byte MTU packets. 


Ian



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - problems! New Information

[Mark A. Williamson]

Just a thought: you are using the BVT scheduler, right?  We haven't tested 
performance with the other schedulers recently but we know something goes 
wrong for IO intensive domains on Atropos.

Mark

On Wednesday 27 Oct 2004 19:26, Ian Pratt wrote:
> > On Fri, Oct 22, 2004 at 06:47:35PM +0100, Ian Pratt wrote:
> > > > We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the samre
> > > > results between
> > > > two xen dom0 nodes. I am not sure if these interrupts tells anything:
> > >
> > > It's the different rate at which the eth0 interrupt counts go up
> > > during your bandwidth tests that is interesting. e.g. poll it
> > > once a second during the test.
> >
> > We tried sending 1 MB and measured:
>
> 1MB isn't really very much with a 128KB socket buffer. Do you get
> the same results with larger transfers?
>
> > non-SMP native Linux:
> >  ~ 130k interrupts
> >  114 kB/s
> > native Linux with compiled-in SMP support, single CPU:
> > 	~ 140k interrupts
> > 	114 kB/s
> > Xen0:
> > 	~ 180k interrupts
> > 	80 kB/s
>
> It's pretty odd that Xen's taking more interrupts. Are you using
> the same native kernel version as you are for Xen?
>
> Also, what happens if you boot xen with 'nosmp' on the Xen
> command line.
>
> We've got Xen tcp performance results from a bunch of machines,
> and dom0 to dom0 performance has always been almost identical to
> native for 1500 byte MTU packets.
>
>
> Ian
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - problems! New Information

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2521 bytes --]

We're getting these results in domain 0, not in a VM. As I understand, BVT or apropos scheduling do not apply in domain 0?

Håvard

On Wed, Oct 27, 2004 at 07:40:27PM +0100, Mark A. Williamson wrote:
> Just a thought: you are using the BVT scheduler, right?  We haven't tested 
> performance with the other schedulers recently but we know something goes 
> wrong for IO intensive domains on Atropos.
> 
> Mark
> 
> On Wednesday 27 Oct 2004 19:26, Ian Pratt wrote:
> > > On Fri, Oct 22, 2004 at 06:47:35PM +0100, Ian Pratt wrote:
> > > > > We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the samre
> > > > > results between
> > > > > two xen dom0 nodes. I am not sure if these interrupts tells anything:
> > > >
> > > > It's the different rate at which the eth0 interrupt counts go up
> > > > during your bandwidth tests that is interesting. e.g. poll it
> > > > once a second during the test.
> > >
> > > We tried sending 1 MB and measured:
> >
> > 1MB isn't really very much with a 128KB socket buffer. Do you get
> > the same results with larger transfers?
> >
> > > non-SMP native Linux:
> > >  ~ 130k interrupts
> > >  114 kB/s
> > > native Linux with compiled-in SMP support, single CPU:
> > > 	~ 140k interrupts
> > > 	114 kB/s
> > > Xen0:
> > > 	~ 180k interrupts
> > > 	80 kB/s
> >
> > It's pretty odd that Xen's taking more interrupts. Are you using
> > the same native kernel version as you are for Xen?
> >
> > Also, what happens if you boot xen with 'nosmp' on the Xen
> > command line.
> >
> > We've got Xen tcp performance results from a bunch of machines,
> > and dom0 to dom0 performance has always been almost identical to
> > native for 1500 byte MTU packets.
> >
> >
> > Ian
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by:
> > Sybase ASE Linux Express Edition - download now for FREE
> > LinuxWorld Reader's Choice Award Winner for best database on Linux.
> > http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click

Re: Xen & I/O in clusters - problems! New Information

[Mark A. Williamson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 3043 bytes --]

Domain 0 is (from the POV of Xen) basically Just Another VM and is scheduled 
pre-emptively.  Some capability flags give the Dom0 VM the privileges it 
needs in order to control devices, screen, Xen management functions etc.  If 
you use the buggy Atropos then you'll lose performance even with just one 
domain.

HTH,
Mark

On Thursday 28 October 2004 11:51, Håvard Bjerke wrote:
> We're getting these results in domain 0, not in a VM. As I understand, BVT
> or apropos scheduling do not apply in domain 0?
>
> HÃ¥vard
>
> On Wed, Oct 27, 2004 at 07:40:27PM +0100, Mark A. Williamson wrote:
> > Just a thought: you are using the BVT scheduler, right?  We haven't
> > tested performance with the other schedulers recently but we know
> > something goes wrong for IO intensive domains on Atropos.
> >
> > Mark
> >
> > On Wednesday 27 Oct 2004 19:26, Ian Pratt wrote:
> > > > On Fri, Oct 22, 2004 at 06:47:35PM +0100, Ian Pratt wrote:
> > > > > > We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the
> > > > > > samre results between
> > > > > > two xen dom0 nodes. I am not sure if these interrupts tells
> > > > > > anything:
> > > > >
> > > > > It's the different rate at which the eth0 interrupt counts go up
> > > > > during your bandwidth tests that is interesting. e.g. poll it
> > > > > once a second during the test.
> > > >
> > > > We tried sending 1 MB and measured:
> > >
> > > 1MB isn't really very much with a 128KB socket buffer. Do you get
> > > the same results with larger transfers?
> > >
> > > > non-SMP native Linux:
> > > >  ~ 130k interrupts
> > > >  114 kB/s
> > > > native Linux with compiled-in SMP support, single CPU:
> > > > 	~ 140k interrupts
> > > > 	114 kB/s
> > > > Xen0:
> > > > 	~ 180k interrupts
> > > > 	80 kB/s
> > >
> > > It's pretty odd that Xen's taking more interrupts. Are you using
> > > the same native kernel version as you are for Xen?
> > >
> > > Also, what happens if you boot xen with 'nosmp' on the Xen
> > > command line.
> > >
> > > We've got Xen tcp performance results from a bunch of machines,
> > > and dom0 to dom0 performance has always been almost identical to
> > > native for 1500 byte MTU packets.
> > >
> > >
> > > Ian
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by:
> > > Sybase ASE Linux Express Edition - download now for FREE
> > > LinuxWorld Reader's Choice Award Winner for best database on Linux.
> > > http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/xen-devel


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click

Re: Xen & I/O in clusters - problems! New Information

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1759 bytes --]

On Wed, Oct 27, 2004 at 07:26:10PM +0100, Ian Pratt wrote:
> > On Fri, Oct 22, 2004 at 06:47:35PM +0100, Ian Pratt wrote:
> > > > 
> > > > We tried to compile xen0 with CONFIG_E1000_NAPI = y and got the samre 
> > > > results between
> > > > two xen dom0 nodes. I am not sure if these interrupts tells anything:
> > > 
> > > It's the different rate at which the eth0 interrupt counts go up
> > > during your bandwidth tests that is interesting. e.g. poll it
> > > once a second during the test.
> > > 
> > 
> > We tried sending 1 MB and measured:
> 
> 1MB isn't really very much with a 128KB socket buffer. Do you get
> the same results with larger transfers?
> 

With 10 MB transfers the results are roughly the same:

native Linux with compiled-in SMP support, single CPU:
	~ 1365k interrupts
	114 kB/s
Xen0 with "nosmp":
	~ 1676 k interrupts
	76 kB/s

> > non-SMP native Linux:
> > 	~ 130k interrupts
> > 	114 kB/s
> > native Linux with compiled-in SMP support, single CPU:
> > 	~ 140k interrupts
> > 	114 kB/s
> > Xen0:
> > 	~ 180k interrupts
> > 	80 kB/s
> 
> It's pretty odd that Xen's taking more interrupts. Are you using
> the same native kernel version as you are for Xen?
> 

Yes, both are 2.4.27

> Also, what happens if you boot xen with 'nosmp' on the Xen
> command line.
> 

There seems to be no change in behaviour. The interrupt layout and count remains roughly the same.

Do you have any more tips? :)


Håvard


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click

Re: Xen & I/O in clusters - problems! New Information

[Ian Pratt]

> Yes, both are 2.4.27
> 
> > Also, what happens if you boot xen with 'nosmp' on the Xen
> > command line.
> > 
> 
> There seems to be no change in behaviour. The interrupt layout and count remains roughly the same.
> 
> Do you have any more tips? :)

Please can you try Xen/linux 2.6.9. Also, please can you remind
me of the spec of your machines. 

Ian
\x1f -=- MIME -=- \x1f\f

On Wed, Oct 27, 2004 at 07:26:10PM +0100, Ian Pratt wrote:
> > On Fri, Oct 22, 2004 at 06:47:35PM +0100, Ian Pratt wrote:
> > > >=20
> > > > We tried to compile xen0 with CONFIG_E1000_NAPI =3D y and got the=
 samre=20
> > > > results between
> > > > two xen dom0 nodes. I am not sure if these interrupts tells anyth=
ing:
> > >=20
> > > It's the different rate at which the eth0 interrupt counts go up
> > > during your bandwidth tests that is interesting. e.g. poll it
> > > once a second during the test.
> > >=20
> >=20
> > We tried sending 1 MB and measured:
>=20
> 1MB isn't really very much with a 128KB socket buffer. Do you get
> the same results with larger transfers?
>=20

With 10 MB transfers the results are roughly the same:

native Linux with compiled-in SMP support, single CPU:
	~ 1365k interrupts
	114 kB/s
Xen0 with "nosmp":
	~ 1676 k interrupts
	76 kB/s

> > non-SMP native Linux:
> > 	~ 130k interrupts
> > 	114 kB/s
> > native Linux with compiled-in SMP support, single CPU:
> > 	~ 140k interrupts
> > 	114 kB/s
> > Xen0:
> > 	~ 180k interrupts
> > 	80 kB/s
>=20
> It's pretty odd that Xen's taking more interrupts. Are you using
> the same native kernel version as you are for Xen?
>=20

Yes, both are 2.4.27

> Also, what happens if you boot xen with 'nosmp' on the Xen
> command line.
>=20

There seems to be no change in behaviour. The interrupt layout and count =
remains roughly the same.

Do you have any more tips? :)


H=E5vard



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - problems! New Information

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1806 bytes --]

On Thu, Oct 28, 2004 at 01:44:44PM +0100, Ian Pratt wrote:
> 
> Please can you try Xen/linux 2.6.9. Also, please can you remind
> me of the spec of your machines. 
> 

Specs:
Ethernet controller: Intel Corp. 82547GI Gigabit Ethernet Controller
CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz single cpu
RAM: 1 GB


The previous results from 2.4.27 are (I wrote kB/s earlier but that was wrong):
Native:
	130k interrupts
	114 MB/s
Xen0:
	180k interrupts
	80 MB/s                                                                                       

New results with 2.6.9:
Native:
	bandwidth: 2048000000 bytes in 17.42 real seconds = 114794.42 KB/sec
	CPU: 0.0user 1.4sys 0:17real 8%
	interrupts: 135k                                                               
Xen0:
	bandwidth: 2048000000 bytes in 21.77 real seconds = 91885.76 KB/sec
	CPU: 0.0user 16.2sys 0:21real 74%
	interrupts: 107k

I also tried sending localhost-localhost in 2.4.27, and interestingly Native performed 7:1 times better than xen0:
Native:
    bandwidth: 2048000000 bytes in 2.70 real seconds = 741704.96 KB/sec
    CPU: 0.0user 1.6sys 0:02real 62%
Xen0:
    bandwidth: 2048000000 bytes in 17.12 real seconds = 116838.03 KB/sec
    CPU: 5.9user 0.0sys 0:17real 34%

We've also recently benchmarked in an SMP cluster and achieved satisfying results, ie. around 114 MB/s with both Native and Xen0. But we're still wondering why we're not achieving full speed in a single-cpu cluster.

Håvard


-------------------------------------------------------
This Newsletter Sponsored by: Macrovision
For reliable Linux application installations, use the industry's leading
setup authoring tool, InstallShield X. Learn more and evaluate
today. http://clk.atdmt.com/MSI/go/ins0030000001msi/direct/01/

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Rune Johan Andresen]

Hi, we have validated the single CPU results in another single CPU 
cluster, and we still get an performance loss about 30%
(ca. 85 000 KB/s between two dom0 nodes (and NOT 114 000 KB/s),
Specs:
Ethernet controller: Intel Corp. 82547GI Gigabit Ethernet Controller
CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz single CPU
RAM: 1 GB)

But, in a dual CPU cluster, Intel Xenon CPU 2.40 GHz  Ethernet 
controller: Intel Corp. 82547GI Gigabit Ethernet Controller, we get 
only 1%
(from 114 000 KB/s -> 110 000 KB/s) performance loss.)

Both the single CPU clusters and the dual cluster are Xen 2.0 beta 
(2.4.26 kernel) with Red Hat Ent. 3

It seems to me that there is an issue with Xen and Gigabit ethernet 
controllers, where Xen is optimized for
a dual core(?). As mentioned before we have more interrupts with Xen 
dom0 on eth0 than Native Linux
(we use bvt, not atropos)

Are there any Xen developers/testers which have tested Xen 2.0 on a 
single CPU cluster and can confirm (or not) these results?

Cheers,
Rune



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Ian Pratt]

> But, in a dual CPU cluster, Intel Xenon CPU 2.40 GHz  Ethernet 
> controller: Intel Corp. 82547GI Gigabit Ethernet Controller, we get 
> only 1%
> (from 114 000 KB/s -> 110 000 KB/s) performance loss.)

Good, that's the result we expect.

> Hi, we have validated the single CPU results in another single CPU 
> cluster, and we still get an performance loss about 30%
> (ca. 85 000 KB/s between two dom0 nodes (and NOT 114 000 KB/s),
> Specs:
> Ethernet controller: Intel Corp. 82547GI Gigabit Ethernet Controller
> CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz single CPU
> RAM: 1 GB)

Hmm, have your systems got an IOAPIC, or is Xen using the legacy
PIC code? The latter probably hasn't been thoroughly performance
tested...

Ian



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1368 bytes --]

On Fri, Oct 29, 2004 at 06:24:35PM +0100, Ian Pratt wrote:
> 
> > But, in a dual CPU cluster, Intel Xenon CPU 2.40 GHz  Ethernet 
> > controller: Intel Corp. 82547GI Gigabit Ethernet Controller, we get 
> > only 1%
> > (from 114 000 KB/s -> 110 000 KB/s) performance loss.)
> 
> Good, that's the result we expect.
> 
> > Hi, we have validated the single CPU results in another single CPU 
> > cluster, and we still get an performance loss about 30%
> > (ca. 85 000 KB/s between two dom0 nodes (and NOT 114 000 KB/s),
> > Specs:
> > Ethernet controller: Intel Corp. 82547GI Gigabit Ethernet Controller
> > CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz single CPU
> > RAM: 1 GB)
> 
> Hmm, have your systems got an IOAPIC, or is Xen using the legacy
> PIC code? The latter probably hasn't been thoroughly performance
> tested...
> 

The native systems can have either XT-PIC or IOAPIC, and it seems that both have equal performance. In Xen0, however, by looking through dmesg, there doesn't seem to be any IOAPIC. Is it possible to enable (or disable) IOAPIC in Xen0?

Håvard


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Ian Pratt]

> The native systems can have either XT-PIC or IOAPIC, and it seems that both have equal performance. In Xen0, however, by looking through dmesg, there doesn't seem to be any IOAPIC. Is it possible to enable (or disable) IOAPIC in Xen0?

If you boot with 'ignorebiostables' on the Xen command line Xen
will ignore the IOAPIC and use the PIC.

I could believe there might be a lurking performance problem with
PIC support as it probably hasn't had the same level of
performance testing as IOAPIC.

Ian
\x1f -=- MIME -=- \x1f\f

On Fri, Oct 29, 2004 at 06:24:35PM +0100, Ian Pratt wrote:
>=20
> > But, in a dual CPU cluster, Intel Xenon CPU 2.40 GHz  Ethernet=20
> > controller: Intel Corp. 82547GI Gigabit Ethernet Controller, we get=20
> > only 1%
> > (from 114 000 KB/s -> 110 000 KB/s) performance loss.)
>=20
> Good, that's the result we expect.
>=20
> > Hi, we have validated the single CPU results in another single CPU=20
> > cluster, and we still get an performance loss about 30%
> > (ca. 85 000 KB/s between two dom0 nodes (and NOT 114 000 KB/s),
> > Specs:
> > Ethernet controller: Intel Corp. 82547GI Gigabit Ethernet Controller
> > CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz single CPU
> > RAM: 1 GB)
>=20
> Hmm, have your systems got an IOAPIC, or is Xen using the legacy
> PIC code? The latter probably hasn't been thoroughly performance
> tested...
>=20

The native systems can have either XT-PIC or IOAPIC, and it seems that bo=
th have equal performance. In Xen0, however, by looking through dmesg, th=
ere doesn't seem to be any IOAPIC. Is it possible to enable (or disable) =
IOAPIC in Xen0?

H=E5vard



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1097 bytes --]

On Fri, Oct 29, 2004 at 08:46:46PM +0100, Ian Pratt wrote:
> > The native systems can have either XT-PIC or IOAPIC, and it seems that both have equal performance. In Xen0, however, by looking through dmesg, there doesn't seem to be any IOAPIC. Is it possible to enable (or disable) IOAPIC in Xen0?
> 
> If you boot with 'ignorebiostables' on the Xen command line Xen
> will ignore the IOAPIC and use the PIC.
> 
> I could believe there might be a lurking performance problem with
> PIC support as it probably hasn't had the same level of
> performance testing as IOAPIC.
> 

'ignorebiostables' did the trick. Thanks for the help :) 

Now I'm curious about live migration. Is it possible to live migrate an MPI application running on a set of nodes to another set of nodes? Has anyone tried that?

Håvard


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id\x12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Mark A. Williamson]

> Now I'm curious about live migration. Is it possible to live migrate an MPI
> application running on a set of nodes to another set of nodes? Has anyone
> tried that?

If you're using MPI over TCP/IP (which I imagine you are) then it should Just 
Work (TM).  We have tried live migration with MPI applications but you 
shouldn't have any problems moving the VMs around with a cluster.

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Mark A. Williamson]

> If you're using MPI over TCP/IP (which I imagine you are) then it should
> Just Work (TM).  We have tried live migration with MPI applications but you
> shouldn't have any problems moving the VMs around with a cluster.

Sorry I meant to say we have *not* tried live migration with MPI applications.

Note to self: read before clicking send!

Cheers,
Mark


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Rune Johan Andresen]

Well, after the issue between two xen dom0 domains is solved there is a 
new case we don't
understand:

With two physical domains and 4 guest OSs (2 on each physical node) we 
get some rare results with ttcp (b=1000000, l = 1000000):

Lets say we have two guest OSs on physical node A, A1 and A2, and two 
guest OSs on physical node B, B1 and B2.

Between A1 and B1 I get 110 000 KB/s (which is almost optimal!)
Between A1 and B2 I get 81 0000 KB/s
Between A2 and B1 I get 94 000 KB/s

Do you have any idea why we get less performance in the last two cases? 
It doesn't make sense. It cant be
a bottleneck in the network either because of case 1.(?)

Cheers,
Rune


On Nov 2, 2004, at 5:51 PM, Mark A. Williamson wrote:

>> If you're using MPI over TCP/IP (which I imagine you are) then it 
>> should
>> Just Work (TM).  We have tried live migration with MPI applications 
>> but you
>> shouldn't have any problems moving the VMs around with a cluster.
>
> Sorry I meant to say we have *not* tried live migration with MPI 
> applications.
>
> Note to self: read before clicking send!
>
> Cheers,
> Mark
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Bin Ren]

Among A1, A2, B1, B2, which ones are domain 0 and which are unpriviledged?

- Bin

On Thu, 4 Nov 2004 18:42:34 +0100, Rune Johan Andresen
<runejoha@idi.ntnu.no> wrote:
> Well, after the issue between two xen dom0 domains is solved there is a
> new case we don't
> understand:
> 
> With two physical domains and 4 guest OSs (2 on each physical node) we
> get some rare results with ttcp (b=1000000, l = 1000000):
> 
> Lets say we have two guest OSs on physical node A, A1 and A2, and two
> guest OSs on physical node B, B1 and B2.
> 
> Between A1 and B1 I get 110 000 KB/s (which is almost optimal!)
> Between A1 and B2 I get 81 0000 KB/s
> Between A2 and B1 I get 94 000 KB/s
> 
> Do you have any idea why we get less performance in the last two cases?
> It doesn't make sense. It cant be
> a bottleneck in the network either because of case 1.(?)
> 
> Cheers,
> Rune
> 
> 
> 
> 
> On Nov 2, 2004, at 5:51 PM, Mark A. Williamson wrote:
> 
> >> If you're using MPI over TCP/IP (which I imagine you are) then it
> >> should
> >> Just Work (TM).  We have tried live migration with MPI applications
> >> but you
> >> shouldn't have any problems moving the VMs around with a cluster.
> >
> > Sorry I meant to say we have *not* tried live migration with MPI
> > applications.
> >
> > Note to self: read before clicking send!
> >
> > Cheers,
> > Mark
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by:
> > Sybase ASE Linux Express Edition - download now for FREE
> > LinuxWorld Reader's Choice Award Winner for best database on Linux.
> > http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Rune Johan Andresen]

A and B are domain 0, A1,A2,B1,B2 are the guest OS systems. I think all 
are privileged (?), They are
created with the same config file, exception of the file loopback 
configuration and ip config.

Cheers,
Rune

On Nov 4, 2004, at 7:41 PM, Bin Ren wrote:

> Among A1, A2, B1, B2, which ones are domain 0 and which are 
> unpriviledged?
>
> - Bin
>
> On Thu, 4 Nov 2004 18:42:34 +0100, Rune Johan Andresen
> <runejoha@idi.ntnu.no> wrote:
>> Well, after the issue between two xen dom0 domains is solved there is 
>> a
>> new case we don't
>> understand:
>>
>> With two physical domains and 4 guest OSs (2 on each physical node) we
>> get some rare results with ttcp (b=1000000, l = 1000000):
>>
>> Lets say we have two guest OSs on physical node A, A1 and A2, and two
>> guest OSs on physical node B, B1 and B2.
>>
>> Between A1 and B1 I get 110 000 KB/s (which is almost optimal!)
>> Between A1 and B2 I get 81 0000 KB/s
>> Between A2 and B1 I get 94 000 KB/s
>>
>> Do you have any idea why we get less performance in the last two 
>> cases?
>> It doesn't make sense. It cant be
>> a bottleneck in the network either because of case 1.(?)
>>
>> Cheers,
>> Rune
>>
>>
>>
>>
>> On Nov 2, 2004, at 5:51 PM, Mark A. Williamson wrote:
>>
>>>> If you're using MPI over TCP/IP (which I imagine you are) then it
>>>> should
>>>> Just Work (TM).  We have tried live migration with MPI applications
>>>> but you
>>>> shouldn't have any problems moving the VMs around with a cluster.
>>>
>>> Sorry I meant to say we have *not* tried live migration with MPI
>>> applications.
>>>
>>> Note to self: read before clicking send!
>>>
>>> Cheers,
>>> Mark
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.Net email is sponsored by:
>>> Sybase ASE Linux Express Edition - download now for FREE
>>> LinuxWorld Reader's Choice Award Winner for best database on Linux.
>>> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/xen-devel
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by:
>> Sybase ASE Linux Express Edition - download now for FREE
>> LinuxWorld Reader's Choice Award Winner for best database on Linux.
>> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xen-devel
>>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Keir Fraser]

> 
> A and B are domain 0, A1,A2,B1,B2 are the guest OS systems. I think all 
> are privileged (?), They are
> created with the same config file, exception of the file loopback 
> configuration and ip config.
> 
> Cheers,
> Rune

Are you running on SMP systems? I would guess the performance differs
based on whether DOM0 and the unprivileged guest are runnign on the
same CPU or different CPUs...

 -- Keir


-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Rune Johan Andresen]

Yes, I am running on a SMP system with two CPUs. The nodes are running 
on different CPUs, but is this a problem? If the two
guest OSs are unprivileged and run on separate CPUs, they should 
perform I/O just as good (or bad) anyway, no?

Cheers,
Rune


On Nov 5, 2004, at 5:47 PM, Keir Fraser wrote:

>>
>> A and B are domain 0, A1,A2,B1,B2 are the guest OS systems. I think 
>> all
>> are privileged (?), They are
>> created with the same config file, exception of the file loopback
>> configuration and ip config.
>>
>> Cheers,
>> Rune
>
> Are you running on SMP systems? I would guess the performance differs
> based on whether DOM0 and the unprivileged guest are runnign on the
> same CPU or different CPUs...
>
>  -- Keir



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click

Re: Xen & I/O in clusters - Single Vs. Dual CPU issue

[Keir Fraser]

Unless your workload maxes out one of the CPUs, or doesn't play as
well as it could with the CPU scheduler. In general it's probably wise
to run your dom0 on another CPU, or at least HyperThread, from other
performance-critical VMs. 

The situation on a single CPU should be improveable with tweaking,
however. 

 -- Keir

> 
> Yes, I am running on a SMP system with two CPUs. The nodes are running 
> on different CPUs, but is this a problem? If the two
> guest OSs are unprivileged and run on separate CPUs, they should 
> perform I/O just as good (or bad) anyway, no?
> 
> Cheers,
> Rune
> 
> 
> On Nov 5, 2004, at 5:47 PM, Keir Fraser wrote:
> 
> >>
> >> A and B are domain 0, A1,A2,B1,B2 are the guest OS systems. I think 
> >> all
> >> are privileged (?), They are
> >> created with the same config file, exception of the file loopback
> >> configuration and ip config.
> >>
> >> Cheers,
> >> Rune
> >
> > Are you running on SMP systems? I would guess the performance differs
> > based on whether DOM0 and the unprivileged guest are runnign on the
> > same CPU or different CPUs...
> >
> >  -- Keir
> 



-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_id=5588&alloc_id=12065&op=click