Report on Virtualization in Clusters

Report on Virtualization in Clusters

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 550 bytes --]

Rune and I just finished writing a report on using virtualization with Xen in clusters, with focus on network performance. You may download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf

We hope you find it interesting!

cheers,
Håvard


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/

Re: Report on Virtualization in Clusters

[Jacob Gorm Hansen]

Håvard Bjerke wrote:
> Rune and I just finished writing a report on using virtualization with Xen in clusters, with focus on network performance. You may download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf
> 
> We hope you find it interesting!

Yes, very interesting. Good work.

It seems from your tests without interrupt coalescing that Xen has a 
higher interrupt-handling overhead than native Linux.  I suppose that 
with the new IO model, Xen needs to flush the TLBs for every interrupt 
that arrives. Do you think this is the reason for the overhead, or are 
the performance problems only due to your hardware being badly supported 
by Xen?

Jacob


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Report on Virtualization in Clusters

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1681 bytes --]

On Fri, Nov 26, 2004 at 11:44:03AM +0100, Jacob Gorm Hansen wrote:
> Håvard Bjerke wrote:
> >Rune and I just finished writing a report on using virtualization with Xen 
> >in clusters, with focus on network performance. You may download it from 
> >http://www.idi.ntnu.no/~havarbj/clust_virt.pdf
> >
> >We hope you find it interesting!
> 
> Yes, very interesting. Good work.
> 
> It seems from your tests without interrupt coalescing that Xen has a 
> higher interrupt-handling overhead than native Linux.  I suppose that 
> with the new IO model, Xen needs to flush the TLBs for every interrupt 
> that arrives. Do you think this is the reason for the overhead, or are 
> the performance problems only due to your hardware being badly supported 
> by Xen?

My guess is that it is the former, because we "solved" the problem with the IOAPIC bug with 'ignorebiostables'. Thus using legacy PIC, everything seemed to behave normally except when not using IC.

I think it boils down to whether or not using legacy PIC as opposed to (a supported) IOAPIC has an impact on interrupt handling. If it does, I think it's the latter reason, otherwise I think it's the former.

It would be quite easy to verify on the other cluster. I just couldn't find out how to disable IC on the tg3 driver.

The Xen developers could probably give a better answer, though.

Håvard


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/

Re: Report on Virtualization in Clusters

[Ian Pratt]

> > It seems from your tests without interrupt coalescing that Xen has a 
> > higher interrupt-handling overhead than native Linux.  I suppose that 
> > with the new IO model, Xen needs to flush the TLBs for every interrupt 
> > that arrives. Do you think this is the reason for the overhead, or are 
> > the performance problems only due to your hardware being badly supported 
> > by Xen?
> 
> My guess is that it is the former, because we "solved" the problem with the IOAPIC bug with 'ignorebiostables'. Thus using legacy PIC, everything seemed to behave normally except when not using IC.
> 
> I think it boils down to whether or not using legacy PIC as opposed to (a supported) IOAPIC has an impact on interrupt handling. If it does, I think it's the latter reason, otherwise I think it's the former.

The fact that you had to disable the IOAPIC with ignorebiostables
is rather concerning. It certainly sounds like Xen is really
struggling on your platform. What motherboard chipset are the
machines in your cluster?

We're hoping that this kind of problem will go away when Xen 2.1
gets released, as we're planing on leaving IOAPIC initialisation
to domain 0, hence using the existing Linux code. Domain 0 will
then have an interface to tell Xen just what it needs to know
about the IOAPIC configuration (i.e. how to ACK interrupts).

Ian


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/

Re: Report on Virtualization in Clusters

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1473 bytes --]

On Sun, Nov 28, 2004 at 11:17:03AM +0000, Ian Pratt wrote:
> 
> > > It seems from your tests without interrupt coalescing that Xen has a 
> > > higher interrupt-handling overhead than native Linux.  I suppose that 
> > > with the new IO model, Xen needs to flush the TLBs for every interrupt 
> > > that arrives. Do you think this is the reason for the overhead, or are 
> > > the performance problems only due to your hardware being badly supported 
> > > by Xen?
> > 
> > My guess is that it is the former, because we "solved" the problem with the IOAPIC bug with 'ignorebiostables'. Thus using legacy PIC, everything seemed to behave normally except when not using IC.
> > 
> > I think it boils down to whether or not using legacy PIC as opposed to (a supported) IOAPIC has an impact on interrupt handling. If it does, I think it's the latter reason, otherwise I think it's the former.
> 
> The fact that you had to disable the IOAPIC with ignorebiostables
> is rather concerning. It certainly sounds like Xen is really
> struggling on your platform. What motherboard chipset are the
> machines in your cluster?

They are Intel 875P chipsets

Håvard


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/

Re: Report on Virtualization in Clusters

[Rune Johan Andresen]

The application benchmarking behaved similar on the two clusters (with 
different HW), which makes
it more probably it is the overhead. As mentioned we didn't disable IC 
on the tg3 drive so we don't
really know.

- Rune

On Nov 26, 2004, at 11:44 AM, Jacob Gorm Hansen wrote:

> Håvard Bjerke wrote:
>> Rune and I just finished writing a report on using virtualization 
>> with Xen in clusters, with focus on network performance. You may 
>> download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf
>> We hope you find it interesting!
>
> Yes, very interesting. Good work.
>
> It seems from your tests without interrupt coalescing that Xen has a 
> higher interrupt-handling overhead than native Linux.  I suppose that 
> with the new IO model, Xen needs to flush the TLBs for every interrupt 
> that arrives. Do you think this is the reason for the overhead, or are 
> the performance problems only due to your hardware being badly 
> supported by Xen?
>
> Jacob



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/

Re: Report on Virtualization in Clusters

[Håvard Bjerke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 1609 bytes --]

Please note that the application benchmarking was performed with IC enabled. The application benchmarking results do not reflect the problems that were experienced with IC disabled.

Håvard

On Fri, Nov 26, 2004 at 04:27:10PM +0100, Rune Johan Andresen wrote:
> 
> The application benchmarking behaved similar on the two clusters (with 
> different HW), which makes
> it more probably it is the overhead. As mentioned we didn't disable IC 
> on the tg3 drive so we don't
> really know.
> 
> - Rune
> 
> On Nov 26, 2004, at 11:44 AM, Jacob Gorm Hansen wrote:
> 
> >Håvard Bjerke wrote:
> >>Rune and I just finished writing a report on using virtualization 
> >>with Xen in clusters, with focus on network performance. You may 
> >>download it from http://www.idi.ntnu.no/~havarbj/clust_virt.pdf
> >>We hope you find it interesting!
> >
> >Yes, very interesting. Good work.
> >
> >It seems from your tests without interrupt coalescing that Xen has a 
> >higher interrupt-handling overhead than native Linux.  I suppose that 
> >with the new IO model, Xen needs to flush the TLBs for every interrupt 
> >that arrives. Do you think this is the reason for the overhead, or are 
> >the performance problems only due to your hardware being badly 
> >supported by Xen?
> >
> >Jacob
> 


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/