Limited IOP/s on Dual Xeon KVM Host

Limited IOP/s on Dual Xeon KVM Host

[Stefan Priebe]

Hello lists,

on a dual Xeon KVM Host i get max 6000 IOP/s random 4k writes AND reads. 
On a Single Xeon KVM Host i get 17.000-18.000 IOP/s write and read. I 
already tried to pin the kvm process using numactl and also the fio 
process but it doesn't help on the dual xeon.

10GBE Network is fine. I get 9.8Gbit/s on both hosts. Kernel is also he 
same on both.

Anybody an idea?

Greets
Stefan

Re: Limited IOP/s on Dual Xeon KVM Host

[Mark Nelson]

On 11/10/2012 02:03 AM, Stefan Priebe wrote:
> Hello lists,
>
> on a dual Xeon KVM Host i get max 6000 IOP/s random 4k writes AND reads.
> On a Single Xeon KVM Host i get 17.000-18.000 IOP/s write and read. I
> already tried to pin the kvm process using numactl and also the fio
> process but it doesn't help on the dual xeon.
>
> 10GBE Network is fine. I get 9.8Gbit/s on both hosts. Kernel is also he
> same on both.
>
> Anybody an idea?

When you say KVM host, do you mean the underlying node or the virtual 
machine instance?

If you mean underlying node, it could be remote memory access or if you 
are on a last gen xeon if you have dual io hubs, you could be hitting a 
remote io hub for the network card.  I wouldn't think that would cause 
such a big hit, but those are things to look into.

If you mean that a single Xeon VM instance on a dual Xeon node is faster 
than a dual Xeon VM instance, I'm not really sure what to tell you since 
you've already tried pinning the processes.

>
> Greets
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Limited IOP/s on Dual Xeon KVM Host

[Stefan Priebe]

Am 10.11.2012 14:41, schrieb Mark Nelson:
> On 11/10/2012 02:03 AM, Stefan Priebe wrote:
>> Hello lists,
>>
>> on a dual Xeon KVM Host i get max 6000 IOP/s random 4k writes AND reads.
>> On a Single Xeon KVM Host i get 17.000-18.000 IOP/s write and read. I
>> already tried to pin the kvm process using numactl and also the fio
>> process but it doesn't help on the dual xeon.
>>
>> 10GBE Network is fine. I get 9.8Gbit/s on both hosts. Kernel is also he
>> same on both.
>>
>> Anybody an idea?
>
> When you say KVM host, do you mean the underlying node or the virtual
> machine instance?
Sorry i'm talking about the vm host regarding the HW. The vm instance is 
always the same.

> If you mean underlying node, it could be remote memory access or if you
> are on a last gen xeon if you have dual io hubs, you could be hitting a
> remote io hub for the network card.  I wouldn't think that would cause
> such a big hit, but those are things to look into.

I'm on E5-Xeon. What means io hub?

Stefan

Re: Limited IOP/s on Dual Xeon KVM Host

[Andrey Korolyov]

On Sat, Nov 10, 2012 at 5:49 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Am 10.11.2012 14:41, schrieb Mark Nelson:
>
>> On 11/10/2012 02:03 AM, Stefan Priebe wrote:
>>>
>>> Hello lists,
>>>
>>> on a dual Xeon KVM Host i get max 6000 IOP/s random 4k writes AND reads.
>>> On a Single Xeon KVM Host i get 17.000-18.000 IOP/s write and read. I
>>> already tried to pin the kvm process using numactl and also the fio
>>> process but it doesn't help on the dual xeon.
>>>
>>> 10GBE Network is fine. I get 9.8Gbit/s on both hosts. Kernel is also he
>>> same on both.
>>>
>>> Anybody an idea?
>>
>>
>> When you say KVM host, do you mean the underlying node or the virtual
>> machine instance?
>
> Sorry i'm talking about the vm host regarding the HW. The vm instance is
> always the same.
>
>
>> If you mean underlying node, it could be remote memory access or if you
>> are on a last gen xeon if you have dual io hubs, you could be hitting a
>> remote io hub for the network card.  I wouldn't think that would cause
>> such a big hit, but those are things to look into.
>
>
> I'm on E5-Xeon. What means io hub?

QPI path length, in other terms, numa distance(hope Mark means the
same). Yes, it is impossible to have such degradation even in worst
case on two-head node. I assume two possible things - you have pinned
many processes on the core set which including default core for the
network card` irq, please check it via /proc/interrupts, or you have
not really did pinning and qemu process losing ticks by switching
cores - it may be checked, say, by top and guest cpu bencmark. For the
network card, it may be generally recommended to move its irq affinity
to entire numa node to which it belongs.

>
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Limited IOP/s on Dual Xeon KVM Host

[Stefan Priebe]

Am 10.11.2012 17:00, schrieb Andrey Korolyov:
>>> If you mean underlying node, it could be remote memory access or if you
>>> are on a last gen xeon if you have dual io hubs, you could be hitting a
>>> remote io hub for the network card.  I wouldn't think that would cause
>>> such a big hit, but those are things to look into.
>>
>>
>> I'm on E5-Xeon. What means io hub?
>
> QPI path length, in other terms, numa distance(hope Mark means the
> same). Yes, it is impossible to have such degradation even in worst
> case on two-head node. I assume two possible things - you have pinned
> many processes on the core set which including default core for the
> network card` irq, please check it via /proc/interrupts
The 10GBE card is alligned with all it's queues to one CPU. (use intel 
irq affinity script).

> not really did pinning and qemu process losing ticks by switching
> cores - it may be checked, say, by top and guest cpu bencmark. For the
> network card, it may be generally recommended to move its irq affinity
> to entire numa node to which it belongs.
Might be but i'm seeing that the 10GBE card is also slower than on other 
systems ;-(

Stefan

Re: Limited IOP/s on Dual Xeon KVM Host

[Stefan Priebe]

Got it fixed by Bios update... crazy.

Am 10.11.2012 17:00, schrieb Andrey Korolyov:
> On Sat, Nov 10, 2012 at 5:49 PM, Stefan Priebe <s.priebe@profihost.ag> wrote:
>> Am 10.11.2012 14:41, schrieb Mark Nelson:
>>
>>> On 11/10/2012 02:03 AM, Stefan Priebe wrote:
>>>>
>>>> Hello lists,
>>>>
>>>> on a dual Xeon KVM Host i get max 6000 IOP/s random 4k writes AND reads.
>>>> On a Single Xeon KVM Host i get 17.000-18.000 IOP/s write and read. I
>>>> already tried to pin the kvm process using numactl and also the fio
>>>> process but it doesn't help on the dual xeon.
>>>>
>>>> 10GBE Network is fine. I get 9.8Gbit/s on both hosts. Kernel is also he
>>>> same on both.
>>>>
>>>> Anybody an idea?
>>>
>>>
>>> When you say KVM host, do you mean the underlying node or the virtual
>>> machine instance?
>>
>> Sorry i'm talking about the vm host regarding the HW. The vm instance is
>> always the same.
>>
>>
>>> If you mean underlying node, it could be remote memory access or if you
>>> are on a last gen xeon if you have dual io hubs, you could be hitting a
>>> remote io hub for the network card.  I wouldn't think that would cause
>>> such a big hit, but those are things to look into.
>>
>>
>> I'm on E5-Xeon. What means io hub?
>
> QPI path length, in other terms, numa distance(hope Mark means the
> same). Yes, it is impossible to have such degradation even in worst
> case on two-head node. I assume two possible things - you have pinned
> many processes on the core set which including default core for the
> network card` irq, please check it via /proc/interrupts, or you have
> not really did pinning and qemu process losing ticks by switching
> cores - it may be checked, say, by top and guest cpu bencmark. For the
> network card, it may be generally recommended to move its irq affinity
> to entire numa node to which it belongs.
>
>>
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: Limited IOP/s on Dual Xeon KVM Host

[Dietmar Maurer]

> Got it fixed by Bios update... crazy.

So you can't see any regression in IOPS with more cores? 
Can you speedup things with process pinning?

Just want to make sure that NUMA does not play any role here.

Re: Limited IOP/s on Dual Xeon KVM Host

[Stefan Priebe]

Am 11.11.2012 10:14, schrieb Dietmar Maurer:
>> Got it fixed by Bios update... crazy.
>
> So you can't see any regression in IOPS with more cores?
In this subject i wasn't talking about simply more cores in VM but about 
having a Dual Xeon instead of a Single Xeon as host.

It seems Supermicro just had a bug in their X9 Dual Socket mainboard bios...

> Can you speedup things with process pinning?
> Just want to make sure that NUMA does not play any role here.
Yes i still can speed up with pinning but this also applies to a Single 
Xeon. When i pin the fio process inside VM i get 18.000 instead of 
15.000 iops.

Greets,
Stefan