Poor performance of ingress filter for separate VMs

Poor performance of ingress filter for separate VMs

[Neil Aggarwal]

Hello everyone:

I have a CentOS server running two virtual machines using KVM.
Each machine has its own virtual device since I am using bridged
networking.

I am trying to allocate bandwidth to each virtual machine using tc.

I have these rules set up:

/sbin/tc qdisc add dev v1262 ingress handle ffff:
/sbin/tc filter add dev v1262 parent ffff: protocol ip prio 50 u32 match ip
src 0.0.0.0/0 police rate 5mbit burst 500k mtu 3100 drop flowid :1

/sbin/tc qdisc add dev v1263 ingress handle ffff:
/sbin/tc filter add dev v1263 parent ffff: protocol ip prio 50 u32 match ip
src 0.0.0.0/0 police rate 5mbit burst 500k mtu 3100 drop flowid :1

You can see them when I do a qdisc ls:

qdisc ingress ffff: dev v1262 parent ffff:fff1 ----------------
qdisc ingress ffff: dev v1263 parent ffff:fff1 ----------------

The problem is when v1262 is doing a large upload, v1263 gets starved.
If I remove the ingress qdisc, v1263 runs at full speed regardless of what
v1262 is doing.
Its as if the two devices are using the same queue.  I did not think that
would happen since
they are totally separate devices.

Any ideas what is happening here?

Thanks,
  Neil

--
Neil Aggarwal, (972)834-1565, http://UnmeteredVPS.net/centos
Virtual private server with CentOS 6 preinstalled
Unmetered bandwidth = no overage charges

Re: Poor performance of ingress filter for separate VMs

[Andy Furniss]

Neil Aggarwal wrote:
> Hello everyone:
>
> I have a CentOS server running two virtual machines using KVM.
> Each machine has its own virtual device since I am using bridged
> networking.
>
> I am trying to allocate bandwidth to each virtual machine using tc.
>
> I have these rules set up:
>
> /sbin/tc qdisc add dev v1262 ingress handle ffff:
> /sbin/tc filter add dev v1262 parent ffff: protocol ip prio 50 u32 match ip
> src 0.0.0.0/0 police rate 5mbit burst 500k mtu 3100 drop flowid :1
>
> /sbin/tc qdisc add dev v1263 ingress handle ffff:
> /sbin/tc filter add dev v1263 parent ffff: protocol ip prio 50 u32 match ip
> src 0.0.0.0/0 police rate 5mbit burst 500k mtu 3100 drop flowid :1
>
> You can see them when I do a qdisc ls:
>
> qdisc ingress ffff: dev v1262 parent ffff:fff1 ----------------
> qdisc ingress ffff: dev v1263 parent ffff:fff1 ----------------
>
> The problem is when v1262 is doing a large upload, v1263 gets starved.
> If I remove the ingress qdisc, v1263 runs at full speed regardless of what
> v1262 is doing.
> Its as if the two devices are using the same queue.  I did not think that
> would happen since
> they are totally separate devices.
>
> Any ideas what is happening here?

I've never tested with a virtual machine so don't know.

I have suggested mtu 3100 in the past as it happened to work on a quick 
test on a real 100m nic - maybe for vm you need to go higher, or just 
turn off any sort of offload for the virtual device with ethtool and see 
if that helps.

ethtool -k <devname>

to see what the settings are and eg.

ethtool -K <devname> gro off

Re: Poor performance of ingress filter for separate VMs

[Andy Furniss]

Neil Aggarwal wrote:
> Hello everyone:
>
> I have a CentOS server running two virtual machines using KVM.
> Each machine has its own virtual device since I am using bridged
> networking.
>
> I am trying to allocate bandwidth to each virtual machine using tc.
>
> I have these rules set up:
>
> /sbin/tc qdisc add dev v1262 ingress handle ffff:
> /sbin/tc filter add dev v1262 parent ffff: protocol ip prio 50 u32 match ip
> src 0.0.0.0/0 police rate 5mbit burst 500k mtu 3100 drop flowid :1
>
> /sbin/tc qdisc add dev v1263 ingress handle ffff:
> /sbin/tc filter add dev v1263 parent ffff: protocol ip prio 50 u32 match ip
> src 0.0.0.0/0 police rate 5mbit burst 500k mtu 3100 drop flowid :1
>
> You can see them when I do a qdisc ls:
>
> qdisc ingress ffff: dev v1262 parent ffff:fff1 ----------------
> qdisc ingress ffff: dev v1263 parent ffff:fff1 ----------------
>
> The problem is when v1262 is doing a large upload, v1263 gets starved.
> If I remove the ingress qdisc, v1263 runs at full speed regardless of what
> v1262 is doing.
> Its as if the two devices are using the same queue.  I did not think that
> would happen since
> they are totally separate devices.
>
> Any ideas what is happening here?

I've never tested with a virtual machine so don't know.

I have suggested mtu 3100 in the past as it happened to work on a quick 
test on a real 100m nic - maybe for vm you need to go higher, or just 
turn off any sort of offload for the virtual device with ethtool and see 
if that helps.

ethtool -k <devname>

to see what the settings are and eg.

ethtool -K <devname> gro off

RE: Poor performance of ingress filter for separate VMs

[Neil Aggarwal]

Andy:

> turn off any sort of offload for the virtual device with ethtool and see 
> if that helps.

When I go to the host and type:
ethtool -k v1262

It gives me this output:

Offload parameters for v1262:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: on
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off

The generic-receive-offload is already off.

The v1263 device also has the same result.

Any other ideas?

Thanks, 
  Neil

--
Neil Aggarwal, (972)834-1565, http://UnmeteredVPS.net/centos
Virtual private server with CentOS 6 preinstalled
Unmetered bandwidth = no overage charges

Re: Poor performance of ingress filter for separate VMs

[Andy Furniss]

Neil Aggarwal wrote:
> Andy:
>
>> turn off any sort of offload for the virtual device with ethtool and see
>> if that helps.
>
> When I go to the host and type:
> ethtool -k v1262
>
> It gives me this output:
>
> Offload parameters for v1262:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: on

> Any other ideas?

Ahh, tso will also need to be turned off if you want to see normal/real 
sized packets.

I don't know if it will help your specific problem, but generally now I 
see it's on for virtual devices I think turning off will be better than 
using a large mtu with policers just to to avoid dropping too much at once.

If it doesn't help then see what's going on with tcpdump.

RE: Poor performance of ingress filter for separate VMs

[Neil Aggarwal]

Andy:

> Ahh, tso will also need to be turned off if you want to see normal/real 
> sized packets.

I tried this:
ethtool -K v1262 tso off

I get back this reponse:
Cannot set device tcp segmentation offload settings: Operation not supported

I guess I can't change the settings for a virtual device.

Neil


--
To unsubscribe from this list: send the line "unsubscribe lartc" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Poor performance of ingress filter for separate VMs

[Andy Furniss]

Neil Aggarwal wrote:
> Andy:
>
>> Ahh, tso will also need to be turned off if you want to see normal/real
>> sized packets.
>
> I tried this:
> ethtool -K v1262 tso off
>
> I get back this reponse:
> Cannot set device tcp segmentation offload settings: Operation not supported
>
> I guess I can't change the settings for a virtual device.

Strange, maybe a KVM/whatever virtual driver  issue - I guess there are 
places you can ask.

FWIW I just noticed lo has tso enabled and it turns off OK, so it 
doesn't have to be a real nic.

RE: Poor performance of ingress filter for separate VMs

[Neil Aggarwal]

Andy:

> Strange, maybe a KVM/whatever virtual driver  issue - I guess there are 
> places you can ask.

I just figured out libvirt has bandwidth controls in the guest machine's
configuration.  I tried it and it worked.  Looking at the qdisc ls report,
it apparently uses tc to achieve the limiting so I am not sure why my
manual commands did not work.

Thanks for your help,
  Neil

--
Neil Aggarwal, (972)834-1565, http://UnmeteredVPS.net/centos
Virtual private server with CentOS 6 preinstalled
Unmetered bandwidth = no overage charges