Ghost traffic seen in all guest on a kvm hypervisor

Ghost traffic seen in all guest on a kvm hypervisor

[Dennis Jacobfeuerborn]

Hi,
I have a problem on a kvm hypervisor that I cannot explain. It appears 
that some traffic shows up in the interface monitoring in all of the 
guests on that hypervisor. We are not using broadcasts in any way and I 
see peaks of 100mbit for several minutes on all of these systems.

Does anybody have an idea what could cause this?

Regards,
   Dennis

Re: Ghost traffic seen in all guest on a kvm hypervisor

[Remy Mudingay]

Hi Dennis,

Please post the KVM bridge topology and also the output of ethtool -k ethX (for each real nic).

Cheers,

Remy


Sent from my Phone

> On 24 Mar 2014, at 02:57, Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> 
> Hi,
> I have a problem on a kvm hypervisor that I cannot explain. It appears that some traffic shows up in the interface monitoring in all of the guests on that hypervisor. We are not using broadcasts in any way and I see peaks of 100mbit for several minutes on all of these systems.
> 
> Does anybody have an idea what could cause this?
> 
> Regards,
>  Dennis
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ghost traffic seen in all guest on a kvm hypervisor

[Wolfgang Hennerbichler]

Make sure your bridge doesn't have an "bridge_maxage 0" configured or otherwhise your software-switch will behave like a hub. 

On Mon, Mar 24, 2014 at 02:57:37AM +0100, Dennis Jacobfeuerborn wrote:
> Hi,
> I have a problem on a kvm hypervisor that I cannot explain. It
> appears that some traffic shows up in the interface monitoring in
> all of the guests on that hypervisor. We are not using broadcasts in
> any way and I see peaks of 100mbit for several minutes on all of
> these systems.
> 
> Does anybody have an idea what could cause this?
> 
> Regards,
>   Dennis
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
http://www.wogri.com

Re: Ghost traffic seen in all guest on a kvm hypervisor

[Dennis Jacobfeuerborn]

Hi Remy,
the only physical interface used is eth0 but I'm using vlans so the 
interface connected to the bridge is eth0.8 and all the guests are 
connected to this bridge as well. The output of ethtool -k for both eth0 
and eth0.8 is:

Offload parameters for eth0.8:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off

Regards,
   Dennis

On 24.03.2014 07:00, Remy Mudingay wrote:
> Hi Dennis,
>
> Please post the KVM bridge topology and also the output of ethtool -k ethX (for each real nic).
>
> Cheers,
>
> Remy
>
>
> Sent from my Phone
>
>> On 24 Mar 2014, at 02:57, Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
>>
>> Hi,
>> I have a problem on a kvm hypervisor that I cannot explain. It appears that some traffic shows up in the interface monitoring in all of the guests on that hypervisor. We are not using broadcasts in any way and I see peaks of 100mbit for several minutes on all of these systems.
>>
>> Does anybody have an idea what could cause this?
>>
>> Regards,
>>   Dennis
>> --
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ghost traffic seen in all guest on a kvm hypervisor

[Dennis Jacobfeuerborn]

This is a Centos 6 System and I haven't set the aging explicitly but 
from the output of showstp it seems the aging is set to a non-zero value 
by default:

# brctl showstp vlanbr8
vlanbr8
  bridge id              8000.00259035bc06
  designated root        8000.00259035bc06
  root port                 0
  path cost                 0
  max age                  19.99
  bridge max age           19.99
  hello time                1.99
  bridge hello time         1.99
  forward delay            14.99
  bridge forward delay     14.99
  ageing time             299.95
  hello timer               1.85
  tcn timer                 0.00
  topology change timer     0.00
  gc timer                 15.85
  hash elasticity           4
  hash max                512
  mc last member count      2
  mc init query count       2
  mc router                 1
  mc snooping               1
  mc last member timer      0.99
  mc membership timer     259.96
  mc querier timer        254.96
  mc query interval       124.98
  mc response interval      9.99
  mc init query interval   31.24
  flags

The question though is that even if the bridge acted as a hub wouldn't 
the guest drop the packets anyway since the MAC doesn't match? In other 
words I would expect to see the packets on the interface with tcpdump 
but not that the traffic gets counted as incoming traffic on the interface.

Regards,
   Dennis

On 24.03.2014 08:04, Wolfgang Hennerbichler wrote:
> Make sure your bridge doesn't have an "bridge_maxage 0" configured or otherwhise your software-switch will behave like a hub.
>
> On Mon, Mar 24, 2014 at 02:57:37AM +0100, Dennis Jacobfeuerborn wrote:
>> Hi,
>> I have a problem on a kvm hypervisor that I cannot explain. It
>> appears that some traffic shows up in the interface monitoring in
>> all of the guests on that hypervisor. We are not using broadcasts in
>> any way and I see peaks of 100mbit for several minutes on all of
>> these systems.
>>
>> Does anybody have an idea what could cause this?
>>
>> Regards,
>>    Dennis
>> --
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: Ghost traffic seen in all guest on a kvm hypervisor

[Joel Gerber]

SWYgdGhlIGJyaWRnZSB3YXMgYWN0aW5nIGFzIGEgaHViLCB0aGVuIHRoZSBmcmFtZXMgd291bGQg
cGFzcyB0aHJvdWdoIHRoZSBoeXBlcnZpc29yIHRvIHRoZSB2aXJ0dWFsIE5JQyBvbiB0aGUgZ3Vl
c3QuIElmIHRoZSBmcmFtZSB3YXMgdW5pY2FzdCwgdGhlbiB0aGUgTklDIHdvdWxkIGRyb3AgaXQs
IGlmIHRoZSBwYWNrZXQgaGFkIGEgYnJvYWRjYXN0IEV0aGVybmV0IGFkZHJlc3MsIHRoZW4gdGhl
IGZyYW1lIHdvdWxkIHBhc3MgdGhlIE5JQyBhbmQgaGl0IHRoZSBPUy4NCg0KSGF2ZSB5b3UgY2hl
Y2tlZCB0aGUgcGFja2V0cyB0byB2ZXJpZnkgd2hldGhlciB0aGUgZGVzdGluYXRpb24gTUFDIGFk
ZHJlc3Mgd2FzIGFsbCBmJ3M/IEhhdmUgeW91IHZlcmlmaWVkIHRoYXQgdGhpcyBpc24ndCBtdWx0
aWNhc3QgdHJhZmZpYz8NCg0KQW5vdGhlciBxdWVzdGlvbiBJIHdvdWxkIGhhdmUsIGFyZSB0aGUg
ZnJhbWVzIHlvdSdyZSBzZWVpbmcgaW4gdW5leHBlY3RlZCBwbGFjZXMgY29taW5nIGluIG9uIGlu
dGVyZmFjZXMgdGhhdCBkbyBub3QgYmVsb25nIHRvIHRoZSBzYW1lIGJyaWRnZWQgbmV0d29yaz8g
SUU6IElmIHlvdXIgaHlwZXJ2aXNvciBoYXMgMiBzZXBhcmF0ZSBicmlkZ2VkIG5ldHdvcmtzIGNh
bGxlZCBBIGFuZCBCLCBhbmQgZ3Vlc3QgQSBvbmx5IGJlbG9uZ3MgdG8gbmV0d29yayBBIHdoaWxl
IGd1ZXN0IEIgb25seSBiZWxvbmdzIHRvIG5ldHdvcmsgQiwgaXMgZ3Vlc3QgQiBzZWVpbmcgcGFj
a2V0cyBjb21lIGluIG9uIGl0cyBpbnRlcmZhY2UgdGhhdCBzaG91bGQndmUgb25seSBiZWxvbmdl
ZCB0byBuZXR3b3JrIEE/DQoNCkpvZWwgR2VyYmVyDQpOZXR3b3JrIFNwZWNpYWxpc3QNCk5ldHdv
cmsgT3BlcmF0aW9ucw0KRWFzdGxpbmsNCkU6IEpvZWwuR2VyYmVyQGNvcnAuZWFzdGxpbmsuY2Eg
VDogNTE5Ljc4Ni4xMjQxDQotLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KRnJvbTogbGFydGMt
b3duZXJAdmdlci5rZXJuZWwub3JnIFttYWlsdG86bGFydGMtb3duZXJAdmdlci5rZXJuZWwub3Jn
XSBPbiBCZWhhbGYgT2YgRGVubmlzIEphY29iZmV1ZXJib3JuDQpTZW50OiBNYXJjaC0yNC0xNCA2
OjQ1IEFNDQpUbzogV29sZmdhbmcgSGVubmVyYmljaGxlcg0KQ2M6IGxhcnRjQHZnZXIua2VybmVs
Lm9yZw0KU3ViamVjdDogUmU6IEdob3N0IHRyYWZmaWMgc2VlbiBpbiBhbGwgZ3Vlc3Qgb24gYSBr
dm0gaHlwZXJ2aXNvcg0KDQpUaGlzIGlzIGEgQ2VudG9zIDYgU3lzdGVtIGFuZCBJIGhhdmVuJ3Qg
c2V0IHRoZSBhZ2luZyBleHBsaWNpdGx5IGJ1dCBmcm9tIHRoZSBvdXRwdXQgb2Ygc2hvd3N0cCBp
dCBzZWVtcyB0aGUgYWdpbmcgaXMgc2V0IHRvIGEgbm9uLXplcm8gdmFsdWUgYnkgZGVmYXVsdDoN
Cg0KIyBicmN0bCBzaG93c3RwIHZsYW5icjgNCnZsYW5icjgNCiAgYnJpZGdlIGlkICAgICAgICAg
ICAgICA4MDAwLjAwMjU5MDM1YmMwNg0KICBkZXNpZ25hdGVkIHJvb3QgICAgICAgIDgwMDAuMDAy
NTkwMzViYzA2DQogIHJvb3QgcG9ydCAgICAgICAgICAgICAgICAgMA0KICBwYXRoIGNvc3QgICAg
ICAgICAgICAgICAgIDANCiAgbWF4IGFnZSAgICAgICAgICAgICAgICAgIDE5Ljk5DQogIGJyaWRn
ZSBtYXggYWdlICAgICAgICAgICAxOS45OQ0KICBoZWxsbyB0aW1lICAgICAgICAgICAgICAgIDEu
OTkNCiAgYnJpZGdlIGhlbGxvIHRpbWUgICAgICAgICAxLjk5DQogIGZvcndhcmQgZGVsYXkgICAg
ICAgICAgICAxNC45OQ0KICBicmlkZ2UgZm9yd2FyZCBkZWxheSAgICAgMTQuOTkNCiAgYWdlaW5n
IHRpbWUgICAgICAgICAgICAgMjk5Ljk1DQogIGhlbGxvIHRpbWVyICAgICAgICAgICAgICAgMS44
NQ0KICB0Y24gdGltZXIgICAgICAgICAgICAgICAgIDAuMDANCiAgdG9wb2xvZ3kgY2hhbmdlIHRp
bWVyICAgICAwLjAwDQogIGdjIHRpbWVyICAgICAgICAgICAgICAgICAxNS44NQ0KICBoYXNoIGVs
YXN0aWNpdHkgICAgICAgICAgIDQNCiAgaGFzaCBtYXggICAgICAgICAgICAgICAgNTEyDQogIG1j
IGxhc3QgbWVtYmVyIGNvdW50ICAgICAgMg0KICBtYyBpbml0IHF1ZXJ5IGNvdW50ICAgICAgIDIN
CiAgbWMgcm91dGVyICAgICAgICAgICAgICAgICAxDQogIG1jIHNub29waW5nICAgICAgICAgICAg
ICAgMQ0KICBtYyBsYXN0IG1lbWJlciB0aW1lciAgICAgIDAuOTkNCiAgbWMgbWVtYmVyc2hpcCB0
aW1lciAgICAgMjU5Ljk2DQogIG1jIHF1ZXJpZXIgdGltZXIgICAgICAgIDI1NC45Ng0KICBtYyBx
dWVyeSBpbnRlcnZhbCAgICAgICAxMjQuOTgNCiAgbWMgcmVzcG9uc2UgaW50ZXJ2YWwgICAgICA5
Ljk5DQogIG1jIGluaXQgcXVlcnkgaW50ZXJ2YWwgICAzMS4yNA0KICBmbGFncw0KDQpUaGUgcXVl
c3Rpb24gdGhvdWdoIGlzIHRoYXQgZXZlbiBpZiB0aGUgYnJpZGdlIGFjdGVkIGFzIGEgaHViIHdv
dWxkbid0IHRoZSBndWVzdCBkcm9wIHRoZSBwYWNrZXRzIGFueXdheSBzaW5jZSB0aGUgTUFDIGRv
ZXNuJ3QgbWF0Y2g/IEluIG90aGVyIHdvcmRzIEkgd291bGQgZXhwZWN0IHRvIHNlZSB0aGUgcGFj
a2V0cyBvbiB0aGUgaW50ZXJmYWNlIHdpdGggdGNwZHVtcCBidXQgbm90IHRoYXQgdGhlIHRyYWZm
aWMgZ2V0cyBjb3VudGVkIGFzIGluY29taW5nIHRyYWZmaWMgb24gdGhlIGludGVyZmFjZS4NCg0K
UmVnYXJkcywNCiAgIERlbm5pcw0KDQpPbiAyNC4wMy4yMDE0IDA4OjA0LCBXb2xmZ2FuZyBIZW5u
ZXJiaWNobGVyIHdyb3RlOg0KPiBNYWtlIHN1cmUgeW91ciBicmlkZ2UgZG9lc24ndCBoYXZlIGFu
ICJicmlkZ2VfbWF4YWdlIDAiIGNvbmZpZ3VyZWQgb3Igb3RoZXJ3aGlzZSB5b3VyIHNvZnR3YXJl
LXN3aXRjaCB3aWxsIGJlaGF2ZSBsaWtlIGEgaHViLg0KPg0KPiBPbiBNb24sIE1hciAyNCwgMjAx
NCBhdCAwMjo1NzozN0FNICswMTAwLCBEZW5uaXMgSmFjb2JmZXVlcmJvcm4gd3JvdGU6DQo+PiBI
aSwNCj4+IEkgaGF2ZSBhIHByb2JsZW0gb24gYSBrdm0gaHlwZXJ2aXNvciB0aGF0IEkgY2Fubm90
IGV4cGxhaW4uIEl0IA0KPj4gYXBwZWFycyB0aGF0IHNvbWUgdHJhZmZpYyBzaG93cyB1cCBpbiB0
aGUgaW50ZXJmYWNlIG1vbml0b3JpbmcgaW4gYWxsIA0KPj4gb2YgdGhlIGd1ZXN0cyBvbiB0aGF0
IGh5cGVydmlzb3IuIFdlIGFyZSBub3QgdXNpbmcgYnJvYWRjYXN0cyBpbiBhbnkgDQo+PiB3YXkg
YW5kIEkgc2VlIHBlYWtzIG9mIDEwMG1iaXQgZm9yIHNldmVyYWwgbWludXRlcyBvbiBhbGwgb2Yg
dGhlc2UgDQo+PiBzeXN0ZW1zLg0KPj4NCj4+IERvZXMgYW55Ym9keSBoYXZlIGFuIGlkZWEgd2hh
dCBjb3VsZCBjYXVzZSB0aGlzPw0KPj4NCj4+IFJlZ2FyZHMsDQo+PiAgICBEZW5uaXMNCj4+IC0t
DQo+PiBUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJz
Y3JpYmUgbGFydGMiIGluIA0KPj4gdGhlIGJvZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2
Z2VyLmtlcm5lbC5vcmcgTW9yZSBtYWpvcmRvbW8gDQo+PiBpbmZvIGF0ICBodHRwOi8vdmdlci5r
ZXJuZWwub3JnL21ham9yZG9tby1pbmZvLmh0bWwNCj4NCg0KLS0NClRvIHVuc3Vic2NyaWJlIGZy
b20gdGhpcyBsaXN0OiBzZW5kIHRoZSBsaW5lICJ1bnN1YnNjcmliZSBsYXJ0YyIgaW4gdGhlIGJv
ZHkgb2YgYSBtZXNzYWdlIHRvIG1ham9yZG9tb0B2Z2VyLmtlcm5lbC5vcmcgTW9yZSBtYWpvcmRv
bW8gaW5mbyBhdCAgaHR0cDovL3ZnZXIua2VybmVsLm9yZy9tYWpvcmRvbW8taW5mby5odG1sDQo