vif tx drops

vif tx drops

[David Becker]

On a 2.4-xen0 domain-0,  netstat reports lots of TX drops on the vif for
packets destined to xenU.  Is that normal? 

xen0# netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
eth1   1500 0   7497330      0      0      0 1031962      0      0      0 BMRU
lo    16436 0     13515      0      0      0   13515      0      0      0 LRU
vif4.  1500 0    574388      0      0      0 6413103      0   5243      0 BMRU
xen-b  1500 0   5759045      0      0      0  317122      0      0      0 BMRU



Here tx drop rate stats from about 50 machines.

ok 5247675 drop 3336  rate 0.064
ok 5354094 drop 3760  rate 0.070
ok 5382219 drop 4041  rate 0.075
ok 5237801 drop 3971  rate 0.076
ok 5365707 drop 4143  rate 0.077
ok 5365726 drop 4185  rate 0.078
ok 4622468 drop 3663  rate 0.079
ok 5380767 drop 4236  rate 0.079
ok 4622951 drop 3725  rate 0.081
ok 4625124 drop 3788  rate 0.082
ok 4625563 drop 3823  rate 0.083
ok 5355340 drop 4503  rate 0.084
ok 4626115 drop 4243  rate 0.092
ok 4634068 drop 4310  rate 0.093
ok 4622073 drop 4393  rate 0.095
ok 4634365 drop 4427  rate 0.096
ok 5377578 drop 5200  rate 0.097
ok 5378639 drop 5227  rate 0.097
ok 4646310 drop 4627  rate 0.100
ok 4634241 drop 4687  rate 0.101
ok 4620564 drop 4719  rate 0.102
ok 5354846 drop 5500  rate 0.103
ok 5354830 drop 5582  rate 0.104
ok 4634278 drop 4947  rate 0.107
ok 5351365 drop 5826  rate 0.109
ok 5349713 drop 5955  rate 0.111
ok 5354567 drop 6175  rate 0.115
ok 5377851 drop 6195  rate 0.115
ok 5378396 drop 6397  rate 0.119
ok 5378232 drop 6569  rate 0.122
ok 5577610 drop 6873  rate 0.123
ok 5348725 drop 6959  rate 0.130
ok 1761944 drop 2487  rate 0.141
ok 4632893 drop 6767  rate 0.146
ok 5347566 drop 8667  rate 0.162
ok 5971517 drop 9684  rate 0.162
ok 5587120 drop 9115  rate 0.163
ok 5977107 drop 10023  rate 0.168
ok 5609610 drop 9885  rate 0.176
ok 5863496 drop 10654  rate 0.182
ok 5603311 drop 10481  rate 0.187
ok 6077008 drop 11523  rate 0.190
ok 5586420 drop 10668  rate 0.191
ok 5612229 drop 11301  rate 0.201
ok 6260650 drop 12869  rate 0.206
ok 5998453 drop 13610  rate 0.227
ok 6298489 drop 14415  rate 0.229
ok 5595978 drop 12872  rate 0.230
ok 6199668 drop 14246  rate 0.230
ok 6459684 drop 15827  rate 0.245
ok 6298486 drop 15656  rate 0.249
ok 5804212 drop 15735  rate 0.271
ok 6215310 drop 18704  rate 0.301
ok 5778922 drop 237733  rate 4.114
ok 5332122 drop 235716  rate 4.421


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

Are the drops happening at a slow, steady rate? Or are they perhaps
all happening during domain startup and then there are no more drops?

 -- Keir

> 
> On a 2.4-xen0 domain-0,  netstat reports lots of TX drops on the vif for
> packets destined to xenU.  Is that normal? 
> 
> xen0# netstat -i
> Kernel Interface table
> Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
> eth1   1500 0   7497330      0      0      0 1031962      0      0      0 BMRU
> lo    16436 0     13515      0      0      0   13515      0      0      0 LRU
> vif4.  1500 0    574388      0      0      0 6413103      0   5243      0 BMRU
> xen-b  1500 0   5759045      0      0      0  317122      0      0      0 BMRU


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[David Becker]

" Are the drops happening at a slow, steady rate? Or are they perhaps
" all happening during domain startup and then there are no more drops?

The drop rate is high right after a xenU startup (16% when I just tried it),
then the rate slows to just under 1% of packets in steady state.



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

> 
> " Are the drops happening at a slow, steady rate? Or are they perhaps
> " all happening during domain startup and then there are no more drops?
> 
> The drop rate is high right after a xenU startup (16% when I just tried it),
> then the rate slows to just under 1% of packets in steady state.
> 

Please try changing the following line in netfront.c:
#define RX_MIN_TARGET 8
to:
#define RX_MIN_TARGET NETIF_RX_RING_SIZE

This will turn off the attempts to dynamically adjust the number of
receive buffers queued up to receive packets. This strategy can lose
packets if traffic is bursty -- also, if the domain is occasionally
preempted, packets can queue up and overflow a buffer allowance that
is otherwise sufficient.

If you find that this sorts out your packet-loss problem then I'll
take a look at the adjustment strategy and/or make it possible to
disable it in the kernel configuration.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[David Becker]

" Please try changing the following line in netfront.c:
" #define RX_MIN_TARGET 8
" to:
" #define RX_MIN_TARGET NETIF_RX_RING_SIZE

WIth this change, netstat -i on the xen0 shows 40 to 60 tx drops on the
vif at boot and no further drops during steady state operation.
A few xenU had no drops at all.

I still see the multiple second pauses in maybe 10% of domains I create,
so the problem is not due to the vif packet drops.
Even on the same machine with two xenU domains, one xenU will 
suffer from pauses and the other not.



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[David Becker]

" " Please try changing the following line in netfront.c:
" " #define RX_MIN_TARGET 8
" " to:
" " #define RX_MIN_TARGET NETIF_RX_RING_SIZE


It looks like xen0 or the bridge stuff is not getting some packets
destined for xenU.

I traced an ssh connection from a machine running stock 2.4.25
to a xenU domain by running tcpdump on xen0 and on the other machine.

The trace on xen0 shows a 4 second gap where no ssh packets were received
for the xenU domain even though the other machine was sending retransmits. 
Only the last retransmit appears in the xen0 dump.

It is unlikely the network dropped the missing packets.  This is low bandwidth
stuff and somewhat repeatable.  I've never seen the effect on xen0, and
in this case, there is another xenU domain on the same host
showing no pause symptoms.

The 'netstat -i' statistics on xen0 and xenU do not show any dropped packets.


On xen0, tcpdump -i eth0 shows the 4 second gap between packets going to xenU:
13:58:24.247209 IP firefly-batch.39292 > batch009.ssh: . ack 8065 win 40544 <nop,nop,timestamp 431700765 1532967>
13:58:28.432536 IP firefly-batch.39292 > batch009.ssh: P 1296:1392(96) ack 8065 win 40544 <nop,nop,timestamp 431701184 1532967>


On the other host tcpdump shows those two packets, plus 5 more
duplicates sent during the gap:
13:58:24.262055 IP firefly-batch.39292 > batch009.ssh: . ack 2145 win 40544 <nop,nop,timestamp 431700765 1532967>
13:58:25.302477 IP firefly-batch.39292 > batch009.ssh: P 720:768(48) ack 2145 win 40544 <nop,nop,timestamp 431700869 1532967>
13:58:25.414312 IP firefly-batch.39292 > batch009.ssh: P 768:816(48) ack 2145 win 40544 <nop,nop,timestamp 431700880 1532967>
13:58:25.507337 IP firefly-batch.39292 > batch009.ssh: P 720:816(96) ack 2145 win 40544 <nop,nop,timestamp 431700890 1532967>
13:58:25.927331 IP firefly-batch.39292 > batch009.ssh: P 720:816(96) ack 2145 win 40544 <nop,nop,timestamp 431700932 1532967>
13:58:26.767331 IP firefly-batch.39292 > batch009.ssh: P 720:816(96) ack 2145 win 40544 <nop,nop,timestamp 431701016 1532967>
13:58:28.447331 IP firefly-batch.39292 > batch009.ssh: P 720:816(96) ack 2145 win 40544 <nop,nop,timestamp 431701184 1532967>



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

> It looks like xen0 or the bridge stuff is not getting some packets
> destined for xenU.
> 
> I traced an ssh connection from a machine running stock 2.4.25
> to a xenU domain by running tcpdump on xen0 and on the other machine.
> 
> The trace on xen0 shows a 4 second gap where no ssh packets were received
> for the xenU domain even though the other machine was sending retransmits. 
> Only the last retransmit appears in the xen0 dump.
> 
> It is unlikely the network dropped the missing packets.  This is low bandwidth
> stuff and somewhat repeatable.  I've never seen the effect on xen0, and
> in this case, there is another xenU domain on the same host
> showing no pause symptoms.
> 
> The 'netstat -i' statistics on xen0 and xenU do not show any dropped packets.

It does very much look as though the packets simply aren't getting to
the host machine. Perhaps a bad interaction with an Ethernet switch?
One thing to check is that, if you are not choosing your own MAC
addresses, that the random ones chosen by xend turn out to be unique
across the cluster --- any duplicates will simply beat each other up
in the switch forwarding caches!! This is *definitely* worth checking
out at this point, and can be fixed by picking your own MAC addresses.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[David Becker]

" This is *definitely* worth checking
" out at this point, and can be fixed by picking your own MAC addresses.

Yep.  How big a space do you randomize for mac addrs?

# grep -i AA:00:00:24:22:F3 a
batch009 (172.16.12.19) at AA:00:00:24:22:F3 [ether] on eth0
batch075 (172.16.12.85) at AA:00:00:24:22:F3 [ether] on eth0



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

> 
> " This is *definitely* worth checking
> " out at this point, and can be fixed by picking your own MAC addresses.
> 
> Yep.  How big a space do you randomize for mac addrs?
> 
> # grep -i AA:00:00:24:22:F3 a
> batch009 (172.16.12.19) at AA:00:00:24:22:F3 [ether] on eth0
> batch075 (172.16.12.85) at AA:00:00:24:22:F3 [ether] on eth0
> 

I think AA:00:00:00:00:00 - AA:00:00:7F:FF:FF (i.e., 8 million
addresses). But the random-number generator is crap.

You're definitely best off generating your own if creating any decent
number of VMs.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[David Becker]

"  But the random-number generator is crap.

I'll say.  Out of 55 recently created VMs, 15 got dup mac addrs.
    count addr
        3 AA:00:00:21:81:CF
        2 AA:00:00:76:32:D9
        2 AA:00:00:68:1F:40
        2 AA:00:00:55:B7:91
        2 AA:00:00:39:E0:21
        2 AA:00:00:24:22:F3
        2 AA:00:00:20:57:40

" You're definitely best off generating your own if creating any decent
" number of VMs.

will do.  Thanks.




-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Ian Pratt]

> 
> "  But the random-number generator is crap.
> 
> I'll say.  Out of 55 recently created VMs, 15 got dup mac addrs.
>     count addr

    mac = [ 0xaa, 0x00, 0x00,
            random.randint(0x00, 0x7f),
            random.randint(0x00, 0xff),
            random.randint(0x00, 0xff) ]
    return ':'.join(map(lambda x: "%02x" % x, mac))

You could try adding the following ahead of the above code in
tools/python/xen/xm/create.py:

random.seed(time.time())

(you may need to add an 'import time' at the top of the file).

Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

> > 
> > "  But the random-number generator is crap.
> > 
> > I'll say.  Out of 55 recently created VMs, 15 got dup mac addrs.
> >     count addr

> You could try adding the following ahead of the above code in
> tools/python/xen/xm/create.py:
> 
> random.seed(time.time())
> 
> (you may need to add an 'import time' at the top of the file).

Yes, this is worth a try. 

I've experimented with random.randint() but I am totally unable to
make it perform so abysmally as to generate 15 dupes in 55 selections
from a space of 8 million. I'm getting about 1 dupe every 2000
selections. 

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

> > > 
> > > "  But the random-number generator is crap.
> > > 
> > > I'll say.  Out of 55 recently created VMs, 15 got dup mac addrs.
> > >     count addr
> 
> > You could try adding the following ahead of the above code in
> > tools/python/xen/xm/create.py:
> > 
> > random.seed(time.time())
> > 
> > (you may need to add an 'import time' at the top of the file).
> 
> Yes, this is worth a try. 
> 
> I've experimented with random.randint() but I am totally unable to
> make it perform so abysmally as to generate 15 dupes in 55 selections
> from a space of 8 million. I'm getting about 1 dupe every 2000
> selections. 
> 
>  -- Keir

Actually theg change is trivial ('random.seed()') so I checked it
in. Hopefully it will avoid embarrassingly bad 'random' choices.

 -- Keir


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Brian Wolfe]

I temporarilly solved this by converting the IP to hexidecimal and
setting the first 2 bytes of the mac to DE:AD. Unfortunately xen
increments the 3rd if you set the mac. We need a way to set the mac on
both ends of the vif device. 

Maybe we can have things this way. options...
a: set the mac on one end and have a standard "increment" bit somewhere
higher up(say first byte).
b: set the macs on both ends
c: set a specific range of macs to use

Either way xen really should test (not certain how exactly) to see if
it's already in existence prior to using a mac that is dynamicly set.
Not testing is playing russian roulette IMHO (as some including myself
have found out already).

Opinions?

On Wed, 2004-10-13 at 15:17, Keir Fraser wrote:
> > 
> > " This is *definitely* worth checking
> > " out at this point, and can be fixed by picking your own MAC addresses.
> > 
> > Yep.  How big a space do you randomize for mac addrs?
> > 
> > # grep -i AA:00:00:24:22:F3 a
> > batch009 (172.16.12.19) at AA:00:00:24:22:F3 [ether] on eth0
> > batch075 (172.16.12.85) at AA:00:00:24:22:F3 [ether] on eth0
> > 
> 
> I think AA:00:00:00:00:00 - AA:00:00:7F:FF:FF (i.e., 8 million
> addresses). But the random-number generator is crap.
> 
> You're definitely best off generating your own if creating any decent
> number of VMs.
> 
>  -- Keir
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

One option is to have the backend interfaces pick their MAC addresses
from a well-known range: e.g., AA:00:FF:00:00:00 + <vif id>.
These wouldn't need to be unique -- they'd never be seen 'on the
wire'.

The random MAC allocator would never pick a MAC in that range, and
xend could ensure that manually-selected MACs never fall in that
range and print an error if they did.

It's a shame that the backend needs a MAC different from the frontend
-- but if it doesn't then the bridge code complains. :-(

 -- Keir

> I temporarilly solved this by converting the IP to hexidecimal and
> setting the first 2 bytes of the mac to DE:AD. Unfortunately xen
> increments the 3rd if you set the mac. We need a way to set the mac on
> both ends of the vif device. 
> 
> Maybe we can have things this way. options...
> a: set the mac on one end and have a standard "increment" bit somewhere
> higher up(say first byte).
> b: set the macs on both ends
> c: set a specific range of macs to use
> 
> Either way xen really should test (not certain how exactly) to see if
> it's already in existence prior to using a mac that is dynamicly set.
> Not testing is playing russian roulette IMHO (as some including myself
> have found out already).
> 
> Opinions?
> 
> On Wed, 2004-10-13 at 15:17, Keir Fraser wrote:
> > > 
> > > " This is *definitely* worth checking
> > > " out at this point, and can be fixed by picking your own MAC addresses.
> > > 
> > > Yep.  How big a space do you randomize for mac addrs?
> > > 
> > > # grep -i AA:00:00:24:22:F3 a
> > > batch009 (172.16.12.19) at AA:00:00:24:22:F3 [ether] on eth0
> > > batch075 (172.16.12.85) at AA:00:00:24:22:F3 [ether] on eth0
> > > 
> > 
> > I think AA:00:00:00:00:00 - AA:00:00:7F:FF:FF (i.e., 8 million
> > addresses). But the random-number generator is crap.
> > 
> > You're definitely best off generating your own if creating any decent
> > number of VMs.
> > 
> >  -- Keir
> > 
> > 
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > Use IT products in your business? Tell us what you think of them. Give us
> > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Christian Limpach]

On Wed, Oct 13, 2004 at 10:07:57PM -0500, Brian Wolfe wrote:
> I temporarilly solved this by converting the IP to hexidecimal and
> setting the first 2 bytes of the mac to DE:AD. Unfortunately xen
> increments the 3rd if you set the mac. We need a way to set the mac on
> both ends of the vif device. 
> 
> Opinions?

Why don't you simply convert an IP address a.b.c.d into
AA:a:00:b:c:d and let Xen do its thing on the 3rd byte?
I even doubt that doing AA:00:a:b:c:d would be a problem
unless you really have IP addresses a.b.c.d and a.(b^1).c.d
on the same switch...

    christian



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Christian Limpach]

On Thu, Oct 14, 2004 at 09:30:51AM +0100, Christian Limpach wrote:
> On Wed, Oct 13, 2004 at 10:07:57PM -0500, Brian Wolfe wrote:
> > I temporarilly solved this by converting the IP to hexidecimal and
> > setting the first 2 bytes of the mac to DE:AD. Unfortunately xen
> > increments the 3rd if you set the mac. We need a way to set the mac on
> > both ends of the vif device. 
> > 
> > Opinions?
> 
> Why don't you simply convert an IP address a.b.c.d into
> AA:a:00:b:c:d and let Xen do its thing on the 3rd byte?
> I even doubt that doing AA:00:a:b:c:d would be a problem
> unless you really have IP addresses a.b.c.d and a.(b^1).c.d

or actually a.b.c.d and (a^1).b.c.d...

> on the same switch...
> 
>     christian
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
> 


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Brian Wolfe]

So the xen side of the vif isn't see on the wire except for a: the
backend driver domain( usually domain-0) and that vif's front and driver
domain eh? (I got the front and back end locations right yes?)

Interesting...

Either way it goes, how would one go about verifying that the MAC is
truly unique on it's visible lan? I'd be i very interested in this as a
safety precaution. Right now I can't think of how to do the uniqueness
test... 8-P if someone can give me an idea or two I could try to code it
since it's something that I want.(brain fried this morning)

On Thu, 2004-10-14 at 03:21, Keir Fraser wrote:
> One option is to have the backend interfaces pick their MAC addresses
> from a well-known range: e.g., AA:00:FF:00:00:00 + <vif id>.
> These wouldn't need to be unique -- they'd never be seen 'on the
> wire'.
> 
> The random MAC allocator would never pick a MAC in that range, and
> xend could ensure that manually-selected MACs never fall in that
> range and print an error if they did.
> 
> It's a shame that the backend needs a MAC different from the frontend
> -- but if it doesn't then the bridge code complains. :-(
> 
>  -- Keir
> 
> > I temporarilly solved this by converting the IP to hexidecimal and
> > setting the first 2 bytes of the mac to DE:AD. Unfortunately xen
> > increments the 3rd if you set the mac. We need a way to set the mac on
> > both ends of the vif device. 
> > 
> > Maybe we can have things this way. options...
> > a: set the mac on one end and have a standard "increment" bit somewhere
> > higher up(say first byte).
> > b: set the macs on both ends
> > c: set a specific range of macs to use
> > 
> > Either way xen really should test (not certain how exactly) to see if
> > it's already in existence prior to using a mac that is dynamicly set.
> > Not testing is playing russian roulette IMHO (as some including myself
> > have found out already).
> > 
> > Opinions?
> > 
> > On Wed, 2004-10-13 at 15:17, Keir Fraser wrote:
> > > > 
> > > > " This is *definitely* worth checking
> > > > " out at this point, and can be fixed by picking your own MAC addresses.
> > > > 
> > > > Yep.  How big a space do you randomize for mac addrs?
> > > > 
> > > > # grep -i AA:00:00:24:22:F3 a
> > > > batch009 (172.16.12.19) at AA:00:00:24:22:F3 [ether] on eth0
> > > > batch075 (172.16.12.85) at AA:00:00:24:22:F3 [ether] on eth0
> > > > 
> > > 
> > > I think AA:00:00:00:00:00 - AA:00:00:7F:FF:FF (i.e., 8 million
> > > addresses). But the random-number generator is crap.
> > > 
> > > You're definitely best off generating your own if creating any decent
> > > number of VMs.
> > > 
> > >  -- Keir
> > > 
> > > 
> > > -------------------------------------------------------
> > > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > > Use IT products in your business? Tell us what you think of them. Give us
> > > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> > > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.sourceforge.net
> > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> > 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> Use IT products in your business? Tell us what you think of them. Give us
> Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> http://productguide.itmanagersjournal.com/guidepromo.tmpl
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

> So the xen side of the vif isn't see on the wire except for a: the
> backend driver domain( usually domain-0) and that vif's front and driver
> domain eh? (I got the front and back end locations right yes?)

Noone ever sees the backend MAC address -- it is never written into
any packet. We only need to give the backend a MAC address because we
hook into the Linux networking code as a normal Ethernet interface,
and normal Ethernet interfaces need a normal MAC address. :-)
It needs to be unique because of sanity checks in the bridge that fail
if it sees a remote address == a local address that it knows about.

> Either way it goes, how would one go about verifying that the MAC is
> truly unique on it's visible lan? I'd be i very interested in this as a
> safety precaution. Right now I can't think of how to do the uniqueness
> test... 8-P if someone can give me an idea or two I could try to code it
> since it's something that I want.(brain fried this morning)

I'm pretty sure there's no way of soliciting a response from an
Ethernet host without using some higher-level protocol; probably IP or
RARP. RARP is hardly ever used, but maybe if you know the IP subnet
you could do a broadcast ping and collect the responses and look at
their source MAC addresses?

 -- Keir

> On Thu, 2004-10-14 at 03:21, Keir Fraser wrote:
> > One option is to have the backend interfaces pick their MAC addresses
> > from a well-known range: e.g., AA:00:FF:00:00:00 + <vif id>.
> > These wouldn't need to be unique -- they'd never be seen 'on the
> > wire'.
> > 
> > The random MAC allocator would never pick a MAC in that range, and
> > xend could ensure that manually-selected MACs never fall in that
> > range and print an error if they did.
> > 
> > It's a shame that the backend needs a MAC different from the frontend
> > -- but if it doesn't then the bridge code complains. :-(
> > 
> >  -- Keir
> > 
> > > I temporarilly solved this by converting the IP to hexidecimal and
> > > setting the first 2 bytes of the mac to DE:AD. Unfortunately xen
> > > increments the 3rd if you set the mac. We need a way to set the mac on
> > > both ends of the vif device. 
> > > 
> > > Maybe we can have things this way. options...
> > > a: set the mac on one end and have a standard "increment" bit somewhere
> > > higher up(say first byte).
> > > b: set the macs on both ends
> > > c: set a specific range of macs to use
> > > 
> > > Either way xen really should test (not certain how exactly) to see if
> > > it's already in existence prior to using a mac that is dynamicly set.
> > > Not testing is playing russian roulette IMHO (as some including myself
> > > have found out already).
> > > 
> > > Opinions?
> > > 
> > > On Wed, 2004-10-13 at 15:17, Keir Fraser wrote:
> > > > > 
> > > > > " This is *definitely* worth checking
> > > > > " out at this point, and can be fixed by picking your own MAC addresses.
> > > > > 
> > > > > Yep.  How big a space do you randomize for mac addrs?
> > > > > 
> > > > > # grep -i AA:00:00:24:22:F3 a
> > > > > batch009 (172.16.12.19) at AA:00:00:24:22:F3 [ether] on eth0
> > > > > batch075 (172.16.12.85) at AA:00:00:24:22:F3 [ether] on eth0
> > > > > 
> > > > 
> > > > I think AA:00:00:00:00:00 - AA:00:00:7F:FF:FF (i.e., 8 million
> > > > addresses). But the random-number generator is crap.
> > > > 
> > > > You're definitely best off generating your own if creating any decent
> > > > number of VMs.
> > > > 
> > > >  -- Keir
> > > > 
> > > > 
> > > > -------------------------------------------------------
> > > > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > > > Use IT products in your business? Tell us what you think of them. Give us
> > > > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> > > > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > > > _______________________________________________
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.sourceforge.net
> > > > https://lists.sourceforge.net/lists/listinfo/xen-devel
> > > 
> > 
> > 
> > 
> > -------------------------------------------------------
> > This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
> > Use IT products in your business? Tell us what you think of them. Give us
> > Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
> > http://productguide.itmanagersjournal.com/guidepromo.tmpl
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> 



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Brian Wolfe]

On Thu, 2004-10-14 at 10:29, Keir Fraser wrote:
> > So the xen side of the vif isn't see on the wire except for a: the
> > backend driver domain( usually domain-0) and that vif's front and driver
> > domain eh? (I got the front and back end locations right yes?)
> 
> Noone ever sees the backend MAC address -- it is never written into
> any packet. We only need to give the backend a MAC address because we
> hook into the Linux networking code as a normal Ethernet interface,
> and normal Ethernet interfaces need a normal MAC address. :-)
> It needs to be unique because of sanity checks in the bridge that fail
> if it sees a remote address == a local address that it knows about.
> 
> > Either way it goes, how would one go about verifying that the MAC is
> > truly unique on it's visible lan? I'd be i very interested in this as a
> > safety precaution. Right now I can't think of how to do the uniqueness
> > test... 8-P if someone can give me an idea or two I could try to code it
> > since it's something that I want.(brain fried this morning)
> 
> I'm pretty sure there's no way of soliciting a response from an
> Ethernet host without using some higher-level protocol; probably IP or
> RARP. RARP is hardly ever used, but maybe if you know the IP subnet
> you could do a broadcast ping and collect the responses and look at
> their source MAC addresses?
> 

Hmm, not a bad idea.... :) I know brctl can be used to query for a list
of MACs on it's physical lan that it has cached. Maybe if xend on
domain-0 did something similar using ARP and arp-reply packets that it
sees on the bridges to vet it's mac choices against, and issue a warning
if it sees a dupe?

>  -- Keir
<snip - backlog shortened >



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Ian Pratt]

> It looks like xen0 or the bridge stuff is not getting some packets
> destined for xenU.
> 
> I traced an ssh connection from a machine running stock 2.4.25
> to a xenU domain by running tcpdump on xen0 and on the other machine.
> 
> The trace on xen0 shows a 4 second gap where no ssh packets were received
> for the xenU domain even though the other machine was sending retransmits. 
> Only the last retransmit appears in the xen0 dump.
> 
> On xen0, tcpdump -i eth0 shows the 4 second gap between packets
going to xenU:


If they're not showing up here, then I think we can rule out
anything to do with the bridge code as we're looking before the
bridge. 

In fact, its hard to see how they could be dropped by anything
other than the driver, and you'd expect to see the dropped count
bumped. What driver is this?

Even if something (e.g. Xen) went wild and took the CPU away for
4 seconds, you'd wouldn't expect to loose packets as the NIC
would buffer them. 

I'll place money that you've still got some duplicate MAC
addresses out there...

Ian


-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Keir Fraser]

I think the mailing list is sending duplicates. :-)

 K.

> If they're not showing up here, then I think we can rule out
> anything to do with the bridge code as we're looking before the
> bridge. 
> 
> In fact, its hard to see how they could be dropped by anything
> other than the driver, and you'd expect to see the dropped count
> bumped. What driver is this?
> 
> Even if something (e.g. Xen) went wild and took the CPU away for
> 4 seconds, you'd wouldn't expect to loose packets as the NIC
> would buffer them. 
> 
> I'll place money that you've still got some duplicate MAC
> addresses out there...
> 
> Ian



-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl

Re: vif tx drops

[Alvin Starr]

Ian Pratt wrote:

>>It looks like xen0 or the bridge stuff is not getting some packets
>>destined for xenU.
>>
>>I traced an ssh connection from a machine running stock 2.4.25
>>to a xenU domain by running tcpdump on xen0 and on the other machine.
>>
>>The trace on xen0 shows a 4 second gap where no ssh packets were received
>>for the xenU domain even though the other machine was sending retransmits. 
>>Only the last retransmit appears in the xen0 dump.
>>
>>On xen0, tcpdump -i eth0 shows the 4 second gap between packets
>>    
>>
>going to xenU:
>
>
>If they're not showing up here, then I think we can rule out
>anything to do with the bridge code as we're looking before the
>bridge. 
>
>In fact, its hard to see how they could be dropped by anything
>other than the driver, and you'd expect to see the dropped count
>bumped. What driver is this?
>
>Even if something (e.g. Xen) went wild and took the CPU away for
>4 seconds, you'd wouldn't expect to loose packets as the NIC
>would buffer them. 
>
>I'll place money that you've still got some duplicate MAC
>addresses out there...
>
>Ian
>  
>
I am seeing a similar thing with only one xenU domain running.
The thing is that I am trying to run an ISCSI net boot. I have much 
bigger problems just now so I have not had a chance to look at what is 
going on with the dropped packets.
But I am sure I do not have an issue with a duplicate mac address. I do 
now believe anybody sihpps with a mac address of aa:00:00:00:00:0X where 
X is 1,2 or 3.

-- 
Alvin Starr                   ||   voice: (416)585-9971
Interlink Connectivity        ||   fax:   (416)785-3668
alvin@iplink.net              ||




-------------------------------------------------------
This SF.net email is sponsored by: IT Product Guide on ITManagersJournal
Use IT products in your business? Tell us what you think of them. Give us
Your Opinions, Get Free ThinkGeek Gift Certificates! Click to find out more
http://productguide.itmanagersjournal.com/guidepromo.tmpl