help with bugs

help with bugs

[Ian Pratt]

I'd like to appeal for some help tracking down a couple of bugs that
we're struggling to reproduce:

 BUG62  eth0 -> veth0 in network script can loose network
 BUG130 time running fast bug
 BUG76  shared irq's fail under high load

These are all pretty serious and it would be good to get fixed before
3.0-testing-r1 

If you can make them exhibit frequently on your system it would be
useful to know.

Thanks,
Ian

Re: help with bugs

[Anthony Liguori]

Ian Pratt wrote:

> BUG76  shared irq's fail under high load
>  
>
I can repro this very easily.  If there's anything I can do to help 
track this down let me know.

Regards,

Anthony Liguori

>These are all pretty serious and it would be good to get fixed before
>3.0-testing-r1 
>
>If you can make them exhibit frequently on your system it would be
>useful to know.
>
>Thanks,
>Ian
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

Re: help with bugs

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 1354 bytes --]

On Thu, Aug 04, 2005 at 04:04:34PM +0100, Ian Pratt wrote:
> 
> 
> I'd like to appeal for some help tracking down a couple of bugs that
> we're struggling to reproduce:
> 
>  BUG62  eth0 -> veth0 in network script can loose network
I can make this bug come and go at will based on which of the 2 network
interfaces are part of the bridge.  I added that information into the
bugzilla bug, hopefully that helps.

>  BUG130 time running fast bug
>  BUG76  shared irq's fail under high load
> 
> These are all pretty serious and it would be good to get fixed before
> 3.0-testing-r1 
> 
> If you can make them exhibit frequently on your system it would be
> useful to know.
> 
> Thanks,
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

-- 
__________________________________________________________________

Sean Dague                                       Mid-Hudson Valley
sean at dague dot net                            Linux Users Group
http://dague.net                                 http://mhvlug.org

There is no silver bullet.  Plus, werewolves make better neighbors
than zombies, and they tend to keep the vampire population down.
__________________________________________________________________

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help with bugs

[David F Barrera]

Ian Pratt wrote:

>I'd like to appeal for some help tracking down a couple of bugs that
>we're struggling to reproduce:
>
> BUG62  eth0 -> veth0 in network script can loose network
>  
>
I've been able to reproduce this problem frequently on SLES 9 SP2 based 
platforms, x86 and x86_64. It seems that when I first reported the 
problem it happened very infrequently, and I could not reliably 
reproduce it. Now, I seems to be happening all the time on my SLES 9 
boxes; curiously, it does not seem to happen on my FC/RH boxes.

> BUG130 time running fast bug
> BUG76  shared irq's fail under high load
>
>These are all pretty serious and it would be good to get fixed before
>3.0-testing-r1 
>
>If you can make them exhibit frequently on your system it would be
>useful to know.
>
>Thanks,
>Ian
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
>  
>

RE: help with bugs

[Ian Pratt]

> >I'd like to appeal for some help tracking down a couple of bugs that 
> >we're struggling to reproduce:
> >
> > BUG62  eth0 -> veth0 in network script can loose network
> >  
> >
> I've been able to reproduce this problem frequently on SLES 9 
> SP2 based platforms, x86 and x86_64. It seems that when I 
> first reported the problem it happened very infrequently, and 
> I could not reliably reproduce it. Now, I seems to be 
> happening all the time on my SLES 9 boxes; curiously, it does 
> not seem to happen on my FC/RH boxes.

Is there anything unusal about the SLES9 setup, e.g. multiple phsical
interfaces, alias addresses on interfaces etc?

Have you tried my suggestion of adding 'promisc' to the ifconfig up
lines in the script?

Thanks,
Ian

Re: help with bugs

[David F Barrera]

Ian Pratt wrote:

>>>I'd like to appeal for some help tracking down a couple of bugs that 
>>>we're struggling to reproduce:
>>>
>>>BUG62  eth0 -> veth0 in network script can loose network
>>> 
>>>
>>>      
>>>
>>I've been able to reproduce this problem frequently on SLES 9 
>>SP2 based platforms, x86 and x86_64. It seems that when I 
>>first reported the problem it happened very infrequently, and 
>>I could not reliably reproduce it. Now, I seems to be 
>>happening all the time on my SLES 9 boxes; curiously, it does 
>>not seem to happen on my FC/RH boxes.
>>    
>>
>
>Is there anything unusal about the SLES9 setup, e.g. multiple phsical
>interfaces, alias addresses on interfaces etc?
>  
>
No, nothing unusual. In fact, in one case, I have a machine that has 
both SLES 9 and FC3. When I build Xen on SLES 9 and boot Dom0, I can 
recreate the problem. On the same machine, when I build Xen on FC3 and 
boot Dom0, the problem does not occur. I built Xen the same way in both 
instances; the setups are the same with regards to Xen, the only 
variable (and a big one) is the distro.

>Have you tried my suggestion of adding 'promisc' to the ifconfig up
>lines in the script?
>  
>
Yes, and the problem still occurred.

>Thanks,
>Ian
>
>  
>

RE: help with bugs

[Ian Pratt]

> >  BUG62  eth0 -> veth0 in network script can loose network
> I can make this bug come and go at will based on which of the 
> 2 network interfaces are part of the bridge.  I added that 
> information into the bugzilla bug, hopefully that helps.

Are you changing the default 'netdev' at the top of the network script?

I take it that if you try and get it to work on eth1 it fails?

BTW: I'd like to see a few changes in the way this stuff works anyhow.

Firstly, rename network to network-bridge.

Next, I'd make it such that it's possible to have multiple
network-script lines, each with parameters e.g. something like:

(network-script ( network-bridge ( bridge xen-br0 ) ( netdev eth0 ) ) )
(network-script ( network-bridge ( bridge xen-br1 ) ( netdev eth1 ) ) )

[having multiple interfaces should result in multiple vif0.x and vethX
devices]

And then the vif-script along with default parameters e.g.

( vif-script ( vif-bridge ( bridge xen-br0 ) ( antispoof no ) ) )

Do others agree?
Could someone work up a patch?

Thanks,
Ian

 

Re: help with bugs

[Paul Larson]

[-- Attachment #1.1: Type: text/plain, Size: 1553 bytes --]

On Thu, 2005-08-04 at 16:29 -0500, David F Barrera wrote:
> 
> Ian Pratt wrote:
> 
> >>>I'd like to appeal for some help tracking down a couple of bugs that 
> >>>we're struggling to reproduce:
> >>>
> >>>BUG62  eth0 -> veth0 in network script can loose network
> >>> 
> >>>
> >>>      
> >>>
> >>I've been able to reproduce this problem frequently on SLES 9 
> >>SP2 based platforms, x86 and x86_64. It seems that when I 
> >>first reported the problem it happened very infrequently, and 
> >>I could not reliably reproduce it. Now, I seems to be 
> >>happening all the time on my SLES 9 boxes; curiously, it does 
> >>not seem to happen on my FC/RH boxes.
> >>    
> >>
> >
> >Is there anything unusal about the SLES9 setup, e.g. multiple phsical
> >interfaces, alias addresses on interfaces etc?
> >  
> >
> No, nothing unusual. In fact, in one case, I have a machine that has 
> both SLES 9 and FC3. When I build Xen on SLES 9 and boot Dom0, I can 
> recreate the problem. On the same machine, when I build Xen on FC3 and 
> boot Dom0, the problem does not occur. I built Xen the same way in both 
> instances; the setups are the same with regards to Xen, the only 
> variable (and a big one) is the distro.
David, is this one of the boxes that has to use eth1 instead of eth0?  I
have some like this and haven't seen the problem you describe, even with
SLES9, so I don't know if either of those is important to the problem.

-- 
Thanks,
Paul Larson
plars@linuxtestproject.org
http://www.linuxtestproject.org

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help with bugs

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 1020 bytes --]

On Thu, Aug 04, 2005 at 10:48:50PM +0100, Ian Pratt wrote:
> > >  BUG62  eth0 -> veth0 in network script can loose network
> > I can make this bug come and go at will based on which of the 
> > 2 network interfaces are part of the bridge.  I added that 
> > information into the bugzilla bug, hopefully that helps.
> 
> Are you changing the default 'netdev' at the top of the network script?

No actually, I guess I'm not even clear why veth0 exists, as everything
works quite nicely for me without it functioning.

	-Sean

-- 
__________________________________________________________________

Sean Dague                                       Mid-Hudson Valley
sean at dague dot net                            Linux Users Group
http://dague.net                                 http://mhvlug.org

There is no silver bullet.  Plus, werewolves make better neighbors
than zombies, and they tend to keep the vampire population down.
__________________________________________________________________

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: help with bugs

[David F Barrera]

Paul Larson wrote:

>On Thu, 2005-08-04 at 16:29 -0500, David F Barrera wrote:
>  
>
>>Ian Pratt wrote:
>>
>>    
>>
>>>>>I'd like to appeal for some help tracking down a couple of bugs that 
>>>>>we're struggling to reproduce:
>>>>>
>>>>>BUG62  eth0 -> veth0 in network script can loose network
>>>>>
>>>>>
>>>>>     
>>>>>
>>>>>          
>>>>>
>>>>I've been able to reproduce this problem frequently on SLES 9 
>>>>SP2 based platforms, x86 and x86_64. It seems that when I 
>>>>first reported the problem it happened very infrequently, and 
>>>>I could not reliably reproduce it. Now, I seems to be 
>>>>happening all the time on my SLES 9 boxes; curiously, it does 
>>>>not seem to happen on my FC/RH boxes.
>>>>   
>>>>
>>>>        
>>>>
>>>Is there anything unusal about the SLES9 setup, e.g. multiple phsical
>>>interfaces, alias addresses on interfaces etc?
>>> 
>>>
>>>      
>>>
>>No, nothing unusual. In fact, in one case, I have a machine that has 
>>both SLES 9 and FC3. When I build Xen on SLES 9 and boot Dom0, I can 
>>recreate the problem. On the same machine, when I build Xen on FC3 and 
>>boot Dom0, the problem does not occur. I built Xen the same way in both 
>>instances; the setups are the same with regards to Xen, the only 
>>variable (and a big one) is the distro.
>>    
>>
>David, is this one of the boxes that has to use eth1 instead of eth0?  I
>have some like this and haven't seen the problem you describe, even with
>SLES9, so I don't know if either of those is important to the problem.
>  
>
No, this box only has one nic, eth0. I should point out that my machines 
have SLES 9 *SP2*, the latest GA version of SLES 9. Come to think of it, 
the problem did not happen on it when it was just SLES 9. Can you update 
your system to SLES 9 SP2?

>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>  
>

Re: help with bugs

[Gerd Knorr]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:

> I'd like to appeal for some help tracking down a couple of bugs that
> we're struggling to reproduce:
> 
>  BUG62  eth0 -> veth0 in network script can loose network

I think the only sane way to fix this is to let the distribution tools
configure the network.  Thats a bit harder to set up, but works more
reliable.  Also the "if{up|down} <interface>" commands and the like
will work as usual then.  Especially in case eth0 is configured via
dhcp the ip address copying is a bad idea.  Unfortunaly it isn't very
good documented how all this works, especially the new veth0 thing.

IMHO it would be good if the network start script checks whenever any
bridges are already present in the system and don't touch the network
setup if that is the case.  That should catch both network setup being
already done by the distro start scripts or by an earlier network
setup script run (when xend is restarted).

The setup I'm running looks like this (classic 2.x setup, no
veth0/vid0.0 used), in boot.local:

	ip link set eth0 name hw-eth0
	brctl addbr eth0
	brctl addif eth0 hw-eth0
	ip link set hw-eth0 up
	ip link set eth0 up

Then let the network scripts setup eth0 (now a bridge) as usual and
tell xend that "eth0" is the bridge device it should add the vif
interfaces to.

cheers,

  Gerd

-- 
panic("it works"); /* avoid being flooded with debug messages */

RE: help with bugs

[Ian Pratt]

> > I'd like to appeal for some help tracking down a couple of 
> bugs that 
> > we're struggling to reproduce:
> > 
> >  BUG62  eth0 -> veth0 in network script can loose network
> 
> I think the only sane way to fix this is to let the 
> distribution tools configure the network.  Thats a bit harder 
> to set up, but works more reliable.  

Yep, having the distro scripts do it always preferable. However, we do
need something that people can use in the interim. 

I guess people would still want the distro scripts to be able to cope
with enabling/disabling xen networking at run time rather than just
configuring it once at boot. 

> Also the "if{up|down} 
> <interface>" commands and the like will work as usual then.  
> Especially in case eth0 is configured via dhcp the ip address 
> copying is a bad idea.  Unfortunaly it isn't very good 
> documented how all this works, especially the new veth0 thing.

I'd be inclined to rename the physical interface to pethN, and create
veth0 as eth0. I think this should solve the DHCP issue.

> IMHO it would be good if the network start script checks 
> whenever any bridges are already present in the system and 
> don't touch the network setup if that is the case.  That 
> should catch both network setup being already done by the 
> distro start scripts or by an earlier network setup script 
> run (when xend is restarted).

I believe (hope?) it already copes wih xend restart.
If you're using distro or other scripts then you can disable it from
xend-config.sxp.

> The setup I'm running looks like this (classic 2.x setup, no 
> veth0/vid0.0 used), in boot.local:
> 
> 	ip link set eth0 name hw-eth0
> 	brctl addbr eth0
> 	brctl addif eth0 hw-eth0
> 	ip link set hw-eth0 up
> 	ip link set eth0 up
> 
> Then let the network scripts setup eth0 (now a bridge) as 
> usual and tell xend that "eth0" is the bridge device it 
> should add the vif interfaces to.

So you don't use veth0? This is dangerous if you run services in dom0
that are being used by the domUs.

Ian

Re: help with bugs

[Gerd Knorr]

> I'd be inclined to rename the physical interface to pethN, and create
> veth0 as eth0. I think this should solve the DHCP issue.

Hmm, just an idea, not sure how well that would work in
practice, maybe the network script should not even attempt to
copy the ip addresses, but instead use ifup and ifdown, i.e.
something like this:

	ifdown eth0
	ip link set eth0 name peth0
	ip link set veth0 name eth0
	brctl addbr xen-br0
	brctl addif xen-br0 peth0
	brctl addif xen-br0 vif0.0
	ip link set peth0 up
	ip link set vif0.0 up
	ip link set xen-br0 up
	ifup eth0

At least suse, fedora and debian have ifup and ifdown to control
interfaces, not sure about others ...

> So you don't use veth0?

No.  Partly because it's not entriely clear to me how the setup
should look like (hope I got it right above), partly because the
same setup works for both xen2 and xen3 then.  And, no, I don't
run services for the domUs on dom0 ...

  Gerd

-- 
panic("it works"); /* avoid being flooded with debug messages */

RE: help with bugs

[Ian Pratt]

> Hmm, just an idea, not sure how well that would work in 
> practice, maybe the network script should not even attempt to 
> copy the ip addresses, but instead use ifup and ifdown, i.e.
> something like this:

Do you think ifup/ifdown will do the right thing with the interface name
changed underneath them?

If so, that's a good soloution and we should adopt it. Please can
someone try?

Thanks,
Ian 
 
> 	ifdown eth0
> 	ip link set eth0 name peth0
> 	ip link set veth0 name eth0
> 	brctl addbr xen-br0
> 	brctl addif xen-br0 peth0
> 	brctl addif xen-br0 vif0.0
> 	ip link set peth0 up
> 	ip link set vif0.0 up
> 	ip link set xen-br0 up
> 	ifup eth0
> 
> At least suse, fedora and debian have ifup and ifdown to 
> control interfaces, not sure about others ...
> 
> > So you don't use veth0?
> 
> No.  Partly because it's not entriely clear to me how the 
> setup should look like (hope I got it right above), partly 
> because the same setup works for both xen2 and xen3 then.  
> And, no, I don't run services for the domUs on dom0 ...
> 
>   Gerd
> 
> --
> panic("it works"); /* avoid being flooded with debug messages */
> 

Re: help with bugs

[Nivedita Singhvi]

Gerd Knorr wrote:
>>I'd be inclined to rename the physical interface to pethN, and create
>>veth0 as eth0. I think this should solve the DHCP issue.
> 
> 
> Hmm, just an idea, not sure how well that would work in
> practice, maybe the network script should not even attempt to
> copy the ip addresses, but instead use ifup and ifdown, i.e.
> something like this:
> 
> 	ifdown eth0
> 	ip link set eth0 name peth0
> 	ip link set veth0 name eth0
> 	brctl addbr xen-br0
> 	brctl addif xen-br0 peth0
> 	brctl addif xen-br0 vif0.0
> 	ip link set peth0 up
> 	ip link set vif0.0 up
> 	ip link set xen-br0 up
> 	ifup eth0

Unfortunately, we might not be able to do this (we do not
want to force a rename of the physical device). Looking into
this...

thanks,
Nivedita

> At least suse, fedora and debian have ifup and ifdown to control
> interfaces, not sure about others ...
> 
> 
>>So you don't use veth0?
> 
> 
> No.  Partly because it's not entriely clear to me how the setup
> should look like (hope I got it right above), partly because the
> same setup works for both xen2 and xen3 then.  And, no, I don't
> run services for the domUs on dom0 ...
> 
>   Gerd
> 

RE: help with bugs

[Ian Pratt]

 
> Unfortunately, we might not be able to do this (we do not 
> want to force a rename of the physical device). Looking into this...

See the new network script I posted yesterday. Works fine for me on a
number of different boxes.

Ian

Re: help with bugs

[Nivedita Singhvi]

Ian Pratt wrote:

>>Unfortunately, we might not be able to do this (we do not 
>>want to force a rename of the physical device). Looking into this...
> 
> 
> See the new network script I posted yesterday. Works fine for me on a
> number of different boxes.

Ian, yup, will have feedback by tmrw.


thanks,
Nivedita