Network still broken, new issue as well , 7468:17a9f111fa93

Network still broken, new issue as well , 7468:17a9f111fa93

[Ted Kaczmarek]

xen_changeset : Fri Oct 21 13:51:42 2005 +0100 7468:17a9f111fa93

Tyan 2462 SMP Athlon - FC4 dom0


Not one vif get mapped to the bridge, but the bridge otherwise appear to
work fine with other nodes on the broadcast domain.


I am also seeing a new issue as well. 

I see 2 vifs for a domU that has nics = 1 , this happens on 4 out of 10
domU's. grepping my configs make me think that something has changed the
way they are interpreted, all the the domU's using mac, bridge have 2
vifs, where the one that ony have mac declared have 1 vif.


grep nics idom*
idom1:nics=1
idom10:nics = 1
idom2:nics = 1
idom3:nics = 1
idom4:nics = 1
idom5:nics = 1
idom6:nics = 1
idom7:nics = 1
idom8:nics = 1
idom9:nics = 1

 grep vif idom*
idom1:vif = ['mac=AA:00:00:00:00:17','bridge=xen-br0']
idom1:#vif = ['mac=AA:00:00:11:11:12','bridge=xen-br0',
'mac=AA:00:00:11:11:22','bridge=xen-br1',
'mac=AA:00:00:11:12:22','bridge=xen-br2']
idom1:#vif = ['mac=AA:00:00:11:11:11' , 'bridge=xen-br0' ,
'bridge=xen-br1' , 'bridge=xen-br2']
idom1:#vif = ['mac=AA:00:00:11:11:11' , 'bridge=xen-br0' ,
'bridge=xen-br1' ]
idom1:#vif = ['mac=AA:00:00:11:11:11','bridge=xen-br0' ,
'mac=AA:00:00:11:11:12', 'bridge=xen-br1']
idom1:#vif = ['mac=AA:00:00:11:11:11' 'mac=AA:00:00:11:11:12'] BAD, eth1
won't come up

idom10:#vif = [ 'mac=aa:00:66:00:01:08, bridge=xen-br0',
'mac=aa:00:66:01:01:08, bridge=xen-br1', 'mac=aa:00:66:02:01:08,
bridge=xen-br2' ]
idom10:vif = [ 'mac=AA:00:00:00:01:08, bridge=xen-br0' ]

idom2:vif = ['mac=AA:00:00:00:01:01']

idom3:vif = ['mac=AA:00:00:00:01:02' , 'bridge=xen-br0']

idom4:vif = ['mac=AA:00:00:00:01:03' , 'bridge=xen-br0']

idom5:vif = ['mac=AA:00:00"00"01:04' , 'bridge=xen-br0']

idom6:vif = [ 'mac=AA:00:00:00:01:05' ]

idom7:vif = [ 'mac=AA:00:00:00:01:07' ]

idom8:vif = [ 'mac=AA:00:00:00:01:08' ]

idom9:vif = [ 'mac=AA:00:00:00:01:09' ]

list_vm_nets_backends
idom1
(backend/local/domain/0/backend/vif/1/0)
(backend/local/domain/0/backend/vif/1/1)

idom2
(backend/local/domain/0/backend/vif/2/0)

idom3
(backend/local/domain/0/backend/vif/11/0)
(backend/local/domain/0/backend/vif/11/1)

idom4
(backend/local/domain/0/backend/vif/4/0)
(backend/local/domain/0/backend/vif/4/1)

idom5
(backend/local/domain/0/backend/vif/5/0)
(backend/local/domain/0/backend/vif/5/1)

idom6
(backend/local/domain/0/backend/vif/6/0)

idom7
(backend/local/domain/0/backend/vif/7/0)

idom8
(backend/local/domain/0/backend/vif/8/0)

idom9
(backend/local/domain/0/backend/vif/9/0)

idom10
(backend/local/domain/0/backend/vif/10/0)



I have to get caught up on the archives as I know lots of network
related changes are in progress. Would be great if we could get a little
change summary, I have tried to throw some kind of user help doc
together but you guys are like rabbits :-)

IMHO the configs should start to get stricter as well, right now one can
get away with spaces anywhere it seems, and until today it didn't matter
if I specified mac or mac/bridge. Same goes for many other options such
as cpu = -1, etc.. In the past I have always seen this come back to bite
you in the a@@. If the rules for the config get tighter, their is less
room for error across the board.

Regards,
Ted

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Ewan Mellor]

On Fri, Oct 21, 2005 at 10:26:30PM -0400, Ted Kaczmarek wrote:

> I see 2 vifs for a domU that has nics = 1 , this happens on 4 out of 10
> domU's. grepping my configs make me think that something has changed the
> way they are interpreted, all the the domU's using mac, bridge have 2
> vifs, where the one that ony have mac declared have 1 vif.
> 
> [Snip nics]
>
>  grep vif idom*

> idom1:vif = ['mac=AA:00:00:00:00:17','bridge=xen-br0']
> [Snip]
> idom10:vif = [ 'mac=AA:00:00:00:01:08, bridge=xen-br0' ]

Here, idom1 is declared using two strings, but idom10 is declared using just
one.  I'm guessing a little here, but from reading the code it looks like this
means that idom1 has two vifs, one with a default bridge and the given MAC,
and one with a random MAC and the given bridge, whereas idom10 has one vif,
with the given MAC and bridge.

I think this is fairly sensible behaviour.  Whether it is a regression or not,
I'm not sure.  It might actually have been a long-standing bug that has been
masked by other long standing bugs, and now we're doing things as intended.

The vifs= setting seems to only apply if the number of entries in vif= is less
than the number specified in vifs=.  In that case, the rest of the vifs are
created, with default details.  This is fairly barbaric, as you've found out.
We probably ought to have a "extra_vifs" setting or something, to make it
clear that they are separate from those configured using vif= and to make it
clear that this number is not a limit, to avoid the kind of confusion you've
suffered.

That said, I'm pushing back on configuration file format changes at the
moment, because we are trying to stabilise for release, and though the format
is pretty hideous, many people have stable configurations now, and I don't
want to break them without strong justification.

> idom5:vif = ['mac=AA:00:00"00"01:04' , 'bridge=xen-br0']

This MAC has quotation marks in it.  Lord only knows what that does, but it
can't be doing what you want, so I'd fix that ;-)


Regarding your other network problems, if you are still suffering once you've
changed your configuration, then please submit a full set of logs.  The
scripts involved have changed recently, as you know, but they are very
difficult to test, because we don't have a great variation of network
topologies available.  I'd be looking for

ifconfig
brctl show
cat /var/log/syslog
cat /var/log/debug
cat /var/log/xend.log

Ewan.

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Ted Kaczmarek]

I have to run a very old changeset just to boot my SMP dom0, opened up a
bug on this a few days ago. Ever since 2.6.12.5 version started showing
up. Have tried many changesets with no joy. On my UP machine things
behave better in general, matter of fast I have a domU up for almost 5
days now with only minor issues on the UP P4 test box, but the
networking testing is limited by the APCI issues.

http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=359

I can understand the issues with configs and the points are valid. My
experience in the past has taught me that stricter configs actually lead
to less human error, less bugs in the code,  and less ghost chasing by
all parties involved, which in turn leads to higher productivity in
general.

I intentionally throw lots of variations in my configs for the purpose
of learning and testing, it is a double edged sword to be sure.

Regards,
Ted
 


On Tue, 2005-10-25 at 18:40 +0100, Ewan Mellor wrote:
> On Fri, Oct 21, 2005 at 10:26:30PM -0400, Ted Kaczmarek wrote:
> 
> > I see 2 vifs for a domU that has nics = 1 , this happens on 4 out of 10
> > domU's. grepping my configs make me think that something has changed the
> > way they are interpreted, all the the domU's using mac, bridge have 2
> > vifs, where the one that ony have mac declared have 1 vif.
> > 
> > [Snip nics]
> >
> >  grep vif idom*
> 
> > idom1:vif = ['mac=AA:00:00:00:00:17','bridge=xen-br0']
> > [Snip]
> > idom10:vif = [ 'mac=AA:00:00:00:01:08, bridge=xen-br0' ]
> 
> Here, idom1 is declared using two strings, but idom10 is declared using just
> one.  I'm guessing a little here, but from reading the code it looks like this
> means that idom1 has two vifs, one with a default bridge and the given MAC,
> and one with a random MAC and the given bridge, whereas idom10 has one vif,
> with the given MAC and bridge.
> 
> I think this is fairly sensible behaviour.  Whether it is a regression or not,
> I'm not sure.  It might actually have been a long-standing bug that has been
> masked by other long standing bugs, and now we're doing things as intended.
> 
> The vifs= setting seems to only apply if the number of entries in vif= is less
> than the number specified in vifs=.  In that case, the rest of the vifs are
> created, with default details.  This is fairly barbaric, as you've found out.
> We probably ought to have a "extra_vifs" setting or something, to make it
> clear that they are separate from those configured using vif= and to make it
> clear that this number is not a limit, to avoid the kind of confusion you've
> suffered.
> 
> That said, I'm pushing back on configuration file format changes at the
> moment, because we are trying to stabilise for release, and though the format
> is pretty hideous, many people have stable configurations now, and I don't
> want to break them without strong justification.
> 
> > idom5:vif = ['mac=AA:00:00"00"01:04' , 'bridge=xen-br0']
> 
> This MAC has quotation marks in it.  Lord only knows what that does, but it
> can't be doing what you want, so I'd fix that ;-)
> 
> 
> Regarding your other network problems, if you are still suffering once you've
> changed your configuration, then please submit a full set of logs.  The
> scripts involved have changed recently, as you know, but they are very
> difficult to test, because we don't have a great variation of network
> topologies available.  I'd be looking for
> 
> ifconfig
> brctl show
> cat /var/log/syslog
> cat /var/log/debug
> cat /var/log/xend.log
> 
> Ewan.

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Paul Larson]

Ewan Mellor wrote:
> Regarding your other network problems, if you are still suffering once you've
> changed your configuration, then please submit a full set of logs.  The
> scripts involved have changed recently, as you know, but they are very
> difficult to test, because we don't have a great variation of network
> topologies available.  I'd be looking for
>
> ifconfig
> brctl show
> cat /var/log/syslog
> cat /var/log/debug
> cat /var/log/xend.log
>   
Please see bug #363 that's currently biting several of us with systems 
which, for various reasons, use eth1 instead of eth0.  I hope you don't 
mind, I attached logfiles and command output to the bug rather than 
spamming here with it.  We have always had to workaround this it seems, 
but were hoping the new script would fix it.  Instead, with the new 
script we just had to find a new workaround.

Basicially, on these systems it assumes eth0 (which we don't use), so we 
never get networking on domU.

Thanks,
Paul Larson

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Ewan Mellor]

On Tue, Oct 25, 2005 at 05:28:17PM -0500, Paul Larson wrote:

> Please see bug #363 that's currently biting several of us with systems 
> which, for various reasons, use eth1 instead of eth0.  I hope you don't 
> mind, I attached logfiles and command output to the bug rather than 
> spamming here with it.

I don't mind that at all, no.  In fact, it's much better for me if the logs
are on the bug, and drawing attention to a new bug on the list is good too.

> We have always had to workaround this it seems, 
> but were hoping the new script would fix it.  Instead, with the new 
> script we just had to find a new workaround.
> 
> Basicially, on these systems it assumes eth0 (which we don't use), so we 
> never get networking on domU.

I think you want to set, in /etc/xen/xend-config.sxp,

(network-script blade-network)

and then create a file /etc/xen/scripts/blade-network that looks like:

#!/bin/sh

dir=$(dirname "$0")
exec "$dir/network-bridge" "$@" "netdev=eth1"

Don't forget to chmod +x /etc/xen/scripts/blade-network, too.

Hopefully, that will do the trick.  Let me know how you get on.

Ewan.

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Michael Lessard]

[-- Attachment #1.1: Type: text/plain, Size: 1567 bytes --]

On 10/25/05, Ewan Mellor <ewan@xensource.com> wrote:
>
> On Tue, Oct 25, 2005 at 05:28:17PM -0500, Paul Larson wrote:
>
> > Please see bug #363 that's currently biting several of us with systems
> > which, for various reasons, use eth1 instead of eth0. I hope you don't
> > mind, I attached logfiles and command output to the bug rather than
> > spamming here with it.
>
> I don't mind that at all, no. In fact, it's much better for me if the logs
> are on the bug, and drawing attention to a new bug on the list is good
> too.
>
> > We have always had to workaround this it seems,
> > but were hoping the new script would fix it. Instead, with the new
> > script we just had to find a new workaround.
> >
> > Basicially, on these systems it assumes eth0 (which we don't use), so we
> > never get networking on domU.
>
> I think you want to set, in /etc/xen/xend-config.sxp,
>
> (network-script blade-network)
>
> and then create a file /etc/xen/scripts/blade-network that looks like:
>
> #!/bin/sh
>
> dir=$(dirname "$0")
> exec "$dir/network-bridge" "$@" "netdev=eth1"
>
> Don't forget to chmod +x /etc/xen/scripts/blade-network, too.
>
> Hopefully, that will do the trick. Let me know how you get on.
>
> Ewan.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Bigelow:/etc/xen/scripts# ./blade-network start
/etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file or
directory

I use Debian Sid ....

[-- Attachment #1.2: Type: text/html, Size: 2108 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Ewan Mellor]

On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote:

> Bigelow:/etc/xen/scripts# ./blade-network start
> /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file or
> directory
> 
> I use Debian Sid ....

There's a patch for that on the way.  For now, you could just comment out that
whole line -- it's SuSE-specific.

Did the script work anyway?  For me, that error is not a fatal one, and the
configuration still succeeds.

Ewan.

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Michael Lessard]

[-- Attachment #1.1: Type: text/plain, Size: 692 bytes --]

On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote:
>
> On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote:
>
> > Bigelow:/etc/xen/scripts# ./blade-network start
> > /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file or
> > directory
> >
> > I use Debian Sid ....
>
> There's a patch for that on the way. For now, you could just comment out
> that
> whole line -- it's SuSE-specific.
>
> Did the script work anyway? For me, that error is not a fatal one, and the
> configuration still succeeds.
>
> Ewan.


I commend out the line, now when i start the script no error reported
but i couldn't ping dom0 from domU, with eth1 bridge too :(

[-- Attachment #1.2: Type: text/html, Size: 1057 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Ewan Mellor]

On Wed, Oct 26, 2005 at 08:14:41AM -0400, Michael Lessard wrote:

> On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote:
> 
>   On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote:
> 
>   > Bigelow:/etc/xen/scripts# ./blade-network start
>   > /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file
>   or
>   > directory
>   >
>   > I use Debian Sid ....
> 
>   There's a patch for that on the way.  For now, you could just comment
>   out that
>   whole line -- it's SuSE-specific.
> 
>   Did the script work anyway?  For me, that error is not a fatal one, and
>   the
>   configuration still succeeds.
> 
>   Ewan.
> 
> I commend out  the  line,  now  when i start the script no error reported
> but i couldn't ping dom0 from domU, with eth1 bridge too :(

So could you repost the output from brctl show and ifconfig, just as you did
for bug #363 before?  Also, it would be worth having route and iptables -L.

It would also be helpful to know whether you can get packets anywhere -- dom0
to network, network to dom0, domU to network, etc, and if you can figure out
where they are getting lost.

Cheers,

Ewan.

Re: Network still broken, new issue as well , 7468:17a9f111fa93

[Michael Lessard]

[-- Attachment #1.1: Type: text/plain, Size: 4062 bytes --]

On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote:
>
> On Wed, Oct 26, 2005 at 08:14:41AM -0400, Michael Lessard wrote:
>
> > On 10/26/05, Ewan Mellor <ewan@xensource.com> wrote:
> >
> > On Tue, Oct 25, 2005 at 08:10:16PM -0400, Michael Lessard wrote:
> >
> > > Bigelow:/etc/xen/scripts# ./blade-network start
> > > /etc/xen/scripts/network-bridge: line 211: /sbin/getcfg: No such file
> > or
> > > directory
> > >
> > > I use Debian Sid ....
> >
> > There's a patch for that on the way. For now, you could just comment
> > out that
> > whole line -- it's SuSE-specific.
> >
> > Did the script work anyway? For me, that error is not a fatal one, and
> > the
> > configuration still succeeds.
> >
> > Ewan.
> >
> > I commend out the line, now when i start the script no error reported
> > but i couldn't ping dom0 from domU, with eth1 bridge too :(
>
> So could you repost the output from brctl show and ifconfig, just as you
> did
> for bug #363 before? Also, it would be worth having route and iptables -L.
>
> It would also be helpful to know whether you can get packets anywhere --
> dom0
> to network, network to dom0, domU to network, etc, and if you can figure
> out
> where they are getting lost.
>
> Cheers,
>
> Ewan.
>

I have two network cards on my computer :
eth0 connect to -> FW (DMZ 192.168.100.254 <http://192.168.100.254>)
eth1 connect to -> FW (internal network 192.168.10.254<http://192.168.10.254>
)

On my DOM0 i haved configure network
eth0 : up (no ip)
eth1 : 192.168.10.20 <http://192.168.10.20>

On my DOMU i haved configure network
eth1 : 192.168.100.1 <http://192.168.100.1>

ON DOM0
#brctl show
xenbr0 8000.feffffffffff no vif8.0
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif1.0
ACCEPT all -- anywhere anywhere PHYSDEV match --physdev-in vif8.0

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

# ifconfig
eth0 Link encap:Ethernet HWaddr 00:08:54:2F:EA:38
inet addr:192.168.100.5 <http://192.168.100.5>
Bcast:192.168.100.255<http://192.168.100.255>Mask:
255.255.255.0 <http://255.255.255.0>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:13 errors:0 dropped:0 overruns:0 frame:0
TX packets:87 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:894 (894.0 b) TX bytes:16522 (16.1 KiB)
Interrupt:17 Base address:0xec00

eth1 Link encap:Ethernet HWaddr 00:08:54:2F:EA:32
inet addr:192.168.10.20 <http://192.168.10.20>
Bcast:192.168.10.255<http://192.168.10.255>Mask:
255.255.255.0 <http://255.255.255.0>
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:7597 errors:0 dropped:0 overruns:0 frame:0
TX packets:4755 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:963119 (940.5 KiB) TX bytes:667126 (651.4 KiB)
Interrupt:18 Base address:0xe800

lo Link encap:Local Loopback
inet addr:127.0.0.1 <http://127.0.0.1> Mask:255.0.0.0 <http://255.0.0.0>
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:76 errors:0 dropped:0 overruns:0 frame:0
TX packets:76 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5064 (4.9 KiB) TX bytes:5064 (4.9 KiB)

vif8.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:72 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3024 (2.9 KiB) TX bytes:0 (0.0 b)

xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:72 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2016 (1.9 KiB) TX bytes:0 (0.0 b)


I haved put ip on eth0 to test connectivty from etho on DOM0 and its working
...
I could ping FW from DOM0
but i could'nt ping DOM0 from DOMU

[-- Attachment #1.2: Type: text/html, Size: 7971 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel