RE: nfsroot and brige

RE: nfsroot and brige

[Ian Pratt]

> hello,
>  	it seems like a catch-22 to me. I boot xen0 with nfsroot,
>  	and then try to setup bridge stuff. something like this
> 
> /sbin/brctl addbr xen-br0
> /sbin/brctl addif xen-br0 eth0
> /sbin/ifconfig xen-br0 10.0.10.103 netmask 255.255.255.0 up
> /sbin/ifconfig eth0 0.0.0.0 promisc up
> 
>  	however, once I start it running it crashes down with error
>  	that nfs server is not available.
> 
>  	does it mean that I must use initramfs to configure bridge
>  	first, or is there some other way?

It really bugs me that there's no atomic way to configure a bridge and
transfer an IP address already active on another interface to it. I'm
not 100% convinced that the current behaviour is actually what the
bridge code author's intend: a the host can still transmit packet's with
an IP associated with an eth interface that is part of a bridge, it just
can't receive packets on that IP. I'd certainly appreciate it if someone
could ask the question on the bridge mailing list...

The workarounds for using nfsroot in dom0 at the moment are: a) use a
ramfs to configure the bridge; b) use a routed rather than bridged
setup.

I'm actually going off the whole idea of bridging rather than routeing
as the default...

Ian


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

nfsroot and brige

[Adam Sulmicki]

hello,
 	it seems like a catch-22 to me. I boot xen0 with nfsroot,
 	and then try to setup bridge stuff. something like this

/sbin/brctl addbr xen-br0
/sbin/brctl addif xen-br0 eth0
/sbin/ifconfig xen-br0 10.0.10.103 netmask 255.255.255.0 up
/sbin/ifconfig eth0 0.0.0.0 promisc up

 	however, once I start it running it crashes down with error
 	that nfs server is not available.

 	does it mean that I must use initramfs to configure bridge
 	first, or is there some other way?


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: nfsroot and brige

[Ronald G. Minnich]

On Wed, 12 Jan 2005, Ian Pratt wrote:

> I'm actually going off the whole idea of bridging rather than routeing
> as the default...

I am with you. I don't think the bridge thing has worked out. I thought it 
was just me, but then I watched this list and realized it's a lot of 
people having trouble with it.

ron


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: nfsroot and brige

[Adam Heath]

On Wed, 12 Jan 2005, Ian Pratt wrote:

> > hello,
> >  	it seems like a catch-22 to me. I boot xen0 with nfsroot,
> >  	and then try to setup bridge stuff. something like this
> >
> > /sbin/brctl addbr xen-br0
> > /sbin/brctl addif xen-br0 eth0
> > /sbin/ifconfig xen-br0 10.0.10.103 netmask 255.255.255.0 up
> > /sbin/ifconfig eth0 0.0.0.0 promisc up
> >
> >  	however, once I start it running it crashes down with error
> >  	that nfs server is not available.
> >
> >  	does it mean that I must use initramfs to configure bridge
> >  	first, or is there some other way?
>
> It really bugs me that there's no atomic way to configure a bridge and
> transfer an IP address already active on another interface to it. I'm
> not 100% convinced that the current behaviour is actually what the
> bridge code author's intend: a the host can still transmit packet's with
> an IP associated with an eth interface that is part of a bridge, it just
> can't receive packets on that IP. I'd certainly appreciate it if someone
> could ask the question on the bridge mailing list...
>
> The workarounds for using nfsroot in dom0 at the moment are: a) use a
> ramfs to configure the bridge; b) use a routed rather than bridged
> setup.
>
> I'm actually going off the whole idea of bridging rather than routeing
> as the default...

Actually, our setup here uses proxy_arp, which is close to bridging anyways.


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Keir Fraser]

> 
> 
> On Wed, 12 Jan 2005, Ian Pratt wrote:
> 
> > I'm actually going off the whole idea of bridging rather than routeing
> > as the default...
> 
> I am with you. I don't think the bridge thing has worked out. I thought it 
> was just me, but then I watched this list and realized it's a lot of 
> people having trouble with it.

Yes, I think we should try routing instead. Should be easy to concoct
some replacement scripts?

Also then no need for etherbridge patch to our 2.4 kernel. :-)

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Ronald G. Minnich]

On Wed, 12 Jan 2005, Keir Fraser wrote:

> Yes, I think we should try routing instead. Should be easy to concoct
> some replacement scripts?

I could not agree more. I think the bridge idea was interesting but I feel 
that it is not working out.

ron


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Nivedita Singhvi]

Keir Fraser wrote:

>>>I'm actually going off the whole idea of bridging rather than routeing
>>>as the default...
>>
>>I am with you. I don't think the bridge thing has worked out. I thought it 
>>was just me, but then I watched this list and realized it's a lot of 
>>people having trouble with it.
> 
> 
> Yes, I think we should try routing instead. Should be easy to concoct
> some replacement scripts?
> 
> Also then no need for etherbridge patch to our 2.4 kernel. :-)

Amen!! Thank you! The bridging stuff approach was causing a
lot of pain. Routing will be a little more flexible.

thanks,
Nivedita


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Leigh Brown]

Keir Fraser said:
>>
>>
>> On Wed, 12 Jan 2005, Ian Pratt wrote:
>>
>> > I'm actually going off the whole idea of bridging rather than routeing
>> > as the default...
>>
>> I am with you. I don't think the bridge thing has worked out. I thought
>> it
>> was just me, but then I watched this list and realized it's a lot of
>> people having trouble with it.
>
> Yes, I think we should try routing instead. Should be easy to concoct
> some replacement scripts?
>
> Also then no need for etherbridge patch to our 2.4 kernel. :-)

Eek.  I am using bridging very successfully and have never had a
problem with it.  I don't use 2.4 any more though.  I hope it will
be possible to retain support for bridging.

I'm hoping I'm in the silent majority ;-)

One tweak I do make is to set up the bridge in Dom0 first.  I'd
be happy if bridge support was changed to require a pre-existing
bridge...

Cheers,

Leigh.



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Keir Fraser]

> Eek.  I am using bridging very successfully and have never had a
> problem with it.  I don't use 2.4 any more though.  I hope it will
> be possible to retain support for bridging.
> 
> I'm hoping I'm in the silent majority ;-)

Well, leaving in the etherbridge patch is easier than removing
it. I'll leave it for the time being.

> One tweak I do make is to set up the bridge in Dom0 first.  I'd
> be happy if bridge support was changed to require a pre-existing
> bridge...

The bridge config scripts are quite simple. The intention is that you
customise them for your site. I'll leave example bridge scripts even
though the default will change to using routing.

I'll send a message here to describe how to switch between bridging
and routing.

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Nuutti Kotivuori]

Keir Fraser wrote:
> Well, leaving in the etherbridge patch is easier than removing
> it. I'll leave it for the time being.

Um, what is the etherbridge patch actually? I missed the references to
it.

But, I'd just like to make sure - this aversion to bridging is because
of bridges getting configured wrong, people having wrong assumptions
on how they work, and bridges being a bad default choice, right? There
aren't any known actual bugs in Xen or bridge code wrt. bridging
setups, are there?

The reason is that we are/will be using bridging really extensively,
possibly soon with Xen virtual machines, although the configuration of
the bridges is going to be totally custom.

-- Naked




-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Keir Fraser]

> > One tweak I do make is to set up the bridge in Dom0 first.  I'd
> > be happy if bridge support was changed to require a pre-existing
> > bridge...
> 
> The bridge config scripts are quite simple. The intention is that you
> customise them for your site. I'll leave example bridge scripts even
> though the default will change to using routing.
> 
> I'll send a message here to describe how to switch between bridging
> and routing.

I've checked in new routing scripts to 2.0-testing and unstable
trees. Note that I have *not* changed the default behaviour, which is
to bridge VIFs at the MAC level.

If you want to route instead of bridge:

 1. Edit /etc/xen/xend-config.sxp. Delete the lines referring to
    bridging, and instead uncomment the lines referring to routing.

 2. You may want to edit /etc/xen/scripts/network-route and
    /etc/xen/scripts/vif-route. The former currently enables IP
    forwarding when xend is started; the latter brings up the VIF
    with a dummy IP address, installs a /32 routing rule (the IP
    address is grabbed from the config file as described in step 3),
    and enables proxy ARP.

 3. You will need to edit your VM config file to include a 'vif list'
    containing the IP address of each VIF you are configuring. i.e.,
    the list must look at least something like:
    vif = [ 'ip=1.2.3.4' ]
    (This just extends the existing 'vif' syntax, but a lot of people
    probably have been able to avoid needing to specify any vif
    details so far).

Let me know how you get on!

 -- Keir


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Chris Andrews]

Leigh Brown wrote on 13/01/05 09:30:

> 
> Eek.  I am using bridging very successfully and have never had a
> problem with it.  I don't use 2.4 any more though.  I hope it will
> be possible to retain support for bridging.
> 
> I'm hoping I'm in the silent majority ;-)
> 
> One tweak I do make is to set up the bridge in Dom0 first.  I'd
> be happy if bridge support was changed to require a pre-existing
> bridge...

AOL. It seems to work fine if you set the bridge up in advance. Also lets 
you boot into non-Xeno-Linux and have the same networking setup work (while 
2.0 was stabilising I flipped between Xen and UML with the same networking 
config).


Chris.


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Nuutti Kotivuori]

Ian Pratt wrote:
> 2.4 needs the bridge-nf to be able to use netfilter (iptables) in
> the presence of a bridge.

Ah right, so it was about bridge-nf and ebtables, not bridging
itself. Thanks for the clarification.

> Briding works fine if configured correctly.

So I assumed. Thank you.

-- Naked



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Jan Kundrát]

Keir Fraser wrote:
>>
>>On Wed, 12 Jan 2005, Ian Pratt wrote:
>>
>>
>>>I'm actually going off the whole idea of bridging rather than routeing
>>>as the default...
>>
>>I am with you. I don't think the bridge thing has worked out. I thought it 
>>was just me, but then I watched this list and realized it's a lot of 
>>people having trouble with it.
> 
> 
> Yes, I think we should try routing instead. Should be easy to concoct
> some replacement scripts?

What about performance comparsion? What is faster, ipv4 routing or L2 
bridging? (-> you'll have to recompile & restart domain0 for adding 
support of more protocols ;-))

You'll have to configure your routers to route traffic for unprivileged 
host via domain0 IP address, right? It's more difficult for newbies than 
setting up bridging properly, IMHO.

-jkt

-- 
cd /local/pub && more beer > /dev/mouth



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Ian Pratt]

> > Yes, I think we should try routing instead. Should be easy to concoct
> > some replacement scripts?
> 
> What about performance comparsion? What is faster, ipv4 routing or L2 
> bridging? (-> you'll have to recompile & restart domain0 for adding 
> support of more protocols ;-))

I seriously doubt there'll be any measurable performance difference.

> You'll have to configure your routers to route traffic for unprivileged 
> host via domain0 IP address, right? It's more difficult for newbies than 
> setting up bridging properly, IMHO.

Obviously routing will only work if you're using IP. (Obviously
you'd need to configure v4 and v6 differently).

I haven't thought through what happens if you're using DHCP to
assign user domains static IPs. It might just work if domain0
forwards traffic to the IP broadcast address (255.255.255.255),
or it might be necessary to use a dhcp relay or do something
fancy with netfilter. I'd need to remind myself of what dhcp
requests/replies look like...

Ian



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Ronald G. Minnich]

On Thu, 13 Jan 2005, Nuutti Kotivuori wrote:

> But, I'd just like to make sure - this aversion to bridging is because
> of bridges getting configured wrong, people having wrong assumptions on
> how they work, and bridges being a bad default choice, right? There
> aren't any known actual bugs in Xen or bridge code wrt. bridging setups,
> are there?


it's probably incompetence on my part, but I've found a lot of 
half-working ways to configure the bridge but none that act like bridges 
(hardware ones I mean) that I've used in the past. 

Also we've seen real problems here with nfs root, when starting a domU and 
cutting over to xen-br0 the connection seems to die. 

So, tell me: I have a dom0 with eth0 at 10.128.107.187, eth1 at
192.168.0.65, and I want to bridge a domU at 192.168.0.66 to the dom0 such
that domU can go out over my 802.11 network to the world. How will my
xen-br0 and vif1.0 configuration look? I never got this to work correctly. 
What I really want is domU to DHCP to 192.168.0.1, and I never saw that 
work correctly either. But, if the bridge is really a bridge, that should 
"just happen" (or did on real hardware bridges).

ron


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Grzegorz Milos]

> So, tell me: I have a dom0 with eth0 at 10.128.107.187, eth1 at
> 192.168.0.65, and I want to bridge a domU at 192.168.0.66 to the dom0 such
> that domU can go out over my 802.11 network to the world. How will my
> xen-br0 and vif1.0 configuration look? I never got this to work correctly.
> What I really want is domU to DHCP to 192.168.0.1, and I never saw that
> work correctly either. But, if the bridge is really a bridge, that should
> "just happen" (or did on real hardware bridges).
>

That is a situation where you will want to use routing, not bridging. The 
reason is that you would need to have domU on the same network as eth0 in 
dom0 (so 10.128.107.0/24 or something like that) in order to use the bridge. 
But probably you:
1) cannot request another IP on this network
2) don't want to give direct access to your domU from the outside world.

Instead use routing and iptables (masquerade) as follows:
dom0 eth0 stays with 10.128.107.187
dom0 eth1 stays with 192.168.0.65
dom0 acts as a NAT for unpriviledged domains

domU eth0 gets an IP on 192.168.0.* subnet (I am not quite sure how would be 
able to get if from 192.168.0.1 using DHCP. Dynamic IPs would be also cause 
some problems with setting up the routing table in dom0).
Routing in domU has 192.168.0.65 as its default gateway.

I've tested it once - it worked with IP addresses statically assigned to domUs 
(remember to echo 1 >/proc/sys/net/ipv4/ip_forward, took me some time to work 
out why dom0 doesn't route anything :) ).

Cheers
Gregor


-- 
Quidquid latine dictum sit, altum viditur --- Anon


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Grzegorz Milos]

> So, tell me: I have a dom0 with eth0 at 10.128.107.187, eth1 at
> 192.168.0.65, and I want to bridge a domU at 192.168.0.66 to the dom0 such
> that domU can go out over my 802.11 network to the world. How will my
> xen-br0 and vif1.0 configuration look? I never got this to work correctly.
> What I really want is domU to DHCP to 192.168.0.1, and I never saw that
> work correctly either. But, if the bridge is really a bridge, that should
> "just happen" (or did on real hardware bridges).
>

That is a situation where you will want to use routing, not bridging. The 
reason is that you would need to have domU on the same network as eth0 in 
dom0 (so 10.128.107.0/24 or something like that) in order to use the bridge. 
But probably you:
1) cannot request another IP on this network
2) don't want to give direct access to your domU from the outside world.

Instead use routing and iptables (masquerade) as follows:
dom0 eth0 stays with 10.128.107.187
dom0 eth1 stays with 192.168.0.65
dom0 acts as a NAT for unpriviledged domains

domU eth0 gets an IP on 192.168.0.* subnet (I am not quite sure how would be 
able to get if from 192.168.0.1 using DHCP. Dynamic IPs would be also cause 
some problems with setting up the routing table in dom0).
Routing in domU has 192.168.0.65 as its default gateway.

I've tested it once - it worked with IP addresses statically assigned to domUs 
(remember to echo 1 >/proc/sys/net/ipv4/ip_forward, took me some time to work 
out why dom0 doesn't route anything :) ).

Cheers
Gregor


-- 
Quidquid latine dictum sit, altum viditur --- Anon


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Ronald G. Minnich]

On Thu, 13 Jan 2005, Grzegorz Milos wrote:

> Instead use routing and iptables (masquerade) as follows:
> dom0 eth0 stays with 10.128.107.187
> dom0 eth1 stays with 192.168.0.65
> dom0 acts as a NAT for unpriviledged domains

that's what I was kind of figuring I would need to do. 

But a real bridge (I used to use them) would transparently bridge packets 
from vif1.0 to eth1, right? This is what I never saw working, unless I did 
things that made no sense (e.g. ifconfig xen-br0 192.168.0.65), and even 
then I only got from domU to dom0. (it makes no real sense to me for a 
*bridge* to have an IP address).

I would expect something working as a real bridge to allow me to do this:

ifconfig eth1 192.168.0.65
brctl  xen-br0 addif eth1
brctl  xen-br0 addif vif1.0

This is essentially wiring the two ifs up to xen-br0. 

then I dhcp from domU and I would think packets ought to flow to 
vif1.0->eth1, and eth1->vif1.0, broadcasts would flow across the bridge 
transparently and, once the right MAC discovery happened, packets from 
vif1.0 would make it to 192.168.0.1

I'm still not sure they didn't -- tcpdump seemed to think the DHCP 
requests were going to eth1, but my home router didn't seem to think it 
was seeing them. I will do a little more fooling around. 

ron


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Adam Sulmicki]

On Thu, 13 Jan 2005, Nuutti Kotivuori wrote:

> But, I'd just like to make sure - this aversion to bridging is because
> of bridges getting configured wrong, people having wrong assumptions
> on how they work, and bridges being a bad default choice, right? There
> aren't any known actual bugs in Xen or bridge code wrt. bridging
> setups, are there?
>
> The reason is that we are/will be using bridging really extensively,
> possibly soon with Xen virtual machines, although the configuration of
> the bridges is going to be totally custom.

what I think is missing is a decent HOWTO which explains how to go from 0 
to a fully setup bridge with xen. 7.2 could be a bit more verbose on this 
with command examples,


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Grzegorz Milos]

> > Instead use routing and iptables (masquerade) as follows:
> > dom0 eth0 stays with 10.128.107.187
> > dom0 eth1 stays with 192.168.0.65
> > dom0 acts as a NAT for unpriviledged domains
>
> that's what I was kind of figuring I would need to do.
>
> But a real bridge (I used to use them) would transparently bridge packets
> from vif1.0 to eth1, right? This is what I never saw working, unless I did
> things that made no sense (e.g. ifconfig xen-br0 192.168.0.65), and even
> then I only got from domU to dom0. (it makes no real sense to me for a
> *bridge* to have an IP address).
> I would expect something working as a real bridge to allow me to do this:
>
> ifconfig eth1 192.168.0.65
> brctl  xen-br0 addif eth1
> brctl  xen-br0 addif vif1.0

I am suprised that does not work. This is roughly what we are doing here. Let 
me just go through the steps you need to do to set up the bridge - maybe that 
will clarify something:

a) create the bridge:
brctl addbr xen-br0
ifconfig xen-br0 up

b) add the ip address of eth1 to the bridge (can also do it with ifconfig, but 
ip is easier to use):
ip addr add 192.168.0.65 brd 10.212.4.255 scope global dev xen-br0

c) setup routing:
route del -net 192.168.0.0/24 eth1
route add -net 192.168.0.0/24 xen-br0

d) add eth1 to the bridge:
brctl addif xen-br0 eth1

The above sets up the bridge, then upon domain creation:
e) add virtual interface to the bridge:
brctl addif xen-br0 vif1.0 
ifconfig vif1.0 up

That is all implemented in the two network scripts:
/etc/xen/scripts/network
/etc/xen/scripts/vif-bridge

So if you decide not to use them make sure to have them disabled.

All that should allow your unpriviledged domains to appear as if they were 
connected to your local network (through a switch or whatever else).

In order to allow domU to access the internet you will have to:
a) set up routing on domU:
route add default gw 192.168.0.65

b) set dom0 to work as a NAT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE


Get back to me if that still does not work.


> This is essentially wiring the two ifs up to xen-br0.
>
> then I dhcp from domU and I would think packets ought to flow to
> vif1.0->eth1, and eth1->vif1.0, broadcasts would flow across the bridge
> transparently and, once the right MAC discovery happened, packets from
> vif1.0 would make it to 192.168.0.1
>
> I'm still not sure they didn't -- tcpdump seemed to think the DHCP
> requests were going to eth1, but my home router didn't seem to think it
> was seeing them. I will do a little more fooling around.
>
> ron

Cheers 
Gregor

-- 
Quidquid latine dictum sit, altum viditur --- Anon


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Mike Wray]

Ronald G. Minnich wrote:
> 
> On Thu, 13 Jan 2005, Grzegorz Milos wrote:
> 
> 
>>Instead use routing and iptables (masquerade) as follows:
>>dom0 eth0 stays with 10.128.107.187
>>dom0 eth1 stays with 192.168.0.65
>>dom0 acts as a NAT for unpriviledged domains
> 
> 
> that's what I was kind of figuring I would need to do. 
> 
> But a real bridge (I used to use them) would transparently bridge packets 
> from vif1.0 to eth1, right? This is what I never saw working, unless I did 
> things that made no sense (e.g. ifconfig xen-br0 192.168.0.65), and even 
> then I only got from domU to dom0. (it makes no real sense to me for a 
> *bridge* to have an IP address).

Remember the default is to have one bridge, xen-br0, and to enslave
eth0 to it. This means that IP on eht0 stops working. IP packets
arriving on eth0 are stolen by the bridge, though output to eth0 still works.
In order to get IP working again in dom0 you ahve to move the IP address and routes
from eth0 onto xen-br0 so that IP will work.

When connecting other interfaces together you should really use another bridge,
xen-br1 say. This bridge will not need an IP address unless a real interface
is connected to it _and_ you want dom0 access to IP on the interface. Otherwise
it doesn't need an IP.

> 
> I would expect something working as a real bridge to allow me to do this:
> 
> ifconfig eth1 192.168.0.65
> brctl  xen-br0 addif eth1
> brctl  xen-br0 addif vif1.0
> 
> This is essentially wiring the two ifs up to xen-br0. 

This sort of things works fine for me. I use vnets a lot
(well I wrote them :-) ), and they rely on bridging.
I have a tunnel interface like vnetif1000 and a bridge vnet1000.
I connect vifs onto the vnet1000 bridge and everything works fine.
The vnet1000 bridge does not have or need an IP address.

> then I dhcp from domU and I would think packets ought to flow to 
> vif1.0->eth1, and eth1->vif1.0, broadcasts would flow across the bridge 
> transparently and, once the right MAC discovery happened, packets from 
> vif1.0 would make it to 192.168.0.1

This is exactly what should be happening.
But if you use xen-br0 remember that eth0 is connected to it too.
You should probably use another bridge for eth1.

> I'm still not sure they didn't -- tcpdump seemed to think the DHCP 
> requests were going to eth1, but my home router didn't seem to think it 
> was seeing them. I will do a little more fooling around. 

Might be good to run ethereal in dom0 and on another machine on the
same LAN segment to see what made it onto the network.

Mike


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Gregor Milos]

I will try to put a document together over the weekend.

Cheers 
Gregor

> any chance to have generalized version of this in User's Manual? i think
> many ppl would find this usefull.
>
> On Fri, 14 Jan 2005, Grzegorz Milos wrote:
> >>> Instead use routing and iptables (masquerade) as follows:
> >>> dom0 eth0 stays with 10.128.107.187
> >>> dom0 eth1 stays with 192.168.0.65
> >>> dom0 acts as a NAT for unpriviledged domains
> >>
> >> that's what I was kind of figuring I would need to do.
> >>
> >> But a real bridge (I used to use them) would transparently bridge
> >> packets from vif1.0 to eth1, right? This is what I never saw working,
> >> unless I did things that made no sense (e.g. ifconfig xen-br0
> >> 192.168.0.65), and even then I only got from domU to dom0. (it makes no
> >> real sense to me for a *bridge* to have an IP address).
> >> I would expect something working as a real bridge to allow me to do
> >> this:
> >>
> >> ifconfig eth1 192.168.0.65
> >> brctl  xen-br0 addif eth1
> >> brctl  xen-br0 addif vif1.0
> >
> > I am suprised that does not work. This is roughly what we are doing here.
> > Let me just go through the steps you need to do to set up the bridge -
> > maybe that will clarify something:
> >
> > a) create the bridge:
> > brctl addbr xen-br0
> > ifconfig xen-br0 up
> >
> > b) add the ip address of eth1 to the bridge (can also do it with
> > ifconfig, but ip is easier to use):
> > ip addr add 192.168.0.65 brd 10.212.4.255 scope global dev xen-br0
> >
> > c) setup routing:
> > route del -net 192.168.0.0/24 eth1
> > route add -net 192.168.0.0/24 xen-br0
> >
> > d) add eth1 to the bridge:
> > brctl addif xen-br0 eth1
> >
> > The above sets up the bridge, then upon domain creation:
> > e) add virtual interface to the bridge:
> > brctl addif xen-br0 vif1.0
> > ifconfig vif1.0 up
> >
> > That is all implemented in the two network scripts:
> > /etc/xen/scripts/network
> > /etc/xen/scripts/vif-bridge
> >
> > So if you decide not to use them make sure to have them disabled.
> >
> > All that should allow your unpriviledged domains to appear as if they
> > were connected to your local network (through a switch or whatever else).
> >
> > In order to allow domU to access the internet you will have to:
> > a) set up routing on domU:
> > route add default gw 192.168.0.65
> >
> > b) set dom0 to work as a NAT
> > iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
> >
> >
> > Get back to me if that still does not work.
> >
> >> This is essentially wiring the two ifs up to xen-br0.
> >>
> >> then I dhcp from domU and I would think packets ought to flow to
> >> vif1.0->eth1, and eth1->vif1.0, broadcasts would flow across the bridge
> >> transparently and, once the right MAC discovery happened, packets from
> >> vif1.0 would make it to 192.168.0.1
> >>
> >> I'm still not sure they didn't -- tcpdump seemed to think the DHCP
> >> requests were going to eth1, but my home router didn't seem to think it
> >> was seeing them. I will do a little more fooling around.
> >>
> >> ron
> >
> > Cheers
> > Gregor


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: Re: nfsroot and brige

[Adam Sulmicki]

any chance to have generalized version of this in User's Manual? i think 
many ppl would find this usefull.

On Fri, 14 Jan 2005, Grzegorz Milos wrote:

>>> Instead use routing and iptables (masquerade) as follows:
>>> dom0 eth0 stays with 10.128.107.187
>>> dom0 eth1 stays with 192.168.0.65
>>> dom0 acts as a NAT for unpriviledged domains
>>
>> that's what I was kind of figuring I would need to do.
>>
>> But a real bridge (I used to use them) would transparently bridge packets
>> from vif1.0 to eth1, right? This is what I never saw working, unless I did
>> things that made no sense (e.g. ifconfig xen-br0 192.168.0.65), and even
>> then I only got from domU to dom0. (it makes no real sense to me for a
>> *bridge* to have an IP address).
>> I would expect something working as a real bridge to allow me to do this:
>>
>> ifconfig eth1 192.168.0.65
>> brctl  xen-br0 addif eth1
>> brctl  xen-br0 addif vif1.0
>
> I am suprised that does not work. This is roughly what we are doing here. Let
> me just go through the steps you need to do to set up the bridge - maybe that
> will clarify something:
>
> a) create the bridge:
> brctl addbr xen-br0
> ifconfig xen-br0 up
>
> b) add the ip address of eth1 to the bridge (can also do it with ifconfig, but
> ip is easier to use):
> ip addr add 192.168.0.65 brd 10.212.4.255 scope global dev xen-br0
>
> c) setup routing:
> route del -net 192.168.0.0/24 eth1
> route add -net 192.168.0.0/24 xen-br0
>
> d) add eth1 to the bridge:
> brctl addif xen-br0 eth1
>
> The above sets up the bridge, then upon domain creation:
> e) add virtual interface to the bridge:
> brctl addif xen-br0 vif1.0
> ifconfig vif1.0 up
>
> That is all implemented in the two network scripts:
> /etc/xen/scripts/network
> /etc/xen/scripts/vif-bridge
>
> So if you decide not to use them make sure to have them disabled.
>
> All that should allow your unpriviledged domains to appear as if they were
> connected to your local network (through a switch or whatever else).
>
> In order to allow domU to access the internet you will have to:
> a) set up routing on domU:
> route add default gw 192.168.0.65
>
> b) set dom0 to work as a NAT
> iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
>
>
> Get back to me if that still does not work.
>
>
>> This is essentially wiring the two ifs up to xen-br0.
>>
>> then I dhcp from domU and I would think packets ought to flow to
>> vif1.0->eth1, and eth1->vif1.0, broadcasts would flow across the bridge
>> transparently and, once the right MAC discovery happened, packets from
>> vif1.0 would make it to 192.168.0.1
>>
>> I'm still not sure they didn't -- tcpdump seemed to think the DHCP
>> requests were going to eth1, but my home router didn't seem to think it
>> was seeing them. I will do a little more fooling around.
>>
>> ron
>
> Cheers
> Gregor
>
>


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

RE: nfsroot and brige

[Ian Pratt]

 
> it seems that it is actually possible to configure bridge 
> over nfsroot,
> but one needs to re-arrange lines slightl. As it is below it 
> works for me
> 
> /sbin/brctl addbr xen-br0
> /sbin/ifconfig xen-br0 $HOSTNAME netmask 255.255.255.0 up
> /sbin/brctl addif xen-br0 eth0
> /sbin/ifconfig eth0 0.0.0.0 promisc up

I can see how this can work if the NFS server is on the local subnet:
Linux is quite happy with two interfaces having the same IP. However, if
you're trying to transfer routes from eth0 to xen-br0 I'm not sure its
possible to do it 'atomically'.

Please could someone experiment and post a patch to vif-bridge.

Thanks,
Ian


-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt

Re: nfsroot and brige

[Adam Sulmicki]

> 	it seems like a catch-22 to me. I boot xen0 with nfsroot,
> 	and then try to setup bridge stuff. something like this
>
> /sbin/brctl addbr xen-br0
> /sbin/brctl addif xen-br0 eth0
> /sbin/ifconfig xen-br0 10.0.10.103 netmask 255.255.255.0 up
> /sbin/ifconfig eth0 0.0.0.0 promisc up
>
> 	however, once I start it running it crashes down with error
> 	that nfs server is not available.
>
> 	does it mean that I must use initramfs to configure bridge
> 	first, or is there some other way?

it seems that it is actually possible to configure bridge over nfsroot,
but one needs to re-arrange lines slightl. As it is below it works for me

/sbin/brctl addbr xen-br0
/sbin/ifconfig xen-br0 $HOSTNAME netmask 255.255.255.0 up
/sbin/brctl addif xen-br0 eth0
/sbin/ifconfig eth0 0.0.0.0 promisc up



-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues
Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek.
It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt