AOE (Ata over ethernet) troubles on xen 2.0.6

AOE (Ata over ethernet) troubles on xen 2.0.6

[Jeb Campbell]

I understand that all work is going into xen3, but I had wanted to note 
that aoe (drivers/block/aoe) is giving me trouble on xen 2.0.6 (so we 
can keep and eye on xen3).

Specifically I can't see nor export AOE devices.  As a quick background 
on AOE, it is not IP (not routable, etc), but works with broadcasts and 
packets to MAC addresses (see http://www.coraid.com).

(for anyone who wants to test or play)
In my dom0 (w/ aoe module), I installed aoetools and vblade ( 
http://aoetools.sourceforge.net ).  aoetools are not necessary, but 
nice.  vblade lets you export a block device to the network.

I exported an lvm device (worked fine on non xen machine) with:
vblade 2 0 eth0 (2 is shelf number and 0 is slot number -- can be anything).

If I export from non-xen, it shows up on other machines as 
/dev/etherd/e2.0, but the xen machine can't see it when others export 
either.  (Note, vblades from your own machine don't show up in your 
/dev/etherd/ -- only on other hosts)

As this is dom0, I'm not sure why the aoe module is not seeing the 
packets (unless the bridge is messing it up).  Any tips for debugging 
this would be appreciated -- I don't really care if we get this working 
in xen2, but I think this is important for xen3.

Thanks for your time,
-- 
Jeb Campbell
jebc@c4solutions.net

RE: AOE (Ata over ethernet) troubles on xen 2.0.6

[James Harper]

For what it's worth, I'm using aoe presently without any problems, and
have been for quite a few months now. In my setup I have vblade running
on a server, with dom0's on two machines reading the same aoe device,
and exporting lvm partitions to the different domains as required. This
allows migration to work nicely.

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Jeb Campbell
> Sent: Saturday, 23 July 2005 01:51
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] AOE (Ata over ethernet) troubles on xen 2.0.6
> 
> I understand that all work is going into xen3, but I had wanted to
note
> that aoe (drivers/block/aoe) is giving me trouble on xen 2.0.6 (so we
> can keep and eye on xen3).
> 
> Specifically I can't see nor export AOE devices.  As a quick
background
> on AOE, it is not IP (not routable, etc), but works with broadcasts
and
> packets to MAC addresses (see http://www.coraid.com).
> 
> (for anyone who wants to test or play)
> In my dom0 (w/ aoe module), I installed aoetools and vblade (
> http://aoetools.sourceforge.net ).  aoetools are not necessary, but
> nice.  vblade lets you export a block device to the network.
> 
> I exported an lvm device (worked fine on non xen machine) with:
> vblade 2 0 eth0 (2 is shelf number and 0 is slot number -- can be
> anything).
> 
> If I export from non-xen, it shows up on other machines as
> /dev/etherd/e2.0, but the xen machine can't see it when others export
> either.  (Note, vblades from your own machine don't show up in your
> /dev/etherd/ -- only on other hosts)
> 
> As this is dom0, I'm not sure why the aoe module is not seeing the
> packets (unless the bridge is messing it up).  Any tips for debugging
> this would be appreciated -- I don't really care if we get this
working
> in xen2, but I think this is important for xen3.
> 
> Thanks for your time,
> --
> Jeb Campbell
> jebc@c4solutions.net
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Kristinn Soffanias Runarsson]

Hi James, could you explain this setup in some detail?

Bestestest regards from Iceland

-soffi-


On 7/23/05, James Harper <james.harper@bendigoit.com.au> wrote:
> For what it's worth, I'm using aoe presently without any problems, and
> have been for quite a few months now. In my setup I have vblade running
> on a server, with dom0's on two machines reading the same aoe device,
> and exporting lvm partitions to the different domains as required. This
> allows migration to work nicely.
>

RE: AOE (Ata over ethernet) troubles on xen 2.0.6

[James Harper]

My setup is:

Server1 - running vanilla debian install + vblade export. vblade is a
userspace app so the kernel doesn't need anything special. A single
partition is being exported, and while it is being used via lvm on the
xen machines, Server1 itself knows nothing about lvm.

Xen1 & Xen2 - running xen 2.0.6 with AoE compiled in. Boot via the
network as they are diskless. Both run lvm on /dev/etherd/e0.0. I need
to take care never to mount the same volume in more than one place. Bad
things happen if I do that (unfortunately I know this from experience!).
My volume names in lvm are things like 'mail-root', 'mail2-root',
'mail-swap', etc. the xen0 domains export those volumes to the other
domains. The other domains don't use AoE directly. Migration works well
as both xen machines can see all the lvm volumes.

One of the xen machines has a 1Gb network adapter and a 100Mb adapter.
The 1G has no ip address and is simply used for AoE. The other network
adapter uses vlan's to service 3 other networks.

The other xen machine has only two 100Mb adapters but is otherwise
configured much the same.

Hope that tells you what you want to know!

James

> -----Original Message-----
> From: Kristinn Soffanias Runarsson [mailto:soffanias@gmail.com]
> Sent: Monday, 25 July 2005 10:37
> To: James Harper
> Cc: Jeb Campbell; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] AOE (Ata over ethernet) troubles on xen 2.0.6
> 
> Hi James, could you explain this setup in some detail?
> 
> Bestestest regards from Iceland
> 
> -soffi-
> 
> 
> On 7/23/05, James Harper <james.harper@bendigoit.com.au> wrote:
> > For what it's worth, I'm using aoe presently without any problems,
and
> > have been for quite a few months now. In my setup I have vblade
running
> > on a server, with dom0's on two machines reading the same aoe
device,
> > and exporting lvm partitions to the different domains as required.
This
> > allows migration to work nicely.
> >

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Jeb Campbell]

James Harper wrote:
> My setup is:  . . .

Thanks for the info.  Per your setup, I have compiled aoe in-kernel and 
it can see my vblades from non-xen hosts.  Don't know why it didn't work 
as a module?

However, exporting a vblade from from my xen host does not work.  This 
is very confusing as it is a userspace app.

While I would like to get to the bottom of this, I have other things to 
work on and this fills my needs.

I will update the list if I figure anything else out (could just be my 
custom kernel missing something).
-- 
Jeb Campbell
jebc@c4solutions.net

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Mark Williamson]

If you have some spare time, could you please add this to 
http://wiki.xensource.com/xenwiki/CoolConfigurations

It's another neat setup that people might want to play around with.

Cheers,
Mark

On Monday 25 July 2005 02:16, James Harper wrote:
> My setup is:
>
> Server1 - running vanilla debian install + vblade export. vblade is a
> userspace app so the kernel doesn't need anything special. A single
> partition is being exported, and while it is being used via lvm on the
> xen machines, Server1 itself knows nothing about lvm.
>
> Xen1 & Xen2 - running xen 2.0.6 with AoE compiled in. Boot via the
> network as they are diskless. Both run lvm on /dev/etherd/e0.0. I need
> to take care never to mount the same volume in more than one place. Bad
> things happen if I do that (unfortunately I know this from experience!).
> My volume names in lvm are things like 'mail-root', 'mail2-root',
> 'mail-swap', etc. the xen0 domains export those volumes to the other
> domains. The other domains don't use AoE directly. Migration works well
> as both xen machines can see all the lvm volumes.
>
> One of the xen machines has a 1Gb network adapter and a 100Mb adapter.
> The 1G has no ip address and is simply used for AoE. The other network
> adapter uses vlan's to service 3 other networks.
>
> The other xen machine has only two 100Mb adapters but is otherwise
> configured much the same.
>
> Hope that tells you what you want to know!
>
> James
>
> > -----Original Message-----
> > From: Kristinn Soffanias Runarsson [mailto:soffanias@gmail.com]
> > Sent: Monday, 25 July 2005 10:37
> > To: James Harper
> > Cc: Jeb Campbell; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] AOE (Ata over ethernet) troubles on xen 2.0.6
> >
> > Hi James, could you explain this setup in some detail?
> >
> > Bestestest regards from Iceland
> >
> > -soffi-
> >
> > On 7/23/05, James Harper <james.harper@bendigoit.com.au> wrote:
> > > For what it's worth, I'm using aoe presently without any problems,
>
> and
>
> > > have been for quite a few months now. In my setup I have vblade
>
> running
>
> > > on a server, with dom0's on two machines reading the same aoe
>
> device,
>
> > > and exporting lvm partitions to the different domains as required.
>
> This
>
> > > allows migration to work nicely.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Avi Kivity]

Jeb Campbell wrote:

> James Harper wrote:
>
>> My setup is:  . . .
>
>
> Thanks for the info.  Per your setup, I have compiled aoe in-kernel 
> and it can see my vblades from non-xen hosts.  Don't know why it 
> didn't work as a module?
>
> However, exporting a vblade from from my xen host does not work.  This 
> is very confusing as it is a userspace app.
>
> While I would like to get to the bottom of this, I have other things 
> to work on and this fills my needs.
>
perhaps the vblade server binds to eth0 instead of xen-br0.