AOE (Ata over ethernet) troubles on xen 2.0.6

AOE (Ata over ethernet) troubles on xen 2.0.6

[Jeb Campbell]

I understand that all work is going into xen3, but I had wanted to note 
that aoe (drivers/block/aoe) is giving me trouble on xen 2.0.6 (so we 
can keep and eye on xen3).

Specifically I can't see nor export AOE devices.  As a quick background 
on AOE, it is not IP (not routable, etc), but works with broadcasts and 
packets to MAC addresses (see http://www.coraid.com).

(for anyone who wants to test or play)
In my dom0 (w/ aoe module), I installed aoetools and vblade ( 
http://aoetools.sourceforge.net ).  aoetools are not necessary, but 
nice.  vblade lets you export a block device to the network.

I exported an lvm device (worked fine on non xen machine) with:
vblade 2 0 eth0 (2 is shelf number and 0 is slot number -- can be anything).

If I export from non-xen, it shows up on other machines as 
/dev/etherd/e2.0, but the xen machine can't see it when others export 
either.  (Note, vblades from your own machine don't show up in your 
/dev/etherd/ -- only on other hosts)

As this is dom0, I'm not sure why the aoe module is not seeing the 
packets (unless the bridge is messing it up).  Any tips for debugging 
this would be appreciated -- I don't really care if we get this working 
in xen2, but I think this is important for xen3.

Thanks for your time,
-- 
Jeb Campbell
jebc@c4solutions.net

RE: AOE (Ata over ethernet) troubles on xen 2.0.6

[James Harper]

For what it's worth, I'm using aoe presently without any problems, and
have been for quite a few months now. In my setup I have vblade running
on a server, with dom0's on two machines reading the same aoe device,
and exporting lvm partitions to the different domains as required. This
allows migration to work nicely.

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Jeb Campbell
> Sent: Saturday, 23 July 2005 01:51
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] AOE (Ata over ethernet) troubles on xen 2.0.6
> 
> I understand that all work is going into xen3, but I had wanted to
note
> that aoe (drivers/block/aoe) is giving me trouble on xen 2.0.6 (so we
> can keep and eye on xen3).
> 
> Specifically I can't see nor export AOE devices.  As a quick
background
> on AOE, it is not IP (not routable, etc), but works with broadcasts
and
> packets to MAC addresses (see http://www.coraid.com).
> 
> (for anyone who wants to test or play)
> In my dom0 (w/ aoe module), I installed aoetools and vblade (
> http://aoetools.sourceforge.net ).  aoetools are not necessary, but
> nice.  vblade lets you export a block device to the network.
> 
> I exported an lvm device (worked fine on non xen machine) with:
> vblade 2 0 eth0 (2 is shelf number and 0 is slot number -- can be
> anything).
> 
> If I export from non-xen, it shows up on other machines as
> /dev/etherd/e2.0, but the xen machine can't see it when others export
> either.  (Note, vblades from your own machine don't show up in your
> /dev/etherd/ -- only on other hosts)
> 
> As this is dom0, I'm not sure why the aoe module is not seeing the
> packets (unless the bridge is messing it up).  Any tips for debugging
> this would be appreciated -- I don't really care if we get this
working
> in xen2, but I think this is important for xen3.
> 
> Thanks for your time,
> --
> Jeb Campbell
> jebc@c4solutions.net
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Kristinn Soffanias Runarsson]

Hi James, could you explain this setup in some detail?

Bestestest regards from Iceland

-soffi-


On 7/23/05, James Harper <james.harper@bendigoit.com.au> wrote:
> For what it's worth, I'm using aoe presently without any problems, and
> have been for quite a few months now. In my setup I have vblade running
> on a server, with dom0's on two machines reading the same aoe device,
> and exporting lvm partitions to the different domains as required. This
> allows migration to work nicely.
>

RE: AOE (Ata over ethernet) troubles on xen 2.0.6

[James Harper]

My setup is:

Server1 - running vanilla debian install + vblade export. vblade is a
userspace app so the kernel doesn't need anything special. A single
partition is being exported, and while it is being used via lvm on the
xen machines, Server1 itself knows nothing about lvm.

Xen1 & Xen2 - running xen 2.0.6 with AoE compiled in. Boot via the
network as they are diskless. Both run lvm on /dev/etherd/e0.0. I need
to take care never to mount the same volume in more than one place. Bad
things happen if I do that (unfortunately I know this from experience!).
My volume names in lvm are things like 'mail-root', 'mail2-root',
'mail-swap', etc. the xen0 domains export those volumes to the other
domains. The other domains don't use AoE directly. Migration works well
as both xen machines can see all the lvm volumes.

One of the xen machines has a 1Gb network adapter and a 100Mb adapter.
The 1G has no ip address and is simply used for AoE. The other network
adapter uses vlan's to service 3 other networks.

The other xen machine has only two 100Mb adapters but is otherwise
configured much the same.

Hope that tells you what you want to know!

James

> -----Original Message-----
> From: Kristinn Soffanias Runarsson [mailto:soffanias@gmail.com]
> Sent: Monday, 25 July 2005 10:37
> To: James Harper
> Cc: Jeb Campbell; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] AOE (Ata over ethernet) troubles on xen 2.0.6
> 
> Hi James, could you explain this setup in some detail?
> 
> Bestestest regards from Iceland
> 
> -soffi-
> 
> 
> On 7/23/05, James Harper <james.harper@bendigoit.com.au> wrote:
> > For what it's worth, I'm using aoe presently without any problems,
and
> > have been for quite a few months now. In my setup I have vblade
running
> > on a server, with dom0's on two machines reading the same aoe
device,
> > and exporting lvm partitions to the different domains as required.
This
> > allows migration to work nicely.
> >

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Jeb Campbell]

James Harper wrote:
> My setup is:  . . .

Thanks for the info.  Per your setup, I have compiled aoe in-kernel and 
it can see my vblades from non-xen hosts.  Don't know why it didn't work 
as a module?

However, exporting a vblade from from my xen host does not work.  This 
is very confusing as it is a userspace app.

While I would like to get to the bottom of this, I have other things to 
work on and this fills my needs.

I will update the list if I figure anything else out (could just be my 
custom kernel missing something).
-- 
Jeb Campbell
jebc@c4solutions.net

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Mark Williamson]

If you have some spare time, could you please add this to 
http://wiki.xensource.com/xenwiki/CoolConfigurations

It's another neat setup that people might want to play around with.

Cheers,
Mark

On Monday 25 July 2005 02:16, James Harper wrote:
> My setup is:
>
> Server1 - running vanilla debian install + vblade export. vblade is a
> userspace app so the kernel doesn't need anything special. A single
> partition is being exported, and while it is being used via lvm on the
> xen machines, Server1 itself knows nothing about lvm.
>
> Xen1 & Xen2 - running xen 2.0.6 with AoE compiled in. Boot via the
> network as they are diskless. Both run lvm on /dev/etherd/e0.0. I need
> to take care never to mount the same volume in more than one place. Bad
> things happen if I do that (unfortunately I know this from experience!).
> My volume names in lvm are things like 'mail-root', 'mail2-root',
> 'mail-swap', etc. the xen0 domains export those volumes to the other
> domains. The other domains don't use AoE directly. Migration works well
> as both xen machines can see all the lvm volumes.
>
> One of the xen machines has a 1Gb network adapter and a 100Mb adapter.
> The 1G has no ip address and is simply used for AoE. The other network
> adapter uses vlan's to service 3 other networks.
>
> The other xen machine has only two 100Mb adapters but is otherwise
> configured much the same.
>
> Hope that tells you what you want to know!
>
> James
>
> > -----Original Message-----
> > From: Kristinn Soffanias Runarsson [mailto:soffanias@gmail.com]
> > Sent: Monday, 25 July 2005 10:37
> > To: James Harper
> > Cc: Jeb Campbell; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] AOE (Ata over ethernet) troubles on xen 2.0.6
> >
> > Hi James, could you explain this setup in some detail?
> >
> > Bestestest regards from Iceland
> >
> > -soffi-
> >
> > On 7/23/05, James Harper <james.harper@bendigoit.com.au> wrote:
> > > For what it's worth, I'm using aoe presently without any problems,
>
> and
>
> > > have been for quite a few months now. In my setup I have vblade
>
> running
>
> > > on a server, with dom0's on two machines reading the same aoe
>
> device,
>
> > > and exporting lvm partitions to the different domains as required.
>
> This
>
> > > allows migration to work nicely.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: AOE (Ata over ethernet) troubles on xen 2.0.6

[Avi Kivity]

Jeb Campbell wrote:

> James Harper wrote:
>
>> My setup is:  . . .
>
>
> Thanks for the info.  Per your setup, I have compiled aoe in-kernel 
> and it can see my vblades from non-xen hosts.  Don't know why it 
> didn't work as a module?
>
> However, exporting a vblade from from my xen host does not work.  This 
> is very confusing as it is a userspace app.
>
> While I would like to get to the bottom of this, I have other things 
> to work on and this fills my needs.
>
perhaps the vblade server binds to eth0 instead of xen-br0.

Re: AOE (Ata over ethernet) Solved!

[Jeb Campbell]

Avi Kivity wrote:
> perhaps the vblade server binds to eth0 instead of xen-br0.

You nailed it!  To export from dom0 (to real network -- not tested with
domU hosts) you have to tell vblade to bind to xen-br0 (on xen2 -- not
tested with xen3) not eth0.  I never would have thought to do this as
dom0 should be as close to "real" linux as possible.

So in summary for aoe on xen:

1. Compile aoe in kernel (Device Drivers/Block devices/ATA over
Ethernet) to be able to see aoe devices (remember you can't see vblade
exports on the host exporting them).  Module did not work for me, but
your milage may vary ;) .

2. If you want to use vblade (http://aoetools.sf.net) to export a
disk/lvm/loopback device, you have to use xen-br0 (for xen2 -- adjust
for xen3) NOT eth0.

Thanks for the help on the list.

PS - I'm getting in the 3U SATA raid from coraid in a week or 2, and
assuming I can get xen3 on my single proccessor Sun V20z (Opteron), I
will post performance numbers from dom0 and domU.  As the sata boxes do
raid on board, it should be a great option for large and/or redundant
shared drives for xen migration.
-- 
Jeb Campbell
jebc@c4solutions.net

Re: AOE (Ata over ethernet) Solved!

[Mark Williamson]

> You nailed it!  To export from dom0 (to real network -- not tested with
> domU hosts) you have to tell vblade to bind to xen-br0 (on xen2 -- not
> tested with xen3) not eth0.  I never would have thought to do this as
> dom0 should be as close to "real" linux as possible.

The bridging code is the standard Linux bridge: I'd expect this to happen on 
native Linux too, when using bridging.

Cheers,
Mark

> So in summary for aoe on xen:
>
> 1. Compile aoe in kernel (Device Drivers/Block devices/ATA over
> Ethernet) to be able to see aoe devices (remember you can't see vblade
> exports on the host exporting them).  Module did not work for me, but
> your milage may vary ;) .
>
> 2. If you want to use vblade (http://aoetools.sf.net) to export a
> disk/lvm/loopback device, you have to use xen-br0 (for xen2 -- adjust
> for xen3) NOT eth0.
>
> Thanks for the help on the list.
>
> PS - I'm getting in the 3U SATA raid from coraid in a week or 2, and
> assuming I can get xen3 on my single proccessor Sun V20z (Opteron), I
> will post performance numbers from dom0 and domU.  As the sata boxes do
> raid on board, it should be a great option for large and/or redundant
> shared drives for xen migration.

vblade in Xen 3

[Steve Traugott]

Hi All,

Has anyone got vblade (server daemon for ATA over Ethernet) working
reliably in a Xen 3 dom0 yet?  There was a thread (excerpt below) last
July about doing this in Xen 2, so I know this worked at some point...

I have it built under xen 3/linux 2.6.12, and mostly running, but the
client aoe module hangs after a few seconds under load (bonnie or dd
of a few hundred MB in an ext3 filesystem, physical is /dev/sdb).  An
strace on the vblade process tends to show it waiting on read() from
the raw ethernet socket; aoeping works fine, aoe-stat from the hung
client shows the device 'up'. but aoe-stat from another client shows
the device 'down,paused'.

I've tried this when binding vblade to eth0.  If I take a hint from
the message below and bind to xenbr0 instead, then the aoe client
can't see the vblade device at all, according to aoe-stat; this is the
reverse of the symptoms which Jeb noticed back then.  If I try to
bypass the internal bridge and bind vblade directly to peth0, then the
client still can't see it.  Still troubleshooting this.

Does anyone recognize this symptom already, or have any hints for
what to look at?

BTW, things work great if I run the vblade server on a vanilla 2.6.11
kernel -- I've had bonnie running in a loop for the last 10 hours on a
Xen 3 dom0 aoe client, talking to a vanilla server; performance over a
100Mb link is better than half of local SCSI disk speeds; I can't wait
to get some GigE cards...

Thanks,

Steve


On Mon, Jul 25, 2005 at 05:28:52PM +0100, Mark Williamson wrote:
> > You nailed it!  To export from dom0 (to real network -- not tested with
> > domU hosts) you have to tell vblade to bind to xen-br0 (on xen2 -- not
> > tested with xen3) not eth0.  I never would have thought to do this as
> > dom0 should be as close to "real" linux as possible.
> 
> The bridging code is the standard Linux bridge: I'd expect this to happen on 
> native Linux too, when using bridging.
> 
> Cheers,
> Mark
> 
> From: Jeb Campbell <jebc@c4solutions.net>
> > So in summary for aoe on xen:
> >
> > 1. Compile aoe in kernel (Device Drivers/Block devices/ATA over
> > Ethernet) to be able to see aoe devices (remember you can't see vblade
> > exports on the host exporting them).  Module did not work for me, but
> > your milage may vary ;) .
> >
> > 2. If you want to use vblade (http://aoetools.sf.net) to export a
> > disk/lvm/loopback device, you have to use xen-br0 (for xen2 -- adjust
> > for xen3) NOT eth0.
> >
> > Thanks for the help on the list.
> >
> > PS - I'm getting in the 3U SATA raid from coraid in a week or 2, and
> > assuming I can get xen3 on my single proccessor Sun V20z (Opteron), I
> > will post performance numbers from dom0 and domU.  As the sata boxes do
> > raid on board, it should be a great option for large and/or redundant
> > shared drives for xen migration.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Stephen G. Traugott  (KG6HDQ)
UNIX/Linux Infrastructure Architect, TerraLuna LLC
stevegt@TerraLuna.Org 
http://www.stevegt.com -- http://Infrastructures.Org

Re: vblade in Xen 3

[Keir Fraser]

On 28 Jan 2006, at 14:45, Steve Traugott wrote:

> Has anyone got vblade (server daemon for ATA over Ethernet) working
> reliably in a Xen 3 dom0 yet?  There was a thread (excerpt below) last
> July about doing this in Xen 2, so I know this worked at some point...
>
> I have it built under xen 3/linux 2.6.12, and mostly running, but the
> client aoe module hangs after a few seconds under load (bonnie or dd
> of a few hundred MB in an ext3 filesystem, physical is /dev/sdb).  An
> strace on the vblade process tends to show it waiting on read() from
> the raw ethernet socket; aoeping works fine, aoe-stat from the hung
> client shows the device 'up'. but aoe-stat from another client shows
> the device 'down,paused'.

What happens if you boot dom0 but don't start xend or bring up the 
bridge? If you bind vblade to eth0 (which is the real physical eth0 in 
this case, since you won;t have run the bridge script) do things work 
better?

  -- Keir