Re: eNBD on loopback [was Re: [PATCH] 2.5.8 IDE 36]

Re: eNBD on loopback [was Re: [PATCH] 2.5.8 IDE 36]

[Sean Reifschneider]

>Don't ask me, I'm not a user, I have just seen the patch submissions, and
>I just want to get real user feedback before I'd merge a new "extended
>nbd".

I haven't used enbd, because the site was down the weekend I was evaluating
the alternatives...  I did try NBD and DRBD, however.  My experience was
that enbd could hardly be worse than nbd, for the following reasons:

   The nbd server software referenced in the Configuration documentation
   (the only I was able to find, and that only after some digging), would
   fail rather quickly because the remote kernel would send a request much
   larger than the server was expecting.

   After a couple of days, the primary machine would just lock up when
   running RAID-1 on top of the NBD.

   The NBD server code is, not pretty...  It sounds like that server was
   written as just a hack, and really hasn't been looked at since then.

This was with kernel version 2.4.18.

DRBD is what I'm currently using and it's been running for a few weeks now
without any problems.  It combines the mirroring and NBD functionality into
a single combined package.  A nice feature of DRBD is that it understands
about the second node and can do things like wait for a RAID mirror to
finish before starting up other processes.

enbd has some nice features, particularly it looks like the server code
has had a lot more development in it.  Particularly nice is that the
client/server will auto-negotiate an optimized mirror mode where they will
exchange MD5 sums of each block, and only transmit the block if the MD5 is
different...  It switches to and from this mode automatically.

I can't really on wether enbd should be in the kernel...  It can't be worse
than nbd, based on my experience.  It's active development makes it a
better choice to include.

Sean
-- 
 Fire at the celuloud factory.  No film at eleven.
                 -- _Kentucky_Fried_Movie_
Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com>
tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python

Re: eNBD on loopback [was Re: [PATCH] 2.5.8 IDE 36]

[Anton Altaparmakov]

At 04:28 21/04/02, Sean Reifschneider wrote:
> >Don't ask me, I'm not a user, I have just seen the patch submissions, and
> >I just want to get real user feedback before I'd merge a new "extended
> >nbd".
>
>I haven't used enbd, because the site was down the weekend I was evaluating
>the alternatives...  I did try NBD and DRBD, however.  My experience was
>that enbd could hardly be worse than nbd, for the following reasons:
>
>    The nbd server software referenced in the Configuration documentation
>    (the only I was able to find, and that only after some digging), would
>    fail rather quickly because the remote kernel would send a request much
>    larger than the server was expecting.

Indeed. The source code reference in th Configuration documentation is very 
much out of data and completely broken for anything that requires 64 bit 
sizes on a 32 bit architecture.

This is all fixed now (I know because I shared your frustration and went 
and fixed it myself (-:), if you want to get a properly working version 
which exhibits no problems under very intensive i/o on a 15GiB partition 
over a 100MBit lan just go to http://sf.net/projects/nbd/ and get the 
latest version from CVS or download the new 2.0 release tarball.

>   After a couple of days, the primary machine would just lock up when
>    running RAID-1 on top of the NBD.

I haven't tried RAID... Interesting idea though.

>   The NBD server code is, not pretty...  It sounds like that server was
>    written as just a hack, and really hasn't been looked at since then.

Yes, I had that impression, too. I hope I made the parts I touched look a 
bit better. (-8

>This was with kernel version 2.4.18.

Simillar here on server side (2.4.18-pre7-ac2+misc patches).

>DRBD is what I'm currently using and it's been running for a few weeks now
>without any problems.  It combines the mirroring and NBD functionality into
>a single combined package.  A nice feature of DRBD is that it understands
>about the second node and can do things like wait for a RAID mirror to
>finish before starting up other processes.
>
>enbd has some nice features, particularly it looks like the server code
>has had a lot more development in it.  Particularly nice is that the
>client/server will auto-negotiate an optimized mirror mode where they will
>exchange MD5 sums of each block, and only transmit the block if the MD5 is
>different...  It switches to and from this mode automatically.
>
>I can't really on wether enbd should be in the kernel...  It can't be worse
>than nbd, based on my experience.  It's active development makes it a
>better choice to include.

nbd is also actively developed. The only problem is nobody has bothered to 
update the kernel documentation to point to the website where development 
happens. )-:

While we are here, I will make patches for 2.4 and 2.5 in a minute and 
submit them...

Anton


-- 
   "I've not lost my mind. It's backed up on tape somewhere." - Unknown
-- 
Anton Altaparmakov <aia21 at cantab.net> (replace at with @)
Linux NTFS Maintainer / IRC: #ntfs on irc.openprojects.net
WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/

Re: eNBD on loopback [was Re: [PATCH] 2.5.8 IDE 36]

[Sean Reifschneider]

On Sun, Apr 21, 2002 at 11:40:09AM +0100, Anton Altaparmakov wrote:
>I haven't tried RAID... Interesting idea though.

My use for it was in a high availability cluster, where I was using NBD to
set up shared storage between the two machines.

Another challenge I ran into, which DRBD solves well, is that it was hard
to get the RAID to operate in degraded mode.  It would try to access the
block device on the first startup of the md device after the primary
failure, even if the raidtab told it the drive was dead...  I finally just
set up a noop nbd server that would return the proper size on startup, but
any subsequent read/writes would fail...

>nbd is also actively developed. The only problem is nobody has bothered to 
>update the kernel documentation to point to the website where development 
>happens. )-:

Updates to the Configuration documentation would really help.  I've asked
google to index nbd.sf.net, and also asked Alan Robertson to add it to his
list of shared storage alternatives on the linux-ha.com site.  When I was
trying to get NBD going, I was unable to find nbd.sf.net through google,
though this may have changed in month since I was doing the work...

Sean
-- 
 Got Source?
Sean Reifschneider, Inimitably Superfluous <jafo@tummy.com>
tummy.com - Linux Consulting since 1995. Qmail, KRUD, Firewalls, Python

Re: eNBD on loopback [was Re: [PATCH] 2.5.8 IDE 36]

[Pavel Machek]

Hi!

> > >Don't ask me, I'm not a user, I have just seen the patch submissions, and
> > >I just want to get real user feedback before I'd merge a new "extended
> > >nbd".
> >
> >I haven't used enbd, because the site was down the weekend I was evaluating
> >the alternatives...  I did try NBD and DRBD, however.  My experience was
> >that enbd could hardly be worse than nbd, for the following reasons:
> >
> >    The nbd server software referenced in the Configuration documentation
> >    (the only I was able to find, and that only after some digging), would
> >    fail rather quickly because the remote kernel would send a request much
> >    larger than the server was expecting.
> 
> Indeed. The source code reference in th Configuration documentation is very 
> much out of data and completely broken for anything that requires 64 bit 
> sizes on a 32 bit architecture.

Can you submit patch to fix that docs? It should point to sourceforge..

> This is all fixed now (I know because I shared your frustration and went 
> and fixed it myself (-:), if you want to get a properly working version 
> which exhibits no problems under very intensive i/o on a 15GiB partition 
> over a 100MBit lan just go to http://sf.net/projects/nbd/ and get the 
> latest version from CVS or download the new 2.0 release tarball.
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: [PATCH] 2.5.8 IDE 36

[Alan Cox]

> Doing it with a loopback like interface at a higher level is the much 
> saner operation - I understand why Martin removed the byteswap support, 
> and agree with it 100%. It just didn't make any sense from a driver 
> standpoint.

We need to support partitioning on loopback devices in that case.

> The only reason byteswapping exists is a rather historical one: Linux did 
> the wrong thing for "insw/outsw" on big-endian architectures at one point 
> (it byteswapped the data).

A small number of other setups people wired the IDE the quick and easy
way and their native format is indeed ass backwards - some M68K disks and
the Tivo are examples of that. Interworking requires byteswapping and the
ability to handle byteswapped partition tables.

Given the ability to see partitions on loop devices all works out I think

Alan

Re: [PATCH] 2.5.8 IDE 36

[Linus Torvalds]

On Tue, 16 Apr 2002, Alan Cox wrote:
>
> We need to support partitioning on loopback devices in that case.

No, you just need to do the loopback over nbd - you need something to do
the byte swapping anyway (ie you can't really use the normal "loop"
device: I really just meant the more generic "loop the data back"
approach).

nbd devices already do partitioning, I'm fairly certain.

(But no, I've never tested it, obviously).

Btw, while I'm at it - who out there actually uses the new "enbd"
(Enhanced NBD)? I have this feeling that that would be the better choice,
since unlike plain nbd it should be deadlock-free on localhost (ie you
don't need a remote machine).

> > The only reason byteswapping exists is a rather historical one: Linux did
> > the wrong thing for "insw/outsw" on big-endian architectures at one point
> > (it byteswapped the data).
>
> A small number of other setups people wired the IDE the quick and easy
> way and their native format is indeed ass backwards - some M68K disks and
> the Tivo are examples of that. Interworking requires byteswapping and the
> ability to handle byteswapped partition tables.

Note that THAT case is an architecture issue, and should probably be
handled by just making the IDE "insw" macro do the byteswapping natively.
That way you don't get the current "it can actually corrupt your
filesystem on SMP" behaviour.

(Although I suspect that none of the architectures in question have ever
even jokingly considered SMP support ;)

			Linus

eNBD on loopback [was Re: [PATCH] 2.5.8 IDE 36]

[Pavel Machek]

Hi!

> Btw, while I'm at it - who out there actually uses the new "enbd"
> (Enhanced NBD)? I have this feeling that that would be the better choice,
> since unlike plain nbd it should be deadlock-free on localhost (ie you
> don't need a remote machine).

How does eNBD manage to do that? It was pretty hard last time I
checked...

What if their enbd server is swapped out, and all memory is in dirty
pages waiting for writeback to eNBD?
									Pavel
-- 
(about SSSCA) "I don't say this lightly.  However, I really think that the U.S.
no longer is classifiable as a democracy, but rather as a plutocracy." --hpa

Re: eNBD on loopback [was Re: [PATCH] 2.5.8 IDE 36]

[Linus Torvalds]

On Thu, 18 Apr 2002, Pavel Machek wrote:
> 
> > Btw, while I'm at it - who out there actually uses the new "enbd"
> > (Enhanced NBD)? I have this feeling that that would be the better choice,
> > since unlike plain nbd it should be deadlock-free on localhost (ie you
> > don't need a remote machine).
> 
> How does eNBD manage to do that? It was pretty hard last time I
> checked...

Don't ask me, I'm not a user, I have just seen the patch submissions, and
I just want to get real user feedback before I'd merge a new "extended
nbd".

So far I haven't heard anything from any actual users..

		Linus