[Advocacy] Re: 3ware 9650 tips

[Advocacy] Re: 3ware 9650 tips

[Bryan J. Smith]

Off-topic, advocacy-level response ...  

On Mon, 2007-07-16 at 11:43 -0400, Joshua Baker-LePain wrote:
> I do so wish that RedHat shared this view...

I've been trying to convince them since Red Hat Linux 7 (and, later, 9)
that they need to realize the limits of Ext3 at the enterprise end of
the scalability spectrum -- you know, that whole market they are
seemingly saying they are the king of and a replacement for Sun?  ;->

The problem with Red Hat is that when anyone brings up an alternative to
Ext3, Red Hat falls back to arguments against other filesystems, which
is rather easy given the various compatibility issues with JFS (ported
from OS/2, requiring a lot of inode compatibility hacks -- don't get me
started with my experiences) and ReiserFS (utter lack of inode
compatibility in structures, requiring kernel-level emulation, etc...
that never seems to work, regardless of what the advocates say, let
alone the almsota always "out-of-sync" off-line repair tools).

But when you bring up XFS and its history of a stable, but advanced
inode structure, quota support from day 1, POSIX ACLs from nearly day 1,
and all the SGI team put into 2.5.3+ that is now stock kernel, they
still try to dance.  One thing I always get is "oh, its extents don't
perform well for /tmp or /var" or countless other arguments, of which I
merely respond, "all the more reason to use Ext3 for those few
filesystems, and XFS when Ext3 doesn't scale -- like for
large /home, /export, etc... filesystems."  No matter how many times I
put forth the argument that XFS complements Ext3, they seem to treat it
as yet another JFS/ReiserFS argument.

Hopeless?

-- Bryan "one of the reasons I still deploy Solaris instead of RHEL for
fileservers, even though RHL7+XFS and RHL9+XFS rocked (and are still
rocking!)" Smith


-- 
Bryan J. Smith         Professional, Technical Annoyance
mailto:b.j.smith@ieee.org   http://thebs413.blogspot.com
--------------------------------------------------------
        Fission Power:  An Inconvenient Solution

Re: [Advocacy] Re: 3ware 9650 tips

[Matthew Wilcox]

On Mon, Jul 16, 2007 at 08:40:00PM +0300, Al Boldi wrote:
> XFS surely rocks, but it's missing one critical component: data=ordered
> And that's one component that's just too critical to overlook for an 
> enterprise environment that is built on data-integrity over performance.
> 
> So that's the secret why people still use ext3, and XFS' reliance on external 
> hardware to ensure integrity is really misplaced.
> 
> Now, maybe when we get the data=ordered onto the VFS level, then maybe XFS 
> may become viable for the enterprise, and ext3 may cease to be KING.

Wow, thanks for bringing an advocacy thread onto linux-fsdevel.  Just what
we wanted.  Do you have any insight into how to "get the data=ordered
onto the VFS level"?  Because to me, that sounds like pure nonsense.

-- 
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Advocacy] Re: 3ware 9650 tips

[Bryan J. Smith]

On Mon, 2007-07-16 at 11:48 -0600, Matthew Wilcox wrote:
> Wow, thanks for bringing an advocacy thread onto linux-fsdevel.  Just what
> we wanted.  Do you have any insight into how to "get the data=ordered
> onto the VFS level"?  Because to me, that sounds like pure nonsense.

First off, I have no idea who decided to respond to my post and CC:
linux-fsdevel on it.

In retrospect, secondly, I should have not posted my post to linux-raid
in the first place (is that list now mirrored to linux-fsdevel or
something?).  I was just sharing in my frustration of the lack of XFS
support by Red Hat.

So, lastly and in any case, my apologies to all, even if I did not
proliferate it to linux-fsdevel, it was probably not ideal for me to
post such to anything on vger.kernel.org (like linux-raid) in the first
place.


-- 
Bryan J. Smith         Professional, Technical Annoyance
mailto:b.j.smith@ieee.org   http://thebs413.blogspot.com
--------------------------------------------------------
        Fission Power:  An Inconvenient Solution

[Advocacy] Re: 3ware 9650 tips

[Bryan J. Smith]

[ I removed linux-raid, the only vgers list I posted to (who
re-posted/responded to my post on linux-fsdevel anyway?), from this
response. ]

On Mon, 2007-07-16 at 12:34 -0500, Stuart Levy wrote:
> So they support XFS in Fedora, but not in RHEL??
> (I've been using Fedora...)

Fedora ships support for all filesystems, and I believe has since Fedora
Core 2 / kernel 2.6.  RHEL only ships with support for Ext3.

There are various support issues with XFS that I, among others, feel
could be quickly addressed if Red Hat took a keen interest in supporting
XFS as a second, supplemental filesystem to Ext3.  I have gone on-record
several times about this, although I've dropped the advocacy over the
last few years and just "given up."  I honestly haven't kept up with the
issues either (like issues with XFS and the 4G/4G kernel model and/or
4KiB stacks -- have they been addressed?).

Luckily I have been doing more and more embedded as of late, so I
haven't had to deploy large filesystems.  But I have put in a few
Solaris 10/Opteron systems as NFS/SMB file server solutions in the last
year.  I'm not saying Solaris is "better," I'm just saying I would
really like RHEL Ext3 with an officially supported XFS release.  That's
all.


-- 
Bryan J. Smith         Professional, Technical Annoyance
mailto:b.j.smith@ieee.org   http://thebs413.blogspot.com
--------------------------------------------------------
        Fission Power:  An Inconvenient Solution

Re: [Advocacy] Re: 3ware 9650 tips

[Simon Matter]

> [ I removed linux-raid, the only vgers list I posted to (who
> re-posted/responded to my post on linux-fsdevel anyway?), from this
> response. ]
>
> On Mon, 2007-07-16 at 12:34 -0500, Stuart Levy wrote:
>> So they support XFS in Fedora, but not in RHEL??
>> (I've been using Fedora...)
>
> Fedora ships support for all filesystems, and I believe has since Fedora
> Core 2 / kernel 2.6.  RHEL only ships with support for Ext3.
>
> There are various support issues with XFS that I, among others, feel
> could be quickly addressed if Red Hat took a keen interest in supporting
> XFS as a second, supplemental filesystem to Ext3.  I have gone on-record
> several times about this, although I've dropped the advocacy over the
> last few years and just "given up."  I honestly haven't kept up with the
> issues either (like issues with XFS and the 4G/4G kernel model and/or
> 4KiB stacks -- have they been addressed?).

I already wrote quite a long mail for this list yesterday but then decided
not to send it because I also gave up on the issue with RedHat. It has all
been said in the past years and it's still true: You can't compare
anything with XFS on Linux and the competitiors aren't sleeping (just
think ZFS for example). I know that XFS had and maybe still has some minor
issues - but, if RedHat decided to use XFS and put the same effort into it
like they did with Ext3, those problems would have gone long ago.
I'm a long time XFS user starting with RedHat 7.x, and I'm still using it
for choosen filesystem on RHEL3,4,5. I just don't want to live without
lvextend and xfs_growfs.

Simon

Re: [Advocacy] Re: 3ware 9650 tips

[Giuseppe Ghibò]

Bryan J Smith wrote:

> Matter, Simon wrote:  
>> I know that XFS had and maybe still has some minor issues -
>> but, if RedHat decided to use XFS and put the same effort into it
>> like they did with Ext3, those problems would have gone long ago.
> 
> Ditto.  XFS was the ultimate, complementary filesystem to Ext3.
> 
> In all honesty, I'm just looking forward to ZFS on RHEL, sorry to say.

ZFS won't appear officially in Linux until there will be change of
license, so right now it's released in userspace.

> Although I don't see that being "proven" for years, so I'm with XFS "on my own"
> when I deploy RHEL - which is, many times, the only viable filesystem.
> 
> --  

Indeed XFS is a lot faster than ext3 on many task (e.g. copy/moving
or delete huge files o creating filesystems or dumping with xfsdump),
and worked fine, until linux kernels around 2.6.15|16|17|18 when it had serious problems about
data corruptions. Furthermore when you run xfs_repair to fix such errors, you find that it lost
all the directory names, and places restored files into "random" dirs
named with "number" names.

See for instance:

http://lkml.org/lkml/2006/8/4/97
http://lkml.org/lkml/2006/8/28/88
or http://qa.mandriva.com/show_bug.cgi?id=24716

and sound the fix is not easy to be backportable to kernel series older than 2.6.18,
such as 2.6.17 (used for instance in ubuntu 7.04 and mandriva 2007.1).

Also in the recent 2.6.20|21 kernel series I found it has serious problems of
performance, especially when used in softraid (e.g. for storing
the vmware huge filedisks images a simple "sync" takes fifteen minutes in a raid1).

Bye
Giuseppe.

Re: [Advocacy] Re: 3ware 9650 tips

[David Chinner]

On Tue, Jul 17, 2007 at 10:20:54PM +0200, Giuseppe Ghibò wrote:
> Indeed XFS is a lot faster than ext3 on many task (e.g.
> copy/moving or delete huge files o creating filesystems or dumping
> with xfsdump), and worked fine, until linux kernels around
> 2.6.15|16|17|18 when it had serious problems about data
> corruptions.

<sigh>

I don't mind advocacy, but misleading FUD about filesystem
corruption, regardless of filesystem type, is *not acceptable* in
any forum.

So, to set the record straight, the only XFS corruption I know of in the
releases you mention above is this:

http://oss.sgi.com/projects/xfs/faq.html#dir2

Which was introduced in 2.6.17-rc1 and fixed in 2.6.18-rc2 (IIRC).i
The only released kernels affected are 2.6.17.0-6. i.e. it was fixed
in 2.6.17.7.

And in the interest of full disclosure, there was another in 2.6.19 (IIRC)
to do with a brand new feature that nobody used (the attr2 bug) - until
it was enabled by default on Fedora and the installer tripped over
it - that was fixed in 2.6.20.

If you know of more, then where are the bug reports?

> Furthermore when you run xfs_repair to fix such errors, you find that it lost
> all the directory names, and places restored files into "random" dirs
> named with "number" names.

Please, a little research would tell you what these mean.

When you lose directory entries on a filesystem for *any* reason,
you'll end up with files named by *inode number* placed in
lost+found because they are guaranteed to be unique.  The names and
the structure that end up in lost+found are certainly not random
and it's not just XFS that does this. e.g. ext2/3/4 does this, too [1]:

"Some of the directory and files may not pop-up at their right
places. Instead they will be located in /lost+found with names after
their inode numbers."

[1] trivial google search "e2fsck lost+found" points to
http://tldp.org/HOWTO/archived/Ext2fs-Undeletion-Dir-Struct/lostnfnd.html

> See for instance:
> 
> http://lkml.org/lkml/2006/8/4/97
> http://lkml.org/lkml/2006/8/28/88

A kernel panic in 2.6.18-rc3/5 due to a bad error handling path that
nobody had hit - or, more correctly, reported - for a couple of
years. This is not a filesystem corrupting bug.

> or http://qa.mandriva.com/show_bug.cgi?id=24716

"------- Comment #3 From Thomas Backlund 2006-08-25 09:32:46 CEST ------- 
What you are hitting is a bug I tried to warn about before releasing 2007b1,
namely kernel.org-2.6.17.6 had a nasty xfs bug, wich mdv 2.6.17.1mdv was based
on, and I tried to point out before beta1 was released that 2.6.17.7 was out
and had this fixed, but no-one with powers to do anything listened..."

And yes, I can see that you raised this bug. I'm sorry that were
affected by this bug, but in reality you should be complaining to
your kernel release team who released a kernel with known serious
corruption bug that they'd been pre-warned about.

IOWs, your evidence points to one data corruption that only affected
2.6.17.0-2.6.17.6 and you *already knew this*. How does this
translate into data corruption problems that span four whole kernel
releases?

Hence in future can you please try to stick to facts as filesystem
corruption is something that we take extremely seriously.

> Also in the recent 2.6.20|21 kernel series I found it has serious
> problems of performance, especially when used in softraid (e.g.
> for storing the vmware huge filedisks images a simple "sync" takes
> fifteen minutes in a raid1).

Where's the bug report? We can't fix what we don't know about.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: [Advocacy] Re: 3ware 9650 tips

[Giuseppe Ghibò]

David Chinner wrote:

> On Tue, Jul 17, 2007 at 10:20:54PM +0200, Giuseppe Ghibò wrote:
>> Indeed XFS is a lot faster than ext3 on many task (e.g.
>> copy/moving or delete huge files o creating filesystems or dumping
>> with xfsdump), and worked fine, until linux kernels around
>> 2.6.15|16|17|18 when it had serious problems about data
>> corruptions.
> 
> <sigh>
> 
> I don't mind advocacy, but misleading FUD about filesystem
> corruption, regardless of filesystem type, is *not acceptable* in
> any forum.
> 
> So, to set the record straight, the only XFS corruption I know of in the
> releases you mention above is this:
> 
> http://oss.sgi.com/projects/xfs/faq.html#dir2
> 
> Which was introduced in 2.6.17-rc1 and fixed in 2.6.18-rc2 (IIRC).i
> The only released kernels affected are 2.6.17.0-6. i.e. it was fixed
> in 2.6.17.7.
> 
> And in the interest of full disclosure, there was another in 2.6.19 (IIRC)
> to do with a brand new feature that nobody used (the attr2 bug) - until
> it was enabled by default on Fedora and the installer tripped over
> it - that was fixed in 2.6.20.

Sorry, but I'm not doing any FUD (if you think so, then my apologies),
I'm just reporting experiences, like any other in this thread.
I was a *strong* estimator of xfs for several reasons (I was even using it for /usr, and
also I was using it everywhere including for /home), but after the experience that lead
to me to that problems I switched back to ext3+dir_index, though it's slower, basically because
I've often to access to the filesystems (e.g. I use it mobile on removable devices)
though different kernels, and I don't know which kernel I would find (e.g.
i might write with a 2.6.12, then reopen with a 2.6.17 or other).

FYI, yes, I was the first who spotted the problem (or rather the first
who did the bug report there), but I was not the only one experiencing that problem, also
Olivier Thauvin, the maintainer of distrib-coffee (which is a 3TB mirror
here http://distrib-coffee.ipsl.jussieu.fr/) and he had the problem
also with kernel based on 2.6.17.14 (final not RC): the problem occurs usually under high|heavy
I/O load/pressure rsync (especially during rsyncing hard and soft links, e.g. over a gigabit
network or an USB external disk). He resolved right now just upgrading
to the final stock 2.6.20|21.

> 
> If you know of more, then where are the bug reports?
> 
>> Furthermore when you run xfs_repair to fix such errors, you find that it lost
>> all the directory names, and places restored files into "random" dirs
>> named with "number" names.
> 
> Please, a little research would tell you what these mean.
> 
> When you lose directory entries on a filesystem for *any* reason,
> you'll end up with files named by *inode number* placed in
> lost+found because they are guaranteed to be unique.  The names and
> the structure that end up in lost+found are certainly not random
> and it's not just XFS that does this. e.g. ext2/3/4 does this, too [1]:
> 
> "Some of the directory and files may not pop-up at their right
> places. Instead they will be located in /lost+found with names after
> their inode numbers."

yes, I knew names should come from the inode numbers, but indeed
were notjust "some" file which takes the inode dir name, but a bunch.
Note that in my case if I shouldn't have done any xfs_repair I problably
wouldn't have lost any file. Indeed the term "lost" is wrong, it didn't
loose any file, the files were there just moved by xfs_repair under the inodes names
(but was not just a couple but thousand of dirs like so, but the problem
originated on try deleting a simple softlink). I know that also other
filesystem place files in lost+found, but I've not experienced a
so high number of recovered|renamed dirs (thousand) in the past, and
if you have the same filename repated under thousand of dir (think to a mirror
of software branches), then it's hard to manually replace them in the right place.

> [...]
> Hence in future can you please try to stick to facts as filesystem
> corruption is something that we take extremely seriously.

so I.

Bye
Giuseppe.