From: Dave Chinner <david@fromorbit.com>
To: Christian Herzog <horeizo@phys.ethz.ch>
Cc: isg@phys.ethz.ch, xfs@oss.sgi.com
Subject: Re: XFS errors on large Infiniband fileserver setup
Date: Fri, 24 Sep 2010 09:53:55 +1000 [thread overview]
Message-ID: <20100923235355.GO2614@dastard> (raw)
In-Reply-To: <29252416bd0d9dc973a909e411dbec6a@phys.ethz.ch>
On Thu, Sep 23, 2010 at 09:22:29AM +0200, Christian Herzog wrote:
>
> Dear all,
>
> we (Physics Dept. at ETH Zurich) are trying to set up a large file
> server combo (two disk backends connected to a frontend by
> Infiniband, all running Ubuntu 10.04) and keep getting XFS internal
> error xfs_da_do_buf(2) messages when copying large amounts of data,
> resulting in 'structure needs cleaning' warnings. We have tried a
> lot of different kernels, iSCSI implementations, LVM configurations,
> whatnot, but these errors persist. The setup right now looks like
> this:
>
> 2 disk backends, each: Quad-Xeon X5550, 12G of RAM, 28T HW
> SATA-RAID6 sliced into 2T chunks by LVM2 and exported via tgt
> 1.0.0-2, Ubuntu 10.04 LTS, connected via Mellanox MHRH19B-XTR
> Infiniband + ISER to
>
> 1 frontend Octo-Xeon E5520, 12G of RAM, open-iscsi 2.0.871
> initiator, Ubuntu 10.04 LTS. LMV2 stitches together the
> 2T-iSCSI-LUNs and provides a 10T test XFS filesystem
Out of curiousity, why are you using such a complex storage
configuration?
IMO, it is unneccessarily complex - you could easily do this (~30
drives) with a single server with a couple of external SAS JBOD
arrays and SAS RAID controllers. That would give you the same
performance (or better), with many fewer points of failure (both
hardware and software), use less rack space, and probably be
significantly cheaper....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
next prev parent reply other threads:[~2010-09-23 23:53 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-09-23 7:22 XFS errors on large Infiniband fileserver setup Christian Herzog
2010-09-23 10:01 ` Christian Herzog
2010-09-23 10:38 ` Christoph Hellwig
2010-09-23 11:52 ` Christian Herzog
2010-09-23 13:54 ` Christoph Hellwig
2010-09-23 14:10 ` Christian Herzog
2010-09-23 23:53 ` Dave Chinner [this message]
2010-09-24 5:41 ` Christian Herzog
2010-09-24 8:56 ` Stan Hoeppner
2010-09-24 13:19 ` Emmanuel Florac
2010-09-24 13:19 ` Emmanuel Florac
2010-09-27 5:49 ` Christian Herzog
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100923235355.GO2614@dastard \
--to=david@fromorbit.com \
--cc=horeizo@phys.ethz.ch \
--cc=isg@phys.ethz.ch \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox