XFS on CoRAID errors with SMB

XFS on CoRAID errors with SMB

[Jon Marshall]

Hi,

We have recently experienced what appear to be XFS filesystem errors on
a samba share. The actual filesystem resides on a network attached
storage device, a Coraid. The attached server locked up totally, and we
forced to hard reset it.

I have the following trace from the kernel logs:

[6128798.051868] smbd: page allocation failure. order:4, mode:0xc0d0
[6128798.051872] Pid: 16908, comm: smbd Not tainted 2.6.32-5-amd64 #1
[6128798.051874] Call Trace:
[6128798.051882]  [<ffffffff810ba5d6>] ? __alloc_pages_nodemask+0x592/0x5f4
[6128798.051885]  [<ffffffff810b959c>] ? __get_free_pages+0x9/0x46
[6128798.051889]  [<ffffffff810e7ea1>] ? __kmalloc+0x3f/0x141
[6128798.051893]  [<ffffffff8110672c>] ? getxattr+0x89/0x117
[6128798.051896]  [<ffffffff810e5b65>] ? virt_to_head_page+0x9/0x2a
[6128798.051899]  [<ffffffff810f9bc4>] ? user_path_at+0x52/0x79
[6128798.051919]  [<ffffffffa0297b17>] ? xfs_xattr_put_listent+0x0/0xe5 [xfs]
[6128798.051922]  [<ffffffff810e5b65>] ? virt_to_head_page+0x9/0x2a
[6128798.051925]  [<ffffffff8118ddcb>] ? _atomic_dec_and_lock+0x33/0x50
[6128798.051928]  [<ffffffff811068b4>] ? sys_getxattr+0x45/0x60
[6128798.051931]  [<ffffffff81010b42>] ? system_call_fastpath+0x16/0x1b

smbd seems to throw these errors for about 15 minutes, then sshd starts
throwing errors and shortly after the system became unresponsive.

Just wondering if anyone had any experience of similar results, with XFS
on a CoRAID device or XFS SMB shares?

Thanks
Jon

-- 
Jon Marshall
Technical Officer
jon@campbell-lange.net
.
Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on CoRAID errors with SMB

[Joe Landman]

On 11/28/2011 08:55 AM, Jon Marshall wrote:
> Hi,
>
> We have recently experienced what appear to be XFS filesystem errors on
> a samba share. The actual filesystem resides on a network attached
> storage device, a Coraid. The attached server locked up totally, and we
> forced to hard reset it.

This is (from our past experience working with these units and the AoE 
system), more likely the AoE driver crashing (or something on the 
underlying network failing).  From there, the file system eventually dies.

This isn't an xfs problem per se, xfs is sort of an uwilling participant 
in a slow motion crash.

> I have the following trace from the kernel logs:
>
> [6128798.051868] smbd: page allocation failure. order:4, mode:0xc0d0
> [6128798.051872] Pid: 16908, comm: smbd Not tainted 2.6.32-5-amd64 #1
> [6128798.051874] Call Trace:
> [6128798.051882]  [<ffffffff810ba5d6>] ? __alloc_pages_nodemask+0x592/0x5f4
> [6128798.051885]  [<ffffffff810b959c>] ? __get_free_pages+0x9/0x46
> [6128798.051889]  [<ffffffff810e7ea1>] ? __kmalloc+0x3f/0x141

If you note the failed kmalloc, something ran you out of memory.  What 
we've run into in the past with this has been a driver memory leak 
(usually older model e1000 or similar drivers)

[...]

> smbd seems to throw these errors for about 15 minutes, then sshd starts
> throwing errors and shortly after the system became unresponsive.
>
> Just wondering if anyone had any experience of similar results, with XFS
> on a CoRAID device or XFS SMB shares?

This is what you see when the AoE stack collapses due to a crash of one 
of the lower block rungs.  XFS can't run if it can't allocate memory for 
itself.  smbd dies when the underlying filesystem goes away.  sshd 
probably gets unresponsive in part, due to all the IOs queuing up that 
the scheduler can't do anything with.  Before sshd stops working, user 
load winds up past 5x number of CPUs, then past 10x, then ...

Once you see this happening, its time to kill the upper level stacks if 
possible, and unmount the file system as rapidly as possible.  If you 
can't kill the stuff above it, a 'umount -l ' is your friend.  You *may* 
be able to regain enough control for a non-crash based reboot.  Even 
with this, I'd recommend changing / to sync before either forcing a reboot

    mount -o remount,sync /

to preserve the integrity of the OS drive.

Then reboot (or if the user load is too high, and a reboot command will 
just hang ... hopefully you have IPMI on you unit so you can do an 
'ipmitool -I open chassis power cycle' hard bounce)





>
> Thanks
> Jon
>


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on CoRAID errors with SMB

[Jon Marshall]

Hi Joe,

Thanks for the rapid response.

Is this something that has been reported often in relation to AoE? Is
there any chance you could point us in the direction of some more
background on the issue? I am checking the AoE mailing list, but if you know
of something specific that would be very helpful.

I am also looking into the ethernet drivers we have in place on the
system in question.

Again, thanks for the quick and informative response.

Jon

On Mon, Nov 28, 2011 at 09:46:22AM -0500, Joe Landman wrote:
> On 11/28/2011 08:55 AM, Jon Marshall wrote:
> >Hi,
> >
> >We have recently experienced what appear to be XFS filesystem errors on
> >a samba share. The actual filesystem resides on a network attached
> >storage device, a Coraid. The attached server locked up totally, and we
> >forced to hard reset it.
> 
> This is (from our past experience working with these units and the
> AoE system), more likely the AoE driver crashing (or something on
> the underlying network failing).  From there, the file system
> eventually dies.
> 
> This isn't an xfs problem per se, xfs is sort of an uwilling
> participant in a slow motion crash.
> 
> >I have the following trace from the kernel logs:
> >
> >[6128798.051868] smbd: page allocation failure. order:4, mode:0xc0d0
> >[6128798.051872] Pid: 16908, comm: smbd Not tainted 2.6.32-5-amd64 #1
> >[6128798.051874] Call Trace:
> >[6128798.051882]  [<ffffffff810ba5d6>] ? __alloc_pages_nodemask+0x592/0x5f4
> >[6128798.051885]  [<ffffffff810b959c>] ? __get_free_pages+0x9/0x46
> >[6128798.051889]  [<ffffffff810e7ea1>] ? __kmalloc+0x3f/0x141
> 
> If you note the failed kmalloc, something ran you out of memory.
> What we've run into in the past with this has been a driver memory
> leak (usually older model e1000 or similar drivers)
> 
> [...]
> 
> >smbd seems to throw these errors for about 15 minutes, then sshd starts
> >throwing errors and shortly after the system became unresponsive.
> >
> >Just wondering if anyone had any experience of similar results, with XFS
> >on a CoRAID device or XFS SMB shares?
> 
> This is what you see when the AoE stack collapses due to a crash of
> one of the lower block rungs.  XFS can't run if it can't allocate
> memory for itself.  smbd dies when the underlying filesystem goes
> away.  sshd probably gets unresponsive in part, due to all the IOs
> queuing up that the scheduler can't do anything with.  Before sshd
> stops working, user load winds up past 5x number of CPUs, then past
> 10x, then ...
> 
> Once you see this happening, its time to kill the upper level stacks
> if possible, and unmount the file system as rapidly as possible.  If
> you can't kill the stuff above it, a 'umount -l ' is your friend.
> You *may* be able to regain enough control for a non-crash based
> reboot.  Even with this, I'd recommend changing / to sync before
> either forcing a reboot
> 
>    mount -o remount,sync /
> 
> to preserve the integrity of the OS drive.
> 
> Then reboot (or if the user load is too high, and a reboot command
> will just hang ... hopefully you have IPMI on you unit so you can do
> an 'ipmitool -I open chassis power cycle' hard bounce)
> 
> 
> 
> 
> 
> >
> >Thanks
> >Jon
> >
> 
> 
> -- 
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman@scalableinformatics.com
> web  : http://scalableinformatics.com
>        http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

-- 
Jon Marshall
Technical Officer
jon@campbell-lange.net
.
Campbell-Lange Workshop
www.campbell-lange.net
0207 6311 555
3 Tottenham Street London W1T 2AF
Registered in England No. 04551928

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: XFS on CoRAID errors with SMB

[Joe Landman]

On 11/28/2011 10:26 AM, Jon Marshall wrote:
> Hi Joe,
>
> Thanks for the rapid response.
>
> Is this something that has been reported often in relation to AoE? Is

We've experienced it in the past when we supported our customers with 
Coraid gear.  Most of that is gone now, so we haven't seen much AoE 
stuff as of late (last 2 years or so).

This said, the AoE stack depends critically upon the network stack, and 
between AoE and the network stack (or possibly something else), you ran 
out of memory for use in the kernel.  Our experience with this is 
usually a leaky network driver.  e1000 and similar Intel drivers shipped 
with default RHEL5/Centos5 are highly problematic.  AoE could be leaking 
itself (early versions were pretty bad in this regard, though I haven't 
looked at the driver in the last few years, they hopefully have improved 
it).

The xfs connection to this (to stay relevant to this group) is that xfs 
is ok atop this, as long as the other layers don't go away.  If you can 
detect problems like this in advance, you might be able to issue an 
xfs_freeze, and preserve the integrity of the underlying filesystem 
(obviating the need for an xfs_repair).  The hard part would be an 
accurate prediction, but if your drivers are grabbing memory and not 
releasing it back, or you have a run-away memory consuming process, 
yeah, you could potentially predict this onset.

> there any chance you could point us in the direction of some more
> background on the issue? I am checking the AoE mailing list, but if you know
> of something specific that would be very helpful.

Not really, we aren't doing much with AoE anymore.  This may or may not 
be an AoE issue per se.  Likely AoE crashed, and the reason for the 
crash is very probably the same reason that xfs crashed, it ran out of 
memory.  If AoE is the culprit, you might find some sort of imprint of 
this in the logs, though our experience has been usually a run-away 
network driver.  Since AoE does its block devices over raw ethernet 
packets, it doesn't take very long for a leaky driver to crash such a 
system under load.

>
> I am also looking into the ethernet drivers we have in place on the
> system in question.
>
> Again, thanks for the quick and informative response.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs