ENOSPC and filesystem shutdowns

ENOSPC and filesystem shutdowns

[Bernard Chan]

[-- Attachment #1.1: Type: text/plain, Size: 4017 bytes --]

Hello,

My apologies if this sounds too amateurish for this mailing list, yet I
would like to get some insights on a recent issue we have on our XFS
installation.

We have an XFS filesystem (on LVM, probably doesn't matter anyway) that is
4TB running on CentOS kernel 2.6.21.7, with about 65% FS utilization and 5%
reported inode utilization (df -i). This filesystem contains a lot of small
files about 5 levels deep with each level fanning out with several hundred
to several thousand nodes on average. While having rolled out to production
for a couple weeks, the XFS suddenly shuts down on a certain mkdir.

Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: Filesystem "dm-1": XFS internal
error xfs_trans_cancel at line 1138 of file fs/xfs/xfs_trans.c.  Caller
0xee27bcac
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2732fe>]
xfs_trans_cancel+0x59/0xe3 [xfs]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee27bcac>] xfs_mkdir+0x5bc/0x60b
[xfs]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee27bcac>] xfs_mkdir+0x5bc/0x60b
[xfs]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee284ee6>]
xfs_vn_mknod+0x1a5/0x28f [xfs]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee284fe2>]
xfs_vn_mkdir+0x12/0x14 [xfs]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<c10786d0>] vfs_mkdir+0xbd/0x125
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2dc1bc>]
nfsd_create+0x297/0x38c [nfsd]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2e5597>]
nfsd4_create+0x1ab/0x34c [nfsd]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2e53ec>]
nfsd4_create+0x0/0x34c [nfsd]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2e4948>]
nfsd4_proc_compound+0x178/0x263 [nfsd]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<c102ebfd>]
groups_alloc+0x42/0xae
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2d720f>]
nfsd_dispatch+0xd4/0x18f [nfsd]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee180805>]
svcauth_unix_set_client+0x16d/0x1a0 [sunrpc]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee17d63f>]
svc_process+0x391/0x656 [sunrpc]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2d7756>] nfsd+0x171/0x277
[nfsd]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<ee2d75e5>] nfsd+0x0/0x277 [nfsd]
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: [<c100598f>]
kernel_thread_helper+0x7/0x10
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: =======================
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: xfs_force_shutdown(dm-1,0x8)
called from line 1139 of file fs/xfs/xfs_trans.c.  Return address =
0xee287778
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: Filesystem "dm-1": Corruption of
in-memory data detected.  Shutting down filesystem: dm-1
Sep  3 04:45:39 ip-10-204-xxx-xxx kernel: Please umount the filesystem, and
rectify the problem(s)


We searched and found this list, and a few patches around kernel
2.6.26-2.6.27 that seem to match our scenario. We were able to log the
specific mkdir command that failed and confirmed it consistently fails that
way that gives "no space left on device", while we did not reproduce the
same issue mkdir in other directories with large inode numbers. We haven't
tried patching or upgrading the kernel yet, but we will do that later.

As the root cause of that patch points to a bug triggered by ENOSPC, we
checked the inode numbers created for some directories and files with "ls
-li" and some of them are pretty close to 2^32.

So, we would like to ascertain if that is the cause for ENOSPC in our case,
and does that mean 32-bit inodes are no longer adequate for us and we should
switch to 64-bit inodes? Will switching it avoid this kind of shutdowns with
successful writes in the future?

And is it true that we don't need a 64-bit OS for 64-bit inodes? How can we
tell if our system supports 64-bit inodes?

Finally, although we all know that "df -i" is sort of nonsense on XFS, how
can we get the output of 5% inode while having inode numbers that are close
to 2^32? So what does that 5% exactly mean, or were I looking at inodes the
wrong way?

Thanks in advance for any insights anyone may shed on this one.

-- 

Regards,
Bernard Chan.
GoAnimate.

[-- Attachment #1.2: Type: text/html, Size: 4744 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC and filesystem shutdowns

[Christoph Hellwig]

On Sun, Sep 04, 2011 at 02:09:49PM +0800, Bernard Chan wrote:
> We have an XFS filesystem (on LVM, probably doesn't matter anyway) that is
> 4TB running on CentOS kernel 2.6.21.7,

Isn't Centos based on RHEL and thus running either 2.6.9, 2.6.18 or
2.6.32-ish kernels?

> We searched and found this list, and a few patches around kernel
> 2.6.26-2.6.27 that seem to match our scenario. We were able to log the
> specific mkdir command that failed and confirmed it consistently fails that
> way that gives "no space left on device", while we did not reproduce the
> same issue mkdir in other directories with large inode numbers. We haven't
> tried patching or upgrading the kernel yet, but we will do that later.
> 
> As the root cause of that patch points to a bug triggered by ENOSPC, we
> checked the inode numbers created for some directories and files with "ls
> -li" and some of them are pretty close to 2^32.
> 
> So, we would like to ascertain if that is the cause for ENOSPC in our case,
> and does that mean 32-bit inodes are no longer adequate for us and we should
> switch to 64-bit inodes? Will switching it avoid this kind of shutdowns with
> successful writes in the future?
> 
> And is it true that we don't need a 64-bit OS for 64-bit inodes? How can we
> tell if our system supports 64-bit inodes?

It doesn't.  On Linux XFS only supports inode64 on 32-bit systems since
Linux 3.0.

> Finally, although we all know that "df -i" is sort of nonsense on XFS, how
> can we get the output of 5% inode while having inode numbers that are close
> to 2^32? So what does that 5% exactly mean, or were I looking at inodes the
> wrong way?

It's based on the available space given that XFS can theoretically use
any inode block for data.

> Thanks in advance for any insights anyone may shed on this one.

I'd move off a 4.5-year old unsupposed kernel.  The real RHEL/Centos
kernel have fairly good xfs support these days if you want a backporting
option.  Even RHEL5 might have inode64 on 32-bit systems as it has a lot
of XFS updates backport, but in doubt I would recommend to move to
a RHEL6/Centos6 kernel at least.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC and filesystem shutdowns

[Bernard Chan]

[-- Attachment #1.1: Type: text/plain, Size: 3536 bytes --]

Hi Christoph and everybody else,

Thanks so much for your response.

We are running this on a customized CentOS image on AWS and thus had a
different kernel version. We learned from the image vendors that these are
based on CentOS and indeed they seem to be, although we do not know the
exact CentOS versions on which those images were based.

We switched to another image that offers a newer kernel version (2.6.35.11),
and re-mounted the LVM with XFS volumes. Earlier on we performed xfs_repair
and did not find any problems, and fragmentation was found to be low.
We retried those mkdir operations that previously resulted in ENOSPC which
in turn shut down the XFS. There is no longer any XFS shutdown now and the
same operations are successful. We feel curious and do not quite understand
why, and we haven't enabled inode64 either with the new setup and remain on
a 32-bit architecture. Essentially we only changed the kernel and a slightly
different 32-bit image.

So should we bother with inode64 and 64-bit servers with NFS4, and should we
anticipate any other issues running on this setup with a 4TB volume without
enabling inode64?

Thanks so much for any possible insights.


On Mon, Sep 5, 2011 at 3:47 PM, Christoph Hellwig <hch@infradead.org> wrote:

> On Sun, Sep 04, 2011 at 02:09:49PM +0800, Bernard Chan wrote:
> > We have an XFS filesystem (on LVM, probably doesn't matter anyway) that
> is
> > 4TB running on CentOS kernel 2.6.21.7,
>
> Isn't Centos based on RHEL and thus running either 2.6.9, 2.6.18 or
> 2.6.32-ish kernels?
>
> > We searched and found this list, and a few patches around kernel
> > 2.6.26-2.6.27 that seem to match our scenario. We were able to log the
> > specific mkdir command that failed and confirmed it consistently fails
> that
> > way that gives "no space left on device", while we did not reproduce the
> > same issue mkdir in other directories with large inode numbers. We
> haven't
> > tried patching or upgrading the kernel yet, but we will do that later.
> >
> > As the root cause of that patch points to a bug triggered by ENOSPC, we
> > checked the inode numbers created for some directories and files with "ls
> > -li" and some of them are pretty close to 2^32.
> >
> > So, we would like to ascertain if that is the cause for ENOSPC in our
> case,
> > and does that mean 32-bit inodes are no longer adequate for us and we
> should
> > switch to 64-bit inodes? Will switching it avoid this kind of shutdowns
> with
> > successful writes in the future?
> >
> > And is it true that we don't need a 64-bit OS for 64-bit inodes? How can
> we
> > tell if our system supports 64-bit inodes?
>
> It doesn't.  On Linux XFS only supports inode64 on 32-bit systems since
> Linux 3.0.
>
> > Finally, although we all know that "df -i" is sort of nonsense on XFS,
> how
> > can we get the output of 5% inode while having inode numbers that are
> close
> > to 2^32? So what does that 5% exactly mean, or were I looking at inodes
> the
> > wrong way?
>
> It's based on the available space given that XFS can theoretically use
> any inode block for data.
>
> > Thanks in advance for any insights anyone may shed on this one.
>
> I'd move off a 4.5-year old unsupposed kernel.  The real RHEL/Centos
> kernel have fairly good xfs support these days if you want a backporting
> option.  Even RHEL5 might have inode64 on 32-bit systems as it has a lot
> of XFS updates backport, but in doubt I would recommend to move to
> a RHEL6/Centos6 kernel at least.
>
>


-- 

Regards,
Bernard Chan.
GoAnimate.

[-- Attachment #1.2: Type: text/html, Size: 4346 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC and filesystem shutdowns

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 946 bytes --]

On Donnerstag, 8. September 2011 Bernard Chan wrote:
> ENOSPC

There have been some "optimizations" for NFS in earlier kernel, leading 
to large space reservation per open file. Marc Lehman yelled at that 
early August in thread "frequent kernel BUG and lockups - 2.6.39 + 
xfs_fsr". Maybe that's your problem too? Do you use "allocsize=" mount 
option? It could be your new kernel version fixed that, but I can't 
tell, as I'm not a dev.

inode64 helps in case there is no more space in the area <1TB. As you 
have "only" 4TB, chances are it wouldn't make a difference for you. 
Somebody once posted a [perl?] script on this list to find out how much 
space is left at <1TB, maybe that can be of help.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

// Haus zu verkaufen: http://zmi.at/langegg/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: ENOSPC and filesystem shutdowns

[Christoph Hellwig]

On Thu, Sep 08, 2011 at 07:09:55PM +0800, Bernard Chan wrote:
> So should we bother with inode64 and 64-bit servers with NFS4, and should we
> anticipate any other issues running on this setup with a 4TB volume without
> enabling inode64?

I would absolutely recommend using inode64 if your are running a pure
64-bit environment.  Even if you are not running out of inodes it will
give you a much better allocation pattern.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs