[PATCH] Printk for ENOSPC due to lack of inodes

[PATCH] Printk for ENOSPC due to lack of inodes

[Raghavendra D Prabhu]

[-- Attachment #1.1: Type: text/plain, Size: 2625 bytes --]

Hi,

While diagnosing a MySQL crash (on a Centos 5.7 box), I noticed 
that it had failed with ENOSPC earlier; it was rebooted after 
that; now after reboot, even though space was there, ENOSPC was 
showing up, I also did df -i and it showed inodes available. At 
this point, mounting with inode64 option was tried, which fixed 
it. 

http://oss.sgi.com/archives/xfs/2011-03/msg00299.html helped me 
here.

So, I have attached a patch here.

=====================================================================


When a ENOSPC is encountered and it is due to lack of inodes (particularly
without inode64), it is not possible to detect this (df -i doesn't help here),
so adding a printk which can aid in detecting this.

Signed-off-by: Raghavendra D Prabhu <rprabhu@wnohang.net>
---
  fs/xfs/xfs_qm.c       |    3 +++
  fs/xfs/xfs_vnodeops.c |   10 ++++++++--
  2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
index c872fea..fbefa87 100644
--- a/fs/xfs/xfs_qm.c
+++ b/fs/xfs/xfs_qm.c
@@ -987,6 +987,9 @@ xfs_qm_qino_alloc(
  
  	error = xfs_dir_ialloc(&tp, NULL, S_IFREG, 1, 0, 0, 1, ip, &committed);
  	if (error) {
+		if (error == ENOSPC)
+			xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+					XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
  		xfs_trans_cancel(tp, XFS_TRANS_RELEASE_LOG_RES |
  				 XFS_TRANS_ABORT);
  		return error;
diff --git a/fs/xfs/xfs_vnodeops.c b/fs/xfs/xfs_vnodeops.c
index ebdb888..7542c36 100644
--- a/fs/xfs/xfs_vnodeops.c
+++ b/fs/xfs/xfs_vnodeops.c
@@ -946,8 +946,11 @@ xfs_create(
  	error = xfs_dir_ialloc(&tp, dp, mode, is_dir ? 2 : 1, rdev,
  			       prid, resblks > 0, &ip, &committed);
  	if (error) {
-		if (error == ENOSPC)
+		if (error == ENOSPC) {
+			xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+					XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
  			goto out_trans_cancel;
+		}
  		goto out_trans_abort;
  	}
  
@@ -1610,8 +1613,11 @@ xfs_symlink(
  	error = xfs_dir_ialloc(&tp, dp, S_IFLNK | (mode & ~S_IFMT), 1, 0,
  			       prid, resblks > 0, &ip, NULL);
  	if (error) {
-		if (error == ENOSPC)
+		if (error == ENOSPC) {
+			xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
+					XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
  			goto error_return;
+		}
  		goto error1;
  	}
  
-- 
1.7.9.2

Regards,
-- 
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net

[-- Attachment #1.2: Type: application/pgp-signature, Size: 490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH] Printk for ENOSPC due to lack of inodes

[Eric Sandeen]

On 2/26/12 6:37 PM, Raghavendra D Prabhu wrote:
> Hi,
> 
> While diagnosing a MySQL crash (on a Centos 5.7 box), I noticed that
> it had failed with ENOSPC earlier; it was rebooted after that; now
> after reboot, even though space was there, ENOSPC was showing up, I
> also did df -i and it showed inodes available. At this point,
> mounting with inode64 option was tried, which fixed it. 
> http://oss.sgi.com/archives/xfs/2011-03/msg00299.html helped me
> here.

Yeah, that's kind of a bummer.  And given the semi-ugly situation
we're in with inode32, maybe a syslog message would be good.

But a few things; I don't think we want to warn on every inode-allocation
ENOSPC.  Ideally I'd probably do a WARN_ON_ONCE or a ratelimited printk.
xfs_warn_once_per_fs()?  :)

I'd probably also want to only do it in the case where the ENOSPC was due
to either inode32, or due to fragmented freespace.

Without looking very hard yet; could this be done in xfs_dialloc() so that
the exact reason for the ENOSPC can be issued?  (maxicount, inode32, or
no free contiguous space...)

-Eric

> 
> So, I have attached a patch here.
> 
> =====================================================================
> 
> 
> When a ENOSPC is encountered and it is due to lack of inodes (particularly
> without inode64), it is not possible to detect this (df -i doesn't help here),
> so adding a printk which can aid in detecting this.
> 
> Signed-off-by: Raghavendra D Prabhu <rprabhu@wnohang.net>
> ---
>  fs/xfs/xfs_qm.c       |    3 +++
>  fs/xfs/xfs_vnodeops.c |   10 ++++++++--
>  2 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
> index c872fea..fbefa87 100644
> --- a/fs/xfs/xfs_qm.c
> +++ b/fs/xfs/xfs_qm.c
> @@ -987,6 +987,9 @@ xfs_qm_qino_alloc(
>  
>      error = xfs_dir_ialloc(&tp, NULL, S_IFREG, 1, 0, 0, 1, ip, &committed);
>      if (error) {
> +        if (error == ENOSPC)
> +            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
> +                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
>          xfs_trans_cancel(tp, XFS_TRANS_RELEASE_LOG_RES |
>                   XFS_TRANS_ABORT);
>          return error;
> diff --git a/fs/xfs/xfs_vnodeops.c b/fs/xfs/xfs_vnodeops.c
> index ebdb888..7542c36 100644
> --- a/fs/xfs/xfs_vnodeops.c
> +++ b/fs/xfs/xfs_vnodeops.c
> @@ -946,8 +946,11 @@ xfs_create(
>      error = xfs_dir_ialloc(&tp, dp, mode, is_dir ? 2 : 1, rdev,
>                     prid, resblks > 0, &ip, &committed);
>      if (error) {
> -        if (error == ENOSPC)
> +        if (error == ENOSPC) {
> +            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
> +                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
>              goto out_trans_cancel;
> +        }
>          goto out_trans_abort;
>      }
>  
> @@ -1610,8 +1613,11 @@ xfs_symlink(
>      error = xfs_dir_ialloc(&tp, dp, S_IFLNK | (mode & ~S_IFMT), 1, 0,
>                     prid, resblks > 0, &ip, NULL);
>      if (error) {
> -        if (error == ENOSPC)
> +        if (error == ENOSPC) {
> +            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
> +                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
>              goto error_return;
> +        }
>          goto error1;
>      }
>  
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Re: [PATCH] Printk for ENOSPC due to lack of inodes

[Raghavendra D Prabhu]

[-- Attachment #1.1: Type: text/plain, Size: 4649 bytes --]

Hi,

Thanks for the review. I have now moved it inside xfs_dialloc. 
Along with adding the message, I noticed that the loop 


	while (!agi->agi_freecount) {
	}

is redundant when noroom=1 and okalloc=0.

Also, xfs_ialloc_ag_alloc function in the loop calls  

============
	if (mp->m_maxicount &&
	    mp->m_sb.sb_icount + XFS_IALLOC_INODES(mp) > mp->m_maxicount) {

===============

condition again.

So I have moved xfs_tran_brelse etc. into the condition along 
with message.

Is this logic valid? If it is, then I will look into 
rate-limiting the message etc. 

* On Wed, Mar 07, 2012 at 11:39:24AM -0600, Eric Sandeen <sandeen@sandeen.net> wrote:
>On 2/26/12 6:37 PM, Raghavendra D Prabhu wrote:
>> Hi,
>>
>> While diagnosing a MySQL crash (on a Centos 5.7 box), I noticed that
>> it had failed with ENOSPC earlier; it was rebooted after that; now
>> after reboot, even though space was there, ENOSPC was showing up, I
>> also did df -i and it showed inodes available. At this point,
>> mounting with inode64 option was tried, which fixed it.
>> http://oss.sgi.com/archives/xfs/2011-03/msg00299.html helped me
>> here.
>
>Yeah, that's kind of a bummer.  And given the semi-ugly situation
>we're in with inode32, maybe a syslog message would be good.
>
>But a few things; I don't think we want to warn on every inode-allocation
>ENOSPC.  Ideally I'd probably do a WARN_ON_ONCE or a ratelimited printk.
>xfs_warn_once_per_fs()?  :)
>
>I'd probably also want to only do it in the case where the ENOSPC was due
>to either inode32, or due to fragmented freespace.
>
>Without looking very hard yet; could this be done in xfs_dialloc() so that
>the exact reason for the ENOSPC can be issued?  (maxicount, inode32, or
>no free contiguous space...)
>
>-Eric
>
>>
>> So, I have attached a patch here.
>>
>> =====================================================================
>>
>>
>> When a ENOSPC is encountered and it is due to lack of inodes (particularly
>> without inode64), it is not possible to detect this (df -i doesn't help here),
>> so adding a printk which can aid in detecting this.
>>
>> Signed-off-by: Raghavendra D Prabhu <rprabhu@wnohang.net>
>> ---
>>  fs/xfs/xfs_qm.c       |    3 +++
>>  fs/xfs/xfs_vnodeops.c |   10 ++++++++--
>>  2 files changed, 11 insertions(+), 2 deletions(-)
>>
>> diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c
>> index c872fea..fbefa87 100644
>> --- a/fs/xfs/xfs_qm.c
>> +++ b/fs/xfs/xfs_qm.c
>> @@ -987,6 +987,9 @@ xfs_qm_qino_alloc(
>>
>>      error = xfs_dir_ialloc(&tp, NULL, S_IFREG, 1, 0, 0, 1, ip, &committed);
>>      if (error) {
>> +        if (error == ENOSPC)
>> +            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
>> +                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
>>          xfs_trans_cancel(tp, XFS_TRANS_RELEASE_LOG_RES |
>>                   XFS_TRANS_ABORT);
>>          return error;
>> diff --git a/fs/xfs/xfs_vnodeops.c b/fs/xfs/xfs_vnodeops.c
>> index ebdb888..7542c36 100644
>> --- a/fs/xfs/xfs_vnodeops.c
>> +++ b/fs/xfs/xfs_vnodeops.c
>> @@ -946,8 +946,11 @@ xfs_create(
>>      error = xfs_dir_ialloc(&tp, dp, mode, is_dir ? 2 : 1, rdev,
>>                     prid, resblks > 0, &ip, &committed);
>>      if (error) {
>> -        if (error == ENOSPC)
>> +        if (error == ENOSPC) {
>> +            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
>> +                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
>>              goto out_trans_cancel;
>> +        }
>>          goto out_trans_abort;
>>      }
>>
>> @@ -1610,8 +1613,11 @@ xfs_symlink(
>>      error = xfs_dir_ialloc(&tp, dp, S_IFLNK | (mode & ~S_IFMT), 1, 0,
>>                     prid, resblks > 0, &ip, NULL);
>>      if (error) {
>> -        if (error == ENOSPC)
>> +        if (error == ENOSPC) {
>> +            xfs_err(mp, "Out of inodes: Required %d, Current %llu, Maximum %llu",
>> +                    XFS_IALLOC_INODES(mp), mp->m_sb.sb_icount, mp->m_maxicount);
>>              goto error_return;
>> +        }
>>          goto error1;
>>      }
>>
>>
>>
>> _______________________________________________
>> xfs mailing list
>> xfs@oss.sgi.com
>> http://oss.sgi.com/mailman/listinfo/xfs
>
>_______________________________________________
>xfs mailing list
>xfs@oss.sgi.com
>http://oss.sgi.com/mailman/listinfo/xfs
>




Regards,
-- 
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net

[-- Attachment #1.2: Type: application/pgp-signature, Size: 490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [PATCH] Printk for ENOSPC due to lack of inodes

[Eric Sandeen]

On 5/31/12 2:36 PM, Raghavendra D Prabhu wrote:
> Hi,
> 
> Thanks for the review. I have now moved it inside xfs_dialloc. Along with adding the message, I noticed that the loop
> 
>     while (!agi->agi_freecount) {
>     }
> 
> is redundant when noroom=1 and okalloc=0.
> 
> Also, xfs_ialloc_ag_alloc function in the loop calls 
> ============
>     if (mp->m_maxicount &&
>         mp->m_sb.sb_icount + XFS_IALLOC_INODES(mp) > mp->m_maxicount) {
> 
> ===============
> 
> condition again.
> 
> So I have moved xfs_tran_brelse etc. into the condition along with message.
> 
> Is this logic valid? If it is, then I will look into rate-limiting the message etc.

It'd be easiest to understand this new change as a patch rather than as a description.

If you are changing logic or flow in addition to adding the messages, it should almost certainly be sent as more than one patch.

Thanks,

-eric

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Re: [PATCH] Printk for ENOSPC due to lack of inodes

[Raghavendra D Prabhu]

[-- Attachment #1.1: Type: text/plain, Size: 1424 bytes --]

Hi,


* On Thu, May 31, 2012 at 10:31:21PM -0500, Eric Sandeen <sandeen@sandeen.net> wrote:
>On 5/31/12 2:36 PM, Raghavendra D Prabhu wrote:
>> Hi,
>>
>> Thanks for the review. I have now moved it inside xfs_dialloc. Along with adding the message, I noticed that the loop
>>
>>     while (!agi->agi_freecount) {
>>     }
>>
>> is redundant when noroom=1 and okalloc=0.
>>
>> Also, xfs_ialloc_ag_alloc function in the loop calls
>> ============
>>     if (mp->m_maxicount &&
>>         mp->m_sb.sb_icount + XFS_IALLOC_INODES(mp) > mp->m_maxicount) {
>>
>> ===============
>>
>> condition again.
>>
>> So I have moved xfs_tran_brelse etc. into the condition along with message.
>>
>> Is this logic valid? If it is, then I will look into rate-limiting the message etc.
>
>It'd be easiest to understand this new change as a patch rather than as a description.
Yeah, I would certainly send it as a patch. This was just for RFC. :)

>
>If you are changing logic or flow in addition to adding the messages, it should almost certainly be sent as more than one patch.
Thanks, I will make it so.
>
>Thanks,
>
>-eric
>
>_______________________________________________
>xfs mailing list
>xfs@oss.sgi.com
>http://oss.sgi.com/mailman/listinfo/xfs
>




Regards,
-- 
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net

[-- Attachment #1.2: Type: application/pgp-signature, Size: 490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Re: [PATCH] Printk for ENOSPC due to lack of inodes

[Raghavendra D Prabhu]

[-- Attachment #1.1: Type: text/plain, Size: 1510 bytes --]

Hi,

Thanks for reviewing earlier.

I have sent the patch series here from: http://oss.sgi.com/archives/xfs/2012-06/msg00020.html to http://oss.sgi.com/archives/xfs/2012-06/msg00023.html


* On Thu, May 31, 2012 at 10:31:21PM -0500, Eric Sandeen <sandeen@sandeen.net> wrote:
>On 5/31/12 2:36 PM, Raghavendra D Prabhu wrote:
>> Hi,
>>
>> Thanks for the review. I have now moved it inside xfs_dialloc. Along with adding the message, I noticed that the loop
>>
>>     while (!agi->agi_freecount) {
>>     }
>>
>> is redundant when noroom=1 and okalloc=0.
>>
>> Also, xfs_ialloc_ag_alloc function in the loop calls
>> ============
>>     if (mp->m_maxicount &&
>>         mp->m_sb.sb_icount + XFS_IALLOC_INODES(mp) > mp->m_maxicount) {
>>
>> ===============
>>
>> condition again.
>>
>> So I have moved xfs_tran_brelse etc. into the condition along with message.
>>
>> Is this logic valid? If it is, then I will look into rate-limiting the message etc.
>
>It'd be easiest to understand this new change as a patch rather than as a description.
>
>If you are changing logic or flow in addition to adding the messages, it should almost certainly be sent as more than one patch.
>
>Thanks,
>
>-eric
>
>_______________________________________________
>xfs mailing list
>xfs@oss.sgi.com
>http://oss.sgi.com/mailman/listinfo/xfs
>




Regards,
-- 
Raghavendra Prabhu
GPG Id : 0xD72BE977
Fingerprint: B93F EBCB 8E05 7039 CD3C A4B8 A616 DCA1 D72B E977
www: wnohang.net

[-- Attachment #1.2: Type: application/pgp-signature, Size: 490 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs