[PATCH] ext4: Fix error handling on inode bitmap corruption

[PATCH] ext4: Fix error handling on inode bitmap corruption

[Jan Kara]

When insert_inode_locked() fails in ext4_new_inode() it most likely means inode
bitmap got corrupted and we allocated again inode which is already in use. Also
doing unlock_new_inode() during error recovery is wrong since the inode does
not have I_NEW set. Fix the problem by jumping to fail: (instead of fail_drop:)
which declares filesystem error and does not call unlock_new_inode().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/ext4/ialloc.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c
index 00beb4f..8fb6844 100644
--- a/fs/ext4/ialloc.c
+++ b/fs/ext4/ialloc.c
@@ -885,8 +885,12 @@ got:
 	if (IS_DIRSYNC(inode))
 		ext4_handle_sync(handle);
 	if (insert_inode_locked(inode) < 0) {
-		err = -EINVAL;
-		goto fail_drop;
+		/*
+		 * Likely a bitmap corruption causing inode to be allocated
+		 * twice.
+		 */
+		err = -EIO;
+		goto fail;
 	}
 	spin_lock(&sbi->s_next_gen_lock);
 	inode->i_generation = sbi->s_next_generation++;
-- 
1.7.1

[PATCH] ext3: Fix error handling on inode bitmap corruption

[Jan Kara]

When insert_inode_locked() fails in ext3_new_inode() it most likely
means inode bitmap got corrupted and we allocated again inode which
is already in use. Also doing unlock_new_inode() during error recovery
is wrong since inode does not have I_NEW set. Fix the problem by jumping
to fail: (instead of fail_drop:) which declares filesystem error and
does not call unlock_new_inode().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/ext3/ialloc.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/fs/ext3/ialloc.c b/fs/ext3/ialloc.c
index 5c866e0..adae962 100644
--- a/fs/ext3/ialloc.c
+++ b/fs/ext3/ialloc.c
@@ -525,8 +525,12 @@ got:
 	if (IS_DIRSYNC(inode))
 		handle->h_sync = 1;
 	if (insert_inode_locked(inode) < 0) {
-		err = -EINVAL;
-		goto fail_drop;
+		/*
+		 * Likely a bitmap corruption causing inode to be allocated
+		 * twice.
+		 */
+		err = -EIO;
+		goto fail;
 	}
 	spin_lock(&sbi->s_next_gen_lock);
 	inode->i_generation = sbi->s_next_generation++;
-- 
1.7.1

Re: [PATCH] ext4: Fix error handling on inode bitmap corruption

[Eric Sandeen]

On 12/8/11 2:28 PM, Jan Kara wrote:
> When insert_inode_locked() fails in ext4_new_inode() it most likely means inode
> bitmap got corrupted and we allocated again inode which is already in use. Also
> doing unlock_new_inode() during error recovery is wrong since the inode does
> not have I_NEW set. Fix the problem by jumping to fail: (instead of fail_drop:)
> which declares filesystem error and does not call unlock_new_inode().

This looks an awful lot like the:

[PATCH 3/6 V2] ext4: fix up error handling for insert_inode_locked

I sent a couple days ago.

Except yours is better ;)  I had overlooked the existing fail: target.

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
>  fs/ext4/ialloc.c |    8 ++++++--
>  1 files changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c
> index 00beb4f..8fb6844 100644
> --- a/fs/ext4/ialloc.c
> +++ b/fs/ext4/ialloc.c
> @@ -885,8 +885,12 @@ got:
>  	if (IS_DIRSYNC(inode))
>  		ext4_handle_sync(handle);
>  	if (insert_inode_locked(inode) < 0) {
> -		err = -EINVAL;
> -		goto fail_drop;
> +		/*
> +		 * Likely a bitmap corruption causing inode to be allocated
> +		 * twice.
> +		 */
> +		err = -EIO;
> +		goto fail;
>  	}
>  	spin_lock(&sbi->s_next_gen_lock);
>  	inode->i_generation = sbi->s_next_generation++;

Re: [PATCH] ext3: Fix error handling on inode bitmap corruption

[Eric Sandeen]

On 12/8/11 2:28 PM, Jan Kara wrote:
> When insert_inode_locked() fails in ext3_new_inode() it most likely
> means inode bitmap got corrupted and we allocated again inode which
> is already in use. Also doing unlock_new_inode() during error recovery
> is wrong since inode does not have I_NEW set. Fix the problem by jumping
> to fail: (instead of fail_drop:) which declares filesystem error and
> does not call unlock_new_inode().
> 
> Signed-off-by: Jan Kara <jack@suse.cz>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

I think ext2 could use the same treatment.

BTW, though, have you recently started seeing the issue?  We have
people hitting this when resuming after suspend; it seems likely
that the bitmap did get corrupted though, based on some other
things seen in similar bugs.

-Eric

> ---
>  fs/ext3/ialloc.c |    8 ++++++--
>  1 files changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/ext3/ialloc.c b/fs/ext3/ialloc.c
> index 5c866e0..adae962 100644
> --- a/fs/ext3/ialloc.c
> +++ b/fs/ext3/ialloc.c
> @@ -525,8 +525,12 @@ got:
>  	if (IS_DIRSYNC(inode))
>  		handle->h_sync = 1;
>  	if (insert_inode_locked(inode) < 0) {
> -		err = -EINVAL;
> -		goto fail_drop;
> +		/*
> +		 * Likely a bitmap corruption causing inode to be allocated
> +		 * twice.
> +		 */
> +		err = -EIO;
> +		goto fail;
>  	}
>  	spin_lock(&sbi->s_next_gen_lock);
>  	inode->i_generation = sbi->s_next_generation++;

Re: [PATCH] ext4: Fix error handling on inode bitmap corruption

[Jan Kara]

On Thu 08-12-11 14:44:48, Eric Sandeen wrote:
> On 12/8/11 2:28 PM, Jan Kara wrote:
> > When insert_inode_locked() fails in ext4_new_inode() it most likely means inode
> > bitmap got corrupted and we allocated again inode which is already in use. Also
> > doing unlock_new_inode() during error recovery is wrong since the inode does
> > not have I_NEW set. Fix the problem by jumping to fail: (instead of fail_drop:)
> > which declares filesystem error and does not call unlock_new_inode().
> 
> This looks an awful lot like the:
> 
> [PATCH 3/6 V2] ext4: fix up error handling for insert_inode_locked
> 
> I sent a couple days ago.
> 
> Except yours is better ;)  I had overlooked the existing fail: target.
> 
> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
  Ah, I haven't catch up with mailing lists after a few days of vacation
yet... Thanks for review.

								Honza

> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
> >  fs/ext4/ialloc.c |    8 ++++++--
> >  1 files changed, 6 insertions(+), 2 deletions(-)
> > 
> > diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c
> > index 00beb4f..8fb6844 100644
> > --- a/fs/ext4/ialloc.c
> > +++ b/fs/ext4/ialloc.c
> > @@ -885,8 +885,12 @@ got:
> >  	if (IS_DIRSYNC(inode))
> >  		ext4_handle_sync(handle);
> >  	if (insert_inode_locked(inode) < 0) {
> > -		err = -EINVAL;
> > -		goto fail_drop;
> > +		/*
> > +		 * Likely a bitmap corruption causing inode to be allocated
> > +		 * twice.
> > +		 */
> > +		err = -EIO;
> > +		goto fail;
> >  	}
> >  	spin_lock(&sbi->s_next_gen_lock);
> >  	inode->i_generation = sbi->s_next_generation++;
> 
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [PATCH] ext3: Fix error handling on inode bitmap corruption

[Jan Kara]

On Thu 08-12-11 14:46:09, Eric Sandeen wrote:
> On 12/8/11 2:28 PM, Jan Kara wrote:
> > When insert_inode_locked() fails in ext3_new_inode() it most likely
> > means inode bitmap got corrupted and we allocated again inode which
> > is already in use. Also doing unlock_new_inode() during error recovery
> > is wrong since inode does not have I_NEW set. Fix the problem by jumping
> > to fail: (instead of fail_drop:) which declares filesystem error and
> > does not call unlock_new_inode().
> > 
> > Signed-off-by: Jan Kara <jack@suse.cz>
> 
> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
> 
> I think ext2 could use the same treatment.
> 
> BTW, though, have you recently started seeing the issue?  We have
> people hitting this when resuming after suspend; it seems likely
> that the bitmap did get corrupted though, based on some other
> things seen in similar bugs.
  Interesting. I've got a report from IBM testing ext3 on SLE11 SP2 kernel
(3.0 based). Their filesystem got damaged (might be HW issue, not sure yet)
and they also observed warnings from unlock_new_inode().

								Honza
> > ---
> >  fs/ext3/ialloc.c |    8 ++++++--
> >  1 files changed, 6 insertions(+), 2 deletions(-)
> > 
> > diff --git a/fs/ext3/ialloc.c b/fs/ext3/ialloc.c
> > index 5c866e0..adae962 100644
> > --- a/fs/ext3/ialloc.c
> > +++ b/fs/ext3/ialloc.c
> > @@ -525,8 +525,12 @@ got:
> >  	if (IS_DIRSYNC(inode))
> >  		handle->h_sync = 1;
> >  	if (insert_inode_locked(inode) < 0) {
> > -		err = -EINVAL;
> > -		goto fail_drop;
> > +		/*
> > +		 * Likely a bitmap corruption causing inode to be allocated
> > +		 * twice.
> > +		 */
> > +		err = -EIO;
> > +		goto fail;
> >  	}
> >  	spin_lock(&sbi->s_next_gen_lock);
> >  	inode->i_generation = sbi->s_next_generation++;
> 
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [PATCH] ext3: Fix error handling on inode bitmap corruption

[Eric Sandeen]

On 12/8/11 4:28 PM, Jan Kara wrote:
> On Thu 08-12-11 14:46:09, Eric Sandeen wrote:
>> On 12/8/11 2:28 PM, Jan Kara wrote:
>>> When insert_inode_locked() fails in ext3_new_inode() it most likely
>>> means inode bitmap got corrupted and we allocated again inode which
>>> is already in use. Also doing unlock_new_inode() during error recovery
>>> is wrong since inode does not have I_NEW set. Fix the problem by jumping
>>> to fail: (instead of fail_drop:) which declares filesystem error and
>>> does not call unlock_new_inode().
>>>
>>> Signed-off-by: Jan Kara <jack@suse.cz>
>>
>> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
>>
>> I think ext2 could use the same treatment.
>>
>> BTW, though, have you recently started seeing the issue?  We have
>> people hitting this when resuming after suspend; it seems likely
>> that the bitmap did get corrupted though, based on some other
>> things seen in similar bugs.
>   Interesting. I've got a report from IBM testing ext3 on SLE11 SP2 kernel
> (3.0 based). Their filesystem got damaged (might be HW issue, not sure yet)
> and they also observed warnings from unlock_new_inode().

It may be that it has been failing in other ways, but now we get the WARN_ON
and the long backtrace so it's reported more frequently...

I think there might be a hibernate issue that is causing the underlying
corruption, trying to look into that now.

-Eric

> 								Honza
>>> ---
>>>  fs/ext3/ialloc.c |    8 ++++++--
>>>  1 files changed, 6 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/fs/ext3/ialloc.c b/fs/ext3/ialloc.c
>>> index 5c866e0..adae962 100644
>>> --- a/fs/ext3/ialloc.c
>>> +++ b/fs/ext3/ialloc.c
>>> @@ -525,8 +525,12 @@ got:
>>>  	if (IS_DIRSYNC(inode))
>>>  		handle->h_sync = 1;
>>>  	if (insert_inode_locked(inode) < 0) {
>>> -		err = -EINVAL;
>>> -		goto fail_drop;
>>> +		/*
>>> +		 * Likely a bitmap corruption causing inode to be allocated
>>> +		 * twice.
>>> +		 */
>>> +		err = -EIO;
>>> +		goto fail;
>>>  	}
>>>  	spin_lock(&sbi->s_next_gen_lock);
>>>  	inode->i_generation = sbi->s_next_generation++;
>>

Re: [PATCH] ext3: Fix error handling on inode bitmap corruption

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 877 bytes --]

On Thu 08-12-11 14:46:09, Eric Sandeen wrote:
> On 12/8/11 2:28 PM, Jan Kara wrote:
> > When insert_inode_locked() fails in ext3_new_inode() it most likely
> > means inode bitmap got corrupted and we allocated again inode which
> > is already in use. Also doing unlock_new_inode() during error recovery
> > is wrong since inode does not have I_NEW set. Fix the problem by jumping
> > to fail: (instead of fail_drop:) which declares filesystem error and
> > does not call unlock_new_inode().
> > 
> > Signed-off-by: Jan Kara <jack@suse.cz>
> 
> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
> 
> I think ext2 could use the same treatment.
  Good point. Attached is a similar patch for ext2 (I didn't use your patch
so that all ext? are consistent and declare filesystem error when
insert_inode_locked() fails). Thanks.

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[-- Attachment #2: 0001-ext2-Fix-error-handling-on-inode-bitmap-corruption.patch --]
[-- Type: text/x-patch, Size: 1234 bytes --]

>From 9d1602d9a8b895d0b6dbb30a6d2a148558912dad Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Fri, 9 Dec 2011 00:08:58 +0100
Subject: [PATCH] ext2: Fix error handling on inode bitmap corruption

When insert_inode_locked() fails in ext2_new_inode() it most likely means inode
bitmap got corrupted and we allocated again inode which is already in use. Also
doing unlock_new_inode() during error recovery is wrong since the inode does
not have I_NEW set. Fix the problem by informing about filesystem error and
jumping to fail: (instead of fail_drop:) which doesn't call unlock_new_inode().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/ext2/ialloc.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/fs/ext2/ialloc.c b/fs/ext2/ialloc.c
index c4e81df..78502c1 100644
--- a/fs/ext2/ialloc.c
+++ b/fs/ext2/ialloc.c
@@ -573,8 +573,11 @@ got:
 	inode->i_generation = sbi->s_next_generation++;
 	spin_unlock(&sbi->s_next_gen_lock);
 	if (insert_inode_locked(inode) < 0) {
-		err = -EINVAL;
-		goto fail_drop;
+		ext2_error(sb, "ext2_new_inode",
+			   "inode number already in use - inode=%lu",
+			   (unsigned long) ino);
+		err = -EIO;
+		goto fail;
 	}
 
 	dquot_initialize(inode);
-- 
1.7.1

Re: [PATCH] ext3: Fix error handling on inode bitmap corruption

[Eric Sandeen]

On 12/8/11 5:13 PM, Jan Kara wrote:
> On Thu 08-12-11 14:46:09, Eric Sandeen wrote:
>> On 12/8/11 2:28 PM, Jan Kara wrote:
>>> When insert_inode_locked() fails in ext3_new_inode() it most likely
>>> means inode bitmap got corrupted and we allocated again inode which
>>> is already in use. Also doing unlock_new_inode() during error recovery
>>> is wrong since inode does not have I_NEW set. Fix the problem by jumping
>>> to fail: (instead of fail_drop:) which declares filesystem error and
>>> does not call unlock_new_inode().
>>>
>>> Signed-off-by: Jan Kara <jack@suse.cz>
>>
>> Reviewed-by: Eric Sandeen <sandeen@redhat.com>
>>
>> I think ext2 could use the same treatment.
>   Good point. Attached is a similar patch for ext2 (I didn't use your patch
> so that all ext? are consistent and declare filesystem error when
> insert_inode_locked() fails). Thanks.
> 
> 								Honza
> 


> 
> From 9d1602d9a8b895d0b6dbb30a6d2a148558912dad Mon Sep 17 00:00:00 2001
> From: Jan Kara <jack@suse.cz>
> Date: Fri, 9 Dec 2011 00:08:58 +0100
> Subject: [PATCH] ext2: Fix error handling on inode bitmap corruption
> 
> When insert_inode_locked() fails in ext2_new_inode() it most likely means inode
> bitmap got corrupted and we allocated again inode which is already in use. Also
> doing unlock_new_inode() during error recovery is wrong since the inode does
> not have I_NEW set. Fix the problem by informing about filesystem error and
> jumping to fail: (instead of fail_drop:) which doesn't call unlock_new_inode().
> 
> Signed-off-by: Jan Kara <jack@suse.cz>

Reviewed-by: Eric Sandeen <sandeen@redhat.com>

> ---
>  fs/ext2/ialloc.c |    7 +++++--
>  1 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/ext2/ialloc.c b/fs/ext2/ialloc.c
> index c4e81df..78502c1 100644
> --- a/fs/ext2/ialloc.c
> +++ b/fs/ext2/ialloc.c
> @@ -573,8 +573,11 @@ got:
>  	inode->i_generation = sbi->s_next_generation++;
>  	spin_unlock(&sbi->s_next_gen_lock);
>  	if (insert_inode_locked(inode) < 0) {
> -		err = -EINVAL;
> -		goto fail_drop;
> +		ext2_error(sb, "ext2_new_inode",
> +			   "inode number already in use - inode=%lu",
> +			   (unsigned long) ino);
> +		err = -EIO;
> +		goto fail;
>  	}
>  
>  	dquot_initialize(inode);
> -- 1.7.1 

Re: [PATCH] ext4: Fix error handling on inode bitmap corruption

[Ted Ts'o]

On Thu, Dec 08, 2011 at 09:28:34PM +0100, Jan Kara wrote:
> When insert_inode_locked() fails in ext4_new_inode() it most likely means inode
> bitmap got corrupted and we allocated again inode which is already in use. Also
> doing unlock_new_inode() during error recovery is wrong since the inode does
> not have I_NEW set. Fix the problem by jumping to fail: (instead of fail_drop:)
> which declares filesystem error and does not call unlock_new_inode().
> 
> Signed-off-by: Jan Kara <jack@suse.cz>

Applied, thanks.

					- Ted