Kernel 3.0.0 + ext4 + ceph == ...

Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

fail. Epic fail.

Absolutely reproducible.

I have ceph cluster with this configuration:

8 physical servers
14 osd servers.
Each osd server have personal fs.
48T total size of ceph cluster.
17T used.

Now, step by step:

1. Stop ceph server osd0
/etc/init.d/ceph stop

2. Make fresh fs for osd
umount /osd.0
mkfs.ext4 /dev/sdc1
tune2fs -o journal_data_writeback /dev/sdc1
mount -a
# string from /etc/fstab:
# /dev/sdc1 /osd.0          ext4    
user_xattr,rw,noexec,nodev,noatime,nodiratime,data=writeback,barrier=0        
0       2
ceph mon getmap -o /tmp/monmap
cosd --mkfs -i 0 --monmap /tmp/monmap

3. Start ceph server osd0
/etc/init.d/ceph start

Now, make a big cup of coffee and begin to wait.

After completion of rebalancing do:
/etc/init.d/ceph stop
umount /osd.0
fsck.ext4 -fy /dev/sdc1

and see many-many messages like:

Inode 238551053, i_blocks is 24, should be 32.  Fix? yes

Inode 238551054, i_blocks is 40, should be 32.  Fix? yes

Inode 238551066, i_blocks is 24, should be 32.  Fix? yes

Inode 238944257, i_blocks is 8, should be 16.  Fix? yes

Inode 239206414, i_blocks is 8, should be 16.  Fix? yes

Inode 239206416, i_blocks is 40, should be 32.  Fix? yes

Inode 239206431, i_blocks is 8, should be 16.  Fix? yes

Inode 239206441, i_blocks is 24, should be 32.  Fix? yes

Voila.

P.S. No any message in syslog. No any message in console.

WBR,
     Fyodor.

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Christian Brunner]

This is also reproducable with a RHEL6.1 kernel (2.6.32-131.6.1.el6.x86_64). :(

Regards,
Christian

2011/7/30 Fyodor Ustinov <ufm@ufm.su>:
> fail. Epic fail.
>
> Absolutely reproducible.
>
> I have ceph cluster with this configuration:
>
> 8 physical servers
> 14 osd servers.
> Each osd server have personal fs.
> 48T total size of ceph cluster.
> 17T used.
>
> Now, step by step:
>
> 1. Stop ceph server osd0
> /etc/init.d/ceph stop
>
> 2. Make fresh fs for osd
> umount /osd.0
> mkfs.ext4 /dev/sdc1
> tune2fs -o journal_data_writeback /dev/sdc1
> mount -a
> # string from /etc/fstab:
> # /dev/sdc1 /osd.0          ext4
>  user_xattr,rw,noexec,nodev,noatime,nodiratime,data=writeback,barrier=0
>    0       2
> ceph mon getmap -o /tmp/monmap
> cosd --mkfs -i 0 --monmap /tmp/monmap
>
> 3. Start ceph server osd0
> /etc/init.d/ceph start
>
> Now, make a big cup of coffee and begin to wait.
>
> After completion of rebalancing do:
> /etc/init.d/ceph stop
> umount /osd.0
> fsck.ext4 -fy /dev/sdc1
>
> and see many-many messages like:
>
> Inode 238551053, i_blocks is 24, should be 32.  Fix? yes
>
> Inode 238551054, i_blocks is 40, should be 32.  Fix? yes
>
> Inode 238551066, i_blocks is 24, should be 32.  Fix? yes
>
> Inode 238944257, i_blocks is 8, should be 16.  Fix? yes
>
> Inode 239206414, i_blocks is 8, should be 16.  Fix? yes
>
> Inode 239206416, i_blocks is 40, should be 32.  Fix? yes
>
> Inode 239206431, i_blocks is 8, should be 16.  Fix? yes
>
> Inode 239206441, i_blocks is 24, should be 32.  Fix? yes
>
> Voila.
>
> P.S. No any message in syslog. No any message in console.
>
> WBR,
>    Fyodor.
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Fwd: Kernel 3.0.0 + ext4 + ceph == ...

[Christian Brunner]

Fyodor and I are struggling to get a fully stable ceph cluster up and running.

When we run an Ceph-Objectstore (OSD) ontop of an ext4 filesystem, we
get fsck errors, when we check the filesystem (see below).

Fyodor is running 3.0.
I am running a RHEL6.1 Kernel (2.6.32-131.6.1.el6.x86_64).

Any help or hints on how to trace the bug would be appreciated.

Thanks,
Christian

2011/7/30 Fyodor Ustinov <ufm@ufm.su>:
> fail. Epic fail.
>
> Absolutely reproducible.
>
> I have ceph cluster with this configuration:
>
> 8 physical servers
> 14 osd servers.
> Each osd server have personal fs.
> 48T total size of ceph cluster.
> 17T used.
>
> Now, step by step:
>
> 1. Stop ceph server osd0
> /etc/init.d/ceph stop
>
> 2. Make fresh fs for osd
> umount /osd.0
> mkfs.ext4 /dev/sdc1
> tune2fs -o journal_data_writeback /dev/sdc1
> mount -a
> # string from /etc/fstab:
> # /dev/sdc1 /osd.0          ext4
>  user_xattr,rw,noexec,nodev,noatime,nodiratime,data=writeback,barrier=0
>    0       2
> ceph mon getmap -o /tmp/monmap
> cosd --mkfs -i 0 --monmap /tmp/monmap
>
> 3. Start ceph server osd0
> /etc/init.d/ceph start
>
> Now, make a big cup of coffee and begin to wait.
>
> After completion of rebalancing do:
> /etc/init.d/ceph stop
> umount /osd.0
> fsck.ext4 -fy /dev/sdc1
>
> and see many-many messages like:
>
> Inode 238551053, i_blocks is 24, should be 32.  Fix? yes
>
> Inode 238551054, i_blocks is 40, should be 32.  Fix? yes
>
> Inode 238551066, i_blocks is 24, should be 32.  Fix? yes
>
> Inode 238944257, i_blocks is 8, should be 16.  Fix? yes
>
> Inode 239206414, i_blocks is 8, should be 16.  Fix? yes
>
> Inode 239206416, i_blocks is 40, should be 32.  Fix? yes
>
> Inode 239206431, i_blocks is 8, should be 16.  Fix? yes
>
> Inode 239206441, i_blocks is 24, should be 32.  Fix? yes
>
> Voila.
>
> P.S. No any message in syslog. No any message in console.
>
> WBR,
>    Fyodor.
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Theodore Tso]

On Jul 30, 2011, at 5:38 AM, Fyodor Ustinov wrote:

> fail. Epic fail
> 
> Inode 238551053, i_blocks is 24, should be 32.  Fix? yes
> 
> Inode 238551054, i_blocks is 40, should be 32.  Fix? yes
> ...

Fyodor, what kernel were you using?  Was it a RHEL 6.x
kernel, like Christian?

I haven't had an opportunity to play with Ceph in about six
months, but when I did I was using a much more recent
ext4 code base (backported onto 2.6.34, where as RHEL
was based on 2.6.32, but more importantly we were much
more aggressive at backporting fixes onto our kernel).

I didn't see anything like what you're reporting.

If this is with a much more recent kernel, I'd definitely be
interested.  Since the Ceph server is a pure userspace
application, if you can cause file system corruption which
e2fsck has to fix, that's definitely an ext4 bug.

Regards,

-- Ted

P.S  Ext4 in the RHEL 6.x kernel is I believe a technology
preview, which means it might not get full support from
Red Hat.  Still, if you do see those sorts of problems with
ext4 on a RHEL kernel, and you have a valid Red Hat 
support contract, I'd definitely encourage you to file a
bug with the distro.

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

On 07/30/2011 06:34 PM, Theodore Tso wrote:
> On Jul 30, 2011, at 5:38 AM, Fyodor Ustinov wrote:
>
>> fail. Epic fail
>>
>> Inode 238551053, i_blocks is 24, should be 32.  Fix? yes
>>
>> Inode 238551054, i_blocks is 40, should be 32.  Fix? yes
>> ...
> Fyodor, what kernel were you using?  Was it a RHEL 6.x
> kernel, like Christian?
As it is written in subject - 3.0.0 release.

It's Ubuntu 11.04 with custom kernel

Linux osd0 3.0.0-ufm+ #1 SMP Fri Jul 22 12:34:13 EEST 2011 x86_64 x86_64 
x86_64 GNU/Linux


WBR,
     Fyodor.

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Ted Ts'o]

On Sat, Jul 30, 2011 at 07:36:12PM +0300, Fyodor Ustinov wrote:
> As it is written in subject - 3.0.0 release.
> 
> It's Ubuntu 11.04 with custom kernel

Right, sorry, I missed that.  And just to be clear this wasn't an -rc
kernel but 3.0 final, right?

Hmm, looking through recent commits which will shortly be merged into
3.1, this one leaps out, but I'm not sure it's the cause --- how full
was your disk at the end of this exercise?

I haven't looked at Ceph in quite a while.  As I recall it was
primarily doing Direct I/O writes, correct?  Or does it use buffered
I/O?  And does it use the new "punch" ioctl to release blocks from the
middle of a file?  Ext4 added punch support in 3.0, and there are some
bug fixes that are going into 3.1, but I don't think there were any
that would lead to the failure mode you are seeing.

					- Ted

commit 7132de744ba76930d13033061018ddd7e3e8cd91
Author: Maxim Patlasov <maxim.patlasov@gmail.com>
Date:   Sun Jul 10 19:37:48 2011 -0400

    ext4: fix i_blocks/quota accounting when extent insertion fails
    
    The current implementation of ext4_free_blocks() always calls
    dquot_free_block This looks quite sensible in the most cases: blocks
    to be freed are associated with inode and were accounted in quota and
    i_blocks some time ago.
    
    However, there is a case when blocks to free were not accounted by the
    time calling ext4_free_blocks() yet:
    
    1. delalloc is on, write_begin pre-allocated some space in quota
    2. write-back happens, ext4 allocates some blocks in ext4_ext_map_blocks()
    3. then ext4_ext_map_blocks() gets an error (e.g.  ENOSPC) from
       ext4_ext_insert_extent() and calls ext4_free_blocks().
    
    In this scenario, ext4_free_blocks() calls dquot_free_block() who, in
    turn, decrements i_blocks for blocks which were not accounted yet (due
    to delalloc) After clean umount, e2fsck reports something like:
    
    > Inode 21, i_blocks is 5080, should be 5128.  Fix<y>?
    because i_blocks was erroneously decremented as explained above.
    
    The patch fixes the problem by passing the new flag
    EXT4_FREE_BLOCKS_NO_QUOT_UPDATE to ext4_free_blocks(), to request
    that the dquot_free_block() call be skipped.
    
    Signed-off-by: Maxim Patlasov <maxim.patlasov@gmail.com>
    Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
    Cc: stable@kernel.org

diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index 49d2cea..d13f3b5 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -526,6 +526,7 @@ struct ext4_new_group_data {
 #define EXT4_FREE_BLOCKS_METADATA	0x0001
 #define EXT4_FREE_BLOCKS_FORGET		0x0002
 #define EXT4_FREE_BLOCKS_VALIDATED	0x0004
+#define EXT4_FREE_BLOCKS_NO_QUOT_UPDATE	0x0008
 
 /*
  * ioctl commands
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 31ae5fb..a862138 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -3565,12 +3565,14 @@ int ext4_ext_map_blocks(handle_t *handle, struct inode *inode,
 
 	err = ext4_ext_insert_extent(handle, inode, path, &newex, flags);
 	if (err) {
+		int fb_flags = flags & EXT4_GET_BLOCKS_DELALLOC_RESERVE ?
+			EXT4_FREE_BLOCKS_NO_QUOT_UPDATE : 0;
 		/* free data blocks we just allocated */
 		/* not a good idea to call discard here directly,
 		 * but otherwise we'd need to call it every free() */
 		ext4_discard_preallocations(inode);
 		ext4_free_blocks(handle, inode, NULL, ext4_ext_pblock(&newex),
-				 ext4_ext_get_actual_len(&newex), 0);
+				 ext4_ext_get_actual_len(&newex), fb_flags);
 		goto out2;
 	}
 
diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 389386b..1900ec7 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -4637,7 +4637,7 @@ do_more:
 	}
 	ext4_mark_super_dirty(sb);
 error_return:
-	if (freed)
+	if (freed && !(flags & EXT4_FREE_BLOCKS_NO_QUOT_UPDATE))
 		dquot_free_block(inode, freed);
 	brelse(bitmap_bh);
 	ext4_std_error(sb, err);

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

Ted Ts'o <tytso <at> mit.edu> writes:


 > Right, sorry, I missed that.  And just to be clear this wasn't an -rc
 > kernel but 3.0 final, right?
Yes.

 >
 > Hmm, looking through recent commits which will shortly be merged into
 > 3.1, this one leaps out, but I'm not sure it's the cause --- how full
 > was your disk at the end of this exercise?
root@osd0:~# df -h /osd.0/
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdc1             3.6T  1.3T  2.2T  37% /osd.0
root@osd0:~#

 >
 > I haven't looked at Ceph in quite a while.  As I recall it was
 > primarily doing Direct I/O writes, correct?  Or does it use buffered
 > I/O?  And does it use the new "punch" ioctl to release blocks from the
 > middle of a file?  Ext4 added punch support in 3.0, and there are some
 > bug fixes that are going into 3.1, but I don't think there were any
 > that would lead to the failure mode you are seeing.

Ted, I'm sorry, I'm not a developer. I can not answer these questions, 
but I can try build 3.1

WBR,
     Fyodor

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Sage Weil]

On Sat, 30 Jul 2011, Ted Ts'o wrote:
> On Sat, Jul 30, 2011 at 07:36:12PM +0300, Fyodor Ustinov wrote:
> > As it is written in subject - 3.0.0 release.
> > 
> > It's Ubuntu 11.04 with custom kernel
> 
> Right, sorry, I missed that.  And just to be clear this wasn't an -rc
> kernel but 3.0 final, right?
> 
> Hmm, looking through recent commits which will shortly be merged into
> 3.1, this one leaps out, but I'm not sure it's the cause --- how full
> was your disk at the end of this exercise?
> 
> I haven't looked at Ceph in quite a while.  As I recall it was
> primarily doing Direct I/O writes, correct?  Or does it use buffered
> I/O?  And does it use the new "punch" ioctl to release blocks from the
> middle of a file?  Ext4 added punch support in 3.0, and there are some
> bug fixes that are going into 3.1, but I don't think there were any
> that would lead to the failure mode you are seeing.

Direct-io is used for the osd journal only; is that on the ext4 partition, 
Fyodor?  Everything else is buffered io.

We don't use the new punch ioctl.

We do use xattrs extensively, though; that was the last extN bug we 
uncovered.  That's where my money is.

Fyodor, if you set 'debug filestore = 10' you'll get a log of every 
operation on the fs in the osd log.  (Or close to it; there may be a few 
that we missed, but to a first approximation at least it'll describe the 
workload pretty well.)

sage

(BTW we'll be really happy if/when the large xattr patches from the Lustre 
guys make it into mainline!  The (4k?) limit on total xattrs is a problem 
for us.)


> 
> 					- Ted
> 
> commit 7132de744ba76930d13033061018ddd7e3e8cd91
> Author: Maxim Patlasov <maxim.patlasov@gmail.com>
> Date:   Sun Jul 10 19:37:48 2011 -0400
> 
>     ext4: fix i_blocks/quota accounting when extent insertion fails
>     
>     The current implementation of ext4_free_blocks() always calls
>     dquot_free_block This looks quite sensible in the most cases: blocks
>     to be freed are associated with inode and were accounted in quota and
>     i_blocks some time ago.
>     
>     However, there is a case when blocks to free were not accounted by the
>     time calling ext4_free_blocks() yet:
>     
>     1. delalloc is on, write_begin pre-allocated some space in quota
>     2. write-back happens, ext4 allocates some blocks in ext4_ext_map_blocks()
>     3. then ext4_ext_map_blocks() gets an error (e.g.  ENOSPC) from
>        ext4_ext_insert_extent() and calls ext4_free_blocks().
>     
>     In this scenario, ext4_free_blocks() calls dquot_free_block() who, in
>     turn, decrements i_blocks for blocks which were not accounted yet (due
>     to delalloc) After clean umount, e2fsck reports something like:
>     
>     > Inode 21, i_blocks is 5080, should be 5128.  Fix<y>?
>     because i_blocks was erroneously decremented as explained above.
>     
>     The patch fixes the problem by passing the new flag
>     EXT4_FREE_BLOCKS_NO_QUOT_UPDATE to ext4_free_blocks(), to request
>     that the dquot_free_block() call be skipped.
>     
>     Signed-off-by: Maxim Patlasov <maxim.patlasov@gmail.com>
>     Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
>     Cc: stable@kernel.org
> 
> diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
> index 49d2cea..d13f3b5 100644
> --- a/fs/ext4/ext4.h
> +++ b/fs/ext4/ext4.h
> @@ -526,6 +526,7 @@ struct ext4_new_group_data {
>  #define EXT4_FREE_BLOCKS_METADATA	0x0001
>  #define EXT4_FREE_BLOCKS_FORGET		0x0002
>  #define EXT4_FREE_BLOCKS_VALIDATED	0x0004
> +#define EXT4_FREE_BLOCKS_NO_QUOT_UPDATE	0x0008
>  
>  /*
>   * ioctl commands
> diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
> index 31ae5fb..a862138 100644
> --- a/fs/ext4/extents.c
> +++ b/fs/ext4/extents.c
> @@ -3565,12 +3565,14 @@ int ext4_ext_map_blocks(handle_t *handle, struct inode *inode,
>  
>  	err = ext4_ext_insert_extent(handle, inode, path, &newex, flags);
>  	if (err) {
> +		int fb_flags = flags & EXT4_GET_BLOCKS_DELALLOC_RESERVE ?
> +			EXT4_FREE_BLOCKS_NO_QUOT_UPDATE : 0;
>  		/* free data blocks we just allocated */
>  		/* not a good idea to call discard here directly,
>  		 * but otherwise we'd need to call it every free() */
>  		ext4_discard_preallocations(inode);
>  		ext4_free_blocks(handle, inode, NULL, ext4_ext_pblock(&newex),
> -				 ext4_ext_get_actual_len(&newex), 0);
> +				 ext4_ext_get_actual_len(&newex), fb_flags);
>  		goto out2;
>  	}
>  
> diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
> index 389386b..1900ec7 100644
> --- a/fs/ext4/mballoc.c
> +++ b/fs/ext4/mballoc.c
> @@ -4637,7 +4637,7 @@ do_more:
>  	}
>  	ext4_mark_super_dirty(sb);
>  error_return:
> -	if (freed)
> +	if (freed && !(flags & EXT4_FREE_BLOCKS_NO_QUOT_UPDATE))
>  		dquot_free_block(inode, freed);
>  	brelse(bitmap_bh);
>  	ext4_std_error(sb, err);
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

On 07/30/2011 08:21 PM, Sage Weil wrote:
>
>> Hmm, looking through recent commits which will shortly be merged into
>> 3.1, this one leaps out, but I'm not sure it's the cause --- how full
>> was your disk at the end of this exercise?
>>
>> I haven't looked at Ceph in quite a while.  As I recall it was
>> primarily doing Direct I/O writes, correct?  Or does it use buffered
>> I/O?  And does it use the new "punch" ioctl to release blocks from the
>> middle of a file?  Ext4 added punch support in 3.0, and there are some
>> bug fixes that are going into 3.1, but I don't think there were any
>> that would lead to the failure mode you are seeing.
> Direct-io is used for the osd journal only; is that on the ext4 partition,
> Fyodor?  Everything else is buffered io.
No, journal placed on tempfs.
> We don't use the new punch ioctl.
>
> We do use xattrs extensively, though; that was the last extN bug we
> uncovered.  That's where my money is.
>
> Fyodor, if you set 'debug filestore = 10' you'll get a log of every
> operation on the fs in the osd log.  (Or close to it; there may be a few
> that we missed, but to a first approximation at least it'll describe the
> workload pretty well.)
Ok, I will try it. But my system fs have only 16G. I'm not sure that it 
fits.

WBR,
     Fyodor.

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

On 07/30/2011 08:21 PM, Sage Weil wrote:
>
> Fyodor, if you set 'debug filestore = 10' you'll get a log of every
> operation on the fs in the osd log.  (Or close to it; there may be a few
> that we missed, but to a first approximation at least it'll describe the
> workload pretty well.)
>
Well, I do it. No need to wait for complete synchronisation.

Who wants to get this small file? :)

-rw------- 1 root root 412M 2011-07-30 20:51 /var/log/ceph/osd.0.log

WBR,
     Fyodor.

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Christian Brunner]

2011/7/30 Theodore Tso <tytso@mit.edu>:

>
> P.S  Ext4 in the RHEL 6.x kernel is I believe a technology
> preview, which means it might not get full support from
> Red Hat.  Still, if you do see those sorts of problems with
> ext4 on a RHEL kernel, and you have a valid Red Hat
> support contract, I'd definitely encourage you to file a
> bug with the distro.

Ext4 should be a standard feature in RHEL6 and we have a support
contract. However I have to wait until monday to file a bug report,
since I can't do this by myself, but we will do this.

Christian
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Ted Ts'o]

On Sat, Jul 30, 2011 at 10:21:13AM -0700, Sage Weil wrote:
> 
> We do use xattrs extensively, though; that was the last extN bug we 
> uncovered.  That's where my money is.

Hmm, yes.  That could very well be.  How big are the xattrs, and are
there cases where you:

a) start with a small xattr (where the total size is less than 128
bytes, so it can be stored in the inode table), and then increase it
something where it needs to be stored in an external block?

b) start with enough xattrs so it's large, and then delete all or most
of them?

I could easily believe we might have some bugs as we transition from
in-inode to external block storage, or vice versa.  I'll take a look
at the code and try to create some reproduction cases, but if you
could give me a handle on workload patterns of ceph around xattrs,
that would be interesting.

Another thing to try might be to format the disk with 128 byte inodes
(mke2fs -t ext4 -I 128 /dev/hdXX) and see if you can reproduce the
problem that way.  The support for in-inode xattrs is a new feature
(to ext4), and so it's a bit more likely that if there is a bug, it's
related to our in-inode xattr handling --- and using a 128 byte inode
would suppress that feature.  I don't recommend running that way, of
course, but it might help tell us if that's where we should be looking
for a bug.

> (BTW we'll be really happy if/when the large xattr patches from the Lustre 
> guys make it into mainline!  The (4k?) limit on total xattrs is a problem 
> for us.)

OK, good to know.  It hadn't been high priority for the ext4 team
(since I thought it was only the Lustre folks that really needed it),
but I'll escalate the priority of that on our todo list.

Thanks, regards,

						- Ted

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Sage Weil]

On Sat, 30 Jul 2011, Ted Ts'o wrote:
> On Sat, Jul 30, 2011 at 10:21:13AM -0700, Sage Weil wrote:
> > 
> > We do use xattrs extensively, though; that was the last extN bug we 
> > uncovered.  That's where my money is.
> 
> Hmm, yes.  That could very well be.  How big are the xattrs, and are
> there cases where you:
> 
> a) start with a small xattr (where the total size is less than 128
> bytes, so it can be stored in the inode table), and then increase it
> something where it needs to be stored in an external block?
> 
> b) start with enough xattrs so it's large, and then delete all or most
> of them?
> 
> I could easily believe we might have some bugs as we transition from
> in-inode to external block storage, or vice versa.  I'll take a look
> at the code and try to create some reproduction cases, but if you
> could give me a handle on workload patterns of ceph around xattrs,
> that would be interesting.

I would guess a, but it could also be a+b. 

Fyodor, can you take some of the corrupt inos that fsck complained about 
and see what files/directories they are?  find /osd.0 -inum NNN.  (I'm 
guessing the largest xattrs are on the collection directories, like 
/osd.0/current/something_head/.)  Then grep that filename out of the log 
to see exactly which operations took place.  The setattr log normally 
includes xattr size.

> Another thing to try might be to format the disk with 128 byte inodes
> (mke2fs -t ext4 -I 128 /dev/hdXX) and see if you can reproduce the
> problem that way.  The support for in-inode xattrs is a new feature
> (to ext4), and so it's a bit more likely that if there is a bug, it's
> related to our in-inode xattr handling --- and using a 128 byte inode
> would suppress that feature.  I don't recommend running that way, of
> course, but it might help tell us if that's where we should be looking
> for a bug.
>
> > (BTW we'll be really happy if/when the large xattr patches from the Lustre 
> > guys make it into mainline!  The (4k?) limit on total xattrs is a problem 
> > for us.)
> 
> OK, good to know.  It hadn't been high priority for the ext4 team
> (since I thought it was only the Lustre folks that really needed it),
> but I'll escalate the priority of that on our todo list.

Wonderful.

Thanks, Ted!
sage

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

On 07/31/2011 07:54 AM, Sage Weil wrote:
> On Sat, 30 Jul 2011, Ted Ts'o wrote:
>> On Sat, Jul 30, 2011 at 10:21:13AM -0700, Sage Weil wrote:
>>> We do use xattrs extensively, though; that was the last extN bug we
>>> uncovered.  That's where my money is.
>> Hmm, yes.  That could very well be.  How big are the xattrs, and are
>> there cases where you:
>>
>> a) start with a small xattr (where the total size is less than 128
>> bytes, so it can be stored in the inode table), and then increase it
>> something where it needs to be stored in an external block?
>>
>> b) start with enough xattrs so it's large, and then delete all or most
>> of them?
>>
>> I could easily believe we might have some bugs as we transition from
>> in-inode to external block storage, or vice versa.  I'll take a look
>> at the code and try to create some reproduction cases, but if you
>> could give me a handle on workload patterns of ceph around xattrs,
>> that would be interesting.
> I would guess a, but it could also be a+b.
>
> Fyodor, can you take some of the corrupt inos that fsck complained about
> and see what files/directories they are?  find /osd.0 -inum NNN.  (I'm
> guessing the largest xattrs are on the collection directories, like
> /osd.0/current/something_head/.)  Then grep that filename out of the log
> to see exactly which operations took place.  The setattr log normally
> includes xattr size.


/etc/init.d/ceph stop
umount /mnt/osd.0
mke2fs -t ext4 -I 128 /dev/sdc1
tune2fs -o journal_data_writeback /dev/sdc1
mount -a
mon getmap -o /tmp/monmap
cosd --mkfs -i 0 --monmap /tmp/monmap
/etc/init.d/ceph start
sleep 300
/etc/init.d/ceph stop
umount /osd.0
fsck.ext4 -f /dev/sdc1

Inode 99356878, i_blocks is 8208, should be 8200.

mount -a
root@osd0:~# find /osd.0 -inum 99356878

/osd.0/current/0.2a4_head/10000000468.0000007e_head

root@osd0:~# grep "10000000468\.0000007e" /var/log/ceph/osd.0.log
2011-07-31 09:57:20.859834 7f624c82a700 filestore(/osd.0) remove 
temp/10000000468.0000007e/head = -1
2011-07-31 09:57:20.861166 7f624c82a700 filestore(/osd.0) write 
temp/10000000468.0000007e/head 0~1048576 = 1048576
2011-07-31 09:57:20.990464 7f624c029700 filestore(/osd.0) write 
temp/10000000468.0000007e/head 1048576~1048576 = 1048576
2011-07-31 09:57:21.121648 7f624c029700 filestore(/osd.0) write 
temp/10000000468.0000007e/head 2097152~1048576 = 1048576
2011-07-31 09:57:21.265879 7f624c029700 filestore(/osd.0) write 
temp/10000000468.0000007e/head 3145728~1048576 = 1048576
2011-07-31 09:57:21.265952 7f624c029700 filestore(/osd.0) remove 
0.2a4_head/10000000468.0000007e/head = -1
2011-07-31 09:57:21.265995 7f624c029700 filestore(/osd.0) collection_add 
0.2a4_head/10000000468.0000007e/head temp/10000000468.0000007e/head = 0
2011-07-31 09:57:21.266025 7f624c029700 filestore(/osd.0) 
collection_remove temp/10000000468.0000007e/head = 0
2011-07-31 09:57:21.266134 7f624c029700 filestore(/osd.0) setattrs 
0.2a4_head/10000000468.0000007e/head = 26


WBR,
     Fyodor.

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Sage Weil]

On Sun, 31 Jul 2011, Fyodor Ustinov wrote:
> On 07/31/2011 07:54 AM, Sage Weil wrote:
> > On Sat, 30 Jul 2011, Ted Ts'o wrote:
> > > On Sat, Jul 30, 2011 at 10:21:13AM -0700, Sage Weil wrote:
> > > > We do use xattrs extensively, though; that was the last extN bug we
> > > > uncovered.  That's where my money is.
> > > Hmm, yes.  That could very well be.  How big are the xattrs, and are
> > > there cases where you:
> > > 
> > > a) start with a small xattr (where the total size is less than 128
> > > bytes, so it can be stored in the inode table), and then increase it
> > > something where it needs to be stored in an external block?
> > > 
> > > b) start with enough xattrs so it's large, and then delete all or most
> > > of them?
> > > 
> > > I could easily believe we might have some bugs as we transition from
> > > in-inode to external block storage, or vice versa.  I'll take a look
> > > at the code and try to create some reproduction cases, but if you
> > > could give me a handle on workload patterns of ceph around xattrs,
> > > that would be interesting.
> > I would guess a, but it could also be a+b.
> > 
> > Fyodor, can you take some of the corrupt inos that fsck complained about
> > and see what files/directories they are?  find /osd.0 -inum NNN.  (I'm
> > guessing the largest xattrs are on the collection directories, like
> > /osd.0/current/something_head/.)  Then grep that filename out of the log
> > to see exactly which operations took place.  The setattr log normally
> > includes xattr size.
> 
> 
> /etc/init.d/ceph stop
> umount /mnt/osd.0
> mke2fs -t ext4 -I 128 /dev/sdc1
> tune2fs -o journal_data_writeback /dev/sdc1
> mount -a
> mon getmap -o /tmp/monmap
> cosd --mkfs -i 0 --monmap /tmp/monmap
> /etc/init.d/ceph start
> sleep 300
> /etc/init.d/ceph stop
> umount /osd.0
> fsck.ext4 -f /dev/sdc1
> 
> Inode 99356878, i_blocks is 8208, should be 8200.
> 
> mount -a
> root@osd0:~# find /osd.0 -inum 99356878
> 
> /osd.0/current/0.2a4_head/10000000468.0000007e_head
> 
> root@osd0:~# grep "10000000468\.0000007e" /var/log/ceph/osd.0.log
> 2011-07-31 09:57:20.859834 7f624c82a700 filestore(/osd.0) remove
> temp/10000000468.0000007e/head = -1
> 2011-07-31 09:57:20.861166 7f624c82a700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 0~1048576 = 1048576
> 2011-07-31 09:57:20.990464 7f624c029700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 1048576~1048576 = 1048576
> 2011-07-31 09:57:21.121648 7f624c029700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 2097152~1048576 = 1048576
> 2011-07-31 09:57:21.265879 7f624c029700 filestore(/osd.0) write
> temp/10000000468.0000007e/head 3145728~1048576 = 1048576
> 2011-07-31 09:57:21.265952 7f624c029700 filestore(/osd.0) remove
> 0.2a4_head/10000000468.0000007e/head = -1
> 2011-07-31 09:57:21.265995 7f624c029700 filestore(/osd.0) collection_add
> 0.2a4_head/10000000468.0000007e/head temp/10000000468.0000007e/head = 0
> 2011-07-31 09:57:21.266025 7f624c029700 filestore(/osd.0) collection_remove
> temp/10000000468.0000007e/head = 0
> 2011-07-31 09:57:21.266134 7f624c029700 filestore(/osd.0) setattrs
> 0.2a4_head/10000000468.0000007e/head = 26

Hrm, I was hoping it wouldn't be a setattrs call.  The below will tell us 
what xattrs it is setting, but sadly you'll need to reproduce the whole 
thing again.  The above is only telling us the size of the last xattr of 
the set.

Ted, I'm assuming the i_blocks mismatch is likely on the same files that 
make the transition from/to in-inode xattrs?

sage


diff --git a/src/os/FileStore.cc b/src/os/FileStore.cc
index 8bdee6b..fb00cff 100644
--- a/src/os/FileStore.cc
+++ b/src/os/FileStore.cc
@@ -3570,6 +3570,7 @@ int FileStore::_setattrs(coll_t cid, const sobject_t& oid, map<string,bufferptr>
       val = "";
     // ??? Why do we skip setting all the other attrs if one fails?
     r = lfn_setxattr(cid, oid, n, val, p->second.length());
+    dout(10) << "setattrs " << cid << "/" << oid << " '" << p->first << "' len " << p->second.length() << " = " << r << dendl;
     if (r < 0) {
       derr << "FileStore::_setattrs: do_setxattr returned " << r << dendl;
       break;

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

On 07/31/2011 08:04 PM, Sage Weil wrote:
>
> Hrm, I was hoping it wouldn't be a setattrs call.  The below will tell us
> what xattrs it is setting, but sadly you'll need to reproduce the whole
> thing again.  The above is only telling us the size of the last xattr of
> the set.

ok, until hour or two I do it.

WBR,
Fyodor.

P.S. muttered under his breath: "я уже лет 15 как не программист. Я 
простой системный администратор. Ну за что мне это всё опять?"

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Fyodor Ustinov]

On 07/31/2011 08:04 PM, Sage Weil wrote:
> dout(10)<<  "setattrs "<<  cid<<  "/"<<  oid<<  " '"<<  p->first<<  "' len "<<  p->second.length()<<  " = "<<  r<<  dendl;
Well.

root@osd0:~# grep "10000000483\.000005d6" /var/log/ceph/osd.0.log
2011-07-31 23:06:49.172838 7f23c048c700 filestore(/osd.0) remove 
temp/10000000483.000005d6/head = -1
2011-07-31 23:06:49.173941 7f23c048c700 filestore(/osd.0) write 
temp/10000000483.000005d6/head 0~1048576 = 1048576
2011-07-31 23:06:49.900837 7f23c048c700 filestore(/osd.0) write 
temp/10000000483.000005d6/head 1048576~1048576 = 1048576
2011-07-31 23:06:49.985600 7f23c048c700 filestore(/osd.0) write 
temp/10000000483.000005d6/head 2097152~1048576 = 1048576
2011-07-31 23:06:50.114185 7f23c048c700 filestore(/osd.0) write 
temp/10000000483.000005d6/head 3145728~1048576 = 1048576
2011-07-31 23:06:50.114239 7f23c048c700 filestore(/osd.0) remove 
0.69_head/10000000483.000005d6/head = -1
2011-07-31 23:06:50.114287 7f23c048c700 filestore(/osd.0) collection_add 
0.69_head/10000000483.000005d6/head temp/10000000483.000005d6/head = 0
2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) 
collection_remove temp/10000000483.000005d6/head = 0
2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs 
0.69_head/10000000483.000005d6/head '_' len 161 = 161
2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs 
0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs 
0.69_head/10000000483.000005d6/head = 26
root@osd0:~#


WBR,
     Fyodor.

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Sage Weil]

On Sun, 31 Jul 2011, Fyodor Ustinov wrote:
> On 07/31/2011 08:04 PM, Sage Weil wrote:
> > dout(10)<<  "setattrs "<<  cid<<  "/"<<  oid<<  " '"<<  p->first<<  "' len
> > "<<  p->second.length()<<  " = "<<  r<<  dendl;
> Well.

Thanks!  Just to clarify a few things:
 
> root@osd0:~# grep "10000000483\.000005d6" /var/log/ceph/osd.0.log
> 2011-07-31 23:06:49.172838 7f23c048c700 filestore(/osd.0) remove
> temp/10000000483.000005d6/head = -1
       ^ The actual filename is a 10000000483.000005d6_head, ignore that 
last /.

> 2011-07-31 23:06:49.173941 7f23c048c700 filestore(/osd.0) write
> temp/10000000483.000005d6/head 0~1048576 = 1048576
> 2011-07-31 23:06:49.900837 7f23c048c700 filestore(/osd.0) write
> temp/10000000483.000005d6/head 1048576~1048576 = 1048576
> 2011-07-31 23:06:49.985600 7f23c048c700 filestore(/osd.0) write
> temp/10000000483.000005d6/head 2097152~1048576 = 1048576
> 2011-07-31 23:06:50.114185 7f23c048c700 filestore(/osd.0) write
> temp/10000000483.000005d6/head 3145728~1048576 = 1048576
> 2011-07-31 23:06:50.114239 7f23c048c700 filestore(/osd.0) remove
> 0.69_head/10000000483.000005d6/head = -1
> 2011-07-31 23:06:50.114287 7f23c048c700 filestore(/osd.0) collection_add
> 0.69_head/10000000483.000005d6/head temp/10000000483.000005d6/head = 0

This is link(2)

> 2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) collection_remove
> temp/10000000483.000005d6/head = 0

This is unlink(2)

> 2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs
> 0.69_head/10000000483.000005d6/head '_' len 161 = 161

The xattr names are prefixed with 'user.ceph.'

> 2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs
> 0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
> 2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs
> 0.69_head/10000000483.000005d6/head = 26

Does that tell you what to need, Ted?

Thanks!
sage

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Theodore Tso]

On Jul 31, 2011, at 4:42 PM, Sage Weil wrote:

> This is link(2)
> 
>> 2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) collection_remove
>> temp/10000000483.000005d6/head = 0
> 
> This is unlink(2)
> 
>> 2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs
>> 0.69_head/10000000483.000005d6/head '_' len 161 = 161
> 
> The xattr names are prefixed with 'user.ceph.'
> 
>> 2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs
>> 0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
>> 2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs
>> 0.69_head/10000000483.000005d6/head = 26
> 
> Does that tell you what to need, Ted?

So let me make sure I understand what happened.  Ceph made sure no file existed with this name, and created with the first write log entry:

2011-07-31 23:06:49.173941 7f23c048c700 filestore(/osd.0) write temp/10000000483.000005d6/head 0~1048576 = 1048576

It then linked it into this directory:

2011-07-31 23:06:50.114287 7f23c048c700 filestore(/osd.0) collection_add 0.69_head/10000000483.000005d6/head temp/10000000483.000005d6/head = 0

… and removed it from the temp directory:

2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) collection_remove temp/10000000483.000005d6/head = 0

and then created three inodes that totaled approximately 240 bytes in length or so (including the name and headers)

2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head '_' len 161 = 161
2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head = 26

… and then presumably the file was closed and that would have been the end of this file being touched by ceph, correct?

Thanks!!

-- Ted--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Sage Weil]

On Mon, 1 Aug 2011, Theodore Tso wrote:
> On Jul 31, 2011, at 4:42 PM, Sage Weil wrote:
> 
> > This is link(2)
> > 
> >> 2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) collection_remove
> >> temp/10000000483.000005d6/head = 0
> > 
> > This is unlink(2)
> > 
> >> 2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs
> >> 0.69_head/10000000483.000005d6/head '_' len 161 = 161
> > 
> > The xattr names are prefixed with 'user.ceph.'
> > 
> >> 2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs
> >> 0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
> >> 2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs
> >> 0.69_head/10000000483.000005d6/head = 26
> > 
> > Does that tell you what to need, Ted?
> 
> So let me make sure I understand what happened.  Ceph made sure no file existed with this name, and created with the first write log entry:
> 
> 2011-07-31 23:06:49.173941 7f23c048c700 filestore(/osd.0) write temp/10000000483.000005d6/head 0~1048576 = 1048576

The fd was actually closed here.

> It then linked it into this directory:
> 
> 2011-07-31 23:06:50.114287 7f23c048c700 filestore(/osd.0) collection_add 0.69_head/10000000483.000005d6/head temp/10000000483.000005d6/head = 0
> 
> ? and removed it from the temp directory:
> 
> 2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) collection_remove temp/10000000483.000005d6/head = 0
> 
> and then created three inodes that totaled approximately 240 bytes in length or so (including the name and headers)
                         xattrs
> 
> 2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head '_' len 161 = 161
> 2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
> 2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head = 26
> 
> ? and then presumably the file was closed and that would have been the 
> end of this file being touched by ceph, correct?

Yep!

sage

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Christian Brunner]

2011/8/1 Sage Weil <sage@newdream.net>:
> On Mon, 1 Aug 2011, Theodore Tso wrote:
>> On Jul 31, 2011, at 4:42 PM, Sage Weil wrote:
>>
>> > This is link(2)
>> >
>> >> 2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) collection_remove
>> >> temp/10000000483.000005d6/head = 0
>> >
>> > This is unlink(2)
>> >
>> >> 2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs
>> >> 0.69_head/10000000483.000005d6/head '_' len 161 = 161
>> >
>> > The xattr names are prefixed with 'user.ceph.'
>> >
>> >> 2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs
>> >> 0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
>> >> 2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs
>> >> 0.69_head/10000000483.000005d6/head = 26
>> >
>> > Does that tell you what to need, Ted?
>>
>> So let me make sure I understand what happened.  Ceph made sure no file existed with this name, and created with the first write log entry:
>>
>> 2011-07-31 23:06:49.173941 7f23c048c700 filestore(/osd.0) write temp/10000000483.000005d6/head 0~1048576 = 1048576
>
> The fd was actually closed here.
>
>> It then linked it into this directory:
>>
>> 2011-07-31 23:06:50.114287 7f23c048c700 filestore(/osd.0) collection_add 0.69_head/10000000483.000005d6/head temp/10000000483.000005d6/head = 0
>>
>> ? and removed it from the temp directory:
>>
>> 2011-07-31 23:06:50.114316 7f23c048c700 filestore(/osd.0) collection_remove temp/10000000483.000005d6/head = 0
>>
>> and then created three inodes that totaled approximately 240 bytes in length or so (including the name and headers)
>                         xattrs
>>
>> 2011-07-31 23:06:50.114384 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head '_' len 161 = 161
>> 2011-07-31 23:06:50.114406 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head 'snapset' len 26 = 26
>> 2011-07-31 23:06:50.114413 7f23c048c700 filestore(/osd.0) setattrs 0.69_head/10000000483.000005d6/head = 26
>>
>> ? and then presumably the file was closed and that would have been the
>> end of this file being touched by ceph, correct?
>
> Yep!

I tried to reproduce this without ceph, but wasn't able to...

In the meantime it seams, that I can also see the side effects on the
librbd side: I get an "librbd: data error!" when I do an "rbd copy".

When I look at the librbd code this is related to a sparse_read not
returning the right size of the object.

I don't know if it helps, but I think that the problem is also related
to sparse file usage.

Regards,
Christian
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Yehuda Sadeh Weinraub]

On Wed, Aug 3, 2011 at 7:16 AM, Christian Brunner <chb@muc.de> wrote:
...
> I tried to reproduce this without ceph, but wasn't able to...
>
> In the meantime it seams, that I can also see the side effects on the
> librbd side: I get an "librbd: data error!" when I do an "rbd copy".
>
> When I look at the librbd code this is related to a sparse_read not
> returning the right size of the object.
>
> I don't know if it helps, but I think that the problem is also related
> to sparse file usage.
>

There were a few sparse-read issues that we fixed not too long ago,
but should have been fixed for at least the previous ceph version. I'm
not sure what version you're using.
There was a ext4 fiemap issue that I was hitting on specific
environments but couldn't determine whether it was fixed in later
kernel versions (I was using 2.6.32). Now is a good time to try and
get to the bottom of it. Here's a script I was using to reproduce it:

#!/bin/sh
dd if=/dev/urandom of=bla bs=1 seek=$((0x6f000)) count=$((0x1000)); sync
dd if=/dev/urandom of=bla bs=1 seek=$((0x70000)) count=$((0x1000)); sync
dd if=/dev/urandom of=bla bs=1 seek=$((0x71000)) count=$((0x1000)); sync
dd if=/dev/urandom of=bla bs=1 seek=$((0x72000)) count=$((0x1000)); sync
dd if=/dev/urandom of=bla bs=1 seek=$((0x73000)) count=$((0x1000)); sync
dd if=/dev/urandom of=bla bs=1 seek=$((0x74000)) count=$((0x2000)); sync
dd if=/dev/urandom of=bla bs=1 seek=$((0x2ae000)) count=$((0x2000)); sync

You can compile and run the following utility to dump all the extents:
http://pastebin.com/h2Cnpk2Q

Thanks,
Yehuda

Oh, btw, You can effectively disable the use of fiemap by setting the
'filestore fiemap threshold' config option with large enough value
(e.g., anything bigger than 4 MB should be enough for rbd).

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Christian Brunner]

I tried 3.0.1 today, which contains the commit Theodore suggested and
was no longer able to reproduce the problem.

So I think the corruption we have seen is indeed related to:

commit 7132de744ba76930d13033061018ddd7e3e8cd91
Author: Maxim Patlasov <maxim.patlasov@gmail.com>
Date:   Sun Jul 10 19:37:48 2011 -0400

   ext4: fix i_blocks/quota accounting when extent insertion fails


I will now try to apply this patch to the RHEL6.1 kernel and see what
happens...

Thanks for your help.

Christian


2011/8/3 Yehuda Sadeh Weinraub <yehuda.sadeh@dreamhost.com>:
> On Wed, Aug 3, 2011 at 7:16 AM, Christian Brunner <chb@muc.de> wrote:
> ...
>> I tried to reproduce this without ceph, but wasn't able to...
>>
>> In the meantime it seams, that I can also see the side effects on the
>> librbd side: I get an "librbd: data error!" when I do an "rbd copy".
>>
>> When I look at the librbd code this is related to a sparse_read not
>> returning the right size of the object.
>>
>> I don't know if it helps, but I think that the problem is also related
>> to sparse file usage.
>>
>
> There were a few sparse-read issues that we fixed not too long ago,
> but should have been fixed for at least the previous ceph version. I'm
> not sure what version you're using.
> There was a ext4 fiemap issue that I was hitting on specific
> environments but couldn't determine whether it was fixed in later
> kernel versions (I was using 2.6.32). Now is a good time to try and
> get to the bottom of it. Here's a script I was using to reproduce it:
>
> #!/bin/sh
> dd if=/dev/urandom of=bla bs=1 seek=$((0x6f000)) count=$((0x1000)); sync
> dd if=/dev/urandom of=bla bs=1 seek=$((0x70000)) count=$((0x1000)); sync
> dd if=/dev/urandom of=bla bs=1 seek=$((0x71000)) count=$((0x1000)); sync
> dd if=/dev/urandom of=bla bs=1 seek=$((0x72000)) count=$((0x1000)); sync
> dd if=/dev/urandom of=bla bs=1 seek=$((0x73000)) count=$((0x1000)); sync
> dd if=/dev/urandom of=bla bs=1 seek=$((0x74000)) count=$((0x2000)); sync
> dd if=/dev/urandom of=bla bs=1 seek=$((0x2ae000)) count=$((0x2000)); sync
>
> You can compile and run the following utility to dump all the extents:
> http://pastebin.com/h2Cnpk2Q
>
> Thanks,
> Yehuda
>
> Oh, btw, You can effectively disable the use of fiemap by setting the
> 'filestore fiemap threshold' config option with large enough value
> (e.g., anything bigger than 4 MB should be enough for rbd).
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel 3.0.0 + ext4 + ceph == ...

[Christian Brunner]

I'm sorry, that I have to correct this:

The problem is still happening with 3.0.1. Although it only seems to
happen under high load now.

I also did some tracing (with 3.0.0 as the problem is easier to
reproduce here). What might be interesting to note is, that the
corruption does not occur, when I do an "strace -f cosd". (Maybe a
race condition?).

To reproduce the problem I have now setup a ceph cluster on a single machine
with replication between /ceph/osd.000 and /ceph/osd.001.

My setup now has only two active placement groups with 2 objects.

The corruption is happening, when I start replication from osd.000 to
osd.001. It is reproducible most of the time (but not allways), when I
do the following:

# mkfs.ext4 -T largefile /dev/sdb1
# mount -o noatime,user_xattr /dev/sdb1 /ceph/osd.001/
# cosd -i 001 --mkjournal --mkfs --monmap /tmp/monmap
# /usr/bin/cosd -d -i 001 -c /etc/ceph/ceph.conf


### wait until replication has finished and then stop the cosd

# umount /dev/sdb1
# fsck.ext4 -f /dev/sdb
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Inode 43, i_blocks is 8, should be 16.  Fix<y>? no

Inode 2078, i_blocks is 24, should be 16.  Fix<y>? no



I can also provide an e2image with the metadata and the strace output
of the cosd, if this would be helpful.

Regards,
Christian


2011/8/8 Christian Brunner <chb@muc.de>:
> I tried 3.0.1 today, which contains the commit Theodore suggested and
> was no longer able to reproduce the problem.
>
> So I think the corruption we have seen is indeed related to:
>
> commit 7132de744ba76930d13033061018ddd7e3e8cd91
> Author: Maxim Patlasov <maxim.patlasov@gmail.com>
> Date:   Sun Jul 10 19:37:48 2011 -0400
>
>   ext4: fix i_blocks/quota accounting when extent insertion fails
>
>
> I will now try to apply this patch to the RHEL6.1 kernel and see what
> happens...
>
> Thanks for your help.
>
> Christian
>
>
> 2011/8/3 Yehuda Sadeh Weinraub <yehuda.sadeh@dreamhost.com>:
>> On Wed, Aug 3, 2011 at 7:16 AM, Christian Brunner <chb@muc.de> wrote:
>> ...
>>> I tried to reproduce this without ceph, but wasn't able to...
>>>
>>> In the meantime it seams, that I can also see the side effects on the
>>> librbd side: I get an "librbd: data error!" when I do an "rbd copy".
>>>
>>> When I look at the librbd code this is related to a sparse_read not
>>> returning the right size of the object.
>>>
>>> I don't know if it helps, but I think that the problem is also related
>>> to sparse file usage.
>>>
>>
>> There were a few sparse-read issues that we fixed not too long ago,
>> but should have been fixed for at least the previous ceph version. I'm
>> not sure what version you're using.
>> There was a ext4 fiemap issue that I was hitting on specific
>> environments but couldn't determine whether it was fixed in later
>> kernel versions (I was using 2.6.32). Now is a good time to try and
>> get to the bottom of it. Here's a script I was using to reproduce it:
>>
>> #!/bin/sh
>> dd if=/dev/urandom of=bla bs=1 seek=$((0x6f000)) count=$((0x1000)); sync
>> dd if=/dev/urandom of=bla bs=1 seek=$((0x70000)) count=$((0x1000)); sync
>> dd if=/dev/urandom of=bla bs=1 seek=$((0x71000)) count=$((0x1000)); sync
>> dd if=/dev/urandom of=bla bs=1 seek=$((0x72000)) count=$((0x1000)); sync
>> dd if=/dev/urandom of=bla bs=1 seek=$((0x73000)) count=$((0x1000)); sync
>> dd if=/dev/urandom of=bla bs=1 seek=$((0x74000)) count=$((0x2000)); sync
>> dd if=/dev/urandom of=bla bs=1 seek=$((0x2ae000)) count=$((0x2000)); sync
>>
>> You can compile and run the following utility to dump all the extents:
>> http://pastebin.com/h2Cnpk2Q
>>
>> Thanks,
>> Yehuda
>>
>> Oh, btw, You can effectively disable the use of fiemap by setting the
>> 'filestore fiemap threshold' config option with large enough value
>> (e.g., anything bigger than 4 MB should be enough for rbd).
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fwd: Kernel 3.0.0 + ext4 + ceph == ...

[Eric Sandeen]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 7/30/11 9:53 AM, Christian Brunner wrote:
> Fyodor and I are struggling to get a fully stable ceph cluster up and running.
> 
> When we run an Ceph-Objectstore (OSD) ontop of an ext4 filesystem, we
> get fsck errors, when we check the filesystem (see below).

BTW, this should be fixed now as of my commit 6d6a435190bdf2e04c9465cde5bdc3ac68cf11a4
ext4: fix race in xattr block allocation path

I think it made its way to a couple older -stable kernels, too.

- -Eric


> Fyodor is running 3.0.
> I am running a RHEL6.1 Kernel (2.6.32-131.6.1.el6.x86_64).
> 
> Any help or hints on how to trace the bug would be appreciated.
> 
> Thanks,
> Christian
> 
> 2011/7/30 Fyodor Ustinov <ufm@ufm.su>:
>> fail. Epic fail.
>>
>> Absolutely reproducible.
>>
>> I have ceph cluster with this configuration:
>>
>> 8 physical servers
>> 14 osd servers.
>> Each osd server have personal fs.
>> 48T total size of ceph cluster.
>> 17T used.
>>
>> Now, step by step:
>>
>> 1. Stop ceph server osd0
>> /etc/init.d/ceph stop
>>
>> 2. Make fresh fs for osd
>> umount /osd.0
>> mkfs.ext4 /dev/sdc1
>> tune2fs -o journal_data_writeback /dev/sdc1
>> mount -a
>> # string from /etc/fstab:
>> # /dev/sdc1 /osd.0          ext4
>>  user_xattr,rw,noexec,nodev,noatime,nodiratime,data=writeback,barrier=0
>>    0       2
>> ceph mon getmap -o /tmp/monmap
>> cosd --mkfs -i 0 --monmap /tmp/monmap
>>
>> 3. Start ceph server osd0
>> /etc/init.d/ceph start
>>
>> Now, make a big cup of coffee and begin to wait.
>>
>> After completion of rebalancing do:
>> /etc/init.d/ceph stop
>> umount /osd.0
>> fsck.ext4 -fy /dev/sdc1
>>
>> and see many-many messages like:
>>
>> Inode 238551053, i_blocks is 24, should be 32.  Fix? yes
>>
>> Inode 238551054, i_blocks is 40, should be 32.  Fix? yes
>>
>> Inode 238551066, i_blocks is 24, should be 32.  Fix? yes
>>
>> Inode 238944257, i_blocks is 8, should be 16.  Fix? yes
>>
>> Inode 239206414, i_blocks is 8, should be 16.  Fix? yes
>>
>> Inode 239206416, i_blocks is 40, should be 32.  Fix? yes
>>
>> Inode 239206431, i_blocks is 8, should be 16.  Fix? yes
>>
>> Inode 239206441, i_blocks is 24, should be 32.  Fix? yes
>>
>> Voila.
>>
>> P.S. No any message in syslog. No any message in console.
>>
>> WBR,
>>    Fyodor.
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJOwoljAAoJECCuFpLhPd7gjxIQAJ7B+f7EYxBZ+48gUrncmB5r
Izkkv2ACza+27g/CUi9ku9j1o3pjZwLNhzo3Fj0gwweB3WaY9T+JMXnfInSFegeR
GCT/8XQqGWFVoRQKKc4wUBKGgW5f+3HTgYLqUY0Z38MqMHpIMXYswXdOSB1Wc4MC
p+jEjHmTWftklpIjv+Vm61AejpoUO93SFE5gUuBeKSZxwjifV1uTUXtaZCQXUG5N
EFz+sS7YvGrttAldK+lbiq7sa7IKINnB5lbDs5ChSZoytSF9hPIRgDOTLrkAZ+k8
YovLWbu2gwGMcZEhu3ZLJ7NdtZbn45A/fh/grNU8nezTo0cTHBTYZCLqtjsUDuMr
mwUIDNUEAv6LIz0OyeJMftDX4TzxjQyEQOgYg5wyCKCjE2Nyktyap2T5sAFKamJJ
pgTUt0JSpXgDnDBL7Y3M6RbY8DQsDHIir3A7aOwdINGKweNiJXBYC3LWYHIXY0bd
yoKXT6e/Bentlj+Peugg51bw91JtlqxJT4qJfk6HMF00uxrfWHlvzht7Lu61YxrW
LBQgNyQ+Gu1drHIHyIFu95UePhzEGQcLXB3YUe7BKFGe4Vde8Jcrwn1RSFmILU6H
o9jPncZVanQYy9URQqnrcHzqpRfViuVeyhuAUh3lPt4Q7jIrr+2Ug6xWxIkBrtTt
/iKT0p8+aR3HhakrGqp4
=VbZG
-----END PGP SIGNATURE-----