* [PATCH] EXT2: Remove superblock lock contention in ext2_statfs @ 2006-09-18 20:36 Dave Kleikamp 2006-09-18 23:38 ` Dave Kleikamp 0 siblings, 1 reply; 3+ messages in thread From: Dave Kleikamp @ 2006-09-18 20:36 UTC (permalink / raw) To: Andrew Morton; +Cc: Valerie Henson, linux-kernel, ext4 development EXT2: Remove superblock lock contention in ext2_statfs Fix a performance degradation introduced in 2.6.17. (30% degradation running dbench with 16 threads) Patch 21730eed11de42f22afcbd43f450a1872a0b5ea1, which claims to make EXT2_DEBUG work again, moves the taking of the kernel lock out of debug-only code in ext2_count_free_inodes and ext2_count_free_blocks and into ext2_statfs. This patch reverses that part of the patch. Signed-off-by: Dave Kleikamp <shaggy@austin.ibm.com> diff --git a/fs/ext2/balloc.c b/fs/ext2/balloc.c index d487043..fddefff 100644 --- a/fs/ext2/balloc.c +++ b/fs/ext2/balloc.c @@ -539,7 +539,6 @@ unsigned long ext2_count_free (struct bu #endif /* EXT2FS_DEBUG */ -/* Superblock must be locked */ unsigned long ext2_count_free_blocks (struct super_block * sb) { struct ext2_group_desc * desc; @@ -549,6 +548,7 @@ #ifdef EXT2FS_DEBUG unsigned long bitmap_count, x; struct ext2_super_block *es; + lock_super(sb); es = EXT2_SB(sb)->s_es; desc_count = 0; bitmap_count = 0; @@ -572,6 +572,7 @@ #ifdef EXT2FS_DEBUG printk("ext2_count_free_blocks: stored = %lu, computed = %lu, %lu\n", (long)le32_to_cpu(es->s_free_blocks_count), desc_count, bitmap_count); + unlock_super(sb); return bitmap_count; #else for (i = 0; i < EXT2_SB(sb)->s_groups_count; i++) { diff --git a/fs/ext2/ialloc.c b/fs/ext2/ialloc.c index de85c61..5d1d1c9 100644 --- a/fs/ext2/ialloc.c +++ b/fs/ext2/ialloc.c @@ -637,7 +637,6 @@ fail: return ERR_PTR(err); } -/* Superblock must be locked */ unsigned long ext2_count_free_inodes (struct super_block * sb) { struct ext2_group_desc *desc; @@ -649,6 +648,7 @@ #ifdef EXT2FS_DEBUG unsigned long bitmap_count = 0; struct buffer_head *bitmap_bh = NULL; + lock_super(sb); es = EXT2_SB(sb)->s_es; for (i = 0; i < EXT2_SB(sb)->s_groups_count; i++) { unsigned x; @@ -671,6 +671,7 @@ #ifdef EXT2FS_DEBUG printk("ext2_count_free_inodes: stored = %lu, computed = %lu, %lu\n", percpu_counter_read(&EXT2_SB(sb)->s_freeinodes_counter), desc_count, bitmap_count); + unlock_super(sb); return desc_count; #else for (i = 0; i < EXT2_SB(sb)->s_groups_count; i++) { diff --git a/fs/ext2/super.c b/fs/ext2/super.c index ca5bfb6..4286ff6 100644 --- a/fs/ext2/super.c +++ b/fs/ext2/super.c @@ -1083,7 +1083,6 @@ static int ext2_statfs (struct dentry * unsigned long overhead; int i; - lock_super(sb); if (test_opt (sb, MINIX_DF)) overhead = 0; else { @@ -1124,7 +1123,6 @@ static int ext2_statfs (struct dentry * buf->f_files = le32_to_cpu(sbi->s_es->s_inodes_count); buf->f_ffree = ext2_count_free_inodes (sb); buf->f_namelen = EXT2_NAME_LEN; - unlock_super(sb); return 0; } -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] EXT2: Remove superblock lock contention in ext2_statfs 2006-09-18 20:36 [PATCH] EXT2: Remove superblock lock contention in ext2_statfs Dave Kleikamp @ 2006-09-18 23:38 ` Dave Kleikamp 2006-09-25 16:46 ` Valerie Henson 0 siblings, 1 reply; 3+ messages in thread From: Dave Kleikamp @ 2006-09-18 23:38 UTC (permalink / raw) To: Andrew Morton; +Cc: Valerie Henson, linux-kernel, ext4 development On Mon, 2006-09-18 at 15:36 -0500, Dave Kleikamp wrote: > EXT2: Remove superblock lock contention in ext2_statfs > > Fix a performance degradation introduced in 2.6.17. (30% degradation running > dbench with 16 threads) > > Patch 21730eed11de42f22afcbd43f450a1872a0b5ea1, which claims to make > EXT2_DEBUG work again, moves the taking of the kernel lock out of debug-only > code in ext2_count_free_inodes and ext2_count_free_blocks and into > ext2_statfs. This patch reverses that part of the patch. > > Signed-off-by: Dave Kleikamp <shaggy@austin.ibm.com> Eric Sandeen pointed out to me that taking the superblock lock in ext2_count_free_* will cause a deadlock when EXT2FS_DEBUG is enabled, since the superblock is locked in write_super(). We found that the same problem was fixed in ext3 with this patch (forgive the long link): http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5b11687924e40790deb0d5f959247ade82196665;hp=2384f55f8aa520172c995965bd2f8a9740d53095 The patch below just removes the use of the superblock lock in the debug code. > diff --git a/fs/ext2/balloc.c b/fs/ext2/balloc.c > index d487043..fddefff 100644 > --- a/fs/ext2/balloc.c > +++ b/fs/ext2/balloc.c > @@ -539,7 +539,6 @@ unsigned long ext2_count_free (struct bu > > #endif /* EXT2FS_DEBUG */ > > -/* Superblock must be locked */ > unsigned long ext2_count_free_blocks (struct super_block * sb) > { > struct ext2_group_desc * desc; > @@ -549,6 +548,7 @@ #ifdef EXT2FS_DEBUG > unsigned long bitmap_count, x; > struct ext2_super_block *es; > > + lock_super(sb); > es = EXT2_SB(sb)->s_es; > desc_count = 0; > bitmap_count = 0; > @@ -572,6 +572,7 @@ #ifdef EXT2FS_DEBUG > printk("ext2_count_free_blocks: stored = %lu, computed = %lu, %lu\n", > (long)le32_to_cpu(es->s_free_blocks_count), > desc_count, bitmap_count); > + unlock_super(sb); > return bitmap_count; > #else > for (i = 0; i < EXT2_SB(sb)->s_groups_count; i++) { > diff --git a/fs/ext2/ialloc.c b/fs/ext2/ialloc.c > index de85c61..5d1d1c9 100644 > --- a/fs/ext2/ialloc.c > +++ b/fs/ext2/ialloc.c > @@ -637,7 +637,6 @@ fail: > return ERR_PTR(err); > } > > -/* Superblock must be locked */ > unsigned long ext2_count_free_inodes (struct super_block * sb) > { > struct ext2_group_desc *desc; > @@ -649,6 +648,7 @@ #ifdef EXT2FS_DEBUG > unsigned long bitmap_count = 0; > struct buffer_head *bitmap_bh = NULL; > > + lock_super(sb); > es = EXT2_SB(sb)->s_es; > for (i = 0; i < EXT2_SB(sb)->s_groups_count; i++) { > unsigned x; > @@ -671,6 +671,7 @@ #ifdef EXT2FS_DEBUG > printk("ext2_count_free_inodes: stored = %lu, computed = %lu, %lu\n", > percpu_counter_read(&EXT2_SB(sb)->s_freeinodes_counter), > desc_count, bitmap_count); > + unlock_super(sb); > return desc_count; > #else > for (i = 0; i < EXT2_SB(sb)->s_groups_count; i++) { > diff --git a/fs/ext2/super.c b/fs/ext2/super.c > index ca5bfb6..4286ff6 100644 > --- a/fs/ext2/super.c > +++ b/fs/ext2/super.c > @@ -1083,7 +1083,6 @@ static int ext2_statfs (struct dentry * > unsigned long overhead; > int i; > > - lock_super(sb); > if (test_opt (sb, MINIX_DF)) > overhead = 0; > else { > @@ -1124,7 +1123,6 @@ static int ext2_statfs (struct dentry * > buf->f_files = le32_to_cpu(sbi->s_es->s_inodes_count); > buf->f_ffree = ext2_count_free_inodes (sb); > buf->f_namelen = EXT2_NAME_LEN; > - unlock_super(sb); > return 0; > } > EXT2: Remove superblock lock contention in ext2_statfs Fix a performance degradation introduced in 2.6.17. (30% degradation running dbench with 16 threads) Patch 21730eed11de42f22afcbd43f450a1872a0b5ea1, which claims to make EXT2_DEBUG work again, moves the taking of the kernel lock out of debug-only code in ext2_count_free_inodes and ext2_count_free_blocks and into ext2_statfs. The same problem was fixed in ext3 by removing the lock completely (patch 5b11687924e40790deb0d5f959247ade82196665) Signed-off-by: Dave Kleikamp <shaggy@austin.ibm.com> diff --git a/fs/ext2/balloc.c b/fs/ext2/balloc.c index d487043..b1981d0 100644 --- a/fs/ext2/balloc.c +++ b/fs/ext2/balloc.c @@ -539,7 +539,6 @@ unsigned long ext2_count_free (struct bu #endif /* EXT2FS_DEBUG */ -/* Superblock must be locked */ unsigned long ext2_count_free_blocks (struct super_block * sb) { struct ext2_group_desc * desc; diff --git a/fs/ext2/ialloc.c b/fs/ext2/ialloc.c index de85c61..695f69c 100644 --- a/fs/ext2/ialloc.c +++ b/fs/ext2/ialloc.c @@ -637,7 +637,6 @@ fail: return ERR_PTR(err); } -/* Superblock must be locked */ unsigned long ext2_count_free_inodes (struct super_block * sb) { struct ext2_group_desc *desc; diff --git a/fs/ext2/super.c b/fs/ext2/super.c index ca5bfb6..4286ff6 100644 --- a/fs/ext2/super.c +++ b/fs/ext2/super.c @@ -1083,7 +1083,6 @@ static int ext2_statfs (struct dentry * unsigned long overhead; int i; - lock_super(sb); if (test_opt (sb, MINIX_DF)) overhead = 0; else { @@ -1124,7 +1123,6 @@ static int ext2_statfs (struct dentry * buf->f_files = le32_to_cpu(sbi->s_es->s_inodes_count); buf->f_ffree = ext2_count_free_inodes (sb); buf->f_namelen = EXT2_NAME_LEN; - unlock_super(sb); return 0; } -- David Kleikamp IBM Linux Technology Center ^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH] EXT2: Remove superblock lock contention in ext2_statfs 2006-09-18 23:38 ` Dave Kleikamp @ 2006-09-25 16:46 ` Valerie Henson 0 siblings, 0 replies; 3+ messages in thread From: Valerie Henson @ 2006-09-25 16:46 UTC (permalink / raw) To: Dave Kleikamp Cc: Andrew Morton, linux-kernel, ext4 development, Theodore Ts'o On Mon, Sep 18, 2006 at 06:38:05PM -0500, Dave Kleikamp wrote: > On Mon, 2006-09-18 at 15:36 -0500, Dave Kleikamp wrote: > > EXT2: Remove superblock lock contention in ext2_statfs > > > > Fix a performance degradation introduced in 2.6.17. (30% degradation running > > dbench with 16 threads) > > > > Patch 21730eed11de42f22afcbd43f450a1872a0b5ea1, which claims to make > > EXT2_DEBUG work again, moves the taking of the kernel lock out of debug-only > > code in ext2_count_free_inodes and ext2_count_free_blocks and into > > ext2_statfs. This patch reverses that part of the patch. > > > > Signed-off-by: Dave Kleikamp <shaggy@austin.ibm.com> > > Eric Sandeen pointed out to me that taking the superblock lock in > ext2_count_free_* will cause a deadlock when EXT2FS_DEBUG is enabled, > since the superblock is locked in write_super(). > > We found that the same problem was fixed in ext3 with this patch > (forgive the long link): > http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=5b11687924e40790deb0d5f959247ade82196665;hp=2384f55f8aa520172c995965bd2f8a9740d53095 > > The patch below just removes the use of the superblock lock in the debug > code. (Sorry for the delay; been on vacation.) Heh, I ran into the same lock nesting issues as you when I first tried to fix this; the lock debugging code found it for me. I asked for feedback on the locking issue when I submitted the patch, but no one had any opinions then, so I chose consistency over possible contention. Al Viro snorted at the idea of consistency in the results of statfs (I paraphrase his IRC remarks), and thinking about it further, I realized the debug code should not be doing these checks in statfs anyway; only on mount and unmount. This is because it appears that the block group accounting and the overall fs accounting are done non-atomically - see group_reserve_blocks() for example - and part of what the code does is reconcile these two numbers. It is legal for a valid fs to have the block group summaries and the fs-wide summaries out of sync, so the debug code could erroneously report an error, leading some poor soul on a wild goose chase. Removing this code from statfs also happens to fix the locking issues nicely. Rewriting this has been on my todo list for about 6 months now - anyone interested in grabbing it? I'm on #linuxfs on irc.oftc.net if anyone wants to chat about it. -VAL ^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2006-09-25 16:48 UTC | newest] Thread overview: 3+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2006-09-18 20:36 [PATCH] EXT2: Remove superblock lock contention in ext2_statfs Dave Kleikamp 2006-09-18 23:38 ` Dave Kleikamp 2006-09-25 16:46 ` Valerie Henson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox