Very very slow rm/unlink of large sparse files

Very very slow rm/unlink of large sparse files

[Stephen D. Williams]

[-- Attachment #1: Type: text/plain, Size: 1868 bytes --]

Linux kernel 2.4.21-99 (SuSe 9 Pro kernel), and also experienced with 2.6.1.

When deleting a large file (4-10GB), at least somewhat sparse (created 
completely sparse and filled in as a reiserfs filesystem under 
user-mode-linux), the filesystem layer spends a very very long time, on 
the order of several hours per GB, with constant disk activity trying to 
delete the file.  If the system is rebooted, the partial unlink is 
continued at the same very very slow speed.  A read-only mount works 
just fine.

This is has happened several times as I have reinstalled and reoriented 
systems.  The filesystems are 40-500GB and are running on top of RAID 
0,1, or 5 (ReiserFS->RAID 0, 1, or 5), sometimes with LVM 
(ReiserFS->LVM->RAID 1).  It seems completely repeatable and 
consistent.  I have been waiting about 24 hours for an unlink of a 9.9GB 
file, formatted as ReiserFS and mostly filled with data.  The filesystem 
involved is accessible, but very very slowly.  An ls of a small 
directory takes at least 30 minutes.  The system has 2GB ram and has at 
least half of that free in buffers and cache.  System load is over 8 
because of things waiting for disk I/O, but CPU is 1% user and 3% 
system.  The disk LED is flashing at about 50% load, constant activity.

dd if=/dev/zero of=u.img bs=1 count=1 seek=$((1024 * 1024 * 4300))
mkfs.reiserfs -f u.img
mount -o loop u.img /mnt
cp -r /etc /mnt   # I'm not sure how much data needs to be present, if 
any, to exacerbate the problem.
umount /mnt
rm -f u.img


Is this a known problem?  Has it been fixed in recent versions?  Is 
there a way to avoid practically (but not completely) locking up a 
filesystem for many hours or days just to delete a file?

sdw

-- 
swilliams@hpti.com http://www.hpti.com Personal: sdw@lig.net http://sdw.st
Stephen D. Williams 703-724-0118W 703-995-0407Fax 20147-4622 AIM: sdw

Re: Very very slow rm/unlink of large sparse files

[Dieter Nützel]

Am Montag, 16. Februar 2004 04:29 schrieb Stephen D. Williams:
> Linux kernel 2.4.21-99 (SuSe 9 Pro kernel), and also experienced with
> 2.6.1.
>
> When deleting a large file (4-10GB), at least somewhat sparse (created
> completely sparse and filled in as a reiserfs filesystem under
> user-mode-linux), the filesystem layer spends a very very long time, on
> the order of several hours per GB, with constant disk activity trying to
> delete the file.  If the system is rebooted, the partial unlink is
> continued at the same very very slow speed.  A read-only mount works
> just fine.
>
> This is has happened several times as I have reinstalled and reoriented
> systems.  The filesystems are 40-500GB and are running on top of RAID
> 0,1, or 5 (ReiserFS->RAID 0, 1, or 5), sometimes with LVM
> (ReiserFS->LVM->RAID 1).  It seems completely repeatable and
> consistent.  I have been waiting about 24 hours for an unlink of a 9.9GB
> file, formatted as ReiserFS and mostly filled with data.  The filesystem
> involved is accessible, but very very slowly.  An ls of a small
> directory takes at least 30 minutes.  The system has 2GB ram and has at
> least half of that free in buffers and cache.  System load is over 8
> because of things waiting for disk I/O, but CPU is 1% user and 3%
> system.  The disk LED is flashing at about 50% load, constant activity.
>
> dd if=/dev/zero of=u.img bs=1 count=1 seek=$((1024 * 1024 * 4300))
> mkfs.reiserfs -f u.img
> mount -o loop u.img /mnt
> cp -r /etc /mnt   # I'm not sure how much data needs to be present, if
> any, to exacerbate the problem.
> umount /mnt
> rm -f u.img

> Is this a known problem?

Yes.

Look here:
http://marc.theaimsgroup.com/?l=reiserfs&m=105781312812025&w=2


> Has it been fixed in recent versions?

No.

But maybe this one helps?
http://marc.theaimsgroup.com/?l=reiserfs&m=105781500513025&w=2
http://marc.theaimsgroup.com/?l=reiserfs&m=105782794622005&w=2

> Is there a way to avoid practically (but not completely) locking up a
> filesystem for many hours or days just to delete a file?

Don't know, yet.

Regards,
	Dieter

Re: Very very slow rm/unlink of large sparse files

[Chris Mason]

On Mon, 2004-02-16 at 08:32, Dieter Nützel wrote:

> > Is there a way to avoid practically (but not completely) locking up a
> > filesystem for many hours or days just to delete a file?

Yes, reiserfs needs to be nicer with the BKL.  This I can fix
easily...patching.

-chris

Re: Very very slow rm/unlink of large sparse files

[Dieter Nützel]

Am Montag, 16. Februar 2004 14:42 schrieb Chris Mason:
> On Mon, 2004-02-16 at 08:32, Dieter Nützel wrote:
> > > Is there a way to avoid practically (but not completely) locking up a
> > > filesystem for many hours or days just to delete a file?
>
> Yes, reiserfs needs to be nicer with the BKL.  This I can fix
> easily...patching.

Ah, I tried the "test" with my current set up (SuSE 9.0, 2.6.2-6, data-logging 
patches for 2.6.1), again.

> daul Athlon MP 1900+

Same with 1 GB RAM.

> U160

I have this, now:

U320 disks (Fujitsu MAS3184NP, 15k RPM) on U160 controller
RAID0 and RAID1

;-)

> Argh! ---I can't kill it!
> No CNTL-C or '"killall -9 dd" or under "top".
> Even as root;-(
> 
> 2.4.21-jam1 (aa1) plus all data-logging
> 
> SOURCE/dri-trunk> time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
> 0.000u 362.760s 8:26.34 71.6%   0+0k 0+0io 124pf+0w

Test was performed on RAID0 (second partitions, from outer to inner 
cylinders):

INSTALL/SOURCE> time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
1+0 Records ein
1+0 Records aus
0.000u 3.038s 0:05.88 51.5%     0+0k 0+0io 0pf+0w

INSTALL/SOURCE> l
insgesamt 430288
drwxrwxr-x    7 root     root          264 2004-02-16 14:53 .
drwxr-xr-x    3 root     root           72 2004-01-12 13:53 ..
drwxr-xr-x    2 nuetzel  users         856 2003-12-27 14:51 bench
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk-3-7-2003
drwxr-xr-x    2 root     root          168 2004-02-14 00:49 NX
-rw-r--r--    1 nuetzel  users    214748364801 2004-02-16 14:53 sparse

INSTALL/SOURCE> time rm sparse
0.000u 4.745s 0:04.74 100.0%    0+0k 0+0io 0pf+0w

System was NOT responseable during "rm".
NO mouse movement, etc.

And this is WITH kernel pre-emption...

> It was runinng with a paralell C++ (-j20) compilation.

Now, same as before but with compilation.

INSTALL/SOURCE> time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
1+0 Records ein
1+0 Records aus
0.000u 2.741s 0:09.30 29.4%     0+0k 0+0io 3pf+0w
INSTALL/SOURCE> l
insgesamt 430288
drwxrwxr-x    7 root     root          264 2004-02-16 15:02 .
drwxr-xr-x    3 root     root           72 2004-01-12 13:53 ..
drwxr-xr-x    2 nuetzel  users         856 2003-12-27 14:51 bench
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk-3-7-2003
drwxr-xr-x    2 root     root          168 2004-02-14 00:49 NX
-rw-r--r--    1 nuetzel  users    214748364801 2004-02-16 15:02 sparse

INSTALL/SOURCE> time rm sparse
0.000u 10.684s 2:15.11 7.9%     0+0k 0+0io 0pf+0w

'rm' time is lost in 'io_schedule' and 'down'.

Cheers,
	Dieter

Re: Very very slow rm/unlink of large sparse files

[Chris Mason]

On Mon, 2004-02-16 at 09:06, Dieter Nützel wrote:

> INSTALL/SOURCE> time rm sparse
> 0.000u 10.684s 2:15.11 7.9%     0+0k 0+0io 0pf+0w
> 
> 'rm' time is lost in 'io_schedule' and 'down'.

Dieter, could you please try with this (shamefully untested and
uncompiled) patch:

--- linux.test/fs/reiserfs/stree.c.1	2004-02-16 09:26:33.643567752 -0500
+++ linux.test/fs/reiserfs/stree.c	2004-02-16 09:27:24.166887040 -0500
@@ -1754,7 +1754,7 @@
 	  reiserfs_update_sd(th, p_s_inode) ;
 
 	  journal_end(th, p_s_inode->i_sb, orig_len_alloc) ;
-	  journal_begin(th, p_s_inode->i_sb, orig_len_alloc) ;
+	  journal_begin(th, p_s_inode->i_sb, JOURNAL_PER_BALANCE_CNT * 5) ;
 	  reiserfs_update_inode_transaction(p_s_inode) ;
 	}
     } while ( n_file_size > ROUND_UP (n_new_file_size) &&

Re: Very very slow rm/unlink of large sparse files

[Dieter Nützel]

Am Montag, 16. Februar 2004 15:29 schrieb Chris Mason:
> On Mon, 2004-02-16 at 09:06, Dieter Nützel wrote:
> > INSTALL/SOURCE> time rm sparse
> > 0.000u 10.684s 2:15.11 7.9%     0+0k 0+0io 0pf+0w
> >
> > 'rm' time is lost in 'io_schedule' and 'down'.
>
> Dieter, could you please try with this (shamefully untested and
> uncompiled) patch:
>
> --- linux.test/fs/reiserfs/stree.c.1	2004-02-16 09:26:33.643567752 -0500
> +++ linux.test/fs/reiserfs/stree.c	2004-02-16 09:27:24.166887040 -0500
> @@ -1754,7 +1754,7 @@
>  	  reiserfs_update_sd(th, p_s_inode) ;
>
>  	  journal_end(th, p_s_inode->i_sb, orig_len_alloc) ;
> -	  journal_begin(th, p_s_inode->i_sb, orig_len_alloc) ;
> +	  journal_begin(th, p_s_inode->i_sb, JOURNAL_PER_BALANCE_CNT * 5) ;
>  	  reiserfs_update_inode_transaction(p_s_inode) ;
>  	}
>      } while ( n_file_size > ROUND_UP (n_new_file_size) &&


INSTALL/SOURCE> time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
1+0 Records ein
1+0 Records aus
0.000u 2.984s 0:05.74 51.9%     0+0k 0+0io 1pf+0w

INSTALL/SOURCE> l
insgesamt 430288
drwxrwxr-x    7 root     root          264 2004-02-16 16:42 .
drwxr-xr-x    3 root     root           72 2004-01-12 13:53 ..
drwxr-xr-x    2 nuetzel  users         856 2003-12-27 14:51 bench
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk-3-7-2003
drwxr-xr-x    2 root     root          168 2004-02-14 00:49 NX
-rw-r--r--    1 nuetzel  users    214748364801 2004-02-16 16:42 sparse
-rw-r--r--    1 root     root     440604456 2004-02-15 08:45 
ut2004-lnx-demo-3120.run
drwxr-xr-x   15 nuetzel  users         392 2004-01-17 04:24 viewperf-6.1.2

INSTALL/SOURCE> time sync
0.000u 0.061s 0:00.41 14.6%     0+0k 0+0io 0pf+0w

INSTALL/SOURCE> time rm sparse
0.000u 4.483s 0:04.48 100.0%    0+0k 0+0io 0pf+0w

Mouse behavior is the same.

With compilation in the "background":

INSTALL/SOURCE> time dd if=/dev/zero of=sparse bs=1 seek=200G count=1
1+0 Records ein
1+0 Records aus
0.000u 2.707s 0:10.30 26.2%     0+0k 0+0io 0pf+0w

INSTALL/SOURCE> l
insgesamt 430288
drwxrwxr-x    7 root     root          264 2004-02-16 16:47 .
drwxr-xr-x    3 root     root           72 2004-01-12 13:53 ..
drwxr-xr-x    2 nuetzel  users         856 2003-12-27 14:51 bench
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk
drwxr-xr-x    3 nuetzel  users          72 2003-09-07 20:37 dri-trunk-3-7-2003
drwxr-xr-x    2 root     root          168 2004-02-14 00:49 NX
-rw-r--r--    1 nuetzel  users    214748364801 2004-02-16 16:48 sparse
-rw-r--r--    1 root     root     440604456 2004-02-15 08:45 
ut2004-lnx-demo-3120.run
drwxr-xr-x   15 nuetzel  users         392 2004-01-17 04:24 viewperf-6.1.2

INSTALL/SOURCE> time sync
0.000u 0.015s 0:01.05 0.9%      0+0k 0+0io 0pf+0w

INSTALL/SOURCE> time rm sparse
0.000u 4.546s 0:04.86 93.4%     0+0k 0+0io 0pf+0w

Thanks,
	Dieter
-- 
Dieter Nützel
@home: <Dieter.Nuetzel () hamburg ! de>

Re: Very very slow rm/unlink of large sparse files

[Chris Mason]

On Mon, 2004-02-16 at 10:49, Dieter Nützel wrote:
> Am Montag, 16. Februar 2004 15:29 schrieb Chris Mason:
> > On Mon, 2004-02-16 at 09:06, Dieter Nützel wrote:
> > > INSTALL/SOURCE> time rm sparse
> > > 0.000u 10.684s 2:15.11 7.9%     0+0k 0+0io 0pf+0w
> > >
> > > 'rm' time is lost in 'io_schedule' and 'down'.
> >
> > Dieter, could you please try with this (shamefully untested and
> > uncompiled) patch:
> >
> 
[ no compile in the backround ]

> INSTALL/SOURCE> time rm sparse
> 0.000u 4.483s 0:04.48 100.0%    0+0k 0+0io 0pf+0w
> 
> Mouse behavior is the same.
> 
Very choppy?  Ok, I didn't expect that to change, but it is fixable.

> With compilation in the "background":

> INSTALL/SOURCE> time rm sparse
> 0.000u 4.546s 0:04.86 93.4%     0+0k 0+0io 0pf+0w

So, just to make sure I'm comparing things correctly, the time to rm the
large file with a compile in the background went from over two minutes
to 4 seconds?

If so, the improvement is from a stupid bug fixed in 2.4.x, basically we
were trying to start a huge transaction, which was always forcing the
current transaction to end, which made for tiny transactions, which are
really slow.

-chris

Re: Very very slow rm/unlink of large sparse files

[Dieter Nützel]

Am Montag, 16. Februar 2004 16:59 schrieb Chris Mason:
> On Mon, 2004-02-16 at 10:49, Dieter Nützel wrote:
> > Am Montag, 16. Februar 2004 15:29 schrieb Chris Mason:
> > > On Mon, 2004-02-16 at 09:06, Dieter Nützel wrote:
> > > > INSTALL/SOURCE> time rm sparse
> > > > 0.000u 10.684s 2:15.11 7.9%     0+0k 0+0io 0pf+0w
> > > >
> > > > 'rm' time is lost in 'io_schedule' and 'down'.
> > >
> > > Dieter, could you please try with this (shamefully untested and
> > > uncompiled) patch:

Now, I'm back...
During "dd" mouse movement works OK.

> [ no compile in the backround ]

Yes. ;-)

> > INSTALL/SOURCE> time rm sparse
> > 0.000u 4.483s 0:04.48 100.0%    0+0k 0+0io 0pf+0w
> >
> > Mouse behavior is the same.
>
> Very choppy?

Yes. No movement at all. 8-(
But only for the _first_ four seconds.

> Ok, I didn't expect that to change, but it is fixable. 

Great.

> > With compilation in the "background":
> >
> > INSTALL/SOURCE> time rm sparse
> > 0.000u 4.546s 0:04.86 93.4%     0+0k 0+0io 0pf+0w
>
> So, just to make sure I'm comparing things correctly, the time to rm the
> large file with a compile in the background went from over two minutes
> to 4 seconds?

Correct.

But ONLY if I have such workload in the background.

Several "rm" runs _without_ compilation:

INSTALL/SOURCE> time rm sparse
0.000u 6.177s 0:10.46 58.9%     0+0k 0+0io 0pf+0w

INSTALL/SOURCE> time rm sparse
0.000u 6.098s 0:09.39 64.8%     0+0k 0+0io 0pf+0w

INSTALL/SOURCE> time rm sparse
0.000u 6.256s 0:12.24 51.0%     0+0k 0+0io 0pf+0w

INSTALL/SOURCE> time rm sparse
0.001u 4.792s 0:10.19 47.0%     0+0k 0+0io 0pf+0w

> If so, the improvement is from a stupid bug fixed in 2.4.x, basically we
> were trying to start a huge transaction, which was always forcing the
> current transaction to end, which made for tiny transactions, which are
> really slow.

Good to know.

-Dieter

Re: Very very slow rm/unlink of large sparse files

[Hans Reiser]

Chris Mason wrote:

>On Mon, 2004-02-16 at 08:32, Dieter Nützel wrote:
>
>  
>
>>>Is there a way to avoid practically (but not completely) locking up a
>>>filesystem for many hours or days just to delete a file?
>>>      
>>>
>
>Yes, reiserfs needs to be nicer with the BKL.  This I can fix
>easily...patching.
>
>-chris
>
>
>
>
>  
>
Just a reminder, reiser4 fixes the V3 performance problems with holes.

Re: Very very slow rm/unlink of large sparse files

[Dieter Nützel]

Am Montag, 16. Februar 2004 16:22 schrieb Hans Reiser:
> Chris Mason wrote:
> >On Mon, 2004-02-16 at 08:32, Dieter Nützel wrote:
> >>>Is there a way to avoid practically (but not completely) locking up a
> >>>filesystem for many hours or days just to delete a file?
> >
> >Yes, reiserfs needs to be nicer with the BKL.  This I can fix
> >easily...patching.
> >
> >-chris
>
> Just a reminder, reiser4 fixes the V3 performance problems with holes.

Sorry Hans,

that dosen't matter.
How stable is Reiser4 for production, now?

It IS a very nice FS, but not yet.
We need a solution, now.

-- 
Dieter Nützel
@home: <Dieter.Nuetzel () hamburg ! de>