reiserfs+acl makes processes hang?

reiserfs+acl makes processes hang?

[Tarmo Tänav]

Hi,

(I've already sent this to lkml and the bug has been confirmed
a couple of times, but I though I should also send it to
reiserfs-list so here goes:)


I think I've found a bug in reiserfs acls. If triggered
it means that any program trying to access the partition,
where the bug occured, will just hang in D state, with
no way to kill the program.

Here's how to reproduce:
1. mount a reiserfs volume (loopmount will do) with "-o acl".
2. create a directory "dir"
3. set some default acl: setfacl -d -m u:username:rwX dir
4. cd dir
5. dd if=/dev/zero of=somefile1 bs=4k count=100000
(the idea is to run out of space)
6. now df should show 0 free space, if not then repeat 5.
7. echo "1" > somefile2 # this should hang infinitely

Now no program will be able to access the partition.

I haven't tried to reproduce it, but the same problem also happened
when a user hit his hard quota limit on my server. Then no program
could access his homedir.

Re: reiserfs+acl makes processes hang?

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tarmo Tänav wrote:
> Hi,
> 
> (I've already sent this to lkml and the bug has been confirmed
> a couple of times, but I though I should also send it to
> reiserfs-list so here goes:)
> 
> 
> I think I've found a bug in reiserfs acls. If triggered
> it means that any program trying to access the partition,
> where the bug occured, will just hang in D state, with
> no way to kill the program.
> 
> Here's how to reproduce:
> 1. mount a reiserfs volume (loopmount will do) with "-o acl".
> 2. create a directory "dir"
> 3. set some default acl: setfacl -d -m u:username:rwX dir
> 4. cd dir
> 5. dd if=/dev/zero of=somefile1 bs=4k count=100000
> (the idea is to run out of space)
> 6. now df should show 0 free space, if not then repeat 5.
> 7. echo "1" > somefile2 # this should hang infinitely
> 
> Now no program will be able to access the partition.
> 
> I haven't tried to reproduce it, but the same problem also happened
> when a user hit his hard quota limit on my server. Then no program
> could access his homedir.
> 
> 

Ok, this is due to the iput in the reiserfs_new_inode failure path
calling reiserfs_delete_xattrs with the lock already held.

I'll post a patch soon.

- -Jeff

- --
Jeff Mahoney
SuSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (GNU/Linux)

iD8DBQFC297mLPWxlyuTD7IRAjO6AJ4wgkdvMnHVablWE7vBL1f+nwJhSACgmG7F
icjFXLkoAIPOVUa02yR7Ys4=
=1Nkh
-----END PGP SIGNATURE-----

reiserfs+acl makes processes hang?

[Tarmo Tänav]

Hi,

I think I've found a bug in reiserfs acls. If triggered
it means that any program trying to access the partition,
where the bug occured, will just hang in D state, with
no way to kill the program.

Here's how to reproduce:
1. mount a reiserfs volume (loopmount will do) with "-o acl".
2. create a directory "dir"
3. set some default acl: setfacl -d -m u:username:rwX dir
4. cd dir
5. dd if=/dev/zero of=somefile1 bs=4k count=100000
(the idea is to run out of space)
6. now df should show 0 free space, if not then repeat 5.
7. echo "1" > somefile2 # this should hang infinitely

Now no program will be able to access the partition.

I haven't tried to reproduce it, but the same problem also happened
when a user hit his hard quota limit on my server. Then no program
could access his homedir.


PS. I'm not subscribed to lkml so please CC

--
Tarmo Tänav
tarmo@itech.ee

Re: reiserfs+acl makes processes hang?

[Kasper Sandberg]

confirmed, i run 2.6.12 with reiserfs, created with reiserfsprogs 3.6.19

On Fri, 2005-07-15 at 23:19 +0000, Tarmo Tänav wrote:
> Hi,
> 
> I think I've found a bug in reiserfs acls. If triggered
> it means that any program trying to access the partition,
> where the bug occured, will just hang in D state, with
> no way to kill the program.
> 
> Here's how to reproduce:
> 1. mount a reiserfs volume (loopmount will do) with "-o acl".
> 2. create a directory "dir"
> 3. set some default acl: setfacl -d -m u:username:rwX dir
> 4. cd dir
> 5. dd if=/dev/zero of=somefile1 bs=4k count=100000
> (the idea is to run out of space)
> 6. now df should show 0 free space, if not then repeat 5.
> 7. echo "1" > somefile2 # this should hang infinitely
> 
> Now no program will be able to access the partition.
> 
> I haven't tried to reproduce it, but the same problem also happened
> when a user hit his hard quota limit on my server. Then no program
> could access his homedir.
> 
> 
> PS. I'm not subscribed to lkml so please CC
> 
> --
> Tarmo Tänav
> tarmo@itech.ee
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: reiserfs+acl makes processes hang?

[Jan Engelhardt]

>Hi,
>
>Here's how to reproduce:
>1. mount a reiserfs volume (loopmount will do) with "-o acl".
>2. create a directory "dir"
>3. set some default acl: setfacl -d -m u:username:rwX dir
>4. cd dir
>5. dd if=/dev/zero of=somefile1 bs=4k count=100000
>(the idea is to run out of space)
>6. now df should show 0 free space, if not then repeat 5.
>7. echo "1" > somefile2 # this should hang infinitely

Can't reproduce. My versions are:
mkreiserfs 3.6.18
Kernel 2.6.13-rc1


---Step 1---
10:25 shanghai:/mnt > dd if=/dev/zero of=blk count=64 bs=1M
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.552862 seconds, 121 MB/s
10:26 shanghai:/mnt > mkreiserfs -f blk
mkreiserfs 3.6.18 (2003 www.namesys.com)

A pair of credits:
...
blk is not a block special device
Continue (y/n):y
Guessing about desired format.. Kernel 2.6.13-rc1 is running.
Format 3.6 with standard journal
Count of blocks on the device: 16384
Number of blocks consumed by mkreiserfs formatting process: 8212
Blocksize: 4096
Hash function used to sort names: "r5"
Journal Size 8193 blocks (first block 18)
Journal Max transaction length 1024
inode generation number: 0
UUID: aa3bd664-fde0-4552-9484-49bac0fb698f
Initializing journal - 0%....20%....40%....60%....80%....100%
Syncing..ok
ReiserFS is successfully created on blk.
10:26 shanghai:/mnt > mount blk loop -o loop,acl

---Step 2-7---
10:26 shanghai:/mnt > cd loop/
10:27 shanghai:/mnt/loop > md dir
10:27 shanghai:/mnt/loop > setfacl -d -m u:daemon:rwX dir
10:27 shanghai:/mnt/loop > cd dir
10:27 shanghai:/mnt/loop/dir > cat /dev/zero >blah
cat: write error: No space left on device
10:27 shanghai:/mnt/loop/dir > df .
Filesystem           1K-blocks      Used Available Use% Mounted on
/mnt/blk                 65528     65528         0 100% /mnt/loop
10:27 shanghai:/mnt/loop/dir > l
total 32684
drwxr-xr-x+ 2 root root       72 Jul 16 10:27 .
drwxr-xr-x  5 root root      104 Jul 16 10:27 ..
-rw-rw-r--+ 1 root root 33435648 Jul 16 10:27 blah
(That's ok, the other 32MB are for the journal)
10:27 shanghai:/mnt/loop/dir > echo "1" >blah
10:28 shanghai:/mnt/loop/dir > l blah
-rw-rw-r--+ 1 root root 2 Jul 16 10:28 blah



Jan Engelhardt
-- 

Re: reiserfs+acl makes processes hang?

[Tarmo Tänav]

You made one mistake, the last echo "1" >blah should not be
to the file you created earlier.. the echo is meant to
create another file which is supposed to fail because there
is no free space but for some reason instead of failing it
will cause the process to enter D state.

Here is how I just reproced it:
#mkdir testdir
#cd testdir/
#dd if=/dev/zero of=blk count=64 bs=1M
64+0 records in
64+0 records out
#mkreiserfs -f blk >/dev/null
mkreiserfs 3.6.19 (2003 www.namesys.com)

blk is not a block special device
Continue (y/n):y
#mkdir mpoint
#mount blk mpoint/ -o loop,acl
#cd mpoint
#mkdir dir
#setfacl -f -m u:sn4ip3r:rwX dir
setfacl: invalid option -- f
Usage: setfacl [-bkndRLP] { -m|-M|-x|-X ... } file ...
Try `setfacl --help' for more information.
#setfacl -d -m u:sn4ip3r:rwX dir
#cd dir
#cat /dev/zero >blah
cat: write error: No space left on device
#df .
Filesystem           1K-blocks      Used Available Use% Mounted on
/home/sn4ip3r/testdir/blk
                         65528     65528         0
100% /home/sn4ip3r/testdir/mpoint
#echo 1 > blah2              <<< and this is the line that never returns

--
Tarmo Tänav
tarmo@itech.ee

On L, 2005-07-16 at 10:24 +0200, Jan Engelhardt wrote:
> >Hi,
> >
> >Here's how to reproduce:
> >1. mount a reiserfs volume (loopmount will do) with "-o acl".
> >2. create a directory "dir"
> >3. set some default acl: setfacl -d -m u:username:rwX dir
> >4. cd dir
> >5. dd if=/dev/zero of=somefile1 bs=4k count=100000
> >(the idea is to run out of space)
> >6. now df should show 0 free space, if not then repeat 5.
> >7. echo "1" > somefile2 # this should hang infinitely
> 
> Can't reproduce. My versions are:
> mkreiserfs 3.6.18
> Kernel 2.6.13-rc1
> 
> 
> ---Step 1---
> 10:25 shanghai:/mnt > dd if=/dev/zero of=blk count=64 bs=1M
> 64+0 records in
> 64+0 records out
> 67108864 bytes (67 MB) copied, 0.552862 seconds, 121 MB/s
> 10:26 shanghai:/mnt > mkreiserfs -f blk
> mkreiserfs 3.6.18 (2003 www.namesys.com)
> 
> A pair of credits:
> ...
> blk is not a block special device
> Continue (y/n):y
> Guessing about desired format.. Kernel 2.6.13-rc1 is running.
> Format 3.6 with standard journal
> Count of blocks on the device: 16384
> Number of blocks consumed by mkreiserfs formatting process: 8212
> Blocksize: 4096
> Hash function used to sort names: "r5"
> Journal Size 8193 blocks (first block 18)
> Journal Max transaction length 1024
> inode generation number: 0
> UUID: aa3bd664-fde0-4552-9484-49bac0fb698f
> Initializing journal - 0%....20%....40%....60%....80%....100%
> Syncing..ok
> ReiserFS is successfully created on blk.
> 10:26 shanghai:/mnt > mount blk loop -o loop,acl
> 
> ---Step 2-7---
> 10:26 shanghai:/mnt > cd loop/
> 10:27 shanghai:/mnt/loop > md dir
> 10:27 shanghai:/mnt/loop > setfacl -d -m u:daemon:rwX dir
> 10:27 shanghai:/mnt/loop > cd dir
> 10:27 shanghai:/mnt/loop/dir > cat /dev/zero >blah
> cat: write error: No space left on device
> 10:27 shanghai:/mnt/loop/dir > df .
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /mnt/blk                 65528     65528         0 100% /mnt/loop
> 10:27 shanghai:/mnt/loop/dir > l
> total 32684
> drwxr-xr-x+ 2 root root       72 Jul 16 10:27 .
> drwxr-xr-x  5 root root      104 Jul 16 10:27 ..
> -rw-rw-r--+ 1 root root 33435648 Jul 16 10:27 blah
> (That's ok, the other 32MB are for the journal)
> 10:27 shanghai:/mnt/loop/dir > echo "1" >blah
> 10:28 shanghai:/mnt/loop/dir > l blah
> -rw-rw-r--+ 1 root root 2 Jul 16 10:28 blah
> 
> 
> 
> Jan Engelhardt

Re: reiserfs+acl makes processes hang?

[Jan Engelhardt]

>You made one mistake, the last echo "1" >blah should not be
>to the file you created earlier.. the echo is meant to

Right. Hangs now.

If someone wants a stack trace, below is one. However, I wonder why my traces 
have become so distorted -- e.g. the proc_lookup in the last one should not be 
there at all.

But I think the problem is simple:
Reiserfs implements ACLs/Xattrs using files, and obviously, creating the ACL 
for the file cannot succeed because there's no space left. Don't know what 
reiser does in case _an acl file_ can't be created...


: bash          D C39335D0     0  4252   4195                     (NOTLB)
: c3de3cb8 00000082 00000000 c39335d0 c0128a60 c3de3cbc bf1ff5dc 000000fd
:        c39335d0 cdc01b84 c3de3cc8 c39335d0 c7101208 c02852a5 c3de3cbc cdc01b88
:        c2d17e0c cdc01b88 c39335d0 00000001 c7101208 c71011bc c71011cc c7101208
: Call Trace:
:  [<c0128a60>] autoremove_wake_function+0x0/0x50
:  [<c02852a5>] rwsem_down_read_failed+0x75/0x150
:  [<c01b2268>] .text.lock.xattr+0x55/0x23d
:  [<c018f5b0>] reiserfs_delete_inode+0x70/0x110
:  [<c01b2ec7>] reiserfs_set_acl+0x167/0x1a0
:  [<c018f540>] reiserfs_delete_inode+0x0/0x110
:  [<c016643c>] generic_delete_inode+0x9c/0x140
:  [<c016665c>] iput+0x4c/0x70
:  [<c0192759>] reiserfs_new_inode+0x149/0x700
:  [<c018cfdd>] reiserfs_find_entry+0xad/0x120
:  [<c01aca00>] wake_queued_writers+0x30/0x40
:  [<c018d8b7>] reiserfs_create+0xe7/0x210
:  [<c01b21ef>] reiserfs_permission+0xf/0x20
:  [<c01595e6>] permission+0xb6/0xe0
:  [<c015b166>] vfs_create+0xc6/0x190
:  [<c015ba9a>] open_namei+0x5fa/0x740
:  [<c014c427>] filp_open+0x27/0x50
:  [<c014c650>] get_unused_fd+0x20/0xa0
:  [<c01593c7>] getname+0x67/0xb0
:  [<c014c779>] sys_open+0x49/0xd0
:  [<c0102b89>] syscall_call+0x7/0xb
: ls            D C2D17E60     0 20215  14083                     (NOTLB)
: c2d17dfc 00000086 c01ad123 c2d17e60 c2d17eb0 00000000 b1b8700c 00000134
:        c3933ac0 cdc01b84 c2d17e0c c3933ac0 00000000 c02852a5 c3933ac0 cdc01b88
:        cdc01b88 c3de3cc8 c3933ac0 00000001 c02c7be0 c7c7c4b4 c2d17e58 00000000
: Call Trace:
:  [<c01ad123>] journal_mark_dirty+0x113/0x250
:  [<c02852a5>] rwsem_down_read_failed+0x75/0x150
:  [<c01b22da>] .text.lock.xattr+0xc7/0x23d
:  [<c016c8dc>] getxattr+0xdc/0x170
:  [<c01b21e0>] reiserfs_permission+0x0/0x20
:  [<c01b21ef>] reiserfs_permission+0xf/0x20
:  [<c01595e6>] permission+0xb6/0xe0
:  [<c0159f54>] __link_path_walk+0x4a4/0xed0
:  [<c013fb78>] handle_mm_fault+0x138/0x190
:  [<c015aa2b>] link_path_walk+0xab/0x1a0
:  [<c015af9d>] __user_walk+0x3d/0x60
:  [<c016c9bb>] sys_getxattr+0x4b/0x70
:  [<c0102b89>] syscall_call+0x7/0xb
: ls            D C0180493     0 22554  22493                     (NOTLB)
: ce079dfc 00000082 cffec560 c0180493 ffffffea ced0fb00 08b060f1 00000170
:        c478aa20 cdc01b84 ce079e0c c478aa20 00000000 c02852a5 c0135316 cdc01b88
:        cdc01b88 c2d17e0c c478aa20 00000001 c02c7be0 c82d7f54 ce079e58 00000000
: Call Trace:
:  [<c0180493>] proc_lookup+0xa3/0xb0
:  [<c02852a5>] rwsem_down_read_failed+0x75/0x150
:  [<c0135316>] prep_new_page+0x46/0x60
:  [<c01b22da>] .text.lock.xattr+0xc7/0x23d
:  [<c016c8dc>] getxattr+0xdc/0x170
:  [<c01b21e0>] reiserfs_permission+0x0/0x20
:  [<c01b21ef>] reiserfs_permission+0xf/0x20
:  [<c01595e6>] permission+0xb6/0xe0
:  [<c0159f54>] __link_path_walk+0x4a4/0xed0
:  [<c013fb78>] handle_mm_fault+0x138/0x190
:  [<c015aa2b>] link_path_walk+0xab/0x1a0
:  [<c015af9d>] __user_walk+0x3d/0x60
:  [<c016c9bb>] sys_getxattr+0x4b/0x70
:  [<c0102b89>] syscall_call+0x7/0xb


Jan Engelhardt
--