btrfs root + mount subvolid=0 problem

btrfs root + mount subvolid=0 problem

[Fajar A. Nugraha]

Hi

I have a system with Ubuntu natty i386 which uses btrfs root. It has
worked mostly well, but I have a problem when I want to create new
snapshot.
Current layout looks something like this

$ mount | grep btrfs
/dev/sda6 on / type btrfs (rw,noatime,subvolid=258,compress-force=lzo)
/dev/sda6 on /home type btrfs (rw,noatime,subvolid=259,compress-force=lzo)
/dev/sda6 on /data type btrfs (rw,noatime,subvolid=260,compress-force=lzo)
/dev/sda6 on /data/vm type btrfs (rw,noatime,subvolid=261,compress-force=lzo)
/dev/sda6 on /boot type btrfs (rw,noatime,subvolid=289,compress-force=lzo)

$ sudo btrfs su li /
[sudo] password for user:
ID 258 top level 5 path subvol/root
ID 259 top level 5 path subvol/home
ID 260 top level 5 path subvol/data
ID 261 top level 5 path subvol/vm
ID 289 top level 5 path boot
ID 318 top level 5 path subvol/oneiric

A few days ago I was able to mount subvolid=0 to a temporary path
(e.g. /mnt/tmp), and on that create a snapshot of subvol/root as
subvol/oneiric (for test upgrade purposes). That worked well. The
problem is now I can't repeat the process

$ sudo mount /dev/sda6 -o subvolid=0 /mnt/tmp
$ ls /mnt/tmp
boot  subvol
$ ls /mnt/tmp/subvol

(hangs at this point, I have to perform hard reset or "reboot -f").

This happens both with natty's 2.6.38-11-generic and kernel 3.0.4
(backported from oneiric). Does anyone know if this is a know problem,
or how to get further information to fix this?

Thanks,

Fajar

Re: btrfs root + mount subvolid=0 problem

[David Sterba]

On Mon, Oct 10, 2011 at 02:30:30PM +0700, Fajar A. Nugraha wrote:
> This happens both with natty's 2.6.38-11-generic and kernel 3.0.4
> (backported from oneiric). Does anyone know if this is a know problem,
> or how to get further information to fix this?

I'm afraid with the kernels that old, nobody will try to fix it unless
you reproduce it on the most recent kernels.

Do you see any relevant messages in the log before/while the ls is
stuck? I'd be good to know where excatly the ls is stuck by 'cat
/proc/<lspid>/stack', and if it's in D state, possibly gethering stacks
of all such processes.

I have seen problems with mounting subvolumes when there is a
non-toplevel one set as deafult, but this was more related to the
'subvol' option, subvolid does not suffer from this.


david

Re: btrfs root + mount subvolid=0 problem

[Fajar A. Nugraha]

On Mon, Oct 10, 2011 at 5:20 PM, David Sterba <dave@jikos.cz> wrote:
> On Mon, Oct 10, 2011 at 02:30:30PM +0700, Fajar A. Nugraha wrote:
>> This happens both with natty's 2.6.38-11-generic and kernel 3.0.4
>> (backported from oneiric). Does anyone know if this is a know problem,
>> or how to get further information to fix this?
>
> I'm afraid with the kernels that old, nobody will try to fix it unless
> you reproduce it on the most recent kernels.

3.0.4 is considered old already? Wow, 3.1 is not even out yet

I'll retry with 3.1-rc9 then.

>
> Do you see any relevant messages in the log before/while the ls is
> stuck?

Nope

> I'd be good to know where excatly the ls is stuck by 'cat
> /proc/<lspid>/stack', and if it's in D state, possibly gethering stacks
> of all such processes.

cpu usage of the ls process is 96-99%

The good news is that it seems to be doing something (the stack
changes somewhat). The bad news is that it seems stuck in a loop. This
is on kernel 3.0.4

$ for n in `seq 1 10`;do echo "=============================";sudo cat
/proc/9354/stack;done
=============================
[<c1041ebb>] __cond_resched+0x1b/0x30
[<c113b3f9>] iget5_locked+0x79/0x1a0
[<f8572ccc>] btrfs_iget+0x3c/0x4a0 [btrfs]
[<f8573278>] btrfs_orphan_cleanup+0x148/0x320 [btrfs]
[<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0
[<c112f977>] path_lookupat+0x107/0x5e0
[<c112fe7c>] do_path_lookup+0x2c/0xb0
[<c11302b3>] user_path_at+0x43/0x80
[<c1128165>] vfs_fstatat+0x55/0xa0
[<c11281d0>] vfs_lstat+0x20/0x30
[<c11285a6>] sys_lstat64+0x16/0x30
[<c150b3e4>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
=============================
[<c1041ebb>] __cond_resched+0x1b/0x30
[<c113b3f9>] iget5_locked+0x79/0x1a0
[<f8572ccc>] btrfs_iget+0x3c/0x4a0 [btrfs]
[<f8573278>] btrfs_orphan_cleanup+0x148/0x320 [btrfs]
[<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0
[<c112f977>] path_lookupat+0x107/0x5e0
[<c112fe7c>] do_path_lookup+0x2c/0xb0
[<c11302b3>] user_path_at+0x43/0x80
[<c1128165>] vfs_fstatat+0x55/0xa0
[<c11281d0>] vfs_lstat+0x20/0x30
[<c11285a6>] sys_lstat64+0x16/0x30
[<c150b3e4>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
=============================
[<f8545273>] generic_bin_search.clone.39+0x1b3/0x210 [btrfs]
[<c102cf7e>] kmap_atomic_prot+0xde/0x100
[<ffffffff>] 0xffffffff
=============================
[<f8545273>] generic_bin_search.clone.39+0x1b3/0x210 [btrfs]
[<f854617a>] bin_search+0x4a/0x90 [btrfs]
[<f854ac44>] btrfs_search_slot+0x104/0x5c0 [btrfs]
[<f85731c7>] btrfs_orphan_cleanup+0x97/0x320 [btrfs]
[<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0
[<c112f977>] path_lookupat+0x107/0x5e0
[<c112fe7c>] do_path_lookup+0x2c/0xb0
[<c11302b3>] user_path_at+0x43/0x80
[<c1128165>] vfs_fstatat+0x55/0xa0
[<c11281d0>] vfs_lstat+0x20/0x30
[<c11285a6>] sys_lstat64+0x16/0x30
[<c150b3e4>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
=============================
[<ffffffff>] 0xffffffff
=============================
[<f8545222>] generic_bin_search.clone.39+0x162/0x210 [btrfs]
[<f854617a>] bin_search+0x4a/0x90 [btrfs]
[<f854accc>] btrfs_search_slot+0x18c/0x5c0 [btrfs]
[<f85731c7>] btrfs_orphan_cleanup+0x97/0x320 [btrfs]
[<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0
[<c112f977>] path_lookupat+0x107/0x5e0
[<c112fe7c>] do_path_lookup+0x2c/0xb0
[<c11302b3>] user_path_at+0x43/0x80
[<c1128165>] vfs_fstatat+0x55/0xa0
[<c11281d0>] vfs_lstat+0x20/0x30
[<c11285a6>] sys_lstat64+0x16/0x30
[<c150b3e4>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
=============================
[<c1041ebb>] __cond_resched+0x1b/0x30
[<c113b3f9>] iget5_locked+0x79/0x1a0
[<f8572ccc>] btrfs_iget+0x3c/0x4a0 [btrfs]
[<f8573278>] btrfs_orphan_cleanup+0x148/0x320 [btrfs]
[<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0
[<c112f977>] path_lookupat+0x107/0x5e0
[<c112fe7c>] do_path_lookup+0x2c/0xb0
[<c11302b3>] user_path_at+0x43/0x80
[<c1128165>] vfs_fstatat+0x55/0xa0
[<c11281d0>] vfs_lstat+0x20/0x30
[<c11285a6>] sys_lstat64+0x16/0x30
[<c150b3e4>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
=============================
[<c1041ebb>] __cond_resched+0x1b/0x30
[<c113b3f9>] iget5_locked+0x79/0x1a0
[<f8572ccc>] btrfs_iget+0x3c/0x4a0 [btrfs]
[<f8573278>] btrfs_orphan_cleanup+0x148/0x320 [btrfs]
[<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0
[<c112f977>] path_lookupat+0x107/0x5e0
[<c112fe7c>] do_path_lookup+0x2c/0xb0
[<c11302b3>] user_path_at+0x43/0x80
[<c1128165>] vfs_fstatat+0x55/0xa0
[<c11281d0>] vfs_lstat+0x20/0x30
[<c11285a6>] sys_lstat64+0x16/0x30
[<c150b3e4>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
=============================
[<c1041ebb>] __cond_resched+0x1b/0x30
[<c113b3f9>] iget5_locked+0x79/0x1a0
[<f8572ccc>] btrfs_iget+0x3c/0x4a0 [btrfs]
[<f8573278>] btrfs_orphan_cleanup+0x148/0x320 [btrfs]
[<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0
[<c112f977>] path_lookupat+0x107/0x5e0
[<c112fe7c>] do_path_lookup+0x2c/0xb0
[<c11302b3>] user_path_at+0x43/0x80
[<c1128165>] vfs_fstatat+0x55/0xa0
[<c11281d0>] vfs_lstat+0x20/0x30
[<c11285a6>] sys_lstat64+0x16/0x30
[<c150b3e4>] syscall_call+0x7/0xb
[<ffffffff>] 0xffffffff
=============================
[<f8545273>] generic_bin_search.clone.39+0x1b3/0x210 [btrfs]
[<ffffffff>] 0xffffffff


>
> I have seen problems with mounting subvolumes when there is a
> non-toplevel one set as deafult, but this was more related to the
> 'subvol' option, subvolid does not suffer from this.

I didn't change the default subvol. The root subvol was explicitly
selected with subvolid option, both in fstab and grub.cfg.

-- 
Fajar

Re: btrfs root + mount subvolid=0 problem

[Fajar A. Nugraha]

On Mon, Oct 10, 2011 at 7:09 PM, Fajar A. Nugraha <list@fajar.net> wrote:
> On Mon, Oct 10, 2011 at 5:20 PM, David Sterba <dave@jikos.cz> wrote:
>> On Mon, Oct 10, 2011 at 02:30:30PM +0700, Fajar A. Nugraha wrote:
>>> This happens both with natty's 2.6.38-11-generic and kernel 3.0.4
>>> (backported from oneiric). Does anyone know if this is a know problem,
>>> or how to get further information to fix this?
>>
>> I'm afraid with the kernels that old, nobody will try to fix it unless
>> you reproduce it on the most recent kernels.
>
> 3.0.4 is considered old already? Wow, 3.1 is not even out yet
>
> I'll retry with 3.1-rc9 then.

Well, apparently 3.0.4 IS old in btrfs world :) With 3.1-rc9, it looks
stuck for a while (same high cpu load), but after a while it completes
succesfully

>> I'd be good to know where excatly the ls is stuck by 'cat
>> /proc/<lspid>/stack', and if it's in D state, possibly gethering stacks
>> of all such processes.
>
> cpu usage of the ls process is 96-99%
>
> The good news is that it seems to be doing something (the stack
> changes somewhat). The bad news is that it seems stuck in a loop. This
> is on kernel 3.0.4
>
> $ for n in `seq 1 10`;do echo "=============================";sudo cat
> /proc/9354/stack;done
> =============================
> [<c1041ebb>] __cond_resched+0x1b/0x30
> [<c113b3f9>] iget5_locked+0x79/0x1a0
> [<f8572ccc>] btrfs_iget+0x3c/0x4a0 [btrfs]
> [<f8573278>] btrfs_orphan_cleanup+0x148/0x320 [btrfs]
> [<f8573757>] btrfs_lookup_dentry+0x307/0x4a0 [btrfs]
> [<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
> [<c112cf27>] d_alloc_and_lookup+0x37/0x70
> [<c112eb89>] do_lookup+0x279/0x2f0
> [<c112f977>] path_lookupat+0x107/0x5e0
> [<c112fe7c>] do_path_lookup+0x2c/0xb0
> [<c11302b3>] user_path_at+0x43/0x80
> [<c1128165>] vfs_fstatat+0x55/0xa0
> [<c11281d0>] vfs_lstat+0x20/0x30
> [<c11285a6>] sys_lstat64+0x16/0x30
> [<c150b3e4>] syscall_call+0x7/0xb
> [<ffffffff>] 0xffffffff


The stack looks a little different. The 3.0.4 sometimes has this while
3.1-rc9 doesn't (at least I wasn't able to capture it)

[<f8545222>] generic_bin_search.clone.39+0x162/0x210 [btrfs]

...and 3.0.4 has these

[<f8573900>] btrfs_lookup+0x10/0x30 [btrfs]
[<c112cf27>] d_alloc_and_lookup+0x37/0x70
[<c112eb89>] do_lookup+0x279/0x2f0

while 3.1-rc has

[<f8955ec8>] btrfs_lookup+0x18/0x60 [btrfs]
[<c112ddaf>] d_inode_lookup.clone.9+0x1f/0x50
[<c113008b>] do_lookup+0x31b/0x360



Anyway, since 3.1-rc9 works I'll stick with this for now. Thanks for
your help, David.

-- 
Fajar

Re: btrfs root + mount subvolid=0 problem

[Chris Samuel]

On 10/10/11 23:09, Fajar A. Nugraha wrote:

> 3.0.4 is considered old already? Wow, 3.1 is not even out yet

I think the essential difference is that none of the btrfs
work gets pushed back to the stable kernel releases, it only
ever ends up in the RC's for new kernels (as it's still very
much experimental and under development).

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC