ls hangs filesystem

ls hangs filesystem

[Jim]

Good morning btrfs list,
I have been rsyncing files from an nfs mount to a btrfs filesystem.  
After an rsync run I ls random subvols or directorys to check the copy.  
About 60% to 70% of the time ls completely hangs.  Ps aux shows it as 
running but even when I let it go for up to an hour it never finishes.  
Kill -9 will not stop the process.  Dmesg shows nothing beyond a 
successful mount at boot.  I can't umount the system because "filesystem 
is busy".  I find that a forced reboot is the only way to recapture the 
system.  /var/log/messages has the only indication (that I can find) 
that anything abnormal is happening.  A tail of the file is below.  
Thank you for any help and advice.
Jim

[root@btrfs ~]# tail -20 /var/log/messages
Oct 18 05:06:36 btrfs kernel: [<ffffffff81142849>] sys_getdents+0x89/0xf0
Oct 18 05:06:36 btrfs kernel: [<ffffffff814b19ab>] 
system_call_fastpath+0x16/0x1b
Oct 18 05:08:36 btrfs kernel: INFO: task ls:7100 blocked for more than 
120 seconds.
Oct 18 05:08:36 btrfs kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 05:08:36 btrfs kernel: ls              D ffff88080bc843d8     0  
7100   7047 0x00000084
Oct 18 05:08:36 btrfs kernel: ffff8804f526dd68 0000000000000086 
ffff8804f526dd18 ffffffff00000000
Oct 18 05:08:36 btrfs kernel: ffff88080bc84000 0000000000011900 
ffff8804f526dfd8 ffff8804f526c010
Oct 18 05:08:36 btrfs kernel: ffff8804f526dfd8 0000000000011900 
ffff88080f64d7f0 ffff88080bc84000
Oct 18 05:08:36 btrfs kernel: Call Trace:
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a0f45>] 
wait_current_trans+0xa5/0x110 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffff8106c0e0>] ? wake_up_bit+0x40/0x40
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a2540>] 
start_transaction+0x1e0/0x2b0 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01a2665>] 
btrfs_join_transaction+0x15/0x20 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffffa01ad360>] 
btrfs_dirty_inode+0x50/0x160 [btrfs]
Oct 18 05:08:36 btrfs kernel: [<ffffffff8115631f>] 
__mark_inode_dirty+0x3f/0x200
Oct 18 05:08:36 btrfs kernel: [<ffffffff81148bf5>] touch_atime+0x115/0x150
Oct 18 05:08:36 btrfs kernel: [<ffffffff81142520>] ? sys_ioctl+0xb0/0xb0
Oct 18 05:08:36 btrfs kernel: [<ffffffff811426ce>] vfs_readdir+0xce/0xe0
Oct 18 05:08:36 btrfs kernel: [<ffffffff81142849>] sys_getdents+0x89/0xf0
Oct 18 05:08:36 btrfs kernel: [<ffffffff814b19ab>] 
system_call_fastpath+0x16/0x1b

Re: ls hangs filesystem

[Josef Bacik]

On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote:
> Good morning btrfs list,
> I have been rsyncing files from an nfs mount to a btrfs filesystem.
> After an rsync run I ls random subvols or directorys to check the
> copy.  About 60% to 70% of the time ls completely hangs.  Ps aux
> shows it as running but even when I let it go for up to an hour it
> never finishes.  Kill -9 will not stop the process.  Dmesg shows
> nothing beyond a successful mount at boot.  I can't umount the
> system because "filesystem is busy".  I find that a forced reboot is
> the only way to recapture the system.  /var/log/messages has the
> only indication (that I can find) that anything abnormal is
> happening.  A tail of the file is below.  Thank you for any help and
> advice.

Hit sysrq+w when this happens and give us all the tracebacks.  Also which kernel
you are on would be helpful.  Thanks,

Josef

Re: ls hangs filesystem

[Jim]

I am on kernel 3.1.0-rc4.  I am ssh'd into a remote machine so have no 
access to keyboard, any other way to get the info you need?  Thanks again.
Jim

On 10/18/2011 09:31 AM, Josef Bacik wrote:
> On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote:
>> Good morning btrfs list,
>> I have been rsyncing files from an nfs mount to a btrfs filesystem.
>> After an rsync run I ls random subvols or directorys to check the
>> copy.  About 60% to 70% of the time ls completely hangs.  Ps aux
>> shows it as running but even when I let it go for up to an hour it
>> never finishes.  Kill -9 will not stop the process.  Dmesg shows
>> nothing beyond a successful mount at boot.  I can't umount the
>> system because "filesystem is busy".  I find that a forced reboot is
>> the only way to recapture the system.  /var/log/messages has the
>> only indication (that I can find) that anything abnormal is
>> happening.  A tail of the file is below.  Thank you for any help and
>> advice.
> Hit sysrq+w when this happens and give us all the tracebacks.  Also which kernel
> you are on would be helpful.  Thanks,
>
> Josef

Re: ls hangs filesystem

[Sander]

Jim wrote (ao):
> I am on kernel 3.1.0-rc4.  I am ssh'd into a remote machine so have
> no access to keyboard, any other way to get the info you need?

echo 'w' > /proc/sysrq-trigger


> On 10/18/2011 09:31 AM, Josef Bacik wrote:
> >On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote:
> >>Good morning btrfs list,
> >>I have been rsyncing files from an nfs mount to a btrfs filesystem.
> >>After an rsync run I ls random subvols or directorys to check the
> >>copy.  About 60% to 70% of the time ls completely hangs.  Ps aux
> >>shows it as running but even when I let it go for up to an hour it
> >>never finishes.  Kill -9 will not stop the process.  Dmesg shows
> >>nothing beyond a successful mount at boot.  I can't umount the
> >>system because "filesystem is busy".  I find that a forced reboot is
> >>the only way to recapture the system.  /var/log/messages has the
> >>only indication (that I can find) that anything abnormal is
> >>happening.  A tail of the file is below.  Thank you for any help and
> >>advice.
> >Hit sysrq+w when this happens and give us all the tracebacks.  Also which kernel
> >you are on would be helpful.  Thanks,
> >
> >Josef
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: ls hangs filesystem

[Jim]

Thank you, I should have thought of that.  I will get back as soon as I 
have info.
Jim

On 10/18/2011 09:49 AM, Sander wrote:
> Jim wrote (ao):
>> I am on kernel 3.1.0-rc4.  I am ssh'd into a remote machine so have
>> no access to keyboard, any other way to get the info you need?
> echo 'w'>  /proc/sysrq-trigger
>
>
>> On 10/18/2011 09:31 AM, Josef Bacik wrote:
>>> On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote:
>>>> Good morning btrfs list,
>>>> I have been rsyncing files from an nfs mount to a btrfs filesystem.
>>>> After an rsync run I ls random subvols or directorys to check the
>>>> copy.  About 60% to 70% of the time ls completely hangs.  Ps aux
>>>> shows it as running but even when I let it go for up to an hour it
>>>> never finishes.  Kill -9 will not stop the process.  Dmesg shows
>>>> nothing beyond a successful mount at boot.  I can't umount the
>>>> system because "filesystem is busy".  I find that a forced reboot is
>>>> the only way to recapture the system.  /var/log/messages has the
>>>> only indication (that I can find) that anything abnormal is
>>>> happening.  A tail of the file is below.  Thank you for any help and
>>>> advice.
>>> Hit sysrq+w when this happens and give us all the tracebacks.  Also which kernel
>>> you are on would be helpful.  Thanks,
>>>
>>> Josef
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ls hangs filesystem

[Jim]

I am getting no output from echo 'w' > /proc/sysrq-trigger.  Ls is hung 
again so I can try any thing else you suggest to get data.
Jim

On 10/18/2011 09:49 AM, Sander wrote:
> Jim wrote (ao):
>> I am on kernel 3.1.0-rc4.  I am ssh'd into a remote machine so have
>> no access to keyboard, any other way to get the info you need?
> echo 'w'>  /proc/sysrq-trigger
>
>
>> On 10/18/2011 09:31 AM, Josef Bacik wrote:
>>> On Tue, Oct 18, 2011 at 09:25:46AM -0400, Jim wrote:
>>>> Good morning btrfs list,
>>>> I have been rsyncing files from an nfs mount to a btrfs filesystem.
>>>> After an rsync run I ls random subvols or directorys to check the
>>>> copy.  About 60% to 70% of the time ls completely hangs.  Ps aux
>>>> shows it as running but even when I let it go for up to an hour it
>>>> never finishes.  Kill -9 will not stop the process.  Dmesg shows
>>>> nothing beyond a successful mount at boot.  I can't umount the
>>>> system because "filesystem is busy".  I find that a forced reboot is
>>>> the only way to recapture the system.  /var/log/messages has the
>>>> only indication (that I can find) that anything abnormal is
>>>> happening.  A tail of the file is below.  Thank you for any help and
>>>> advice.
>>> Hit sysrq+w when this happens and give us all the tracebacks.  Also which kernel
>>> you are on would be helpful.  Thanks,
>>>
>>> Josef
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ls hangs filesystem

[Josef Bacik]

On Tue, Oct 18, 2011 at 10:03:40AM -0400, Jim wrote:
> I am getting no output from echo 'w' > /proc/sysrq-trigger.  Ls is
> hung again so I can try any thing else you suggest to get data.

Do

echo 1> /proc/sys/kernel/sysrq

and then try doing the echo w thing.  Thanks,

Josef

Re: ls hangs filesystem

[Jim]

Yes, I had thought of that as well but still no output.  Ls has been 
hung for about 30 min now and shows as D+ in ps.  Do you think it would 
get us any information if I forced a reboot and redid the entire 
sequence running sysrq-trigger right after starting ls?
Jim

On 10/18/2011 10:04 AM, Josef Bacik wrote:
> On Tue, Oct 18, 2011 at 10:03:40AM -0400, Jim wrote:
>> I am getting no output from echo 'w'>  /proc/sysrq-trigger.  Ls is
>> hung again so I can try any thing else you suggest to get data.
> Do
>
> echo 1>  /proc/sys/kernel/sysrq
>
> and then try doing the echo w thing.  Thanks,
>
> Josef

Re: ls hangs filesystem

[Josef Bacik]

On Tue, Oct 18, 2011 at 10:08:30AM -0400, Jim wrote:
> Yes, I had thought of that as well but still no output.  Ls has been
> hung for about 30 min now and shows as D+ in ps.  Do you think it
> would get us any information if I forced a reboot and redid the
> entire sequence running sysrq-trigger right after starting ls?

Maybe sysrq+t will get it?  Thanks,

Josef

Re: ls hangs filesystem

[Jim]

Thank you both (Sander and Josef) I can forget anything so all reminders 
are appreciated.  I will try again and also use the t argument.
Jim

On 10/18/2011 10:09 AM, Josef Bacik wrote:
> On Tue, Oct 18, 2011 at 10:08:30AM -0400, Jim wrote:
>> Yes, I had thought of that as well but still no output.  Ls has been
>> hung for about 30 min now and shows as D+ in ps.  Do you think it
>> would get us any information if I forced a reboot and redid the
>> entire sequence running sysrq-trigger right after starting ls?
> Maybe sysrq+t will get it?  Thanks,
>
> Josef

Re: ls hangs filesystem

[Jim]

Thank you all for your prompt responses.  I forced a reboot, remounted 
the fs and now cannot recreate the behavior.  Ls appears to be working 
as expected.  If it hangs later today I will get all the info I can and 
repost under this subject.  Thank you again for your assistance.
Jim

On 10/18/2011 10:09 AM, Josef Bacik wrote:
> On Tue, Oct 18, 2011 at 10:08:30AM -0400, Jim wrote:
>> Yes, I had thought of that as well but still no output.  Ls has been
>> hung for about 30 min now and shows as D+ in ps.  Do you think it
>> would get us any information if I forced a reboot and redid the
>> entire sequence running sysrq-trigger right after starting ls?
> Maybe sysrq+t will get it?  Thanks,
>
> Josef

Re: ls hangs filesystem

[Sander]

Josef Bacik wrote (ao):
> On Tue, Oct 18, 2011 at 10:03:40AM -0400, Jim wrote:
> > I am getting no output from echo 'w' > /proc/sysrq-trigger.  Ls is
> > hung again so I can try any thing else you suggest to get data.
> 
> Do
> 
> echo 1> /proc/sys/kernel/sysrq
> 
> and then try doing the echo w thing.  Thanks,

Jim, sorry if I'm stating the obvious here, but the output will be in
dmesg (and syslog if possible).

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net