Occassional problems with unfreeze

Occassional problems with unfreeze

[Hrishikesh Barua]

[-- Attachment #1.1: Type: text/plain, Size: 938 bytes --]

We use XFS on Amazon EBS on Ubuntu 12.04.3, kernel "3.2.0-54-virtual".

Our backup scripts invoke "xfs_freeze -f" before taking a snapshot to
ensure filesystem consistency. After the snapshot is taken, we invoke
"xfs_freeze -u".

Recently, we have observed that some of these unfreeze calls exit with code
255. This results in a frozen filesystem, processes queuing up and us
having to reboot the EC2 instance. Invoking freeze -u manually results in
multiple processes like this -

     17628 ?        D      0:00 /usr/sbin/xfs_io -F -r -p xfs_freeze -x -c
thaw /mysql-vol-a
     17694 ?        D      0:00 /usr/sbin/xfs_io -F -r -p xfs_freeze -x -c
thaw /mysql-vol-a

A search on this list reveals at least one other instance of this problem -
http://oss.sgi.com/archives/xfs/2013-01/msg00437.html

No resolution seems to have happened there. Is this a known bug, or has it
been fixed since (I'm running xfsprogs 3.1.7) ?

Regards
Hrish

[-- Attachment #1.2: Type: text/html, Size: 1902 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Occassional problems with unfreeze

[Brian Foster]

On Wed, Jul 16, 2014 at 04:50:22PM +0530, Hrishikesh Barua wrote:
> We use XFS on Amazon EBS on Ubuntu 12.04.3, kernel "3.2.0-54-virtual".
> 
> Our backup scripts invoke "xfs_freeze -f" before taking a snapshot to
> ensure filesystem consistency. After the snapshot is taken, we invoke
> "xfs_freeze -u".
> 
> Recently, we have observed that some of these unfreeze calls exit with code
> 255. This results in a frozen filesystem, processes queuing up and us
> having to reboot the EC2 instance. Invoking freeze -u manually results in
> multiple processes like this -
> 
>      17628 ?        D      0:00 /usr/sbin/xfs_io -F -r -p xfs_freeze -x -c
> thaw /mysql-vol-a
>      17694 ?        D      0:00 /usr/sbin/xfs_io -F -r -p xfs_freeze -x -c
> thaw /mysql-vol-a
> 
> A search on this list reveals at least one other instance of this problem -
> http://oss.sgi.com/archives/xfs/2013-01/msg00437.html

That one looks like a freeze hang whereas you see an unfreeze hang.
Perhaps similar to this:

https://bugzilla.redhat.com/show_bug.cgi?id=1052004

> 
> No resolution seems to have happened there. Is this a known bug, or has it
> been fixed since (I'm running xfsprogs 3.1.7) ?
> 

Do you have the hung task output for when this occurs (echo w >
/proc/sysrq-trigger)?

Brian

> Regards
> Hrish

> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Occassional problems with unfreeze

[Hrishikesh Barua]

[-- Attachment #1.1: Type: text/plain, Size: 926 bytes --]

On 16 July 2014 18:17, Brian Foster <bfoster@redhat.com> wrote:

>
> That one looks like a freeze hang whereas you see an unfreeze hang.
>
You are right. My bad.


> Perhaps similar to this:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1052004
>
Looks like I can't access that bug report, even after logging in. The error
message says  "Most likely the bug has been restricted for internal
development processes and we cannot grant access."

>
> >
> > No resolution seems to have happened there. Is this a known bug, or has
> it
> > been fixed since (I'm running xfsprogs 3.1.7) ?
> >
>
> Do you have the hung task output for when this occurs (echo w >
> /proc/sysrq-trigger)?
>
No, but I'll get it the next time it happens. Thanks for the tip.

>
> Brian
>
> > Regards
> > Hrish
>
> > _______________________________________________
> > xfs mailing list
> > xfs@oss.sgi.com
> > http://oss.sgi.com/mailman/listinfo/xfs
>
>

[-- Attachment #1.2: Type: text/html, Size: 2060 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Occassional problems with unfreeze

[Hrishikesh Barua]

[-- Attachment #1.1: Type: text/plain, Size: 1354 bytes --]

This happened again yesterday and I took the "echo w >
/proc/sysrq-trigger".
Output - http://pastebin.com/rpKNSYY7

- Hrish

On 16 July 2014 18:53, Hrishikesh Barua <talonx@gmail.com> wrote:

>
> On 16 July 2014 18:17, Brian Foster <bfoster@redhat.com> wrote:
>
>>
>> That one looks like a freeze hang whereas you see an unfreeze hang.
>>
> You are right. My bad.
>
>
>> Perhaps similar to this:
>>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1052004
>>
> Looks like I can't access that bug report, even after logging in. The
> error message says  "Most likely the bug has been restricted for internal
> development processes and we cannot grant access."
>
>>
>> >
>> > No resolution seems to have happened there. Is this a known bug, or has
>> it
>> > been fixed since (I'm running xfsprogs 3.1.7) ?
>> >
>>
>> Do you have the hung task output for when this occurs (echo w >
>> /proc/sysrq-trigger)?
>>
> No, but I'll get it the next time it happens. Thanks for the tip.
>
>>
>> Brian
>>
>> > Regards
>> > Hrish
>>
>> > _______________________________________________
>> > xfs mailing list
>> > xfs@oss.sgi.com
>> > http://oss.sgi.com/mailman/listinfo/xfs
>>
>>
>


-- 
You can't be normal and expect abnormal returns - Jeffrey Pfeffer
------------------------------------------
http://code.deepinspace.net
------------------------------------------

[-- Attachment #1.2: Type: text/html, Size: 2992 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Occassional problems with unfreeze

[Dave Chinner]

On Thu, Sep 25, 2014 at 11:37:47AM +0530, Hrishikesh Barua wrote:
> This happened again yesterday and I took the "echo w >
> /proc/sysrq-trigger".
> Output - http://pastebin.com/rpKNSYY7

How long have you been ignoring these messages for?

[30278027.798067] XFS (xvdj): xlog_space_left: head behind tail
[30278027.798069]   tail_cycle = 1085, tail_bytes = 1041920
[30278027.798070]   GH   cycle = 1085, GH   bytes = 1039296

XFS is emitting that warning several times a second. Isn't that a
pretty good sign that there's something not quite right and is worth
reporting?

Indeed, you are running an old kern (3.2.0-54) which means it won't
have this fix:

3948659 xfs: Account log unmount transaction correctly

And that's likely to be the source of all your freeze/unfreeze
problems because it will eventually hang the log when it's leaked
all it's space in the ether....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Occassional problems with unfreeze

[Hrishikesh Barua]

[-- Attachment #1.1: Type: text/plain, Size: 494 bytes --]

> Indeed, you are running an old kern (3.2.0-54) which means it won't
> have this fix:
>
> 3948659 xfs: Account log unmount transaction correctly
>
> And that's likely to be the source of all your freeze/unfreeze
> problems because it will eventually hang the log when it's leaked
> all it's space in the ether....
>

We did get a few pointers earlier that 3948659 might be the cause of the
freeze/unfreeze problems but now that you have confirmed it we'll upgrade
the kernel. Thanks.

- Hrish

[-- Attachment #1.2: Type: text/html, Size: 791 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs