Sudden "hangs"....

Sudden "hangs"....

[Rogier Wolff]

Hi,

It seems my home workstation hangs on a "mkdir" once every morning for
about half a minute. It'll freeze whatever I'm doing and continue
happily 30 seconds later, but I can't figure out what's going on
because it's frozen...

My server did it just now and I was able to grab some more
debugging info..... 

obelix:~> time mkdir /recover3/testdir
0.000u 0.004s 2:11.83 0.0%      0+0k 0+0io 5pf+0w
obelix:~> 

Over two minutes to make a directory! (I have the impression that
the wait is for something to finish, so if I had started the mkdir
a minute earlier, it would've taken a minute longer). 

The process:
28649 root      20   0     0    0    0 D  6.3  0.0   0:11.69 flush-9:0 

was in "top", and as shown here in "disk wait" mode. 

While this is going on, "vmstat 1"  shows:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  1   5368  12764 324804 1616932    0    0   464     0  244  477  0  2  0 98
 0  1   5368  12764 324556 1617168    0    0   448     0  255  474  0  1  0 99
 0  1   5368  12764 324952 1616832    0    0   472     0  257  487  0  1  0 99

... about 400 kbytes of IO per second. This is similar to the
single-disk machine at home.

The server runs: 

Linux version 2.6.34 (wolff@abra2) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue Jun 1 14:44:01 CEST 2010

at home I run: 
Linux version 2.6.28-19-generic (buildd@palmer) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #61-Ubuntu SMP Wed May 26 23:35:15 UTC 2010


Does anybody know what's going on? 

	Roger. 


-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ

Re: Sudden "hangs"....

[Satoru Takeuchi]

Hi,

(2010/09/16 23:33), Rogier Wolff wrote:
>
> Hi,
>
> It seems my home workstation hangs on a "mkdir" once every morning for
> about half a minute. It'll freeze whatever I'm doing and continue
> happily 30 seconds later, but I can't figure out what's going on
> because it's frozen...
>
> My server did it just now and I was able to grab some more
> debugging info.....
>
> obelix:~>  time mkdir /recover3/testdir
> 0.000u 0.004s 2:11.83 0.0%      0+0k 0+0io 5pf+0w
> obelix:~>
>
> Over two minutes to make a directory! (I have the impression that
> the wait is for something to finish, so if I had started the mkdir
> a minute earlier, it would've taken a minute longer).
>
> The process:
> 28649 root      20   0     0    0    0 D  6.3  0.0   0:11.69 flush-9:0
>
> was in "top", and as shown here in "disk wait" mode.
>
> While this is going on, "vmstat 1"  shows:
>
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>   r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
>   0  1   5368  12764 324804 1616932    0    0   464     0  244  477  0  2  0 98
>   0  1   5368  12764 324556 1617168    0    0   448     0  255  474  0  1  0 99
>   0  1   5368  12764 324952 1616832    0    0   472     0  257  487  0  1  0 99

It seems normal data. I suggest you to do the check dmesg and
iostat to confirm whether something wrong is occuring on your
disk which you issued mkdir.

Thanks,
Satoru

>
> ... about 400 kbytes of IO per second. This is similar to the
> single-disk machine at home.
>
> The server runs:
>
> Linux version 2.6.34 (wolff@abra2) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) #5 SMP Tue Jun 1 14:44:01 CEST 2010
>
> at home I run:
> Linux version 2.6.28-19-generic (buildd@palmer) (gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) ) #61-Ubuntu SMP Wed May 26 23:35:15 UTC 2010
>
>
> Does anybody know what's going on?
>
> 	Roger.
>
>

Re: Sudden "hangs"....

[Wakko Warner]

Rogier Wolff wrote:
> It seems my home workstation hangs on a "mkdir" once every morning for
> about half a minute. It'll freeze whatever I'm doing and continue
> happily 30 seconds later, but I can't figure out what's going on
> because it's frozen...

One of my volumes was doing this.  As someone mentioned earlier, check your
kernel log to see if you have a disk error.  On mine, I had no errors.  The
activity led for that volume would be on solid until the mkdir completed. 
My fix was to create a new directory and move everything over and remove the
old directory.  Obviously this won't work if you're having problems with
more than 1 directory or the offending directory is the root of the volume.

Just for information, this volume I have is on a single disk created on the
whole disk (no lvm nor partition).

-- 
 Microsoft has beaten Volkswagen's world record.  Volkswagen only created 22
 million bugs.

Re: Sudden "hangs"....

[Rogier Wolff]

On Fri, Sep 17, 2010 at 12:47:23PM -0400, Wakko Warner wrote:
> Rogier Wolff wrote:
> > It seems my home workstation hangs on a "mkdir" once every morning for
> > about half a minute. It'll freeze whatever I'm doing and continue
> > happily 30 seconds later, but I can't figure out what's going on
> > because it's frozen...
> 
> One of my volumes was doing this.  As someone mentioned earlier, check your
> kernel log to see if you have a disk error.

I work at "harddisk-recovery.nl". I know how to recognize bad disks. You can
recognize a bad block in the vmstat output in that the bi and bo numbers
go to zero for at least 10 seconds. This doesn't happen, as you can see
in the original post. 

> On mine, I had no errors.  The activity led for that volume would be
> on solid until the mkdir completed. 

Besides that I can't see the activity led from my chair. But vmstat shows
that it would be solid indeed. 

> My fix was to create a new
> directory and move everything over and remove the old directory.
> Obviously this won't work if you're having problems with more than 1
> directory or the offending directory is the root of the volume.

This helps for the "very big directory" case. Directories are never
shrunk. So if a directory has become very large because of thousands
of files at one point in time, you can reduce that with the move-all-files
trick. 

When it happened and I recognized the effect, I typed the "time mkdir"
command. I just chose the root of the partition for this. The directory
there is NOT large: 

drwxrwsr-x 32 root hdr 4096 2010-09-17 22:05 .

In fact it is as small as possible. 

IMHO, the hint is that the flush process for the device ends up in
top. Now if I understand the word "flush" correctly I'd expect it to
write dirty buffers to the device. In this case, vmstat however
reports mostly data coming INTO the computer. So why would the flush
process need to READ data from the device?



	Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
Q: It doesn't work. A: Look buddy, doesn't work is an ambiguous statement. 
Does it sit on the couch all day? Is it unemployed? Please be specific! 
Define 'it' and what it isn't doing. --------- Adapted from lxrbot FAQ