Finding what is stuck...

Finding what is stuck...

[J.A. Magallón]

Hi all...

I'm running 2.6.27-rc5-git2 on an Aspire One.
The system is in general pretty responsive, but sometimes it just gets
totally stuck. Even the mouse stops. 

It looks related to disk (SSD) access, but I'm not totally sure.
Is there any way to find what is getting stuck ? I know that SSDs can
be slow on write, I don't mind if the system is faster or slower
(it's small :))), but if the speed is constant. That occasional
pauses are strange, like if SSD flushing gets stuck on BKL
(I know, no idea about what I talk...).

I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the
disk writes are the culprit.

Any idea about how to find this ?

TIA

-- 
J.A. Magallon                       \                     Software is like sex:
                                     \               It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux kernel 2.6.23-jam01,  gcc version 4.2.2 20070909

Re: Finding what is stuck...

[Arjan van de Ven]

On Tue, 2 Sep 2008 02:04:47 +0200
"J.A. Magallón" <jamagallon@ono.com> wrote:

> Hi all...
> 
> I'm running 2.6.27-rc5-git2 on an Aspire One.
> The system is in general pretty responsive, but sometimes it just gets
> totally stuck. Even the mouse stops. 
> 
> It looks related to disk (SSD) access, but I'm not totally sure.
> Is there any way to find what is getting stuck ? I know that SSDs can
> be slow on write, I don't mind if the system is faster or slower
> (it's small :))), but if the speed is constant. That occasional
> pauses are strange, like if SSD flushing gets stuck on BKL
> (I know, no idea about what I talk...).
> 
> I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the
> disk writes are the culprit.
> 
> Any idea about how to find this ?

Have you tried to run "latencytop"?
(you need to enable this in the kernel config as well)

it tends to (for me at least) point out very well where stalls happen,
or at least, what the system is doing when they happen.

(hint: make sure you do "make install" before running it)

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: Finding what is stuck...

[J.A. Magallón]

On Mon, 1 Sep 2008 17:08:12 -0700, Arjan van de Ven <arjan@infradead.org> wrote:

> On Tue, 2 Sep 2008 02:04:47 +0200
> "J.A. Magallón" <jamagallon@ono.com> wrote:
> 
> > Hi all...
> > 
> > I'm running 2.6.27-rc5-git2 on an Aspire One.
> > The system is in general pretty responsive, but sometimes it just gets
> > totally stuck. Even the mouse stops. 
> > 
> > It looks related to disk (SSD) access, but I'm not totally sure.
> > Is there any way to find what is getting stuck ? I know that SSDs can
> > be slow on write, I don't mind if the system is faster or slower
> > (it's small :))), but if the speed is constant. That occasional
> > pauses are strange, like if SSD flushing gets stuck on BKL
> > (I know, no idea about what I talk...).
> > 
> > I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the
> > disk writes are the culprit.
> > 
> > Any idea about how to find this ?
> 
> Have you tried to run "latencytop"?
> (you need to enable this in the kernel config as well)
> 
> it tends to (for me at least) point out very well where stalls happen,
> or at least, what the system is doing when they happen.
> 
> (hint: make sure you do "make install" before running it)
> 

Thanks, I'm a bit limited to distro kernels 'cause I don't want to build
my own kernel on the One, but I have just looked and Mandriva kernel is
built with LATENCYTOP=y. I will try that and report...

-- 
J.A. Magallon                       \                     Software is like sex:
                                     \               It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux kernel 2.6.23-jam01,  gcc version 4.2.2 20070909

Re: Finding what is stuck...

[Jan Knutar]

On Tuesday 02 September 2008, J.A. Magallón wrote:

> I'm running 2.6.27-rc5-git2 on an Aspire One.
> The system is in general pretty responsive, but sometimes it just
> gets totally stuck. Even the mouse stops.
>
> It looks related to disk (SSD) access, but I'm not totally sure.

Are you hitting swap at all? Does it have swap?

On my N810 tablet there can be a huge slowdown if it starts swapping. 
The first time it dips into swap is relatively painless when stuff gets 
written out sequentially, but after that when writes are mostly random 
the throughput drops to a few kilobytes per second, causing massive 
slowdown... The MMC/SD cards hate random write, and I expect SSDs are 
no better.

iostat -x -k 10
in a terminal is useful. Check the iowait and util% numbers after/during 
slowdown...

Re: Finding what is stuck...

[Alan Jenkins]

Jan Knutar wrote:

> On Tuesday 02 September 2008, J.A. Magallón wrote:
>
>> I'm running 2.6.27-rc5-git2 on an Aspire One.
>> The system is in general pretty responsive, but sometimes it just
>> gets totally stuck. Even the mouse stops.
>>
>> It looks related to disk (SSD) access, but I'm not totally sure.
>
> Are you hitting swap at all? Does it have swap?
>
> On my N810 tablet there can be a huge slowdown if it starts swapping.
> The first time it dips into swap is relatively painless when stuff gets
> written out sequentially, but after that when writes are mostly random
> the throughput drops to a few kilobytes per second, causing massive
> slowdown... The MMC/SD cards hate random write, and I expect SSDs are
> no better.
>
> iostat -x -k 10
> in a terminal is useful. Check the iowait and util% numbers after/during
> slowdown...

My experience on the EeePC is similar.  I didn't look into it much but I
think it helped a lot switching to the "noop" ioscheduler

(echo noop > /sys/block/sda/queue/scheduler)

as opposed to the default CFQ, if you haven't already done so.

Alan

Re: Finding what is stuck...

[J.A. Magallón]

On Mon, 1 Sep 2008 17:08:12 -0700, Arjan van de Ven <arjan@infradead.org> wrote:

> On Tue, 2 Sep 2008 02:04:47 +0200
> "J.A. Magallón" <jamagallon@ono.com> wrote:
> 
> > Hi all...
> > 
> > I'm running 2.6.27-rc5-git2 on an Aspire One.
> > The system is in general pretty responsive, but sometimes it just gets
> > totally stuck. Even the mouse stops. 
> > 
> > It looks related to disk (SSD) access, but I'm not totally sure.
> > Is there any way to find what is getting stuck ? I know that SSDs can
> > be slow on write, I don't mind if the system is faster or slower
> > (it's small :))), but if the speed is constant. That occasional
> > pauses are strange, like if SSD flushing gets stuck on BKL
> > (I know, no idea about what I talk...).
> > 
> > I'm using ext3 fs, noop iosched. But as I say, I'm not sure that the
> > disk writes are the culprit.
> > 
> > Any idea about how to find this ?
> 
> Have you tried to run "latencytop"?
> (you need to enable this in the kernel config as well)
> 
> it tends to (for me at least) point out very well where stalls happen,
> or at least, what the system is doing when they happen.
> 
> (hint: make sure you do "make install" before running it)
> 

These are some shots of latencytop while working. I copied the screen
when I saw any very high timing...

Cause                                                Maximum     Percentage
Walking directory tree                            790.2 msec         10.6 %
Scheduler: waiting for cpu                         40.5 msec         38.5 %

fsync() on a file                                 3191.6 msec         36.7 %
Reading EXT3 directory                            1553.3 msec          3.9 %
EXT3: Waiting for journal access                  1518.7 msec          3.8 %
Deleting an inode                                 657.5 msec          1.7 %
EXT3: Looking for file                            576.3 msec          1.4 %
Writing a page to disk                            363.0 msec          6.1 %
call_usermodehelper_exec request_module __sock_cre 69.4 msec          0.2 %
Scheduler: waiting for cpull                       28.0 msec         18.9 %
Page fault                                          7.9 msec          0.3 %

fsync() on a file                                 8843.7 msec         81.8 %
Writing a page to disk                            1548.6 msec          2.9 %
EXT3: Waiting for journal access                  356.5 msec          1.1 %
Scheduler: waiting for cpu                         40.6 msec          6.0 %

EXT3: Waiting for journal access                  6153.6 msec         17.2 %
fsync() on a file                                 4902.7 msec         53.5 %
Writing a page to disk                            3487.2 msec          9.8 %
Writing buffer to disk (synchronous)              2288.7 msec          4.6 %
synchronous write                                 1320.6 msec          2.6 %
Truncating file                                   302.6 msec          0.6 %
Page fault                                        290.5 msec          0.6 %
Scheduler: waiting for cpu                         13.4 msec          4.5 %
Waiting for event (select)                          5.0 msec          5.0 %

Deleting an inode                                 5584.7 msec         27.5 %
Writing a page to disk                            829.2 msec          5.4 %
fsync() on a file                                 265.1 msec         11.2 %
Scheduler: waiting for cpu                         37.2 msec         11.5 %
stat() operation                                    5.6 msec          0.1 %
Page fault                                          5.5 msec          0.2 %
Waiting for event (select)                          5.0 msec         39.8 %
Waiting for event (poll)                            5.0 msec          2.5 %
synchronous write                                   4.2 msec          0.0 %

All look fs access:

one:~# df
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/sda1     ext3    6.9G  3.7G  3.2G  55% /
tmpfs        tmpfs    246M   12K  246M   1% /tmp
/dev/mmcblk0p1
              ext3    1.9G   35M  1.8G   2% /store

I will try mounting root as ext2, to rule out jorunaling...

TIA

-- 
J.A. Magallon                       \                     Software is like sex:
                                     \               It's better when it's free
Mandriva Linux release 2008.1 (Cooker) for i586
Linux kernel 2.6.23-jam01,  gcc version 4.2.2 20070909

Re: Finding what is stuck...

[Arjan van de Ven]

On Thu, 4 Sep 2008 01:00:43 +0200
"J.A. Magallón" <jamagallon@ono.com> wrote:

> On Mon, 1 Sep 2008 17:08:12 -0700, Arjan van de Ven
> <arjan@infradead.org> wrote:
> 
> > On Tue, 2 Sep 2008 02:04:47 +0200
> > "J.A. Magallón" <jamagallon@ono.com> wrote:
> > 
> > > Hi all...
> > > 
> > > I'm running 2.6.27-rc5-git2 on an Aspire One.
> > > The system is in general pretty responsive, but sometimes it just
> > > gets totally stuck. Even the mouse stops. 
> > > 
> > > It looks related to disk (SSD) access, but I'm not totally sure.
> > > Is there any way to find what is getting stuck ? I know that SSDs
> > > can be slow on write, I don't mind if the system is faster or
> > > slower (it's small :))), but if the speed is constant. That
> > > occasional pauses are strange, like if SSD flushing gets stuck on
> > > BKL (I know, no idea about what I talk...).
> > > 
> > > I'm using ext3 fs, noop iosched. But as I say, I'm not sure that
> > > the disk writes are the culprit.
> > > 
> > > Any idea about how to find this ?
> > 
> > Have you tried to run "latencytop"?
> > (you need to enable this in the kernel config as well)
> > 
> > it tends to (for me at least) point out very well where stalls
> > happen, or at least, what the system is doing when they happen.
> > 
> > (hint: make sure you do "make install" before running it)
> > 
> 
> These are some shots of latencytop while working. I copied the screen
> when I saw any very high timing...
> 

wow bad ones..

one thing to note.. you have *something* doing fsync() a lot it seems
(latencytop is likely to tell you which one it is); fsync on ext3 is
really expensive, especially on an ssd that is slow to write to.


-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org