Poor desktop responsiveness with background I/O-operations

Poor desktop responsiveness with background I/O-operations

[Ulrich Lukas]

[-- Attachment #1: Type: text/plain, Size: 1848 bytes --]

Hi,


using a recent hard-/software setup, I observed that continuous
read/write operations severely degrade the overall system responsiveness
in typical desktop-PC use cases.

Merely doing write/read operations on a data volume leads to stuck text
and mouse cursors, seconds-long delays for simple window-context
switches in X11, dropouts in low-resolution video playback etc.



Test case:
- 64-bit dual-core PC, SATA harddrive, plenty of free RAM
- vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit


How to reproduce:
- start KDE/GNOME-session
- open a terminal window and do as a non-root user:
  dd if=/dev/zero of=/home/john-doe/testfile
  (or dd if=/home/john-doe/big-testfile of=/dev/null)

- a real use scenario would be a daily disk-backup or the
  simple extraction of a tarball containing slightly bigger files


Observation:
- The system becomes _really_ slow as described above; unusable for
  any multimedia tasks.

- Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers)
  compounds the issue to a painful extent.



Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.

I realize that a single heavyweight transfer slows down I/O for the
corresponding transactions/processes/volumes etc.

But there needs to be a fair distribution of I/O or CPU time which
leaves enough for other basic operations. And this doesn't seem to be
the case with recent Linux versions.




On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This
yields significantly higher throughput when using disk encryption (50%
improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But
with these patches, the responsiveness was even worse during my quick
test. Switching to a text-mode console: several /minutes/ delay...


I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)


[-- Attachment #2: kernelconfig-linux-2.6.31.bz2 --]
[-- Type: application/octet-stream, Size: 13204 bytes --]

Re: Poor desktop responsiveness with background I/O-operations

[Thomas Fjellstrom]

On Sat September 19 2009, Ulrich Lukas wrote:
> Hi,
> 
> 
> using a recent hard-/software setup, I observed that continuous
> read/write operations severely degrade the overall system responsiveness
> in typical desktop-PC use cases.
> 
> Merely doing write/read operations on a data volume leads to stuck text
> and mouse cursors, seconds-long delays for simple window-context
> switches in X11, dropouts in low-resolution video playback etc.
> 
> 
> 
> Test case:
> - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
> 
> 
> How to reproduce:
> - start KDE/GNOME-session
> - open a terminal window and do as a non-root user:
>   dd if=/dev/zero of=/home/john-doe/testfile
>   (or dd if=/home/john-doe/big-testfile of=/dev/null)
> 
> - a real use scenario would be a daily disk-backup or the
>   simple extraction of a tarball containing slightly bigger files
> 
> 
> Observation:
> - The system becomes _really_ slow as described above; unusable for
>   any multimedia tasks.
> 
> - Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers)
>   compounds the issue to a painful extent.
> 
> 
> 
> Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.
> 
> I realize that a single heavyweight transfer slows down I/O for the
> corresponding transactions/processes/volumes etc.
> 
> But there needs to be a fair distribution of I/O or CPU time which
> leaves enough for other basic operations. And this doesn't seem to be
> the case with recent Linux versions.
> 
> 
> 
> 
> On a side note, I've tried the BFS-patches (bfs-221 on 2.6.31): This
> yields significantly higher throughput when using disk encryption (50%
> improvement with dm-crypt/LUKS, 512 bit aes-xts-plain cipher mode). But
> with these patches, the responsiveness was even worse during my quick
> test. Switching to a text-mode console: several /minutes/ delay...
> 
> 
> I'm attaching my .config for linux 2.6.31 (grep ^C and bzip2-ed)
> 

I've seen similar things with my Intel p35 (q6600) system, especially under 
64bit. Heavy i/o makes the rest of the system perform very badly. I have 
however not noticed nearly as much of a problem with my AMD (SB700&SB750) 
systems.

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: Poor desktop responsiveness with background I/O-operations

[Arjan van de Ven]

On Sun, 20 Sep 2009 05:08:43 +0200
Ulrich Lukas <stellplatz-nr.13a@datenparkplatz.de> wrote:
> 
> Observation:
> - The system becomes _really_ slow as described above; unusable for
>   any multimedia tasks.
> 
> - Using an encrypted (dm-crypt/LUKS) /home (e.g. on mobile computers)
>   compounds the issue to a painful extent.
> 
> 
> 
> Possible culprits (I'm guessing) are the Linux I/O- or CPU scheduler.

this has nothing to do with the cpu scheduler.... 
can you try the following two things?

echo 4096 > /sys/block/sda/queue/nr_requests
for i in `pidof kjournald` ; do ionice -c1 -p $i ; done

to see if they help?

they tweak the IO scheduler... I assume you're using CFQ.

Re: Poor desktop responsiveness with background I/O-operations

[Ulrich Lukas]

Arjan van de Ven wrote:
> can you try the following two things?
> 
> echo 4096 > /sys/block/sda/queue/nr_requests
> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done

I tried this just now, but it doesn't change things much. I can do a
more testing later today.


> I assume you're using CFQ.

Yes, it's CFQ. (see .config of my previous posting)

Filesystem is EXT4.

Re: Poor desktop responsiveness with background I/O-operations

[Nikos Chantziaras]

(Please don't strip the CC list when you reply.  Use the "Reply to all" 
functionality of your mail client and keep all CCs.)

On 09/20/2009 11:50 AM, Ulrich Lukas wrote:
> Arjan van de Ven wrote:
>> can you try the following two things?
>>
>> echo 4096>  /sys/block/sda/queue/nr_requests
>> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
>
> I tried this just now, but it doesn't change things much. I can do a
> more testing later today.
>
>
>> I assume you're using CFQ.
>
> Yes, it's CFQ. (see .config of my previous posting)
>
> Filesystem is EXT4.

I can also confirm total lock-up when copying a big file (Intel Core 2 
Duo, Intel X38 chipset).  Everything locks-up for several seconds during 
the copying.  I'm also on ext4.  I've been told on various mailing lists 
that this is normal, expected and unavoidable so I pretty much think the 
only solution is to avoid heavy I/O :P

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

[-- Attachment #1: Type: text/plain, Size: 2748 bytes --]

On Sun, 2009-09-20 at 10:50 +0200, Ulrich Lukas wrote:
> Arjan van de Ven wrote:
> > can you try the following two things?
> > 
> > echo 4096 > /sys/block/sda/queue/nr_requests
> > for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
> 
> I tried this just now, but it doesn't change things much. I can do a
> more testing later today.

Try ionice -c3 for your dd.

> > I assume you're using CFQ.
> 
> Yes, it's CFQ. (see .config of my previous posting)
> 
> Filesystem is EXT4.

Mine is ext3.

I can easily reproduce pain.  The attached cheezy test script, when run
back to back with no args (stock IO), regularly produces beauties:

 Performance counter stats for 'sh -c konsole -e exit':

     257.670601  task-clock-msecs         #      0.004 CPUs
           1036  context-switches         #      0.004 M/sec
             30  CPU-migrations           #      0.000 M/sec
           6317  page-faults              #      0.025 M/sec
      471734719  cycles                   #   1830.767 M/sec
      393365060  instructions             #      0.834 IPC
        7849464  cache-references         #     30.463 M/sec
        1038976  cache-misses             #      4.032 M/sec

   65.766937233  seconds time elapsed

Yup, a full minute to get a binary loaded and fired up.  Seems it and
others get evicted by the VM.  Getting whatever back into ram is painful
to listen to (poor disk/me.. seek latency eating MUCH time).

Calling script back to back with args 1 (realtime foreground) 3 (idle
background) does best:

 Performance counter stats for 'sh -c konsole -e exit':

     169.915036  task-clock-msecs         #      0.077 CPUs
            439  context-switches         #      0.003 M/sec
              9  CPU-migrations           #      0.000 M/sec
           6314  page-faults              #      0.037 M/sec
      363128252  cycles                   #   2137.117 M/sec
      367410945  instructions             #      1.012 IPC
        5822302  cache-references         #     34.266 M/sec
         437189  cache-misses             #      2.573 M/sec

    2.213079979  seconds time elapsed

Args 2 (best effort foreground) and 3 (idle background);

 Performance counter stats for 'sh -c konsole -e exit':

     176.689856  task-clock-msecs         #      0.015 CPUs
            357  context-switches         #      0.002 M/sec
             15  CPU-migrations           #      0.000 M/sec
           6314  page-faults              #      0.036 M/sec
      374861947  cycles                   #   2121.582 M/sec
      368929476  instructions             #      0.984 IPC
        5806286  cache-references         #     32.861 M/sec
         496651  cache-misses             #      2.811 M/sec

   12.036165968  seconds time elapsed

	-Mike

[-- Attachment #2: testo.sh --]
[-- Type: application/x-shellscript, Size: 1702 bytes --]

Re: Poor desktop responsiveness with background I/O-operations

[Justin P. Mattock]

Mike Galbraith wrote:
> On Sun, 2009-09-20 at 10:50 +0200, Ulrich Lukas wrote:
>    
>> Arjan van de Ven wrote:
>>      
>>> can you try the following two things?
>>>
>>> echo 4096>  /sys/block/sda/queue/nr_requests
>>> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
>>>        
>> I tried this just now, but it doesn't change things much. I can do a
>> more testing later today.
>>      
>
> Try ionice -c3 for your dd.
>
>    
>>> I assume you're using CFQ.
>>>        
>> Yes, it's CFQ. (see .config of my previous posting)
>>
>> Filesystem is EXT4.
>>      
>
> Mine is ext3.
>
> I can easily reproduce pain.  The attached cheezy test script, when run
> back to back with no args (stock IO), regularly produces beauties:
>
>   Performance counter stats for 'sh -c konsole -e exit':
>
>       257.670601  task-clock-msecs         #      0.004 CPUs
>             1036  context-switches         #      0.004 M/sec
>               30  CPU-migrations           #      0.000 M/sec
>             6317  page-faults              #      0.025 M/sec
>        471734719  cycles                   #   1830.767 M/sec
>        393365060  instructions             #      0.834 IPC
>          7849464  cache-references         #     30.463 M/sec
>          1038976  cache-misses             #      4.032 M/sec
>
>     65.766937233  seconds time elapsed
>
> Yup, a full minute to get a binary loaded and fired up.  Seems it and
> others get evicted by the VM.  Getting whatever back into ram is painful
> to listen to (poor disk/me.. seek latency eating MUCH time).
>
> Calling script back to back with args 1 (realtime foreground) 3 (idle
> background) does best:
>
>   Performance counter stats for 'sh -c konsole -e exit':
>
>       169.915036  task-clock-msecs         #      0.077 CPUs
>              439  context-switches         #      0.003 M/sec
>                9  CPU-migrations           #      0.000 M/sec
>             6314  page-faults              #      0.037 M/sec
>        363128252  cycles                   #   2137.117 M/sec
>        367410945  instructions             #      1.012 IPC
>          5822302  cache-references         #     34.266 M/sec
>           437189  cache-misses             #      2.573 M/sec
>
>      2.213079979  seconds time elapsed
>
> Args 2 (best effort foreground) and 3 (idle background);
>
>   Performance counter stats for 'sh -c konsole -e exit':
>
>       176.689856  task-clock-msecs         #      0.015 CPUs
>              357  context-switches         #      0.002 M/sec
>               15  CPU-migrations           #      0.000 M/sec
>             6314  page-faults              #      0.036 M/sec
>        374861947  cycles                   #   2121.582 M/sec
>        368929476  instructions             #      0.984 IPC
>          5806286  cache-references         #     32.861 M/sec
>           496651  cache-misses             #      2.811 M/sec
>
>     12.036165968  seconds time elapsed
>
> 	-Mike
>    
Wow, at first I did not notice anything
from your *.sh script, but then a few moments
later, the system became vary sluggish. firefox
takes about 15 seconds to load. OS is mint linux
with the latest HEAD on an imac9,1.

Justin P. Mattock

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

On Sun, 2009-09-20 at 17:22 -0700, Justin P. Mattock wrote:
> Wow, at first I did not notice anything
> from your *.sh script, but then a few moments
> later, the system became vary sluggish. firefox
> takes about 15 seconds to load. OS is mint linux
> with the latest HEAD on an imac9,1.

CFQ inherits IO priority from the parent on fork(), so nicing a shell or
the dd should (and does) help a LOT.  It also inherits the CPU scheduler
class.  I just checked, and IO does indeed inherit SCHED_IDLE as well as
the RT classes (SCHED_BATCH does nothing).  Setting the dd to SCHED_IDLE
makes a big difference, as it should, so there's a bigger hammer than
nice + ionice.  SCHED_IDLE is CPU_adorable + IO_nicest.

Anyway, the problem seems to have to have at least two facets:
1.  streaming IO causing pagecache reclaim.  That problem doesn't seem
to want to go away.  Fixing that won't help 2 one bit though.

2.  reads are sync, more heavily affected by seek latency than writes.
If the disk your binaries live on is seeking a lot, you pay a LOT.  

Hohum, IO + VM = haaaaard ;-)

	-Mike

Re: Poor desktop responsiveness with background I/O-operations

[Ulrich Lukas]

Hi and thanks for your reply!


Mike Galbraith wrote:
> nicing a shell or the dd should (and does) help a LOT.

If this is the only way to influence this, maybe the default settings
for the niceness of interactive and non-interactive tasks are not the
best choice. (Maybe a distribution problem in this case)



> reads are sync, more heavily affected by seek latency than writes.

But how does this explain the seconds-long delays?

If an interactive process causes a lot of seeks because of reads/writes
which "are sync", I see how this can greatly slow down otherwise
pipelined write operations, but the other way around?

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

On Mon, 2009-09-21 at 09:48 +0200, Ulrich Lukas wrote:
> Hi and thanks for your reply!

Hi.

> Mike Galbraith wrote:
> > nicing a shell or the dd should (and does) help a LOT.
> 
> If this is the only way to influence this, maybe the default settings
> for the niceness of interactive and non-interactive tasks are not the
> best choice. (Maybe a distribution problem in this case)

There is no knowledge in the CPU nor IO scheduler wrt interactive vs
non-interactive.  I've tinkered many times with a SCHED_INTERACTIVE
class, but it's not at all an easy problem, so keeps landing on the
trash heap.  I could ramble on _for ever_ about that subject, but it's
thankfully irrelevant to this thread ;-)
 
> > reads are sync, more heavily affected by seek latency than writes.
> 
> But how does this explain the seconds-long delays?

Seek latencies are cumulative is my (wild arsed) theory.

> If an interactive process causes a lot of seeks because of reads/writes
> which "are sync", I see how this can greatly slow down otherwise
> pipelined write operations, but the other way around?

If you seek between tiny reads, the more little reads you do, the more
seeks hurt.  Readahead is supposed to help, but.. the pain is there.

	-Mike 

Re: Poor desktop responsiveness with background I/O-operations

[James Cloos]

FWIW, I did not notice any problems with the last kernel I had compiled
in the -- IIRC -- 29-rc timeframe.  Or maybe it was the 30-rc timeframe.

I had that kernel up for soemthing like five weeks, on a busy, low-ram
laptop, w/o and long pauses or other noticeable pain.  For at least two
years before that the best this laptop could manage before latency
became horrific was a fortnight, and weekly reboots were desirable.

Unfortunately, before I could send out a congratulatory note (I wanted
to first confirm that the result remained) I pulled up and started see-
ing even worse latency then ever.

Turning ext4 barriers off helped avoid some of the new latency, and my
current compile (master as of the 17th) is much better than .31 was.

But long pauses still occur whenever paging and backgound i/o-bound
tasks interact.

There was a major regression in paging performance either .30 or .31
(see above).  Interestingly, top(1) also shows less swap usage for
the same usage patterns than with the older kernels.  I don't know
which is cause or which is effect.

A git pull on a clone of Linus' tree is a great way to trigger the
problem.  As is using Gentoo's portage to install/upgrade anything.
(Bad python; no alligator!¹)  (Yes, that is a burlesque of the no
doughnut joke.)

[1] Pythons which have invaded Florida have reportedly been
    witnessed eating alligators, according to a recent wire
    service article.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Poor desktop responsiveness with background I/O-operations

[Nikos Chantziaras]

On 09/21/2009 10:47 PM, James Cloos wrote:
>[...]
> I had that kernel up for soemthing like five weeks, on a busy, low-ram
> laptop, w/o and long pauses or other noticeable pain.  For at least two
> years before that the best this laptop could manage before latency
> became horrific was a fortnight, and weekly reboots were desirable.
>
> Unfortunately, before I could send out a congratulatory note (I wanted
> to first confirm that the result remained) I pulled up and started see-
> ing even worse latency then ever.
>[...]
> A git pull on a clone of Linus' tree is a great way to trigger the
> problem.  As is using Gentoo's portage to install/upgrade anything.
> (Bad python; no alligator!¹)

Fortunately, this is completely (at least here) solved by putting those 
in make.conf:

PORTAGE_NICENESS=19
PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

Re: Poor desktop responsiveness with background I/O-operations

[James Cloos]

>>>>> "Nikos" == Nikos Chantziaras <realnc@arcor.de> writes:

Nikos> On 09/21/2009 10:47 PM, James Cloos wrote:
>> is a great way to trigger the problem.  As is using Gentoo's portage
>> to install/upgrade anything.

Nikos> Fortunately, this is completely (at least here) solved by putting
Nikos> those in make.conf:

Nikos> PORTAGE_NICENESS=19
Nikos> PORTAGE_IONICE_COMMAND="ionice -c 3 -p \${PID}"

I had NICENESS set, but had never noticed that PORTAGE_IONICE_COMMAND
was available.  I'll give that a test.  Thanks.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

[-- Attachment #1: Type: text/plain, Size: 2470 bytes --]

On Mon, 2009-09-21 at 15:47 -0400, James Cloos wrote:
> FWIW, I did not notice any problems with the last kernel I had compiled
> in the -- IIRC -- 29-rc timeframe.  Or maybe it was the 30-rc timeframe.
> 
> I had that kernel up for soemthing like five weeks, on a busy, low-ram
> laptop, w/o and long pauses or other noticeable pain.  For at least two
> years before that the best this laptop could manage before latency
> became horrific was a fortnight, and weekly reboots were desirable.
> 
> Unfortunately, before I could send out a congratulatory note (I wanted
> to first confirm that the result remained) I pulled up and started see-
> ing even worse latency then ever.
> 
> Turning ext4 barriers off helped avoid some of the new latency, and my
> current compile (master as of the 17th) is much better than .31 was.
> 
> But long pauses still occur whenever paging and backgound i/o-bound
> tasks interact.
> 
> There was a major regression in paging performance either .30 or .31
> (see above).  Interestingly, top(1) also shows less swap usage for
> the same usage patterns than with the older kernels.  I don't know
> which is cause or which is effect.

If someone has a definite good/bad cutoff for the problem they're
seeing, a git bisect (yeah, takes time) would be a good thing to try.

The "interactivity while saturating root disk" thing isn't a new problem
though, I can recall that being a problem to varying degrees forever.
(forever being defined as feb '93 -> today;)

> A git pull on a clone of Linus' tree is a great way to trigger the
> problem.  As is using Gentoo's portage to install/upgrade anything.
> (Bad python; no alligator!¹)  (Yes, that is a burlesque of the no
> doughnut joke.)

Git doesn't cause me any woes whatsoever, but my box's primary drive
isn't a slow laptop drive, ram isn't tight, and it has 4 cores.

Dunno.

If you're using the CFQ IO scheduler, IO bandwidth follows CPU nice
level, so tweaking nice tweaks both CPU and IO in one go.  ionice can
also be used to tweak IO independently.

I don't know if distros provide a tool that supports SCHED_IDLE, if so,
you can use that to set both CPU and IO usage to IDLE class in one go.

If you don't have such a tool, you can try the attached for background
loads.  Someone posted it to LKML in '98, and I've been updating/using
it for experiments ever since.  Just do schedctl -I background-job, and
the job will be as polite as CPU and IO schedulers can make it.

	-Mike

[-- Attachment #2: schedctl.c --]
[-- Type: text/x-csrc, Size: 12308 bytes --]

#include <unistd.h>
#include <sched.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>

#ifndef SCHED_BATCH
#define SCHED_BATCH 3
#endif
#ifndef SCHED_ISO
#define SCHED_ISO 4
#endif
#ifndef SCHED_IDLE
#define SCHED_IDLE 5
#endif
#ifndef SCHED_SYNC
#define SCHED_SYNC 6
#endif
#ifndef SCHED_RESET_ON_FORK
#define SCHED_RESET_ON_FORK 0x40000000
#endif

void usage();
void version();
void info(int pid);
void setsched(int pid, int policy, int pri, char flags);

unsigned char *ourname;
static int reset_flag;

/* flags */
#define	GOTPID		1
#define	GOTPOLICY	2
#define	GOTPRI		4
#define GOTCMD		8
#define GOTINFO		16

int main(argc, argv)
unsigned int argc;
unsigned char *argv[];
{
	int pid;
	int policy;
	int pri = 0;
	int i1;
	int i2;
	unsigned long l1;
	char *ptr;
	char flags = 0;

	ourname = argv[0];

	if (argc <= 1) {
		usage();
		exit(1);
	}

	for (i1 = 1; i1 < argc; i1++)
		if (argv[i1][0] == '-') {	/* switch */
			if (argv[i1][1] >= '0' && argv[i1][1] <= '9') {
				l1 = strtol(&argv[i1][1], &ptr, 10);
				if (*ptr || l1 < 0 || l1 > 0x7FFF) {
					fprintf(stderr,
						"invalid priority: %s\n",
						argv[i1]);
					exit(1);
				}
				pri = l1;
				flags |= GOTPRI;
				continue;
			}
			if (argv[i1][1] == '-') {	/* long opt. */
				if (!strcasecmp("help", &argv[i1][2])) {
					usage();
					exit(0);
				}
				if (!strcasecmp("version", &argv[i1][2])) {
					version();
					exit(0);
				}
				if (!strcasecmp("info", &argv[i1][2])) {
				      INFO:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
							}
							flags |= GOTPID;
						}
					}
					if (flags & GOTPID) {
						info(pid);
						exit(0);
					} else {
						flags |= GOTINFO;
//                                              fprintf(stderr,"no PID given\n");
//                                              exit(1);
					}
					continue;
				}	/* end of --info */
				if (!strcasecmp("pid", &argv[i1][2])) {
				      PID:
					i1++;
					if (i1 >= argc) {
						fprintf(stderr,
							"option requires an argument\n");
						exit(1);
					} else {
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr || l1 < 0
						    || l1 > 0x7FFF) {
							fprintf(stderr,
								"invalid PID: %s\n",
								argv[i1]);
							exit(1);
						}
						if (!l1)
							pid = getpid();
						else
							pid = l1;
						flags |= GOTPID;
					}
					continue;
				}	/* end of --pid */
				if (!strcasecmp("priority", &argv[i1][2])
				    || !strcasecmp("pri", &argv[i1][2])) {
				      PRIORITY:
					i1++;
					if (i1 >= argc) {
						fprintf(stderr,
							"option requires an argument\n");
						exit(1);
					} else {
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							fprintf(stderr,
								"invalid priority: %s\n",
								argv[i1]);
							exit(1);
						}
						pri = l1;
						flags |= GOTPRI;
					}
					continue;
				}	/* end of --priority */
				if (!strcasecmp("other", &argv[i1][2])
				    || !strcasecmp("normal", &argv[i1][2])) {
				      OTHER:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_OTHER;
					continue;
				}	/* end of --other */
				if (!strcasecmp("batch", &argv[i1][2])) {
				      BATCH:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_BATCH;
					continue;
				}	/* end of --batch */
				if (!strcasecmp("idle", &argv[i1][2])) {
				      IDLE:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_IDLE;
					continue;
				}	/* end of --idle */
				if (!strcasecmp("sync", &argv[i1][2])) {
				      SYNC:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_SYNC;
					continue;
				}	/* end of --sync */
				if (!strcasecmp("reset", &argv[i1][2])) {
				      RESET:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					reset_flag = SCHED_RESET_ON_FORK;
					continue;
				}	/* end of --reset */
				if (!strcasecmp("fifo", &argv[i1][2])) {
				      FIFO:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_FIFO;
					continue;
				}	/* end of --fifo */
				if (!strcasecmp("rr", &argv[i1][2])
				    || !strcasecmp("round-robin",
						   &argv[i1][2])) {
				      RR:
					if (i1 < argc - 1) {
						i1++;
						l1 = strtol(argv[i1], &ptr, 0);
						if (*ptr) {
							i1--;
						} else {
							if (l1 < 0
							    || l1 > 0x7FFF) {
								fprintf(stderr,
									"invalid PID: %s\n",
									argv
									[i1]);
								exit(1);
							}
							if (!l1)
								pid = getpid();
							else {
								pid = l1;
								flags |= GOTPID;
							}
						}
					}
					flags |= GOTPOLICY;
					policy = SCHED_RR;
					continue;
				}
				/* end of --rr */
				fprintf(stderr,
					"warning: unknown long switch: %s\n",
					&argv[i1][2]);
				continue;
			}
			for (i2 = 1; i2 < strlen(argv[i1]); i2++)
				switch (argv[i1][i2]) {
				case 'h':
				case 'H':
				case '?':
					usage();
					exit(0);
				case 'i':
					goto INFO;
				case 'p':
					goto PID;
				case 'P':
					goto PRIORITY;
				case 'r':
					goto RESET;
				case 'N':
				case 'O':
					goto OTHER;
				case 'B':
					goto BATCH;
				case 'I':
					goto IDLE;
				case 'S':
					goto SYNC;
				case 'F':
					goto FIFO;
				case 'R':
					goto RR;
				default:
					fprintf(stderr,
						"warning: unknown switch: %c\n",
						argv[i1][i2]);
					break;
				}
		} else {	/* command */

//                      printf("command: %s\n",argv[i1]);
			if (flags & GOTPID) {
				fprintf(stderr,
					"warning: both PID and command given\n");
			} else
				pid = getpid();
			if (!(flags & (GOTPOLICY | GOTPRI))) {
				fprintf(stderr,
					"warning: neither policy nor priority given\n");
			}

			flags |= GOTCMD;

			setsched(pid, policy, pri, flags);

			execvp(argv[i1], (char **)(&argv[i1]));
			if (errno) {
				printf("%s: error: %s\n", argv[i1],
				       strerror(errno));
				exit(1);
			}
			break;
		}

	if (reset_flag && !(flags & GOTPOLICY)) {
		policy = sched_getscheduler(pid);
		flags |= GOTPOLICY;
	}

	if (flags & GOTINFO) {
		if (!(flags & GOTPID)) {
			fprintf(stderr, "no PID given\n");
			exit(1);
		}
		info(pid);
	}

	if (!(flags & (GOTCMD | GOTINFO))) {
		if (!(flags & GOTPID)) {
			fprintf(stderr, "neither PID nor command given\n");
			exit(1);
		}
		if (!(flags & (GOTPOLICY | GOTPRI))) {
			fprintf(stderr,
				"warning: neither policy nor priority given\n");
		}
		setsched(pid, policy, pri, flags);
	}
	exit(0);
}

void usage()
{
	printf("Usage: %s [options] [command [params...]]\n\n", ourname);
	printf("  --pid, -p <n>		specify process id\n");
	printf("  --fifo, -F [pid]	set scheduling policy to FIFO\n");
	printf("  --rr, -R [pid]	set scheduling policy to RR\n");
	printf("  --batch, -B [pid]	set scheduling policy to BATCH\n");
	printf("  --idle, -I [pid]	set scheduling policy to IDLE\n");
	printf("  --sync, -S [pid]	set scheduling policy to SYNC\n");
	printf("  --other, --normal,\n");
	printf("      -N, -O [pid]	specify normal scheduling\n");
	printf("  --reset, -r 		set SCHED_RESET_ON_FORK flag\n");
	printf("  --priority, --pri,\n");
	printf("      -<n>		set priority\n");
	printf("  --info, -i [pid]	show scheduling data\n");
	printf("  --help, -h, -H, -?	display this text\n");
	printf("  --version		show version data\n\n");
}

void version()
{
	printf("schedctl, version 0.95 (beta), 8-May-1998\n");
	printf("report bugs to aw@mail1.bet1.puv.fi or awik@freenet.fi\n");
}

void info(int pid)
{
	struct sched_param sp;
	int i1;

	i1 = sched_getscheduler(pid);
	if (errno == ESRCH) {
		fprintf(stderr, "process not found\n");
		return;
	}

	if (i1 & SCHED_RESET_ON_FORK) {
		i1 &= ~SCHED_RESET_ON_FORK;
		reset_flag = SCHED_RESET_ON_FORK;
	}

	printf("scheduling policy of process %d is ", pid);
	switch (i1) {
	case SCHED_FIFO:
		printf("FIFO");
		break;
	case SCHED_RR:
		printf("RR");
		break;
	case SCHED_OTHER:
		printf("OTHER (normal)");
		break;
	case SCHED_BATCH:
		printf("BATCH");
		break;
	case SCHED_ISO:
		printf("ISO");
		break;
	case SCHED_IDLE:
		printf("IDLE");
		break;
	case SCHED_SYNC:
		printf("SYNC");
		break;
	default:
		printf("**UNKNOWN**");
	}
	if (reset_flag)
		printf(" SCHED_RESET_ON_FORK");

	sched_getparam(pid, &sp);
	printf(", priority is %d\n", sp.sched_priority);
}

void setsched(int pid, int policy, int pri, char flags)
{
	struct sched_param sp;
	int i1;

	i1 = sched_getscheduler(pid);

	if (errno == ESRCH) {
		fprintf(stderr, "process not found\n");
		return;
	}

	if (!(flags & GOTPOLICY))
		policy = i1;
	sched_getparam(pid, &sp);
	if (!(flags & GOTPRI))
		pri = sp.sched_priority;
	else if (pri < sched_get_priority_min(policy)
		 || pri > sched_get_priority_max(policy)) {
		fprintf(stderr, "priority (%d) out of range\n", pri);
		return;
	}
	if (policy == SCHED_FIFO || policy == SCHED_RR) {
		if (!pri) {
			if (!sp.sched_priority)
				sp.sched_priority++;
		} else
			sp.sched_priority = pri;
	} else if (pri && (policy == SCHED_OTHER || policy == SCHED_BATCH ||
			   policy == SCHED_ISO || policy == SCHED_IDLE || policy == SCHED_SYNC))
		sp.sched_priority = 0;
	else
		sp.sched_priority = pri;
	if (sched_setscheduler(pid, policy|reset_flag, &sp)) {
		fprintf(stderr, "error #%d: %s\n", errno, strerror(errno));
		return;
	}

	i1 = sched_getscheduler(pid);
	if (i1 & SCHED_RESET_ON_FORK)
		i1 &= ~SCHED_RESET_ON_FORK;
	sched_getparam(pid, &sp);
	if (flags & GOTCMD)
		return;
	printf("scheduling policy of process %d set to ", pid);
	switch (i1) {
		case SCHED_FIFO:
			printf("FIFO");
			break;
		case SCHED_RR:
			printf("RR");
			break;
		case SCHED_OTHER:
			printf("OTHER");
			break;
		case SCHED_BATCH:
			printf("BATCH");
			break;
		case SCHED_ISO:
			printf("ISO");
			break;
		case SCHED_IDLE:
			printf("IDLE");
			break;
		case SCHED_SYNC:
			printf("SYNC");
			break;
	}

	if (reset_flag)
		printf(" SCHED_RESET_ON_FORK");
	printf("\n");
}

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

On Tue, 2009-09-22 at 09:06 +0200, Mike Galbraith wrote:

> I don't know if distros provide a tool that supports SCHED_IDLE, if so,
> you can use that to set both CPU and IO usage to IDLE class in one go.

P.S.  I just remembered, don't use SCHED_IDLE with kernels < 2.6.28.
There are two patches wrt this scheduling class which did not go to the
stable trees less than 2.6.28.

	-Mike

Re: Poor desktop responsiveness with background I/O-operations

[Henrique de Moraes Holschuh]

On Tue, 22 Sep 2009, Mike Galbraith wrote:
> On Tue, 2009-09-22 at 09:06 +0200, Mike Galbraith wrote:
> > I don't know if distros provide a tool that supports SCHED_IDLE, if so,
> > you can use that to set both CPU and IO usage to IDLE class in one go.
> 
> P.S.  I just remembered, don't use SCHED_IDLE with kernels < 2.6.28.
> There are two patches wrt this scheduling class which did not go to the
> stable trees less than 2.6.28.

Since 2.6.27 is long-term, maybe they should be sent to 2.6.27?  It is weird
to have a long-term stable kernel for which the recommendation is "don't use
SCHED_IDLE".

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

On Tue, 2009-09-22 at 08:22 -0300, Henrique de Moraes Holschuh wrote:
> On Tue, 22 Sep 2009, Mike Galbraith wrote:
> > On Tue, 2009-09-22 at 09:06 +0200, Mike Galbraith wrote:
> > > I don't know if distros provide a tool that supports SCHED_IDLE, if so,
> > > you can use that to set both CPU and IO usage to IDLE class in one go.
> > 
> > P.S.  I just remembered, don't use SCHED_IDLE with kernels < 2.6.28.
> > There are two patches wrt this scheduling class which did not go to the
> > stable trees less than 2.6.28.
> 
> Since 2.6.27 is long-term, maybe they should be sent to 2.6.27?  It is weird
> to have a long-term stable kernel for which the recommendation is "don't use
> SCHED_IDLE".

Yeah, that's what I thought too, but apparently maintainers didn't think
SCHED_IDLE was common/important enough.

	-Mike

Re: Poor desktop responsiveness with background I/O-operations

[James Cloos]

>>>>> "Mike" == Mike Galbraith <efault@gmx.de> writes:

Mike> If someone has a definite good/bad cutoff for the problem they're
Mike> seeing, a git bisect (yeah, takes time) would be a good thing to try.

Yes, I wanted to try that for this, but given how long it takes to
compile a kernel on this box, and how long it takes to get everything
up so that I can test my real-life load, added to the difference in
performance after each day of uptime, I could do at best one kernel
per day; it would take weeks to finish.

That said, I jsut took a look at the git log for my /boot/grub repo
and can confirm that the last really good kernel -- the one I had up
for more than a month for the first time in years -- was pulled 
between 2.6.30-rc6 and 2.6.30-rc7 and installed on 2009/05/17 which,
since I keep .config in git in my compile repo, lets me narrow down
the make oldconfig to 2009/05/17 18:37:24Z.

So, whatever Linus' tree looked like on that date is GOOD.  His last
commit before that was 0f6f49a8cd0163fdb1723ed29f01fc65177108dc.

My next make oldconfig was on 2009/06/25 04:32:03Z, when I started
testing the radeon kms.  (I'm back to non-KMS now.)  There were too
many issues directly caused by KMS itself for me to comment on non-
KMS kernel issues from the kernels I ran with KMS enabled.

My return from KMS as on 2009/08/31; Linus' last commit before I did
that was adda766193ea1cf3137484a9521972d080d0b7af.

To give an idea of how little that narrows things:

  :; git log 0f6f49a8cd..adda766193|egrep -c ^commit
  12094

For those who use the tars, that means 30-rc6 was fast and 31-rc8 slow.

Mike> Git doesn't cause me any woes whatsoever, but my box's primary
Mike> drive isn't a slow laptop drive, ram isn't tight, 

That does make a difference.  Git is *great* when it fits into ram, but
a pull or merge on a clone of Linus' tree takes a quarter gig of VM on
x86-32, which guarantees paging, including to the swap partitions. (The
laptop is old enough to support three platters — including the optical —
so I have my primary swap partition on the second drive to reduce some
of the load to the / /var /boot drive.  The first disk's swap partition
only gets hit when I boot w/o the second drive.  It helps.)

The problem isn't that paging occurs, but rather that now, when
paging occurs, said paging is much slower than it had been.

IOW, the priority of paging vs other disk I/O has changed for the worse,
at some point after 0f6f49a8cd and before adda766193.  But, as I wrote
above, to actually test kernels between those to determine where would
take weeks.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Poor desktop responsiveness with background I/O-operations

[Ulrich Lukas]

Arjan van de Ven wrote:
> > can you try the following two things?
> >
> > echo 4096 > /sys/block/sda/queue/nr_requests
> > for i in `pidof kjournald` ; do ionice -c1 -p $i ; done

No, just tested again. Only a marginal improvement if there is any.

Re: Poor desktop responsiveness with background I/O-operations

[Jiri Kosina]

On Sun, 20 Sep 2009, Ulrich Lukas wrote:

> Test case:
> - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
> 
> 
> How to reproduce:
> - start KDE/GNOME-session
> - open a terminal window and do as a non-root user:
>   dd if=/dev/zero of=/home/john-doe/testfile
>   (or dd if=/home/john-doe/big-testfile of=/dev/null)
> 
> - a real use scenario would be a daily disk-backup or the
>   simple extraction of a tarball containing slightly bigger files
> 
> 
> Observation:
> - The system becomes _really_ slow as described above; unusable for
>   any multimedia tasks.

I guess that switching from CFQ to deadline I/O scheduler improves the 
situation, right?

-- 
Jiri Kosina
SUSE Labs, Novell Inc.

Re: Poor desktop responsiveness with background I/O-operations

[Jan Kara]

On Sun 20-09-09 22:22:20, Jiri Kosina wrote:
> On Sun, 20 Sep 2009, Ulrich Lukas wrote:
> 
> > Test case:
> > - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> > - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
> > 
> > 
> > How to reproduce:
> > - start KDE/GNOME-session
> > - open a terminal window and do as a non-root user:
> >   dd if=/dev/zero of=/home/john-doe/testfile
> >   (or dd if=/home/john-doe/big-testfile of=/dev/null)
> > 
> > - a real use scenario would be a daily disk-backup or the
> >   simple extraction of a tarball containing slightly bigger files
> > 
> > 
> > Observation:
> > - The system becomes _really_ slow as described above; unusable for
> >   any multimedia tasks.
> 
> I guess that switching from CFQ to deadline I/O scheduler improves the 
> situation, right?
  For example on my desktop, switching to deadline slightly improves the
situation but the machine is still mostly unusable while dd is running...
I'll try to debug it some more to see whether it can be somehow helped.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

On Mon, 2009-09-21 at 00:04 +0200, Jan Kara wrote:
> On Sun 20-09-09 22:22:20, Jiri Kosina wrote:
> > On Sun, 20 Sep 2009, Ulrich Lukas wrote:
> > 
> > > Test case:
> > > - 64-bit dual-core PC, SATA harddrive, plenty of free RAM
> > > - vanilla Linux 2.6.31, Kubuntu 9.10 packages, all software 64-bit
> > > 
> > > 
> > > How to reproduce:
> > > - start KDE/GNOME-session
> > > - open a terminal window and do as a non-root user:
> > >   dd if=/dev/zero of=/home/john-doe/testfile
> > >   (or dd if=/home/john-doe/big-testfile of=/dev/null)
> > > 
> > > - a real use scenario would be a daily disk-backup or the
> > >   simple extraction of a tarball containing slightly bigger files
> > > 
> > > 
> > > Observation:
> > > - The system becomes _really_ slow as described above; unusable for
> > >   any multimedia tasks.
> > 
> > I guess that switching from CFQ to deadline I/O scheduler improves the 
> > situation, right?
>   For example on my desktop, switching to deadline slightly improves the
> situation but the machine is still mostly unusable while dd is running...
> I'll try to debug it some more to see whether it can be somehow helped.

Datapoint:

echo 1 > /sys/block/$disk/queue/iosched/quantum seems to make a huge
difference.  Stock is 4.  (wisdom of setting it to 1? no idea)

This...
marge:/root/tmp # ./testo.sh
quantum is 4, setting to 1
dd if=3DMark2000.mkv of=/tmp/3DMark2000.mkv
204529+1 records in
204529+1 records out
104718895 bytes (105 MB) copied, 1.62672 s, 64.4 MB/s
perf sched record -o /tmp/mplayer-c2-c2.data (tmpfs)&
mplayer /tmp/3DMark2000.mkv&
dd if=/dev/zero of=crud-25859&

sleep 180
timing konsole -e exit

 Performance counter stats for 'sh -c konsole -e exit':

     173.615837  task-clock-msecs         #      0.031 CPUs
            431  context-switches         #      0.002 M/sec
             10  CPU-migrations           #      0.000 M/sec
           6314  page-faults              #      0.036 M/sec
      368422027  cycles                   #   2122.053 M/sec
      376283291  instructions             #      1.021 IPC
        5883195  cache-references         #     33.886 M/sec
         444333  cache-misses             #      2.559 M/sec

    5.597628358  seconds time elapsed

...is pretty typical, and a LOT better than the 60-70 seconds I was
seeing, or the minutes I sometimes saw when playing with evolution.

I still occasionally see some largish numbers though, and suspect that
depends on how much cache was evicted.

	-Mike

Re: Poor desktop responsiveness with background I/O-operations

[Arjan van de Ven]

On Mon, 21 Sep 2009 09:25:08 +0200
Mike Galbraith <efault@gmx.de> wrote:
> I still occasionally see some largish numbers though, and suspect that
> depends on how much cache was evicted.

if you run latencytop it'll at least tell you what kind of operating
was going on that hit the delay...

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

On Mon, 2009-09-21 at 09:33 +0200, Arjan van de Ven wrote:
> On Mon, 21 Sep 2009 09:25:08 +0200
> Mike Galbraith <efault@gmx.de> wrote:
> > I still occasionally see some largish numbers though, and suspect that
> > depends on how much cache was evicted.
> 
> if you run latencytop it'll at least tell you what kind of operating
> was going on that hit the delay...

One of the really bad ones for evolution is below.  It was so bad that I
had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before
killing the darn dd.

[ 5112.427012] SysRq : Show Blocked State
[ 5112.430995]   task                        PC stack   pid father
[ 5112.430995] kjournald     D 0000000000000002     0  1230      2 0x00000000
[ 5112.430995]  ffff8800378d7c50 0000000000000046 0000000000000000 ffff8800a5ed53b0
[ 5112.430995]  ffff8800bd2c0000 ffff8800378d7fd8 000000000000dd38 0000000000013480
[ 5112.430995]  0000000000004000 0000000000013480 ffff8800378d7bf0 00001000810e0c0a
[ 5112.430995] Call Trace:
[ 5112.430995]  [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.430995]  [<ffffffff8110df99>] sync_buffer+0x40/0x44
[ 5112.430995]  [<ffffffff812cb373>] __wait_on_bit+0x4c/0x7e
[ 5112.430995]  [<ffffffff8110df59>] ? sync_buffer+0x0/0x44
[ 5112.430995]  [<ffffffff8110df59>] ? sync_buffer+0x0/0x44
[ 5112.430995]  [<ffffffff812cb414>] out_of_line_wait_on_bit+0x6f/0x7c
[ 5112.430995]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.430995]  [<ffffffff8110deff>] __wait_on_buffer+0x24/0x26
[ 5112.430995]  [<ffffffffa0077e91>] journal_commit_transaction+0xca4/0x138b [jbd]
[ 5112.430995]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.430995]  [<ffffffffa007ba02>] kjournald+0x168/0x397 [jbd]
[ 5112.430995]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.430995]  [<ffffffffa007b89a>] ? kjournald+0x0/0x397 [jbd]
[ 5112.430995]  [<ffffffff81065acb>] kthread+0x82/0x8a
[ 5112.625071]  [<ffffffff8100ccca>] child_rip+0xa/0x20
[ 5112.625071]  [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071]  [<ffffffff81065a49>] ? kthread+0x0/0x8a
[ 5112.625071]  [<ffffffff8100ccc0>] ? child_rip+0x0/0x20
[ 5112.625071] flush-8:16    D ffff8800bd2097f0     0  1236      2 0x00000000
[ 5112.625071]  ffff8800bd209790 0000000000000046 0000000000000000 ffffffff810beb58
[ 5112.625071]  ffff8800bd209740 ffff8800bd209fd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 0000000001080001 0000000000000080
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff810beb58>] ? get_page_from_freelist+0x33d/0x4a6
[ 5112.625071]  [<ffffffff81174cb6>] ? cfq_may_queue+0x60/0xe1
[ 5112.625071]  [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.625071]  [<ffffffff81169184>] get_request_wait+0xf3/0x18f
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8115f142>] ? elv_merge+0x37/0x19d
[ 5112.625071]  [<ffffffff8116954d>] __make_request+0x32d/0x45b
[ 5112.625071]  [<ffffffff811678ae>] generic_make_request+0x339/0x38b
[ 5112.625071]  [<ffffffff811679d4>] submit_bio+0xd4/0xdd
[ 5112.625071]  [<ffffffff8110ccfc>] submit_bh+0x106/0x129
[ 5112.625071]  [<ffffffff8110f7a4>] __block_write_full_page+0x1eb/0x2d1
[ 5112.625071]  [<ffffffff8110e197>] ? end_buffer_async_write+0x0/0x13f
[ 5112.625071]  [<ffffffff8110f94f>] block_write_full_page_endio+0xc5/0xca
[ 5112.625071]  [<ffffffff8110f969>] block_write_full_page+0x15/0x17
[ 5112.625071]  [<ffffffffa0095e1f>] ext3_writeback_writepage+0x71/0x142 [ext3]
[ 5112.625071]  [<ffffffff810bf647>] __writepage+0x17/0x34
[ 5112.625071]  [<ffffffff810bfd9f>] write_cache_pages+0x250/0x393
[ 5112.625071]  [<ffffffff810bf630>] ? __writepage+0x0/0x34
[ 5112.625071]  [<ffffffff810bff06>] generic_writepages+0x24/0x2a
[ 5112.625071]  [<ffffffff810bff34>] do_writepages+0x28/0x2a
[ 5112.625071]  [<ffffffff81106b94>] writeback_single_inode+0x181/0x395
[ 5112.625071]  [<ffffffff811077fc>] writeback_inodes_wb+0x341/0x42f
[ 5112.625071]  [<ffffffff81107a2b>] wb_writeback+0x141/0x194
[ 5112.625071]  [<ffffffff81095031>] ? call_rcu_sched+0x15/0x17
[ 5112.625071]  [<ffffffff81095041>] ? call_rcu+0xe/0x10
[ 5112.625071]  [<ffffffff81107bb9>] wb_do_writeback+0x7a/0x16b
[ 5112.625071]  [<ffffffff81107ce9>] bdi_writeback_task+0x3f/0xad
[ 5112.625071]  [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7
[ 5112.625071]  [<ffffffff810ccad3>] bdi_start_fn+0x76/0xd7
[ 5112.625071]  [<ffffffff810cca5d>] ? bdi_start_fn+0x0/0xd7
[ 5112.625071]  [<ffffffff81065acb>] kthread+0x82/0x8a
[ 5112.625071]  [<ffffffff8100ccca>] child_rip+0xa/0x20
[ 5112.625071]  [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071]  [<ffffffff81065a49>] ? kthread+0x0/0x8a
[ 5112.625071]  [<ffffffff8100ccc0>] ? child_rip+0x0/0x20
[ 5112.625071] console-kit-d D ffff8800b788c480     0  2963      1 0x00000000
[ 5112.625071]  ffff8800bc59f6b8 0000000000000082 0000000000000000 ffffffffa0076655
[ 5112.625071]  ffff8800bc59f628 ffff8800bc59ffd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff8800bd2f3c00 0000000000000000
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffffa0076655>] ? do_get_write_access+0x443/0x48b [jbd]
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa0092901>] ext3_new_blocks+0x333/0x669 [ext3]
[ 5112.625071]  [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3]
[ 5112.625071]  [<ffffffff8110c866>] ? alloc_buffer_head+0x1e/0x51
[ 5112.625071]  [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3]
[ 5112.625071]  [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2
[ 5112.625071]  [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffffa007709a>] ? journal_start+0x60/0xca [jbd]
[ 5112.625071]  [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5
[ 5112.625071]  [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277
[ 5112.625071]  [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385
[ 5112.625071]  [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071]  [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071]  [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8117c94f>] ? __up_read+0x9e/0xa7
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071]  [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] dd            D ffff8800b788c480     0  8587   6686 0x00000000
[ 5112.625071]  ffff8800a5e2faa8 0000000000000086 0000000000000000 ffffffff812cab8c
[ 5112.625071]  ffff8800a5e2fbe0 ffff8800a5e2ffd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff8800a5e2fa28 ffffffff810b9b1c
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff812cab8c>] ? thread_return+0x60/0x108
[ 5112.625071]  [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3]
[ 5112.625071]  [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3]
[ 5112.625071]  [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3]
[ 5112.625071]  [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d
[ 5112.625071]  [<ffffffff810fc6af>] file_update_time+0xbe/0x102
[ 5112.625071]  [<ffffffff810baef7>] __generic_file_aio_write+0x267/0x385
[ 5112.625071]  [<ffffffff810b8427>] ? perf_swcounter_ctx_event+0x16c/0x1c6
[ 5112.625071]  [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071]  [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8117ffa0>] ? __clear_user+0x17/0x55
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071]  [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] perf          D ffff8800b788c480     0  8588   6686 0x00000000
[ 5112.625071]  ffff8800a5f115d8 0000000000000082 0000000000000000 000004a57bce3b78
[ 5112.625071]  ffff8800be035f40 ffff8800a5f11fd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff8800a5f11558 ffffffff810b9b1c
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff810b9b1c>] ? find_get_page+0x28/0x89
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa00953f5>] ext3_reserve_inode_write+0x44/0x80 [ext3]
[ 5112.625071]  [<ffffffffa0095460>] ext3_mark_inode_dirty+0x2f/0x4c [ext3]
[ 5112.625071]  [<ffffffffa0095605>] ext3_dirty_inode+0xab/0xc2 [ext3]
[ 5112.625071]  [<ffffffff81107392>] __mark_inode_dirty+0x34/0x15d
[ 5112.625071]  [<ffffffffa0092672>] ext3_new_blocks+0xa4/0x669 [ext3]
[ 5112.625071]  [<ffffffff81110195>] ? __bread+0x13/0x92
[ 5112.625071]  [<ffffffffa00959c4>] ? ext3_get_branch+0x7b/0xee [ext3]
[ 5112.625071]  [<ffffffffa0096654>] ext3_get_blocks_handle+0x3ec/0x92d [ext3]
[ 5112.625071]  [<ffffffffa0096c3a>] ext3_get_block+0xa5/0xe3 [ext3]
[ 5112.625071]  [<ffffffff8110ec0a>] __block_prepare_write+0x1f0/0x3d2
[ 5112.625071]  [<ffffffffa0076e49>] ? start_this_handle+0x37d/0x474 [jbd]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff8110ef7c>] block_write_begin+0x85/0xd5
[ 5112.625071]  [<ffffffffa0098130>] ext3_write_begin+0xf0/0x1e7 [ext3]
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff810ba9e8>] generic_file_buffered_write+0x11f/0x277
[ 5112.625071]  [<ffffffff810bafe1>] __generic_file_aio_write+0x351/0x385
[ 5112.625071]  [<ffffffff8100c669>] ? restore_args+0x0/0x30
[ 5112.625071]  [<ffffffff812cb922>] ? mutex_lock+0x1c/0x3b
[ 5112.625071]  [<ffffffff810bb07b>] generic_file_aio_write+0x66/0xad
[ 5112.625071]  [<ffffffff810e75b4>] do_sync_write+0xec/0x132
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff812cab6a>] ? thread_return+0x3e/0x108
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e8053>] vfs_write+0xb3/0x16f
[ 5112.625071]  [<ffffffff810e81dd>] sys_write+0x4c/0x74
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] evolution.bin D 0000000000000082     0  8638   6627 0x00000000
[ 5112.625071]  ffff880099339c08 0000000000000086 0000000000000000 ffffffff811157a6
[ 5112.625071]  ffffffffa0096b95 ffff880099339fd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffffea0001c5ae08 00000000009ff29b
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff811157a6>] ? mpage_readpages+0x10a/0x120
[ 5112.625071]  [<ffffffffa0096b95>] ? ext3_get_block+0x0/0xe3 [ext3]
[ 5112.625071]  [<ffffffff812cacb4>] io_schedule+0x80/0xc6
[ 5112.625071]  [<ffffffff810b9d22>] sync_page+0x46/0x4a
[ 5112.625071]  [<ffffffff810b9d34>] sync_page_killable+0xe/0x35
[ 5112.625071]  [<ffffffff812cb263>] __wait_on_bit_lock+0x4a/0x92
[ 5112.625071]  [<ffffffff810b9d26>] ? sync_page_killable+0x0/0x35
[ 5112.625071]  [<ffffffff810b9c54>] __lock_page_killable+0x68/0x6f
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffff810bb480>] generic_file_aio_read+0x3be/0x5bb
[ 5112.625071]  [<ffffffff810e76e6>] do_sync_read+0xec/0x132
[ 5112.625071]  [<ffffffff810c176a>] ? lru_cache_add_lru+0x2b/0x2d
[ 5112.625071]  [<ffffffff81065e32>] ? autoremove_wake_function+0x0/0x3d
[ 5112.625071]  [<ffffffff81151f8c>] ? security_file_permission+0x16/0x18
[ 5112.625071]  [<ffffffff810e82b5>] vfs_read+0xb0/0x16c
[ 5112.625071]  [<ffffffff810e843f>] sys_read+0x4c/0x75
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b
[ 5112.625071] hal-acl-tool  D ffff8800b788c480     0  8675   2964 0x00000000
[ 5112.625071]  ffff880037b3db78 0000000000000082 ffff88007af907e0 ffff880062b7f7e0
[ 5112.625071]  ffff880037b3dac8 ffff880037b3dfd8 000000000000dd38 0000000000013480
[ 5112.625071]  0000000000004000 0000000000013480 ffff880037b3db18 ffffffff8110d414
[ 5112.625071] Call Trace:
[ 5112.625071]  [<ffffffff8110d414>] ? __getblk+0x2d/0x2ae
[ 5112.625071]  [<ffffffffa0076490>] do_get_write_access+0x27e/0x48b [jbd]
[ 5112.625071]  [<ffffffff81065e6f>] ? wake_bit_function+0x0/0x2f
[ 5112.625071]  [<ffffffffa00766c4>] journal_get_write_access+0x27/0x38 [jbd]
[ 5112.625071]  [<ffffffffa00a15fb>] __ext3_journal_get_write_access+0x24/0x4d [ext3]
[ 5112.625071]  [<ffffffffa0093ef8>] ext3_new_inode+0x4d4/0x96d [ext3]
[ 5112.625071]  [<ffffffffa009afca>] ext3_create+0x76/0xf3 [ext3]
[ 5112.625071]  [<ffffffff810f2be9>] vfs_create+0xa2/0x117
[ 5112.625071]  [<ffffffff810f50fb>] do_filp_open+0x327/0xa39
[ 5112.625071]  [<ffffffff810f2e69>] ? getname+0x31/0x1b3
[ 5112.625071]  [<ffffffff810e513e>] do_sys_open+0x62/0x109
[ 5112.625071]  [<ffffffff810e5218>] sys_open+0x20/0x22
[ 5112.625071]  [<ffffffff8100bc1b>] system_call_fastpath+0x16/0x1b

Re: Poor desktop responsiveness with background I/O-operations

[Mike Galbraith]

On Mon, 2009-09-21 at 09:41 +0200, Mike Galbraith wrote:
> On Mon, 2009-09-21 at 09:33 +0200, Arjan van de Ven wrote:
> > On Mon, 21 Sep 2009 09:25:08 +0200
> > Mike Galbraith <efault@gmx.de> wrote:
> > > I still occasionally see some largish numbers though, and suspect that
> > > depends on how much cache was evicted.
> > 
> > if you run latencytop it'll at least tell you what kind of operating
> > was going on that hit the delay...
> 
> One of the really bad ones for evolution is below.  It was so bad that I
> had _plenty_ of time to get to a GUI free shell, and poke SysRq-W before
> killing the darn dd.

P.S.  evolution is always journal journal and more journal.

Re: Poor desktop responsiveness with background I/O-operations

[Ulrich Lukas]

Jiri Kosina wrote:
> I guess that switching from CFQ to deadline I/O scheduler improves the 
> situation, right?


Indeed, there is an improvement. But, especially in the case with an
encrypted /home, copying a large file from a fast source means I better
go get another cup of coffee rather than continue working on something
else in the meantime.


BTW, I've tested this with a default install of
OpenSUSE 11.1, and I get basically the same behaviour.

Re: Poor desktop responsiveness with background I/O-operations

[Sanjoy Mahajan]

Arjan van de Ven <arjan@infradead.org> wrote:

> can you try the following two things?
>
> echo 4096 > /sys/block/sda/queue/nr_requests
> for i in `pidof kjournald` ; do ionice -c1 -p $i ; done
>
> to see if they help?

That helped here.  The hardware is a Thinkpad T60 with 1.5GB RAM and
this hard drive:

  ata1.00: ATA-7: Hitachi HTE541616J9SA00, SB4OC7KP, max UDMA/100
  ata1.00: 312581808 sectors, multi 16: LBA48 NCQ (depth 31/32)

When the 'locate' daily cron job runs, or I run 'aptitude
dist-upgrade' (after all packages are downloaded), or I back up /home
to a USB hard drive using rsync, the desktop response badly degrades.

I'm not using a fancy desktop---just twm.  An rxvt can take up to 10
seconds to start up, and I often see emacs repainting its window when it
comes to the foreground.  Using your settings, rxvt takes about 1 second
to start up, which is a large improvement.

(The kernel is 2.6.30 with debian patches)

-Sanjoy