Re: bug (trouble?) report on high mem support

Randy.Dunlap
2002-03-15T23:39:38Z

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2327 bytes --]

Hi-

If someone (Martin or Alan ?) hasn't already told you,
there is a block-highmem patch for 2.4.teens, so if you
can upgrade your kernel to 2.4.19-pre3, for example,
the block-highmem patch is at
http://www.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.19pre3aa2/
file: 00_block-highmem-all-18b-7.gz

Also, as suggested a day or two ago, you could profile the
kernel to see where it is spending time, although I'm not
sure how useful that would be.

A third alternative for you is to apply the attached patch.
I applied it to 2.4.9 (it applies with a little "fuzz"),
but I haven't tested it on 2.4.9, just 2.4.teens.

It counts bounce IOs, both normal IOs and swap IOs.
They can be displayed by printing /proc/stats .
This patch doesn't work with the block-highmem
patch applied -- I'm working on a different patch for that.

This patch also prints (by major:minor) which device(s) are
causing bounce IO. This printing could become excessive
for you, so don't hesitate to disable it (comment it out, or
let me know if you need help with it).

Regards,
~Randy

On Fri, 15 Mar 2002, John Helms wrote:

| Alan,
|
| Ok, how do I go about determining that? The machine
| I have is a brand-spankin' new IBM x-series 350 with
| 4 900MHz Xeon processors. The system bios can
| recognize all of the 16320MB of memory at startup.
| If those patches work, it will save our butts as
| we have a major conversion project that hinges on
| this.
|
| Thanks,
| jwh
|
| >>>>>>>>>>>>>>>>>> Original Message <<<<<<<<<<<<<<<<<<
|
| On 3/15/02, 2:30:22 PM, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote regarding
| Re: bug (trouble?) report on high mem support:
|
|
| > > Here is a top output. We have 16Gb of ram.
| > > I have also tried a 2.4.9-31 enterprise=20
| > > kernel rpm from RedHat with the same=20
| > > results.
|
| > Ok that would make sense. Next question is do you have an I/O controller
| > that can use all the 64bit address space on the PCI bus ?
|
| > What is happening is that you are using a lot of CPU copying buffers down
| > into lower memory to transfer to/from disk - as well probably as that
| > causing a lot of competition for low memory. If your I/O controller can
| hit
| > the full 64bit space there are some rather nice test patches that should
| > completely obliterate the problem.
|
| > Alan

[-- Attachment #2: Type: TEXT/PLAIN, Size: 2629 bytes --]

--- linux/include/linux/kernel_stat.h.org Mon Nov 26 10:19:29 2001
+++ linux/include/linux/kernel_stat.h Thu Dec 20 13:26:50 2001
@@ -26,12 +26,14 @@
unsigned int dk_drive_wblk[DK_MAX_MAJOR][DK_MAX_DISK];
unsigned int pgpgin, pgpgout;
unsigned int pswpin, pswpout;
+ unsigned int bouncein, bounceout;
+ unsigned int bounceswapin, bounceswapout;
#if !defined(CONFIG_ARCH_S390)
unsigned int irqs[NR_CPUS][NR_IRQS];
#endif
- unsigned int ipackets, opackets;
- unsigned int ierrors, oerrors;
- unsigned int collisions;
+/// unsigned int ipackets, opackets;
+/// unsigned int ierrors, oerrors;
+/// unsigned int collisions;
unsigned int context_swtch;
};

--- linux/fs/proc/proc_misc.c.org Tue Nov 20 21:29:09 2001
+++ linux/fs/proc/proc_misc.c Thu Dec 20 13:34:44 2001
@@ -310,6 +310,12 @@
xtime.tv_sec - jif / HZ,
total_forks);

+ len += sprintf(page + len,
+ "bounce io %u %u\n"
+ "bounce swap io %u %u\n",
+ kstat.bouncein, kstat.bounceout,
+ kstat.bounceswapin, kstat.bounceswapout);
+
return proc_calc_metrics(page, start, off, count, eof, len);
}

--- linux/mm/page_io.c.org Mon Nov 19 15:19:42 2001
+++ linux/mm/page_io.c Thu Dec 20 15:59:41 2001
@@ -10,6 +10,7 @@
* Always use brw_page, life becomes simpler. 12 May 1998 Eric Biederman
*/

+#include <linux/config.h>
#include <linux/mm.h>
#include <linux/kernel_stat.h>
#include <linux/swap.h>
@@ -68,6 +69,13 @@
dev = swapf->i_dev;
} else {
return 0;
+ }
+
+ if (PageHighMem(page)) {
+ if (rw == WRITE)
+ kstat.bounceswapout++;
+ else
+ kstat.bounceswapin++;
}

/* block_size == PAGE_SIZE/zones_used */
--- linux/drivers/block/ll_rw_blk.c.org Mon Oct 29 12:11:17 2001
+++ linux/drivers/block/ll_rw_blk.c Thu Dec 20 17:45:19 2001
@@ -936,6 +936,7 @@
} while (q->make_request_fn(q, rw, bh));
}

+static int bmsg_count = 0;

/**
* submit_bh: submit a buffer_head to the block device later for I/O
@@ -953,6 +954,7 @@
void submit_bh(int rw, struct buffer_head * bh)
{
int count = bh->b_size >> 9;
+ int bounce = PageHighMem(bh->b_page);

if (!test_bit(BH_Lock, &bh->b_state))
BUG();
@@ -971,10 +973,19 @@
switch (rw) {
case WRITE:
kstat.pgpgout += count;
+ if (bounce) kstat.bounceout += count;
break;
default:
kstat.pgpgin += count;
+ if (bounce) kstat.bouncein += count;
break;
+ }
+ if (bounce) {
+ bmsg_count++;
+ if ((bmsg_count % 1000) == 1)
+ printk ("bounce io (%c) for %d:%d\n",
+ (rw == WRITE) ? 'W' : 'R',
+ MAJOR(bh->b_rdev), MINOR(bh->b_rdev));
}
}