2.5.47-mm3 IO rate question

2.5.47-mm3 IO rate question

[Badari Pulavarty]

Hi,

I am using 2.5.47-mm3 and qlogic driver qla2x00src-v6.03.00b8.
I have 2 qlogic controllers connected to 2 trays (10 disks/tray).

I do 256K IO on raw devices using "dd".

dd of=/dev/null bs=256k count=2000 if=/dev/raw/raw1 &
dd of=/dev/null bs=256k count=2000 if=/dev/raw/raw2 &
dd of=/dev/null bs=256k count=2000 if=/dev/raw/raw3 &
...
dd of=/dev/null bs=256k count=2000 if=/dev/raw/raw20 &

IO throughput changes just by adjusting SG_SEGMENTS (from 32 to 64).
I was wondering why such a significant difference. (20MB/sec).

I used to get 194-198MB/sec with 2.5.37 (with DIO code sending
(fixed) 64K requests - qlogic max_sector=512, SG_SEGMETS=32).

Any thoughts ?

Thanks,
Badari

max_sectors = 512
SG_SEGMENTS = 64
[root]# vmstat 5
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0 20  0      0 3642680   8832  42836    0    0 194765    15 1762  2183  0 100  0
 0 20  0      0 3642680   8836  42836    0    0 194611     3 1761  2122  0 100  0
 0 20  0      0 3642680   8840  42836    0    0 194509     3 1774  2150 10 81  8
 0 20  0      0 3642680   8844  42836    0    0 194816     3 1762  2132  2 98  0
 0 20  0      0 3642680   8848  42836    0    0 194765     3 1764  2177  2 98  0


max_sectors = 512
SG_SEGMENTS = 32
[root]# vmstat 5
   procs                      memory      swap          io     system      cpu
 r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
 0 20  0      0 3652472   7752  34464    0    0 170755    15 2354  3477  2 98  0
 0 20  0      0 3652472   7756  34464    0    0 172771     3 2363  3512  2 98  0
 0 20  0      0 3652472   7760  34464    0    0 174031     3 2372  3576  0 100  0
 0 20  0      0 3652408   7764  34464    0    0 173074     7 2384  3538 10 87  3
 0 20  0      0 3652408   7768  34464    0    0 172670     3 2364  3502  2 98  0

Re: 2.5.47-mm3 IO rate question

[Andrew Morton]

Badari Pulavarty wrote:
> 
> ...
> max_sectors = 512
> SG_SEGMENTS = 32
> [root]# vmstat 5
>    procs                      memory      swap          io     system      cpu
>  r  b  w   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
>  0 20  0      0 3652472   7752  34464    0    0 170755    15 2354  3477  2 98  0
>  0 20  0      0 3652472   7756  34464    0    0 172771     3 2363  3512  2 98  0
>  0 20  0      0 3652472   7760  34464    0    0 174031     3 2372  3576  0 100  0
>  0 20  0      0 3652408   7764  34464    0    0 173074     7 2384  3538 10 87  3
>  0 20  0      0 3652408   7768  34464    0    0 172670     3 2364  3502  2 98  0

I don't know what happened to your bandwidth, but that context switch rate
is excessive.  qlogic's small BIOs are hurting.

We don't actually need to perform a wakeup-per-BIO in there.  It is sufficient
to deliver a single wakeup on the very last BIO.

This should drop your CPU load a bit.  (vmstat numbers might not change - the
statistical process accounting may not be very accurate for this sort of 
thing.  Use cyclesoak: http://www.zip.com.au/~akpm/linux/#zc)


 fs/direct-io.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

--- 25/fs/direct-io.c~dio-reduce-context-switch-rate	Sat Nov 16 16:55:25 2002
+++ 25-akpm/fs/direct-io.c	Sat Nov 16 16:55:25 2002
@@ -105,7 +105,8 @@ struct dio {
 	int page_errors;		/* errno from get_user_pages() */
 
 	/* BIO completion state */
-	atomic_t bio_count;		/* nr bios in flight */
+	atomic_t bio_count;		/* nr bios to be completed */
+	atomic_t bios_in_flight;	/* nr bios in flight */
 	spinlock_t bio_list_lock;	/* protects bio_list */
 	struct bio *bio_list;		/* singly linked via bi_private */
 	struct task_struct *waiter;	/* waiting task (NULL if none) */
@@ -238,7 +239,8 @@ static int dio_bio_end_io(struct bio *bi
 	spin_lock_irqsave(&dio->bio_list_lock, flags);
 	bio->bi_private = dio->bio_list;
 	dio->bio_list = bio;
-	if (dio->waiter)
+	atomic_dec(&dio->bios_in_flight);
+	if (dio->waiter && atomic_read(&dio->bios_in_flight) == 0)
 		wake_up_process(dio->waiter);
 	spin_unlock_irqrestore(&dio->bio_list_lock, flags);
 	return 0;
@@ -271,6 +273,7 @@ static void dio_bio_submit(struct dio *d
 
 	bio->bi_private = dio;
 	atomic_inc(&dio->bio_count);
+	atomic_inc(&dio->bios_in_flight);
 	submit_bio(dio->rw, bio);
 
 	dio->bio = NULL;
@@ -852,6 +855,7 @@ direct_io_worker(int rw, struct kiocb *i
 	 * still submittion BIOs.
 	 */
 	atomic_set(&dio->bio_count, 1);
+	atomic_set(&dio->bios_in_flight, 0);
 	spin_lock_init(&dio->bio_list_lock);
 	dio->bio_list = NULL;
 	dio->waiter = NULL;

_