* [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. @ 2012-07-16 1:31 majianpeng 2012-07-16 5:40 ` NeilBrown 0 siblings, 1 reply; 8+ messages in thread From: majianpeng @ 2012-07-16 1:31 UTC (permalink / raw) To: Neil Brown, viro; +Cc: linux-raid, linux-fsdevel In commit e9c7469bb4f502dafc092166201bea1ad5fc0fbf: Tejun Heo introduced "implment REQ_FLUSH/FUA support". But for direct-write-blocks, it maybe for other purpose which like the regular file. And this flag will set STRIPE_PREREAD_ACTIVE which decreaed the change to full write. But this patch remove REQ_SYNC only judging the WRITE_ODIRECT,it will contail regular file.So it maybe not correctly. How can difference odriect_write between regular file or block file? Signed-off-by: Jianpeng Ma <majianpeng@gmail.com> --- drivers/md/raid5.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 04348d7..8d2d4d1 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -4010,6 +4010,9 @@ static void make_request(struct mddev *mddev, struct bio * bi) chunk_aligned_read(mddev,bi)) return; + if (bi->bi_rw & WRITE_ODIRECT) + bi->bi_rw &= ~REQ_SYNC; + logical_sector = bi->bi_sector & ~((sector_t)STRIPE_SECTORS-1); last_sector = bi->bi_sector + (bi->bi_size>>9); bi->bi_next = NULL; -- 1.7.5.4 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. 2012-07-16 1:31 [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect majianpeng @ 2012-07-16 5:40 ` NeilBrown 2012-07-16 5:47 ` majianpeng 2012-07-16 6:42 ` majianpeng 0 siblings, 2 replies; 8+ messages in thread From: NeilBrown @ 2012-07-16 5:40 UTC (permalink / raw) To: majianpeng; +Cc: viro, linux-raid, linux-fsdevel [-- Attachment #1: Type: text/plain, Size: 2601 bytes --] On Mon, 16 Jul 2012 09:31:55 +0800 majianpeng <majianpeng@gmail.com> wrote: > In commit e9c7469bb4f502dafc092166201bea1ad5fc0fbf: > Tejun Heo introduced "implment REQ_FLUSH/FUA support". > But for direct-write-blocks, it maybe for other purpose which like the > regular file. > And this flag will set STRIPE_PREREAD_ACTIVE which decreaed the change > to full write. > > But this patch remove REQ_SYNC only judging the WRITE_ODIRECT,it will > contail regular file.So it maybe not correctly. > How can difference odriect_write between regular file or block file? Hi, I think you are saying the when REQ_SYNC is used with O_DIRECT writes it is having a negative effect on throughput because it allows the stripe to be processed immediately without waiting for more requests to be added to the stripe. Normal 'sync' requests use WRITE_SYNC which includes "REQ_NOIDLE" which means /* don't anticipate more IO after this one */ O_DIRECT request use WRITE_ODIRECT which does not include this flag. So maybe we should simply change raid5 to only set STRIPE_PREREAD_ACTIVE if REQ_NOIDLE is set on the bio. I think this would have the same effect as what you are trying to achieve. Could you please try that and see if it has the desired effect on performance? Thanks, NeilBrown i.e. something like this: diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index d56d74d..2d72a57 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -4178,7 +4178,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) finish_wait(&conf->wait_for_overlap, &w); set_bit(STRIPE_HANDLE, &sh->state); clear_bit(STRIPE_DELAYED, &sh->state); - if ((bi->bi_rw & REQ_SYNC) && + if ((bi->bi_rw & REQ_NOIDLE) && !test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state)) atomic_inc(&conf->preread_active_stripes); release_stripe_plug(mddev, sh); > > Signed-off-by: Jianpeng Ma <majianpeng@gmail.com> > --- > drivers/md/raid5.c | 3 +++ > 1 files changed, 3 insertions(+), 0 deletions(-) > > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c > index 04348d7..8d2d4d1 100644 > --- a/drivers/md/raid5.c > +++ b/drivers/md/raid5.c > @@ -4010,6 +4010,9 @@ static void make_request(struct mddev *mddev, struct bio * bi) > chunk_aligned_read(mddev,bi)) > return; > > + if (bi->bi_rw & WRITE_ODIRECT) > + bi->bi_rw &= ~REQ_SYNC; > + > logical_sector = bi->bi_sector & ~((sector_t)STRIPE_SECTORS-1); > last_sector = bi->bi_sector + (bi->bi_size>>9); > bi->bi_next = NULL; [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: Re: [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. 2012-07-16 5:40 ` NeilBrown @ 2012-07-16 5:47 ` majianpeng 2012-07-16 6:42 ` majianpeng 1 sibling, 0 replies; 8+ messages in thread From: majianpeng @ 2012-07-16 5:47 UTC (permalink / raw) To: Neil Brown; +Cc: viro, linux-raid, linux-fsdevel On 2012-07-16 13:40 NeilBrown <neilb@suse.de> Wrote: >On Mon, 16 Jul 2012 09:31:55 +0800 majianpeng <majianpeng@gmail.com> wrote: > >> In commit e9c7469bb4f502dafc092166201bea1ad5fc0fbf: >> Tejun Heo introduced "implment REQ_FLUSH/FUA support". >> But for direct-write-blocks, it maybe for other purpose which like the >> regular file. >> And this flag will set STRIPE_PREREAD_ACTIVE which decreaed the change >> to full write. >> >> But this patch remove REQ_SYNC only judging the WRITE_ODIRECT,it will >> contail regular file.So it maybe not correctly. >> How can difference odriect_write between regular file or block file? > >Hi, > I think you are saying the when REQ_SYNC is used with O_DIRECT writes it is > having a negative effect on throughput because it allows the stripe to be > processed immediately without waiting for more requests to be added to the > stripe. > > Normal 'sync' requests use WRITE_SYNC which includes "REQ_NOIDLE" which means > /* don't anticipate more IO after this one */ > O_DIRECT request use WRITE_ODIRECT which does not include this flag. > > So maybe we should simply change raid5 to only set STRIPE_PREREAD_ACTIVE if > REQ_NOIDLE is set on the bio. I think this would have the same effect as > what you are trying to achieve. > Yes, thanks. my object is to so.But i didn't notice the REQ_NOIDLE so i used the REQ_FLAG. I'll tested and resend the patch. > Could you please try that and see if it has the desired effect on > performance? > >Thanks, >NeilBrown > >i.e. something like this: > >diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c >index d56d74d..2d72a57 100644 >--- a/drivers/md/raid5.c >+++ b/drivers/md/raid5.c >@@ -4178,7 +4178,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) > finish_wait(&conf->wait_for_overlap, &w); > set_bit(STRIPE_HANDLE, &sh->state); > clear_bit(STRIPE_DELAYED, &sh->state); >- if ((bi->bi_rw & REQ_SYNC) && >+ if ((bi->bi_rw & REQ_NOIDLE) && > !test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state)) > atomic_inc(&conf->preread_active_stripes); > release_stripe_plug(mddev, sh); > > >> >> Signed-off-by: Jianpeng Ma <majianpeng@gmail.com> >> --- >> drivers/md/raid5.c | 3 +++ >> 1 files changed, 3 insertions(+), 0 deletions(-) >> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c >> index 04348d7..8d2d4d1 100644 >> --- a/drivers/md/raid5.c >> +++ b/drivers/md/raid5.c >> @@ -4010,6 +4010,9 @@ static void make_request(struct mddev *mddev, struct bio * bi) >> chunk_aligned_read(mddev,bi)) >> return; >> >> + if (bi->bi_rw & WRITE_ODIRECT) >> + bi->bi_rw &= ~REQ_SYNC; >> + >> logical_sector = bi->bi_sector & ~((sector_t)STRIPE_SECTORS-1); >> last_sector = bi->bi_sector + (bi->bi_size>>9); >> bi->bi_next = NULL; > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Re: [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. 2012-07-16 5:40 ` NeilBrown 2012-07-16 5:47 ` majianpeng @ 2012-07-16 6:42 ` majianpeng 2012-07-16 7:07 ` NeilBrown 1 sibling, 1 reply; 8+ messages in thread From: majianpeng @ 2012-07-16 6:42 UTC (permalink / raw) To: Neil Brown; +Cc: viro, linux-raid, linux-fsdevel On 2012-07-16 13:40 NeilBrown <neilb@suse.de> Wrote: >On Mon, 16 Jul 2012 09:31:55 +0800 majianpeng <majianpeng@gmail.com> wrote: > >> In commit e9c7469bb4f502dafc092166201bea1ad5fc0fbf: >> Tejun Heo introduced "implment REQ_FLUSH/FUA support". >> But for direct-write-blocks, it maybe for other purpose which like the >> regular file. >> And this flag will set STRIPE_PREREAD_ACTIVE which decreaed the change >> to full write. >> >> But this patch remove REQ_SYNC only judging the WRITE_ODIRECT,it will >> contail regular file.So it maybe not correctly. >> How can difference odriect_write between regular file or block file? > >Hi, > I think you are saying the when REQ_SYNC is used with O_DIRECT writes it is > having a negative effect on throughput because it allows the stripe to be > processed immediately without waiting for more requests to be added to the > stripe. > > Normal 'sync' requests use WRITE_SYNC which includes "REQ_NOIDLE" which means > /* don't anticipate more IO after this one */ > O_DIRECT request use WRITE_ODIRECT which does not include this flag. > Using REQ_NOIDEL to difference odirect and sync.Why not using: + if (bi->bi_rw & WRITE_ODIRECT) + bi->bi_rw &= ~REQ_SYNC; The flag WRITE_ODIRECT is only used in odirect-write. > So maybe we should simply change raid5 to only set STRIPE_PREREAD_ACTIVE if > REQ_NOIDLE is set on the bio. I think this would have the same effect as > what you are trying to achieve. > > Could you please try that and see if it has the desired effect on > performance? > I tested and the performance is the same. >Thanks, >NeilBrown > >i.e. something like this: > >diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c >index d56d74d..2d72a57 100644 >--- a/drivers/md/raid5.c >+++ b/drivers/md/raid5.c >@@ -4178,7 +4178,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) > finish_wait(&conf->wait_for_overlap, &w); > set_bit(STRIPE_HANDLE, &sh->state); > clear_bit(STRIPE_DELAYED, &sh->state); >- if ((bi->bi_rw & REQ_SYNC) && >+ if ((bi->bi_rw & REQ_NOIDLE) && > !test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state)) > atomic_inc(&conf->preread_active_stripes); > release_stripe_plug(mddev, sh); > > >> >> Signed-off-by: Jianpeng Ma <majianpeng@gmail.com> >> --- >> drivers/md/raid5.c | 3 +++ >> 1 files changed, 3 insertions(+), 0 deletions(-) >> >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c >> index 04348d7..8d2d4d1 100644 >> --- a/drivers/md/raid5.c >> +++ b/drivers/md/raid5.c >> @@ -4010,6 +4010,9 @@ static void make_request(struct mddev *mddev, struct bio * bi) >> chunk_aligned_read(mddev,bi)) >> return; >> >> + if (bi->bi_rw & WRITE_ODIRECT) >> + bi->bi_rw &= ~REQ_SYNC; >> + >> logical_sector = bi->bi_sector & ~((sector_t)STRIPE_SECTORS-1); >> last_sector = bi->bi_sector + (bi->bi_size>>9); >> bi->bi_next = NULL; > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. 2012-07-16 6:42 ` majianpeng @ 2012-07-16 7:07 ` NeilBrown 2012-07-16 7:11 ` majianpeng 0 siblings, 1 reply; 8+ messages in thread From: NeilBrown @ 2012-07-16 7:07 UTC (permalink / raw) To: majianpeng; +Cc: viro, linux-raid, linux-fsdevel [-- Attachment #1: Type: text/plain, Size: 3456 bytes --] On Mon, 16 Jul 2012 14:42:54 +0800 majianpeng <majianpeng@gmail.com> wrote: > On 2012-07-16 13:40 NeilBrown <neilb@suse.de> Wrote: > >On Mon, 16 Jul 2012 09:31:55 +0800 majianpeng <majianpeng@gmail.com> wrote: > > > >> In commit e9c7469bb4f502dafc092166201bea1ad5fc0fbf: > >> Tejun Heo introduced "implment REQ_FLUSH/FUA support". > >> But for direct-write-blocks, it maybe for other purpose which like the > >> regular file. > >> And this flag will set STRIPE_PREREAD_ACTIVE which decreaed the change > >> to full write. > >> > >> But this patch remove REQ_SYNC only judging the WRITE_ODIRECT,it will > >> contail regular file.So it maybe not correctly. > >> How can difference odriect_write between regular file or block file? > > > >Hi, > > I think you are saying the when REQ_SYNC is used with O_DIRECT writes it is > > having a negative effect on throughput because it allows the stripe to be > > processed immediately without waiting for more requests to be added to the > > stripe. > > > > Normal 'sync' requests use WRITE_SYNC which includes "REQ_NOIDLE" which means > > /* don't anticipate more IO after this one */ > > O_DIRECT request use WRITE_ODIRECT which does not include this flag. > > > Using REQ_NOIDEL to difference odirect and sync.Why not using: > + if (bi->bi_rw & WRITE_ODIRECT) > + bi->bi_rw &= ~REQ_SYNC; Because that code is wrong. WRITE_ODIRECT is not one flag, it is two flags 'or'ed together. So this code does not do what you expect. > > The flag WRITE_ODIRECT is only used in odirect-write. > > > So maybe we should simply change raid5 to only set STRIPE_PREREAD_ACTIVE if > > REQ_NOIDLE is set on the bio. I think this would have the same effect as > > what you are trying to achieve. > > > > Could you please try that and see if it has the desired effect on > > performance? > > > I tested and the performance is the same. "The same" as what? The same are your original patch, or the same as without any patch? NeilBrown > >Thanks, > >NeilBrown > > > >i.e. something like this: > > > >diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c > >index d56d74d..2d72a57 100644 > >--- a/drivers/md/raid5.c > >+++ b/drivers/md/raid5.c > >@@ -4178,7 +4178,7 @@ static void make_request(struct mddev *mddev, struct bio * bi) > > finish_wait(&conf->wait_for_overlap, &w); > > set_bit(STRIPE_HANDLE, &sh->state); > > clear_bit(STRIPE_DELAYED, &sh->state); > >- if ((bi->bi_rw & REQ_SYNC) && > >+ if ((bi->bi_rw & REQ_NOIDLE) && > > !test_and_set_bit(STRIPE_PREREAD_ACTIVE, &sh->state)) > > atomic_inc(&conf->preread_active_stripes); > > release_stripe_plug(mddev, sh); > > > > > >> > >> Signed-off-by: Jianpeng Ma <majianpeng@gmail.com> > >> --- > >> drivers/md/raid5.c | 3 +++ > >> 1 files changed, 3 insertions(+), 0 deletions(-) > >> > >> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c > >> index 04348d7..8d2d4d1 100644 > >> --- a/drivers/md/raid5.c > >> +++ b/drivers/md/raid5.c > >> @@ -4010,6 +4010,9 @@ static void make_request(struct mddev *mddev, struct bio * bi) > >> chunk_aligned_read(mddev,bi)) > >> return; > >> > >> + if (bi->bi_rw & WRITE_ODIRECT) > >> + bi->bi_rw &= ~REQ_SYNC; > >> + > >> logical_sector = bi->bi_sector & ~((sector_t)STRIPE_SECTORS-1); > >> last_sector = bi->bi_sector + (bi->bi_size>>9); > >> bi->bi_next = NULL; > > > > [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Re: [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. 2012-07-16 7:07 ` NeilBrown @ 2012-07-16 7:11 ` majianpeng 2012-07-16 7:30 ` NeilBrown 0 siblings, 1 reply; 8+ messages in thread From: majianpeng @ 2012-07-16 7:11 UTC (permalink / raw) To: Neil Brown; +Cc: viro, linux-raid, linux-fsdevel On 2012-07-16 15:07 NeilBrown <neilb@suse.de> Wrote: >On Mon, 16 Jul 2012 14:42:54 +0800 majianpeng <majianpeng@gmail.com> wrote: > >> On 2012-07-16 13:40 NeilBrown <neilb@suse.de> Wrote: >> >On Mon, 16 Jul 2012 09:31:55 +0800 majianpeng <majianpeng@gmail.com> wrote: >> > [snip] >> > Normal 'sync' requests use WRITE_SYNC which includes "REQ_NOIDLE" which means >> > /* don't anticipate more IO after this one */ >> > O_DIRECT request use WRITE_ODIRECT which does not include this flag. >> > > >> Using REQ_NOIDEL to difference odirect and sync.Why not using: >> + if (bi->bi_rw & WRITE_ODIRECT) >> + bi->bi_rw &= ~REQ_SYNC; > >Because that code is wrong. WRITE_ODIRECT is not one flag, it is two flags >'or'ed together. So this code does not do what you expect. > No, I used those code test and it's ok. The code used & not &&. Maybe I wrong? > >> [snip] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. 2012-07-16 7:11 ` majianpeng @ 2012-07-16 7:30 ` NeilBrown 2012-07-16 8:14 ` majianpeng 0 siblings, 1 reply; 8+ messages in thread From: NeilBrown @ 2012-07-16 7:30 UTC (permalink / raw) To: majianpeng; +Cc: viro, linux-raid, linux-fsdevel [-- Attachment #1: Type: text/plain, Size: 1506 bytes --] On Mon, 16 Jul 2012 15:11:29 +0800 majianpeng <majianpeng@gmail.com> wrote: > On 2012-07-16 15:07 NeilBrown <neilb@suse.de> Wrote: > >On Mon, 16 Jul 2012 14:42:54 +0800 majianpeng <majianpeng@gmail.com> wrote: > > > >> On 2012-07-16 13:40 NeilBrown <neilb@suse.de> Wrote: > >> >On Mon, 16 Jul 2012 09:31:55 +0800 majianpeng <majianpeng@gmail.com> wrote: > >> > > [snip] > >> > Normal 'sync' requests use WRITE_SYNC which includes "REQ_NOIDLE" which means > >> > /* don't anticipate more IO after this one */ > >> > O_DIRECT request use WRITE_ODIRECT which does not include this flag. > >> > > > > >> Using REQ_NOIDEL to difference odirect and sync.Why not using: > >> + if (bi->bi_rw & WRITE_ODIRECT) > >> + bi->bi_rw &= ~REQ_SYNC; > > > >Because that code is wrong. WRITE_ODIRECT is not one flag, it is two flags > >'or'ed together. So this code does not do what you expect. > > > No, I used those code test and it's ok. > The code used & not &&. > Maybe I wrong? Think about it... #define REQ_WRITE (1 << __REQ_WRITE) #define REQ_SYNC (1 << __REQ_SYNC) #define RW_MASK REQ_WRITE #define WRITE RW_MASK #define WRITE_ODIRECT (WRITE | REQ_SYNC) So (bi->bi_rw & WRITE_ODIRECT) will be true if either REQ_WRITE or REQ_SYNC are set in bi_rw So whenever REQ_SYNC is set, your code clears the flag. So your code is functionally identical to bi->bi_rw &= ~REQ_SYNC; NeilBrown [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 828 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Re: [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect. 2012-07-16 7:30 ` NeilBrown @ 2012-07-16 8:14 ` majianpeng 0 siblings, 0 replies; 8+ messages in thread From: majianpeng @ 2012-07-16 8:14 UTC (permalink / raw) To: Neil Brown; +Cc: viro, linux-raid, linux-fsdevel On 2012-07-16 15:30 NeilBrown <neilb@suse.de> Wrote: >On Mon, 16 Jul 2012 15:11:29 +0800 majianpeng <majianpeng@gmail.com> wrote: > >> On 2012-07-16 15:07 NeilBrown <neilb@suse.de> Wrote: >> >On Mon, 16 Jul 2012 14:42:54 +0800 majianpeng <majianpeng@gmail.com> wrote: >> > >> >> On 2012-07-16 13:40 NeilBrown <neilb@suse.de> Wrote: >> >> >On Mon, 16 Jul 2012 09:31:55 +0800 majianpeng <majianpeng@gmail.com> wrote: >> >> > >> [snip] >> >> > Normal 'sync' requests use WRITE_SYNC which includes "REQ_NOIDLE" which means >> >> > /* don't anticipate more IO after this one */ >> >> > O_DIRECT request use WRITE_ODIRECT which does not include this flag. >> >> > >> > >> >> Using REQ_NOIDEL to difference odirect and sync.Why not using: >> >> + if (bi->bi_rw & WRITE_ODIRECT) >> >> + bi->bi_rw &= ~REQ_SYNC; >> > >> >Because that code is wrong. WRITE_ODIRECT is not one flag, it is two flags >> >'or'ed together. So this code does not do what you expect. >> > >> No, I used those code test and it's ok. >> The code used & not &&. >> Maybe I wrong? > >Think about it... > >#define REQ_WRITE (1 << __REQ_WRITE) >#define REQ_SYNC (1 << __REQ_SYNC) > >#define RW_MASK REQ_WRITE >#define WRITE RW_MASK > >#define WRITE_ODIRECT (WRITE | REQ_SYNC) > >So > (bi->bi_rw & WRITE_ODIRECT) > >will be true if either REQ_WRITE or REQ_SYNC are set in bi_rw >So whenever REQ_SYNC is set, your code clears the flag. >So your code is functionally identical to > > bi->bi_rw &= ~REQ_SYNC; > >NeilBrown > > Yes, thanks your time.I maked a stupid mistake. I'll corrected by your suggestion and resend to you. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-07-16 8:14 UTC | newest] Thread overview: 8+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-07-16 1:31 [PATCH 2/2] raid5: For write performance, remove REQ_SYNC when write was odirect majianpeng 2012-07-16 5:40 ` NeilBrown 2012-07-16 5:47 ` majianpeng 2012-07-16 6:42 ` majianpeng 2012-07-16 7:07 ` NeilBrown 2012-07-16 7:11 ` majianpeng 2012-07-16 7:30 ` NeilBrown 2012-07-16 8:14 ` majianpeng
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).