* Bugreport mdadm-2.0-devel-1
@ 2005-07-06 9:13 Maxim Kozover
2005-07-07 0:10 ` Neil Brown
0 siblings, 1 reply; 7+ messages in thread
From: Maxim Kozover @ 2005-07-06 9:13 UTC (permalink / raw)
To: linux-raid
Hi!
I'm getting problems with raid1 creation using new superblock format.
Kernel used is 2.6.12.2 (with drivers/md and include/linux/raid taken from
2.6.13-rc1 to have a support for bitmaps and bug correction of md
failing when starting resync), all compiled together.
For mdadm I tried both mdadm-2.0-devel-1 and mdadm-2.0-devel-1a.
Compiles only if -Werror is removed from the Makefile, otherwise
/usr/include/asm/byteorder.h:6:2: #warning using private kernel header; include <endian.h> instead!
raid1 creation with superblock 0.90 goes OK.
However, when I try using superflock version 1, I get the following from
mdadm-2.0-devel-1a:
# mdadm -C -e 1 /dev/md2 -l 1 -n 2 /dev/sdb /dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=1 devices=2 ctime=Tue Jul 5 18:55:26 2005
mdadm: /dev/sdc appears to be part of a raid array:
level=1 devices=2 ctime=Tue Jul 5 18:55:26 2005
Continue creating array? y
mdadm: internal error - sb_offset is wrong
Aborted
(Note that md_p.h already has md_p.h: __u8 pad1[128-100]; /* set
to 0 when written */)
and the following from mdadm-2.0-devel-1:
# mdadm -C -e 1 /dev/md2 -l 1 -n 2 /dev/sdb /dev/sdc
mdadm: /dev/sdb appears to be part of a raid array:
level=1 devices=2 ctime=Tue Jul 5 18:55:26 2005
mdadm: /dev/sdc appears to be part of a raid array:
level=1 devices=2 ctime=Tue Jul 5 18:55:26 2005
Continue creating array? y
VERS = 9002
mdadm: ADD_NEW_DISK for /dev/sdb failed: Device or resource busy
Also I think --zero-superblock doesn't do the job since I tried it,
but the superblock still remains.
Please advise.
Thanks,
Maxim.
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Bugreport mdadm-2.0-devel-1 2005-07-06 9:13 Bugreport mdadm-2.0-devel-1 Maxim Kozover @ 2005-07-07 0:10 ` Neil Brown 2005-07-07 1:30 ` Re[2]: " Maxim Kozover 0 siblings, 1 reply; 7+ messages in thread From: Neil Brown @ 2005-07-07 0:10 UTC (permalink / raw) To: Maxim Kozover; +Cc: linux-raid On Wednesday July 6, maximkoz@netvision.net.il wrote: > Hi! > I'm getting problems with raid1 creation using new superblock format. > Kernel used is 2.6.12.2 (with drivers/md and include/linux/raid taken from > 2.6.13-rc1 to have a support for bitmaps and bug correction of md > failing when starting resync), all compiled together. > > For mdadm I tried both mdadm-2.0-devel-1 and mdadm-2.0-devel-1a. > Compiles only if -Werror is removed from the Makefile, otherwise > /usr/include/asm/byteorder.h:6:2: #warning using private kernel header; include <endian.h> instead! Hmmm... I don't get that. I must have different header files. And my <endian.h> doesn't provide the needed functionality :-( Anyway, try http://www.cse.unsw.edu.au/~neilb/source/mdadm/mdadm-2.0-devel-2.tgz (and remove -Werror). It works for me with the command line that you give. NeilBrown ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re[2]: Bugreport mdadm-2.0-devel-1 2005-07-07 0:10 ` Neil Brown @ 2005-07-07 1:30 ` Maxim Kozover 2005-07-09 1:33 ` Neil Brown 0 siblings, 1 reply; 7+ messages in thread From: Maxim Kozover @ 2005-07-07 1:30 UTC (permalink / raw) To: linux-raid Hi Neil! Thanks much for your help, array creation using devel-2 just works, however, the array can't be assembled again after it's stopped:( # mdadm --zero-superblock -e 1 /dev/sdb # mdadm --zero-superblock -e 1 /dev/sdc # /bin/rm -rf /bitmap # mdadm -C /dev/md0 -e 1 --bitmap /bitmap -l 1 -n 2 /dev/sdb /dev/sdc mdadm: array /dev/md0 started. # more /proc/mdstat Personalities : [raid1] md0 : active raid1 sdc[1] sdb[0] 33554424 blocks [2/2] [UU] [>....................] resync = 0.6% (206400/33554424) finish=16.1min s peed=34400K/sec bitmap: 4096/4096 pages [16384KB], 4KB chunk, file: /bitmap unused devices: <none> # mdadm -S /dev/md0 # more /proc/mdstat Personalities : [raid1] unused devices: <none> # mdadm -A /dev/md0 -e 1 --bitmap /bitmap /dev/sdb /dev/sdc mdadm: device 1 in /dev/md0 has wrong state in superblock, but /dev/sdc seems ok mdadm: failed to add /dev/sdc to /dev/md0: Invalid argument mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument mdadm: /dev/md0 assembled from 0 drives - not enough to start the array. (even if you stop the array after resync completes, it doesn't matter as expected). Please advise. BTW, my endian.h (and the files it includes) also doesn't provide various conversion functions, although byteorder.h advises to use it. Thanks, Maxim. Thursday, July 7, 2005, 3:10:13 AM, you wrote: NB> On Wednesday July 6, maximkoz@netvision.net.il wrote: >> Hi! >> I'm getting problems with raid1 creation using new superblock format. >> Kernel used is 2.6.12.2 (with drivers/md and include/linux/raid taken from >> 2.6.13-rc1 to have a support for bitmaps and bug correction of md >> failing when starting resync), all compiled together. >> >> For mdadm I tried both mdadm-2.0-devel-1 and mdadm-2.0-devel-1a. >> Compiles only if -Werror is removed from the Makefile, otherwise >> /usr/include/asm/byteorder.h:6:2: #warning using private kernel >> header; include <endian.h> instead! NB> Hmmm... I don't get that. I must have different header files. And my NB> <endian.h> doesn't provide the needed functionality :-( NB> Anyway, try NB> NB> http://www.cse.unsw.edu.au/~neilb/source/mdadm/mdadm-2.0-devel-2.tgz NB> (and remove -Werror). It works for me with the command line that you NB> give. NB> NeilBrown ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Re[2]: Bugreport mdadm-2.0-devel-1 2005-07-07 1:30 ` Re[2]: " Maxim Kozover @ 2005-07-09 1:33 ` Neil Brown 2005-07-15 1:54 ` Neil Brown 0 siblings, 1 reply; 7+ messages in thread From: Neil Brown @ 2005-07-09 1:33 UTC (permalink / raw) To: Maxim Kozover; +Cc: linux-raid On Thursday July 7, maximkoz@netvision.net.il wrote: > Hi Neil! > Thanks much for your help, array creation using devel-2 just works, > however, the array can't be assembled again after it's stopped:( Hmm, yeh, nor it can :-( I'm not sure when I'll have time to look at this (I'm on leave at the moment with family visiting and such) but I'll definitely get back to you by Thursday if not before. Thanks, NeilBrown > > # mdadm --zero-superblock -e 1 /dev/sdb > # mdadm --zero-superblock -e 1 /dev/sdc > # /bin/rm -rf /bitmap > # mdadm -C /dev/md0 -e 1 --bitmap /bitmap -l 1 -n 2 /dev/sdb /dev/sdc > mdadm: array /dev/md0 started. > # more /proc/mdstat > Personalities : [raid1] > md0 : active raid1 sdc[1] sdb[0] > 33554424 blocks [2/2] [UU] > [>....................] resync = 0.6% (206400/33554424) finish=16.1min s > peed=34400K/sec > bitmap: 4096/4096 pages [16384KB], 4KB chunk, file: /bitmap > > unused devices: <none> > # mdadm -S /dev/md0 > # more /proc/mdstat > Personalities : [raid1] > unused devices: <none> > # mdadm -A /dev/md0 -e 1 --bitmap /bitmap /dev/sdb /dev/sdc > mdadm: device 1 in /dev/md0 has wrong state in superblock, but /dev/sdc seems ok > mdadm: failed to add /dev/sdc to /dev/md0: Invalid argument > mdadm: failed to add /dev/sdb to /dev/md0: Invalid argument > mdadm: /dev/md0 assembled from 0 drives - not enough to start the array. > (even if you stop the array after resync completes, it doesn't matter > as expected). ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Re[2]: Bugreport mdadm-2.0-devel-1 2005-07-09 1:33 ` Neil Brown @ 2005-07-15 1:54 ` Neil Brown 2005-07-18 21:08 ` Maxim Kozover 0 siblings, 1 reply; 7+ messages in thread From: Neil Brown @ 2005-07-15 1:54 UTC (permalink / raw) To: Maxim Kozover, linux-raid On Saturday July 9, neilb@cse.unsw.edu.au wrote: > On Thursday July 7, maximkoz@netvision.net.il wrote: > > Hi Neil! > > Thanks much for your help, array creation using devel-2 just works, > > however, the array can't be assembled again after it's stopped:( > > Hmm, yeh, nor it can :-( > > I'm not sure when I'll have time to look at this (I'm on leave at the > moment with family visiting and such) but I'll definitely get back to > you by Thursday if not before. Sorry for the delay. The following patch against -devel2 should fix these problems if (when?) you get more, please let me know. NeilBrown Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au> ### Diffstat output ./Assemble.c | 13 ++++++++++++- ./Query.c | 33 +++++++++++++++++++-------------- ./mdadm.h | 2 +- ./super0.c | 1 + ./super1.c | 4 ++-- 5 files changed, 35 insertions(+), 18 deletions(-) diff ./Assemble.c~current~ ./Assemble.c --- ./Assemble.c~current~ 2005-07-15 10:13:04.000000000 +1000 +++ ./Assemble.c 2005-07-15 10:37:59.000000000 +1000 @@ -473,6 +473,7 @@ int Assemble(struct supertype *st, char if (!devices[j].uptodate) continue; info.disk.number = i; + info.disk.raid_disk = i; info.disk.state = desired_state; if (devices[j].uptodate && @@ -526,7 +527,17 @@ int Assemble(struct supertype *st, char /* Almost ready to actually *do* something */ if (!old_linux) { - if (ioctl(mdfd, SET_ARRAY_INFO, NULL) != 0) { + int rv; + if ((vers % 100) >= 1) { /* can use different versions */ + mdu_array_info_t inf; + memset(&inf, 0, sizeof(inf)); + inf.major_version = st->ss->major; + inf.minor_version = st->minor_version; + rv = ioctl(mdfd, SET_ARRAY_INFO, &inf); + } else + rv = ioctl(mdfd, SET_ARRAY_INFO, NULL); + + if (rv) { fprintf(stderr, Name ": SET_ARRAY_INFO failed for %s: %s\n", mddev, strerror(errno)); return 1; diff ./Query.c~current~ ./Query.c --- ./Query.c~current~ 2005-07-07 09:19:53.000000000 +1000 +++ ./Query.c 2005-07-15 11:38:18.000000000 +1000 @@ -105,26 +105,31 @@ int Query(char *dev) if (superror == 0) { /* array might be active... */ st->ss->getinfo_super(&info, super); - mddev = get_md_name(info.array.md_minor); - disc.number = info.disk.number; - activity = "undetected"; - if (mddev && (fd = open(mddev, O_RDONLY))>=0) { - if (md_get_version(fd) >= 9000 && - ioctl(fd, GET_ARRAY_INFO, &array)>= 0) { - if (ioctl(fd, GET_DISK_INFO, &disc) >= 0 && - makedev((unsigned)disc.major,(unsigned)disc.minor) == stb.st_rdev) - activity = "active"; - else - activity = "mismatch"; + if (st->ss->major == 0) { + mddev = get_md_name(info.array.md_minor); + disc.number = info.disk.number; + activity = "undetected"; + if (mddev && (fd = open(mddev, O_RDONLY))>=0) { + if (md_get_version(fd) >= 9000 && + ioctl(fd, GET_ARRAY_INFO, &array)>= 0) { + if (ioctl(fd, GET_DISK_INFO, &disc) >= 0 && + makedev((unsigned)disc.major,(unsigned)disc.minor) == stb.st_rdev) + activity = "active"; + else + activity = "mismatch"; + } + close(fd); } - close(fd); + } else { + activity = "unknown"; + mddev = "array"; } - printf("%s: device %d in %d device %s %s md%d. Use mdadm --examine for more detail.\n", + printf("%s: device %d in %d device %s %s %s. Use mdadm --examine for more detail.\n", dev, info.disk.number, info.array.raid_disks, activity, map_num(pers, info.array.level), - info.array.md_minor); + mddev); } return 0; } diff ./mdadm.h~current~ ./mdadm.h --- ./mdadm.h~current~ 2005-07-07 09:19:53.000000000 +1000 +++ ./mdadm.h 2005-07-15 10:15:51.000000000 +1000 @@ -73,7 +73,7 @@ struct mdinfo { mdu_array_info_t array; mdu_disk_info_t disk; __u64 events; - unsigned int uuid[4]; + int uuid[4]; }; #define Name "mdadm" diff ./super0.c~current~ ./super0.c --- ./super0.c~current~ 2005-07-07 09:19:53.000000000 +1000 +++ ./super0.c 2005-07-15 11:27:12.000000000 +1000 @@ -205,6 +205,7 @@ static void getinfo_super0(struct mdinfo info->disk.major = sb->this_disk.major; info->disk.minor = sb->this_disk.minor; info->disk.raid_disk = sb->this_disk.raid_disk; + info->disk.number = sb->this_disk.number; info->events = md_event(sb); diff ./super1.c~current~ ./super1.c --- ./super1.c~current~ 2005-07-07 09:19:53.000000000 +1000 +++ ./super1.c 2005-07-15 11:25:04.000000000 +1000 @@ -278,7 +278,7 @@ static void getinfo_super1(struct mdinfo info->disk.major = 0; info->disk.minor = 0; - + info->disk.number = __le32_to_cpu(sb->dev_number); if (__le32_to_cpu(sb->dev_number) >= __le32_to_cpu(sb->max_dev) || __le32_to_cpu(sb->max_dev) > 512) role = 0xfffe; @@ -303,7 +303,7 @@ static void getinfo_super1(struct mdinfo for (i=0; i< __le32_to_cpu(sb->max_dev); i++) { role = __le16_to_cpu(sb->dev_roles[i]); - if (role == 0xFFFF || role < info->array.raid_disks) + if (/*role == 0xFFFF || */role < info->array.raid_disks) working++; } ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Re: Re[2]: Bugreport mdadm-2.0-devel-1 2005-07-15 1:54 ` Neil Brown @ 2005-07-18 21:08 ` Maxim Kozover 2005-07-19 11:49 ` Maxim Kozover 0 siblings, 1 reply; 7+ messages in thread From: Maxim Kozover @ 2005-07-18 21:08 UTC (permalink / raw) To: linux-raid Thanks much, Neil, will try. Thanks, Maxim. NB> On Saturday July 9, neilb@cse.unsw.edu.au wrote: >> On Thursday July 7, maximkoz@netvision.net.il wrote: >> > Hi Neil! >> > Thanks much for your help, array creation using devel-2 just works, >> > however, the array can't be assembled again after it's stopped:( >> >> Hmm, yeh, nor it can :-( >> >> I'm not sure when I'll have time to look at this (I'm on leave at the >> moment with family visiting and such) but I'll definitely get back to >> you by Thursday if not before. NB> Sorry for the delay. NB> The following patch against -devel2 should fix these problems if NB> (when?) you get more, please let me know. NB> NeilBrown NB> Signed-off-by: Neil Brown <neilb@cse.unsw.edu.au> NB> ### Diffstat output NB> ./Assemble.c | 13 ++++++++++++- NB> ./Query.c | 33 +++++++++++++++++++-------------- NB> ./mdadm.h | 2 +- NB> ./super0.c | 1 + NB> ./super1.c | 4 ++-- NB> 5 files changed, 35 insertions(+), 18 deletions(-) NB> diff ./Assemble.c~current~ ./Assemble.c NB> --- ./Assemble.c~current~ 2005-07-15 10:13:04.000000000 +1000 NB> +++ ./Assemble.c 2005-07-15 10:37:59.000000000 +1000 NB> @@ -473,6 +473,7 @@ int Assemble(struct supertype *st, char NB> if (!devices[j].uptodate) NB> continue; NB> info.disk.number = i; NB> + info.disk.raid_disk = i; NB> info.disk.state = desired_state; NB> if (devices[j].uptodate && NB> @@ -526,7 +527,17 @@ int Assemble(struct supertype *st, char NB> /* Almost ready to actually *do* something */ NB> if (!old_linux) { NB> - if (ioctl(mdfd, SET_ARRAY_INFO, NULL) != 0) { NB> + int rv; NB> + if ((vers % 100) >= 1) { /* can use different versions */ NB> + mdu_array_info_t inf; NB> + memset(&inf, 0, sizeof(inf)); NB> + inf.major_version = st->ss->major; NB> + inf.minor_version = st->minor_version; NB> + rv = ioctl(mdfd, SET_ARRAY_INFO, &inf); NB> + } else NB> + rv = ioctl(mdfd, SET_ARRAY_INFO, NULL); NB> + NB> + if (rv) { NB> fprintf(stderr, Name ": NB> SET_ARRAY_INFO failed for %s: %s\n", NB> mddev, strerror(errno)); NB> return 1; NB> diff ./Query.c~current~ ./Query.c NB> --- ./Query.c~current~ 2005-07-07 09:19:53.000000000 +1000 NB> +++ ./Query.c 2005-07-15 11:38:18.000000000 +1000 NB> @@ -105,26 +105,31 @@ int Query(char *dev) NB> if (superror == 0) { NB> /* array might be active... */ NB> st->ss->getinfo_super(&info, super); NB> - mddev = get_md_name(info.array.md_minor); NB> - disc.number = info.disk.number; NB> - activity = "undetected"; NB> - if (mddev && (fd = open(mddev, O_RDONLY))>=0) { NB> - if (md_get_version(fd) >= 9000 && NB> - ioctl(fd, GET_ARRAY_INFO, &array)>= 0) { NB> - if (ioctl(fd, GET_DISK_INFO, &disc) >= 0 && NB> - NB> makedev((unsigned)disc.major,(unsigned)disc.minor) == stb.st_rdev) NB> - activity = "active"; NB> - else NB> - activity = "mismatch"; NB> + if (st->ss->major == 0) { NB> + mddev = get_md_name(info.array.md_minor); NB> + disc.number = info.disk.number; NB> + activity = "undetected"; NB> + if (mddev && (fd = open(mddev, O_RDONLY))>=0) { NB> + if (md_get_version(fd) >= 9000 && NB> + ioctl(fd, GET_ARRAY_INFO, &array)>= 0) { NB> + if (ioctl(fd, GET_DISK_INFO, &disc) >= 0 && NB> + NB> makedev((unsigned)disc.major,(unsigned)disc.minor) == stb.st_rdev) NB> + activity = "active"; NB> + else NB> + activity = "mismatch"; NB> + } NB> + close(fd); NB> } NB> - close(fd); NB> + } else { NB> + activity = "unknown"; NB> + mddev = "array"; NB> } NB> - printf("%s: device %d in %d device %s %s NB> md%d. Use mdadm --examine for more detail.\n", NB> + printf("%s: device %d in %d device %s %s %s. NB> Use mdadm --examine for more detail.\n", NB> dev, NB> info.disk.number, info.array.raid_disks, NB> activity, NB> map_num(pers, info.array.level), NB> - info.array.md_minor); NB> + mddev); NB> } NB> return 0; NB> } NB> diff ./mdadm.h~current~ ./mdadm.h NB> --- ./mdadm.h~current~ 2005-07-07 09:19:53.000000000 +1000 NB> +++ ./mdadm.h 2005-07-15 10:15:51.000000000 +1000 NB> @@ -73,7 +73,7 @@ struct mdinfo { NB> mdu_array_info_t array; NB> mdu_disk_info_t disk; NB> __u64 events; NB> - unsigned int uuid[4]; NB> + int uuid[4]; NB> }; NB> #define Name "mdadm" NB> diff ./super0.c~current~ ./super0.c NB> --- ./super0.c~current~ 2005-07-07 09:19:53.000000000 +1000 NB> +++ ./super0.c 2005-07-15 11:27:12.000000000 +1000 NB> @@ -205,6 +205,7 @@ static void getinfo_super0(struct mdinfo NB> info->disk.major = sb->this_disk.major; NB> info->disk.minor = sb->this_disk.minor; NB> info->disk.raid_disk = sb->this_disk.raid_disk; + info->>disk.number = sb->this_disk.number; NB> info->events = md_event(sb); NB> diff ./super1.c~current~ ./super1.c NB> --- ./super1.c~current~ 2005-07-07 09:19:53.000000000 +1000 NB> +++ ./super1.c 2005-07-15 11:25:04.000000000 +1000 NB> @@ -278,7 +278,7 @@ static void getinfo_super1(struct mdinfo NB> info->disk.major = 0; NB> info->disk.minor = 0; NB> - + info->>disk.number = __le32_to_cpu(sb->dev_number); NB> if (__le32_to_cpu(sb->dev_number) >= __le32_to_cpu(sb->max_dev) || NB> __le32_to_cpu(sb->max_dev) > 512) NB> role = 0xfffe; NB> @@ -303,7 +303,7 @@ static void getinfo_super1(struct mdinfo NB> for (i=0; i< __le32_to_cpu(sb->max_dev); i++) { NB> role = __le16_to_cpu(sb->dev_roles[i]); NB> - if (role == 0xFFFF || role < info->array.raid_disks) NB> + if (/*role == 0xFFFF || */role < info->array.raid_disks) NB> working++; NB> } ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Re: Re: Re[2]: Bugreport mdadm-2.0-devel-1 2005-07-18 21:08 ` Maxim Kozover @ 2005-07-19 11:49 ` Maxim Kozover 0 siblings, 0 replies; 7+ messages in thread From: Maxim Kozover @ 2005-07-19 11:49 UTC (permalink / raw) To: linux-raid Hi Neil! Thanks much for the patch, the array can now be started again. However, I observe some strange behaviour with raid1 intelligent resync. I'm not sure if your new patch (md/raid1: clear bitmap when fullsync completes) http://marc.theaimsgroup.com/?l=linux-raid&m=112140143401678&w=2 addresses this strangeness, cause I still didn't apply it. When we create a raid1 array with bitmap it starts a full resync. Let's wait a couple of minutes, so the resync will proceed to, say, 50%. Then stop the array and assemble it again with the same bitmap. We expect it should resync only the remaining part, but it does a full resync again instead. Please see a log below. What do you think? Thanks, Maxim Kozover. # mdadm -C /dev/md0 --bitmap /bitmap -e 1 -l 1 -n 2 /dev/sdb /dev/sdc mdadm: array /dev/md0 started. # more /proc/mdstat Personalities : [raid1] md0 : active raid1 sdc[1] sdb[0] 16777208 blocks [2/2] [UU] [===========>.........] resync = 55.6% (9339328/16777208) finish=4.2min speed=29091K/sec bitmap: 2048/2048 pages [8192KB], 4KB chunk, file: /bitmap unused devices: <none> # mdadm -S /dev/md0 # mdadm -A /dev/md0 --bitmap /bitmap /dev/sdb /dev/sdc mdadm: /dev/md0 has been started with 2 drives. # more /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb[0] sdc[1] 16777208 blocks [2/2] [UU] [>....................] resync = 0.3% (53248/16777208) finish=5.2min spe ed=53248K/sec bitmap: 2048/2048 pages [8192KB], 4KB chunk, file: /bitmap unused devices: <none> MK> Thanks much, Neil, will try. MK> Thanks, MK> Maxim. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2005-07-19 11:49 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2005-07-06 9:13 Bugreport mdadm-2.0-devel-1 Maxim Kozover 2005-07-07 0:10 ` Neil Brown 2005-07-07 1:30 ` Re[2]: " Maxim Kozover 2005-07-09 1:33 ` Neil Brown 2005-07-15 1:54 ` Neil Brown 2005-07-18 21:08 ` Maxim Kozover 2005-07-19 11:49 ` Maxim Kozover
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).