* Aacraid - terrible performance on RHES3 pedge 2450
@ 2004-10-25 10:31 Zoe May
2004-10-25 12:22 ` Matt Domsch
0 siblings, 1 reply; 6+ messages in thread
From: Zoe May @ 2004-10-25 10:31 UTC (permalink / raw)
To: linux-raid
I am experiencing disk performance issues (very slow) with hardware and
software combo listed below.
Poweredge 2450
Perc 3/si controller.
RH linux ES3 (2.4.21-4.Elsmp)
4 x 73GB scsi (In a RAID 5 set)
This server was previously running RH 7.3 with a 34GB mirror set with no
issue
So far I have updated the firmware of the card to Version 2.8.0.6092,
Rev. A13
And have updated the aacraid driver to aacraid-1.1.4-2302
Neither of the above have had any impact.
This machine is freshly installed and has no load to speak of yet
As an indicator of performance here is what I get from hdparm.
/dev/sda2:
Timing buffer-cache reads: 780 MB in 2.00 seconds = 390.00 MB/sec
Timing buffered disk reads: 22 MB in 3.20 seconds = 6.88 MB/sec
[root@benidorm root]# hdparm -tT /dev/sda1
/dev/sda1:
Timing buffer-cache reads: 788 MB in 2.00 seconds = 394.00 MB/sec
Timing buffered disk reads: 26 MB in 3.02 seconds = 8.61 MB/sec
When I do disk writes the system pretty much locks up in terms of doing
anything else until it's finished.
The following is a vmstat when the system is idle
vmstat 1
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us sy
wa id
0 0 0 1541840 23028 127984 0 0 22 4 80 85 1
0 1 98
0 0 0 1541840 23028 127984 0 0 0 0 166 165 0
0 0 100
0 0 0 1541840 23028 127984 0 0 0 0 171 172 0
0 0 100
0 0 0 1541840 23028 127984 0 0 0 0 170 179 0
0 0 100
0 0 0 1541824 23044 127984 0 0 0 48 174 180 0
0 0 100
0 0 0 1541824 23044 127984 0 0 0 0 166 165 0
0 0 100
0 0 0 1541824 23044 127984 0 0 0 0 171 160 0
0 0 100
0 0 0 1541824 23044 127984 0 0 0 0 164 170 0
0 0 100
2 0 0 1541824 23044 127984 0 0 0 0 169 167 0
0 0 100
0 0 0 1541816 23052 127984 0 0 0 44 178 178 0
0 0 100
0 0 0 1541816 23052 127984 0 0 0 0 163 161 0
0 0 100
0 0 0 1541816 23052 127984 0 0 0 0 164 160 0
0 0 100
1 0 0 1541816 23052 127984 0 0 0 0 163 165 0
0 0 100
This is what happens when I run 'dd if=/dev/zero of=test bs=1024000
count=500'
[root@benidorm raid]# vmstat 1
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us sy
wa id
0 0 0 1539248 23168 127984 0 0 22 4 80 86 1
0 1 98
0 0 0 1539248 23168 127984 0 0 0 0 202 226 0
0 0 100
0 0 0 1539248 23168 127984 0 0 0 0 445 458 0
0 0 99
0 1 0 1539196 23196 127984 0 0 28 0 239 258 0
0 5 95
2 0 0 1438304 23356 227984 0 0 28 14992 561 329 0
46 11 43
1 0 0 1438304 23356 227984 0 0 0 0 254 238 0
0 0 100
2 0 0 1438304 23356 227984 0 0 0 0 348 331 0
0 0 100
0 0 0 1438304 23356 227984 0 0 0 0 311 404 0
0 0 100
0 0 0 1438304 23356 227984 0 0 0 0 213 218 0
0 0 100
1 2 0 1437672 23360 227984 0 0 0 63704 826 371 0
4 57 38
0 2 0 1437668 23360 227984 0 0 0 8192 313 184 0
0 50 50
0 2 0 1437664 23360 227984 0 0 0 8192 310 166 0
0 50 50
0 2 0 1437652 23360 227984 0 0 0 5040 337 199 0
1 49 50
2 2 0 1437652 23360 227984 0 0 0 0 332 190 0
0 50 50
0 2 0 1437652 23360 227984 0 0 0 370 522 379 0
0 50 50
0 2 0 1437652 23360 227984 0 0 0 0 417 285 0
0 50 50
0 0 0 1437644 23372 227984 0 0 0 120 322 216 0
0 7 92
0 0 0 1437644 23372 227984 0 0 0 0 275 254 1
0 0 100
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us sy
wa id
3 0 0 1437644 23372 227984 0 0 0 0 279 282 0
0 0 100
0 0 0 1437636 23380 227984 0 0 0 16 597 617 1
0 15 83
0 0 0 1437632 23380 227984 0 0 0 0 603 598 3
0 0 97
0 0 0 1437632 23380 227984 0 0 0 0 192 188 0
0 0 100
0 0 0 1437632 23380 227984 0 0 0 0 200 200 0
1 0 99
1 0 0 1437632 23380 227984 0 0 0 0 188 189 0
0 0 100
1 0 0 1437628 23388 227984 0 0 0 16 215 204 0
0 0 100
0 0 0 1437628 23388 227984 0 0 0 0 289 296 1
0 0 99
[root@benidorm raid]# vmstat 1
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us sy
wa id
2 0 0 1225320 23700 431252 0 0 22 12 81 86 1
0 1 98
1 2 0 1113708 23816 538284 0 0 4 42376 334 176 0
51 8 42
1 3 0 1053544 23876 595980 0 0 4 23108 607 289 0
29 72 0
0 2 0 1020184 23908 627984 0 0 0 10492 541 388 0
14 50 36
0 2 0 1020184 23908 627984 0 0 0 8192 457 319 1
0 50 49
0 2 0 1020184 23908 627984 0 0 0 8192 374 244 2
0 50 47
0 2 0 1020184 23908 627984 0 0 0 10560 411 284 0
0 50 50
1 2 0 1020184 23908 627984 0 0 0 6720 307 178 2
0 50 47
0 2 0 1020184 23908 627984 0 0 0 8256 398 333 1
0 50 48
0 2 0 1020184 23908 627984 0 0 0 10240 305 174 1
0 50 48
0 2 0 1020184 23908 627984 0 0 0 8256 371 233 1
1 49 49
0 2 0 1020184 23908 627984 0 0 0 10368 319 182 2
1 48 48
0 2 0 1020184 23908 627984 0 0 0 10432 313 168 5
1 49 45
0 2 0 1020184 23908 627984 0 0 0 8640 294 171 1
3 52 44
0 2 0 1020184 23908 627984 0 0 0 8384 297 171 0
0 49 50
0 2 0 1020184 23908 627984 0 0 0 8256 309 178 0
1 49 50
0 2 0 1020184 23908 627984 0 0 0 8512 297 179 0
1 49 50
0 2 0 1020184 23908 627984 0 0 0 8832 369 213 0
1 49 50
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us sy
wa id
1 2 0 1020184 23908 627984 0 0 0 10240 361 238 1
0 49 50
0 2 0 1020184 23908 627984 0 0 0 8128 300 181 0
0 50 50
0 2 0 1020184 23908 627984 0 0 0 8192 332 225 0
0 51 50
0 2 0 1020184 23908 627984 0 0 0 10624 488 343 1
1 48 50
0 2 0 1020184 23908 627984 0 0 0 8320 304 167 0
1 48 50
0 2 0 1020184 23908 627984 0 0 0 8704 302 180 1
1 49 50
0 2 0 1020184 23908 627984 0 0 0 10304 308 178 0
1 49 50
0 2 0 1020184 23908 627984 0 0 0 8384 304 178 1
0 48 50
0 2 0 1020184 23908 627984 0 0 0 8960 306 196 0
2 48 50
0 2 0 1020184 23908 627984 0 0 0 8192 316 191 1
0 49 50
0 2 0 1020184 23908 627984 0 0 0 8192 348 239 1
1 48 50
0 2 0 1020184 23908 627984 0 0 0 8192 301 176 1
1 48 50
0 2 0 1020184 23908 627984 0 0 0 10240 390 263 1
1 48 50
0 3 0 1020184 23908 627984 0 0 0 8188 399 271 2
2 47 50
0 3 0 1020184 23908 627984 0 0 0 11136 331 185 2
0 48 50
1 3 0 1020184 23908 627984 0 0 0 8256 353 233 1
1 47 50
0 3 0 1020184 23908 627984 0 0 0 8192 302 182 2
0 54 43
0 3 0 1020180 23908 627984 0 0 0 7132 292 186 3
2 95 0
procs memory swap io system
cpu
r b swpd free buff cache si so bi bo in cs us sy
wa id
1 4 0 1019896 23908 627988 0 0 0 8132 350 217 3
1 96 0
0 4 0 1019896 23908 627988 0 0 0 7380 299 183 2
1 96 0
1 4 0 1019896 23908 627988 0 0 0 7292 361 223 2
3 96 0
1 4 0 1019896 23908 627988 0 0 0 10920 357 207 1
3 96 0
0 4 0 1019892 23916 627988 0 0 0 9916 327 192 2
8 90 0
1 4 0 1019892 23916 627988 0 0 0 8832 333 180 5
3 92 0
0 4 0 1019892 23916 627988 0 0 0 10560 327 178 0
0 100 0
0 4 0 1019880 23924 627988 0 0 0 10892 326 175 0
1 99 0
0 4 0 1019880 23924 627988 0 0 0 8384 294 173 0
0 100 0
1 4 0 1019880 23924 627988 0 0 0 8832 313 168 0
1 99 0
1 4 0 1019880 23924 627988 0 0 0 10368 432 332 0
0 100 0
0 4 0 1019880 23924 627988 0 0 0 8128 371 290 0
0 100 0
0 4 0 1019880 23924 627988 0 0 0 8192 307 166 0
1 100 0
0 4 0 1019880 23924 627988 0 0 0 8576 297 168 0
0 100 0
0 4 0 1019880 23924 627988 0 0 0 8768 299 175 0
0 100 0
1 3 0 1019880 23924 627988 0 0 0 5944 299 177 0
0 100 0
1 3 0 1019880 23924 627988 0 0 0 0 306 156 0
0 100 0
0 3 0 1019880 23924 627988 0 0 0 0 300 158 0
0 100 0
This is the container config output from aafcli
_container show > cache 0
Executing: container show cache 0
Global Container Read Cache Size : 0
Global Container Write Cache Size : 51380224
Read Cache Setting : ENABLE
Write Cache Setting : ENABLE ALWAYS
Write Cache Status : Active, not protected, battery not present
The write cache setting was previously 'enable when protected'. I
changed it to enable always but it had no impact.
I'm not sure where to go with this now. Possibly trying to boot off
knoppix or something to see if a different kernel changes things.
If anyone has any suggestions I'd be very grateful.
Thanks
Zoe
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Aacraid - terrible performance on RHES3 pedge 2450
2004-10-25 10:31 Zoe May
@ 2004-10-25 12:22 ` Matt Domsch
0 siblings, 0 replies; 6+ messages in thread
From: Matt Domsch @ 2004-10-25 12:22 UTC (permalink / raw)
To: Zoe May; +Cc: linux-raid
On Mon, Oct 25, 2004 at 11:31:07AM +0100, Zoe May wrote:
>
> I am experiencing disk performance issues (very slow) with hardware and
> software combo listed below.
>
> Poweredge 2450
> Perc 3/si controller.
> RH linux ES3 (2.4.21-4.Elsmp)
up2date your system to kernel-smp-2.4.21-20.EL and see how much that
helps.
You're not the only person who has seen slow disk performance with the
-4.EL kernel.
Thanks,
Matt
--
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Aacraid - terrible performance on RHES3 pedge 2450
@ 2004-10-25 12:28 Salyzyn, Mark
2004-10-25 13:01 ` Matt Domsch
0 siblings, 1 reply; 6+ messages in thread
From: Salyzyn, Mark @ 2004-10-25 12:28 UTC (permalink / raw)
To: Matt Domsch, Zoe May; +Cc: linux-raid
Vary_io may be the reason for this.
The driver needs to have .vary_io set to 1.
There are no kernel clues to its existence (that I know of), but the
driver has an SCSI_HAS_VARY_IO to enable. Our build system will produce
correct binaries, but the DKMS tool does not differentiate.
There is a need to ship a different driver source in the DKMS package
for RHEL3 as a result (Add a #define SCSI_HAS_VARY_IO in the aacraid.h
file).
Sincerely -- Mark Salyzyn
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Matt Domsch
Sent: Monday, October 25, 2004 8:22 AM
To: Zoe May
Cc: linux-raid@vger.kernel.org
Subject: Re: Aacraid - terrible performance on RHES3 pedge 2450
On Mon, Oct 25, 2004 at 11:31:07AM +0100, Zoe May wrote:
>
> I am experiencing disk performance issues (very slow) with hardware
and
> software combo listed below.
>
> Poweredge 2450
> Perc 3/si controller.
> RH linux ES3 (2.4.21-4.Elsmp)
up2date your system to kernel-smp-2.4.21-20.EL and see how much that
helps.
You're not the only person who has seen slow disk performance with the
-4.EL kernel.
Thanks,
Matt
--
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Aacraid - terrible performance on RHES3 pedge 2450
2004-10-25 12:28 Aacraid - terrible performance on RHES3 pedge 2450 Salyzyn, Mark
@ 2004-10-25 13:01 ` Matt Domsch
0 siblings, 0 replies; 6+ messages in thread
From: Matt Domsch @ 2004-10-25 13:01 UTC (permalink / raw)
To: Salyzyn, Mark; +Cc: Zoe May, linux-raid
On Mon, Oct 25, 2004 at 08:28:47AM -0400, Salyzyn, Mark wrote:
> Vary_io may be the reason for this.
>
> The driver needs to have .vary_io set to 1.
>
> There are no kernel clues to its existence (that I know of), but the
> driver has an SCSI_HAS_VARY_IO to enable. Our build system will produce
> correct binaries, but the DKMS tool does not differentiate.
>
> There is a need to ship a different driver source in the DKMS package
> for RHEL3 as a result (Add a #define SCSI_HAS_VARY_IO in the aacraid.h
> file).
We could do this easily in DKMS with its PATCH_MATCH mechanism, making
it match RHEL kernel names and apply the patch only then which sets
.vary_io=1.
--
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux
Linux on Dell mailing lists @ http://lists.us.dell.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Aacraid - terrible performance on RHES3 pedge 2450
@ 2004-11-01 15:51 Zoe May
0 siblings, 0 replies; 6+ messages in thread
From: Zoe May @ 2004-11-01 15:51 UTC (permalink / raw)
To: Matt Domsch, Salyzyn, Mark; +Cc: linux-raid
Thanks for the info
I've upgraded the kernel to 2.4.21-20 and have a very slight increase in
performance but unfortunately not much to write home about.
I have limited experience with kernel compilation and I'm a little
confused as to which aacraid.h I would have to add the #define
SCSI_HAS_VARY_IO 1 to.
I have the following aacraid.h files
/usr/src/aacraid-1.1.4-2302/aacraid.h
/usr/src/linux-2.4.21-20.EL/drivers/addon/aacraid_10102/aacraid.h
/usr/src/linux-2.4.21-20.EL/drivers/scsi/aacraid/aacraid.h
Ordinarily I would modify the .h file in the scsi/aacraid dir and then
go through the full kernel recompile but I have updated aacraid using
the Dell rpm. I assume the version in this folder is the old version
that shipped with the 2.4.21-20 source. If there is a way that I can
recompile just the aacraid module without doing a full kernel recompile
I would prefer to do this .
Zoe
-----Original Message-----
From: Matt Domsch [mailto:Matt_Domsch@dell.com]
Sent: Monday, October 25, 2004 2:02 PM
To: Salyzyn, Mark
Cc: Zoe May; linux-raid@vger.kernel.org
Subject: Re: Aacraid - terrible performance on RHES3 pedge 2450
On Mon, Oct 25, 2004 at 08:28:47AM -0400, Salyzyn, Mark wrote:
> Vary_io may be the reason for this.
>
> The driver needs to have .vary_io set to 1.
>
> There are no kernel clues to its existence (that I know of), but the
> driver has an SCSI_HAS_VARY_IO to enable. Our build system will
> produce correct binaries, but the DKMS tool does not differentiate.
>
> There is a need to ship a different driver source in the DKMS package
> for RHEL3 as a result (Add a #define SCSI_HAS_VARY_IO in the aacraid.h
> file).
We could do this easily in DKMS with its PATCH_MATCH mechanism, making
it match RHEL kernel names and apply the patch only then which sets
.vary_io=1.
--
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell
mailing lists @ http://lists.us.dell.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: Aacraid - terrible performance on RHES3 pedge 2450
@ 2004-12-08 14:58 Zoe May
0 siblings, 0 replies; 6+ messages in thread
From: Zoe May @ 2004-12-08 14:58 UTC (permalink / raw)
To: linux-raid
After trying various Linux distros without success I decided to take the
hardware RAID out of the equation.
If I put the raid card into scsi mode and configure the disks (my system
has 4) as volumes my problem goes away - except that I now can't boot
into Linux. (install completes without issue).
I get as far as the controller picking up the volumes then at the point
when it would normally go into grub I just get a blinking cursor.
I can see my install by using linux rescue and when I run my disk i/o
tests the results are good - virtually no CPU wait at all.
I've tried using one volume with no software RAID, 4 volumes with
software RAID 5, 2 volumes with software RAID 1, all had the same
result.
Tried clearing nvram, flashing the BIOS, flashing the controller. In
desperation removed the RAID rom and ram and set the main BIOS
controller setting to SCSI. The system won't boot (stops just after
processor detection) or let me into
BIOS when I have this option set. I had to put the RAID hardware back in
just to access the BIOS!
Anyone any suggestions? I can't see any way out of this and it's held up
an upgrade project for weeks now.
At this stage I would be willing to suggest buying another controller if
there is a better one that's
guaranteed to work with the 2450 and RHEL3.
Thing I'm worried about is that the next server in the upgrade chain has
exactly the same hardware...
Cheers
Zoe
-----Original Message-----
From: Zoe May
Sent: Wednesday, November 24, 2004 2:07 PM
To: 'Salyzyn, Mark'
Cc: 'Matt_Domsch@dell.com'
Subject: RE: Aacraid - terrible performance on RHES3 pedge 2450
I tried this but unfortunately no improvement.
I really need to get this box in to production so I'm going to drag
myself away from AS3 try some different distros. Tried Fedora C2 this
morning but had the same issues.
Interestingly when I tried using Knoppix I didn't see much improvement
in disk i/o but vmstat didn't output the same high CPU wait times that
seem to be the cause of stopping everything else on the box running as
soon as I start writing to disk.
Thanks for your help
Zoe
-----Original Message-----
From: Salyzyn, Mark [mailto:mark_salyzyn@adaptec.com]
Sent: Thursday, November 18, 2004 4:06 PM
To: Zoe May
Cc: Matt_Domsch@dell.com
Subject: RE: Aacraid - terrible performance on RHES3 pedge 2450
Yes on the directory. No need to rebuild the kernel (if a past build is
sitting there), just do (2.4 kernel):
# make SUBDIRS=drivers/scsi/aacraid modules
and use the replacement module.
Sincerely -- Mark Salyzyn
-----Original Message-----
From: Zoe May [mailto:Zoe.May@Cartesian.co.uk]
Sent: Thursday, November 18, 2004 10:26 AM
To: Salyzyn, Mark
Cc: Matt_Domsch@dell.com
Subject: RE: Aacraid - terrible performance on RHES3 pedge 2450
Hi
After saying I would try the driver attached I opened the attachment and
realised I wasn't sure what to do with the files. Do I need to replace
the files in /usr/src/linux-xxx/drivers/scsi/aacraid with these and
rebuild the kernel?
Thanks
Zoe
-----Original Message-----
From: Salyzyn, Mark [mailto:mark_salyzyn@adaptec.com]
Sent: Monday, November 01, 2004 4:08 PM
To: Zoe May; Matt Domsch
Cc: linux-raid@vger.kernel.org
Subject: RE: Aacraid - terrible performance on RHES3 pedge 2450
Vary_io was added in 1.1.5-2326 to the adaptec aacraid sources, so
probably *none* of these. Adding a ".vary_io = 1," to the
scsi_host_template in the linit.c file would have to be manually added.
I have enclosed the sources of our latest driver that my comments were
based on. This driver has several performance enhancements (depending on
the adapter, between 15% and 45% improvement separate from the .vary_io
RHEL3 enhancement).
Sincerely -- Mark Salyzyn
-----Original Message-----
From: Zoe May [mailto:Zoe.May@Cartesian.co.uk]
Sent: Monday, November 01, 2004 10:52 AM
To: Matt Domsch; Salyzyn, Mark
Cc: linux-raid@vger.kernel.org
Subject: RE: Aacraid - terrible performance on RHES3 pedge 2450
Thanks for the info
I've upgraded the kernel to 2.4.21-20 and have a very slight increase in
performance but unfortunately not much to write home about.
I have limited experience with kernel compilation and I'm a little
confused as to which aacraid.h I would have to add the #define
SCSI_HAS_VARY_IO 1 to.
I have the following aacraid.h files
/usr/src/aacraid-1.1.4-2302/aacraid.h
/usr/src/linux-2.4.21-20.EL/drivers/addon/aacraid_10102/aacraid.h
/usr/src/linux-2.4.21-20.EL/drivers/scsi/aacraid/aacraid.h
Ordinarily I would modify the .h file in the scsi/aacraid dir and then
go through the full kernel recompile but I have updated aacraid using
the Dell rpm. I assume the version in this folder is the old version
that shipped with the 2.4.21-20 source. If there is a way that I can
recompile just the aacraid module without doing a full kernel recompile
I would prefer to do this .
Zoe
-----Original Message-----
From: Matt Domsch [mailto:Matt_Domsch@dell.com]
Sent: Monday, October 25, 2004 2:02 PM
To: Salyzyn, Mark
Cc: Zoe May; linux-raid@vger.kernel.org
Subject: Re: Aacraid - terrible performance on RHES3 pedge 2450
On Mon, Oct 25, 2004 at 08:28:47AM -0400, Salyzyn, Mark wrote:
> Vary_io may be the reason for this.
>
> The driver needs to have .vary_io set to 1.
>
> There are no kernel clues to its existence (that I know of), but the
> driver has an SCSI_HAS_VARY_IO to enable. Our build system will
> produce correct binaries, but the DKMS tool does not differentiate.
>
> There is a need to ship a different driver source in the DKMS package
> for RHEL3 as a result (Add a #define SCSI_HAS_VARY_IO in the aacraid.h
> file).
We could do this easily in DKMS with its PATCH_MATCH mechanism, making
it match RHEL kernel names and apply the patch only then which sets
.vary_io=1.
--
Matt Domsch
Sr. Software Engineer, Lead Engineer
Dell Linux Solutions linux.dell.com & www.dell.com/linux Linux on Dell
mailing lists @ http://lists.us.dell.com
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2004-12-08 14:58 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-10-25 12:28 Aacraid - terrible performance on RHES3 pedge 2450 Salyzyn, Mark
2004-10-25 13:01 ` Matt Domsch
-- strict thread matches above, loose matches on Subject: below --
2004-12-08 14:58 Zoe May
2004-11-01 15:51 Zoe May
2004-10-25 10:31 Zoe May
2004-10-25 12:22 ` Matt Domsch
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.