Re: RAID performance - 5x SSD RAID5 - effects of stripe cache sizing

[Stan Hoeppner <stan@hardwarefreak.com>]

On 3/7/2013 6:17 PM, Adam Goryachev wrote:
> On 07/03/13 18:36, Stan Hoeppner wrote:

>> Run FIO on the DC itself and see what your NTFS throughput is to this
>> 300GB filesystem.  Use a small file, say 2GB, since the FS is nearly
>> full.  Post results.
> 
> Can't, I don't see a version of fio that runs on win2000...

Well find another utility, or just do a file copy with a stop watch.
The point here is to determine if NTFS is part of the bottleneck, as you
already verified, IIRC, that FIO at the hypervisor level is doing ~100MB/s.

> I had somewhat forgotten about FTP, and it does provide nice simple
> performance results/numbers too. I'll give this a try, talking to
> another linux box on the network, it should achieve 100MB/s (gigabit
> speeds) or close to it. I'll also run the same FTP test from one of the
> 2003 boxes for comparison.

I think you missed the point.  SMB with TS<->DC is ~10MB/s, but should
be more like 100MB/s.  Run the FTP client on TS against the FTP service
on the DC.  Get and put files from/to the 300GB NTFS volume that is
shared.  If FTP is significantly faster then we know SMB is the problem,
or something related to SMB, not TCP.

> Well, during the day, under normal user load, the CPU frequently rises
> to around 70 to 80%, while this is not as clear cut as 100%, it makes me
> worry that it is limiting performance.

Agreed.  CPU burn shouldn't be that high for SMB serving.  The cause
could be any number of things, or a combination of things.  The steps
I'm suggesting will allow us to identify the cause of the burn and the
low SMB throughput.

> NTFS compression is already disabled on all volumes.... I've *never*
> enabled it on any system I've ever been responsible for, and never seen
> anyone else do that. However, due to the age of this system, it is
> possible that it has been enabled and then disabled again at some point.

Ok, good.  No compression.  How about NTFS encryption?

> I only bumped it up a small amount just in case I got burned by windows
> 2000 having an upper limit on disk size supported. I couldn't find a
> clear answer on the maximum size supported.... I'll probably increase it
> to at least 400 or even 500 as soon as I complete the upgrade to win2003.

2TB is the minimum ceiling.  If it's a Dynamic Disk the limit is 16TB:
http://technet.microsoft.com/en-us/library/cc779002%28v=ws.10%29.aspx

>> Growing filesystem in small chunks like this is a recipe for disaster.
>>  Your free space map is always heavily fragmented and is very large.
>> The more entries the filesystem driver must walk the more CPU you burn.
>>  Recall we just discussed the table walking overhead of the md/RAID
>> stripe cache?  Filesystem maps/tables/B+ trees are much, much larger
>> structures.  When they don't fit it cache we read memory, and when they
>> don't fit in memory (remember you "pool" problem) we must read from disk.
> 
> Yes, I did try and run a defrag (win2000 version) on the volume. I was

*NEVER* run a defragger against a filesystem residing on SSDs.  It will
shorten the life of the flash cells due to wear leveling, and thus the
SSDs themselves:  http://www.intel.com/support/ssdc/hpssd/sb/CS-029623.htm#5

And it will not fix the problem I was describing.

>> If you've been expanding this NTFS this way for a while it would also
>> explain some of your CPU burn at the DC.  FYI, XFS is a MUCH higher
>> performance and much more efficient filesystem than NTFS ever dreamed of
>> becoming, but even XFS suffers slow IO and CPU burn due to heavily
>> fragmented free space.
> 
> Nope, it has had the same 300G HDD (279G) for at least 6 years (as far
> as I know). I'm pretty sure is has only been extended by physical
> replacement of the HDD from time to time.

Ok. Good.  We're narrowing things down a bit.

> The current plan is to upgrade to win2003, I'm hoping this will improve
> performance equivalent to what is being achieved on the other 2003
> servers, which should make the users happy again. I may increase the
> disk space and have another crack at defrag prior to the upgrade since
> the upgrade won't happen until next weekend at the earliest.

Do NOT defrag.  This should be well driven home at this point.

When you upgrade to 2003, make sure you use the open source Xen
paravirtualized SCSI and NIC drivers:

http://jolokianetworks.com/70Knowledge/Virtualization/Open_Source_Windows_Paravirtualization_Drivers_for_Xen

-- 
Stan