Re: Growing RAID5 SSD Array

[Adam Goryachev <mailinglists@websitemanagers.com.au>]

On 18/03/14 22:22, Stan Hoeppner wrote:
> On 3/17/2014 8:41 PM, Adam Goryachev wrote:
>> On 18/03/14 08:43, Stan Hoeppner wrote:
>>> On 3/17/2014 12:43 AM, Adam Goryachev wrote:
>>>> On 13/03/14 22:58, Stan Hoeppner wrote:
>>>>> On 3/12/2014 9:49 PM, Adam Goryachev wrote:
>>>>>
>>>>>
>>>>>
> But again you should have had no iSCSI sessions active, and if you 
> didn't shutdown DRBD during a reshape then you're asking for it 
> anyway. Recall in my initial response I recommended you shutdown DRBD 
> before doing the reshapes? 

Yes, and I did ignore that very good, sane advice. However, it should 
have worked... So a kernel bug somewhere happened to bite me, hopefully 
it has been fixed in a newer kernel already, and I will definitely learn 
from the experience and shutdown all iscsi clients prior to the next 
upgrade. However, I don't think I can stop drbd when I grow the array, 
because drbd uses information at the "end" of the block device, and if 
the array has grown, then it won't find the right information. I need to 
grow the MD device, and then grow drbd while it is on-line to work 
smoothly. Though if I shutdown iscsi, then stop lvm, then nothing will 
even have the drbd device open, so it should be totally idle.
>>>> It was usually around 250 to
>>>> 300MB/s, the maximum achieved was around 420MB/s. I also noticed that
>>>> idle CPU time on one of the cores was relatively low, though I never saw
>>>> it hit 0 (minimum I saw was 12% idle, average around 20%).
>>> Never look at idle, but what's eating the CPU.  Was that 80+% being
>>> eaten by sys, wa, or a process?  Without that information it's not
>>> possible to definitely answer your questions below.
>> Unfortunately I should have logged the info but didn't. I am pretty sure
>> md1_resync was at the top of the task list...
> A reshape reads and writes all drives concurrently.  You're likely not
> going to get even one drive worth of write throughput.  Your FIO testing
> under my direction showed 1.6GB/s div 4 = 400MB/s peak per drive write
> throughput with a highly parallel workload, i.e. queue depth >4.  I'd
> say these reshape numbers are pretty good.  If it peaked at 420MB/s and
> average 250-300 then other processes were accessing the drives.  If DRBD
> was active that would probably explain it.  This isn't something to
> spend any time worrying about because it's not relevant to your
> production issues.

OK, good :) Less to worry about is a good thing.
>>>> Currently I'm looking to replace at least the motherboard with
>>>> http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCM-F.cfm
>>>> in
>>>> order to get 2 of the PCIe 2.0 8x slots (one for the existing LSI SATA
>>>> controller and one for a dual port 10Gb ethernet card. This will provide
>>>> a 10Gb cross-over connection between the two server, plus replace the 8
>>>> x 1G ports with a single 10Gb port (solving the load balancing across
>>>> the multiple links issue). Finally, this 28 port (4 x 10G + 24 x 1G)
>>>> switch
>>> Adam if you have the budget now I absolutely agree that 10 GbE is a much
>>> better solution than the multi-GbE setup.
>> Well, I've been tasked to fix the problem..... Whatever it takes. I just
>> don't know what I should be targetting....
>>> But you don't need a new
>>> motherboard.  The S1200BTLR has 4 PCIe 2.0 slots: one x8 electrical in
>>> x16 physical slot, and three x4 electrical in x8 physical slots.  Your
>>> bandwidth per slot is:
>>>
>>> x8    4 GB/s unidirectional x2  <-  occupied by LSI SAS HBA
>>> x4    2 GB/s unidirectional x2  <-  occupied by quad port GbE cards
>>>
>>> 10 Gbps Ethernet has a 1 GB/s effective data rate one way.  Inserting an
>>> x8 PCIe card into an x4 electrical/x8 physical slot gives you 4 active
>>> lanes for 2+2 GB/s bandwidth.  This is an exact match for a dual port 10
>>> GbE card.  You could install up to three dual port 10 GbE cards into
>>> these 3 slots of the S1200BTLR.
>> This is somewhat beyond my knowledge, but I'm trying to understand, so
>> thank you for the information. From
>> http://en.wikipedia.org/wiki/PCI_Express#PCI_Express_2.0 it says:
>>
>> "Like 1.x, PCIe 2.0 uses an 8b/10b encoding
>> <http://en.wikipedia.org/wiki/8b/10b_encoding> scheme, therefore
>> delivering, per-lane, an effective 4 Gbit/s max transfer rate from its 5
>> GT/s raw data rate."
>>
>> So, it suggests that we can get 4Gbit/s * 4 (using the x4 slots) which
>> provides a maximum throughput of 16Gbit/s  which wouldn't quite manage
>> the full 20Gb/s capable from a dual port 10Gb card.
> Except for the fact that you'll never get close to 10 Gbps with TCP due
> to protocol overhead, host latency, etc.  Your goal in switching to 10
> GbE should not be achieving 10 Gb/s throughput, as that's not possible
> with your workload.  Your goal should be achieving more bandwidth more
> of the time than what you can achieve now with 8 GbE interfaces, and
> simplifying your topology.
>
> Again, your core problem isn't lack of bandwidth in the storage network.
I'm still somewhat concerned that this might cause problems, given a new 
motherboard is around $350, I'd prefer to replace it if that is going to 
help at all. Even if I solve the "other" problem, I'd prefer the users 
to *really* notice the difference, rather than just "normal". ie, I want 
the end result to be excellent rather than good, considering all the 
time, money and effort... For now, I've just ordered the 2 x Intel cards 
plus 1 of the cables (only one in stock right now, the other three are 
on back order) plus the switch. I should have all that by tomorrow, and 
if all goes well and I can use the single cable as a direct connect 
between the two machines, then that's great, if not I will have to wait 
for more cables.

>> One option is to
>> only use a single port for the cross connect, but it would probably help
>> to be able to use the second port to replace the 8x1Gb ports. (BTW, the
>> pci and ethernet bandwidth is apparently full duplex, so that shouldn't
>> be a problem AFAIK).
>>
>> Or, I'm reading something wrong?
> Everything is full duplex today, has been for many years.  Yes, you'd
> use one port on each 2-port 10 GbE NIC for DRBD traffic and the other to
> replace the 8 GbE ports.  Again, this won't solve the current core
> problem but it will provide benefits.
>
>>>> http://www.netgear.com.au/business/products/switches/stackable-smart-switches/GS728TXS.aspx#
>>>>
>>>> should allow the 2 x 10G connections to be connected through to the 8
>>>> servers with 2 x 1G connections each using multipath scsi to setup two
>>>> connections (one on each 1G port) with the same destination (10G port)
>>>>
>>>> Any suggestions/comments would be welcome.
>> Finally, can you suggest a reasonable solution on how or what to monitor
>> to rule out the various components?
> You don't need to.  You already found the problem, a year ago.  I'm
> guessing you simply forgot to fix it, or didn't sufficiently fix it.
>
>> I know in the past I've used fio on the server itself, and got excellent
>> results (2.5GB/s read + 1.6GB/s write), I know I've done multiple
>> parallel fio tests from the linux clients and each gets around 180+MB/s
>> read and write, I know I can do fio tests within my windows VM's, and
>> still get 200MB/s read/write (one at a time recently). Yet at times I am
>> seeing *really* slow disk IO from the windows VM's (and linux VM's),
>> where in windows you can wait 30 seconds for the command prompt to
>> change to another drive, or 2 minutes for the "My Computer" window to
>> show the list of drives. I have all this hardware, and yet performance
>> feels really bad, if it's not hardware, then it must be some config
>> option that I've seriously stuffed up...
> I may have some details incorrect as I'm going strictly from organic
> memory here, so please pardon me if I fubar a detail or two.
>
> You had a Windows 2000 Directory Controller VM that hosts all of your
> SMB file shares.  You were giving it only one virtual CPU, i.e. one
> core, and not enough RAM.  It was peaking the core during any sustained
> SMB file copy in either direction while achieving less than 100 MB/s SMB
> throughput IIRC.  In addition, your topology limits SMB traffic between
> the hypervisor nodes to a single GbE link, 100 MB/s.
I only ran win2000 for a very minimal time, I think less than one day, 
which was part of the process of migrating from the old winNT 4.0 
physical machine to the VM. It has been running win2003 for over a year 
now.
I know in the initial period I did have an issue where I couldn't 
upgrade to multiple CPU's, but it seems I did eventually manage to solve 
that, because it is now running with 4 vCPU's and has been for a long 
time. I think I also had issues with running win2003sp1, but an upgrade 
to sp2 resolved that issue, something to do with the way the CPU was 
being used the the virtualisation layer.

Generally, it has always been the only VM running on the physical 
machine (to ensure network/cpu/etc priority), and has 4 vCPU's mapped to 
4 physical CPU's which are not shared with anything. The dom0 has two 
dedicated CPU's as well. All the win2003 machines are allocated 4GB RAM 
(maximum for win2003 32bit).
The windows VM is limited to 1Gbps for SMB traffic, in fact the entire 
"user" LAN is 1Gbps, at least for all the VM's.

> The W2K VM simply couldn't handle more than 200 MB/s of combined SMB and
> block IO processing.  I did some research at that time and found that
> 2003/2008 had many enhancements for running in VMs that solved many of
> the virtualization performance problems of W2K.  I suggested you
> wholesale move SMB file sharing directly to the storage servers running
> Samba to fix this once and for all, with a sledgehammer, but you did not
> want to part with a Windows VM hosting the SMB shares.  I said your next
> best option was to upgrade and give the DC VM 4 virtual CPUs and 2GB of
> RAM.  IIRC you said you needed to allocate as much CPU/RAM as possible
> to the other VMs on that box and you couldn't spare it.

Yes, I was (still am) very scared to replace the DC with a Linux box. 
Moving the SMB shares would have resulted in changing the "location" of 
all the files, and means finding and fixing every config file or spot 
which relies on that. Though I have thought about this a number of 
times. Currently, the plan is to migrate the authentication, DHCP, DNS, 
etc to a new win2008R2 machine this weekend. Once that is done, next 
weekend I will try and migrate the shares to a new win2012R2 machine. 
The goal being to resolve any issues caused by upgrading the old win NT 
era machine over and over and over again, by using brand new 
installations of more modern versions. When the time comes, I may 
consider migrating the file sharing to a linux VM, I've very slightly 
played with samba4, but I'm not particularly confident about it yet (it 
isn't included in Debian stable yet).

> So, as of the last information I have, you had not fixed this.  Given
> the nature of the end user issues you describe, which are pretty much
> identical to a year ago, I can only assume you didn't properly upgrade
> or replace this Windows DC file server VM and it is still the
> bottleneck.  The long delays you mention tend to indicate it is trying
> to swap heavily but is experiencing tremendous latency in doing so.  Is
> the swap file for this DC VM physically located on the iSCSI server?  If
> so the round trip latency is exacerbating the VM's attempts to swap.

The VM isn't swapping at all. At one stage I allocated an additional 4GB 
ram drive for each VM (DC plus terminal servers), which simply looked 
like a normal 4GB hard drive to windows. Then moved the pagefile to this 
drive. It didn't make any difference to the performance issues, and in 
the end I removed it because it meant I couldn't live migrate VM's to 
different physical boxes, etc. In any case, swap is not in use for the 
DC at all, right now (9:15am) there is 17% physical memory in use, and 
CPU load is under 1%. The next time things are running slowly I'll take 
another look at these numbers, but I don't suspect the issue is memory 
or cpu on this box.

> Get out your medical examiner's kit and perform an autopsy on this
> Windows DC/SMB server VM.  This is where you'll find the problem I
> think.  If not it's somewhere in your Windows infrastructure.
>
> Two minutes to display the mapped drive list in Explorer?  That might be
> a master browser issue.  Go through all the Windows Event logs for the
> Terminal Services VMs with a fine toothed comb.
The performance issue impacts on unrelated linux VM's as well. I 
recently setup a new Linux VM to run a new application. When the issue 
is happening, if I login to this VM, disk IO is severely slow, like 
running ls will take a long time etc...

I see the following event logs on the DC:
NTDS (764)NTDSA: A request to write to the file "C:\WINNT\NTDS\edb.chk" 
at offset 0 (0x0000000000000000) for 4096 (0x00001000) bytes succeeded 
but took an abnormally long time (72 seconds) to be serviced by the OS. 
This problem is likely due to faulty hardware. Please contact your 
hardware vendor for further assistance diagnosing the problem.

That type of event hasn't happened often:
20140314 11:15:35   72 seconds
20131124 17:55:48   55 minutes 12 seconds
20130422 20:45:23   367 seconds
20130410 23:57:16   901 seconds

Though these look like they may have happened at times when DRBD crashed 
or similar, since I've definitely had a lot more times of very slow 
performance....

Also looking on the terminal servers has produced a similar lack of 
events, except some auth errors when the DC has crashed recently.

The newest terminal servers (running Win 2012R2) show this event for 
every logon:
Remote Desktop services has taken too long to load the user 
configuration from server \\DC for user xyz

Although the logins actually do work, and seems mostly normal after 
login, except for times when it runs really slow again.

Finally, on the old terminal servers, the PST file for outlook contained 
*all* of the email and was stored on the SMB server, on the new terminal 
servers, the PST file on the SMB server only contains contacts and 
calendars (ie, very small) and the email is stored in the "local" 
profile on the C: (which is iSCSI still). I'm hopeful that this will 
reduce the file sharing load on the domain controller. (If the C: pst 
file is lost, then it is automatically re-created and all the email is 
re-downloaded from the IMAP server, so nothing is lost, but it 
drastically increases the SAN load to re-download 2GB of email for each 
user, which had a massive impact on performance on Friday last week!).
>> Firstly I want to rule out MD, so far I am graphing the read/write
>> sectors per second for each physical disk as well as md1, drbd2 and each
>> LVM. I am also graphing BackLog and ActiveTime taken from
>> /sys/block/DEVICE/stat
>> These stats clearly show significantly higher IO during the backups than
>> during peak times, so again it suggests that the system should be
>> capable of performing really well.
> You're troubleshooting what you know because you know how to do it, even
> though you know deep down that's not where the problem is.  You're
> comfortable with it so that's the path you take.  You're avoiding
> troubleshooting Windows, but this is where the heart of this problem is,
> so you simply must.
>
>> Thanks again for any advice or suggestions.
> I hope I helped steer you toward the right path Adam.  Always keep in
> mind that the apparent cause of problems within a virtual machine guest
> are not always what they appear to be.
I'm really not sure, I still don't like the domain controller and file 
server being on the same box, and the fact it has been upgraded so many 
times, but I'm doubtful that it is the real cause.

On Thursday night after the failed RAID5 grow, I decided not to increase 
the allocated space for the two new terminal servers (in case I caused 
more problems), and simply deleted a number of user profiles on each 
system. (I assumed the roaming profile would simply copy back when the 
user logged in the next day). However, the roaming profile didn't copy, 
and windows logged users in with a temp profile, so eventually the only 
fix was to restore the profile from the backup server. Once I did this, 
the user could login normally, except the backup doesn't save the pst 
file, so outlook was forced to re-download all of the users email from 
IMAP. This then caused the really, really, really bad performance across 
the SAN, yet it didn't generate any traffic on the SMB shares from the 
domain controller. In addition, as I mentioned, disk IO on the newest 
Linux VM was also badly delayed. Also, copying from a smb share on a 
different windows 2008 VM (basically idle and unused) showed equally bad 
performance copying to my desktop (linux), etc.

So, essentially the current plans are:
Install the Intel 10Gb network cards
Replace the existing 1Gbps crossover connection with one 10Gbps connection
Replace the existing 8 x 1Gbps connections with 1 x 10Gbps connection
Migrate the win2003sp2 authentication etc to a new win2008R2 server
Migrate the win2003sp2 SMB to a new win2012R2 server

I'd still like to clarify whether there is any benefit to replacing the 
motherboard, if needed, I would prefer to do that now rather than later. 
Mainly I wanted to confirm that the rest of the interfaces on the 
motherboard were not interconnected "worse" than the current one. I 
think from the manual the 2 x PCIe x8 and one PCIe x4 and memory were 
directly connected to the CPU, while everything else including onboard 
sata, onboard ethernet, etc are all connected via another chip.

Thanks again for all your advice, much appreciated.

Regards,
Adam

-- 
Adam Goryachev Website Managers www.websitemanagers.com.au