Re: RAID performance

[Stan Hoeppner <stan@hardwarefreak.com>]

On 2/14/2013 6:22 AM, Stan Hoeppner wrote:

> Then create 8 table entries with names, such as port_0 thru port_7:
> 
> ~$ echo 100 port_0 >> /etc/iproute2/rt_tables
> ......
> ~$ echo 101 port_7 >> /etc/iproute2/rt_tables

Correcting a typo here, this 2nd line above should read:

~$ echo 107 port_7 >> /etc/iproute2/rt_tables


These 7 commands result in a routing table like this:

100	port_0
101	port_1
102	port_2
103	port_3
104	port_4
105	port_5
106	port_6
107	port_7

The commands below this in the previous email populate the table with
the source routing rules.  With arp_filter enabled, what all of this
does is allow each of the 8 interfaces to behave just like 8 individual
hosts on the same subnet would.  And thinking about this for a brief
moment, you realize this should work just fine on a single switch
without any special switch configuration.  arp_filter docs tell us:

arp_filter - BOOLEAN
	1 - Allows you to have multiple network interfaces on the same
	subnet, and have the ARPs for each interface be answered
	based on whether or not the kernel would route a packet from
	the ARP'd IP out that interface (therefore you must use source
	based routing for this to work). In other words it allows
	control of which cards (usually 1) will respond to an arp
	request.


	0 - (default) The kernel can respond to arp requests with
	addresses from other interfaces. This may seem wrong but it
	usually makes sense, because it increases the chance of
	successful communication.  IP addresses are owned by the
	complete host on Linux, not by particular interfaces. Only for
	more complex setups like load-balancing, does this behaviour
	cause problems.

	arp_filter for the interface will be enabled if at least one of
	conf/{all,interface}/arp_filter is set to TRUE,
	it will be disabled otherwise

As you have other interfaces on the user subnet, we're enabling this
only for the SAN subnet, on a per interface basis, otherwise it would
cause problems with the user subnet interfaces.

So now all SAN subnet traffic from a given interface is properly sent
from that interface.  With your previous arp tweaks it seems each
interface was responding to arps, but TCP packets were still all going
out a single interface.  This configuration fixes that.

** IMPORTANT **
All of the work you've done with iscsiadm to this point has been with
clients having a single iSCSI ethernet port and single server target
port, and everything "just worked" without specifying local and target
addresses (BTW, don't use the server hostname for any of these
operations, obviously, only the IP addresses as they won't map).  Since
you will now have two local iSCSI addresses and potentially 8 target
addresses, discovery and possibly operations should probably be done on
a 1:1 port basis to make sure both client ports are working and both are
logging into the correct remote ports and mapping the correct LUNs.
Executing the same shell command 128 times across 8 hosts, changing
source and port IP addresses each time, seems susceptible to input
errors.  Two per host less so.

On paper, if multipath will fan all 8 remote ports from each client
port, theoretically you could getter better utilization in some client
access pattern scenarios.  But in real world use, you won't see a
difference.  Given the complexity of trying to use all 8 server ports
per client port, if this was my network, I'd do it like this,
conceptually:  http://www.hardwarefreak.com/lun-mapping.png
Going the "all 8" route you'd add another 112 lines to that diagram atop
the current 16.  That seems a little "busy" and unnecessary, more
difficult to troubleshoot.

Yes, I originally suggested fanning across all 8 ports, but after
weighing the marginal potential benefit against the many negatives, it's
clear to me that it's not the way to go.

So during your next trip to the client, once you have all of your new
cables and ties, it should be relatively quick to set this up.  Going
the "all 8" route maybe not so quick.

-- 
Stan