Re: Infiniband 40GB

[Alexandre DERUMIER <aderumier@odiso.com>]

Hi,
about this:
>> Turning off Virtualisation extension in BIOS. Don't know why, but it 
>>gaves us crappy performance. We usually put it on, because we use KVM a 
>>lot. In our case, OSD are in bare metal and disabling virtualisation 
>>extension gives us a very big boost. 
>>It may be a BIOS bug in our machines (DELL M610). 

It could be related to iommu, if you pass intel_iommu=on in grub.
I have already had this kind of problem.

When intel_iommu=on, Linux (completely unrelated to KVM) adds a new level
of protection which didn't exist without an IOMMU - the network card, which
without an IOMMU could write (via DMA) to any memory location, now is
not allowed - the card can only write to memory locates which the OS
wanted it to write. Theoretically, this can protect the OS against
various kinds of attacks. But what happens now is that every time that
Linux passes a new buffer to the card, it needs to change the IOMMU
mappings. This noticably slows down I/O, unfortunately. 


----- Mail original ----- 

De: "Yann Dupont" <Yann.Dupont@univ-nantes.fr> 
À: "Stefan Majer" <stefan.majer@gmail.com> 
Cc: "Hannes Reinecke" <hare@suse.de>, "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag>, "Mark Nelson" <mark.nelson@inktank.com>, ceph-devel@vger.kernel.org 
Envoyé: Lundi 4 Juin 2012 11:21:56 
Objet: Re: Infiniband 40GB 

Le 04/06/2012 10:23, Stefan Majer a écrit : 
> Hi Hannes, 
> 
> our production environment is running on 10GB infrastructure. We had a 
> lot of troubles till we got to where we are today. 
> We use Intel X520 D2 cards on our OSD´s and nexus switch 
> infrastructure. All other cards we where testing failed horrible. 
> 

we have Intel Corporation 82599EB 10 Gigabit Dual Port Backplane 
Connection (rev 01)... Don't know the 'commercial name'. ixgbe driver. 


> Some of the problems we encountered have been: 
> - page allocation failures in the ixgbe driver --> fixed in upstream 
> - problems with jumbo frames, we had to disable tso, gro, lro -- > 
> this is the most obscure thing 
> - various tuning via sysctl in the net.tcp and net.ipv4 area --> this 
> was also the outcome of stefan´s benchmarking odysee. 

some tuning we made : 

-> Turning off Virtualisation extension in BIOS. Don't know why, but it 
gaves us crappy performance. We usually put it on, because we use KVM a 
lot. In our case, OSD are in bare metal and disabling virtualisation 
extension gives us a very big boost. 
It may be a BIOS bug in our machines (DELL M610). 

-> One of my colleague played with receive flow steeting ; the intel 
card supports multi queue, so it seems we can gain a little with it : 

!/bin/sh 

for x in $(seq 0 23); do echo FFFFFFFF > 
/sys/class/net/eth2/queues/rx-${x}/rps_cpus; done 
echo 16384 > /proc/sys/net/core/rps_sock_flow_entries 
for x in $(seq 0 23); do echo 16384 > 
/sys/class/net/eth2/queues/rx-${x}/rps_flow_cnt; done 


> 
> But after all this we a quite happy actully and are only limited by 
> the speed of the drives (2TB SATA). 
> The fsync is a fdatasync in fact which is available in newer glibc. If 
> you dont use btrfs (we use xfs) you need to use a recent glibc with 
> fdatasync support. 

Does it may explain why we see loosy performance with xfs right now ? 
That the main reason we're stuck with btrfs for the moment. 

we're using debian 'stable' : libc is 
libc6 2.11.3-3 
probably too old ? 

Cheers, 
-- 
Yann Dupont - Service IRTS, DSI Université de Nantes 
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr 


-- 
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at http://vger.kernel.org/majordomo-info.html 



-- 

-- 




	Alexandre D erumier 
Ingénieur Système 
Fixe : 03 20 68 88 90 
Fax : 03 20 68 90 81 
45 Bvd du Général Leclerc 59100 Roubaix - France 
12 rue Marivaux 75002 Paris - France 
	
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html