From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Mason Subject: Re: swiotlb=force in Konrad's xen-pcifront-0.8.2 pvops domU kernel with PCI passthrough Date: Thu, 18 Nov 2010 12:28:34 -0500 Message-ID: <1290101068-sup-598@think> References: <20101112165541.GA10339@dumpdata.com> <20101112223333.GD26189@dumpdata.com> <20101116185748.GA11549@dumpdata.com> <20101116201349.GA18315@dumpdata.com> <20101118171936.GA29275@dumpdata.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Return-path: In-reply-to: <20101118171936.GA29275@dumpdata.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: Jeremy Fitzhardinge , Xen-devel , "mathieu.desnoyers" , "andrew.thomas" , "keir.fraser" , "mukesh.rathor" , Dante Cinco List-Id: xen-devel@lists.xenproject.org Excerpts from Konrad Rzeszutek Wilk's message of 2010-11-18 12:19:36 -050= 0: > Keir, Dan, Mathieu, Chris, Mukesh, >=20 > This fellow is passing in a PCI device to his Xen PV guest and trying > to get high IOPS. The kernel he is using is a 2.6.36 with tglx's > sparse_irq rework. >=20 > > I wanted to confirm that bounce buffering was indeed occurring so I > > modified swiotlb.c in the kernel and added printks in the following > > functions: > > swiotlb_bounce > > swiotlb_tbl_map_single > > swiotlb_tbl_unmap_single > > Sure enough we were calling all 3 five times per I/O. We took your > > suggestion and replaced pci_map_single with pci_pool_alloc. The > > swiotlb calls were gone but the I/O performance only improved 6% (29k > > IOPS to 31k IOPS) which is still abysmal. >=20 > Hey! 6% that is nothing to sneeze at. How fast does it go on bare metal? I usually do four things: 1) perf record -g -a -f 'sleep 15' (use perf report to look at the biggest CPU hogs) 2) mpstat -P ALL 1 to find the CPU doing all the softirq processing 3) perf record -g -C N -f 'sleep 15' where N was the CPU in mpstat -P ALL that was doing all the softirq processing 4) Turn off the intel iommu. This isn't an option of for virtualized though, but I'd try it on/off on bare metal. -chris