From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Sacha Subject: IOTLB page size question Date: Mon, 13 Oct 2014 13:50:35 -0700 Message-ID: <543C3B1B.8050802@alcatel-lucent.com> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit To: Return-path: Received: from us-hpatc-esg-02.alcatel-lucent.com ([135.245.18.28]:54558 "EHLO smtp-us.alcatel-lucent.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751188AbaJMVBD (ORCPT ); Mon, 13 Oct 2014 17:01:03 -0400 Received: from us70uusmtp4.zam.alcatel-lucent.com (unknown [135.5.2.66]) by Websense Email Security Gateway with ESMTPS id 036FD54171553 for ; Mon, 13 Oct 2014 20:50:37 +0000 (GMT) Received: from US70TWXCHHUB04.zam.alcatel-lucent.com (us70twxchhub04.zam.alcatel-lucent.com [135.5.2.36]) by us70uusmtp4.zam.alcatel-lucent.com (GMO) with ESMTP id s9DKoc2G016178 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Mon, 13 Oct 2014 16:50:39 -0400 Sender: kvm-owner@vger.kernel.org List-ID: Hi, I have a question about IOTLB. We are running KVM/Qemu VMs with huge page memory backing on Intel Xeon Ivy Bridge machines. Our VMs use 10G Ether NICs in Intel VT-d mode. We actually see that IOTLB becomes a performance bottleneck when IOMMU uses 4k pages. We get much better packet throughput with 2M IOTLB pages. We have tried two different Linux distributions (CentOS and Fedora). An older CentOS kernel maps everything using 4k IOTLB pages. Our newer Fedora kernel initially maps guest memory using 2M IOTLB pages, but we see that a couple of seconds later it remaps the first 0xE000 pages (3.75GB) of memory using 4k IOTLB pages. We still have 2M IOTLB page mappings for memory above 4GB. Why would the kernel change the IOTLB page size from 2M to 4k? How can we make sure that all memory (except for some non-aligned bits) gets mapped using 2M IOTLB pages? As I mentioned, we are using huge-page memory backing for all our VMs. Any advice, also for debugging and further diagnosis, would be appreciated. Jan