From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FDA3C56201 for ; Thu, 19 Nov 2020 15:56:31 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D6F24246AA for ; Thu, 19 Nov 2020 15:56:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ipM+oEkI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D6F24246AA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:39996 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kfmIN-00026a-Li for qemu-devel@archiver.kernel.org; Thu, 19 Nov 2020 10:56:27 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:41502) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kfm2I-0007bm-6i for qemu-devel@nongnu.org; Thu, 19 Nov 2020 10:39:51 -0500 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:47379) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1kfm2E-0001AF-TL for qemu-devel@nongnu.org; Thu, 19 Nov 2020 10:39:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1605800383; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=OGJjBmAMVccGCD/44cFyB2b7ZSHWOW7Ler+oZW3HThc=; b=ipM+oEkIeNA9fgP/1udXWB+2/p3WaA30l/hiGgY/7o+a6B6oODCrDPOj1FWziMVgLmv1qP /HZNJk0hY/7+YCP9nrjQfS9kXAP6pvwestVXlZR50rt7acXAzj753YifgU9+y8iVWKpXos 3xq4rGorST63ZQlgPIum91M98beVDio= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-125-jZ4_qLRrPfuNL_iENiYcOA-1; Thu, 19 Nov 2020 10:39:41 -0500 X-MC-Unique: jZ4_qLRrPfuNL_iENiYcOA-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id BBEDD9CC05; Thu, 19 Nov 2020 15:39:39 +0000 (UTC) Received: from t480s.redhat.com (ovpn-114-158.ams2.redhat.com [10.36.114.158]) by smtp.corp.redhat.com (Postfix) with ESMTP id DFF4D5C1A1; Thu, 19 Nov 2020 15:39:18 +0000 (UTC) From: David Hildenbrand To: qemu-devel@nongnu.org Subject: [PATCH v1 0/9] virtio-mem: vfio support Date: Thu, 19 Nov 2020 16:39:09 +0100 Message-Id: <20201119153918.120976-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=david@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII" Received-SPF: pass client-ip=216.205.24.124; envelope-from=david@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/11/19 03:44:58 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] [fuzzy] X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Pankaj Gupta , Wei Yang , David Hildenbrand , "Michael S. Tsirkin" , "Dr. David Alan Gilbert" , Peter Xu , Luiz Capitulino , Auger Eric , Alex Williamson , teawater , Paolo Bonzini , Igor Mammedov , Marek Kedzierski Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This is the follow-up of: https://lkml.kernel.org/r/20200924160423.106747-1-david@redhat.com to make vfio and virtio-mem play together. The basic idea was the result of Alex brainstorming with me on how to tackle this. A virtio-mem device manages a memory region in guest physical address space, represented as a single (currently large) memory region in QEMU, mapped into system memory address space. Before the guest is allowed to use memory blocks, it must coordinate with the hypervisor (plug blocks). After a reboot, all memory is usually unplugged - when the guest comes up, it detects the virtio-mem device and selects memory blocks to plug (based on resize requests from the hypervisor). Memory hot(un)plug consists of (un)plugging memory blocks via a virtio-mem device (triggered by the guest). When unplugging blocks, we discard the memory - similar to memory balloon inflation. In contrast to memory ballooning, we always know which memory blocks a guest may actually use - especially during a reboot, after a crash, or after kexec (and during hibernation as well). Guests agreed to no access unplugged memory again, especially not via DMA. The issue with vfio is, that it cannot deal with random discards - for this reason, virtio-mem and vfio can currently only run mutually exclusive. Especially, vfio would currently map the whole memory region (with possible only little/no plugged blocks), resulting in all pages getting pinned and therefore resulting in a higher memory consumption than expected (turning virtio-mem basically useless in these environments). To make vfio work nicely with virtio-mem, we have to map only the plugged blocks, and map/unmap properly when plugging/unplugging blocks (including discarding of RAM when unplugging). We achieve that by using a new notifier mechanism that communicates changes. It's important to map memory in the granularity in which we could see unmaps again (-> virtio-mem block size) - so when e.g., plugging consecutive 100 MB with a block size of 2MB, we need 50 mappings. When unmapping, we can use a single vfio_unmap call for the applicable range. We expect that the block size of virtio-mem devices will be fairly large in the future (to not run out of mappings and to improve hot(un)plug performance), configured by the user, when used with vfio (e.g., 128MB, 1G, ...). More info regarding virtio-mem can be found at: https://virtio-mem.gitlab.io/ I'll add a guide for virtio-mem+vfio soonish. v1 is located at: git@github.com:davidhildenbrand/qemu.git virtio-mem-vfio-v1 RFC - v1: - VFIO migration code. Due to missing kernel support, I cannot really test if that part works. - Understand/test/document vIOMMU implications, also regarding migration - Nicer ram_block_discard_disable/require handling. - s/SparseRAMHandler/RamDiscardMgr/, refactorings, cleanups, documentation, testing, ... David Hildenbrand (9): memory: Introduce RamDiscardMgr for RAM memory regions virtio-mem: Factor out traversing unplugged ranges virtio-mem: Implement RamDiscardMgr interface vfio: Support for RamDiscardMgr in the !vIOMMU case vfio: Support for RamDiscardMgr in the vIOMMU case softmmu/physmem: Don't use atomic operations in ram_block_discard_(disable|require) softmmu/physmem: Extend ram_block_discard_(require|disable) by two discard types virtio-mem: Require only coordinated discards vfio: Disable only uncoordinated discards hw/vfio/common.c | 282 ++++++++++++++++++++++++++++- hw/virtio/virtio-mem.c | 319 ++++++++++++++++++++++++++++----- include/exec/memory.h | 243 ++++++++++++++++++++++++- include/hw/vfio/vfio-common.h | 12 ++ include/hw/virtio/virtio-mem.h | 3 + softmmu/memory.c | 22 +++ softmmu/physmem.c | 108 ++++++++--- 7 files changed, 913 insertions(+), 76 deletions(-) -- 2.26.2