* [PATCH net-next] virtio-net: avoid unnecessary sg initialzation
From: Jason Wang @ 2015-08-27 6:53 UTC (permalink / raw)
To: mst, virtualization, netdev, linux-kernel
Usually an skb does not have up to MAX_SKB_FRAGS frags. So no need to
initialize the unuse part of sg. This patch initialize the sg based on
the real number it will used:
- during xmit, it could be inferred from nr_frags and can_push.
- for small receive buffer, it will also be 2.
Cc: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
drivers/net/virtio_net.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 53f5660..c006ae4 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -540,7 +540,7 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
skb_put(skb, GOOD_PACKET_LEN);
hdr = skb_vnet_hdr(skb);
- sg_init_table(rq->sg, MAX_SKB_FRAGS + 2);
+ sg_init_table(rq->sg, 2);
sg_set_buf(rq->sg, hdr, vi->hdr_len);
skb_to_sgvec(skb, rq->sg + 1, 0, skb->len);
@@ -893,7 +893,7 @@ static int xmit_skb(struct send_queue *sq, struct sk_buff *skb)
if (vi->mergeable_rx_bufs)
hdr->num_buffers = 0;
- sg_init_table(sq->sg, MAX_SKB_FRAGS + 2);
+ sg_init_table(sq->sg, skb_shinfo(skb)->nr_frags + (can_push ? 1 : 2));
if (can_push) {
__skb_push(skb, hdr_len);
num_sg = skb_to_sgvec(skb, sq->sg, 0, skb->len);
--
2.1.4
^ permalink raw reply related
* Call for Papers - WorldCIST'15: 4td World Conference on Information Systems and Technologies
From: Maria Lemos @ 2015-08-27 12:20 UTC (permalink / raw)
To: virtualization
[-- Attachment #1: Type: text/plain, Size: 7165 bytes --]
WorldCIST'16 - 4th World Conference on Information Systems and Technologies
Recife, PE, Brazil
22th-24th of March 2016
http://www.aisti.eu/worldcist16/
-------------------------------------------
SCOPE
The WorldCist'16 - 4th World Conference on Information Systems and Technologies, to be held at Recife, PE, Brazil, 22 - 24 March 2016, is a global forum for researchers and practitioners to present and discuss the most recent innovations, trends, results, experiences and concerns in the several perspectives of Information Systems and Technologies.
We are pleased to invite you to submit your papers to WorldCist'16. All submissions will be reviewed on the basis of relevance, originality, importance and clarity.
THEMES
Submitted papers should be related with one or more of the main themes proposed for the Conference:
A) Information and Knowledge Management (IKM);
B) Organizational Models and Information Systems (OMIS);
C) Software and Systems Modeling (SSM);
D) Software Systems, Architectures, Applications and Tools (SSAAT);
E) Multimedia Systems and Applications (MSA);
F) Computer Networks, Mobility and Pervasive Systems (CNMPS);
G) Intelligent and Decision Support Systems (IDSS);
H) Big Data Analytics and Applications (BDAA);
I) Human-Computer Interaction (HCI);
J) Health Informatics (HIS);
K) Information Technologies in Education (ITE);
L) Information Technologies in Radiocommunications (ITR).
TYPES OF SUBMISSIONS AND DECISIONS
Four types of papers can be submitted:
- Full paper: Finished or consolidated R&D works, to be included in one of the Conference themes. These papers are assigned a 10-page limit.
- Short paper: Ongoing works with relevant preliminary results, open to discussion. These papers are assigned a 7-page limit.
-Poster paper: Initial work with relevant ideas, open to discussion. These papers are assigned to a 4-page limit.
- Company paper: Companies' papers that show practical experience, R & D, tools, etc., focused on some topics of the conference. These papers are assigned to a 4-page limit.
Submitted papers must comply with the format of Advances in Intelligent Systems and Computing Series (see Instructions for Authors at Springer Website or download a DOC example) be written in English, must not have been published before, not be under review for any other conference or publication and not include any information leading to the authors’ identification. Therefore, the authors’ names, affiliations and bibliographic references should not be included in the version for evaluation by the Program Committee. This information should only be included in the camera-ready version, saved in Word or Latex format and also in PDF format. These files must be accompanied by the Consent to Publication form filled out, in a ZIP file, and uploaded at the conference management system.
All papers will be subjected to a “double-blind review” by at least two members of the Program Committee.
Based on Program Committee evaluation, a paper can be rejected or accepted by the Conference Chairs. In the later case, it can be accepted as the type originally submitted or as another type. Thus, full papers can be accepted as short papers or poster papers only. Similarly, short papers can be accepted as poster papers only. In these cases, the authors will be allowed to maintain the original number of pages in the camera-ready version.
The authors of accepted poster papers must also build and print a poster to be exhibited during the Conference. This poster must follow an A1 or A2 vertical format. The Conference can includes Work Sessions where these posters are presented and orally discussed, with a 5 minute limit per poster.
The authors of accepted full papers will have 15 minutes to present their work in a Conference Work Session; approximately 5 minutes of discussion will follow each presentation. The authors of accepted short papers and company papers will have 11 minutes to present their work in a Conference Work Session; approximately 4 minutes of discussion will follow each presentation.
PUBLICATION AND INDEXING
To ensure that a full paper, short paper, poster paper or company paper is published in the Proceedings, at least one of the authors must be fully registered by the 27th of December 2015, and the paper must comply with the suggested layout and page-limit. Additionally, all recommended changes must be addressed by the authors before they submit the camera-ready version.
No more than one paper per registration will be published in the Conference Proceedings. An extra fee must be paid for publication of additional papers, with a maximum of one additional paper per registration.
Full and short papers will be published in Proceedings by Springer, in a book of Advances in Intelligent Systems and Computing series. Poster and company papers will be published by AISTI.
Published full and short papers will be submitted for indexation by ISI, EI-Compendex, SCOPUS and DBLP, among others, and will be available in the SpringerLink Digital Library.
The authors of the best selected papers will be invited to extend them for publication in international journals indexed by ISI/SCI, SCOPUS and DBLP, among others, such as:
- International Journal of Neural Systems (IF: 6.507)
- Integrated Computer-Aided Engineering (IF: 4.698)
- Computers in Human Behavior (IF: 2.694)
- Journal of Medical Systems (IF: 2.213)
- International Journal of Computer-Supported Collaborative Learning (IF: 1.841)
- Journal of Intelligent & Fuzzy Systems (IF: 1.812)
- Telemedicine and e-Health (IF: 1.668)
- International Journal of Information Management (IF: 1.550)
- Engineering Computations (IF: 1.495)
- Electronic Commerce Research and Applications (IF: 1.482)
- Telematics and Informatics (IF: 1.120)
- Journal of Evaluation in Clinical Practice (IF: 1.084)
- Ethics and Information Technology (IF: 1.021)
- Int. Journal of Computers Communications & Control (IF: 0.746)
- IET Software (IF: 0.595)
- Knowledge Management Research & Practice (IF: 0.554)
- AI Communications (IF: 0.547)
- Computing and Informatics (IF: 0.504)
- Universal Access in the Information Society (IF: 0.475)
- Journal of Global Information Management (IF: 0.424)
- Journal of Software Engineering and Knowledge Engineering (IF: 0.362)
- Journal of Internet Services and Applications (SJR: 0.88)
- Journal of Hospitality and Tourism Technology (SJR: 0.41)
- VINE - The Journal of Information and Knowledge Management Systems (SJR: 0.24)
- International Journal of Online Engineering (SJR: 0.21)
- Int. Journal of Emerging Technologies in Learning (SJR: 0.12)
- Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization
IMPORTANT DATES
Paper Submission: October 31, 2015
Notification of Acceptance: December 13, 2015
Payment of Registration, to ensure the inclusion of an accepted paper in the conference proceedings: December 27, 2015.
Camera-ready Submission: December 31, 2015
-
Maria Lemos
WorldCIST'16
http://www.aisti.eu/worldcist16/
[-- Attachment #2: Type: text/plain, Size: 183 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: [PATCH] virtio_mmio: add ACPI probing
From: Graeme Gregory @ 2015-08-27 14:55 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: linux-kernel, virtualization
In-Reply-To: <20150729211637-mutt-send-email-mst@redhat.com>
On Wed, Jul 29, 2015 at 09:17:08PM +0300, Michael S. Tsirkin wrote:
> On Tue, Jul 28, 2015 at 10:44:02AM +0100, Graeme Gregory wrote:
> > Added the match table and pointers for ACPI probing to the driver.
> >
> > This uses the same identifier for virt devices as being used for qemu
> > ARM64 ACPI support.
> >
> > http://git.linaro.org/people/shannon.zhao/qemu.git/commit/d0bf1955a3ecbab4b51d46f8c5dda02b7e14a17e
> >
> > Signed-off-by: Graeme Gregory <graeme.gregory@linaro.org>
>
> To summarize, let's wait a bit with this until QEMU 2.4
> with the relevant code is out, to make sure ID does not
> change.
>
It seems qemu 2.4 released with the LNRO0005 identifier.
Graeme
> > ---
> > drivers/virtio/virtio_mmio.c | 10 ++++++++++
> > 1 file changed, 10 insertions(+)
> >
> > diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
> > index 10189b5..f499d9d 100644
> > --- a/drivers/virtio/virtio_mmio.c
> > +++ b/drivers/virtio/virtio_mmio.c
> > @@ -58,6 +58,7 @@
> >
> > #define pr_fmt(fmt) "virtio-mmio: " fmt
> >
> > +#include <linux/acpi.h>
> > #include <linux/highmem.h>
> > #include <linux/interrupt.h>
> > #include <linux/io.h>
> > @@ -732,12 +733,21 @@ static struct of_device_id virtio_mmio_match[] = {
> > };
> > MODULE_DEVICE_TABLE(of, virtio_mmio_match);
> >
> > +#ifdef CONFIG_ACPI
> > +static const struct acpi_device_id virtio_mmio_acpi_match[] = {
> > + { "LNRO0005", },
> > + { }
> > +};
> > +MODULE_DEVICE_TABLE(acpi, virtio_mmio_acpi_match);
> > +#endif
> > +
> > static struct platform_driver virtio_mmio_driver = {
> > .probe = virtio_mmio_probe,
> > .remove = virtio_mmio_remove,
> > .driver = {
> > .name = "virtio-mmio",
> > .of_match_table = virtio_mmio_match,
> > + .acpi_match_table = ACPI_PTR(virtio_mmio_acpi_match),
> > },
> > };
> >
> > --
> > 2.1.4
^ permalink raw reply
* Re: [PATCH net-next] virtio-net: avoid unnecessary sg initialzation
From: David Miller @ 2015-08-27 22:52 UTC (permalink / raw)
To: jasowang; +Cc: netdev, virtualization, linux-kernel, mst
In-Reply-To: <1440658386-9544-1-git-send-email-jasowang@redhat.com>
From: Jason Wang <jasowang@redhat.com>
Date: Thu, 27 Aug 2015 14:53:06 +0800
> Usually an skb does not have up to MAX_SKB_FRAGS frags. So no need to
> initialize the unuse part of sg. This patch initialize the sg based on
> the real number it will used:
>
> - during xmit, it could be inferred from nr_frags and can_push.
> - for small receive buffer, it will also be 2.
>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
This looks fine, thanks Jason.
^ permalink raw reply
* Re: [PATCH] x86/paravirt: remove unused operation
From: Juergen Gross @ 2015-08-28 9:08 UTC (permalink / raw)
To: linux-kernel, jeremy, chrisw, akataria, rusty, virtualization
In-Reply-To: <1438862127-19674-1-git-send-email-jgross@suse.com>
Ping?
On 08/06/2015 01:55 PM, Juergen Gross wrote:
> Remove the paravirt operation "get_tsc_khz" as it is used nowhere.
>
> Signed-off-by: Juergen Gross <jgross@suse.com>
> ---
> arch/x86/include/asm/paravirt_types.h | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
> index a6b8f9f..5a18a66 100644
> --- a/arch/x86/include/asm/paravirt_types.h
> +++ b/arch/x86/include/asm/paravirt_types.h
> @@ -97,7 +97,6 @@ struct pv_lazy_ops {
> struct pv_time_ops {
> unsigned long long (*sched_clock)(void);
> unsigned long long (*steal_clock)(int cpu);
> - unsigned long (*get_tsc_khz)(void);
> };
>
> struct pv_cpu_ops {
>
^ permalink raw reply
* Re: [PATCH] x86/paravirt: remove unused operation
From: Rusty Russell @ 2015-08-31 0:05 UTC (permalink / raw)
To: Juergen Gross, linux-kernel, jeremy, chrisw, akataria,
virtualization; +Cc: x86
In-Reply-To: <55E02501.5030105@suse.com>
Juergen Gross <jgross@suse.com> writes:
> Ping?
Acked-by: Rusty Russell <rusty@rustcorp.com.au>
Cheers,
Rusty.
> On 08/06/2015 01:55 PM, Juergen Gross wrote:
>> Remove the paravirt operation "get_tsc_khz" as it is used nowhere.
>>
>> Signed-off-by: Juergen Gross <jgross@suse.com>
>> ---
>> arch/x86/include/asm/paravirt_types.h | 1 -
>> 1 file changed, 1 deletion(-)
>>
>> diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h
>> index a6b8f9f..5a18a66 100644
>> --- a/arch/x86/include/asm/paravirt_types.h
>> +++ b/arch/x86/include/asm/paravirt_types.h
>> @@ -97,7 +97,6 @@ struct pv_lazy_ops {
>> struct pv_time_ops {
>> unsigned long long (*sched_clock)(void);
>> unsigned long long (*steal_clock)(int cpu);
>> - unsigned long (*get_tsc_khz)(void);
>> };
>>
>> struct pv_cpu_ops {
>>
^ permalink raw reply
* Re: [PATCH 0/2] virtio_balloon: do not change memory amount visible via /proc/meminfo
From: Denis V. Lunev @ 2015-08-31 7:33 UTC (permalink / raw)
To: Michael S. Tsirkin; +Cc: virtualization
In-Reply-To: <1440020989-10824-1-git-send-email-den@openvz.org>
On 08/20/2015 12:49 AM, Denis V. Lunev wrote:
> Though there is a problem in this setup. The end-user and hosting provider
> have signed SLA agreement in which some amount of memory is guaranted for
> the guest. The good thing is that this memory will be given to the guest
> when the guest will really need it (f.e. with OOM in guest and with
> VIRTIO_BALLOON_F_DEFLATE_ON_OOM configuration flag set). The bad thing
> is that end-user does not know this.
>
> Balloon by default reduce the amount of memory exposed to the end-user
> each time when the page is stolen from guest or returned back by using
> adjust_managed_page_count and thus /proc/meminfo shows reduced amount
> of memory.
>
> Fortunately the solution is simple, we should just avoid to call
> adjust_managed_page_count with VIRTIO_BALLOON_F_DEFLATE_ON_OOM set.
>
> Please note that neither VMWare ballon nor HyperV balloon do not care
> about proper handling of adjust_managed_page_count at all.
>
> Signed-off-by: Denis V. Lunev <den@openvz.org>
> CC: Michael S. Tsirkin <mst@redhat.com>
ping
Michael, the issue is important for us. Can you pls look?
Den
^ permalink raw reply
* rfc: vhost user enhancements for vm2vm communication
From: Michael S. Tsirkin @ 2015-08-31 14:11 UTC (permalink / raw)
To: qemu-devel, virtualization, virtio-dev, opnfv-tech-discuss
Cc: Jan Kiszka, Claudio.Fontana
Hello!
During the KVM forum, we discussed supporting virtio on top
of ivshmem. I have considered it, and came up with an alternative
that has several advantages over that - please see below.
Comments welcome.
-----
Existing solutions to userspace switching between VMs on the
same host are vhost-user and ivshmem.
vhost-user works by mapping memory of all VMs being bridged into the
switch memory space.
By comparison, ivshmem works by exposing a shared region of memory to all VMs.
VMs are required to use this region to store packets. The switch only
needs access to this region.
Another difference between vhost-user and ivshmem surfaces when polling
is used. With vhost-user, the switch is required to handle
data movement between VMs, if using polling, this means that 1 host CPU
needs to be sacrificed for this task.
This is easiest to understand when one of the VMs is
used with VF pass-through. This can be schematically shown below:
+-- VM1 --------------+ +---VM2-----------+
| virtio-pci +-vhost-user-+ virtio-pci -- VF | -- VFIO -- IOMMU -- NIC
+---------------------+ +-----------------+
With ivshmem in theory communication can happen directly, with two VMs
polling the shared memory region.
I won't spend time listing advantages of vhost-user over ivshmem.
Instead, having identified two advantages of ivshmem over vhost-user,
below is a proposal to extend vhost-user to gain the advantages
of ivshmem.
1: virtio in guest can be extended to allow support
for IOMMUs. This provides guest with full flexibility
about memory which is readable or write able by each device.
By setting up a virtio device for each other VM we need to
communicate to, guest gets full control of its security, from
mapping all memory (like with current vhost-user) to only
mapping buffers used for networking (like ivshmem) to
transient mappings for the duration of data transfer only.
This also allows use of VFIO within guests, for improved
security.
vhost user would need to be extended to send the
mappings programmed by guest IOMMU.
2. qemu can be extended to serve as a vhost-user client:
remote VM mappings over the vhost-user protocol, and
map them into another VM's memory.
This mapping can take, for example, the form of
a BAR of a pci device, which I'll call here vhost-pci -
with bus address allowed
by VM1's IOMMU mappings being translated into
offsets within this BAR within VM2's physical
memory space.
Since the translation can be a simple one, VM2
can perform it within its vhost-pci device driver.
While this setup would be the most useful with polling,
VM1's ioeventfd can also be mapped to
another VM2's irqfd, and vice versa, such that VMs
can trigger interrupts to each other without need
for a helper thread on the host.
The resulting channel might look something like the following:
+-- VM1 --------------+ +---VM2-----------+
| virtio-pci -- iommu +--+ vhost-pci -- VF | -- VFIO -- IOMMU -- NIC
+---------------------+ +-----------------+
comparing the two diagrams, a vhost-user thread on the host is
no longer required, reducing the host CPU utilization when
polling is active. At the same time, VM2 can not access all of VM1's
memory - it is limited by the iommu configuration setup by VM1.
Advantages over ivshmem:
- more flexibility, endpoint VMs do not have to place data at any
specific locations to use the device, in practice this likely
means less data copies.
- better standardization/code reuse
virtio changes within guests would be fairly easy to implement
and would also benefit other backends, besides vhost-user
standard hotplug interfaces can be used to add and remove these
channels as VMs are added or removed.
- migration support
It's easy to implement since ownership of memory is well defined.
For example, during migration VM2 can notify hypervisor of VM1
by updating dirty bitmap each time is writes into VM1 memory.
Thanks,
--
MST
^ permalink raw reply
* CFP: 8th Workshop on Many-Task Computing on Clouds, Grids, and Supercomputers (MTAGS) 2015 -- co-located with IEEE/ACM Supercomputing/SC 2015
From: Ioan Raicu @ 2015-08-31 14:53 UTC (permalink / raw)
To: publicity, virtualization, micro_publicity, performance,
sigplan-l
Call for Papers
---------------------------------------------------------------------------------------
The 8th Workshop on Many-Task Computing on Clouds, Grids, and
Supercomputers (MTAGS) 2015
http://datasys.cs.iit.edu/events/MTAGS15/
---------------------------------------------------------------------------------------
November 15th, 2015
Austin, Texas, USA
Co-located with with IEEE/ACM International Conference for
High Performance Computing, Networking, Storage and Analysis (SC15)
=======================================================================================
The 8th workshop on Many-Task Computing on Clouds, Grids, and
Supercomputers (MTAGS) will provide the scientific community a dedicated
forum for presenting new research, development, and deployment efforts
of large-scale many-task computing (MTC) applications on large scale
clusters, Grids, Supercomputers, and Cloud Computing infrastructure.
MTC, the theme of the workshop encompasses loosely coupled applications,
which are generally composed of many tasks (both independent and
dependent tasks) to achieve some larger application goal. This workshop
will cover challenges that can hamper efficiency and utilization in
running applications on large-scale systems, such as local resource
manager scalability and granularity, efficient utilization of raw
hardware, parallel file system contention and scalability, data
management, I/O management, reliability at scale, and application
scalability. We welcome paper submissions on all theoretical,
simulations, and systems topics related to MTC, but we give special
consideration to papers addressing petascale to exascale challenges.
Papers will be peer-reviewed for novelty, scientific merit, and scope
for the workshop. The workshop will be co-located with the IEEE/ACM
Supercomputing 2015 Conference in Austin Texas on November 15th, 2015.
For more information, please see http://datasys.cs.iit.edu/events/MTAGS15/.
For more information on past workshops, please see MTAGS14, MTAGS13,
MTAGS12, MTAGS11, MTAGS10, MTAGS09, and MTAGS08. We also ran a Special
Issue on Many-Task Computing in the IEEE Transactions on Parallel and
Distributed Systems (TPDS) which appeared in June 2011; the proceedings
can be found online at
http://www.computer.org/portal/web/csdl/abs/trans/td/2011/06/ttd201106toc.htm.
We are also currently assembling a new special issue in the IEEE
Transaction on Cloud Computing on Many-Task Computing in the Cloud, see
http://datasys.cs.iit.edu/events/TCC-MTC15/index.html for more
information. We, the workshop organizers, also published a highly
relevant paper that defines Many-Task Computing which was published in
MTAGS08,titled “Many-Task Computing for Grids and Supercomputers”; we
encourage potential authors to read this paper, and to clearly
articulate in your paper submissions how your papers are related to
Many-Task Computing.
Topics
---------------------------------------------------------------------------------------
We invite the submission of original work that is related to the topics
below. The papers should be 6 pages, including all figures and
references.We aim to cover topics related to Many-Task Computing on each
of the three major distributed systems paradigms, Cloud Computing, Grid
Computing and Supercomputing. Topics of interest include:
* Compute Resource Management
* Scheduling
* Job execution frameworks
* Local resource manager extensions
* Performance evaluation of resource managers in use on large scale
systems
* Dynamic resource provisioning
* Techniques to manage many-core resources and/or GPUs
* Challenges and opportunities in running many-task workloads on HPC
systems
* Challenges and opportunities in running many-task workloads on
Cloud Computing infrastructure
* Storage architectures and implementations
* Distributed file systems
* Parallel file systems
* Distributed meta-data management
* Content distribution systems for large data
* Data caching frameworks and techniques
* Data management within and across data centers
* Data-aware scheduling
* Data-intensive computing applications
* Eventual-consistency storage usage and management
* Programming models and tools
* Map-reduce and its generalizations
* Many-task computing middleware and applications
* Parallel programming frameworks
* Ensemble MPI techniques and frameworks
* Service-oriented science applications
* Large-Scale Workflow Systems
* Workflow system performance and scalability analysis
* Scalability of workflow systems
* Workflow infrastructure and e-Science middleware
* Programming Paradigms and Models
* Large-Scale Many-Task Applications
* High-throughput computing (HTC) applications
* Data-intensive applications
* Quasi-supercomputing applications, deployments, and experiences
* Performance Evaluation
* Performance evaluation
* Real systems
* Simulations
* Reliability of large systems
Paper Submission and Publication
---------------------------------------------------------------------------------------
Authors are invited to submit papers with unpublished, original work of
not more than 6 pages of double column text using single spaced 10 point
size on 8.5 x 11 inch pages, as per ACM 8.5 x 11 manuscript guidelines;
document templates can be found at
http://www.acm.org/sigs/publications/proceedings-templates. The final 6
page papers in PDF format must be submitted online at
https://cmt.research.microsoft.com/MTAGS2015/ before the deadline.
Papers will be peer-reviewed for novelty, scientific merit, and scope
for the workshop. Submission implies the willingness of at least one of
the authors to register and present the paper. For more information,
please see http://datasys.cs.iit.edu/events/MTAGS15/.
Important Dates
---------------------------------------------------------------------------------------
* Abstracts due: September 4th, 2015
* Full paper due: September 11th, 2015
* Acceptance notification: October 2nd, 2015
* Early Registration deadline: October 15th, 2015
* Final papers due: November 6th, 2015
* Workshop date: November 15th, 2015
Committee Members
---------------------------------------------------------------------------------------
Workshop Chairs
* Ioan Raicu, Illinois Institute of Technology & Argonne National Laboratory
* Justin Wozniak, University of Chicago & Argonne National Laboratory
* Ian Foster, University of Chicago & Argonne National Laboratory
* Yong Zhao, University of Electronic Science and Technology of China
Steering Committee
* David Abramson, Monash University, Australia
* Jack Dongarra, University of Tennessee, USA
* Geoffrey Fox, Indiana University, USA
* Manish Parashar, Rutgers University, USA
* Marc Snir, University of Illinois at Urbana Champaign, USA
* Xian-He Sun, Illinois Institute of Technology, USA
* Weimin Zheng, Tsinghua University, China
Technical Committee
* Hasan Abbasi, Oak Ridge National Labs, USA
* Tim Armstrong, University of Chicago, USA
* Roger Barga, Microsoft, USA
* Mihai Budiu, Microsoft Research, USA
* Rajkumar Buyya University of Melbourne, Australia
* Kyle Chard, University of Chicago, USA
* Evangelinos Constantinos, Massachusetts Institute of Technology, USA
* Catalin Dumitrescu, Fermi National Labs, USA
* Haryadi Gunawi, University of Chicago, USA
* Indranil Gupta, University of Illinois at Urbana Champaign, USA
* Alexandru Iosup, Delft University of Technology, Netherlands
* Florin Isaila, Universidad Carlos III de Madrid, Spain & Argonne
National Laboratory, USA
* Kamil Iskra, Argonne National Laboratory, USA
* Daniel S. Katz, University of Chicago, USA
* Jik-Soo Kim, Kristi, Korea
* Scott A. Klasky, Oak Ridge National Labs, USA
* Mike Lang, Los Alamos National Laboratory, USA
* Tonglin Li, Illinois Institute of Technology, USA
* Chris Moretti, Princeton University, USA
* David O'Hallaron, Carnegie Mellon University, USA
* Marlon Pierce, Indiana University, USA
* Judy Qiu, Indiana University, USA
* Lavanya Ramakrishnan, Lawrence Berkeley National Laboratory, USA
* Matei Ripeanu, University of British Columbia, Canada
* Iman Sadooghi, Illinois Institute of Technology, USA
* Wei Tang, Argonne National Laboratory, USA
* Edward Walker, Whitworth University, USA
* Ke Wang, Illinois Institute of Technology, USA
* Matthew Woitaszek, Walmart Labs, USA
* Rich Wolski, University of California, Santa Barbara, USA
* Zhifeng Yun, University of Houston, USA
* Zhao Zhang, University of California at Berkeley, USA
* Ziming Zheng, University of Chicago, USA
--
=================================================================
Ioan Raicu, Ph.D.
Assistant Professor, Illinois Institute of Technology (IIT)
Guest Research Faculty, Argonne National Laboratory (ANL)
=================================================================
Data-Intensive Distributed Systems Laboratory, CS/IIT
Distributed Systems Laboratory, MCS/ANL
=================================================================
Editor: IEEE TCC, Springer Cluster, Springer JoCCASA
Chair: IEEE/ACM MTAGS, ACM ScienceCloud
=================================================================
Office: 1-312-567-5704
Email: iraicu@cs.iit.edu
Web: http://www.cs.iit.edu/~iraicu/
Web: http://datasys.cs.iit.edu/
LinkedIn: http://www.linkedin.com/in/ioanraicu
Google: http://scholar.google.com/citations?user=jE73HYAAAAAJ
=================================================================
=================================================================
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Nakajima, Jun @ 2015-08-31 18:35 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, Jan Kiszka, Claudio.Fontana, qemu-devel,
Linux Virtualization, opnfv-tech-discuss
In-Reply-To: <20150831160655-mutt-send-email-mst@redhat.com>
On Mon, Aug 31, 2015 at 7:11 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> Hello!
> During the KVM forum, we discussed supporting virtio on top
> of ivshmem. I have considered it, and came up with an alternative
> that has several advantages over that - please see below.
> Comments welcome.
Hi Michael,
I like this, and it should be able to achieve what I presented at KVM
Forum (vhost-user-shmem).
Comments below.
>
> -----
>
> Existing solutions to userspace switching between VMs on the
> same host are vhost-user and ivshmem.
>
> vhost-user works by mapping memory of all VMs being bridged into the
> switch memory space.
>
> By comparison, ivshmem works by exposing a shared region of memory to all VMs.
> VMs are required to use this region to store packets. The switch only
> needs access to this region.
>
> Another difference between vhost-user and ivshmem surfaces when polling
> is used. With vhost-user, the switch is required to handle
> data movement between VMs, if using polling, this means that 1 host CPU
> needs to be sacrificed for this task.
>
> This is easiest to understand when one of the VMs is
> used with VF pass-through. This can be schematically shown below:
>
> +-- VM1 --------------+ +---VM2-----------+
> | virtio-pci +-vhost-user-+ virtio-pci -- VF | -- VFIO -- IOMMU -- NIC
> +---------------------+ +-----------------+
>
>
> With ivshmem in theory communication can happen directly, with two VMs
> polling the shared memory region.
>
>
> I won't spend time listing advantages of vhost-user over ivshmem.
> Instead, having identified two advantages of ivshmem over vhost-user,
> below is a proposal to extend vhost-user to gain the advantages
> of ivshmem.
>
>
> 1: virtio in guest can be extended to allow support
> for IOMMUs. This provides guest with full flexibility
> about memory which is readable or write able by each device.
I assume that you meant VFIO only for virtio by "use of VFIO". To get
VFIO working for general direct-I/O (including VFs) in guests, as you
know, we need to virtualize IOMMU (e.g. VT-d) and the interrupt
remapping table on x86 (i.e. nested VT-d).
> By setting up a virtio device for each other VM we need to
> communicate to, guest gets full control of its security, from
> mapping all memory (like with current vhost-user) to only
> mapping buffers used for networking (like ivshmem) to
> transient mappings for the duration of data transfer only.
And I think that we can use VMFUNC to have such transient mappings.
> This also allows use of VFIO within guests, for improved
> security.
>
> vhost user would need to be extended to send the
> mappings programmed by guest IOMMU.
Right. We need to think about cases where other VMs (VM3, etc.) join
the group or some existing VM leaves.
PCI hot-plug should work there (as you point out at "Advantages over
ivshmem" below).
>
> 2. qemu can be extended to serve as a vhost-user client:
> remote VM mappings over the vhost-user protocol, and
> map them into another VM's memory.
> This mapping can take, for example, the form of
> a BAR of a pci device, which I'll call here vhost-pci -
> with bus address allowed
> by VM1's IOMMU mappings being translated into
> offsets within this BAR within VM2's physical
> memory space.
I think it's sensible.
>
> Since the translation can be a simple one, VM2
> can perform it within its vhost-pci device driver.
>
> While this setup would be the most useful with polling,
> VM1's ioeventfd can also be mapped to
> another VM2's irqfd, and vice versa, such that VMs
> can trigger interrupts to each other without need
> for a helper thread on the host.
>
>
> The resulting channel might look something like the following:
>
> +-- VM1 --------------+ +---VM2-----------+
> | virtio-pci -- iommu +--+ vhost-pci -- VF | -- VFIO -- IOMMU -- NIC
> +---------------------+ +-----------------+
>
> comparing the two diagrams, a vhost-user thread on the host is
> no longer required, reducing the host CPU utilization when
> polling is active. At the same time, VM2 can not access all of VM1's
> memory - it is limited by the iommu configuration setup by VM1.
>
>
> Advantages over ivshmem:
>
> - more flexibility, endpoint VMs do not have to place data at any
> specific locations to use the device, in practice this likely
> means less data copies.
> - better standardization/code reuse
> virtio changes within guests would be fairly easy to implement
> and would also benefit other backends, besides vhost-user
> standard hotplug interfaces can be used to add and remove these
> channels as VMs are added or removed.
> - migration support
> It's easy to implement since ownership of memory is well defined.
> For example, during migration VM2 can notify hypervisor of VM1
> by updating dirty bitmap each time is writes into VM1 memory.
Also, the ivshmem functionality could be implemented by this proposal:
- vswitch (or some VM) allocates memory regions in its address space, and
- it sets up that IOMMU mappings on the VMs be translated into the regions
>
> Thanks,
>
> --
> MST
> _______________________________________________
> Virtualization mailing list
> Virtualization@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/virtualization
--
Jun
Intel Open Source Technology Center
^ permalink raw reply
* RE: [Qemu-devel] rfc: vhost user enhancements for vm2vm communication
From: Varun Sethi @ 2015-09-01 3:03 UTC (permalink / raw)
To: Nakajima, Jun, Michael S. Tsirkin
Cc: virtio-dev@lists.oasis-open.org, Jan Kiszka,
Claudio.Fontana@huawei.com, qemu-devel@nongnu.org,
Linux Virtualization, opnfv-tech-discuss@lists.opnfv.org
In-Reply-To: <CAL54oT2udFxDnY4jwFb1mZEEZ7vQzco7XL2QiDNv-kYQSVFH1Q@mail.gmail.com>
Hi Michael,
When you talk about VFIO in guest, is it with a purely emulated IOMMU in Qemu?
Also, I am not clear on the following points:
1. How transient memory would be mapped using BAR in the backend VM
2. How would the backend VM update the dirty page bitmap for the frontend VM
Regards
Varun
> -----Original Message-----
> From: qemu-devel-bounces+varun.sethi=freescale.com@nongnu.org
> [mailto:qemu-devel-bounces+varun.sethi=freescale.com@nongnu.org] On
> Behalf Of Nakajima, Jun
> Sent: Monday, August 31, 2015 1:36 PM
> To: Michael S. Tsirkin
> Cc: virtio-dev@lists.oasis-open.org; Jan Kiszka;
> Claudio.Fontana@huawei.com; qemu-devel@nongnu.org; Linux
> Virtualization; opnfv-tech-discuss@lists.opnfv.org
> Subject: Re: [Qemu-devel] rfc: vhost user enhancements for vm2vm
> communication
>
> On Mon, Aug 31, 2015 at 7:11 AM, Michael S. Tsirkin <mst@redhat.com>
> wrote:
> > Hello!
> > During the KVM forum, we discussed supporting virtio on top of
> > ivshmem. I have considered it, and came up with an alternative that
> > has several advantages over that - please see below.
> > Comments welcome.
>
> Hi Michael,
>
> I like this, and it should be able to achieve what I presented at KVM Forum
> (vhost-user-shmem).
> Comments below.
>
> >
> > -----
> >
> > Existing solutions to userspace switching between VMs on the same host
> > are vhost-user and ivshmem.
> >
> > vhost-user works by mapping memory of all VMs being bridged into the
> > switch memory space.
> >
> > By comparison, ivshmem works by exposing a shared region of memory to
> all VMs.
> > VMs are required to use this region to store packets. The switch only
> > needs access to this region.
> >
> > Another difference between vhost-user and ivshmem surfaces when
> > polling is used. With vhost-user, the switch is required to handle
> > data movement between VMs, if using polling, this means that 1 host
> > CPU needs to be sacrificed for this task.
> >
> > This is easiest to understand when one of the VMs is used with VF
> > pass-through. This can be schematically shown below:
> >
> > +-- VM1 --------------+ +---VM2-----------+
> > | virtio-pci +-vhost-user-+ virtio-pci -- VF | -- VFIO -- IOMMU -- NIC
> > +---------------------+ +-----------------+
> >
> >
> > With ivshmem in theory communication can happen directly, with two VMs
> > polling the shared memory region.
> >
> >
> > I won't spend time listing advantages of vhost-user over ivshmem.
> > Instead, having identified two advantages of ivshmem over vhost-user,
> > below is a proposal to extend vhost-user to gain the advantages of
> > ivshmem.
> >
> >
> > 1: virtio in guest can be extended to allow support for IOMMUs. This
> > provides guest with full flexibility about memory which is readable or
> > write able by each device.
>
> I assume that you meant VFIO only for virtio by "use of VFIO". To get VFIO
> working for general direct-I/O (including VFs) in guests, as you know, we
> need to virtualize IOMMU (e.g. VT-d) and the interrupt remapping table on
> x86 (i.e. nested VT-d).
>
> > By setting up a virtio device for each other VM we need to communicate
> > to, guest gets full control of its security, from mapping all memory
> > (like with current vhost-user) to only mapping buffers used for
> > networking (like ivshmem) to transient mappings for the duration of
> > data transfer only.
>
> And I think that we can use VMFUNC to have such transient mappings.
>
> > This also allows use of VFIO within guests, for improved security.
> >
> > vhost user would need to be extended to send the mappings programmed
> > by guest IOMMU.
>
> Right. We need to think about cases where other VMs (VM3, etc.) join the
> group or some existing VM leaves.
> PCI hot-plug should work there (as you point out at "Advantages over
> ivshmem" below).
>
> >
> > 2. qemu can be extended to serve as a vhost-user client:
> > remote VM mappings over the vhost-user protocol, and map them into
> > another VM's memory.
> > This mapping can take, for example, the form of a BAR of a pci device,
> > which I'll call here vhost-pci - with bus address allowed by VM1's
> > IOMMU mappings being translated into offsets within this BAR within
> > VM2's physical memory space.
>
> I think it's sensible.
>
> >
> > Since the translation can be a simple one, VM2 can perform it within
> > its vhost-pci device driver.
> >
> > While this setup would be the most useful with polling, VM1's
> > ioeventfd can also be mapped to another VM2's irqfd, and vice versa,
> > such that VMs can trigger interrupts to each other without need for a
> > helper thread on the host.
> >
> >
> > The resulting channel might look something like the following:
> >
> > +-- VM1 --------------+ +---VM2-----------+
> > | virtio-pci -- iommu +--+ vhost-pci -- VF | -- VFIO -- IOMMU -- NIC
> > +---------------------+ +-----------------+
> >
> > comparing the two diagrams, a vhost-user thread on the host is no
> > longer required, reducing the host CPU utilization when polling is
> > active. At the same time, VM2 can not access all of VM1's memory - it
> > is limited by the iommu configuration setup by VM1.
> >
> >
> > Advantages over ivshmem:
> >
> > - more flexibility, endpoint VMs do not have to place data at any
> > specific locations to use the device, in practice this likely
> > means less data copies.
> > - better standardization/code reuse
> > virtio changes within guests would be fairly easy to implement
> > and would also benefit other backends, besides vhost-user
> > standard hotplug interfaces can be used to add and remove these
> > channels as VMs are added or removed.
> > - migration support
> > It's easy to implement since ownership of memory is well defined.
> > For example, during migration VM2 can notify hypervisor of VM1
> > by updating dirty bitmap each time is writes into VM1 memory.
>
> Also, the ivshmem functionality could be implemented by this proposal:
> - vswitch (or some VM) allocates memory regions in its address space, and
> - it sets up that IOMMU mappings on the VMs be translated into the regions
>
> >
> > Thanks,
> >
> > --
> > MST
> > _______________________________________________
> > Virtualization mailing list
> > Virtualization@lists.linux-foundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/virtualization
>
>
> --
> Jun
> Intel Open Source Technology Center
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Jan Kiszka @ 2015-09-01 7:35 UTC (permalink / raw)
To: Michael S. Tsirkin, qemu-devel, virtualization, virtio-dev,
opnfv-tech-discuss
Cc: Varun Sethi, Claudio.Fontana
In-Reply-To: <20150831160655-mutt-send-email-mst@redhat.com>
On 2015-08-31 16:11, Michael S. Tsirkin wrote:
> Hello!
> During the KVM forum, we discussed supporting virtio on top
> of ivshmem.
No, not on top of ivshmem. On top of shared memory. Our model is
different from the simplistic ivshmem.
> I have considered it, and came up with an alternative
> that has several advantages over that - please see below.
> Comments welcome.
>
> -----
>
> Existing solutions to userspace switching between VMs on the
> same host are vhost-user and ivshmem.
>
> vhost-user works by mapping memory of all VMs being bridged into the
> switch memory space.
>
> By comparison, ivshmem works by exposing a shared region of memory to all VMs.
> VMs are required to use this region to store packets. The switch only
> needs access to this region.
>
> Another difference between vhost-user and ivshmem surfaces when polling
> is used. With vhost-user, the switch is required to handle
> data movement between VMs, if using polling, this means that 1 host CPU
> needs to be sacrificed for this task.
>
> This is easiest to understand when one of the VMs is
> used with VF pass-through. This can be schematically shown below:
>
> +-- VM1 --------------+ +---VM2-----------+
> | virtio-pci +-vhost-user-+ virtio-pci -- VF | -- VFIO -- IOMMU -- NIC
> +---------------------+ +-----------------+
>
>
> With ivshmem in theory communication can happen directly, with two VMs
> polling the shared memory region.
>
>
> I won't spend time listing advantages of vhost-user over ivshmem.
> Instead, having identified two advantages of ivshmem over vhost-user,
> below is a proposal to extend vhost-user to gain the advantages
> of ivshmem.
>
>
> 1: virtio in guest can be extended to allow support
> for IOMMUs. This provides guest with full flexibility
> about memory which is readable or write able by each device.
> By setting up a virtio device for each other VM we need to
> communicate to, guest gets full control of its security, from
> mapping all memory (like with current vhost-user) to only
> mapping buffers used for networking (like ivshmem) to
> transient mappings for the duration of data transfer only.
> This also allows use of VFIO within guests, for improved
> security.
>
> vhost user would need to be extended to send the
> mappings programmed by guest IOMMU.
>
> 2. qemu can be extended to serve as a vhost-user client:
> remote VM mappings over the vhost-user protocol, and
> map them into another VM's memory.
> This mapping can take, for example, the form of
> a BAR of a pci device, which I'll call here vhost-pci -
> with bus address allowed
> by VM1's IOMMU mappings being translated into
> offsets within this BAR within VM2's physical
> memory space.
>
> Since the translation can be a simple one, VM2
> can perform it within its vhost-pci device driver.
>
> While this setup would be the most useful with polling,
> VM1's ioeventfd can also be mapped to
> another VM2's irqfd, and vice versa, such that VMs
> can trigger interrupts to each other without need
> for a helper thread on the host.
>
>
> The resulting channel might look something like the following:
>
> +-- VM1 --------------+ +---VM2-----------+
> | virtio-pci -- iommu +--+ vhost-pci -- VF | -- VFIO -- IOMMU -- NIC
> +---------------------+ +-----------------+
>
> comparing the two diagrams, a vhost-user thread on the host is
> no longer required, reducing the host CPU utilization when
> polling is active. At the same time, VM2 can not access all of VM1's
> memory - it is limited by the iommu configuration setup by VM1.
>
>
> Advantages over ivshmem:
>
> - more flexibility, endpoint VMs do not have to place data at any
> specific locations to use the device, in practice this likely
> means less data copies.
> - better standardization/code reuse
> virtio changes within guests would be fairly easy to implement
> and would also benefit other backends, besides vhost-user
> standard hotplug interfaces can be used to add and remove these
> channels as VMs are added or removed.
> - migration support
> It's easy to implement since ownership of memory is well defined.
> For example, during migration VM2 can notify hypervisor of VM1
> by updating dirty bitmap each time is writes into VM1 memory.
>
> Thanks,
>
This sounds like a different interface to a concept very similar to
Xen's grant table, no? Well, there might be benefits for some use cases,
for ours this is too dynamic, in fact. We'd like to avoid remappings
during runtime controlled by guest activities, which is clearly required
for this model.
Another shortcoming: If VM1 does not trust (security or safety-wise) VM2
while preparing a message for it, it has to keep the buffer invisible
for VM2 until it is completed and signed, hashed etc. That means it has
to reprogram the IOMMU frequently. With the concept we discussed at KVM
Forum, there would be shared memory mapped read-only to VM2 while being
R/W for VM1. That would resolve this issue without the need for costly
remappings.
Leaving all the implementation and interface details aside, this
discussion is first of all about two fundamentally different approaches:
static shared memory windows vs. dynamically remapped shared windows (a
third one would be copying in the hypervisor, but I suppose we all agree
that the whole exercise is about avoiding that). Which way do we want or
have to go?
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Michael S. Tsirkin @ 2015-09-01 8:01 UTC (permalink / raw)
To: Jan Kiszka
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <55E55539.4030008@siemens.com>
On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
> On 2015-08-31 16:11, Michael S. Tsirkin wrote:
> > Hello!
> > During the KVM forum, we discussed supporting virtio on top
> > of ivshmem.
>
> No, not on top of ivshmem. On top of shared memory. Our model is
> different from the simplistic ivshmem.
>
> > I have considered it, and came up with an alternative
> > that has several advantages over that - please see below.
> > Comments welcome.
> >
> > -----
> >
> > Existing solutions to userspace switching between VMs on the
> > same host are vhost-user and ivshmem.
> >
> > vhost-user works by mapping memory of all VMs being bridged into the
> > switch memory space.
> >
> > By comparison, ivshmem works by exposing a shared region of memory to all VMs.
> > VMs are required to use this region to store packets. The switch only
> > needs access to this region.
> >
> > Another difference between vhost-user and ivshmem surfaces when polling
> > is used. With vhost-user, the switch is required to handle
> > data movement between VMs, if using polling, this means that 1 host CPU
> > needs to be sacrificed for this task.
> >
> > This is easiest to understand when one of the VMs is
> > used with VF pass-through. This can be schematically shown below:
> >
> > +-- VM1 --------------+ +---VM2-----------+
> > | virtio-pci +-vhost-user-+ virtio-pci -- VF | -- VFIO -- IOMMU -- NIC
> > +---------------------+ +-----------------+
> >
> >
> > With ivshmem in theory communication can happen directly, with two VMs
> > polling the shared memory region.
> >
> >
> > I won't spend time listing advantages of vhost-user over ivshmem.
> > Instead, having identified two advantages of ivshmem over vhost-user,
> > below is a proposal to extend vhost-user to gain the advantages
> > of ivshmem.
> >
> >
> > 1: virtio in guest can be extended to allow support
> > for IOMMUs. This provides guest with full flexibility
> > about memory which is readable or write able by each device.
> > By setting up a virtio device for each other VM we need to
> > communicate to, guest gets full control of its security, from
> > mapping all memory (like with current vhost-user) to only
> > mapping buffers used for networking (like ivshmem) to
> > transient mappings for the duration of data transfer only.
> > This also allows use of VFIO within guests, for improved
> > security.
> >
> > vhost user would need to be extended to send the
> > mappings programmed by guest IOMMU.
> >
> > 2. qemu can be extended to serve as a vhost-user client:
> > remote VM mappings over the vhost-user protocol, and
> > map them into another VM's memory.
> > This mapping can take, for example, the form of
> > a BAR of a pci device, which I'll call here vhost-pci -
> > with bus address allowed
> > by VM1's IOMMU mappings being translated into
> > offsets within this BAR within VM2's physical
> > memory space.
> >
> > Since the translation can be a simple one, VM2
> > can perform it within its vhost-pci device driver.
> >
> > While this setup would be the most useful with polling,
> > VM1's ioeventfd can also be mapped to
> > another VM2's irqfd, and vice versa, such that VMs
> > can trigger interrupts to each other without need
> > for a helper thread on the host.
> >
> >
> > The resulting channel might look something like the following:
> >
> > +-- VM1 --------------+ +---VM2-----------+
> > | virtio-pci -- iommu +--+ vhost-pci -- VF | -- VFIO -- IOMMU -- NIC
> > +---------------------+ +-----------------+
> >
> > comparing the two diagrams, a vhost-user thread on the host is
> > no longer required, reducing the host CPU utilization when
> > polling is active. At the same time, VM2 can not access all of VM1's
> > memory - it is limited by the iommu configuration setup by VM1.
> >
> >
> > Advantages over ivshmem:
> >
> > - more flexibility, endpoint VMs do not have to place data at any
> > specific locations to use the device, in practice this likely
> > means less data copies.
> > - better standardization/code reuse
> > virtio changes within guests would be fairly easy to implement
> > and would also benefit other backends, besides vhost-user
> > standard hotplug interfaces can be used to add and remove these
> > channels as VMs are added or removed.
> > - migration support
> > It's easy to implement since ownership of memory is well defined.
> > For example, during migration VM2 can notify hypervisor of VM1
> > by updating dirty bitmap each time is writes into VM1 memory.
> >
> > Thanks,
> >
>
> This sounds like a different interface to a concept very similar to
> Xen's grant table, no?
Yes in a sense that grant tables are also memory sharing and
include permissions.
But we are emulating an IOMMU, and keep the PV part
as simple as possible (e.g. offset within BAR)
without attaching any policy to it.
Xen is fundamentally a PV interface.
> Well, there might be benefits for some use cases,
> for ours this is too dynamic, in fact. We'd like to avoid remappings
> during runtime controlled by guest activities, which is clearly required
> for this model.
The dynamic part is up to the guest. For example, userspace pmd within guest would
create mostly static mappings using VFIO.
> Another shortcoming: If VM1 does not trust (security or safety-wise) VM2
> while preparing a message for it, it has to keep the buffer invisible
> for VM2 until it is completed and signed, hashed etc. That means it has
> to reprogram the IOMMU frequently. With the concept we discussed at KVM
> Forum, there would be shared memory mapped read-only to VM2 while being
> R/W for VM1. That would resolve this issue without the need for costly
> remappings.
IOMMU allows read-only mappings too. It's all up to the guest.
> Leaving all the implementation and interface details aside, this
> discussion is first of all about two fundamentally different approaches:
> static shared memory windows vs. dynamically remapped shared windows (a
> third one would be copying in the hypervisor, but I suppose we all agree
> that the whole exercise is about avoiding that). Which way do we want or
> have to go?
>
> Jan
Dynamic is a superset of static: you can always make it static if you
wish. Static has the advantage of simplicity, but that's lost once you
realize you need to invent interfaces to make it work. Since we can use
existing IOMMU interfaces for the dynamic one, what's the disadvantage?
Let me put it another way: any security model you come up with
should also be useful for bare-metal OS isolation from a device.
That's a useful test for checking whether whatever we come
up with makes sense, and it's much better than inventing our own.
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Michael S. Tsirkin @ 2015-09-01 8:17 UTC (permalink / raw)
To: Nakajima, Jun
Cc: virtio-dev, Jan Kiszka, Claudio.Fontana, qemu-devel,
Linux Virtualization, opnfv-tech-discuss
In-Reply-To: <CAL54oT2udFxDnY4jwFb1mZEEZ7vQzco7XL2QiDNv-kYQSVFH1Q@mail.gmail.com>
On Mon, Aug 31, 2015 at 11:35:55AM -0700, Nakajima, Jun wrote:
> On Mon, Aug 31, 2015 at 7:11 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > Hello!
> > During the KVM forum, we discussed supporting virtio on top
> > of ivshmem. I have considered it, and came up with an alternative
> > that has several advantages over that - please see below.
> > Comments welcome.
>
> Hi Michael,
>
> I like this, and it should be able to achieve what I presented at KVM
> Forum (vhost-user-shmem).
> Comments below.
>
> >
> > -----
> >
> > Existing solutions to userspace switching between VMs on the
> > same host are vhost-user and ivshmem.
> >
> > vhost-user works by mapping memory of all VMs being bridged into the
> > switch memory space.
> >
> > By comparison, ivshmem works by exposing a shared region of memory to all VMs.
> > VMs are required to use this region to store packets. The switch only
> > needs access to this region.
> >
> > Another difference between vhost-user and ivshmem surfaces when polling
> > is used. With vhost-user, the switch is required to handle
> > data movement between VMs, if using polling, this means that 1 host CPU
> > needs to be sacrificed for this task.
> >
> > This is easiest to understand when one of the VMs is
> > used with VF pass-through. This can be schematically shown below:
> >
> > +-- VM1 --------------+ +---VM2-----------+
> > | virtio-pci +-vhost-user-+ virtio-pci -- VF | -- VFIO -- IOMMU -- NIC
> > +---------------------+ +-----------------+
> >
> >
> > With ivshmem in theory communication can happen directly, with two VMs
> > polling the shared memory region.
> >
> >
> > I won't spend time listing advantages of vhost-user over ivshmem.
> > Instead, having identified two advantages of ivshmem over vhost-user,
> > below is a proposal to extend vhost-user to gain the advantages
> > of ivshmem.
> >
> >
> > 1: virtio in guest can be extended to allow support
> > for IOMMUs. This provides guest with full flexibility
> > about memory which is readable or write able by each device.
>
> I assume that you meant VFIO only for virtio by "use of VFIO". To get
> VFIO working for general direct-I/O (including VFs) in guests, as you
> know, we need to virtualize IOMMU (e.g. VT-d) and the interrupt
> remapping table on x86 (i.e. nested VT-d).
Not necessarily: if pmd is used, mappings stay mostly static,
and there are no interrupts, so existing IOMMU emulation in qemu
will do the job.
> > By setting up a virtio device for each other VM we need to
> > communicate to, guest gets full control of its security, from
> > mapping all memory (like with current vhost-user) to only
> > mapping buffers used for networking (like ivshmem) to
> > transient mappings for the duration of data transfer only.
>
> And I think that we can use VMFUNC to have such transient mappings.
Interesting. There are two points to make here:
1. To create transient mappings, VMFUNC isn't strictly required.
Instead, mappings can be created when first access by VM2
within BAR triggers a page fault.
I guess VMFUNC could remove this first pagefault by hypervisor mapping
host PTE into the alternative view, then VMFUNC making
VM2 PTE valid - might be important if mappings are very dynamic
so there are many pagefaults.
2. To invalidate mappings, VMFUNC isn't sufficient since
translation cache of other CPUs needs to be invalidated.
I don't think VMFUNC can do this.
> > This also allows use of VFIO within guests, for improved
> > security.
> >
> > vhost user would need to be extended to send the
> > mappings programmed by guest IOMMU.
>
> Right. We need to think about cases where other VMs (VM3, etc.) join
> the group or some existing VM leaves.
> PCI hot-plug should work there (as you point out at "Advantages over
> ivshmem" below).
>
> >
> > 2. qemu can be extended to serve as a vhost-user client:
> > remote VM mappings over the vhost-user protocol, and
> > map them into another VM's memory.
> > This mapping can take, for example, the form of
> > a BAR of a pci device, which I'll call here vhost-pci -
> > with bus address allowed
> > by VM1's IOMMU mappings being translated into
> > offsets within this BAR within VM2's physical
> > memory space.
>
> I think it's sensible.
>
> >
> > Since the translation can be a simple one, VM2
> > can perform it within its vhost-pci device driver.
> >
> > While this setup would be the most useful with polling,
> > VM1's ioeventfd can also be mapped to
> > another VM2's irqfd, and vice versa, such that VMs
> > can trigger interrupts to each other without need
> > for a helper thread on the host.
> >
> >
> > The resulting channel might look something like the following:
> >
> > +-- VM1 --------------+ +---VM2-----------+
> > | virtio-pci -- iommu +--+ vhost-pci -- VF | -- VFIO -- IOMMU -- NIC
> > +---------------------+ +-----------------+
> >
> > comparing the two diagrams, a vhost-user thread on the host is
> > no longer required, reducing the host CPU utilization when
> > polling is active. At the same time, VM2 can not access all of VM1's
> > memory - it is limited by the iommu configuration setup by VM1.
> >
> >
> > Advantages over ivshmem:
> >
> > - more flexibility, endpoint VMs do not have to place data at any
> > specific locations to use the device, in practice this likely
> > means less data copies.
> > - better standardization/code reuse
> > virtio changes within guests would be fairly easy to implement
> > and would also benefit other backends, besides vhost-user
> > standard hotplug interfaces can be used to add and remove these
> > channels as VMs are added or removed.
> > - migration support
> > It's easy to implement since ownership of memory is well defined.
> > For example, during migration VM2 can notify hypervisor of VM1
> > by updating dirty bitmap each time is writes into VM1 memory.
>
> Also, the ivshmem functionality could be implemented by this proposal:
> - vswitch (or some VM) allocates memory regions in its address space, and
> - it sets up that IOMMU mappings on the VMs be translated into the regions
I agree it's possible, but that's not something that exists on real
hardware. It's not clear to me what are the security implications
of having VM2 control IOMMU of VM1. Having each VM control its own IOMMU
seems more straight-forward.
> >
> > Thanks,
> >
> > --
> > MST
> > _______________________________________________
> > Virtualization mailing list
> > Virtualization@lists.linux-foundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/virtualization
>
>
> --
> Jun
> Intel Open Source Technology Center
^ permalink raw reply
* Re: [Qemu-devel] rfc: vhost user enhancements for vm2vm communication
From: Michael S. Tsirkin @ 2015-09-01 8:30 UTC (permalink / raw)
To: Varun Sethi
Cc: virtio-dev@lists.oasis-open.org, Jan Kiszka,
Claudio.Fontana@huawei.com, qemu-devel@nongnu.org,
Linux Virtualization, opnfv-tech-discuss@lists.opnfv.org
In-Reply-To: <BN1PR0301MB0627D7AE90FC7A08545848CAEA6A0@BN1PR0301MB0627.namprd03.prod.outlook.com>
On Tue, Sep 01, 2015 at 03:03:12AM +0000, Varun Sethi wrote:
> Hi Michael,
> When you talk about VFIO in guest, is it with a purely emulated IOMMU in Qemu?
This can use the emulated IOMMU in Qemu.
That's probably fast enough if mappings are mostly static.
We can also add a PV-IOMMU if necessary.
> Also, I am not clear on the following points:
> 1. How transient memory would be mapped using BAR in the backend VM
The simplest way is that
each update sends a vhost-user message. backend gets it and
mmaps it into backend QEMU and make it part of RAM memory slot.
Or - backend QEMU could detect a pagefault on access and get the
IOMMU from frontend QEMU - using vhost-user messages or
from shared memory.
> 2. How would the backend VM update the dirty page bitmap for the frontend VM
>
> Regards
> Varun
The easiest to implement way is probably for backend QEMU to setup dirty tracking
for the relevant slot (upon getting vhost user message
from the frontend) then retrieve the dirty map
from kvm and record it in a shared memory region
(when do it? We could have an eventfd and/or vhost-user message to
trigger this from the frontend QEMU, or just use a timer).
An alternative is for backend VM to get access to dirty log
(e.g. map it within BAR) and update it directly in shared memory.
Seems like more work.
Marc-André Lureau recently sent patches to support passing
dirty log around, these would be useful.
> > -----Original Message-----
> > From: qemu-devel-bounces+varun.sethi=freescale.com@nongnu.org
> > [mailto:qemu-devel-bounces+varun.sethi=freescale.com@nongnu.org] On
> > Behalf Of Nakajima, Jun
> > Sent: Monday, August 31, 2015 1:36 PM
> > To: Michael S. Tsirkin
> > Cc: virtio-dev@lists.oasis-open.org; Jan Kiszka;
> > Claudio.Fontana@huawei.com; qemu-devel@nongnu.org; Linux
> > Virtualization; opnfv-tech-discuss@lists.opnfv.org
> > Subject: Re: [Qemu-devel] rfc: vhost user enhancements for vm2vm
> > communication
> >
> > On Mon, Aug 31, 2015 at 7:11 AM, Michael S. Tsirkin <mst@redhat.com>
> > wrote:
> > > Hello!
> > > During the KVM forum, we discussed supporting virtio on top of
> > > ivshmem. I have considered it, and came up with an alternative that
> > > has several advantages over that - please see below.
> > > Comments welcome.
> >
> > Hi Michael,
> >
> > I like this, and it should be able to achieve what I presented at KVM Forum
> > (vhost-user-shmem).
> > Comments below.
> >
> > >
> > > -----
> > >
> > > Existing solutions to userspace switching between VMs on the same host
> > > are vhost-user and ivshmem.
> > >
> > > vhost-user works by mapping memory of all VMs being bridged into the
> > > switch memory space.
> > >
> > > By comparison, ivshmem works by exposing a shared region of memory to
> > all VMs.
> > > VMs are required to use this region to store packets. The switch only
> > > needs access to this region.
> > >
> > > Another difference between vhost-user and ivshmem surfaces when
> > > polling is used. With vhost-user, the switch is required to handle
> > > data movement between VMs, if using polling, this means that 1 host
> > > CPU needs to be sacrificed for this task.
> > >
> > > This is easiest to understand when one of the VMs is used with VF
> > > pass-through. This can be schematically shown below:
> > >
> > > +-- VM1 --------------+ +---VM2-----------+
> > > | virtio-pci +-vhost-user-+ virtio-pci -- VF | -- VFIO -- IOMMU -- NIC
> > > +---------------------+ +-----------------+
> > >
> > >
> > > With ivshmem in theory communication can happen directly, with two VMs
> > > polling the shared memory region.
> > >
> > >
> > > I won't spend time listing advantages of vhost-user over ivshmem.
> > > Instead, having identified two advantages of ivshmem over vhost-user,
> > > below is a proposal to extend vhost-user to gain the advantages of
> > > ivshmem.
> > >
> > >
> > > 1: virtio in guest can be extended to allow support for IOMMUs. This
> > > provides guest with full flexibility about memory which is readable or
> > > write able by each device.
> >
> > I assume that you meant VFIO only for virtio by "use of VFIO". To get VFIO
> > working for general direct-I/O (including VFs) in guests, as you know, we
> > need to virtualize IOMMU (e.g. VT-d) and the interrupt remapping table on
> > x86 (i.e. nested VT-d).
> >
> > > By setting up a virtio device for each other VM we need to communicate
> > > to, guest gets full control of its security, from mapping all memory
> > > (like with current vhost-user) to only mapping buffers used for
> > > networking (like ivshmem) to transient mappings for the duration of
> > > data transfer only.
> >
> > And I think that we can use VMFUNC to have such transient mappings.
> >
> > > This also allows use of VFIO within guests, for improved security.
> > >
> > > vhost user would need to be extended to send the mappings programmed
> > > by guest IOMMU.
> >
> > Right. We need to think about cases where other VMs (VM3, etc.) join the
> > group or some existing VM leaves.
> > PCI hot-plug should work there (as you point out at "Advantages over
> > ivshmem" below).
> >
> > >
> > > 2. qemu can be extended to serve as a vhost-user client:
> > > remote VM mappings over the vhost-user protocol, and map them into
> > > another VM's memory.
> > > This mapping can take, for example, the form of a BAR of a pci device,
> > > which I'll call here vhost-pci - with bus address allowed by VM1's
> > > IOMMU mappings being translated into offsets within this BAR within
> > > VM2's physical memory space.
> >
> > I think it's sensible.
> >
> > >
> > > Since the translation can be a simple one, VM2 can perform it within
> > > its vhost-pci device driver.
> > >
> > > While this setup would be the most useful with polling, VM1's
> > > ioeventfd can also be mapped to another VM2's irqfd, and vice versa,
> > > such that VMs can trigger interrupts to each other without need for a
> > > helper thread on the host.
> > >
> > >
> > > The resulting channel might look something like the following:
> > >
> > > +-- VM1 --------------+ +---VM2-----------+
> > > | virtio-pci -- iommu +--+ vhost-pci -- VF | -- VFIO -- IOMMU -- NIC
> > > +---------------------+ +-----------------+
> > >
> > > comparing the two diagrams, a vhost-user thread on the host is no
> > > longer required, reducing the host CPU utilization when polling is
> > > active. At the same time, VM2 can not access all of VM1's memory - it
> > > is limited by the iommu configuration setup by VM1.
> > >
> > >
> > > Advantages over ivshmem:
> > >
> > > - more flexibility, endpoint VMs do not have to place data at any
> > > specific locations to use the device, in practice this likely
> > > means less data copies.
> > > - better standardization/code reuse
> > > virtio changes within guests would be fairly easy to implement
> > > and would also benefit other backends, besides vhost-user
> > > standard hotplug interfaces can be used to add and remove these
> > > channels as VMs are added or removed.
> > > - migration support
> > > It's easy to implement since ownership of memory is well defined.
> > > For example, during migration VM2 can notify hypervisor of VM1
> > > by updating dirty bitmap each time is writes into VM1 memory.
> >
> > Also, the ivshmem functionality could be implemented by this proposal:
> > - vswitch (or some VM) allocates memory regions in its address space, and
> > - it sets up that IOMMU mappings on the VMs be translated into the regions
> >
> > >
> > > Thanks,
> > >
> > > --
> > > MST
> > > _______________________________________________
> > > Virtualization mailing list
> > > Virtualization@lists.linux-foundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/virtualization
> >
> >
> > --
> > Jun
> > Intel Open Source Technology Center
>
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Jan Kiszka @ 2015-09-01 9:11 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <20150901104914-mutt-send-email-mst@redhat.com>
On 2015-09-01 10:01, Michael S. Tsirkin wrote:
> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
>> Leaving all the implementation and interface details aside, this
>> discussion is first of all about two fundamentally different approaches:
>> static shared memory windows vs. dynamically remapped shared windows (a
>> third one would be copying in the hypervisor, but I suppose we all agree
>> that the whole exercise is about avoiding that). Which way do we want or
>> have to go?
>>
>> Jan
>
> Dynamic is a superset of static: you can always make it static if you
> wish. Static has the advantage of simplicity, but that's lost once you
> realize you need to invent interfaces to make it work. Since we can use
> existing IOMMU interfaces for the dynamic one, what's the disadvantage?
Complexity. Having to emulate even more of an IOMMU in the hypervisor
(we already have to do a bit for VT-d IR in Jailhouse) and doing this
per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
sense, generic grant tables would be more appealing. But what we would
actually need is an interface that is only *optionally* configured by a
guest for dynamic scenarios, otherwise preconfigured by the hypervisor
for static setups. And we need guests that support both. That's the
challenge.
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Michael S. Tsirkin @ 2015-09-01 9:24 UTC (permalink / raw)
To: Jan Kiszka
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <55E56BD8.5010202@siemens.com>
On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote:
> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
> > On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
> >> Leaving all the implementation and interface details aside, this
> >> discussion is first of all about two fundamentally different approaches:
> >> static shared memory windows vs. dynamically remapped shared windows (a
> >> third one would be copying in the hypervisor, but I suppose we all agree
> >> that the whole exercise is about avoiding that). Which way do we want or
> >> have to go?
> >>
> >> Jan
> >
> > Dynamic is a superset of static: you can always make it static if you
> > wish. Static has the advantage of simplicity, but that's lost once you
> > realize you need to invent interfaces to make it work. Since we can use
> > existing IOMMU interfaces for the dynamic one, what's the disadvantage?
>
> Complexity. Having to emulate even more of an IOMMU in the hypervisor
> (we already have to do a bit for VT-d IR in Jailhouse) and doing this
> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
> sense, generic grant tables would be more appealing.
That's not how we do things for KVM, PV features need to be
modular and interchangeable with emulation.
If you just want something that's cross-platform and easy to
implement, just build a PV IOMMU. Maybe use virtio for this.
> But what we would
> actually need is an interface that is only *optionally* configured by a
> guest for dynamic scenarios, otherwise preconfigured by the hypervisor
> for static setups. And we need guests that support both. That's the
> challenge.
>
> Jan
That's already there for IOMMUs: vfio does the static setup by default,
enabling iommu by guests is optional.
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Jan Kiszka @ 2015-09-01 14:09 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <20150901121743-mutt-send-email-mst@redhat.com>
On 2015-09-01 11:24, Michael S. Tsirkin wrote:
> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote:
>> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
>>>> Leaving all the implementation and interface details aside, this
>>>> discussion is first of all about two fundamentally different approaches:
>>>> static shared memory windows vs. dynamically remapped shared windows (a
>>>> third one would be copying in the hypervisor, but I suppose we all agree
>>>> that the whole exercise is about avoiding that). Which way do we want or
>>>> have to go?
>>>>
>>>> Jan
>>>
>>> Dynamic is a superset of static: you can always make it static if you
>>> wish. Static has the advantage of simplicity, but that's lost once you
>>> realize you need to invent interfaces to make it work. Since we can use
>>> existing IOMMU interfaces for the dynamic one, what's the disadvantage?
>>
>> Complexity. Having to emulate even more of an IOMMU in the hypervisor
>> (we already have to do a bit for VT-d IR in Jailhouse) and doing this
>> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
>> sense, generic grant tables would be more appealing.
>
> That's not how we do things for KVM, PV features need to be
> modular and interchangeable with emulation.
I know, and we may have to make some compromise for Jailhouse if that
brings us valuable standardization and broad guest support. But we will
surely not support an arbitrary amount of IOMMU models for that reason.
>
> If you just want something that's cross-platform and easy to
> implement, just build a PV IOMMU. Maybe use virtio for this.
That is likely required to keep the complexity manageable and to allow
static preconfiguration.
Well, we could declare our virtio-shmem device to be an IOMMU device
that controls access of a remote VM to RAM of the one that owns the
device. In the static case, this access may at most be enabled/disabled
but not moved around. The static regions would have to be discoverable
for the VM (register read-back), and the guest's firmware will likely
have to declare those ranges reserved to the guest OS.
In the dynamic case, the guest would be able to create an alternative
mapping. We would probably have to define a generic page table structure
for that. Or do you rather have some MPU-like control structure in mind,
more similar to the memory region descriptions vhost is already using?
Also not yet clear to me are how the vhost-pci device and the
translations it will have to do should look like for VM2.
>
>> But what we would
>> actually need is an interface that is only *optionally* configured by a
>> guest for dynamic scenarios, otherwise preconfigured by the hypervisor
>> for static setups. And we need guests that support both. That's the
>> challenge.
>>
>> Jan
>
> That's already there for IOMMUs: vfio does the static setup by default,
> enabling iommu by guests is optional.
Cannot follow yet how vfio comes into play regarding some preconfigured
virtual IOMMU.
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Michael S. Tsirkin @ 2015-09-01 14:34 UTC (permalink / raw)
To: Jan Kiszka
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <55E5B1A8.9080506@siemens.com>
On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote:
> On 2015-09-01 11:24, Michael S. Tsirkin wrote:
> > On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote:
> >> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
> >>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
> >>>> Leaving all the implementation and interface details aside, this
> >>>> discussion is first of all about two fundamentally different approaches:
> >>>> static shared memory windows vs. dynamically remapped shared windows (a
> >>>> third one would be copying in the hypervisor, but I suppose we all agree
> >>>> that the whole exercise is about avoiding that). Which way do we want or
> >>>> have to go?
> >>>>
> >>>> Jan
> >>>
> >>> Dynamic is a superset of static: you can always make it static if you
> >>> wish. Static has the advantage of simplicity, but that's lost once you
> >>> realize you need to invent interfaces to make it work. Since we can use
> >>> existing IOMMU interfaces for the dynamic one, what's the disadvantage?
> >>
> >> Complexity. Having to emulate even more of an IOMMU in the hypervisor
> >> (we already have to do a bit for VT-d IR in Jailhouse) and doing this
> >> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
> >> sense, generic grant tables would be more appealing.
> >
> > That's not how we do things for KVM, PV features need to be
> > modular and interchangeable with emulation.
>
> I know, and we may have to make some compromise for Jailhouse if that
> brings us valuable standardization and broad guest support. But we will
> surely not support an arbitrary amount of IOMMU models for that reason.
>
> >
> > If you just want something that's cross-platform and easy to
> > implement, just build a PV IOMMU. Maybe use virtio for this.
>
> That is likely required to keep the complexity manageable and to allow
> static preconfiguration.
Real IOMMU allow static configuration just fine. This is exactly
what VFIO uses.
> Well, we could declare our virtio-shmem device to be an IOMMU device
> that controls access of a remote VM to RAM of the one that owns the
> device. In the static case, this access may at most be enabled/disabled
> but not moved around. The static regions would have to be discoverable
> for the VM (register read-back), and the guest's firmware will likely
> have to declare those ranges reserved to the guest OS.
> In the dynamic case, the guest would be able to create an alternative
> mapping.
I don't think we want a special device just to support the
static case. It might be a bit less code to write, but
eventually it should be up to the guest.
Fundamentally, it's policy that host has no business
dictating.
> We would probably have to define a generic page table structure
> for that. Or do you rather have some MPU-like control structure in mind,
> more similar to the memory region descriptions vhost is already using?
I don't care much. Page tables use less memory if a lot of memory needs
to be covered. OTOH if you want to use virtio (e.g. to allow command
batching) that likely means commands to manipulate the IOMMU, and
maintaining it all on the host. You decide.
> Also not yet clear to me are how the vhost-pci device and the
> translations it will have to do should look like for VM2.
I think we can use vhost-pci BAR + VM1 bus address as the
VM2 physical address. In other words, all memory exposed to
virtio-pci by VM1 through it's IOMMU is mapped into BAR of
vhost-pci.
Bus addresses can be validated to make sure they fit
in the BAR.
One issue to consider is that VM1 can trick VM2 into writing
into bus address that isn't mapped in the IOMMU, or
is mapped read-only.
We probably would have to teach KVM to handle this somehow,
e.g. exit to QEMU, or even just ignore. Maybe notify guest
e.g. by setting a bit in the config space of the device,
to avoid easy DOS.
> >
> >> But what we would
> >> actually need is an interface that is only *optionally* configured by a
> >> guest for dynamic scenarios, otherwise preconfigured by the hypervisor
> >> for static setups. And we need guests that support both. That's the
> >> challenge.
> >>
> >> Jan
> >
> > That's already there for IOMMUs: vfio does the static setup by default,
> > enabling iommu by guests is optional.
>
> Cannot follow yet how vfio comes into play regarding some preconfigured
> virtual IOMMU.
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Jan Kiszka @ 2015-09-01 15:34 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <20150901172239-mutt-send-email-mst@redhat.com>
On 2015-09-01 16:34, Michael S. Tsirkin wrote:
> On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote:
>> On 2015-09-01 11:24, Michael S. Tsirkin wrote:
>>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote:
>>>> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
>>>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
>>>>>> Leaving all the implementation and interface details aside, this
>>>>>> discussion is first of all about two fundamentally different approaches:
>>>>>> static shared memory windows vs. dynamically remapped shared windows (a
>>>>>> third one would be copying in the hypervisor, but I suppose we all agree
>>>>>> that the whole exercise is about avoiding that). Which way do we want or
>>>>>> have to go?
>>>>>>
>>>>>> Jan
>>>>>
>>>>> Dynamic is a superset of static: you can always make it static if you
>>>>> wish. Static has the advantage of simplicity, but that's lost once you
>>>>> realize you need to invent interfaces to make it work. Since we can use
>>>>> existing IOMMU interfaces for the dynamic one, what's the disadvantage?
>>>>
>>>> Complexity. Having to emulate even more of an IOMMU in the hypervisor
>>>> (we already have to do a bit for VT-d IR in Jailhouse) and doing this
>>>> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
>>>> sense, generic grant tables would be more appealing.
>>>
>>> That's not how we do things for KVM, PV features need to be
>>> modular and interchangeable with emulation.
>>
>> I know, and we may have to make some compromise for Jailhouse if that
>> brings us valuable standardization and broad guest support. But we will
>> surely not support an arbitrary amount of IOMMU models for that reason.
>>
>>>
>>> If you just want something that's cross-platform and easy to
>>> implement, just build a PV IOMMU. Maybe use virtio for this.
>>
>> That is likely required to keep the complexity manageable and to allow
>> static preconfiguration.
>
> Real IOMMU allow static configuration just fine. This is exactly
> what VFIO uses.
Please specify more precisely which feature in which IOMMU you are
referring to. Also, given that you refer to VFIO, I suspect we have
different thing in mind. I'm talking about an IOMMU device model, like
the one we have in QEMU now for VT-d. That one is not at all
preconfigured by the host for VFIO.
>
>> Well, we could declare our virtio-shmem device to be an IOMMU device
>> that controls access of a remote VM to RAM of the one that owns the
>> device. In the static case, this access may at most be enabled/disabled
>> but not moved around. The static regions would have to be discoverable
>> for the VM (register read-back), and the guest's firmware will likely
>> have to declare those ranges reserved to the guest OS.
>> In the dynamic case, the guest would be able to create an alternative
>> mapping.
>
>
> I don't think we want a special device just to support the
> static case. It might be a bit less code to write, but
> eventually it should be up to the guest.
> Fundamentally, it's policy that host has no business
> dictating.
"A bit less" is to be validated, and I doubt its just "a bit". But if
KVM and its guests will also support some PV-IOMMU that we can reuse for
our scenarios, than that is fine. KVM would not have to mandate support
for it while we would, that's all.
>
>> We would probably have to define a generic page table structure
>> for that. Or do you rather have some MPU-like control structure in mind,
>> more similar to the memory region descriptions vhost is already using?
>
> I don't care much. Page tables use less memory if a lot of memory needs
> to be covered. OTOH if you want to use virtio (e.g. to allow command
> batching) that likely means commands to manipulate the IOMMU, and
> maintaining it all on the host. You decide.
I don't care very much about the dynamic case as we won't support it
anyway. However, if the configuration concept used for it is applicable
to static mode as well, then we could reuse it. But preconfiguration
will required register-based region description, I suspect.
>
>> Also not yet clear to me are how the vhost-pci device and the
>> translations it will have to do should look like for VM2.
>
> I think we can use vhost-pci BAR + VM1 bus address as the
> VM2 physical address. In other words, all memory exposed to
> virtio-pci by VM1 through it's IOMMU is mapped into BAR of
> vhost-pci.
>
> Bus addresses can be validated to make sure they fit
> in the BAR.
Sounds simple but may become challenging for VMs that have many of such
devices (in order to connect to many possibly large VMs).
>
>
> One issue to consider is that VM1 can trick VM2 into writing
> into bus address that isn't mapped in the IOMMU, or
> is mapped read-only.
> We probably would have to teach KVM to handle this somehow,
> e.g. exit to QEMU, or even just ignore. Maybe notify guest
> e.g. by setting a bit in the config space of the device,
> to avoid easy DOS.
Well, that would be trivial for VM1 to check if there are only one or
two memory windows. Relying on the hypervisor to handle it may be
unacceptable for real-time VMs.
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Michael S. Tsirkin @ 2015-09-01 16:02 UTC (permalink / raw)
To: Jan Kiszka
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <55E5C58D.6000104@siemens.com>
On Tue, Sep 01, 2015 at 05:34:37PM +0200, Jan Kiszka wrote:
> On 2015-09-01 16:34, Michael S. Tsirkin wrote:
> > On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote:
> >> On 2015-09-01 11:24, Michael S. Tsirkin wrote:
> >>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote:
> >>>> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
> >>>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
> >>>>>> Leaving all the implementation and interface details aside, this
> >>>>>> discussion is first of all about two fundamentally different approaches:
> >>>>>> static shared memory windows vs. dynamically remapped shared windows (a
> >>>>>> third one would be copying in the hypervisor, but I suppose we all agree
> >>>>>> that the whole exercise is about avoiding that). Which way do we want or
> >>>>>> have to go?
> >>>>>>
> >>>>>> Jan
> >>>>>
> >>>>> Dynamic is a superset of static: you can always make it static if you
> >>>>> wish. Static has the advantage of simplicity, but that's lost once you
> >>>>> realize you need to invent interfaces to make it work. Since we can use
> >>>>> existing IOMMU interfaces for the dynamic one, what's the disadvantage?
> >>>>
> >>>> Complexity. Having to emulate even more of an IOMMU in the hypervisor
> >>>> (we already have to do a bit for VT-d IR in Jailhouse) and doing this
> >>>> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
> >>>> sense, generic grant tables would be more appealing.
> >>>
> >>> That's not how we do things for KVM, PV features need to be
> >>> modular and interchangeable with emulation.
> >>
> >> I know, and we may have to make some compromise for Jailhouse if that
> >> brings us valuable standardization and broad guest support. But we will
> >> surely not support an arbitrary amount of IOMMU models for that reason.
> >>
> >>>
> >>> If you just want something that's cross-platform and easy to
> >>> implement, just build a PV IOMMU. Maybe use virtio for this.
> >>
> >> That is likely required to keep the complexity manageable and to allow
> >> static preconfiguration.
> >
> > Real IOMMU allow static configuration just fine. This is exactly
> > what VFIO uses.
>
> Please specify more precisely which feature in which IOMMU you are
> referring to. Also, given that you refer to VFIO, I suspect we have
> different thing in mind. I'm talking about an IOMMU device model, like
> the one we have in QEMU now for VT-d. That one is not at all
> preconfigured by the host for VFIO.
I really just mean that VFIO creates a mostly static IOMMU configuration.
It's configured by the guest, not the host.
I don't see host control over configuration as being particularly important.
> >
> >> Well, we could declare our virtio-shmem device to be an IOMMU device
> >> that controls access of a remote VM to RAM of the one that owns the
> >> device. In the static case, this access may at most be enabled/disabled
> >> but not moved around. The static regions would have to be discoverable
> >> for the VM (register read-back), and the guest's firmware will likely
> >> have to declare those ranges reserved to the guest OS.
> >> In the dynamic case, the guest would be able to create an alternative
> >> mapping.
> >
> >
> > I don't think we want a special device just to support the
> > static case. It might be a bit less code to write, but
> > eventually it should be up to the guest.
> > Fundamentally, it's policy that host has no business
> > dictating.
>
> "A bit less" is to be validated, and I doubt its just "a bit". But if
> KVM and its guests will also support some PV-IOMMU that we can reuse for
> our scenarios, than that is fine. KVM would not have to mandate support
> for it while we would, that's all.
Someone will have to do this work.
> >
> >> We would probably have to define a generic page table structure
> >> for that. Or do you rather have some MPU-like control structure in mind,
> >> more similar to the memory region descriptions vhost is already using?
> >
> > I don't care much. Page tables use less memory if a lot of memory needs
> > to be covered. OTOH if you want to use virtio (e.g. to allow command
> > batching) that likely means commands to manipulate the IOMMU, and
> > maintaining it all on the host. You decide.
>
> I don't care very much about the dynamic case as we won't support it
> anyway. However, if the configuration concept used for it is applicable
> to static mode as well, then we could reuse it. But preconfiguration
> will required register-based region description, I suspect.
I don't know what you mean by preconfiguration exactly.
Do you want the host to configure the IOMMU? Why not let the
guest do this?
> >
> >> Also not yet clear to me are how the vhost-pci device and the
> >> translations it will have to do should look like for VM2.
> >
> > I think we can use vhost-pci BAR + VM1 bus address as the
> > VM2 physical address. In other words, all memory exposed to
> > virtio-pci by VM1 through it's IOMMU is mapped into BAR of
> > vhost-pci.
> >
> > Bus addresses can be validated to make sure they fit
> > in the BAR.
>
> Sounds simple but may become challenging for VMs that have many of such
> devices (in order to connect to many possibly large VMs).
You don't need to be able to map all guest memory if you know
guest won't try to allow device access to all of it.
It's a question of how good is the bus address allocator.
> >
> >
> > One issue to consider is that VM1 can trick VM2 into writing
> > into bus address that isn't mapped in the IOMMU, or
> > is mapped read-only.
> > We probably would have to teach KVM to handle this somehow,
> > e.g. exit to QEMU, or even just ignore. Maybe notify guest
> > e.g. by setting a bit in the config space of the device,
> > to avoid easy DOS.
>
> Well, that would be trivial for VM1 to check if there are only one or
> two memory windows. Relying on the hypervisor to handle it may be
> unacceptable for real-time VMs.
>
> Jan
Why? real-time != fast. I doubt you can avoid vm exits completely.
> --
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Jan Kiszka @ 2015-09-01 16:28 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, Claudio.Fontana, qemu-devel, virtualization,
Varun Sethi, opnfv-tech-discuss
In-Reply-To: <20150901184922-mutt-send-email-mst@redhat.com>
On 2015-09-01 18:02, Michael S. Tsirkin wrote:
> On Tue, Sep 01, 2015 at 05:34:37PM +0200, Jan Kiszka wrote:
>> On 2015-09-01 16:34, Michael S. Tsirkin wrote:
>>> On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote:
>>>> On 2015-09-01 11:24, Michael S. Tsirkin wrote:
>>>>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan Kiszka wrote:
>>>>>> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
>>>>>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200, Jan Kiszka wrote:
>>>>>>>> Leaving all the implementation and interface details aside, this
>>>>>>>> discussion is first of all about two fundamentally different approaches:
>>>>>>>> static shared memory windows vs. dynamically remapped shared windows (a
>>>>>>>> third one would be copying in the hypervisor, but I suppose we all agree
>>>>>>>> that the whole exercise is about avoiding that). Which way do we want or
>>>>>>>> have to go?
>>>>>>>>
>>>>>>>> Jan
>>>>>>>
>>>>>>> Dynamic is a superset of static: you can always make it static if you
>>>>>>> wish. Static has the advantage of simplicity, but that's lost once you
>>>>>>> realize you need to invent interfaces to make it work. Since we can use
>>>>>>> existing IOMMU interfaces for the dynamic one, what's the disadvantage?
>>>>>>
>>>>>> Complexity. Having to emulate even more of an IOMMU in the hypervisor
>>>>>> (we already have to do a bit for VT-d IR in Jailhouse) and doing this
>>>>>> per platform (AMD IOMMU, ARM SMMU, ...) is out of scope for us. In that
>>>>>> sense, generic grant tables would be more appealing.
>>>>>
>>>>> That's not how we do things for KVM, PV features need to be
>>>>> modular and interchangeable with emulation.
>>>>
>>>> I know, and we may have to make some compromise for Jailhouse if that
>>>> brings us valuable standardization and broad guest support. But we will
>>>> surely not support an arbitrary amount of IOMMU models for that reason.
>>>>
>>>>>
>>>>> If you just want something that's cross-platform and easy to
>>>>> implement, just build a PV IOMMU. Maybe use virtio for this.
>>>>
>>>> That is likely required to keep the complexity manageable and to allow
>>>> static preconfiguration.
>>>
>>> Real IOMMU allow static configuration just fine. This is exactly
>>> what VFIO uses.
>>
>> Please specify more precisely which feature in which IOMMU you are
>> referring to. Also, given that you refer to VFIO, I suspect we have
>> different thing in mind. I'm talking about an IOMMU device model, like
>> the one we have in QEMU now for VT-d. That one is not at all
>> preconfigured by the host for VFIO.
>
> I really just mean that VFIO creates a mostly static IOMMU configuration.
>
> It's configured by the guest, not the host.
OK, that resolves my confusion.
>
> I don't see host control over configuration as being particularly important.
We do, see below.
>
>
>>>
>>>> Well, we could declare our virtio-shmem device to be an IOMMU device
>>>> that controls access of a remote VM to RAM of the one that owns the
>>>> device. In the static case, this access may at most be enabled/disabled
>>>> but not moved around. The static regions would have to be discoverable
>>>> for the VM (register read-back), and the guest's firmware will likely
>>>> have to declare those ranges reserved to the guest OS.
>>>> In the dynamic case, the guest would be able to create an alternative
>>>> mapping.
>>>
>>>
>>> I don't think we want a special device just to support the
>>> static case. It might be a bit less code to write, but
>>> eventually it should be up to the guest.
>>> Fundamentally, it's policy that host has no business
>>> dictating.
>>
>> "A bit less" is to be validated, and I doubt its just "a bit". But if
>> KVM and its guests will also support some PV-IOMMU that we can reuse for
>> our scenarios, than that is fine. KVM would not have to mandate support
>> for it while we would, that's all.
>
> Someone will have to do this work.
>
>>>
>>>> We would probably have to define a generic page table structure
>>>> for that. Or do you rather have some MPU-like control structure in mind,
>>>> more similar to the memory region descriptions vhost is already using?
>>>
>>> I don't care much. Page tables use less memory if a lot of memory needs
>>> to be covered. OTOH if you want to use virtio (e.g. to allow command
>>> batching) that likely means commands to manipulate the IOMMU, and
>>> maintaining it all on the host. You decide.
>>
>> I don't care very much about the dynamic case as we won't support it
>> anyway. However, if the configuration concept used for it is applicable
>> to static mode as well, then we could reuse it. But preconfiguration
>> will required register-based region description, I suspect.
>
> I don't know what you mean by preconfiguration exactly.
>
> Do you want the host to configure the IOMMU? Why not let the
> guest do this?
We simply freeze GPA-to-HPA mappings during runtime. Avoids having to
validate and synchronize guest-triggered changes.
>>>
>>>> Also not yet clear to me are how the vhost-pci device and the
>>>> translations it will have to do should look like for VM2.
>>>
>>> I think we can use vhost-pci BAR + VM1 bus address as the
>>> VM2 physical address. In other words, all memory exposed to
>>> virtio-pci by VM1 through it's IOMMU is mapped into BAR of
>>> vhost-pci.
>>>
>>> Bus addresses can be validated to make sure they fit
>>> in the BAR.
>>
>> Sounds simple but may become challenging for VMs that have many of such
>> devices (in order to connect to many possibly large VMs).
>
> You don't need to be able to map all guest memory if you know
> guest won't try to allow device access to all of it.
> It's a question of how good is the bus address allocator.
But those BARs need to allocate a guest-physical address range as large
as the other guest's RAM is, possibly even larger if that RAM is not
contiguous, and you can't put other resources into potential holes
because VM2 does not know where those holes will be.
>
>>>
>>>
>>> One issue to consider is that VM1 can trick VM2 into writing
>>> into bus address that isn't mapped in the IOMMU, or
>>> is mapped read-only.
>>> We probably would have to teach KVM to handle this somehow,
>>> e.g. exit to QEMU, or even just ignore. Maybe notify guest
>>> e.g. by setting a bit in the config space of the device,
>>> to avoid easy DOS.
>>
>> Well, that would be trivial for VM1 to check if there are only one or
>> two memory windows. Relying on the hypervisor to handle it may be
>> unacceptable for real-time VMs.
>>
>> Jan
>
> Why? real-time != fast. I doubt you can avoid vm exits completely.
We can, one property of Jailhouse (on x86, ARM is waiting for GICv4).
Real-time == deterministic. And if you have such vm exits potentially in
your code path, you have them always - for worst-case analysis. One may
argue about probability in certain scenarios, but if the triggering side
is malicious, probability may become 1.
Jan
--
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Nakajima, Jun @ 2015-09-01 22:56 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, Jan Kiszka, Claudio.Fontana, qemu-devel,
Linux Virtualization, opnfv-tech-discuss
In-Reply-To: <20150901110627-mutt-send-email-mst@redhat.com>
My previous email has been bounced by virtio-dev@lists.oasis-open.org.
I tried to subscribed it, but to no avail...
On Tue, Sep 1, 2015 at 1:17 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Mon, Aug 31, 2015 at 11:35:55AM -0700, Nakajima, Jun wrote:
>> On Mon, Aug 31, 2015 at 7:11 AM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> > 1: virtio in guest can be extended to allow support
>> > for IOMMUs. This provides guest with full flexibility
>> > about memory which is readable or write able by each device.
>>
>> I assume that you meant VFIO only for virtio by "use of VFIO". To get
>> VFIO working for general direct-I/O (including VFs) in guests, as you
>> know, we need to virtualize IOMMU (e.g. VT-d) and the interrupt
>> remapping table on x86 (i.e. nested VT-d).
>
> Not necessarily: if pmd is used, mappings stay mostly static,
> and there are no interrupts, so existing IOMMU emulation in qemu
> will do the job.
OK. It would work, although we need to engage additional/complex code
in the guests when we are making just memory operations under the
hood.
>> > By setting up a virtio device for each other VM we need to
>> > communicate to, guest gets full control of its security, from
>> > mapping all memory (like with current vhost-user) to only
>> > mapping buffers used for networking (like ivshmem) to
>> > transient mappings for the duration of data transfer only.
>>
>> And I think that we can use VMFUNC to have such transient mappings.
>
> Interesting. There are two points to make here:
>
>
> 1. To create transient mappings, VMFUNC isn't strictly required.
> Instead, mappings can be created when first access by VM2
> within BAR triggers a page fault.
> I guess VMFUNC could remove this first pagefault by hypervisor mapping
> host PTE into the alternative view, then VMFUNC making
> VM2 PTE valid - might be important if mappings are very dynamic
> so there are many pagefaults.
I agree that VMFUNC isn't strictly required. It would provide
performance optimization.
And I think it can add some level of protection as well because you
might want to keep mapping guest physical memory (which is partial or
entire VM1's memory) at BAR of VM2 all the time. IOMMU on VM1 can
limit the address ranges accessed by VM2, but such restriction becomes
loose as you want them static and thus large enough.
>
> 2. To invalidate mappings, VMFUNC isn't sufficient since
> translation cache of other CPUs needs to be invalidated.
> I don't think VMFUNC can do this.
I don't think we need to invalidate mappings often. And if we do, we
need to invalidate EPT anyway.
>>
>> Also, the ivshmem functionality could be implemented by this proposal:
>> - vswitch (or some VM) allocates memory regions in its address space, and
>> - it sets up that IOMMU mappings on the VMs be translated into the regions
>
> I agree it's possible, but that's not something that exists on real
> hardware. It's not clear to me what are the security implications
> of having VM2 control IOMMU of VM1. Having each VM control its own IOMMU
> seems more straight-forward.
I meant the vswitch's IOMMU. It can a bare-metal (or host) process or
VM. For a bare-metal process, it's basically VFIO, where virtual
address is used as bus address. Each VM accesses the shared memory
using vhost-pci BAR + bus (i.e. virtual) address.
--
Jun
Intel Open Source Technology Center
^ permalink raw reply
* Re: rfc: vhost user enhancements for vm2vm communication
From: Nakajima, Jun @ 2015-09-02 0:01 UTC (permalink / raw)
To: Jan Kiszka
Cc: virtio-dev, Michael S. Tsirkin, Claudio.Fontana, qemu-devel,
Linux Virtualization, Varun Sethi, opnfv-tech-discuss
In-Reply-To: <55E5D22C.6020108@siemens.com>
On Tue, Sep 1, 2015 at 9:28 AM, Jan Kiszka <jan.kiszka@siemens.com> wrote:
> On 2015-09-01 18:02, Michael S. Tsirkin wrote:
...
>> You don't need to be able to map all guest memory if you know
>> guest won't try to allow device access to all of it.
>> It's a question of how good is the bus address allocator.
>
> But those BARs need to allocate a guest-physical address range as large
> as the other guest's RAM is, possibly even larger if that RAM is not
> contiguous, and you can't put other resources into potential holes
> because VM2 does not know where those holes will be.
>
I think you can allocate such guest-physical address ranges
efficiently if each BAR sets the base of each memory region reported
by VHOST_SET_MEM_TABLE, for example. The issue is that we would need
to 8 (VHOST_MEMORY_MAX_NREGIONS) of them vs. 6 (defined by PCI-SIG).
--
Jun
Intel Open Source Technology Center
^ permalink raw reply
* [PATCH] virtio-gpu: fix compilation warnings
From: Mike Rapoport @ 2015-09-02 9:30 UTC (permalink / raw)
To: David Airlie; +Cc: linux-kernel, Mike Rapoport, dri-devel, virtualization
Update snprintf format in virtgpu_fence.c and virtgpu_debugfs.c to fix the
following compilation warnings:
C [M] drivers/gpu/drm/virtio/virtgpu_fence.o
drivers/gpu/drm/virtio/virtgpu_fence.c: In function ‘virtio_timeline_value_str’ :
drivers/gpu/drm/virtio/virtgpu_fence.c:64:2: warning: format ‘%lu’ expects argument of type ‘long unsigned int’, but argument 4 has type ‘long long int’ [-Wformat=]
snprintf(str, size, "%lu", atomic64_read(&fence->drv->last_seq));
^
CC [M] drivers/gpu/drm/virtio/virtgpu_debugfs.o
drivers/gpu/drm/virtio/virtgpu_debugfs.c: In function ‘virtio_gpu_debugfs_irq_info’:
drivers/gpu/drm/virtio/virtgpu_debugfs.c:39:6: warning: format ‘%ld’ expects argument of type ‘long int’, but argument 3 has type ‘long long int’ [-Wformat=]
vgdev->fence_drv.sync_seq);
^
Signed-off-by: Mike Rapoport <mike.rapoport@gmail.com>
---
drivers/gpu/drm/virtio/virtgpu_debugfs.c | 2 +-
drivers/gpu/drm/virtio/virtgpu_fence.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/drm/virtio/virtgpu_debugfs.c b/drivers/gpu/drm/virtio/virtgpu_debugfs.c
index db8b491..d87b27c 100644
--- a/drivers/gpu/drm/virtio/virtgpu_debugfs.c
+++ b/drivers/gpu/drm/virtio/virtgpu_debugfs.c
@@ -34,7 +34,7 @@ virtio_gpu_debugfs_irq_info(struct seq_file *m, void *data)
struct drm_info_node *node = (struct drm_info_node *) m->private;
struct virtio_gpu_device *vgdev = node->minor->dev->dev_private;
- seq_printf(m, "fence %ld %lld\n",
+ seq_printf(m, "fence %lld %lld\n",
atomic64_read(&vgdev->fence_drv.last_seq),
vgdev->fence_drv.sync_seq);
return 0;
diff --git a/drivers/gpu/drm/virtio/virtgpu_fence.c b/drivers/gpu/drm/virtio/virtgpu_fence.c
index 1da6326..98dd385 100644
--- a/drivers/gpu/drm/virtio/virtgpu_fence.c
+++ b/drivers/gpu/drm/virtio/virtgpu_fence.c
@@ -61,7 +61,7 @@ static void virtio_timeline_value_str(struct fence *f, char *str, int size)
{
struct virtio_gpu_fence *fence = to_virtio_fence(f);
- snprintf(str, size, "%lu", atomic64_read(&fence->drv->last_seq));
+ snprintf(str, size, "%llu", atomic64_read(&fence->drv->last_seq));
}
static const struct fence_ops virtio_fence_ops = {
--
1.8.3.1
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox