All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Jan Beulich" <JBeulich@suse.com>
To: Quan Xu <quan.xu@intel.com>
Cc: Yang Z Zhang <yang.z.zhang@intel.com>,
	"andrew.cooper3@citrix.com" <andrew.cooper3@citrix.com>,
	Kevin Tian <kevin.tian@intel.com>,
	DonaldD Dugger <donald.d.dugger@intel.com>,
	xen-devel <xen-devel@lists.xenproject.org>
Subject: Re: VT-d async invalidation for Device-TLB.
Date: Fri, 12 Jun 2015 07:46:31 +0100	[thread overview]
Message-ID: <557A9C670200007800083FAC@mail.emea.novell.com> (raw)
In-Reply-To: <945CA011AD5F084CBEA3E851C0AB28890E928F23@SHSMSX101.ccr.corp.intel.com>

>>> On 12.06.15 at 04:40, <quan.xu@intel.com> wrote:
>> > >>> On 10.06.15 at 16:05, <JBeulich@suse.com> wrote:
>> >>> On 03.06.15 at 09:49, <quan.xu@intel.com> wrote:
>> >     For Context Invalidation and IOTLB invalidation without Device-TLB
>> > invalidation, Invalidation Queue flushes synchronous invalidation as
>> > before(This is a tradeoff and the cost of interrupt is overhead).
>> 
>> DMAR_OPERATION_TIMEOUT being 1s, are you saying that you're not intending
>> to replace the current spinning for the non-ATS case?
> 
> Yes, we are not intending to replace the current spinning for the non-ATS 
> case.

I'm not really happy about that.

>> Considering that expiring these loops results in panic()s, I would expect 
> these to
>> become asynchronous _and_ contained to the affected VM alongside the ATS
>> induced changed behavior. You talking of overhead - can you quantify that?
>>
> 
> I tested it by a Myri-10G Dual-Protocol NIC, which is an ATS device. 
> for an invalidation:
>  By sync way, it takes about 1.4 ms.
>  By async way, it takes about 4.3 ms.

What's the theory on why this is? After all, it shouldn't matter how
the completion of the invalidation gets signaled.

Apart from that measuring the ATS case (in which case we're set to
use async mode anyway) is kind of pointless here - we'd need to
know the overhead of non-ATS async compared to non-ATS sync.

>> > More details:
>> >
>> > 1. invalidation table. We define iommu _invl structure in domain.
>> > Struct iommu _invl {
>> >     volatile u64 iommu _invl _poll_slot :62;
>> >     domid_t dom_id;
>> >     u64 iommu _invl _status_data :32;
>> > }__attribute__ ((aligned (64)));
>> >
>> >    iommu _invl _poll_slot: Set it equal to the status address of wait
>> > descriptor when the invalidation queue is with Device-TLB.
>> >    dom_id: Keep the id of the domain.
>> >    iommu _invl _status_data: Keep the count of in-flight queue with
>> > Device-TLB invalidation.
>> 
>> Without further explanation above/below I don't think I really understand 
> the
>> purpose of this structure, nor its organization: Is this something imposed 
> by the
>> VT-d specification? If so, a reference to the respective section in the spec 
> would
>> be useful. If not, I can't see why the structure is laid out the (odd) way 
> it is.
>> 
> 
> Refer to the explanation above. If it is still not clear, I will continue to 
> explain in next email.

The explanation above helped for what I asked above, but didn't
make clear to me what the structure here is, how it relates to hw
defined structures, and hence (as said) why it is laid out the way it
is.

>> > 4. New interrupt handler for invalidation completion:
>> >     - when hardware completes the invalidations with Device IOTLB, it
>> > generates an interrupt to notify hypervisor.
>> >     - In interrupt handler, we will schedule a soft-irq to handle the
>> > finished invalidations.
>> >     - soft-irq to handle finished invalidation:
>> >         Scan the pending flush list
>> > 	    for each entry in list
>> >             check the values of iommu _invl _poll_slot and iommu _invl
>> > _status_data in each domain's invalidation table.
>> >             if yes, clear iommu_pending_flush and invalidation table,
>> > then wakeup the domain.
>> 
>> Did you put some consideration into how long this list may get, and hence 
> how
>> long it may take you to iterate through the entire list?
>> 
> 
> Only the domain which has the ATS device assigned will be tracked in this 
> list. So the list length shouldn't be very long.

Okay, if this is a list of domains (or of devices), that would hopefully
be acceptable (albeit on a huge system this could still be dozens). If
this was a list of pending flush requests, it might be worse.

> Besides, the DEVICE-IOTLB 
> invalidation doesn't happened frequently so the cost should be acceptable.

That's not a valid consideration: At no time must any processing
inside the hypervisor take arbitrarily long. This requirement is
entirely independent of how frequently such cases may occur.

Jan

  reply	other threads:[~2015-06-12  6:46 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2015-06-03  7:49 VT-d async invalidation for Device-TLB Xu, Quan
2015-06-10  8:04 ` Jan Beulich
2015-06-12  2:40   ` Xu, Quan
2015-06-12  6:46     ` Jan Beulich [this message]
2015-06-13 14:44       ` FW: " Xu, Quan
2015-06-15  7:03         ` Jan Beulich
2015-06-16  3:07           ` Zhang, Yang Z
2015-06-16  7:55             ` Jan Beulich
2015-06-16  7:59               ` Zhang, Yang Z
2015-06-16  8:11                 ` Jan Beulich
2015-06-18  8:09                   ` Xu, Quan
2015-06-18  9:18                     ` Jan Beulich
2015-06-18 11:31                       ` Xu, Quan
2015-06-18 11:39                         ` Jan Beulich
2015-06-18 12:02                           ` Xu, Quan
  -- strict thread matches above, loose matches on Subject: below --
2015-06-16  3:55 Xu, Quan
2015-06-16  7:57 ` Jan Beulich
2015-06-16  8:29   ` Xu, Quan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=557A9C670200007800083FAC@mail.emea.novell.com \
    --to=jbeulich@suse.com \
    --cc=andrew.cooper3@citrix.com \
    --cc=donald.d.dugger@intel.com \
    --cc=kevin.tian@intel.com \
    --cc=quan.xu@intel.com \
    --cc=xen-devel@lists.xenproject.org \
    --cc=yang.z.zhang@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.