From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jan Beulich" Subject: Re: FW: VT-d async invalidation for Device-TLB. Date: Tue, 16 Jun 2015 08:55:57 +0100 Message-ID: <557FF2AD0200007800085447@mail.emea.novell.com> References: <945CA011AD5F084CBEA3E851C0AB28890E914E2C@SHSMSX101.ccr.corp.intel.com> <55780BBB0200007800082E90@mail.emea.novell.com> <945CA011AD5F084CBEA3E851C0AB28890E928F23@SHSMSX101.ccr.corp.intel.com> <557A9C670200007800083FAC@mail.emea.novell.com> <945CA011AD5F084CBEA3E851C0AB28890E92AE62@SHSMSX101.ccr.corp.intel.com> <557E94EB0200007800084A14@mail.emea.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Z4lj0-0004pI-2g for xen-devel@lists.xenproject.org; Tue, 16 Jun 2015 07:56:02 +0000 In-Reply-To: Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Quan Xu , Yang Z Zhang Cc: "andrew.cooper3@citrix.com" , Kevin Tian , DonaldD Dugger , xen-devel List-Id: xen-devel@lists.xenproject.org >>> On 16.06.15 at 05:07, wrote: > Jan Beulich wrote on 2015-06-15: >>>>> On 13.06.15 at 16:44, wrote: >>>> On 12.06.15 at 14:47, wrote: >>>>>>> On 12.06.15 at 04:40, wrote: >>>>> I tested it by a Myri-10G Dual-Protocol NIC, which is an ATS device. >>>>> for an invalidation: >>>>> By sync way, it takes about 1.4 ms. >>>>> By async way, it takes about 4.3 ms. > > It is a typo for the time: should be 1.4 us and 4.3 us. > >>>> >>>> What's the theory on why this is? After all, it shouldn't matter >>>> how the completion of the invalidation gets signaled. >>>> >>> >>> Let me introduce more about how I get these test data. >>> >>> For sync way, I get the time by NOW() macro, when I update Queue >>> Tail Register (qinval_update_qtail()). >>> Then get time by NOW() macro in current spinning when it has wrote >>> the value in the Status Data field to the address specified in the Status Address. >>> The difference of these 2 value is the time of an sync invalidation. >>> >>> >>> For async way, as the previous email, I have introduced the IF bit >>> of Invalidation Wait Descriptor. >>> (IF: Indicate the invalidation wait descriptor completion by >>> generating an invalidation completion event per the programing of >>> the Invalidation Completion Event Registers.) I have implemented an >>> interrupt for invalidation completion event. >>> Also I get the time by NOW() macro, when I update Queue Tail >>> Register (by qinval_update_qtail()). >>> Then get time by NOW() macro in invalidation completion event handler. >>> The difference of these 2 value is the time of an async invalidation. >> >> Okay, thanks for the explanation. As this includes the time it takes >> to deliver and (partly) handle the interrupt, the difference is of >> course within what one would expect (and also of what would seem > acceptable). > > The time doesn't include the cost of handling of interrupt. We just record > it at the entry of interrupt handler. So the cost should bigger than 4.3 us > if taking the handing cost into consideration. And the costs will much bigger > if there are more pass-through VMs runs. We can start from ATS case firstly. > And apply it to non-ATS case later if the async approach doesn't hurt the > performance. In which case we're back to the question I raised originally: How do you explain the time difference if the interrupt delivery overhead isn't included? Jan