From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751677AbbJELrY (ORCPT ); Mon, 5 Oct 2015 07:47:24 -0400 Received: from mail-wi0-f176.google.com ([209.85.212.176]:34470 "EHLO mail-wi0-f176.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751330AbbJELrW (ORCPT ); Mon, 5 Oct 2015 07:47:22 -0400 Subject: Re: [PATCH v3 2/3] uio_pci_generic: add MSI/MSI-X support To: Vlad Zolotarov , Greg KH References: <1443991398-23761-1-git-send-email-vladz@cloudius-systems.com> <1443991398-23761-3-git-send-email-vladz@cloudius-systems.com> <20151005031159.GB27303@kroah.com> <561229B3.7000109@cloudius-systems.com> <20151005075628.GA1747@kroah.com> <56125587.40104@cloudius-systems.com> <20151005105715.GA23459@kroah.com> <561261EF.8010101@cloudius-systems.com> Cc: linux-kernel@vger.kernel.org, mst@redhat.com, hjk@hansjkoch.de, corbet@lwn.net, bruce.richardson@intel.com, avi@cloudius-systems.com, gleb@cloudius-systems.com, stephen@networkplumber.org, alexander.duyck@gmail.com From: Avi Kivity Message-ID: <56126346.3090605@scylladb.com> Date: Mon, 5 Oct 2015 14:47:18 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <561261EF.8010101@cloudius-systems.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/05/2015 02:41 PM, Vlad Zolotarov wrote: > > > On 10/05/15 13:57, Greg KH wrote: >> On Mon, Oct 05, 2015 at 01:48:39PM +0300, Vlad Zolotarov wrote: >>> >>> On 10/05/15 10:56, Greg KH wrote: >>>> On Mon, Oct 05, 2015 at 10:41:39AM +0300, Vlad Zolotarov wrote: >>>>>>> +struct msix_info { >>>>>>> + int num_irqs; >>>>>>> + struct msix_entry *table; >>>>>>> + struct uio_msix_irq_ctx { >>>>>>> + struct eventfd_ctx *trigger; /* MSI-x vector to >>>>>>> eventfd */ >>>>>> Why are you using eventfd for msi vectors? What's the reason for >>>>>> needing this? >>>>> A small correction - for MSI-X vectors. There may be only one MSI >>>>> vector per >>>>> PCI function and if it's used it would use the same interface as a >>>>> legacy >>>>> INT#x interrupt uses at the moment. >>>>> So, for MSI-X case the reason is that there may be (in most cases >>>>> there will >>>>> be) more than one interrupt vector. Thus, as I've explained in a >>>>> PATCH1 >>>>> thread we need a way to indicated each of them separately. eventfd >>>>> seems >>>>> like a good way of doing so. If u have better ideas, pls., share. >>>> You need to document what you are doing here, I don't see any >>>> explaination for using eventfd at all. >>>> >>>> And no, I don't know of any other solution as I don't know what you >>>> are >>>> trying to do here (hint, the changelog didn't document it...) >>>> >>>>>> You haven't documented how this api works at all, you are going >>>>>> to have >>>>>> to a lot more work to justify this, as this greatly increases the >>>>>> complexity of the user/kernel api in unknown ways. >>>>> I actually do documented it a bit. Pls., check PATCH3 out. >>>> That provided no information at all about how to use the api. >>>> >>>> If it did, you would see that your api is broken for 32/64bit kernels >>>> and will fall over into nasty pieces the first time you try to use it >>>> there, which means it hasn't been tested at all :( >>> It has been tested of course ;) >>> I tested it only in 64 bit environment however where both kernel and >>> user >>> space applications were compiled on the same machine with the same >>> compiler >>> and it could be that "int" had the same number of bytes both in >>> kernel and >>> in user space application. Therefore it worked perfectly - I patched >>> DPDK to >>> use the new uio_pci_generic MSI-X API to test this and I have >>> verified that >>> all 3 interrupt modes work: MSI-X with SR-IOV VF device in Amazon >>> EC2 guest >>> and INT#x and MSI with a PF device on bare metal server. >>> >>> However I agree using uint32_t for "vec" and "fd" would be much more >>> correct. >> I don't think file descriptors are __u32 on a 64bit arch, are they? > > I think they are "int" on all platforms and as far as I know u32 > should be enough to contain int on any platform. > You need to make sure structures have the same layout on both 32-bit and 64-bit systems, or you'll have to code compat ioctl translations for them. The best way to do that is to use __u32 so the sizes are obvious, even for int, and to pad everything to 64 bit: > +struct msix_info { + __u32 num_irqs; + __u32 pad; // so pointer below is aligned to 64-bit on both 32-bit and 64-bit userspace > > + struct msix_entry *table; > + struct uio_msix_irq_ctx { > + struct eventfd_ctx *trigger; /* MSI-x vector to eventfd */ >> >> And NEVER use the _t types in kernel code, > > Never meant it - it was for a user space interface. For a kernel it's > u32 of course. > For interfaces, use __u32. You can't use uint32_t because if someone uses C89 in 2015, they may not have .