From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751912AbbJELxb (ORCPT ); Mon, 5 Oct 2015 07:53:31 -0400 Received: from mail-wi0-f182.google.com ([209.85.212.182]:38536 "EHLO mail-wi0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750942AbbJELxa (ORCPT ); Mon, 5 Oct 2015 07:53:30 -0400 Subject: Re: [PATCH v3 2/3] uio_pci_generic: add MSI/MSI-X support To: Avi Kivity , Greg KH References: <1443991398-23761-1-git-send-email-vladz@cloudius-systems.com> <1443991398-23761-3-git-send-email-vladz@cloudius-systems.com> <20151005031159.GB27303@kroah.com> <561229B3.7000109@cloudius-systems.com> <20151005075628.GA1747@kroah.com> <56125587.40104@cloudius-systems.com> <20151005105715.GA23459@kroah.com> <561261EF.8010101@cloudius-systems.com> <56126346.3090605@scylladb.com> Cc: linux-kernel@vger.kernel.org, mst@redhat.com, hjk@hansjkoch.de, corbet@lwn.net, bruce.richardson@intel.com, avi@cloudius-systems.com, gleb@cloudius-systems.com, stephen@networkplumber.org, alexander.duyck@gmail.com From: Vlad Zolotarov Message-ID: <561264B6.5020009@cloudius-systems.com> Date: Mon, 5 Oct 2015 14:53:26 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <56126346.3090605@scylladb.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/05/15 14:47, Avi Kivity wrote: > On 10/05/2015 02:41 PM, Vlad Zolotarov wrote: >> >> >> On 10/05/15 13:57, Greg KH wrote: >>> On Mon, Oct 05, 2015 at 01:48:39PM +0300, Vlad Zolotarov wrote: >>>> >>>> On 10/05/15 10:56, Greg KH wrote: >>>>> On Mon, Oct 05, 2015 at 10:41:39AM +0300, Vlad Zolotarov wrote: >>>>>>>> +struct msix_info { >>>>>>>> + int num_irqs; >>>>>>>> + struct msix_entry *table; >>>>>>>> + struct uio_msix_irq_ctx { >>>>>>>> + struct eventfd_ctx *trigger; /* MSI-x vector to >>>>>>>> eventfd */ >>>>>>> Why are you using eventfd for msi vectors? What's the reason for >>>>>>> needing this? >>>>>> A small correction - for MSI-X vectors. There may be only one MSI >>>>>> vector per >>>>>> PCI function and if it's used it would use the same interface as >>>>>> a legacy >>>>>> INT#x interrupt uses at the moment. >>>>>> So, for MSI-X case the reason is that there may be (in most cases >>>>>> there will >>>>>> be) more than one interrupt vector. Thus, as I've explained in a >>>>>> PATCH1 >>>>>> thread we need a way to indicated each of them separately. >>>>>> eventfd seems >>>>>> like a good way of doing so. If u have better ideas, pls., share. >>>>> You need to document what you are doing here, I don't see any >>>>> explaination for using eventfd at all. >>>>> >>>>> And no, I don't know of any other solution as I don't know what >>>>> you are >>>>> trying to do here (hint, the changelog didn't document it...) >>>>> >>>>>>> You haven't documented how this api works at all, you are going >>>>>>> to have >>>>>>> to a lot more work to justify this, as this greatly increases the >>>>>>> complexity of the user/kernel api in unknown ways. >>>>>> I actually do documented it a bit. Pls., check PATCH3 out. >>>>> That provided no information at all about how to use the api. >>>>> >>>>> If it did, you would see that your api is broken for 32/64bit kernels >>>>> and will fall over into nasty pieces the first time you try to use it >>>>> there, which means it hasn't been tested at all :( >>>> It has been tested of course ;) >>>> I tested it only in 64 bit environment however where both kernel >>>> and user >>>> space applications were compiled on the same machine with the same >>>> compiler >>>> and it could be that "int" had the same number of bytes both in >>>> kernel and >>>> in user space application. Therefore it worked perfectly - I >>>> patched DPDK to >>>> use the new uio_pci_generic MSI-X API to test this and I have >>>> verified that >>>> all 3 interrupt modes work: MSI-X with SR-IOV VF device in Amazon >>>> EC2 guest >>>> and INT#x and MSI with a PF device on bare metal server. >>>> >>>> However I agree using uint32_t for "vec" and "fd" would be much more >>>> correct. >>> I don't think file descriptors are __u32 on a 64bit arch, are they? >> >> I think they are "int" on all platforms and as far as I know u32 >> should be enough to contain int on any platform. >> > > You need to make sure structures have the same layout on both 32-bit > and 64-bit systems, or you'll have to code compat ioctl translations > for them. The best way to do that is to use __u32 so the sizes are > obvious, even for int, and to pad everything to 64 bit: Sure, but the structure below is not the one that is passed in ioctl() - it's an internal uio_pci_generic state and there is nothing to worry about. The one in question is struct uio_pci_generic_irq_set from uio_pci_generic.h: struct uio_pci_generic_irq_set { int vec; /* index of the IRQ to connect to starting from 0 */ int fd; }; It should be struct uio_pci_generic_irq_set { __u32 vec; /* index of the IRQ to connect to starting from 0 */ __u32 fd; }; instead. > >> +struct msix_info { > > + __u32 num_irqs; > + __u32 pad; // so pointer below is aligned to 64-bit on both > 32-bit and 64-bit userspace >> >> + struct msix_entry *table; >> + struct uio_msix_irq_ctx { >> + struct eventfd_ctx *trigger; /* MSI-x vector to eventfd */ > > >>> >>> And NEVER use the _t types in kernel code, >> >> Never meant it - it was for a user space interface. For a kernel it's >> u32 of course. >> > > For interfaces, use __u32. You can't use uint32_t because if someone > uses C89 in 2015, they may not have . >