From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C50C0C10F14 for ; Fri, 12 Apr 2019 15:31:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 936542171F for ; Fri, 12 Apr 2019 15:31:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727428AbfDLPbA (ORCPT ); Fri, 12 Apr 2019 11:31:00 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:41672 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726965AbfDLPbA (ORCPT ); Fri, 12 Apr 2019 11:31:00 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x3CFQPHT076478 for ; Fri, 12 Apr 2019 11:30:59 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0b-001b2d01.pphosted.com with ESMTP id 2rtu4wf3ax-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 12 Apr 2019 11:30:58 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 12 Apr 2019 16:30:57 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Fri, 12 Apr 2019 16:30:55 +0100 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x3CFUsHK49086480 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Apr 2019 15:30:54 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1BB63A4051; Fri, 12 Apr 2019 15:30:54 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8A0D6A4057; Fri, 12 Apr 2019 15:30:53 +0000 (GMT) Received: from oc2783563651 (unknown [9.152.224.40]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 12 Apr 2019 15:30:53 +0000 (GMT) Date: Fri, 12 Apr 2019 17:30:17 +0200 From: Halil Pasic To: Sebastian Ott Cc: kvm@vger.kernel.org, linux-s390@vger.kernel.org, Cornelia Huck , Martin Schwidefsky , virtualization@lists.linux-foundation.org, Christian Borntraeger , Viktor Mihajlovski , Vasily Gorbik , Janosch Frank , Claudio Imbrenda , Farhan Ali , Eric Farman Subject: Re: [RFC PATCH 04/12] s390/cio: introduce cio DMA pool In-Reply-To: References: <20190404231622.52531-1-pasic@linux.ibm.com> <20190404231622.52531-5-pasic@linux.ibm.com> <20190412132010.3c74cb63@oc2783563651> Organization: IBM X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.31; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-TM-AS-GCONF: 00 x-cbid: 19041215-0028-0000-0000-000003602D31 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19041215-0029-0000-0000-0000241F58F5 Message-Id: <20190412173017.04b768bb@oc2783563651> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-12_09:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1904120102 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Fri, 12 Apr 2019 14:12:31 +0200 (CEST) Sebastian Ott wrote: > On Fri, 12 Apr 2019, Halil Pasic wrote: > > On Thu, 11 Apr 2019 20:25:01 +0200 (CEST) > > Sebastian Ott wrote: > > > I don't think we should use this global DMA pool. I guess it's OK for > > > stuff like airq (where we don't have a struct device at hand) but for > > > CCW we should use the device we have. Yes, this way we waste some memory > > > but all dma memory a device uses should fit in a page - so the wastage > > > is not too much. Regarding the wastage. Let us do the math together in search for an upper (wastage) limit. #define __MAX_CSSID 0 #define __MAX_SUBCHANNEL 65535 #define __MAX_SSID 3 that is a maximum of 2^16 devices per subchannel set x 2^2 subchannel sets * 2^0 css == 2^18 devices at most. With your a page per device we have max 2^30 = 2^18 + 2^12 bytes. That is exactly 1GB. And because some of the stuff needs to be 31 bit addressable all of that 1GB would have to be under 2GB. Currently we need at least 224 bytes per device that is ~ 6% of a PAGE_SIZE. > > > > > > > Is what you envision an own gen_pool on for each subchannel (e.g. a > > struct io_subchannel member)? > > Either that or if that's too much overhead simply map a page and create > a struct containing the few dma areas for that device. In virtio-ccw we do dynamic allocations of DMA memory. And we could theoretically do the same elsewhere. So I don't think a struct with a few dma areas would do (assumed I understood you correctly, which isn't very likely). > > > I'm struggling with understanding the expected benefits of a > > per-subchannel pool/allocator. Can you please tell me what benefits do > > you expect (over the current approach)? > > Logically DMA is a capability of a device and the whole DMA API is build > around devices. I agree the whole DMA API is built around devices. IMHO DMA is indeed on one hand a capability of a device, but it is also a capability of a machine (system). Normally (PCI) limitations can come either from the device side (e.g. device supports only 32 bit addressing) or from the machine/bus side. In our case however all ccw devices have the same limitations/requirements. E.g. if you want to do a SSCH your ORB and your CCWs will have to have to be 31 bit addressable (physical). Your data can be above 2G if MIDA is available and used. None of this depends on the device we are talking to, but dictated by the properties of the architecture and the machine. Another important thing to notice is that for CCW IO isolation is done very differently than for PCI. > Working around that just feels wrong. Then the airq iv->vector should be a per-device thing as well, or? > For practical > matters: DMA debugging will complain about misuse of a specific device or > driver. > Do you mean CONFIG_DMA_API_DEBUG and CONFIG_DMA_API_DEBUG_SG? I've been running with those and did not see any complaints. Maybe we should clarify this one offline... > > I understand you idea is to keep the CIO global pool for stuff that can > > not be tied to a single device (i.e. ariq). So the per device stuff would > > also mean more code. Would you be OK with postponing this alleged > > enhancement (i.e. implement it as a patch on top of this series)? > > I don't like it but it's just in-kernel usage which we can change at any > time. So if it helps you to do it that way, why not.. > Right. There should be no external interface implications to changing this AFAICT. I prefer seeing this matter as not substantial for what I'm trying to accomplish with this series. Regards, Halil