From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yx1-f52.google.com (mail-yx1-f52.google.com [74.125.224.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9624A45BD78 for ; Wed, 1 Apr 2026 16:13:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.224.52 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775060015; cv=none; b=Ef+hjkjNjANSMee+jt21Z6e6HuhYgLLBM0K/aNf9+pjL9doZbMjVVKlsBwbjd0FWFKmzDzYh0d7Z+waej33V42Dn6xC9r8lm1qDXzaHbliBFkweOS3f+1CsuGh75nx94vUbP2m2TZ2x/0OvPoHHYD2nek9cUyWbUcQ1XdtBo89g= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775060015; c=relaxed/simple; bh=cnK6EEPHajF0pbqfPzN+z3vn0QShkKkn4JXq6163h3Y=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=JGEXo7mI6kW58Fj8lYgp2H3ucZQ5ORXgmP172qKXQgPmRDIyS+0HVqNJpdBKYu8asEbpqwlk6nZtm0ad2hzrhtyfVKVd3sw/5qwRrq7u1TbBnLJYCkLTTLbXPNWEdNe0eiLCfePdnsv2f2qYx5vMxWdlZpVbZsWb1t+RsEhsImU= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=AF59w76/; arc=none smtp.client-ip=74.125.224.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AF59w76/" Received: by mail-yx1-f52.google.com with SMTP id 956f58d0204a3-65003f40a22so8787119d50.2 for ; Wed, 01 Apr 2026 09:13:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1775060011; x=1775664811; darn=vger.kernel.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=C7CaeRefHVdJqe/blozX5Q0mpA6fFJRjORc2uH8uvuo=; b=AF59w76/f8xlogYR+1wDAZa7DrC03nhlxPFj/+Dl1VvSfv9LVX+yaB5YNL5oXYF0PR jX/a6R/FZxvfZbUs+6PPmRrjYSO/om0hYu/GHP59rpVJiTqHYDtbkQytGQK4hkj1xAE6 Z+S8+SSTgJGq3VVpjhTmusKnwhzhWzW6SheL8oV/St1jRygpwhrTbyq8WBwi+f3uLGnv sNZD0mnAcSfLl1zWJA439sba7+m6jEimyLiu5Qy8ikDIOD6roY3fNDpUDFym4MRjESmy o0lzGSsakoIB/7vqO3fKAElPEgRwLPwE7NNgjlQnltuzw2Dy7IEO1tBazDS+lEVNnJrb XMGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775060011; x=1775664811; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=C7CaeRefHVdJqe/blozX5Q0mpA6fFJRjORc2uH8uvuo=; b=J3oJ5e5hHHJcBYFxBlIcM8d33Uk4OPeR8zZIUFk/GbgkF6GQowHz8VANALWmqzPbdg uPvX6ek/9h5b7ElYY7PwfOFXumIBfV8b0lsDyWMfAoqtZhi3AKEDNb1qnt66mwaNIA7o xQumMfgSsEK9L+GcCz6klrwd3O6VG2u+PfMPmY0NkvIlD+YAKvPxnFGL0MtQxLqG5NxM 867dincHj+eKOYMwweOJ/bbimRhXI+AP5dqZG7rVrAwqp6F+auWhIMN3SNBKdM6t4ePJ ZknH9x08VphTc9fW6bOXoXZSR3P+VFksNqOY4xq0s0kAwqeNs7VuFaryj31k4IJnR3XX cNCw== X-Forwarded-Encrypted: i=1; AJvYcCUD/6dskvF6KJJkEKrPxJzKk6Eb5wZ4FF+4yKPqiS5ZFHrRZvhDgSZW2UiJpoEL1nM/8tM3XGyWVD8FcPo=@vger.kernel.org X-Gm-Message-State: AOJu0Yznu4iD9ZaXFwdFa+YiHxQrTjCUqKH7147E1VqUc0Xc3Nx4eA3Q T0izV7iR7uiAI5vpxxGjKsRa+L5Ia6LgnlDQi3VDZY/26TQLrLxlc4Ko X-Gm-Gg: ATEYQzwIH3WRL2JvvMVutx/mKi0Cgdxg8dcUv6VEFauhXbPj8LCMpb03a+8euRkL91K kLS1PHODdBlbRgvOLFAfodLekMzRaAIgqxUlCCZ+8hmRtJI/uOeF6TF0mcuVBn5+fdJa43Qg4cN XE5gWkKAWCrj2tOxM+ke7cGD9WzF6UyhBsrgyflIIZY4tkc1i38SnwLRFYXjw34jrermcfJ6BKd rq5QhjMNZHwBsmqGWASCx+TovXEK+0K3/42dcWbslZcWtAPMEDldwHiVBPl6r6Uos7PEnyqgDf5 yUoVri413BVmpxAHiYYX2vQG8CzzMZe2DPM9oadSBUMgbg/bH2NuuemFBXIMeLIa7NbXYBiQsce eGpW/Ou99o4zgzMSNHfFUCxBhvPRzHNRHC9+L/qlDDG7k7UmWRC9DaUDSbYedM//uUUCOPEQGUG p0xyX8D5kUT3Gqlg== X-Received: by 2002:a05:690e:144a:b0:650:314f:110d with SMTP id 956f58d0204a3-650314f13c5mr3348992d50.57.1775060011460; Wed, 01 Apr 2026 09:13:31 -0700 (PDT) Received: from nsa ([45.94.208.239]) by smtp.gmail.com with ESMTPSA id 956f58d0204a3-6503a9b34f7sm100224d50.16.2026.04.01.09.13.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Apr 2026 09:13:30 -0700 (PDT) Date: Wed, 1 Apr 2026 17:14:16 +0100 From: Nuno =?utf-8?B?U8Oh?= To: Frank Li Cc: Nuno =?utf-8?B?U8Oh?= , dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, Lars-Peter Clausen , Vinod Koul , Frank Li , Eliza Balas Subject: Re: [PATCH v2 4/4] dmaengine: dma-axi-dmac: Defer freeing DMA descriptors Message-ID: References: <20260327-dma-dmac-handle-vunmap-v2-0-021f95f0e87b@analog.com> <20260327-dma-dmac-handle-vunmap-v2-4-021f95f0e87b@analog.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Tue, Mar 31, 2026 at 04:21:06PM +0100, Nuno Sá wrote: > On Tue, Mar 31, 2026 at 10:16:09AM -0400, Frank Li wrote: > > On Tue, Mar 31, 2026 at 09:53:45AM +0100, Nuno Sá wrote: > > > On Mon, Mar 30, 2026 at 11:24:34AM -0400, Frank Li wrote: > > > > On Fri, Mar 27, 2026 at 04:58:41PM +0000, Nuno Sá wrote: > > > > > From: Eliza Balas > > > > > > > > > > This IP core can be used in architectures (like Microblaze) where DMA > > > > > descriptors are allocated with vmalloc(). > > > > > > > > strage, why use vmalloc()? > > > > > > It's just one of the paths in dma_alloc_coherent(). It should be > > > architecture dependant. > > > > Which architectures, this may common problem, dma_alloc/free_coherent() is > > quite common at other dma-engine driver. > > I'll double check this but I believe this was triggered on microblaze > where we also use this IP. Will come back with confirmation! > Hi Frank, I now went to the bottom of the issue! The problem is that for archs like microblaze and arm64 we have DMA_DIRECT_REMAP which means that when calling dma_alloc_coherent() in [1] we will get into the code path in [2]. Now I did some research and we might have other solution for this that does not involve this refcount craziness plus async work. But I need to test it. FYI, what I have in mind is similar to the what loongson2-apb-dma.c does. This means, using the dma pool API. IIUC, with the pool we only actually free the memory (dma_free_coherent()) in the .terminate_all() callback (when destroying the pool) which should not happen in interrupt context right? [1]: https://elixir.bootlin.com/linux/v7.0-rc6/source/drivers/dma/dma-axi-dmac.c#L549 [2]: https://elixir.bootlin.com/linux/v7.0-rc6/source/kernel/dma/direct.c#L278 - Nuno Sá > - Nuno Sá > > > > Frank > > > > > > > > - Nuno Sá > > > > > > > > > > > Frank > > > > > > > > > Hence, given that freeing the > > > > > descriptors happen in softirq context, vunmpap() will BUG(). > > > > > > > > > > To solve the above, we setup a work item during allocation of the > > > > > descriptors and schedule in softirq context. Hence, the actual freeing > > > > > happens in threaded context. > > > > > > > > > > Also note that to account for the possible race where the struct axi_dmac > > > > > object is gone between scheduling the work and actually running it, we > > > > > now save and get a reference of struct device when allocating the > > > > > descriptor (given that's all we need in axi_dmac_free_desc()) and > > > > > release it in axi_dmac_free_desc(). > > > > > > > > > > Signed-off-by: Eliza Balas > > > > > Co-developed-by: Nuno Sá > > > > > Signed-off-by: Nuno Sá > > > > > --- > > > > > drivers/dma/dma-axi-dmac.c | 50 ++++++++++++++++++++++++++++++++++------------ > > > > > 1 file changed, 37 insertions(+), 13 deletions(-) > > > > > > > > > > diff --git a/drivers/dma/dma-axi-dmac.c b/drivers/dma/dma-axi-dmac.c > > > > > index 70d3ad7e7d37..46f1ead0c7d7 100644 > > > > > --- a/drivers/dma/dma-axi-dmac.c > > > > > +++ b/drivers/dma/dma-axi-dmac.c > > > > > @@ -25,6 +25,7 @@ > > > > > #include > > > > > #include > > > > > #include > > > > > +#include > > > > > > > > > > #include > > > > > > > > > > @@ -133,6 +134,9 @@ struct axi_dmac_sg { > > > > > struct axi_dmac_desc { > > > > > struct virt_dma_desc vdesc; > > > > > struct axi_dmac_chan *chan; > > > > > + struct device *dev; > > > > > + > > > > > + struct work_struct sched_work; > > > > > > > > > > bool cyclic; > > > > > bool cyclic_eot; > > > > > @@ -666,6 +670,25 @@ static void axi_dmac_issue_pending(struct dma_chan *c) > > > > > spin_unlock_irqrestore(&chan->vchan.lock, flags); > > > > > } > > > > > > > > > > +static void axi_dmac_free_desc(struct axi_dmac_desc *desc) > > > > > +{ > > > > > + struct axi_dmac_hw_desc *hw = desc->sg[0].hw; > > > > > + dma_addr_t hw_phys = desc->sg[0].hw_phys; > > > > > + > > > > > + dma_free_coherent(desc->dev, PAGE_ALIGN(desc->num_sgs * sizeof(*hw)), > > > > > + hw, hw_phys); > > > > > + put_device(desc->dev); > > > > > + kfree(desc); > > > > > +} > > > > > + > > > > > +static void axi_dmac_free_desc_schedule_work(struct work_struct *work) > > > > > +{ > > > > > + struct axi_dmac_desc *desc = container_of(work, struct axi_dmac_desc, > > > > > + sched_work); > > > > > + > > > > > + axi_dmac_free_desc(desc); > > > > > +} > > > > > + > > > > > static struct axi_dmac_desc * > > > > > axi_dmac_alloc_desc(struct axi_dmac_chan *chan, unsigned int num_sgs) > > > > > { > > > > > @@ -681,6 +704,7 @@ axi_dmac_alloc_desc(struct axi_dmac_chan *chan, unsigned int num_sgs) > > > > > return NULL; > > > > > desc->num_sgs = num_sgs; > > > > > desc->chan = chan; > > > > > + desc->dev = get_device(dmac->dma_dev.dev); > > > > > > > > > > hws = dma_alloc_coherent(dev, PAGE_ALIGN(num_sgs * sizeof(*hws)), > > > > > &hw_phys, GFP_ATOMIC); > > > > > @@ -703,21 +727,18 @@ axi_dmac_alloc_desc(struct axi_dmac_chan *chan, unsigned int num_sgs) > > > > > /* The last hardware descriptor will trigger an interrupt */ > > > > > desc->sg[num_sgs - 1].hw->flags = AXI_DMAC_HW_FLAG_LAST | AXI_DMAC_HW_FLAG_IRQ; > > > > > > > > > > + /* > > > > > + * We need to setup a work item because this IP can be used on archs > > > > > + * that rely on vmalloced memory for descriptors. And given that freeing > > > > > + * the descriptors happens in softirq context, vunmpap() will BUG(). > > > > > + * Hence, setup the worker so that we can queue it and free the > > > > > + * descriptor in threaded context. > > > > > + */ > > > > > + INIT_WORK(&desc->sched_work, axi_dmac_free_desc_schedule_work); > > > > > + > > > > > return desc; > > > > > } > > > > > > > > > > -static void axi_dmac_free_desc(struct axi_dmac_desc *desc) > > > > > -{ > > > > > - struct axi_dmac *dmac = chan_to_axi_dmac(desc->chan); > > > > > - struct device *dev = dmac->dma_dev.dev; > > > > > - struct axi_dmac_hw_desc *hw = desc->sg[0].hw; > > > > > - dma_addr_t hw_phys = desc->sg[0].hw_phys; > > > > > - > > > > > - dma_free_coherent(dev, PAGE_ALIGN(desc->num_sgs * sizeof(*hw)), > > > > > - hw, hw_phys); > > > > > - kfree(desc); > > > > > -} > > > > > - > > > > > static struct axi_dmac_sg *axi_dmac_fill_linear_sg(struct axi_dmac_chan *chan, > > > > > enum dma_transfer_direction direction, dma_addr_t addr, > > > > > unsigned int num_periods, unsigned int period_len, > > > > > @@ -958,7 +979,10 @@ static void axi_dmac_free_chan_resources(struct dma_chan *c) > > > > > > > > > > static void axi_dmac_desc_free(struct virt_dma_desc *vdesc) > > > > > { > > > > > - axi_dmac_free_desc(to_axi_dmac_desc(vdesc)); > > > > > + struct axi_dmac_desc *desc = to_axi_dmac_desc(vdesc); > > > > > + > > > > > + /* See the comment in axi_dmac_alloc_desc() for the why! */ > > > > > + schedule_work(&desc->sched_work); > > > > > } > > > > > > > > > > static bool axi_dmac_regmap_rdwr(struct device *dev, unsigned int reg) > > > > > > > > > > -- > > > > > 2.53.0 > > > > >