RE: How to resolve an issue in swiotlb environment?

Biju Das
2019-06-10T07:31:41Z

Hi All,

Any update on the below issue. I am seeing similar issue on RZ/G2M board with Linux version 5.2.0-rc3.

root@hihope-rz-g2m:~# [ 35.414177] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[ 35.449402] usb-storage 2-1:1.0: USB Mass Storage device detected
[ 35.455915] scsi host0: usb-storage 2-1:1.0
[ 36.482585] scsi 0:0:0:0: Direct-Access SanDisk Extreme 0001 PQ: 0 ANSI: 6
[ 36.491260] sd 0:0:0:0: [sda] 125045424 512-byte logical blocks: (64.0 GB/59.6 GiB)
[ 36.499823] sd 0:0:0:0: [sda] Write Protect is off
[ 36.505474] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 36.518074] sda: sda1
[ 36.523163] sd 0:0:0:0: [sda] Attached SCSI disk

root@hihope-rz-g2m:~# mkdir -p /tmp/rmnt/sda1
root@hihope-rz-g2m:~# mount -t auto /dev/sda1 /tmp/rmnt/sda1
root@hihope-rz-g2m:~# dd if=/dev/urandom of=/tmp/sda1-random bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.187696 s, 55.9 MB/s
root@hihope-rz-g2m:~# cp /tmp/sda1-random /tmp/rmnt/sda1/sda1-random
root@hihope-rz-g2m:~# [ 218.861212] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 218.871885] xhci-hcd ee000000.usb: overflow 0x000000067430b000+1003520 of DMA mask ffffffff bus mask 0
[ 218.881233] WARNING: CPU: 0 PID: 258 at kernel/dma/direct.c:43 report_addr+0x38/0xa8
[ 218.888974] Modules linked in: renesas_usb3 usb_dmac phy_rcar_gen3_usb3
[ 218.895594] CPU: 0 PID: 258 Comm: usb-storage Not tainted 5.2.0-rc3-00017-gc80b083-dirty #5
[ 218.903940] Hardware name: HopeRun HiHope RZ/G2M with sub board (DT)
[ 218.910291] pstate: 40000005 (nZcv daif -PAN -UAO)
[ 218.915078] pc : report_addr+0x38/0xa8
[ 218.918821] lr : report_addr+0xa0/0xa8
[ 218.922564] sp : ffff0000125fb970
[ 218.925872] x29: ffff0000125fb970 x28: 0000000000000000
[ 218.931180] x27: 0000000000000000 x26: 000000001f020280
[ 218.936487] x25: ffff8006394a82a8 x24: 0000000000000000
[ 218.941794] x23: 0000000000000001 x22: 0000000000000000
[ 218.947101] x21: 00000000000f5000 x20: ffff000011309000
[ 218.952408] x19: ffff80063a600010 x18: ffffffffffffffff
[ 218.957714] x17: 0000000000000000 x16: 0000000000000000
[ 218.963023] x15: ffff0000113096c8 x14: 4d4420666f203032
[ 218.968331] x13: 35333030312b3030 x12: 3062303334373630
[ 218.973638] x11: 3030303030307830 x10: ffff000011309f20
[ 218.978945] x9 : ffff0000112e3018 x8 : 0000000000000123
[ 218.984252] x7 : 0000000000000005 x6 : ffff80063b578180
[ 218.989559] x5 : ffff80063b578180 x4 : 0000000000000000
[ 218.994865] x3 : ffff80063b57ef10 x2 : eed25f279b69f300
[ 219.000172] x1 : eed25f279b69f300 x0 : 0000000000000000
[ 219.005481] Call trace:
[ 219.007923] report_addr+0x38/0xa8
[ 219.011321] dma_direct_map_page+0x148/0x158
[ 219.015586] dma_direct_map_sg+0x78/0xe0
[ 219.019510] usb_hcd_map_urb_for_dma+0x2fc/0x468
[ 219.024124] xhci_map_urb_for_dma+0x54/0x68
[ 219.028303] usb_hcd_submit_urb+0x88/0x968
[ 219.032394] usb_submit_urb+0x3b0/0x570
[ 219.036226] usb_sg_wait+0x98/0x158
[ 219.039711] usb_stor_bulk_transfer_sglist.part.3+0x94/0x128
[ 219.045366] usb_stor_bulk_srb+0x48/0x88
[ 219.049283] usb_stor_Bulk_transport+0x10c/0x390
[ 219.053896] usb_stor_invoke_transport+0x3c/0x500
[ 219.058595] usb_stor_transparent_scsi_command+0xc/0x18
[ 219.063816] usb_stor_control_thread+0x1c4/0x260
[ 219.068431] kthread+0x124/0x128
[ 219.071660] ret_from_fork+0x10/0x18
[ 219.075229] ---[ end trace dd9ef2a6b7fef860 ]---
[ 219.080087] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.090810] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.101510] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.112209] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.122901] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.133591] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.144283] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.154973] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 219.165674] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.861717] swiotlb_tbl_map_single: 67451 callbacks suppressed
[ 223.861721] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.878249] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.888940] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.899630] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.910318] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.921005] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.931695] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.942387] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.953077] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 223.963765] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.865664] swiotlb_tbl_map_single: 70409 callbacks suppressed
[ 228.865668] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.882188] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.892878] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.903567] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.914256] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.924944] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.935636] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.946326] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.957015] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[ 228.967705] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)

Regards,
Biju

> Subject: RE: How to resolve an issue in swiotlb environment?
>
> Hi Christoph,
>
> I think we should continue to discuss on this email thread instead of the fixed
> DMA-API.txt patch [1]
>
> [1]
> https://marc.info/?t=155989412200001&r=1&w=2
>
> > From: Yoshihiro Shimoda, Sent: Monday, June 3, 2019 3:42 PM
> >
> > Hi linux-block and iommu mailing lists,
> >
> > I have an issue that a USB SSD with xHCI on R-Car H3 causes "swiotlb is full"
> like below.
> >
> > [ 36.745286] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 524288
> bytes), total 32768 (slots), used 1338 (slots)
> >
> > I have investigated this issue by using git bisect, and then I found the
> following commit:
> >
> > ---
> > commit 09324d32d2a0843e66652a087da6f77924358e62
> > Author: Christoph Hellwig <hch@lst.de>
> > Date: Tue May 21 09:01:41 2019 +0200
> >
> > block: force an unlimited segment size on queues with a virt
> > boundary
> > ---
>
> Thank you for your comment on other email thread [2] like below:
> ---
> Turns out it isn't as simple as I thought, as there doesn't seem to be an easy
> way to get to the struct device used for DMA mapping from USB drivers. I'll
> need to think a bit more how to handle that best.
> ---
>
> [2]
> https://marc.info/?l=linux-doc&m=155989651620473&w=2
>
> I'm not sure this is a correct way, but the issue disappears if I applied a patch
> below to USB storage driver. Especially, WARNING happened on
> blk_queue_max_segment_size().
> Maybe we need to expand the argument "struct device *" of
> blk_queue_virt_boundary() to call dma_max_mapping_size()?
> ---
> diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
> index 59190d8..fa37b39 100644
> --- a/drivers/usb/storage/scsiglue.c
> +++ b/drivers/usb/storage/scsiglue.c
> @@ -28,6 +28,7 @@
> * status of a command.
> */
>
> +#include <linux/dma-mapping.h>
> #include <linux/module.h>
> #include <linux/mutex.h>
>
> @@ -83,6 +84,15 @@ static int slave_alloc (struct scsi_device *sdev)
> maxp = usb_maxpacket(us->pusb_dev, us->recv_bulk_pipe, 0);
> blk_queue_virt_boundary(sdev->request_queue, maxp - 1);
>
> +{
> + struct device *dev = us->pusb_dev->bus->controller;
> +
> + dev_info(dev, "%s: size = %zu\n", __func__,
> dma_max_mapping_size(dev));
> + blk_queue_max_segment_size(sdev->request_queue,
> + dma_max_mapping_size(dev));
> +}
> +
> +
> /*
> * Some host controllers may have alignment requirements.
> * We'll play it safe by requiring 512-byte alignment always.
> ---
>
> Best regards,
> Yoshihiro Shimoda