Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

Bart Van Assche
2016-06-15T07:40:23Z

On 06/14/2016 08:41 PM, Laurence Oberman wrote:
> This may be a data point here
> After each change I have rebooted the host as its required.
> I am at swiotlb=16 and after the first reboot with maxed out tuning I had no alerts.
> On the second controller restart without a system reboot I got them again.
>
> Again, I never see these other than when I am in the reconnect loop, and they seem to be non-intrusive as each time I recover fully.
>
> When I first changed to 4 and had not increased the ib_srp paramaters I had two restarts with no messages so that was what led me to report that this seems to have worked.
> I can see now that this was not the case and already mentioned, the claim that the change fixed this was wrong.
> Apologies for that.
>
> I am continuing to research and debug now.

Hello Laurence,

In the kernel source tree I found the following:

From include/linux/swiotlb.h:

#define IO_TLB_SHIFT 11

From lib/swiotlb.c:

#define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
[ ... ]
#define IO_TLB_DEFAULT_SIZE (64UL<<20)
[ ... ]
static int __init
setup_io_tlb_npages(char *str)
{
if (isdigit(*str)) {
io_tlb_nslabs = simple_strtoul(str, &str, 0);
/* avoid tail segment of size < IO_TLB_SEGSIZE */
io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
}
[ ... ]
}
early_param("swiotlb", setup_io_tlb_npages);
[ ... ]
void __init
swiotlb_init(int verbose)
{
size_t default_size = IO_TLB_DEFAULT_SIZE;
[ ... ]

if (!io_tlb_nslabs) {
io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
}
[ ... ]
}

I think this means that the swiotlb parameter has to be set to a value
above 32768 to increase the number of swiotlb buffers above the default.

Bart.