Re: Deadlock under load with Linux 5.9 and other recent kernels

From: Jens Axboe <axboe@kernel.dk>
To: Christian Hewitt <christianshewitt@gmail.com>
Cc: linux-block@vger.kernel.org, linux-amlogic@lists.infradead.org,
	linux-usb@vger.kernel.org, furkan@fkardame.com,
	Brad Harper <bjharper@gmail.com>
Subject: Re: Deadlock under load with Linux 5.9 and other recent kernels
Date: Sat, 26 Sep 2020 06:13:56 -0600	[thread overview]
Message-ID: <c431ae48-6913-acc0-dc0a-b52065e9eaed@kernel.dk> (raw)
In-Reply-To: <EE8C801E-0AD8-43E2-9C65-92CD512904BE@gmail.com>

On 9/26/20 5:55 AM, Christian Hewitt wrote:
>>
>> On 26 Sep 2020, at 2:51 pm, Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 9/26/20 1:55 AM, Christian Hewitt wrote:
>>> I am using an ARM SBC device with Amlogic S922X chip (Beelink
>>> GS-King-X, an Android STB) to boot the Kodi mediacentre distro
>>> LibreELEC (which I work on) although the issue is also reproducible
>>> with Manjaro and Armbian on the same hardware, and with the GT-King
>>> and GT-King Pro devices from the same vendor - all three devices are
>>> using a common dtsi:
>>>
>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-gsking-x.dts
>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-gtking-pro.dts
>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-gtking.dts
>>> https://github.com/chewitt/linux/blob/amlogic-5.9-integ/arch/arm64/boot/dts/amlogic/meson-g12b-w400.dtsi
>>>
>>> I have schematics for the devices, but can only share those privately
>>> on request.
>>>
>>> For testing I am booting LibreELEC from SD card. The box has a 4TB
>>> SATA drive internally connected with a USB > SATA bridge, see dmesg:
>>> http://ix.io/2yLh and I connect a USB stick with a 4GB ISO file that I
>>> copy to the internal SATA drive. Within 10-20 seconds of starting the
>>> copy the box deadlocks needing a hard power cycle to recover. The
>>> timing of the deadlock is variable but the device _always_ deadlocks.
>>> Although I am using a simple copy use-case, there are similar reports
>>> in Armbian forums performing tasks like installs/updates that involve
>>> I/O loads.
>>>
>>> Following advice in the #linux-amlogic IRC channel I added
>>> CONFIG_SOFTLOCKUP_DETECTOR and CONFIG_DETECT_HUNG_TASK and was able to
>>> get output on the HDMI screen (it is not possible to connect to UART
>>> pins without destroying the box case). If you advance the following
>>> video frame by frame in VLC you can see the output:
>>>
>>> https://www.dropbox.com/s/klvcizim8cs5lze/lockup_clip.mov?dl=0
>>
>> Try with this patch:
>>
>> https://lore.kernel.org/linux-block/20200925191902.543953-1-shakeelb@google.com/
> 
> It still locks up approx. 25 seconds into the copy operation. Here’s the output in video again (a little blurry):
> 
> https://www.dropbox.com/s/3j2czaq509arg6g/lockup_clip2.mov?dl=0

Can you try and set CONFIG_SLUB in your .config instead of CONFIG_SLAB?

Also, just take a picture, should be easier to get readable than a video.
And the static trace is all that is needed.

-- 
Jens Axboe

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic