Re: uml segfault during I/O

[Anton Ivanov <anton.ivanov@cambridgegreys.com>]

On 18/10/2019 11:51, Anton Ivanov wrote:
>
>
> On 18/10/2019 10:55, Anton Ivanov wrote:
>> Adding Jens and Christoph
>>
>> On 18/10/2019 08:35, Anton Ivanov wrote:
>>>
>>>
>>> On 17/10/2019 17:29, Anton Ivanov wrote:
>>>>
>>>>
>>>> On 17/10/2019 16:03, Anton Ivanov wrote:
>>>>>
>>>>>
>>>>> On 17/10/2019 14:33, Anton Ivanov wrote:
>>>>>>
>>>>>>
>>>>>> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>>>>>>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>>>>>>> Looking into that. I have not run into anything like that, but 
>>>>>>>> I have
>>>>>>>> not used any of the legacy networking for 5 odd years now.
>>>>>>>>
>>>>>>>
>>>>>>> Do you think this is related to networking ? I ask because there 
>>>>>>> was no
>>>>>>> network activity going on.
>>>>>>
>>>>>> no, it's disk somewhere. I managed to reproduce it with 5.2 stock 
>>>>>> on Debian 5.2 host.
>>>>>>
>>>>>>>
>>>>>>> apt is just an example. The packages were all already downloaded 
>>>>>>> and
>>>>>>> there was no network activity. Rather, there was block I/O.
>>>>>>
>>>>>> I am looking into that.
>>>>>>
>>>>>>>
>>>>>>> One thing I noticed, which may or may not be useful to this 
>>>>>>> report. I
>>>>>>> was booting the uml guest and immediately logging into it and 
>>>>>>> running
>>>>>>> the block I/O. And the segfault would occur.
>>>>>>>
>>>>>>> If I, instead, booted the uml vm and let it remain idle for a 
>>>>>>> minute or
>>>>>>> so and then did the I/O, it worked fine. So I am not sure if 
>>>>>>> there is
>>>>>>> any lazy initialization happening in the background which gets
>>>>>>> corrupted during immediate hot boot I/O.
>>>>>>
>>>>>> Interesting...
>>>>>>
>>>>>>>
>>>>>>> On the other hand, if you think there can be any number of 
>>>>>>> commands to
>>>>>>> run locally that could generate more information, please assist 
>>>>>>> me so.
>>>>>>
>>>>>> As I said - I managed to reproduce it, I am looking at it. In 
>>>>>> first instance I am trying with a couple of version up/down to 
>>>>>> see if this is 5.2 specific.
>>>>>
>>>>> I cannot even get it to start on 5.4-rc1, 5.3 shows the same 
>>>>> symptoms.
>>>>
>>>> This is something outside the UBD driver as such. There were no 
>>>> notable changes to it since we ported it to block-mq and added 
>>>> DISCARD which was quite a while back.
>>>>
>>>> I am going to check the other usual suspects such as IRQs, but that 
>>>> is something I test quite extensively when working on the 
>>>> networking side.
>>>>
>>>> So I suspect that this is something outside UML which is showing 
>>>> only in UML for some reason.
>>>
>>> Still happening with 5.2.21, albeit a bit more difficult to reproduce.
>>>
>>> Looking at the backtraces it is ALWAYS a result of a re-queue.
>>>
>>> [  263.990000]  [<60440633>] blk_mq_dispatch_rq_list+0xf3/0x5f0
>>> [  263.990000]  [<60440501>] ? blk_mq_get_driver_tag+0xc1/0x100
>>> [  263.990000]  [<60440540>] ? blk_mq_dispatch_rq_list+0x0/0x5f0
>>> [  263.990000]  [<60445966>] blk_mq_do_dispatch_sched+0x66/0xe0
>>> [  263.990000]  [<60446107>] blk_mq_sched_dispatch_requests+0x107/0x190
>>> [  263.990000]  [<6043f130>] ? blk_mq_run_hw_queue+0x0/0x120
>>> [  263.990000]  [<6043efb4>] __blk_mq_run_hw_queue+0x74/0xd0
>>> [  263.990000]  [<6043f0d4>] __blk_mq_delay_run_hw_queue+0xc4/0xd0
>>> [  263.990000]  [<6043f1d9>] blk_mq_run_hw_queue+0xa9/0x120
>>> [  263.990000]  [<6043f292>] blk_mq_run_hw_queues+0x42/0x60
>>> [  263.990000]  [<60440bc0>] ? blk_mq_request_bypass_insert+0x0/0x90
>>> [  263.990000]  [<604462a0>] ? blk_mq_sched_insert_request+0x0/0x1c0
>>> ALWAYS >>[  263.990000]  [<604414b0>] blk_mq_requeue_work+0x160/0x170
>>> [  263.990000]  [<6008c91b>] process_one_work+0x1eb/0x490
>>> [  263.990000]  [<607f5aa0>] ? __schedule+0x0/0x620
>>> [  263.990000]  [<6008e000>] ? wq_worker_running+0x10/0x40
>>> [  263.990000]  [<6008c730>] ? process_one_work+0x0/0x490
>>> [  263.990000]  [<6008cc06>] worker_thread+0x46/0x670
>>> [  263.990000]  [<600931c1>] ? __kthread_parkme+0xa1/0xd0
>>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>>> [  263.990000]  [<60093bc4>] kthread+0x194/0x1c0
>>> [  263.990000]  [<6002a091>] new_thread_handler+0x81/0xc0
>>>
>>>
>>> A.
>>>
>>>>
>>>> A.
>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ritesh
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> linux-um mailing list
>>>>>>> linux-um@lists.infradead.org
>>>>>>> http://lists.infradead.org/mailman/listinfo/linux-um
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> To put a long story short we have a reproducible segfault when UML 
>> re-queues a block request in 5.x
>>
>> This used to work in 4.x
>>
>> We found a couple of minor things in UML when looking at it which we 
>> will fix shortly, but the root problem seems to be either in block_mq 
>> or in the way we are using it to re-queue requests.
>>
>
> My (probably wrong) guess is that it is something related to the 
> changes from 4.x to 5.x where blk-mq is no longer getting the hw 
> context out of the cpu map.
>
> For some reason it gets a null deref at some point.
>
> I may be wrong of course.
>
> I will not be able to pick this up again before Tuesday so if someone 
> can have a go before then it will be greatly appreciated.
>
Unless I am mistaken It should not re-queue.

Xen does not: 
https://elixir.bootlin.com/linux/latest/source/drivers/block/xen-blkfront.c#L910

Unless I am mistaken, if you return a form of BUSY the the upper layers 
will re-queue for you.

That explains the crash as we have the same bio enqueued multiple times 
when the device stalls.

I am testing a patch and if it all tests out I will submit it shortly.

-- 

Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um