Re: uml segfault during I/O

[Anton Ivanov <anton.ivanov@cambridgegreys.com>]

On 18/10/2019 10:55, Anton Ivanov wrote:
> Adding Jens and Christoph
> 
> On 18/10/2019 08:35, Anton Ivanov wrote:
>>
>>
>> On 17/10/2019 17:29, Anton Ivanov wrote:
>>>
>>>
>>> On 17/10/2019 16:03, Anton Ivanov wrote:
>>>>
>>>>
>>>> On 17/10/2019 14:33, Anton Ivanov wrote:
>>>>>
>>>>>
>>>>> On 17/10/2019 14:30, Ritesh Raj Sarraf wrote:
>>>>>> On Thu, 2019-10-17 at 14:02 +0100, Anton Ivanov wrote:
>>>>>>> Looking into that. I have not run into anything like that, but I 
>>>>>>> have
>>>>>>> not used any of the legacy networking for 5 odd years now.
>>>>>>>
>>>>>>
>>>>>> Do you think this is related to networking ? I ask because there 
>>>>>> was no
>>>>>> network activity going on.
>>>>>
>>>>> no, it's disk somewhere. I managed to reproduce it with 5.2 stock 
>>>>> on Debian 5.2 host.
>>>>>
>>>>>>
>>>>>> apt is just an example. The packages were all already downloaded and
>>>>>> there was no network activity. Rather, there was block I/O.
>>>>>
>>>>> I am looking into that.
>>>>>
>>>>>>
>>>>>> One thing I noticed, which may or may not be useful to this report. I
>>>>>> was booting the uml guest and immediately logging into it and running
>>>>>> the block I/O. And the segfault would occur.
>>>>>>
>>>>>> If I, instead, booted the uml vm and let it remain idle for a 
>>>>>> minute or
>>>>>> so and then did the I/O, it worked fine. So I am not sure if there is
>>>>>> any lazy initialization happening in the background which gets
>>>>>> corrupted during immediate hot boot I/O.
>>>>>
>>>>> Interesting...
>>>>>
>>>>>>
>>>>>> On the other hand, if you think there can be any number of 
>>>>>> commands to
>>>>>> run locally that could generate more information, please assist me 
>>>>>> so.
>>>>>
>>>>> As I said - I managed to reproduce it, I am looking at it. In first 
>>>>> instance I am trying with a couple of version up/down to see if 
>>>>> this is 5.2 specific.
>>>>
>>>> I cannot even get it to start on 5.4-rc1, 5.3 shows the same symptoms.
>>>
>>> This is something outside the UBD driver as such. There were no 
>>> notable changes to it since we ported it to block-mq and added 
>>> DISCARD which was quite a while back.
>>>
>>> I am going to check the other usual suspects such as IRQs, but that 
>>> is something I test quite extensively when working on the networking 
>>> side.
>>>
>>> So I suspect that this is something outside UML which is showing only 
>>> in UML for some reason.
>>
>> Still happening with 5.2.21, albeit a bit more difficult to reproduce.
>>
>> Looking at the backtraces it is ALWAYS a result of a re-queue.
>>
>> [  263.990000]  [<60440633>] blk_mq_dispatch_rq_list+0xf3/0x5f0
>> [  263.990000]  [<60440501>] ? blk_mq_get_driver_tag+0xc1/0x100
>> [  263.990000]  [<60440540>] ? blk_mq_dispatch_rq_list+0x0/0x5f0
>> [  263.990000]  [<60445966>] blk_mq_do_dispatch_sched+0x66/0xe0
>> [  263.990000]  [<60446107>] blk_mq_sched_dispatch_requests+0x107/0x190
>> [  263.990000]  [<6043f130>] ? blk_mq_run_hw_queue+0x0/0x120
>> [  263.990000]  [<6043efb4>] __blk_mq_run_hw_queue+0x74/0xd0
>> [  263.990000]  [<6043f0d4>] __blk_mq_delay_run_hw_queue+0xc4/0xd0
>> [  263.990000]  [<6043f1d9>] blk_mq_run_hw_queue+0xa9/0x120
>> [  263.990000]  [<6043f292>] blk_mq_run_hw_queues+0x42/0x60
>> [  263.990000]  [<60440bc0>] ? blk_mq_request_bypass_insert+0x0/0x90
>> [  263.990000]  [<604462a0>] ? blk_mq_sched_insert_request+0x0/0x1c0
>> ALWAYS >>[  263.990000]  [<604414b0>] blk_mq_requeue_work+0x160/0x170
>> [  263.990000]  [<6008c91b>] process_one_work+0x1eb/0x490
>> [  263.990000]  [<607f5aa0>] ? __schedule+0x0/0x620
>> [  263.990000]  [<6008e000>] ? wq_worker_running+0x10/0x40
>> [  263.990000]  [<6008c730>] ? process_one_work+0x0/0x490
>> [  263.990000]  [<6008cc06>] worker_thread+0x46/0x670
>> [  263.990000]  [<600931c1>] ? __kthread_parkme+0xa1/0xd0
>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>> [  263.990000]  [<6008cbc0>] ? worker_thread+0x0/0x670
>> [  263.990000]  [<60093bc4>] kthread+0x194/0x1c0
>> [  263.990000]  [<6002a091>] new_thread_handler+0x81/0xc0
>>
>>
>> A.
>>
>>>
>>> A.
>>>
>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Ritesh
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> linux-um mailing list
>>>>>> linux-um@lists.infradead.org
>>>>>> http://lists.infradead.org/mailman/listinfo/linux-um
>>>>>>
>>>>>
>>>>
>>>
>>
> 
> To put a long story short we have a reproducible segfault when UML 
> re-queues a block request in 5.x
> 
> This used to work in 4.x
> 
> We found a couple of minor things in UML when looking at it which we 
> will fix shortly, but the root problem seems to be either in block_mq or 
> in the way we are using it to re-queue requests.
> 

My (probably wrong) guess is that it is something related to the changes 
from 4.x to 5.x where blk-mq is no longer getting the hw context out of 
the cpu map.

For some reason it gets a null deref at some point.

I may be wrong of course.

I will not be able to pick this up again before Tuesday so if someone 
can have a go before then it will be greatly appreciated.

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um