Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

["Jun'ichi Nomura" <j-nomura@ce.jp.nec.com>]

On 11/02/11 22:47, Heiko Carstens wrote:
> On Wed, Nov 02, 2011 at 09:37:06PM +0900, Jun'ichi Nomura wrote:
>> On 10/31/11 22:00, Heiko Carstens wrote:
>>> On Mon, Oct 31, 2011 at 08:46:06PM +0900, Jun'ichi Nomura wrote:
>>>> Hm, dm_softirq_done is generic completion code of original
>>>> request in dm-multipath.
>>>> So oops here might be another manifestation of use-after-free.
>>>>
>>>> Do you always hit the oops at the same address?
>>>
>>> I think we saw this bug the first time. But before that the scsi
>>> logging level was higher. Gonzalo is trying to recreate it with
>>> the same (old) scsi logging level.
>>> Afterwards we will try with barrier=0.
>>>
>>> Both on v3.0.7 btw.
>>>
>>>> Could you find corresponding source code line for
>>>> the crashed address, dm_softirq_done+0x72/0x140,
>>>> and which pointer was invalid?
>>>
>>> It crashes in the inlined function dm_done() when trying to
>>> dereference tio (aka clone->end_io_data):
>>>
>>> static void dm_done(struct request *clone, int error, bool mapped)
>>> {
>>>         int r = error;
>>>         struct dm_rq_target_io *tio = clone->end_io_data;
>>>         dm_request_endio_fn rq_end_io = tio->ti->type->rq_end_io;
>>
>> Thank you. But, hmm. I have no idea about scenario.
>>
>> struct dm_rq_target_io is a container of clone request
>> and clone->end_io_data points to its container.
>>
>>   struct dm_rq_target_io {
>>         struct mapped_device *md;
>>         struct dm_target *ti;
>>         struct request *orig, clone;
>>         int error;
>>         union map_info info;
>>   };
>>
>> If clone can be dereferenced, clone->end_io_data should be, too.
> 
> If it helps: the above *ti pointer is the only one that points to
> an (invalid) vmalloc area address. Invalid means the page was unmapped
> because it was freed because of DEBUG_PAGEALLOC.
> All other addresses I followed to get to this one belong to
> the 1:1 mapping of the kernel, so no vmalloc involved.

Thanks, ok it was ti which was invalid. Not tio.
ti is a pointer to dm table entry, which is vmalloc-ed.
So it means the dm table was replaced while I/O was in-flight.

dm has a machanism to prevent it: in dm_suspend(),
stop_queue() is called to stop block queue processing
and no new I/O becomes in-flight after that.
Then all in-flight I/Os are waited to be completed or requeued
(dm_wait_for_completion()).
If the wait was successful, the table can become "suspended",
i.e. ready to be replaced.

So ti should be always valid.
Hmm..

-- 
Jun'ichi Nomura, NEC Corporation