Unmapping of UIO logical memory causing a trace

Unmapping of UIO logical memory causing a trace

[Ankit Jindal]

Hi,

We have observed an issue where kmalloc of a small sized memory causes
an occasional trace when unmapping the mmaped memory via UIO framework
This trace is coming when kernel sees a negative value in
page->_mapcount. Trace is pasted at the end of the mail.

After debugging this issue further, we realized following sequence
occurs when kmalloc is used to allocate small memory using slub
allocator:
1. Frozen bit (msb) of the page from which memory has been allocated
is set (which is an union with _mapcount).
2. If there are free objects in the the same page then this frozen bit
remains set even after kernel boots completely.
3. When user space calls unmap of this memory, vma_unmap_single()
treats the _mapcount as a negative (as frozen bit is set), causing a
trace.

We are not sure whether exposing kernel memory of size
less than PAGE_SIZE via UIO is a valid use case ? In case this is an invalid
use case then shouldn't the UIO framework restrict mapping of non
PAGE_SIZE aligned memory and size not in order of PAGE_SIZE.

Trace:

BUG: Bad page map in process test-mumap  pte:2a00040ef92af53 pmd:43ad50a003
page:ffffffbec3be4a80 count:2 mapcount:0x82000201 mapping:
(null) index:0x0
flags: 0x94(referenced|dirty|slab)
page dumped because: bad pte
addr:0000007fad20a000 vm_flags:040400ff anon_vma:          (null)
mapping:ffffffc3ad9c6630 index:3
file:uio0 fault:uio_vma_fault mmap:uio_mmap readpage:          (null)
CPU: 0 PID: 2179 Comm: test-xgene-qmtm Not tainted 4.1.0 #102
Hardware name: APM board (DT)
Call trace:
[<ffffffc000088b40>] dump_backtrace+0x0/0x124
[<ffffffc000088c74>] show_stack+0x10/0x1c
[<ffffffc000908328>] dump_stack+0x84/0xc8
[<ffffffc000150404>] print_bad_pte+0x15c/0x1f8
[<ffffffc000151e90>] unmap_single_vma+0x478/0x678
[<ffffffc000152934>] unmap_vmas+0x64/0xd4
[<ffffffc000157a7c>] unmap_region+0x8c/0x140
[<ffffffc000159b1c>] do_munmap+0x174/0x3c4
[<ffffffc000159dac>] vm_munmap+0x40/0x64
[<ffffffc00015ac20>] SyS_munmap+0x20/0x34

Thanks,
Ankit

Re: Unmapping of UIO logical memory causing a trace

[Greg Kroah-Hartman]

On Tue, Aug 11, 2015 at 04:39:08PM +0530, Ankit Jindal wrote:
> Hi,
> 
> We have observed an issue where kmalloc of a small sized memory causes
> an occasional trace when unmapping the mmaped memory via UIO framework
> This trace is coming when kernel sees a negative value in
> page->_mapcount. Trace is pasted at the end of the mail.
> 
> After debugging this issue further, we realized following sequence
> occurs when kmalloc is used to allocate small memory using slub
> allocator:
> 1. Frozen bit (msb) of the page from which memory has been allocated
> is set (which is an union with _mapcount).
> 2. If there are free objects in the the same page then this frozen bit
> remains set even after kernel boots completely.
> 3. When user space calls unmap of this memory, vma_unmap_single()
> treats the _mapcount as a negative (as frozen bit is set), causing a
> trace.
> 
> We are not sure whether exposing kernel memory of size
> less than PAGE_SIZE via UIO is a valid use case ? In case this is an invalid
> use case then shouldn't the UIO framework restrict mapping of non
> PAGE_SIZE aligned memory and size not in order of PAGE_SIZE.

We've had a few discussions about this in the past, and one proposed
patch which had to be reverted because it broke some working systems, so
it's a messy thing.

What UIO driver are you using that causes this behavior?

thanks,

greg k-h

Re: Unmapping of UIO logical memory causing a trace

[Ankit Jindal]

Hi Greg,

On Wed, Aug 12, 2015 at 3:15 AM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Tue, Aug 11, 2015 at 04:39:08PM +0530, Ankit Jindal wrote:
>> Hi,
>>
>> We have observed an issue where kmalloc of a small sized memory causes
>> an occasional trace when unmapping the mmaped memory via UIO framework
>> This trace is coming when kernel sees a negative value in
>> page->_mapcount. Trace is pasted at the end of the mail.
>>
>> After debugging this issue further, we realized following sequence
>> occurs when kmalloc is used to allocate small memory using slub
>> allocator:
>> 1. Frozen bit (msb) of the page from which memory has been allocated
>> is set (which is an union with _mapcount).
>> 2. If there are free objects in the the same page then this frozen bit
>> remains set even after kernel boots completely.
>> 3. When user space calls unmap of this memory, vma_unmap_single()
>> treats the _mapcount as a negative (as frozen bit is set), causing a
>> trace.
>>
>> We are not sure whether exposing kernel memory of size
>> less than PAGE_SIZE via UIO is a valid use case ? In case this is an invalid
>> use case then shouldn't the UIO framework restrict mapping of non
>> PAGE_SIZE aligned memory and size not in order of PAGE_SIZE.
>
> We've had a few discussions about this in the past, and one proposed
> patch which had to be reverted because it broke some working systems, so
> it's a messy thing.
>
> What UIO driver are you using that causes this behavior?

We have observed this during the development of new UIO driver for our
soc. In our driver, we need to parse non probable properties of device
and provide these details to our user application. For this we do a
kmalloc of device info(approx size 80 bytes) and pass this address to
user space via UIO mem logical.

>
> thanks,
>
> greg k-h

Thanks,
Ankit

Re: Unmapping of UIO logical memory causing a trace

[Greg Kroah-Hartman]

On Wed, Aug 12, 2015 at 01:25:53PM +0530, Ankit Jindal wrote:
> Hi Greg,
> 
> On Wed, Aug 12, 2015 at 3:15 AM, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Tue, Aug 11, 2015 at 04:39:08PM +0530, Ankit Jindal wrote:
> >> Hi,
> >>
> >> We have observed an issue where kmalloc of a small sized memory causes
> >> an occasional trace when unmapping the mmaped memory via UIO framework
> >> This trace is coming when kernel sees a negative value in
> >> page->_mapcount. Trace is pasted at the end of the mail.
> >>
> >> After debugging this issue further, we realized following sequence
> >> occurs when kmalloc is used to allocate small memory using slub
> >> allocator:
> >> 1. Frozen bit (msb) of the page from which memory has been allocated
> >> is set (which is an union with _mapcount).
> >> 2. If there are free objects in the the same page then this frozen bit
> >> remains set even after kernel boots completely.
> >> 3. When user space calls unmap of this memory, vma_unmap_single()
> >> treats the _mapcount as a negative (as frozen bit is set), causing a
> >> trace.
> >>
> >> We are not sure whether exposing kernel memory of size
> >> less than PAGE_SIZE via UIO is a valid use case ? In case this is an invalid
> >> use case then shouldn't the UIO framework restrict mapping of non
> >> PAGE_SIZE aligned memory and size not in order of PAGE_SIZE.
> >
> > We've had a few discussions about this in the past, and one proposed
> > patch which had to be reverted because it broke some working systems, so
> > it's a messy thing.
> >
> > What UIO driver are you using that causes this behavior?
> 
> We have observed this during the development of new UIO driver for our
> soc. In our driver, we need to parse non probable properties of device
> and provide these details to our user application.

What exactly do you mean by this?

> For this we do a kmalloc of device info(approx size 80 bytes) and pass
> this address to user space via UIO mem logical.

For such tiny sizes, why not just use a normal sysfs file?

Do you have a pointer to your driver so that I can see exactly what it
is doing here?

thanks,

greg k-h

Re: Unmapping of UIO logical memory causing a trace

[Ankit Jindal]

Hi Greg,

On Wed, Aug 12, 2015 at 10:41 PM, Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> On Wed, Aug 12, 2015 at 01:25:53PM +0530, Ankit Jindal wrote:
>> Hi Greg,
>>
>> On Wed, Aug 12, 2015 at 3:15 AM, Greg Kroah-Hartman
>> <gregkh@linuxfoundation.org> wrote:
>> > On Tue, Aug 11, 2015 at 04:39:08PM +0530, Ankit Jindal wrote:
>> >> Hi,
>> >>
>> >> We have observed an issue where kmalloc of a small sized memory causes
>> >> an occasional trace when unmapping the mmaped memory via UIO framework
>> >> This trace is coming when kernel sees a negative value in
>> >> page->_mapcount. Trace is pasted at the end of the mail.
>> >>
>> >> After debugging this issue further, we realized following sequence
>> >> occurs when kmalloc is used to allocate small memory using slub
>> >> allocator:
>> >> 1. Frozen bit (msb) of the page from which memory has been allocated
>> >> is set (which is an union with _mapcount).
>> >> 2. If there are free objects in the the same page then this frozen bit
>> >> remains set even after kernel boots completely.
>> >> 3. When user space calls unmap of this memory, vma_unmap_single()
>> >> treats the _mapcount as a negative (as frozen bit is set), causing a
>> >> trace.
>> >>
>> >> We are not sure whether exposing kernel memory of size
>> >> less than PAGE_SIZE via UIO is a valid use case ? In case this is an invalid
>> >> use case then shouldn't the UIO framework restrict mapping of non
>> >> PAGE_SIZE aligned memory and size not in order of PAGE_SIZE.
>> >
>> > We've had a few discussions about this in the past, and one proposed
>> > patch which had to be reverted because it broke some working systems, so
>> > it's a messy thing.
>> >
>> > What UIO driver are you using that causes this behavior?
>>
>> We have observed this during the development of new UIO driver for our
>> soc. In our driver, we need to parse non probable properties of device
>> and provide these details to our user application.
>
> What exactly do you mean by this?

There are certain the device specific properties (coming from device
tree) which we are exposing using UIO_LOGICAL_MEM to the user space.

>
>> For this we do a kmalloc of device info(approx size 80 bytes) and pass
>> this address to user space via UIO mem logical.
>
> For such tiny sizes, why not just use a normal sysfs file?

We did not think of sysfs as an option, since UIO_MEM_LOGICAL seems to
address our use case.

>
> Do you have a pointer to your driver so that I can see exactly what it
> is doing here?

Below is the code snippet of my driver.

struct reserved_qgroup {
        uint32_t qstart;
        uint32_t qend;
        uint32_t valid;
};

struct my_dev_info {
        uint32_t num_queues;
        uint32_t version;
        struct reserved_qgroup resvd_qgrps[MAX_RESVD_QGROUPS];
};

static int my_dev_probe(struct platform_device *pdev) {

struct property *prop;
struct my_dev_info *dev_info;

dev_info = devm_kzalloc(&pdev->dev, sizeof(*dev_info), GFP_KERNEL);

if (!dev_info)
      return -ENOMEM;

prop = of_find_property(pdev->dev.of_node, "reserved-qgroups", NULL);

if (prop) {
        const __be32 *reserved_qid = NULL;
        unsigned int i;
        uint32_t qid_start, qid_end;

        for (i = 0; i < MAX_RESVD_QGROUPS; i++) {
                reserved_qid = of_prop_next_u32(prop, reserved_qid, &qid_start);
                if (!reserved_qid)
                        break;
                reserved_qid = of_prop_next_u32(prop, reserved_qid, &qid_end);
                if (!reserved_qid)
                        break;

                dev_info->resvd_qgrps[i].qstart = qid_start;
                dev_info->resvd_qgrps[i].qend = qid_end;
                dev_info->resvd_qgrps[i].valid  = 1;
        }
}

ret = of_property_read_u32(pdev->dev.of_node, "num-queues",
                dev_info->num_queues);
if (ret < 0) {
        dev_err(&pdev->dev, "No num-queues resource specified\n");
        goto out_err;
}

info->mem[3].name = "dev-info";
info->mem[3].addr = (phys_addr_t) dev_info;
info->mem[3].size = sizeof(struct dev_info);
info->mem[3].memtype = UIO_MEM_LOGICAL;

......
}

Thanks,
Ankit

Re: Unmapping of UIO logical memory causing a trace

[Greg Kroah-Hartman]

On Wed, Aug 19, 2015 at 03:25:09PM +0530, Ankit Jindal wrote:
> Hi Greg,
> 
> On Wed, Aug 12, 2015 at 10:41 PM, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > On Wed, Aug 12, 2015 at 01:25:53PM +0530, Ankit Jindal wrote:
> >> Hi Greg,
> >>
> >> On Wed, Aug 12, 2015 at 3:15 AM, Greg Kroah-Hartman
> >> <gregkh@linuxfoundation.org> wrote:
> >> > On Tue, Aug 11, 2015 at 04:39:08PM +0530, Ankit Jindal wrote:
> >> >> Hi,
> >> >>
> >> >> We have observed an issue where kmalloc of a small sized memory causes
> >> >> an occasional trace when unmapping the mmaped memory via UIO framework
> >> >> This trace is coming when kernel sees a negative value in
> >> >> page->_mapcount. Trace is pasted at the end of the mail.
> >> >>
> >> >> After debugging this issue further, we realized following sequence
> >> >> occurs when kmalloc is used to allocate small memory using slub
> >> >> allocator:
> >> >> 1. Frozen bit (msb) of the page from which memory has been allocated
> >> >> is set (which is an union with _mapcount).
> >> >> 2. If there are free objects in the the same page then this frozen bit
> >> >> remains set even after kernel boots completely.
> >> >> 3. When user space calls unmap of this memory, vma_unmap_single()
> >> >> treats the _mapcount as a negative (as frozen bit is set), causing a
> >> >> trace.
> >> >>
> >> >> We are not sure whether exposing kernel memory of size
> >> >> less than PAGE_SIZE via UIO is a valid use case ? In case this is an invalid
> >> >> use case then shouldn't the UIO framework restrict mapping of non
> >> >> PAGE_SIZE aligned memory and size not in order of PAGE_SIZE.
> >> >
> >> > We've had a few discussions about this in the past, and one proposed
> >> > patch which had to be reverted because it broke some working systems, so
> >> > it's a messy thing.
> >> >
> >> > What UIO driver are you using that causes this behavior?
> >>
> >> We have observed this during the development of new UIO driver for our
> >> soc. In our driver, we need to parse non probable properties of device
> >> and provide these details to our user application.
> >
> > What exactly do you mean by this?
> 
> There are certain the device specific properties (coming from device
> tree) which we are exposing using UIO_LOGICAL_MEM to the user space.

Why would you do that?  Userspace has the device tree info already
through other means.

> >> For this we do a kmalloc of device info(approx size 80 bytes) and pass
> >> this address to user space via UIO mem logical.
> >
> > For such tiny sizes, why not just use a normal sysfs file?
> 
> We did not think of sysfs as an option, since UIO_MEM_LOGICAL seems to
> address our use case.

Not for tiny memory chunks like you have seen :)

> > Do you have a pointer to your driver so that I can see exactly what it
> > is doing here?
> 
> Below is the code snippet of my driver.

<snip>

How about the whole driver please.

thanks,

greg k-h