Re: bade page state while calling munmap() for kmalloc'ed UIO memory

Re: bade page state while calling munmap() for kmalloc'ed UIO memory

[Hans J. Koch]

On Mon, Aug 29, 2011 at 05:05:47PM +0200, Jan Altenberg wrote:

[Since we got no reply on linux-mm, I added lkml and Andrew to Cc: (mm doesn't
seem to have a maintainer...)]

> Hi,
> 
> I'm currently analysing a problem similar to some mmap() issue reported
> in the past: https://lkml.org/lkml/2010/7/11/140

The arch there was microblaze, and you are working on arm. That means
the problem appears on at least to archs.

> 
> So, what I'm trying to do is mapping some physically continuous memory
> (allocated by kmalloc) to userspace, using a trivial UIO driver (the
> idea is that a device can directly DMA to that buffer):
> 
> [...]
> #define MEM_SIZE (4 * PAGE_SIZE)
> 
> addr = kmalloc(MEM_SIZE, GFP_KERNEL)
> [...]
> info.mem[0].addr = (unsigned long) addr;
> info.mem[0].internal_addr = addr;
> info.mem[0].size = MEM_SIZE;
> info.mem[0].memtype = UIO_MEM_LOGICAL;
> [...]
> ret = uio_register_device(&pdev->dev, &info);
> 
> Userspace maps that memory range and writes its contents to a file:
> 
> [...]
> 
> fd = open("/dev/uio0", O_RDWR);
> if (fd < 0) {
>            perror("Can't open UIO device\n");
>            exit(1);
> }
> 
> mem_map = mmap(NULL, MAP_SIZE, PROT_READ | PROT_WRITE,
>                   MAP_PRIVATE, fd, 0);
> 
> if(mem_map == MAP_FAILED) {
>            perror("Can't map UIO memory\n");
>            ret = -ENOMEM;
>            goto out_file;
> }
> [...]
> bytes_written = write(fd_file, mem_map, MAP_SIZE)
> [...]
> 
> munmap(mem_map);

>From my point of view (I've got Jan's full test case code), this
is a completely correct UIO use case.

> 
> So, what happens is (I'm currently testing with 3.0.3 on ARM
> VersatilePB): When I do the munmap(), I run into the following error:
> 
> BUG: Bad page state in process uio_test  pfn:078ed
> page:c0409154 count:0 mapcount:0 mapping:  (null) index:0x0
> page flags: 0x284(referenced|slab|arch_1)
> [<c0033e50>] (unwind_backtrace+0x0/0xe4) from [<c0079938>] (bad_page+0xcc/0xf8)
> [<c0079938>] (bad_page+0xcc/0xf8) from [<c007a5f0>] (free_pages_prepare+0x6c/0xcc)
> [<c007a5f0>] (free_pages_prepare+0x6c/0xcc) from [<c007a778>] (free_hot_cold_page+0x20/0x18c)
> [<c007a778>] (free_hot_cold_page+0x20/0x18c) from [<c008ccb4>] (unmap_vmas+0x338/0x564)
> [<c008ccb4>] (unmap_vmas+0x338/0x564) from [<c008f0f4>] (unmap_region+0xa4/0x1e0)
> [<c008f0f4>] (unmap_region+0xa4/0x1e0) from [<c0090428>] (do_munmap+0x20c/0x274)
> [<c0090428>] (do_munmap+0x20c/0x274) from [<c00904cc>] (sys_munmap+0x3c/0x50)
> [<c00904cc>] (sys_munmap+0x3c/0x50) from [<c002e680>] (ret_fast_syscall+0x0/0x2c)
> Disabling lock debugging due to kernel taint
> BUG: Bad page state in process uio_test  pfn:078ee
> page:c0409178 count:0 mapcount:0 mapping:  (null) index:0x0
> page flags: 0x284(referenced|slab|arch_1)
> [<c0033e50>] (unwind_backtrace+0x0/0xe4) from [<c0079938>] (bad_page+0xcc/0xf8)
> [<c0079938>] (bad_page+0xcc/0xf8) from [<c007a5f0>] (free_pages_prepare+0x6c/0xcc)
> [<c007a5f0>] (free_pages_prepare+0x6c/0xcc) from [<c007a778>] (free_hot_cold_page+0x20/0x18c)
> [<c007a778>] (free_hot_cold_page+0x20/0x18c) from [<c008ccb4>] (unmap_vmas+0x338/0x564)
> [<c008ccb4>] (unmap_vmas+0x338/0x564) from [<c008f0f4>] (unmap_region+0xa4/0x1e0)
> [<c008f0f4>] (unmap_region+0xa4/0x1e0) from [<c0090428>] (do_munmap+0x20c/0x274)
> [<c0090428>] (do_munmap+0x20c/0x274) from [<c00904cc>] (sys_munmap+0x3c/0x50)
> [<c00904cc>] (sys_munmap+0x3c/0x50) from [<c002e680>] (ret_fast_syscall+0x0/0x2c)
> BUG: Bad page state in process uio_test  pfn:078ef
> page:c040919c count:0 mapcount:0 mapping:  (null) index:0x0
> page flags: 0x284(referenced|slab|arch_1)
> [<c0033e50>] (unwind_backtrace+0x0/0xe4) from [<c0079938>] (bad_page+0xcc/0xf8)
> [<c0079938>] (bad_page+0xcc/0xf8) from [<c007a5f0>] (free_pages_prepare+0x6c/0xcc)
> [<c007a5f0>] (free_pages_prepare+0x6c/0xcc) from [<c007a778>] (free_hot_cold_page+0x20/0x18c)
> [<c007a778>] (free_hot_cold_page+0x20/0x18c) from [<c008ccb4>] (unmap_vmas+0x338/0x564)
> [<c008ccb4>] (unmap_vmas+0x338/0x564) from [<c008f0f4>] (unmap_region+0xa4/0x1e0)
> [<c008f0f4>] (unmap_region+0xa4/0x1e0) from [<c0090428>] (do_munmap+0x20c/0x274)
> [<c0090428>] (do_munmap+0x20c/0x274) from [<c00904cc>] (sys_munmap+0x3c/0x50)
> [<c00904cc>] (sys_munmap+0x3c/0x50) from [<c002e680>] (ret_fast_syscall+0x0/0x2c)

Quite strange that memory that could be mapped with mmap() cannot be
unmapped with munmap().

> 
> This happens for every page except the first one.

...which is the next strange thing.

> If I change the code
> and just touch the first page, everything's working fine. As soon as I
> touch one of the other pages, I can see the "bad page state error" for
> that page.

The pages are mapped when you access them through the UIO core page fault
handler.

> The kernel is currently built using CONFIG_SLAB (my .config
> is based on the versatile_defconfig); if I change to CONFIG_SLUB,
> munmap() seems to be happy and I can't see the "bad page state" error.

That is more than strange, that points to some things going really wrong.

> 
> Any idea what might be wrong here? Am I missing something obvious? (I've
> prepared some brown paperbags for that case ;-))
> 
> Thanks,
> 	Jan
> 

Thanks, Jan, for reporting this!

Hans

Re: bade page state while calling munmap() for kmalloc'ed UIO memory

[Andrew Morton]

On Wed, 31 Aug 2011 11:58:25 +0200
"Hans J. Koch" <hjk@hansjkoch.de> wrote:

> On Mon, Aug 29, 2011 at 05:05:47PM +0200, Jan Altenberg wrote:
> 
> [Since we got no reply on linux-mm, I added lkml and Andrew to Cc: (mm doesn't
> seem to have a maintainer...)]
> 
> > Hi,
> > 
> > I'm currently analysing a problem similar to some mmap() issue reported
> > in the past: https://lkml.org/lkml/2010/7/11/140
> 
> The arch there was microblaze, and you are working on arm. That means
> the problem appears on at least to archs.
> 
> > 
> > So, what I'm trying to do is mapping some physically continuous memory
> > (allocated by kmalloc) to userspace, using a trivial UIO driver (the
> > idea is that a device can directly DMA to that buffer):
> > 
> > [...]
> > #define MEM_SIZE (4 * PAGE_SIZE)
> > 
> > addr = kmalloc(MEM_SIZE, GFP_KERNEL)
> > [...]
> > info.mem[0].addr = (unsigned long) addr;
> > info.mem[0].internal_addr = addr;
> > info.mem[0].size = MEM_SIZE;
> > info.mem[0].memtype = UIO_MEM_LOGICAL;
> > [...]
> > ret = uio_register_device(&pdev->dev, &info);
> > 
> > Userspace maps that memory range and writes its contents to a file:
> > 
> > [...]
> > 
> > fd = open("/dev/uio0", O_RDWR);
> > if (fd < 0) {
> >            perror("Can't open UIO device\n");
> >            exit(1);
> > }
> > 
> > mem_map = mmap(NULL, MAP_SIZE, PROT_READ | PROT_WRITE,
> >                   MAP_PRIVATE, fd, 0);
> > 
> > if(mem_map == MAP_FAILED) {
> >            perror("Can't map UIO memory\n");
> >            ret = -ENOMEM;
> >            goto out_file;
> > }
> > [...]
> > bytes_written = write(fd_file, mem_map, MAP_SIZE)
> > [...]
> > 
> > munmap(mem_map);
> 
> >From my point of view (I've got Jan's full test case code), this
> is a completely correct UIO use case.
> 
> > 
> > So, what happens is (I'm currently testing with 3.0.3 on ARM
> > VersatilePB): When I do the munmap(), I run into the following error:
> > 
> > BUG: Bad page state in process uio_test  pfn:078ed
> > page:c0409154 count:0 mapcount:0 mapping:  (null) index:0x0
> > page flags: 0x284(referenced|slab|arch_1)

PG_slab is set.  The kernel is complaining because a page which was
allocated via kmalloc/kmem_cache_alloc was directly passed to the page
allocator for freeing.  It should have been passed to kfree().

Presumably the uio driver expects that its memory was allocated via
alloc_pages(), not via kmalloc().

Re: bade page state while calling munmap() for kmalloc'ed UIO memory

[Hans J. Koch]

On Wed, Aug 31, 2011 at 04:13:07PM -0700, Andrew Morton wrote:
> On Wed, 31 Aug 2011 11:58:25 +0200
> "Hans J. Koch" <hjk@hansjkoch.de> wrote:
> 
> > On Mon, Aug 29, 2011 at 05:05:47PM +0200, Jan Altenberg wrote:
> > 
> > [Since we got no reply on linux-mm, I added lkml and Andrew to Cc: (mm doesn't
> > seem to have a maintainer...)]
> > 
> > > Hi,
> > > 
> > > I'm currently analysing a problem similar to some mmap() issue reported
> > > in the past: https://lkml.org/lkml/2010/7/11/140
> > 
> > The arch there was microblaze, and you are working on arm. That means
> > the problem appears on at least to archs.
> > 
> > > 
> > > So, what I'm trying to do is mapping some physically continuous memory
> > > (allocated by kmalloc) to userspace, using a trivial UIO driver (the
> > > idea is that a device can directly DMA to that buffer):
> > > 
> > > [...]
> > > #define MEM_SIZE (4 * PAGE_SIZE)
> > > 
> > > addr = kmalloc(MEM_SIZE, GFP_KERNEL)
> > > [...]
> > > info.mem[0].addr = (unsigned long) addr;
> > > info.mem[0].internal_addr = addr;
> > > info.mem[0].size = MEM_SIZE;
> > > info.mem[0].memtype = UIO_MEM_LOGICAL;
> > > [...]
> > > ret = uio_register_device(&pdev->dev, &info);
> > > 
> > > Userspace maps that memory range and writes its contents to a file:
> > > 
> > > [...]
> > > 
> > > fd = open("/dev/uio0", O_RDWR);
> > > if (fd < 0) {
> > >            perror("Can't open UIO device\n");
> > >            exit(1);
> > > }
> > > 
> > > mem_map = mmap(NULL, MAP_SIZE, PROT_READ | PROT_WRITE,
> > >                   MAP_PRIVATE, fd, 0);
> > > 
> > > if(mem_map == MAP_FAILED) {
> > >            perror("Can't map UIO memory\n");
> > >            ret = -ENOMEM;
> > >            goto out_file;
> > > }
> > > [...]
> > > bytes_written = write(fd_file, mem_map, MAP_SIZE)
> > > [...]
> > > 
> > > munmap(mem_map);
> > 
> > >From my point of view (I've got Jan's full test case code), this
> > is a completely correct UIO use case.
> > 
> > > 
> > > So, what happens is (I'm currently testing with 3.0.3 on ARM
> > > VersatilePB): When I do the munmap(), I run into the following error:
> > > 
> > > BUG: Bad page state in process uio_test  pfn:078ed
> > > page:c0409154 count:0 mapcount:0 mapping:  (null) index:0x0
> > > page flags: 0x284(referenced|slab|arch_1)
> 
> PG_slab is set.  The kernel is complaining because a page which was
> allocated via kmalloc/kmem_cache_alloc was directly passed to the page
> allocator for freeing.  It should have been passed to kfree().
> 
> Presumably the uio driver expects that its memory was allocated via
> alloc_pages(), not via kmalloc().

Thanks for that hint. I'll check and will update UIO documentation
accordingly if necessary.

BUT the following problems are still threatening my mental health:

If userspace gets a valid and working pointer from mmap(), it is
entitled to expect munmap() to work on that pointer, too, isn't it?
Shouldn't mmap() fail in such a case?

The kernel's behavior should be the same, no matter which SLAB or
SLUB is chosen.

Or am I following wrong philosophies?

Thanks,
Hans