[uml-devel] "Sleeping in atomic section" with ubd aio patch

[uml-devel] "Sleeping in atomic section" with ubd aio patch

[Blaisorblade]

Today I was running latest 2.6 UML git snapshot I have (from 8 September), 
compiled with spinlock debugging, and got a big number of long series of 
traces like this one (they varied a bit, but the core is 
do_io->kmem_cache_alloc).

It's just because things are called under spinlocks and GFP_KERNEL is used - 
but if possible, it'd be better to use something better than GFP_ATOMIC 
(since this is performance critical).

Debug: sleeping function called from invalid context 
at /home/paolo/Admin/kernel/6/VCS/linux-2.6.git/mm/slab.c:2097
in_atomic():0, irqs_disabled():1
a05bb5f0:  [<a0015c52>] dump_stack+0x22/0x30
a05bb610:  [<a0048ddc>] __might_sleep+0xac/0xd0
a05bb630:  [<a0094a38>] kmem_cache_alloc+0x98/0xb0
a05bb660:  [<a003b266>] do_io+0x3b6/0x520
a05bb760:  [<a003a6ba>] do_ubd_request+0x12a/0x180
a05bb7f0:  [<a01d146d>] __generic_unplug_device+0x2d/0x40
a05bb800:  [<a01d1500>] generic_unplug_device+0x80/0x180
a05bb830:  [<a01d1621>] blk_backing_dev_unplug+0x21/0x30
a05bb840:  [<a00bca6a>] block_sync_page+0x3a/0x50
a05bb850:  [<a0087556>] sync_page+0x46/0x60
a05bb860:  [<a02c79e9>] __wait_on_bit_lock+0x49/0x70
a05bb880:  [<a0087e39>] __lock_page+0x89/0xa0
a05bb900:  [<a008866a>] do_generic_mapping_read+0x3ca/0x770
a05bb9e0:  [<a0088e5e>] __generic_file_aio_read+0x26e/0x2c0
a05bba60:  [<a0088f03>] generic_file_aio_read+0x53/0x90
a05bba90:  [<a00b53bc>] do_sync_read+0xdc/0x130
a05bbb60:  [<a00b54f3>] vfs_read+0xe3/0x290
a05bbba0:  [<a00b5a2b>] sys_read+0x4b/0x80
a05bbbd0:  [<a001e687>] handle_syscall+0x117/0x140
a05bbc70:  [<a001d5a6>] userspace+0x236/0x3c0
a05bbcf0:  [<a001e1d8>] fork_handler+0xb8/0xd0
a05bbd20:  [<ffffe420>] _etext+0x5fd3398a/0x0

-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

	

	
		
___________________________________ 
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB 
http://mail.yahoo.it



-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] Re: "Sleeping in atomic section" with ubd aio patch

[Jeff Dike]

On Tue, Sep 13, 2005 at 08:01:40PM +0200, Blaisorblade wrote:
> It's just because things are called under spinlocks and GFP_KERNEL is used - 
> but if possible, it'd be better to use something better than GFP_ATOMIC 
> (since this is performance critical).

This would be ubd_io_lock, which is taken for us by the block layer before 
entering do_ubd_request.  We could make that a GFP_ATOMIC allocation, except
that we have already dequeued the request, so would need to do
something with it if the allocation failed.

I'm wondering if we can drop the lock before do_io and pick it up
afterwards.  The bitmap argument is a global, which is a problem.  Two
racing requests could result in one writing out a bitmap which is
either too old or too new.

So, I'm not sure how to handle this.  We could requeue what's left of
the request if an allocation failure happens.  We could drop the lock
and pick it up later, but we have to be careful to write out the
current bitmap, not one which we saved earlier or one which has been
updated to reflect a racing request which is coming through later.

				Jeff




-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

[uml-devel] Re: "Sleeping in atomic section" with ubd aio patch

[Blaisorblade]

On Wednesday 14 September 2005 21:22, Jeff Dike wrote:
> On Tue, Sep 13, 2005 at 08:01:40PM +0200, Blaisorblade wrote:
> > It's just because things are called under spinlocks and GFP_KERNEL is
> > used - but if possible, it'd be better to use something better than
> > GFP_ATOMIC (since this is performance critical).

> This would be ubd_io_lock, which is taken for us by the block layer before
> entering do_ubd_request.  We could make that a GFP_ATOMIC allocation,
> except that we have already dequeued the request, so would need to do
> something with it if the allocation failed.

> I'm wondering if we can drop the lock before do_io and pick it up
> afterwards.  The bitmap argument
the COW bitmap, right?
> is a global, which is a problem.  Two 
> racing requests could result in one writing out a bitmap which is
> either too old or too new.

> So, I'm not sure how to handle this.  We could requeue what's left of
> the request if an allocation failure happens.
Assuming it can be done, it'd be ok.

However, why have we already dequeued the request?

Looking for examples, via grep blkdev_dequeue_request drivers/ide/*.c and 
going opening drivers/ide/ide-io.c, I found blkdev_dequeue_request() called 
*after* end_that_request_first(), while there it's called before.

A comment in blk_start_pre_flush() says there's some freedom, but probably 
leaving the request in the queue would be ok....
> We could drop the lock 
> and pick it up later, but we have to be careful to write out the
> current bitmap, not one which we saved earlier or one which has been
> updated to reflect a racing request which is coming through later.
If you work on the second, be also sure to ask Jens Axboe (see MAINTAINERS for 
email) whether it's legal for the block layer.

CC'ing him on the change in first place would not have been a bad idea, either 
(IMHO).
> 				Jeff

-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

		
___________________________________ 
Yahoo! Messenger: chiamate gratuite in tutto il mondo 
http://it.messenger.yahoo.com



-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: "Sleeping in atomic section" with ubd aio patch

[Jeff Dike]

On Wed, Sep 14, 2005 at 10:00:39PM +0200, Blaisorblade wrote:
> Assuming it can be done, it'd be ok.
> 
> However, why have we already dequeued the request?

It is easier than removing segments from the request and finally
dequeueing the request when it is finally empty.

> Looking for examples, via grep blkdev_dequeue_request drivers/ide/*.c and 
> going opening drivers/ide/ide-io.c, I found blkdev_dequeue_request() called 
> *after* end_that_request_first(), while there it's called before.
> 
> A comment in blk_start_pre_flush() says there's some freedom, but probably 
> leaving the request in the queue would be ok....

There is a good amount of freedom.  The driver is given a set of
requests, and has the job of making them happen.

				Jeff


-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel

Re: [uml-devel] Re: "Sleeping in atomic section" with ubd aio patch

[Blaisorblade]

On Wednesday 14 September 2005 23:18, Jeff Dike wrote:
> On Wed, Sep 14, 2005 at 10:00:39PM +0200, Blaisorblade wrote:
> > Assuming it can be done, it'd be ok.
> >
> > However, why have we already dequeued the request?
>
> It is easier than removing segments from the request and finally
> dequeueing the request when it is finally empty.
Why can't you just move that exact call later, without removing segments?

However, even if it's "easier" this way, IMHO it's still easier to defer the 
removal than to readd the thing into the queue (assuming the driver is 
supposed to do this).

Not that I've looked well at the code or at the API recently. I'm busy with 
other stuff.
> > Looking for examples, via grep blkdev_dequeue_request drivers/ide/*.c and
> > going opening drivers/ide/ide-io.c, I found blkdev_dequeue_request()
> > called *after* end_that_request_first(), while there it's called before.

> > A comment in blk_start_pre_flush() says there's some freedom, but
> > probably leaving the request in the queue would be ok....

> There is a good amount of freedom.  The driver is given a set of
> requests, and has the job of making them happen.

> 				Jeff

-- 
Inform me of my mistakes, so I can keep imitating Homer Simpson's "Doh!".
Paolo Giarrusso, aka Blaisorblade (Skype ID "PaoloGiarrusso", ICQ 215621894)
http://www.user-mode-linux.org/~blaisorblade

	

	
		
___________________________________ 
Yahoo! Mail: gratis 1GB per i messaggi e allegati da 10MB 
http://mail.yahoo.it



-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
User-mode-linux-devel mailing list
User-mode-linux-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/user-mode-linux-devel