System hangs after echo value > /sys/block/dm-0/queue/nr_requests

System hangs after echo value > /sys/block/dm-0/queue/nr_requests

[Miquel van Smoorenburg]

If you echo a value (any value; for example the default 128) to
/sys/block/dm-0/queue/nr_requests the shell you are in hangs.
After about 5 seconds, the whole system hangs 100%.

The system responds to interrupts (ping / sysrq) but that's it.
For the rest it's dead in the water.

This is with Linux 2.6.0 and the deadline IO scheduler.

There is no oops or panic. With sysrq I managed to get a backtrace
of the process doing the echo:

bash          R 00000001    16   657      1                 656 (NOTLB)
f6d45e60 00000082 c2432c80 00000001 00000003 00000000 c03ba4a0 00000000
       00000000 00000000 00000000 00000000 00000000 f738b680 f738b714 c01484d4
       f738b714 00000000 00000000 00000000 00000000 f738b680 00001000 00000000
Call Trace:
 [<c01484d4>] vmtruncate+0x79/0x112
 [<c016f0b2>] inode_setattr+0x106/0x17d
 [<c016f29b>] notify_change+0x119/0x16a
 [<c013b0a5>] unlock_page+0x15/0x57
 [<c01482c2>] do_wp_page+0x2c7/0x2f2
 [<c013ee4b>] buffered_rmqueue+0xc4/0x143
 [<c013ef71>] __alloc_pages+0xa7/0x317
 [<c01ff557>] queue_var_store+0x23/0x31
 [<c01ff690>] queue_requests_store+0x11b/0x17d
 [<c01ff751>] queue_attr_store+0x32/0x36
 [<c0185a3a>] flush_write_buffer+0x3b/0x47
 [<c0185aa0>] sysfs_write_file+0x5a/0x69
 [<c0185a46>] sysfs_write_file+0x0/0x69
 [<c0156300>] vfs_write+0xb0/0x119
 [<c015640e>] sys_write+0x42/0x63
 [<c010b20b>] syscall_call+0x7/0xb


A "show regs" shows:

SysRq : Show Regs
 
Pid: 6, comm:             events/0
EIP: 0060:[<c01197b2>] CPU: 0
EIP is at smp_call_function+0x130/0x15b
 EFLAGS: 00000297    Not tainted
EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: 00000002
ESI: 00000001 EDI: 00000297 EBP: f7fda580 DS: 007b ES: 007b
CR0: 8005003b CR2: 400c6fe0 CR3: 36c1d000 CR4: 000006d0
Call Trace:
 [<c0113780>] mce_checkregs+0x0/0x92
 [<c0113839>] do_mce_timer+0x27/0x2b
 [<c0113780>] mce_checkregs+0x0/0x92
 [<c01319f5>] worker_thread+0x1be/0x283
 [<c0113812>] do_mce_timer+0x0/0x2b
 [<c011f07d>] default_wake_function+0x0/0x12
 [<c011f07d>] default_wake_function+0x0/0x12
 [<c0131837>] worker_thread+0x0/0x283
 [<c0109269>] kernel_thread_helper+0x5/0xb

The sysrq reboot function doesn't work - it you use it, the kernel
stops responding to sysrq requests as well.

Mike.
-- 
When life hands you lemons, grab the salt and pass the tequila.

Re: System hangs after echo value > /sys/block/dm-0/queue/nr_requests

[Andrew Morton]

Miquel van Smoorenburg <miquels@cistron.nl> wrote:
>
> If you echo a value (any value; for example the default 128) to
> /sys/block/dm-0/queue/nr_requests the shell you are in hangs.
> After about 5 seconds, the whole system hangs 100%.

hm, nice.  It does the same thing for /sys/block/md0/queue/nr-requests.

With CONFIG_DEBUG_SPINLOCK enabled we go BUG in __wake_up():

Program received signal SIGTRAP, Trace/breakpoint trap.
__wake_up (q=0xcf299e90, mode=3, nr_exclusive=1) at include/asm/spinlock.h:137
137                     BUG();
(gdb) bt
#0  __wake_up (q=0xcf299e90, mode=3, nr_exclusive=1) at include/asm/spinlock.h:137
#1  0xc02cc847 in queue_requests_store (q=0xcf299df8, page=0xe <Address 0xe out of bounds>, count=14)
    at drivers/block/ll_rw_blk.c:2843
#2  0xc02cc907 in queue_attr_store (kobj=0xcf299f68, attr=0xe, page=0xe <Address 0xe out of bounds>, 
    length=14) at drivers/block/ll_rw_blk.c:2892
#3  0xc01aa84f in flush_write_buffer (file=0xe, buffer=0xc0456100, count=14) at fs/sysfs/file.c:205
#4  0xc01aa8ac in sysfs_write_file (file=0xc0493550, buf=0xe <Address 0xe out of bounds>, count=2, 
    ppos=0xc851cf84) at fs/sysfs/file.c:233
#5  0xc016f458 in vfs_write (file=0xc0493550, buf=0x80b2d00 "128\n", count=4, pos=0xc851cf84)
    at fs/read_write.c:257
#6  0xc016f55e in sys_write (fd=14, buf=0xe <Address 0xe out of bounds>, count=14) at fs/read_write.c:293


Where queue_requests_store() does wake_up(&rl->wait[READ]);

It looks like nobody has called blk_init_queue() for this queue and the
waitqueue head is uninitialised.

No md was in use on this machine: it was simply enabled in kernel config.

Re: System hangs after echo value > /sys/block/dm-0/queue/nr_requests

[Mike Christie]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

Andrew Morton wrote:

> Where queue_requests_store() does wake_up(&rl->wait[READ]);
> 
> It looks like nobody has called blk_init_queue() for this queue and the
> waitqueue head is uninitialised.
> 

DM, MD, rd and loop use blk_alloc_queue and blk_queue_make_request to 
initialize their queue, because they only use the make_request_fn. The 
attached patch prevents the queue from being registered if only 
blk_alloc_queue was called.

Mike Christie

[-- Attachment #2: queue_attr.patch --]
[-- Type: text/plain, Size: 573 bytes --]

--- linux-2.6.0-orig/drivers/block/ll_rw_blk.c	2003-12-28 23:09:16.000000000 -0800
+++ linux-2.6.0/drivers/block/ll_rw_blk.c	2003-12-30 16:11:00.690504036 -0800
@@ -2902,7 +2902,7 @@ int blk_register_queue(struct gendisk *d
 
 	request_queue_t *q = disk->queue;
 
-	if (!q)
+	if (!q || !q->request_fn)
 		return -ENXIO;
 
 	q->kobj.parent = kobject_get(&disk->kobj);
@@ -2929,7 +2929,7 @@ void blk_unregister_queue(struct gendisk
 {
 	request_queue_t *q = disk->queue;
 
-	if (q) {
+	if (q && q->request_fn) {
 		elv_unregister_queue(q);
 
 		kobject_unregister(&q->kobj);

Re: System hangs after echo value > /sys/block/dm-0/queue/nr_requests

[Jens Axboe]

On Tue, Dec 30 2003, Mike Christie wrote:
> Andrew Morton wrote:
> 
> >Where queue_requests_store() does wake_up(&rl->wait[READ]);
> >
> >It looks like nobody has called blk_init_queue() for this queue and the
> >waitqueue head is uninitialised.
> >
> 
> DM, MD, rd and loop use blk_alloc_queue and blk_queue_make_request to 
> initialize their queue, because they only use the make_request_fn. The 
> attached patch prevents the queue from being registered if only 
> blk_alloc_queue was called.

I'm fine with that patch since we don't have any attributes in there
that apply to just remappers.

-- 
Jens Axboe