From: Alex Elder <elder@inktank.com>
To: Andrey Korolyov <andrey@xdel.ru>
Cc: ceph-devel@vger.kernel.org
Subject: Re: 'rbd map' asynchronous behavior
Date: Tue, 15 May 2012 08:23:55 -0500 [thread overview]
Message-ID: <4FB258EB.7080904@inktank.com> (raw)
In-Reply-To: <CABYiri-Whv7tGLVb-AcwMKcA1=iMTyguOYokg2xKt9r6hzKRuQ@mail.gmail.com>
On 05/15/2012 06:49 AM, Andrey Korolyov wrote:
> Hi,
>
> There are strange bug when I tried to map excessive amounts of block
> devices inside the pool, like following
>
> for vol in $(rbd ls); do rbd map $vol; [some-microsleep]; [some
> operation or nothing, I have stubbed guestfs mount here] ;
> [some-microsleep]; unmap /dev/rbd/rbd/$vol ; [some-microsleep]; done,
This is most likely due to a recently-fixed problem.
The fix is found in this commit, although there were
other changes that led up to it:
32eec68d2f rbd: don't drop the rbd_id too early
It is present starting in Linux kernel 3.3; it appears
you are running 2.6?
-Alex
> udev or rbd seems to be somehow late and mapping fails. There is no
> real-world harm at all, and such case can be easily avoided, but on
> busy cluster timeout increases and I was able to catch same thing on
> two-osd config in recovering state. For 0.1 second on healthy cluster,
> all works okay, for 0.05 it may fail with following trace(just for me,
> because I am testing on relatively old and crappy hardware, so others
> may catch that on smaller intervals):
>
> [ 2130.450044] libceph: client0 fsid 70204128-4328-47e7-9df7-c7253c833fc1
> [ 2130.450643] libceph: mon0 192.168.10.129:6789 session established
> [ 2130.454542] rbd0: p1 p2
> [ 2130.454772] rbd: rbd0: added with size 0x80000000
> [ 2137.783484] libceph: client0 fsid 70204128-4328-47e7-9df7-c7253c833fc1
> [ 2137.784095] libceph: mon0 192.168.10.129:6789 session established
> [ 2137.787801] rbd0: p1 p2
> [ 2137.788028] rbd: rbd0: added with size 0x7d000000
> [ 2138.044490] ------------[ cut here ]------------
> [ 2138.044499] WARNING: at
> /build/kernel/linux-2.6-3.2.15/debian/build/source_amd64_none/fs/sysfs/dir.c:481
> sysfs_add_one+0x83/0x96()
> [ 2138.044503] Hardware name: System Product Name
> [ 2138.044505] sysfs: cannot create duplicate filename
> '/devices/virtual/block/rbd0'
> [ 2138.044508] Modules linked in: ip6table_filter ip6_tables
> iptable_filter acpi_cpufreq mperf ip_tables cpufreq_powersave
> ebtable_nat cpufreq_userspace ebtables x_tables cpufreq_conservative
> cpufreq_stats cn microcode ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa
> ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> fuse nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc kvm_intel kvm
> bridge stp ext2 rbd libceph coretemp tcp_yeah tcp_vegas loop dm_crypt
> snd_hda_codec_realtek nvidia(P) snd_hda_intel snd_hda_codec snd_hwdep
> nv_tco snd_pcm psmouse i2c_nforce2 i2c_core snd_page_alloc evdev
> serio_raw snd_timer snd pcspkr soundcore processor button asus_atk0110
> ext4 crc16 jbd2 mbcache btrfs crc32c libcrc32c zlib_deflate dm_mod
> sd_mod sr_mod cdrom crc_t10dif ohci_hcd ata_generic pata_amd sata_nv
> ehci_hcd libata fan forcedeth thermal thermal_sys scsi_mod usbcore
> usb_common [last unloaded: scsi_wait_scan]
> [ 2138.044607] Pid: 16891, comm: rbd Tainted: P O 3.2.0-2-amd64 #1
> [ 2138.044610] Call Trace:
> [ 2138.044616] [<ffffffff81046811>] ? warn_slowpath_common+0x78/0x8c
> [ 2138.044620] [<ffffffff810468bd>] ? warn_slowpath_fmt+0x45/0x4a
> [ 2138.044624] [<ffffffff8114e918>] ? sysfs_add_one+0x83/0x96
> [ 2138.044628] [<ffffffff8114e991>] ? create_dir+0x66/0xa0
> [ 2138.044631] [<ffffffff8114ea66>] ? sysfs_create_dir+0x85/0x9b
> [ 2138.044636] [<ffffffff811afb6b>] ? vsnprintf+0x7c/0x427
> [ 2138.044640] [<ffffffff811a9aa2>] ? kobject_add_internal+0xc8/0x181
> [ 2138.044643] [<ffffffff811a9e77>] ? kobject_add+0x95/0xa4
> [ 2138.044647] [<ffffffff810363c7>] ? should_resched+0x5/0x23
> [ 2138.044651] [<ffffffff811a99d5>] ? kobject_get+0x12/0x17
> [ 2138.044655] [<ffffffff8119ccb4>] ? get_disk+0x8d/0x8d
> [ 2138.044659] [<ffffffff8124bd13>] ? device_add+0xd6/0x587
> [ 2138.044663] [<ffffffff8124ad4e>] ? dev_set_name+0x42/0x47
> [ 2138.044667] [<ffffffff8119d8db>] ? register_disk+0x37/0x147
> [ 2138.044670] [<ffffffff8119ce48>] ? blk_register_region+0x22/0x27
> [ 2138.044674] [<ffffffff8119db6b>] ? add_disk+0x180/0x26c
> [ 2138.044681] [<ffffffffa0e994d0>] ? rbd_add+0x7b2/0xa32 [rbd]
> [ 2138.044685] [<ffffffff810363c7>] ? should_resched+0x5/0x23
> [ 2138.044689] [<ffffffff8114d477>] ? sysfs_write_file+0xe0/0x11c
> [ 2138.044693] [<ffffffff810f92a7>] ? vfs_write+0xa2/0xe9
> [ 2138.044697] [<ffffffff810f9484>] ? sys_write+0x45/0x6b
> [ 2138.044701] [<ffffffff8134e492>] ? system_call_fastpath+0x16/0x1b
> [ 2138.044704] ---[ end trace b7a29490cafc363d ]---
> [ 2138.044708] kobject_add_internal failed for rbd0 with -EEXIST,
> don't try to register things with the same name in the same directory.
> [ 2138.044723] Pid: 16891, comm: rbd Tainted: P W O 3.2.0-2-amd64 #1
> [ 2138.044725] Call Trace:
> [ 2138.044729] [<ffffffff811a9b31>] ? kobject_add_internal+0x157/0x181
> [ 2138.044733] [<ffffffff811a9e77>] ? kobject_add+0x95/0xa4
> [ 2138.044736] [<ffffffff810363c7>] ? should_resched+0x5/0x23
> [ 2138.044740] [<ffffffff811a99d5>] ? kobject_get+0x12/0x17
> [ 2138.044743] [<ffffffff8119ccb4>] ? get_disk+0x8d/0x8d
> [ 2138.044746] [<ffffffff8124bd13>] ? device_add+0xd6/0x587
> [ 2138.044750] [<ffffffff8124ad4e>] ? dev_set_name+0x42/0x47
> [ 2138.044757] [<ffffffff8119d8db>] ? register_disk+0x37/0x147
> [ 2138.044760] [<ffffffff8119ce48>] ? blk_register_region+0x22/0x27
> [ 2138.044763] [<ffffffff8119db6b>] ? add_disk+0x180/0x26c
> [ 2138.044769] [<ffffffffa0e994d0>] ? rbd_add+0x7b2/0xa32 [rbd]
> [ 2138.044772] [<ffffffff810363c7>] ? should_resched+0x5/0x23
> [ 2138.044776] [<ffffffff8114d477>] ? sysfs_write_file+0xe0/0x11c
> [ 2138.044780] [<ffffffff810f92a7>] ? vfs_write+0xa2/0xe9
> [ 2138.044783] [<ffffffff810f9484>] ? sys_write+0x45/0x6b
> [ 2138.044787] [<ffffffff8134e492>] ? system_call_fastpath+0x16/0x1b
> [ 2138.044928] ------------[ cut here ]------------
> [ 2138.044937] kernel BUG at
> /build/kernel/linux-2.6-3.2.15/debian/build/source_amd64_none/fs/sysfs/group.c:65!
> [ 2138.044947] invalid opcode: 0000 [#1] SMP
> [ 2138.044962] CPU 1
> [ 2138.044967] Modules linked in: ip6table_filter ip6_tables
> iptable_filter acpi_cpufreq mperf ip_tables cpufreq_powersave
> ebtable_nat cpufreq_userspace ebtables x_tables cpufreq_conservative
> cpufreq_stats cn microcode ib_iser rdma_cm ib_addr iw_cm ib_cm ib_sa
> ib_mad ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi
> fuse nfsd nfs nfs_acl auth_rpcgss fscache lockd sunrpc kvm_intel kvm
> bridge stp ext2 rbd libceph coretemp tcp_yeah tcp_vegas loop dm_crypt
> snd_hda_codec_realtek nvidia(P) snd_hda_intel snd_hda_codec snd_hwdep
> nv_tco snd_pcm psmouse i2c_nforce2 i2c_core snd_page_alloc evdev
> serio_raw snd_timer snd pcspkr soundcore processor button asus_atk0110
> ext4 crc16 jbd2 mbcache btrfs crc32c libcrc32c zlib_deflate dm_mod
> sd_mod sr_mod cdrom crc_t10dif ohci_hcd ata_generic pata_amd sata_nv
> ehci_hcd libata fan forcedeth thermal thermal_sys scsi_mod usbcore
> usb_common [last unloaded: scsi_wait_scan]
> [ 2138.045366]
> [ 2138.045380] Pid: 16891, comm: rbd Tainted: P W O
> 3.2.0-2-amd64 #1 System manufacturer System Product Name/P5N-D
> [ 2138.045420] RIP: 0010:[<ffffffff8114fdb7>] [<ffffffff8114fdb7>]
> internal_create_group+0x27/0x11f
> [ 2138.045454] RSP: 0018:ffff8800b19e3d38 EFLAGS: 00010246
> [ 2138.045472] RAX: 00000000ffffffef RBX: ffff8800a19aac00 RCX: 0000000000002019
> [ 2138.045491] RDX: ffffffff81624cb0 RSI: 0000000000000000 RDI: ffff8800a19aac78
> [ 2138.045511] RBP: ffff8800a19aac78 R08: 0000000000000002 R09: 00000000fffffffe
> [ 2138.045530] R10: 0000000000000001 R11: 0000000000000002 R12: ffffffff81624cb0
> [ 2138.045549] R13: ffff8800a19aac68 R14: 0000000000000000 R15: ffff880037770038
> [ 2138.045569] FS: 00007ffc5219a760(0000) GS:ffff88012fc80000(0000)
> knlGS:0000000000000000
> [ 2138.045598] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2138.045616] CR2: 00007f16285260f2 CR3: 00000000a1946000 CR4: 00000000000006e0
> [ 2138.045635] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 2138.045655] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 2138.045675] Process rbd (pid: 16891, threadinfo ffff8800b19e2000,
> task ffff880117c76830)
> [ 2138.045702] Stack:
> [ 2138.045716] 0000000000000000 0000000000000010 ffff8800b19e3d98
> 0000000069dd99c3
> [ 2138.045755] ffff88006f63d3c0 ffff8800a19aac00 ffff880037770038
> ffff880037770038
> [ 2138.045793] ffff8800a19aac68 ffff8800a19aac00 ffff880037770038
> ffffffff811990b7
> [ 2138.045831] Call Trace:
> [ 2138.045849] [<ffffffff811990b7>] ? blk_register_queue+0x41/0xe1
> [ 2138.045868] [<ffffffff8119db73>] ? add_disk+0x188/0x26c
> [ 2138.045889] [<ffffffffa0e994d0>] ? rbd_add+0x7b2/0xa32 [rbd]
> [ 2138.045909] [<ffffffff810363c7>] ? should_resched+0x5/0x23
> [ 2138.045928] [<ffffffff8114d477>] ? sysfs_write_file+0xe0/0x11c
> [ 2138.045947] [<ffffffff810f92a7>] ? vfs_write+0xa2/0xe9
> [ 2138.045965] [<ffffffff810f9484>] ? sys_write+0x45/0x6b
> [ 2138.045984] [<ffffffff8134e492>] ? system_call_fastpath+0x16/0x1b
> [ 2138.046002] Code: 59 5b 5d c3 41 57 41 56 41 89 f6 41 55 41 54 49
> 89 d4 55 48 89 fd 53 48 83 ec 28 48 85 ff 74 0b 85 f6 75 09 48 83 7f
> 30 00 75 12<0f> 0b 48 83 7f 30 00 b8 ea ff ff ff 0f 84 d7 00 00 00 49
> 8b 34
> [ 2138.046248] RIP [<ffffffff8114fdb7>] internal_create_group+0x27/0x11f
> [ 2138.046270] RSP<ffff8800b19e3d38>
> [ 2138.046587] ---[ end trace b7a29490cafc363e ]---
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
next prev parent reply other threads:[~2012-05-15 13:23 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-05-15 11:49 'rbd map' asynchronous behavior Andrey Korolyov
2012-05-15 13:23 ` Alex Elder [this message]
2012-05-15 15:40 ` Josh Durgin
2012-05-16 8:24 ` Andrey Korolyov
2012-05-25 10:15 ` Andrey Korolyov
2012-05-25 20:07 ` Greg Farnum
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FB258EB.7080904@inktank.com \
--to=elder@inktank.com \
--cc=andrey@xdel.ru \
--cc=ceph-devel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox