From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrey Borzenkov Subject: Re: [Linux-fbdev-devel] [2.6.29-rc2] fb_mmap: circular locking dependency on hibernation Date: Sun, 1 Feb 2009 10:15:36 +0300 Message-ID: <200902011015.45562.arvidjaar@mail.ru> References: <200901272137.57757.arvidjaar@mail.ru> <200901300716.06137.arvidjaar@mail.ru> <498485E0.4080501@gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1966327.fp6U0JGfBU"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <498485E0.4080501@gmail.com> Sender: linux-kernel-owner@vger.kernel.org List-ID: To: righi.andrea@gmail.com Cc: Geert Uytterhoeven , Linux Frame Buffer Device Development , "Antonino A. Daplas" , linux-pm@lists.linux-foundation.org, Linux Kernel Development --nextPart1966327.fp6U0JGfBU Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On 31 of January 2009 20:09:52 Andrea Righi wrote: > On 2009-01-30 05:15, Andrey Borzenkov wrote: > > On 29 of January 2009 12:10:11 Geert Uytterhoeven wrote: > >> On Tue, 27 Jan 2009, Andrey Borzenkov wrote: > >>> I do not know if this is new; I usually do not hibernate at all > >>> and was testing another driver actually. I do not remember such > >>> issues in 2.6.28 though. Dmesg follows; config attached. > >>> > >>> [11715.652498] > >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D > >>> [11715.652520] [ INFO: possible circular locking dependency > >>> detected ] [11715.652534] 2.6.29-rc2-1avb #11 > >>> [11715.652541] > >>> ------------------------------------------------------- > >>> [11715.652550] s2disk/25519 is trying to acquire lock: > >>> [11715.652562] (&fb_info->lock){--..}, at: [] > >>> fb_mmap+0x97/0x170 [11715.652623] > >>> [11715.652625] but task is already holding lock: > >>> [11715.652632] (&mm->mmap_sem){----}, at: [] > >>> sys_mmap2+0x8e/0xc0 [11715.652669] > >>> [11715.652671] which lock already depends on the new lock. > >>> [11715.652675] > >>> [11715.652681] > >>> [11715.652683] the existing dependency chain (in reverse order) > >>> is: [11715.652691] > >>> [11715.652693] -> #1 (&mm->mmap_sem){----}: > >>> [11715.652707] [] __lock_acquire+0x129f/0x1930 > >>> [11715.652730] [] lock_acquire+0x5c/0x80 > >>> [11715.652742] [] might_fault+0x77/0xa0 > >>> [11715.652764] [] copy_from_user+0x35/0x130 > >>> [11715.652782] [] fb_ioctl+0x1d5/0x3f0 > >>> [11715.652797] [] vfs_ioctl+0x20/0x80 > >>> [11715.652832] [] do_vfs_ioctl+0x72/0x570 > >>> [11715.652845] [] sys_ioctl+0x39/0x70 > >>> [11715.652857] [] sysenter_do_call+0x12/0x31 > >>> [11715.652870] [] 0xffffffff > >>> [11715.652973] > >>> [11715.652975] -> #0 (&fb_info->lock){--..}: > >>> [11715.652988] [] __lock_acquire+0x140c/0x1930 > >>> [11715.653000] [] lock_acquire+0x5c/0x80 > >>> [11715.653011] [] mutex_lock_nested+0xba/0x2f0 > >>> [11715.653032] [] fb_mmap+0x97/0x170 > >>> [11715.653044] [] mmap_region+0x1d6/0x530 > >>> [11715.653067] [] do_mmap_pgoff+0x1d3/0x2f0 > >>> [11715.653079] [] sys_mmap2+0xad/0xc0 > >>> [11715.653091] [] sysenter_do_call+0x12/0x31 > >>> [11715.653102] [] 0xffffffff > >>> [11715.653116] > >>> [11715.653119] other info that might help us debug this: > >>> [11715.653122] > >>> [11715.653130] 1 lock held by s2disk/25519: > >>> [11715.653137] #0: (&mm->mmap_sem){----}, at: [] > >>> sys_mmap2+0x8e/0xc0 > >>> [11715.653156] > >>> [11715.653158] stack backtrace: > >>> [11715.653170] Pid: 25519, comm: s2disk Not tainted > >>> 2.6.29-rc2-1avb #11 [11715.653179] Call Trace: > >>> [11715.653190] [] ? printk+0x18/0x20 > >>> [11715.653202] [] print_circular_bug_tail+0xcf/0xe0 > >>> [11715.653213] [] __lock_acquire+0x140c/0x1930 > >>> [11715.653224] [] ? trace_hardirqs_on+0xb/0x10 > >>> [11715.653242] [] ? _spin_unlock_irqrestore+0x35/0x60 > >>> [11715.653254] [] ? lock_release_holdtime+0x35/0x210 > >>> [11715.653265] [] lock_acquire+0x5c/0x80 > >>> [11715.653277] [] ? fb_mmap+0x97/0x170 > >>> [11715.653287] [] mutex_lock_nested+0xba/0x2f0 > >>> [11715.653298] [] ? fb_mmap+0x97/0x170 > >>> [11715.653309] [] ? fb_mmap+0x97/0x170 > >>> [11715.653335] [] ? kmem_cache_alloc+0xad/0x100 > >>> [11715.653347] [] fb_mmap+0x97/0x170 > >>> [11715.653358] [] mmap_region+0x1d6/0x530 > >>> [11715.653371] [] ? > >>> arch_get_unmapped_area_topdown+0x10d/0x170 [11715.653383] > >>> [] do_mmap_pgoff+0x1d3/0x2f0 > >>> [11715.653395] [] sys_mmap2+0xad/0xc0 > >>> [11715.653406] [] sysenter_do_call+0x12/0x31 > >> > >> Is this fixed by http://patchwork.kernel.org/patch/3259/? > > > > No. It just makes stack a bit more deep :) > > > > [ 408.267429] > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D [=20 > > 408.267448] [ INFO: possible circular locking dependency detected ] > > [ 408.267462] 2.6.29-rc3-1avb #14 > > [ 408.267469] > > ------------------------------------------------------- [=20 > > 408.267478] s2disk/5695 is trying to acquire lock: > > [ 408.267487] (&fb_info->lock){--..}, at: [] > > fb_mmap+0x97/0x170 > > [ 408.267544] > > [ 408.267546] but task is already holding lock: > > [ 408.267553] (&mm->mmap_sem){----}, at: [] > > sys_mmap2+0x8e/0xc0 > > [ 408.267591] > > [ 408.267593] which lock already depends on the new lock. > > [ 408.267596] > > [ 408.267603] > > [ 408.267604] the existing dependency chain (in reverse order) is: > > [ 408.267612] > > [ 408.267614] -> #3 (&mm->mmap_sem){----}: > > [ 408.267628] [] __lock_acquire+0x129f/0x1930 > > [ 408.267650] [] lock_acquire+0x5c/0x80 > > [ 408.267661] [] might_fault+0x77/0xa0 > > [ 408.267683] [] copy_to_user+0x36/0x120 > > [ 408.267699] [] filldir+0x97/0xe0 > > [ 408.267735] [] sysfs_readdir+0x129/0x220 > > [ 408.267751] [] vfs_readdir+0x86/0xa0 > > [ 408.267763] [] sys_getdents+0x68/0xc0 > > [ 408.267775] [] syscall_call+0x7/0xb > > [ 408.267787] [] 0xffffffff > > [ 408.267862] > > [ 408.267864] -> #2 (sysfs_mutex){--..}: > > [ 408.267877] [] __lock_acquire+0x129f/0x1930 > > [ 408.267889] [] lock_acquire+0x5c/0x80 > > [ 408.267899] [] mutex_lock_nested+0xba/0x2f0 > > [ 408.267921] [] sysfs_addrm_start+0x2c/0xc0 > > [ 408.267933] [] create_dir+0x40/0x90 > > [ 408.267944] [] sysfs_create_dir+0x2b/0x50 > > [ 408.267955] [] kobject_add_internal+0xbc/0x1b0 > > [ 408.267984] [] kobject_add_varg+0x31/0x50 > > [ 408.267996] [] kobject_add+0x2c/0x60 > > [ 408.268008] [] device_add+0xa8/0x550 > > [ 408.268041] [] device_register+0x12/0x20 > > [ 408.268053] [] device_create_vargs+0xab/0xc0 > > [ 408.268065] [] device_create+0x28/0x30 > > [ 408.268077] [] register_con_driver+0xed/0x130 > > [ 408.268090] [] take_over_console+0x1b/0x50 > > [ 408.268102] [] fbcon_takeover+0x5d/0xb0 > > [ 408.268125] [] fbcon_event_notify+0x820/0x900 > > [ 408.268137] [] notifier_call_chain+0x53/0xa0 > > [ 408.268161] [] > > __blocking_notifier_call_chain+0x44/0x60 > > [ 408.268184] [] > > blocking_notifier_call_chain+0x1a/0x20 > > [ 408.268197] [] fb_notifier_call_chain+0x11/0x20 > > [ 408.268209] [] register_framebuffer+0x168/0x220 > > [ 408.268223] [] vesafb_probe+0x542/0x783 > > [ 408.268250] [] platform_drv_probe+0xf/0x20 > > [ 408.268269] [] driver_probe_device+0x87/0x1b0 > > [ 408.268281] [] __device_attach+0x8/0x10 > > [ 408.268291] [] bus_for_each_drv+0x5b/0x80 > > [ 408.268308] [] device_attach+0x76/0x80 > > [ 408.268319] [] bus_attach_device+0x47/0x70 > > [ 408.268331] [] device_add+0x323/0x550 > > [ 408.268342] [] platform_device_add+0x175/0x1c0 > > [ 408.268358] [] vesafb_init+0x9a/0x1ec > > [ 408.268370] [] do_one_initcall+0x2a/0x160 > > [ 408.268382] [] kernel_init+0x83/0xd5 > > [ 408.268411] [] kernel_thread_helper+0x7/0x10 > > [ 408.268423] [] 0xffffffff > > [ 408.268436] > > [ 408.268438] -> #1 ((fb_notifier_list).rwsem){----}: > > [ 408.268451] [] __lock_acquire+0x129f/0x1930 > > [ 408.268463] [] lock_acquire+0x5c/0x80 > > [ 408.268474] [] down_read+0x49/0x90 > > [ 408.268484] [] > > __blocking_notifier_call_chain+0x2a/0x60 > > [ 408.268497] [] > > blocking_notifier_call_chain+0x1a/0x20 > > [ 408.268509] [] fb_notifier_call_chain+0x11/0x20 > > [ 408.268521] [] do_fb_ioctl+0x2ae/0x5d0 > > [ 408.268533] [] fb_ioctl+0x1d/0x20 > > [ 408.268545] [] vfs_ioctl+0x20/0x80 > > [ 408.268558] [] do_vfs_ioctl+0x72/0x570 > > [ 408.268570] [] sys_ioctl+0x39/0x70 > > [ 408.268581] [] sysenter_do_call+0x12/0x31 > > [ 408.268592] [] 0xffffffff > > [ 408.268623] > > [ 408.268625] -> #0 (&fb_info->lock){--..}: > > [ 408.268638] [] __lock_acquire+0x140c/0x1930 > > [ 408.268649] [] lock_acquire+0x5c/0x80 > > [ 408.268661] [] mutex_lock_nested+0xba/0x2f0 > > [ 408.268672] [] fb_mmap+0x97/0x170 > > [ 408.268684] [] mmap_region+0x1d6/0x530 > > [ 408.268706] [] do_mmap_pgoff+0x1d3/0x2f0 > > [ 408.268718] [] sys_mmap2+0xad/0xc0 > > [ 408.268729] [] sysenter_do_call+0x12/0x31 > > [ 408.268740] [] 0xffffffff > > [ 408.268752] > > [ 408.268754] other info that might help us debug this: > > [ 408.268758] > > [ 408.268767] 1 lock held by s2disk/5695: > > [ 408.268773] #0: (&mm->mmap_sem){----}, at: [] > > sys_mmap2+0x8e/0xc0 > > [ 408.268792] > > [ 408.268794] stack backtrace: > > [ 408.268805] Pid: 5695, comm: s2disk Not tainted 2.6.29-rc3-1avb > > #14 [ 408.268814] Call Trace: > > [ 408.268825] [] ? printk+0x18/0x20 > > [ 408.268836] [] print_circular_bug_tail+0xcf/0xe0 > > [ 408.268847] [] __lock_acquire+0x140c/0x1930 > > [ 408.268862] [] ? futex_wait+0x170/0x450 > > [ 408.268881] [] ? try_to_wake_up+0x10c/0x120 > > [ 408.268894] [] ? _spin_unlock_irqrestore+0x35/0x60 > > [ 408.268906] [] ? lock_release_holdtime+0x35/0x210 > > [ 408.268917] [] lock_acquire+0x5c/0x80 > > [ 408.268928] [] ? fb_mmap+0x97/0x170 > > [ 408.268939] [] mutex_lock_nested+0xba/0x2f0 > > [ 408.268950] [] ? fb_mmap+0x97/0x170 > > [ 408.268961] [] ? fb_mmap+0x97/0x170 > > [ 408.268987] [] ? kmem_cache_alloc+0xad/0x100 > > [ 408.268999] [] fb_mmap+0x97/0x170 > > [ 408.269010] [] mmap_region+0x1d6/0x530 > > [ 408.269023] [] ? > > arch_get_unmapped_area_topdown+0x10d/0x170 > > [ 408.269035] [] do_mmap_pgoff+0x1d3/0x2f0 > > [ 408.269047] [] sys_mmap2+0xad/0xc0 > > [ 408.269058] [] sysenter_do_call+0x12/0x31 > > Anyway, looking at this trace in detail, it seems that in the fbcon > notifier the FB_EVENT_SET_CONSOLE_MAP handler doesn't use fb_info at > all. > > So, just for this particular case, we could unlock fb_info->lock > before calling the routine that might acquire mm->mmap_sem, and avoid > the circular locking dependency. > > Any chance to do a test with the following patch? > Step by step ... :) With both patches applied: [ 113.949848] =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D [ 113.950054] [ INFO: possible circular locking dependency detected ] [ 113.950054] 2.6.29-rc3-1avb #16 [ 113.950054] ------------------------------------------------------- [ 113.950054] s2disk/4168 is trying to acquire lock: [ 113.950054] (&fb_info->lock){--..}, at: []=20 fbcon_event_notify+0x2d6/0x950 [ 113.950054] [ 113.950054] but task is already holding lock: [ 113.950054] ((fb_notifier_list).rwsem){----}, at: []=20 __blocking_notifier_call_chain+0x2a/0x60 [ 113.950054] [ 113.950054] which lock already depends on the new lock. [ 113.950054] [ 113.950054] [ 113.950054] the existing dependency chain (in reverse order) is: [ 113.950054] [ 113.950054] -> #1 ((fb_notifier_list).rwsem){----}: [ 113.950054] [] __lock_acquire+0x129f/0x1930 [ 113.950054] [] lock_acquire+0x5c/0x80 [ 113.950054] [] down_read+0x49/0x90 [ 113.950054] [] __blocking_notifier_call_chain+0x2a/0x60 [ 113.950054] [] blocking_notifier_call_chain+0x1a/0x20 [ 113.950054] [] fb_notifier_call_chain+0x11/0x20 [ 113.950054] [] do_fb_ioctl+0x2ae/0x5d0 [ 113.950054] [] fb_ioctl+0x1d/0x20 [ 113.950054] [] vfs_ioctl+0x20/0x80 [ 113.950054] [] do_vfs_ioctl+0x72/0x570 [ 113.950054] [] sys_ioctl+0x39/0x70 [ 113.950054] [] sysenter_do_call+0x12/0x31 [ 113.950054] [] 0xffffffff [ 113.950054] [ 113.950054] -> #0 (&fb_info->lock){--..}: [ 113.950054] [] __lock_acquire+0x140c/0x1930 [ 113.950054] [] lock_acquire+0x5c/0x80 [ 113.950054] [] mutex_lock_nested+0xba/0x2f0 [ 113.950054] [] fbcon_event_notify+0x2d6/0x950 [ 113.950054] [] notifier_call_chain+0x53/0xa0 [ 113.950054] [] __blocking_notifier_call_chain+0x44/0x60 [ 113.950054] [] blocking_notifier_call_chain+0x1a/0x20 [ 113.950054] [] fb_notifier_call_chain+0x11/0x20 [ 113.950054] [] do_fb_ioctl+0x3da/0x5d0 [ 113.950054] [] fb_ioctl+0x1d/0x20 [ 113.950054] [] vfs_ioctl+0x20/0x80 [ 113.950054] [] do_vfs_ioctl+0x72/0x570 [ 113.950054] [] sys_ioctl+0x39/0x70 [ 113.950054] [] sysenter_do_call+0x12/0x31 [ 113.950054] [] 0xffffffff [ 113.950054] [ 113.950054] other info that might help us debug this: [ 113.950054] [ 113.950054] 1 lock held by s2disk/4168: [ 113.950054] #0: ((fb_notifier_list).rwsem){----}, at: []=20 __blocking_notifier_call_chain+0x2a/0x60 [ 113.950054] [ 113.950054] stack backtrace: [ 113.950054] Pid: 4168, comm: s2disk Not tainted 2.6.29-rc3-1avb #16 [ 113.950054] Call Trace: [ 113.950054] [] ? printk+0x18/0x20 [ 113.950054] [] print_circular_bug_tail+0xcf/0xe0 [ 113.950054] [] __lock_acquire+0x140c/0x1930 [ 113.950054] [] ? lock_release_holdtime+0x35/0x210 [ 113.950054] [] ? check_noncircular+0x92/0x150 [ 113.950054] [] ? lock_release_non_nested+0x16e/0x2a0 [ 113.950054] [] ? __blocking_notifier_call_chain+0x2a/0x60 [ 113.950054] [] lock_acquire+0x5c/0x80 [ 113.950054] [] ? fbcon_event_notify+0x2d6/0x950 [ 113.950054] [] mutex_lock_nested+0xba/0x2f0 [ 113.950054] [] ? fbcon_event_notify+0x2d6/0x950 [ 113.950054] [] ? fbcon_event_notify+0x2d6/0x950 [ 113.950054] [] fbcon_event_notify+0x2d6/0x950 [ 113.950054] [] ? save_stack_trace+0x2b/0x50 [ 113.950054] [] ? save_trace+0x3c/0xa0 [ 113.950054] [] ? add_lock_to_list+0x3d/0xb0 [ 113.950054] [] ? lock_release_holdtime+0x35/0x210 [ 113.950054] [] ? lock_release_non_nested+0x9d/0x2a0 [ 113.950054] [] ? __blocking_notifier_call_chain+0x2a/0x60 [ 113.950054] [] notifier_call_chain+0x53/0xa0 [ 113.950054] [] ? down_read+0x65/0x90 [ 113.950054] [] __blocking_notifier_call_chain+0x44/0x60 [ 113.950054] [] blocking_notifier_call_chain+0x1a/0x20 [ 113.950054] [] fb_notifier_call_chain+0x11/0x20 [ 113.950054] [] do_fb_ioctl+0x3da/0x5d0 [ 113.950054] [] ? might_fault+0x46/0xa0 [ 113.950054] [] ? might_fault+0x46/0xa0 [ 113.950054] [] ? might_fault+0x91/0xa0 [ 113.950054] [] ? might_fault+0x46/0xa0 [ 113.950054] [] ? lock_release_holdtime+0x35/0x210 [ 113.950054] [] ? lock_release_non_nested+0x9d/0x2a0 [ 113.950054] [] ? might_fault+0x46/0xa0 [ 113.950054] [] ? might_fault+0x46/0xa0 [ 113.950054] [] ? might_fault+0x91/0xa0 [ 113.950054] [] ? might_fault+0x46/0xa0 [ 113.950054] [] ? copy_to_user+0x36/0x120 [ 113.950054] [] ? fb_ioctl+0x0/0x20 [ 113.950054] [] fb_ioctl+0x1d/0x20 [ 113.950054] [] vfs_ioctl+0x20/0x80 [ 113.950054] [] do_vfs_ioctl+0x72/0x570 [ 113.950054] [] ? sys_fstat64+0x22/0x30 [ 113.950054] [] ? __link_path_walk+0x240/0xd40 [ 113.950054] [] ? sysenter_exit+0xf/0x16 [ 113.950054] [] sys_ioctl+0x39/0x70 [ 113.950054] [] sysenter_do_call+0x12/0x31 [ 117.846986] Syncing filesystems ... done. In case it helps - suspend to RAM does not exhibit this issue. > Thanks, > -Andrea > > --- > fbcon: avoid circular locking dependency between fb_info->lock and > mm->mmap_sem > > In fbcon notifier the handler for FB_EVENT_SET_CONSOLE_MAP doesn't > need to hold fb_info->lock. > > Simply unlock it before calling set_con2fb_map(), that could try to > acquire mm->mmap_sem to avoid a circular locking dependency with > fb_mmap() (that acquires mm->mmap_sem -> fb_info-lock). > > Reported-by: Andrey Borzenkov > Signed-off-by: Andrea Righi > --- > drivers/video/console/fbcon.c | 6 ++++++ > 1 files changed, 6 insertions(+), 0 deletions(-) > > diff --git a/drivers/video/console/fbcon.c > b/drivers/video/console/fbcon.c index 1657b96..81fc2f7 100644 > --- a/drivers/video/console/fbcon.c > +++ b/drivers/video/console/fbcon.c > @@ -3169,9 +3169,15 @@ static int fbcon_event_notify(struct > notifier_block *self, ret =3D fbcon_fb_unregistered(info); > break; > case FB_EVENT_SET_CONSOLE_MAP: > + /* > + * XXX: info is not used here, we can unlock it to avoid > + * circular locking with ¤t->mm->mmap_sem in fb_mmap(). > + */ > + unlock_fb_info(info); > con2fb =3D event->data; > ret =3D set_con2fb_map(con2fb->console - 1, > con2fb->framebuffer, 1); > + lock_fb_info(info); > break; > case FB_EVENT_GET_CONSOLE_MAP: > con2fb =3D event->data; --nextPart1966327.fp6U0JGfBU Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEABECAAYFAkmFTBsACgkQR6LMutpd94z+FgCgv6si1jQKCzxv7rBA29XAWaRu 4G4An1lhDDR7NQFiaGyJrG5qn4WJdEcy =gVyB -----END PGP SIGNATURE----- --nextPart1966327.fp6U0JGfBU--