* Query - state of memory hotplug with RT kernel [not found] <1997236940.56779393.1464861408686.JavaMail.zimbra@redhat.com> @ 2016-06-02 10:10 ` Pankaj Gupta 2016-06-03 16:04 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 5+ messages in thread From: Pankaj Gupta @ 2016-06-02 10:10 UTC (permalink / raw) To: linux-rt-users Hello, Recently, I have been debugging some of the issues with memory hotplug with RT kernel. I want to know state of memory hotplug with RT kernel and if there are any known issues or work going on upstream? I came to know that there is rework of 'cpu hotplug' going on. Not sure about 'memory hotplug'. Any inputs or pointers on this? Best regards, Pankaj ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Query - state of memory hotplug with RT kernel 2016-06-02 10:10 ` Query - state of memory hotplug with RT kernel Pankaj Gupta @ 2016-06-03 16:04 ` Sebastian Andrzej Siewior 2016-06-08 7:05 ` Pankaj Gupta 0 siblings, 1 reply; 5+ messages in thread From: Sebastian Andrzej Siewior @ 2016-06-03 16:04 UTC (permalink / raw) To: Pankaj Gupta; +Cc: linux-rt-users * Pankaj Gupta | 2016-06-02 06:10:41 [-0400]: >Hello, Hi, >Recently, I have been debugging some of the issues with memory >hotplug with RT kernel. I want to know state of memory hotplug >with RT kernel and if there are any known issues or work going >on upstream? I never tried memory hotplug. You are the first one that complains or mentions it. >I came to know that there is rework of 'cpu hotplug' going on. >Not sure about 'memory hotplug'. > >Any inputs or pointers on this? I have been looking into CPU hotplug but got interrupted by v4.6 and a few other things. Regarding memory hotplug, as I said I don't know what is broken. Is this something that can be tested in kvm? >Best regards, >Pankaj Sebastian ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Query - state of memory hotplug with RT kernel 2016-06-03 16:04 ` Sebastian Andrzej Siewior @ 2016-06-08 7:05 ` Pankaj Gupta 2016-06-09 13:05 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 5+ messages in thread From: Pankaj Gupta @ 2016-06-08 7:05 UTC (permalink / raw) To: Sebastian Andrzej Siewior; +Cc: linux-rt-users Hello Sebastian, Sorry! for replying late as I am on vacation this week. > >Hello, > Hi, > > >Recently, I have been debugging some of the issues with memory > >hotplug with RT kernel. I want to know state of memory hotplug > >with RT kernel and if there are any known issues or work going > >on upstream? > > I never tried memory hotplug. You are the first one that complains or > mentions it. > > >I came to know that there is rework of 'cpu hotplug' going on. > >Not sure about 'memory hotplug'. > > > >Any inputs or pointers on this? > > I have been looking into CPU hotplug but got interrupted by v4.6 and a > few other things. Regarding memory hotplug, as I said I don't know what > is broken. Is this something that can be tested in kvm? Yes, KVM guest hangs when I tried to hotplug 4 GB of memory. When I tried to find root cause of it, I got below calltrace: //call trace of running task #0 [ffff8801638ab7e0] get_page_from_freelist at ffffffff81165607 #1 [ffff8801638ab8d8] cpuacct_charge at ffffffff810b9a61 #2 [ffff8801638ab908] __switch_to at ffffffff810018b2 #3 [ffff8801638ab968] __schedule at ffffffff81625984 #4 [ffff8801638ab9c8] preempt_schedule_irq at ffffffff816264a1 #5 [ffff8801638ab9f0] retint_kernel at ffffffff81628277 #6 [ffff8801638aba38] migrate_enable at ffffffff810aa0eb #7 [ffff8801638abaa8] __switch_to at ffffffff810018b2 #8 [ffff8801638abb08] __schedule at ffffffff81625984 #9 [ffff8801638abb68] preempt_schedule_irq at ffffffff816264a1 #10 [ffff8801638abb90] retint_kernel at ffffffff81628277 #11 [ffff8801638abbe8] isolate_pcp_pages at ffffffff81160489 #12 [ffff8801638abc50] free_hot_cold_page at ffffffff8116463c--------\ //pa_lock is used with local_lock_irqsave #13 [ffff8801638abcb8] __free_pages at ffffffff811647df #14 [ffff8801638abcd8] __online_page_free at ffffffff811b6dcc #15 [ffff8801638abce8] generic_online_page at ffffffff811b6e0b #16 [ffff8801638abcf8] online_pages_range at ffffffff811b6ce5 #17 [ffff8801638abd38] walk_system_ram_range at ffffffff8107914c #18 [ffff8801638abda8] online_pages at ffffffff81614a84 #19 [ffff8801638abe20] memory_subsys_online at ffffffff813f1268 #20 [ffff8801638abe50] device_online at ffffffff813d9745 #21 [ffff8801638abe78] store_mem_state at ffffffff813f0ef4 #22 [ffff8801638abea0] dev_attr_store at ffffffff813d6698 #23 [ffff8801638abeb0] sysfs_write_file at ffffffff81247499 #24 [ffff8801638abef8] vfs_write at ffffffff811ca7ad #25 [ffff8801638abf38] sys_write at ffffffff811cb24f #26 [ffff8801638abf80] system_call_fastpath at ffffffff8162fe89 It looks to me "local_lock_irqsave(pa_lock, flags)" is taken in function 'free_hot_cold_page' which is again attempted by function 'get_page_from_freelist'=>'buffered_rmqueue' during interrupt. Looks like same lock(pa_lock) is allowed to work on same critical section multiple times in the calltrace which can result in undefined behaviour. => p (( struct local_irq_lock *) 0xffff88023fd11580)->nestcnt $5 = 1 ----------> nested local_irq_lock is set I will also try to reproduce this issue with latest upstream next week after I come back from vacation. But code looks same. If I am thinking in right direction, Could you please share your thoughts. Thanks, Pankaj > > >Best regards, > >Pankaj > > Sebastian > -- > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Query - state of memory hotplug with RT kernel 2016-06-08 7:05 ` Pankaj Gupta @ 2016-06-09 13:05 ` Sebastian Andrzej Siewior 2016-06-17 13:08 ` Pankaj Gupta 0 siblings, 1 reply; 5+ messages in thread From: Sebastian Andrzej Siewior @ 2016-06-09 13:05 UTC (permalink / raw) To: Pankaj Gupta; +Cc: linux-rt-users * Pankaj Gupta | 2016-06-08 03:05:14 [-0400]: >Hello Sebastian, Hi Pankaj, >Sorry! for replying late as I am on vacation this week. No, you aren't. If you were on vacation you would neither be sorry nor would you reply at all :) >> I have been looking into CPU hotplug but got interrupted by v4.6 and a >> few other things. Regarding memory hotplug, as I said I don't know what >> is broken. Is this something that can be tested in kvm? > >Yes, KVM guest hangs when I tried to hotplug 4 GB of memory. >When I tried to find root cause of it, I got below calltrace: > >//call trace of running task > > #0 [ffff8801638ab7e0] get_page_from_freelist at ffffffff81165607 > #1 [ffff8801638ab8d8] cpuacct_charge at ffffffff810b9a61 > #2 [ffff8801638ab908] __switch_to at ffffffff810018b2 > #3 [ffff8801638ab968] __schedule at ffffffff81625984 > #4 [ffff8801638ab9c8] preempt_schedule_irq at ffffffff816264a1 > #5 [ffff8801638ab9f0] retint_kernel at ffffffff81628277 > #6 [ffff8801638aba38] migrate_enable at ffffffff810aa0eb > #7 [ffff8801638abaa8] __switch_to at ffffffff810018b2 > #8 [ffff8801638abb08] __schedule at ffffffff81625984 > #9 [ffff8801638abb68] preempt_schedule_irq at ffffffff816264a1 >#10 [ffff8801638abb90] retint_kernel at ffffffff81628277 >#11 [ffff8801638abbe8] isolate_pcp_pages at ffffffff81160489 >#12 [ffff8801638abc50] free_hot_cold_page at ffffffff8116463c--------\ > //pa_lock is used with local_lock_irqsave >#13 [ffff8801638abcb8] __free_pages at ffffffff811647df >#14 [ffff8801638abcd8] __online_page_free at ffffffff811b6dcc >#15 [ffff8801638abce8] generic_online_page at ffffffff811b6e0b >#16 [ffff8801638abcf8] online_pages_range at ffffffff811b6ce5 >#17 [ffff8801638abd38] walk_system_ram_range at ffffffff8107914c >#18 [ffff8801638abda8] online_pages at ffffffff81614a84 >#19 [ffff8801638abe20] memory_subsys_online at ffffffff813f1268 >#20 [ffff8801638abe50] device_online at ffffffff813d9745 >#21 [ffff8801638abe78] store_mem_state at ffffffff813f0ef4 >#22 [ffff8801638abea0] dev_attr_store at ffffffff813d6698 >#23 [ffff8801638abeb0] sysfs_write_file at ffffffff81247499 >#24 [ffff8801638abef8] vfs_write at ffffffff811ca7ad >#25 [ffff8801638abf38] sys_write at ffffffff811cb24f >#26 [ffff8801638abf80] system_call_fastpath at ffffffff8162fe89 >It looks to me "local_lock_irqsave(pa_lock, flags)" is taken in >function 'free_hot_cold_page' which is again attempted by function >'get_page_from_freelist'=>'buffered_rmqueue' during interrupt. >Looks like same lock(pa_lock) is allowed to work on same critical >section multiple times in the calltrace which can result in >undefined behaviour. The pa_lock is not (or should not be) take with interrupts disabled on -RT. >=> p (( struct local_irq_lock *) 0xffff88023fd11580)->nestcnt >$5 = 1 ----------> nested local_irq_lock is set > >I will also try to reproduce this issue with latest upstream next week >after I come back from vacation. But code looks same. > >If I am thinking in right direction, Could you please share your thoughts. So I tried memory hotplug on v4.6.1-rt3: |# free -m | total used free shared buff/cache available |Mem: 827 67 678 0 80 741 | |# echo 0 > /sys/bus/memory/devices/memory4/online |[ 587.028476] Offlined Pages 32768 |[ 587.029210] remove from free list 20000 1024 28000 |[ 587.030235] remove from free list 20400 1024 28000 |[ 587.031174] remove from free list 20800 1024 28000 |[ 587.032137] remove from free list 20c00 1024 28000 |[ 587.033219] remove from free list 21000 1024 28000 |[ 587.034174] remove from free list 21400 1024 28000 |[ 587.035109] remove from free list 21800 1024 28000 |[ 587.036080] remove from free list 21c00 1024 28000 |[ 587.037233] remove from free list 22000 1024 28000 |[ 587.038235] remove from free list 22400 1024 28000 |[ 587.039217] remove from free list 22800 1024 28000 |[ 587.040183] remove from free list 22c00 1024 28000 |[ 587.041149] remove from free list 23000 1024 28000 |[ 587.042068] remove from free list 23400 1024 28000 |[ 587.042957] remove from free list 23800 1024 28000 |[ 587.043864] remove from free list 23c00 1024 28000 |[ 587.044962] remove from free list 24000 1024 28000 |[ 587.046327] remove from free list 24400 1024 28000 |[ 587.047705] remove from free list 24800 1024 28000 |[ 587.049110] remove from free list 24c00 1024 28000 |[ 587.050398] remove from free list 25000 1024 28000 |[ 587.051769] remove from free list 25400 1024 28000 |[ 587.053206] remove from free list 25800 1024 28000 |[ 587.054570] remove from free list 25c00 1024 28000 |[ 587.055950] remove from free list 26000 1024 28000 |[ 587.057326] remove from free list 26400 1024 28000 |[ 587.058630] remove from free list 26800 1024 28000 |[ 587.059947] remove from free list 26c00 1024 28000 |[ 587.061337] remove from free list 27000 1024 28000 |[ 587.062636] remove from free list 27400 1024 28000 |[ 587.063970] remove from free list 27800 1024 28000 |[ 587.065287] remove from free list 27c00 1024 28000 |# free -m | total used free shared buff/cache available |Mem: 699 67 550 0 80 614 |Swap: 0 0 0 | |# echo 1 > /sys/bus/memory/devices/memory4/online |# free -m | total used free shared buff/cache available |Mem: 827 68 677 0 80 740 |Swap: 0 0 0 seems to work. I also enabled CONFIG_CGROUP_CPUACCT and tried again and it still removed & added memory. >Thanks, >Pankaj Sebastian ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Query - state of memory hotplug with RT kernel 2016-06-09 13:05 ` Sebastian Andrzej Siewior @ 2016-06-17 13:08 ` Pankaj Gupta 0 siblings, 0 replies; 5+ messages in thread From: Pankaj Gupta @ 2016-06-17 13:08 UTC (permalink / raw) To: Sebastian Andrzej Siewior; +Cc: linux-rt-users Hello Sabastian, I also tested memory hotplug with upstream '4.6.2-rt5' and it looks memory hotplug is working fine. I will try to look for and isolate the cause in downstream kernel. Thanks for the confirmation. Best regards, Pankaj > > >Hello Sebastian, > Hi Pankaj, > > >Sorry! for replying late as I am on vacation this week. > > No, you aren't. If you were on vacation you would neither be sorry nor > would you reply at all :) > > >> I have been looking into CPU hotplug but got interrupted by v4.6 and a > >> few other things. Regarding memory hotplug, as I said I don't know what > >> is broken. Is this something that can be tested in kvm? > > > >Yes, KVM guest hangs when I tried to hotplug 4 GB of memory. > >When I tried to find root cause of it, I got below calltrace: > > > >//call trace of running task > > > > #0 [ffff8801638ab7e0] get_page_from_freelist at ffffffff81165607 > > #1 [ffff8801638ab8d8] cpuacct_charge at ffffffff810b9a61 > > #2 [ffff8801638ab908] __switch_to at ffffffff810018b2 > > #3 [ffff8801638ab968] __schedule at ffffffff81625984 > > #4 [ffff8801638ab9c8] preempt_schedule_irq at ffffffff816264a1 > > #5 [ffff8801638ab9f0] retint_kernel at ffffffff81628277 > > #6 [ffff8801638aba38] migrate_enable at ffffffff810aa0eb > > #7 [ffff8801638abaa8] __switch_to at ffffffff810018b2 > > #8 [ffff8801638abb08] __schedule at ffffffff81625984 > > #9 [ffff8801638abb68] preempt_schedule_irq at ffffffff816264a1 > >#10 [ffff8801638abb90] retint_kernel at ffffffff81628277 > >#11 [ffff8801638abbe8] isolate_pcp_pages at ffffffff81160489 > >#12 [ffff8801638abc50] free_hot_cold_page at ffffffff8116463c--------\ > > //pa_lock is used with > > local_lock_irqsave > >#13 [ffff8801638abcb8] __free_pages at ffffffff811647df > >#14 [ffff8801638abcd8] __online_page_free at ffffffff811b6dcc > >#15 [ffff8801638abce8] generic_online_page at ffffffff811b6e0b > >#16 [ffff8801638abcf8] online_pages_range at ffffffff811b6ce5 > >#17 [ffff8801638abd38] walk_system_ram_range at ffffffff8107914c > >#18 [ffff8801638abda8] online_pages at ffffffff81614a84 > >#19 [ffff8801638abe20] memory_subsys_online at ffffffff813f1268 > >#20 [ffff8801638abe50] device_online at ffffffff813d9745 > >#21 [ffff8801638abe78] store_mem_state at ffffffff813f0ef4 > >#22 [ffff8801638abea0] dev_attr_store at ffffffff813d6698 > >#23 [ffff8801638abeb0] sysfs_write_file at ffffffff81247499 > >#24 [ffff8801638abef8] vfs_write at ffffffff811ca7ad > >#25 [ffff8801638abf38] sys_write at ffffffff811cb24f > >#26 [ffff8801638abf80] system_call_fastpath at ffffffff8162fe89 > > >It looks to me "local_lock_irqsave(pa_lock, flags)" is taken in > >function 'free_hot_cold_page' which is again attempted by function > >'get_page_from_freelist'=>'buffered_rmqueue' during interrupt. > >Looks like same lock(pa_lock) is allowed to work on same critical > >section multiple times in the calltrace which can result in > >undefined behaviour. > > The pa_lock is not (or should not be) take with interrupts disabled on -RT. > > >=> p (( struct local_irq_lock *) 0xffff88023fd11580)->nestcnt > >$5 = 1 ----------> nested local_irq_lock is set > > > >I will also try to reproduce this issue with latest upstream next week > >after I come back from vacation. But code looks same. > > > >If I am thinking in right direction, Could you please share your thoughts. > > So I tried memory hotplug on v4.6.1-rt3: > |# free -m > | total used free shared buff/cache > | available > |Mem: 827 67 678 0 80 > |741 > | > |# echo 0 > /sys/bus/memory/devices/memory4/online > |[ 587.028476] Offlined Pages 32768 > |[ 587.029210] remove from free list 20000 1024 28000 > |[ 587.030235] remove from free list 20400 1024 28000 > |[ 587.031174] remove from free list 20800 1024 28000 > |[ 587.032137] remove from free list 20c00 1024 28000 > |[ 587.033219] remove from free list 21000 1024 28000 > |[ 587.034174] remove from free list 21400 1024 28000 > |[ 587.035109] remove from free list 21800 1024 28000 > |[ 587.036080] remove from free list 21c00 1024 28000 > |[ 587.037233] remove from free list 22000 1024 28000 > |[ 587.038235] remove from free list 22400 1024 28000 > |[ 587.039217] remove from free list 22800 1024 28000 > |[ 587.040183] remove from free list 22c00 1024 28000 > |[ 587.041149] remove from free list 23000 1024 28000 > |[ 587.042068] remove from free list 23400 1024 28000 > |[ 587.042957] remove from free list 23800 1024 28000 > |[ 587.043864] remove from free list 23c00 1024 28000 > |[ 587.044962] remove from free list 24000 1024 28000 > |[ 587.046327] remove from free list 24400 1024 28000 > |[ 587.047705] remove from free list 24800 1024 28000 > |[ 587.049110] remove from free list 24c00 1024 28000 > |[ 587.050398] remove from free list 25000 1024 28000 > |[ 587.051769] remove from free list 25400 1024 28000 > |[ 587.053206] remove from free list 25800 1024 28000 > |[ 587.054570] remove from free list 25c00 1024 28000 > |[ 587.055950] remove from free list 26000 1024 28000 > |[ 587.057326] remove from free list 26400 1024 28000 > |[ 587.058630] remove from free list 26800 1024 28000 > |[ 587.059947] remove from free list 26c00 1024 28000 > |[ 587.061337] remove from free list 27000 1024 28000 > |[ 587.062636] remove from free list 27400 1024 28000 > |[ 587.063970] remove from free list 27800 1024 28000 > |[ 587.065287] remove from free list 27c00 1024 28000 > |# free -m > | total used free shared buff/cache > | available > |Mem: 699 67 550 0 80 > |614 > |Swap: 0 0 0 > | > |# echo 1 > /sys/bus/memory/devices/memory4/online > |# free -m > | total used free shared buff/cache > | available > |Mem: 827 68 677 0 80 > |740 > |Swap: 0 0 0 > > seems to work. I also enabled CONFIG_CGROUP_CPUACCT and tried again and > it still removed & added memory. > > >Thanks, > >Pankaj > > Sebastian > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-06-17 13:08 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <1997236940.56779393.1464861408686.JavaMail.zimbra@redhat.com>
2016-06-02 10:10 ` Query - state of memory hotplug with RT kernel Pankaj Gupta
2016-06-03 16:04 ` Sebastian Andrzej Siewior
2016-06-08 7:05 ` Pankaj Gupta
2016-06-09 13:05 ` Sebastian Andrzej Siewior
2016-06-17 13:08 ` Pankaj Gupta
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).