[Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Jan Kiszka]

Hi,

any thoughts on this BUG? Happens with ipipe-2.0-07 on 2.6.24.7,
obviously during module loading.

 kernel BUG at arch/x86/mm/fault_64.c:258!
 invalid opcode: 0000 [1] SMP
 CPU 3
 Modules linked in: ide_core ide_disk scsi_mod sd_mod serverworks libata
 sata_svw scsi_transport_sas mptbase mptscsih mptsas sg fan edd
 pata_serverworks jbd mbcache ext3 usbcore hwmon i2c_core k8temp
 pci_hotplug i2c_piix4 shpchp ehci_hcd ohci_hcd rtc_lib rtc_core rtc_cmos
 tg3
 Pid: 1683, comm: modprobe Not tainted 2.6.24.7-xeno #1
 RIP: 0010:[<ffffffff80224e8c>]  [<ffffffff80224e8c>]
 vmalloc_sync_one+0x6f/0x197
 RSP: 0018:ffff81023b0c1c98  EFLAGS: 00010287
 RAX: 00003ffffffff000 RBX: ffff81023feeea88 RCX: ffff810000000000
 RDX: ffff81023c423000 RSI: 000000023c423000 RDI: ffff81023b1e7c20
 RBP: ffff81023b0c1cc8 R08: ffffffff80201c20 R09: 0000000000000800
 R10: ffffffff8099a380 R11: 0000000000000002 R12: 0000000000000c20
 R13: ffffc20001888000 R14: ffffc20001888000 R15: 0000000000000000
 FS:  00002ac2367716d0(0000) GS:ffff81023c31d5c0(0000)
 knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
 CR2: 00002ac236442000 CR3: 000000023b139000 CR4: 00000000000006e0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 Process modprobe (pid: 1683, threadinfo ffff81023b0c0000, task
 ffff81023c3ba7f0)
 Stack:  ffffc2000188bfff ffff81023feeea88 0000000000000c20
 ffffc20001888000
  ffffc2000188c000 0000000000000000 ffff81023b0c1d08 ffffffff802252ac
  ffffc2000188c000 0000000000000000 ffffc2000188c000 ffff81013a1cf468
 Call Trace:
  [<ffffffff802252ac>] __ipipe_pin_range_globally+0x9a/0xe4
  [<ffffffff802dac08>] map_vm_area+0x29f/0x2b0
  [<ffffffff802db28b>] __vmalloc_area_node+0x173/0x199
  [<ffffffff802db30e>] __vmalloc_node+0x5d/0x6a
  [<ffffffff802db34d>] __vmalloc+0x11/0x13
  [<ffffffff802db40a>] vmalloc+0x1d/0x1f
  [<ffffffff8025c73b>] sys_init_module+0x71/0x18ba
  [<ffffffff8022453c>] mcount+0x4c/0x72
  [<ffffffff8022453c>] mcount+0x4c/0x72
  [<ffffffff80223f54>] __ipipe_syscall_root+0xc/0x197
  [<ffffffff8047fb11>] __ipipe_syscall_root_thunk+0x35/0x6a
  [<ffffffff8020c172>] system_call+0x92/0x97


 Code: 0f 0b eb fe 49 8b 00 4c 89 f2 49 bf 00 f0 ff ff ff 3f 00 00
 RIP  [<ffffffff80224e8c>] vmalloc_sync_one+0x6f/0x197
  RSP <ffff81023b0c1c98>


The relevant code in fault_64.c:

static int vmalloc_sync_one(pgd_t *pgd, unsigned long address)
{
        pgd_t *pgd_ref;
        pud_t *pud, *pud_ref;
        pmd_t *pmd, *pmd_ref;
        pte_t *pte, *pte_ref;

        /* Copy kernel mappings over when needed. This can also
           happen within a race in page table update. In the later
           case just flush. */

        pgd_ref = pgd_offset_k(address);
        if (pgd_none(*pgd_ref))
                return -1;
        if (pgd_none(*pgd))
                set_pgd(pgd, *pgd_ref);
        else
                BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));

This one triggers.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Hi,
> 
> any thoughts on this BUG? Happens with ipipe-2.0-07 on 2.6.24.7,
> obviously during module loading.
> 
>  kernel BUG at arch/x86/mm/fault_64.c:258!
>  invalid opcode: 0000 [1] SMP
>  CPU 3
>  Modules linked in: ide_core ide_disk scsi_mod sd_mod serverworks libata
>  sata_svw scsi_transport_sas mptbase mptscsih mptsas sg fan edd
>  pata_serverworks jbd mbcache ext3 usbcore hwmon i2c_core k8temp
>  pci_hotplug i2c_piix4 shpchp ehci_hcd ohci_hcd rtc_lib rtc_core rtc_cmos
>  tg3
>  Pid: 1683, comm: modprobe Not tainted 2.6.24.7-xeno #1
>  RIP: 0010:[<ffffffff80224e8c>]  [<ffffffff80224e8c>]
>  vmalloc_sync_one+0x6f/0x197
>  RSP: 0018:ffff81023b0c1c98  EFLAGS: 00010287
>  RAX: 00003ffffffff000 RBX: ffff81023feeea88 RCX: ffff810000000000
>  RDX: ffff81023c423000 RSI: 000000023c423000 RDI: ffff81023b1e7c20
>  RBP: ffff81023b0c1cc8 R08: ffffffff80201c20 R09: 0000000000000800
>  R10: ffffffff8099a380 R11: 0000000000000002 R12: 0000000000000c20
>  R13: ffffc20001888000 R14: ffffc20001888000 R15: 0000000000000000
>  FS:  00002ac2367716d0(0000) GS:ffff81023c31d5c0(0000)
>  knlGS:0000000000000000
>  CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>  CR2: 00002ac236442000 CR3: 000000023b139000 CR4: 00000000000006e0
>  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>  DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>  Process modprobe (pid: 1683, threadinfo ffff81023b0c0000, task
>  ffff81023c3ba7f0)
>  Stack:  ffffc2000188bfff ffff81023feeea88 0000000000000c20
>  ffffc20001888000
>   ffffc2000188c000 0000000000000000 ffff81023b0c1d08 ffffffff802252ac
>   ffffc2000188c000 0000000000000000 ffffc2000188c000 ffff81013a1cf468
>  Call Trace:
>   [<ffffffff802252ac>] __ipipe_pin_range_globally+0x9a/0xe4
>   [<ffffffff802dac08>] map_vm_area+0x29f/0x2b0
>   [<ffffffff802db28b>] __vmalloc_area_node+0x173/0x199
>   [<ffffffff802db30e>] __vmalloc_node+0x5d/0x6a
>   [<ffffffff802db34d>] __vmalloc+0x11/0x13
>   [<ffffffff802db40a>] vmalloc+0x1d/0x1f
>   [<ffffffff8025c73b>] sys_init_module+0x71/0x18ba
>   [<ffffffff8022453c>] mcount+0x4c/0x72
>   [<ffffffff8022453c>] mcount+0x4c/0x72
>   [<ffffffff80223f54>] __ipipe_syscall_root+0xc/0x197
>   [<ffffffff8047fb11>] __ipipe_syscall_root_thunk+0x35/0x6a
>   [<ffffffff8020c172>] system_call+0x92/0x97
> 
> 
>  Code: 0f 0b eb fe 49 8b 00 4c 89 f2 49 bf 00 f0 ff ff ff 3f 00 00
>  RIP  [<ffffffff80224e8c>] vmalloc_sync_one+0x6f/0x197
>   RSP <ffff81023b0c1c98>
> 
> 
> The relevant code in fault_64.c:
> 
> static int vmalloc_sync_one(pgd_t *pgd, unsigned long address)
> {
>         pgd_t *pgd_ref;
>         pud_t *pud, *pud_ref;
>         pmd_t *pmd, *pmd_ref;
>         pte_t *pte, *pte_ref;
> 
>         /* Copy kernel mappings over when needed. This can also
>            happen within a race in page table update. In the later
>            case just flush. */
> 
>         pgd_ref = pgd_offset_k(address);
>         if (pgd_none(*pgd_ref))
>                 return -1;
>         if (pgd_none(*pgd))
>                 set_pgd(pgd, *pgd_ref);
>         else
>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
> 
> This one triggers.

I think there is something missing in the I-pipe patch: when a vmalloc
occurs we update all page directories, but when a vfree occurs, we do
nothing. Is there any chance that the bug you observed is in fact a
vmalloc which reuses an address which has been vfreed recently ?

-- 
                                                 Gilles.

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Hi,
>>
>> any thoughts on this BUG? Happens with ipipe-2.0-07 on 2.6.24.7,
>> obviously during module loading.
>>
>>  kernel BUG at arch/x86/mm/fault_64.c:258!
>>  invalid opcode: 0000 [1] SMP
>>  CPU 3
>>  Modules linked in: ide_core ide_disk scsi_mod sd_mod serverworks libata
>>  sata_svw scsi_transport_sas mptbase mptscsih mptsas sg fan edd
>>  pata_serverworks jbd mbcache ext3 usbcore hwmon i2c_core k8temp
>>  pci_hotplug i2c_piix4 shpchp ehci_hcd ohci_hcd rtc_lib rtc_core rtc_cmos
>>  tg3
>>  Pid: 1683, comm: modprobe Not tainted 2.6.24.7-xeno #1
>>  RIP: 0010:[<ffffffff80224e8c>]  [<ffffffff80224e8c>]
>>  vmalloc_sync_one+0x6f/0x197
>>  RSP: 0018:ffff81023b0c1c98  EFLAGS: 00010287
>>  RAX: 00003ffffffff000 RBX: ffff81023feeea88 RCX: ffff810000000000
>>  RDX: ffff81023c423000 RSI: 000000023c423000 RDI: ffff81023b1e7c20
>>  RBP: ffff81023b0c1cc8 R08: ffffffff80201c20 R09: 0000000000000800
>>  R10: ffffffff8099a380 R11: 0000000000000002 R12: 0000000000000c20
>>  R13: ffffc20001888000 R14: ffffc20001888000 R15: 0000000000000000
>>  FS:  00002ac2367716d0(0000) GS:ffff81023c31d5c0(0000)
>>  knlGS:0000000000000000
>>  CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>  CR2: 00002ac236442000 CR3: 000000023b139000 CR4: 00000000000006e0
>>  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>  DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>  Process modprobe (pid: 1683, threadinfo ffff81023b0c0000, task
>>  ffff81023c3ba7f0)
>>  Stack:  ffffc2000188bfff ffff81023feeea88 0000000000000c20
>>  ffffc20001888000
>>   ffffc2000188c000 0000000000000000 ffff81023b0c1d08 ffffffff802252ac
>>   ffffc2000188c000 0000000000000000 ffffc2000188c000 ffff81013a1cf468
>>  Call Trace:
>>   [<ffffffff802252ac>] __ipipe_pin_range_globally+0x9a/0xe4
>>   [<ffffffff802dac08>] map_vm_area+0x29f/0x2b0
>>   [<ffffffff802db28b>] __vmalloc_area_node+0x173/0x199
>>   [<ffffffff802db30e>] __vmalloc_node+0x5d/0x6a
>>   [<ffffffff802db34d>] __vmalloc+0x11/0x13
>>   [<ffffffff802db40a>] vmalloc+0x1d/0x1f
>>   [<ffffffff8025c73b>] sys_init_module+0x71/0x18ba
>>   [<ffffffff8022453c>] mcount+0x4c/0x72
>>   [<ffffffff8022453c>] mcount+0x4c/0x72
>>   [<ffffffff80223f54>] __ipipe_syscall_root+0xc/0x197
>>   [<ffffffff8047fb11>] __ipipe_syscall_root_thunk+0x35/0x6a
>>   [<ffffffff8020c172>] system_call+0x92/0x97
>>
>>
>>  Code: 0f 0b eb fe 49 8b 00 4c 89 f2 49 bf 00 f0 ff ff ff 3f 00 00
>>  RIP  [<ffffffff80224e8c>] vmalloc_sync_one+0x6f/0x197
>>   RSP <ffff81023b0c1c98>
>>
>>
>> The relevant code in fault_64.c:
>>
>> static int vmalloc_sync_one(pgd_t *pgd, unsigned long address)
>> {
>>         pgd_t *pgd_ref;
>>         pud_t *pud, *pud_ref;
>>         pmd_t *pmd, *pmd_ref;
>>         pte_t *pte, *pte_ref;
>>
>>         /* Copy kernel mappings over when needed. This can also
>>            happen within a race in page table update. In the later
>>            case just flush. */
>>
>>         pgd_ref = pgd_offset_k(address);
>>         if (pgd_none(*pgd_ref))
>>                 return -1;
>>         if (pgd_none(*pgd))
>>                 set_pgd(pgd, *pgd_ref);
>>         else
>>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
>>
>> This one triggers.
> 
> I think there is something missing in the I-pipe patch: when a vmalloc
> occurs we update all page directories, but when a vfree occurs, we do
> nothing. Is there any chance that the bug you observed is in fact a
> vmalloc which reuses an address which has been vfreed recently ?

Maybe. This happens during boot-up, probably while issuing modprobes in
a row where you also tend to release some temporary memory again. That
said, I cannot provide a precise test case. And according to the
reporter, this only happens fairly sporadically.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> Hi,
>>>
>>> any thoughts on this BUG? Happens with ipipe-2.0-07 on 2.6.24.7,
>>> obviously during module loading.
>>>
>>>  kernel BUG at arch/x86/mm/fault_64.c:258!
>>>  invalid opcode: 0000 [1] SMP
>>>  CPU 3
>>>  Modules linked in: ide_core ide_disk scsi_mod sd_mod serverworks libata
>>>  sata_svw scsi_transport_sas mptbase mptscsih mptsas sg fan edd
>>>  pata_serverworks jbd mbcache ext3 usbcore hwmon i2c_core k8temp
>>>  pci_hotplug i2c_piix4 shpchp ehci_hcd ohci_hcd rtc_lib rtc_core rtc_cmos
>>>  tg3
>>>  Pid: 1683, comm: modprobe Not tainted 2.6.24.7-xeno #1
>>>  RIP: 0010:[<ffffffff80224e8c>]  [<ffffffff80224e8c>]
>>>  vmalloc_sync_one+0x6f/0x197
>>>  RSP: 0018:ffff81023b0c1c98  EFLAGS: 00010287
>>>  RAX: 00003ffffffff000 RBX: ffff81023feeea88 RCX: ffff810000000000
>>>  RDX: ffff81023c423000 RSI: 000000023c423000 RDI: ffff81023b1e7c20
>>>  RBP: ffff81023b0c1cc8 R08: ffffffff80201c20 R09: 0000000000000800
>>>  R10: ffffffff8099a380 R11: 0000000000000002 R12: 0000000000000c20
>>>  R13: ffffc20001888000 R14: ffffc20001888000 R15: 0000000000000000
>>>  FS:  00002ac2367716d0(0000) GS:ffff81023c31d5c0(0000)
>>>  knlGS:0000000000000000
>>>  CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>  CR2: 00002ac236442000 CR3: 000000023b139000 CR4: 00000000000006e0
>>>  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>  DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>  Process modprobe (pid: 1683, threadinfo ffff81023b0c0000, task
>>>  ffff81023c3ba7f0)
>>>  Stack:  ffffc2000188bfff ffff81023feeea88 0000000000000c20
>>>  ffffc20001888000
>>>   ffffc2000188c000 0000000000000000 ffff81023b0c1d08 ffffffff802252ac
>>>   ffffc2000188c000 0000000000000000 ffffc2000188c000 ffff81013a1cf468
>>>  Call Trace:
>>>   [<ffffffff802252ac>] __ipipe_pin_range_globally+0x9a/0xe4
>>>   [<ffffffff802dac08>] map_vm_area+0x29f/0x2b0
>>>   [<ffffffff802db28b>] __vmalloc_area_node+0x173/0x199
>>>   [<ffffffff802db30e>] __vmalloc_node+0x5d/0x6a
>>>   [<ffffffff802db34d>] __vmalloc+0x11/0x13
>>>   [<ffffffff802db40a>] vmalloc+0x1d/0x1f
>>>   [<ffffffff8025c73b>] sys_init_module+0x71/0x18ba
>>>   [<ffffffff8022453c>] mcount+0x4c/0x72
>>>   [<ffffffff8022453c>] mcount+0x4c/0x72
>>>   [<ffffffff80223f54>] __ipipe_syscall_root+0xc/0x197
>>>   [<ffffffff8047fb11>] __ipipe_syscall_root_thunk+0x35/0x6a
>>>   [<ffffffff8020c172>] system_call+0x92/0x97
>>>
>>>
>>>  Code: 0f 0b eb fe 49 8b 00 4c 89 f2 49 bf 00 f0 ff ff ff 3f 00 00
>>>  RIP  [<ffffffff80224e8c>] vmalloc_sync_one+0x6f/0x197
>>>   RSP <ffff81023b0c1c98>
>>>
>>>
>>> The relevant code in fault_64.c:
>>>
>>> static int vmalloc_sync_one(pgd_t *pgd, unsigned long address)
>>> {
>>>         pgd_t *pgd_ref;
>>>         pud_t *pud, *pud_ref;
>>>         pmd_t *pmd, *pmd_ref;
>>>         pte_t *pte, *pte_ref;
>>>
>>>         /* Copy kernel mappings over when needed. This can also
>>>            happen within a race in page table update. In the later
>>>            case just flush. */
>>>
>>>         pgd_ref = pgd_offset_k(address);
>>>         if (pgd_none(*pgd_ref))
>>>                 return -1;
>>>         if (pgd_none(*pgd))
>>>                 set_pgd(pgd, *pgd_ref);
>>>         else
>>>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
>>>
>>> This one triggers.
>> I think there is something missing in the I-pipe patch: when a vmalloc
>> occurs we update all page directories, but when a vfree occurs, we do
>> nothing. Is there any chance that the bug you observed is in fact a
>> vmalloc which reuses an address which has been vfreed recently ?
> 
> Maybe. This happens during boot-up, probably while issuing modprobes in
> a row where you also tend to release some temporary memory again. That
> said, I cannot provide a precise test case. And according to the
> reporter, this only happens fairly sporadically.

Ok. Maybe printing pgd_page_vaddr(*pgd) and pgd_page_vaddr(*pgd_ref)
would help ?

-- 
                                                 Gilles.

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Hi,
>>>>
>>>> any thoughts on this BUG? Happens with ipipe-2.0-07 on 2.6.24.7,
>>>> obviously during module loading.
>>>>
>>>>  kernel BUG at arch/x86/mm/fault_64.c:258!
>>>>  invalid opcode: 0000 [1] SMP
>>>>  CPU 3
>>>>  Modules linked in: ide_core ide_disk scsi_mod sd_mod serverworks libata
>>>>  sata_svw scsi_transport_sas mptbase mptscsih mptsas sg fan edd
>>>>  pata_serverworks jbd mbcache ext3 usbcore hwmon i2c_core k8temp
>>>>  pci_hotplug i2c_piix4 shpchp ehci_hcd ohci_hcd rtc_lib rtc_core rtc_cmos
>>>>  tg3
>>>>  Pid: 1683, comm: modprobe Not tainted 2.6.24.7-xeno #1
>>>>  RIP: 0010:[<ffffffff80224e8c>]  [<ffffffff80224e8c>]
>>>>  vmalloc_sync_one+0x6f/0x197
>>>>  RSP: 0018:ffff81023b0c1c98  EFLAGS: 00010287
>>>>  RAX: 00003ffffffff000 RBX: ffff81023feeea88 RCX: ffff810000000000
>>>>  RDX: ffff81023c423000 RSI: 000000023c423000 RDI: ffff81023b1e7c20
>>>>  RBP: ffff81023b0c1cc8 R08: ffffffff80201c20 R09: 0000000000000800
>>>>  R10: ffffffff8099a380 R11: 0000000000000002 R12: 0000000000000c20
>>>>  R13: ffffc20001888000 R14: ffffc20001888000 R15: 0000000000000000
>>>>  FS:  00002ac2367716d0(0000) GS:ffff81023c31d5c0(0000)
>>>>  knlGS:0000000000000000
>>>>  CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>>>  CR2: 00002ac236442000 CR3: 000000023b139000 CR4: 00000000000006e0
>>>>  DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>>  DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
>>>>  Process modprobe (pid: 1683, threadinfo ffff81023b0c0000, task
>>>>  ffff81023c3ba7f0)
>>>>  Stack:  ffffc2000188bfff ffff81023feeea88 0000000000000c20
>>>>  ffffc20001888000
>>>>   ffffc2000188c000 0000000000000000 ffff81023b0c1d08 ffffffff802252ac
>>>>   ffffc2000188c000 0000000000000000 ffffc2000188c000 ffff81013a1cf468
>>>>  Call Trace:
>>>>   [<ffffffff802252ac>] __ipipe_pin_range_globally+0x9a/0xe4
>>>>   [<ffffffff802dac08>] map_vm_area+0x29f/0x2b0
>>>>   [<ffffffff802db28b>] __vmalloc_area_node+0x173/0x199
>>>>   [<ffffffff802db30e>] __vmalloc_node+0x5d/0x6a
>>>>   [<ffffffff802db34d>] __vmalloc+0x11/0x13
>>>>   [<ffffffff802db40a>] vmalloc+0x1d/0x1f
>>>>   [<ffffffff8025c73b>] sys_init_module+0x71/0x18ba
>>>>   [<ffffffff8022453c>] mcount+0x4c/0x72
>>>>   [<ffffffff8022453c>] mcount+0x4c/0x72
>>>>   [<ffffffff80223f54>] __ipipe_syscall_root+0xc/0x197
>>>>   [<ffffffff8047fb11>] __ipipe_syscall_root_thunk+0x35/0x6a
>>>>   [<ffffffff8020c172>] system_call+0x92/0x97
>>>>
>>>>
>>>>  Code: 0f 0b eb fe 49 8b 00 4c 89 f2 49 bf 00 f0 ff ff ff 3f 00 00
>>>>  RIP  [<ffffffff80224e8c>] vmalloc_sync_one+0x6f/0x197
>>>>   RSP <ffff81023b0c1c98>
>>>>
>>>>
>>>> The relevant code in fault_64.c:
>>>>
>>>> static int vmalloc_sync_one(pgd_t *pgd, unsigned long address)
>>>> {
>>>>         pgd_t *pgd_ref;
>>>>         pud_t *pud, *pud_ref;
>>>>         pmd_t *pmd, *pmd_ref;
>>>>         pte_t *pte, *pte_ref;
>>>>
>>>>         /* Copy kernel mappings over when needed. This can also
>>>>            happen within a race in page table update. In the later
>>>>            case just flush. */
>>>>
>>>>         pgd_ref = pgd_offset_k(address);
>>>>         if (pgd_none(*pgd_ref))
>>>>                 return -1;
>>>>         if (pgd_none(*pgd))
>>>>                 set_pgd(pgd, *pgd_ref);
>>>>         else
>>>>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
>>>>
>>>> This one triggers.
>>> I think there is something missing in the I-pipe patch: when a vmalloc
>>> occurs we update all page directories, but when a vfree occurs, we do
>>> nothing. Is there any chance that the bug you observed is in fact a
>>> vmalloc which reuses an address which has been vfreed recently ?
>> Maybe. This happens during boot-up, probably while issuing modprobes in
>> a row where you also tend to release some temporary memory again. That
>> said, I cannot provide a precise test case. And according to the
>> reporter, this only happens fairly sporadically.
> 
> Ok. Maybe printing pgd_page_vaddr(*pgd) and pgd_page_vaddr(*pgd_ref)
> would help ?

I don't see yet where you want to go. As I said, the issue is rare. I
rather think we need to approach it theoretically.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Jan Kiszka wrote:
>>>>>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
>> Ok. Maybe printing pgd_page_vaddr(*pgd) and pgd_page_vaddr(*pgd_ref)
>> would help ?
> 
> I don't see yet where you want to go. As I said, the issue is rare. I
> rather think we need to approach it theoretically.

Well the bug happens when the two virtual addresses are different (so, I
do not think my original remark applies). So, I want to see what the two
addresses look like, for instance if they are not 0 or ~0.

-- 
                                                 Gilles.

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Jan Kiszka wrote:
>>>>>>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
>>> Ok. Maybe printing pgd_page_vaddr(*pgd) and pgd_page_vaddr(*pgd_ref)
>>> would help ?
>> I don't see yet where you want to go. As I said, the issue is rare. I
>> rather think we need to approach it theoretically.
> 
> Well the bug happens when the two virtual addresses are different (so, I
> do not think my original remark applies). So, I want to see what the two
> addresses look like, for instance if they are not 0 or ~0.

OK, will forward the request.

Not sure if this gives more context, but here is also the ipipe trace
that came with the oops:

...
 |   #func                  -14 notify_die+0xd (do_invalid_op+0x7a)
 |   #func                  -15 do_invalid_op+0x10 (__ipipe_handle_exception+0x18e)
 |   +func                  -15 __ipipe_handle_exception+0x16 (error_sti+0x1e)
     +func                  -16 vmalloc_sync_one+0x16 (__ipipe_pin_range_globally+0x9a)
     +func                  -16 page_to_pfn+0x9 (__ipipe_pin_range_globally+0x7d)
     +func                  -17 _spin_lock+0x9 (__ipipe_pin_range_globally+0x68)
     +func                  -17 __ipipe_pin_range_globally+0x16 (map_vm_area+0x29f)
     +func                  -17 page_to_pfn+0x9 (map_vm_area+0x226)
     +func                  -18 page_to_pfn+0x9 (map_vm_area+0x226)
     +func                  -18 page_to_pfn+0x9 (map_vm_area+0x226)
     +func                  -18 page_to_pfn+0x9 (map_vm_area+0x226)
     +func                  -19 map_vm_area+0x16 (__vmalloc_area_node+0x173)
     +func                  -19 page_to_pfn+0x9 (__alloc_pages+0x357)
 |   +end     0x80000000    -19 __ipipe_unstall_root+0x5e (__ipipe_restore_root+0x2c)
 |   #begin   0x80000000    -19 __ipipe_unstall_root+0x1b (__ipipe_restore_root+0x2c)
     #func                  -20 __ipipe_unstall_root+0x9 (__ipipe_restore_root+0x2c)
     #func                  -20 __ipipe_restore_root+0x9 (get_page_from_freelist+0x3c8)
     #func                  -20 __inc_zone_state+0x9 (zone_statistics+0x69)
     #func                  -20 __inc_zone_state+0x9 (zone_statistics+0x4a)
     #func                  -20 zone_statistics+0xc (get_page_from_freelist+0x3b3)
     +func                  -20 ipipe_check_context+0xe (get_page_from_freelist+0x26a)
     +func                  -20 ipipe_check_context+0xe (get_page_from_freelist+0x239)
     +func                  -20 zone_watermark_ok+0xa (get_page_from_freelist+0x169)
     +func                  -20 get_page_from_freelist+0x16 (__alloc_pages+0x6f)
     +func                  -21 cond_resched+0x9 (__alloc_pages+0x47)
     +func                  -21 ipipe_check_context+0xe (__alloc_pages+0x42)
     +func                  -21 __alloc_pages+0x16 (alloc_pages_current+0x7d)
     +func                  -21 zonelist_policy+0x9 (alloc_pages_current+0x70)
     +func                  -21 alloc_pages_current+0xc (__vmalloc_area_node+0xc9)
     +func                  -21 page_to_pfn+0x9 (__alloc_pages+0x357)
 |   +end     0x80000000    -21 __ipipe_unstall_root+0x5e (__ipipe_restore_root+0x2c)
 |   #begin   0x80000000    -21 __ipipe_unstall_root+0x1b (__ipipe_restore_root+0x2c)
     #func                  -21 __ipipe_unstall_root+0x9 (__ipipe_restore_root+0x2c)
     #func                  -21 __ipipe_restore_root+0x9 (get_page_from_freelist+0x3c8)
     #func                  -22 __inc_zone_state+0x9 (zone_statistics+0x69)
     #func                  -22 __inc_zone_state+0x9 (zone_statistics+0x4a)
     #func                  -22 zone_statistics+0xc (get_page_from_freelist+0x3b3)
     +func                  -22 ipipe_check_context+0xe (get_page_from_freelist+0x26a)
     +func                  -22 ipipe_check_context+0xe (get_page_from_freelist+0x239)
     +func                  -22 zone_watermark_ok+0xa (get_page_from_freelist+0x169)
     +func                  -22 get_page_from_freelist+0x16 (__alloc_pages+0x6f)
     +func                  -22 cond_resched+0x9 (__alloc_pages+0x47)
     +func                  -22 ipipe_check_context+0xe (__alloc_pages+0x42)
     +func                  -23 __alloc_pages+0x16 (alloc_pages_current+0x7d)
     +func                  -23 zonelist_policy+0x9 (alloc_pages_current+0x70)
     +func                  -23 alloc_pages_current+0xc (__vmalloc_area_node+0xc9)
     +func                  -23 page_to_pfn+0x9 (__alloc_pages+0x357)
 |   +end     0x80000000    -23 __ipipe_unstall_root+0x5e (__ipipe_restore_root+0x2c)
 |   #begin   0x80000000    -23 __ipipe_unstall_root+0x1b (__ipipe_restore_root+0x2c)
     #func                  -23 __ipipe_unstall_root+0x9 (__ipipe_restore_root+0x2c)
     #func                  -23 __ipipe_restore_root+0x9 (get_page_from_freelist+0x3c8)
     #func                  -23 __inc_zone_state+0x9 (zone_statistics+0x69)
     #func                  -24 __inc_zone_state+0x9 (zone_statistics+0x4a)
     #func                  -24 zone_statistics+0xc (get_page_from_freelist+0x3b3)
     +func                  -24 ipipe_check_context+0xe (get_page_from_freelist+0x26a)
     +func                  -24 ipipe_check_context+0xe (get_page_from_freelist+0x239)
     +func                  -24 zone_watermark_ok+0xa (get_page_from_freelist+0x169)
     +func                  -24 get_page_from_freelist+0x16 (__alloc_pages+0x6f)
     +func                  -24 cond_resched+0x9 (__alloc_pages+0x47)
     +func                  -24 ipipe_check_context+0xe (__alloc_pages+0x42)
     +func                  -25 __alloc_pages+0x16 (alloc_pages_current+0x7d)
     +func                  -25 zonelist_policy+0x9 (alloc_pages_current+0x70)
     +func                  -25 alloc_pages_current+0xc (__vmalloc_area_node+0xc9)
     +func                  -25 page_to_pfn+0x9 (__alloc_pages+0x357)
 |   +end     0x80000000    -25 __ipipe_unstall_root+0x5e (__ipipe_restore_root+0x2c)
 |   #begin   0x80000000    -25 __ipipe_unstall_root+0x1b (__ipipe_restore_root+0x2c)
     #func                  -25 __ipipe_unstall_root+0x9 (__ipipe_restore_root+0x2c)
     #func                  -25 __ipipe_restore_root+0x9 (get_page_from_freelist+0x3c8)
     #func                  -26 __inc_zone_state+0x9 (zone_statistics+0x69)
     #func                  -26 __inc_zone_state+0x9 (zone_statistics+0x4a)
     #func                  -26 zone_statistics+0xc (get_page_from_freelist+0x3b3)
     +func                  -27 ipipe_check_context+0xe (get_page_from_freelist+0x26a)
     +func                  -27 ipipe_check_context+0xe (get_page_from_freelist+0x239)
     +func                  -27 zone_watermark_ok+0xa (get_page_from_freelist+0x169)
     +func                  -27 get_page_from_freelist+0x16 (__alloc_pages+0x6f)
     +func                  -27 cond_resched+0x9 (__alloc_pages+0x47)
     +func                  -27 ipipe_check_context+0xe (__alloc_pages+0x42)
     +func                  -27 __alloc_pages+0x16 (alloc_pages_current+0x7d)


-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Gilles Chanteperdrix]

Jan Kiszka wrote:
> Gilles Chanteperdrix wrote:
>> Jan Kiszka wrote:
>>> Gilles Chanteperdrix wrote:
>>>> Jan Kiszka wrote:
>>>>> Gilles Chanteperdrix wrote:
>>>>>> Jan Kiszka wrote:
>>>>>>>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
>>>> Ok. Maybe printing pgd_page_vaddr(*pgd) and pgd_page_vaddr(*pgd_ref)
>>>> would help ?
>>> I don't see yet where you want to go. As I said, the issue is rare. I
>>> rather think we need to approach it theoretically.
>> Well the bug happens when the two virtual addresses are different (so, I
>> do not think my original remark applies). So, I want to see what the two
>> addresses look like, for instance if they are not 0 or ~0.
> 
> OK, will forward the request.

Ok. Something else: x86_32 and vmalloc_sync_all use spinlock_irqsave,
so, maybe __ipipe_pin_range_globally should use irqsave as well ?

-- 
                                                 Gilles.

Re: [Adeos-main] [BUG] vmalloc_sync_one complains about __ipipe_pin_range_globally

[Jan Kiszka]

Gilles Chanteperdrix wrote:
> Jan Kiszka wrote:
>> Gilles Chanteperdrix wrote:
>>> Jan Kiszka wrote:
>>>> Gilles Chanteperdrix wrote:
>>>>> Jan Kiszka wrote:
>>>>>> Gilles Chanteperdrix wrote:
>>>>>>> Jan Kiszka wrote:
>>>>>>>>                 BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref));
>>>>> Ok. Maybe printing pgd_page_vaddr(*pgd) and pgd_page_vaddr(*pgd_ref)
>>>>> would help ?
>>>> I don't see yet where you want to go. As I said, the issue is rare. I
>>>> rather think we need to approach it theoretically.
>>> Well the bug happens when the two virtual addresses are different (so, I
>>> do not think my original remark applies). So, I want to see what the two
>>> addresses look like, for instance if they are not 0 or ~0.
>> OK, will forward the request.
> 
> Ok. Something else: x86_32 and vmalloc_sync_all use spinlock_irqsave,
> so, maybe __ipipe_pin_range_globally should use irqsave as well ?

Hmm. In 2.6.24 this lock was not used with irqsave, but in 2.6.26 it is.
One would have to check if this is a bug fix or due to other changes.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux