Re: Vanilla Linux and has_foreign_mapping

[Michael Abd-El-Malek <mabdelmalek@cmu.edu>]

Jeremy Fitzhardinge wrote:
> Michael Abd-El-Malek wrote:
>> I'm trying to add support to Linux 2.6.25 for the 
>> "has_foreign_mappings" MMU context flag.  Xen's Linux 2.6.18 tree uses 
>> this flag, so that page tables are properly disposed of when an 
>> application exits when it has foreign mappings.
> 
> I was hoping to avoid having to introduce that flag, but I have to admit 
> I haven't given it much analysis.  How are you using it?

A user-level application has a page table entry pointing to another domain's 
page.  My virtual memory area's zap_pte handler (which I added to 2.6.25, a la 
2.6.18) unmaps the grant.  But on application exit on 2.6.25, my zap_pte 
function runs too late.  There's a comment in gntdev.c that explains the need 
for has_foreign_mappings:
	/* This flag ensures that the page tables are not unpinned before the
	 * VM area is unmapped. Therefore Xen still recognises the PTE as
	 * belonging to an L1 pagetable, and the grant unmap operation will
	 * succeed, even if the process does not exit cleanly.
	 */

>>   See:
>> http://lists.xensource.com/archives/html/xen-devel/2006-08/msg00038.html
>>
>> Unfortunately, I got the following kernel crash on process exit:
>>
>> BUG: unable to handle kernel paging request at ebdae008
>> IP: [<c01157f9>] pgd_mop_up_pmds+0x6a/0xd8
> 
> Which line is that?

My original 2.6.25 kernel didn't have debugging support, preventing objdump -S 
from giving me address-to-source-line translations.  I rebuilt the kernel and 
here is the new stack dump:

BUG: unable to handle kernel paging request at ebb11008
IP: [<c0115b7b>] pgd_mop_up_pmds+0x7b/0xe6
*pdpt = 000000007f01a027
Oops: 0003 [#1] SMP
Modules linked in: efsvm(F) nfs lockd sunrpc dm_snapshot dm_mirror dm_mod

Pid: 2607, comm: a.out Tainted: GF        (2.6.25 #12)
EIP: 0061:[<c0115b7b>] EFLAGS: 00010246 CPU: 0
EIP is at pgd_mop_up_pmds+0x7b/0xe6
EAX: ebb11000 EBX: 00000000 ECX: 00000001 EDX: 00000000
ESI: 7edc3007 EDI: eb8533f4 EBP: ebaedf28 ESP: ebaedefc
  DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
Process a.out (pid: 2607, ti=ebaec000 task=ed3f4ed0 task.ti=ebaec000)
Stack: 2bb01007 00000000 00000000 00000000 ebb11000 00000001 00000000 2bb01007
        ebb11000 ed3f4ed0 eb8533f4 ebaedf34 c0115c44 eb8533c0 ebaedf40 c0121bad
        eb8533c0 ebaedf4c c0121c34 eb8533c0 ebaedf60 c01250f2 00000000 ed3f4ed0
Call Trace:
  [<c0115c44>] ? pgd_free+0xb/0x1e
  [<c0121bad>] ? __mmdrop+0x19/0x2f
  [<c0121c34>] ? mmput+0x71/0x74
  [<c01250f2>] ? exit_mm+0xd5/0xda
  [<c01263c4>] ? do_exit+0x1b3/0x56f
  [<c01267ed>] ? do_group_exit+0x6d/0x84
  [<c0126813>] ? sys_exit_group+0xf/0x11
  [<c0107012>] ? syscall_call+0x7/0xb
  =======================
Code: 00 00 00 89 55 dc 8b 45 dc 0b 45 d4 89 4d e0 09 c1 74 61 89 f0 89 da e8 8f 
d8 fe ff 90 89 45 f0 8b 4d e8 8b 45 e4 89 55 ec 89 da <c7> 04 c8 00 00 00 00 c7 
44 c8 04 00 00 00 00 89 f0 e8 6a d8 fe
EIP: [<c0115b7b>] pgd_mop_up_pmds+0x7b/0xe6 SS:ESP 0069:ebaedefc
---[ end trace b8f5f274f55408cd ]---
Fixing recursive fault but reboot is needed!


pgd_free calls pgd_mop_up_pmds, where the crash is occurring at c0115b7b. 
Here's the relevant assembly and source code:

			pmd_t *pmd = (pmd_t *)pgd_page_vaddr(pgd);

			pgdp[i] = native_make_pgd(0);
c0115b70:	8b 4d e8             	mov    -0x18(%ebp),%ecx
c0115b73:	8b 45 e4             	mov    -0x1c(%ebp),%eax
c0115b76:	89 55 ec             	mov    %edx,-0x14(%ebp)
c0115b79:	89 da                	mov    %ebx,%edx
c0115b7b:	c7 04 c8 00 00 00 00 	movl   $0x0,(%eax,%ecx,8)
c0115b82:	c7 44 c8 04 00 00 00 	movl   $0x0,0x4(%eax,%ecx,8)
c0115b89:	00
c0115b8a:	89 f0                	mov    %esi,%eax
c0115b8c:	ff 15 a0 5b 4c c0    	call   *0xc04c5ba0
{

Any hints?

Thanks,
Mike