Re: c/s 22402 ("86 hvm: Refuse to perform __hvm_copy() work in atomic context.") breaks HVM, race possible in other code - any ideas?

[Daniel De Graaf <dgdegra@tycho.nsa.gov>]

On 01/11/2011 01:00 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Jan 11, 2011 at 09:52:19AM -0500, Daniel De Graaf wrote:
>> On 01/11/2011 08:15 AM, Daniel De Graaf wrote:
>>> On 01/10/2011 05:41 PM, Konrad Rzeszutek Wilk wrote:
>>>>> @@ -284,8 +304,25 @@ static void unmap_grant_pages(struct grant_map *map, int offset, int pages)
>>>>>  		goto out;
>>>>>  
>>>>>  	for (i = 0; i < pages; i++) {
>>>>> +		uint32_t check, *tmp;
>>>>>  		WARN_ON(unmap_ops[i].status);
>>>>> -		__free_page(map->pages[offset+i]);
>>>>> +		if (!map->pages[i])
>>>>> +			continue;
>>>>> +		/* XXX When unmapping, Xen will sometimes end up mapping the GFN
>>>>> +		 * to an invalid MFN. In this case, writes will be discarded and
>>>>> +		 * reads will return all 0xFF bytes. Leak these unusable GFNs
>>>>> +		 * until a way to restore them is found.
>>>>> +		 */
>>>>> +		tmp = kmap(map->pages[i]);
>>>>> +		tmp[0] = 0xdeaddead;
>>>>> +		mb();
>>>>> +		check = tmp[0];
>>>>> +		kunmap(map->pages[i]);
>>>>> +		if (check == 0xdeaddead)
>>>>> +			__free_page(map->pages[i]);
>>>>> +		else if (debug)
>>>>> +			printk("%s: Discard page %d=%ld\n", __func__,
>>>>> +				i, page_to_pfn(map->pages[i]));
>>>>
>>>> Whoa. Any leads to when the "sometimes" happens? Does the status report an
>>>> error or is it silent?
>>>
>>> Status is silent in this case. I can produce it quite reliably on my
>>> test system where I am mapping a framebuffer (1280 pages) between two
>>> HVM guests - in this case, about 2/3 of the released pages will end up
>>> being invalid. It doesn't seem to be size-related as I have also seen
>>> it on the small 3-page page index mapping. There is a message on xm
>>> dmesg that may be related:
>>>
>>> (XEN) sh error: sh_remove_all_mappings(): can't find all mappings of mfn 7cbc6: c=8000000000000004 t=7400000000000002
>>>
>>> This appears about once per page, with different MFNs but the same c/t.
>>> One of the two HVM guests (the one doing the mapping) has the PCI
>>> graphics card forwarded to it.
>>>
>>
>> Just tested on the latest xen 4.1 (with 22402:7d2fdc083c9c reverted as
>> that breaks HVM grants), which produces different output:
> 
> Keir, the c/s 22402 has your name on it.
> 
> Any ideas on the problem that Daniel is hitting with unmapping grants?

c/s 22402 was discussed at the beginning of December:
http://lists.xensource.com/archives/html/xen-devel/2010-12/msg00063.html

Since I don't use xenpaging (which is the reason for the change), this
revert shouldn't be relevant to the problems I am seeing.

>> ...
>> (XEN) mm.c:889:d1 Error getting mfn b803e (pfn 25a3e) from L1 entry 00000000b803e021 for l1e_owner=1, pg_owner=1
>> (XEN) mm.c:889:d1 Error getting mfn b8038 (pfn 25a38) from L1 entry 00000000b8038021 for l1e_owner=1, pg_owner=1
>> (XEN) mm.c:889:d1 Error getting mfn b803d (pfn 25a3d) from L1 entry 00000000b803d021 for l1e_owner=1, pg_owner=1
>> (XEN) mm.c:889:d1 Error getting mfn 10829 (pfn 25a29) from L1 entry 0000000010829021 for l1e_owner=1, pg_owner=1
>> (XEN) mm.c:889:d1 Error getting mfn 1081c (pfn 25a1c) from L1 entry 000000001081c021 for l1e_owner=1, pg_owner=1
>> (XEN) mm.c:889:d1 Error getting mfn 10816 (pfn 25a16) from L1 entry 0000000010816021 for l1e_owner=1, pg_owner=1
>> (XEN) mm.c:889:d1 Error getting mfn 1081a (pfn 25a1a) from L1 entry 000000001081a021 for l1e_owner=1, pg_owner=1
>> ...
>>
>> This appears on the map; nothing is printed on the unmap. If the
>> unmap happens while the domain is up, it seems to be invalid more often;
>> most (perhaps all) of the destination-valid unmaps happen when the domain
>> is being destroyed. Exactly which pages are valid or invalid seems to be
>> mostly random, although nearby GFNs tend to have the same validity.
>>
>> If you have any thoughts as to the cause, I can test patches or provide
>> output as needed; it would be better if this workaround weren't needed.
>>
>> -- 
>> Daniel De Graaf
>> National Security Agency